AGIC deletes a working backend when a different pod fails to start

11/9/2020

Deployment overview

We are using the Azure Gateway Ingress Controller (AGIC) to automatically create listeners and back-ends on an app gateway for ingresses in our AKS cluster

ArgoCD is deployed to the K8s cluster to create applications. When ArgoCD creates an app, it pulls a helm chart from a git repo created for that instance of our app, and creates the app

The app is created with a Persistent Volume Claim to an Azure Storage File folder to store user data. It also gets an Ingress for the app that is labelled so that AGIC creates it in the App Gateway.

When everything works, all is well. I can access my argocd on one hostname, and each of my deployed apps on their hostnames - all through the App Gateway that is being maintained by AGIC

Problem description

When one of my pods fails to start (because the storage key used by the PVC is incorrect), then AGIC updates the app gateway to remove my argoCD backend, which still works correctly.

AGIC deletes my working ARGOCD back-end.

If I delete the failed pod, AGIC deploys my HTTP back-end for ArgoCD again on the app gateway.

Questions:

  1. How can I troubleshoot why AGIC removes the ArgoCD back-end? Is there a log I can enable that will tell me in detail how it is making deployment decisions?
  2. Is there anything I can do on AKS to try and separate the ArgoCD from the pods so that AGIC doesn't remove the back-end for ArgoCD when a pod is broken? (they are already deployed in different namespaces)
-- Joon
argocd
azure-aks
azure-application-gateway
kubernetes

1 Answer

11/10/2020

There appears to be a bug in AGIC where when some back-ends are resolved, and some are not, as soon as the first back-end in the list is unresolved, the rest of the backends are not created.

I have logged the following issue in Github to get it fixed: https://github.com/Azure/application-gateway-kubernetes-ingress/issues/1054

I found this by setting the logging parameter for AGIC to level 5, reviewing the logs and matching up the log messages to the AGIC source code in that repo.

-- Joon
Source: StackOverflow