We are using the Azure Gateway Ingress Controller (AGIC) to automatically create listeners and back-ends on an app gateway for ingresses in our AKS cluster
ArgoCD is deployed to the K8s cluster to create applications. When ArgoCD creates an app, it pulls a helm chart from a git repo created for that instance of our app, and creates the app
The app is created with a Persistent Volume Claim to an Azure Storage File folder to store user data. It also gets an Ingress for the app that is labelled so that AGIC creates it in the App Gateway.
When everything works, all is well. I can access my argocd on one hostname, and each of my deployed apps on their hostnames - all through the App Gateway that is being maintained by AGIC
When one of my pods fails to start (because the storage key used by the PVC is incorrect), then AGIC updates the app gateway to remove my argoCD backend, which still works correctly.
AGIC deletes my working ARGOCD back-end.
If I delete the failed pod, AGIC deploys my HTTP back-end for ArgoCD again on the app gateway.
There appears to be a bug in AGIC where when some back-ends are resolved, and some are not, as soon as the first back-end in the list is unresolved, the rest of the backends are not created.
I have logged the following issue in Github to get it fixed: https://github.com/Azure/application-gateway-kubernetes-ingress/issues/1054
I found this by setting the logging parameter for AGIC to level 5, reviewing the logs and matching up the log messages to the AGIC source code in that repo.