I'm trying to setup and expose a service (ArgoCD) to outside a cluster. Note: I'm fairly new to Kubernetes, so quite probably I have some misconceptions. If you can see one, please help me get rid of it. If more information is needed to diagnose what's happening, please let me know, I'll add it.
I have nginx-ingress ingress controller installed in the cluster in the namespace nginx
. I have installed ArgoCD via helm into argocd
namespace*. kubectl get service -n argocd
shows (omitting AGE column):
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
projectname-argocd-application-controller ClusterIP 10.100.249.133 <none> 8082/TCP
projectname-argocd-dex-server ClusterIP 10.100.80.187 <none> 5556/TCP,5557/TCP
projectname-argocd-redis ClusterIP 10.100.230.170 <none> 6379/TCP
projectname-argocd-repo-server ClusterIP 10.100.221.87 <none> 8081/TCP
projectname-argocd-server ClusterIP 10.100.22.26 <none> 80/TCP,443/TCP
As far as I understand, service projectname-argocd-server
is the one I should expose to get ArgoCD WebUI. Trying to do so, I've created an ingress (based on docs):
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-routing
spec:
rules:
- host: test2.projectname.org
http:
paths:
- path: /
pathType: Prefix # presumably may comment this out
backend:
service:
name: projectname-argocd-server
port:
number: 80
# this was added later while trying to figure the problem out
defaultBackend:
service:
name: projectname-argocd-server
port:
number: 80
ingressClassName: nginx
and applied it via kubectl apply -f routing.yaml -n argocd
. Now I can see the ingress is created along with the one created by deployment of ArgoCD, and the output of kubectl get ing -A
is (omitting AGE, and PORTS that are 80; <url>
is url of LoadBalancer shown in AWS console):
NAMESPACE NAME CLASS HOSTS ADDRESS
argocd projectname-argocd-server nginx test.projectname.org <url>
argocd ingress-routing nginx test2.projectname.org <url>
By the way, kubectl get svc -n nginx
shows that nginx-ingress-ingress-nginx-controller
is LoadBalancer with url <url>
(80:30538/TCP
).
kubectl describe ingress -n argocd
shows that ingress ingress-routing
is ok, with correct address, default backend and rules; for ingress projectname-argocd-server
it shows ok address and rules (path /
), although Default backend
is shown as default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
.
Now let me also show the DNS settings to complete the picture:
test.projectname.org
to <url>
test2.projectname.org
, selected the load balancer from the list and so it points to dualstack.<url>
I expected to see ArgoCD interface at least at one of http://test.projectname.org/
and http://test2.projectname.org/
. What actually happens is:
when I open http://test.projectname.org/
, it redirects me to https url and shows NET::ERR_CERT_AUTHORITY_INVALID
. If I insist on visiting, browser shows ERR_TOO_MANY_REDIRECTS
.
Before I added ingress class and moved ingress-routing
from nginx
namespace to argocd
namespace, http://test2.projectname.org/
gave me 404; now it also redirects to https and then gives ERR_TOO_MANY_REDIRECTS
I've also checked the /healthz
addresses but they give the same result as the /
ones. (in contrast, http://<url>/healthz
gives an empty page)
My question is: what else am I missing, why I don't get the UI?
Is it impossible to expose a service before setting some SSL certificate? Can 2 ingresses conflict when trying to expose the same thing on different subdomains (test.projectname.org and test2.projectname.org)? Can I see at least one service (ArgoCD) without using projectname.org to check if it is configured and deployed properly? (to separate if it's an ingress/routing/dns issue or a configuration issue)
(*) Here's the chart that I used to install ArgoCD:
apiVersion: v2
name: argo-cd
appVersion: v2.1.5
description: A declarative, GitOps continuous delivery tool for Kubernetes
version: 3.26.3
dependencies:
- name: argo-cd
version: 3.26.3
repository: https://argoproj.github.io/argo-helm
and values-overwrite.yaml that I've used is just default values wrapped into argo-cd:
thing since these should be applied to the dependency. Notably, those have enabled: false
in ingress:
, so the fact that ingress projectname-argocd-server
is created is somewhat unexpected.
PS the nginx
IngressClass was generated, not created manually, so it may be useful to see it as well (I've substituted ids and timestamps with "..."), as shown by kubectl get IngressClass nginx -o yaml
:
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
annotations:
meta.helm.sh/release-name: nginx-ingress
meta.helm.sh/release-namespace: nginx
creationTimestamp: ...
generation: 1
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: nginx-ingress
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/version: 1.0.3
helm.sh/chart: ingress-nginx-4.0.5
name: nginx
resourceVersion: "5750"
uid: ...
spec:
controller: k8s.io/ingress-nginx
Right, the issue was somewhat complicated, but I've figured it out. Basically, it consists of 2 problems:
The main problem about https configuration was solved in a separate question and is reduced to switching ACME server from staging to production. I've provided more details it in my answer.
Now, the ingress configuration is somewhat tricky since ArgoCD has some redirections, ~inner TLS requirements~, and also serves more than one protocol at :443. Fortunately, I've found this tutorial which shows ssl-passthrough settings, more ingress annotations, including nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
which fixes the ERR_TOO_MANY_REDIRECTS
error. Here's my ingress config which works fine with https set up (note also changes in port and tls secret):
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-argocd-routing
namespace: argocd
annotations:
cert-manager.io/cluster-issuer: <cluster issuer name>
kubernetes.io/tls-acme: "true"
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
ingressClassName: nginx
tls:
- hosts:
- test2.projectname.org # switched to argocd. later
secretName: argocd-secret # do not change, this is provided by Argo CD
rules:
- host: test2.projectname.org # switched to argocd. later
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: projectname-argocd-server
port:
number: 443