Unable to get ArgoCD working on EC2 running centos 7

9/24/2020

I am trying to run argocd on my EC2 instance running centos 7 by following official documentation and EKS workshop from AWS, but it is in pending state, all pods from kube-system namespace are running fine.

below is the output of kubectl get pods --all-namespaces

NAMESPACE     NAME                                                              READY   STATUS    RESTARTS   AGE
argocd        argocd-application-controller-5785f6b79-nvg7n                     0/1     Pending   0          29s
argocd        argocd-dex-server-7f5d7d6645-gprpd                                0/1     Pending   0          19h
argocd        argocd-redis-cccbb8f7-vb44n                                       0/1     Pending   0          19h
argocd        argocd-repo-server-67ddb49495-pnw5k                               0/1     Pending   0          19h
argocd        argocd-server-6bcbf7997d-jqqrw                                    0/1     Pending   0          19h
kube-system   calico-kube-controllers-56b44cd6d5-tzgdm                          1/1     Running   0          19h
kube-system   calico-node-4z9tx                                                 1/1     Running   0          19h
kube-system   coredns-f9fd979d6-8d6hm                                           1/1     Running   0          19h
kube-system   coredns-f9fd979d6-p9dq6                                           1/1     Running   0          19h
kube-system   etcd-ip-10-1-3-94.us-east-2.compute.internal                      1/1     Running   0          19h
kube-system   kube-apiserver-ip-10-1-3-94.us-east-2.compute.internal            1/1     Running   0          19h
kube-system   kube-controller-manager-ip-10-1-3-94.us-east-2.compute.internal   1/1     Running   0          19h
kube-system   kube-proxy-tkp7k                                                  1/1     Running   0          19h
kube-system   kube-scheduler-ip-10-1-3-94.us-east-2.compute.internal            1/1     Running   0          19h

While same configuration is working fine on my local mac, I've made sure that docker, kubernetes services are up and runnning. Tried deleting pods, reconfigured argocd, however everytime result remained same.

Being new to ArgoCD I am unable to figure out the reason for the same. Please let me know where I am going wrong. Thanks!

-- Shubham Namdeo
amazon-ec2
amazon-web-services
argocd
kubernetes
kubernetes-pod

1 Answer

9/24/2020

I figured out what the problem was by running:

kubectl describe pods <name>  -n argocd

It gave output ending with FailedScheduling:

...
Events:
  Type     Reason            Age                From               Message
  ----     ------            ----               ----               -------
  Warning  FailedScheduling  3m (x5 over 7m2s)  default-scheduler  0/1 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate.

henceforth, by referring this GitHub issue, I figured out to run:

kubectl taint nodes --all node-role.kubernetes.io/master-

After this command, pods started to work and transitioned from Pending state to Running with kubectl describe pods showing output as:

...
Events:
  Type     Reason            Age                From               Message
  ----     ------            ----               ----               -------
  Warning  FailedScheduling  3m (x5 over 7m2s)  default-scheduler  0/1 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate.
  Normal   Scheduled         106s               default-scheduler  Successfully assigned argocd/argocd-server-7d44dfbcc4-qfj6m to ip-XX-XX-XX-XX.<region>.compute.internal
  Normal   Pulling           105s               kubelet            Pulling image "argoproj/argocd:v1.7.6"
  Normal   Pulled            81s                kubelet            Successfully pulled image "argoproj/argocd:v1.7.6" in 23.779457251s
  Normal   Created           72s                kubelet            Created container argocd-server
  Normal   Started           72s                kubelet            Started container argocd-server

From this error and resolution I've learned to always use kubectl describe pods to resolve the errors.

-- Shubham Namdeo
Source: StackOverflow