Can't install autoscaler on AWS now?

1/25/2018

When install autoscaler on AWS as:

https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler/cloudprovider/aws

Got error:

cluster-autoscaler-5f69cdcd84-4kpqw  0/1  RunContainerError   0   3s

See detail:

$ kubectl describe po cluster-autoscaler-5b454d874c-4f85w -n kube-system
...
Last State:  Terminated
      Reason:    ContainerCannotRun
      Message:   oci runtime error: container_linux.go:247: starting container process caused "process_linux.go:359: container init caused \"rootfs_linux.go:54: mounting \\\"/etc/ssl/certs/ca-certificates.crt\\\" to rootfs \\\"/var/lib/docker/overlay/f45f8b9b739167c3b6bb5
275c7ca6285508b52ecf940b3759e3ca99b87fadd53/merged\\\" at \\\"/var/lib/docker/overlay/f45f8b9b739167c3b6bb5275c7ca6285508b52ecf940b3759e3ca99b87fadd53/merged/etc/ssl/certs/ca-certificates.crt\\\" caused \\\"not a directory\\\"\""
: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type
...
Events:
  Type     Reason                 Age   From                                                      Message
  ----     ------                 ----  ----                                                      -------
  Normal   Scheduled              55s   default-scheduler                                         Successfully assigned cluster-autoscaler-5b454d874c-4f85w to ip-100.200.0.1.ap-northeast-1.compute.internal
  Normal   SuccessfulMountVolume  55s   kubelet, ip-100.200.0.1.ap-northeast-1.compute.internal  MountVolume.SetUp succeeded for volume "ssl-certs"
  Normal   SuccessfulMountVolume  55s   kubelet, ip-100.200.0.1.ap-northeast-1.compute.internal  MountVolume.SetUp succeeded for volume "default-token-2wmct"
  Warning  Failed                 53s   kubelet, ip-100.200.0.1.ap-northeast-1.compute.internal  Error: failed to start container "cluster-autoscaler": Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "process_linux
.go:359: container init caused \"rootfs_linux.go:54: mounting \\\"/etc/ssl/certs/ca-certificates.crt\\\" to rootfs \\\"/var/lib/docker/overlay/3796432b43abb86f70886e31d3bc555bd6beb54a2854d1e09ee6cdc74cab3af3/merged\\\" at \\\"/var/lib/docker/overlay/3796432b43abb86f70886e
31d3bc555bd6beb54a2854d1e09ee6cdc74cab3af3/merged/etc/ssl/certs/ca-certificates.crt\\\" caused \\\"not a directory\\\"\""
: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type
  Warning  Failed  51s  kubelet, ip-100.200.0.1.ap-northeast-1.compute.internal  Error: failed to start container "cluster-autoscaler": Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "process_linux.go:359: contain
er init caused \"rootfs_linux.go:54: mounting \\\"/etc/ssl/certs/ca-certificates.crt\\\" to rootfs \\\"/var/lib/docker/overlay/2c1fac03d81e1e77df060a70035adf2442840705198e5c887825bc3b1eb80f8f/merged\\\" at \\\"/var/lib/docker/overlay/2c1fac03d81e1e77df060a70035adf24428407
05198e5c887825bc3b1eb80f8f/merged/etc/ssl/certs/ca-certificates.crt\\\" caused \\\"not a directory\\\"\""
: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type
  Warning  Failed  33s  kubelet, ip-100.200.0.1.ap-northeast-1.compute.internal  Error: failed to start container "cluster-autoscaler": Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "process_linux.go:359: contain
er init caused \"rootfs_linux.go:54: mounting \\\"/etc/ssl/certs/ca-certificates.crt\\\" to rootfs \\\"/var/lib/docker/overlay/f45f8b9b739167c3b6bb5275c7ca6285508b52ecf940b3759e3ca99b87fadd53/merged\\\" at \\\"/var/lib/docker/overlay/f45f8b9b739167c3b6bb5275c7ca6285508b52
ecf940b3759e3ca99b87fadd53/merged/etc/ssl/certs/ca-certificates.crt\\\" caused \\\"not a directory\\\"\""
: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type
  Warning  BackOff     22s (x2 over 47s)  kubelet, ip-100.200.0.1.ap-northeast-1.compute.internal  Back-off restarting failed container
  Normal   Pulling     8s (x4 over 55s)   kubelet, ip-100.200.0.1.ap-northeast-1.compute.internal  pulling image "k8s.gcr.io/cluster-autoscaler:v0.6.0"
  Normal   Created     7s (x4 over 53s)   kubelet, ip-100.200.0.1.ap-northeast-1.compute.internal  Created container
  Warning  FailedSync  7s (x6 over 53s)   kubelet, ip-100.200.0.1.ap-northeast-1.compute.internal  Error syncing pod
  Normal   Pulled      7s (x4 over 53s)   kubelet, ip-100.200.0.1.ap-northeast-1.compute.internal  Successfully pulled image "k8s.gcr.io/cluster-autoscaler:v0.6.0"

Is it the image k8s.gcr.io/cluster-autoscaler:v0.6.0 issue?

-- online
amazon-web-services
autoscaling
containers
image
kubernetes

3 Answers

1/25/2018

What version of Kubernetes are you running on AWS? Please refer to the compatibility matrix. You also may find my pull request to Kops helpful if you are running Kubernetes 1.8+

-- SteveCoffman
Source: StackOverflow

7/14/2018

On AWS EKS (Elastic Kubernetes Service), the sslCertPath required by the cluster-autoscaler seems to be indeed /etc/ssl/certs/ca-bundle.crt

Exmple:

helm install stable/cluster-autoscaler 
  --set "autoscalingGroups[0].name=myasgname-worker-nodes-3-NodeGroup-HHTVNI2VF9DF,autoscalingGroups[0].maxSize=10,autoscalingGroups[0].minSize=2" 
  --name cluster-autoscaler
  --namespace kube-system 
  --set rbac.create=true 
  --set sslCertPath=/etc/ssl/certs/ca-bundle.crt
-- Costi Muraru
Source: StackOverflow

1/25/2018

The issue appears to be with the CA certificate mount. It's saying Check if the specified host path exists and is the expected type.

The cluster-autoscaler image does not come with any CA certificates, so you have to mount them from the host into the container. Maybe the OS that you're using puts those certificates in a different location? Check if /etc/ssl/certs/ca-certificates.crt exists on your host, and if not find out what the correct path is for the CA certs on your host and update the volume: configuration of the deployment accordingly.

-- nerdherd
Source: StackOverflow