I am using kubernetes 1.15.7 version.
I am trying to follow the link https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/#understanding-the-encryption-at-rest-configuration to enable 'encryption-provider-config' option on 'kube-apiserver'.
I edited file '/etc/kubernetes/manifests/kube-apiserver.yaml' and provided below option
- --encryption-provider-config=/home/rtonukun/secrets.yaml
But after that I am getting below error.
The connection to the server 171.69.225.87:6443 was refused - did you specify the right host or port?
with all kubectl commands like 'kubectl get no'.
Mainy, how do I do these below two steps?
3. Set the --encryption-provider-config flag on the kube-apiserver to point to the location of the config file.
4. Restart your API server.
I've reproduced exactly your scenario, and I'll try to explain how can I fixed it
/home/rabello/secrets.yaml
:apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
providers:
- aescbc:
keys:
- name: key1
secret: r48bixfj02BvhhnVktmJJiuxmQZp6c0R60ZQBFE7558=
- identity: {}
Edit the file /etc/kubernetes/manifests/kube-apiserver.yaml
and set the --encryption-provider-config
flag:
- --encryption-provider-config=/home/rabello/encryption.yaml
Save the file and exit.
When I checked the pods status got the same error:
$ kubectl get pods -A
The connection to the server 10.128.0.62:6443 was refused - did you specify the right host or port?
Since kubectl is not working anymore, I tried to look directly the running containers using docker command, then I see kube-apiserver container was recently recreated:
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
54203ea95e39 k8s.gcr.io/pause:3.1 "/pause" 1 minutes ago Up 1 minutes k8s_POD_kube-apiserver-lab-1_kube-system_015d9709c9881516d6ecf861945f6a10_0
...
Kubernetes store the logs of created pods on /var/log/pods
directory, I've checked the kube-apiserver log file and found a valuable information:
{"log":"Error: error opening encryption provider configuration file \"/home/rabello/encryption.yaml\": open /home/rabello/encryption.yaml: no such file or directory\n","stream":"stderr","time":"2020-01-22T13:28:46.772768108Z"}
Taking a look at manifest file kube-apiserver.yaml
is possible to see the command kube-apiserver
, it runs into container, so they need to have the encryption.yaml
file mounted into container.
If you check the volumeMounts
in this file, you could see that only the paths below is mounted in container by default:
- /etc/ssl/certs
- /etc/ca-certificates
- /etc/kubernetes/pki
- /usr/local/share/ca-certificates
- /usr/share/ca-certificates
...
volumeMounts:
- mountPath: /etc/ssl/certs
name: ca-certs
readOnly: true
- mountPath: /etc/ca-certificates
name: etc-ca-certificates
readOnly: true
- mountPath: /etc/kubernetes/pki
name: k8s-certs
readOnly: true
- mountPath: /usr/local/share/ca-certificates
name: usr-local-share-ca-certificates
readOnly: true
- mountPath: /usr/share/ca-certificates
name: usr-share-ca-certificates
readOnly: true
...
Based on the facts above, we can assume that apiserver failed to start because /home/rabello/encryption.yaml
doesn't actually mounted into container.
I can see 2 ways to solve this issue:
1st - Copy the encryption file to /etc/kubernetes/pki
(or any of the path above) and change the path in /etc/kubernetes/kube-apiserver.yaml
:
- --encryption-provider-config=/etc/kubernetes/encryption.yaml
Save the file and wait apiserver restart.
2nd - Create a new volumeMounts
in the kube-apiserver.yaml manifest to mount a custom directory from node into container.
Let's create a new directory in /etc/kubernetes/secret
(home folder isn't a good location to leave config files =)).
Edit /etc/kubernetes/manifests/kube-apiserver.yaml
:
...
- --encryption-provider-config=/etc/kubernetes/secret/encryption.yaml
...
volumeMounts:
- mountPath: /etc/kubernetes/secret
name: secret
readOnly: true
...
volumes:
- hostPath:
path: /etc/kubernetes/secret
type: DirectoryOrCreate
name: secret
...
After save the file kubernetes will mount the node path /etc/kubernetes/secret
into the same path into the apiserver container, wait start completely and try to list your node again.
Please let know if that helped!