Can I replace the client-ca-file to invalidate all users on kubernetes?

6/5/2019

When setting up our cluster we use kubespray which creates an kubernetes-admin user for me. I believe the code is here.

For some reason now this admin.conf was leaked to all of our developers and I somehow need to revoke it.

What (I think) I understand:

In our kubernetes cluster we use x509 to authenticate our users. For our users we create a private key, then create a CSR with that key and sign it with the client-ca-file and key from our kubernetes installation. Like this:

openssl genrsa -out $K8S_USER-key.pem 2048
openssl req -new -key $K8S_USER-key.pem -out $K8S_USER.csr -subj "/CN=$K8S_USER"
openssl x509 -req -in $K8S_USER.csr -CA $cert_dir/ca.pem -CAkey $cert_dir/ca-key.pem -set_serial $SERIAL -out $K8S_USER.pem -days $DAYS

I assume the same was done for the kubernetes-admin user and I assume that when I change the client-ca-file the admin.conf cannot be used anymore to use the kubernetes API.

Is this correct? That changing the client-ca-file will invalidate the kubernetes-admin?

I also assume I need to recreate all my service accounts as they also will be invalidated.

EDIT: So, I spent some time creating a new CA cert, then issuing new certs for my user and the kube-apiserver. Not sure a restart of the apiserver pods was enough though. My user is being rejected as Unable to connect to the server: x509: issuer name does not match subject from issuing certificate. This doesn't make too much sense to me though. When I connect to the apiserver pod via exec and inspect the apiserver cert it has the same issuer as my kubeconfig user cert.

-- Moritz Schmitz v. Hülst
kubernetes
rbac
ssl
x509
x509certificate

1 Answer

6/11/2019

Changing the client-ca-file will not invalidate the kubernetes-admin.

Referring to your case:

During creating a config file for generating a Certificate Signing Request you need to (CSR substitute the values marked with angle brackets (e.g. <MASTER_IP>) with real values before saving this to a file (e.g. csr.conf). Value for MASTER_CLUSTER_IP is the service cluster IP for the API server. I assume that you are using cluster.local as the default DNS domain name.

Did you add the same parameters into the API server start parameters ?

Submit the CSR to the CA, just like you would with any CSR, but with the -selfsign option. This requires your CA directory structure to be prepared first, which you will have to do anyway if you want to set up your own CA. You can find an tutorial on that here, for example. Submitting the request can be done as follows:

ca -selfsign -keyfile dist/ca_key.pem -in ca_csr.pem -out dist/ca_cert.pem \
    -outdir root-ca/cert -config openssl/ca.cnf

A client node may refuse to recognize a self-signed CA certificate as valid. For a non-production deployment, or for a deployment that runs behind a company firewall, you can distribute a self-signed CA certificate to all clients and refresh the local list for valid certificates.

On each client, perform the following operations:

$ sudo cp ca.crt /usr/local/share/ca-certificates/kubernetes.crt

$ sudo update-ca-certificates

Updating certificates in /etc/ssl/certs...
1 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d....
done.

To make the leaked certificate useless is to replace the CA in the cluster. This would require a restart of the cluster, though. And it would require to re-issue all the certificates. You will have to recreate service account again.

To invalidate leaked tokens just delete the secret that corresponds to the user's token. Remember of certificate expiration date you have added.

I hope it helps.

-- MaggieO
Source: StackOverflow