I'm having trouble with the final step in creating user certificates for use with NodeJS pg client to access a secure CockroachDB running on GKE. (it all works fine using the root user client key and cert but I don't want to use root for pg access).

The container's /cockroach-certs directory looks like this

ca.crt  client.root.crt  client.root.key

and

kubectl exec -it cockroachdb-client-secure -- ./cockroach  --certs-dir=/cockroach-certs cert list

shows

+-----------------------+------------------+-----------------+------------+--------------+-------+
|         Usage         | Certificate File |    Key File     |  Expires   |    Notes     | Error |
+-----------------------+------------------+-----------------+------------+--------------+-------+
| Certificate Authority | ca.crt           |                 | 2023/09/03 | num certs: 1 |       |
| Client                | client.root.crt  | client.root.key | 2023/09/03 | user: root   |       |
+-----------------------+------------------+-----------------+------------+--------------+-------+

I used cockroach's client-secure.yaml (https://github.com/cockroachdb/cockroach/blob/master/cloud/kubernetes/client-secure.yaml) - the same one used to set up the CSR for root - but changed the username to xyz (who has been added as a user to the DB). This successfully generated a CSR for xyz, which I then approved.

default.client.xyz                               8m        system:serviceaccount:default:cockroachdb   Approved,Issued

and which created the expected secret, from which I can export both a client key and cert.

default.client.xyz                Opaque                                2         9m

The problem is that cert list doesn't show client.xyz.key or client.xyz.crt and they are not in the /cockroach-certs directory. If I extract them from the default.client.xyz secret and copy them there manually, they show up in cert list but not assigned to a specific user.

The cockroach docs use "cockroach cert" to create a user, but do not show the specific process when using kubernetes. So I'm missing this last piece of the puzzle - why does client-secure.yaml work through the whole process with -user=root but miss the final step with -user=xyz, and what is the step I am missing?

.... Added more information The container claims to have written the cert files, but they are actually not there.

$ kubectl logs fidserver-csr  -c init-certs

+ /request-cert '-namespace=default' '-certs-dir=/cockroach-certs' '-type=client' '-user=fidserver' '-symlink-ca-from=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt'
2018/09/07 15:32:11 Looking up cert and key under secret default.client.fidserver
W0907 15:32:11.700733       1 client_config.go:529] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2018/09/07 15:32:11 Writing cert and key to local files
wrote key file: /cockroach-certs/client.fidserver.key
wrote certificate file: /cockroach-certs/client.fidserver.crt
symlinked CA certificate file: /cockroach-certs/ca.crt -> /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

.... Updated to try again in a different certs directory - no files were actually written even though log claims they were. The symlink didn't happen either.

+ /request-cert '-namespace=default' '-certs-dir=/cockroach-client-certs' '-type=client' '-user=fidserver' '-symlink-ca-from=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt'
2018/09/08 09:35:56 Looking up cert and key under secret default.client.fidserver
W0908 09:35:56.160525       1 client_config.go:529] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2018/09/08 09:35:56 Secret default.client.fidserver not found, sending CSR
Sending create request: default.client.fidserver for 
Request sent, waiting for approval. To approve, run 'kubectl certificate approve default.client.fidserver'
2018-09-08 09:36:26.718183601 +0000 UTC m=+30.564697651: waiting for 'kubectl certificate approve default.client.fidserver'
2018-09-08 09:36:56.718422397 +0000 UTC m=+60.564936446: waiting for 'kubectl certificate approve default.client.fidserver'
2018-09-08 09:37:26.718657743 +0000 UTC m=+90.565171864: waiting for 'kubectl certificate approve default.client.fidserver'
2018-09-08 09:37:56.718959817 +0000 UTC m=+120.565473905: waiting for 'kubectl certificate approve default.client.fidserver'
CSR approved, but no certificate in response. Waiting some more
request default.client.fidserver Approved at 2018-09-08 09:38:00 +0000 UTC
  reason:   KubectlApprove
  message:  This CSR was approved by kubectl certificate approve.
2018/09/08 09:38:00 Storing cert and key under secret default.client.fidserver
2018/09/08 09:38:01 Writing cert and key to local files
wrote key file: /cockroach-client-certs/client.fidserver.key
wrote certificate file: /cockroach-client-certs/client.fidserver.crt
symlinked CA certificate file: /cockroach-client-certs/ca.crt -> /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

Now logged as an issue with cockroach. https://github.com/cockroachdb/cockroach/issues/29968