I get an error in ml pipeline pod when building kubeflow.

cache-server

cache-deployer

ml-pipeline

minio

The following error log is taken from cache-deployer pod.

Start deploying cache service to existing cluster:
+ echo 'Start deploying cache service to existing cluster:'
+ NAMESPACE=kubeflow
+ MUTATING_WEBHOOK_CONFIGURATION_NAME=cache-webhook-kubeflow
+ WEBHOOK_SECRET_NAME=webhook-server-tls
+ mkdir -p /root/bin
+ export 'PATH=/root/bin:/google-cloud-sdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
+ kubectl version --output json
+ jq --raw-output '(.serverVersion.major + "." + .serverVersion.minor)'
+ tr -d '"+'
+ server_version_major_minor=1.20
+ curl -s https://storage.googleapis.com/kubernetes-release/release/stable-1.20.txt
+ stable_build_version=v1.20.8
+ kubectl_url=https://storage.googleapis.com/kubernetes-release/release/v1.20.8/bin/linux/amd64/kubectl
+ curl -L -o /root/bin/kubectl https://storage.googleapis.com/kubernetes-release/release/v1.20.8/bin/linux/amd64/kubectl
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 68 38.3M 68 26.3M 0 0 28.9M 0 0:00:01 --:--:-- 0:00:01 28.9M 100 38.3M 100 38.3M 0 0 32.1M 0 0:00:01 0:00:01 --:--:-- 32.1M
+ chmod +x /root/bin/kubectl
+ kubectl get mutatingwebhookconfigurations cache-webhook-kubeflow --namespace kubeflow --ignore-not-found
+ kubectl get secrets webhook-server-tls --namespace kubeflow --ignore-not-found
+ webhook_config_exists=false
+ grep cache-webhook-kubeflow -w
+ webhook_secret_exists=false
+ grep webhook-server-tls -w
+ '[' false '==' true ]
+ '[' false '==' true ]
+ '[' false '==' true ]
+ export 'CA_FILE=ca_cert'
+ rm -f ca_cert
+ touch ca_cert
+ ./webhook-create-signed-cert.sh --namespace kubeflow --cert_output_path ca_cert --secret webhook-server-tls
+ [[ 6 -gt 0 ]]
+ case ${1} in
+ namespace=kubeflow
+ shift
+ shift
+ [[ 4 -gt 0 ]]
+ case ${1} in
+ cert_output_path=ca_cert
+ shift
+ shift
+ [[ 2 -gt 0 ]]
+ case ${1} in
+ secret=webhook-server-tls
+ shift
+ shift
+ [[ 0 -gt 0 ]]
+ '[' -z ']'
+ service=cache-server
+ '[' -z webhook-server-tls ']'
+ '[' -z kubeflow ']'
+ '[' -z ca_cert ']'
++ command -v openssl
+ '[' '!' -x /usr/bin/openssl ']'
+ csrName=cache-server.kubeflow
++ mktemp -d
+ tmpdir=/tmp/tmp.cJCiPG
+ echo 'creating certs in tmpdir /tmp/tmp.cJCiPG '
+ cat
creating certs in tmpdir /tmp/tmp.cJCiPG
+ openssl genrsa -out /tmp/tmp.cJCiPG/server-key.pem 2048
Generating RSA private key, 2048 bit long modulus (2 primes)
..................................+++++
.......................+++++
e is 65537 (0x010001)
+ openssl req -new -key /tmp/tmp.cJCiPG/server-key.pem -subj /CN=cache-server.kubeflow.svc -out /tmp/tmp.cJCiPG/server.csr -config /tmp/tmp.cJCiPG/csr.conf
+ echo 'start running kubectl...'
start running kubectl...
+ kubectl delete csr cache-server.kubeflow
certificatesigningrequest.certificates.k8s.io "cache-server.kubeflow" deleted
+ cat
+ kubectl create -f -
++ cat /tmp/tmp.cJCiPG/server.csr
++ base64
++ tr -d '\n'
Warning: certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
certificatesigningrequest.certificates.k8s.io/cache-server.kubeflow created
+ true
+ kubectl get csr cache-server.kubeflow
NAME AGE SIGNERNAME REQUESTOR CONDITION
cache-server.kubeflow 0s kubernetes.io/legacy-unknown system:serviceaccount:kubeflow:kubeflow-pipelines-cache-deployer-sa Pending
+ '[' 0 -eq 0 ']'
+ break
+ kubectl certificate approve cache-server.kubeflow
certificatesigningrequest.certificates.k8s.io/cache-server.kubeflow approved
++ seq 10
+ for x in $(seq 10)
++ kubectl get csr cache-server.kubeflow -o 'jsonpath={.status.certificate}'
+ serverCert=
+ [[ '' != '' ]]
+ sleep 1
+ for x in $(seq 10)
++ kubectl get csr cache-server.kubeflow -o 'jsonpath={.status.certificate}'
+ serverCert=
+ [[ '' != '' ]]
+ sleep 1
+ for x in $(seq 10)
++ kubectl get csr cache-server.kubeflow -o 'jsonpath={.status.certificate}'
+ serverCert=
+ [[ '' != '' ]]
+ sleep 1
+ for x in $(seq 10)
++ kubectl get csr cache-server.kubeflow -o 'jsonpath={.status.certificate}'
+ serverCert=
+ [[ '' != '' ]]
+ sleep 1
+ for x in $(seq 10)
++ kubectl get csr cache-server.kubeflow -o 'jsonpath={.status.certificate}'
+ serverCert=
+ [[ '' != '' ]]
+ sleep 1
+ for x in $(seq 10)
++ kubectl get csr cache-server.kubeflow -o 'jsonpath={.status.certificate}'
+ serverCert=
+ [[ '' != '' ]]
+ sleep 1
+ for x in $(seq 10)
++ kubectl get csr cache-server.kubeflow -o 'jsonpath={.status.certificate}'
+ serverCert=
+ [[ '' != '' ]]
+ sleep 1
+ for x in $(seq 10)
++ kubectl get csr cache-server.kubeflow -o 'jsonpath={.status.certificate}'
+ serverCert=
+ [[ '' != '' ]]
+ sleep 1
+ for x in $(seq 10)
++ kubectl get csr cache-server.kubeflow -o 'jsonpath={.status.certificate}'
+ serverCert=
+ [[ '' != '' ]]
+ sleep 1
+ for x in $(seq 10)
++ kubectl get csr cache-server.kubeflow -o 'jsonpath={.status.certificate}'
+ serverCert=
+ [[ '' != '' ]]
+ sleep 1
+ [[ '' == '' ]]
+ echo 'ERROR: After approving csr cache-server.kubeflow, the signed certificate did not appear on the resource. Giving up after 10 attempts.'
ERROR: After approving csr cache-server.kubeflow, the signed certificate did not appear on the resource. Giving up after 10 attempts.
+ exit 1

If you look up the csr, you will see the following:

# kubectl get csr/cache-server.kubeflow -o 'json'
{
    "apiVersion": "certificates.k8s.io/v1",
    "kind": "CertificateSigningRequest",
    "metadata": {
        "creationTimestamp": "2021-06-23T05:31:33Z",
        "name": "cache-server.kubeflow",
        "resourceVersion": "856361",
        "uid": "a62a0d32-f34b-4342-9af0-9a6b6f1d1d33"
    },
    "spec": {
        "groups": [
            "system:serviceaccounts",
            "system:serviceaccounts:kubeflow",
            "system:authenticated"
        ],
        "request": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQzlEQ0NBZHdDQVFBd0pERWlNQ0FHQTFVRUF3d1pZMkZqYUdVdGMyVnlkbVZ5TG10MVltVm1iRzkzTG5OMgpZekNDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFEZ2dFUEFEQ0NBUW9DZ2dFQkFOb0duQ0xCaC9ZZkRKYUNuSEt4CnNiaW5vYTFjeVJBdHl1aWFPY3RjeTJWODdKeDhYYVlsODcwdHBtOHpxKzN6aVJrT3JOWk9qNC9Ja3QvOWlxd1MKYkw0RG5nbGNKSjljbWVVdXZMemFaN25XdEZwdERTSVNGZC9zMDV5RUt5R2N4WVdtdkRCSlFuK1NaSmtwcnpIVApWQWV4eGlTdVcyYjl1T0pXUW8zMWpFa2dJeHh3Sys1OFlneTc0TDM2SGFPUmJMbGd4bm4wZGR4b1NBOEQ5OFFyCjEwbEg0RFRmSTdsUG5tOWpBbWZtQnRTNHZFdGU0RUQvR1c0TElkQkFIVFdtVklGZHVpZWhXZTgxL2ZiTmZwaVoKWTBDazFia0dwcW9NT2h4NHhkeWZYaFNPK05TNkVXK3ljUm5hK3VxQ001M094aXlOTzYvVXlHdEo2YWUyS2hreQptWnNDQXdFQUFhQ0JpakNCaHdZSktvWklodmNOQVFrT01Yb3dlREFKQmdOVkhSTUVBakFBTUFzR0ExVWREd1FFCkF3SUY0REFUQmdOVkhTVUVEREFLQmdnckJnRUZCUWNEQVRCSkJnTlZIUkVFUWpCQWdneGpZV05vWlMxelpYSjIKWlhLQ0ZXTmhZMmhsTFhObGNuWmxjaTVyZFdKbFpteHZkNElaWTJGamFHVXRjMlZ5ZG1WeUxtdDFZbVZtYkc5MwpMbk4yWXpBTkJna3Foa2lHOXcwQkFRc0ZBQU9DQVFFQVdTRERwL2FCbHAzdUZIcDl2SUFxdmY1NkdFQnRhMDlJCnE4OEpXOEhxUWJJekJpbHVWSGRxSEhtYmZGcVJ0clpGQlpYS3RFcnFlaDU4UXR6M3BmYUxQUEJjQ1o4eHk0N0MKNHJhbjR5Zy80K2FWblhBb0w3QkJOTXRRVUhCUDlkYndIZEp3dWlGV3JXejNpL2FFaDFCQ0xwUHYzaDUrbklzbApDekczSVdISGdkZ08vSC9Uc1B2VlRQbGZGNk9hd1hJRC9CcG4zUjZ2bUZVZTNXSTRMYzNRMDlKOEhQNFlmMkpxCjRuOWd3NlAwTzRERkhTeUNXWGozYXRpZGI1ek9IUk5EcWZ4OVNGRU5jZU0wQTU4dFBJRHk2R0hMbWdWWUVBbi8KcjZ3TzFqblNRaEpQamRSWlhwcXRyZXYwNVZoM3dXYUV0dHkxSXEzSDdkbGM2eCttS1J0QVN3PT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUgUkVRVUVTVC0tLS0tCg==",
        "signerName": "kubernetes.io/legacy-unknown",
        "uid": "e317eb83-fafb-472a-b323-33db038edcd3",
        "usages": [
            "digital signature",
            "key encipherment",
            "server auth"
        ],
        "username": "system:serviceaccount:kubeflow:kubeflow-pipelines-cache-deployer-sa"
    },
    "status": {
        "conditions": [
            {
                "lastTransitionTime": "2021-06-23T05:31:33Z",
                "lastUpdateTime": "2021-06-23T05:31:33Z",
                "message": "This CSR was approved by kubectl certificate approve.",
                "reason": "KubectlApprove",
                "status": "True",
                "type": "Approved"
            }
        ]
    }
}

But when I enter it as a command in the pod, nothing comes up. I'm assuming this is the cause of the error. How can I solve this?

kubectl get csr cache-server.kubeflow -o 'jsonpath={.status.certificate}'