Missing kubernetes default API Resource Definitions

2/4/2020
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.2", GitCommit:"59603c6e503c87169aea6106f57b9f242f64df89", GitTreeState:"clean", BuildDate:"2020-01-21T22:17:28Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.6", GitCommit:"7015f71e75f670eb9e7ebd4b5749639d42e20079", GitTreeState:"clean", BuildDate:"2019-11-13T11:11:50Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

In the process of trying to get cert-manager v0.13 running I ran into the following errors:

unable to recognize "reproduce.yaml": no matches for kind "MutatingWebhookConfiguration" in version "admissionregistration.k8s.io/v1beta1"
unable to recognize "reproduce.yaml": no matches for kind "ValidatingWebhookConfiguration" in version "admissionregistration.k8s.io/v1beta1"

Running kubectl api-resources yields the resource definitions as described here except MutatingWebhookConfiguration, ValidatingWebhookConfiguration, and CertificateSigningRequest.

I have run through my shell history and don't see a line where I would have deleted those resources (if that is at all possible). What is going on here? Is there a way to re-create those definitions and would all internal services relying on them gracefully recover?

The reproduce.yaml:

---
apiVersion: admissionregistration.k8s.io/v1beta1
kind: MutatingWebhookConfiguration
metadata:
  annotations:
    cert-manager.io/inject-ca-from-secret: logs/cert-manager-webhook-tls
  labels:
    app: webhook
    app.kubernetes.io/instance: cert-manager
    app.kubernetes.io/managed-by: skaffold-v1.3.1
    app.kubernetes.io/name: webhook
    skaffold.dev/builder: local
    skaffold.dev/cleanup: "true"
    skaffold.dev/deployer: kustomize
    skaffold.dev/docker-api-version: "1.40"
    skaffold.dev/run-id: a4c99b76-7a04-41c3-b1c7-ea74f8b2ec83
    skaffold.dev/tag-policy: git-commit
    skaffold.dev/tail: "true"
  name: cert-manager-webhook
webhooks:
- clientConfig:
    service:
      name: cert-manager-webhook
      namespace: cert-manager
      path: /mutate
  failurePolicy: Fail
  name: webhook.cert-manager.io
  rules:
  - apiGroups:
    - cert-manager.io
    - acme.cert-manager.io
    apiVersions:
    - v1alpha2
    operations:
    - CREATE
    - UPDATE
    resources:
    - '*/*'
  sideEffects: None
---
apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingWebhookConfiguration
metadata:
  annotations:
    cert-manager.io/inject-ca-from-secret: logs/cert-manager-webhook-tls
  labels:
    app: webhook
    app.kubernetes.io/instance: cert-manager
    app.kubernetes.io/managed-by: skaffold-v1.3.1
    app.kubernetes.io/name: webhook
    skaffold.dev/builder: local
    skaffold.dev/cleanup: "true"
    skaffold.dev/deployer: kustomize
    skaffold.dev/docker-api-version: "1.40"
    skaffold.dev/run-id: a4c99b76-7a04-41c3-b1c7-ea74f8b2ec83
    skaffold.dev/tag-policy: git-commit
    skaffold.dev/tail: "true"
  name: cert-manager-webhook
webhooks:
- clientConfig:
    service:
      name: cert-manager-webhook
      namespace: cert-manager
      path: /mutate
  failurePolicy: Fail
  name: webhook.cert-manager.io
  namespaceSelector:
    matchExpressions:
    - key: cert-manager.io/disable-validation
      operator: NotIn
      values:
      - "true"
    - key: name
      operator: NotIn
      values:
      - logs
  rules:
  - apiGroups:
    - cert-manager.io
    - acme.cert-manager.io
    apiVersions:
    - v1alpha2
    operations:
    - CREATE
    - UPDATE
    resources:
    - '*/*'
  sideEffects: None

EDIT: As per the comments, yes both webhooks are enabled through the --enable-admission-plugins flag. Full kube-apiserver as found in ps:

/usr/local/bin/kube-apiserver \
  --allow-privileged=true \
  --anonymous-auth=false \
  --apiserver-count=3 \
  --authorization-mode=RBAC \
  --basic-auth-file=/srv/kubernetes/basic_auth.csv \
  --bind-address=0.0.0.0 \
  --client-ca-file=/srv/kubernetes/ca.crt \
  --cloud-provider=aws \
  --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,NodeRestriction,ResourceQuota \
  --etcd-cafile=/etc/kubernetes/pki/kube-apiserver/etcd-ca.crt \
  --etcd-certfile=/etc/kubernetes/pki/kube-apiserver/etcd-client.crt \
  --etcd-keyfile=/etc/kubernetes/pki/kube-apiserver/etcd-client.key \
  --etcd-servers-overrides=/events#https://127.0.0.1:4002 \
  --etcd-servers=https://127.0.0.1:4001 \
  --insecure-bind-address=127.0.0.1 \
  --insecure-port=8080 \
  --kubelet-client-certificate=/srv/kubernetes/kubelet-api.pem \
  --kubelet-client-key=/srv/kubernetes/kubelet-api-key.pem \
  --kubelet-preferred-address-types=InternalIP,Hostname,ExternalIP \
  --oidc-client-id=..... \
  --oidc-issuer-url=..... \
  --proxy-client-cert-file=/srv/kubernetes/apiserver-aggregator.cert \
  --proxy-client-key-file=/srv/kubernetes/apiserver-aggregator.key \
  --requestheader-allowed-names=aggregator \
  --requestheader-client-ca-file=/srv/kubernetes/apiserver-aggregator-ca.cert \
  --requestheader-extra-headers-prefix=X-Remote-Extra- \
  --requestheader-group-headers=X-Remote-Group \
  --requestheader-username-headers=X-Remote-User \
  --secure-port=443 \
  --service-cluster-ip-range=..... \
  --storage-backend=etcd3 \
  --tls-cert-file=/srv/kubernetes/server.cert \
  --tls-private-key-file=/srv/kubernetes/server.key \
  --token-auth-file=/srv/kubernetes/known_tokens.csv \
  --v=2 \
  --logtostderr=false \
  --alsologtostderr \
  --log-file=/var/log/kube-apiserver.log
-- andsens
kubernetes

1 Answer

2/4/2020

I saw that kops had a minor version update ready and decided that this would be the best way to "reboot" the cluster. Lo and behold, all three resources are once again present after the upgrade.

I do not have the foggiest how this could have happened, and it doesn't seem like anybody else has ever experienced this issue. Many thanks to @DT. for providing some great troubleshooting pointers.

-- andsens
Source: StackOverflow