$ kubectl version
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.2", GitCommit:"59603c6e503c87169aea6106f57b9f242f64df89", GitTreeState:"clean", BuildDate:"2020-01-21T22:17:28Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.6", GitCommit:"7015f71e75f670eb9e7ebd4b5749639d42e20079", GitTreeState:"clean", BuildDate:"2019-11-13T11:11:50Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
In the process of trying to get cert-manager v0.13 running I ran into the following errors:
unable to recognize "reproduce.yaml": no matches for kind "MutatingWebhookConfiguration" in version "admissionregistration.k8s.io/v1beta1"
unable to recognize "reproduce.yaml": no matches for kind "ValidatingWebhookConfiguration" in version "admissionregistration.k8s.io/v1beta1"
Running kubectl api-resources
yields the resource definitions as described here except MutatingWebhookConfiguration
, ValidatingWebhookConfiguration
, and CertificateSigningRequest
.
I have run through my shell history and don't see a line where I would have deleted those resources (if that is at all possible). What is going on here? Is there a way to re-create those definitions and would all internal services relying on them gracefully recover?
The reproduce.yaml:
---
apiVersion: admissionregistration.k8s.io/v1beta1
kind: MutatingWebhookConfiguration
metadata:
annotations:
cert-manager.io/inject-ca-from-secret: logs/cert-manager-webhook-tls
labels:
app: webhook
app.kubernetes.io/instance: cert-manager
app.kubernetes.io/managed-by: skaffold-v1.3.1
app.kubernetes.io/name: webhook
skaffold.dev/builder: local
skaffold.dev/cleanup: "true"
skaffold.dev/deployer: kustomize
skaffold.dev/docker-api-version: "1.40"
skaffold.dev/run-id: a4c99b76-7a04-41c3-b1c7-ea74f8b2ec83
skaffold.dev/tag-policy: git-commit
skaffold.dev/tail: "true"
name: cert-manager-webhook
webhooks:
- clientConfig:
service:
name: cert-manager-webhook
namespace: cert-manager
path: /mutate
failurePolicy: Fail
name: webhook.cert-manager.io
rules:
- apiGroups:
- cert-manager.io
- acme.cert-manager.io
apiVersions:
- v1alpha2
operations:
- CREATE
- UPDATE
resources:
- '*/*'
sideEffects: None
---
apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingWebhookConfiguration
metadata:
annotations:
cert-manager.io/inject-ca-from-secret: logs/cert-manager-webhook-tls
labels:
app: webhook
app.kubernetes.io/instance: cert-manager
app.kubernetes.io/managed-by: skaffold-v1.3.1
app.kubernetes.io/name: webhook
skaffold.dev/builder: local
skaffold.dev/cleanup: "true"
skaffold.dev/deployer: kustomize
skaffold.dev/docker-api-version: "1.40"
skaffold.dev/run-id: a4c99b76-7a04-41c3-b1c7-ea74f8b2ec83
skaffold.dev/tag-policy: git-commit
skaffold.dev/tail: "true"
name: cert-manager-webhook
webhooks:
- clientConfig:
service:
name: cert-manager-webhook
namespace: cert-manager
path: /mutate
failurePolicy: Fail
name: webhook.cert-manager.io
namespaceSelector:
matchExpressions:
- key: cert-manager.io/disable-validation
operator: NotIn
values:
- "true"
- key: name
operator: NotIn
values:
- logs
rules:
- apiGroups:
- cert-manager.io
- acme.cert-manager.io
apiVersions:
- v1alpha2
operations:
- CREATE
- UPDATE
resources:
- '*/*'
sideEffects: None
EDIT: As per the comments, yes both webhooks are enabled through the --enable-admission-plugins
flag. Full kube-apiserver as found in ps
:
/usr/local/bin/kube-apiserver \
--allow-privileged=true \
--anonymous-auth=false \
--apiserver-count=3 \
--authorization-mode=RBAC \
--basic-auth-file=/srv/kubernetes/basic_auth.csv \
--bind-address=0.0.0.0 \
--client-ca-file=/srv/kubernetes/ca.crt \
--cloud-provider=aws \
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,NodeRestriction,ResourceQuota \
--etcd-cafile=/etc/kubernetes/pki/kube-apiserver/etcd-ca.crt \
--etcd-certfile=/etc/kubernetes/pki/kube-apiserver/etcd-client.crt \
--etcd-keyfile=/etc/kubernetes/pki/kube-apiserver/etcd-client.key \
--etcd-servers-overrides=/events#https://127.0.0.1:4002 \
--etcd-servers=https://127.0.0.1:4001 \
--insecure-bind-address=127.0.0.1 \
--insecure-port=8080 \
--kubelet-client-certificate=/srv/kubernetes/kubelet-api.pem \
--kubelet-client-key=/srv/kubernetes/kubelet-api-key.pem \
--kubelet-preferred-address-types=InternalIP,Hostname,ExternalIP \
--oidc-client-id=..... \
--oidc-issuer-url=..... \
--proxy-client-cert-file=/srv/kubernetes/apiserver-aggregator.cert \
--proxy-client-key-file=/srv/kubernetes/apiserver-aggregator.key \
--requestheader-allowed-names=aggregator \
--requestheader-client-ca-file=/srv/kubernetes/apiserver-aggregator-ca.cert \
--requestheader-extra-headers-prefix=X-Remote-Extra- \
--requestheader-group-headers=X-Remote-Group \
--requestheader-username-headers=X-Remote-User \
--secure-port=443 \
--service-cluster-ip-range=..... \
--storage-backend=etcd3 \
--tls-cert-file=/srv/kubernetes/server.cert \
--tls-private-key-file=/srv/kubernetes/server.key \
--token-auth-file=/srv/kubernetes/known_tokens.csv \
--v=2 \
--logtostderr=false \
--alsologtostderr \
--log-file=/var/log/kube-apiserver.log
I saw that kops had a minor version update ready and decided that this would be the best way to "reboot" the cluster. Lo and behold, all three resources are once again present after the upgrade.
I do not have the foggiest how this could have happened, and it doesn't seem like anybody else has ever experienced this issue. Many thanks to @DT. for providing some great troubleshooting pointers.