After a few days of running Couchbase 6.5.1 in kubernetes the defined "CouchbaseCluster" resource disappears, resulting in the deletion of my couchbase pods.
After a bit of digging i found that the Admissions Operator pod logs contained continual updates to the CouchbaseCluster resource more than once per second:
I1021 15:05:20.013984 1 admission.go:185] Mutating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example
I1021 15:05:20.061531 1 admission.go:137] Validating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example
I1021 15:05:20.613922 1 admission.go:185] Mutating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example
I1021 15:05:20.620427 1 admission.go:137] Validating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example
I1021 15:05:21.414017 1 admission.go:185] Mutating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example
I1021 15:05:21.460600 1 admission.go:137] Validating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example
I1021 15:05:22.013887 1 admission.go:185] Mutating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example
I1021 15:05:22.060931 1 admission.go:137] Validating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example
I1021 15:05:22.413665 1 admission.go:185] Mutating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example
I1021 15:05:22.420773 1 admission.go:137] Validating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example
I1021 15:05:23.014797 1 admission.go:185] Mutating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example
I1021 15:05:23.023459 1 admission.go:137] Validating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example
I1021 15:05:23.614544 1 admission.go:185] Mutating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example
I1021 15:05:23.661482 1 admission.go:137] Validating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example
I1021 15:05:24.014503 1 admission.go:185] Mutating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example
I1021 15:05:24.021428 1 admission.go:137] Validating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example
I1021 15:05:24.613723 1 admission.go:185] Mutating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example
I1021 15:05:24.639612 1 admission.go:137] Validating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example
I1021 15:05:25.217866 1 admission.go:185] Mutating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example
I1021 15:05:25.223814 1 admission.go:137] Validating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example
I1021 15:05:25.614774 1 admission.go:185] Mutating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example
I1021 15:05:25.662553 1 admission.go:137] Validating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example
I1021 15:05:26.213481 1 admission.go:185] Mutating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example
I1021 15:05:26.221502 1 admission.go:137] Validating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example
I1021 15:05:26.813576 1 admission.go:185] Mutating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example
I1021 15:05:26.820181 1 admission.go:137] Validating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example
This causes the generation number on the CouchbaseCluster type definition to climb rapidly. After just 15 mins it gets to 1500. I suspect that this behaviour is not normal and eventually the CouchbaseCluster resource is deleted by kubernetes.
This behaviour occurs with the most basic CouchbaseCluster definition:
apiVersion: couchbase.com/v2
kind: CouchbaseCluster
metadata:
name: cb-example
spec:
image: couchbase/server:6.5.1
security:
adminSecret: cb-auth
networking:
exposeAdminConsole: true
adminConsoleServices:
- data
buckets:
managed: true
servers:
- size: 3
name: all_services
services:
- data
- index
- query
- search
- eventing
- analytics
Are the operator admissions logs normal?
How do I debug it further?
The disappearing cluster turned out to be this issue: https://forums.couchbase.com/t/multiple-couchbase-environments-in-one-kubernetes-cluster/28343