CouchbaseCluster resource in kubernetes disappears after a few days

10/21/2020

After a few days of running Couchbase 6.5.1 in kubernetes the defined "CouchbaseCluster" resource disappears, resulting in the deletion of my couchbase pods.

After a bit of digging i found that the Admissions Operator pod logs contained continual updates to the CouchbaseCluster resource more than once per second:

I1021 15:05:20.013984       1 admission.go:185] Mutating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example
I1021 15:05:20.061531       1 admission.go:137] Validating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example
I1021 15:05:20.613922       1 admission.go:185] Mutating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example
I1021 15:05:20.620427       1 admission.go:137] Validating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example
I1021 15:05:21.414017       1 admission.go:185] Mutating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example
I1021 15:05:21.460600       1 admission.go:137] Validating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example
I1021 15:05:22.013887       1 admission.go:185] Mutating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example
I1021 15:05:22.060931       1 admission.go:137] Validating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example
I1021 15:05:22.413665       1 admission.go:185] Mutating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example
I1021 15:05:22.420773       1 admission.go:137] Validating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example
I1021 15:05:23.014797       1 admission.go:185] Mutating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example
I1021 15:05:23.023459       1 admission.go:137] Validating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example
I1021 15:05:23.614544       1 admission.go:185] Mutating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example
I1021 15:05:23.661482       1 admission.go:137] Validating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example
I1021 15:05:24.014503       1 admission.go:185] Mutating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example
I1021 15:05:24.021428       1 admission.go:137] Validating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example
I1021 15:05:24.613723       1 admission.go:185] Mutating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example
I1021 15:05:24.639612       1 admission.go:137] Validating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example
I1021 15:05:25.217866       1 admission.go:185] Mutating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example
I1021 15:05:25.223814       1 admission.go:137] Validating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example
I1021 15:05:25.614774       1 admission.go:185] Mutating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example
I1021 15:05:25.662553       1 admission.go:137] Validating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example
I1021 15:05:26.213481       1 admission.go:185] Mutating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example
I1021 15:05:26.221502       1 admission.go:137] Validating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example
I1021 15:05:26.813576       1 admission.go:185] Mutating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example
I1021 15:05:26.820181       1 admission.go:137] Validating resource: UPDATE couchbase.com/v2, Kind=CouchbaseCluster test/cb-example

This causes the generation number on the CouchbaseCluster type definition to climb rapidly. After just 15 mins it gets to 1500. I suspect that this behaviour is not normal and eventually the CouchbaseCluster resource is deleted by kubernetes.

This behaviour occurs with the most basic CouchbaseCluster definition:

apiVersion: couchbase.com/v2
kind: CouchbaseCluster
metadata:
  name: cb-example  
spec:
  image: couchbase/server:6.5.1  
  security:
    adminSecret: cb-auth
  networking:
    exposeAdminConsole: true
    adminConsoleServices:
    - data
  buckets:
    managed: true
  servers:
  - size: 3  
    name: all_services
    services:
    - data
    - index
    - query
    - search
    - eventing
    - analytics

Are the operator admissions logs normal?

How do I debug it further?

-- Paul Clark
couchbase
kubernetes

1 Answer

11/17/2020
-- Paul Clark
Source: StackOverflow