K
Q

Using Couchbase operator 1.1 with Istio Service Mesh

4/2/2019

I am trying to create a Couchbase cluster in GKE with Istio (envoy proxy) using the Autonomous Operator 1.1. The operator starts up fine, and after running the yaml to create the couchbasecluster the first node starts up and then the 2nd node starts. Issue is that the 2nd node appears to fail to join the cluster and additional nodes are not being started.

I am not sure how to debug what is happening or what needs to be done to get the cluster to start up in my gke cluster. Any assistance is appreceated.

Thank you

Here are some of the logs from one of the couchbase node pods:

I  [2019-04-02T14:58:00.706Z] "POST /engageCluster2HTTP/1.1" 404 NR 0 0 0 - "-" "-" "782bde60-c611-4bfb-a0f4-9975300c71a4" "cb-example-0003.cb-example.couchbase.svc:8091" "-" - - 10.36.9.15:8091 10.36.8.13:37221

I  [2019-04-02T14:58:05.706Z] "POST /engageCluster2HTTP/1.1" 404 NR 0 0 0 - "-" "-" "382b6163-e8bc-4259-baaa-e854c36af1bd" "cb-example-0003.cb-example.couchbase.svc:8091" "-" - - 10.36.9.15:8091 10.36.8.13:55515

I  [2019-04-02T14:58:10.707Z] "POST /engageCluster2HTTP/1.1" 404 NR 0 0 0 - "-" "-" "390e417e-b179-4bbf-81d8-02cc28d2bc98" "cb-example-0003.cb-example.couchbase.svc:8091" "-" - - 10.36.9.15:8091 10.36.8.13:34377

I  [2019-04-02T14:53:13.605Z] - 210 4281 300015 "127.0.0.1:8091" inbound|8091||cb-example.couchbase.svc.cluster.local 127.0.0.1:45756 10.36.8.13:8091 10.36.9.12:49792

I  [2019-04-02T14:58:15.709Z] "POST /engageCluster2HTTP/1.1" 404 NR 0 0 0 - "-" "-" "037d1791-9feb-47be-b699-10269aaf36e9" "cb-example-0003.cb-example.couchbase.svc:8091" "-" - - 10.36.9.15:8091 10.36.8.13:55307

I  [2019-04-02T14:58:20.708Z] "POST /engageCluster2HTTP/1.1" 404 NR 0 0 0 - "-" "-" "5ca29b59-ff25-4a13-a0c1-62668d40c681" "cb-example-0003.cb-example.couchbase.svc:8091" "-" - - 10.36.9.15:8091 10.36.8.13:51205

I  [2019-04-02T14:58:25.706Z] "POST /engageCluster2HTTP/1.1" 404 NR 0 0 0 - "-" "-" "9e21bc4d-1367-4d25-b674-39ae6341c9b4" "cb-example-0003.cb-example.couchbase.svc:8091" "-" - - 10.36.9.15:8091 10.36.8.13:41435

I  [2019-04-02T14:58:30.710Z] "POST /engageCluster2HTTP/1.1" 404 NR 0 0 0 - "-" "-" "c2f8e866-e0a5-43ff-b54f-e5c504b17cdf" "cb-example-0003.cb-example.couchbase.svc:8091" "-" - - 10.36.9.15:8091 10.36.8.13:40203

I  [2019-04-02T14:58:35.708Z] "POST /engageCluster2HTTP/1.1" 404 NR 0 0 0 - "-" "-" "4b02e855-cc72-49dc-99e1-a8644fdf1af8" "cb-example-0003.cb-example.couchbase.svc:8091" "-" - - 10.36.9.15:8091 10.36.8.13:56433

I  [2019-04-02T14:53:13.641Z] - 16628 40061 324989 "127.0.0.1:8091" inbound|8091||cb-example.couchbase.svc.cluster.local 127.0.0.1:45760 10.36.8.13:8091 10.36.9.12:49796

I  [2019-04-02T14:56:45.698Z] - 9490 13635 112934 "127.0.0.1:8091" inbound|8091||cb-example.couchbase.svc.cluster.local 127.0.0.1:46218 10.36.8.13:8091 10.36.9.12:50534

I  [2019-04-02T14:56:45.665Z] - 210 4281 112967 "127.0.0.1:8091" inbound|8091||cb-example.couchbase.svc.cluster.local 127.0.0.1:46216 10.36.8.13:8091 10.36.9.12:50528

And a portion of the error.log from inside the couchbase container.

[ns_server:error,2019-04-03T16:09:47.398Z,ns_1@cb-example-0000.cb-example.couchbase.svc:service_agent-index<0.24974.68>:service_agent:handle_call:182]Got rebalance-only call {if_rebalance,<0.23572.68>,unset_rebalancer} that doesn't match rebalancer pid undefined
[ns_server:error,2019-04-03T16:09:47.398Z,ns_1@cb-example-0000.cb-example.couchbase.svc:service_rebalancer-index<0.23572.68>:service_agent:process_bad_results:810]Service call unset_rebalancer (service index) failed on some nodes:
[{'ns_1@cb-example-0000.cb-example.couchbase.svc',nack}]
[ns_server:error,2019-04-03T16:09:47.398Z,ns_1@cb-example-0000.cb-example.couchbase.svc:cleanup_process<0.23562.68>:service_janitor:maybe_init_topology_aware_service:87]Initial rebalance for `index` failed: {error,
                                       {initial_rebalance_failed,index,
                                        {linked_process_died,<0.23516.68>,
                                         {no_connection,
                                          "index-service_api"}}}}
[ns_server:error,2019-04-03T16:10:47.399Z,ns_1@cb-example-0000.cb-example.couchbase.svc:<0.24979.68>:service_agent:wait_for_connection_loop:299]No connection with label "index-service_api" after 60000ms. Exiting.
[ns_server:error,2019-04-03T16:10:47.399Z,ns_1@cb-example-0000.cb-example.couchbase.svc:service_agent-index<0.24974.68>:service_agent:handle_info:231]Linked process <0.24979.68> died with reason {no_connection,
                                              "index-service_api"}. Terminating
[ns_server:error,2019-04-03T16:10:47.399Z,ns_1@cb-example-0000.cb-example.couchbase.svc:service_agent-index<0.24974.68>:service_agent:terminate:260]Terminating abnormally
[ns_server:error,2019-04-03T16:10:47.399Z,ns_1@cb-example-0000.cb-example.couchbase.svc:service_rebalancer-index<0.25043.68>:service_rebalancer:run_rebalance:82]Agent terminated during the rebalance: {'DOWN',#Ref<0.0.48.97712>,process,
                                        <0.24974.68>,
                                        {linked_process_died,<0.24979.68>,
                                         {no_connection,"index-service_api"}}}
[ns_server:error,2019-04-03T16:10:47.400Z,ns_1@cb-example-0000.cb-example.couchbase.svc:service_agent-index<0.26461.68>:service_agent:handle_call:182]Got rebalance-only call {if_rebalance,<0.25043.68>,unset_rebalancer} that doesn't match rebalancer pid undefined
[ns_server:error,2019-04-03T16:10:47.400Z,ns_1@cb-example-0000.cb-example.couchbase.svc:service_rebalancer-index<0.25043.68>:service_agent:process_bad_results:810]Service call unset_rebalancer (service index) failed on some nodes:
[{'ns_1@cb-example-0000.cb-example.couchbase.svc',nack}]
[ns_server:error,2019-04-03T16:10:47.400Z,ns_1@cb-example-0000.cb-example.couchbase.svc:cleanup_process<0.25042.68>:service_janitor:maybe_init_topology_aware_service:87]Initial rebalance for `index` failed: {error,
                                       {initial_rebalance_failed,index,
                                        {linked_process_died,<0.24979.68>,
                                         {no_connection,
                                          "index-service_api"}}}}
[ns_server:error,2019-04-03T16:11:47.401Z,ns_1@cb-example-0000.cb-example.couchbase.svc:<0.26456.68>:service_agent:wait_for_connection_loop:299]No connection with label "index-service_api" after 60000ms. Exiting.
[ns_server:error,2019-04-03T16:11:47.401Z,ns_1@cb-example-0000.cb-example.couchbase.svc:service_agent-index<0.26461.68>:service_agent:handle_info:231]Linked process <0.26456.68> died with reason {no_connection,
                                              "index-service_api"}. Terminating
[ns_server:error,2019-04-03T16:11:47.401Z,ns_1@cb-example-0000.cb-example.couchbase.svc:service_agent-index<0.26461.68>:service_agent:terminate:260]Terminating abnormally
[ns_server:error,2019-04-03T16:11:47.401Z,ns_1@cb-example-0000.cb-example.couchbase.svc:service_rebalancer-index<0.26515.68>:service_rebalancer:run_rebalance:82]Agent terminated during the rebalance: {'DOWN',#Ref<0.0.48.106235>,process,
                                        <0.26461.68>,
                                        {linked_process_died,<0.26456.68>,
                                         {no_connection,"index-service_api"}}}
[ns_server:error,2019-04-03T16:11:47.402Z,ns_1@cb-example-0000.cb-example.couchbase.svc:service_agent-index<0.27939.68>:service_agent:handle_call:182]Got rebalance-only call {if_rebalance,<0.26515.68>,unset_rebalancer} that doesn't match rebalancer pid undefined
[ns_server:error,2019-04-03T16:11:47.402Z,ns_1@cb-example-0000.cb-example.couchbase.svc:service_rebalancer-index<0.26515.68>:service_agent:process_bad_results:810]Service call unset_rebalancer (service index) failed on some nodes:
[{'ns_1@cb-example-0000.cb-example.couchbase.svc',nack}]
[ns_server:error,2019-04-03T16:11:47.402Z,ns_1@cb-example-0000.cb-example.couchbase.svc:cleanup_process<0.26517.68>:service_janitor:maybe_init_topology_aware_service:87]Initial rebalance for `index` failed: {error,
                                       {initial_rebalance_failed,index,
                                        {linked_process_died,<0.26456.68>,
                                         {no_connection,
                                          "index-service_api"}}}}

And this is from the current portion of the couchbase-operator log:

I  [2019-04-03T16:15:13.959Z] "GET /poolsHTTP/1.1" 404 NR 0 0 0 - "-" "Go-http-client/1.1" "cc976505-818a-4930-9fc8-8bdcb047185d" "cb-example-0000.cb-example.couchbase.svc:8091" "-" - - 10.36.8.13:8091 10.36.9.12:59280

I  [2019-04-03T16:15:13.963Z] "GET /poolsHTTP/1.1" 404 NR 0 0 0 - "-" "Go-http-client/1.1" "bfd981b2-9356-4132-a7f8-2a6c0d8ba15f" "cb-example-0003.cb-example.couchbase.svc:8091" "-" - - 10.36.9.15:8091 10.36.9.12:57624

I  [2019-04-03T16:15:14.939Z] - 119 135 0 "127.0.0.1:8080" inbound|8080||mgmtCluster 127.0.0.1:37568 10.36.9.12:8080 10.36.9.1:44810

I  [2019-04-03T16:15:17.939Z] - 119 135 0 "127.0.0.1:8080" inbound|8080||mgmtCluster 127.0.0.1:37574 10.36.9.12:8080 10.36.9.1:44816

I  [2019-04-03T16:15:18.959Z] "GET /poolsHTTP/1.1" 404 NR 0 0 0 - "-" "Go-http-client/1.1" "997c061a-d5d2-425d-b123-bf76073d148a" "cb-example-0000.cb-example.couchbase.svc:8091" "-" - - 10.36.8.13:8091 10.36.9.12:59298

I  [2019-04-03T16:15:18.962Z] "GET /poolsHTTP/1.1" 404 NR 0 0 0 - "-" "Go-http-client/1.1" "b03def85-726f-4107-8b27-9fc8b5bddea7" "cb-example-0003.cb-example.couchbase.svc:8091" "-" - - 10.36.9.15:8091 10.36.9.12:57642

I  [2019-04-03T16:15:20.939Z] - 119 135 0 "127.0.0.1:8080" inbound|8080||mgmtCluster 127.0.0.1:37586 10.36.9.12:8080 10.36.9.1:44828

E  time="2019-04-03T16:15:26Z" level=warning msg="cluster status: failed with error [Get http://cb-example-0000.cb-example.couchbase.svc:8091/pools/default: uuid check: unexpected status code '404 Not Found' from cb-example-0000.cb-example.couchbase.svc:8091], [Get http://cb-example-0003.cb-example.couchbase.svc:8091/pools/default: uuid check: unexpected status code '404 Not Found' from cb-example-0003.cb-example.couchbase.svc:8091] ...retrying" cluster-name=cb-example module=cluster

E  time="2019-04-03T16:15:31Z" level=warning msg="cluster status: failed with error [Get http://cb-example-0000.cb-example.couchbase.svc:8091/pools/default: uuid check: unexpected status code '404 Not Found' from cb-example-0000.cb-example.couchbase.svc:8091], [Get http://cb-example-0003.cb-example.couchbase.svc:8091/pools/default: uuid check: unexpected status code '404 Not Found' from cb-example-0003.cb-example.couchbase.svc:8091] ...retrying" cluster-name=cb-example module=cluster

I  [2019-04-03T16:15:23.939Z] - 119 135 0 "127.0.0.1:8080" inbound|8080||mgmtCluster 127.0.0.1:37592 10.36.9.12:8080 10.36.9.1:44834

I  [2019-04-03T16:15:26.939Z] - 119 135 0 "127.0.0.1:8080" inbound|8080||mgmtCluster 127.0.0.1:37604 10.36.9.12:8080 10.36.9.1:44846

I  [2019-04-03T16:15:26.987Z] "GET /poolsHTTP/1.1" 404 NR 0 0 0 - "-" "Go-http-client/1.1" "c20d057e-646d-4eb3-8931-a220126c27d5" "cb-example-0000.cb-example.couchbase.svc:8091" "-" - - 10.36.8.13:8091 10.36.9.12:59326

I  [2019-04-03T16:15:26.991Z] "GET /poolsHTTP/1.1" 404 NR 0 0 0 - "-" "Go-http-client/1.1" "c46e9753-1249-4b8c-8fc9-889e74a0d70b" "cb-example-0003.cb-example.couchbase.svc:8091" "-" - - 10.36.9.15:8091 10.36.9.12:57670

I  [2019-04-03T16:15:29.939Z] - 119 135 1 "127.0.0.1:8080" inbound|8080||mgmtCluster 127.0.0.1:37616 10.36.9.12:8080 10.36.9.1:44858

I  [2019-04-03T16:15:31.986Z] "GET /poolsHTTP/1.1" 404 NR 0 0 0 - "-" "Go-http-client/1.1" "30c8522f-5797-488c-b234-d1c5a43d9826" "cb-example-0000.cb-example.couchbase.svc:8091" "-" - - 10.36.8.13:8091 10.36.9.12:59338

I  [2019-04-03T16:15:31.990Z] "GET /poolsHTTP/1.1" 404 NR 0 0 0 - "-" "Go-http-client/1.1" "a4f74a5e-c026-46b4-b9e3-8b20f65477b4" "cb-example-0003.cb-example.couchbase.svc:8091" "-" - - 10.36.9.15:8091 10.36.9.12:57682

E  time="2019-04-03T16:15:36Z" level=warning msg="cluster status: failed with error [Get http://cb-example-0000.cb-example.couchbase.svc:8091/pools/default: uuid check: unexpected status code '404 Not Found' from cb-example-0000.cb-example.couchbase.svc:8091], [Get http://cb-example-0003.cb-example.couchbase.svc:8091/pools/default: uuid check: unexpected status code '404 Not Found' from cb-example-0003.cb-example.couchbase.svc:8091] ...retrying" cluster-name=cb-example module=cluster

E  time="2019-04-03T16:15:41Z" level=warning msg="cluster status: failed with error [Get http://cb-example-0000.cb-example.couchbase.svc:8091/pools/default: uuid check: unexpected status code '404 Not Found' from cb-example-0000.cb-example.couchbase.svc:8091], [Get http://cb-example-0003.cb-example.couchbase.svc:8091/pools/default: uuid check: unexpected status code '404 Not Found' from cb-example-0003.cb-example.couchbase.svc:8091] ...retrying" cluster-name=cb-example module=cluster

I  [2019-04-03T16:15:32.939Z] - 119 135 0 "127.0.0.1:8080" inbound|8080||mgmtCluster 127.0.0.1:37624 10.36.9.12:8080 10.36.9.1:44866
-- mmphantom
couchbase
google-kubernetes-engine
istio
kubernetes

Similar Questions

Kubernetes: Id or size of image "XXX.dkr.ecr.eu-west-1.amazonaws.com/msg/my_image:v1.0" is not set
How to fail a helm release based on inputs in values.yaml
How to setup Spring backend with JWT and Kubernetes
MongoDB replica set - Secondary repeatedly goes down

0 Answers