How to handle ip address change in couchbase cluster?

9/27/2019

I have couchbase cluster on k8s with operator 1.2 , I see following error today continuously

IP address seems to have changed. Unable to listen on 'ns_1@couchbase-cluster-couchbase-cluster-0001.couchbase-cluster-couchbase-cluster.default.svc'. (POSIX error code: 'nxdomain') (repeated 3 times)

-- Vishnu Gopal Singhal
couchbase
kubernetes

1 Answer

10/16/2019

The “IP address change” message is an alert message generated by Couchbase Server. The server checks for this situation as follows: it tries to listen on a free port on the interface that is the node’s address.

It does this every 3 seconds. If the host name of the node can’t be resolved you get an nxdomain error which is the most common reason that users see this alert message.

However, the alert would also fire if the user stopped the server, renamed the host and restarted - a much more serious configuration error that we would want to alert the user to right away. Because this check runs every three seconds, if you have any flakiness in your DNS you are likely to see this alert message every now and then.

As long as the DNS glitch doesn’t persist for long (a few seconds) there probably won’t be any adverse issues. However, it is an indication that you may want to take a look at your DNS to make sure it’s reliable enough to run a distributed system such as Couchbase Server against. In the worst case, DNS that is unavailable for a significant length of time could result in lack of availability or auto failover.

Ps:Thanks to Dave Finlay who actually answered this question to me.

-- deniswsrosa
Source: StackOverflow