standalone network endpoint group (NEG) on GKE not working

9/28/2019

i am running a minimal stateful database service on GKE. single node cluster. i've setup a database as a stateful set on a single pod as of now. the database has exposed a management console on a particular port along with the mandatory database port. i am attempting to do two things.

  • expose management port over a global HTTP(S) load balancer
  • expose database port outside of GKE to be consumed by the likes of Cloud Functions or App Engine Applications.

My stateful set is running fine and i can see from the container logs that the database is properly booted up and is listening on required ports.

i am attempting to setup a standalone NEG (ref: https://cloud.google.com/kubernetes-engine/docs/how-to/standalone-neg) using a simple ClusterIP service.

the cluster service comes up fine and i can see it using

kubectl get service service-name

but i dont see the NEG setup as such... the following command returns nothing

$ gcloud compute network-endpoint-groups list
Listed 0 items.

my pod exposes the port 8080 my service maps 51000 to 8080 and i have provided the neg annotation

cloud.google.com/neg: '{"exposed_ports": {"51000":{}}'

I dont see any errors as such but neither do i see a NEG created/listed.

Any suggestions on how i would go about debugging this.

As a followup question...

  • when exposing NEG over global load balancer, how do i enforce authn? im ok with either of service account roles or oauth/openid.

  • would i be able to expose multiple ports using a single NEG? for e.g. if i wanted to expose one port to my global load balancer and another to local services, is this possible with a single NEG or should i expose each port using a dedicated ClusterIP service?

  • where can i find documentation/specification for google kubernetes annotations. i tried to expose two ports on the neg using the following annotation syntax. is that even supported/meaningful?

    cloud.google.com/neg: '{"exposed_ports": {"51000":{},"51010":{}}'

Thanks in advance!

-- Denounce'IN
google-cloud-platform
google-kubernetes-engine
load-balancing

1 Answer

10/3/2019

In order to create the service that is backed by a network endpoint group, you need to be working on a GKE Cluster that is VPC Native:

When you create a new cluster, this option is disabled by default and you must enable it upon creation. You can confirm if your cluster is VPC Native going to your Cluster details in GKE. It should appear like this:

VPC-native (alias IP)   Enabled

If the cluster is not VPC Native, you won’t be able to use this feature as described on their restrictions:

In case you have VPC Native enabled, make sure as well that the pods have the same labels “purpose:” and “topic:” to make sure they are members of the service:

kubectl get pods --show-labels

You can also create multi-port services as it is described on Kubernetes documentation:

-- Josep Nadal
Source: StackOverflow