What prevents the 'instance-groups' annotation from being set when creating a multi-cluster ingress on Google Kubernetes Engine?

1/25/2020

I'm trying to create a multi-cluster ingress on Google Kubernetes Engine using kubemci, however when running the following command the program waits indefinitely for the ingress service to get the ingress.gcp.kubernetes.io/instance-groups annotation (as illustrated in the output below).

What is preventing this annotation from being set?

Input

./kubemci create app-mci \
    --ingress=ingress.yaml \
    --gcp-project=app-prod \
    --kubeconfig=mcikubeconfig

Output

% ./kubemci create app-mci --ingress=ingress.yaml --gcp-project=app-prod --kubeconfig=clusters.yaml        
Created Ingress in cluster: gke_app-prod_europe-west4-a_app-europe-west4
Created Ingress in cluster: gke_app-prod_us-east4-a_app-us-east4
Ensuring health checks
Pod app-deployment-c99578769-xdmql matching service selectors app=app (targetport ): lacks a matching HTTP probe for use in health checks.
Pod app-deployment-c99578769-xgq2m matching service selectors app=app (targetport ): lacks a matching HTTP probe for use in health checks.
Pod app-deployment-c99578769-qms7r matching service selectors app=app (targetport ): lacks a matching HTTP probe for use in health checks.
Pod app-deployment-c99578769-tsrsw matching service selectors app=app (targetport ): lacks a matching HTTP probe for use in health checks.
Path for healthcheck is /
Ensuring health check for port: {SvcName:default/app-service SvcPort:{Type:0 IntVal:80 StrVal:} NodePort:30061 Protocol:HTTP SvcTargetPort: NEGEnabled:false}
Health check mci1-hc-30061--app-mci exists already. Checking if it matches our desired health check
Desired health check exists already
Determining instance groups for cluster gke_app-prod_europe-west4-a_app-europe-west4
Waiting for ingress ( default : app-ingress ) to get ingress.gcp.kubernetes.io/instance-groups annotation.....
Waiting for ingress ( default : app-ingress ) to get ingress.gcp.kubernetes.io/instance-groups annotation.....
Waiting for ingress ( default : app-ingress ) to get ingress.gcp.kubernetes.io/instance-groups annotation.....

As you can see my configuration is identical (aside from resource names) to that in the multi-cluster ingress guide:

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-deployment
spec:
  selector:
    matchLabels:
      app: app
  replicas: 2
  template:
    metadata:
      labels:
        app: app
    spec:
      containers:
        - name: app
          image: gcr.io/app-prod/app:tag
          ports:
            - containerPort: 8080

service.yaml

apiVersion: v1
kind: Service
metadata:
  labels:
    app: app
  name: app-service
spec:
  ports:
    - port: 80
      protocol: TCP
      targetPort: 8080
      name: http
      nodePort: 30061
  selector:
    app: app
  type: NodePort

ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: app-ingress
  annotations:
    kubernetes.io/ingress.global-static-ip-name: app-ip
    kubernetes.io/ingress.class: gce-multi-cluster
spec:
  backend:
    serviceName: app-service
    servicePort: 80
-- Luke G. Ribchester
google-kubernetes-engine
kubernetes
kubernetes-ingress

1 Answer

1/25/2020

Enable HTTP load balancing

Enable the HTTP load balancing add-on to allow the load balancer controller to set the ingress.gcp.kubernetes.io/instance-groups annotation.

Console

  1. Edit a cluster.
  2. Expand add-ons.
  3. Enable HTTP load balancing:

enter image description here

Command line

% gcloud container clusters update [CLUSTER_NAME] --update-addons HttpLoadBalancing=ENABLED

Updating ***...done.                                                                                                                                                              
Updated [https://container.googleapis.com/v1/projects/***/zones/us-east4-a/clusters/***].
To inspect the contents of your cluster, go to: https://console.cloud.google.com/kubernetes/workload_/gcloud/us-east4-a/***?project=***

View the cluster configuration:

% gcloud container clusters describe [CLUSTER_NAME]

# ENABLED
addonsConfig:
  httpLoadBalancing: {}

# DISABLED
addonsConfig:
  httpLoadBalancing:
    disabled: true

Configure Services

Ensure that the backend Services used in the multi-cluster ingress are configured correctly.

Services must:

  • Have the same name in all of the clusters.
  • Be in the same namespace in all of the clusters.
  • Be of type NodePort.
  • Use the same port number for all of the clusters.

    Setting up a multi-cluster Ingress, Google

Credit

  • Nikhil Jindal for his insight.
  • Ivan for raising this issue.
-- Luke G. Ribchester
Source: StackOverflow