How to wait until Kubernetes assigned an external IP to a LoadBalancer service?

2/3/2016

Creating a Kubernetes LoadBalancer returns immediatly (ex: kubectl create -f ... or kubectl expose svc NAME --name=load-balancer --port=80 --type=LoadBalancer).

I know a manual way to wait in shell:

external_ip=""
while [ -z $external_ip ]; do
    sleep 10
    external_ip=$(kubectl get svc load-balancer --template="{{range .status.loadBalancer.ingress}}{{.ip}}{{end}}")
done

This is however not ideal:

  • Requires at least 5 lines Bash script.
  • Infinite wait even in case of error (else requires a timeout which increases a lot line count).
  • Probably not efficient; could use --wait or --wait-once but using those the command never returns.

Is there a better way to wait until a service external IP (aka LoadBalancer Ingress IP) is set or failed to set?

-- Wernight
bash
kubernetes

4 Answers

5/12/2019

Really just a clean-up of @Dan Garfield's working example; My OCD wouldn't let this slide. In this case:

  • on GCP
  • requesting an internal lb
  • with an annotation in a service definition

apiVersion: v1
kind: Service
metadata:
  name: yo
  annotations:
    cloud.google.com/load-balancer-type: "Internal"
    # external-dns.alpha.kubernetes.io/hostname: vault.stage.domain.tld.
...

NOTE: I've only been able to get external-dns to associate names to public IP addresses.


This has been scripted to accept a few arguments, now it's a library; example:

myServiceLB=$1
while true; do                                                                     
    successCond="$(kubectl get svc "$myServiceLB" \                                
        --template="{{range .status.loadBalancer.ingress}}{{.ip}}{{end}}")"        
    if [[ -z "$successCond" ]]; then                                               
        echo "Waiting for endpoint readiness..."                                   
        sleep 10                                                                   
    else                                                                           
        sleep 2                                                                    
        export lbIngAdd="$successCond"                                             
        pMsg """
            The Internal LoadBalancer is up!
        """                                                                        
        break                                                                      
    fi                                                                             
done

Later, $lbIngAdd can be used to set records. Seems like -o jsonpath="{.status.loadBalancer.ingress[*].ip}" would work as well; whatever works.

Thanks for getting us started Dan :-)

-- todd_dsm
Source: StackOverflow

4/23/2018

Just to add to the answers here, the best option right now is to use a bash script. For convenience, I've put it into a single line that includes exporting an environmental variable.

Command to wait and find Kubernetes service endpoint

bash -c 'external_ip=""; while [ -z $external_ip ]; do echo "Waiting for end point..."; external_ip=$(kubectl get svc NAME_OF_YOUR_SERVICE --template="{{range .status.loadBalancer.ingress}}{{.ip}}{{end}}"); [ -z "$external_ip" ] && sleep 10; done; echo "End point ready-" && echo $external_ip; export endpoint=$external_ip'

I've also modified your script so it only executes a wait if the ip isn't available. The last bit will export an environment variable called "endpoint"

Bash Script to Check a Given Service

Save this as check-endpoint.sh and then you can execute $sh check-endpoint.sh SERVICE_NAME

#!/bin/bash
# Pass the name of a service to check ie: sh check-endpoint.sh staging-voting-app-vote
# Will run forever...
external_ip=""
while [ -z $external_ip ]; do
  echo "Waiting for end point..."
  external_ip=$(kubectl get svc $1 --template="{{range .status.loadBalancer.ingress}}{{.ip}}{{end}}")
  [ -z "$external_ip" ] && sleep 10
done
echo 'End point ready:' && echo $external_ip

Using this in a Codefresh Step

I'm using this for a Codefresh pipeline and it passes a variable $endpoint when it's done.

  GrabEndPoint:
    title: Waiting for endpoint to be ready
    image: codefresh/plugin-helm:2.8.0
    commands:
      - bash -c 'external_ip=""; while [ -z $external_ip ]; do echo "Waiting for end point..."; external_ip=$(kubectl get svc staging-voting-app-vote --template="{{range .status.loadBalancer.ingress}}{{.ip}}{{end}}"); [ -z "$external_ip" ] && sleep 10; done; echo "End point ready-" && echo $external_ip; cf_export endpoint=$external_ip'
-- Dan Garfield
Source: StackOverflow

3/12/2020

Here's a generic bash function to watch with timeout, for any regexp in the output of a given command:

function watch_for() {
  CMD="$1" # Command to watch. Variables should be escaped \$
  REGEX="$2" # Pattern to search
  ATTEMPTS=${3:-10} # Timeout. Default is 10 attempts (interval of second)
  COUNT=0;

  echo -e "# Watching for /$REGEX/ during $ATTEMPTS seconds, on the output of command:\n# $CMD"
  until eval "$CMD" | grep -m 1 "$REGEX" || [[ $COUNT -eq $ATTEMPTS ]]; do
    echo -e "$(( COUNT++ ))... \c"
    sleep 1
  done
  if [[ $COUNT -eq $ATTEMPTS ]]; then
    echo "# Limit of $ATTEMPTS attempts has exceeded."
    return 1
  fi
  return 0
}

And here's how I used it to wait until a worker node gets an external IP (which took more than a minute):

$ watch_for "kubectl get nodes -l node-role.kubernetes.io/worker -o wide | awk '{print \$7}'" \
"[0-9]" 100

0... 1... 2... 3... .... 63... 64... 3.22.37.41

-- Noam Manos
Source: StackOverflow

2/11/2016

There's not really a "failed to set" condition because we will retry it forever. A failure might have been a transient error in the cloud provider or a quota issue that gets resolved over the course of hours or days, or any number of things. The only failure comes from "how long are you willing to wait?" - which only you can know.

We don't have a general "wait for expression" command because it ends up being arbitrarily complex and you're better off just coding that in a real language. Ergo the bash loop above. We could do better about having a 'watch' command, but it's still a timeout in the end.

-- Tim Hockin
Source: StackOverflow