K
Q

Question

Restart pods in deployment A when deployment B changes

10/12/2019

Stuck deploying a legacy application I don't control to k8s that requires a list of IP addresses on the command line, those IPs being the pods in deployment B, eg.:

./legacy_app -s 10.1.0.1 10.1.0.2 10.2.0.2 - call this app deployment A

(the IP addresses are gathered by querying the .../pods endpoint of the k8s API during pod launch)

When deployment B changes (scales out / in, pods restart, etc), the list of IPs changes, and I need the pods in deployment A to restart, in order to re-query the API for the correct list of IPs.

How can I cleanly achieve this, ideally using standard k8s primitives?

What I've tried so far:

an app that calls watch on deployment B, and on detecting a MODIFIED event, updates a label on deployment A, forcing a restart. This kinda works, but requires the watcher to pause for a few seconds before restarting deploy A - without the pause, the list of IPs is often not up to date by the time deploy A restarts, resulting in an incomplete list. However, the longer it pauses, the more data I'm losing. This adds a bit more operational complexity than I like.

What I'm going to try next:

replacing pid 1 on deployment A with a monitoring loop that (re-) starts the legacy app with the new list of IPs when the list changes.
can I update a configMap or a label on deployment A pods containing the list of IPs, and somehow use that to signal when a restart is needed?

Is there a better way? Coming into k8s I expected there to be some kind of hook / watch I could subscribe to, and run a "restart all pods in deployment" type command, but that feature doesn't appear to exist.

I'm clearly new to k8s, any input much appreciated.

k8s 1.14 on AWS EKS

-- Danielle M.

kubernetes

kubernetes-pod

1 Answer

10/14/2019

I ended up solving this using a bash entry script in front of deployment A, p much this pseudo code:

#!/bin/bash

get_deployment_B_ips() {
    echo $(curl https://$K8S_API/api/v1/namespaces/my_namespace/pods/ | \
           jq -r '[.items[] | select(.metadata.labels.app=="deployment-B") | select(.status.phase=="Running")] | map(.status.podIP + ":9125") |  join(" ")')
}

while true; do
CURRENT_LIST=$(get_deployment_B_ips)

  if [[ "$IP_LIST" == "$CURRENT_LIST" ]]; then
    sleep 5
  else
   # restart the process with new IP LIST
  fi
done

This works great for now, but going forward I'm going to read about implementation details of k8s Operators, and see if they can provide a cleaner fix than this.

Marking this as the answer to my question, unless a better solution comes along.

-- Danielle M.

Source: StackOverflow

KQ

Restart pods in deployment A when deployment B changes

Similar Questions

1 Answer

K
Q