What are the options to scale up/scale down a kubernetes statefulset during runtime?

7/13/2019

In my use case i need to scale up a stateful set upon specific logic, and scale it down. I'm not sure HPA will help as i need to create exactly one replica each time, and later raise a notification to scale it down, meaning - the application will be scaled, and then for a specific amount of time there will be no scaling down (HPA will continue spinning pods until its conditions are met, if they are met, it will scale down the pod which is a problem).

I'm deploying using helm.

Should i:

  • Make the pod access K8s API and change the replica count?
  • Use helm controller to maybe create/update a CRD that contains the replica count?
  • run kubectl scale inside the pod when the conditions are met?

what's the best practices here?

-- ArielB
kubernetes
kubernetes-helm
scale

1 Answer

7/18/2019

You can setup a CronJob that will spawn a pod each (x minutes) and check, for example by using ConfigMap if it needs to scale up/down the StatefulSet.

This Job will use the REST API, with it you can use API reference docs to Replace or Patch your StatefulSet.

You can do that by:

Using kubectl proxy

$ kubectl proxy --port=8080 & See kubectl proxy for more details.

Then you can explore the API with curl, wget, or a browser, like so:

$ curl http://localhost:8080/api/ The output is similar to this:

{
  "versions": [
    "v1"
  ],
  "serverAddressByClientCIDRs": [
    {
      "clientCIDR": "0.0.0.0/0",
      "serverAddress": "10.0.1.149:443"
    }
  ]
}

Without kubectl proxy

It is possible to avoid using kubectl proxy by passing an authentication token directly to the API server, like this:

Using grep/cut approach:

# Check all possible clusters, as you .KUBECONFIG may have multiple contexts:
kubectl config view -o jsonpath='{"Cluster name\tServer\n"}{range .clusters[*]}{.name}{"\t"}{.cluster.server}{"\n"}{end}'

# Select name of cluster you want to interact with from above output:
export CLUSTER_NAME="some_server_name"

# Point to the API server refering the cluster name
APISERVER=$(kubectl config view -o jsonpath="{.clusters[?(@.name==\"$CLUSTER_NAME\")].cluster.server}")

# Gets the token value
TOKEN=$(kubectl get secrets -o jsonpath="{.items[?(@.metadata.annotations['kubernetes\.io/service-account\.name']=='default')].data.token}"|base64 -d)

# Explore the API with TOKEN
curl -X GET $APISERVER/api --header "Authorization: Bearer $TOKEN" --insecure

The output is similar to this:

{
  "kind": "APIVersions",
  "versions": [
    "v1"
  ],
  "serverAddressByClientCIDRs": [
    {
      "clientCIDR": "0.0.0.0/0",
      "serverAddress": "10.0.1.149:443"
    }
  ]
}

Or with programmatic access to the API, Kubernetes officially supports Go and Python client libraries.

I hope this helps you a bit, if you have further questions please ask.

-- Crou
Source: StackOverflow