K
Q

Changing the CPU Manager Policy in Kubernetes

1/17/2019

I'm trying to change the CPU Manager Policy for a Kubernetes cluster that I manage, as described here however, I've run into numerous issues while doing so.

The cluster is running in DigitalOcean and here is what I've tried so far.

  • 1. Since the article mentions that --cpu-manager-policy is a kubelet option I assume that I cannot change it via the API Server and have to change it manually on each node. (Is this assumption BTW?)
  • 2. I ssh into one of the nodes (droplets in DigitalOcean lingo) and run kubelet --cpu-manager-policy=static command as described in the kubelet CLI reference here. It gives me the message Flag --cpu-manager-policy has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
  • 3. So I check the file pointed at by the --config flag by running ps aux | grep kubelet and find that its /etc/kubernetes/kubelet.conf.
  • 4. I edit the file and add a line cpuManagerPolicy: static to it, and also kubeReserved and systemReserved because they become required fields if specifying cpuManagerPolicy.
  • 5. Then I kill the process that was running the process and restart it. A couple other things showed up (delete this file and drain the node etc) that I was able to get through and was able to restart the kubelet ultimately

I'm a little lost about the following things

  • How do I need to do this for all nodes? My cluster has 12 of them and doing all of these steps for each seems very inefficient.
  • Is there any way I can set these params from the globally i.e. cluster-wide rather than doing node by node?
  • How can I even confirm that what I did actually changed the CPU Manager policy?
-- satnam
devops
digital-ocean
kubelet
kubernetes

Similar Questions

EKS cluster doesn't provide client-ca-file
Kubernetes pattern canary deployment smoke test best practice?
How can I mirror traffic to two places with Istio?
How to add php-redis for a dockerfile of laravel to kubernetes?
Kubernetes Pod is not access Service via Name on minikube

2 Answers

1/18/2019

It might not be the global way of doing stuff, but I think it will be much more comfortable than what you are currently doing.

First you need to run kubectl proxy --port=8001 &

Download the configuration:

NODE_NAME="the-name-of-the-node-you-are-reconfiguring"; curl -sSL "http://localhost:8001/api/v1/nodes/${NODE_NAME}/proxy/configz" | jq '.kubeletconfig|.kind="KubeletConfiguration"|.apiVersion="kubelet.config.k8s.io/v1beta1"' > kubelet_configz_${NODE_NAME}

Edit it accordingly, and push the configuration to control plane. You will see a valid response if everything went well. Then you will have to edit the configuration so the Node starts to use the new ConfigMap. There are many more possibilities, for example you can go back to default settings if anything goes wrong.

This process is described with all the details in this documentation section.

Hope this helps.

-- aurelius
Source: StackOverflow
2/11/2019

One issue with Dynamic Configuration is that in case the node fails to restart, the API does not give a reasonable response back that tells you what you did wrong, you'll have to ssh into the node and tail the kubelet logs. Plus, you have to ssh into every node and set the --dynamic-config-dir flag anyways.

The folllowing worked best for me

  1. SSH into the node. Edit
vim /etc/systemd/system/kubelet.service
  1. Add the following lines
  --cpu-manager-policy=static \
  --kube-reserved=cpu=1,memory=2Gi,ephemeral-storage=1Gi \
  --system-reserved=cpu=1,memory=2Gi,ephemeral-storage=1Gi \

We need to set the --kube-reserved and --system-reserved flags because they're prerequisties to setting the --cpu-manager-policy flag

  1. Then drain the node and delete the following folder
rm -rf /var/lib/kubelet/cpu_manager_state
  1. Restart the kubelet
sudo systemctl daemon-reload
sudo systemctl stop kubelet
sudo systemctl start kubelet
  1. uncordon the node and check the policy. This assumes that you're running kubectl proxy on port 8001.
curl -sSL "http://localhost:8001/api/v1/nodes/${NODE_NAME}/proxy/configz" | grep cpuManager
-- satnam
Source: StackOverflow