K
Q

kubernetes autoscaling not working

12/28/2015

I have a kubernetes cluster that I'm trying to build from scratch without using their build scripts. Pretty much everything is working except for autoscaling. For some reason the control-manager can't find or doesn't know heapster is running.

I have a ticket open but it seems no responses

https://github.com/kubernetes/kubernetes/issues/18652

Things I have setup.

Here is a list of all the pods currently

[root@kube-master test] [dev] # kubectl get pods --all-namespaces
NAMESPACE        NAME                                  READY     STATUS    RESTARTS   AGE
default          my-nginx-8kmlz                        1/1       Running   0          11h
default          my-nginx-z8cxb                        1/1       Running   0          11h
kube-system      heapster-v10-vdc1v                    3/3       Running   0          11h
kube-system      kube-apiserver-10.122.0.20            1/1       Running   0          4d
kube-system      kube-controller-manager-10.122.0.20   1/1       Running   1          9h
kube-system      kube-dns-6iw3a                        4/4       Running   0          4d
kube-system      kube-proxy-10.122.0.20                1/1       Running   0          3d
kube-system      kube-proxy-10.122.42.163              1/1       Running   0          4d
kube-system      kube-proxy-10.122.43.138              1/1       Running   1          4d
kube-system      kube-scheduler-10.122.0.20            1/1       Running   1          4d

So heapster is running against my proxy and I can access

http://10.122.0.20:8080/api/v1/proxy/namespaces/kube-system/services/heapster/api/v1/model/namespaces/default/pods/my-nginx-8kmlz/stats

It returns stats about the pod.

I'm really not sure what i am missing.

Here is what the output of a autoscale looks like

[root@kube-master test] [dev] # kubectl get hpa
NAME       REFERENCE                              TARGET    CURRENT     MINPODS   MAXPODS   AGE
my-nginx   ReplicationController/my-nginx/scale   80%       <waiting>   1         5         22h

In my controller logs the only thing I really see is

W1224 18:27:43.425126       1 horizontal.go:185] Failed to reconcile my-nginx: failed to compute desired number of replicas based on CPU utilization for ReplicationController/default/my-nginx: failed to get cpu utilization: failed to get CPU consumption and request: some pods do not have request for cpu
-- Mike
kubernetes

Similar Questions

How to update/check the ubuntu kernel configuration?
Pod "mysql" is forbidden: no API token found for service account default/default
Permission Denied for Kubernetes Secrets with SELINUX Enabled
Kubernetes on AWS cloud provider

3 Answers

12/2/2019

Some time this happen because resource metrics not enabled.

you can verify with blow command:

kubectl top pod -n <namespace>

if you are getting pods then metrics is enabled:

https://kubernetes.io/docs/tasks/debug-application-cluster/resource-usage-monitoring/#resource-metrics-pipeline

-- Ramakant Chandrakar
Source: StackOverflow
12/2/2019

you need to assign cpu request / limit for pods in deployment file example

resources: requests: cpu: "100m"
limits: cpu: "250m"

-- Ramakant Chandrakar
Source: StackOverflow
12/30/2015

This error might be caused if you are missing a CPU request on your pods. You can confirm that by running the following command:

kubectl get dc $YOUR_DC -o yaml

In order to use the CPU autoscaling, you'll need to specify a CPU request under the resources section for your pod spec (CPU autoscaling is based on a percentage of the requested CPU). For example:

...
    spec:
      containers:
      - image: nginx
        name: nginx
        resources:
          requests:
            cpu: 400m
...
-- George
Source: StackOverflow