K
Q

Kubernetes autoscaling not working - showing unkown

4/4/2019

There are a lot of questions about this topic and I tried a lot of stuff but it's still not working.

I'm new to Kubernetes. I have a Kubernetes Cluster with 2 Nodes (1 PC, 1 VM on a different PC), 1 Master, 1 Node. I started a website (Docker Image) and the Pod is running on the Node (not master). Now I wanted to autoscale the Pod, this is what I did:

  1. Set --request='cpu=50m' to the pod
  2. Created hpa: kubectl autoscale deployment testwebsite --min=1 --max=4 --cpu-percent=25
  3. Used: kubectl get hpa -w

And now there is the problem that the output shows as current value. I read a lot that I have to assign the request to the pod, as I did. Checked it via: kubectl get pod testwebsite --out=yaml

I also created the metrics-server version 1.8+. Made sure it's running via: kubectl get pods --all-namespaces

After that I tried to debug the HPA and looked into it via:

kubectl describe hpa testwebsite

That showed me this:

Name:                                                  testwebsite
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           <none>
CreationTimestamp:                                     Thu, 04 Apr 2019 14:08:57 +0200
Reference:                                             Deployment/testwebsite
Metrics:                                               ( current / target )   resource cpu on pods  (as a percentage of request):  <unknown> / 25%
Min replicas:                                          1
Max replicas:                                          4
Deployment pods:                                       1 current / 0 desired
Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
 AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
 ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: unable to get metrics for resource cpu: no metrics returned from resource metrics API
Events:
  Type     Reason                        Age                   From                       Message
  ----     ------                        ----                  ----                       -------
  Warning  FailedComputeMetricsReplicas  7m24s (x12 over 10m)  horizontal-pod-autoscaler  failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
  Warning  FailedGetResourceMetric       5s (x41 over 10m)     horizontal-pod-autoscaler  unable to get metrics for resource cpu: no metrics returned from resource metrics API

After 10+ mins it's not showing the current value and it won't scale up even when the cpu has 50-100% usage.

As I said I'm new to Kubernetes and I really hope someone can help me.

Best regards,

Nico aka Myridor

EDIT: Using Ubuntu 18.04 LTS on both Nodes | Kubernetes Version: 1.14.0

-- Myridor
autoscaling
kubernetes
kubernetes-hpa
metrics

Similar Questions

Is there a way to share a configMap in kubernetes between namespaces?
How to host multiple applications in one Istio service mesh?
RabbitMQ, .NET Core and Kubernetes (configuration)
Problem listing Google Container Registry images via Docker Registry HTTP API V2
command restriction in container Kubernetes

2 Answers

4/5/2019

are your sure on kubernetes metrics server & metrics api is running? which is necessary first to and it use to check cpu usage and limits

You can remove the LIMITS from your deployments if any there and try it.

-- Harsh Manvar
Source: StackOverflow
4/5/2019

The issue is not in HPS itself but in metrics server, which is not able to scrape metrics.

I recreated the issue by cloning metrics-server git and then creating it by kubectl create -f deploy/1.8+/.

Next I edited the metrics server deployment by running: kubectl edit deployment metrics-server -n kube-system

Under spec: -> containers: add following flag:

spec:
      containers:
      - command:
        - /metrics-server
        - --kubelet-insecure-tls

As described in git:

--kubelet-insecure-tls: skip verifying Kubelet CA certificates. Not recommended for production usage, but can be useful in test clusters with self-signed Kubelet serving certificates.

After the deployment is updated and metrics server scraped the metrics you should see HPA updated with current targets. Default scrape interval is 60s, you can change it by adding following flag to the above --metric-resolution=10s <- interval set to 10s.

-- MWZ
Source: StackOverflow