Horizontal Pod Autoscaler can't read CPU usage

12/13/2017

I don't understand why my horizontal pod autoscaler cannot read the CPU usage stats from my deployment.

A little about my environment:

  • kubernetes 1.8.4 on AWS, deployed with k8s
  • 3 nodes and 1 master
  • heapster installed (version 1.5.0) and working (it writes data in influxdb and I can see it in grafana)

Here is a list of the active HPAs:

NAME       REFERENCE        TARGETS           MINPODS   MAXPODS   REPLICAS   AGE
api        Deployment/api   <unknown> / 70%   2         5         0          5h
web        Deployment/web   0% / 70%          2         5         2          22h

As you can see only one of the HPAs can detect the resources usage for the deployment.

The web deployment (the one that works):

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
labels:
    name: web
name: web
namespace: default
spec:
replicas: 1
selector:
    matchLabels:
    name: web
template:
    metadata:
    labels:
        name: web
    spec:
    containers:
    - image: <private repository image path>
        imagePullPolicy: IfNotPresent
        name: node
        ports:
        - containerPort: 3000
        protocol: TCP
        resources:
        limits:
            cpu: 200m
            memory: 200M
        requests:
            cpu: 200m
            memory: 200M

The api deployment that does not work:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
labels:
    name: api
name: api
namespace: default
spec:
replicas: 1
selector:
    matchLabels:
    name: api
template:
    metadata:
    labels:
        app: api
    spec:
    containers:
        image: <private repository image path>
        imagePullPolicy: IfNotPresent
        name: node
        ports:
        - containerPort: 3000
        protocol: TCP
        resources:
        limits:
            cpu: 250m
            memory: 200M
        requests:
            cpu: 250m
            memory: 200M

The web HPA (the one that works):

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
name: web
namespace: default
spec:
maxReplicas: 5
minReplicas: 2
scaleTargetRef:
    apiVersion: apps/v1beta1
    kind: Deployment
    name: web
targetCPUUtilizationPercentage: 70

The api HPA (the one that doesn't work):

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
name: api
namespace: default
spec:
maxReplicas: 5
minReplicas: 2
scaleTargetRef:
    apiVersion: apps/v1beta2
    kind: Deployment
    name: api
targetCPUUtilizationPercentage: 70

As you can see the deployments and hpa are almost identical, that's why I can't figure out why only one of the two works. Any help in troubleshooting this would be very appreciated

EDIT

I just tried to remove the working hpa and re-applying it and it doesn't work either. That makes me think it's a problem with heapster. I updated to the latest heapster version (1.5.0) but still no luck.

-- whites11
kubernetes

2 Answers

12/13/2017

The reason could be because of no pods running:

NAME       REFERENCE        TARGETS           MINPODS   MAXPODS   REPLICAS   AGE
api        Deployment/api   <unknown> / 70%   2         5         0          5h
web        Deployment/web   0% / 70%          2         5         2          22h

As the number of replicas for "api" is ZERO which means the app is not running and for "web" it is TWO.

Try to check "kubectl get pods {namespace}" for running pods.

-- Here_2_learn
Source: StackOverflow

12/14/2017

Everything started working again by restarting the kube-controller-manager pod.

-- whites11
Source: StackOverflow