I don't understand why my horizontal pod autoscaler cannot read the CPU usage stats from my deployment.
A little about my environment:
Here is a list of the active HPAs:
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
api Deployment/api <unknown> / 70% 2 5 0 5h
web Deployment/web 0% / 70% 2 5 2 22h
As you can see only one of the HPAs can detect the resources usage for the deployment.
The web
deployment (the one that works):
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
labels:
name: web
name: web
namespace: default
spec:
replicas: 1
selector:
matchLabels:
name: web
template:
metadata:
labels:
name: web
spec:
containers:
- image: <private repository image path>
imagePullPolicy: IfNotPresent
name: node
ports:
- containerPort: 3000
protocol: TCP
resources:
limits:
cpu: 200m
memory: 200M
requests:
cpu: 200m
memory: 200M
The api deployment that does not work:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
labels:
name: api
name: api
namespace: default
spec:
replicas: 1
selector:
matchLabels:
name: api
template:
metadata:
labels:
app: api
spec:
containers:
image: <private repository image path>
imagePullPolicy: IfNotPresent
name: node
ports:
- containerPort: 3000
protocol: TCP
resources:
limits:
cpu: 250m
memory: 200M
requests:
cpu: 250m
memory: 200M
The web HPA (the one that works):
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
name: web
namespace: default
spec:
maxReplicas: 5
minReplicas: 2
scaleTargetRef:
apiVersion: apps/v1beta1
kind: Deployment
name: web
targetCPUUtilizationPercentage: 70
The api HPA (the one that doesn't work):
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
name: api
namespace: default
spec:
maxReplicas: 5
minReplicas: 2
scaleTargetRef:
apiVersion: apps/v1beta2
kind: Deployment
name: api
targetCPUUtilizationPercentage: 70
As you can see the deployments and hpa are almost identical, that's why I can't figure out why only one of the two works. Any help in troubleshooting this would be very appreciated
EDIT
I just tried to remove the working hpa and re-applying it and it doesn't work either. That makes me think it's a problem with heapster. I updated to the latest heapster version (1.5.0) but still no luck.
The reason could be because of no pods running:
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
api Deployment/api <unknown> / 70% 2 5 0 5h
web Deployment/web 0% / 70% 2 5 2 22h
As the number of replicas for "api" is ZERO which means the app is not running and for "web" it is TWO.
Try to check "kubectl get pods {namespace}" for running pods.
Everything started working again by restarting the kube-controller-manager pod.