We are encountering some Internal Server errors (500) on our web services API, seemingly when new pods are introduced or deleted as part of our autoscaler configuration.
Naturally, this sort of thing is very hard to troubleshoot as you are not always able to control the moments pods start and stop.
Is there a mechanism one could use to instruct the cluster to scale up and down upon?
Our hpa.yaml for interest.
kind: HorizontalPodAutoscaler
apiVersion: autoscaling/v2beta2
metadata:
name: xyz
spec:
minReplicas: 1
maxReplicas: 12
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: xyz
behavior:
scaleUp:
stabilizationWindowSeconds: 0
policies:
- type: Percent
value: 100
periodSeconds: 15
scaleDown:
stabilizationWindowSeconds: 300
policies:
- type: Percent
value: 50
periodSeconds: 60
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: 85If you're interested in causing your HPA to forcibly create or destroy Pods for debugging purposes, you can use custom metrics in your Horizontal Pod Autoscaler .yaml definition.
If the problem is that newly created Pods cause errors, you can implement readiness probes in the Pod definition that perform an httpGet check. That way, you can avoid redirecting traffic on faulty Pods until the probe check returns a status = 200.