Kubernetes scale rules for queue consumer with keda

12/22/2021

I have question about scaling rules for service processing tasks form queue. Let's say we have very simple and common situation:

  • service A push tasks to queue with rate ~100 tasks per second
  • service B process tasks with rate ~20 tasks per second

Scaling for service B with keda looks like simple solution:

  • configure trigger based on queue
  • set queueLenth to some random value (?) like 10

But in real world we have number of replicas jumping around 5 up and down all time, because of such flow:

  1. If we have 5 replicas - queue is empty
  2. if queue is empty - scale down
  3. if we have 4 replicas - queue is growing very fast
  4. keda set 5 or more replicas because queue can be very big
  5. goto step 1

Is it common problem. Is it possible to solve it?

Currently best that we can develop with keda is:

spec:
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleDown:
          policies:
          - periodSeconds: 600
            type: Pods
            value: 1
          stabilizationWindowSeconds: 1200
        scaleUp:
          policies:
          - periodSeconds: 1200
            type: Pods
            value: 1
          stabilizationWindowSeconds: 1
  maxReplicaCount: 10
  minReplicaCount: 1
  pollingInterval: 30
  scaleTargetRef:
    envSourceContainerName: celery
    name: celery
  triggers:
  - metadata:
      queueName: celery
      listLength: "2"
    type: redis
-- Danil Gusev
celery
hpa
keda
kubernetes

0 Answers