vertical pod autoscaler not working when receiving load from a benchmark tool

1/21/2020

I have a container based application that is used to receive load from a benchmark tool and generate memory and cpu utilization. So I created this deployment file to the app:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: slave-leech-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: slave-leech-deployment
  template:
    metadata:
      labels:
        app: slave-leech-deployment
    spec:
      containers:
      - name: slave-leech
        image: kewynakshlley/slave-leech
        ports:
          - containerPort: 3000
        imagePullPolicy: Always
        resources:
          limits:
            cpu: 100m
            memory: 50Mi
          requests:
            cpu: 100m
            memory: 50Mi

Also I have the service and the vertical pod autoscaler manifest:

Service:

kind: Service
metadata:
  name: slave-leech-deployment
spec:
  selector:
    app: slave-leech-deployment
  ports:
    - port: 80
      targetPort: 3000
  type: LoadBalancer```


vertical pod auto scaler manifest:

``` apiVersion: autoscaling.k8s.io/v1beta2
kind: VerticalPodAutoscaler
metadata:
  name: my-vpa
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind:       Deployment
    name:       slave-leech-deployment
  updatePolicy:
    updateMode: "Auto"

When I run the benchmark tool I can see the cpu utilization reaching the limit but the VPA isn't triggered and the container is not recreated with more resources.

cpu utilization

output of kubectl describe vpa my-vpa

Namespace:    default
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"autoscaling.k8s.io/v1beta2","kind":"VerticalPodAutoscaler","metadata":{"annotations":{},"name":"my-vpa","namespace":"defaul...
API Version:  autoscaling.k8s.io/v1beta2
Kind:         VerticalPodAutoscaler
Metadata:
  Creation Timestamp:  2020-01-21T22:59:28Z
  Generation:          19
  Resource Version:    20357
  Self Link:           /apis/autoscaling.k8s.io/v1beta2/namespaces/default/verticalpodautoscalers/my-vpa
  UID:                 a5958daa-9274-4904-b706-bb81e4948599
Spec:
  Target Ref:
    API Version:  apps/v1
    Kind:         Deployment
    Name:         slave-leech-deployment
  Update Policy:
    Update Mode:  Recreate
Status:
  Conditions:
    Last Transition Time:  2020-01-21T23:00:07Z
    Message:               No pods match this VPA object
    Reason:                NoPodsMatched
    Status:                True
    Type:                  NoPodsMatched
    Last Transition Time:  2020-01-21T23:00:07Z
    Status:                True
    Type:                  RecommendationProvided
  Recommendation:
    Container Recommendations:
      Container Name:  slave-leech
      Lower Bound:
        Cpu:     25m
        Memory:  262144k
      Target:
        Cpu:     126m
        Memory:  262144k
      Uncapped Target:
        Cpu:     126m
        Memory:  262144k
      Upper Bound:
        Cpu:     1490m
        Memory:  428775530
Events: 
         <none>
-- Kewyn Akshlley
autoscaling
devops
infrastructure
kubernetes
kubernetes-pod

2 Answers

1/22/2020

Pod's max CPU utilization is .097 which is less than 1 core, upperbound for your recommendation is 1490m (1.4 core), so VPA will not reschedule. The VerticalPodAutoscaler uses the lowerBound and upperBound recommendations to decide whether to delete a Pod and replace it with a new Pod. If a Pod has requests less than the lower bound or greater than the upper bound, the VerticalPodAutoscaler deletes the Pod and replaces it with a Pod that has the target recommendation

Vertical Pod Autoscaling

Understanding Resource Limits in Kubernetes

-- ffran09
Source: StackOverflow

2/6/2020

ffran09 is right with his/her answer but I would like to add some more info.

As already stated in my comment:

Try running this command to see the exact CPU requests on the pods:

kubectl get pod -n dev -o=custom-columns=NAME:.metadata.name,PHASE:.status.phase,CPU-REQUEST:.spec.containers\[0\].resources.requests.cpu 

Than compare it with your configured recommendations.

This will help you to observe the actual resource utilization and therefore setup the proper limits for your VPA.

I also recommend you to use a metrics-server for ingesting a pod metrics. Minikube includes it as a bundled add-on:

minikube addons enable metrics-server

For more details regarding that part and other vital elements of VPA I highly recommend you to go through these resources:

Please let me know if that helped.

-- OhHiMark
Source: StackOverflow