I have a container based application that is used to receive load from a benchmark tool and generate memory and cpu utilization. So I created this deployment file to the app:
apiVersion: apps/v1
kind: Deployment
metadata:
name: slave-leech-deployment
spec:
replicas: 1
selector:
matchLabels:
app: slave-leech-deployment
template:
metadata:
labels:
app: slave-leech-deployment
spec:
containers:
- name: slave-leech
image: kewynakshlley/slave-leech
ports:
- containerPort: 3000
imagePullPolicy: Always
resources:
limits:
cpu: 100m
memory: 50Mi
requests:
cpu: 100m
memory: 50Mi
Also I have the service and the vertical pod autoscaler manifest:
Service:
kind: Service
metadata:
name: slave-leech-deployment
spec:
selector:
app: slave-leech-deployment
ports:
- port: 80
targetPort: 3000
type: LoadBalancer```
vertical pod auto scaler manifest:
``` apiVersion: autoscaling.k8s.io/v1beta2
kind: VerticalPodAutoscaler
metadata:
name: my-vpa
spec:
targetRef:
apiVersion: "apps/v1"
kind: Deployment
name: slave-leech-deployment
updatePolicy:
updateMode: "Auto"
When I run the benchmark tool I can see the cpu utilization reaching the limit but the VPA isn't triggered and the container is not recreated with more resources.
output of kubectl describe vpa my-vpa
Namespace: default
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"autoscaling.k8s.io/v1beta2","kind":"VerticalPodAutoscaler","metadata":{"annotations":{},"name":"my-vpa","namespace":"defaul...
API Version: autoscaling.k8s.io/v1beta2
Kind: VerticalPodAutoscaler
Metadata:
Creation Timestamp: 2020-01-21T22:59:28Z
Generation: 19
Resource Version: 20357
Self Link: /apis/autoscaling.k8s.io/v1beta2/namespaces/default/verticalpodautoscalers/my-vpa
UID: a5958daa-9274-4904-b706-bb81e4948599
Spec:
Target Ref:
API Version: apps/v1
Kind: Deployment
Name: slave-leech-deployment
Update Policy:
Update Mode: Recreate
Status:
Conditions:
Last Transition Time: 2020-01-21T23:00:07Z
Message: No pods match this VPA object
Reason: NoPodsMatched
Status: True
Type: NoPodsMatched
Last Transition Time: 2020-01-21T23:00:07Z
Status: True
Type: RecommendationProvided
Recommendation:
Container Recommendations:
Container Name: slave-leech
Lower Bound:
Cpu: 25m
Memory: 262144k
Target:
Cpu: 126m
Memory: 262144k
Uncapped Target:
Cpu: 126m
Memory: 262144k
Upper Bound:
Cpu: 1490m
Memory: 428775530
Events:
<none>
Pod's max CPU utilization is .097 which is less than 1 core, upperbound for your recommendation is 1490m (1.4 core), so VPA will not reschedule. The VerticalPodAutoscaler uses the lowerBound and upperBound recommendations to decide whether to delete a Pod and replace it with a new Pod. If a Pod has requests less than the lower bound or greater than the upper bound, the VerticalPodAutoscaler deletes the Pod and replaces it with a Pod that has the target recommendation
ffran09 is right with his/her answer but I would like to add some more info.
As already stated in my comment:
Try running this command to see the exact CPU requests on the pods:
kubectl get pod -n dev -o=custom-columns=NAME:.metadata.name,PHASE:.status.phase,CPU-REQUEST:.spec.containers\[0\].resources.requests.cpu
Than compare it with your configured recommendations.
This will help you to observe the actual resource utilization and therefore setup the proper limits for your VPA.
I also recommend you to use a metrics-server
for ingesting a pod metrics. Minikube includes it as a bundled add-on:
minikube addons enable metrics-server
For more details regarding that part and other vital elements of VPA I highly recommend you to go through these resources:
Please let me know if that helped.