I've recently set up a Kubernetes cluster using GKE Version 1.19.9-gke.1900 for the purpose of spinning up groups of microservices in different namespaces. Each service's pod contains a service image and an Istio proxy image. The initial CPU resource request for each pod is 100 mCPU, but this is greater than most of services in a given namespace will ever use. This has caused the cluster to run out of available CPU for services to launch, even while I'm only utilizing around 25% of the total CPU available at any given time.
Vertical Auto-Scaling is disabled and I've tried setting resource limits of 25 mCPU using the Deployment, ResourceQuota and LimitRange manifests.
# First tried this
apiVersion: apps/v1
kind: Deployment
spec:
...
template:
...
spec:
containers:
- name: service-name
image: service-name:dev
resources:
requests:
cpu: 25m
memory: 128Mi
limits:
cpu: 2000m
memory: 512Mi
...
---
# Then this
apiVersion: v1
kind: LimitRange
metadata:
name: limit-range-limiter
spec:
limits:
- default:
cpu: 100m
memory: 512Mi
defaultRequest:
cpu: 25m
memory: 128Mi
type: Container
---
# Then this finally
apiVersion: v1
kind: ResourceQuota
metadata:
name: resource-limiter
spec:
hard:
requests.cpu: 25m
limits.cpu: 2000m
requests.memory: 128Mi
limits.memory: 512Mi
With the Deployment and LimitRange, instead of requesting just 25 mCPU for the service's pod, the pod will request 125 mCPU. And with the ResourceQuota, I receive an error that I must set the the cpu & memory requests and limits, despite it being in the manifest. This causes a replica failure and prevents my services from being launched.
I've also noticed each namespace has a ResourceQuota installed by default from GKE called gke-resource-quotas
that can't be edited. But if I apply a ResourceQuota manifest directly after the namespace has been created but before any service get deployed, the gke-resource-quotas
file doesn't get added.
Is it possible for pods to request anything lower than 100 mCPU on GKE?
EDIT: We managed to figure out what was going on. Istio was requesting 100 mCPU by default for the Pod. The actual config value was being set in the istio-sidecar-injector
ConfigMap. Once we changed that value, the Pods were able to start with a lower value.
Would like to extend what guillaume blaquiere said,It’s possible for pods to request anything lower than 100 mCPU . For example see, Resource requests and limits
1- The default request for CPU is 100m. This is too small for many applications, and is probably much smaller than the amount of CPU available on the node.
2- There is no default request for memory. A Pod with no default memory request could be scheduled onto a node without enough memory to run the Pod's workloads.
3- Setting too small a value for CPU or memory requests could cause too many Pods or a suboptimal combination of Pods to be scheduled onto a given node and reduce performance.
4-Setting too large a value for CPU or memory requests could cause the Pod to be unschedulable and increase the cost of the cluster's resources.
For additional information refer.