GKE resource request can't be lowered past 100 mCPU

7/16/2021

I've recently set up a Kubernetes cluster using GKE Version 1.19.9-gke.1900 for the purpose of spinning up groups of microservices in different namespaces. Each service's pod contains a service image and an Istio proxy image. The initial CPU resource request for each pod is 100 mCPU, but this is greater than most of services in a given namespace will ever use. This has caused the cluster to run out of available CPU for services to launch, even while I'm only utilizing around 25% of the total CPU available at any given time.

Vertical Auto-Scaling is disabled and I've tried setting resource limits of 25 mCPU using the Deployment, ResourceQuota and LimitRange manifests.

# First tried this
apiVersion: apps/v1
kind: Deployment
spec:
  ...
  template:
    ...
    spec:
      containers:
        - name: service-name
          image: service-name:dev
          resources:
            requests:
              cpu: 25m
              memory: 128Mi
            limits:
              cpu: 2000m
              memory: 512Mi
      ...
---
# Then this
apiVersion: v1
kind: LimitRange
metadata:
  name: limit-range-limiter
spec:
  limits:
    - default:
        cpu: 100m
        memory: 512Mi
      defaultRequest:
        cpu: 25m
        memory: 128Mi
      type: Container
---
# Then this finally
apiVersion: v1
kind: ResourceQuota
metadata:
  name: resource-limiter
spec:
  hard:
    requests.cpu: 25m
    limits.cpu: 2000m
    requests.memory: 128Mi
    limits.memory: 512Mi

With the Deployment and LimitRange, instead of requesting just 25 mCPU for the service's pod, the pod will request 125 mCPU. And with the ResourceQuota, I receive an error that I must set the the cpu & memory requests and limits, despite it being in the manifest. This causes a replica failure and prevents my services from being launched.

I've also noticed each namespace has a ResourceQuota installed by default from GKE called gke-resource-quotas that can't be edited. But if I apply a ResourceQuota manifest directly after the namespace has been created but before any service get deployed, the gke-resource-quotas file doesn't get added.

Is it possible for pods to request anything lower than 100 mCPU on GKE?

EDIT: We managed to figure out what was going on. Istio was requesting 100 mCPU by default for the Pod. The actual config value was being set in the istio-sidecar-injector ConfigMap. Once we changed that value, the Pods were able to start with a lower value.

-- Alex Montolio
google-cloud-platform
google-kubernetes-engine
kubernetes

1 Answer

7/20/2021

Would like to extend what guillaume blaquiere said,It’s possible for pods to request anything lower than 100 mCPU . For example see, Resource requests and limits

1- The default request for CPU is 100m. This is too small for many applications, and is probably much smaller than the amount of CPU available on the node.

2- There is no default request for memory. A Pod with no default memory request could be scheduled onto a node without enough memory to run the Pod's workloads.

3- Setting too small a value for CPU or memory requests could cause too many Pods or a suboptimal combination of Pods to be scheduled onto a given node and reduce performance.

4-Setting too large a value for CPU or memory requests could cause the Pod to be unschedulable and increase the cost of the cluster's resources.

For additional information refer.

-- Fariya Rahmat
Source: StackOverflow