How to enforce policies for manifests in Kubernetes?

11/11/2019

I have built a self-service platform based on Kubernetes, where we create namespaces for each team and allow them to 'do whatever they want within the namespace' (we set resource limits so no one can kill the whole cluster).

However, now I want to implement some kind of standard across the organization. For example, I want every PodSpec to define its own resource limits, and I want every resource to have a label that specifies what application it belongs to.

Is there a mechanism that will allow the API server to check the manifests being applied against a set of rules, and if it fails the check the manifest is rejected.

For example, the following manifest would be rejected because it has neither a label nor are resource limits set.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80

But the following manifest would succeed because it satisfies all the rules:

apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment labels: app: foobar spec: selector: matchLabels: app: nginx replicas: 2 template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.7.9 ports: - containerPort: 80 resources: limits: cpu: "1" requests: cpu: "0.5"

-- dayuloli
kube-apiserver
kubernetes
kubernetes-apiserver

2 Answers

11/12/2019

You can enforce custom policies on resource manifests with Open Policy Agent (OPA) and Gatekeeper.

OPA is a general-purpose policy engine and Gatekeeper provides CRDs and a validating admission control webhook to define and enforce OPA policies in Kubernetes.

You can describe policies about literally every aspect of a resource manifest and Gatekeeper rejects any resource that violates violates a policy.

Here is a demo that shows how Gatekeeper works.

-- weibeld
Source: StackOverflow

11/11/2019

Is there a mechanism that will allow the API server to check the manifests being applied against a set of rules, and if it fails the check the manifest is rejected.

In general, this may be solved by an custom admission controller alternatively by a custom proxy. It depends on your needs and may not be so easy.

Resource limits by namespace

we create namespaces for each team and allow them to 'do whatever they want within the namespace' (we set resource limits so no one can kill the whole cluster).

I want every PodSpec to define its own resource limits

What you are looking for here is probably Limit Ranges per namespace, and possibly default values.

With Resource quotas, cluster administrators can restrict the resource consumption and creation on a namespace basis. Within a namespace, a Pod or Container can consume as much CPU and memory as defined by the namespace’s resource quota. There is a concern that one Pod or Container could monopolize all of the resources. Limit Range is a policy to constrain resource by Pod or Container in a namespace.

Mandatory Labels As what I know this is not possibly, yet

-- Jonas
Source: StackOverflow