On our cluster, we use the following PSP and ClusterRoleBinding for almost all our services:
apiVersion: extensions/v1beta1
kind: PodSecurityPolicy
metadata:
name: restricted
spec:
allowPrivilegeEscalation: false
fsGroup:
ranges:
- max: 65535
min: 1
rule: MustRunAs
requiredDropCapabilities:
- ALL
runAsUser:
rule: MustRunAsNonRoot
seLinux:
rule: RunAsAny
supplementalGroups:
ranges:
- max: 65535
min: 1
rule: MustRunAs
volumes:
- configMap
- emptyDir
- projected
- secret
- downwardAPI
- persistentVolumeClaim
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: psp:restricted
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: psp:restricted
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: system:authenticated
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: psp:restricted
rules:
- apiGroups:
- extensions
resourceNames:
- restricted
resources:
- podsecuritypolicies
verbs:
- use
In other words, a restrictive PSP that doesn't allow anything exciting. This works well for us and is a well-understood and mature configuration. However, when creating pods with the Airflow GKEPodOperator, we get the following error:
Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "e1ab15d888c52b90c2fea73fec3a75cd3d79125f21f7840e2e6ca3c00f3b3b16" network for pod "braintree-to-gcs-11bd354a": NetworkPlugin cni failed to set up pod "braintree-to-gcs-11bd354a_private" network: connection is unauthorized: pods "braintree-to-gcs-11bd354a" is forbidden: unable to validate against any pod security policy: []
Unless I badly miss my guess, our Pod specification is failing to validate against this PSP. However, I'm stuck for why this might be the case - it's essentially the same kind of container we routinely run in production without any issues at all. My only guess is that we haven't set the container securityContext (as Airflow doesn't permit this) but I don't think this is actually necessary for the Pod to validate against the PSP. The Pod specification created by Airflow is as follows:
apiVersion: v1
kind: Pod
metadata:
annotations:
# note that the pod is attempting to validate against a different PSP (but still failing)
kubernetes.io/psp: dynatrace-oneagent
name: braintree-to-gcs-11bd354a
namespace: private
spec:
affinity: {}
containers:
- args:
- "2011-02-19T00:00:00+00:00"
env:
REDACTED
image: gcr.io/prod-233014/c-braintree-etl:6ac00fdba4cf75ac3b13002d5d343f5427bbd44b
imagePullPolicy: IfNotPresent
name: base
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: default-token-pqmf6
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
imagePullSecrets:
- name: gcr-json-key
nodeName: gke-main-cluster-preemptible-nodes-1-e414988e-dn0d
priority: 0
restartPolicy: Never
schedulerName: default-scheduler
securityContext:
runAsUser: 65532
serviceAccount: default
serviceAccountName: default
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- name: default-token-pqmf6
secret:
defaultMode: 420
secretName: default-token-pqmf6