Pod serviceaccount not using defined PSP profile

4/26/2020

I have the following serviceaccount, role & role binding defined:

    # role.yaml
    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      name: concourse-worker
      namespace: k8s-01
    rules:
    - apiGroups: ['policy']
      resources: ['podsecuritypolicies']
      verbs: ['use']
      resourceNames:
        - wcp-privileged-psp
    ---
    # role-binding.yaml
    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      name: concourse-worker
      namespace: k8s-01
      labels:
        app: concourse-worker
        release: concourse
    subjects:
    - kind: ServiceAccount
      name: concourse-worker
      namespace: k8s-01
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: Role
      name: concourse-worker
    ---
    # service-account.yaml
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: concourse-worker
      namespace: k8s-01
      labels:
        app: concourse-worker
        release: concourse

To confirm the serviceaccount has access to the wcp-privileged-psp PSP I ran below:

» kubectl --as=system:serviceaccount:k8s-01:concourse-worker auth can-i use podsecuritypolicy/wcp-privileged-psp
Warning: resource 'podsecuritypolicies' is not namespace scoped in group 'policy'
yes

Deploying my stateful application:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: concourse-worker
  labels:
    app: concourse-worker
    release: "concourse"
spec:
  serviceName: concourse-worker
  replicas: 2
  selector:
    matchLabels:
      app: concourse-worker
      release: concourse
  template:
    metadata:
      labels:
        app: concourse-worker
        release: "concourse"
        tier: middletier
    spec:
      serviceAccountName: concourse-worker
      terminationGracePeriodSeconds: 60
      initContainers:
        - name: concourse-worker-init-rm
          image: "x.x.x.x/k8s-01/concourse:6.0.0"
          imagePullPolicy: "IfNotPresent"
          securityContext:
           privileged: true
          command:
            - /bin/bash
          args:
            - -ce
            - |-
              for v in $((btrfs subvolume list --sort=-ogen "/concourse-work-dir" || true) | awk '{print $9}'); do
                (btrfs subvolume show "/concourse-work-dir/$v" && btrfs subvolume delete "/concourse-work-dir/$v") || true
              done
              rm -rf "/concourse-work-dir/*"
          volumeMounts:
            - name: concourse-work-dir
              mountPath: "/concourse-work-dir"
      containers:
        - name: concourse-worker
          image: "x.x.x.x/k8s-01/concourse:6.0.0"
          imagePullPolicy: "IfNotPresent"
          args:
            - worker
          livenessProbe:
            failureThreshold: 5
            httpGet:
              path: /
              port: worker-hc
            initialDelaySeconds: 10
            periodSeconds: 15
            timeoutSeconds: 3
          lifecycle:
            preStop:
              exec:
                command:
                  - "/bin/bash"
                  - "/pre-stop-hook.sh"
          env:
            - name: CONCOURSE_SWEEP_INTERVAL
              value: "30s"
            - name: CONCOURSE_CONNECTION_DRAIN_TIMEOUT
              value: "1h"
            - name: CONCOURSE_HEALTHCHECK_BIND_IP
              value: "0.0.0.0"
            - name: CONCOURSE_HEALTHCHECK_BIND_PORT
              value: "8888"
            - name: CONCOURSE_HEALTHCHECK_TIMEOUT
              value: "5s"
            - name: CONCOURSE_DEBUG_BIND_IP
              value: "127.0.0.1"
            - name: CONCOURSE_DEBUG_BIND_PORT
              value: "7776"
            - name: CONCOURSE_WORK_DIR
              value: "/concourse-work-dir"
            - name: CONCOURSE_BIND_IP
              value: "127.0.0.1"
            - name: CONCOURSE_BIND_PORT
              value: "7777"
            - name: CONCOURSE_LOG_LEVEL
              value: "debug"
            - name: CONCOURSE_TSA_HOST
              value: "concourse-web:2222"
            - name: CONCOURSE_TSA_PUBLIC_KEY
              value: "/concourse-keys/host_key.pub"
            - name: CONCOURSE_TSA_WORKER_PRIVATE_KEY
              value: "/concourse-keys/worker_key"
            - name: CONCOURSE_BAGGAGECLAIM_LOG_LEVEL
              value: "info"
            - name: CONCOURSE_BAGGAGECLAIM_BIND_IP
              value: "127.0.0.1"
            - name: CONCOURSE_BAGGAGECLAIM_BIND_PORT
              value: "7788"
            - name: CONCOURSE_BAGGAGECLAIM_DEBUG_BIND_IP
              value: "127.0.0.1"
            - name: CONCOURSE_BAGGAGECLAIM_DEBUG_BIND_PORT
              value: "7787"
            - name: CONCOURSE_BAGGAGECLAIM_DRIVER
              value: "naive"
            - name: CONCOURSE_BAGGAGECLAIM_BTRFS_BIN
              value: "btrfs"
            - name: CONCOURSE_BAGGAGECLAIM_MKFS_BIN
              value: "mkfs.btrfs"
            - name: CONCOURSE_VOLUME_SWEEPER_MAX_IN_FLIGHT
              value: "5"
            - name: CONCOURSE_CONTAINER_SWEEPER_MAX_IN_FLIGHT
              value: "5"
          ports:
            - name: worker-hc
              containerPort: 8888
          resources:
            requests:
              cpu: 100m
              memory: 512Mi
          securityContext:
            privileged: true
          volumeMounts:
            - name: concourse-keys
              mountPath: "/concourse-keys"
              readOnly: true
            - name: concourse-work-dir
              mountPath: "/concourse-work-dir"
            - name: pre-stop-hook
              mountPath: /pre-stop-hook.sh
              subPath: pre-stop-hook.sh
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              topologyKey: kubernetes.io/hostname
              labelSelector:
                matchLabels:
                  app: concourse-worker
                  release: "concourse"
      volumes:
        - name: pre-stop-hook
          configMap:
            name: concourse-worker
        - name: concourse-keys
          secret:
            secretName: concourse-worker
            defaultMode: 0400
            items:
              - key: host-key-pub
                path: host_key.pub
              - key: worker-key
                path: worker_key
        - name: concourse-work-dir
          emptyDir:
            sizeLimit: 20Gi
  updateStrategy:
    type: RollingUpdate
  podManagementPolicy: Parallel

However when it's deployed and I check it's PSP it's not showing "wcp-privileged-psp" as it should but the default "wcp-default-psp"

 »?  kubectl describe pods concourse-worker-0                                                                                            /opt/k8s/apps/concourse
Name:         concourse-worker-0
Namespace:    k8s-01
Priority:     0
Node:         x.x.x.x
Start Time:   Sun, 26 Apr 2020 15:51:15 -0400
Labels:       app=concourse-worker
              controller-revision-hash=concourse-worker-6847cb88c5
              release=concourse
              statefulset.kubernetes.io/pod-name=concourse-worker-0
              tier=middletier
Annotations:  
              kubernetes.io/psp: **wcp-default-psp**
              mac: xxx
              vlan: None
-- Alex Willander
kubernetes

1 Answer

4/27/2020

As documented here the policies are chosen in this order.

  1. PodSecurityPolicies which allow the pod as-is, without changing defaults or mutating the pod, are preferred. The order of these non-mutating PodSecurityPolicies doesn’t matter.
  2. If the pod must be defaulted or mutated, the first PodSecurityPolicy (ordered by name) to allow the pod is selected.

In this case wcp-default-psp comes before wcp-privileged-psp in the order.

-- Arghya Sadhu
Source: StackOverflow