K8S cert-manager error creating acme challenge pods

2/23/2021

I've been trying for the last 3 days to setup cert-manager on a K8S cluster (v1.19.8) in an OpenStack environment with 1 master and 2 nodes. It worked before (like 1 month ago), but since I re-created the cluster, pod ACME challenges cannot be created due to this error:

Status:
  Presented:   false
  Processing:  true
  Reason:      pods "cm-acme-http-solver-" is forbidden: PodSecurityPolicy: unable to admit pod: []
  State:       pending
Events:
  Type     Reason        Age                    From          Message
  ----     ------        ----                   ----          -------
  Normal   Started       8m25s                  cert-manager  Challenge scheduled for processing
  Warning  PresentError  3m18s (x7 over 8m23s)  cert-manager  Error presenting challenge: pods "cm-acme-http-solver-" is forbidden: PodSecurityPolicy: unable to admit pod: []

I've tried different versions of the ingress-nginx, different versions of cert-manager, different versions of k8s, but to no avail. I'm getting crazy..., please help. Many thanks :)

Cluster setup

kubectl create namespace ingress-nginx && \
helm install ingress-nginx ingress-nginx/ingress-nginx -n ingress-nginx && \
kubectl create namespace cert-manager && \
helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --version v1.1.0 \
  --set installCRDs=true

Issuer

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: email@example.com
    preferredChain: "ISRG Root X1"
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
      - http01:
          ingress:
            class: nginx

Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: main-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/from-to-www-redirect: "true"
    cert-manager.io/issuer: "letsencrypt-prod"
spec:
  tls:
  - hosts:
      - host.com
    secretName: the-secret-name
  rules:
  - host: host.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: api-nginx
            port: 
              number: 80
-- Andrei Ciceu
cert-manager
kubernetes
nginx-ingress
openstack

2 Answers

2/17/2022

A year late, but adding another solution in case it helps others finding this. I had the same issue of the challenge pod being blocked by PSP, but really didn't want to have to recreate/reconfigure my cluster, so I eventually solved the issue by adding this to the helm chart values.yaml: https://github.com/cert-manager/cert-manager/blob/master/deploy/charts/cert-manager/values.yaml

  global:
    podSecurityPolicy:
      enabled: true 
      useAppArmor: false

In my case, this is part of a Gitlab deployment so I added it under the certmanager key, as follows:

certmanager:
  install: true
  global:
    podSecurityPolicy:
      enabled: true 
      useAppArmor: false

(tags for search: gitlab helm chart certmanager PodSecurityPolicy "unable to admit pod" blocked)

-- Ozone
Source: StackOverflow

2/25/2021

After some debugging and much help from the hosting provider, we found the problem and the solution.

We were using the latest (from master) version of Magnum/OpenStack, which got an update that installed by default a PodSecurityPolicy controller. That prevented ACME pods to be created by cert-manager.

Recreating the cluster without a policy controller solved the issue:

openstack coe cluster create \
  --cluster-template v1.kube1.20.4 \
  --labels \
admission_control_list="NodeRestriction,NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,PersistentVolumeClaimResize,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,RuntimeClass" \
  --merge-labels
  ...
-- Andrei Ciceu
Source: StackOverflow