I have created a k3s cluster on some raspberry pis and am currently struggling to use the traefik service mesh to resolve to my domain.

I have previously got it working but upon recreating my cluster I cannot get pass the following problem:

The error message I am encountering in the traefik pod is the following:

Failed to watch *v1alpha1.IngressRouteUDP: failed to list *v1alpha1.IngressRouteUDP: the server could not find the requested resource 

This error message is repeated many times with different resource types.

I am also getting the following error when trying to get a tls certificate from lets encrypt:

Waiting for HTTP-01 challenge propagation: wrong status code '404', expected '200

I have created a clusterissuer for a staging certificate with the following YAML:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
  namespace: cert-manager
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: myemail@example.com
    privateKeySecretRef:
      name: letsencrypt-staging
    solvers:
    - http01:
        ingress:
          class: traefik

This appears to be working and returns the status READY=true

I have then requested a staging certificate with the YAML below:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: example-com
  namespace: cert-manager
  annotations:
    certmanager.k8s.io/cluster-issuer: letsencrypt-staging
spec:
  secretName: example-com-tls
  issuerRef:
    name: letsencrypt-staging
    kind: ClusterIssuer
  commonName: example.com
  dnsNames:
  - example.com

This is where I encounter the 404 error which is outputted by the challenge.

I also have the following arguments in my traefik deployment:

      - --certificatesresolvers.myresolver.acme.email=myemail@example.com
        - --global.checknewversion
        - --global.sendanonymoususage
        - --entryPoints.traefik.address=:9000/tcp
        - --entryPoints.web.address=:8000/tcp
        - --entryPoints.websecure.address=:8443/tcp
        - --api.dashboard=true
        - --ping=true
        - --providers.kubernetescrd
        - --providers.kubernetesingress
        - --providers.kubernetesingress.ingressendpoint.publishedservice=kube-system/traefik
        - --entrypoints.websecure.http.tls=true
        - --certificatesresolvers.default.acme.tlschallenge
        - --certificatesresolvers.default.acme.storage=acme.json

I am stumped. Have spent over a week trying to fix this and am sure it is something simple I am missing but I cannot work it out. Any help is greatly appreciated. Thank you.

Sample of logs from traefik pod:

1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.3/tools/cache/reflector.go:105: Failed to list *v1alpha1.TraefikService: traefikservices.traefik.containo.us is forbidden: User "system:serviceaccount:kube-system:traefik-ingress-controller" cannot list resource "traefikservices" in API group "traefik.containo.us" at the cluster scope
E1019 11:15:04.610288       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.3/tools/cache/reflector.go:105: Failed to list *v1.Endpoints: endpoints is forbidden: User "system:serviceaccount:kube-system:traefik-ingress-controller" cannot list resource "endpoints" in API group "" at the cluster scope
E1019 11:15:04.610542       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.3/tools/cache/reflector.go:105: Failed to list *v1.Service: services is forbidden: User "system:serviceaccount:kube-system:traefik-ingress-controller" cannot list resource "services" in API group "" at the cluster scope
E1019 11:15:04.610902       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.3/tools/cache/reflector.go:105: Failed to list *v1beta1.Ingress: ingresses.extensions is forbidden: User "system:serviceaccount:kube-system:traefik-ingress-controller" cannot list resource "ingresses" in API group "extensions" at the cluster scope
E1019 11:15:04.610959       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.3/tools/cache/reflector.go:105: Failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:kube-system:traefik-ingress-controller" cannot list resource "secrets" in API group "" at the cluster scope
E1019 11:15:04.658001       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.3/tools/cache/reflector.go:105: Failed to list *v1alpha1.IngressRouteTCP: ingressroutetcps.traefik.containo.us is forbidden: User "system:serviceaccount:kube-system:traefik-ingress-controller" cannot list resource "ingressroutetcps" in API group "traefik.containo.us" at the cluster scope
E1019 11:15:04.861684       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.3/tools/cache/reflector.go:105: Failed to list *v1alpha1.IngressRoute: ingressroutes.traefik.containo.us is forbidden: User "system:serviceaccount:kube-system:traefik-ingress-controller" cannot list resource "ingressroutes" in API group "traefik.containo.us" at the cluster scope
E1019 11:15:05.060807       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.3/tools/cache/reflector.go:105: Failed to list *v1alpha1.IngressRouteUDP: ingressrouteudps.traefik.containo.us is forbidden: User "system:serviceaccount:kube-system:traefik-ingress-controller" cannot list resource "ingressrouteudps" in API group "traefik.containo.us" at the cluster scope
E1019 11:15:05.278868       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.3/tools/cache/reflector.go:105: Failed to list *v1alpha1.Middleware: middlewares.traefik.containo.us is forbidden: User "system:serviceaccount:kube-system:traefik-ingress-controller" cannot list resource "middlewares" in API group "traefik.containo.us" at the cluster scope