K
Q

'realpath(): Permission denied' when running dotnet ef command as non-root in dotnet/sdk:3.1-bullseye container on k8s

1/24/2022

Problem

A container meant to run database migrations works locally, on Docker for Mac, but fails on Kubernetes, repeatedly logging 

realpath(): Permission denied
Failed to resolve full path of the current executable [/proc/self/exe]

Reproduction

I have an image built from the following Dockerfiles (other descendents of base not included)

#Dockerfile.base

FROM mcr.microsoft.com/dotnet/sdk:3.1-bullseye

RUN dotnet tool install --global dotnet-ef
ENV PATH="$PATH:/root/.dotnet/tools"

ADD event_processor/dotnet /app
ADD classification_registry/topic_registry  /app/topic_registry

WORKDIR /app

and 

#Dockerfile.migrate
FROM app-base

ENV DOTNET_CLI_HOME=/app
RUN addgroup --system --gid 1000 app \
    && adduser --home /app --system --uid 2000 --ingroup app --shell /bin/sh appmigrate
RUN chown -R appmigrate /app
RUN chown -R appmigrate /root/.dotnet/tools
RUN chown -R appmigrate /tmp
USER appmigrate
ENV PATH="$PATH:/app/.dotnet/tools"
RUN dotnet tool install --global dotnet-ef

so if we let unique-image-ref be a unique tag for this built image, I am able to run the container locally, as I expect with 

$ docker run -it --rm --user 2000 unique-image-ref dotnet-ef database update
Build started...
Build succeeded.
Configuring DB Access for migrations...
No migrations were applied. The database is already up to date.
Done.

So far so good. The problem arises in the Kubernetes cluster, when a Job is configured to run this container, with the following container definition

containers:
        - name: my-app-migration
          image: unique-image-ref
          imagePullPolicy: Always
          workingDir: /app
          command: ["dotnet-ef"]
          args:
            - database
            - update
          envFrom:
            - configMapRef:
                name: app-conf
            - secretRef:
                name: app-secret
      restartPolicy: OnFailure
      securityContext:
        runAsNonRoot: true
        runAsUser: 2000

When I examine the logs from the container, I see nothing but the error (at the top of this post).

Any suggestions would be welcomed.

-- Ben
.net-core
c#
docker
entity-framework
kubernetes

Similar Questions

setting up rabbitmq in k8s with nginx ingress controller
http request always blocked in container of k8s cluster pod
RBAC role to manage single pod with dynamic name
kubernetes pull image getting error "ImagePullBackOff"
Helm doesn't take values YAMLs

1 Answer

1/25/2022

I did find a solution, though I cannot articulate exactly how this change effects the fix.

    containers:
        - name: my-app-migration
          image: unique-image-ref
          imagePullPolicy: Always
          workingDir: /app
          command: ["/bin/sh"]
          args: ["-c", "dotnet-ef --project=Seismic.NotificationEventProcessor.RecoveryQueue database update"]
          envFrom:
            - configMapRef:
                name: app-conf
            - secretRef:
                name: app-secret
      restartPolicy: OnFailure
      securityContext:
        runAsNonRoot: true
        runAsUser: 2000

So the fix is to invoke the command from a new shell, and pass the command to execute as a string (hence -c). But realistically, the details of this escape me.

I hope this helps someone else out with a very obscure error message. If anyone cares to expand on the underlying behavior, I would be glad to understand the relationships among docker, kubernetes, and bash in this regard.

-- Ben
Source: StackOverflow