OCI runtime create failed: container_linux.go:345

2/21/2020

Currently we are migrating from GitLab to GitHub and we've decided to move the CI/CD process to GitHub actions. The pipeline process works like a charm but when GKE tries to spin up the newly pushed image it gives back this error:

'OCI runtime create failed: container_linux.go:345: starting container process caused "exec: \"/socket-server\": permission denied": unknown'

It's important to note here that this whole process was working on GitLab. Anyway, the GitHub workflow yaml file looks like this:

name: Build and deploy
on:
  - push
  - pull_request

jobs:
  build:
    name: Build
    runs-on: ubuntu-latest
    steps:
      - uses: fusion-engineering/setup-git-credentials@v2
        with:
          credentials: https://${{ secrets.MACHINE_ACCOUNT_ACCESS_TOKEN }}:x-oauth-basic@github.com/

      - name: Setup environment
        shell: bash
        run: |
          echo "::set-env name=GOPATH::${{ github.workspace }}/go"
          echo "::add-path::${{ github.workspace }}/go/bin"

      - name: Install Go
        uses: actions/setup-go@v1
        with:
          go-version: 1.12.4

      - name: Checkout code
        uses: actions/checkout@v2
        with:
          path: go/src/github.com/${{ github.repository }}

      - name: Prepare environment
        run: |
          cd $GOPATH/src/github.com/${{ github.repository }}
          make prepare

      - name: Format code
        run: |
          cd $GOPATH/src/github.com/${{ github.repository }}
          make fmt && git diff --exit-code

      - name: Lint code
        run: |
          cd $GOPATH/src/github.com/${{ github.repository }}
          make lint

      - name: Vet code
        run: |
          cd $GOPATH/src/github.com/${{ github.repository }}
          make vet

      - name: Test code
        run: |
          cd $GOPATH/src/github.com/${{ github.repository }}
          make cover

      - name: Build code
        run: |
          cd $GOPATH/src/github.com/${{ github.repository }}
          make build

      - name: Upload artifact
        uses: actions/upload-artifact@v1
        with:
          name: socket-server
          path: go/src/github.com/${{ github.repository }}/socket-server

  deploy:
    name: Deploy
    runs-on: ubuntu-latest
    needs: [build]
    if: contains(github.ref, 'refs/tags')

    steps:
      - name: Set release version
        run: echo ::set-env name=CI_COMMIT_TAG::${GITHUB_REF/refs\/tags\//}

      - name: Checkout code
        uses: actions/checkout@v2

      - name: Get artifact from build step
        uses: actions/download-artifact@v1
        with:
          name: socket-server

      - name: Set ci auth
        run: echo ::set-env name=CI_AUTH::$(cat ci_auth.json | base64)

      - uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
        with:
          service_account_key: ${{ env.CI_AUTH }}

      - name: Configure gcloud docker authentication
        run: |
          gcloud config set project foo
          gcloud auth configure-docker

      - name: Build, push and deploy container
        run: |
          bash deploy.sh

The deploy.sh file looks like this:

#!/bin/bash

if [[ -z "${CI_COMMIT_TAG}" ]]; then
    echo "CI_COMMIT_TAG is empty, this stage should not run"
    exit 0
fi

export ENV="stage"

if [[ "$CI_COMMIT_TAG" != "${CI_COMMIT_TAG%-release}" ]]; then
    export ENV="prod"
fi

echo "Current environment: $ENV"

make deploy

The deploy step in the Makefile looks like this:

deploy:
    ( echo "cat <<EOF" ; cat k8s.yml.template; ) | sh > k8s-${ENV}.yml
    docker build --no-cache \
    --build-arg RELEASE=${CI_COMMIT_TAG} \
    --build-arg ENV=${ENV} \
    -t gcr.io/foo/socket-server:${CI_COMMIT_TAG} .
    docker push gcr.io/foo/socket-server:${CI_COMMIT_TAG}
    gcloud container clusters get-credentials api-${ENV} --zone=europe-west1-b
    kubectl apply -f k8s-${ENV}.yml

And the Dockerfile looks like this:

FROM alpine:latest as certs
RUN apk --update add ca-certificates

FROM scratch
COPY --from=certs /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt

ARG RELEASE
ARG ENV

ADD ./socket-server /socket-server
ADD ./config.yml /config.yml
ADD ./dbconfig.yml /dbconfig.yml
ADD ./migrations /migrations

ENV SOCKET_SERVER_SENTRY_DSN https://foo@sentry.io/bar
ENV SOCKET_SERVER_SENTRY_RELEASE $RELEASE
ENV SOCKET_SERVER_SENTRY_ENVIRONMENT $ENV

CMD ["/socket-server", "--port", "9345", "--host", ""]

I have already tried to chmod +x socket-server on the pipeline and also in the Dockerfile (as suggested here). When I do it in the Dockerfile it fails with the following error:

Step 14/15 : RUN chmod +x socket-server
 ---> Running in 9c66aef0c35b
OCI runtime create failed: container_linux.go:349: starting container process caused "exec: \"/bin/sh\": stat /bin/sh: no such file or directory": unknown

Am I missing something on the GitHub workflow or is there anybody who is seeing something that I don't? Some help is appreciated!

-- Sefa
docker
github
github-actions
go
google-kubernetes-engine

2 Answers

2/28/2020

So I have found out that the uploaded artifact is put in a folder by the pipeline. So the socket-server was a directory and not the expected executable file. I fixed it by changing this:

- name: Get artifact from build step
        uses: actions/download-artifact@v1
        with:
          name: socket-server

to

- name: Get artifact from build step
        uses: actions/download-artifact@v1
        with:
          name: socket-server
          path: .
-- Sefa
Source: StackOverflow

2/21/2020

You can run a chmod in another stage that includes the chmod binary and then copy the file with the corrected permissions into your final scratch based stage:

FROM alpine:latest as certs
RUN apk --update add ca-certificates

FROM alpine:latest as binaries
COPY ./socket-server /socket-server
RUN chmod 755 /socket-server

FROM scratch
COPY --from=certs /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt

COPY --from=binaries /socket-server /
COPY ./config.yml /config.yml
COPY ./dbconfig.yml /dbconfig.yml
COPY ./migrations /migrations

ARG RELEASE
ARG ENV

ENV SOCKET_SERVER_SENTRY_DSN https://foo@sentry.io/bar
ENV SOCKET_SERVER_SENTRY_RELEASE $RELEASE
ENV SOCKET_SERVER_SENTRY_ENVIRONMENT $ENV

CMD ["/socket-server", "--port", "9345", "--host", ""]

Note that I've also switched from ADD to COPY since you don't want to unzip the binaries/yml files or pull them from a remote http server. And I've also moved the ARG entries down to where you use them to avoid breaking the cache prematurely (not an issue here, but would be with a RUN command).

-- BMitch
Source: StackOverflow