I'm running a flask application that serves an ML model that loads in a wordembeddings file (2GB) on k8s. This file is being loaded with gcsfuse and we have the application running for about 2 years.

Since a recent restart of the pod, this setup is not working anymore while nothing has changed in our code/deployment settings. While debugging, I notice that we even have the problem with the following dockerfile that does not even use the python script:

Dockerfile:

FROM levkuznetsov/gcsfuse-docker

RUN apt-get update && apt-get install -y

build-essential

COPY . /app

WORKDIR /app

RUN /bin/bash -c "mkdir -p /app/wordembeddingtest"

COPY ./serviceacc.json /

ADD /serviceacc.json /etc/gcloud/serviceacc.json

ADD /serviceacc.json /etc/gcloud/service-account.json

EXPOSE 8080

ENTRYPOINT ["/bin/bash", "-c", "gcsfuse bucket_name wordembeddingtest ; ls wordembeddingtest"]

What is even more strange, is that we have other deployments that use the same set-up, and they can be restarted and still work..

The logs show the following error:

And with --foreground --debug_invariants --debug_http --debug_gcs --debug_fuse we get the following:

What I have checked so far:

Service account permissions are ok

What I have tried so far:

Different storage bucket gcsfuse command with Implicit-dirs, -o allow_other Different kubernetes cluster Other mount folder locations