Gunicorn issues on gcloud. Memory faults and restarts thread

9/1/2019

I am deploying a django application to gcloud using gunicorn without nginx.

Running the container locally works fine, the application boots and does a memory consuming job on startup in its own thread (building a cache). Approx. 900 MB of memory is used after the job is finished.

Gunicorn is started with:CMD gunicorn -b 0.0.0.0:8080 app.wsgi:application -k eventlet --workers=1 --threads=4 --timeout 1200 --log-file /gunicorn.log --log-level debug --capture-output --worker-tmp-dir /dev/shm

Now I want to deploy this to gcloud. Creating a running container with the following manifest:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: app
  namespace: default
spec:
  selector:
    matchLabels:
      run: app
  template:
    metadata:
      labels:
      run: app
    spec:
      containers:
      - image: gcr.io/app-numbers/app:latest
        imagePullPolicy: Always
        resources:
          limits:
            memory: "2Gi"
          requests:
            memory: "2Gi"        
        name: app
        ports:
        - containerPort: 8080
          protocol: TCP

Giving the container 2 GB of memory.

Looking at the logs, guniucorn is booting workers [2019-09-01 11:37:48 +0200] [17] [INFO] Booting worker with pid: 17

Using free -m in the container shows the memory slowly being consumed and dmesg shows:

[497886.626932] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
[497886.636597] [1452813]     0 1452813      256        1       4       2        0          -998 pause
[497886.646332] [1452977]     0 1452977      597      175       5       3        0           447 sh
[497886.656064] [1452989]     0 1452989    10195     7426      23       4        0           447 gunicorn
[497886.666376] [1453133]     0 1453133      597      360       5       3        0           447 sh
[497886.676959] [1458304]     0 1458304   543235   520309    1034       6        0           447 gunicorn
[497886.686727] Memory cgroup out of memory: Kill process 1458304 (gunicorn) score 1441 or sacrifice child
[497886.697411] Killed process 1458304 (gunicorn) total-vm:2172940kB, anon-rss:2075432kB, file-rss:5804kB, shmem-rss:0kB
[497886.858875] oom_reaper: reaped process 1458304 (gunicorn), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

What could be going on creating a memory leak on gcloud and not locally?

-- OMA
django
docker
gcloud
gunicorn
kubernetes

0 Answers