I'm trying to migrate a django app to google kubernetes engine and can't get it to work.

The response from the app is: Exception Value:
(1045, "Access denied for user 'root'@'cloudsqlproxy~[cloudsql instance ip]' (using password: NO)")

If I fire up a bash shell into the running container in which the django app lives, I can connect to the cloud sql instance.

I have my database credentials in environment variables but django doesn't seem to care about the username and password specifically.

django connection settings from inside the container:

from django.db import connection
connection.settings_dict
{'ENGINE': 'django.db.backends.mysql', 'NAME': 'aesh_db', 'USER': None, 'PASSWORD': None, 'HOST': '127.0.0.1', 'PORT': '3306', 'OPTIONS': {'charset': 'utf8mb4'}, 'TEST': {'CHARSET': 'utf8mb4', 'COLLATION': 'utf8mb4_unicode_ci', 'NAME': None, 'MIRROR': None}, 'ATOMIC_REQUESTS': False, 'AUTOCOMMIT': True, 'CONN_MAX_AGE': 0, 'TIME_ZONE': None}

the env vars are set correctly in the container, this works:

>> import _mysql
>> import os
>> conn = _mysql.connect(db=os.getenv('DB_NAME'), user=os.getenv('DB_USER'), passwd=os.getenv('DB_PASS'), host=os.getenv('DB_HOST'))

django db settings:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': os.getenv('DB_NAME'),
        'USER': os.getenv('DB_USER'),
        'PASSWORD': os.getenv('DB_PASS'),
        'HOST': os.getenv('DB_HOST'),
        'PORT': os.getenv('DB_PORT'),
        'OPTIONS': {
            # Tell MySQLdb to connect with 'utf8mb4' character set
            'charset': 'utf8mb4',
         },
        'TEST': {
            'CHARSET': 'utf8mb4',
            'COLLATION': 'utf8mb4_unicode_ci',
        }
    }
}

deployment.yaml:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: aesh-web
  labels:
    app: aesh-web
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: aesh-web
    spec:
      containers:
      - name: aesh-web
        image: gcr.io/[my-project]/aesh_web:1.0.0
        ports:
          - containerPort: 8000
        env:
          - name: LOAD_BALANCER_IP
            value: [service ip]
          - name: DB_HOST
            value: 127.0.0.1
          - name: DB_PORT
            value: "3306"
          - name: DB_NAME
            value: aesh_db
          - name: DB_USER
            valueFrom:
              secretKeyRef:
                name: cloudsql-db-credentials
                key: username
          - name: DB_PASS
            valueFrom:
              secretKeyRef:
                name: cloudsql-db-credentials
                key: password
      - name: cloudsql-proxy
        image: gcr.io/cloudsql-docker/gce-proxy:1.11
        command: ["/cloud_sql_proxy",
                  "-instances=[instance-connection-name]:aesh-web-db=tcp:3306",
                  "-credential_file=/secrets/cloudsql/credentials.json"]
        volumeMounts:
          - name: cloudsql-instance-credentials
            mountPath: /secrets/cloudsql
            readOnly: true
      volumes:
        - name: cloudsql-instance-credentials
          secret:
            secretName: cloudsql-instance-credentials