K
Q

I made django application but can't orchestration in GKE

3/22/2020

I made django appilcation in reference to official manual.

https://cloud.google.com/python/django/kubernetes-engine?hl=ja

After pushing the Docker image, I got bellow message in EXTERNAL-IP page.

Django Version: 2.2.5
Exception Type: OperationalError
Exception Value:     (2013, "Lost connection to MySQL server at 'reading initial communication packet', system error: 0")

When I develop in local, I can connect to Cloud SQL instance. But, it seems that GKE resource can't connect to Cloud SQL instance.

I think I made secret for instance-level access and database access correctly according to manual.

When I create secret for instance-level access, I type bellow command.

kubectl create secret generic cloudsql-oauth-credentials --from-file=credentials.json=appname-0000-*******.json

When I create secret for database access, I type bellow command.

kubectl create secret generic cloudsql --from-literal=username=user --from-literal=password=***

I edited in polls.yaml.

In polls.yaml, I edited like bellow.

  # [START proxy_container]
  - image: gcr.io/cloudsql-docker/gce-proxy:1.05
    name: cloudsql-proxy
    command: ***

In settings.py, I coded about DATABASES like bellow.

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'polls',
        'USER': os.getenv('DATABASE_USER'),
        'PASSWORD': os.getenv('DATABASE_PASSWORD'),
        'HOST': '127.0.0.1',
        'PORT': '5432',
    }
}

What can cause this error? Please let me know the solution!

When I enter

ss -ntlp | grep 5432

I get bellow.

root@polls-55456ddbbb-6vg8g:/home/vmagent/app# ss -ntlp | grep 5432
LISTEN     0      1024   127.0.0.1:5432                     *:*

root@polls-55456ddbbb-r9l9x:/home/vmagent/app# ss -ntlp | grep 5432
LISTEN     0      1024   127.0.0.1:5432                     *:*

root@polls-55456ddbbb-wz4ml:/home/vmagent/app# ss -ntlp | grep 5432
LISTEN     0      1024   127.0.0.1:5432                     *:*

Unfortunately it is not helpful.

I output pod's log.

kubectl describe pods polls-55456ddbbb-6vg8g

Name:           polls-55456ddbbb-6vg8g
Namespace:      default
Priority:       0
Node:           gke-polls-default-pool-2c3ca2d7-9m9d/10.146.0.29
Start Time:     Mon, 23 Mar 2020 18:11:03 +0900
Labels:         app=polls
                pod-template-hash=55456ddbbb
Annotations:    kubernetes.io/limit-ranger: LimitRanger plugin set: cpu request for container polls-app; cpu request for container cloudsql-proxy
Status:         Running
IP:             10.60.3.4
Controlled By:  ReplicaSet/polls-55456ddbbb
Containers:
  polls-app:
    Container ID:   ***
    Image:          ***
    Image ID:       ***
    Port:           8080/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Mon, 23 Mar 2020 18:12:01 +0900
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:  100m
    Environment:
      DATABASE_USER:      <set to the key 'username' in secret 'cloudsql'>  Optional: false
      DATABASE_PASSWORD:  <set to the key 'password' in secret 'cloudsql'>  Optional: false
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-vkv88 (ro)
  cloudsql-proxy:
    Container ID:  docker://d5458d325b071937d12c0cefe2daaf31630d4ab9d565bc3888686cb53127b3e6
    Image:         gcr.io/cloudsql-docker/gce-proxy:1.05
    Image ID:      docker-pullable://gcr.io/cloudsql-docker/gce-proxy@sha256:af9140a11b2593d4be77323f25c0c2abb136133d2bbef76204e7bdad82f05ee7
    Port:          <none>
    Host Port:     <none>
    Command:
      ***
    State:          Running
      Started:      Mon, 23 Mar 2020 18:12:03 +0900
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:        100m
    Environment:  <none>
    Mounts:
      /cloudsql from cloudsql (rw)
      /etc/ssl/certs from ssl-certs (rw)
      /secrets/cloudsql from cloudsql-oauth-credentials (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-vkv88 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  cloudsql-oauth-credentials:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  cloudsql-oauth-credentials
    Optional:    false
  ssl-certs:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/ssl/certs
    HostPathType:
  cloudsql:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  default-token-vkv88:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-vkv88
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>

kubectl describe pods polls-55456ddbbb-r9l9x

Name:           polls-55456ddbbb-r9l9x
Namespace:      default
Priority:       0
Node:           gke-polls-default-pool-2c3ca2d7-9m9d/10.146.0.29
Start Time:     Mon, 23 Mar 2020 18:11:03 +0900
Labels:         app=polls
                pod-template-hash=55456ddbbb
Annotations:    kubernetes.io/limit-ranger: LimitRanger plugin set: cpu request for container polls-app; cpu request for container cloudsql-proxy
Status:         Running
IP:             10.60.3.5
Controlled By:  ReplicaSet/polls-55456ddbbb
Containers:
  polls-app:
    Container ID:   ***
    Image:          ***
    Image ID:       ***
    Port:           8080/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Mon, 23 Mar 2020 18:12:01 +0900
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:  100m
    Environment:
      DATABASE_USER:      <set to the key 'username' in secret 'cloudsql'>  Optional: false
      DATABASE_PASSWORD:  <set to the key 'password' in secret 'cloudsql'>  Optional: false
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-vkv88 (ro)
  cloudsql-proxy:
    Container ID:  ***
    Image:         ***
    Image ID:      ***
    Port:          <none>
    Host Port:     <none>
    Command:
      ***
    State:          Running
      Started:      Mon, 23 Mar 2020 18:12:04 +0900
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:        100m
    Environment:  <none>
    Mounts:
      /cloudsql from cloudsql (rw)
      /etc/ssl/certs from ssl-certs (rw)
      /secrets/cloudsql from cloudsql-oauth-credentials (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-vkv88 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  cloudsql-oauth-credentials:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  cloudsql-oauth-credentials
    Optional:    false
  ssl-certs:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/ssl/certs
    HostPathType:
  cloudsql:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  default-token-vkv88:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-vkv88
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>

kubectl describe pods polls-55456ddbbb-wz4ml

Name:           polls-55456ddbbb-wz4ml
Namespace:      default
Priority:       0
Node:           gke-polls-default-pool-2c3ca2d7-ztcd/10.146.0.27
Start Time:     Mon, 23 Mar 2020 18:11:03 +0900
Labels:         app=polls
                pod-template-hash=55456ddbbb
Annotations:    kubernetes.io/limit-ranger: LimitRanger plugin set: cpu request for container polls-app; cpu request for container cloudsql-proxy
Status:         Running
IP:             10.60.1.3
Controlled By:  ReplicaSet/polls-55456ddbbb
Containers:
  polls-app:
    Container ID:   ***
    Image:          ***
    Image ID:       ***
    Port:           8080/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Mon, 23 Mar 2020 18:11:59 +0900
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:  100m
    Environment:
      DATABASE_USER:      <set to the key 'username' in secret 'cloudsql'>  Optional: false
      DATABASE_PASSWORD:  <set to the key 'password' in secret 'cloudsql'>  Optional: false
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-vkv88 (ro)
  cloudsql-proxy:
    Container ID:  docker://375a66aafccc2cab413b9eaf9b3bf18a68234c353c9e33082cc879d2edd7639f
    Image:         gcr.io/cloudsql-docker/gce-proxy:1.05
    Image ID:      docker-pullable://gcr.io/cloudsql-docker/gce-proxy@sha256:af9140a11b2593d4be77323f25c0c2abb136133d2bbef76204e7bdad82f05ee7
    Port:          <none>
    Host Port:     <none>
    Command:
      ***
    State:          Running
      Started:      Mon, 23 Mar 2020 18:12:03 +0900
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:        100m
    Environment:  <none>
    Mounts:
      /cloudsql from cloudsql (rw)
      /etc/ssl/certs from ssl-certs (rw)
      /secrets/cloudsql from cloudsql-oauth-credentials (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-vkv88 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  cloudsql-oauth-credentials:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  cloudsql-oauth-credentials
    Optional:    false
  ssl-certs:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/ssl/certs
    HostPathType:
  cloudsql:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  default-token-vkv88:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-vkv88
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>
-- Yoshiki.M
deployment
django
google-kubernetes-engine
mysql
python

Similar Questions

Applying TLS to a working ingress results in default backend - 404
Exception in thread "main" java.lang.NoSuchMethodError: io.fabric8.kubernetes.api.model.AuthInfo.getExec()Lio/fabric8/kubernetes/api/model/ExecConfig;
kubernetes pod yaml set as env var the pod ip and port
How to avoid coredns resolving overhead in kubernetes

1 Answer

3/28/2020

I can connect to Cloud SQL instance after some process.

1.I changed setting.py like below.

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'polls',
        'USER': os.getenv('DATABASE_USER'),
        'PASSWORD': os.getenv('DATABASE_PASSWORD'),
        'HOST': '127.0.0.1',
        'PORT': '3306',
    }
}

And I changed polls.yaml like below.

  - image: gcr.io/cloudsql-docker/gce-proxy:1.16
    name: cloudsql-proxy
    command: ***:mysqlinstance=tcp:3306",***

Port number was changed from 5432 to 3306. When I use mysql instance, I must change port number.

-- Yoshiki.M
Source: StackOverflow