App not communicating with mysql db pod kubernetes

1/10/2022

I'm following the steps here to deploy a multi container application in Kubernetes https://stackoverflow.com/questions/70637470/kubernetes-deployment-not-reachable-via-browser-exposed-with-service

When I browse to nodeIP:nodePort, the application says "database connection mysql is missing or could not be created". I suspect the db is not talking to the app pod.

I've been able to exec into the db using the credentials provisioned in the app ENV. The database.php file also has the correct credentials in the app pod.

DB pod logs

root@osboxes:/home/osboxes# kubectl logs eramba-mariadb-0 -n eramba-1
mariadb 03:25:18.67 
mariadb 03:25:18.67 Welcome to the Bitnami mariadb container
mariadb 03:25:18.67 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-mariadb
mariadb 03:25:18.68 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-mariadb/issues
mariadb 03:25:18.68 
mariadb 03:25:18.68 INFO  ==> ** Starting MariaDB setup **
mariadb 03:25:18.70 INFO  ==> Validating settings in MYSQL_*/MARIADB_* env vars
mariadb 03:25:18.71 INFO  ==> Initializing mariadb database
mariadb 03:25:18.73 WARN  ==> The mariadb configuration file '/opt/bitnami/mariadb/conf/my.cnf' is not writable. Configurations based on environment variables will not be applied for this file.
mariadb 03:25:18.73 INFO  ==> Using persisted data
mariadb 03:25:18.78 INFO  ==> Running mysql_upgrade
mariadb 03:25:18.79 INFO  ==> Starting mariadb in background
mariadb 03:25:20.87 INFO  ==> Stopping mariadb
mariadb 03:25:21.89 INFO  ==> ** MariaDB setup finished! **

mariadb 03:25:21.95 INFO  ==> ** Starting MariaDB **
2022-01-10  3:25:21 0 [Note] /opt/bitnami/mariadb/sbin/mysqld (mysqld 10.5.13-MariaDB) starting as process 1 ...
2022-01-10  3:25:22 0 [Note] InnoDB: Uses event mutexes
2022-01-10  3:25:22 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2022-01-10  3:25:22 0 [Note] InnoDB: Number of pools: 1
2022-01-10  3:25:22 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2022-01-10  3:25:22 0 [Note] mysqld: O_TMPFILE is not supported on /opt/bitnami/mariadb/tmp (disabling future attempts)
2022-01-10  3:25:22 0 [Note] InnoDB: Using Linux native AIO
2022-01-10  3:25:22 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
2022-01-10  3:25:22 0 [Note] InnoDB: Completed initialization of buffer pool
2022-01-10  3:25:22 0 [Note] InnoDB: 128 rollback segments are active.
2022-01-10  3:25:22 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2022-01-10  3:25:22 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2022-01-10  3:25:22 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2022-01-10  3:25:22 0 [Note] InnoDB: 10.5.13 started; log sequence number 3829693; transaction id 2899
2022-01-10  3:25:22 0 [Note] Plugin 'FEEDBACK' is disabled.
2022-01-10  3:25:22 0 [Note] InnoDB: Loading buffer pool(s) from /bitnami/mariadb/data/ib_buffer_pool
2022-01-10  3:25:22 0 [Note] Server socket created on IP: '::'.
2022-01-10  3:25:22 0 [Warning] 'proxies_priv' entry '@% root@eramba-mariadb-0' ignored in --skip-name-resolve mode.
2022-01-10  3:25:22 0 [Note] Reading of all Master_info entries succeeded
2022-01-10  3:25:22 0 [Note] Added new Master_info '' to hash table
2022-01-10  3:25:22 0 [Note] /opt/bitnami/mariadb/sbin/mysqld: ready for connections.
Version: '10.5.13-MariaDB'  socket: '/opt/bitnami/mariadb/tmp/mysql.sock'  port: 3306  Source distribution
2022-01-10  3:25:22 0 [Note] InnoDB: Buffer pool(s) load completed at 220110  3:25:2

describe for db service

root@osboxes:/home/osboxes# kubectl describe svc eramba-mariadb -n eramba-1
Name:              eramba-mariadb
Namespace:         eramba-1
Labels:            app.kubernetes.io/component=primary
                   app.kubernetes.io/instance=eramba
                   app.kubernetes.io/managed-by=Helm
                   app.kubernetes.io/name=mariadb
                   helm.sh/chart=mariadb-10.3.0
Annotations:       meta.helm.sh/release-name: eramba
                   meta.helm.sh/release-namespace: eramba-1
Selector:          app.kubernetes.io/component=primary,app.kubernetes.io/instance=eramba,app.kubernetes.io/name=mariadb
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.107.247.50
IPs:               10.107.247.50
Port:              mysql  3306/TCP
TargetPort:        mysql/TCP
Endpoints:         10.20.0.28:3306
Session Affinity:  None
Events:            <none>

describe db pod

root@osboxes:/home/osboxes# kubectl describe pod eramba-mariadb-0 -n eramba-1
Name:         eramba-mariadb-0
Namespace:    eramba-1
Priority:     0
Node:         osboxes/172.16.42.135
Start Time:   Sun, 09 Jan 2022 20:59:24 -0500
Labels:       app.kubernetes.io/component=primary
              app.kubernetes.io/instance=eramba
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=mariadb
              controller-revision-hash=eramba-mariadb-64579c97cc
              helm.sh/chart=mariadb-10.3.0
              statefulset.kubernetes.io/pod-name=eramba-mariadb-0
Annotations:  checksum/configuration: dcf82b6bca1415cc418030332ecbc15240b060c806b11c7c0319fa2fa568ca69
Status:       Running
IP:           10.20.0.28
IPs:
  IP:           10.20.0.28
Controlled By:  StatefulSet/eramba-mariadb
Init Containers:
  volume-permissions:
    Container ID:  docker://81e4c5f3b6cfc2d638cb366a4c64b22088e3a205fffd3ef15f4eafac804690f6
    Image:         docker.io/bitnami/bitnami-shell:10-debian-10-r279
    Image ID:      docker-pullable://bitnami/bitnami-shell@sha256:b9dba2d6bf011513a8914bdbd2c3307c9253d235d2c50d158b5d1c1359a6fe39
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/bash
      -ec
      chown -R 1001:1001 /bitnami/mariadb
      
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sun, 09 Jan 2022 22:25:17 -0500
      Finished:     Sun, 09 Jan 2022 22:25:18 -0500
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /bitnami/mariadb from data (rw)
Containers:
  mariadb:
    Container ID:   docker://ece671c8646d9c12c5a56b490145fe8ac8de223f2c5ca18a89041cdce92bbe13
    Image:          docker.io/bitnami/mariadb:10.5.13-debian-10-r32
    Image ID:       docker-pullable://bitnami/mariadb@sha256:4969eda3a6cbb8007b4e52992979d9e1f1685cbed7c21afd2b44a64797c9e400
    Port:           3306/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Sun, 09 Jan 2022 22:25:18 -0500
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sun, 09 Jan 2022 20:59:26 -0500
      Finished:     Sun, 09 Jan 2022 21:59:43 -0500
    Ready:          True
    Restart Count:  1
    Liveness:       exec [/bin/bash -ec password_aux="${MARIADB_ROOT_PASSWORD:-}"
if [[ -f "${MARIADB_ROOT_PASSWORD_FILE:-}" ]]; then
    password_aux=$(cat "$MARIADB_ROOT_PASSWORD_FILE")
fi
mysqladmin status -uroot -p"${password_aux}"
] delay=120s timeout=1s period=10s #success=1 #failure=3
    Readiness:  exec [/bin/bash -ec password_aux="${MARIADB_ROOT_PASSWORD:-}"
if [[ -f "${MARIADB_ROOT_PASSWORD_FILE:-}" ]]; then
    password_aux=$(cat "$MARIADB_ROOT_PASSWORD_FILE")
fi
mysqladmin status -uroot -p"${password_aux}"
] delay=30s timeout=1s period=10s #success=1 #failure=3
    Environment:
      BITNAMI_DEBUG:          false
      MARIADB_ROOT_PASSWORD:  <set to the key 'mariadb-root-password' in secret 'eramba-mariadb'>  Optional: false
      MARIADB_DATABASE:       erambadb
    Mounts:
      /bitnami/mariadb from data (rw)
      /docker-entrypoint-initdb.d from custom-init-scripts (rw)
      /opt/bitnami/mariadb/conf/my.cnf from config (rw,path="my.cnf")
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      eramba-mariadb
    Optional:  false
  custom-init-scripts:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      eramba
    Optional:  false
  data:
    Type:        PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:   eramba-storage
    ReadOnly:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason          Age                    From               Message
  ----     ------          ----                   ----               -------
  Normal   Scheduled       93m                    default-scheduler  Successfully assigned eramba-1/eramba-mariadb-0 to osboxes
  Normal   Pulled          93m                    kubelet            Container image "docker.io/bitnami/bitnami-shell:10-debian-10-r279" already present on machine
  Normal   Created         93m                    kubelet            Created container volume-permissions
  Normal   Started         93m                    kubelet            Started container volume-permissions
  Normal   Pulled          93m                    kubelet            Container image "docker.io/bitnami/mariadb:10.5.13-debian-10-r32" already present on machine
  Normal   Created         93m                    kubelet            Created container mariadb
  Normal   Started         93m                    kubelet            Started container mariadb
  Normal   SandboxChanged  7m52s                  kubelet            Pod sandbox changed, it will be killed and re-created.
  Warning  Failed          7m51s (x2 over 7m52s)  kubelet            Error: services have not yet been read at least once, cannot construct envvars
  Normal   Pulled          7m49s (x3 over 7m52s)  kubelet            Container image "docker.io/bitnami/bitnami-shell:10-debian-10-r279" already present on machine
  Normal   Created         7m49s                  kubelet            Created container volume-permissions
  Normal   Started         7m49s                  kubelet            Started container volume-permissions
  Normal   Pulled          7m48s                  kubelet            Container image "docker.io/bitnami/mariadb:10.5.13-debian-10-r32" already present on machine
  Normal   Created         7m48s                  kubelet            Created container mariadb
  Normal   Started         7m48s                  kubelet            Started container mariadb
root@osboxes:/home/osboxes# 

Logs for the web-app pod

root@osboxes:/home/osboxes# kubectl logs eramba-web-5d7ffc87fd-lsrck -n eramba-1
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 10.20.0.30. Set the 'ServerName' directive globally to suppress this message

describe web service

root@osboxes:/home/osboxes# kubectl describe svc eramba-web -n eramba-1
Name:                     eramba-web
Namespace:                eramba-1
Labels:                   app.kubernetes.io/name=eramba-web
Annotations:              <none>
Selector:                 app.kubernetes.io/name=eramba-web
Type:                     NodePort
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.100.24.86
IPs:                      10.100.24.86
Port:                     http  8080/TCP
TargetPort:               8080/TCP
NodePort:                 http  31622/TCP
Endpoints:                10.20.0.30:8080
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

describe web-app pod

root@osboxes:/home/osboxes# kubectl describe pod eramba-web-5d7ffc87fd-lsrck -n eramba-1
Name:         eramba-web-5d7ffc87fd-lsrck
Namespace:    eramba-1
Priority:     0
Node:         osboxes/172.16.42.135
Start Time:   Sun, 09 Jan 2022 21:17:47 -0500
Labels:       app.kubernetes.io/name=eramba-web
              pod-template-hash=5d7ffc87fd
Annotations:  <none>
Status:       Running
IP:           10.20.0.30
IPs:
  IP:           10.20.0.30
Controlled By:  ReplicaSet/eramba-web-5d7ffc87fd
Containers:
  eramba-web:
    Container ID:   docker://e76bcea65ab2f5e50877e3a5504ec515024a1728fcd31465588605dc2fe74a35
    Image:          markz0r/eramba-app:c281
    Image ID:       docker-pullable://markz0r/eramba-app@sha256:747f89930be922c7a6ca51a140890d22f2c54de74fa391e421b074fb14037198
    Port:           8080/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Sun, 09 Jan 2022 22:25:16 -0500
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sun, 09 Jan 2022 21:17:48 -0500
      Finished:     Sun, 09 Jan 2022 21:59:43 -0500
    Ready:          True
    Restart Count:  1
    Environment:
      MYSQL_HOSTNAME:   eramba-mariadb
      MYSQL_DATABASE:   erambadb
      MYSQL_USER:       root
      MYSQL_PASSWORD:   eramba
      DATABASE_PREFIX:  
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-bzjp6 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  kube-api-access-bzjp6:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason          Age                From               Message
  ----     ------          ----               ----               -------
  Normal   Scheduled       80m                default-scheduler  Successfully assigned eramba-1/eramba-web-5d7ffc87fd-lsrck to osboxes
  Normal   Pulled          80m                kubelet            Container image "markz0r/eramba-app:c281" already present on machine
  Normal   Created         80m                kubelet            Created container eramba-web
  Normal   Started         80m                kubelet            Started container eramba-web
  Normal   SandboxChanged  12m                kubelet            Pod sandbox changed, it will be killed and re-created.
  Warning  Failed          12m                kubelet            Error: services have not yet been read at least once, cannot construct envvars
  Normal   Pulled          12m (x2 over 12m)  kubelet            Container image "markz0r/eramba-app:c281" already present on machine
  Normal   Created         12m                kubelet            Created container eramba-web
  Normal   Started         12m                kubelet            Started container eramba-web

output

root@osboxes:/home/osboxes# kubectl get cm,pvc,pv,svc,pods -n eramba-1
NAME                         DATA   AGE
configmap/eramba             1      91m
configmap/eramba-mariadb     1      90m
configmap/kube-root-ca.crt   1      91m

NAME                                   STATUS   VOLUME           CAPACITY   ACCESS MODES   STORAGECLASS     AGE
persistentvolumeclaim/eramba-storage   Bound    eramba-storage   5Gi        RWO            eramba-storage   91m

NAME                              CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                     STORAGECLASS     REASON   AGE
persistentvolume/eramba-storage   5Gi        RWO            Retain           Bound    eramba-1/eramba-storage   eramba-storage            91m

NAME                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
service/eramba-mariadb   ClusterIP   10.107.247.50   <none>        3306/TCP         90m
service/eramba-web       NodePort    10.100.24.86    <none>        8080:31622/TCP   72m

NAME                              READY   STATUS    RESTARTS      AGE
pod/eramba-mariadb-0              1/1     Running   1 (30m ago)   90m
pod/eramba-web-5d7ffc87fd-lsrck   1/1     Running   1 (30m ago)   72m
-- Bryan
kubernetes
mariadb
mysql

0 Answers