At task: Waiting for auth-pdp to start auth-pdp pod could not reach mariadb-0. Has anyone faced this issue before?

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:02:09 --:--:--     0curl: (7) Failed to connect to mariadb port 3306: Operation timed out

I have checked mariadb container if it is running, then realized it has no ip address.

kubectl -n kube-system get pods -o wide
...
mariadb-0                                             1/1       Running             0          7m        <none>       34.65.234.2
...

Then I tried port-forward mariadb I could reach from local terminal

kubectl -n kube-system port-forward mariadb-0 8080:3306 &
[1] 9483
Forwarding from 127.0.0.1:8080 -> 3306
Forwarding from [::1]:8080 -> 3306
^C
curl localhost:3306
5.5.5-10.2.17-MariaDB-1:10.2.17+maria~bionic~rHO8ewF5���vtf~is&`4#M=mysql_native_password!��#08S01 Got packets out of order

I exed auth-pdp pod try curl another services

bash-4.4# env | grep -i mongo
MONGODB_SERVICE_PORT=27017
MONGODB_PORT_27017_TCP=tcp://10.0.0.181:27017
MONGODB_PORT_27017_TCP_PROTO=tcp
MONGODB_PORT_27017_TCP_ADDR=10.0.0.181
MONGODB_SERVICE_HOST=10.0.0.181
MONGODB_PORT_27017_TCP_PORT=27017
MONGODB_PORT=tcp://10.0.0.181:27017
bash-4.4# curl 10.0.0.181:27017
It looks like you are trying to access MongoDB over HTTP on the native driver port.

I reviewed the interfaces of mariadb pod and found the following:

calie6c068a340a: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::ecee:eeff:feee:eeee  prefixlen 64  scopeid 0x20<link>
        ether ee:ee:ee:ee:ee:ee  txqueuelen 0  (Ethernet)
        RX packets 451  bytes 27552 (27.5 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 348  bytes 27089 (27.0 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

caliee45ca146c2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::ecee:eeff:feee:eeee  prefixlen 64  scopeid 0x20<link>
        ether ee:ee:ee:ee:ee:ee  txqueuelen 0  (Ethernet)
        RX packets 191  bytes 14176 (14.1 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 87  bytes 9236 (9.2 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

calif6e93cd4328: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::ecee:eeff:feee:eeee  prefixlen 64  scopeid 0x20<link>
        ether ee:ee:ee:ee:ee:ee  txqueuelen 0  (Ethernet)
        RX packets 7302  bytes 1203027 (1.2 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 7645  bytes 4818796 (4.8 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
        inet6 fe80::42:1ff:feee:54b4  prefixlen 64  scopeid 0x20<link>
        ether 02:42:01:ee:54:b4  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 5  bytes 446 (446.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1460
        inet 10.240.0.11  netmask 255.255.255.255  broadcast 10.240.0.11
        inet6 fe80::4001:aff:fef0:b  prefixlen 64  scopeid 0x20<link>
        ether 42:01:0a:f0:00:0b  txqueuelen 1000  (Ethernet)
        RX packets 753548  bytes 3753160696 (3.7 GB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 697800  bytes 219630854 (219.6 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 214601  bytes 54627282 (54.6 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 214601  bytes 54627282 (54.6 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

The ens4 NIC has the same IP address as the Node where the pod is running. It does not got a unique address. Then the following resolves:

k exec -it auth-idp-bkqwf -c init-mariadb bash
bash-4.4# curl 10.240.0.11:3306 --output dummy.txt
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   148    0   148    0     0   144k      0 --:--:-- --:--:-- --:--:--  144k