Kubernetes service not working as expected

1/7/2019

I am failing to deploy postgres (single node, official image) on kubernetes and allow services to access postgres via ClusterIP service.

The config is rather simple - Namespace, Deployment, Service:

---
apiVersion: v1
kind: Namespace
metadata:
  name: database
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  namespace: database
  name: postgres
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:11.1
          imagePullPolicy: "IfNotPresent"
          ports:
            - containerPort: 5432
---
apiVersion: v1
kind: Service
metadata:
  name: pg
  namespace: database
  labels:
    app: postgres
spec:
  selector:
    app: postgres
  ports:
  - protocol: TCP
    name: postgres
    port: 5432
    targetPort: 5432

To test is executed a "/bin/bash" into the pod and ran a simple psql command to test the connection. All works well locally:

kubectl --kubeconfig $k8sconf -n database exec -it $(kubectl --kubeconfig $k8sconf -n database get pods -o jsonpath='{.items[*].metadata.name}') -- psql -U admin postgresdb -c "\t"
Tuples only is on.

But as soon as I try to access postgres via service, the command fails:

kubectl --kubeconfig $k8sconf -n database exec -it $(kubectl --kubeconfig $k8sconf -n database get pods -o jsonpath='{.items[*].metadata.name}') -- psql -h pg -U admin postgresdb -c "\t"
psql: could not connect to server: Connection timed out
    Is the server running on host "pg" (10.245.102.15) and accepting
    TCP/IP connections on port 5432?

This is tested on a DigitalOcean a single node cluster (1.12.3).

Postgres listened on * on the correct port, pg_hba.conf looks by default like this:

...
local   all             all                                     trust
# IPv4 local connections:
host    all             all             127.0.0.1/32            trust
# IPv6 local connections:
host    all             all             ::1/128                 trust
# Allow replication connections from localhost, by a user with the
# replication privilege.
local   replication     all                                     trust
host    replication     all             127.0.0.1/32            trust
host    replication     all             ::1/128                 trust
host all all all md5

To reproduce see this gist

Execute via (please use a fresh cluster and read thru):

export k8sconf=/path/to/your/k8s/confic/file
kubectl --kubeconfig $k8sconf apply -f https://gist.githubusercontent.com/sontags/c364751e7f0d8ba1a02a9805efc68db6/raw/01b1808348541d743d6a861402cfba224bee8971/database.yaml
kubectl --kubeconfig $k8sconf -n database exec -it $(kubectl --kubeconfig $k8sconf -n database get pods -o jsonpath='{.items[*].metadata.name}') -- /bin/bash /reproducer/runtest.sh

Any hint why the service does not allow to connect or other tests to perform?

-- sontags
kubernetes
postgresql

1 Answer

1/7/2019

Hard to tell without access to your cluster. This works fine on my AWS cluster. Some things to look at:

  • Is the kube-proxy running on all nodes?
  • Is your network overlay/CNI running on all nodes?
  • Does this happen with the pg pod only? what about other pods?
  • DNS seems to be fine since pg is being resolved to 10.245.102.15
  • Are your nodes allowing IP forwarding from the Linux side?
  • Are your Digital Ocean firewall rules allowing traffic from any source on port 5432? Note that the PodCidr and K8s Service IP range is different the hostCidr (of your droplets).
-- Rico
Source: StackOverflow