K
Q

Question

Running scripts using lifecycle hook does not work properly in kubernetes

9/25/2019

I am new to kubernetes. And I am learning kubernetes using minikube. I have a situation where I have to configure fdw (foreign table concept) in PostgreSQL.

I am trying to implement this fdw concept in kubernetes. For that I have two pods meta pod where we create the foreign table and server, Data pod where the actual table exist. To create a meta pod you should have an Docker image consist of following script.

Script.sh

#!/bin/bash
psql -d metap -U papu -c "CREATE EXTENSION if not exists postgres_fdw;"
psql -d metap -U papu -c "CREATE SERVER if not exists dataserver FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host 'datap.default.svc.cluster.local', dbname 'datap', port '5432');"
psql -d metap -U papu -c "CREATE USER MAPPING if not exists FOR ais SERVER dataserver OPTIONS (user 'papu', password 'papu');"
psql -d metap -U papu -c "CREATE FOREIGN TABLE if not exists dream (id integer, val text) SERVER dataserver OPTIONS (schema_name 'public', table_name 'dream');"

The yaml file for the two pods is given below.

p-config.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: p-config
  labels:
    app: post
data:
  POSTGRES_DB: datap
  POSTGRES_USER: papu
  POSTGRES_PASSWORD: papu

datap.yaml

apiVersion: v1
kind: Service
metadata:
  name: datap
  labels:
    app: datap
spec:
  ports:
  - port: 5432
    name: datap
  clusterIP: None
  selector:
    app: datap
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: datap
spec:
  serviceName: "datap"
  replicas: 1
  selector:
    matchLabels:
      app: datap
  template:
    metadata:
      labels:
        app: datap
    spec:
      containers:
      - name: datap
        image: postgres:latest
        envFrom:
          - configMapRef:
              name: p-config
        ports:
        - containerPort: 5432
          name: datap
        volumeMounts:
        - name: datap 
          mountPath: /var/lib/postgresql/data
          subPath: datap
  volumeClaimTemplates:
    - metadata:
        name: datap 
      spec:
        accessModes: [ "ReadWriteOnce" ]
        resources:
          requests:
            storage: 2Gi

My use case is that I already have data pod up and running, now I have to create the foreign table through running script in meta pod dynamically.

For that I am using life cycle hook. when I run this configuration, the foreign table is created and fdw connection is established. But the logs say life-cycle hook is not Executed. Is this a bug? Or any problem in my configuration?

$ kubectl describe pod metap-0

Name:           metap-0
Namespace:      default
Priority:       0
Node:           minikube/10.0.2.15
Start Time:     Fri, 20 Sep 2019 15:50:41 +0530
Labels:         app=metap
                controller-revision-hash=metap-648ddb5465
                statefulset.kubernetes.io/pod-name=metap-0
Annotations:    <none>
Status:         Running
IP:             172.17.0.10
Controlled By:  StatefulSet/metap
Containers:
  metap:
    Container ID:  
    Image:          <script containing image >:latest
    Port:           5432/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Fri, 20 Sep 2019 15:51:29 +0530
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Fri, 20 Sep 2019 15:51:14 +0530
      Finished:     Fri, 20 Sep 2019 15:51:15 +0530
    Ready:          True
    Restart Count:  2
    Environment:    <none>
    Mounts:
      /var/lib/postgresql/data from metap (rw,path="metap")
      /var/run/secrets/kubernetes.io/serviceaccount from default-token
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  mpostgredb:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  metap-metap-0
    ReadOnly:   false
  default-token-r2ncm:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-r2ncm
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason               Age                From               Message
  ----     ------               ----               ----               -------
  Warning  FailedScheduling     71s (x2 over 71s)  default-scheduler  pod has unbound immediate PersistentVolumeClaims
  Normal   Scheduled            69s                default-scheduler  Successfully assigned default/metap-0 to minikube
  Warning  FailedPostStartHook  67s                kubelet, minikube  Exec lifecycle hook ([/bin/sh -c script.sh]) for Container "metap" in Pod "metap-0_default(6a367766-cd7e-4bab-826a-908e33622bcf)" failed - error: command '/bin/sh -c script.sh' exited with 2: psql: could not connect to server: No such file or directory
           Is the server running locally and accepting
           connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
psql: could not connect to server: No such file or directory
  Is the server running locally and accepting
  connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
psql: could not connect to server: No such file or directory
  Is the server running locally and accepting
  connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
psql: could not connect to server: No such file or directory
  Is the server running locally and accepting
  connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
, message: "psql: could not connect to server: No such file or directory\n\tIs the server running locally and accepting\n\tconnections on Unix domain socket \"/var/run/postgresql/.s.PGSQL.5432\"?\npsql: could not connect to server: No such file or directory\n\tIs the server running locally and accepting\n\tconnections on Unix domain socket \"/var/run/postgresql/.s.PGSQL.5432\"?\npsql: could not connect to server: No such file or directory\n\tIs the server running locally and accepting\n\tconnections on Unix domain socket \"/var/run/postgresql/.s.PGSQL.5432\"?\npsql: could not connect to server: No such file or directory\n\tIs the server running locally and accepting\n\tconnections on Unix domain socket \"/var/run/postgresql/.s.PGSQL.5432\"?\n"
  Warning  FailedPostStartHook  35s  kubelet, minikube  Exec lifecycle hook ([/bin/sh -c script.sh]) for Container "metap" in Pod "metap-0_default(6a367766-cd7e-4bab-826a-908e33622bcf)" failed - error: command '/bin/sh -c script.sh' exited with 1: psql: FATAL:  the database system is starting up
psql: FATAL:  the database system is starting up
ERROR:  server "dataserver" does not exist
ERROR:  server "dataserver" does not exist
, message: "psql: FATAL:  the database system is starting up\npsql: FATAL:  the database system is starting up\nERROR:  server \"dataserver\" does not exist\nERROR:  server \"dataserver\" does not exist\n"
  Normal   Killing  35s (x2 over 67s)  kubelet, minikube  FailedPostStartHook
  Warning  BackOff  33s (x2 over 34s)  kubelet, minikube  Back-off restarting failed container
  Normal   Created  21s (x3 over 68s)  kubelet, minikube  Created container metap
  Normal   Started  21s (x3 over 68s)  kubelet, minikube  Started container metap
  Normal   Pulling  21s (x3 over 68s)  kubelet, minikube  Pulling image " <script containing image >:latest"
  Normal   Pulled   21s (x3 over 68s)  kubelet, minikube  Successfully pulled image " <script containing image >:latest"



$ kubectl logs metap-0 
2019-09-20 10:21:29.500 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2019-09-20 10:21:29.500 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2019-09-20 10:21:29.502 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2019-09-20 10:21:29.514 UTC [22] LOG:  database system was shut down at 2019-09-20 10:21:15 UTC
2019-09-20 10:21:29.518 UTC [1] LOG:  database system is ready to accept connections

kubectl version

Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:13:54Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568", GitTreeState:"clean", BuildDate:"2019-08-05T09:15:22Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}

* Minikube version *

minikube version: v1.3.1
commit: ca60a424ce69a4d79f502650199ca2b52f29e631

-- SHARON XAVIER

kubernetes

kubernetes-pod

lifecycle-hook

minikube

postgresql

1 Answer

10/23/2019

I have reproduced your issue and found that the problem is in the lifecycle hook configuration.

The script was executed before database was ready and that's why it thrown errors: command '/bin/sh -c script.sh' exited with 2: psql: could not connect to server: No such file or directory,

failed - error: command '/bin/sh -c script.sh' exited with 1: psql: FATAL: the database system is starting up You should add sleep in the ``beginning of your script to give database some time to be ready.

Example:

#!/bin/bash
sleep 5
psql -d metap -U papu -c "CREATE EXTENSION if not exists postgres_fdw;"
psql -d metap -U papu -c "CREATE SERVER if not exists dataserver FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host 'datap.default.svc.cluster.local', dbname 'datap', port '5432');"
psql -d metap -U papu -c "CREATE USER MAPPING if not exists FOR ais SERVER dataserver OPTIONS (user 'papu', password 'papu');"
psql -d metap -U papu -c "CREATE FOREIGN TABLE if not exists dream (id integer, val text) SERVER dataserver OPTIONS (schema_name 'public', table_name 'dream');

-- KFC_

Source: StackOverflow

KQ

Running scripts using lifecycle hook does not work properly in kubernetes

Similar Questions

1 Answer

K
Q