I have a Go Lang REST service and ETCD DB in one container, deployed in kubernetes cluster using Deployment type. Whenever I try to restart the service pod, the service loses connectivity to ETCD, I have tried using stateful sets instead of deployment but still didn't help. My deployment looks something like below.
The ETCD fails restarting due to this issue: https://github.com/etcd-io/etcd/issues/10487
PVC :
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: XXXX
namespace: XXXX
annotations:
volume.beta.kubernetes.io/storage-class: glusterfs-storage
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
Deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: XXX
namespace: XXX
spec:
replicas: X
XXXXXXX
template:
metadata:
labels:
app: rest-service
version: xx
spec:
hostAliases:
- ip: 127.0.0.1
hostnames:
- "etcd.xxxxx"
containers:
- name: rest-service
image: xxxx
imagePullPolicy: IfNotPresent
ports:
- containerPort: xxx
securityContext:
readOnlyRootFilesystem: false
capabilities:
add:
- IPC_LOCK
- name: etcd-db
image: quay.io/coreos/etcd:v3.3.11
imagePullPolicy: IfNotPresent
command:
- etcd
- --name=etcd-db
- --listen-client-urls=https://0.0.0.0:2379
- --advertise-client-urls=https://etcd.xxxx:2379
- --data-dir=/var/etcd/data
- --client-cert-auth
- --trusted-ca-file=xxx/ca.crt
- --cert-file=xxx/tls.crt
- --key-file=xxx/tls.key
volumeMounts:
- mountPath: /var/etcd/data
name: etcd-data
XXXX
ports:
- containerPort: 2379
volumes:
- name: etcd-data
persistentVolumeClaim:
claimName: XXXX
I would expect the DB to be able to connect to pod even when it restarts
Keeping application and database in one pod is one of the worst practices in Kubernetes. If you update application code - you have to restart pod to apply changes. So you restart database also just for nothing.
Solution is very simple - you should run application in one deployment and database - in another. That way you can update application without restarting database. In that case you can also scale app and DB separately, like add more replicas to app while keeping DB at 1 replicas or vice versa.