Error from server: Get https://10.128.15.203:10250/containerLogs/default/postgres-54db6bdb8b-cmrsb/postgres: EOF
How could I solve this issue ? And what can be reason . I've used this tutrial for configuring all stuff (https://severalnines.com/database-blog/using-kubernetes-deploy-postgresql). If you need more info , pls let me know !
kubectl describe pods postgres-54db6bdb8b-cmrsb
Name: postgres-54db6bdb8b-cmrsb
Namespace: default
Priority: 0
Node: gke-booknotes-pool-2-c1d23e62-r6nb/10.128.15.203
Start Time: Sat, 14 Dec 2019 23:27:20 +0700
Labels: app=postgres
pod-template-hash=54db6bdb8b
Annotations: kubernetes.io/limit-ranger: LimitRanger plugin set: cpu request for container postgres
Status: Running
IP: 10.56.1.3
IPs: <none>
Controlled By: ReplicaSet/postgres-54db6bdb8b
Containers:
postgres:
Container ID: docker://1a607cfb9a8968d708ff79419ec8bfc7233fb5ad29fb1055034ddaacfb793d6a
Image: postgres:10.4
Image ID: docker-pullable://postgres@sha256:9625c2fb34986a49cbf2f5aa225d8eb07346f89f7312f7c0ea19d82c3829fdaa
Port: 5432/TCP
Host Port: 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: ContainerCannotRun
Message: error while creating mount source path '/mnt/data': mkdir /mnt/data: read-only file system
Exit Code: 128
Started: Sat, 14 Dec 2019 23:54:00 +0700
Finished: Sat, 14 Dec 2019 23:54:00 +0700
Ready: False
Restart Count: 25
Requests:
cpu: 100m
Environment Variables from:
postgres-config ConfigMap Optional: false
Environment: <none>
Mounts:
/var/lib/postgresql/data from postgredb (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-t48dw (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
postgredb:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: postgres-pv-claim
ReadOnly: false
default-token-t48dw:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-t48dw
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 32m default-scheduler Successfully assigned default/postgres-54db6bdb8b-cmrsb to gke-booknotes-pool-2-c1d23e62-r6nb
Normal Pulled 28m (x5 over 30m) kubelet, gke-booknotes-pool-2-c1d23e62-r6nb Container image "postgres:10.4" already present on machine
Normal Created 28m (x5 over 30m) kubelet, gke-booknotes-pool-2-c1d23e62-r6nb Created container postgres
Warning Failed 28m (x5 over 30m) kubelet, gke-booknotes-pool-2-c1d23e62-r6nb Error: failed to start container "postgres": Error response from daemon: error while creating mount source path '/mnt/data': mkdir /mnt/data: read-only file system
Warning BackOff 27m (x10 over 29m) kubelet, gke-booknotes-pool-2-c1d23e62-r6nb Back-off restarting failed container
Warning Failed 23m (x4 over 25m) kubelet, gke-booknotes-pool-2-c1d23e62-r6nb Error: failed to start container "postgres": Error response from daemon: error while creating mount source path '/mnt/data': mkdir /mnt/data: read-only file system
Warning BackOff 22m (x11 over 25m) kubelet, gke-booknotes-pool-2-c1d23e62-r6nb Back-off restarting failed container
Normal Pulled 22m (x5 over 25m) kubelet, gke-booknotes-pool-2-c1d23e62-r6nb Container image "postgres:10.4" already present on machine
Normal Created 22m (x5 over 25m) kubelet, gke-booknotes-pool-2-c1d23e62-r6nb Created container postgres
Normal Pulled 19m (x4 over 20m) kubelet, gke-booknotes-pool-2-c1d23e62-r6nb Container image "postgres:10.4" already present on machine
Normal Created 19m (x4 over 20m) kubelet, gke-booknotes-pool-2-c1d23e62-r6nb Created container postgres
Warning Failed 19m (x4 over 20m) kubelet, gke-booknotes-pool-2-c1d23e62-r6nb Error: failed to start container "postgres": Error response from daemon: error while creating mount source path '/mnt/data': mkdir /mnt/data: read-only file system
Warning BackOff 18m (x11 over 20m) kubelet, gke-booknotes-pool-2-c1d23e62-r6nb Back-off restarting failed container
Normal Created 15m (x4 over 17m) kubelet, gke-booknotes-pool-2-c1d23e62-r6nb Created container postgres
Warning Failed 15m (x4 over 17m) kubelet, gke-booknotes-pool-2-c1d23e62-r6nb Error: failed to start container "postgres": Error response from daemon: error while creating mount source path '/mnt/data': mkdir /mnt/data: read-only file system
Normal Pulled 14m (x5 over 17m) kubelet, gke-booknotes-pool-2-c1d23e62-r6nb Container image "postgres:10.4" already present on machine
Warning BackOff 12m (x19 over 17m) kubelet, gke-booknotes-pool-2-c1d23e62-r6nb Back-off restarting failed container
Normal Pulled 5m38s (x5 over 8m29s) kubelet, gke-booknotes-pool-2-c1d23e62-r6nb Container image "postgres:10.4" already present on machine
Normal Created 5m38s (x5 over 8m27s) kubelet, gke-booknotes-pool-2-c1d23e62-r6nb Created container postgres
Warning Failed 5m37s (x5 over 8m24s) kubelet, gke-booknotes-pool-2-c1d23e62-r6nb Error: failed to start container "postgres": Error response from daemon: error while creating mount source path '/mnt/data': mkdir /mnt/data: read-only file system
Warning BackOff 5m24s (x10 over 7m58s) kubelet, gke-booknotes-pool-2-c1d23e62-r6nb Back-off restarting failed container
Here is also my yaml files:
deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: postgres
spec:
replicas: 1
template:
metadata:
labels:
app: postgres
spec:
containers:
- name: postgres
image: postgres:10.4
imagePullPolicy: "IfNotPresent"
ports:
- containerPort: 5432
envFrom:
- configMapRef:
name: postgres-config
volumeMounts:
- mountPath: /var/lib/postgresql/data
name: postgredb
volumes:
- name: postgredb
persistentVolumeClaim:
claimName: postgres-pv-claim
postgres-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: postgres-config
labels:
app: postgres
data:
POSTGRES_DB: postgresdb
POSTGRES_USER: postgresadmin
POSTGRES_PASSWORD: some_password
postgres-service.yaml
apiVersion: v1
kind: Service
metadata:
name: postgres
labels:
app: postgres
spec:
type: NodePort
ports:
- port: 5432
selector:
app: postgres
postgres-storage.yaml
kind: PersistentVolume
apiVersion: v1
metadata:
name: postgres-pv-volume
labels:
type: local
app: postgres
spec:
storageClassName: manual
capacity:
storage: 5Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/mnt/data"
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: postgres-pv-claim
labels:
app: postgres
spec:
storageClassName: manual
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
After I've changed RWM to RWO - then I this (I've deleted old instances and have created new one):
Name: postgres-54db6bdb8b-wgvr2
Namespace: default
Priority: 0
Node: gke-booknotes-pool-1-3e566443-dc08/10.128.15.236
Start Time: Sun, 15 Dec 2019 04:56:57 +0700
Labels: app=postgres
pod-template-hash=54db6bdb8b
Annotations: kubernetes.io/limit-ranger: LimitRanger plugin set: cpu request for container postgres
Status: Running
IP: 10.56.6.13
IPs: <none>
Controlled By: ReplicaSet/postgres-54db6bdb8b
Containers:
postgres:
Container ID: docker://1070018c2a670cc7e0248e6269c271c3cba022fdd2c9cc5099a8eb4da44f7d65
Image: postgres:10.4
Image ID: docker-pullable://postgres@sha256:9625c2fb34986a49cbf2f5aa225d8eb07346f89f7312f7c0ea19d82c3829fdaa
Port: 5432/TCP
Host Port: 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: ContainerCannotRun
Message: error while creating mount source path '/mnt/data': mkdir /mnt/data: read-only file system
Exit Code: 128
Started: Sun, 15 Dec 2019 10:56:21 +0700
Finished: Sun, 15 Dec 2019 10:56:21 +0700
Ready: False
Restart Count: 76
Requests:
cpu: 100m
Environment Variables from:
postgres-config ConfigMap Optional: false
Environment: <none>
Mounts:
/var/lib/postgresql/data from postgredb (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-t48dw (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
postgredb:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: postgres-pv-claim
ReadOnly: false
default-token-t48dw:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-t48dw
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning BackOff 81s (x1629 over 6h) kubelet, gke-booknotes-pool-1-3e566443-dc08 Back-off restarting failed container
The issue is here: error while creating mount source path '/mnt/data': mkdir /mnt/data: read-only file system.
You need to make sure postgres-pv-claim
is writable. You need to recreate the pv and pv claim with RWO access (you must have mistyped it to RO instead which is why you ran into the issue) then try to deploy postgres pod which should fix the issue.
Provisioning persistent volume in GKE you don't need to create PersistentVolume objects they are dynamically created by GKE. So solve the Warning FailedScheduling 69s (x10 over 7m35s)
issue
storageClassName
property from your pvc andwhich should fix the issue. Please see below revised postgres-storage.yaml
.
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: postgres-pv-claim
labels:
app: postgres
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi