Kubernetes repeatedly pull image and stay stuck at containercreating

11/7/2017

I'm trying to create a Single-Instance Stateful Application along the lines of https://kubernetes.io/docs/tasks/run-application/run-single-instance-stateful-application/, with the exception that I would like to use an Oracle database.

I run W10 and use Minikube to set up the cluster, Hyper-V as the driver and I also created a Virtual switch for Minikube.

I edited the deployment YAML accordingly to use Oracle DB image and ports, as well as added a pull secret for Docker Hub (as Oracle wants you to log in to pull the inmage).

The final YAML:

apiVersion: v1
kind: Service
metadata:
  name: orcldb
spec:
  ports:
  - port: 1521
    name: sqlnet
  - port: 5500
    name: oraclexml
  selector:
    app: orcldb
  clusterIP: None
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: orcldb-pv-claim
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
---
apiVersion: apps/v1beta2
kind: Deployment
metadata:
  name: orcldb
spec:
  selector:
    matchLabels:
      app: orcldb
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: orcldb
    spec:
      containers:
      - image: store/oracle/database-enterprise:12.2.0.1
        name: orcldb
        ports:
        - containerPort: 1521
          name: sqlnet
        - containerPort: 5500
          name: oraclexml
        volumeMounts:
        - name: orcldb-persistent-storage
          mountPath: /var/lib/orcldb
      imagePullSecrets:
      - name: dockerhubkey
      volumes:
      - name: orcldb-persistent-storage
        persistentVolumeClaim:
          claimName: orcldb-pv-claim

So after I run kubectl create -f deployment.yaml, everything is created accordingly:

service "orcldb" created
persistentvolumeclaim "orcldb-pv-claim" created
deployment "orcldb" created

But then, if I run kubectl describe pods orcldb, I see this in the events:

Type     Reason                 Age              From               Message
  ----     ------                 ----             ----               -------
  Warning  FailedScheduling       1h (x2 over 1h)  default-scheduler  PersistentVolumeClaim is not bound: "orcldb-pv-claim"
  Normal   Scheduled              1h               default-scheduler  Successfully assigned orcldb-7d96df68c8-bjwql to minikube
  Normal   SuccessfulMountVolume  1h               kubelet, minikube  MountVolume.SetUp succeeded for volume "pvc-c91fd4a0-c3da-11e7-9300-00155d050c0a"
  Normal   SuccessfulMountVolume  1h               kubelet, minikube  MountVolume.SetUp succeeded for volume "default-token-rszmg"
  Normal   Pulling                1h               kubelet, minikube  pulling image "store/oracle/database-enterprise:12.2.0.1"
  Normal   SuccessfulMountVolume  1h               kubelet, minikube  MountVolume.SetUp succeeded for volume "pvc-c91fd4a0-c3da-11e7-9300-00155d050c0a"
  Normal   SuccessfulMountVolume  1h               kubelet, minikube  MountVolume.SetUp succeeded for volume "default-token-rszmg"
  Normal   Pulling                1h               kubelet, minikube  pulling image "store/oracle/database-enterprise:12.2.0.1"
  Normal   SuccessfulMountVolume  43m              kubelet, minikube  MountVolume.SetUp succeeded for volume "pvc-c91fd4a0-c3da-11e7-9300-00155d050c0a"
  Normal   SuccessfulMountVolume  43m              kubelet, minikube  MountVolume.SetUp succeeded for volume "default-token-rszmg"
  Normal   Pulling                43m              kubelet, minikube  pulling image "store/oracle/database-enterprise:12.2.0.1"
  Normal   SuccessfulMountVolume  32m              kubelet, minikube  MountVolume.SetUp succeeded for volume "pvc-c91fd4a0-c3da-11e7-9300-00155d050c0a"
  Normal   SuccessfulMountVolume  32m              kubelet, minikube  MountVolume.SetUp succeeded for volume "default-token-rszmg"
  Normal   Pulling                32m              kubelet, minikube  pulling image "store/oracle/database-enterprise:12.2.0.1" 

Also, the minikube cluster sometimes stops (?) / refuses connection, I usually restart minikube after that, but once I left it be and it came back up afte a while (with the repeating pulling process still going on).

I haven't been able to find anyone facing this kind of issue, so I would greatly appreciate any insigh as to why this might be happening.

Part of the minikube logs with errors about disk:

Nov 07 18:37:22 minikube localkube[16252]: E1107 18:37:22.727122   16252 fsHandler.go:121] failed to collect filesystem stats - rootDiskErr: du command failed on /var/lib/docker/overlay2/611dbda511b2771528eff890891445fe89fdd171e6e1e944596b82b5a031d4c6 with output stdout: , stderr: du: cannot access '/var/lib/docker/overlay2/611dbda511b2771528eff890891445fe89fdd171e6e1e944596b82b5a031d4c6': No such file or directory
Nov 07 18:37:22 minikube localkube[16252]:  - exit status 1, rootInodeErr: cmd [find /var/lib/docker/overlay2/611dbda511b2771528eff890891445fe89fdd171e6e1e944596b82b5a031d4c6 -xdev -printf .] failed. stderr: find: '/var/lib/docker/overlay2/611dbda511b2771528eff890891445fe89fdd171e6e1e944596b82b5a031d4c6': No such file or directory
Nov 07 18:37:22 minikube localkube[16252]: ; err: exit status 1, extraDiskErr: du command failed on /var/lib/docker/containers/97fc2e9f49de23415fe6494bf3684672aeb1213930de02314b1f2ce411f4dc6a with output stdout: , stderr: du: cannot access '/var/lib/docker/containers/97fc2e9f49de23415fe6494bf3684672aeb1213930de02314b1f2ce411f4dc6a': No such file or directory
-- EyfI
containers
docker
kubernetes
minikube
oracle

1 Answer

12/4/2017

I moved away from the huge official images by Oracle and used a custom made image of the Oracle Express 11g database. The image is now pulled and container started without any problems.

Also, the issue where Minikube would stop responding after a while was solved by disabling dynamic memory in Hyper-V, as described in this github issue.

-- EyfI
Source: StackOverflow