I have two Fedora installed VMs where i am exploring Kubernetes- one being master node and one being slave node.

a) I created a docker private registry on slave node using basic ssl authentication. Lets call it abc.def.com:1234

b) Then i logged into docker using docker login. Login succeeded.

c) I downloaded opensuse:latest image from docker hub, tagged it and uploaded to the registry. Docker push succeeded. I deleted from docker local cache and tried docker pull to verify if image push was successful and yes, it was successful.

d) I am able to use the image downloaded from my private registry and run it with basic command /bin/bash. Eg: docker run -it taggedimage /bin/bash

e) Now, I stopped the container. I deleted the image from local cache as well.

f) Then, i created a basic yaml file to create the same type of container- just start the container with /bin/bash.

g) I provided appropriate REGSECRET into my yaml file and also restarted kubelet service after updating the argument --pod-container-infra-image.

$ ps -ef | grep kubelet
root     23683     1  2 05:42 ?        00:01:12 /usr/bin/kubelet --logtostderr=true --v=0 --api-servers=http://wer.txy.com:8080 --address=0.0.0.0 --hostname-override=abc.def.com --allow-privileged=false --pod-infra-container-image=abc.def.com:1234/s5678:test --cluster-dns=x.y.z.b --cgroup-driver=systemd

(Note: don't go by the values i have provided, it's just to maintain privacy)

h) Then i ran the command - kubectl create -f test.yaml; where test.yaml contents look like below:

apiVersion: v1  
kind: Pod  
metadata:  
  name: test1  
spec:  
  containers:  
  - image: abc.def.com:1234/s5678:test  
    imagePullPolicy: IfNotPresent  
    ports:  
      - containerPort: 80  
    command: ["/bin/bash"]  
  imagePullSecrets:  
    - name: regsecret  

i) The pod got created successfully, but status is containercreating.

$ kubectl get pods  
NAME      READY     STATUS              RESTARTS   AGE  
test1     0/1       ContainerCreating   0          22m  

j) When checked kubectl logs:

Error from server (BadRequest): container "utest1" in pod "test1" is waiting to start: ContainerCreating    

k) When checked kubectl describe:

  47m           9s              216     kubelet, abc.def.com                     Warning         FailedSync      Error syncing pod, skipping: failed to "CreatePodSandbox" for "test1_default(8961fa88-a4fa-11e7-95b4-0050568dcdb2)" with CreatePodSandboxError: "CreatePodSandbox for pod \"test1_default(8961fa88-a4fa-11e7-95b4-0050568dcdb2)\" failed: rpc error: code = 2 desc = failed to create a sandbox for pod \"test1\": Error response from daemon: {\"message\":\"No command specified\"}"  

l) I further went ahead and checked the /var/log/messages. It specified that kubelet was able to receive the required arguments:

kubelet[23683]: I0929 06:31:14.465534   23683 kuberuntime_manager.go:458] Container {Name:utest1 Image:abc.def.com:1234/s5678:test Command:[/bin/sh] Args:[] WorkingDir: Ports:[{Name: HostPort:0 ContainerPort:80 Protocol:TCP HostIP:}] EnvFrom:[] Env:[] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[] LivenessProbe:nil ReadinessProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false}  

AS you can see above command is received by Kubelet. But is not passed to dockerd-current daemon:

dockerd-current[23526]: time="2017-09-29T06:31:14.474656750-04:00" level=error msg="Handler for POST /v1.26/containers/create?name=k8s_POD_test1_default_8961fa88-a4fa-11e7-95b4-0050568dcdb2_0 returned error: No command specified"    

m) Also, further to this, i went ahead and checked if image retrieval from private registry was successful. And yes - it was.

Before pod creation:

$ docker images    
REPOSITORY           TAG                 IMAGE ID            CREATED             SIZE  
docker.io/registry   2                   28525f9a6e46        10 days ago         33.2 MB  

After pod creation:

$ docker images    
REPOSITORY                        TAG                 IMAGE ID            CREATED             SIZE  
abc.def.com:1234/s5678   test                54ae12a89367        8 days ago          108 MB  
docker.io/registry                2                   28525f9a6e46        10 days ago         33.2 MB    

So, image retrieval was successful. And also, after checking /var/run/container* directory and associated docker directories, i found that container create event was issued, but had failed - since no command was specified.

Can any of you please help me to triage this issue further and move into closure?