I have multiple .NET microservices in my architecture and for each one I have created a deployment object and now I attempt to deploy to the Azure container service running kubernetes. When I run a kubectl apply -f services.yml I always have a few failing pods. But if I run these deployments individually then they all work. The error I receive in summary is: failed to start container and The system cannot find the file specified. The pods status shows as: CrashLoopBackOff
My services file looks like this:
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: service1-deployment
labels:
app: service1
spec:
replicas: 1
selector:
matchLabels:
app: service1
template:
metadata:
labels:
app: service1
spec:
containers:
- name: service1
image: iolregistry.azurecr.io/panviva/doc:v1
command: ["service1.exe"]
imagePullSecrets:
- name: regsecret
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: service2-deployment
labels:
app: service2
spec:
replicas: 1
selector:
matchLabels:
app: service2
template:
metadata:
labels:
app: service2
spec:
containers:
- name: service2
image: iolregistry.azurecr.io/panviva/doc:v1
command: ["service2.exe"]
imagePullSecrets:
- name: regsecret
In reality I have many more deployments in the file but you get the Idea. I have tried splitting these deployments into different files and calling kubectl apply -f service1.yml -f service2.yml but I still receive the same error. I believe it is something to do with the kubernetes starting multiple pods at once. How can I fix this?
EDIT: Describing a failing pod yields the following result:
Name: getdocumentservice-deployment-528941145-cpt30
Namespace: default
Node: e527bacs9002/10.240.0.5
Start Time: Fri, 29 Dec 2017 05:02:31 +0000
Labels: app=getdocumentservice
pod-template-hash=528941145
Annotations: kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"default","name":"getdocumentservice-deployment-528941145","uid":"793ff632-ec55-11...
Status: Running
IP:
Controlled By: ReplicaSet/getdocumentservice-deployment-528941145
Containers:
getdocumentservice:
Container ID: docker://774d7bd23ce3da64a747db4c3737123a56069de97c7b3c3cd11e898e3c9e0e42
Image: iolregistry.azurecr.io/panviva/doc:v1
Image ID: docker-pullable://iolregistry.azurecr.io/panviva/doc@sha256:1bc4f4840707c0174a6d9665828042b04045da2d30e77d96fa325c2f3ae245a6
Port: <none>
Command:
./modules/GetDocumentService/GetDocumentService.exe
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: ContainerCannotRun
Message: container 774d7bd23ce3da64a747db4c3737123a56069de97c7b3c3cd11e898e3c9e0e42 encountered an error during CreateProcess: failure in a Windows system call: The system cannot find the file specified. (0x2) extra info: {"ApplicationName":"","CommandLine":"./modules/GetDocumentService/GetDocumentService.exe","User":"","WorkingDirectory":"C:\\app\\m
odules","Environment":{"KUBERNETES_PORT":"tcp://10.0.0.1:443","KUBERNETES_PORT_443_TCP":"tcp://10.0.0.1:443","KUBERNETES_PORT_443_TCP_ADDR":"10.0.0.1","KUBERNETES_PORT_443_TCP_PORT":"443","KUBERNETES_PORT_443_TCP_PROTO":"tcp","KUBERNETES_SERVICE_HOST":"10.0.0.1","KUBERNETES_SERVICE_PORT":"443","KUBERNETES_SERVICE_PORT_HTTPS":"443"},"EmulateConsole":false,"Creat
eStdInPipe":true,"CreateStdOutPipe":true,"CreateStdErrPipe":true,"ConsoleSize":[0,0]}
Exit Code: 128
Started: Fri, 29 Dec 2017 05:18:41 +0000
Finished: Fri, 29 Dec 2017 05:18:41 +0000
Ready: False
Restart Count: 8
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-9l4dp (ro)
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
default-token-9l4dp:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-9l4dp
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 20m default-scheduler Successfully assigned getdocumentservice-deployment-528941145-cpt30 to e527bacs9002
Normal SuccessfulMountVolume 20m kubelet, e527bacs9002 MountVolume.SetUp succeeded for volume "default-token-9l4dp"
Normal Pulled 4m (x9 over 20m) kubelet, e527bacs9002 Container image "iolregistry.azurecr.io/panviva/doc:v1" already present on machine
Normal Created 4m (x9 over 20m) kubelet, e527bacs9002 Created container
Warning Failed 4m (x9 over 20m) kubelet, e527bacs9002 Error: failed to start container "getdocumentservice": Error response from daemon: {"message":"container getdocumentservice encountered an error during CreateProcess: failure in a Windows system call: The system cannot find the file specified. (0x2) extra info: {\"ApplicationName\":\"\"
,\"CommandLine\":\"./modules/GetDocumentService/GetDocumentService.exe\",\"User\":\"\",\"WorkingDirectory\":\"C:\\\\app\\\\modules\",\"Environment\":{\"KUBERNETES_PORT\":\"tcp://10.0.0.1:443\",\"KUBERNETES_PORT_443_TCP\":\"tcp://10.0.0.1:443\",\"KUBERNETES_PORT_443_TCP_ADDR\":\"10.0.0.1\",\"KUBERNETES_PORT_443_TCP_PORT\":\"443\",\"KUBERNETES_PORT_443_TCP_PROTO\
":\"tcp\",\"KUBERNETES_SERVICE_HOST\":\"10.0.0.1\",\"KUBERNETES_SERVICE_PORT\":\"443\",\"KUBERNETES_SERVICE_PORT_HTTPS\":\"443\"},\"EmulateConsole\":false,\"CreateStdInPipe\":true,\"CreateStdOutPipe\":true,\"CreateStdErrPipe\":true,\"ConsoleSize\":[0,0]}"}