I have a running container showing running but not ready and I can't for the life of my figure out what's wrong. I am on google container engine running kubectl server 1.64 and kubectl client 1.62

My healthcheck route is indicating that the healthcheck is being requested, albeit directly on the pod ip and not the service ip. I say this because when within a different pod, I am unable to curl the service that is not ready.

Nothing seems to be wrong when I look at my yaml files which I've included below. I've also included some commands I've run and their outputs.

Within the container, I can confirm secrets exist because I can echo $MYSQL_USER so I know they are being set.

Finally, my docker file is extremely simply and runs the server on port 7000 so not sure that could be it. What am I doing wrong? How can I go about debugging this?

from a different pod

wget -qO- blahblah

wget: can't connect to remote host (10.55.252.109): Operation timed out

nslookup blahblah:

nslookup: can't resolve '(null)': Name does not resolve

Name:      blahblah-service
Address 1: 10.55.252.109 blahblah-service.staging.svc.cluster.local

from within the gke vm

curl 10.55.252.109

Failed to connect to 10.55.252.109 port 80: Connection refused

from my terminal:

kubectl get svc blahblah

NAME                       CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
blahblah-service   10.55.252.109   <nodes>       80:31003/TCP   41m

kubectl get endpoints blahblah-service

NAME                       ENDPOINTS   AGE
blahblah-service               29m

kubectl get pods -l name=blahblah

NAME                                           READY     STATUS    RESTARTS   AGE
blahblah-3521139244-6tkj5   0/1       Running   0          29m

kubectl describe svc blahblah-service

Name:           blahblah-service
Namespace:          staging
Labels:         <none>
Annotations:        <none>
Selector:           name=blahblah
Type:           NodePort
IP:             10.55.252.109
Port:           http    80/TCP
NodePort:           http    31003/TCP
Endpoints:
Session Affinity:       None
Events:         <none>

service & deployment yamls

kind: Service
apiVersion: v1
metadata:
  name: blahblah-service
spec:
  selector:
    name: blahblah
  type: NodePort
  ports:
    - port: 80
      targetPort: "http"
      name: http

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: blahblah-deployment
spec:
  replicas: 1
  minReadySeconds: 30
  template:
    metadata:
      labels:
        name: blahblah
    spec:
      containers:
      - name: blahblah
        image: eu.gcr.io/company_name/blahblah-service
        ports:
          - containerPort: 7000
            name: http
        readinessProbe:
          httpGet:
            path: /_internal_/ok
            port: http
        env:
          - name: NODE_ENV
            valueFrom:
              configMapKeyRef:
                name: env-variables
                key: ENV_NAME
          - name: MYSQL_USER
            valueFrom:
              secretKeyRef:
                name: blahblah-mysql-secrets
                key: MYSQL_USER
          - name: MYSQL_PASS
            valueFrom:
              secretKeyRef:
                name: blahblah-mysql-secrets
                key: MYSQL_PASS

dockerfile

FROM node:7.5.0-alpine

RUN apk update && apk upgrade \
    && apk add --no-cache --update-cache git ca-certificates openssl\
    && update-ca-certificates

RUN cd /tmp && wget https://yarnpkg.com/latest.tar.gz && tar zxf latest.tar.gz

COPY package.json /workspace/package.json
WORKDIR /workspace
COPY yarn.lock /workspace/yarn.lock
RUN /tmp/dist/bin/yarn

COPY . /workspace

ARG NODE_ENV=production
ENV NODE_ENV ${NODE_ENV}

ENV PORT 7000

EXPOSE $PORT

ENTRYPOINT exec node_modules/pm2/bin/pm2-docker start pm2.json --only blahblah-service-$NODE_ENV

EDIT

I also noticed that it only shows not ready when another service is run at the same time. I don't see any similarities between the two services so I'm not sure how running both would conflict? Both services use a configmap and have their own secrets but I don't see why that would cause conflicts.