One Deployment is reported with unhealthy upstream

4/16/2018

Using Istio with Kubernetes and facing an issue with a simple routerule splitting traffic between two Deployments. Only one of the Deployment (howdy) is returning results correctly. The other Deployment (hello) is reporting "no healthy upstream".

Here is the k8s manifest:

apiVersion: v1
kind: Service
metadata:
  name: greeting
  labels:
    name: greeting
spec:
  selector:
    app: greeting
  ports:
  - name: http
    protocol: TCP
    port: 8081
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: greeting-hello
  labels:
    name: greeting-hello
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: greeting
        greeting: hello
    spec:
      containers:
      - name: greeting
        image: arungupta/greeting:hello
        imagePullPolicy: Always
        ports:
        - containerPort: 8081
        readinessProbe:
          httpGet:
            path: /resources/greeting
            port: 8081
          initialDelaySeconds: 50
          periodSeconds: 5
--- apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: greeting-howdy
  labels:
    name: greeting-howdy
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: greeting
        greeting: howdy
    spec:
      containers:
      - name: greeting
        image: arungupta/greeting:howdy
        imagePullPolicy: Always
        ports:
        - containerPort: 8081
        readinessProbe:
          httpGet:
            path: /resources/greeting
            port: 8081
          initialDelaySeconds: 50
          periodSeconds: 5

And the route is:

apiVersion: config.istio.io/v1alpha2
kind: RouteRule
metadata:
  name: greeting-50-50
spec:
  destination:
    name: greeting
  route:
  - labels:
      app: greeting
      greeting: hello
    weight: 100
  - labels:
      app: greeting
      greeting: howdy
    weight: 0

Any idea?

This is also tracked at https://github.com/aws-samples/aws-microservices-deploy-options/issues/239

-- Arun Gupta
istio
kubernetes

1 Answer

4/17/2018

I've reproduced your deployment issue and found the following errors during the process:

Warning Unhealthy 48m (x7 over 49m) kubelet, gke-cluster-1-default-pool-e44042f6-kndh Readiness probe failed: Get http://10.0.2.30:8081/resources/greeting: dial tcp 10.0.2.30:8081: getsockopt: connection refused

I would recommend you to check if docker containers in this particular configuration are exposing endpoints on the specified (8081) port. I also found that pods are not registered in any of the endpoints:

$ kubectl describe service
[...]
IP:                10.3.247.97
Port:              http  8081/TCP
TargetPort:        8081/TCP
Endpoints:
Session Affinity:  None
Events:            <none>

The problem is that the application did not bind to the port which was set as a container port, and the application did not receive connection.

-- d0bry
Source: StackOverflow