K
Q

Readiness probe failed: HTTP probe failed with statuscode: 503 in ISTIO

5/6/2019

I have deployed istio service mesh in my AKS cluster. i have ui and backend services both configured in istio for service communication. i am able to invoke backend service through istio, and no problem in accessing my backend services.

But i am facing issue in test-ui pod, when i see pod status

NAME                                  READY   STATUS    RESTARTS   AGE
Test-api-deployment-59f6c6f67-ml4xm   2/2     Running   0          3d21h
Test-ui-deployment-b54fd89b-2ndsv     1/2     Running   0          52m

In above status, my ui pod , one of my istio container is not in ready state to serve my request.. when i looked into the container state, i get below error..

Warning  Unhealthy  2m24s (x299 over 12m)  kubelet, aks-wmsdevk8s-25812762-4  
Readiness probe failed: HTTP probe failed with statuscode: 503

i see it an open issue issue in github.. is there any workaround for this...

EDIT

my istio version

version.BuildInfo{Version:"1.1.5", GitRevision:"9b6d31b74d1c0cc9358cc82d395b53f71393326b", User:"root", Host:"3e29fde4-6c3f-11e9-b00d-0a580a2c0205", GolangVersion:"go1.10.4", DockerHub:"docker.io/istio", BuildStatus:"Clean", GitTag:"1.1.4-10-g9b6d31b"}

istio proxy version -   Image:         docker.io/istio/proxyv2:1.1.2
-- pappu_kutty
azure
azure-aks
azure-kubernetes
istio
kubernetes

Similar Questions

grpc-dart - how to set the HEADER in grpc-dart - I can only do it via a secure connection
Prometheus alert manager configuration in openshift
Kubernetes - validate deployments
how to pass environment variable in kubectl deployment?

4 Answers

5/30/2019

In my case, I tested on OpenShift 3.11 with ServiceMesh TP 0.10 and used ovs-multitenant(SDN plugin) so inter-projects communications were blocked. I had same problem and set netid of istio-system namespace to "0", then problem was solved. Check network communications between your application's namespace and namespace which has istio-sidecar-injector.

-- Insu Yun
Source: StackOverflow
3/5/2020

You need to enable liveness probe for pods:

If you already installed Istio, you can change config map in namespace istio-system

kubectl get cm istio-sidecar-injector -n istio-system -o yaml | \ 
sed -e 's/rewriteAppHTTPProbe:false/rewriteAppHTTPProbe:true/' | \
kubectl apply -f -

If you are going to install Istio, you can enable it globally using istioctl

istioctl manifest apply --set values.sidecarInjectorWebhook.rewriteAppHTTPProbe=true

If you want to enable HTTP probe for specific deployment or pod without globally affecting on other installed resources, simple annotate pod with sidecar.istio.io/rewriteAppHTTPProbers: "true". in case you want to customize probe section, take a look at the following example:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: liveness-http
spec:
  selector:
    matchLabels:
      app: liveness-http
      version: v1
  template:
    metadata:
      labels:
        app: liveness-http
        version: v1
      annotations:
        sidecar.istio.io/rewriteAppHTTPProbers: "true"
    spec:
      containers:
      - name: liveness-http
        image: docker.io/istio/health:example
        ports:
        - containerPort: 8001
        livenessProbe:
          httpGet:
            path: /foo
            port: 8001
          initialDelaySeconds: 5
          periodSeconds: 5

For more information about Health Checking of Istio Services, check out their documentation here.

-- Muhammad Soliman
Source: StackOverflow
11/21/2019

You can inject your sidecar proxy and rewrite the app probe endpoint using istioctl

kb get <deployment/statefulset> -n <namespace> <resource_name> -o yaml | istioctl kube-inject --rewriteAppProbe -f - | kubectl apply -f -

-- Elieser Jose Pereira Reyes
Source: StackOverflow
5/6/2019

It should work with istio 1.1.5. I suppose you didnt configure istio to rewrite HTTPProbes. You need to install Istio with the sidecarInjectorWebhook.rewriteAppHTTPProbe=true (source).

You can check your istio-sidecar-injector configmap, it should have rewriteAppHTTPProbe: true

ps. it works for me with istio 1.1.2, i believe

-- 4c74356b41
Source: StackOverflow