K
Q

Timeouts when creating resources in Istio 1.4.0 when installed with istioctl

12/2/2019

I am attempting to install Istio 1.4.0 on a new GKE cluster. When I use the soon to be deprecated Helm installation process, Istio works correctly and I can apply my own resources (gateways, deployments etc) without any issues, e.g.:

# Create namespace
kubectl create namespace istio-system

# Install Istio CRDs
helm template install/kubernetes/helm/istio-init --name istio-init --namespace istio-system | kubectl apply -f -

# Install Istio
helm template install/kubernetes/helm/istio --name istio --namespace istio-system \
  --set gateways.enabled=true \
  --set gateways.istio-ingressgateway.enabled=true \
  --set gateways.istio-ingressgateway.sds.enabled=true \
  --set gateways.istio-ingressgateway.externalTrafficPolicy="Local" \
  --set global.disablePolicyChecks=false \
  --set global.proxy.accessLogFile="/dev/stdout" \
  --set global.proxy.accessLogEncoding="TEXT" \
  --set grafana.enabled=true \
  --set grafana.security.enabled=true \
  --set kiali.enabled=true \
  --set prometheus.enabled=true \
  --set tracing.enabled=true \
  | kubectl apply -f -

However, when I attempt to install Istio using the istioctl process, e.g.:

istioctl manifest apply \
  --set values.gateways.enabled=true \
  --set values.gateways.istio-ingressgateway.enabled=true \
  --set values.gateways.istio-ingressgateway.sds.enabled=true \
  --set values.global.disablePolicyChecks=false \
  --set values.global.proxy.accessLogFile="/dev/stdout" \
  --set values.global.proxy.accessLogEncoding="TEXT" \
  --set values.grafana.enabled=true \
  --set values.grafana.security.enabled=true \
  --set values.kiali.enabled=true \
  --set values.prometheus.enabled=true \
  --set values.tracing.enabled=true

...I am unable to create resources as the kubectl apply command times out, e.g.:

$ kubectl apply -f default-gateway.yaml
Error from server (Timeout): error when creating "default-gateway.yaml": Timeout: request did not complete within requested timeout 30s

This happens for every type of resource that I try to create. Has anyone else experienced something similar or is aware of what the underlying issue is?

Running Istio analyze does not reveal any issues:

$ istioctl x analyze -k
✔ No validation issues found.

-- Edit --

Trying to run busybox also errors:

$ kubectl run -i --tty busybox --image=busybox --restart=Never -- sh
Error from server (InternalError): Internal error occurred: failed calling webhook "sidecar-injector.istio.io": Post https://istio-sidecar-injector.istio-system.svc:443/inject?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
-- Andrew Ridout
google-kubernetes-engine
istio
kubernetes

Similar Questions

Kubernetes is not discovering the service through feignclient
Kubernetes Dashboard - Unknown server error after login
How do we Look up service-cluster-ip-range in Azure AKS Cluster
How to install only Fluentd on Openshift 3.11
load distribution between pods in hpa
Can not access selenium gird console

2 Answers

12/3/2019

I tried to replicate your issue but was not able too. I was able to install istio with the following commands:

curl -L https://istio.io/downloadIstio | sh -
cd istio-1.*
export PATH=$PWD/bin:$PATH
istioctl manifest apply

I was then able to run this command and it completed successfully:

kubectl run -i --tty busybox --image=busybox --restart=Never -- sh
-- Frederic G
Source: StackOverflow
12/3/2019

I figured this out. The problem is the istio-sidecar-injector service and that I'm using a private GKE cluster which has more restrictive default firewall rules than a non-private cluster.

When installing Istio via the Helm process, the istio-sidecar-injector service targets port 443:

$ kubectl get svc istio-sidecar-injector -n istio-system -o jsonpath='{.spec.ports[0]}'
map[name:https-inject port:443 protocol:TCP targetPort:443]

Port 443 is open by default on the master firewall rule (e.g. gke-<cluster-name>-XXXXXXXX-master), so the istio-sidecar-injector can operate successfully.

However, when installing Istio via the new Istioctl process, the istio-sidecar-injector service targets port 9443 instead of 443:

~ $ kubectl get svc istio-sidecar-injector -n istio-system -o jsonpath="{.spec.ports[0]}"
map[port:443 protocol:TCP targetPort:9443]

This port is not open by default on the master firewall rule and is the cause of the timeout errors when trying to deploy resources, such as:

$ kubectl run -i --tty busybox --image=busybox --restart=Never -- sh
Error from server (InternalError): Internal error occurred: failed calling webhook "sidecar-injector.istio.io": Post https://istio-sidecar-injector.istio-system.svc:443/inject?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

After installing Istio, I opened port 9443 on the firewall like this:

VPC_PROJECT=<project containing VPC network>
CLUSTER_NAME=<name of the GKE cluster>

FIREWALL_RULE_NAME=$(gcloud compute firewall-rules list --project $VPC_PROJECT --filter="name~gke-$CLUSTER_NAME-[0-9a-z]*-master" --format="value(name)")
gcloud compute firewall-rules update $FIREWALL_RULE_NAME --project $VPC_PROJECT --allow tcp:10250,tcp:443,tcp:9443

I was then able to create resources without error:

$ kubectl run -i --tty busybox --image=busybox --restart=Never -- sh
If you don't see a command prompt, try pressing enter.
/ # ls
bin   dev   etc   home  proc  root  sys   tmp   usr   var
/ #
-- Andrew Ridout
Source: StackOverflow