Edit

The root problem was not what I initially thought. If I rename the backend hostname to blaapi.example.com instead of api.example.com, it works perfectly.

What could cause istio to not find the wanted api.example.com and that it would resolve the blaapi.example.com ?

I couldn't find any documentation on how to get the logs for the istio dns resolver...

k8s version

 curl http://localhost:8001/version -k
{
  "major": "1",
  "minor": "18",
  "gitVersion": "v1.18.3",
  "gitCommit": "2e7996e3e2712684bc73f0dec0200d64eec7fe40",
  "gitTreeState": "clean",
  "buildDate": "2020-05-20T12:43:34Z",
  "goVersion": "go1.13.9",
  "compiler": "gc",
  "platform": "linux/amd64"
}

istioctl version
client version: 1.6.3
control plane version: 1.6.3
data plane version: 1.6.3 (13 proxies)

I have a jenkins build pipeline that builds docker images and deploy them automatically to a minikube cluster. I have a multi-branch pipeline that will deploy a full environment for each branch. And I have a simple javascript file that outputs the k8s templates which is piped to kubectl.

I am using istio as a service mesh, and cert-manager.

I am trying to deploy 2 simple containers, frontend, and backend

Here are the resources created 1. Namespace for the deployment (with istio injection enabled) 3. Issuer (with type selfsigned) 4. Certificate (for the frontend app) 5. Certificate (for the backend app) 6. Istio Gateway (for the frontend app) 7. Istio Gateway (for the backend app) 8. Deployment (for the frontend app) 9. Deployment (for the backend app) 10. Service (for the frontend app) 11. Service (for the backend app) 12. VirtualService (for the frontend app) 13. VirtualService (for the backend app)

kubectl get all -l=project=myproject --all-namespaces

(Istio Gateway)
NAMESPACE      NAME                      AGE
istio-system   master-gateway-backend    3h52m
istio-system   master-gateway-frontend   3h52m

(Istio VirtualService)
NAMESPACE   NAME                  GATEWAYS                                 HOSTS                  AGE
master      myproject-backend    [istio-system/master-gateway-backend]    [api.example.com]   25h
master      myproject-frontend   [istio-system/master-gateway-frontend]   [example.com]       25h

NAMESPACE   NAME                                       READY   STATUS    RESTARTS   AGE
master      pod/myproject-backend-54d55c5795-b9st2    2/2     Running   0          13m
master      pod/myproject-frontend-85dfbdc48f-mdxqh   2/2     Running   0          13m

NAMESPACE   NAME                          TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
master      service/myproject-backend    ClusterIP   10.98.114.81   <none>        80/TCP    23h
master      service/myproject-frontend   ClusterIP   10.109.78.55   <none>        80/TCP    23h

NAMESPACE   NAME                                  READY   UP-TO-DATE   AVAILABLE   AGE
master      deployment.apps/myproject-backend    1/1     1            1           23h
master      deployment.apps/myproject-frontend   1/1     1            1           23h

NAMESPACE   NAME                                             DESIRED   CURRENT   READY   AGE
master      replicaset.apps/myproject-backend-54d55c5795    1         1         1       13m
master      replicaset.apps/myproject-frontend-85dfbdc48f   1         1         1       13m

(Secrets)
NAMESPACE      NAME                          TYPE                DATA   AGE
istio-system   ingress-cert-master-frontend  kubernetes.io/tls   2      24h
istio-system   ingress-cert-master-backend   kubernetes.io/tls   2      24h

(Certificates)
NAMESPACE      NAME                           READY   SECRET                         AGE
istio-system   ingress-cert-master-backend    True    ingress-cert-master-backend    152m
istio-system   ingress-cert-master-frontend   True    ingress-cert-master-frontend   152m

(CertificateRequests)
NAMESPACE      NAME                                     READY   AGE
istio-system   ingress-cert-master-backend-563723203    True    147m
istio-system   ingress-cert-master-frontend-556751135   True    147m

There are no validation errors while deploying the templates from jenkins.

The frontend app host is defined as example.com while the backend app host is defined as api.example.com

The problem is that, the frontend app deploys successfully and is reachable through the istio ingressgateway at https://example.com. Though, for some obscure reason, the backend app is not reachable at https://api.example.com

wget --no-check-certificate  -O- https://example.com -> 200 OK
wget --no-check-certificate  -O- https://api.example.com -> TIMEOUT
kubectl exec -it -n admin network-multitool-659588b964-d5zfc -- wget -O- myproject-backend.master.svc.cluster.local -> 200 OK
INGRESS=$(kubectl get svc -n istio-system istio-ingressgateway -o json | jq -r  .status.loadBalancer.ingress[0].ip)
wget --no-check-certificate --header="Host: api.example.com" http://${INGRESS} -> TIMEOUT
wget --no-check-certificate --header="Host: example.com" http://${INGRESS} -> 200 OK

getent hosts example.com api.example.com
10.99.153.121   example.com
19.99.153.121   api.example.com

Initially, I thought that it was related to the fact that I have multiple Gateways that have ports with the same name. But I fixed that and it didn't resolve anything: https://istio.io/latest/docs/ops/common-problems/network-issues/#404-errors-occur-when-multiple-gateways-configured-with-same-tls-certificate

I know that the Service and Deployment part work because I can wget myproject-backend.master.svc.cluster.local and wget myproject-frontend.master.svc.cluster.local from any pod. So the problem must lie at the istio level.

The weird thing is that if I disable the backend httpsRedirect, then if I wget --header='Host: api.example.com' -O- http://10.99.153.121, then I do resolve the index.html file. But then if I simply wget -O- api.commonpoo.lab, it times out.

Here are the configurations for the Gateways, VirtualServices, Issuer, Certificates

apiVersion: cert-manager.io/v1alpha3
kind: Issuer
metadata:
  labels:
    branch: master
    project: myproject
  name: ca-issuer-master
  namespace: istio-system
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1alpha3
kind: Certificate
metadata:
  labels:
    branch: master
    project: myproject
  name: ingress-cert-master-backend
  namespace: istio-system
spec:
  commonName: api.example.com
  dnsNames:
  - api.example.com
  issuerRef:
    name: ca-issuer-master
  secretName: ingress-cert-master-backend
  subject:
    organizations:
    - myproject
---
apiVersion: cert-manager.io/v1alpha3
kind: Certificate
metadata:
  labels:
    branch: master
    project: myproject
  name: ingress-cert-master-frontend
  namespace: istio-system
spec:
  commonName: example.com
  dnsNames:
  - example.com
  issuerRef:
    name: ca-issuer-master
  secretName: ingress-cert-master-frontend
  subject:
    organizations:
    - myproject
---
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:  
  labels:
    branch: master
    project: myproject
  name: master-gateway-frontend
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - example.com
    port:
      name: http-master-frontend
      number: 80
      protocol: HTTP
    tls:
      httpsRedirect: true
  - hosts:
    - example.com
    port:
      name: https-master-frontend
      number: 443
      protocol: HTTPS
    tls:
      credentialName: ingress-cert-master-frontend
      mode: SIMPLE
---
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  labels:
    branch: master
    project: myproject
  name: master-gateway-backend
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - api.example.com
    port:
      name: http-master-backend
      number: 80
      protocol: HTTP
    tls:
      httpsRedirect: true
  - hosts:
    - api.example.com
    port:
      name: https-master-backend
      number: 443
      protocol: HTTPS
    tls:
      credentialName: ingress-cert-master-backend
      mode: SIMPLE
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  labels:
    app: myproject-backend
    branch: master
    commit: 49cf75c4d79bd1987aea2a753481d94133f99cc6
    project: myproject
  name: myproject-backend
  namespace: master
spec:
  gateways:
  - istio-system/master-gateway-backend
  hosts:
  - api.example.com
  http:
  - route:
    - destination:
        host: myproject-backend.master.svc.cluster.local
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  labels:
    app: myproject-frontend
    branch: master
    commit: 49cf75c4d79bd1987aea2a753481d94133f99cc6
    project: myproject
  name: myproject-frontend
  namespace: master
spec:
  gateways:
  - istio-system/master-gateway-frontend
  hosts:
  - example.com
  http:
  - route:
    - destination:
        host: myproject-frontend.master.svc.cluster.local

Here is my hosts /etc/hosts file

# 10.99.153.121 is the istio-ingressgateway external IP

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
127.0.0.1   host.minikube.internal
10.8.0.12   control-plane.minikube.internal
10.99.153.121 keycloak.example.com
10.99.153.121 pgadmin.example.com
10.99.153.121 registry.example.com
10.99.153.121 jenkins.example.com
10.99.153.121 frontend.example.com
10.99.153.121 sonarqube.example.com
19.99.153.121 api.example.com
10.99.153.121 example.com

Even istioctl recognizes that there is a route to the backend pod

istioctl x describe pod myproject-backend-54d55c5795-b9st2.master
Pod: myproject-backend-54d55c5795-b9st2.master
   Pod Ports: 80 (myproject-backend), 15090 (istio-proxy)
Suggestion: add 'version' label to pod for Istio telemetry.
--------------------
Service: myproject-backend.master
   Port: http 80/HTTP targets pod port 80


Exposed on Ingress Gateway http://10.99.153.121
VirtualService: myproject-backend.master
   1 HTTP route(s)

---

istioctl x describe pod myproject-frontend-85dfbdc48f-mdxqh.master
Pod: myproject-frontend-85dfbdc48f-mdxqh.master
   Pod Ports: 80 (myproject-frontend), 15090 (istio-proxy)
Suggestion: add 'version' label to pod for Istio telemetry.
--------------------
Service: myproject-frontend.master
   Port: http 80/HTTP targets pod port 80


Exposed on Ingress Gateway http://10.99.153.121
VirtualService: myproject-frontend.master
   1 HTTP route(s)

Istioctl proxy status :

istioctl proxy-status
NAME                                                   CDS        LDS        EDS        RDS          PILOT                       VERSION
myproject-backend-54d55c5795-b9st2.master             SYNCED     SYNCED     SYNCED     SYNCED       istiod-7b69ff6f8c-2jdpx     1.6.3
myproject-frontend-85dfbdc48f-mdxqh.master            SYNCED     SYNCED     SYNCED     SYNCED       istiod-7b69ff6f8c-2jdpx     1.6.3
istio-egressgateway-77c7d594c5-2nsz5.istio-system      SYNCED     SYNCED     SYNCED     NOT SENT     istiod-7b69ff6f8c-2jdpx     1.6.3
istio-ingressgateway-766c84dfdc-dmmd8.istio-system     SYNCED     SYNCED     SYNCED     SYNCED       istiod-7b69ff6f8c-2jdpx     1.6.3
keycloak-0.admin                                       SYNCED     SYNCED     SYNCED     SYNCED       istiod-7b69ff6f8c-2jdpx     1.6.3
network-multitool-659588b964-d5zfc.admin               SYNCED     SYNCED     SYNCED     SYNCED       istiod-7b69ff6f8c-2jdpx     1.6.3
pgadmin-549db8c6fb-bjpn9.admin                         SYNCED     SYNCED     SYNCED     SYNCED       istiod-7b69ff6f8c-2jdpx     1.6.3
postgres-postgresql-master-0.admin                     SYNCED     SYNCED     SYNCED     SYNCED       istiod-7b69ff6f8c-2jdpx     1.6.3
postgres-postgresql-slave-0.admin                      SYNCED     SYNCED     SYNCED     SYNCED       istiod-7b69ff6f8c-2jdpx     1.6.3
postgres-postgresql-slave-1.admin                      SYNCED     SYNCED     SYNCED     SYNCED       istiod-7b69ff6f8c-2jdpx     1.6.3
prometheus-5fdfc44fb7-kkm8h.istio-system               SYNCED     SYNCED     SYNCED     SYNCED       istiod-7b69ff6f8c-2jdpx     1.6.3
sonarqube-sonarqube-5b99564cb6-khccq.admin             SYNCED     SYNCED     SYNCED     SYNCED       istiod-7b69ff6f8c-2jdpx     1.6.3

istioctl analyze --all-namespaces
Warn [IST0102] (Namespace default) The namespace is not enabled for Istio injection. Run 'kubectl label namespace default istio-injection=enabled' to enable it, or 'kubectl label namespace default istio-injection=disabled' to explicitly mark it as not needing injection
Warn [IST0102] (Namespace flux-system) The namespace is not enabled for Istio injection. Run 'kubectl label namespace flux-system istio-injection=enabled' to enable it, or 'kubectl label namespace flux-system istio-injection=disabled' to explicitly mark it as not needing injection
Warn [IST0102] (Namespace istio-system) The namespace is not enabled for Istio injection. Run 'kubectl label namespace istio-system istio-injection=enabled' to enable it, or 'kubectl label namespace istio-system istio-injection=disabled' to explicitly mark it as not needing injection
Warn [IST0102] (Namespace kube-node-lease) The namespace is not enabled for Istio injection. Run 'kubectl label namespace kube-node-lease istio-injection=enabled' to enable it, or 'kubectl label namespace kube-node-lease istio-injection=disabled' to explicitly mark it as not needing injection
Info [IST0118] (Service cert-manager.cert-manager) Port name  (port: 9402, targetPort: 9402) doesn't follow the naming convention of Istio port.
Info [IST0118] (Service flux-memcached.flux-system) Port name memcached (port: 11211, targetPort: memcached) doesn't follow the naming convention of Istio port.
Info [IST0118] (Service jenkins-agent.jenkins-system) Port name slavelistener (port: 50000, targetPort: 50000) doesn't follow the naming convention of Istio port.
Error: Analyzers found issues when analyzing all namespaces.
See https://istio.io/docs/reference/config/analysis for more information about causes and resolutions.

I ran out of ideas on how to troubleshoot this. Any help would be greatly appreciated.