• Istio: 1.3 (also tried 1.1 before update to 1.3)
• K8s: 1.16.2
• Cloud provider: DigitalOcean

I have a cluster setup with Istio. I have enabled grafana/kiali and also installed kibana and RabbitMQ management UI and for all of those I have gateways and virtual services configured (all in istio-system namespace) along with HTTPS using SDS and cert-manager and all works fine. It means I can access these resources in the browser over HTTPS with a sub domain.

Then I deployed a microservice (part of a real application) and created Service, VirtualService and Gateway resources for it (for now it is the only one service and gateway except rabbitmq which uses different sub domain and differend port). And it is located in default namespace.

$ kubectl get gateway

NAME             AGE
gateway-rabbit   131m
tg-gateway       45m

$ kubectl get po

NAME                           READY     STATUS    RESTARTS   AGE
rabbit-rabbitmq-0              2/2       Running   2          134m
tg-app-auth-79c578b94f-mqsz9   2/2       Running   0          46m

If I try to connect to my service with port forwarding I can get a success response from localhost:8000/api/me (also healthz, readyz both return 200 and pod has 0 restarts) so it is working fine.

kubectl port-forward $(kubectl get pod --selector="app=tg-app-auth" --output jsonpath='{.items[0].metadata.name}') 8000:8000

But I can't access it neither via HTTP nor HTTPS. I get 404 using HTTP and the following response using HTTPS:

*   Trying MYIP...
* TCP_NODELAY set
* Connected to example.com (MYIP) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to www.example.com:443 
* Closing connection 0
curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to www.example.com:443

Here are my yaml files:

Gateway:

apiVersion: networking.istio.io/v1alpha3

kind: Gateway

metadata:
  name: tg-gateway
  namespace: default

spec:
  selector:
    istio: ingressgateway

  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - www.example.com
    tls:
      httpsRedirect: true
  - port:
      number: 443
      name: https
      protocol: HTTPS
    hosts:
    - www.example.com
    tls:
      mode: SIMPLE
      serverCertificate: sds
      privateKey: sds
      credentialName: tg-certificate

Service:

apiVersion: v1

kind: Service

metadata:
  name: tg-app-auth
  namespace: default
  labels:
    app: tg-app-auth

spec:
  selector:
    app: tg-app-auth

  ports:
  - name: http
    port: 8000

VirtualService

apiVersion: networking.istio.io/v1alpha3

kind: VirtualService

metadata:
  name: tg-app-auth-vs
  namespace: default

spec:
  hosts:
  - www.example.com

  gateways:
  - tg-gateway

  http:
  - match:
    - port: 443
    - uri:
        prefix: /api/auth
    rewrite:
      uri: /api
    route:
    - destination:
        host: tg-app-auth
        port:
          number: 8000

---

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: tg-app-auth-dr
  namespace: default
spec:
  host: tg-app-auth
  trafficPolicy:
    tls:
      mode: DISABLE

I tried to remove all the HTTPS and TLS details and configure it with HTTP only but still can not get any response. I read all the issues on github but nothing helps and it seems like I have a very silly mistake. All these configurations are pretty much the same as I have for grafana/kibana/kiali/rabbit and all of them works fine.

UPD: Tried to get response with and it also works fine but I can't get response from LB IP or domain

kubectl exec $(kubectl get pod --selector app=tg-app-auth --output jsonpath='{.items[0].metadata.name}') -c istio-proxy -- curl -v http://$(kubectl get endpoints tg-app-auth -o jsonpath='{.subsets[0].addresses[0].ip}'):8000/api/me

$ kubectl get endpoints tg-app-auth

NAME          ENDPOINTS          AGE
tg-app-auth   10.244.0.37:8000   22h

UPD

All statuses are OK. There are a lot more with different ports but I copied 80/443 only.

$ istioctl authn tls-check <pod_name>

cert-manager-webhook.istio-system.svc.cluster.local:443 
istio-galley.istio-system.svc.cluster.local:443
istio-ingressgateway.istio-system.svc.cluster.local:80
istio-ingressgateway.istio-system.svc.cluster.local:443
istio-sidecar-injector.istio-system.svc.cluster.local:443
kubernetes.default.svc.cluster.local:443

$ kubectl get ingress --all-namespaces
No resources found.

$ kubectl get gateways --all-namespaces

default        gateway-rabbit                    3d2h
default        tg-gateway                        17h
istio-system   gateway-grafana                   3d2h
istio-system   gateway-kiali                     3d2h
istio-system   istio-autogenerated-k8s-ingress   3d2h
logging        gateway-kibana                    3d2h