I have the following routing.yaml file and two services running on GKE cluster
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: frontend
  namespace: prefix
spec:
  hosts:
    - frontend
  http:
    - route:
      - destination:
          host: frontend
          subset: prod
        weight: 100
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: backend
  namespace: prefix
spec:
  hosts:
    - backend
  http:
    - route:
      - destination:
          host: backend
          subset: prod
        weight: 100Now whenever I hit the http request I expect that my frontend-prod should get all the request and response should be the same. But it turns out it is different and worst part is other server gets hit randomly Following is the output
http GET xx.xx.xx.xx:6756/get_prefix class==d 'Authorization: Token 95cd418693b14ddc87220430e7225ab5'
HTTP/1.1 403 Forbidden
content-length: 159
content-type: text/html
date: Thu, 24 Oct 2019 09:32:25 GMT
server: istio-envoy
x-envoy-decorator-operation: frontend.prefix.svc.cluster.local:6756/*
x-envoy-upstream-service-time: 4
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>403 Forbidden</title>
<h1>Forbidden</h1>
<p>YOUR TOKEN IS INVALID, YOU CAN'T ASK FOR PREFIX</p>But some other time I get following
http GET xx.xx.xx.xx:6756/get_prefix class==d 'Authorization: Token 95cd418693b14ddc87220430e7225ab5'
HTTP/1.1 400 Bad Request
content-length: 143
content-type: text/html
date: Thu, 24 Oct 2019 09:32:26 GMT
server: istio-envoy
x-envoy-decorator-operation: frontend.prefix.svc.cluster.local:6756/*
x-envoy-upstream-service-time: 12
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>400 Bad Request</title>
<h1>Bad Request</h1>
<p>Could not parse the request</p>This does not make sense all the traffic should go to prod front end not the canary one. I can see the logs in both of the containers which confirms that request goes there.
Below is my service.yaml if it helps
apiVersion: v1
kind: Service
metadata:
  name: frontend
  labels:
    app: frontend
  namespace: prefix
spec:
  selector:
    app: frontend
  type: LoadBalancer
  ports:
    - port: 6756
      targetPort: 6756
      name: http
Backend service file is similar with port 6757. How to make sure that traffic goes to one pod only?
kubectl get pods --namespace prefix
NAME                               READY   STATUS    RESTARTS   AGE
backend-canary-7978f77b58-7qzh6    2/2     Running   0          126m
backend-prod-5ff66456f9-zwxxn      2/2     Running   0          126m
frontend-canary-7dd5c45dfc-lcfh6   2/2     Running   0          126m
frontend-prod-7f6d9b5ddc-bkk5h     2/2     Running   0          126m