How to get an Istio VirtualService to vary routes by header along with uri

12/8/2019

I'm still experimenting with Istio in a dev cluster, along with a couple of other people. We have a sample virtualservice, deployment, and destinationrule, and requests to the specified uri are going to the pod associated with the deployment. This is working fine.

I am attempting a variation of this that goes to an alternate deployment if a particular header and value are present. If the header is not set, or the specified header value is not present, it will go to the original deployment, otherwise to the alternate deployment. I eventually intend for it to check for several different values of the specified header, each going to different deployments.

I've created a deployment and destination rule that are copies of the original, with consistent variations. I attempted to modify the virtualservice with this alternate routing. Thus far, it isn't working properly. I determine which deployment a request goes to by tailing the container log of the container associated with each deployment. When I sent a request with the specified header and value, it does go to the alternate deployment. However, when I send the request without the specified header, or without the matching value, it ALSO goes to the alternate deployment. In fact, I can't get it to reach the main deployment at all.

Note that I understand that another way to do this is to have one virtualservice for the "default" route, and an additional virtualservice for each alternate route, specifying a different header value. I've seen something basically like that working. However, that seems like a lot of duplication to get something that should be simpler to set up in a single VirtualService.

The following is the current state of the virtualservice, with some elisions:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  annotations: {}
  name: fooms-vs-ingress
  namespace: com-example
spec:
  gateways:
  - ingress-gateway
  hosts:
  - '*'
  http:
  - match:
    - uri:
        prefix: /msapi/foo
    - headers:
        ctx-debug-route-fooms:
          exact: myuid-1
    route:
    - destination:
        host: fooms.com-example.svc.cluster.local
        port:
          number: 80
        subset: myuid-1
  - route:
    - destination:
        host: fooms.com-example.svc.cluster.local
        port:
          number: 80
        subset: blue

I could show the deployments and destinationrules, but I don't know if that will be helpful.

Update:

Since I wrote this, I discovered that in order to make two conditions AND in a route match, I have to have both conditions in a single match rule. I'm still getting used to how YAML works. I'm going to provide here an updated version of the virtualservice, along with the gateway, destination rule, and much of the deployment. There's a lot of stuff in the deployment that probably isn't helpful.

When I sent a request to the service from Postman, with or without the routing header, I get a 503 back. Before I made these changes to check for the routing header, it was properly routing requests to the "blue" instance (I am tailing the logs for both pods). When I first tried making these changes, I inadvertently defined two match blocks, one with the uri condition, and one with the header match condition. When I did that, all of the requests were going to the alternate pod.

Here are elided versions of the objects that might be relevant, with some transient properties removed.

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  annotations: {}
  name: fooms-vs-ingress
  namespace: com-example
spec:
  gateways:
  - comp-ingress-gateway
  hosts:
  - '*'
  http:
  - match:
    - headers:
        compctx-debug-route-fooms:
          exact: myuid-1
      name: match-myuid-1
      uri:
        prefix: /msapi/foo
    route:
    - destination:
        host: fooms.com-example.svc.cluster.local
        port:
          number: 80
        subset: myuid-1
  - name: default
    route:
    - destination:
        host: fooms.com-example.svc.cluster.local
        port:
          number: 80
        subset: blue

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  annotations: {}
  name: comp-ingress-gateway
  namespace: com-example
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - '*'
    port:
      name: http
      number: 80
      protocol: HTTP
  - hosts:
    - '*'
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      mode: SIMPLE
      privateKey: /etc/istio/ingressgateway-certs/tls.key
      serverCertificate: /etc/istio/ingressgateway-certs/tls.crt

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
  labels:
    app: FooMS
    role: blue
    routeoffer: DEFAULT
    seed: COMPv2.2.0
    version: 2.2.0-myuid-1
  name: fooms-myuid-1
  namespace: com-example
spec:
  replicas: 1
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app: FooMS
      role: blue
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      annotations:
        sidecar.istio.io/inject: "true"
      creationTimestamp: null
      labels:
        app: FooMS
        role: blue
        routeoffer: DEFAULT
        seed: COMPv2.2.0
        version: 2.2.0-myuid-1
    spec:
      containers:
      - env:
        - name: SERVICE_NAME
          value: fooms
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        envFrom:
        - configMapRef:
            name: global-config
        - secretRef:
            name: global-secrets
        image: dockercentral.it....
        imagePullPolicy: Always
        name: fooms
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
      - image: ...
        imagePullPolicy: IfNotPresent
        name: info
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: default
      serviceAccountName: default
      terminationGracePeriodSeconds: 30

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  annotations: {}
  name: fooms-destination-myuid-1
  namespace: com-example
spec:
  host: fooms.com-example.svc.cluster.local
  subsets:
  - labels:
      version: 2.2.0-myuid-1
    name: myuid-1
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
-- David M. Karr
istio
kubernetes

1 Answer

12/12/2019

Since I don't have all informations from this command kubectl get pods,ep,svc -o wide i'm not sure that's destination rule error because i don't know everything about your pods,ep,svc and your destination rule applies to same deployment with only 1 replica.

Probably the problem there is destination rule with only 1 subset

Based on your virtual service i think that's how your destination rule should look

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: fooms-destination-myuid-1
  namespace: com-example
spec:
  host: fooms.com-example.svc.cluster.local
  subsets:
  - labels:
      version: 2.2.0-myuid-1
    name: myuid-1
  subsets:
  - labels:
       role: blue
    name: blue
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL

I made some example, everything needed to make this below

Kubernetes version 1.13.11-gke.14

Istio version 1.4.1

kubectl label namespace default istio-injection=enabled

Deployment 1

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx1
spec:
  selector:
    matchLabels:
      run: nginx1
  replicas: 1
  template:
    metadata:
      labels:
        run: nginx1
        app: frontend
    spec:
      containers:
      - name: nginx1
        image: nginx
        ports:
        - containerPort: 80
        lifecycle:
          postStart:
            exec:
              command: ["/bin/sh", "-c", "echo Hello nginx1 > /usr/share/nginx/html/index.html"]

Deployment 2

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx2
spec:
  selector:
    matchLabels:
      run: nginx2
  replicas: 1
  template:
    metadata:
      labels:
        run: nginx2
        app: frontend
    spec:
      containers:
      - name: nginx2
        image: nginx
        ports:
        - containerPort: 80
        lifecycle:
          postStart:
            exec:
              command: ["/bin/sh", "-c", "echo Hello nginx2 > /usr/share/nginx/html/index.html"]

Service

apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: frontend
spec:
  ports:
  - port: 80
    protocol: TCP
  selector:
    app: frontend

Gateway

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: comp-ingress-gateway
  namespace: default
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - '*'
    port:
      name: http
      number: 80
      protocol: HTTP
  - hosts:
    - '*'
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      mode: SIMPLE
      privateKey: /etc/istio/ingressgateway-certs/tls.key
      serverCertificate: /etc/istio/ingressgateway-certs/tls.crt

Virtual Service

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: nginxvirt
spec:
  gateways:
  - comp-ingress-gateway #outside cluster
  - mesh #inside cluster
  hosts:
  - nginx.default.svc.cluster.local #inside cluster
  - nginx.com #outside cluster
  http:
  - name: match-myuid
    match:
    - uri:
        prefix: /msapi
      headers:
        compctx:
          exact: myuid
    rewrite:
      uri: /
    route:
    - destination:
        host: nginx.default.svc.cluster.local
        port:
          number: 80
        subset: v1
  - name: default
    route:
    - destination:
        host: nginx.default.svc.cluster.local
        port:
          number: 80
        subset: v2

Destination Rule

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: nginxdest
spec:
  host: nginx.default.svc.cluster.local
  subsets:
  - name: v1
    labels:
      run: nginx1
  - name: v2
    labels:
      run: nginx2
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL

Pod for testing

apiVersion: v1
kind: Pod
metadata:
  name: ubu1
spec:
  containers:
  - name: ubu1
    image: ubuntu
    command: ["/bin/sh"]
    args: ["-c", "sleep 3000"]

And results from curl:

Outside:

curl -H "host: nginx.com" -H "compctx: myuid" ingress_gateway_ip/msapi
Hello nginx1
curl -H "host: nginx.com" -H "compctx: myuid" ingress_gateway_ip/msapi
Hello nginx1
curl -H "host: nginx.com" -H "compctx: myuid" ingress_gateway_ip/msapi
Hello nginx1

curl -H "host: nginx.com" ingress_gateway_ip
Hello nginx2
curl -H "host: nginx.com" ingress_gateway_ip
Hello nginx2
curl -H "host: nginx.com" ingress_gateway_ip
Hello nginx2

Inside:

kubectl exec -ti ubu1 -- /bin/bash

root@ubu1:/# curl -H "compctx: myuid " nginx/msapi
Hello nginx1
root@ubu1:/# curl -H "compctx: myuid " nginx/msapi
Hello nginx1
root@ubu1:/# curl -H "compctx: myuid " nginx/msapi
Hello nginx1
-- jt97
Source: StackOverflow