I was able to get it working using Istio without mTLS using the following definitions:

ConfigMap

Just allowing the pod and service CIDRs for BAN requests and expecting them to come from the Varnish service FQDN.

apiVersion: v1
kind: ConfigMap
metadata:
  name: varnish-configuration
data:
  default.vcl: |
    vcl 4.0;

    import std;

    backend default {
      .host = "varnish-service";
      .port = "80";      
    }

    acl purge {
        "localhost";
        "10.x.0.0"/14; #Pod CIDR
        "10.x.0.0"/16; #Service CIDR
    }

    sub vcl_recv {
            # this code below allows PURGE from localhost and x.x.x.x

            if (req.method == "BAN") {
                    if (!client.ip ~ purge) {
                            return(synth(405,"Not allowed."));
                    }
                    return (purge);
            }
    }

Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: varnish
spec:
  replicas: 1
  selector:
    matchLabels:
      app: varnish
  template:
    metadata:
      labels:
        app: varnish
    spec:
      containers:
      - name: varnish
        image: varnish:6.3
        ports:        
        - containerPort: 80
          name: varnish-port
        imagePullPolicy: IfNotPresent        
        volumeMounts:
        - name: varnish-conf
          mountPath: /etc/varnish
      volumes:
      - name: varnish-conf
        configMap:
          name: varnish-configuration

Service

apiVersion: v1
kind: Service
metadata:
  name: varnish-service
  labels:
    workload: varnish
spec:
  selector:
    app: varnish
  ports:    
    - name: varnish-port
      protocol: TCP
      port: 80
      targetPort: 80

After deploying these, you can run a cURL enabled pod:

kubectl run bb-$RANDOM --rm -i --image=yauritux/busybox-curl --restart=Never --tty -- /bin/sh

And then, from the tty try curling it:

curl -v -X BAN http://varnish-service

From here, either you'll get 200 purged or 405 Not allowed. Either way, you've hit the Varnish pod across the mesh.

Your issue might be related to mTLS in your cluster. You can check if it's enabled by issuing this command*:

istioctl authn tls-check $(k get pod -l app=varnish -o jsonpath={.items..metadata.name}) varnish-service.default.svc.cluster.local

*The command assumess that you're using the definitions shared in this post. If not, you can adjust accordingly.

I tested this running GKE twice: One with open source Istio via Helm install and another using the Google managed Istio installation (in permissive mode).