For the Google managed version of Istio (enabled by checking the box on your GKE cluster) then versions 1.13 and above have the access logs disabled by default, having the configmap accessLogFile: "". On 1.12 or older versions, the access logs are enabled by default so the configmap has accessLogFile: "/dev/stdout".

As you have noted, you are unable to change it as the reconciliation will wipe the change.

I logged a support case with Google to find out the best approach and they suggested using the Mixer logs instead. To access these, you need Stackdriver enabled on your GKE cluster (either legacy or the newer Kubernetes Engine monitoring). You can then use the filter logName="projects/[PROJECT-NAME]/logs/server-accesslog-stackdriver.logentry.istio-system".

To see the requests between two services you would use this Stackdriver query:

logName="projects/[PROJECT-NAME]/logs/server-accesslog-stackdriver.logentry.istio-system"
labels.destination_app="[YOUR-SERVICE]"
labels.source_app="[YOUR-OTHER-SERVICE]"

To see the requests originating from outside GKE and flowing through the Istio Ingress Gateway:

logName="projects/[YOUR-PROJECT]/logs/server-accesslog-stackdriver.logentry.istio-system"
labels.destination_app="[YOUR-SERVICE]"
labels.source_app="istio-ingressgateway"

These logs aren't 100% equivalent to the proxy access logs however, and may not help troubleshooting all scenarios. There is a feature request open with Google to support customization of the Istio config-map including the accessLogFile setting: https://issuetracker.google.com/issues/126527530

I'd suggest anyone interested in this feature should vote for it by adding a star.

Hope that helps!

In GKE, all stdout and stderr gets collected and sent to the node's log-rotate for later parsing and exporting into Stackdriver, via Fluentd.

The Access Logs are available using kubectl logs command, meaning that they're in the node and they're being parsed and exported using the Fluentd agent.

I replicated this and was able to find the Access Logs using the following Stackdriver advaced filter (change it according to your own resources):

resource.type="container"
resource.labels.cluster_name="gke-cluster"
resource.labels.namespace_id="application-namespace"
resource.labels.project_id="project-id"
resource.labels.zone:"gcp-zone1-a"
resource.labels.container_name="istio-proxy"
resource.labels.pod_id:"sleep-"

The important lines are the resource.labels.container_name="istio-proxy" to query the istio-proxy container and looking into each replica of the interested pod with resource.labels.pod_id:"sleep-".

Regarding the configMap, since GKE is a managed Kubernetes implementation, you're not supposed to change many configurations, including Fluentd. A reconciliation loop will automatically reset any changes attempted to these resources.

If you really need it, you can deploy your own unmanaged version of Fluentd when using GKE.