Can't see Istio metrics in Apache SkyWalking

10/10/2019

Couldn't see in skywalking ui data from istio metrics.

Using below guide to installed istio and skywalking backend to Kubernetes: https://github.com/apache/skywalking/tree/master/docs/en/setup/istio. Made all steps in the guide and deployed app. Deployed app is example book app from istio. Could see traces in Jaeger and works good, but can't see istio metrics in skywalking ui.

How I can provide metrics from istio to skywalking ui or what I need to configure to make metrics from istio come to skywalking ui.

Mixer logs:

gc 1847 @19617.610s 0%: 0.043+22+0.035 ms clock, 0.17+1.9/28/17+0.14 ms cpu, 16->16->8 MB, 17 MB goal, 6 P
2019-10-11T01:28:44.153821Z warn    grpcAdapter unable to connect to:/metric.HandleMetricService/HandleMetric, oap.skywalking.svc.cluster.local:11800
{"level":"info","time":"2019-10-11T01:28:43.143390Z","instance":"accesslog.instance.istio-system","apiClaims":"","apiKey":"","clientTraceId":"","connection_security_policy":"none","destinationApp":"telemetry","destinationIp":"10.1.3.13","destinationName":"istio-telemetry-7c88764b9c-fncph","destinationNamespace":"istio-system","destinationOwner":"kubernetes://apis/apps/v1/namespaces/istio-system/deployments/istio-telemetry","destinationPrincipal":"","destinationServiceHost":"istio-telemetry.istio-system.svc.cluster.local","destinationWorkload":"istio-telemetry","grpcMessage":"3 errors occurred:%0A%09* fieldEncoder: dimensions - fieldEncoder: value - lookup failed: 'response.time'%0A%09* fieldEncoder: dimensions - fieldEncoder: value - lookup failed: 'response.time'%0A%09* fieldEncoder: dimensions - fieldEncoder: value - lookup failed: 'response.time'%0A%0A","grpcStatus":"2","httpAuthority":"mixer","latency":"6.5979ms","method":"POST","permissiveResponseCode":"none","permissiveResponsePolicyID":"none","protocol":"http","receivedBytes":1317,"referer":"","reporter":"destination","requestId":"b45592ff-0c49-49fd-af64-3c9229b60466","requestSize":931,"requestedServerName":"","responseCode":200,"responseFlags":"-","responseSize":0,"responseTimestamp":"2019-10-11T01:28:43.149845Z","sentBytes":421,"sourceApp":"elasticsearch","sourceIp":"10.1.3.36","sourceName":"skywalking-elasticsearch-data-1","sourceNamespace":"default","sourceOwner":"kubernetes://apis/apps/v1/namespaces/default/statefulsets/skywalking-elasticsearch-data","sourcePrincipal":"","sourceWorkload":"skywalking-elasticsearch-data","url":"/istio.mixer.v1.Mixer/Report","userAgent":"","xForwardedFor":"10.1.3.36"}
{"level":"info","time":"2019-10-11T01:28:43.913917Z","instance":"accesslog.instance.istio-system","apiClaims":"","apiKey":"","clientTraceId":"","connection_security_policy":"none","destinationApp":"telemetry","destinationIp":"10.1.3.13","destinationName":"istio-telemetry-7c88764b9c-fncph","destinationNamespace":"istio-system","destinationOwner":"kubernetes://apis/apps/v1/namespaces/istio-system/deployments/istio-telemetry","destinationPrincipal":"","destinationServiceHost":"istio-telemetry.istio-system.svc.cluster.local","destinationWorkload":"istio-telemetry","grpcMessage":"1 error occurred:%0A%09* fieldEncoder: dimensions - fieldEncoder: value - lookup failed: 'response.time'%0A%0A","grpcStatus":"2","httpAuthority":"mixer","latency":"13.906ms","method":"POST","permissiveResponseCode":"none","permissiveResponsePolicyID":"none","protocol":"http","receivedBytes":698,"referer":"","reporter":"destination","requestId":"ff7ae8af-f279-46b3-ae86-bde2b61477dd","requestSize":312,"requestedServerName":"","responseCode":200,"responseFlags":"-","responseSize":0,"responseTimestamp":"2019-10-11T01:28:43.927780Z","sentBytes":247,"sourceApp":"elasticsearch","sourceIp":"10.1.3.37","sourceName":"skywalking-elasticsearch-master-2","sourceNamespace":"default","sourceOwner":"kubernetes://apis/apps/v1/namespaces/default/statefulsets/skywalking-elasticsearch-master","sourcePrincipal":"","sourceWorkload":"skywalking-elasticsearch-master","url":"/istio.mixer.v1.Mixer/Report","userAgent":"","xForwardedFor":"10.1.3.37"}
{"level":"info","time":"2019-10-11T01:28:43.913675Z","instance":"accesslog.instance.istio-system","apiClaims":"","apiKey":"","clientTraceId":"","connection_security_policy":"none","destinationApp":"telemetry","destinationIp":"10.1.3.13","destinationName":"istio-telemetry-7c88764b9c-fncph","destinationNamespace":"istio-system","destinationOwner":"kubernetes://apis/apps/v1/namespaces/istio-system/deployments/istio-telemetry","destinationPrincipal":"","destinationServiceHost":"istio-telemetry.istio-system.svc.cluster.local","destinationWorkload":"istio-telemetry","grpcMessage":"8 errors occurred:%0A%09* fieldEncoder: dimensions - fieldEncoder: value - lookup failed: 'response.time'%0A%09* fieldEncoder: dimensions - fieldEncoder: value - lookup failed: 'response.time'%0A%09* fieldEncoder: dimensions - fieldEncoder: value - lookup failed: 'response.time'%0A%09* fieldEncoder: dimensions - fieldEncoder: value - lookup failed: 'response.time'%0A%09* fieldEncoder: dimensions - fieldEncoder: value - lookup failed: 'response.time'%0A%09* fieldEncoder: dimensions - fieldEncoder: value - lookup failed: 'response.time'%0A%09* fieldEncoder: dimensions - fieldEncoder: value - lookup failed: 'response.time'%0A%09* fieldEncoder: dimensions - fieldEncoder: value - lookup failed: 'response.time'%0A%0A","grpcStatus":"2","httpAuthority":"mixer","latency":"29.8101ms","method":"POST","permissiveResponseCode":"none","permissiveResponsePolicyID":"none","protocol":"http","receivedBytes":2430,"referer":"","reporter":"destination","requestId":"b0eec606-c91e-4193-aba2-39b069b76474","requestSize":2044,"requestedServerName":"","responseCode":200,"responseFlags":"-","responseSize":0,"responseTimestamp":"2019-10-11T01:28:43.943383Z","sentBytes":857,"sourceApp":"elasticsearch","sourceIp":"10.1.3.37","sourceName":"skywalking-elasticsearch-master-2","sourceNamespace":"default","sourceOwner":"kubernetes://apis/apps/v1/namespaces/default/statefulsets/skywalking-elasticsearch-master","sourcePrincipal":"","sourceWorkload":"skywalking-elasticsearch-master","url":"/istio.mixer.v1.Mixer/Report","userAgent":"","xForwardedFor":"10.1.3.37"}
2019-10-11T01:28:44.154193Z error   api Report failed: 1 error occurred:
    * rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp: lookup oap.skywalking.svc.cluster.local on 10.96.0.10:53: no such host"

Istio policy:

GC forced
gc 181 @20366.930s 0%: 3.9+19+0.030 ms clock, 15+0/30/67+0.12 ms cpu, 13->13->7 MB, 15 MB goal, 6 P
scvg135: 0 MB released
scvg135: inuse: 15, idle: 43, sys: 58, released: 3, consumed: 55 (MB)
2019-10-11T01:42:28.493381Z info    base.baseBalancer: handle SubConn state change: 0xc0011280a0, CONNECTING
2019-10-11T01:42:28.493442Z info    roundrobinPicker: newPicker called with readySCs: map[]
2019-10-11T01:42:28.577142Z info    grpc: addrConn.createTransport failed to connect to {oap.skywalking.svc.cluster.local:11800 0  <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp: lookup oap.skywalking.svc.cluster.local on 10.96.0.10:53: no such host". Reconnecting...
2019-10-11T01:42:28.577256Z info    base.baseBalancer: handle SubConn state change: 0xc0011280a0, TRANSIENT_FAILURE
GC forced
gc 182 @20486.971s 0%: 0.13+13+0.056 ms clock, 0.55+0/13/26+0.22 ms cpu, 13->13->7 MB, 15 MB goal, 6 P

somehow I have failed to connect to my skywalking backend, problem was with name and namespace but still getting error but now skywalking backend getting data but ui no updates. Istio telemetry:

rpc error: code = Unavailable desc = upstream connect error or disconnect/reset before headers. reset reason: connection failure

-- DO IT
distributed-tracing
istio
jaeger
kubernetes
microservices

1 Answer

10/28/2019
-- DO IT
Source: StackOverflow