Couldn't see in skywalking ui
data from istio
metrics.
Using below guide to installed istio
and skywalking
backend to Kubernetes
: https://github.com/apache/skywalking/tree/master/docs/en/setup/istio. Made all steps in the guide and deployed app. Deployed app is example book app
from istio
. Could see traces in Jaeger
and works good, but can't see istio
metrics in skywalking ui
.
How I can provide metrics from istio
to skywalking ui
or what I need to configure to make metrics from istio
come to skywalking ui
.
Mixer logs:
gc 1847 @19617.610s 0%: 0.043+22+0.035 ms clock, 0.17+1.9/28/17+0.14 ms cpu, 16->16->8 MB, 17 MB goal, 6 P
2019-10-11T01:28:44.153821Z warn grpcAdapter unable to connect to:/metric.HandleMetricService/HandleMetric, oap.skywalking.svc.cluster.local:11800
{"level":"info","time":"2019-10-11T01:28:43.143390Z","instance":"accesslog.instance.istio-system","apiClaims":"","apiKey":"","clientTraceId":"","connection_security_policy":"none","destinationApp":"telemetry","destinationIp":"10.1.3.13","destinationName":"istio-telemetry-7c88764b9c-fncph","destinationNamespace":"istio-system","destinationOwner":"kubernetes://apis/apps/v1/namespaces/istio-system/deployments/istio-telemetry","destinationPrincipal":"","destinationServiceHost":"istio-telemetry.istio-system.svc.cluster.local","destinationWorkload":"istio-telemetry","grpcMessage":"3 errors occurred:%0A%09* fieldEncoder: dimensions - fieldEncoder: value - lookup failed: 'response.time'%0A%09* fieldEncoder: dimensions - fieldEncoder: value - lookup failed: 'response.time'%0A%09* fieldEncoder: dimensions - fieldEncoder: value - lookup failed: 'response.time'%0A%0A","grpcStatus":"2","httpAuthority":"mixer","latency":"6.5979ms","method":"POST","permissiveResponseCode":"none","permissiveResponsePolicyID":"none","protocol":"http","receivedBytes":1317,"referer":"","reporter":"destination","requestId":"b45592ff-0c49-49fd-af64-3c9229b60466","requestSize":931,"requestedServerName":"","responseCode":200,"responseFlags":"-","responseSize":0,"responseTimestamp":"2019-10-11T01:28:43.149845Z","sentBytes":421,"sourceApp":"elasticsearch","sourceIp":"10.1.3.36","sourceName":"skywalking-elasticsearch-data-1","sourceNamespace":"default","sourceOwner":"kubernetes://apis/apps/v1/namespaces/default/statefulsets/skywalking-elasticsearch-data","sourcePrincipal":"","sourceWorkload":"skywalking-elasticsearch-data","url":"/istio.mixer.v1.Mixer/Report","userAgent":"","xForwardedFor":"10.1.3.36"}
{"level":"info","time":"2019-10-11T01:28:43.913917Z","instance":"accesslog.instance.istio-system","apiClaims":"","apiKey":"","clientTraceId":"","connection_security_policy":"none","destinationApp":"telemetry","destinationIp":"10.1.3.13","destinationName":"istio-telemetry-7c88764b9c-fncph","destinationNamespace":"istio-system","destinationOwner":"kubernetes://apis/apps/v1/namespaces/istio-system/deployments/istio-telemetry","destinationPrincipal":"","destinationServiceHost":"istio-telemetry.istio-system.svc.cluster.local","destinationWorkload":"istio-telemetry","grpcMessage":"1 error occurred:%0A%09* fieldEncoder: dimensions - fieldEncoder: value - lookup failed: 'response.time'%0A%0A","grpcStatus":"2","httpAuthority":"mixer","latency":"13.906ms","method":"POST","permissiveResponseCode":"none","permissiveResponsePolicyID":"none","protocol":"http","receivedBytes":698,"referer":"","reporter":"destination","requestId":"ff7ae8af-f279-46b3-ae86-bde2b61477dd","requestSize":312,"requestedServerName":"","responseCode":200,"responseFlags":"-","responseSize":0,"responseTimestamp":"2019-10-11T01:28:43.927780Z","sentBytes":247,"sourceApp":"elasticsearch","sourceIp":"10.1.3.37","sourceName":"skywalking-elasticsearch-master-2","sourceNamespace":"default","sourceOwner":"kubernetes://apis/apps/v1/namespaces/default/statefulsets/skywalking-elasticsearch-master","sourcePrincipal":"","sourceWorkload":"skywalking-elasticsearch-master","url":"/istio.mixer.v1.Mixer/Report","userAgent":"","xForwardedFor":"10.1.3.37"}
{"level":"info","time":"2019-10-11T01:28:43.913675Z","instance":"accesslog.instance.istio-system","apiClaims":"","apiKey":"","clientTraceId":"","connection_security_policy":"none","destinationApp":"telemetry","destinationIp":"10.1.3.13","destinationName":"istio-telemetry-7c88764b9c-fncph","destinationNamespace":"istio-system","destinationOwner":"kubernetes://apis/apps/v1/namespaces/istio-system/deployments/istio-telemetry","destinationPrincipal":"","destinationServiceHost":"istio-telemetry.istio-system.svc.cluster.local","destinationWorkload":"istio-telemetry","grpcMessage":"8 errors occurred:%0A%09* fieldEncoder: dimensions - fieldEncoder: value - lookup failed: 'response.time'%0A%09* fieldEncoder: dimensions - fieldEncoder: value - lookup failed: 'response.time'%0A%09* fieldEncoder: dimensions - fieldEncoder: value - lookup failed: 'response.time'%0A%09* fieldEncoder: dimensions - fieldEncoder: value - lookup failed: 'response.time'%0A%09* fieldEncoder: dimensions - fieldEncoder: value - lookup failed: 'response.time'%0A%09* fieldEncoder: dimensions - fieldEncoder: value - lookup failed: 'response.time'%0A%09* fieldEncoder: dimensions - fieldEncoder: value - lookup failed: 'response.time'%0A%09* fieldEncoder: dimensions - fieldEncoder: value - lookup failed: 'response.time'%0A%0A","grpcStatus":"2","httpAuthority":"mixer","latency":"29.8101ms","method":"POST","permissiveResponseCode":"none","permissiveResponsePolicyID":"none","protocol":"http","receivedBytes":2430,"referer":"","reporter":"destination","requestId":"b0eec606-c91e-4193-aba2-39b069b76474","requestSize":2044,"requestedServerName":"","responseCode":200,"responseFlags":"-","responseSize":0,"responseTimestamp":"2019-10-11T01:28:43.943383Z","sentBytes":857,"sourceApp":"elasticsearch","sourceIp":"10.1.3.37","sourceName":"skywalking-elasticsearch-master-2","sourceNamespace":"default","sourceOwner":"kubernetes://apis/apps/v1/namespaces/default/statefulsets/skywalking-elasticsearch-master","sourcePrincipal":"","sourceWorkload":"skywalking-elasticsearch-master","url":"/istio.mixer.v1.Mixer/Report","userAgent":"","xForwardedFor":"10.1.3.37"}
2019-10-11T01:28:44.154193Z error api Report failed: 1 error occurred:
* rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp: lookup oap.skywalking.svc.cluster.local on 10.96.0.10:53: no such host"
Istio policy:
GC forced
gc 181 @20366.930s 0%: 3.9+19+0.030 ms clock, 15+0/30/67+0.12 ms cpu, 13->13->7 MB, 15 MB goal, 6 P
scvg135: 0 MB released
scvg135: inuse: 15, idle: 43, sys: 58, released: 3, consumed: 55 (MB)
2019-10-11T01:42:28.493381Z info base.baseBalancer: handle SubConn state change: 0xc0011280a0, CONNECTING
2019-10-11T01:42:28.493442Z info roundrobinPicker: newPicker called with readySCs: map[]
2019-10-11T01:42:28.577142Z info grpc: addrConn.createTransport failed to connect to {oap.skywalking.svc.cluster.local:11800 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp: lookup oap.skywalking.svc.cluster.local on 10.96.0.10:53: no such host". Reconnecting...
2019-10-11T01:42:28.577256Z info base.baseBalancer: handle SubConn state change: 0xc0011280a0, TRANSIENT_FAILURE
GC forced
gc 182 @20486.971s 0%: 0.13+13+0.056 ms clock, 0.55+0/13/26+0.22 ms cpu, 13->13->7 MB, 15 MB goal, 6 P
somehow I have failed to connect to my skywalking
backend, problem was with name and namespace but still getting error but now skywalking
backend getting data but ui no updates. Istio
telemetry:
rpc error: code = Unavailable desc = upstream connect error or disconnect/reset before headers. reset reason: connection failure
Explanation how to set up skywalking
properly: https://github.com/apache/skywalking/issues/3589#issuecomment-543268029