Modified the helm chart of alerta
to spin it up on an istio
- enabled GKE cluster.
The alerta pod and its sidecar are created OK
▶ k get pods | grep alerta
alerta-758bc87dcf-tp5nv 2/2 Running 0 22m
When I try to access the url that my virtual service is pointing to
I get the following error
upstream connect error or disconnect/reset before headers. reset reason: connection termination
▶ k get vs alerta-virtual-service -o yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
annotations:
helm.fluxcd.io/antecedent: mynamespace:helmrelease/alerta
creationTimestamp: "2020-04-23T14:45:04Z"
generation: 1
name: alerta-virtual-service
namespace: mynamespace
resourceVersion: "46844125"
selfLink: /apis/networking.istio.io/v1alpha3/namespaces/mynamespace/virtualservices/alerta-virtual-service
uid: 2a3caa13-3900-4da1-a3a1-9f07322b52b0
spec:
gateways:
- mynamespace/istio-ingress-gateway
hosts:
- alerta.myurl.com
http:
- appendHeaders:
x-request-start: t=%START_TIME(%s.%3f)%
match:
- uri:
prefix: /
route:
- destination:
host: alerta
port:
number: 80
timeout: 60s
and here is the service
▶ k get svc alerta -o yaml
apiVersion: v1
kind: Service
metadata:
annotations:
helm.fluxcd.io/antecedent: mynamespace:helmrelease/alerta
creationTimestamp: "2020-04-23T14:45:04Z"
labels:
app: alerta
chart: alerta-0.1.0
heritage: Tiller
release: alerta
name: alerta
namespace: mynamespace
resourceVersion: "46844120"
selfLink: /api/v1/namespaces/mynamespace/services/alerta
uid: 4d4a3c73-ee42-49e3-a4cb-8c51536a0508
spec:
clusterIP: 10.8.58.228
ports:
- name: http
port: 80
protocol: TCP
targetPort: http
selector:
app: alerta
release: alerta
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
However, when I exec via another pod into the cluster and try to reach the alerta svc endpoint:
/ # curl -IL http://alerta
curl: (56) Recv failure: Connection reset by peer
/ # nc -zv -w 3 alerta 80
alerta (10.8.58.228:80) open
although as it is evident the port is open
any suggestion?
could it be that the chaining of the 2 proxies is creating issues? nginx behind envoy?
The container logs seem normal
2020-04-23 15:34:40,272 DEBG 'nginx' stdout output:
ip=\- [\23/Apr/2020:15:34:40 +0000] "\GET / HTTP/1.1" \200 \994 "\-" "\kube-probe/1.15+"
/web | /index.html | > GET / HTTP/1.1
edit: Here is a verbose curl
with host header explicitly set
/ # curl -v -H "host: alerta.myurl.com" http://alerta:80
* Rebuilt URL to: http://alerta:80/
* Trying 10.8.58.228...
* TCP_NODELAY set
* Connected to alerta (10.8.58.228) port 80 (#0)
> GET / HTTP/1.1
> host: alerta.myurl.com
> User-Agent: curl/7.57.0
> Accept: */*
>
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer
The nginx
config file used by the app/pod is the following FWIW
worker_processes 4;
pid /tmp/nginx.pid;
daemon off;
error_log /dev/stderr info;
events {
worker_connections 1024;
}
http {
client_body_temp_path /tmp/client_body;
fastcgi_temp_path /tmp/fastcgi_temp;
proxy_temp_path /tmp/proxy_temp;
scgi_temp_path /tmp/scgi_temp;
uwsgi_temp_path /tmp/uwsgi_temp;
include /etc/nginx/mime.types;
gzip on;
gzip_disable "msie6";
log_format main 'ip=\$http_x_real_ip [\$time_local] '
'"\$request" \$status \$body_bytes_sent "\$http_referer" '
'"\$http_user_agent"' ;
log_format scripts '$document_root | $uri | > $request';
default_type application/octet-stream;
server {
listen 8080 default_server;
access_log /dev/stdout main;
access_log /dev/stdout scripts;
location ~ /api {
include /etc/nginx/uwsgi_params;
uwsgi_pass unix:/tmp/uwsgi.sock;
proxy_set_header Host $host:$server_port;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
root /web;
index index.html;
location / {
try_files $uri $uri/ /index.html;
}
}
}
edit 2: Trying to get the istio
authentication policy
✔ 18h55m ⍉
▶ kubectl get peerauthentication.security.istio.io
No resources found.
✔ 18h55m
▶ kubectl get peerauthentication.security.istio.io/default -o yaml
Error from server (NotFound): peerauthentications.security.istio.io "default" not found
edit 3: when performing curl
to the service from within the istio proxy container
▶ k exec -it alerta-758bc87dcf-jzjgj -c istio-proxy bash
istio-proxy@alerta-758bc87dcf-jzjgj:/$ curl -v http://alerta:80
* Rebuilt URL to: http://alerta:80/
* Trying 10.8.58.228...
* Connected to alerta (10.8.58.228) port 80 (#0)
> GET / HTTP/1.1
> Host: alerta
> User-Agent: curl/7.47.0
> Accept: */*
>
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer
I created new gke cluster with istio 1.5.2, in fact if you check for mtls, there are no resources found
kubectl get peerauthentication --all-namespaces
No resources found.
kubectl get peerauthentication.security.istio.io/default
Error from server (NotFound): peerauthentications.security.istio.io "default" not found
So I tried to make this example and that clearly shows istio is in strict tls mode when you installed it with global.mtls.enabled=true
.
If you add pods,namespaces as mentioned here it should be 200 for every request, but it's not
sleep.foo to httpbin.foo: 200
sleep.foo to httpbin.bar: 200
sleep.foo to httpbin.legacy: 200
sleep.bar to httpbin.foo: 200
sleep.bar to httpbin.bar: 200
sleep.bar to httpbin.legacy: 200
sleep.legacy to httpbin.foo: 000
command terminated with exit code 56
sleep.legacy to httpbin.bar: 000
command terminated with exit code 56
sleep.legacy to httpbin.legacy: 200
So if you change the mtls from strict to permissive with above below yaml
apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
name: "default"
namespace: "istio-system"
spec:
mtls:
mode: PERMISSIVE
it works now
sleep.foo to httpbin.foo: 200
sleep.foo to httpbin.bar: 200
sleep.foo to httpbin.legacy: 200
sleep.bar to httpbin.foo: 200
sleep.bar to httpbin.bar: 200
sleep.bar to httpbin.legacy: 200
sleep.legacy to httpbin.foo: 200
sleep.legacy to httpbin.bar: 200
sleep.legacy to httpbin.legacy: 200
Additionaly github issue with error you provided.
About the question
why the pod fails to mtls authenticate with itself, when curling from inside it
There is a github issue about this.
Additionally take a look at this istio docs.
When exec-ing into the corresponding istio-proxy
container and turning trace
- level logging on
curl http://localhost:15000/logging?level=trace
I see the following error:
alerta-758bc87dcf-jzjgj istio-proxy [2020-04-24 13:15:21.180][29][debug][connection] [external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:168] [C1484] handshake error: 1
alerta-758bc87dcf-jzjgj istio-proxy [2020-04-24 13:15:21.180][29][debug][connection] [external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:201] [C1484] TLS error: 268435612:SSL routines:OPENSSL_internal:HTTP_REQUEST