I'm new to k8s and slowly picking it up. I have built out a web api that runs on .net 5 and uses HTTPS (a previous version used http) only, I built the image in docker compose and everything works as expected locally with the default aspnetapp.pfx cert. What I am struggling with is that my ingress routing seems to terminaate the connection early.
I have created a cert pfx for kestrel to run on with the CN name of a.b.com this was from the crt and key files that are needed to create secrets in the documentation. but from my understanding kestrel needs a pfx to run (straight out of the box).
Below are my ingress, service and deployment snippets as well as the entries from the logs: I believe that my issue is that in the logs it is showing as a "http" request but it should be https
Logs:
2021/02/24 09:44:11 [error] 3231#3231: *44168360 upstream prematurely closed connection while reading response header from upstream, client: 127.0.0.1, server: _, request: "GET /dayofweek/api/1/dayofweek/action HTTP/2.0", upstream: "http://<podip>/api/1/dayofweek/action", host: "<clusterip>:<nodePort>"
Ingress:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: dayofweek-ingress-path
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$1
nginx.ingress.kubernetes.io/auth-type: basic
nginx.ingress.kubernetes.io/auth-secret: basic-auth
#kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
rules:
- host: a.b.com
- http:
paths:
- backend:
service:
name: dayofweek-svc
port:
number: 9057
path: /dayofweek/?(.*)
pathType: Prefix
Service + Deployment
apiVersion: v1
kind: Service
metadata:
name: dayofweek-svc
labels:
run: dayofweek-svc
spec:
ports:
- port: 9057
targetPort: 3441
protocol: TCP
name: https
selector:
app: dayofweek
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: dayofweek
spec:
replicas: 1
selector:
matchLabels:
app: dayofweek
template:
metadata:
labels:
app: dayofweek
spec:
volumes:
- name: cert-volume
persistentVolumeClaim:
claimName: persistentcerts-claim
containers:
- name: dayofweek
image: ...omitted
ports:
- containerPort: 3441
env:
- name: DOTNET_ENVIRONMENT
value: Development
- name: Culture
value: en-US #English by default
- name: ASPNETCORE_Kestrel__Certificates__Default__Path
value: /https/aspnetapp.pfx
- name: ASPNETCORE_Kestrel__Certificates__Default__Password
value: password
volumeMounts:
- mountPath: /https
name: cert-volume
I followed through with the guide here: https://kubernetes.io/docs/concepts/services-networking/connect-applications-service/
And I seem to have it up and running, but for some reason when I'm not sure if I've overcomplicated it by adding in the "-host" element of the ingress.
Any help would be greatly appreciated!
Service and deployment look correct, but I can see some issues with ingress.
When using ssl-passthrough path based routing doesn't work so you can skip it.
Also, there is a typo in your config:
- host: a.b.com
- http: # <- HERE
there shouldn't be the second dash.
Here is how it should look like:
spec:
rules:
- host: a.b.com
http:
paths:
Additionally, have a look what nginx ingres docs has to say about ssl-passthrough:
SSL Passthrough
The --enable-ssl-passthrough flag enables the SSL Passthrough feature, which is disabled by default. This is required to enable passthrough backends in Ingress objects.
Warning
This feature is implemented by intercepting all traffic on the configured HTTPS port (default: 443) and handing it over to a local TCP proxy. This bypasses NGINX completely and introduces a non-negligible performance penalty.
SSL Passthrough leverages SNI and reads the virtual domain from the TLS negotiation, which requires compatible clients. After a connection has been accepted by the TLS listener, it is handled by the controller itself and piped back and forth between the backend and the client.
If there is no hostname matching the requested host name, the request is handed over to NGINX on the configured passthrough proxy port (default: 442), which proxies the request to the default backend.
There is also this in docs:
SSL Passthrough
nginx.ingress.kubernetes.io/ssl-passthrough instructs the controller to send TLS connections directly to the backend instead of letting NGINX decrypt the communication. See also TLS/HTTPS in the User guide.
**Note SSL Passthrough is disabled by default and requires starting the controller with the --enable-ssl-passthrough flag.
Attention
Because SSL Passthrough works on layer 4 of the OSI model (TCP) and not on the layer 7 (HTTP), using SSL Passthrough invalidates all the other annotations set on an Ingress object.
So, according to the docs, in order for it to work you need to enable ssl-passthrough feature first. After this is done, you can use ssl-passthrough annotation but this invalidates all the other annotations and path based routing stops working.