I have a cluster running on my server VM's on my local network. On front of it I have an other VM setup running nginx on a docker container to manage subdomains and so. My issue is that I would like to deploy apps on my cluster for different domains. I already have metallb and ingress controller set up on the cluster and it works when I query directly metallb's ip.

My nginx config:

server {
    listen 80;
    return 301 https://$host$request_uri;
}
 server {
        listen   443 ssl;
        server_name domain.com;
        ssl_certificate /etc/nginx/ssl/domain.com/server.crt;
        ssl_certificate_key /etc/nginx/ssl/domain.com/server.key;
         location / {
                        proxy_pass      https://10.20.0.240;
                    }
}
server {

        listen   80;
        listen   443 ssl;
        listen   6443;
        server_name k8s.domain.com;
        ssl_certificate /etc/nginx/ssl/domain.com/server.crt;
        ssl_certificate_key /etc/nginx/ssl/domain.com/server.key;
         location / {
                        proxy_pass      https://10.20.0.130:6443;
                    }
}
server {

        listen   80;
        listen   443 ssl;
        server_name docker.domain.com;
        client_max_body_size 0;
        
        ssl_certificate /etc/nginx/ssl/domain.com/server.crt;
        ssl_certificate_key /etc/nginx/ssl/domain.com/server.key;
         location / {
                        proxy_pass      http://10.20.0.140;
                    }
}

I need to have both my k8s api exposed and docker registry to work with my CI/CD solution.

in the future I might want to do dev.domain.com and point it to a different cluster or namespace.

My ingress config look like this right now but I will change it once it work so wevery app have their own ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: main-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    kubernetes.io/ingress.class: nginx
spec:
  rules:
  - host: "domain.com"
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: demo
            port:
              number: 80
  - host: "domain.com"
    http:
      paths:
      - pathType: Prefix
        path: "/someservice"
        backend:
          service:
            name: someservice-service
            port:
              number: 80

I want to be able to manage domain on the ingress level but since on my nginx I redirect to an IP I get 404 not found from ingress. I guess I could add the host on my container so my domain.com would equal the right ip and just proxy pass to the domain but I am sure that there is a more elegant solution to this problem.

I also though of using ingress as my entry point but I don't think I will be able to redirect the traffic to ip outside of the cluster.

EDIT

So I changed my view on how achieve what I wanna do. I decided to use ingress as an entry point instead of my NGINX VM. I used service external name to redirect some request outside of the cluster (for docker registry and k8s api) the issue is with the API. I can't get it to work.

There are the services I used.

apiVersion: v1
kind: Service
metadata:
  name: private-registry
spec:
  type: ExternalName
  externalName: 10.20.0.140
---
apiVersion: v1
kind: Service
metadata:
  name: k8s-api
  namespace: default
spec:
  clusterIP: None
  ports:
  - name: api
    port: 6443
    protocol: TCP
---
kind: Endpoints
apiVersion: v1
metadata:
  name: k8s-api
  namespace: default
subsets:
  - addresses:
      - ip: 10.20.0.130
    ports:
      - port: 6443
        name: api
        protocol: TCP

There is my new ingress config

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: main-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
spec:
  tls:
  - hosts:
    - domain.com
    secretName: domain-ssl
  - hosts:
    - docker.domain.com
    secretName: domain-ssl
  - hosts:
    - k8s.domain.com
    secretName: domain-ssl
  rules:
  - host: "domain.com"
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: demo
            port:
              number: 80
  - host: "domain.com"
    http:
      paths:
      - pathType: Prefix
        path: "/vcdbservice"
        backend:
          service:
            name: vcdbservice-service
            port:
              number: 8080
  - host: "docker.domain.com"
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: private-registry
            port:
              number: 80
  - host: "k8s.domain.com"
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: k8s-api
            port:
              number: 80