I have tentatively decided upon Kubernetes, ideally with Rancher, so I can readily spin up and take down containerised applications and run web servers etc on my home office setup. I have a fairly normal internet setup with a router connected to the internet via a static IP from my ISP. I want to be able to install web applications on a bare metal server on the office LAN such that some of them are accessible via the internet and others just via the LAN. I have settled on k3s as the Kubernetes distribution.

I intend to use port-forwarding from the router to an IP address on the server and have multiple web servers for different domains hosted on it. The router will find the applications via Hostname entries on the router initially, but maybe by a DNS server hosted on k3s later.

Reading around, I need MetalLB and an Ingress controller of some kind on the Kubernetes installation.
MetalLB to provide IP addresses for the k3s "cluster" and the ingress controller to perform Layer 7 routing to the individual servers in the k3s cluster. I have found ingress-nginx as a potentially suitable ingress but my install has several issues that I cannot get past. I want MetalLB to broadcast for several IP Addresses on the server network i/f :

One IP for the externally-accessible applications via the port forwarding and another IP for those servers only having internally accessible applications.

I have gone for eaxctness rather than brevity in the following as I think is is a sufficiently complex setup that is necessary. I give the exact install instructions for the various parts so people can see exactly what I am doing:

Install k3s:

curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.21.4+k3s1 sh -s -

Then disable Traefik and servicelb by editing /etc/systemd/system/k3s.service so that the ExecStart line becomes:

ExecStart=/usr/local/bin/k3s server --disable traefik --disable servicelb

Install CertManager as follows:

kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v1.5.3/cert-manager.crds.yaml
kubectl create namespace cert-manager
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install cert-manager jetstack/cert-manager --namespace cert-manager --version v1.5.3

Install MetalLB:

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.10.2/manifests/namespace.yaml
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.10.2/manifests/metallb.yaml
kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"

With the following in metallb-config.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: lan-facing
      protocol: layer2
      addresses:
      - 192.168.1.17/32
    - name: inetfa
      protocol: layer2
      addresses:
      - 192.168.1.16/32

run:

kubectl apply -f metallb-config.yaml

Install nginx:

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm install ingress-nginx-plain ingress-nginx/ingress-nginx --version 4.0.1

kubectl apply -f ingress-nginx-baremetal-github.yaml

Where ingress-nginx-baremetal-github.yaml is the file at https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.0.1/deploy/static/provider/baremetal/deploy.yaml which is referenced on the ingress-nginx installation page at https://kubernetes.github.io/ingress-nginx/deploy/#bare-metal section Bare-metal with the following changes :

# Source: ingress-nginx/templates/controller-service.yaml
apiVersion: v1
kind: Service
metadata:
  annotations:
    metallb.universe.tf/address-pool: inetfa
...
spec:
  type: LoadBalancer

Ie, where the annotation "metallb.universe.tf/address-pool: inetfa" has been added and NodePort changed to LoadBalancer.

I have installed servers using bitnami Apache installs as follows:

helm install my-release -f apache1-configmap.yaml bitnami/apache

The intent of this setup changes was to make the ingress-nginx installation use IP address 192.168.1.16 handed out by MetalLB by virtue of the matching address-pool name "inetfa"

BUT

the IP address used by my server applications is 192.168.1.17/32 though the Layer 7 routing is working OK based upon the "apache1-configmap.yaml" files, eg:

...
ingress:
...
  hostname: apache1.local
...
  ingressClassName: nginx
...

So, to recap what I've got:

I can have several apache servers with different hostnames. Ingress-nginx host-based routing works to each server according to the hostname in the respective apache configmap.yaml file

BUT

they all come out n the first IP that MetalLB sees in its address-pools, not the one selected by the annotation in the ingress-nginx setup as required.

It seems that it is the first address-pool that is used which does not have "auto-assign: false" set, regardless of the metallb.universe.tf/address-pool: annotation and the address-pool name. This is no good as I will need to use "auto-assign: false" on all my address-pools to prevent the wrong service applications being accessible on the web.

Can anyone see what I am doing wrong and what I need to change to get address-pool selection working between MetalLB and ingress-nginx ?

When I get the selection of address-pools for the internet-facing applications to work I then want to get the other address-pool to work for LAN-facing applications and I would appreciate advice on whether I need a separate install of nginx or whether these is a way to get one ingress-nginx install to support several IngressClass'es so the "nginx" IngressClass provided by the "ingress-nginx-baremetal-github.yaml" could be supplemented by eg an "nginxlanonly" IngressClass that uses a different address-pool. Anyone know how to do that or what to do to achieve the same goal if that idea cannot work ?