Kubernetes Pod with hostNetwork True cannot reach external IPs of services in the same cluster

11/11/2020

Problem

I have two pods A and B running in a cluster on minikube, both have external IPs www.service-a.com and www.service-b.com. Both external IPs are accessible from outside.

I need A to be able to call B with it's external IP rather than its cluster DNS, that is A needs to use www.service-b.com rather than b.svc.cluster.local (which does work but I can't use it).

I set A to use hostNetwork: true and dnsPolicy: ClusterFirstWithHostNet. If I spin up a NodeJS docker container manually, it indeed can connect and find it. However, A is still unable to connect to service-b.com. Am I using hostNetwork wrong? How can I configure my pod to connect to b in that way?

A's Deployment YAML

...
spec:
  replicas: 1
  selector:
    matchLabels:
      app: a-app
  template:
    metadata:
      labels:
        app: a-app
    spec:
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
...

B's service YAML

...
spec:
  externalTrafficPolicy: Cluster
  type: LoadBalancer
  ports:
  - port: ...
    targetPort: ...
    protocol: TCP
    name: http
...

Background:

I'm using Minio (a local S3-like solution) and I need to presign the URLs to get and put objects. Minio's pods are running in the same cluster as my authentication pod which would generate the presigned urls. The presigned urls would be used from outside the cluster. Hence I can't sign the url with the cluster dns names like minio.svc.cluster.local because this URL would not be accessible from outside the cluster and replacing the host with my-minio.com and keeping the signature does not work because I guess minio signs the entire host and path. Hence I need to have my authentication pod connect to Minio's public facing my-minio.com instead which does not seem to work.

-- Math is Hard
docker
kubernetes
minio

2 Answers

11/11/2020

Regarding hostNetwork, it looks like you misunderstood it. Setting it to true means that Pod will have access to the host where it's running. In case of minikube it's VM and not your host where actual containers are running.

Also, i'm not sure how you expose your services to external world, but i suggest you to try Ingress for that.

-- Grigoriy Mikhalkin
Source: StackOverflow

11/17/2020

As Grigoriy suggested, I used an ingress with nginx.ingress.kubernetes.io/upstream-vhost annotation to forward all requests into the cluster with Host: service-b to resolve my issue. Previously I had nginx.ingress.kubernetes.io/rewrite-target: /$1 which stripped the path from the request that caused a serious of issues, so I removed that. The details of how I got it working are here:

https://stackoverflow.com/questions/64815229/nginx-controller-kubernetes-need-to-change-host-header-within-ingress

-- Math is Hard
Source: StackOverflow