Gist: I am struggling to get a pod to connect to a service outside the cluster. Basically the pod manages to resolve the ClusterIp of the selectorless service, but traffic does not go through. Traffic does go through if i hit the ClusterIp of the selectorless service from the cluster host.

I'm fairly new with microk8s and k8s in general. I hope i am making some sense though...

Background:

I am attempting to move parts of my infrastructure from a docker-compose setup on one virtual machine, to a microk8s cluster (with 2 nodes).

In the docker compose, i have a Grafana Container, connecting to an InfluxDb container.

kubectl version:

Client Version: version.Info{Major:"1", Minor:"22+", GitVersion:"v1.22.2-3+9ad9ee77396805", GitCommit:"9ad9ee77396805781cd0ae076d638b9da93477fd", GitTreeState:"clean", BuildDate:"2021-09-30T09:52:57Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}

I now want to setup a Grafana container on the microk8s cluster, and have it connect to the InfluxDb that is still running on the docker-compose vm.

All of these VM's are running on an ESXi host.

• InfluxDb is exposed at 10.1.2.220:8086
• microk8s-master has ip 10.1.2.50
• microk8s-slave-1 has ip 10.1.2.51

I have enabled ingress and dns. I have also enabled metallb, though i don't intend to use it here.

I have configured a selectorless service, a remote endpoint and an egress Network Policy (currently allowing all).

From microk8s-master and slave-1, i can

• telnet directly to 10.1.2.220:8086 successfully
• telnet to the ClusterIP(10.152.183.26):8086 of the service, successfully reaching influxdb
• wget ClusterIp:8086

Inside the Pod, if i do a wget to influxdb-service:8086, it will resolve to the ClusterIP, but after that it times out. I can however reach (wget), services pointing to other pods in the same namespace

Update:

I have been able to get it to work through a workaround, but i dont think this is the correct way.

My temporary solution is to expose the selectorless service on metallb, then use that exposed ip inside the pod.

Service and Endpoints for InfluxDb

---
apiVersion: v1
kind: Service
metadata:
  name: influxdb-service
  labels:
    app: grafana
spec:
  ports:
    - protocol: TCP
      port: 8086
      targetPort: 8086
---
apiVersion: v1
kind: Endpoints
metadata:
  name: influxdb-service
subsets:
  - addresses:
      - ip: 10.1.2.220
    ports:
      - port: 8086

The service and endpoint shows up fine

eso@microk8s-master:~/k8s-grafana$ microk8s.kubectl get endpoints
NAME                ENDPOINTS                             AGE
neo4j-service-lb    10.1.166.176:7687,10.1.166.176:7474   25h
influxdb-service    10.1.2.220:8086                       127m
questrest-service   10.1.166.178:80                       5d
kubernetes          10.1.2.50:16443,10.1.2.51:16443       26d
grafana-service     10.1.237.120:3000                     3h11m

eso@microk8s-master:~/k8s-grafana$ microk8s.kubectl get svc
NAME                TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                         AGE
kubernetes          ClusterIP      10.152.183.1     <none>        443/TCP                         26d
questrest-service   ClusterIP      10.152.183.56    <none>        80/TCP                          5d
neo4j-service-lb    LoadBalancer   10.152.183.166   10.1.2.60     7474:31974/TCP,7687:32688/TCP   25h
grafana-service     ClusterIP      10.152.183.75    <none>        3000/TCP                        3h13m
influxdb-service    ClusterIP      10.152.183.26    <none>        8086/TCP                        129m

eso@microk8s-master:~/k8s-grafana$ microk8s.kubectl get networkpolicy
NAME                            POD-SELECTOR    AGE
grafana-allow-egress-influxdb   app=grafana     129m
test-egress-influxdb            app=questrest   128m

Describe:

eso@microk8s-master:~/k8s-grafana$ microk8s.kubectl describe svc influxdb-service
Name:              influxdb-service
Namespace:         default
Labels:            app=grafana
Annotations:       <none>
Selector:          <none>
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.152.183.26
IPs:               10.152.183.26
Port:              <unset>  8086/TCP
TargetPort:        8086/TCP
Endpoints:         10.1.2.220:8086
Session Affinity:  None
Events:            <none>

eso@microk8s-master:~/k8s-grafana$ microk8s.kubectl describe endpoints influxdb-service
Name:         influxdb-service
Namespace:    default
Labels:       <none>
Annotations:  <none>
Subsets:
  Addresses:          10.1.2.220
  NotReadyAddresses:  <none>
  Ports:
    Name     Port  Protocol
    ----     ----  --------
    <unset>  8086  TCP

Events:  <none>

eso@microk8s-master:~/k8s-grafana$ microk8s.kubectl describe networkpolicy grafana-allow-egress-influxdb
Name:         grafana-allow-egress-influxdb
Namespace:    default
Created on:   2021-11-03 20:53:00 +0000 UTC
Labels:       <none>
Annotations:  <none>
Spec:
  PodSelector:     app=grafana
  Not affecting ingress traffic
  Allowing egress traffic:
    To Port: <any> (traffic allowed to all ports)
    To: <any> (traffic not restricted by destination)
  Policy Types: Egress

Grafana.yml:

eso@microk8s-master:~/k8s-grafana$ cat grafana.yml
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: grafana-pv
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteMany
  storageClassName: ""
  claimRef:
    name: grafana-pvc
    namespace: default
  persistentVolumeReclaimPolicy: Retain
  nfs:
    path: /mnt/MainVol/grafana
    server: 10.2.0.1
    readOnly: false
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: grafana-pvc
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: ""
  volumeName: grafana-pv
  resources:
    requests:
      storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: grafana
  name: grafana
spec:
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      labels:
        app: grafana
    spec:
      securityContext:
        fsGroup: 472
        supplementalGroups:
          - 0
      containers:
        - name: grafana
          image: grafana/grafana:7.5.2
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 3000
              name: http-grafana
              protocol: TCP
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /robots.txt
              port: 3000
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 30
            successThreshold: 1
            timeoutSeconds: 2
          livenessProbe:
            failureThreshold: 3
            initialDelaySeconds: 30
            periodSeconds: 10
            successThreshold: 1
            tcpSocket:
              port: 3000
            timeoutSeconds: 1
          resources:
            requests:
              cpu: 250m
              memory: 750Mi
          volumeMounts:
            - mountPath: /var/lib/grafana
              name: grafana-pv
      volumes:
        - name: grafana-pv
          persistentVolumeClaim:
            claimName: grafana-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: grafana-service
spec:
  ports:
    - port: 3000
      protocol: TCP
      targetPort: http-grafana
  selector:
    app: grafana
  #sessionAffinity: None
  #type: LoadBalancer
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: grafana-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: "g2.some.domain.com"
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: grafana-service
            port:
              number: 3000
---
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: grafana-allow-egress-influxdb
  namespace: default
spec:
  podSelector:
    matchLabels:
      app: grafana
  ingress:
  - {}
  egress:
  - {}
  policyTypes:
  - Egress