Metallb LoadBalancer is stuck on pending

2/9/2021

I am trying to set up a bare metal Kubernetes server including the metallb LoadBalancer. Therefore, I followed the instructions published on the website of the kind project https://kind.sigs.k8s.io/docs/user/loadbalancer/. To test my installation I tried to deploy the default nginx web-server and reach it via the browser. Unfortunately the created service of type LoadBalancer is stuck on EXTERNAL IP <Pending>. I will attach as much information as possible. I hope someone could help me with this issue. If you need more info please let me know.

This is what I did:

  1. Installation of metallb LoadBalancer according to https://kind.sigs.k8s.io/docs/user/loadbalancer/
  2. Created a nginx deployment via kubectl create deployment nginx --image nginx
  3. Created a corresponding service kubectl expose deployment/nginx --type="LoadBalancer" --port 8080

Outputs

Version and Nodes

$ kubectl version --short
Client Version: v1.20.2
Server Version: v1.20.2

$ kubectl get nodes -o wide
NAME      STATUS   ROLES                  AGE     VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE       KERNEL-VERSION     CONTAINER-RUNTIME
gemini    Ready    control-plane,master   3d      v1.20.2   192.168.2.203   <none>        Ubuntu 20.10   5.8.0-1013-raspi   docker://19.3.13
phoenix   Ready    <none>                 2d23h   v1.20.2   192.168.2.129   <none>        Ubuntu 20.10   5.8.0-1013-raspi   docker://19.3.13
taurus    Ready    <none>                 2d23h   v1.20.2   192.168.2.201   <none>        Ubuntu 20.10   5.8.0-1013-raspi   docker://19.3.13
virgo     Ready    <none>                 2d23h   v1.20.2   192.168.2.202   <none>        Ubuntu 20.10   5.8.0-1013-raspi   docker://19.3.13

All namespaces

$ kubectl get all --all-namespaces

NAMESPACE              NAME                                             READY   STATUS             RESTARTS   AGE
default                pod/nginx-6799fc88d8-62cjd                       1/1     Running            0          59m
kube-system            pod/calico-kube-controllers-86bddfcff-ccrhg      1/1     Running            5          3d
kube-system            pod/calico-node-djt5s                            0/1     CrashLoopBackOff   1007       3d
kube-system            pod/calico-node-jddnl                            1/1     Running            3          3d
kube-system            pod/calico-node-nxwlw                            1/1     Running            5          3d
kube-system            pod/calico-node-zrxzl                            1/1     Running            3          3d
kube-system            pod/coredns-74ff55c5b-kb2nm                      1/1     Running            5          3d1h
kube-system            pod/coredns-74ff55c5b-wsgs5                      1/1     Running            5          3d1h
kube-system            pod/etcd-gemini                                  1/1     Running            6          3d1h
kube-system            pod/kube-apiserver-gemini                        1/1     Running            6          3d1h
kube-system            pod/kube-controller-manager-gemini               1/1     Running            6          3d1h
kube-system            pod/kube-proxy-7fcjz                             1/1     Running            6          3d1h
kube-system            pod/kube-proxy-84rr7                             1/1     Running            3          3d
kube-system            pod/kube-proxy-lc88w                             1/1     Running            3          3d
kube-system            pod/kube-proxy-v4qd9                             1/1     Running            3          3d
kube-system            pod/kube-scheduler-gemini                        1/1     Running            6          3d1h
kubernetes-dashboard   pod/dashboard-metrics-scraper-79c5968bdc-mlb4s   1/1     Running            5          3d
kubernetes-dashboard   pod/kubernetes-dashboard-7448ffc97b-nq5c9        1/1     Running            5          3d
metallb-system         pod/controller-5c797bc4f8-2zx7l                  1/1     Running            0          101m
metallb-system         pod/speaker-5kxc9                                1/1     Running            0          101m
metallb-system         pod/speaker-b6lbh                                1/1     Running            0          101m
metallb-system         pod/speaker-c8sr7                                1/1     Running            0          101m
metallb-system         pod/speaker-fwbgt                                1/1     Running            1          101m

NAMESPACE              NAME                                TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                  AGE
default                service/kubernetes                  ClusterIP      10.96.0.1        <none>        443/TCP                  23h
default                service/nginx                       LoadBalancer   10.110.25.14     <pending>     8080:30444/TCP           24m
kube-system            service/calico-etcd                 ClusterIP      10.96.232.136    <none>        6666/TCP                 3d
kube-system            service/calico-typha                ClusterIP      10.109.108.233   <none>        5473/TCP                 3d
kube-system            service/kube-dns                    ClusterIP      10.96.0.10       <none>        53/UDP,53/TCP,9153/TCP   3d1h
kubernetes-dashboard   service/dashboard-metrics-scraper   ClusterIP      10.110.70.52     <none>        8000/TCP                 3d
kubernetes-dashboard   service/kubernetes-dashboard        NodePort       10.106.194.127   <none>        443:31741/TCP            3d

NAMESPACE        NAME                         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
kube-system      daemonset.apps/calico-node   4         4         3       4            3           kubernetes.io/os=linux   3d
kube-system      daemonset.apps/kube-proxy    4         4         4       4            4           kubernetes.io/os=linux   3d1h
metallb-system   daemonset.apps/speaker       4         4         4       4            4           kubernetes.io/os=linux   101m

NAMESPACE              NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
default                deployment.apps/nginx                       1/1     1            1           59m
kube-system            deployment.apps/calico-kube-controllers     1/1     1            1           3d
kube-system            deployment.apps/coredns                     2/2     2            2           3d1h
kubernetes-dashboard   deployment.apps/dashboard-metrics-scraper   1/1     1            1           3d
kubernetes-dashboard   deployment.apps/kubernetes-dashboard        1/1     1            1           3d
metallb-system         deployment.apps/controller                  1/1     1            1           101m

NAMESPACE              NAME                                                   DESIRED   CURRENT   READY   AGE
default                replicaset.apps/nginx-6799fc88d8                       1         1         1       59m
kube-system            replicaset.apps/calico-kube-controllers-56b44cd6d5     0         0         0       3d
kube-system            replicaset.apps/calico-kube-controllers-86bddfcff      1         1         1       3d
kube-system            replicaset.apps/coredns-74ff55c5b                      2         2         2       3d1h
kubernetes-dashboard   replicaset.apps/dashboard-metrics-scraper-79c5968bdc   1         1         1       3d
kubernetes-dashboard   replicaset.apps/kubernetes-dashboard-7448ffc97b        1         1         1       3d
metallb-system         replicaset.apps/controller-5c797bc4f8                  1         1         1       101m

Namespace default:

$ kubectl get all
NAME                         READY   STATUS    RESTARTS   AGE
pod/nginx-6799fc88d8-62cjd   1/1     Running   0          29m

NAME                 TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
service/kubernetes   ClusterIP      10.96.0.1        <none>        443/TCP          23h
service/nginx        LoadBalancer   10.110.25.14     <pending>     8080:30444/TCP   29m

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx   1/1     1            1           29m

NAME                               DESIRED   CURRENT   READY   AGE
replicaset.apps/nginx-6799fc88d8   1         1         1       29m

Namespace metallb-system:

$ kubectl get all -n metallb-system
NAME                              READY   STATUS    RESTARTS   AGE
pod/controller-5c797bc4f8-2zx7l   1/1     Running   0          73m
pod/speaker-5kxc9                 1/1     Running   0          73m
pod/speaker-b6lbh                 1/1     Running   0          73m
pod/speaker-c8sr7                 1/1     Running   0          73m
pod/speaker-fwbgt                 1/1     Running   1          73m

NAME                     DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
daemonset.apps/speaker   4         4         4       4            4           kubernetes.io/os=linux   73m

NAME                         READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/controller   1/1     1            1           73m

NAME                                    DESIRED   CURRENT   READY   AGE
replicaset.apps/controller-5c797bc4f8   1         1         1       73m

Config map

$ kubectl describe configmap config -n metallb-system
Name:         config
Namespace:    metallb-system
Labels:       <none>
Annotations:  <none>

Data
====
config:
----
address-pools:
- name: default
  protocol: layer2
  addresses:
  - 192.168.2.210-192.168.2.250

Events:  <none>

Kube-proxy

$ kubectl describe configmap -n kube-system kube-proxy

Name:         kube-proxy
Namespace:    kube-system
Labels:       app=kube-proxy
Annotations:  kubeadm.kubernetes.io/component-config.hash: sha256:038b6c93b0ae1da89eb9c0c589c8e3439e9c91b849975435c29f905041b9b9fc

Data
====
config.conf:
----
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
bindAddressHardFail: false
clientConnection:
  acceptContentTypes: ""
  burst: 0
  contentType: ""
  kubeconfig: /var/lib/kube-proxy/kubeconfig.conf
  qps: 0
clusterCIDR: 192.168.0.0/16
configSyncPeriod: 0s
conntrack:
  maxPerCore: null
  min: null
  tcpCloseWaitTimeout: null
  tcpEstablishedTimeout: null
detectLocalMode: ""
enableProfiling: false
healthzBindAddress: ""
hostnameOverride: ""
iptables:
  masqueradeAll: false
  masqueradeBit: null
  minSyncPeriod: 0s
  syncPeriod: 0s
ipvs:
  excludeCIDRs: null
  minSyncPeriod: 0s
  scheduler: ""
  strictARP: true
  syncPeriod: 0s
  tcpFinTimeout: 0s
  tcpTimeout: 0s
  udpTimeout: 0s
kind: KubeProxyConfiguration
metricsBindAddress: ""
mode: ""
nodePortAddresses: null
oomScoreAdj: null
portRange: ""
showHiddenMetricsForVersion: ""
udpIdleTimeout: 0s
winkernel:
  enableDSR: false
  networkName: ""
  sourceVip: ""
kubeconfig.conf:
----
apiVersion: v1
kind: Config
clusters:
- cluster:
    certificate-authority: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    server: https://192.168.2.203:6443
  name: default
contexts:
- context:
    cluster: default
    namespace: default
    user: default
  name: default
current-context: default
users:
- name: default
  user:
    tokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
Events:  <none>

Metallb Controller and one of the Speaker Pods

$ kubectl describe pods -n metallb-system
Name:         controller-5c797bc4f8-2zx7l
Namespace:    metallb-system
Priority:     0
Node:         virgo/192.168.2.202
Start Time:   Tue, 09 Feb 2021 14:10:16 +0000
Labels:       app=metallb
              component=controller
              pod-template-hash=5c797bc4f8
Annotations:  cni.projectcalico.org/podIP: 192.168.9.246/32
              cni.projectcalico.org/podIPs: 192.168.9.246/32
              prometheus.io/port: 7472
              prometheus.io/scrape: true
Status:       Running
IP:           192.168.9.246
IPs:
  IP:           192.168.9.246
Controlled By:  ReplicaSet/controller-5c797bc4f8
Containers:
  controller:
    Container ID:  docker://b10ee06c88facfc97fcd246bcd214c746deea60073dc8928f478c89ca3a4feb9
    Image:         quay.io/metallb/controller:main
    Image ID:      docker-pullable://quay.io/metallb/controller@sha256:1e1a1bd75cebcecfe84db28c9a8605caeac3e30e57905185822c79fc00e5dcaf
    Port:          7472/TCP
    Host Port:     0/TCP
    Args:
      --port=7472
      --config=config
    State:          Running
      Started:      Tue, 09 Feb 2021 14:10:43 +0000
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     100m
      memory:  100Mi
    Requests:
      cpu:        100m
      memory:     100Mi
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from controller-token-ghmtk (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  controller-token-ghmtk:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  controller-token-ghmtk
    Optional:    false
QoS Class:       Guaranteed
Node-Selectors:  kubernetes.io/os=linux
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  89m   default-scheduler  Successfully assigned metallb-system/controller-5c797bc4f8-2zx7l to virgo
  Normal  Pulling    89m   kubelet            Pulling image "quay.io/metallb/controller:main"
  Normal  Pulled     89m   kubelet            Successfully pulled image "quay.io/metallb/controller:main" in 23.412886883s
  Normal  Created    89m   kubelet            Created container controller
  Normal  Started    89m   kubelet            Started container controller


Name:         speaker-5kxc9
Namespace:    metallb-system
Priority:     0
Node:         taurus/192.168.2.201
Start Time:   Tue, 09 Feb 2021 14:10:16 +0000
Labels:       app=metallb
              component=speaker
              controller-revision-hash=7548d88c64
              pod-template-generation=1
Annotations:  prometheus.io/port: 7472
              prometheus.io/scrape: true
Status:       Running
IP:           192.168.2.201
IPs:
  IP:           192.168.2.201
Controlled By:  DaemonSet/speaker
Containers:
  speaker:
    Container ID:  docker://e5ff811c47191f67c7089ce2464abff4fb8e4d3084f8053eb591c9d21e9a2276
    Image:         quay.io/metallb/speaker:main
    Image ID:      docker-pullable://quay.io/metallb/speaker@sha256:90a8cbdd543870e7db841a49743f3d8cd72b7074b5299086e09c5c1d92331eb4
    Port:          7472/TCP
    Host Port:     7472/TCP
    Args:
      --port=7472
      --config=config
    State:          Running
      Started:      Tue, 09 Feb 2021 14:10:33 +0000
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     100m
      memory:  100Mi
    Requests:
      cpu:     100m
      memory:  100Mi
    Environment:
      METALLB_NODE_NAME:       (v1:spec.nodeName)
      METALLB_HOST:            (v1:status.hostIP)
      METALLB_ML_BIND_ADDR:    (v1:status.podIP)
      METALLB_ML_LABELS:      app=metallb,component=speaker
      METALLB_ML_NAMESPACE:   metallb-system (v1:metadata.namespace)
      METALLB_ML_SECRET_KEY:  <set to the key 'secretkey' in secret 'memberlist'>  Optional: false
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from speaker-token-xfz9p (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  speaker-token-xfz9p:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  speaker-token-xfz9p
    Optional:    false
QoS Class:       Guaranteed
Node-Selectors:  kubernetes.io/os=linux
Tolerations:     node-role.kubernetes.io/master:NoSchedule
                 node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                 node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                 node.kubernetes.io/network-unavailable:NoSchedule op=Exists
                 node.kubernetes.io/not-ready:NoExecute op=Exists
                 node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                 node.kubernetes.io/unreachable:NoExecute op=Exists
                 node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type     Reason            Age                   From               Message
  ----     ------            ----                  ----               -------
  Normal   Scheduled         89m                   default-scheduler  Successfully assigned metallb-system/speaker-5kxc9 to taurus
  Normal   Pulling           89m                   kubelet            Pulling image "quay.io/metallb/speaker:main"
  Normal   Pulled            89m                   kubelet            Successfully pulled image "quay.io/metallb/speaker:main" in 14.647284366s
  Normal   Created           89m                   kubelet            Created container speaker
  Normal   Started           89m                   kubelet            Started container speaker
  Warning  DNSConfigForming  4m14s (x71 over 89m)  kubelet            Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 8.8.8.8 1.1.1.1 192.168.2.1
-- Pat Zio
kubernetes
load-balancing

3 Answers

2/9/2021

There seems to be an issue with the metallb config map you've created. In the address pools section you've mentioned the allocated IP range as 192.168.2.210 - 192.168.250, which is wrong because the second one is not a valid IP (192.168.250 is not a valid IP address).

Please edit the configmap using the following command,

$ kubectl edit configmap config -n metallb-system

and correct the range to 192.168.2.210 - 192.168.2.250. Then delete the service and create it again. If it still fails post to this thread again. Hope this helps :)

-- skogul1997
Source: StackOverflow

3/11/2022

I followed Kind documentation too, of using master version of metallb. My cluster was 1.21.1 and made LB works by using a specific version of metallb, i.e. v0.10.

-- apito
Source: StackOverflow

3/3/2021

The problem concern about MetalLB controller. I have face this issue as well for my action reintall MetalLB again. refer: https://github.com/metallb/metallb/issues/673

Remove

kubectl delete -f https://raw.githubusercontent.com/metallb/metallb/v0.9.5/manifests/metallb.yaml
kubectl delete -f metallb-configMap.yaml

Install

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.5/manifests/metallb.yaml
kubectl apply -f metallb-configMap.yaml

Don't worry about existing service that was assigned IP. As of you deleting the MetalLB controller the service can serve and still work as normal.

-- 6LYTH3
Source: StackOverflow