K
Q

How Pod status becomes MatchNodeSelector

6/15/2019

In our staging environment, there is a defect reported. After kubernetes updated, some Pods status become 'MatchNodeSelector'.

But I have no idea why and how some Pods become 'MatchNodeSelector'. So I do some research 'if a Pod have a field 'nodeSelector' and doesn't have any node have this label. These Pods will become 'MatchNodeSelector'.

But I cannot reproduce it. The Pods status always becomes 'Pending', not 'MatchNodeSelector'. So my question is that how can I make a Pod status become 'MatchNodeSelector'?

-- Cain
kubernetes
kubernetes-pod

Similar Questions

kubectl get pods returns inconsistent results
Kubernetes network policy wtih external services
How to kill a docker container when it reaches set of memory usage or CPU limit
Data from volumes as kubernetes secrets
How to get list of NGINX VirtualServer and VirtualServerRoute in K8S?
Kubernetes No external IP
Run powershell Commands in Dockerfile

2 Answers

4/24/2020

MatchNodeSelector is a status of a pod that I've observed after worker node restarts. It appears on pods which use the .spec.nodeSelector stanza. What happens:

  1. kubelet starts and loads the labels from the command-line, e.g. /usr/bin/kubelet ... --node-labels=node-kind.foo.io/master=
  2. during a start-up among other things kubelet asks etcd for:
    • list of the additional node labels that were created via API in the past
    • list of the pods to run on the node
  3. sometimes it starts bringing up the pods before the full set of labels is applied on the node
  4. as a result, if your pod is supposed to be started on this node, and its .spec.NodeSelector is using labels from etcd/API, not from command-line, kubelet sees a mismatch between the expectations of the pod and the labels applied on this node (because the full refresh from etcd hasn't been processed yet)
  5. typically shortly after the full set of labels is merged (from cmd line and from etcd) and the next pods starts correctly

References: here and here

-- Bernard Halas
Source: StackOverflow
2/19/2020

I have been able to reproduce this by:

  1. For any running pod with no node affinity provided, export its yaml or json with kubectl get po -o json --export. Do note the node on which it is running

  2. Delete the pod.

  3. Explicitly provide the node affinity to a node which is not the same as the previous node on which it was running. Add it in spec with something like this:

 "spec": {
     "affinity": {
        "nodeAffinity": {
          "requiredDuringSchedulingIgnoredDuringExecution": {
            "nodeSelectorTerms": [
              {
                "matchExpressions": [
                  {
                    "key": "kubernetes.io/hostname",
                    "operator": "In",
                    "values": [
                      "new node name"
                    ]
                  }
                ]
              }
            ]
          }
        }
      }
    }
  1. Create the pod with kubectl create -f pod.json
  2. Pod status is now MatchNodeSelector.

This is because there is a node option in the same spec as well, which also needs to be changed. It is not removed by the --export option and would still have the older node name in that field.

I am guessing if it conflicts with whatever node name you gave in affinity, it will result in this status.

These are the versions: `Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.3", GitCommit:"2bba0127d85d5a46ab4b778548be28623b32d0b0", GitTreeState:"clean", BuildDate:"2018-05-21T09:17:39Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"darwin/amd64"}

Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.1", GitCommit:"eec55b9ba98609a46fee712359c7b5b365bdd920", GitTreeState:"clean", BuildDate:"2018-12-13T10:31:33Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}`

-- hitman__o_o
Source: StackOverflow