I have a healthy k8s cluster. The kube-apiserver is using port 8443. Then I do something like below. I create/edit a Sevice and assign a fixed nodePort 8443, too.
apiVersion: v1
kind: Service
metadata:
name: my-service
spec:
selector:
app: MyApp
ports:
- protocol: TCP
port: 80
targetPort: 9376
nodePort: 8443The cluster becomes unreachable. If I run command kubectl get pods. It returns Unable to connect to the server: EOF
Cause I need run kubectl command to change the nodePort back. But I cannot use kubectl. It becomes a dead loop. I have 2 questions. 1. What can I do to get my k8s cluster back? 2. Why kube-apiserver or other components doesn't prevent me to change the nodePort to 8443? Indicate the port 8443 is already in use.
It should not be possible when port range is set with default service-node-port-range api-server flag.
--service-node-port-range portRange Default: 30000-32767
A port range to reserve for services with NodePort visibility. Example: '30000-32767'. Inclusive at both ends of the range.
What can I do to get my k8s cluster back?
You can change /etc/kubernetes/manifests/kube-apiserver.yaml file and reaplace all occurrences of 8443 to e.g. 6443.
Then connect to apiserver with:
$ kubectl --server https://<ip>:6443 ...Delete the my-service service and restore kube-apiserver.yaml file to its previous form.
Why kube-apiserver or other components doesn't prevent me to change the nodePort to 8443? Indicate the port 8443 is already in use.
I don't think many people encounter this problem because people usually use api-server with default values of --service-node-port-range flag which is: 30000-32767. In such case you would see this error:
The Service "my-service" is invalid: spec.ports[0].nodePort: Invalid value: 8443: provided port is not in the valid range. The range of valid ports is 30000-32767Probably the best thing you can do with it is to ask k8s developers on k8s github repo. Just open an issue and as this question there.
kube-apiserver pod is a static pod brought up by kubelet using /etc/kubernetes/manifests/kube-apiserver.yaml file on master nodes
so you can edit /etc/kubernetes/manifests/kube-apiserver.yaml , where 8443 port is referenced and change it to a unused port. so that kube-apiserver will be available on that port. ideally kubelet tries to restart a static pod when its manifest file is changed .
you need to edit ~/.kube/config , where 'server' address needs to be changed to new port.
delete the problematic nodeport service .
revert your changes in /etc/kubernetes/manifests/kube-apiserver.yaml & .kube/config file to go back to original port (8443)