Kubernetes service under default namespace is unreachable or throws connection refused error

2/1/2021

I am trying to deploy the 'lighthouse' application in kubernetes cluster with 'v1.18.14' and the pods are up and running but the logs shows that there is a connection refused error on kubernettes service ip 10.233.0.1 on port 443.

Logs of lighthouse pod -

[centos@master elk_stack_6.x]$ kubectl logs lighthouse-webhooks-7f58c9897c-tbvfz -n jx
{"fields.level":"info","level":"info","msg":"setting the log level","time":"2021-01-21T03:35:52Z"}
{"level":"info","msg":"updating the Lighthouse core configuration","time":"2021-01-21T03:35:52Z"}
{"level":"warning","msg":"unknown plugin","plugin":"blunderbuss","time":"2021-01-21T03:35:52Z"}
{"level":"warning","msg":"unknown plugin","plugin":"heart","time":"2021-01-21T03:35:52Z"}
{"level":"info","msg":"updating the Lighthouse plugins configuration","time":"2021-01-21T03:35:52Z"}
{"level":"warning","msg":"not pushing metrics as there is no push_gateway defined in the config.yaml","time":"2021-01-21T03:35:52Z"}
{"level":"info","msg":"Lighthouse is now listening on path /hook and port 8080 for WebHooks","time":"2021-01-21T03:35:52Z"}
{"level":"info","msg":"Lighthouse is serving prometheus metrics on port 2112","time":"2021-01-21T03:35:52Z"}
E0129 07:23:34.030912       1 reflector.go:309] pkg/mod/k8s.io/client-go@v0.17.6/tools/cache/reflector.go:105: Failed to watch *v1.ConfigMap: Get https://10.233.0.1:443/api/v1/namespaces/jx/configmaps?watch=true: dial tcp 10.233.0.1:443: connect: connection refused
E0129 07:23:35.031560       1 reflector.go:309] pkg/mod/k8s.io/client-go@v0.17.6/tools/cache/reflector.go:105: Failed to watch *v1.ConfigMap: Get https://10.233.0.1:443/api/v1/namespaces/jx/configmaps?watch=true: dial tcp 10.233.0.1:443: connect: connection refused

Service under default namespace -

[centos@master elk_stack_6.x]$ kubectl get svc -n default
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.233.0.1   <none>        443/TCP   12d

Endpoints of default namespace -

[centos@master elk_stack_6.x]$ kubectl get endpoints -n default
NAME         ENDPOINTS             AGE
kubernetes   167.254.204.56:6443   12d

Trying to connect kubernetes service using curl -

[centos@master ~]$ curl -kv https://10.233.0.1:443
* About to connect() to 10.233.0.1 port 443 (#0)
*   Trying 10.233.0.1...
* Connected to 10.233.0.1 (10.233.0.1) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* NSS: client certificate not found (nickname not specified)
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
*       subject: CN=kube-apiserver
*       start date: Jan 20 13:49:12 2021 GMT
*       expire date: Jan 20 13:49:12 2022 GMT
*       common name: kube-apiserver
*       issuer: CN=kubernetes
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 10.233.0.1
> Accept: */*
>
< HTTP/1.1 403 Forbidden
< Cache-Control: no-cache, private
< Content-Type: application/json
< X-Content-Type-Options: nosniff
< Date: Tue, 02 Feb 2021 07:57:21 GMT
< Content-Length: 233
<
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
  "reason": "Forbidden",
  "details": {

  },
  "code": 403
* Connection #0 to host 10.233.0.1 left intact

I am facing 403 forbidden error may be because of some missing RBAC policies. Any suggestions would be appreciated.

-- Bhavani Prasad
kubernetes
kubernetes-service

2 Answers

2/2/2021

As per pod logs and Curl command output, it may be due to permission issue with Service Account the pod is running with. Check if Service Account of the pod has proper permission to watch/read configMap in the default namespace. If not there, then RBAC role and role binding K8S objects need to be created.

-- Rushikesh
Source: StackOverflow

2/2/2021

Kindly try the HTTPS endpoint as mentioned in the error instead of HTTP.

400 error would mean that the request is indeed reaching the server but not in a format that the server is expecting.

Updated to reflect updated question:-

403 means the user via which the cluster is being accessed is not authorised to use the IP for accessing the resource.

Detailed explanation can be found protecting your kubernetes api server

-- rohatgisanat
Source: StackOverflow