I am trying to deploy the 'lighthouse' application in kubernetes cluster with 'v1.18.14' and the pods are up and running but the logs shows that there is a connection refused error on kubernettes service ip 10.233.0.1 on port 443.
Logs of lighthouse pod -
[centos@master elk_stack_6.x]$ kubectl logs lighthouse-webhooks-7f58c9897c-tbvfz -n jx
{"fields.level":"info","level":"info","msg":"setting the log level","time":"2021-01-21T03:35:52Z"}
{"level":"info","msg":"updating the Lighthouse core configuration","time":"2021-01-21T03:35:52Z"}
{"level":"warning","msg":"unknown plugin","plugin":"blunderbuss","time":"2021-01-21T03:35:52Z"}
{"level":"warning","msg":"unknown plugin","plugin":"heart","time":"2021-01-21T03:35:52Z"}
{"level":"info","msg":"updating the Lighthouse plugins configuration","time":"2021-01-21T03:35:52Z"}
{"level":"warning","msg":"not pushing metrics as there is no push_gateway defined in the config.yaml","time":"2021-01-21T03:35:52Z"}
{"level":"info","msg":"Lighthouse is now listening on path /hook and port 8080 for WebHooks","time":"2021-01-21T03:35:52Z"}
{"level":"info","msg":"Lighthouse is serving prometheus metrics on port 2112","time":"2021-01-21T03:35:52Z"}
E0129 07:23:34.030912 1 reflector.go:309] pkg/mod/k8s.io/client-go@v0.17.6/tools/cache/reflector.go:105: Failed to watch *v1.ConfigMap: Get https://10.233.0.1:443/api/v1/namespaces/jx/configmaps?watch=true: dial tcp 10.233.0.1:443: connect: connection refused
E0129 07:23:35.031560 1 reflector.go:309] pkg/mod/k8s.io/client-go@v0.17.6/tools/cache/reflector.go:105: Failed to watch *v1.ConfigMap: Get https://10.233.0.1:443/api/v1/namespaces/jx/configmaps?watch=true: dial tcp 10.233.0.1:443: connect: connection refused
Service under default namespace -
[centos@master elk_stack_6.x]$ kubectl get svc -n default
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.233.0.1 <none> 443/TCP 12d
Endpoints of default namespace -
[centos@master elk_stack_6.x]$ kubectl get endpoints -n default
NAME ENDPOINTS AGE
kubernetes 167.254.204.56:6443 12d
Trying to connect kubernetes service using curl -
[centos@master ~]$ curl -kv https://10.233.0.1:443
* About to connect() to 10.233.0.1 port 443 (#0)
* Trying 10.233.0.1...
* Connected to 10.233.0.1 (10.233.0.1) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* NSS: client certificate not found (nickname not specified)
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
* subject: CN=kube-apiserver
* start date: Jan 20 13:49:12 2021 GMT
* expire date: Jan 20 13:49:12 2022 GMT
* common name: kube-apiserver
* issuer: CN=kubernetes
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 10.233.0.1
> Accept: */*
>
< HTTP/1.1 403 Forbidden
< Cache-Control: no-cache, private
< Content-Type: application/json
< X-Content-Type-Options: nosniff
< Date: Tue, 02 Feb 2021 07:57:21 GMT
< Content-Length: 233
<
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {
},
"status": "Failure",
"message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
"reason": "Forbidden",
"details": {
},
"code": 403
* Connection #0 to host 10.233.0.1 left intact
I am facing 403 forbidden error may be because of some missing RBAC policies. Any suggestions would be appreciated.
As per pod logs and Curl command output, it may be due to permission issue with Service Account the pod is running with. Check if Service Account of the pod has proper permission to watch/read configMap in the default namespace. If not there, then RBAC role and role binding K8S objects need to be created.
Kindly try the HTTPS endpoint as mentioned in the error instead of HTTP.
400 error would mean that the request is indeed reaching the server but not in a format that the server is expecting.
Updated to reflect updated question:-
403 means the user via which the cluster is being accessed is not authorised to use the IP for accessing the resource.
Detailed explanation can be found protecting your kubernetes api server