K
Q

API Server: Unable to authenticate the request due to an error: invalid bearer token

5/7/2019

I set up a sample Kubernetes Cluster with 3 master nodes and 2 worker nodes. I tried to connect it to an OpenId Provider (in my case Keycloak). But when querying the API I get the following message from kubectl:

error: You must be logged in to the server (Unauthorized)

or via curl:

curl -H "Authorization: Bearer $token" -k https://192.168.178.30:6443/api/v1/namespaces/default/pods
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "Unauthorized",
  "reason": "Unauthorized",
  "code": 401
}

and in the API server logs it says

Unable to authenticate the request due to an error: invalid bearer token

According to jwt.io my token seems to be valid.

Configuration of the API server

In the API server I specified the following parameters:

oidc-issuer-url: https://192.168.178.25:8443/auth/realms/master
oidc-client-id: account
oidc-ca-file: /etc/kubernetes/ssl/keycloak-rootCA.pem
oidc-username-claim: sub
oidc-groups-claim: groups

I had to specify the CA file, because the certificate used for keycloak was a self signed one

The certificate seems to work, because a curl without the CA file results in an error; where specifying the CA file doesn't:

ansible@node1:~$ curl https://192.168.178.25:8443/auth/realms/master
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.


ansible@node1:~$ curl --cacert /etc/kubernetes/ssl/keycloak-rootCA.pem https://192.168.178.25:8443/auth/realms/master
{"realm":"master","public_key":"...","token-service":"https://192.168.178.25:8443/auth/realms/master/protocol/openid-connect","account-service":"https://192.168.178.25:8443/auth/realms/master/account","tokens-not-before":0}

JWT Token

The payload of my JWT token looks the following:

{
  "jti": "455feab7-5828-49f3-9243-a2ce0a1ae6f9",
  "exp": 1557246081,
  "nbf": 0,
  "iat": 1557246021,
  "iss": "https://192.168.178.25:8443/auth/realms/master",
  "aud": "account",
  "sub": "78f9bdc8-d1a4-43c4-ab83-f1b4645519f6",
  "typ": "ID",
  "azp": "account",
  "auth_time": 1557246020,
  "session_state": "ccafaa62-a888-43dd-9135-1c5a31766e0b",
  "acr": "1",
  "email_verified": true,
  "name": "Testuser",
  "groups": [
    "group1",
    "k8s-admin"
  ],
  "preferred_username": "dummy-username",
  "given_name": "Firstname1",
  "family_name": "Lastname1",
  "email": "me@example.com"
}

When I try to login with the token of a generated service account (from kubernetes) everything works fine.

additional Information

The environment was set up via Kubespray and the API server parameters specified in the following way:

kube_oidc_url: https://192.168.178.25:8443/auth/realms/master
kube_oidc_client_id: account
kube_oidc_ca_file: "{{ kube_cert_dir }}/keycloak-rootCA.pem"
kube_oidc_username_claim: sub
kube_oidc_groups_claim: groups
#kube_oidc_username_prefix: "oidc:"
#kube_oidc_groups_prefix: "oidc:"

Thanks for your help.

-- chresse
keycloak
kubernetes
kubernetes-apiserver
kubespray

Similar Questions

how to run docker-compose.yaml on kubernetes?
The connection to the server rancher.xxx.xxx was refused - did you specify the right host or port?
How to distinguish different nodes with cpu/mem
Isito Ingress Controller Virtual Service returning 503
Azure Keyvault Bad Request When Called From Kubernetes Pod

2 Answers

5/7/2019

I think in your case kube_oidc_url should be https://192.168.178.25:8443

Also make note of CA certificates requirement from documentation:

A note about requirement #3 above, requiring a CA signed certificate. If you deploy your own identity provider (as opposed to one of the cloud providers like Google or Microsoft) you MUST have your identity provider’s web server certificate signed by a certificate with the CA flag set to TRUE, even if it is self signed. This is due to GoLang’s TLS client implementation being very strict to the standards around certificate validation. If you don’t have a CA handy, you can use this script from the CoreOS team to create a simple CA and a signed certificate and key pair. Or you can use this similar script that generates SHA256 certs with a longer life and larger key size.

-- Vasily Angapov
Source: StackOverflow
5/8/2019

The issue was that Kubespray did not overtake my settings into the Pod spec of the api-server (/etc/kubernetes/manifests/kube-apiserver.yaml). When I added the parameters in the Pod spec of each api-server (I have a multi master setup), it worked as expected:

- --oidc-issuer-url=https://192.168.178.25:8443/auth/realms/master
- --oidc-client-id=account
- --oidc-username-claim=email
- '--oidc-username-prefix=oidc:'
- --oidc-groups-claim=groups
- '--oidc-groups-prefix=oidc:'
- --oidc-ca-file=/etc/kubernetes/ssl/keycloak-rootCA.pem
-- chresse
Source: StackOverflow