How to configure Jenkins master and dynamic agents in different clouds? With Kubernetes plugin
For context, I work at a large company with teams in different clouds (Azure and IBM cloud formerly Bluemix). The Ops team will have control of Jenkins master eventually, but since my team is the first needing this, we are setting it up.
What we have:
- Jenkins master is in Azure (cloud A), latest version, and we have the Kubernetes plugin (among others) to have dynamic provisioning of agents.
- My team works in IBM Kubernetes clusters (cloud B). In this cloud, we have a namespace for devops in a cluster of k8s, where we want the slave agents to be created and run. In this cloud we also created a service accounts in that namespace, a role, and rolebinding.
Problem: Configuring Jenkins master to connect with such namespace in IBM cloud to provision agents and run the jobs as needed. Especifically, in Jenkins > Manage > COnfigure System > Cloud. Here I have:
- Kubernetes URL set to the IP:port of the master node of the cluster
- Kubernetes server certificate key, I put here the certificate I got when I created the service account.
- Namespace, Credentials I have the service account and finally the Jenkins URL.
When I try that configuration, and test the connection I get:
Error testing connection https://ip:port: Failure executing: GET at: https://ip:port/api/v1/namespaces/devops/pods. Message: Unauthorized! Configured service account doesn't have access. Service account may have been revoked. Unauthorized.
Questions:
- Anybody knows how to do this or has links to something similar?
- In the image, what should I put in kubernetes URL? According to the docs, I should enter the container engine cluster endpoint, what is that in IBM Cloud? I have the IP and port of the Master node.
I´ve read a lot of documentation from IBM, the kubernetes plugin for Jenkins (https://github.com/jenkinsci/kubernetes-plugin) and tons of other posts explaining how to configure jenkins with kubernetes and dynamic provisioning and many of them said it is possible to have jenkins and the slaves in different clouds, but none of them explained how to do it..
Thank you in advance.
++++++++++++++++++ UPDATE +++++++++++++++++++++++
In the following screenshot I show the configurations that I am using. In particular, the fields Kubernetes URL and namespace.
Thanks to @samhain1138 for his help so far, but I cannot get a connection test successful in the Jenkins Kubernetes plugin configuration section. I think I may not be entering the correct info in some of the fields in that section.
Note: Please keep in mind that my setup is as follows: Jenkins master is in Cloud A (Azure) and I want the agent nodes to run is in a different cluster in another cloud, call it Cloud B (which in my case is IBM Cloud).
In the screenshot above I am certain that I have the correct values for the Kubernetes URL and namespace, but I am unsure about the other fields (Kubernetes server certificate key and Credentials.)
In the Kubernetes server certificate key field I tried putting:
- The ca.cert of the Service Account in the kubernetes cluster in IBM. (Obtained by 1. creating service account, getting the secret of that service account, and extracting the ca.cert from such secret)
- The "Token" from the server, which I get doing a kubectl config view in the kubernetes cluster in IBM. The token is in the field users>user>auth-provider>config>id-token when you execute the command "kubectl config view"
In Credentials I created and tried different Kinds:
- Username and Password: I tried this with my credentials for the IBM cluster and with the service account credentials (When I created the service account, role, and role-binding, I noticed that user was created which was serviceaccountname-namespace-cluster)
- Kubernetes Service Account
I tried all the combinations of those, and could never get Connection Successful when Testing the Connection.
Erros I get:
Error testing connection https://APISERVER: Failure executing: GET at: https://APISERVER/api/v1/namespaces/a-devops-namespace/pods. Message: Unauthorized! Configured service account doesn't have access. Service account may have been revoked. Unauthorized.
Error testing connection https://APISERVER: Failure executing: GET at: https://APISERVER/api/v1/namespaces/a-devops-namespace/pods. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods is forbidden: User "system:anonymous" cannot list pods in the namespace "a-devops-namespace".
Similar Questions
1 Answer
Alright, you'll need to change a few things in the Jenkins master UI which if I understand correctly runs within your Azure account.
First of all, the Jenkins master is trying to connect to the local (Azure) Kubernetes master.
You'll have to set it to something that resolves to the Kubernetes master running on IBM.
For for that matter, the hostname will not end with ".local".
I doubt the credentials you're using in the screenshot you posted correspond with the Kubernetes master on IBM (they correspond to the Azure Kubernetes master, otherwise you'd get a different error), so you'll need to update those, as well.
This is covered here, and when you're done, here (under "Then, set up the HTTP credentials by following these steps", ignore the gcloud command used to fetch the password).
Make sure the Kubernetes master on IBM is accessible from the Jenkins master on Azure using the "test connection" button.
Then you also seemed to confuse the Kubernetes namespace setting.
Your service account uses the devops namespace while your Jenkins master is configured to use the kubernetes-plugin namespace.
Service accounts correspond to namespaces so you'll want to update that setting to devops.
This is still an extremely bad idea, especially in terms of security (if I sniff your connection I can copy and use your IBM k8s master's password + service account, which allows me to launch pods!).
You really should be running that Jenkins master on IBM, or if you have a good reason for this weird setup, at least make sure this connection goes over a VPN...