Exposing Jenkins in GKE using Ingress NGINX Controller with Google login plugin

3/24/2020

I'm trying to setup my Jenkins instance in Google Kubernetes Engine, also I am using Google login plugin so that I could login with my GCP user to Jenkins, I have installed Ingress controller which is NGINX and exposed Jenkins service using ingress.

Domain under which I want to access Jenkins is : util.my-app.com/jenkins

In Jenkins config under parameter Jenkins URL I also set this domain name util.my-app.com/jenkins

And here's my Ingress:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: jenkins-ing
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: util.my-app.com
    http:
      paths:
      - path: /jenkins/*
        backend:
          serviceName: jenkins-svc
          servicePort: 80

In GCP Credentials page under Authorized JavaScript origins I set http://util.my-app.com and under Authorized redirect URIs I set http://util.my-app.com/jenkins/securityRealm/finishLogin

It's either returning me 404 status or doing infinite redirects, what I noticed that when Jenkins Google login plugin does redirect it is like this http://util.my-app.com/securityRealm/finishLogin without "jenkins" part, what is wrong with my setup ?

-- Laimis
jenkins
kubernetes
nginx
proxy

1 Answer

3/24/2020

Welcome to Stack Laimis!

I tested your ingress object, and there one issue.

Your Ingress is missing the rewrite-target:

In some scenarios the exposed URL in the backend service differs from the specified path in the Ingress rule. Without a rewrite any request will return 404. Set the annotation nginx.ingress.kubernetes.io/rewrite-target to the path expected by the service.

This example from documentation shows the structure required:

Here is your ingress with the edit:

  • added the line nginx.ingress.kubernetes.io/rewrite-target: /$1
apiVersion: networking.k8s.io/v1beta1 # for versions before 1.14 use extensions/v1beta1
kind: Ingress
metadata:
  name: jenkins-ing
  annotations:
     nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
  rules:
    - host: util.my-app.com
      http:
        paths:
          - path: /jenkins/*
            backend:
              serviceName: jenkins-svc
              servicePort: 80

Reproduction:

  • First I created a deployment. For that I'm using echo-app for it's elucidative output.
  • Add a service to expose it inside the cluster on port 8080 and outside as NodePort.
apiVersion: apps/v1
kind: Deployment
metadata:
 name: echo1-deploy
spec:
 selector:
   matchLabels:
     app: echo1-app
 template:
   metadata:
     labels:
       app: echo1-app
   spec:
     containers:
     - name: echo1-app
       image: mendhak/http-https-echo
       ports:
       - name: http
         containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
 name: echo1-svc
spec:
 type: NodePort
 selector:
   app: echo1-app
 ports:
   - protocol: TCP
     port: 8080
     targetPort: 80
  • I'll create one more deployment and service, this way we can try a little further with Ingress and to demonstrate what to do when you have more than one service that you need to expose on ingress
apiVersion: apps/v1
kind: Deployment
metadata:
 name: echo2-deploy
spec:
 selector:
   matchLabels:
     app: echo2-app
 template:
   metadata:
     labels:
       app: echo2-app
   spec:
     containers:
     - name: echo2-app
       image: mendhak/http-https-echo
       ports:
       - name: http
         containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
 name: echo2-svc
spec:
 type: NodePort
 selector:
   app: echo2-app
 ports:
   - protocol: TCP
     port: 8080
     targetPort: 80
  • I'll use an ingress, just like yours, the only diferences are:
    • Changed the service to echo1-svc emulating your jenkins-svc
    • Added another service to echo2-svc to redirect all http requests except the ones that match the first rule.
apiVersion: networking.k8s.io/v1beta1 # for versions before 1.14 use extensions/v1beta1
kind: Ingress
metadata:
  name: jenkins-ing
  annotations:
     nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
  rules:
    - host: util.my-app.com
      http:
        paths:
          - path: /jenkins/*
            backend:
              serviceName: echo1-svc
              servicePort: 80
          - path: /(.*)
            backend:
              serviceName: echo2-svc
              servicePort: 80
  • Now I'll deploy this apps and the ingress:
$ kubectl apply -f echo1-deploy.yaml 
deployment.apps/echo1-deploy created
service/echo1-svc created

$ kubectl apply -f echo2-deploy.yaml 
deployment.apps/echo2-deploy created
service/echo2-svc created

$ kubectl apply -f jenkins-ing.yaml 
ingress.networking.k8s.io/jenkins-ing created
  • Now let's check if everything is running:
$ kubectl get all
NAME                               READY   STATUS    RESTARTS   AGE
pod/echo1-deploy-989766d57-8pmhj   1/1     Running   0          27m
pod/echo2-deploy-65b6ffbcf-lfgzk   1/1     Running   0          27m

NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
service/echo1-svc    NodePort    10.101.127.78   <none>        8080:30443/TCP   27m
service/echo2-svc    NodePort    10.106.34.91    <none>        8080:32628/TCP   27m

NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/echo1-deploy   1/1     1            1           27m
deployment.apps/echo2-deploy   1/1     1            1           27m

NAME                                     DESIRED   CURRENT   READY   AGE
replicaset.apps/echo1-deploy-989766d57   1         1         1       27m
replicaset.apps/echo2-deploy-65b6ffbcf   1         1         1       27m

$ kubectl get ingress
NAME          HOSTS             ADDRESS   PORTS   AGE
jenkins-ing   util.my-app.com             80      4s
  • As you can see the echo1-svc is exposed outside kubernetes on port 30443 and echo2-svc on port 32628
  • With the ingress rule we can Curl on port 80 and it will redirect to the specified service.
  • Since I don't have this domain, I'll add a record on my /etc/hosts file to emulate the DNS resolution directing it to my kubernetes IP.
$ cat /etc/hosts
127.0.0.1       localhost
::1             localhost ip6-localhost ip6-loopback
ff02::1         ip6-allnodes
ff02::2         ip6-allrouters

192.168.39.240  util.my-app.com

$ curl util.my-app.com/jenkins
{
  "headers": {
    "host": "util.my-app.com",
    "x-real-ip": "192.168.39.1",
    "x-forwarded-host": "util.my-app.com",
    "x-forwarded-port": "80",
    "x-forwarded-proto": "http",
    "user-agent": "curl/7.52.1",
  },
  "method": "GET",
  "hostname": "util.my-app.com",
  "ip": "::ffff:172.17.0.6",
  "protocol": "http",
   "subdomains": [
    "util"
  ],
  "os": {
    "hostname": "echo1-deploy-989766d57-8pmhj"
  }

You can see the HTTP GET was redirected to the pod on the backend of echo1-svc - Now let's check what happens when we curl the domain without the /jenkins/

$ curl util.my-app.com
{
  "headers": {
    "host": "util.my-app.com",
    "x-real-ip": "192.168.39.1",
    "x-forwarded-host": "util.my-app.com",
    "x-forwarded-port": "80",
    "x-forwarded-proto": "http",
    "user-agent": "curl/7.52.1",
 },
  "method": "GET",
  "hostname": "util.my-app.com",
  "ip": "::ffff:172.17.0.6",
  "protocol": "http",
  "subdomains": [
    "util"
  ],
  "os": {
    "hostname": "echo2-deploy-65b6ffbcf-lfgzk"

You can see the HTTP GET was redirected to the pod on the backend of echo2-svc.


  • I know your question also addresses issues about your google authentication, but I'd suggest you to first correct your ingress, then you can paste here the status it will return if you still have problems with the ingress redirecting correctly.

If you have any doubt let me know in the comments.

-- willrof
Source: StackOverflow