Summary: Jenkins in K8s minikkube works fine and scales well in case of default jnlp agent but stuck with "Waiting for agent to connect" in case of custom jnlp image.

Detailed description:

I'm running the local minikube with Jenkins setup.

Jenkins master dockerfile:

from jenkins/jenkins:alpine

# Distributed Builds plugins
RUN /usr/local/bin/install-plugins.sh ssh-slaves

# install Notifications and Publishing plugins
RUN /usr/local/bin/install-plugins.sh email-ext
RUN /usr/local/bin/install-plugins.sh mailer
RUN /usr/local/bin/install-plugins.sh slack

# Artifacts
RUN /usr/local/bin/install-plugins.sh htmlpublisher

# UI
RUN /usr/local/bin/install-plugins.sh greenballs
RUN /usr/local/bin/install-plugins.sh simple-theme-plugin

# Scaling
RUN /usr/local/bin/install-plugins.sh kubernetes

# install Maven
USER root
RUN apk update && \
    apk upgrade && \
    apk add maven
USER jenkins

Deployment:

 apiVersion: extensions/v1beta1
    kind: Deployment
    metadata:
      name: jenkins
    spec:
      replicas: 1
      template:
        metadata:
          labels:
            app: jenkins
        spec:
          containers:
            - name: jenkins
              image: ybushnev/my-jenkins-image:1.3
              env:
                - name: JAVA_OPTS
                  value: -Djenkins.install.runSetupWizard=false
              ports:
                - name: http-port
                  containerPort: 8080
                - name: jnlp-port
                  containerPort: 50000
              volumeMounts:
                - name: jenkins-home
                  mountPath: /var/jenkins_home
          volumes:
            - name: jenkins-home
              emptyDir: {}

Service:

apiVersion: v1
kind: Service
metadata:
  name: jenkins
spec:
  type: NodePort
  ports:
    - port: 8080
      name: "http"
      targetPort: 8080
    - port: 50000
      name: "slave"
      targetPort: 50000
  selector:
    app: jenkins

After deployment I have such services:

Yuris-MBP-2% kubectl get services       
NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                          AGE
jenkins      NodePort    10.108.30.10   <none>        8080:30267/TCP,50000:31588/TCP   1h
kubernetes   ClusterIP   10.96.0.1      <none>        443/TCP                          1h

Kubernetes master running on:

Yuris-MBP-2%  kubectl cluster-info | grep master
Kubernetes master is running at https://192.168.99.100:8443

Based on configuration above I specify the cloud config in Jenkins:

And finally I put such configuration for slave pod template:

As a result, via k8s logs I see such logs on the master:

Waiting for agent to connect (41/100): kubernetes-agent-tgskx
Waiting for agent to connect (42/100): kubernetes-agent-tgskx
Waiting for agent to connect (43/100): kubernetes-agent-tgskx
Waiting for agent to connect (44/100): kubernetes-agent-tgskx
Waiting for agent to connect (45/100): kubernetes-agent-tgskx

Jenkins container seems to be green. No logs in K8s but there are such events happened:

Successfully assigned kubernetes-agent-517tl to minikube
MountVolume.SetUp succeeded for volume "workspace-volume"
MountVolume.SetUp succeeded for volume "default-token-8sgh6"

IMPORTANT If I do not put 'jnlp' inside the container name (I guess this is the important as in another case it takes some default jnlp agent image) slave is spinning up and connecting to the master just fine but even if I have custom docker image inside the 'Docker image' field it doesn't take it as a reference as I can see that Jenkins slave doesn't have such tools/files which it suppose to have based in provided image. Last time I tried to use this image: "gcr.io/cloud-solutions-images/jenkins-k8s-slave" but for me it fails for any image in case I put 'jnlp' as container template name. I tried to play with many images with no luck... Will be very glad for any hint!