I recently enabled RBAC at Kubernetes. Since than, Jenkins (running on Kubernetes, creating agent-pods on the very same Kubernetes) is able to create agent-pods, but is unable to connect to JNLP via Port 50'000.
I noticed a reference for Connecting to jenkins.example.de:50000
, but did not find where this is configured, as it must resolve Kubernetes-Internal (Kube-DNS), as the port is not exposed from outside.
I noticed (and updated) configuration at Configure System
> Jenkins Location
> Jenkins URL
, leading to failed RBAC logins (Keycloak), as redirect URL is set incorrectly. Futher it does not feel correct for configuring cluster-internal endpoints for JNLP. I can chose between JNLP being able to work with cluster-internal URL or Being able to login, using RBAC:
kubectl get all -o wide -n jenkins
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/jenkins-64ff7ff784-nq8jh 2/2 Running 0 22h 192.168.0.35 kubernetes-slave02 <none> <none>
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/jenkins-svc ClusterIP 10.105.132.134 <none> 8080/TCP,50000/TCP 68d app=jenkins
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
deployment.apps/jenkins 1/1 1 1 68d jenkins jenkins/jenkins:latest app=jenkins
NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR
replicaset.apps/jenkins-64ff7ff784 1 1 1 68d jenkins jenkins/jenkins:latest app=jenkins,pod-template-hash=64ff7ff784
kubectl describe -n jenkins pod/worker-c82ea4bd-52e1-47c6-bad7-4a416a1e6897-z1bn0-2jm7b
Name: worker-c82ea4bd-52e1-47c6-bad7-4a416a1e6897-z1bn0-2jm7b
Namespace: jenkins
Priority: 0
Node: kubernetes-slave/192.168.190.116
Start Time: Fri, 08 Jan 2021 17:16:56 +0100
Labels: istio.io/rev=default
jenkins=jenkins-slave
jenkins/label=worker-c82ea4bd-52e1-47c6-bad7-4a416a1e6897
jenkins/label-digest=9f81f8f2dabeba69de7d48422a0fc3cbdbaa8ce0
security.istio.io/tlsMode=istio
service.istio.io/canonical-name=worker-c82ea4bd-52e1-47c6-bad7-4a416a1e6897-z1bn0-2jm7b
service.istio.io/canonical-revision=latest
Annotations: buildUrl: https://jenkins.example.de/job/APP-Kiali/job/master/63/
cni.projectcalico.org/podIP: 192.168.4.247/32
cni.projectcalico.org/podIPs: 192.168.4.247/32
prometheus.io/path: /stats/prometheus
prometheus.io/port: 15020
prometheus.io/scrape: true
runUrl: job/APP-Kiali/job/master/63/
sidecar.istio.io/status:
{"version":"e2cb9d4837cda9584fd272bfa1f348525bcaacfadb7e9b9efbd21a3bb44ad7a1","initContainers":["istio-init"],"containers":["istio-proxy"]...
Status: Terminating (lasts <invalid>)
Termination Grace Period: 30s
IP: 192.168.4.247
IPs:
IP: 192.168.4.247
Init Containers:
istio-init:
Container ID: docker://182de6a71b33e7350263b0677f510f85bd8da9c7938ee5c6ff43b083efeffed6
Image: docker.io/istio/proxyv2:1.8.1
Image ID: docker-pullable://istio/proxyv2@sha256:0a407ecee363d8d31957162b82738ae3dd09690668a0168d660044ac8fc728f0
Port: <none>
Host Port: <none>
Args:
istio-iptables
-p
15001
-z
15006
-u
1337
-m
REDIRECT
-i
*
-x
-b
*
-d
15090,15021,15020
State: Terminated
Reason: Completed
Exit Code: 0
Started: Fri, 08 Jan 2021 17:17:01 +0100
Finished: Fri, 08 Jan 2021 17:17:02 +0100
Ready: True
Restart Count: 0
Limits:
cpu: 2
memory: 1Gi
Requests:
cpu: 100m
memory: 128Mi
Environment:
DNS_AGENT:
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-7htdh (ro)
Containers:
kubectl:
Container ID: docker://fb2b1ce8374799b6cc59db17fec0bb993b62369cd7cb2b71ed9bb01c363649cd
Image: lachlanevenson/k8s-kubectl:latest
Image ID: docker-pullable://lachlanevenson/k8s-kubectl@sha256:47e2096ae077b6fe7fdfc135c53feedb160d3b08001b8c855d897d0d37fa8c7e
Port: <none>
Host Port: <none>
Command:
cat
State: Running
Started: Fri, 08 Jan 2021 17:17:03 +0100
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/home/jenkins/agent from workspace-volume (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-7htdh (ro)
jnlp:
Container ID: docker://58ee7b399077701f3f0a99ed97eb6f1e400976b7946d209d2bee64be32a94885
Image: jenkins/inbound-agent:4.3-4
Image ID: docker-pullable://jenkins/inbound-agent@sha256:62f48a12d41e02e557ee9f7e4ffa82c77925b817ec791c8da5f431213abc2828
Port: <none>
Host Port: <none>
State: Terminated
Reason: Error
Exit Code: 255
Started: Fri, 08 Jan 2021 17:17:04 +0100
Finished: Fri, 08 Jan 2021 17:17:15 +0100
Ready: False
Restart Count: 0
Requests:
cpu: 100m
memory: 256Mi
Environment:
JENKINS_PROTOCOLS: JNLP4-connect
JENKINS_SECRET: ****
JENKINS_AGENT_NAME: worker-c82ea4bd-52e1-47c6-bad7-4a416a1e6897-z1bn0-2jm7b
JENKINS_DIRECT_CONNECTION: jenkins.example.de:50000
JENKINS_INSTANCE_IDENTITY: ****
JENKINS_NAME: worker-c82ea4bd-52e1-47c6-bad7-4a416a1e6897-z1bn0-2jm7b
JENKINS_AGENT_WORKDIR: /home/jenkins/agent
Mounts:
/home/jenkins/agent from workspace-volume (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-7htdh (ro)
istio-proxy:
Container ID: docker://9a87cafa07779cfc98c58678f484e48e28e354060573c19db9d3d9c86be7a496
Image: docker.io/istio/proxyv2:1.8.1
Image ID: docker-pullable://istio/proxyv2@sha256:0a407ecee363d8d31957162b82738ae3dd09690668a0168d660044ac8fc728f0
Port: 15090/TCP
Host Port: 0/TCP
Args:
proxy
sidecar
--domain
$(POD_NAMESPACE).svc.cluster.local
--serviceCluster
worker-c82ea4bd-52e1-47c6-bad7-4a416a1e6897-z1bn0-2jm7b.jenkins
--proxyLogLevel=warning
--proxyComponentLogLevel=misc:error
--concurrency
2
State: Running
Started: Fri, 08 Jan 2021 17:17:11 +0100
Ready: True
Restart Count: 0
Limits:
cpu: 2
memory: 1Gi
Requests:
cpu: 100m
memory: 128Mi
Readiness: http-get http://:15021/healthz/ready delay=1s timeout=3s period=2s #success=1 #failure=30
Environment:
JWT_POLICY: first-party-jwt
PILOT_CERT_PROVIDER: istiod
CA_ADDR: istiod.istio-system.svc:15012
POD_NAME: worker-c82ea4bd-52e1-47c6-bad7-4a416a1e6897-z1bn0-2jm7b (v1:metadata.name)
POD_NAMESPACE: jenkins (v1:metadata.namespace)
INSTANCE_IP: (v1:status.podIP)
SERVICE_ACCOUNT: (v1:spec.serviceAccountName)
HOST_IP: (v1:status.hostIP)
CANONICAL_SERVICE: (v1:metadata.labels['service.istio.io/canonical-name'])
CANONICAL_REVISION: (v1:metadata.labels['service.istio.io/canonical-revision'])
PROXY_CONFIG: {"proxyMetadata":{"DNS_AGENT":""}}
ISTIO_META_POD_PORTS: [
]
ISTIO_META_APP_CONTAINERS: kubectl,jnlp
ISTIO_META_CLUSTER_ID: Kubernetes
ISTIO_META_INTERCEPTION_MODE: REDIRECT
ISTIO_METAJSON_ANNOTATIONS: {"buildUrl":"https://jenkins.example.de/job/APP-Kiali/job/master/63/","runUrl":"job/APP-Kiali/job/master/63/"}
ISTIO_META_WORKLOAD_NAME: worker-c82ea4bd-52e1-47c6-bad7-4a416a1e6897-z1bn0-2jm7b
ISTIO_META_OWNER: kubernetes://apis/v1/namespaces/jenkins/pods/worker-c82ea4bd-52e1-47c6-bad7-4a416a1e6897-z1bn0-2jm7b
ISTIO_META_MESH_ID: cluster.local
TRUST_DOMAIN: cluster.local
DNS_AGENT:
Mounts:
/etc/istio/pod from istio-podinfo (rw)
/etc/istio/proxy from istio-envoy (rw)
/var/lib/istio/data from istio-data (rw)
/var/run/secrets/istio from istiod-ca-cert (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-7htdh (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
workspace-volume:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
default-token-7htdh:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-7htdh
Optional: false
istio-envoy:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: <unset>
istio-data:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
istio-podinfo:
Type: DownwardAPI (a volume populated by information about the pod)
Items:
metadata.labels -> labels
metadata.annotations -> annotations
istiod-ca-cert:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: istio-ca-root-cert
Optional: false
QoS Class: Burstable
Node-Selectors: kubernetes.io/os=linux
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 26s default-scheduler Successfully assigned jenkins/worker-c82ea4bd-52e1-47c6-bad7-4a416a1e6897-z1bn0-2jm7b to kubernetes-slave
Normal Pulling 24s kubelet Pulling image "docker.io/istio/proxyv2:1.8.1"
Normal Pulled 21s kubelet Successfully pulled image "docker.io/istio/proxyv2:1.8.1" in 2.897659504s
Normal Created 21s kubelet Created container istio-init
Normal Started 21s kubelet Started container istio-init
Normal Pulled 19s kubelet Container image "lachlanevenson/k8s-kubectl:latest" already present on machine
Normal Created 19s kubelet Created container kubectl
Normal Started 19s kubelet Started container kubectl
Normal Pulled 19s kubelet Container image "jenkins/inbound-agent:4.3-4" already present on machine
Normal Created 19s kubelet Created container jnlp
Normal Started 18s kubelet Started container jnlp
Normal Pulling 18s kubelet Pulling image "docker.io/istio/proxyv2:1.8.1"
Normal Pulled 11s kubelet Successfully pulled image "docker.io/istio/proxyv2:1.8.1" in 7.484694118s
Normal Created 11s kubelet Created container istio-proxy
Normal Started 11s kubelet Started container istio-proxy
Warning Unhealthy 9s kubelet Readiness probe failed: Get "http://192.168.4.247:15021/healthz/ready": dial tcp 192.168.4.247:15021: connect: connection refused
Normal Killing 6s kubelet Stopping container kubectl
Normal Killing 6s kubelet Stopping container istio-proxy
fabiansc@Kubernetes-Master:~$ kubectl logs -n jenkins pod/worker-c82ea4bd-52e1-47c6-bad7-4a416a1e6897-z1bn0-2jm7b
error: a container name must be specified for pod worker-c82ea4bd-52e1-47c6-bad7-4a416a1e6897-z1bn0-2jm7b, choose one of: [kubectl jnlp istio-proxy] or one of the init containers: [istio-init]
fabiansc@Kubernetes-Master:~$ kubectl logs -n jenkins pod/worker-c82ea4bd-52e1-47c6-bad7-4a416a1e6897-z1bn0-2jm7b -c kubectl
fabiansc@Kubernetes-Master:~$ kubectl logs -n jenkins pod/worker-c82ea4bd-52e1-47c6-bad7-4a416a1e6897-z1bn0-2jm7b -c jnlp
unable to retrieve container logs for docker://58ee7b399077701f3f0a99ed97eb6f1e400976b7946d209d2bee64be32a94885fabiansc@Kubernetes-Master:~$ kubectl logs -n jenkins pod/worker-c82ea4bd-52e1-47c6-bad7-4a416a1e6897-z1bn0-2jm7b -c jnlp -c jnlppod/worker-c82ea4bd-52e1-47c6-bad7-4a416a1e6897-z1bn0-t57rw
error: expected 'logs [-f] [-p] (POD | TYPE/NAME) [-c CONTAINER]'.
POD or TYPE/NAME is a required argument for the logs command
See 'kubectl logs -h' for help and examples
fabiansc@Kubernetes-Master:~$ kubectl logs -n jenkins -c jnlp pod/worker-c82ea4bd-52e1-47c6-bad7-4a416a1e6897-z1bn0-t57rw
Error from server (BadRequest): container "jnlp" in pod "worker-c82ea4bd-52e1-47c6-bad7-4a416a1e6897-z1bn0-t57rw" is waiting to start: PodInitializing
fabiansc@Kubernetes-Master:~$ kubectl logs -n jenkins -c jnlp pod/worker-c82ea4bd-52e1-47c6-bad7-4a416a1e6897-z1bn0-t57rw
Jan 08, 2021 4:18:07 PM hudson.remoting.jnlp.Main createEngine
INFO: Setting up agent: worker-c82ea4bd-52e1-47c6-bad7-4a416a1e6897-z1bn0-t57rw
Jan 08, 2021 4:18:07 PM hudson.remoting.jnlp.Main$CuiListener <init>
INFO: Jenkins agent is running in headless mode.
Jan 08, 2021 4:18:07 PM hudson.remoting.Engine startEngine
INFO: Using Remoting version: 4.3
Jan 08, 2021 4:18:07 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using /home/jenkins/agent/remoting as a remoting work directory
Jan 08, 2021 4:18:07 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
INFO: Both error and output logs will be printed to /home/jenkins/agent/remoting
Jan 08, 2021 4:18:07 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among []
Jan 08, 2021 4:18:07 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Agent discovery successful
Agent address: jenkins.example.de
Agent port: 50000
Identity: cd:35:f9:1a:60:54:e4:91:07:86:59:49:0b:b6:73:c4
Jan 08, 2021 4:18:07 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Jan 08, 2021 4:18:07 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to jenkins.example.de:50000
fabiansc@Kubernetes-Master:~$ kubectl logs -f -n jenkins -c jnlp pod/worker-c82ea4bd-52e1-47c6-bad7-4a416a1e6897-z1bn0-t57rw
Jan 08, 2021 4:18:07 PM hudson.remoting.jnlp.Main createEngine
INFO: Setting up agent: worker-c82ea4bd-52e1-47c6-bad7-4a416a1e6897-z1bn0-t57rw
Jan 08, 2021 4:18:07 PM hudson.remoting.jnlp.Main$CuiListener <init>
INFO: Jenkins agent is running in headless mode.
Jan 08, 2021 4:18:07 PM hudson.remoting.Engine startEngine
INFO: Using Remoting version: 4.3
Jan 08, 2021 4:18:07 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using /home/jenkins/agent/remoting as a remoting work directory
Jan 08, 2021 4:18:07 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
INFO: Both error and output logs will be printed to /home/jenkins/agent/remoting
Jan 08, 2021 4:18:07 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among []
Jan 08, 2021 4:18:07 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Agent discovery successful
Agent address: jenkins.example.de
Agent port: 50000
Identity: cd:35:f9:1a:60:54:e4:91:07:86:59:49:0b:b6:73:c4
Jan 08, 2021 4:18:07 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Jan 08, 2021 4:18:07 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to jenkins.example.de:50000
Jan 08, 2021 4:18:17 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to jenkins.example.de:50000 (retrying:2)
java.io.IOException: Failed to connect to jenkins.example.de:50000
at org.jenkinsci.remoting.engine.JnlpAgentEndpoint.open(JnlpAgentEndpoint.java:247)
at hudson.remoting.Engine.connectTcp(Engine.java:844)
at hudson.remoting.Engine.innerRun(Engine.java:722)
at hudson.remoting.Engine.run(Engine.java:518)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:454)
at sun.nio.ch.Net.connect(Net.java:446)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:645)
at java.nio.channels.SocketChannel.open(SocketChannel.java:189)
at org.jenkinsci.remoting.engine.JnlpAgentEndpoint.open(JnlpAgentEndpoint.java:205)
... 3 more
Jan 08, 2021 4:18:17 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP4-connect
Jan 08, 2021 4:18:18 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Protocol JNLP4-connect encountered an unexpected exception
java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Connection closed before acknowledgement sent
at org.jenkinsci.remoting.util.SettableFuture.get(SettableFuture.java:223)
at hudson.remoting.Engine.innerRun(Engine.java:743)
at hudson.remoting.Engine.run(Engine.java:518)
Caused by: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Connection closed before acknowledgement sent
at org.jenkinsci.remoting.protocol.impl.AckFilterLayer.onRecvClosed(AckFilterLayer.java:283)
at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:816)
at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154)
at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer.access$1500(BIONetworkLayer.java:48)
at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader.run(BIONetworkLayer.java:247)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:117)
at java.lang.Thread.run(Thread.java:748)
Jan 08, 2021 4:18:18 PM hudson.remoting.jnlp.Main$CuiListener error
SEVERE: The server rejected the connection: None of the protocols were accepted
java.lang.Exception: The server rejected the connection: None of the protocols were accepted
at hudson.remoting.Engine.onConnectionRejected(Engine.java:828)
at hudson.remoting.Engine.innerRun(Engine.java:768)
at hudson.remoting.Engine.run(Engine.java:518)
INFO: Connecting to jenkins.example.de:50000 (retrying:2)
java.io.IOException: Failed to connect to jenkins.example.de:50000
at org.jenkinsci.remoting.engine.JnlpAgentEndpoint.open(JnlpAgentEndpoint.java:247)
at hudson.remoting.Engine.connectTcp(Engine.java:844)
at hudson.remoting.Engine.innerRun(Engine.java:722)
at hudson.remoting.Engine.run(Engine.java:518)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:454)
at sun.nio.ch.Net.connect(Net.java:446)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:645)
at java.nio.channels.SocketChannel.open(SocketChannel.java:189)
at org.jenkinsci.remoting.engine.JnlpAgentEndpoint.open(JnlpAgentEndpoint.java:205)
... 3 more
Jan 08, 2021 4:18:17 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP4-connect
Jan 08, 2021 4:18:18 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Protocol JNLP4-connect encountered an unexpected exception
java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Connection closed before acknowledgement sent
at org.jenkinsci.remoting.util.SettableFuture.get(SettableFuture.java:223)
at hudson.remoting.Engine.innerRun(Engine.java:743)
at hudson.remoting.Engine.run(Engine.java:518)
Caused by: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Connection closed before acknowledgement sent
at org.jenkinsci.remoting.protocol.impl.AckFilterLayer.onRecvClosed(AckFilterLayer.java:283)
at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:816)
at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154)
at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer.access$1500(BIONetworkLayer.java:48)
at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader.run(BIONetworkLayer.java:247)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:117)
at java.lang.Thread.run(Thread.java:748)
Jan 08, 2021 4:18:18 PM hudson.remoting.jnlp.Main$CuiListener error
SEVERE: The server rejected the connection: None of the protocols were accepted
java.lang.Exception: The server rejected the connection: None of the protocols were accepted
at hudson.remoting.Engine.onConnectionRejected(Engine.java:828)
at hudson.remoting.Engine.innerRun(Engine.java:768)
at hudson.remoting.Engine.run(Engine.java:518)
Found the answer. Istio
was delaying connectivity of JNLP
. Details on Github Issue #146. Further, Jenkins URL
and Jenkins Tunnel
must be configured (otherwise it fails, see Github Issue #788):
Two solutions:
Istio
JNPLP
image, utilizing delay / retry (graceful degradation). None is provided since February 2020.