K
Q

Docker Jenkins Worker Nodes missing docker group

3/27/2019

Using the helm chart: https://github.com/helm/charts/tree/master/stable/jenkins

Previously I had everything working with this chart on my k8s cluster. I recently re-installed the chart using the same persistent storage volume. I only changed the resource requests for the Agents in the values.yaml file. Now when I try to run a deployment through the pipeline, the master pod fails to connect to the worker pods that it is creating. I have verified there is space available on the Nodes, and they seem to get added without a problem. Just the startup for the Worker pods looks like it is having some issues. Most notably with the missing docker group. Not sure how I am suppose to configure that to create the group, or why this is an issue in the first place?

What am I doing wrong to get this deployment pipeline working again?

overriding values.yaml for the chart:

rbac:
  install: true

Master:
  HostName: jenkins.grapenuts.flytedesk.com
  ServiceType: ClusterIP
  NodePort:
  Ingress:
    Annotations:
      kubernetes.io/ingress.class: traefik
  JavaOpts: "-Dorg.jenkinsci.plugins.durabletask.BourneShellScript.HEARTBEAT_CHECK_INTERVAL=1000"
  resources:
    requests:
      cpu: "500m"
      memory: "512Mi"
    limits:
      cpu: "2000m"
      memory: "2048Mi"
  InstallPlugins:
    - amazon-ecr:1.6
    - blueocean:1.7.1
    - kubernetes:1.12.1
    - kubernetes-cli:1.2.0
    - workflow-aggregator:2.5
    - workflow-job:2.23
    - credentials-binding:1.16
    - git:3.9.1
    - slack:2.3

Agent:
  resources:
    requests:
      cpu: "200m"
      memory: "256Mi"
    limits:
      cpu: "1000m"
      memory: "4096Mi"

Persistence:
  Enabled: true
  Size: 8Gi
  ExistingClaim: jenkins-master-pvc

Logs from the Docker Worker Node created by the Jenkins Master node

WARN[2019-03-27T01:20:30.099307297Z] could not change group /var/run/docker.sock to docker: group docker not found 
 WARN[2019-03-27T01:20:30.099429310Z] [!] DON'T BIND ON ANY IP ADDRESS WITHOUT setting --tlsverify IF YOU DON'T KNOW WHAT YOU'RE DOING [!] 
 INFO[2019-03-27T01:20:30.111486018Z] libcontainerd: started new containerd process  pid=17
 INFO[2019-03-27T01:20:30.111603708Z] parsed scheme: "unix"                         module=grpc
 INFO[2019-03-27T01:20:30.111622892Z] scheme "unix" not registered, fallback to default scheme  module=grpc
 INFO[2019-03-27T01:20:30.117760193Z] ccResolverWrapper: sending new addresses to cc: [{unix:///var/run/docker/containerd/containerd.sock 0  <nil>}]  module=grpc
 INFO[2019-03-27T01:20:30.117825438Z] ClientConn switching balancer to "pick_first"  module=grpc
 INFO[2019-03-27T01:20:30.117931427Z] pickfirstBalancer: HandleSubConnStateChange: 0xc420740170, CONNECTING  module=grpc
 INFO[2019-03-27T01:20:30.700526951Z] starting containerd                           revision=e6b3f5632f50dbc4e9cb6288d911bf4f5e95b18e version=v1.2.4
 INFO[2019-03-27T01:20:30.701746137Z] loading plugin "io.containerd.content.v1.content"...  type=io.containerd.content.v1
 INFO[2019-03-27T01:20:30.701897179Z] loading plugin "io.containerd.snapshotter.v1.btrfs"...  type=io.containerd.snapshotter.v1
 WARN[2019-03-27T01:20:30.702171413Z] failed to load plugin io.containerd.snapshotter.v1.btrfs  error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter"
 INFO[2019-03-27T01:20:30.702215021Z] loading plugin "io.containerd.snapshotter.v1.aufs"...  type=io.containerd.snapshotter.v1
 WARN[2019-03-27T01:20:30.712326057Z] failed to load plugin io.containerd.snapshotter.v1.aufs  error="modprobe aufs failed: "ip: can't find device 'aufs'\nmodprobe: can't change directory to '/lib/modules': No such file or directory\n": exit status 1"
 INFO[2019-03-27T01:20:30.712352225Z] loading plugin "io.containerd.snapshotter.v1.native"...  type=io.containerd.snapshotter.v1
 INFO[2019-03-27T01:20:30.712464182Z] loading plugin "io.containerd.snapshotter.v1.overlayfs"...  type=io.containerd.snapshotter.v1
 INFO[2019-03-27T01:20:30.712635973Z] loading plugin "io.containerd.snapshotter.v1.zfs"...  type=io.containerd.snapshotter.v1
 WARN[2019-03-27T01:20:30.716216413Z] failed to load plugin io.containerd.snapshotter.v1.zfs  error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter"
 INFO[2019-03-27T01:20:30.716242462Z] loading plugin "io.containerd.metadata.v1.bolt"...  type=io.containerd.metadata.v1
INFO[2019-03-27T01:20:30.766699317Z] scheme "unix" not registered, fallback to default scheme  module=grpc
 INFO[2019-03-27T01:20:30.768163376Z] ccResolverWrapper: sending new addresses to cc: [{unix:///var/run/docker/containerd/containerd.sock 0  <nil>}]  module=grpc
 INFO[2019-03-27T01:20:30.768190514Z] ClientConn switching balancer to "pick_first"  module=grpc
 INFO[2019-03-27T01:20:30.768365344Z] ccResolverWrapper: sending new addresses to cc: [{unix:///var/run/docker/containerd/containerd.sock 0  <nil>}]  module=grpc
 INFO[2019-03-27T01:20:30.768639743Z] ClientConn switching balancer to "pick_first"  module=grpc
 INFO[2019-03-27T01:20:30.768644457Z] pickfirstBalancer: HandleSubConnStateChange: 0xc420740a90, CONNECTING  module=grpc
 INFO[2019-03-27T01:20:30.768760912Z] pickfirstBalancer: HandleSubConnStateChange: 0xc4207f3450, CONNECTING  module=grpc
 INFO[2019-03-27T01:20:30.769170856Z] pickfirstBalancer: HandleSubConnStateChange: 0xc420740a90, READY  module=grpc
 INFO[2019-03-27T01:20:30.769869637Z] pickfirstBalancer: HandleSubConnStateChange: 0xc4207f3450, READY  module=grpc
 INFO[2019-03-27T01:20:30.822166956Z] Graph migration to content-addressability took 0.00 seconds 
 INFO[2019-03-27T01:20:30.825122094Z] Loading containers: start.                   
 WARN[2019-03-27T01:20:30.854711776Z] Running modprobe bridge br_netfilter failed with message: ip: can't find device 'bridge'
 bridge                172032  1 br_netfilter
 stp                    16384  1 bridge
 llc                    16384  2 bridge,stp
 ipv6                  544768 497 bridge,[permanent]
 ip: can't find device 'br_netfilter'
 br_netfilter           24576  0 
 bridge                172032  1 br_netfilter
 modprobe: can't change directory to '/lib/modules': No such file or directory
 , error: exit status 1 
 WARN[2019-03-27T01:20:30.887386825Z] Running modprobe nf_nat failed with message: `ip: can't find device 'nf_nat'
 nf_nat_masquerade_ipv4    16384  1 ipt_MASQUERADE
 nf_nat_ipv4            16384  1 iptable_nat
 nf_nat                 36864  3 xt_nat,nf_nat_masquerade_ipv4,nf_nat_ipv4
 nf_conntrack          151552  8 xt_nat,ipt_MASQUERADE,nf_nat_masquerade_ipv4,nf_conntrack_netlink,nf_conntrack_ipv4,nf_nat_ipv4,xt_conntrack,nf_nat
 libcrc32c              16384  3 nf_nat,nf_conntrack,xfs
 modprobe: can't change directory to '/lib/modules': No such file or directory`, error: exit status 1 
 WARN[2019-03-27T01:20:30.895238719Z] Running modprobe xt_conntrack failed with message: `ip: can't find device 'xt_conntrack'
 xt_conntrack           16384  2 
 nf_conntrack          151552  8 xt_nat,ipt_MASQUERADE,nf_nat_masquerade_ipv4,nf_conntrack_netlink,nf_conntrack_ipv4,nf_nat_ipv4,xt_conntrack,nf_nat
 x_tables               45056 12 xt_statistic,xt_nat,xt_recent,ipt_REJECT,xt_tcpudp,xt_comment,xt_mark,ipt_MASQUERADE,xt_addrtype,iptable_filter,xt_conntrack,ip_tables
 modprobe: can't change directory to '/lib/modules': No such file or directory`, error: exit status 1 
 INFO[2019-03-27T01:20:31.093705399Z] Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address 
 INFO[2019-03-27T01:20:31.332494859Z] Loading containers: done.                    
 INFO[2019-03-27T01:20:31.470554395Z] Docker daemon                                 commit=774a1f4 graphdriver(s)=overlay2 version=18.09.3
 INFO[2019-03-27T01:20:31.470683993Z] Daemon has completed initialization          
 INFO[2019-03-27T01:20:31.485026627Z] API listen on [::]:2375                      
 INFO[2019-03-27T01:20:31.485180400Z] API listen on /var/run/docker.sock
-- newms87
docker
jenkins
kubernetes-helm

Similar Questions

helm upgrade error "Error: This command needs 2 arguments: release name, chart path"
kubernetes go-client PersistentVolumeClaim not provisioned on request, stuck in Pending state
Kubernetes pod can't access other pods exposed by a service
How to access a container port of another container using kubernetes-plugin in pipeline
Openshift route redirects to cluster console
Prometheus context deadline exceeded
CoreDNS pods stuck in ContainerCreating - Kubernetes

1 Answer

3/27/2019

I think that the problem would be that user jenkins cannot run docker commands.

  1. for the log " could not change group /var/run/docker.sock to docker: group docker not found ", according to https://access.redhat.com/solutions/3745881. It's informative messages and not an error.

  2. Create the docker group. You may need to add jenkins user to docker group. $ sudo groupadd docker Add your user to the docker group. $ sudo usermod -aG docker $USER https://docs.docker.com/install/linux/linux-postinstall/#manage-docker-as-a-non-root-user

In https://github.com/helm/charts/tree/master/stable/jenkins, the default settings of this helm chart let Jenkins run as root user with uid 0. In other cases, Jenkins is run with user jenkins, uid = 1000

You may provide more errors that you meet ?

-- paco alcacer
Source: StackOverflow