Install Jenkins using Helm in Kubernetes (kubeadm)

4/18/2018

First of all, I have Kubernetes cluster behind the proxy environment.

I have three servers, master, node1, and node2.

I installed Jenkins using below command.

Create jenkine-project namespace and then

helm install --name jenkins -f jenkins-values.yaml stable/jenkins --namespace jenkins-project

jenkins-values.yaml is

Master:
  Name: jenkins-master
  Image: "jenkins/jenkins"
  ImageTag: "lts"
  ImagePullPolicy: "Always"
# ImagePullSecret: jenkins
  Component: "jenkins-master"
  UseSecurity: true
  AdminUser: admin
  AdminPassword: 1qaz2wsx
  Cpu: "200m"
  Memory: "512Mi"
  # Environment variables that get added to the init container (useful for e.g. http_proxy)
  InitContainerEnv:
    - name: http_proxy
      value: "http://168.219.yyy.zzz:8080"
    - name: https_proxy
      value: "http://168.219.yyy.zzz:8080"
    - name: no_proxy
      value: "localhost,127.0.0.1,10.251.141.*,"
  ContainerEnv:
    - name: http_proxy
      value: "http://168.219.yyy.zzz:8080"
    - name: https_proxy
      value: "http://168.219.yyy.zzz:8080"
  JavaOpts: >-
    -Dhttp.proxyHost=168.219.yyy.zzz
    -Dhttp.proxyPort=8080
    -Dhttps.proxyHost=168.219.yyy.zzz
    -Dhttps.proxyPort=8080
  # Set min/max heap here if needed with:
  # JavaOpts: "-Xms512m -Xmx512m"
  # JenkinsOpts: ""
  # JenkinsUriPrefix: "/jenkins"
  # Set RunAsUser to 1000 to let Jenkins run as non-root user 'jenkins' which exists in 'jenkins/jenkins' docker image.
  # When setting RunAsUser to a different value than 0 also set FsGroup to the same value:
  # RunAsUser: <defaults to 0>
  # FsGroup: <will be omitted in deployment if RunAsUser is 0>
  ServicePort: 8080
  # For minikube, set this to NodePort, elsewhere use LoadBalancer
  # Use ClusterIP if your setup includes ingress controller
  ServiceType: LoadBalancer
  # Master Service annotations
  ServiceAnnotations: {}
  #   service.beta.kubernetes.io/aws-load-balancer-backend-protocol: https
  # Used to create Ingress record (should used with ServiceType: ClusterIP)
  # HostName: jenkins.cluster.local
  # NodePort: <to set explicitly, choose port between 30000-32767
  ContainerPort: 8080
  # Enable Kubernetes Liveness and Readiness Probes
  HealthProbes: true
  HealthProbesTimeout: 60
  SlaveListenerPort: 50000
  # Kubernetes service type for the JNLP slave service
  # SETTING THIS TO "LoadBalancer" IS A HUGE SECURITY RISK: https://github.com/kubernetes/charts/issues/1341
  SlaveListenerServiceType: ClusterIP
  SlaveListenerServiceAnnotations: {}
  LoadBalancerSourceRanges:
  - 0.0.0.0/0
  # Optionally assign a known public LB IP
  # LoadBalancerIP: 1.2.3.4
  # Optionally configure a JMX port
  # requires additional JavaOpts, ie
  # JavaOpts: >
  #   -Dcom.sun.management.jmxremote.port=4000
  #   -Dcom.sun.management.jmxremote.authenticate=false
  #   -Dcom.sun.management.jmxremote.ssl=false
  # JMXPort: 4000
  # List of plugins to be install during Jenkins master start
  InstallPlugins:
    - kubernetes:1.4
    - workflow-aggregator:2.5
    - workflow-job:2.17
    - credentials-binding:1.16
    - p4:1.8.7
    - blueocean:1.4.2
  # Used to approve a list of groovy functions in pipelines used the script-security plugin. Can be viewed under /scriptApproval
  ScriptApproval:
    - "method groovy.json.JsonSlurperClassic parseText java.lang.String"
    - "new groovy.json.JsonSlurperClassic"
    - "staticMethod org.codehaus.groovy.runtime.DefaultGroovyMethods leftShift java.util.Map java.util.Map"
    - "staticMethod org.codehaus.groovy.runtime.DefaultGroovyMethods split java.lang.String"
  # List of groovy init scripts to be executed during Jenkins master start
  InitScripts:
  #  - |
  #    print 'adding global pipeline libraries, register properties, bootstrap jobs...'
  # Kubernetes secret that contains a 'credentials.xml' for Jenkins
  # CredentialsXmlSecret: jenkins-credentials
  # Kubernetes secret that contains files to be put in the Jenkins 'secrets' directory,
  # useful to manage encryption keys used for credentials.xml for instance (such as
  # master.key and hudson.util.Secret)
  # SecretsFilesSecret: jenkins-secrets
  # Jenkins XML job configs to provision
  # Jobs: |-
  #   test: |-
  #     <<xml here>>
  CustomConfigMap: false
  # Node labels and tolerations for pod assignment
  # ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector
  # ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#taints-and-tolerations-beta-feature
  NodeSelector: {}
  Tolerations: {}

  Ingress:
    Annotations:
#     kubernetes.io/ingress.class: nginx
    # kubernetes.io/tls-acme: "true"

    TLS:
    # - secretName: jenkins.cluster.local
    #   hosts:
    #     - jenkins.cluster.local

Agent:
  Enabled: true
  Image: jenkins/jnlp-slave
  ImageTag: 3.10-1
# ImagePullSecret: jenkins
  Component: "jenkins-slave"
  Privileged: false
  Cpu: "200m"
  Memory: "256Mi"
  # You may want to change this to true while testing a new image
  AlwaysPullImage: false
  # You can define the volumes that you want to mount for this container
  # Allowed types are: ConfigMap, EmptyDir, HostPath, Nfs, Pod, Secret
  # Configure the attributes as they appear in the corresponding Java class for that type
  # https://github.com/jenkinsci/kubernetes-plugin/tree/master/src/main/java/org/csanchez/jenkins/plugins/kubernetes/volumes
  volumes:
   - type: HostPath
     secretName: /var/run/docker.sock
     mountPath: /var/run/docker.sock
  NodeSelector: {}
  # Key Value selectors. Ex:
  # jenkins-agent: v1

Persistence:
  Enabled: true
  ## A manually managed Persistent Volume and Claim
  ## Requires Persistence.Enabled: true
  ## If defined, PVC must be created manually before volume will be bound
  # ExistingClaim: pvc-jenkins-master

  ## jenkins data Persistent Volume Storage Class
  ## If defined, storageClassName: <storageClass>
  ## If set to "-", storageClassName: "", which disables dynamic provisioning
  ## If undefined (the default) or set to null, no storageClassName spec is
  ##   set, choosing the default provisioner.  (gp2 on AWS, standard on
  ##   GKE, AWS & OpenStack)
  ##
  StorageClass: "jenkins-pv"

  Annotations: {}
  AccessMode: ReadWriteOnce
  Size: 20Gi
  volumes:
  #  - name: nothing
  #    emptyDir: {}
  mounts:
  #  - mountPath: /var/nothing
  #    name: nothing
  #    readOnly: true

NetworkPolicy:
  # Enable creation of NetworkPolicy resources.
  Enabled: false
  # For Kubernetes v1.4, v1.5 and v1.6, use 'extensions/v1beta1'
  # For Kubernetes v1.7, use 'networking.k8s.io/v1'
  ApiVersion: networking.k8s.io/v1

## Install Default RBAC roles and bindings
rbac:
  install: true
  serviceAccountName: default
  # RBAC api version (currently either v1beta1 or v1alpha1)
  apiVersion: v1beta1
  # Cluster role reference
  roleRef: cluster-admin

And pod jenkins-694674f4bd-zqfpq is created.

I run kubectl logs jenkins-694674f4bd-zqfpq -n jenkins-project command here is problem.

# kubectl logs jenkins-694674f4bd-zqfpq -n jenkins-project
Error from server: Get https://10.251.141.74:10250/containerLogs/jenkins-project/jenkins-694674f4bd-zqfpq/jenkins: read tcp 10.251.141.xxx:34630->168.219.yyy.zzz:8080: read: connection reset by peer

In this error message, 10.251.141.xxx is the master server's IP address and 168.219.yyy.zzz:8080 is proxy address.

And (I guess) because of this problem plugin will not be installed normally.

What is the problem and how can I fix this?

-- hokwang
jenkins
kubeadm
kubernetes
kubernetes-helm
proxy

1 Answer

4/18/2018

As I understood, you have a cluster behind a proxy, so it looks like that:

You | Proxy| All Kubernetes nodes and master

When you call kubectl logs command, kubectl is connecting to the API server and then the API server is getting logs of your pod from the node.

As I see from the output of the command, the API server is trying to connect to the node through the proxy server instead of direct connection, that's why I think you have a bit incorrect setup of proxy settings on your master.

Try to add all cluster internal IP ranges to exception using no_proxy on the master and on nodes, I think it should help.

-- Anton Kostenko
Source: StackOverflow