K
Q

Setting up HTTP+HTTPs and TCP ports on a LoadBalancer (Want to forward GitLab SSH and Web)

7/6/2018

The setup I want is rather simple. I am hosting gitlab inside a kubernetes cluster and the ingress for it is defined as:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: gitlab-public
  namespace: gitlab
  annotations:
    kubernetes.io/ingress.global-static-ip-name: git-public
spec:
  tls:
    - secretName: gitlab-cert
  backend:
    serviceName: gitlab
    servicePort: 80

This sets up everything for me - SSL termination is enabled, it uses my letsencrypt certificates. The problem is that (clearly) SSH is not working for the repository. The ports for 22 are exposed by the pods and I have verified it:

Both SSH and HTTP ports are forwarded

However, it looks like Ingress does not support a unified L4+L7 specification. A few other options I have considered so far:

  1. Create another load balancer which only forwards 22 -> 22 as a TCP (or an L4 ingress) balancer. The problem is that multiple load balancers cannot share the same static IP.. which of course makes sense.
  2. Basically do the previous thing, but get a new DNS and static IP, probably call it ssh.git.mycompany.com and use it to forward SSH traffic. The huge problem is that there are a lot of developers using this right now and they will have to change their .git/config files and it is going to be extremely disruptive (not in a Silicon Valley way).
  3. Setup an L4 balancer (outside of any k8s config) and let it handle the SSL and setup three ports to be forwarded to 80 and 22. This is the most confusing step possible.. because I have no idea what the Google LoadBalancer is trying to do. I am not able to map frontends to backends, or even select my own port. Apparently, it has to be from a list of pre-defined ports, which is pretty dumb I guess? In contrast, we are currently hosted on AWS and this is how the load balancer looks:

enter image description here

What is the best way to get around this?

EDIT: Modifying the title since the answer is a bit GitLab specific so might help people looking for the exact problem.

-- Rohan Prabhu
google-cloud-platform
google-kubernetes-engine
kubernetes

Similar Questions

change the annotation for node-selector for openshift namespace
kubectl - cert manager - credentials not found
How to install a specific Chart version
Mounting k8 permanent volume fails silently
How do I update an on-premise Kubernetes API server to enable OIDC with Dex?
Gitlab AutoDeploy unable to deploy K8s cluster on GKE
kubernetes flannel network is not working
Error extracting container id - source value does not contain matcher's logs_path '/var/lib/docker/containers/'
Can't find replicaset kube state metrics in Prometheus

1 Answer

7/7/2018

So, I am not entirely sure if this works as an answer, because while it achieves everything we wanted to do; it does not support SSL termination for the load balancer.

So the basic compromise is this:

  1. Make GitLab serve SSL connections with the bundled nginx controller. NGINX Settings for GitLab
  2. Setup GitLab to forward HTTP -> HTTPs, standard stuff. Documentation covers most of it.
  3. When letting the LoadBalancer handle SSL, you just had to point to a secret and that would work (it happens to be in a specific format with keys tls.crt and tls.key, both in pem format). You can carry over the same structure by mounting these two keys as a file and then changing the nginx settings (ssl_certificate and ssl_certificate_key):

In your volumes section:

volumes: - name: gitlab-cert secret: secretName: items: - key: tls.key path: git.mycompany.com.key - key: tls.crt path: git.mycompany.com.crt

And mount it, in whichever location you want:

volumeMounts: - name: gitlab-certs mountPath: /etc/gitlab/ssl/

In the above case, this would make your gitlab.rb look like:

nginx['ssl_certificate'] = "/etc/gitlab/ssl/git.mycompany.com.crt"
nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/git.mycompany.com.key"

In case your domain on which you are hosting is actually git.mycompany.com, then you can skip this step because the location showed above is what it defaults to.

And then finally instead of using an Ingress, just expose your deployment:

apiVersion: v1
kind: Service
metadata:
  name: gitlab
  namespace: gitlab
spec:
  type: LoadBalancer
  loadBalancerIP: <your-static-ip>
  ports:
    - port: 80
      name: http
      targetPort: 80
      protocol: TCP
    - port: 443
      name: https
      targetPort: 443
      protocol: TCP
    - port: 22
      name: ssh
      targetPort: 22
      protocol: TCP
  selector:
    app: gitlab

The static ip for loadBalancerIP is something you need to get from your google console. Do note that this will not work for a global reserved IP. You have to get a regional IP in the same region as your cluster.

I would've still preferred SSL termination because that's how all of our other services work and the non-homogeneity is kind of annoying me a bit :(

-- Rohan Prabhu
Source: StackOverflow