Kubernetes sub-domain provisioning

4/18/2020

I'm working on a SaaS app that will be running in Kubernetes. We're using a Helm chart that deploys all the components into the cluster (for simplicity's sake let's assume it's a frontend service, a backend and a database). App architecture is multi-tenant (we have a single instance of each service that are being shared by all tenants) and we would like to keep it that way. What I'm currently struggling with and would like to ask for advice/best practice on is how does one go about automating the provisioning of custom sub-domains for the tenants?

Imagine the app is hosted at exampleapp.com. A brand new customer comes and registers a new organisation some-company. At that moment, in addition to creating new tenant in the system, I would also like to provision a new subdomain some-company.exampleapp.com. I would like this provisioning to be done automatically and not require any manual intervention.

  • What options do I have for implementing automated sub-domain provisioning in Kubernetes?
  • How does our (exampleapp.com) domain registrar/nameserver provider fit into the solution? Does it have to provide an API for dynamic DNS record creation/modification?

I appreciate that the questions I'm asking are quite broad so I'm not expecting anything more than a high-level conceptual answer or pointers to some services/libraries/tools that might help me achieve this.

-- IvanR
dns
kubernetes
saas
subdomain

1 Answer

4/21/2020

Note: Since this is more of a theoretical question, I'll give you some points from a Kubernetes Engineer, I divided your question in blocks to ease the understanding.

  • About your multi-tenancy architecture: Keeping "it that way" is achievable. It simplifies the Kubernetes structure, on the other hand it relies more on your app.

Question 1:

Imagine the app is hosted at exampleapp.com. A brand new customer comes and registers a new organisation some-company. At that moment, in addition to creating new tenant in the system, I would also like to provision a new subdomain some-company.exampleapp.com. I would like this provisioning to be done automatically and not require any manual intervention.

Suggestion:

  • For that, you will have to give your app admin privileges and the tools required for it to add Ingress Rules entries to your Ingress when a new client is added. A script using kubectl patch is the simpler solution from my viewpoint.

For this approach I suggest installing the Nginx Ingress Controller for it's versatility.

Here is an Example:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: <ingress-name>
spec:
  rules:
  - host: client1.exampleapp.com
    http:
      paths:
      - path: /client1
        backend:
          serviceName: <main-service>
          servicePort: <main-service-port>

  - host: client2.exampleapp.com
    http:
      paths:
      - path: /client2
        backend:
          serviceName: <main-service>
          servicePort: <main-service-port>
  • And here is the one-liner command using kubectl patch on how to add new rules:
kubectl patch ingress demo-ingress --type "json" -p '[{"op":"add","path":"/spec/rules/-","value":{"host":"client3.exampleapp.com","http":{"paths":[{"path":"/client3","backend":{"serviceName":"main-service","servicePort":80}}]}}}]'

POC:

$ kubectl get ingress
NAME           HOSTS                                           ADDRESS         PORTS   AGE
demo-ingress   client1.exampleapp.com,client2.exampleapp.com   192.168.39.39   80      15m

$ kubectl patch ingress demo-ingress --type "json" -p '[{"op":"add","path":"/spec/rules/-","value":{"host":"client3.exampleapp.com","http":{"paths":[{"path":"/client3","backend":{"serviceName":"main-service","servicePort":80}}]}}}]'
ingress.extensions/demo-ingress patched

$ kubectl describe ingress demo-ingress
Rules:
  Host                    Path  Backends
  ----                    ----  --------
  client1.exampleapp.com  
                          /client1   main-service:80 (<none>)
  client2.exampleapp.com  
                          /client2   main-service:80 (<none>)
  client3.exampleapp.com  
                          /client3   main-service:80 (<none>)

This rule redirects the traffic incoming from the subdomains to subpaths inside your main app.


Question2 :

How does our (exampleapp.com) domain registrar/nameserver provider fit into the solution? Does it have to provide an API for dynamic DNS record creation/modification?

Suggestion:

  • I believe you already has something similar, but you need a wildcard record in your DNS to point *.example.app to the IP of the ingress, I don't believe you need anything more than that, because it redirects all to the ingress and the ingress forwards it internally.

Question 3:

If there are some strong arguments why multi-tenancy + Kubernetes don't go along very well, those are welcome as well.

Opinion:

  • I don't see any major reason why it would be a problem. You just have, once again, to adequate your app to handle scaling because I believe in the long run you would want your app to be able to scale to multi-pod structure to provide elastic availability.

These are my 2 cents to your question, I hope it helps you!

-- willrof
Source: StackOverflow