Flask with Gunicorn on Kubernetes ingress yields 502 nginx error

11/1/2018

I have built a flask app that I would like to add to a Kubernetes ingress. Currently, I have 2 questions I cannot seem to get my head around:

  1. In order for the flask app to be able to handle several requests, I figured I would add gunicorn. Do I need this, or can I mitigate this by using some kind of automatic horizontal scaling and the ingress routing layer handle it? I am new to Kubernetes, and perhaps the solution is simpler than what I am trying below.
  2. With the presumption that I do need gunicorn, I have proceeded and added it to the flask docker. The problem I have with this is that I now get a 502 Bad Gateway Error nginx and the log of the pod have not printed any error. If I create a load balancer service instead of the clusterIP I use with the ingress, the flask app with unicorn works fine, just as the flask app does on the ingress without adding gunicorn. I have no idea why hence writing this question. The dockerfile installs all dependencies to run flask and finishes with:

    EXPOSE 8080
    
    CMD ["gunicorn", "--config", "/flaskapp/gunicorn_config.py", "run:app"]

    I have configured my ingress like this:

    apiVersion: v1
    items:
    - apiVersion: extensions/v1beta1
      kind: Ingress
     metadata:
       annotations:
         ingress.bluemix.net/client-max-body-size: 128m
         ingress.bluemix.net/rewrite-path: serviceName=flask-service rewrite=/; 
    spec:
      rules:
      - host: <my-domain>
        http:
          paths:
          - backend:
            serviceName: flask-service
            servicePort: 8080
          path: /flask/
      tls:
      - hosts:
        - <my-domain>
        secretName: <my-secret>
    status:
      loadBalancer:
        ingress:
        - ip: <ip>

    The service looks like this:

    apiVersion: v1
    kind: Service
    metadata:
      name: flask-service
      labels:
        app: flask-service
    spec:
      type: ClusterIP
      ports:
      - port: 8080
        protocol: TCP
      selector:
        app: flask

    The deployment is also very simple specifying the correct image and port.

Given that I need gunicorn(or similar), how can I solve the 502 Bad Gateway Error I get?

-- jawwe
flask
gunicorn
ibm-cloud
kubernetes
kubernetes-ingress

1 Answer

11/1/2018
  1. IMO, you don't need gunicorn scaling (it's an overkill) since an HPA will do the scaling if your single application instances already. This depending on CPUs, memory or custom metrics.

  2. The 502 errors seem to me it's more of how gunicorn is configured issue (is there a limit on the workers? can you see the workers to just 1 to test? how is it scaling inside the container? What are the resource limits on the container?). Hard to tell without looking at logs or the environment, but it could be that you gunicorn workers are thrashing in the container thus returning an invalid response. You might want to try --log-level debug on the gunicorn command line.

Hope it helps.

-- Rico
Source: StackOverflow