I am using a flask microservice with gunicorn to make requests to a service hosted on Google Kubernetes engine. The microservice is dockerised and is hosted as a pod on google kubernetes engine as well. After testing it locally, I deployed it but I am getting a CrashLoopBackOff error. The logs of my pod are :
[2019-03-15 08:41:13 +0000] [1] [INFO] Starting gunicorn 19.9.0
[2019-03-15 08:41:13 +0000] [1] [INFO] Listening at: http://0.0.0.0:8080 (1)
[2019-03-15 08:41:13 +0000] [1] [INFO] Using worker: threads
[2019-03-15 08:41:13 +0000] [12] [INFO] Booting worker with pid: 12
Failed to find application object 'app' in 'app'
[2019-03-15 08:41:13 +0000] [12] [INFO] Worker exiting (pid: 12)
[2019-03-15 08:41:13 +0000] [13] [INFO] Booting worker with pid: 13
Failed to find application object 'app' in 'app'
[2019-03-15 08:41:13 +0000] [13] [INFO] Worker exiting (pid: 13)
[2019-03-15 08:41:13 +0000] [1] [INFO] Shutting down: Master
[2019-03-15 08:41:13 +0000] [1] [INFO] Reason: App failed to load.
This seems to be an error with gunicorn.
My folder structure is :
.
├── app.py
├── app.yaml
├── config.py
├── data
│ └── object_detection.pbtxt
├── Dockerfile
├── filename.jpg
├── helper.py
├── object_2.py
├── object_detection
│ ├── core
│ │ ├── anchor_generator.py
│ └── vrd_evaluation_test.py
├── object_detection_grpc_client.py
├── requirements.txt
└── tensorflow_serving
├── apis
└── regression_pb2.py
The app.py code is:
import logging
from flask import Flask, request, jsonify
from config import auth_secret_token, PORT, DEBUG_MODE
from helper import check_parameters
from object_detection_grpc_client import main
app = Flask(__name__)
def check_authorization(request):
try:
if not 'Auth-token' in request.headers:
return jsonify({'error': 'unauthorized access'}), 401
token = request.headers['Auth-token']
if token != auth_secret_token:
return jsonify({'error': 'unauthorized access'}), 401
return "ok", 200
except Exception as e:
return jsonify({'error': 'unauthorized access'}), 401
@app.route("/", methods=['POST'])
def hello():
info, status_code = check_authorization(request)
if status_code != 200:
return info, status_code
else:
status, status_code = check_parameters(request.form)
if status_code != 200:
return status, status_code
else:
score = main()
response = {"status": "success", "score": score, "customer_id":(request.form["cust_id"])}
return jsonify(response), status_code
if __name__ == "__main__":
app.run(host="0.0.0.0", port=PORT, debug=DEBUG_MODE)
The config.py code is:
from os import environ as env
import multiprocessing
PORT = int(env.get("PORT", 8080))
DEBUG_MODE = int(env.get("DEBUG_MODE", 1))
# Gunicorn config
bind = ":" + str(PORT)
workers = multiprocessing.cpu_count() * 2 + 1
threads = 2 * multiprocessing.cpu_count()
auth_secret_token = "token"
server='A.B.C.D:9000'
model_name="mymodel"
input_image='filename.jpg'
label_map="./data/object_detection.pbtxt"
The Dockerfile is :
FROM python:3.5.2
RUN apt update
WORKDIR /app
ADD requirements.txt /app/requirements.txt
RUN pip install -r /app/requirements.txt
ADD . /app
ENV PORT 8080
CMD ["gunicorn", "app:app", "--config=config.py"]
The deployment file app.yaml:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: mymodel-client
labels:
name: mymodel-client
spec:
replicas: 1
selector:
matchLabels:
name: mymodel-client
template:
metadata:
name: mymodel-client
labels:
name: mymodel-client
spec:
containers:
- name: mymodel-client
image: gcr.io/image-identification/mymodel-client:v1
ports:
- containerPort: 8080
resources:
requests:
memory: 256Mi
limits:
memory: 512Mi
env:
- name: DEBUG_MODE
value: "1"
What is wrong here?
Link to the tutorial I referred to https://medium.com/google-cloud/a-guide-to-deploy-flask-app-on-google-kubernetes-engine-bfbbee5c6fb
It was most likely a memory issue. I deleted the deployment and pushed it again with more resources allocated to it and it started running without any errors.
I think the problem might be similar to this one. https://stackoverflow.com/a/50157417/4229159
The app
folder (the one you create in docker) and the app
file have the same name. Could you try it again renaming one of them?
The rest of the files look ok, and it seems only to be a problem of gunicorn