Airflow webserver not starting while using helm chart on minikube

9/12/2019

I'm trying to run Airflow locally (to test it before deployment) using minikube and helm chart stable/airflow. But airflow-webserver doesn't start due to gunicorn issue.

Helm: v2.14.3

Kubernetes: v1.15.2

Minikube: v1.3.1

Helm chart image: puckel/docker-airflow

These are the steps:

minikube start

helm install --namespace "airflow" --name "airflow" stable/airflow

Logs are:

Thu Sep 12 07:29:54 UTC 2019 - waiting for Postgres... 1/20
Thu Sep 12 07:30:00 UTC 2019 - waiting for Postgres... 2/20
waiting 60s...
executing webserver...
[2019-09-12 07:31:05,745] {{settings.py:213}} INFO - settings.configure_orm(): Using pool settings. pool_size=5, max_overflow=10, pool_recycle=1800, pid=1
/usr/local/lib/python3.7/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
[2019-09-12 07:31:06,030] {{__init__.py:51}} INFO - Using executor CeleryExecutor
  ____________       _____________
 ____    |__( )_________  __/__  /________      __
____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
 _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
[2019-09-12 07:31:06,585] {{dagbag.py:90}} INFO - Filling up the DagBag from /usr/local/airflow/dags
Running the Gunicorn Server with:
Workers: 4 sync
Host: 0.0.0.0:8080
Timeout: 120
Logfiles: - -
=================================================================            
[2019-09-12 07:31:07,676] {{settings.py:213}} INFO - settings.configure_orm(): Using pool settings. pool_size=5, max_overflow=10, pool_recycle=1800, pid=21
/usr/local/lib/python3.7/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
[2019-09-12 07:31:07 +0000] [21] [INFO] Starting gunicorn 19.9.0
[2019-09-12 07:31:07 +0000] [21] [INFO] Listening at: http://0.0.0.0:8080 (21)
[2019-09-12 07:31:07 +0000] [21] [INFO] Using worker: sync
[2019-09-12 07:31:07 +0000] [25] [INFO] Booting worker with pid: 25
[2019-09-12 07:31:07 +0000] [26] [INFO] Booting worker with pid: 26
[2019-09-12 07:31:07 +0000] [27] [INFO] Booting worker with pid: 27
[2019-09-12 07:31:07 +0000] [28] [INFO] Booting worker with pid: 28
[2019-09-12 07:31:08,444] {{__init__.py:51}} INFO - Using executor CeleryExecutor
[2019-09-12 07:31:08,446] {{__init__.py:51}} INFO - Using executor CeleryExecutor
[2019-09-12 07:31:08,545] {{__init__.py:51}} INFO - Using executor CeleryExecutor
[2019-09-12 07:31:08,669] {{__init__.py:51}} INFO - Using executor CeleryExecutor
[2019-09-12 07:31:10,047] {{dagbag.py:90}} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2019-09-12 07:31:20,932] {{cli.py:825}} ERROR - [0 / 0] some workers seem to have died and gunicorndid not restart them as expected
[2019-09-12 07:31:22,095] {{dagbag.py:90}} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2019-09-12 07:31:22 +0000] [25] [INFO] Parent changed, shutting down: <Worker 25>
[2019-09-12 07:31:22 +0000] [25] [INFO] Worker exiting (pid: 25)
[2019-09-12 07:31:32 +0000] [28] [INFO] Parent changed, shutting down: <Worker 28>
[2019-09-12 07:31:32 +0000] [28] [INFO] Worker exiting (pid: 28)
[2019-09-12 07:31:33,289] {{dagbag.py:90}} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2019-09-12 07:31:33,324] {{dagbag.py:90}} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2019-09-12 07:31:35 +0000] [26] [INFO] Parent changed, shutting down: <Worker 26>
[2019-09-12 07:31:35 +0000] [26] [INFO] Worker exiting (pid: 26)
[2019-09-12 07:31:35 +0000] [27] [INFO] Parent changed, shutting down: <Worker 27>
[2019-09-12 07:31:35 +0000] [27] [INFO] Worker exiting (pid: 27)
[2019-09-12 07:33:32,017] {{cli.py:832}} ERROR - No response from gunicorn master within 120 seconds
[2019-09-12 07:33:32,018] {{cli.py:833}} ERROR - Shutting down webserver

I can run that docker image locally with docker-compose with no issues, but no luck using helm, it fails and restarts constantly.

-- frm
airflow
gunicorn
kubernetes-helm
minikube

1 Answer

9/13/2019

Turns out that the issue was that the minikube configuration wasn't making postgres' pod available, editing the pod deployment with the ip of the postgres instance it worked.

-- frm
Source: StackOverflow