K
Q

Airflow services are not getting started

8/28/2019

We are trying to deploy airflow (1.9.0) version which is using Postgres as its DB and Redis for connectivity between pods in Kubernetes.

The same setup is working fine in staging environment and getting failed in prod environment , upon investigating I came to know that Postgres base version for image 9.6 got changed recently , will it cause any issue?

attached is the log of postgress and WebServer(airflow)

Tried using different images of postures but still same

WebServer log

Collecting botocore
  Downloading https://files.pythonhosted.org/packages/a1/b0/7a8794d914b95ef3335a5a4ba20595b46081dbd1e29f13812eceacf091ca/botocore-1.12.215-py2.py3-none-any.whl (5.7MB)
Collecting docutils<0.16,>=0.10 (from botocore)
  Downloading https://files.pythonhosted.org/packages/22/cd/a6aa959dca619918ccb55023b4cb151949c64d4d5d55b3f4ffd7eee0c6e8/docutils-0.15.2-py3-none-any.whl (547kB)
Collecting jmespath<1.0.0,>=0.7.1 (from botocore)
  Downloading https://files.pythonhosted.org/packages/83/94/7179c3832a6d45b266ddb2aac329e101367fbdb11f425f13771d27f225bb/jmespath-0.9.4-py2.py3-none-any.whl
Requirement already satisfied: urllib3<1.26,>=1.20; python_version >= "3.4" in /usr/lib/python3/dist-packages (from botocore) (1.22)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1; python_version >= "2.7" in /usr/local/lib/python3.6/dist-packages (from botocore) (2.8.0)
Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil<3.0.0,>=2.1; python_version >= "2.7"->botocore) (1.11.0)
Installing collected packages: docutils, jmespath, botocore
Successfully installed botocore-1.12.215 docutils-0.15.2 jmespath-0.9.4
Multi-tenant details not configured in this instance - Exiting
Cluster "abc" set.
User "abc@airflow.com" set.
Context "abc" created.
Switched to context "cedp".
[2019-08-26 14:01:03,391] {{driver.py:124}} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2019-08-26 14:01:03,415] {{driver.py:124}} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 17, in <module>
    from airflow.bin.cli import CLIFactory
  File "/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py", line 47, in <module>
    from airflow import jobs, settings
  File "/usr/local/lib/python2.7/dist-packages/airflow/jobs.py", line 64, in <module>
    class BaseJob(Base, LoggingMixin):
  File "/usr/local/lib/python2.7/dist-packages/airflow/jobs.py", line 96, in BaseJob
    executor=executors.GetDefaultExecutor(),
  File "/usr/local/lib/python2.7/dist-packages/airflow/executors/__init__.py", line 42, in GetDefaultExecutor
    DEFAULT_EXECUTOR = _get_executor(executor_name)
  File "/usr/local/lib/python2.7/dist-packages/airflow/executors/__init__.py", line 60, in _get_executor
    from airflow.executors.celery_executor import CeleryExecutor
  File "/usr/local/lib/python2.7/dist-packages/airflow/executors/celery_executor.py", line 18, in <module>
    from celery import Celery
  File "/usr/local/lib/python2.7/dist-packages/celery/local.py", line 509, in __getattr__
    module = __import__(self._object_origins[name], None, None, [name])
  File "/usr/local/lib/python2.7/dist-packages/celery/app/__init__.py", line 5, in <module>
    from celery import _state
  File "/usr/local/lib/python2.7/dist-packages/celery/_state.py", line 17, in <module>
    from celery.utils.threads import LocalStack
  File "/usr/local/lib/python2.7/dist-packages/celery/utils/__init__.py", line 9, in <module>
    from .nodenames import worker_direct, nodename, nodesplit
  File "/usr/local/lib/python2.7/dist-packages/celery/utils/nodenames.py", line 9, in <module>
    from kombu.entity import Exchange, Queue
  File "/usr/local/lib/python2.7/dist-packages/kombu/entity.py", line 9, in <module>
    from .serialization import prepare_accept_content
  File "/usr/local/lib/python2.7/dist-packages/kombu/serialization.py", line 456, in <module>
    for ep, args in entrypoints('kombu.serializers'):  # pragma: no cover
  File "/usr/local/lib/python2.7/dist-packages/kombu/utils/compat.py", line 89, in entrypoints
    for ep in importlib_metadata.entry_points().get(namespace, [])
  File "/usr/local/lib/python2.7/dist-packages/importlib_metadata/__init__.py", line 456, in entry_points
    ordered = sorted(eps, key=by_group)
  File "/usr/local/lib/python2.7/dist-packages/importlib_metadata/__init__.py", line 454, in <genexpr>
    dist.entry_points for dist in distributions())
  File "/usr/local/lib/python2.7/dist-packages/importlib_metadata/__init__.py", line 364, in <genexpr>
    cls._search_path(path, pattern)
  File "/usr/local/lib/python2.7/dist-packages/importlib_metadata/__init__.py", line 373, in _switch_path
    return pathlib.Path(path)
  File "/usr/local/lib/python2.7/dist-packages/pathlib2/__init__.py", line 1256, in __new__
    self = cls._from_parts(args, init=False)
  File "/usr/local/lib/python2.7/dist-packages/pathlib2/__init__.py", line 898, in _from_parts
    drv, root, parts = self._parse_args(args)
  File "/usr/local/lib/python2.7/dist-packages/pathlib2/__init__.py", line 891, in _parse_args
    return cls._flavour.parse_parts(parts)
  File "/usr/local/lib/python2.7/dist-packages/pathlib2/__init__.py", line 250, in parse_parts
    parsed.append(intern(x))
TypeError: can't intern subclass of string
[2019-08-26 14:01:04,253] {{driver.py:124}} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2019-08-26 14:01:04,277] {{driver.py:124}} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
Traceback (most recent call last):
  File "/usr/local/airflow/set_auth.py", line 16, in <module>
    session.commit()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 927, in commit
    self.transaction.commit()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 467, in commit
    self._prepare_impl()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 447, in _prepare_impl
    self.session.flush()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2209, in flush
    self._flush(objects)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2329, in _flush
    transaction.rollback(_capture_exception=True)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2293, in _flush
    flush_context.execute()
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 389, in execute
    rec.execute(self)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 548, in execute
    uow
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 181, in save_obj
    mapper, table, insert)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 835, in _emit_insert_statements
    execute(statement, params)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 945, in execute
    return meth(self, multiparams, params)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 263, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1053, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1189, in _execute_context
    context)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1402, in _handle_dbapi_exception
    exc_info
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context
    context)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 470, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedTable) relation "users" does not exist
LINE 1: INSERT INTO users (username, email, password) VALUES ('admin...
                    ^
 [SQL: 'INSERT INTO users (username, email, password) VALUES (%(username)s, %(email)s, %(password)s) RETURNING users.id'] [parameters: {'username': 'admin', 'password': '$2b$12$F.8CTth9cL5G9f.pd180Duz/nC8S5KwTctwf/jG1Y/QB8PZagkTa.', 'email': 'abc@airflow.com'}]
[2019-08-26 14:01:15,229] {{driver.py:124}} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2019-08-26 14:01:15,253] {{driver.py:124}} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 17, in <module>
    from airflow.bin.cli import CLIFactory
  File "/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py", line 47, in <module>
    from airflow import jobs, settings
  File "/usr/local/lib/python2.7/dist-packages/airflow/jobs.py", line 64, in <module>
    class BaseJob(Base, LoggingMixin):
  File "/usr/local/lib/python2.7/dist-packages/airflow/jobs.py", line 96, in BaseJob
    executor=executors.GetDefaultExecutor(),
  File "/usr/local/lib/python2.7/dist-packages/airflow/executors/__init__.py", line 42, in GetDefaultExecutor
    DEFAULT_EXECUTOR = _get_executor(executor_name)
  File "/usr/local/lib/python2.7/dist-packages/airflow/executors/__init__.py", line 60, in _get_executor
    from airflow.executors.celery_executor import CeleryExecutor
  File "/usr/local/lib/python2.7/dist-packages/airflow/executors/celery_executor.py", line 18, in <module>
    from celery import Celery
  File "/usr/local/lib/python2.7/dist-packages/celery/local.py", line 509, in __getattr__
    module = __import__(self._object_origins[name], None, None, [name])
  File "/usr/local/lib/python2.7/dist-packages/celery/app/__init__.py", line 5, in <module>
    from celery import _state
  File "/usr/local/lib/python2.7/dist-packages/celery/_state.py", line 17, in <module>
    from celery.utils.threads import LocalStack
  File "/usr/local/lib/python2.7/dist-packages/celery/utils/__init__.py", line 9, in <module>
    from .nodenames import worker_direct, nodename, nodesplit
  File "/usr/local/lib/python2.7/dist-packages/celery/utils/nodenames.py", line 9, in <module>
    from kombu.entity import Exchange, Queue
  File "/usr/local/lib/python2.7/dist-packages/kombu/entity.py", line 9, in <module>
    from .serialization import prepare_accept_content
  File "/usr/local/lib/python2.7/dist-packages/kombu/serialization.py", line 456, in <module>
    for ep, args in entrypoints('kombu.serializers'):  # pragma: no cover
  File "/usr/local/lib/python2.7/dist-packages/kombu/utils/compat.py", line 89, in entrypoints
    for ep in importlib_metadata.entry_points().get(namespace, [])
  File "/usr/local/lib/python2.7/dist-packages/importlib_metadata/__init__.py", line 456, in entry_points
    ordered = sorted(eps, key=by_group)
  File "/usr/local/lib/python2.7/dist-packages/importlib_metadata/__init__.py", line 454, in <genexpr>
    dist.entry_points for dist in distributions())
  File "/usr/local/lib/python2.7/dist-packages/importlib_metadata/__init__.py", line 364, in <genexpr>
    cls._search_path(path, pattern)
  File "/usr/local/lib/python2.7/dist-packages/importlib_metadata/__init__.py", line 373, in _switch_path
    return pathlib.Path(path)
  File "/usr/local/lib/python2.7/dist-packages/pathlib2/__init__.py", line 1256, in __new__
    self = cls._from_parts(args, init=False)
  File "/usr/local/lib/python2.7/dist-packages/pathlib2/__init__.py", line 898, in _from_parts
    drv, root, parts = self._parse_args(args)
  File "/usr/local/lib/python2.7/dist-packages/pathlib2/__init__.py", line 891, in _parse_args
    return cls._flavour.parse_parts(parts)
  File "/usr/local/lib/python2.7/dist-packages/pathlib2/__init__.py", line 250, in parse_parts
    parsed.append(intern(x))
TypeError: can't intern subclass of string



Postgress LOg

The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default timezone ... Etc/UTC
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok


WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:

    pg_ctl -D /var/lib/postgresql/data -l logfile start

waiting for server to start....LOG:  database system was shut down at 2019-08-26 13:42:41 UTC
LOG:  MultiXact member wraparound protections are now enabled
LOG:  database system is ready to accept connections
LOG:  autovacuum launcher started
 done
server started
CREATE DATABASE


/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*

LOG:  received fast shutdown request
waiting for server to shut down...LOG:  aborting any active transactions
.LOG:  autovacuum launcher shutting down
LOG:  shutting down
LOG:  database system is shut down
 done
server stopped

PostgreSQL init process complete; ready for start up.

LOG:  database system was shut down at 2019-08-26 13:42:43 UTC
LOG:  MultiXact member wraparound protections are now enabled
LOG:  database system is ready to accept connections
LOG:  autovacuum launcher started
LOG:  incomplete startup packet
LOG:  incomplete startup packet
ERROR:  relation "users" does not exist at character 13
STATEMENT:  INSERT INTO users (username, email, password) VALUES ('admin', 'abc@airflow.com', '$2b$12$Vmkgo0OBgjLmylPMi3yrCOhVIWhWAgrEpCCojRZw0weeP..3nneg.') RETURNING users.id
LOG:  incomplete startup packet
LOG:  incomplete startup packet
LOG:  incomplete startup packet
LOG:  incomplete startup packet
ERROR:  relation "users" does not exist at character 13
STATEMENT:  INSERT INTO users (username, email, password) VALUES ('admin', 'abc@airflow.com', '$2b$12$y2DtC8uEM5coowQZP3GZsOIw/QFkqKZqvV4TcOkCSJ0wM.QbiwbA2') RETURNING users.id
LOG:  incomplete startup packet
LOG:  incomplete startup packet
LOG:  incomplete startup packet
LOG:  incomplete startup packet
ERROR:  relation "users" does not exist at character 13
STATEMENT:  INSERT INTO users (username, email, password) VALUES ('admin', 'abc@airflow.com', '$2b$12$0gj.4OfVy5y.xVt2FpVny.mRfCD/1wYnAbdMA22Xj4aI54tATo4Nu') RETURNING users.id
LOG:  incomplete startup packet
LOG:  incomplete startup packet
ERROR:  relation "users" does not exist at character 13
STATEMENT:  INSERT INTO users (username, email, password) VALUES ('admin', 'abc@airflow.com', '$2b$12$KXKdhuhdt5rmehEPxuX1He8uwE2fvgMcWoS4rg4oGzL5xWfn8Cgd6') RETURNING users.id
LOG:  incomplete startup packet
LOG:  incomplete startup packet
LOG:  incomplete startup packet
ERROR:  relation "users" does not exist at character 13
STATEMENT:  INSERT INTO users (username, email, password) VALUES ('admin', 'abc@airflow.com', '$2b$12$QVtj0DHd6uLOnlIlwbE3kezYDzP.Y8m/Ln9H9of77pEKCihOiLhnq') RETURNING users.id
LOG:  incomplete startup packet
-- Sandhya
airflow
kubernetes
postgres-9.6
webserver

Similar Questions

postgres/data/pgdata is not a valid data directory in kubernetes workload
How to access External DB from kubernate pods
Kibana 6.8.2 deploying on K8S ( using security xpack feature )
Cannot find the specs of Kubernetes Manifests

2 Answers

8/29/2019

You'll likely need to downgrade your kombu version. kombu==4.5.0 works.

How are you installing packages? If you're using a requirements.txt file, then it'll install kombu>4.4.0,<5.0 apache-airflow==1.9.0 specifies celery~=4.3

see: https://github.com/celery/celery/blob/v4.3.0/requirements/default.txt

If possible, use a Python package manager that employs some form of lockfile, such as pipenv or poetry.

-- Alec
Source: StackOverflow
8/28/2019

looks you postgresql server is up, but airflow database need init. (especaily 'user' relation "users" does not exist at character 13)

you can try to re-init your airflow database

initialize the database

airflow initdb

https://airflow.apache.org/start.html

-- Yong Wang
Source: StackOverflow