Job with Kubernetes Pod Operator fail while running with kubernetes Executor

11/26/2019

I am currently using kubernetes executor with Airflow and all the operators in airflow are working correctly but All the dags scheduled in airflow containing kubernetes pod operator are failing.No logs are being displayed on airflow UI. I can only find Following error logs on the scheduler pod.

[2019-11-26 13:32:44,874] {kubernetes_executor.py:362} INFO - Event: kubernetesoperatorpassingtask-94a53570a63440e9ac11206e5a5e5da9 had an event of type MODIFIED
[2019-11-26 13:32:44,881] {kubernetes_executor.py:398} INFO - Event: kubernetesoperatorpassingtask-94a53570a63440e9ac11206e5a5e5da9 Failed
[2019-11-26 13:32:44,907] {kubernetes_executor.py:362} INFO - Event: kubernetesoperatorpassingtask-94a53570a63440e9ac11206e5a5e5da9 had an event of type DELETED
[2019-11-26 13:32:44,915] {kubernetes_executor.py:398} INFO - Event: kubernetesoperatorpassingtask-94a53570a63440e9ac11206e5a5e5da9 Failed
[2019-11-26 13:32:46,850] {kubernetes_executor.py:506} INFO - Attempting to finish pod; pod_id: kubernetesoperatorpassingtask-94a53570a63440e9ac11206e5a5e5da9; stat
e: failed; labels: {u'try_number': '1', u'execution_date': '2019-11-26T13_32_35.285788_plus_00_00', u'airflow-worker': '45390e71-9831-4ea4-9077-a2753eff16ae', u'tas
k_id': 'passing-task', u'dag_id': 'kubernetes-operator'}
[2019-11-26 13:32:46,854] {kubernetes_executor.py:625} INFO - Checking 1 task instances.
[2019-11-26 13:32:46,855] {kubernetes_executor.py:635} INFO - Found matching task kubernetes-operator-passing-task (2019-11-26 13:32:35.285788+00:00) with current s
tate of queued
[2019-11-26 13:32:46,858] {kubernetes_executor.py:506} INFO - Attempting to finish pod; pod_id: kubernetesoperatorpassingtask-94a53570a63440e9ac11206e5a5e5da9; stat
e: failed; labels: {u'try_number': '1', u'execution_date': '2019-11-26T13_32_35.285788_plus_00_00', u'airflow-worker': '45390e71-9831-4ea4-9077-a2753eff16ae', u'tas
k_id': 'passing-task', u'dag_id': 'kubernetes-operator'}
[2019-11-26 13:32:46,862] {kubernetes_executor.py:625} INFO - Checking 1 task instances.
[2019-11-26 13:32:46,863] {kubernetes_executor.py:635} INFO - Found matching task kubernetes-operator-passing-task (2019-11-26 13:32:35.285788+00:00) with current s
tate of queued
[2019-11-26 13:32:46,866] {kubernetes_executor.py:819} INFO - Changing state of ((u'kubernetes-operator', u'passing-task', datetime.datetime(2019, 11, 26, 13, 32, 3
5, 285788, tzinfo=tzlocal()), 1), u'failed', 'kubernetesoperatorpassingtask-94a53570a63440e9ac11206e5a5e5da9', '688810') to failed
[2019-11-26 13:32:46,871] {kubernetes_executor.py:855} INFO - Deleted pod: (u'kubernetes-operator', u'passing-task', datetime.datetime(2019, 11, 26, 13, 32, 35, 285
788, tzinfo=tzlocal()), 1)
[2019-11-26 13:32:46,872] {kubernetes_executor.py:819} INFO - Changing state of ((u'kubernetes-operator', u'passing-task', datetime.datetime(2019, 11, 26, 13, 32, 3
5, 285788, tzinfo=tzlocal()), 1), u'failed', 'kubernetesoperatorpassingtask-94a53570a63440e9ac11206e5a5e5da9', '688811') to failed
[2019-11-26 13:32:46,876] {kubernetes_executor.py:855} INFO - Deleted pod: (u'kubernetes-operator', u'passing-task', datetime.datetime(2019, 11, 26, 13, 32, 35, 285
788, tzinfo=tzlocal()), 1)
[2019-11-26 13:32:50,854] {scheduler_job.py:1288} INFO - Executor reports execution of kubernetes-operator.passing-task execution_date=2019-11-26 13:32:35.285788+00
:00 exited with status failed for try_number 1
[2019-11-26 13:32:50,859] {scheduler_job.py:1304} ERROR - Executor reports task instance <TaskInstance: kubernetes-operator.passing-task 2019-11-26 13:32:35.285788+
00:00 [queued]> finished (failed) although the task says its queued. Was the task killed externally?
[2019-11-26 13:32:50,860] {dagbag.py:92} INFO - Filling up the DagBag from /root/airflow/dags/kubernetes_pod_operator_test.py
[2019-11-26 13:32:50,866] {taskinstance.py:1058} ERROR - Executor reports task instance <TaskInstance: kubernetes-operator.passing-task 2019-11-26 13:32:35.285788+0
0:00 [queued]> finished (failed) although the task says its queued. Was the task killed externally?
None
[2019-11-26 13:32:50,868] {taskinstance.py:1089} INFO - Marking task as FAILED. 

The code of my Dag is:

import airflow
from airflow import DAG
from airflow.contrib.operators.kubernetes_pod_operator import KubernetesPodOperator
from datetime import datetime, timedelta
default_args = {
   'owner': 'airflow',
   'start_date': airflow.utils.dates.days_ago(2),
}
dag = DAG(
   dag_id='kubernetes-operator',
   default_args=default_args,
   schedule_interval=None,
)
task1 = KubernetesPodOperator(namespace='default',
                          image="Python:3.6",
                          cmds=["Python","-c"],
                          arguments=["print('hello world')"],
                          labels={"foo": "bar"},
                          name="passing-test",
                          task_id="passing-task",
                          get_logs=True,
                          dag=dag
)
-- Anmol Kalra
airflow
airflow-operator
airflow-scheduler
kubernetes
kubernetes-pod

0 Answers