Using Airflow to run the KubernetesPodOperator on AWS EKS getting kubernetes.client.rest.ApiException: (400), states that Pod in version \"v1\" cannot be handled as a Pod: v1.Pod.Spec. Is aws-eks not supporting this type of pod creation?
Here is the full log
[2019-03-13 01:13:47,991] {models.py:1593} INFO - Executing
<Task(KubernetesPodOperator): failing-task> on 2019-03-
13T01:12:33.980710+00:00
[2019-03-13 01:13:47,991] {base_task_runner.py:118} INFO - Running:
['bash', '-c', 'airflow run testkuberete2 failing-task 2019-03-
13T01:12:33.980710+00:00 --job_id 20 --raw -sd DAGS_FOLDER/test-
kubepodopptest5.py --cfg_path /tmp/tmpmwb4uzk0']
[2019-03-13 01:13:48,415] {base_task_runner.py:101} INFO - Job 20:
Subtask failing-task [2019-03-13 01:13:48,415] {settings.py:174} INFO -
settings.configure_orm(): Using pool settings. pool_size=5,
pool_recycle=3600, pid=408
[2019-03-13 01:13:48,864] {base_task_runner.py:101} INFO - Job 20:
Subtask failing-task [2019-03-13 01:13:48,863] {__init__.py:51} INFO -
Using executor CeleryExecutor
[2019-03-13 01:13:49,115] {base_task_runner.py:101} INFO - Job 20:
Subtask failing-task [2019-03-13 01:13:49,114] {models.py:273} INFO -
Filling up the DagBag from /usr/local/airflow/dags/test-
kubepodopptest5.py
[2019-03-13 01:13:49,473] {base_task_runner.py:101} INFO - Job 20:
Subtask failing-task [2019-03-13 01:13:49,473] {cli.py:520} INFO -
Running <TaskInstance: testkuberete2.failing-task 2019-03-
13T01:12:33.980710+00:00 [running]> on host airflow-worker-1.airflow-
worker.rguonew.svc.cluster.local
I wrote some custom log to inspect what got send to the EKS:
[2019-03-13 01:13:49,782] {logging_mixin.py:95} INFO - [2019-03-13
01:13:49,781] {api_client.py:117} ERROR - header_params {Accept
application/json}
[2019-03-13 01:13:49,782] {logging_mixin.py:95} INFO - [2019-03-13
01:13:49,782] {api_client.py:117} ERROR - header_params {Content-Type
application/json}
[2019-03-13 01:13:49,782] {logging_mixin.py:95} INFO - [2019-03-13
01:13:49,782] {api_client.py:117} ERROR - header_params {User-Agent
Swagger-Codegen/8.0.1/python}
[2019-03-13 01:13:49,782] {logging_mixin.py:95} INFO - [2019-03-13
01:13:49,782] {api_client.py:128} ERROR - resource_path {namespace
rguonew}
[2019-03-13 01:13:49,782] {logging_mixin.py:95} INFO - [2019-03-13
01:13:49,782] {api_client.py:151} ERROR - body {{'apiVersion': 'v1',
'kind': 'Pod', 'metadata': {'name': 'fail-422f98fc', 'labels': {},
'annotations': {}}, 'spec': {'containers': [{'name': 'base',
'image':
'python:3.6.6-stretch', 'command': 'echo', 'imagePullPolicy':
'IfNotPresent', 'args': '10'}], 'restartPolicy': 'Never',
'nodeSelector': {}, 'volumes': [], 'serviceAccountName': 'default',
'affinity': {}}}}
[2019-03-13 01:13:49,782] {logging_mixin.py:95} INFO - [2019-03-13
01:13:49,782] {api_client.py:155} ERROR - url
{https://xxxxxxxxxxxx.xxxx.us-east-
1.eks.amazonaws.com/api/v1/namespaces/rguonew/pods}
[2019-03-13 01:13:49,838] {logging_mixin.py:95} INFO - [2019-03-13
01:13:49,823] {pod_launcher.py:58} ERROR - Exception when attempting to
create Namespaced Pod.
Traceback
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-
packages/airflow/contrib/kubernetes/pod_launcher.py", line 55, in
run_pod_async
resp = self._client.create_namespaced_pod(body=req,
namespace=pod.namespace)
File "/usr/local/lib/python3.6/site-
packages/kubernetes/client/apis/core_v1_api.py", line 6115, in
create_namespaced_pod
(data) = self.create_namespaced_pod_with_http_info(namespace, body,
**kwargs)
File "/usr/local/lib/python3.6/site-
packages/kubernetes/client/apis/core_v1_api.py", line 6206, in
create_namespaced_pod_with_http_info
collection_formats=collection_formats)
File "/usr/local/lib/python3.6/site-
packages/kubernetes/client/api_client.py", line 329, in call_api
_return_http_data_only, collection_formats, _preload_content,
_request_timeout)
File "/usr/local/lib/python3.6/site-packages/kubernetes/client/ap
i_client.py", line 163, in __call_api
_request_timeout=_request_timeout)
File "/usr/local/lib/python3.6/site-
packages/kubernetes/client/api_client.py", line 372, in request
body=body)
File "/usr/local/lib/python3.6/site-
packages/kubernetes/client/rest.py", line 266, in POST
body=body)
File "/usr/local/lib/python3.6/site-
packages/kubernetes/client/rest.py", line 222, in request
raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (400)
Reason: Bad Request
The Response:
HTTP response headers: HTTPHeaderDict({'Audit-Id': '96f8ffee-3e65-47a7-
bb3c-fe519281ee45', 'Content-Type': 'application/json', 'Date': 'Wed,
13 Mar 2019 01:13:49 GMT', 'Content-Length': '491'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":
{},"status":"Failure","message":"Pod in version \"v1\" cannot be
handled as a Pod: v1.Pod.Spec: v1.PodSpec.Containers: []v1.Container:
v1.Container.Command: []string: decode slice: expect [ or n, but found
\", error found in #10 byte of ...|ommand\": \"echo\", \"im|..., bigger
context ...|se\", \"image\": \"python:3.6.6-stretch\", \"command\":
\"echo\", \"imagePullPolicy\": \"IfNotPresent\", \"args\":
|...","reason":"BadRequest","code":400}
[2019-03-13 01:13:49,838] {logging_mixin.py:95} INFO -
Ah, great to find another using Airflow on EKS!!
I am also running a similar Airflow workload and can confirm that KubernetesPodOperator is supported with AWS EKS.
Looking at the response, I think perhaps the API call to create the pod either got mangled or your python code included incorrect argument types when instantiating the KubernetesPodOperator.
I decoded your pod definition (correct me if wrong):
---
apiVersion: v1
kind: Pod
metadata:
name: fail-422f98fc
labels: {}
annotations: {}
spec:
containers:
- name: base
image: python:3.6.6-stretch
command: echo
imagePullPolicy: IfNotPresent
args: '10'
restartPolicy: Never
nodeSelector: {}
volumes: []
serviceAccountName: default
affinity: {}
Your should wrap your "command" and "args" in square brackets as below:
---
apiVersion: v1
kind: Pod
metadata:
name: fail-422f98fc
labels: {}
annotations: {}
spec:
containers:
- name: base
image: python:3.6.6-stretch
command: ['echo']
imagePullPolicy: IfNotPresent
args: ['10']
restartPolicy: Never
nodeSelector: {}
volumes: []
serviceAccountName: default
affinity: {}
Hope this helps!