KubernetesPodOperator Pod in version \"v1\" cannot be handled as a Pod: v1.Pod.Spec

3/13/2019

Using Airflow to run the KubernetesPodOperator on AWS EKS getting kubernetes.client.rest.ApiException: (400), states that Pod in version \"v1\" cannot be handled as a Pod: v1.Pod.Spec. Is aws-eks not supporting this type of pod creation?

Here is the full log

[2019-03-13 01:13:47,991] {models.py:1593} INFO - Executing 
<Task(KubernetesPodOperator): failing-task> on 2019-03- 
13T01:12:33.980710+00:00
[2019-03-13 01:13:47,991] {base_task_runner.py:118} INFO - Running: 
['bash', '-c', 'airflow run testkuberete2 failing-task 2019-03- 
13T01:12:33.980710+00:00 --job_id 20 --raw -sd DAGS_FOLDER/test- 
kubepodopptest5.py --cfg_path /tmp/tmpmwb4uzk0']
[2019-03-13 01:13:48,415] {base_task_runner.py:101} INFO - Job 20: 
Subtask failing-task [2019-03-13 01:13:48,415] {settings.py:174} INFO - 
settings.configure_orm(): Using pool settings. pool_size=5, 
pool_recycle=3600, pid=408
[2019-03-13 01:13:48,864] {base_task_runner.py:101} INFO - Job 20: 
Subtask failing-task [2019-03-13 01:13:48,863] {__init__.py:51} INFO - 
Using executor CeleryExecutor
[2019-03-13 01:13:49,115] {base_task_runner.py:101} INFO - Job 20: 
Subtask failing-task [2019-03-13 01:13:49,114] {models.py:273} INFO - 
Filling up the DagBag from /usr/local/airflow/dags/test- 
kubepodopptest5.py
[2019-03-13 01:13:49,473] {base_task_runner.py:101} INFO - Job 20: 
Subtask failing-task [2019-03-13 01:13:49,473] {cli.py:520} INFO - 
Running <TaskInstance: testkuberete2.failing-task 2019-03- 
13T01:12:33.980710+00:00 [running]> on host airflow-worker-1.airflow- 
worker.rguonew.svc.cluster.local

I wrote some custom log to inspect what got send to the EKS:

[2019-03-13 01:13:49,782] {logging_mixin.py:95} INFO - [2019-03-13 
01:13:49,781] {api_client.py:117} ERROR - header_params {Accept 
application/json}
[2019-03-13 01:13:49,782] {logging_mixin.py:95} INFO - [2019-03-13 
01:13:49,782] {api_client.py:117} ERROR - header_params {Content-Type 
application/json}
[2019-03-13 01:13:49,782] {logging_mixin.py:95} INFO - [2019-03-13 
01:13:49,782] {api_client.py:117} ERROR - header_params {User-Agent 
Swagger-Codegen/8.0.1/python}

 [2019-03-13 01:13:49,782] {logging_mixin.py:95} INFO - [2019-03-13 
 01:13:49,782] {api_client.py:128} ERROR - resource_path {namespace 
 rguonew}

[2019-03-13 01:13:49,782] {logging_mixin.py:95} INFO - [2019-03-13 
   01:13:49,782] {api_client.py:151} ERROR - body {{'apiVersion': 'v1', 
   'kind': 'Pod', 'metadata': {'name': 'fail-422f98fc', 'labels': {}, 
    'annotations': {}}, 'spec': {'containers': [{'name': 'base', 
  'image': 
   'python:3.6.6-stretch', 'command': 'echo', 'imagePullPolicy': 
   'IfNotPresent', 'args': '10'}], 'restartPolicy': 'Never', 
   'nodeSelector': {}, 'volumes': [], 'serviceAccountName': 'default', 
    'affinity': {}}}}

[2019-03-13 01:13:49,782] {logging_mixin.py:95} INFO - [2019-03-13 
01:13:49,782] {api_client.py:155} ERROR - url 
{https://xxxxxxxxxxxx.xxxx.us-east- 
1.eks.amazonaws.com/api/v1/namespaces/rguonew/pods}
[2019-03-13 01:13:49,838] {logging_mixin.py:95} INFO - [2019-03-13 
01:13:49,823] {pod_launcher.py:58} ERROR - Exception when attempting to 
create Namespaced Pod.

Traceback

Traceback (most recent call last):
File "/usr/local/lib/python3.6/site- 
packages/airflow/contrib/kubernetes/pod_launcher.py", line 55, in 
run_pod_async
resp = self._client.create_namespaced_pod(body=req, 
namespace=pod.namespace)
File "/usr/local/lib/python3.6/site- 
packages/kubernetes/client/apis/core_v1_api.py", line 6115, in 
create_namespaced_pod
(data) = self.create_namespaced_pod_with_http_info(namespace, body, 
**kwargs)

File "/usr/local/lib/python3.6/site- 
packages/kubernetes/client/apis/core_v1_api.py", line 6206, in 
create_namespaced_pod_with_http_info
collection_formats=collection_formats)
File "/usr/local/lib/python3.6/site- 
packages/kubernetes/client/api_client.py", line 329, in call_api
_return_http_data_only, collection_formats, _preload_content, 
_request_timeout)
File "/usr/local/lib/python3.6/site-packages/kubernetes/client/ap

i_client.py", line 163, in __call_api
    _request_timeout=_request_timeout)
  File "/usr/local/lib/python3.6/site- 
packages/kubernetes/client/api_client.py", line 372, in request
    body=body)
  File "/usr/local/lib/python3.6/site- 
packages/kubernetes/client/rest.py", line 266, in POST
    body=body)
  File "/usr/local/lib/python3.6/site- 
packages/kubernetes/client/rest.py", line 222, in request
    raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (400)
Reason: Bad Request

The Response:

HTTP response headers: HTTPHeaderDict({'Audit-Id': '96f8ffee-3e65-47a7- 
bb3c-fe519281ee45', 'Content-Type': 'application/json', 'Date': 'Wed, 
13 Mar 2019 01:13:49 GMT', 'Content-Length': '491'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata": 
{},"status":"Failure","message":"Pod in version \"v1\" cannot be 
handled as a Pod: v1.Pod.Spec: v1.PodSpec.Containers: []v1.Container: 
v1.Container.Command: []string: decode slice: expect [ or n, but found 
\", error found in #10 byte of ...|ommand\": \"echo\", \"im|..., bigger 
context ...|se\", \"image\": \"python:3.6.6-stretch\", \"command\": 
\"echo\", \"imagePullPolicy\": \"IfNotPresent\", \"args\": 
|...","reason":"BadRequest","code":400}
[2019-03-13 01:13:49,838] {logging_mixin.py:95} INFO - 
-- Ricc
airflow
amazon-eks
kubernetes

1 Answer

3/13/2019

Ah, great to find another using Airflow on EKS!!

I am also running a similar Airflow workload and can confirm that KubernetesPodOperator is supported with AWS EKS.

Looking at the response, I think perhaps the API call to create the pod either got mangled or your python code included incorrect argument types when instantiating the KubernetesPodOperator.

I decoded your pod definition (correct me if wrong):

---
apiVersion: v1
kind: Pod
metadata:
  name: fail-422f98fc
  labels: {}
  annotations: {}
spec:
  containers:
  - name: base
    image: python:3.6.6-stretch
    command: echo
    imagePullPolicy: IfNotPresent
    args: '10'
  restartPolicy: Never
  nodeSelector: {}
  volumes: []
  serviceAccountName: default
  affinity: {}

Your should wrap your "command" and "args" in square brackets as below:

---
apiVersion: v1
kind: Pod
metadata:
  name: fail-422f98fc
  labels: {}
  annotations: {}
spec:
  containers:
  - name: base
    image: python:3.6.6-stretch
    command: ['echo']
    imagePullPolicy: IfNotPresent
    args: ['10']
  restartPolicy: Never
  nodeSelector: {}
  volumes: []
  serviceAccountName: default
  affinity: {}

Hope this helps!

-- Frank Yucheng Gu
Source: StackOverflow