Airflow KubernetesPodOperator AirflowException('Pod Launching failed: {error}'.format(error=ex))

4/14/2020

I'm running a KubernetesPodOperator with a custome docker image that was imported with microk8s. When running the DAG in airflow I see the following log until I get the time out message AirflowException('Pod Launching failed: {error}'.format(error=ex))

[2020-04-14 23:06:39,875] {logging_mixin.py:95} INFO - [[34m2020-04-14 23:06:39,875[0m] {[34mpod_launcher.py:[0m141} INFO[0m - Event: [1mk8s-pod-ml-3a541375[0m had an event of type [1mPending[0m[0m

Searching for the problem I ran microk8s kubectl get pods on ubuntus terminal and got: enter image description here

as you can see the name of the pod is almost exactly the same in the logs (except fro the 1m at the begining) but the status is ErrImageNeverPull. I couldn't find anything in the internte regarding that status or the error except for this that says 'The default timeout for KubernetesPodOperator is 120 seconds, which can result in timeouts occurring before larger images download.' I change the default timeout to 600 seconds but did not work.

-- Alejandro Andrade
airflow
kubernetes
microk8s
ubuntu-18.04

1 Answer

4/16/2020

The problem

Microk8s was not having access to docker. I found out about it by running microk8s.inspect and getting the

WARNING:  Docker is installed. 
File "/etc/docker/daemon.json" does not exist. 
You should create it and add the following lines: 
{
    "insecure-registries" : ["localhost:32000"] 
}

and then restart docker with: sudo systemctl restart docker
Building the report tarball
  Report tarball is at /var/snap/microk8s/1320/inspection-report-20200415_141500.tar.gz

After doing as the warning said airflow manage to deploy the image in the pod.

-- Alejandro Andrade
Source: StackOverflow