I want to run a papermill command in a pod and get the errors (if any). This has to be done from an Airflow worker using the bash operator i Airflow.
I have tried running the command as needed, I have tried running multiple bash commands from my own worker, commands that were meant to be executed inside the worker. But this just gives me the idea that something like this is possible. Although, if there is any better approach that would solve this, I am all ears.
The bash command to be run is:
papermill check1.ipynb check1_output.ipynb -p params '{"aviral":"srivastava", "apoorv":"srivastava"}'Manually, I exec in my pod and then run this command. I want to make this a bash command in airflow's dag.
To use kubectl from within a pod you need to have binaries available in your pod.
You can either Create a Docker image with installation of kubectl.
I think the Dockerfile might look like this:
FROM ubuntu:14.04
# Install.
RUN \
sed -i 's/# \(.*multiverse$\)/\1/g' /etc/apt/sources.list && \
apt-get update && \
apt-get -y upgrade && \
apt-get install -y build-essential && \
apt-get install -y software-properties-common && \
apt-get install -y byobu curl git htop man unzip vim wget && \
rm -rf /var/lib/apt/lists/* && \
# Installing kubectl using native package management
apt-get install -y apt-transport-https && \
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - && \
echo 'deb https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee -a /etc/apt/sources.list.d/kubernetes.list && \
apt-get update && \
apt-get install -y kubectl
# Add files.
ADD root/.bashrc /root/.bashrc
ADD root/.gitconfig /root/.gitconfig
ADD root/.scripts /root/.scripts
# Set environment variables.
ENV HOME /root
# Define working directory.
WORKDIR /root
# Define default command.
CMD ["bash"]Or Define a Command and Arguments for a Container and use it to install kubectl when pod is starting.
If you want to use it from within Kubernetes Deployment the part of .yaml might look like this:
...
spec:
containers:
- name: ubuntu-with-kubectl
image: ubuntu
command: ["/bin/sh","-c"]
args:
- apt-get update && apt-get install -y apt-transport-https;
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -;
echo 'deb https://apt.kubernetes.io/ kubernetes-xenial main' | tee -a /etc/apt/sources.list.d/kubernetes.list;
apt-get update;
apt-get install -y kubectl;
...
You can also kubectl exec -it <pod_name> bash into the pod and install is manually using this guide Install and Set Up kubectl, but it will be gone on container restart.