I'm trying to create a Kubeflow Pipeline component. My command works fine in a local Docker container, but within a Pod in a Kubeflow Pipeline it terminates with:
ipython /tmp/input-postgresql.ipynb: 1: ipython /tmp/input-postgresql.ipynb: Syntax error: end of file unexpected
To reproduce on Docker:
docker run -it continuumio/anaconda3:2020.07 /bin/sh
curl -o /tmp/input-postgresql.ipynb https://raw.githubusercontent.com/IBM/claimed/master/component-library/input/input-postgresql.ipynb && ipython /tmp/input-postgresql.ipynb
To reproduce on Kubeflow 1.6.0 please use the pipeline.yaml file
Had to change the component.yaml now it works
Changed syntax for command definition and therefore also needed to find out how to pass parameters:
implementation:
container:
image: continuumio/anaconda3:2020.07
command:
- sh
- -ec
- |
host=$0
database=$1
user=$2
password=$3
port=$4
sql=$5
data_dir=$6
output_data_csv=$7
mkdir -p $output_data_csv
wget https://raw.githubusercontent.com/IBM/claimed/master/component-library/input/input-postgresql.ipynb
ipython ./input-postgresql.ipynb host=$host database=$database user=$user password=$password port=$port sql=$sql data_dir=$data_dir output_data_csv=$output_data_csv
- {inputValue: host}
- {inputValue: database}
- {inputValue: user}
- {inputValue: password}
- {inputValue: port}
- {inputValue: sql}
- {inputValue: data_dir}
- {outputPath: output_data_csv}