K
Q

Connecting to AI Platform Pipelines using the Kubeflow Pipelines SDK with Python and PyCharm

11/25/2020

Using windows 10 and Python 3.9 with PyCharm IDE

I am trying to list the pipelines using the following code:

import kfp
client = kfp.Client(host='.......bb92ea05-dot-us-central2.pipelines.googleusercontent.com')
    
client.list_pipelines()

I know the host variable is correct because I copied it from the AI platform Pipeline. I am getting the following problem:

FileNotFoundError: [WinError 2] The system cannot find the file specified

I think that the file can't be found since the program can't connect to the GCP's AI PLatform Pipeline.

I have installed Kubeflow Pipeline SDK by doing the following commands in Pycharm terminal:

  • conda create --name mlpipeline python=3.7
  • conda activate mlpipeline
  • pip3 install kfp --upgrade --user

When I run:

kfp diagnose_me

I get the following command error:

Google Cloud SDK is not installed, gcloud, gsutil and kubectl are required for this app to run. Please follow instructions at https://cloud.google.com/sdk/install to install the SDK

I have tried

  • Installing Google Cloud SDK and initializing gcloud(Although when I run a command it says 'Python not found')
  • Installing kubectl and gsutil with gcloud commands
-- Franco
google-cloud-ai-platform-pipelines
kubeflow
kubeflow-pipelines
kubernetes
python

Similar Questions

Mount configmap file onto hostpath volume
Kubernetes Pod OOMKilled Solution
Deploying a data driven web app on Kubernetes
How can I serve a static file to my local Kubernetes deployed service from my controller file?
Setting separate values for environment variable in HELM for DEV, PRESTAGING, STAGING and PROD

2 Answers

12/6/2020

I transitioned over to Ubuntu and from there the problem was fixed.

What I did:

  • Installed Google Cloud SDK and configured it: https://cloud.google.com/sdk/docs/quickstart

    Also install gsutil and kubectl using gcloud components install component_id command

  • Install KFP SDK: Run the following commands in terminal:

    apt-get update; apt-get install -y wget bzip2
    wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
    bash Miniconda3-latest-Linux-x86_64.sh
    
    conda create --name mlpipeline python=3.7	
    conda activate mlpipeline
    
    pip3 install kfp --upgrade
-- Franco
Source: StackOverflow
1/12/2021

I ran into the exact same problem. I fixed it by using Ubuntu OS rather than Windows OS. Use a VM and run Ubuntu in the VM. Ubuntu environment is way easier to set up and manage.

-- Franco
Source: StackOverflow