Kubeflow - error in create_run_from_pipeline_func


I'm new to Kubeflow and k8s. I have setup a single node k8s cluster and installed Kubeflow on this. I'm now trying the 'conditional pipeline' simple example from "Kubeflow for Machine Learning" book but I am getting "cannot post /apis/v1beta1/experiments" error ...

Reason: Not Found
HTTP response headers: HTTPHeaderDict({'x-powered-by': 'Express', 'content-security-policy': "default-src 'none'", 'x-content-type-options': 'nosniff', 'content-type': 'text/html; charset=utf-8', 'content-length': '164', 'date': 'Fri, 11 Jun 2021 20:47:13 GMT', 'x-envoy-upstream-service-time': '2', 'server': 'envoy'})
HTTP response body: <!DOCTYPE html>
<html lang="en">
<meta charset="utf-8">
<pre>Cannot POST /apis/v1beta1/experiments</pre>

Any pointer what could be going wrong?

If I do "kubectl get -A svc | grep dashboard", I only see Kubeflow central dashboard. Could this error be related to k8s dashboard not running?

This is the example I am trying:


Before this I've tried below MNIST example also and I faced exact same issue - https://github.com/anjuls/fashion-mnist-kfp-lab/blob/master/KF_Fashion_MNIST.ipynb

Finally, I tried to modify the kfp.Client() line to following: kfp.Client(host='').create_run_from_pipeline_func(conditional_pipeline, arguments={})

After this I'm getting error related to 'healtz' -

MaxRetryError: HTTPConnectionPool(host='', port=8001): Max retries exceeded with url: /apis/v1beta1/healthz (Caused by ProtocolError('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')))

I did following: kubectl logs ml-pipeline-ui-7ddcd74489-xrss8 -c ml-pipeline-ui -n kubeflow

It seems ml-pipeline is running on http://localhost:3000. So I modified the client call to following: client = kfp.Client(host='http://localhost:3000')

I still get an error - this time "connection refused".

MaxRetryError: HTTPConnectionPool(host='localhost', port=3000): Max retries exceeded with url: /apis/v1beta1/healthz (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fffd1ca0f28>: Failed to establish a new connection: [Errno 111] Connection refused',))
-- soumeng78

2 Answers


I've made some progress. However, issues are not fully resolved but I can at least proceed with client creation, do health check, list existing pipelines.

I could find ml-pipeline service is running on following internal IP:

kubeflow service/ml-pipeline ClusterIP

I then used this IP in kfp.Client() API - this resulted in RBAC access issue. I then patched my k8s with following with some hint from another issue -

apiVersion: rbac.istio.io/v1alpha1
kind: ClusterRbacConfig
  name: default
  mode: "OFF"

This resolved issues I was facing with kfp.Client(). But now, I'm facing below error when I try to create_experiment():

ApiException: (400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'content-type': 'application/json', 'trailer': 'Grpc-Trailer-Content-Type', 'date': 'Wed, 16 Jun 2021 22:40:54 GMT', 'x-envoy-upstream-service-time': '2', 'server': 'envoy', 'transfer-encoding': 'chunked'})
HTTP response body: {"error":"Invalid input error: Invalid resource references for experiment. ListExperiment requires filtering by namespace.","message":"Invalid input error: Invalid resource references for experiment. ListExperiment requires filtering by namespace.","code":3,"details":[{"@type":"type.googleapis.com/api.Error","error_message":"Invalid resource references for experiment. ListExperiment requires filtering by namespace.","error_details":"Invalid input error: Invalid resource references for experiment. ListExperiment requires filtering by namespace."}]}
-- soumeng78
Source: StackOverflow


Have you tried to port-forward ml-pipeline-ui?

kubectl port-forward svc/ml-pipeline-ui 3000:80 --namespace kubeflow

Then check whether you can do

import kfp
client = kfp.Client(host='http://localhost:3000')

See the following doc: https://www.kubeflow.org/docs/components/pipelines/sdk/connect-api/

-- Ark-kun
Source: StackOverflow