I'm new to Kubeflow and k8s. I have setup a single node k8s cluster and installed Kubeflow on this. I'm now trying the 'conditional pipeline' simple example from "Kubeflow for Machine Learning" book but I am getting "cannot post /apis/v1beta1/experiments" error ...
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'x-powered-by': 'Express', 'content-security-policy': "default-src 'none'", 'x-content-type-options': 'nosniff', 'content-type': 'text/html; charset=utf-8', 'content-length': '164', 'date': 'Fri, 11 Jun 2021 20:47:13 GMT', 'x-envoy-upstream-service-time': '2', 'server': 'envoy'})
HTTP response body: <!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Error</title>
</head>
<body>
<pre>Cannot POST /apis/v1beta1/experiments</pre>
</body>
</html>
Any pointer what could be going wrong?
If I do "kubectl get -A svc | grep dashboard", I only see Kubeflow central dashboard. Could this error be related to k8s dashboard not running?
This is the example I am trying:
Before this I've tried below MNIST example also and I faced exact same issue - https://github.com/anjuls/fashion-mnist-kfp-lab/blob/master/KF_Fashion_MNIST.ipynb
Finally, I tried to modify the kfp.Client() line to following: kfp.Client(host='http://127.0.0.1:8001').create_run_from_pipeline_func(conditional_pipeline, arguments={})
After this I'm getting error related to 'healtz' -
MaxRetryError: HTTPConnectionPool(host='127.0.0.1', port=8001): Max retries exceeded with url: /apis/v1beta1/healthz (Caused by ProtocolError('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')))I did following: kubectl logs ml-pipeline-ui-7ddcd74489-xrss8 -c ml-pipeline-ui -n kubeflow
It seems ml-pipeline is running on http://localhost:3000. So I modified the client call to following: client = kfp.Client(host='http://localhost:3000')
I still get an error - this time "connection refused".
MaxRetryError: HTTPConnectionPool(host='localhost', port=3000): Max retries exceeded with url: /apis/v1beta1/healthz (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fffd1ca0f28>: Failed to establish a new connection: [Errno 111] Connection refused',))I've made some progress. However, issues are not fully resolved but I can at least proceed with client creation, do health check, list existing pipelines.
I could find ml-pipeline service is running on following internal IP:
kubeflow service/ml-pipeline ClusterIP 172.19.31.229
I then used this IP in kfp.Client() API - this resulted in RBAC access issue. I then patched my k8s with following with some hint from another issue -
apiVersion: rbac.istio.io/v1alpha1
kind: ClusterRbacConfig
metadata:
name: default
spec:
mode: "OFF"This resolved issues I was facing with kfp.Client(). But now, I'm facing below error when I try to create_experiment():
ApiException: (400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'content-type': 'application/json', 'trailer': 'Grpc-Trailer-Content-Type', 'date': 'Wed, 16 Jun 2021 22:40:54 GMT', 'x-envoy-upstream-service-time': '2', 'server': 'envoy', 'transfer-encoding': 'chunked'})
HTTP response body: {"error":"Invalid input error: Invalid resource references for experiment. ListExperiment requires filtering by namespace.","message":"Invalid input error: Invalid resource references for experiment. ListExperiment requires filtering by namespace.","code":3,"details":[{"@type":"type.googleapis.com/api.Error","error_message":"Invalid resource references for experiment. ListExperiment requires filtering by namespace.","error_details":"Invalid input error: Invalid resource references for experiment. ListExperiment requires filtering by namespace."}]}Have you tried to port-forward ml-pipeline-ui?
kubectl port-forward svc/ml-pipeline-ui 3000:80 --namespace kubeflowThen check whether you can do
import kfp
client = kfp.Client(host='http://localhost:3000')
print(client.list_experiments())See the following doc: https://www.kubeflow.org/docs/components/pipelines/sdk/connect-api/