I am building Kubernetes cluster with Jupyterhub and Ray and want users to access Jupyterhub and use Ray cluster on k8s. My plan was conencting Ray cluster using Ray API from Jupyterhub notebook "https://ray.readthedocs.io/en/latest/api.html"
kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 6d19h
ray-head ClusterIP 10.100.19.93 <none> 6379/TCP,6380/TCP,6381/TCP,12345/TCP,12346/TCP 4d21h
However, when I ran
import ray
ray.init(redis_address="10.100.19.93:6379")
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-3-f25708b1f128> in <module>
----> 1 ray.init(redis_address="10.100.19.93:6379")
/opt/conda/lib/python3.7/site-packages/ray/worker.py in init(redis_address, num_cpus, num_gpus, resources, object_store_memory, redis_max_memory, log_to_driver, node_ip_address, object_id_seed, local_mode, redirect_worker_output, redirect_output, ignore_reinit_error, num_redis_shards, redis_max_clients, redis_password, plasma_directory, huge_pages, include_webui, driver_id, configure_logging, logging_level, logging_format, plasma_store_socket_name, raylet_socket_name, temp_dir, load_code_from_local, _internal_config)
1434 load_code_from_local=load_code_from_local)
1435 _global_node = ray.node.Node(
-> 1436 ray_params, head=False, shutdown_at_exit=False, connect_only=True)
1437
1438 connect(
/opt/conda/lib/python3.7/site-packages/ray/node.py in __init__(self, ray_params, head, shutdown_at_exit, connect_only)
100 redis_client = self.create_redis_client()
101 self.session_name = ray.utils.decode(
--> 102 redis_client.get("session_name"))
103
104 self._init_temp(redis_client)
/opt/conda/lib/python3.7/site-packages/ray/utils.py in decode(byte_str, allow_none)
175 if not isinstance(byte_str, bytes):
176 raise ValueError(
--> 177 "The argument {} must be a bytes object.".format(byte_str))
178 if sys.version_info >= (3, 0):
179 return byte_str.decode("ascii")
ValueError: The argument None must be a bytes object.
I was wondering if my approach is the right way to do and how to fix the eror.
As long as you are running ray.init(redis_address="10.100.19.93:6379")
not inside your cluster, you have to expose your ray-head service via LoadBalancer
or NodePort
depending on where your cluster is running.
More details on Publishing Services
So, do kubectl edit svc ray-head
and change
type: ClusterIP
to
type: NodePort
Once done, try ray.init(redis_address="<node-ip-address>:<node-port>")