Using ray with Jupyterhub on Kubernets cluster

7/2/2019

I am building Kubernetes cluster with Jupyterhub and Ray and want users to access Jupyterhub and use Ray cluster on k8s. My plan was conencting Ray cluster using Ray API from Jupyterhub notebook "https://ray.readthedocs.io/en/latest/api.html"

kubectl get svc
NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                                          AGE
kubernetes   ClusterIP   10.96.0.1      <none>        443/TCP                                          6d19h
ray-head     ClusterIP   10.100.19.93   <none>        6379/TCP,6380/TCP,6381/TCP,12345/TCP,12346/TCP   4d21h

However, when I ran

import ray
ray.init(redis_address="10.100.19.93:6379")
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-3-f25708b1f128> in <module>
----> 1 ray.init(redis_address="10.100.19.93:6379")

/opt/conda/lib/python3.7/site-packages/ray/worker.py in init(redis_address, num_cpus, num_gpus, resources, object_store_memory, redis_max_memory, log_to_driver, node_ip_address, object_id_seed, local_mode, redirect_worker_output, redirect_output, ignore_reinit_error, num_redis_shards, redis_max_clients, redis_password, plasma_directory, huge_pages, include_webui, driver_id, configure_logging, logging_level, logging_format, plasma_store_socket_name, raylet_socket_name, temp_dir, load_code_from_local, _internal_config)
   1434             load_code_from_local=load_code_from_local)
   1435         _global_node = ray.node.Node(
-> 1436             ray_params, head=False, shutdown_at_exit=False, connect_only=True)
   1437 
   1438     connect(

/opt/conda/lib/python3.7/site-packages/ray/node.py in __init__(self, ray_params, head, shutdown_at_exit, connect_only)
    100             redis_client = self.create_redis_client()
    101             self.session_name = ray.utils.decode(
--> 102                 redis_client.get("session_name"))
    103 
    104         self._init_temp(redis_client)

/opt/conda/lib/python3.7/site-packages/ray/utils.py in decode(byte_str, allow_none)
    175     if not isinstance(byte_str, bytes):
    176         raise ValueError(
--> 177             "The argument {} must be a bytes object.".format(byte_str))
    178     if sys.version_info >= (3, 0):
    179         return byte_str.decode("ascii")

ValueError: The argument None must be a bytes object.

I was wondering if my approach is the right way to do and how to fix the eror.

-- Mike
jupyterhub
kubernetes
ray

1 Answer

7/5/2019

As long as you are running ray.init(redis_address="10.100.19.93:6379") not inside your cluster, you have to expose your ray-head service via LoadBalancer or NodePort depending on where your cluster is running.

More details on Publishing Services

So, do kubectl edit svc ray-head and change

type: ClusterIP

to

type: NodePort

Once done, try ray.init(redis_address="<node-ip-address>:<node-port>")

-- A_Suh
Source: StackOverflow