Jupyterhub: Permission denied (Zero to Jupyterhub)

8/7/2018

I am working on creating a Jupyterhub using Kubernetes and followed the instructions provided in Zero to Jupyterhub documentation. The basic installation of jupyterhub worked fine and I was even able to use the jupyterhub. Later on, when I created my own docker container using a Dockerfile that I wrote, the jupyter-test pod crashes. After checking the logs, I found out that it is crashing because of a permission issue. Now, I know that in docker every command is run as sudo, so why is this issue present? I tried installing jupyterhub with --user flag but then the jupyterhub-singleuser is not found in PATH.

Please find below the outputs of some commands I ran for debugging:

kubectl describe pod jupyter-test

Name:         jupyter-test
Namespace:    test-cv4faces-jupyterhub
Node:         gke-test-cv4faces-cluste-default-pool-7198f95e-hrkp/10.128.0.2
Start Time:   Mon, 06 Aug 2018 12:53:37 +0000
Labels:       app=jupyterhub
              component=singleuser-server
              heritage=jupyterhub
Annotations:  hub.jupyter.org/username=test
Status:       Running
IP:           10.48.2.11
Init Containers:
  block-cloud-metadata:
    Container ID:  docker://50123375a6748a03d93a4d441901ea5bdcb3d660b8c254224826d491f5a16221
    Image:         jupyterhub/k8s-network-tools:v0.6
    Image ID:      docker-pullable://jupyterhub/k8s-network-tools@sha256:271e7f7c9b4aa319788ca1bbce3428cad7b9da852a8c3af49178abcc2cc45b28
    Port:          <none>
    Command:
      iptables
      -A
      OUTPUT
      -d
      169.254.169.254
      -j
      DROP
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Mon, 06 Aug 2018 12:53:55 +0000
      Finished:     Mon, 06 Aug 2018 12:53:55 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:         <none>
Containers:
  notebook:
    Container ID:  docker://5efb329f6e91d31b71a941281fecf873433abdaf290e71354ee88a0877e62a5d
    Image:         gcr.io/eng-spot-212108/test-cv4faces-image:version2
    Image ID:      docker-pullable://gcr.io/eng-spot-212108/test-cv4faces-image@sha256:e7baf466e96c4865f50b442b49ca91fcb00f3de59c02a43c756e291ffd69161b
    Port:          8888/TCP
    Args:
      jupyterhub-singleuser
      --ip="0.0.0.0"
      --port=8888
    State:          Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Mon, 06 Aug 2018 12:56:50 +0000
      Finished:     Mon, 06 Aug 2018 12:56:50 +0000
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Mon, 06 Aug 2018 12:55:29 +0000
      Finished:     Mon, 06 Aug 2018 12:55:29 +0000
    Ready:          False
    Restart Count:  5
    Requests:
      memory:  1073741824
    Environment:
      EMAIL:                          test@local
      GIT_AUTHOR_NAME:                test
      GIT_COMMITTER_NAME:             test
      JUPYTERHUB_API_TOKEN:           284ac1090eac420f850f94a0a64b79db
      JPY_API_TOKEN:                  284ac1090eac420f850f94a0a64b79db
      JUPYTERHUB_ADMIN_ACCESS:        1
      JUPYTERHUB_CLIENT_ID:           user-test
      JUPYTERHUB_HOST:                
      JUPYTERHUB_OAUTH_CALLBACK_URL:  /user/test/oauth_callback
      JUPYTERHUB_USER:                test
      JUPYTERHUB_API_URL:             http://10.51.246.119:8081/hub/api
      JUPYTERHUB_BASE_URL:            /
      JUPYTERHUB_SERVICE_PREFIX:      /user/test/
      MEM_GUARANTEE:                  1073741824
    Mounts:
      /home/jovyan from volume-test (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from no-api-access-please (ro)
Conditions:
  Type           Status
  Initialized    True 
  Ready          False 
  PodScheduled   True 
Volumes:
  volume-test:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  claim-test
    ReadOnly:   false
  no-api-access-please:
    Type:        EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:      
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                 Age              From                                                          Message
  ----     ------                 ----             ----                                                          -------
  Normal   Scheduled              3m               default-scheduler                                             Successfully assigned jupyter-test to gke-test-cv4faces-cluste-default-pool-7198f95e-hrkp
  Normal   SuccessfulMountVolume  3m               kubelet, gke-test-cv4faces-cluste-default-pool-7198f95e-hrkp  MountVolume.SetUp succeeded for volume "no-api-access-please"
  Normal   SuccessfulMountVolume  3m               kubelet, gke-test-cv4faces-cluste-default-pool-7198f95e-hrkp  MountVolume.SetUp succeeded for volume "pvc-01c3a951-9976-11e8-b54e-42010a800122"
  Normal   Pulled                 3m               kubelet, gke-test-cv4faces-cluste-default-pool-7198f95e-hrkp  Container image "jupyterhub/k8s-network-tools:v0.6" already present on machine
  Normal   Created                3m               kubelet, gke-test-cv4faces-cluste-default-pool-7198f95e-hrkp  Created container
  Normal   Started                3m               kubelet, gke-test-cv4faces-cluste-default-pool-7198f95e-hrkp  Started container
  Normal   Pulled                 2m (x4 over 3m)  kubelet, gke-test-cv4faces-cluste-default-pool-7198f95e-hrkp  Container image "gcr.io/eng-spot-212108/test-cv4faces-image:version2" already present on machine
  Normal   Created                2m (x4 over 3m)  kubelet, gke-test-cv4faces-cluste-default-pool-7198f95e-hrkp  Created container
  Normal   Started                2m (x4 over 3m)  kubelet, gke-test-cv4faces-cluste-default-pool-7198f95e-hrkp  Started container
  Warning  BackOff                1m (x8 over 3m)  kubelet, gke-test-cv4faces-cluste-default-pool-7198f95e-hrkp  Back-off restarting failed container

kubectl logs jupyter-test

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/traitlets/traitlets.py", line 528, in get
    value = obj._trait_values[self.name]
KeyError: 'runtime_dir'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/jupyterhub-singleuser", line 6, in <module>
    main()
  File "/usr/local/lib/python3.5/dist-packages/jupyterhub/singleuser.py", line 455, in main
    return SingleUserNotebookApp.launch_instance(argv)
  File "/usr/local/lib/python3.5/dist-packages/jupyter_core/application.py", line 266, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/traitlets/config/application.py", line 657, in launch_instance
    app.initialize(argv)
  File "<decorator-gen-7>", line 2, in initialize
  File "/usr/local/lib/python3.5/dist-packages/traitlets/config/application.py", line 87, in catch_config_error
    return method(app, *args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/notebook/notebookapp.py", line 1600, in initialize
    self.init_configurables()
  File "/usr/local/lib/python3.5/dist-packages/notebook/notebookapp.py", line 1293, in init_configurables
    connection_dir=self.runtime_dir,
  File "/usr/local/lib/python3.5/dist-packages/traitlets/traitlets.py", line 556, in __get__
    return self.get(obj, cls)
  File "/usr/local/lib/python3.5/dist-packages/traitlets/traitlets.py", line 535, in get
    value = self._validate(obj, dynamic_default())
  File "/usr/local/lib/python3.5/dist-packages/jupyter_core/application.py", line 99, in _runtime_dir_default
    ensure_dir_exists(rd, mode=0o700)
  File "/usr/local/lib/python3.5/dist-packages/jupyter_core/utils/__init__.py", line 13, in ensure_dir_exists
    os.makedirs(path, mode=mode)
  File "/usr/lib/python3.5/os.py", line 231, in makedirs
    makedirs(head, mode, exist_ok)
  File "/usr/lib/python3.5/os.py", line 231, in makedirs
    makedirs(head, mode, exist_ok)
  File "/usr/lib/python3.5/os.py", line 231, in makedirs
    makedirs(head, mode, exist_ok)
  File "/usr/lib/python3.5/os.py", line 241, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/.local'

I would really appreciate any help regarding this.

Thanks in advance

Vishwesh

-- Vishwesh
docker
google-kubernetes-engine
jupyterhub

1 Answer

11/27/2018

Permission of /home/{user} ( for e.g. /home/jovyan ) might have been affected while installing some packages in your jupyter-test container. you can fix a permission by simply giving necessary permission to /home/{user}

for quick fix, simply add this to your dockerfile

chown -R {user} /home/{user}
-- saurssaurav
Source: StackOverflow