I am working on creating a Jupyterhub using Kubernetes and followed the instructions provided in Zero to Jupyterhub documentation. The basic installation of jupyterhub worked fine and I was even able to use the jupyterhub. Later on, when I created my own docker container using a Dockerfile that I wrote, the jupyter-test pod crashes. After checking the logs, I found out that it is crashing because of a permission issue. Now, I know that in docker every command is run as sudo, so why is this issue present? I tried installing jupyterhub with --user flag but then the jupyterhub-singleuser is not found in PATH.
Please find below the outputs of some commands I ran for debugging:
kubectl describe pod jupyter-test
Name: jupyter-test
Namespace: test-cv4faces-jupyterhub
Node: gke-test-cv4faces-cluste-default-pool-7198f95e-hrkp/10.128.0.2
Start Time: Mon, 06 Aug 2018 12:53:37 +0000
Labels: app=jupyterhub
component=singleuser-server
heritage=jupyterhub
Annotations: hub.jupyter.org/username=test
Status: Running
IP: 10.48.2.11
Init Containers:
block-cloud-metadata:
Container ID: docker://50123375a6748a03d93a4d441901ea5bdcb3d660b8c254224826d491f5a16221
Image: jupyterhub/k8s-network-tools:v0.6
Image ID: docker-pullable://jupyterhub/k8s-network-tools@sha256:271e7f7c9b4aa319788ca1bbce3428cad7b9da852a8c3af49178abcc2cc45b28
Port: <none>
Command:
iptables
-A
OUTPUT
-d
169.254.169.254
-j
DROP
State: Terminated
Reason: Completed
Exit Code: 0
Started: Mon, 06 Aug 2018 12:53:55 +0000
Finished: Mon, 06 Aug 2018 12:53:55 +0000
Ready: True
Restart Count: 0
Environment: <none>
Mounts: <none>
Containers:
notebook:
Container ID: docker://5efb329f6e91d31b71a941281fecf873433abdaf290e71354ee88a0877e62a5d
Image: gcr.io/eng-spot-212108/test-cv4faces-image:version2
Image ID: docker-pullable://gcr.io/eng-spot-212108/test-cv4faces-image@sha256:e7baf466e96c4865f50b442b49ca91fcb00f3de59c02a43c756e291ffd69161b
Port: 8888/TCP
Args:
jupyterhub-singleuser
--ip="0.0.0.0"
--port=8888
State: Terminated
Reason: Error
Exit Code: 1
Started: Mon, 06 Aug 2018 12:56:50 +0000
Finished: Mon, 06 Aug 2018 12:56:50 +0000
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Mon, 06 Aug 2018 12:55:29 +0000
Finished: Mon, 06 Aug 2018 12:55:29 +0000
Ready: False
Restart Count: 5
Requests:
memory: 1073741824
Environment:
EMAIL: test@local
GIT_AUTHOR_NAME: test
GIT_COMMITTER_NAME: test
JUPYTERHUB_API_TOKEN: 284ac1090eac420f850f94a0a64b79db
JPY_API_TOKEN: 284ac1090eac420f850f94a0a64b79db
JUPYTERHUB_ADMIN_ACCESS: 1
JUPYTERHUB_CLIENT_ID: user-test
JUPYTERHUB_HOST:
JUPYTERHUB_OAUTH_CALLBACK_URL: /user/test/oauth_callback
JUPYTERHUB_USER: test
JUPYTERHUB_API_URL: http://10.51.246.119:8081/hub/api
JUPYTERHUB_BASE_URL: /
JUPYTERHUB_SERVICE_PREFIX: /user/test/
MEM_GUARANTEE: 1073741824
Mounts:
/home/jovyan from volume-test (rw)
/var/run/secrets/kubernetes.io/serviceaccount from no-api-access-please (ro)
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
volume-test:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: claim-test
ReadOnly: false
no-api-access-please:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 3m default-scheduler Successfully assigned jupyter-test to gke-test-cv4faces-cluste-default-pool-7198f95e-hrkp
Normal SuccessfulMountVolume 3m kubelet, gke-test-cv4faces-cluste-default-pool-7198f95e-hrkp MountVolume.SetUp succeeded for volume "no-api-access-please"
Normal SuccessfulMountVolume 3m kubelet, gke-test-cv4faces-cluste-default-pool-7198f95e-hrkp MountVolume.SetUp succeeded for volume "pvc-01c3a951-9976-11e8-b54e-42010a800122"
Normal Pulled 3m kubelet, gke-test-cv4faces-cluste-default-pool-7198f95e-hrkp Container image "jupyterhub/k8s-network-tools:v0.6" already present on machine
Normal Created 3m kubelet, gke-test-cv4faces-cluste-default-pool-7198f95e-hrkp Created container
Normal Started 3m kubelet, gke-test-cv4faces-cluste-default-pool-7198f95e-hrkp Started container
Normal Pulled 2m (x4 over 3m) kubelet, gke-test-cv4faces-cluste-default-pool-7198f95e-hrkp Container image "gcr.io/eng-spot-212108/test-cv4faces-image:version2" already present on machine
Normal Created 2m (x4 over 3m) kubelet, gke-test-cv4faces-cluste-default-pool-7198f95e-hrkp Created container
Normal Started 2m (x4 over 3m) kubelet, gke-test-cv4faces-cluste-default-pool-7198f95e-hrkp Started container
Warning BackOff 1m (x8 over 3m) kubelet, gke-test-cv4faces-cluste-default-pool-7198f95e-hrkp Back-off restarting failed container
kubectl logs jupyter-test
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/traitlets/traitlets.py", line 528, in get
value = obj._trait_values[self.name]
KeyError: 'runtime_dir'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/bin/jupyterhub-singleuser", line 6, in <module>
main()
File "/usr/local/lib/python3.5/dist-packages/jupyterhub/singleuser.py", line 455, in main
return SingleUserNotebookApp.launch_instance(argv)
File "/usr/local/lib/python3.5/dist-packages/jupyter_core/application.py", line 266, in launch_instance
return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/traitlets/config/application.py", line 657, in launch_instance
app.initialize(argv)
File "<decorator-gen-7>", line 2, in initialize
File "/usr/local/lib/python3.5/dist-packages/traitlets/config/application.py", line 87, in catch_config_error
return method(app, *args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/notebook/notebookapp.py", line 1600, in initialize
self.init_configurables()
File "/usr/local/lib/python3.5/dist-packages/notebook/notebookapp.py", line 1293, in init_configurables
connection_dir=self.runtime_dir,
File "/usr/local/lib/python3.5/dist-packages/traitlets/traitlets.py", line 556, in __get__
return self.get(obj, cls)
File "/usr/local/lib/python3.5/dist-packages/traitlets/traitlets.py", line 535, in get
value = self._validate(obj, dynamic_default())
File "/usr/local/lib/python3.5/dist-packages/jupyter_core/application.py", line 99, in _runtime_dir_default
ensure_dir_exists(rd, mode=0o700)
File "/usr/local/lib/python3.5/dist-packages/jupyter_core/utils/__init__.py", line 13, in ensure_dir_exists
os.makedirs(path, mode=mode)
File "/usr/lib/python3.5/os.py", line 231, in makedirs
makedirs(head, mode, exist_ok)
File "/usr/lib/python3.5/os.py", line 231, in makedirs
makedirs(head, mode, exist_ok)
File "/usr/lib/python3.5/os.py", line 231, in makedirs
makedirs(head, mode, exist_ok)
File "/usr/lib/python3.5/os.py", line 241, in makedirs
mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/.local'I would really appreciate any help regarding this.
Thanks in advance
Vishwesh
Permission of /home/{user} ( for e.g. /home/jovyan ) might have been affected while installing some packages in your jupyter-test container. you can fix a permission by simply giving necessary permission to /home/{user}
for quick fix, simply add this to your dockerfile
chown -R {user} /home/{user}