I'm having a very strange issue running openmpi commands through the jupyter notebook.

Here is the problem context: I set up an openmpi cluster running on kubernetes. The head of the cluster has a notebook exposed so that I can run jobs through the interactive console. This is much more desirable than sshing into the containers to issue commands.

Now, when I issue the mpirun command through the notebook:

In [18]: !cd primo-files; pwd; mpirun --hostfile /tmp/hostfile -np 2 primo primo-inputs
Out[18]: /files/primo-files

The cell runs but just prints the current directory. Here, MPI does not start and there is no error message/output from MPI. The cell does not hang.

Now, when I run the same command from the same directory within the container:

mpirun@notebook-f9b7495f5-2vgks:/files/primo-files$ mpirun --hostfile /tmp/hostfile -np 2 primo primo-inputs

MPI starts and runs as expected. This is a very frustrating problem since the notebook does not output any error message or hints as to what is going wrong. Any help would be greatly appreciated!!