Snakemake "Auto-scaling Azure Kubernetes cluster without shared filesystem" fails with dependency error

8/6/2021

I am trying to run the https://snakemake.readthedocs.io/en/stable/executor_tutorial/azure_aks.html tutorial on snakemake.

Now the pipeline starts, but fails on an import error in python/snakemake.

WorkflowError:
The Python 3 package 'azure-storage-blob' need to be installed to use Azure Storage remote() file functionality. No module named 'azure'
  File "/opt/conda/envs/snakemake/lib/python3.9/importlib/__init__.py", line 127, in import_module
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed

In python, importing azure seems possible.

Python 3.9.6 | packaged by conda-forge | (default, Jul 11 2021, 03:39:48) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from azure.storage.blob import BlobServiceClient, BlobClient, ContainerClient, __version__
>>> print("Azure Blob Storage v" + __version__ + " - Python quickstart sample")
Azure Blob Storage v12.8.1 - Python quickstart sample
>>> 

I tried putting python + azure-storage-blob to the env files in snakemake, but doing this resulted in new problems. I also tried older versions of python and at some point a later version of kubernetes.

-- EricKuijt
azure
kubernetes
python
snakemake

1 Answer

8/9/2021

Ok, I found the answer.

My snakemake installation with the --kubernetes version ran an older image of snakemake. e.g.

snakemake/snakemake v6.6.1 67c6faaeec07 3 weeks ago 2.29GB

by forcing the latest snakemake image using the container-image argument

--container-image docker.io/snakemake/snakemake:latest

I was able to get the tutorial to work.

-- EricKuijt
Source: StackOverflow