I am trying to run the https://snakemake.readthedocs.io/en/stable/executor_tutorial/azure_aks.html tutorial on snakemake.
Now the pipeline starts, but fails on an import error in python/snakemake.
WorkflowError:
The Python 3 package 'azure-storage-blob' need to be installed to use Azure Storage remote() file functionality. No module named 'azure'
File "/opt/conda/envs/snakemake/lib/python3.9/importlib/__init__.py", line 127, in import_module
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 850, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
In python, importing azure seems possible.
Python 3.9.6 | packaged by conda-forge | (default, Jul 11 2021, 03:39:48)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from azure.storage.blob import BlobServiceClient, BlobClient, ContainerClient, __version__
>>> print("Azure Blob Storage v" + __version__ + " - Python quickstart sample")
Azure Blob Storage v12.8.1 - Python quickstart sample
>>>
I tried putting python + azure-storage-blob to the env files in snakemake, but doing this resulted in new problems. I also tried older versions of python and at some point a later version of kubernetes.
Ok, I found the answer.
My snakemake installation with the --kubernetes version ran an older image of snakemake. e.g.
snakemake/snakemake v6.6.1 67c6faaeec07 3 weeks ago 2.29GB
by forcing the latest snakemake image using the container-image argument
--container-image docker.io/snakemake/snakemake:latest
I was able to get the tutorial to work.