K
Q

Question

shutil.copystat() fails inside Docker on Azure

7/31/2018

The failing code runs inside a Docker container based on python:3.6-stretch debian. It happens while Django moves a file from one Docker volume to another.

When I test on MacOS 10, it works without error. Here, the Docker containers are started with docker-compose and use regular Docker volumes on the local machine.

Deployed into Azure (AKS - Kubernetes on Azure), moving the file succeeds but copying the stats fails with the following error:

  File "/usr/local/lib/python3.6/site-packages/django/core/files/move.py", line 70, in file_move_safe
    copystat(old_file_name, new_file_name)
  File "/usr/local/lib/python3.6/shutil.py", line 225, in copystat
    _copyxattr(src, dst, follow_symlinks=follow)
  File "/usr/local/lib/python3.6/shutil.py", line 157, in _copyxattr
    names = os.listxattr(src, follow_symlinks=follow_symlinks)
OSError: [Errno 38] Function not implemented: '/some/path/file.pdf'

The volumes on Azure are persistent volume claims with ReadWriteMany access mode.

Now, copystat is documented as:

copystat() never returns failure.

https://docs.python.org/3/library/shutil.html

My questions are:

Is this a "bug" because the documentation says that it should "never return failure"?
Can I savely try/except this error because the file in question is moved (it only fails later on, while trying to copy the stats)
Can I change something about the Azure settings that fix this? (probably not)

Here some small test on the machine in Azure itself:

root:/media/documents# ls -al
insgesamt 267
drwxrwxrwx 2 1000 1000      0 Jul 31 15:29 .
drwxrwxrwx 2 1000 1000      0 Jul 31 15:29 ..
-rwxrwxrwx 1 1000 1000 136479 Jul 31 16:48 orig.pdf
-rwxrwxrwx 1 1000 1000 136479 Jul 31 15:29 testfile
root:/media/documents# lsattr 
--S-----c-jI------- ./orig.pdf
--S-----c-jI------- ./testfile
root:/media/documents# python
Python 3.6.6 (default, Jul 17 2018, 11:12:33) 
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import shutil
>>> shutil.copystat('orig.pdf', 'testfile')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/shutil.py", line 225, in copystat
    _copyxattr(src, dst, follow_symlinks=follow)
  File "/usr/local/lib/python3.6/shutil.py", line 157, in _copyxattr
    names = os.listxattr(src, follow_symlinks=follow_symlinks)
OSError: [Errno 38] Function not implemented: 'orig.pdf'
>>> shutil.copystat('orig.pdf', 'testfile', follow_symlinks=False)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/shutil.py", line 225, in copystat
    _copyxattr(src, dst, follow_symlinks=follow)
  File "/usr/local/lib/python3.6/shutil.py", line 157, in _copyxattr
    names = os.listxattr(src, follow_symlinks=follow_symlinks)
OSError: [Errno 38] Function not implemented: 'orig.pdf'
>>>

-- Risadinha

azure

azure-kubernetes

kubernetes

python-3.x

shutil

1 Answer

8/1/2018

The following solution is a hotfix. It would have to be applied to any method that calls copystat directly or indirectly (or any shutil method that produces an ignorable errno.ENOSYS).

if hasattr(os, 'listxattr'):
    LOGGER.warning('patching listxattr to avoid ERROR 38 (errno.ENOSYS)')
    # avoid "ERROR 38 function not implemented on Azure"
    with mock.patch('os.listxattr', return_value=[]):
        file_field.save(name=name, content=GeneratedFile(fresh, content_type=content_type), save=True)
else:
    file_field.save(name=name, content=GeneratedFile(fresh, content_type=content_type), save=True)

file_field.save is the Django method that calls the shutil code in question. It's the last location in my code before the error.

-- Risadinha

Source: StackOverflow

KQ

shutil.copystat() fails inside Docker on Azure

Similar Questions

1 Answer

K
Q