I have built an image from: https://archive.apache.org/dist/spark/spark-$2.4.4/spark-$2.4.4-bin-hadoop2.7.tgz
Once is downloaded, I do:
cd spark-2.4.4-bin-hadoop2.7 && bin/docker-image-tool.sh buildThen my image spark-py:latest is built.
I want to install pyarrow in it using this docker file:
FROM spark-py:latest
COPY *.jar /opt/spark/jars/
RUN rm /opt/spark/jars/kubernetes-*-4.1.2.jar
RUN pip install --no-cache pyarrow
but I have an error:
------
> [11/11] RUN python3 -m pip install --no-cache --no-deps pyarrow:
#15 0.772 Collecting pyarrow
#15 0.887 Downloading pyarrow-2.0.0.tar.gz (58.9 MB)
#15 6.265 Installing build dependencies: started
#15 12.45 Installing build dependencies: finished with status 'error'
#15 12.45 ERROR: Command errored out with exit status 1:
#15 12.45 command: /usr/bin/python3 /usr/lib/python3.6/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-iz28sqos/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'cython >= 0.29' 'numpy==1.14.5; python_version<'"'"'3.7'"'"'' 'numpy==1.16.0; python_version>='"'"'3.7'"'"'' setuptools setuptools_scm wheel
#15 12.45 cwd: None
#15 12.45 Complete output (687 lines):
#15 12.45 Ignoring numpy: markers 'python_version >= "3.7"' don't match your environment
#15 12.45 Collecting cython>=0.29
#15 12.45 Downloading Cython-0.29.21-py2.py3-none-any.whl (974 kB)
#15 12.45 Collecting numpy==1.14.5
#15 12.45 Downloading numpy-1.14.5.zip (4.9 MB)
#15 12.45 Collecting setuptools
#15 12.45 Downloading setuptools-50.3.2-py3-none-any.whl (785 kB)
#15 12.45 Collecting setuptools_scm
#15 12.45 Downloading setuptools_scm-4.1.2-py2.py3-none-any.whl (27 kB)
#15 12.45 Collecting wheel
#15 12.45 Using cached wheel-0.35.1-py2.py3-none-any.whl (33 kB)
#15 12.45 Building wheels for collected packages: numpy
#15 12.45 Building wheel for numpy (setup.py): started
#15 12.45 Building wheel for numpy (setup.py): finished with status 'error'
#15 12.45 ERROR: Command errored out with exit status 1:
#15 12.45 command: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-_awsg160/numpy/setup.py'"'"'; __file__='"'"'/tmp/pip-install-_awsg160/numpy/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-rrkjosdy
#15 12.45 cwd: /tmp/pip-install-_awsg160/numpy/
#15 12.45 Complete output (323 lines):
#15 12.45 Running from numpy source directory.
#15 12.45 blas_opt_info:
#15 12.45 blas_mkl_info:
#15 12.45 customize UnixCCompiler
#15 12.45 libraries mkl_rt not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/']
#15 12.45 NOT AVAILABLE
#15 12.45
#15 12.45 blis_info:
#15 12.45 customize UnixCCompiler
#15 12.45 libraries blis not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/']
#15 12.45 NOT AVAILABLE
#15 12.45
#15 12.45
#15 12.45 /usr/lib/python3.6/distutils/dist.py:261: UserWarning: Unknown distribution option: 'define_macros'
#15 12.45 warnings.warn(msg)
#15 12.45 running bdist_wheel
#15 12.45 running build
#15 12.45 running config_cc
#15 12.45 unifing config_cc, config, build_clib, build_ext, build commands --compiler options
#15 12.45 running config_fc
#15 12.45 unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options
#15 12.45 running build_src
#15 12.45 build_src
#15 12.45 building py_modules sources
#15 12.45 creating build
#15 12.45 creating build/src.linux-x86_64-3.6
#15 12.45 creating build/src.linux-x86_64-3.6/numpy
#15 12.45 creating build/src.linux-x86_64-3.6/numpy/distutils
#15 12.45 building library "npymath" sources
#15 12.45 get_default_fcompiler: matching types: '['gnu95', 'intel', 'lahey', 'pg', 'absoft', 'nag', 'vast', 'compaq', 'intele', 'intelem', 'gnu', 'g95', 'pathf95', 'nagfor']'
#15 12.45 customize Gnu95FCompiler
#15 12.45 Could not locate executable gfortran
#15 12.45 Could not locate executable f95
#15 12.45 customize IntelFCompiler
#15 12.45 Could not locate executable ifort
#15 12.45 Could not locate executable ifc
#15 12.45 customize LaheyFCompiler
#15 12.45 Could not locate executable lf95
#15 12.45 customize PGroupFCompiler
#15 12.45 Could not locate executable pgfortran
#15 12.45 customize AbsoftFCompiler
#15 12.45 Could not locate executable f90
#15 12.45 Could not locate executable f77
#15 12.45 customize NAGFCompiler
#15 12.45 customize VastFCompiler
#15 12.45 customize CompaqFCompiler
#15 12.45 Could not locate executable fort
#15 12.45 customize IntelItaniumFCompiler
#15 12.45 Could not locate executable efort
#15 12.45 Could not locate executable efc
#15 12.45 customize IntelEM64TFCompiler
#15 12.45 customize GnuFCompiler
#15 12.45 Could not locate executable g77
#15 12.45 customize G95FCompiler
#15 12.45 Could not locate executable g95
#15 12.45 customize PathScaleFCompiler
#15 12.45 Could not locate executable pathf95
#15 12.45 customize NAGFORCompiler
#15 12.45 Could not locate executable nagfor
#15 12.45 don't know how to compile Fortran code on platform 'posix'
#15 12.45 C compiler: gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -DTHREAD_STACK_SIZE=0x100000 -fPIC
#15 12.45
#15 12.45 compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/usr/include/python3.6m -c'
#15 12.45 gcc: _configtest.c
#15 12.45 gcc _configtest.o -o _configtest
#15 12.45 success!
#15 12.45 removing: _configtest.c _configtest.o _configtest
#15 12.45 C compiler: gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -DTHREAD_STACK_SIZE=0x100000 -fPIC
#15 12.45
#15 12.45 compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/usr/include/python3.6m -c'
#15 12.45 gcc: _configtest.c
#15 12.45 _configtest.c:1:5: warning: conflicting types for built-in function 'exp' [-Wbuiltin-declaration-mismatch]
#15 12.45 int exp (void);
#15 12.45 ^~~
#15 12.45 gcc _configtest.o -o _configtest
#15 12.45 success!
#15 12.45 removing: _configtest.c _configtest.o _configtest
#15 12.45 creating build/src.linux-x86_64-3.6/numpy/core
#15 12.45 creating build/src.linux-x86_64-3.6/numpy/core/src
#15 12.45 creating build/src.linux-x86_64-3.6/numpy/core/src/npymath
#15 12.45 conv_template:> build/src.linux-x86_64-3.6/numpy/core/src/npymath/npy_math_internal.h
#15 12.45 adding 'build/src.linux-x86_64-3.6/numpy/core/src/npymath' to include_dirs.
#15 12.45 conv_template:> build/src.linux-x86_64-3.6/numpy/core/src/npymath/ieee754.c
#15 12.45 conv_template:> build/src.linux-x86_64-3.6/numpy/core/src/npymath/npy_math_complex.c
#15 12.45 None - nothing done with h_files = ['build/src.linux-x86_64-3.6/numpy/core/src/npymath/npy_math_internal.h']
#15 12.45 building library "npysort" sources
#15 12.45 creating build/src.linux-x86_64-3.6/numpy/core/src/npysort
#15 12.45 conv_template:> build/src.linux-x86_64-3.6/numpy/core/src/npysort/quicksort.c
#15 12.45 conv_template:> build/src.linux-x86_64-3.6/numpy/core/src/npysort/mergesort.c
#15 12.45 conv_template:> build/src.linux-x86_64-3.6/numpy/core/src/npysort/heapsort.c
#15 12.45 creating build/src.linux-x86_64-3.6/numpy/core/src/private
#15 12.45 conv_template:> build/src.linux-x86_64-3.6/numpy/core/src/private/npy_partition.h
#15 12.45 adding 'build/src.linux-x86_64-3.6/numpy/core/src/private' to include_dirs.
#15 12.45 conv_template:> build/src.linux-x86_64-3.6/numpy/core/src/npysort/selection.c
#15 12.45 conv_template:> build/src.linux-x86_64-3.6/numpy/core/src/private/npy_binsearch.h
#15 12.45 conv_template:> build/src.linux-x86_64-3.6/numpy/core/src/npysort/binsearch.c
#15 12.45 None - nothing done with h_files = ['build/src.linux-x86_64-3.6/numpy/core/src/private/npy_partition.h', 'build/src.linux-x86_64-3.6/numpy/core/src/private/npy_binsearch.h']
#15 12.45 building extension "numpy.core._dummy" sources
#15 12.45 Generating build/src.linux-x86_64-3.6/numpy/core/include/numpy/config.h
#15 12.45 C compiler: gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -DTHREAD_STACK_SIZE=0x100000 -fPIC
#15 12.45
#15 12.45 compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/usr/include/python3.6m -c'
#15 12.45 gcc: _configtest.c
#15 12.45 _configtest.c:1:10: fatal error: Python.h: No such file or directory
#15 12.45 #include <Python.h>
#15 12.45 ^~~~~~~~~~
#15 12.45 compilation terminated.
#15 12.45 _configtest.c:1:10: fatal error: Python.h: No such file or directory
#15 12.45 #include <Python.h>
#15 12.45 ^~~~~~~~~~
#15 12.45 compilation terminated.
#15 12.45 failure.
#15 12.45 removing: _configtest.c _configtest.o
#15 12.45 Traceback (most recent call last):
#15 12.45 File "<string>", line 1, in <module>
#15 12.45 File "/tmp/pip-install-_awsg160/numpy/setup.py", line 394, in <module>
#15 12.45 setup_package()
#15 12.45 File "/tmp/pip-install-_awsg160/numpy/setup.py", line 386, in setup_package
#15 12.45 setup(**metadata)
#15 12.45 File "/tmp/pip-install-_awsg160/numpy/numpy/distutils/core.py", line 169, in setup
#15 12.45 return old_setup(**new_attr)
#15 12.45 File "/usr/lib/python3.6/site-packages/setuptools/__init__.py", line 143, in setup
#15 12.45 return distutils.core.setup(**attrs)
#15 12.45 File "/usr/lib/python3.6/distutils/core.py", line 148, in setup
#15 12.45 dist.run_commands()
#15 12.45 File "/usr/lib/python3.6/distutils/dist.py", line 955, in run_commands
#15 12.45 self.run_command(cmd)
#15 12.45 File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
#15 12.45 cmd_obj.run()
#15 12.45 File "/usr/lib/python3.6/site-packages/wheel/bdist_wheel.py", line 290, in run
#15 12.45 self.run_command('build')
#15 12.45 File "/usr/lib/python3.6/distutils/cmd.py", line 313, in run_command
#15 12.45 self.distribution.run_command(command)
#15 12.45 File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
#15 12.45 cmd_obj.run()
#15 12.45 File "/tmp/pip-install-_awsg160/numpy/numpy/distutils/command/build.py", line 47, in run
#15 12.45 old_build.run(self)
#15 12.45 File "/usr/lib/python3.6/distutils/command/build.py", line 135, in run
#15 12.45 self.run_command(cmd_name)
#15 12.45 File "/usr/lib/python3.6/distutils/cmd.py", line 313, in run_command
#15 12.45 self.distribution.run_command(command)
#15 12.45 File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
#15 12.45 cmd_obj.run()
#15 12.45 File "/tmp/pip-install-_awsg160/numpy/numpy/distutils/command/build_src.py", line 148, in run
#15 12.45 self.build_sources()
#15 12.45 File "/tmp/pip-install-_awsg160/numpy/numpy/distutils/command/build_src.py", line 165, in build_sources
#15 12.45 self.build_extension_sources(ext)
#15 12.45 File "/tmp/pip-install-_awsg160/numpy/numpy/distutils/command/build_src.py", line 324, in build_extension_sources
#15 12.45 sources = self.generate_sources(sources, ext)
#15 12.45 File "/tmp/pip-install-_awsg160/numpy/numpy/distutils/command/build_src.py", line 377, in generate_sources
#15 12.45 source = func(extension, build_dir)
#15 12.45 File "numpy/core/setup.py", line 422, in generate_config_h
#15 12.45 moredefs, ignored = cocache.check_types(config_cmd, ext, build_dir)
#15 12.45 File "numpy/core/setup.py", line 48, in check_types
#15 12.45 out = check_types(*a, **kw)
#15 12.45 File "numpy/core/setup.py", line 281, in check_types
#15 12.45 "install {0}-dev|{0}-devel.".format(python))
#15 12.45 SystemError: Cannot compile 'Python.h'. Perhaps you need to install python-dev|python-devel.
#15 12.45 ----------------------------------------
#15 12.45 ERROR: Failed building wheel for numpy
#15 12.45 Running setup.py clean for numpy
#15 12.45 ERROR: Command errored out with exit status 1:
#15 12.45 command: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-_awsg160/numpy/setup.py'"'"'; __file__='"'"'/tmp/pip-install-_awsg160/numpy/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' clean --all
#15 12.45 cwd: /tmp/pip-install-_awsg160/numpy
#15 12.45 Complete output (10 lines):
#15 12.45 Running from numpy source directory.
#15 12.45
#15 12.45 `setup.py clean` is not supported, use one of the following instead:
#15 12.45
#15 12.45 - `git clean -xdf` (cleans all files)
#15 12.45 - `git clean -Xdf` (cleans all versioned files, doesn't touch
#15 12.45 files that aren't checked into the git repo)
#15 12.45
#15 12.45 Add `--force` to your command to use it anyway if you must (unsupported).
#15 12.45
#15 12.45 ----------------------------------------
#15 12.45 ERROR: Failed cleaning build dir for numpy
#15 12.45 Failed to build numpy
#15 12.45 Installing collected packages: cython, numpy, setuptools, setuptools-scm, wheel
#15 12.45 Running setup.py install for numpy: started
#15 12.45 Running setup.py install for numpy: finished with status 'error'
#15 12.45 ERROR: Command errored out with exit status 1:
#15 12.45 command: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-_awsg160/numpy/setup.py'"'"'; __file__='"'"'/tmp/pip-install-_awsg160/numpy/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-0kkz6dsp/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-iz28sqos/overlay --compile --install-headers /tmp/pip-build-env-iz28sqos/overlay/include/python3.6m/numpy
#15 12.45 cwd: /tmp/pip-install-_awsg160/numpy/
#15 12.45 Complete output (317 lines):
#15 12.45 Running from numpy source directory.
#15 12.45
#15 12.45 Note: if you need reliable uninstall behavior, then install
#15 12.45 with pip instead of using `setup.py install`:
#15 12.45
#15 12.45 - `pip install .` (from a git repo or downloaded source
#15 12.45 release)
#15 12.45 - `pip install numpy` (last NumPy release on PyPi)
#15 12.45
#15 12.45
#15 12.45 blas_opt_info:
#15 12.45 blas_mkl_info:
#15 12.45 customize UnixCCompiler
#15 12.45 libraries mkl_rt not found in ['/usr/local/lib', '/usr/lib', '/usr/lib/']
#15 12.45 NOT AVAILABLE
#15 12.45
#15 12.45 blis_info:
#15 12.45
#15 12.45 /tmp/pip-install-_awsg160/numpy/numpy/distutils/system_info.py:624: UserWarning:
#15 12.45 Lapack (http://www.netlib.org/lapack/) sources not found.
#15 12.45 Directories to search for the sources can be specified in the
#15 12.45 numpy/distutils/site.cfg file (section [lapack_src]) or by setting
#15 12.45 the LAPACK_SRC environment variable.
#15 12.45 self.calc_info()
#15 12.45 NOT AVAILABLE
#15 12.45
#15 12.45 /usr/lib/python3.6/distutils/dist.py:261: UserWarning: Unknown distribution option: 'define_macros'
#15 12.45 warnings.warn(msg)
#15 12.45 running install
#15 12.45 running build
#15 12.45 running config_cc
#15 12.45 unifing config_cc, config, build_clib, build_ext, build commands --compiler options
#15 12.45 running config_fc
#15 12.45 unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options
#15 12.45 running build_src
#15 12.45 build_src
#15 12.45 building py_modules sources
#15 12.45 building library "npymath" sources
#15 12.45 get_default_fcompiler: matching types: '['gnu95', 'intel', 'lahey', 'pg', 'absoft', 'nag', 'vast', 'compaq', 'intele', 'intelem', 'gnu', 'g95', 'pathf95', 'nagfor']'
#15 12.45 customize Gnu95FCompiler
#15 12.45 Could not locate executable gfortran
#15 12.45 customize G95FCompiler
#15 12.45 Could not locate executable g95
#15 12.45 customize PathScaleFCompiler
#15 12.45 Could not locate executable pathf95
#15 12.45 customize NAGFORCompiler
#15 12.45 Could not locate executable nagfor
#15 12.45 don't know how to compile Fortran code on platform 'posix'
#15 12.45 C compiler: gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -DTHREAD_STACK_SIZE=0x100000 -fPIC
#15 12.45
#15 12.45 compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/usr/include/python3.6m -c'
#15 12.45 gcc: _configtest.c
#15 12.45 gcc _configtest.o -o _configtest
#15 12.45 success!
#15 12.45 removing: _configtest.c _configtest.o _configtest
#15 12.45 C compiler: gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -DTHREAD_STACK_SIZE=0x100000 -fPIC
#15 12.45
#15 12.45 compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/usr/include/python3.6m -c'
#15 12.45 adding 'build/src.linux-x86_64-3.6/numpy/core/src/private' to include_dirs.
#15 12.45 None - nothing done with h_files = ['build/src.linux-x86_64-3.6/numpy/core/src/private/npy_partition.h', 'build/src.linux-x86_64-3.6/numpy/core/src/private/npy_binsearch.h']
#15 12.45 building extension "numpy.core._dummy" sources
#15 12.45 Generating build/src.linux-x86_64-3.6/numpy/core/include/numpy/config.h
#15 12.45 C compiler: gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -DTHREAD_STACK_SIZE=0x100000 -fPIC
#15 12.45
#15 12.45 compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/usr/include/python3.6m -c'
#15 12.45 gcc: _configtest.c
#15 12.45 _configtest.c:1:10: fatal error: Python.h: No such file or directory
#15 12.45 #include <Python.h>
#15 12.45 ^~~~~~~~~~
#15 12.45 compilation terminated.
#15 12.45 _configtest.c:1:10: fatal error: Python.h: No such file or directory
#15 12.45 #include <Python.h>
#15 12.45 ^~~~~~~~~~
#15 12.45 compilation terminated.
#15 12.45 failure.
#15 12.45 removing: _configtest.c _configtest.o
#15 12.45 Traceback (most recent call last):
#15 12.45 File "<string>", line 1, in <module>
#15 12.45 File "/tmp/pip-install-_awsg160/numpy/setup.py", line 394, in <module>
#15 12.45 setup_package()
run
#15 12.45 r = self.setuptools_run()
#15 12.45 File "/tmp/pip-install-_awsg160/numpy/numpy/distutils/command/install.py", line 36, in setuptools_run
#15 12.45 return distutils_install.run(self)
#15 12.45 File "/usr/lib/python3.6/distutils/command/install.py", line 545, in run
#15 12.45 self.run_command('build')
#15 12.45 File "/usr/lib/python3.6/distutils/cmd.py", line 313, in run_command
#15 12.45 self.distribution.run_command(command)
#15 12.45 File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
#15 12.45 cmd_obj.run()
#15 12.45 old_build.run(self)
#15 12.45 File "/usr/lib/python3.6/distutils/command/build.py", line 135, in run
#15 12.45 self.run_command(cmd_name)
#15 12.45 File "/usr/lib/python3.6/distutils/cmd.py", line 313, in run_command
#15 12.45 self.distribution.run_command(command)
#15 12.45 File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
#15 12.45 cmd_obj.run()
#15 12.45 File "/tmp/pip-install-_awsg160/numpy/numpy/distutils/command/build_src.py", line 148, in run
#15 12.45 self.build_sources()
#15 12.45 File "/tmp/pip-install-_awsg160/numpy/numpy/distutils/command/build_src.py", line 165, in build_sources
#15 12.45 self.build_extension_sources(ext)
#15 12.45 File "/tmp/pip-install-_awsg160/numpy/numpy/distutils/command/build_src.py", line 324, in build_extension_sources
#15 12.45 sources = self.generate_sources(sources, ext)
#15 12.45 File "/tmp/pip-install-_awsg160/numpy/numpy/distutils/command/build_src.py", line 377, in generate_sources
#15 12.45 source = func(extension, build_dir)
#15 12.45 File "numpy/core/setup.py", line 422, in generate_config_h
#15 12.45 moredefs, ignored = cocache.check_types(config_cmd, ext, build_dir)
#15 12.45 File "numpy/core/setup.py", line 48, in check_types
#15 12.45 out = check_types(*a, **kw)
#15 12.45 File "numpy/core/setup.py", line 281, in check_types
#15 12.45 "install {0}-dev|{0}-devel.".format(python))
#15 12.45 SystemError: Cannot compile 'Python.h'. Perhaps you need to install python-dev|python-devel.
#15 12.45 ----------------------------------------
#15 12.45 ERROR: Command errored out with exit status 1: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-_awsg160/numpy/setup.py'"'"'; __file__='"'"'/tmp/pip-install-_awsg160/numpy/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-0kkz6dsp/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-iz28sqos/overlay --compile --install-headers /tmp/pip-build-env-iz28sqos/overlay/include/python3.6m/numpy Check the logs for full command output.
#15 12.45 ----------------------------------------
#15 12.45 ERROR: Command errored out with exit status 1: /usr/bin/python3 /usr/lib/python3.6/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-iz28sqos/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'cython >= 0.29' 'numpy==1.14.5; python_version<'"'"'3.7'"'"'' 'numpy==1.16.0; python_version>='"'"'3.7'"'"'' setuptools setuptools_scm wheel Check the logs for full command output.
------
failed to solve with frontend dockerfile.v0: failed to build LLB: executor failed running [/bin/sh -c python3 -m pip install --no-cache --no-deps pyarrow]: runc did not terminate sucessfully
C:\Users\j.crespoguzman\OneDrive - Pamoja\Documents\spark_intento1>docker build -t intento1 .
I am trying to use pip3 like this:
FROM spark-py:latest
COPY *.jar /opt/spark/jars/
RUN rm /opt/spark/jars/kubernetes-*-4.1.2.jar
RUN apk add --no-cache \
build-base \
cmake \
bash \
boost-dev \
autoconf \
zlib-dev \
flex \
bison \
g++
RUN apk add py3-pip
RUN apk add --no-cache --virtual .build-deps g++ python3-dev libffi-dev openssl-dev && \
apk add --no-cache --update python3 && \
pip3 install --upgrade pip setuptools
RUN pip3 install --no-cache-dir six pytest cython
RUN apk add --update --no-cache py3-numpy
ENV PYTHONPATH=/usr/lib/python3.7/site-packages
RUN python3 -m pip install --upgrade pip
RUN pip install wheel
RUN pip install cython
COPY requirements.txt .
RUN python3 -m pip install --no-cache --no-deps pyarrow
but I have all the time this error:
#10 4.030 /bin/sh: pip3: command not foundAlso, I have done this docker file following this Github issue: https://github.com/apache/arrow/issues/2080
and
https://github.com/mbrobbel/libarrow/blob/0.10.0/Dockerfile
I have tried with version 0.10.0 but it was failing so what I did is this:
FROM spark-py:latest
COPY *.jar /opt/spark/jars/
RUN rm /opt/spark/jars/kubernetes-*-4.1.2.jar
ENV ARROW_VERSION 0.16.0
ENV BUILD_PACKAGES git make cmake g++
ENV RUNTIME_PACKAGES boost-dev
RUN apk add --no-cache $BUILD_PACKAGES $RUNTIME_PACKAGES && \
git clone \
--single-branch \
--depth 1 \
--branch apache-arrow-$ARROW_VERSION \
https://github.com/apache/arrow && \
cd arrow && \
cmake \
-DCMAKE_BUILD_TYPE=Release \
-DARROW_BUILD_TESTS=Off \
cpp/ && \
make && make install && \
cd .. && rm -rf arrow && \
apk del $BUILD_PACKAGES
The image is built correctly but if I execute it and try to import pyarrow I obtain the same error:
ModuleNotFoundError: No module named 'pyarrow'In the pod terminal if I do pip list I obtain:
ash-4.4# pip list
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
Package Version
---------- -------
pip 20.2.4
setuptools 44.1.1