K
Q

Pod error - Failed building wheel for fastparquet

11/22/2019

I am following link: https://kubernetes.dask.org/en/latest/, to run dask array on Kubernetes cluster.

Steps:

  1. Installed Kubernetes on 3 nodes(1 Master and 2 workers).
  2. install miniconda3
  3. pip install dask-kubernetes
  4. dask_example.py with code to run dask array (same as example given on link)
  5. Worker-spec.yml file with pod configuration (same as example given on link)

While running the example code, the worker pod is showing error status as below:

(base) [root@k8s-master example]# kubectl logs workerpod
+ '[' '' ']'
+ '[' -e /opt/app/environment.yml ']'
+ echo 'no environment.yml'
no environment.yml
+ '[' '' ']'
+ '[' 'fastparquet git+https://github.com/dask/distributed' ']'
+ echo 'EXTRA_PIP_PACKAGES environment variable found.  Installing.'
+ /opt/conda/bin/pip install fastparquet git+https://github.com/dask/distributed
EXTRA_PIP_PACKAGES environment variable found.  Installing.
Collecting git+https://github.com/dask/distributed
  Cloning https://github.com/dask/distributed to /tmp/pip-req-build-9pgvdhjf
  Running command git clone -q https://github.com/dask/distributed /tmp/pip-req-build-9pgvdhjf
Collecting fastparquet
  Downloading https://files.pythonhosted.org/packages/58/49/dccb790fa17ab3fbf84a6b848050083c7a1899e9586000e34e3e4fbf5538/fastparquet-0.3.2.tar.gz (151kB)
Requirement already satisfied: click>=6.6 in /opt/conda/lib/python3.7/site-packages (from distributed==2.8.0+8.g5b33d54c) (7.0)
Requirement already satisfied: cloudpickle>=0.2.2 in /opt/conda/lib/python3.7/site-packages (from distributed==2.8.0+8.g5b33d54c) (1.2.2)
Requirement already satisfied: dask>=2.7.0 in /opt/conda/lib/python3.7/site-packages (from distributed==2.8.0+8.g5b33d54c) (2.8.0)
Requirement already satisfied: msgpack in /opt/conda/lib/python3.7/site-packages (from distributed==2.8.0+8.g5b33d54c) (0.6.2)
Requirement already satisfied: psutil>=5.0 in /opt/conda/lib/python3.7/site-packages (from distributed==2.8.0+8.g5b33d54c) (5.6.5)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /opt/conda/lib/python3.7/site-packages (from distributed==2.8.0+8.g5b33d54c) (2.1.0)
Requirement already satisfied: tblib in /opt/conda/lib/python3.7/site-packages (from distributed==2.8.0+8.g5b33d54c) (1.4.0)
Requirement already satisfied: toolz>=0.7.4 in /opt/conda/lib/python3.7/site-packages (from distributed==2.8.0+8.g5b33d54c) (0.10.0)
Requirement already satisfied: tornado>=5 in /opt/conda/lib/python3.7/site-packages (from distributed==2.8.0+8.g5b33d54c) (6.0.3)
Requirement already satisfied: zict>=0.1.3 in /opt/conda/lib/python3.7/site-packages (from distributed==2.8.0+8.g5b33d54c) (1.0.0)
Requirement already satisfied: pyyaml in /opt/conda/lib/python3.7/site-packages (from distributed==2.8.0+8.g5b33d54c) (5.1.2)
Requirement already satisfied: pandas>=0.19 in /opt/conda/lib/python3.7/site-packages (from fastparquet) (0.25.2)
Collecting numba>=0.28 (from fastparquet)
  Downloading https://files.pythonhosted.org/packages/57/66/7ebc88e87b4ddf9b1c204e24d467cb0d13a7a890ecb4f83d20949f768929/numba-0.46.0-cp37-cp37m-manylinux1_x86_64.whl (3.6MB)
Requirement already satisfied: numpy>=1.11 in /opt/conda/lib/python3.7/site-packages (from fastparquet) (1.17.3)
Collecting thrift>=0.11.0 (from fastparquet)
  Downloading https://files.pythonhosted.org/packages/97/1e/3284d19d7be99305eda145b8aa46b0c33244e4a496ec66440dac19f8274d/thrift-0.13.0.tar.gz (59kB)
Requirement already satisfied: six in /opt/conda/lib/python3.7/site-packages (from fastparquet) (1.12.0)
Requirement already satisfied: heapdict in /opt/conda/lib/python3.7/site-packages (from zict>=0.1.3->distributed==2.8.0+8.g5b33d54c) (1.0.1)
Requirement already satisfied: pytz>=2017.2 in /opt/conda/lib/python3.7/site-packages (from pandas>=0.19->fastparquet) (2019.3)
Requirement already satisfied: python-dateutil>=2.6.1 in /opt/conda/lib/python3.7/site-packages (from pandas>=0.19->fastparquet) (2.8.1)
Collecting llvmlite>=0.30.0dev0 (from numba>=0.28->fastparquet)
  Downloading https://files.pythonhosted.org/packages/1f/3e/642ffb29ed35ca5e93f171ba327452bdee81ec76f2d711ef0f15b411928a/llvmlite-0.30.0-cp37-cp37m-manylinux1_x86_64.whl (20.2MB)
Building wheels for collected packages: fastparquet, distributed, thrift
  Building wheel for fastparquet (setup.py): started
  Building wheel for fastparquet (setup.py): finished with status 'error'
  ERROR: Command errored out with exit status 1:
   command: /opt/conda/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-m6berq44/fastparquet/setup.py'"'"'; __file__='"'"'/tmp/pip-install-m6berq44/fastparquet/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-tcex8z1e --python-tag cp37
       cwd: /tmp/pip-install-m6berq44/fastparquet/
  Complete output (61 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.7
  creating build/lib.linux-x86_64-3.7/fastparquet
  copying fastparquet/__init__.py -> build/lib.linux-x86_64-3.7/fastparquet
  copying fastparquet/api.py -> build/lib.linux-x86_64-3.7/fastparquet
  copying fastparquet/compression.py -> build/lib.linux-x86_64-3.7/fastparquet
  copying fastparquet/converted_types.py -> build/lib.linux-x86_64-3.7/fastparquet
  copying fastparquet/core.py -> build/lib.linux-x86_64-3.7/fastparquet
  copying fastparquet/dataframe.py -> build/lib.linux-x86_64-3.7/fastparquet
  copying fastparquet/encoding.py -> build/lib.linux-x86_64-3.7/fastparquet
  copying fastparquet/schema.py -> build/lib.linux-x86_64-3.7/fastparquet
  copying fastparquet/thrift_structures.py -> build/lib.linux-x86_64-3.7/fastparquet
  copying fastparquet/util.py -> build/lib.linux-x86_64-3.7/fastparquet
  copying fastparquet/writer.py -> build/lib.linux-x86_64-3.7/fastparquet
  running egg_info
  writing fastparquet.egg-info/PKG-INFO
  writing dependency_links to fastparquet.egg-info/dependency_links.txt
  writing requirements to fastparquet.egg-info/requires.txt
  writing top-level names to fastparquet.egg-info/top_level.txt
  reading manifest file 'fastparquet.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  no previously-included directories found matching 'docs/_build'
  writing manifest file 'fastparquet.egg-info/SOURCES.txt'
  copying fastparquet/parquet.thrift -> build/lib.linux-x86_64-3.7/fastparquet
  copying fastparquet/speedups.c -> build/lib.linux-x86_64-3.7/fastparquet
  copying fastparquet/speedups.pyx -> build/lib.linux-x86_64-3.7/fastparquet
  creating build/lib.linux-x86_64-3.7/fastparquet/benchmarks
  copying fastparquet/benchmarks/columns.py -> build/lib.linux-x86_64-3.7/fastparquet/benchmarks
  creating build/lib.linux-x86_64-3.7/fastparquet/parquet_thrift
  copying fastparquet/parquet_thrift/__init__.py -> build/lib.linux-x86_64-3.7/fastparquet/parquet_thrift
  creating build/lib.linux-x86_64-3.7/fastparquet/parquet_thrift/parquet
  copying fastparquet/parquet_thrift/parquet/__init__.py -> build/lib.linux-x86_64-3.7/fastparquet/parquet_thrift/parquet
  copying fastparquet/parquet_thrift/parquet/constants.py -> build/lib.linux-x86_64-3.7/fastparquet/parquet_thrift/parquet
  copying fastparquet/parquet_thrift/parquet/ttypes.py -> build/lib.linux-x86_64-3.7/fastparquet/parquet_thrift/parquet
  creating build/lib.linux-x86_64-3.7/fastparquet/test
  copying fastparquet/test/__init__.py -> build/lib.linux-x86_64-3.7/fastparquet/test
  copying fastparquet/test/test_api.py -> build/lib.linux-x86_64-3.7/fastparquet/test
  copying fastparquet/test/test_aroundtrips.py -> build/lib.linux-x86_64-3.7/fastparquet/test
  copying fastparquet/test/test_compression.py -> build/lib.linux-x86_64-3.7/fastparquet/test
  copying fastparquet/test/test_converted_types.py -> build/lib.linux-x86_64-3.7/fastparquet/test
  copying fastparquet/test/test_dataframe.py -> build/lib.linux-x86_64-3.7/fastparquet/test
  copying fastparquet/test/test_encoding.py -> build/lib.linux-x86_64-3.7/fastparquet/test
  copying fastparquet/test/test_output.py -> build/lib.linux-x86_64-3.7/fastparquet/test
  copying fastparquet/test/test_partition_filters_specialstrings.py -> build/lib.linux-x86_64-3.7/fastparquet/test
  copying fastparquet/test/test_read.py -> build/lib.linux-x86_64-3.7/fastparquet/test
  copying fastparquet/test/test_schema.py -> build/lib.linux-x86_64-3.7/fastparquet/test
  copying fastparquet/test/test_speedups.py -> build/lib.linux-x86_64-3.7/fastparquet/test
  copying fastparquet/test/test_thrift_structures.py -> build/lib.linux-x86_64-3.7/fastparquet/test
  copying fastparquet/test/test_util.py -> build/lib.linux-x86_64-3.7/fastparquet/test
  copying fastparquet/test/test_with_n.py -> build/lib.linux-x86_64-3.7/fastparquet/test
  copying fastparquet/test/util.py -> build/lib.linux-x86_64-3.7/fastparquet/test
  running build_ext
  building 'fastparquet.speedups' extension
  creating build/temp.linux-x86_64-3.7
  creating build/temp.linux-x86_64-3.7/fastparquet
  gcc -pthread -B /opt/conda/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/opt/conda/include/python3.7m -I/opt/conda/lib/python3.7/site-packages/numpy/core/include -c fastparquet/speedups.c -o build/temp.linux-x86_64-3.7/fastparquet/speedups.o
  unable to execute 'gcc': No such file or directory
  error: command 'gcc' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for fastparquet
  Running setup.py clean for fastparquet
  Building wheel for distributed (setup.py): started
  Building wheel for distributed (setup.py): finished with status 'done'
  Created wheel for distributed: filename=distributed-2.8.0+8.g5b33d54c-cp37-none-any.whl size=568764 sha256=9712974396e1221fa5dd195616e85031da70894222c2c7ff574bcfb318b5f80c
  Stored in directory: /tmp/pip-ephem-wheel-cache-v5jnd4bs/wheels/aa/21/a7/d9548d684f8e074360b7ad1bd8633843dba9658288b68b3dd5
  Building wheel for thrift (setup.py): started
  Building wheel for thrift (setup.py): finished with status 'done'
  Created wheel for thrift: filename=thrift-0.13.0-cp37-none-any.whl size=154884 sha256=c32af6aa5c4cfced68fadc2997e173f14ed0595a4bb9bb407eb7ef62794fafd8
  Stored in directory: /root/.cache/pip/wheels/02/a2/46/689ccfcf40155c23edc7cdbd9de488611c8fdf49ff34b1706e
Successfully built distributed thrift
Failed to build fastparquet
Installing collected packages: llvmlite, numba, thrift, fastparquet, distributed
  Running setup.py install for fastparquet: started
    Running setup.py install for fastparquet: finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /opt/conda/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-m6berq44/fastparquet/setup.py'"'"'; __file__='"'"'/tmp/pip-install-m6berq44/fastparquet/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-qrgpngpc/install-record.txt --single-version-externally-managed --compile
         cwd: /tmp/pip-install-m6berq44/fastparquet/
    Complete output (19 lines):
    running install
    running build
    running build_py
    running egg_info
    writing fastparquet.egg-info/PKG-INFO
    writing dependency_links to fastparquet.egg-info/dependency_links.txt
    writing requirements to fastparquet.egg-info/requires.txt
    writing top-level names to fastparquet.egg-info/top_level.txt
    reading manifest file 'fastparquet.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    no previously-included directories found matching 'docs/_build'
    writing manifest file 'fastparquet.egg-info/SOURCES.txt'
    running build_ext
    building 'fastparquet.speedups' extension
    creating build/temp.linux-x86_64-3.7
    creating build/temp.linux-x86_64-3.7/fastparquet
    gcc -pthread -B /opt/conda/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/opt/conda/include/python3.7m -I/opt/conda/lib/python3.7/site-packages/numpy/core/include -c fastparquet/speedups.c -o build/temp.linux-x86_64-3.7/fastparquet/speedups.o
    unable to execute 'gcc': No such file or directory
    error: command 'gcc' failed with exit status 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /opt/conda/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-m6berq44/fastparquet/setup.py'"'"'; __file__='"'"'/tmp/pip-install-m6berq44/fastparquet/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-qrgpngpc/install-record.txt --single-version-externally-managed --compile Check the logs for full command output.
+ exec dask-worker --nthreads 2 --no-bokeh --memory-limit 6GB --death-timeout 60
/opt/conda/lib/python3.7/site-packages/distributed/cli/dask_worker.py:252: UserWarning: The --bokeh/--no-bokeh flag has been renamed to --dashboard/--no-dashboard.
  "The --bokeh/--no-bokeh flag has been renamed to --dashboard/--no-dashboard. "
distributed.nanny - INFO -         Start Nanny at: 'tcp://10.44.0.1:46597'
distributed.worker - INFO -       Start worker at:      tcp://10.44.0.1:36897
distributed.worker - INFO -          Listening to:      tcp://10.44.0.1:36897
distributed.worker - INFO - Waiting to connect to:    tcp://172.16.0.76:42143
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          2
distributed.worker - INFO -                Memory:                    6.00 GB
distributed.worker - INFO -       Local Directory:           /worker-vlqvrk15
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - Waiting to connect to:    tcp://172.16.0.76:42143
distributed.worker - INFO - Waiting to connect to:    tcp://172.16.0.76:42143
distributed.worker - INFO - Waiting to connect to:    tcp://172.16.0.76:42143
distributed.worker - INFO - Waiting to connect to:    tcp://172.16.0.76:42143
distributed.worker - INFO - Waiting to connect to:    tcp://172.16.0.76:42143
distributed.nanny - INFO - Closing Nanny at 'tcp://10.44.0.1:46597'
distributed.worker - INFO - Stopping worker at tcp://10.44.0.1:36897
distributed.worker - INFO - Closed worker has not yet started: None
distributed.dask_worker - INFO - Timed out starting worker
distributed.dask_worker - INFO - End worker
-- Manvi
dask
dask-distributed
fastparquet
kubernetes
python

Similar Questions

KubernetesPodOperator privileged security_context in Airflow
Java-client: Best way to check operation delete operation completion?
How do I execute dynamic code in a container?
OpenShift: change ConfigMap in java runtime
helm install | Error: failed pre-install: job failed: BackoffLimitExceeded

1 Answer

11/26/2019

Adding below env resolved the issue:

env:
      - name: EXTRA_APT_PACKAGES
        value: gcc
-- Manvi
Source: StackOverflow