I am trying to run a bit of python code that uses tensorflow-gpu. However, when the process tries to run, I'm getting the following error:
2018-04-13 20:03:49.215876: E tensorflow/stream_executor/cuda/cuda_dnn.cc:396] Loaded runtime CuDNN library: 7102 (compatibility version 7100) but source was compiled with 7005 (compatibility version 7000). If using a binary install, upgrade your CuDNN library to match. If building from sources, make sure the library loaded at runtime matches a compatible version specified during compile configuration.
2018-04-13 20:03:49.220783: F tensorflow/core/kernels/conv_ops.cc:712] Check failed: stream->parent()->GetConvolveAlgorithms( conv_parameters.ShouldIncludeWinogradNonfusedAlgo(), &algorithms)
However, I typed env and it listed CUDNN_VERSION=7.0.5.15 and LD_LIBRARY_PATH=/usr/local/cuda/extras/CUPTI/lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
I installed cudnn 7.0.5 by downloading and copying the relevant files into /user/local/cuda/
Why is this error occurring? I am using a kubernetes backed cluster
I fixed it using this post from askubuntu
Pasting the instructions from that post here:
Step 0: Install cuda from the standard repositories. (See How can I install CUDA on Ubuntu 16.04?)
Step 1: Register an nvidia developer account and download cudnn here (about 80 MB)
Step 2: Check where your cuda installation is. For the installation from the repository it is /usr/lib/... and /usr/include. Otherwise, it will be /usr/local/cuda/ or /usr/local/cuda-<version>. You can check it with which nvcc or ldconfig -p | grep cuda
Step 3: Copy the files:
$ cd folder/extracted/contents
$ sudo cp -P include/cudnn.h /usr/include
$ sudo cp -P lib64/libcudnn* /usr/lib/x86_64-linux-gnu/
$ sudo chmod a+r /usr/lib/x86_64-linux-gnu/libcudnn*Basically, on the cudnn installation instruction, it only tells you to copy your cudnn.h and libcudnn* files to the cuda folder. However, in addition to that, one also needs to copy those files inside the systems main include and lib64 folders. That will fix this problem.