Collectives™ on Stack Overflow
Find centralized, trusted content and collaborate around the technologies you use most.
Learn more about Collectives
Teams
Q&A for work
Connect and share knowledge within a single location that is structured and easy to search.
Learn more about Teams
Ask Question
I got this error when running training a deep learning model and although looking at many solutions over the Internet, they did not help me. The log is as follows:
Traceback (most recent call last):
File "main.py", line 208, in <module>
main()
File "main.py", line 100, in main
model = nn.DataParallel(model).cuda()
File "/home/dexter/miniconda3/envs/VideoSum/lib/python3.5/site-packages/torch/nn/parallel/data_parallel.py", line 105, in __init__
self.module.cuda(device_ids[0])
File "/home/dexter/miniconda3/envs/VideoSum/lib/python3.5/site-packages/torch/nn/modules/module.py", line 249, in cuda
return self._apply(lambda t: t.cuda(device))
File "/home/dexter/miniconda3/envs/VideoSum/lib/python3.5/site-packages/torch/nn/modules/module.py", line 176, in _apply
module._apply(fn)
File "/home/dexter/miniconda3/envs/VideoSum/lib/python3.5/site-packages/torch/nn/modules/rnn.py", line 112, in _apply
self.flatten_parameters()
File "/home/dexter/miniconda3/envs/VideoSum/lib/python3.5/site-packages/torch/nn/modules/rnn.py", line 78, in flatten_parameters
if not any_param.is_cuda or not torch.backends.cudnn.is_acceptable(any_param):
File "/home/dexter/miniconda3/envs/VideoSum/lib/python3.5/site-packages/torch/backends/cudnn/__init__.py", line 87, in is_acceptable
if _libcudnn() is None:
File "/home/dexter/miniconda3/envs/VideoSum/lib/python3.5/site-packages/torch/backends/cudnn/__init__.py", line 58, in _libcudnn
'but linked against {}'.format(compile_version, __cudnn_version))
RuntimeError: cuDNN version mismatch: PyTorch was compiled against 7102 but linked against 7604
The question is, no matter the versions showed in the log are, 7.6.4 is my cudnn version and 7.1.2 is the cudnn version that the code is originally compiled. What I need is just downgrade (or upgrade my current cudnn version) by:
conda install cudnn=7.1.2
It works, if any, please correct me.
I had a similar problem...downgrading cudnn
worked for me. As mentioned here, the error message
RuntimeError: cuDNN version mismatch: PyTorch was compiled against 7102 but linked against 7604
...indicates Pytorch was linked to a newer version of the cudnn library. It only caused a problem deep in my model. I would guess it was something of an archaic Pytorch function which wasn't updated recently. After downgrading cudnn to the same version as Pytorch, it worked again...no other noticeable loss in performance.
conda install cudnn=7.1.2
Sorry to be repetitive, but my StackOverflow rep is too low to add a comment.
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.