Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.

Teams

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

I am running multiple python ML jobs at same time. Most of the time my python script runs fine but when there are too many jobs starting at the same time, it fails with this error.

It is also hard to reproduce this error as I am unable to find the cause of it. The file is present at the location.

Error:

Intel MKL FATAL ERROR: Cannot load /usr/local/miniconda-2.7.13/lib/python2.7/site-packages/numpy/core/../../../../libmkl_rt.so.
                I have the same issue, but with unittesting together with Apache Spark. When I run each test separately, they all work. But when I run the entire test suite, the first test works, the rest does not. Seems to be that concurrent access is a problem?!
– akoeltringer
                Aug 31, 2018 at 9:01
First I created a new conda virtual enviornment to use with multithreading:
conda create -n nomkl_env python=3.7 numpy nomkl
I activated this environment and ran this command via the conda MKL docs:
conda install numpy scipy scikit-learn numexpr
Now I am able to run multiple instances of Python at the same time. 
Earlier when I ran conda remove mkl mkl-service it somehow removed python from the bin of an existing virutal env. This was sub-optimal. I would up creating a dedicated conda env and this did the trick. 
        
Thanks for contributing an answer to Stack Overflow!
Please be sure to answer the question. Provide details and share your research!
But avoid …
Asking for help, clarification, or responding to other answers.
Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.