Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.

Teams

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

I'm trying to run multiple repetitive tasks in parallel under python. I'm pretty new to multiprocessing but as all the tasks are independant, I used the simple following piece of code :

    import numpy as np
    import sys
    import os
    import glob
    import matplotlib.pyplot as plt
    import concurrent.futures as Cfut
    def analize(simul, N_thread):
        path = os.getcwd()
        print('Analizing topo ...')
        Data = output of some calculations
        print('Writing Data ...')
        np.save('Data', Data)
        print('Data saved')
        print('Printing figures')
        plt.figure()
        plt.plot(Data[0])
        plt.savefig('figure1.pdf')
        plt.clf()
        plt.plot(Data[0])
        plt.savefig('figure1.pdf')
        plt.close('all')
        print('figures saved')
        os.chdir(path
if __name__ == '__main__':
    list_simul = sorted(glob.glob('Layer*'))
    executor = Cfut.ProcessPoolExecutor(5)
   #executor = Cfut.ThreadPoolExecutor(N_jobs)
    futures = [executor.submit(analize, item, 10) for item in list_simul]
    Cfut.wait(futures)
It ran very well during 3 months, and since since morning I get sometimes the following error :
Exception ignored in: <bound method Image.__del__ of <tkinter.PhotoImage object at 0x7fac6c187f98>>
    Traceback (most recent call last):
      File "/usr/lib/python3.5/tkinter/__init__.py", line 3359, in __del__
        self.tk.call('image', 'delete', self.name)
    RuntimeError: main thread is not in main loop
What is odd is that some jobs are done without any problem, and it happens not all the time. With a bit of research, I found a lot of posts with this problem, all of them were related to GUI. I understood the problem, but I guess I should be able to find a workaround since I'm not showing any figure, just creating the object and saving it, and since all the tasks are independent.
Any hint ?
As seen here, have you tried adding the following code to the top of your imports?
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt
matplotlib uses TkAgg for default. Backends like TkAgg, FltkAgg, GTK, GTKAgg, GTKCairo, Wx, and WxAgg are all GUI based. And most GUI backends require being run from the main thread. Thus, if you are running on an environment without GUI, it will throw RuntimeError: main thread is not in main loop.
Therefore, just switch to backends that do not use GUI: Agg, Cairo, PS, PDF, or SVG.
For example:
import matplotlib.pyplot as plt
plt.switch_backend('agg')
References: https://matplotlib.org/stable/faq/howto_faq.html#work-with-threads
        
Thanks for contributing an answer to Stack Overflow!
Please be sure to answer the question. Provide details and share your research!
But avoid …
Asking for help, clarification, or responding to other answers.
Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.