I'm trying to run multiple repetitive tasks in parallel under python. I'm pretty new to multiprocessing but as all the tasks are independant, I used the simple following piece of code :
import numpy as np
import sys
import os
import glob
import matplotlib.pyplot as plt
import concurrent.futures as Cfut
def analize(simul, N_thread):
path = os.getcwd()
print('Analizing topo ...')
Data = output of some calculations
print('Writing Data ...')
np.save('Data', Data)
print('Data saved')
print('Printing figures')
plt.figure()
plt.plot(Data[0])
plt.savefig('figure1.pdf')
plt.clf()
plt.plot(Data[0])
plt.savefig('figure1.pdf')
plt.close('all')
print('figures saved')
os.chdir(path
if __name__ == '__main__':
list_simul = sorted(glob.glob('Layer*'))
executor = Cfut.ProcessPoolExecutor(5)
#executor = Cfut.ThreadPoolExecutor(N_jobs)
futures = [executor.submit(analize, item, 10) for item in list_simul]
Cfut.wait(futures)
It ran very well during 3 months, and since since morning I get sometimes the following error :
Exception ignored in: <bound method Image.__del__ of <tkinter.PhotoImage object at 0x7fac6c187f98>>
Traceback (most recent call last):
File "/usr/lib/python3.5/tkinter/__init__.py", line 3359, in __del__
self.tk.call('image', 'delete', self.name)
RuntimeError: main thread is not in main loop
What is odd is that some jobs are done without any problem, and it happens not all the time. With a bit of research, I found a lot of posts with this problem, all of them were related to GUI. I understood the problem, but I guess I should be able to find a workaround since I'm not showing any figure, just creating the object and saving it, and since all the tasks are independent.
Any hint ?
Answers
matplotlib uses TkAgg for default. Backends like TkAgg, FltkAgg, GTK, GTKAgg, GTKCairo, Wx, and WxAgg are all GUI based. And most GUI backends require being run from the main thread. Thus, if you are running on an environment without GUI, it will throw RuntimeError: main thread is not in main loop.
Therefore, just switch to backends that do not use GUI: Agg, Cairo, PS, PDF, or SVG.
For example:
import matplotlib.pyplot as plt
plt.switch_backend('agg')
