Collectives™ on Stack Overflow
Find centralized, trusted content and collaborate around the technologies you use most.
Learn more about Collectives
Teams
Q&A for work
Connect and share knowledge within a single location that is structured and easy to search.
Learn more about Teams
CNNNet=[]
img = cv2.imread('../datasets/1.jpg')
sym, arg_params, aux_params = mx.model.load_checkpoint('det1', 0)
for i in range(num_worker):
worker_net = mx.mod.Module(symbol=sym,label_names=None)
worker_net.bind(data_shapes=[('data', (1, 3, 1000, 1000))],for_training=False)
worker_net.set_params(arg_params,aux_params)
CNNNet.append(worker_net)
pool = Pool(num_worker)
threshold = 0.6
res = pool.map(do_work_warpper,zip(repeat(img),CNNNet[:num_worker],repeat(threshold)))
and the
do_work_warpper()
function is:
def do_work_warpper(args):
return do_work(*args)
def do_work(img,net,threshold):
#do image predict job here
return res
I am puzzled by the question that when using multiprocessing.Pool with the mx.mod.Module object, I get the error in python3.6:
TypeError: can't pickle module objects
or in python2.7:
PicklingError: Can't pickle <type 'module'>: attribute lookup __builtin__.module failed
any suggestion will be appreciated.
The reason why you're getting this exception is because multiprocessing needs to be able to pickle the variables you pass to your workers in order to pass them between various processes it spawns.
The error:
TypeError: can't pickle module objects
Suggests that one of the variables you're passing to your Pool contains a module (or a class that has a module as an attribute).
To demonstrate the issue, have a look at these two classes:
import os
class Pickable:
a = 1
class UnPickable:
def __init__(self):
self.mod = os
If you try to pickle instances of these two classes, you'll get:
In [11]: pickle.dumps(Pickable())
Out[11]: b'\x80\x03c__main__\nPickable\nq\x00)\x81q\x01.'
In [10]: pickle.dumps(UnPickable())
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-10-7d4d725a1c6d> in <module>()
----> 1 pickle.dumps(UnPickable())
TypeError: can't pickle module objects
That being said - either you create your own class that mimicks the functionality of mx.mod.Module, but is serializable - OR (better solution in my opinion) use simple (https://docs.python.org/3.1/library/pickle.html#what-can-be-pickled-and-unpickled) python-builtin types to pass variables into workers of your Pool and construct mx.mod.Module instances inside them on their own.
–
–
–
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
