Collectives™ on Stack Overflow
Find centralized, trusted content and collaborate around the technologies you use most.
Learn more about Collectives
Teams
Q&A for work
Connect and share knowledge within a single location that is structured and easy to search.
Learn more about Teams
I read that message queue is preferred over
subprocess.Popen()
. It is said that message queue is scalable solution. I want to understand how is it so.
I just want to list the benefits of message queue over
subeprocess.Popen()
so that I can convince my superiors to use message queue instead of
subprocess
These are completely separate and different things.
subprocess.Popen()
simply spawns (by calling
fork
and
exec
) new OS process for a specific command you passed to it. So it's perfect for cases when you need something to execute in separate process and (optionally) get result of execution (in somewhat awkward way, via pipe).
Queues (like Celery or ActiveJob) gives you two main things:
Storage (more precisely, an interface to some existing storage, like PostgreSQL or MongoDB) for your tasks (or messages), that are going to be serialized automatically before going to that storage.
Workers that polling this storage and actually perform those tasks (deserializing them before performing, also automatically).
So, it's possible to have a lot of workers, even maybe in distributed environment. It gives you not only a vertical scalability but a horizontal one (by keeping your workers on separate machines).
On the other hand, queues are more suited for asynchronous processing (i.e. for jobs that need to be executed later and you don't need results right now) and are more heavyweight than simple process spawning.
So, if you have simple one-off jobs just to be executed somewhere outside your main process - use processes.
If you have a bunch of different jobs that are needed to be executed asynchronously and you want to have an ability to scale that process, you should use queues, they'll make life easier.
Thanks for contributing an answer to Stack Overflow!
-
Please be sure to
answer the question
. Provide details and share your research!
But
avoid
…
-
Asking for help, clarification, or responding to other answers.
-
Making statements based on opinion; back them up with references or personal experience.
To learn more, see our
tips on writing great answers
.
