Django + Gunicorn: How to go about running shared theading.Thread() + subprocess.popen() on startup

相关文章推荐

干练的煎饼果子 · 如何在Django模板中使用jQuery捕获 ...· 1 月前 ·

斯文的杯子 · Python+Django 核心介绍 - ...· 1 周前 ·

不开心的烤面包 · Unity3D图形界面与Django ...· 1 周前 ·

严肃的橡皮擦 · linux发送post请求json数据格式_ ...· 8 月前 ·

一身肌肉的烤红薯 · 无限遍历，Python实现在多维嵌套字典、列 ...· 8 月前 ·

曾深爱过的黄瓜 · mysql ...· 1 年前 ·

强悍的核桃 · 在服务器的同一IP地址和端口中绑定多站点但证 ...· 3 年前 ·

Django + Gunicorn: How to go about running shared theading.Thread() + subprocess.popen() on startup #3202 Django + Gunicorn: How to go about running shared theading.Thread() + subprocess.popen() on startup #3202

I have a django backend that, at startup, does two things:

Runs a


    threading.Thread()

that calls a function every 10 seconds to read RPi's sensor information and saves it to database.

Runs a


    subprocess.Popen()

to call a binary linux program ( go2rtc to be specific) that starts the streaming service indefinetely until it's manually stopped.

These calls are performed in each django app's AppConfig def ready() method.

When running django's development server, everything works as expected. However, when running through Gunicorn with multiple workers, things get a bit messy.

I'd like some help in the following:

With multiple workers, the def ready() method gets called for each worker, resulting in redundant creation of threads and Popen processes. Using --preload seems to solve the issue. However, the threading.Thread() output to journalctl seems to be blocked; only shows in journalctl after gunicorn is stopped/restared. What is going on here?

I need the Thread and Popen instances to be accessible by all workers so that when I send a stop request from the frontend, these threads/processes can halt gracefully. How would I go about sharing these instances between all the workers?

My current gunicorn service is as follows:

[Service]
User=pi
Group=www-data
WorkingDirectory=/home/pi/homecontrol
ExecStart=/home/pi/homecontrol/venv/bin/gunicorn \
          --access-logfile - \
          --workers 3 \
          --bind unix:/run/gunicorn.sock \
          --preload \
          homecontrol.wsgi:application
Appreciate some insights on this.
        Sounds like there is nothing in the threads dealing with external sensors/executables that require it to share process/thread context with the HTTP serving, other than managing start/stop of dependent services. Which is one of the key features of systemd, which you already use.
Why have it interfere with your flexibility in controlling HTTP thread/process management, then? Wrap it inside what Django calls "management commands" and have those launched by entirely separate systemd services, declaring dependencies as needed to have systemd manage launching / stopping it.
should I store the PID
I don't, but have not spent much thought on that. In my case, not a performance issue to query (by name, or to systemd) every time.

Note that retrieving the PID and then separately signalling (potentially since-recycled) that PID is a race condition. If you want to store it, minimize that window in the service crash case by storing the PID in a text file, and instructing systemd to delete that file once the service exits (using PIDFile=).
        Querying the PID by service name seems to be the best way to go.
If you don't mind helping with the other issue now that the subprocess issue is out of the question.
At startup of my django project, I instantiate a pyhap  driver object, which creates homekit accessories and broadcast the service to the network.
Unlike the subprocess binary program, this object must stay within django in order to dynamically exchange information with the front end. Again, in my MVP django project, the driver instance lives in the program as a global variable, which the frontend can access through API endpoints to start/stop/update using the driver's instance methods. Also, the driver object needs to continuously make calls to the django's DB in order to make changes as necessary.
What is the best practice for sharing the driver object between workers? I've read about Celery and Redis, but not sure if that's the easiest way to go about it.
        gthreads might actually be the most suitable answer to all my concerns!
I went ahead and read about the different worker types. Ended up specifying only 1 worker, 8 threads, and removed the preload directive; this caused everything to work as expected. I'm quite new to gunicorn so I'm not sure about the side effects this have over running single-threaded multiple workers, but this will do for now.
gunicorn --bind 0.0.0.0:8000 --workers 1 --thread 8 --access-logfile - homecontrol.wsgi:application --graceful-timeout 5
I'm sure that it'd be wiser to adjust my django project for separate django commands for scalability, but that's a problem for another day.
Thanks a lot @pajod for the great support; very much appreciated.