Are there max requests settings for Daphne?

[alamorre]

I have this ASGI applicate with Django and Daphne, and I'm using the following command.

daphne -b 0.0.0.0 -p $PORT server.asgi:application

Is there a way to add --max-requests and --max-requests-jitter to actively prevent memory leaks?

Maybe something like:
daphne -b 0.0.0.0 -p $PORT --max-requests 1000 --max-requests-jitter 50 server.asgi:application

[carltongibson]

Not currently, but it might be a good addition. 🤔

This is ref:

django/channels#1789

Were you able to investigate why you're seeing a leak there? Also did it stabilise at 4%? (In which case it may be the Python GC rather than a leak per se — for which it'd keep going up.)

[alamorre]

It's really a verbatim implementation of the tutorial project in the docs. In production it leaks in the picture below.

That's why i think --max-requests with daphne would be awesome. We can let an image teardown before consuming too many resources.

[alamorre]

I created this draft PR #405

Please let me know if this is of interest. I'll polish the code and make some unit tests 👍

[WellingtonNico]

hello guys, i'm using gunicorn now, it has an option to choose the class of workers, in my case i use gthreads(green thread), is there an option like this in daphne?

[carltongibson]

@WellingtonNico No. It wouldn't make much sense. ASGI uses asyncio, with multiple requests handled in the single event loop. You can run multiple instances of Daphne (but you should be a while before you'd need to do that...)

[carltongibson]

The PR in #405 adds a shutdown, but it doesn't reload.

For that we'd need something like Django autoreloader's restart_with_reloader():

https://github.com/django/django/blob/747b417a220b0412ed806001a383959449aac6da/django/utils/autoreload.py#L270-L276

If someone wanted to take that on, it would be worth a look.

[ghomem]

Hi,

I was reviewing the possibility of replacing public nginx -> nginx+gunicorn+daphne with simply public nginx -> daphne, with daphne as the main, single process, entry point. In theory this is a simplification, because several moving parts are removed.

I found several issues where memory consumption is discussed but all of them seem inconclusive.

I have indeed observed that memory allocated with a large upload is not deallocated, at least immediately. This may or may not be the correct behaviour, but I could not find a discussion with the definitive answer.

Having a --max-requests flag would help, but only if we had several daphne "workers" in parallel, because otherwise, there would be a service interruption, right?

(unlike gunicorn where the main process forks and provides the forked workers, which are then recycled after max-requests)

The documentation says this:

if you want to bind multiple Daphne instances to the same port on a machine, use a process supervisor that can listen on >ports and pass the file descriptors to launched processes, and then pass the file descriptor with --fd NUM.

So... is the need for --max-requests confirmed?

Will this be implemented?

What is the recommended supervisor for multiple daphne processes?

Thanks in advance!

[carltongibson]

Hi @ghomem.

There's no real plan here as it doesn't make much sense to implement a full supervisor process in Daphne itself (and you're right about the forking issue).

File uploads are a tricky one. Happy to look at anything if you can dig down.

Sorry to not have more to say (but that's why it's sat here.)

[ghomem]

So does daphne not make sense for serving generic traffic ( websocket and non-websocket related), behind a public secure nginx with SSL termination?

There seems to be a lot of interest in this but I don't find the discussions conclusive. And it does work, but don't know if it survives production scenarios.

I looks like the intention of this ticket was making daphne more production ready.

If daphne is not mature for this, should it only be used for websockets and should gunicorn be used for the other cases?

Thanks for the support.

[carltongibson]

@ghomem You're asking questions for which there are no one-size fits all answers. As always, "it depends" — measure and see.

Daphne is perfectly fine for production scenarios. It's built on twisted, which is as battle tested as they come, and is stable and mature.

The intention of this ticket is based on a misconception: that resetting the server is a stable strategy. (It's not.) Rather, whilst it's possible we could find a way to reduce memory usage when handling file uploads, what you're seeing here is a symptom of the way ASGI treats request bodies, and you'll find all ASGI servers have some difficulties with file uploads — both from resource usage and even loop blocking on processing.

The solution folks usually go to is to use pre-signed URLs to do client-side uploads directly to object storage (so bypassing the application entirely). If you need to handle big files, that's likely the most common approach.

An alternative is to use WSGI for parts of your application and ASGI for others. WSGI, whilst using a worker-per-connection model, has known scaling patterns for these kind of scenarios, so may what you need. (Although, still, sending giant files through your Python application makes little sense when there are alternatives.)

This isn't really an appropriate location for these kind of discussions — they're only disguised as issues about Daphne. (I suspect a good chat with an LLM would help you work out your thinking.) Good luck!

[ghomem]

client-side uploads directly to object storage (so bypassing the application entirely).

I know about this, but the "it depends" also applies here. Maintenance of N moving parts / multiple ways of doing X / number of failure modes are also points of concern for system engineering.

I think it is fair to conclude from your comment that ASGI might not be ideal for file uploads as of today but also that YMMV and that one needs to test.

This isn't really an appropriate location for these kind of discussions — they're only disguised as issues about Daphne

Perhaps they are only borderline Daphne issues but your comment helped me and I suspect others will also benefit from it.

Thank you very much for taking your time to share your perspective @carltongibson .