Help with RabbitMQ and the amount of queues created

[maxacarvalho]

Shlink version

v4.4.4

PHP version

8.4.4

How do you serve Shlink

Docker image

Database engine

MySQL

Database version

8.4.3

Summary

Hi,

I have the need to use the RabbitMQ integration so I can process visits to the short ulrs.
One thing that's bothering me is the amount of queues that are going to be created.

Although I see the use for queues pointing to specific short_code, I generally only need the "global" one, https://shlink.io/new-visit.

The fact that every click creates a new queue on Rabbit means that during a month's period I'm going to end up having hundreds of thousands of queues, I don't think that it will scale well enough.

I wonder if there's an alternative to that approach, a way to only create the main https://shlink.io/new-visit queue?

Thank you.

[acelaya]

The fact that every click creates a new queue on Rabbit

That's not correct. Shlink will create one queue per short URL and a global one, so that you can subscribe to visits globally or per short URL.

So you'll end up with as many queues as short urls you create.

[maxacarvalho]

Hi @acelaya
Sorry, I didn't explain myself properly.
That's exactly what I meant. For every short URL a queue will be created. So during a month, if I create 400 thousand short URLs and those get clicked at least once, I'll have 400 thousands queues created on RabbitMQ instance.

The messages on those queues will never be consumed since I only process messages from the global one.
If I have 400 thousand short URLs per month I'll end up with 4 800 000 queues by the end of an year.

Yes, I can do maintenance on the RabbitMQ server and clean up those queues, I was just wondering if there's an option to turn that behaviour of. I tried to investigate the code but didn't find anything yet.

[acelaya]

In that case you are correct, and I'm afraid Shlink does not provide a mechanism to disable that at the moment.

I think it should be relatively simple to allow this to be configured, and reasonable enough.

As a reference, this article with common mistakes explains it's better to span your messages into multiple queues, but does in fact mention having thousands upon thousands of queues can impact CPU usage, and also RAM, if you never consume their messages.

https://www.cloudamqp.com/blog/part1-rabbitmq-best-practice.html#number-of-queues

In general, Shlink has been historically optimized for a larger number of visits, but a lesser number of urls. However, this is definitely a supported use case and the intention is to polish rough edges as they are found.

[maxacarvalho]

Thank for your help here @acelaya
I'll implement a script that deletes the unnecessary (my use case) queues every X period of time. It should be enough.

[acelaya]

I have created an issue to track a potential improvement for this #2387