[3.5.0 regression] Rsync fails to run with --max-alloc=0

[realsimix]

Hi,

Our backup system uses --max-alloc=0 to run rsync. This is not possible anymore with rsync 3.5.0 and it says I should specify a very large value instead. So, how can I determine the size of the value I should give. I have absolutely no idea what the right value is.

This is on a server with multi millions of files and running rsync without --max-alloc=0 always fails to run.

Simon

[seks99x]

--max-alloc=0 option was removed on 3.5.0 for security reasons.
You don't need to calculate an exact number! The --max-alloc option does not reserve memory upfront; it just sets a safety ceiling to prevent a single operation from requesting infinite RAM.

[realsimix]

But I have no idea what size to use. All I know is that without --max-alloc=0 the rsync process would terminate before it could finish. Can you tell me what value I should try with?

[seks99x]

I actually do not know your system or how it works or how much files it handles. But as far as i know —max-alloc=0 would leave the system with no limit (and thats one of the reasons it was removed) so you could try a large number normally like —max-alloc=1g (1GB) or more it should not make any difference

[steadytao]

1g is already the default so that will not help here. --max-alloc=0 was removed due to a security concern around memory exhaustion from a hostile peer. That said, an explicitly enormous value can produce much the same result so I do not think making legitimate users guess a replacement value is a good final answer. We should improve the diagnostic to report the attempted allocation and review whether the restriction should apply differently to locally selected and peer-supplied values.

[seks99x]

I agree that we absolutely need to improve the diagnostic to report the attempted allocation size, finding a way to calculate a specific accurate number so we don’t fall into the same vulnerability again (which i hope we could find some time for that).
However, my point was specific to this user's immediate issue. He is already handling multi-million files and was previously running with ⁠--max-alloc=0⁠ (meaning he didn't care about size constraints to begin with). Asking him to calculate a highly specific value right now just isn't realistic for his environment. That's why I was telling him to just not calculate an exact number, if he sets it too tight, his system and backups could become fragile and stop working.

[realsimix]

The source system in my case has 1.4 billion of files and directories. From these, some dozens of millions are synced over WAN to a remote destination. I'm now running it with --max-alloc=10G. I hope it runs as good as it has worked with --max-alloc=0.

[steadytao]

With 1.4 billion entries, exceeding the default per-allocation ceiling is plausible. 10G is still only a limit on one allocation, not total process memory, so it may work but there is no useful way to size it. The man page also still documents 0 as SIZE_MAX, contrary to 3.5.0. I think we should restore zero for trusted local/remote-shell use while preventing an untrusted daemon peer from raising or disabling the administrators ceiling and report the attempted allocation size when the limit is hit.

[steadytao]

Thoughts? @tridge @seks99x - finish your break first; both of you