feat: add --bfs option to mc mirror for layer-by-layer traversal …

[Laisky]

Community Contribution License

All community contributions in this pull request are licensed to the project maintainers
under the terms of the Apache 2 license.
By creating this pull request I represent that I have the right to license the
contributions to the project maintainers under the Apache 2 license.

Description

In some cases, a Minio bucket can contain so many files that the server struggles to perform the list objects recursively operation, resulting in a complete freeze of mc mirror. This pull request introduces the --bfs parameter, enabling mc mirror to traverse folders layer by layer, thus avoiding overwhelming the server by listing all files at once.

closes #4873

Motivation and Context

discuss in #4873 (comment)

How to test this PR?

mc mirror source target --bfs --dry

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Optimization (provides speedup with no functional changes)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• Fixes a regression (If yes, please add commit-id or PR # here)
• Unit tests added/updated
• Internal documentation updated
• Create a documentation update request here

[harshavardhana]

@Laisky what kind of benchmarks have you done to show the difference?

[harshavardhana]

you have to test with URLs @Laisky just thinking that base and element are just paths is not going to work.

[harshavardhana]

@Laisky can you let met know why this change is needed?

[klauspost]

I don't really see why this would be any faster than a recursive listing. This is pretty much what is done on the server side. This just adds several requests instead of one.