Cache is very slow

[Napolitain]

Description

Caching some very lightweight files with this syntax
path/to/folder/**/*.yaml

Version

Nightly

Operating system

Ubuntu

Experiments Enabled

No response

Example Taskfile

[andreynering]

@Napolitain How many files you have in path/to/folder? Is it a big directory? How many files? Also, how long it takes to run a task?

[Napolitain]

@Napolitain How many files you have in path/to/folder? Is it a big directory? How many files? Also, how long it takes to run a task?

I have created a benchmark PR for trying to add a baseline and then I think we can find some areas of improvements. I think there are many redundancy and allocs.

The benchmark is 20k small files and a few large files. It shows that many small files is dramatically slower than few large files. Of course, it is expected, but that doesn't mean there is no issues either. Maybe with the benchmark then we can try to improve speed against baseline.

I have tried more simple commands like sha256sum and it is really much faster (again, to be expected, but nonetheless, there may be area of improvements).

[trulede]

@Napolitain I'm developing a concurrent-streaming-zeroalloc algo for timestamping. The technique is to create a ReadDir func and inject that into the existing glob mechanism, which effectively bypasses all the allocations and allows early exit if a newer file is found. Since ReadDir loads file info, the repeated calls to os.Stat are no longer required.

I don't really trust the results, some of the boilerplate code is AI generated, but it seems like a good strategy ... but also requires more tests probably. At least one test is failing .... so there may be more effort to get the algorithm correct.

The gist is here:
https://gist.github.com/trulede/bb3f23892b116a91bbfc21a665d5353b

The results:

Benchmark Configuration Tier	Original Strategy	Interceptor Strategy	Real World
2,000 Files / Depth 1	16.02 ms	0.049 ms	~320x Faster
2,000 Files / Depth 10	19.11 ms	0.054 ms	~350x Faster
20,000 Files / Depth 1	252.81 ms	0.117 ms	~2,160x Faster
20,000 Files / Depth 10 (0% Chg)	249.79 ms	0.138 ms	~1,800x Faster
20,000 Files / Depth 10 (1% Chg)	252.64 ms	0.062 ms	~4,000x Faster

Allocations were reduced from 84Mb to 12.7Kb (20.000 files).