fix(selectivity): reduce master-thread RAM usage#1209
Conversation
commit: |
| isCachedOnFs(source) ? getCachedSelectivityFile(CacheType.Asset, url) : source, | ||
| isCachedOnFs(sourceMaps) | ||
| ? getCachedSelectivityFile(CacheType.Asset, sourceMapUrl as string) |
There was a problem hiding this comment.
These 2 should be added to "p-limit" at the same event loop iteration, so scripts would be processed one by one instead of "all fetch source first, then all fetch source maps"
|
|
||
| const hashPromise = provider.calculateForFile(filePath); | ||
|
|
||
| await new Promise(setImmediate); |
There was a problem hiding this comment.
"p-limit" does not run passed functions instantly
We need to wait for the next event loop iteration
| shouldDisableBrowserSelectivity.cache.clear?.(); | ||
| shouldDisableTestBySelectivity.cache.clear?.(); |
There was a problem hiding this comment.
We dont need this cache after we started tests
| startOffsets: number[], | ||
| sourceRoot: string, | ||
| ): Promise<Set<string>> => { | ||
| ): Set<string> => { |
| export class HashProvider { | ||
| private static readonly _fileHashStore: Map<string, Promise<string>> = new Map(); | ||
| private static readonly _patternHashStore: Map<string, Promise<string>> = new Map(); | ||
| private static readonly _limited = pLimit(10); |
There was a problem hiding this comment.
Limiting "calculating hash" operation because node.js can buffer file contents (even if they are processed as stream) if cpu is busy (for example, by calculating md5)
|
|
||
| this._processingTestPromises.push( | ||
| (async (): Promise<[Test, string] | null> => { | ||
| this._processingTestLimit(async () => { |
There was a problem hiding this comment.
limiting "checking if test changed" operation because it reads test dependencies file and file can be buffered
| // https://nodejs.org/api/cli.html#uv_threadpool_sizesize | ||
| const libUVLimited = pLimit((process.env.UV_THREADPOOL_SIZE && Number(process.env.UV_THREADPOOL_SIZE)) || 16); |
There was a problem hiding this comment.
There is no point in reading/writing more than "UV_THREADPOOL_SIZE" files at once, because libuv creates 4 threads
By default we set limit to 16, because node.js has its own queue and we want to make sure when libuv thread completes the task, it immediately takes the other one, prepared by node.js
KuznetsovRoman
force-pushed
the
TESTPLANE-903.restrict_selectivity_fs_parallelism
branch
from
89e2c4e to
d989eaf
Compare
KuznetsovRoman
deleted the
TESTPLANE-903.restrict_selectivity_fs_parallelism
branch
2 participants
No description provided.