ZING-43245: parallelize file signing

[KostashchukIryna]

Tested in Dev comparing with current master version running. Used identical CDF version ranges to get the same number of rows. Verified that the number of rows is the same to check if there is a regression. So I saw an improvement in time spent on signing (- ~70%) based on perf logs that were added before. It's more noticable change in smaller queries though, as in big ones cdfReplay and cdcSpecBuild are still quite high. Maybe we should look if that can be optimized

[Copilot]

Pull request overview

Parallelizes URL signing for table and CDF queries to reduce latency.

Changes:

• Adds parallel signing using a shared thread pool.
• Preserves action ordering and pagination.
• Adds configurable signing concurrency.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
DeltaSharedTable.scala	Implements parallel file signing.
DeltaSharingService.scala	Configures the shared signing pool.
ServerConfig.scala	Adds the pool-size setting.
manifests/base/configmap.yaml	Sets deployment pool size to 32.

Suppressed comments (1)

server/src/main/scala/io/delta/standalone/internal/DeltaSharedTable.scala:1223

• newFixedThreadPool bounds only the worker count; its LinkedBlockingQueue is unbounded. Because parallelSign eagerly submits one future per file (up to 10,000 per request), concurrent requests can accumulate an unbounded backlog, retain all query state, and starve later requests. Timed-out futures also remain queued and continue running. Use bounded submission with explicit backpressure/rejection handling, or process each request in bounded batches.

  // Shared, bounded thread pool for parallel file signing across all tables/requests.
  private lazy val signingExecutionContext: ExecutionContext = ExecutionContext.fromExecutorService(
    java.util.concurrent.Executors.newFixedThreadPool(signingThreadPoolSize.get()))

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.