Skip to content

Parallel Write performance improvements#695

Draft
MarkWolters wants to merge 4 commits into
mainfrom
io_improvements
Draft

Parallel Write performance improvements#695
MarkWolters wants to merge 4 commits into
mainfrom
io_improvements

Conversation

@MarkWolters

@MarkWolters MarkWolters commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

This PR updates the graph parallel write process to take full advantage of the async write process. Whereas before although the write tasks were batched and handed off to achieve better performance they were not truly asynchronous. This update changes that to allow fully async graph index writes and increases the benefit of using parallel writes significantly. As an example prior to this update graph write time showed speedups of 4x - 8x using parallel writes

=== Testing Write Performance ===
Writing with NVQ + FUSED_ADC features...
Sequential write: 6074.18 ms
Parallel write:   1373.73 ms
Speedup:          4.42x

After this update the speedup is more along the lines of 24x - 32x

=== Testing Write Performance ===
Writing with NVQ + FUSED_ADC features...
Sequential write: 20147.29 ms
Parallel write:   627.52 ms
Speedup:          32.11x

This PR also introduces the ability to specify parallel writes as part of the test configuration through yaml files which is standard for BenchYAML and AutoBenchYAML test. The new parameter parallelGraphConstruction has been added to ConstructionParameters as a Boolean value. If set to Yes Grid will use parallel writes in graph construction. If set to No Grid will use the legacy serial writing pattern. If the parameter is omitted it will default to the legacy serial writes during graph construction.

@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Before you submit for review:

  • Does your PR follow guidelines from CONTRIBUTIONS.md?
  • Did you summarize what this PR does clearly and concisely?
  • Did you include performance data for changes which may be performance impacting?
  • Did you include useful docs for any user-facing changes or features?
  • Did you include useful javadocs for developer oriented changes, explaining new concepts or key changes?
  • Did you rebase your branch onto the latest main for regression testing and PR submission?
  • Did you trigger regression testing via Run Bench Main and review results?
  • Did you adhere to the code formatting guidelines (TBD)
  • Did you group your changes for easy review, providing meaningful descriptions for each commit?
  • Did you ensure that all files contain the correct copyright header?
  • Did you add documentation for this feature to the release notes directory?

If you did not complete any of these, then please explain below.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant