DAOS-18633 rebuild: global barrier for ec agg by wangshilong · Pull Request #17861 · daos-stack/daos

wangshilong · 2026-04-01T14:18:24Z

Pause EC agg globally before triggering rebuild and set rebuild fence after we confirmed all ec agg has been paused.

After rebuild, we don't need trigger full scan for ec agg any more, because we gurantee there is no ec aggregation after rebuild fence.

Steps for the author:

Commit message follows the guidelines.
Appropriate Features or Test-tag pragmas were used.
Appropriate Functional Test Stages were run.
At least two positive code reviews including at least one code owner from each category referenced in the PR.
Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

Gatekeeper requested (daos-gatekeeper added as a reviewer).

github-actions · 2026-04-01T14:18:46Z

Errors are Unable to load ticket data
https://daosio.atlassian.net/browse/DAOS-18633

Pause EC agg globally before triggering rebuild and set rebuild fence after we confirmed all ec agg has been paused. After rebuild, we don't need trigger full scan for ec agg any more, because we gurantee there is no ec aggregation after rebuild fence. Signed-off-by: Wang Shilong <shilong.wang@hpe.com>

Signed-off-by: Wang Shilong <shilong.wang@hpe.com>

NiuYawei · 2026-04-02T06:35:13Z

src/rebuild/srv.c

+	}
+
+	if (!dpc->spc_no_storage) {
+		dpc->spc_ec_agg_pause_gate = d_hlc_get();


Instead of acquiring a new HLC, can't we use the 'rebuild version' (pool version) to identify this rebuild?

rebuild_version is not safe, as retied might reuse same rebuild_version.

NiuYawei · 2026-04-02T06:38:10Z

src/rebuild/srv.c

+		dpc->spc_ec_agg_pause_gate = d_hlc_get();
+		D_INFO(DF_RB ": set local EC agg pause gate token " DF_U64 "\n", DP_RB_RPT(rpt),
+		       dpc->spc_ec_agg_pause_gate);
+		ds_cont_child_wait_ec_agg_pause(dpc, rebuild_wait_ec_pause);


This could lead to RPC timeout.

See agg_rate_ctl(), it should check the 'spc_ec_agg_pause_gate' as well? BTW, "rebuild_ver" looks a better name.

Problem is we don't guarantee rebuild_ver always bump for retrying rebuild. I don't want to change that Semantics. Or I bump it locally to ensure it increase every time.

adding spc_ec_agg_pause_gate check to ds_pool_is_rebuilding() can make it affect agg_rate_ctl().

NiuYawei · 2026-04-02T06:41:22Z

src/rebuild/srv.c

+	/* phase-2: broadcast scan RPC to all targets */
+	D_INFO(DF_RB ": start rebuild scan\n", DP_RB_RGT(*p_rgt));
+	rc = rebuild_scan_broadcast(pool, *p_rgt, &task->dst_tgts, task->dst_new_layout_version,
+				    task->dst_rebuild_op, RB_SCAN_START);


Given that we turned to use two rounds of scan broadcast, getting the global 'rebuild epoch' could be easier now, @liuxuezhao ?

right, I also thought that, I can check some details, may push another PR or another commit to this PR later (will confirm with shilong which one is better).

NiuYawei · 2026-04-02T06:43:28Z

src/rebuild/rebuild_internal.h

+rebuild_tgt_prepare_pause(crt_rpc_t *rpc, uint64_t *stable_epoch);
+
+int
+rebuild_tgt_cancel_pause(crt_rpc_t *rpc);


rebuild_tgt_stop_agg()/rebuild_tgt_resume_agg() sounds more straightforward.

liuxuezhao · 2026-04-02T06:57:53Z

src/container/srv_target.c

 	} else {
-		D_ALLOC(snapshots, (cont->sc_snapshots_nr + 1) *
-			sizeof(daos_epoch_t));
+		D_ALLOC(snapshots, (cont->sc_snapshots_nr + 1) * sizeof(daos_epoch_t));


as rebuild_fence removed now, so looks need not + 1, for both L382 and 385

liuxuezhao · 2026-04-02T06:59:24Z

src/container/srv_target.c

 	D_DEBUG(DB_MD, DF_UUID "[%d]: wait for pausing EC aggregation\n",
 		DP_UUID(pool_child->spc_uuid), dss_get_module_info()->dmi_tgt_id);
+	D_INFO(DF_UUID "[%d]: wait for local EC aggregation to pause\n",
+	       DP_UUID(pool_child->spc_uuid), dss_get_module_info()->dmi_tgt_id);


looks fine to use INFO, then L976 can remove

liuxuezhao · 2026-04-02T07:21:34Z

src/rebuild/srv.c

-		dpc->spc_rebuild_end_hlc = d_hlc_get();
+	if (rpt->rt_ec_pause_token == dpc->spc_ec_agg_pause_gate) {
+		dpc->spc_ec_agg_pause_gate = 0;
+		dpc->spc_rebuild_end_hlc   = d_hlc_get();


spc_rebuild_end_hlc is no usage now, looks fine to remove it also.

liuxuezhao · 2026-04-02T07:27:44Z

src/rebuild/srv.c

+		dpc->spc_ec_agg_pause_gate = d_hlc_get();
+		D_INFO(DF_RB ": set local EC agg pause gate token " DF_U64 "\n", DP_RB_RPT(rpt),
+		       dpc->spc_ec_agg_pause_gate);
+		ds_cont_child_wait_ec_agg_pause(dpc, rebuild_wait_ec_pause);


adding spc_ec_agg_pause_gate check to ds_pool_is_rebuilding() can make it affect agg_rate_ctl().

liuxuezhao · 2026-04-02T07:37:50Z

src/rebuild/srv.c

+	/* phase-2: broadcast scan RPC to all targets */
+	D_INFO(DF_RB ": start rebuild scan\n", DP_RB_RGT(*p_rgt));
+	rc = rebuild_scan_broadcast(pool, *p_rgt, &task->dst_tgts, task->dst_new_layout_version,
+				    task->dst_rebuild_op, RB_SCAN_START);


right, I also thought that, I can check some details, may push another PR or another commit to this PR later (will confirm with shilong which one is better).

Signed-off-by: Wang Shilong <shilong.wang@hpe.com>

…uild_ec_barrier

Signed-off-by: Wang Shilong <shilong.wang@hpe.com>

wangshilong · 2026-04-08T10:57:00Z

Replace by #17941

wangshilong force-pushed the shilongw/rebuild_ec_barrier branch from 1be479b to af424f3 Compare April 1, 2026 15:04

Remove rebuild fence

14228ab

Signed-off-by: Wang Shilong <shilong.wang@hpe.com>

NiuYawei reviewed Apr 2, 2026

View reviewed changes

liuxuezhao reviewed Apr 2, 2026

View reviewed changes

Address comments and more cleanups

e1b73d8

Signed-off-by: Wang Shilong <shilong.wang@hpe.com>

wangshilong requested review from NiuYawei and liuxuezhao April 2, 2026 09:01

liuxuezhao previously approved these changes Apr 3, 2026

View reviewed changes

wangshilong added 2 commits April 7, 2026 00:57

Merge branch 'master' of github.com:daos-stack/daos into shilongw/reb…

faf6fc4

…uild_ec_barrier

resolve conflicts

6545896

Signed-off-by: Wang Shilong <shilong.wang@hpe.com>

wangshilong dismissed liuxuezhao’s stale review via 6545896 April 7, 2026 01:22

wangshilong marked this pull request as ready for review April 7, 2026 01:22

wangshilong requested review from a team as code owners April 7, 2026 01:22

NiuYawei approved these changes Apr 7, 2026

View reviewed changes

wangshilong closed this Apr 8, 2026

wangshilong deleted the shilongw/rebuild_ec_barrier branch April 8, 2026 10:57

Conversation

wangshilong commented Apr 1, 2026

Steps for the author:

After all prior steps are complete:

Uh oh!

github-actions bot commented Apr 1, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wangshilong commented Apr 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants