DAOS-17535 chk: misc improvements for CR logic by Nasf-Fan · Pull Request #17329 · daos-stack/daos

Nasf-Fan · 2025-12-30T17:05:20Z

Include the followings:

When create CHK IV namespace, make the secondary group to be same as
the primary group. Otherwise, CHK logic may hit DER_NONEXIST trouble
when communicate via IV.
Integrate CHK IV namespace create and destroy API, cleanup related
logic, redefine the version.
Get ranks list and IV namespace version from CHK leader when rejoin.
Adjust CHK_REJOIN RPC for related changes.
Remove unsupported functionality for checking the specified 'phase'.
Add new test for case of lost some engine(s) before start checker.
Dedicated ULT to handle dead rank event, that will not be affected
by checker start or stop. Then even if check scheduler exited, the
subsequent check query still can work against the latest rank list.

Test-tag: recovery

Signed-off-by: Fan Yong fan.yong@hpe.com

Steps for the author:

Commit message follows the guidelines.
Appropriate Features or Test-tag pragmas were used.
Appropriate Functional Test Stages were run.
At least two positive code reviews including at least one code owner from each category referenced in the PR.
Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

Gatekeeper requested (daos-gatekeeper added as a reviewer).

github-actions · 2025-12-30T17:08:31Z

Ticket title is 'DAOS checker cannot completed on Aurora after some engines excluded'
Status is 'In Review'
Labels: 'scrubbed_2.6.5'
https://daosio.atlassian.net/browse/DAOS-17535

daosbuild3 · 2025-12-30T17:57:45Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17329/1/testReport/

daosbuild3 · 2025-12-31T03:18:18Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17329/1/execution/node/1388/log

daosbuild3 · 2025-12-31T05:01:56Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17329/2/testReport/

daosbuild3 · 2025-12-31T07:34:33Z

Test stage Functional on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17329/2/execution/node/1076/log

daosbuild3 · 2026-01-01T04:35:55Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17329/5/testReport/

daosbuild3 · 2026-01-01T08:42:03Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17329/5/execution/node/1324/log

daosbuild3 · 2026-01-04T03:37:40Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17329/6/testReport/

Nasf-Fan · 2026-01-04T09:34:14Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17329/6/testReport/

NLT failure for DAOS-17435, not related with the patch. All required CR tests passed.

Nasf-Fan · 2026-01-09T03:13:27Z

Ping reviewers, thanks!

kjacque

Overall looks good, just a couple minor comments/questions.

kjacque · 2026-01-10T00:57:09Z

+	/* Let secondary rank == primary rank. */
+	rc = crt_group_secondary_modify(ins->ci_iv_group, ins->ci_ranks, ins->ci_ranks,
+					CRT_GROUP_MOD_OP_REPLACE, ns_ver);


Since this function call is repeated in a few places with the same comment, you could put it in a simple (maybe even inline) wrapper function that takes only ins and ns_ver as params. Also, I think it would be good to add detail to the comment as to why the primary and secondary groups must be the same for the checker.

Good idea. I will do that when need to refresh the patch next time, or in subsequent CR related PR.

OK with me. Thanks!

Nasf-Fan · 2026-01-12T15:56:36Z

Ping reviewers, thanks!

Nasf-Fan · 2026-01-14T15:50:50Z

Resolve merge conflict.

daosbuild3 · 2026-01-14T15:55:55Z

Test stage Build on Leap 15.5 with Intel-C and TARGET_PREFIX completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17329/7/execution/node/301/log

daosbuild3 · 2026-01-19T04:22:23Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17329/13/testReport/

daosbuild3 · 2026-01-19T09:05:49Z

Test stage Functional Hardware Medium MD on SSD completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17329/13/testReport/

daosbuild3 · 2026-01-21T07:52:41Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17329/15/execution/node/1139/log

daosbuild3 · 2026-01-21T08:29:57Z

Test stage Functional Hardware Medium MD on SSD completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17329/15/testReport/

Nasf-Fan · 2026-01-21T09:57:49Z

A lot of CI tests failed for DAOS-18004, need to retest.

daosbuild3 · 2026-01-21T22:06:51Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17329/17/execution/node/1098/log

daosbuild3 · 2026-01-21T23:21:42Z

Test stage Functional Hardware Medium MD on SSD completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17329/17/testReport/

daosbuild3 · 2026-01-22T05:16:58Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17329/18/testReport/

daosbuild3 · 2026-01-22T13:10:34Z

Test stage Functional Hardware Medium MD on SSD completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17329/18/testReport/

daosbuild3 · 2026-01-23T05:48:58Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17329/23/execution/node/646/log

Nasf-Fan · 2026-01-23T08:04:07Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17329/23/execution/node/646/log

test_dangling_rank_entry failed because test logic cannot detect md-on-ssd mode properly. It is DAOS-18505, not related with the patch. All the other required CR related tests passed.

Nasf-Fan · 2026-01-23T08:05:09Z

Ping reviewers. Please help to review the patch, that is important for 2.8 release. Thanks!

daltonbohning · 2026-01-23T16:46:43Z

+pool:
+  scm_size: 1G


Is it intentional to only use SCM?

No, that is wrong. I will refresh the patch.

Include the followings: 1. When create CHK IV namespace, make the secondary group to be same as the primary group. Otherwise, CHK logic may hit DER_NONEXIST trouble when communicate via IV. 2. Integrate CHK IV namespace create and destroy API, cleanup related logic, redefine the version. 3. Get ranks list and IV namespace version from CHK leader when rejoin. Adjust CHK_REJOIN RPC for related changes. 4. Remove unsupported functionality for checking the specified 'phase'. 5. Add new test for case of lost some engine(s) before start checker. 6. Dedicated ULT to handle dead rank event, that will not be affected by checker start or stop. Then even if check scheduler exited, the subsequent check query still can work against the latest rank list. Test-tag: recovery Signed-off-by: Fan Yong <fan.yong@hpe.com>

Nasf-Fan · 2026-01-26T14:02:41Z

It is very slow to load the long PR push history. So I cloned it into another PR#17427 (#17427). Let's use such PR for further review and landing.

Nasf-Fan changed the title ~~DAOS-17535 chk: secondary group should be same as primary group for C…~~ DAOS-17535 chk: misc improvements for CR logic Dec 31, 2025

Nasf-Fan force-pushed the Nasf-Fan/DAOS-17535_7 branch from 8e4ad6a to 639a8ec Compare December 31, 2025 03:16

Nasf-Fan force-pushed the Nasf-Fan/DAOS-17535_7 branch from 639a8ec to 78579dd Compare December 31, 2025 07:33

Nasf-Fan force-pushed the Nasf-Fan/DAOS-17535_7 branch 2 times, most recently from aa39da7 to 476d0f9 Compare January 1, 2026 03:09

Nasf-Fan force-pushed the Nasf-Fan/DAOS-17535_7 branch from 476d0f9 to 09aaf91 Compare January 4, 2026 02:41

Nasf-Fan marked this pull request as ready for review January 4, 2026 09:34

Nasf-Fan requested review from gnailzenh, kjacque and wangshilong and removed request for kjacque January 4, 2026 09:34

kjacque reviewed Jan 10, 2026

View reviewed changes

frostedcmos mentioned this pull request Jan 12, 2026

DAOS-17535 cart: not exclude self when create corpc for IV #16357

Closed

6 tasks

kjacque self-requested a review January 13, 2026 00:21

kjacque previously approved these changes Jan 13, 2026

View reviewed changes

Nasf-Fan dismissed kjacque’s stale review via 680c95c January 14, 2026 15:50

Nasf-Fan force-pushed the Nasf-Fan/DAOS-17535_7 branch from baa1ffc to 40d16b7 Compare January 20, 2026 10:28

Nasf-Fan force-pushed the Nasf-Fan/DAOS-17535_7 branch from 40d16b7 to ec23619 Compare January 21, 2026 09:58

Nasf-Fan force-pushed the Nasf-Fan/DAOS-17535_7 branch from ec23619 to 8b375bb Compare January 22, 2026 03:42

Nasf-Fan force-pushed the Nasf-Fan/DAOS-17535_7 branch 4 times, most recently from 1226b77 to 14caebf Compare January 22, 2026 17:35

Nasf-Fan requested review from daltonbohning and kjacque January 23, 2026 08:04

daltonbohning reviewed Jan 23, 2026

View reviewed changes

Nasf-Fan force-pushed the Nasf-Fan/DAOS-17535_7 branch from 14caebf to 14f35b5 Compare January 26, 2026 06:20

Nasf-Fan requested review from a team as code owners January 26, 2026 06:20

Nasf-Fan closed this Jan 26, 2026

Nasf-Fan deleted the Nasf-Fan/DAOS-17535_7 branch January 28, 2026 04:06

		pool:
		scm_size: 1G

Conversation

Nasf-Fan commented Dec 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Steps for the author:

After all prior steps are complete:

Uh oh!

github-actions Bot commented Dec 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

daosbuild3 commented Dec 30, 2025

Uh oh!

daosbuild3 commented Dec 31, 2025

Uh oh!

daosbuild3 commented Dec 31, 2025

Uh oh!

daosbuild3 commented Dec 31, 2025

Uh oh!

daosbuild3 commented Jan 1, 2026

Uh oh!

daosbuild3 commented Jan 1, 2026

Uh oh!

daosbuild3 commented Jan 4, 2026

Uh oh!

Nasf-Fan commented Jan 4, 2026

Uh oh!

Nasf-Fan commented Jan 9, 2026

Uh oh!

kjacque left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kjacque Jan 10, 2026

Choose a reason for hiding this comment

Uh oh!

Nasf-Fan Jan 11, 2026

Choose a reason for hiding this comment

Uh oh!

kjacque Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

Nasf-Fan commented Jan 12, 2026

Uh oh!

Nasf-Fan commented Jan 14, 2026

Uh oh!

daosbuild3 commented Jan 14, 2026

Uh oh!

daosbuild3 commented Jan 19, 2026

Uh oh!

daosbuild3 commented Jan 19, 2026

Uh oh!

daosbuild3 commented Jan 21, 2026

Uh oh!

daosbuild3 commented Jan 21, 2026

Uh oh!

Nasf-Fan commented Jan 21, 2026

Uh oh!

daosbuild3 commented Jan 21, 2026

Uh oh!

daosbuild3 commented Jan 21, 2026

Uh oh!

daosbuild3 commented Jan 22, 2026

Uh oh!

daosbuild3 commented Jan 22, 2026

Uh oh!

daosbuild3 commented Jan 23, 2026

Uh oh!

Nasf-Fan commented Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Nasf-Fan commented Jan 23, 2026

Uh oh!

daltonbohning Jan 23, 2026

Choose a reason for hiding this comment

Uh oh!

Nasf-Fan Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

Nasf-Fan commented Jan 26, 2026

Uh oh!

Nasf-Fan commented Dec 30, 2025 •

edited

Loading

github-actions Bot commented Dec 30, 2025 •

edited

Loading

Nasf-Fan commented Jan 23, 2026 •

edited

Loading