Skip to content

[fix](inverted-index) Avoid global null bitmap in sub-reader context#65549

Open
hoshinojyunn wants to merge 1 commit into
apache:masterfrom
hoshinojyunn:fix/multi-segment-null-resolver
Open

[fix](inverted-index) Avoid global null bitmap in sub-reader context#65549
hoshinojyunn wants to merge 1 commit into
apache:masterfrom
hoshinojyunn:fix/multi-segment-null-resolver

Conversation

@hoshinojyunn

@hoshinojyunn hoshinojyunn commented Jul 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: N/A

Related PR: N/A

Problem Summary:

create_segment_context() creates a QueryExecutionContext for each CLucene sub-reader. The root NULL bitmap resolver uses table-level global docids, while the child scorer evaluates its boolean expression with sub-reader-local docids. Copying that resolver into the child context mixes these coordinate systems.

Reproduction:

  1. Build a MultiReader with two sub-readers, each containing two documents.
  2. Set global doc 0 to NULL. The root NULL bitmap is {0}.
  3. In the first sub-reader, local doc 0 maps to global doc 0; in the second sub-reader, local doc 0 maps to global doc 2.
  4. Run NOT PREFIX(title, "missing").

Incorrect result before this fix:

  • The second sub-reader receives the root NULL bitmap {0}.
  • Its boolean scorer treats bitmap entry 0 as local doc 0 rather than global doc 0.
  • It therefore excludes global doc 2 and contributes only global doc 3.
  • The collected TRUE bitmap becomes {1,3}.
  • The final root-level NULL mask {0} cannot restore the incorrectly omitted doc 2.

Expected result:

  • The TRUE bitmap before the root NULL mask is {0,1,2,3}.
  • Applying the root NULL bitmap {0} produces {1,2,3}.

Fix:

  • Keep null_resolver on the root execution context.
  • Do not copy it into the per-sub-reader context created by create_segment_context().
  • Add a unit test that reproduces the two-sub-reader NOT case and verifies global doc 2 is retained.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
      • ./run-be-ut.sh --run --filter=MultiSegmentCollectorTest.* -j16
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hoshinojyunn
[fix](be) Avoid global null bitmap in sub-reader context
8b1034c 
### What problem does this PR solve?

Issue Number: N/A

Related PR: N/A

Problem Summary: Multi-segment collection creates a QueryExecutionContext for each CLucene sub-reader. Copying the root NULL bitmap resolver into that context lets nullable boolean scorers interpret a table-level NULL bitmap with local docids. For example, consider two sub-readers with two documents each. In the first sub-reader, local doc 0 maps to global doc 0 and is NULL. In the second, local doc 0 maps to global doc 2. The root NULL bitmap is therefore {0}. For NOT PREFIX(title, "missing"), the second sub-reader incorrectly treats global bitmap entry 0 as its local doc 0, excludes global doc 2, and contributes only global doc 3. The collected TRUE bitmap becomes {1,3}; masking the actual root NULL bitmap {0} cannot restore doc 2, while the correct result is {1,2,3}. Keep the resolver on the root context only. The collector test covers this two-sub-reader case and verifies that the final NOT result retains global doc 2.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - ./run-be-ut.sh --run --filter=MultiSegmentCollectorTest.* -j16
- Behavior changed: Yes (fixes incorrect multi-segment boolean query results with NULLs)
- Does this need documentation: No
@hello-stephen

hello-stephen commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hoshinojyunn hoshinojyunn changed the title [fix](be) Avoid global null bitmap in sub-reader context [fix](inverted-index) Avoid global null bitmap in sub-reader context Jul 13, 2026
@hoshinojyunn

hoshinojyunn commented Jul 13, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

hello-stephen commented Jul 13, 2026

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 29274 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8b1034cd17c794385e76f46cef43cfedf2e7eec1, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17735	4136	4058	4058
q2	2014	316	199	199
q3	10353	1384	790	790
q4	4686	472	339	339
q5	7550	841	550	550
q6	188	177	135	135
q7	774	844	615	615
q8	9394	1550	1530	1530
q9	5719	4382	4400	4382
q10	6828	1783	1522	1522
q11	496	345	319	319
q12	717	551	438	438
q13	18076	3395	2772	2772
q14	263	259	247	247
q15	q16	782	791	710	710
q17	954	928	916	916
q18	6766	5713	5495	5495
q19	1162	1246	1006	1006
q20	711	675	580	580
q21	5722	2569	2374	2374
q22	422	350	297	297
Total cold run time: 101312 ms
Total hot run time: 29274 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4320	4370	4267	4267
q2	283	313	220	220
q3	4615	4955	4402	4402
q4	2055	2139	1367	1367
q5	4406	4298	4291	4291
q6	223	171	128	128
q7	1745	1628	1879	1628
q8	2659	2237	2152	2152
q9	8049	8121	7836	7836
q10	4709	4761	4208	4208
q11	572	396	393	393
q12	797	750	535	535
q13	3281	3578	3020	3020
q14	312	328	308	308
q15	q16	711	719	655	655
q17	1314	1311	1283	1283
q18	7993	7350	7177	7177
q19	1178	1084	1092	1084
q20	2244	2205	1944	1944
q21	5143	4509	4297	4297
q22	520	454	408	408
Total cold run time: 57129 ms
Total hot run time: 51603 ms

@hello-stephen

hello-stephen commented Jul 13, 2026

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 180045 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8b1034cd17c794385e76f46cef43cfedf2e7eec1, data reload: false

query5	4331	620	496	496
query6	452	229	205	205
query7	4890	581	339	339
query8	333	180	178	178
query9	8761	4005	3987	3987
query10	457	342	302	302
query11	5931	2351	2136	2136
query12	161	112	110	110
query13	1259	606	445	445
query14	6252	5302	4937	4937
query14_1	4284	4301	4292	4292
query15	211	224	179	179
query16	986	470	363	363
query17	929	702	562	562
query18	2454	460	338	338
query19	209	182	146	146
query20	112	109	106	106
query21	226	155	131	131
query22	13589	13515	13427	13427
query23	17375	16471	16263	16263
query23_1	16281	16336	16315	16315
query24	7515	1763	1269	1269
query24_1	1301	1295	1292	1292
query25	544	458	378	378
query26	1327	363	214	214
query27	2617	618	390	390
query28	4557	1999	2001	1999
query29	1057	617	478	478
query30	338	267	224	224
query31	1113	1101	972	972
query32	110	62	60	60
query33	527	332	262	262
query34	1170	1153	644	644
query35	768	799	670	670
query36	1442	1420	1207	1207
query37	158	107	92	92
query38	1866	1705	1646	1646
query39	946	938	899	899
query39_1	879	871	890	871
query40	251	172	145	145
query41	73	68	68	68
query42	95	94	92	92
query43	321	325	285	285
query44	1412	794	797	794
query45	207	188	186	186
query46	1058	1180	773	773
query47	2413	2355	2222	2222
query48	417	393	297	297
query49	584	427	315	315
query50	1057	421	340	340
query51	10867	10507	11883	10507
query52	83	86	75	75
query53	262	284	202	202
query54	274	236	210	210
query55	72	72	70	70
query56	306	287	291	287
query57	1458	1433	1320	1320
query58	283	254	243	243
query59	1577	1622	1484	1484
query60	300	267	246	246
query61	148	156	143	143
query62	716	649	576	576
query63	237	199	203	199
query64	2847	1034	866	866
query65	4843	4755	4770	4755
query66	1839	498	379	379
query67	29608	29383	29355	29355
query68	3181	1538	1042	1042
query69	403	315	275	275
query70	1086	952	970	952
query71	336	319	302	302
query72	3199	2817	2625	2625
query73	833	786	427	427
query74	5105	4970	4737	4737
query75	2613	2584	2215	2215
query76	2325	1152	772	772
query77	362	378	285	285
query78	12324	12306	11873	11873
query79	1363	1102	749	749
query80	1278	539	453	453
query81	523	325	281	281
query82	813	152	120	120
query83	362	313	288	288
query84	323	159	132	132
query85	960	585	516	516
query86	424	293	283	283
query87	1835	1816	1749	1749
query88	3678	2744	2768	2744
query89	448	395	353	353
query90	1951	195	192	192
query91	199	187	157	157
query92	63	60	56	56
query93	1605	1454	1022	1022
query94	703	353	319	319
query95	766	474	529	474
query96	1019	843	341	341
query97	2660	2676	2543	2543
query98	215	214	202	202
query99	1143	1176	1040	1040
Total cold run time: 265787 ms
Total hot run time: 180045 ms

@hello-stephen

hello-stephen commented Jul 13, 2026

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.99 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8b1034cd17c794385e76f46cef43cfedf2e7eec1, data reload: false

query1	0.01	0.01	0.01
query2	0.09	0.05	0.05
query3	0.26	0.13	0.13
query4	1.60	0.13	0.13
query5	0.26	0.24	0.22
query6	1.91	1.10	1.05
query7	0.04	0.01	0.01
query8	0.06	0.03	0.03
query9	0.38	0.30	0.30
query10	0.58	0.57	0.57
query11	0.19	0.13	0.13
query12	0.17	0.15	0.15
query13	0.48	0.49	0.49
query14	1.01	1.02	1.00
query15	0.60	0.60	0.60
query16	0.31	0.33	0.31
query17	1.10	1.08	1.12
query18	0.22	0.20	0.20
query19	2.04	1.95	1.92
query20	0.01	0.01	0.02
query21	15.43	0.24	0.13
query22	4.76	0.05	0.05
query23	16.14	0.31	0.12
query24	2.96	0.42	0.31
query25	0.12	0.05	0.05
query26	0.74	0.20	0.16
query27	0.04	0.04	0.03
query28	3.52	0.92	0.55
query29	12.50	4.13	3.26
query30	0.27	0.15	0.15
query31	2.77	0.58	0.32
query32	3.22	0.59	0.48
query33	3.16	3.18	3.28
query34	15.68	4.25	3.56
query35	3.55	3.50	3.53
query36	0.56	0.45	0.41
query37	0.10	0.06	0.07
query38	0.05	0.03	0.04
query39	0.04	0.03	0.03
query40	0.18	0.16	0.14
query41	0.08	0.03	0.03
query42	0.04	0.02	0.02
query43	0.04	0.03	0.04
Total cold run time: 97.27 s
Total hot run time: 24.99 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@airborne12 airborne12 Awaiting requested review from airborne12 airborne12 is a code owner
@eldenmoon eldenmoon Awaiting requested review from eldenmoon eldenmoon is a code owner
@csun5285 csun5285 Awaiting requested review from csun5285 csun5285 is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@hoshinojyunn @hello-stephen