Skip to content

[tool](meta_tool) Add gen_empty_segment operation to generate empty segment files#61703

Open
gavinchou wants to merge 1 commit intoapache:masterfrom
gavinchou:gavin-gen-empty-segment
Open

[tool](meta_tool) Add gen_empty_segment operation to generate empty segment files#61703
gavinchou wants to merge 1 commit intoapache:masterfrom
gavinchou:gavin-gen-empty-segment

Conversation

@gavinchou
Copy link
Contributor

Proposed changes

Add a new operation gen_empty_segment to the meta_tool that allows generating empty segment files (0 rows). This is useful for testing and debugging purposes.

Usage

./meta_tool --operation=gen_empty_segment [--output_path=/path/to/output]

Default output file name: empty.dat

Generated file structure

  • Short key index page with 0 keys
  • Segment footer with num_rows=0
  • Total file size: ~47 bytes

Example output

Generated empty segment file: /tmp/empty.dat
  - Index page size: 27 bytes
  - Footer size: 4 slices, 20 bytes
  - Total file size: 47 bytes
  - num_rows: 0

Checklist

  • I have created an issue and described the feature request.
  • I have checked the version labels which the pr will be auto-backported to the target branch.
    • 3.0
    • 2.1
    • 2.0
  • I have added test cases for my feature.

🤖 Generated with Claude Code

@Thearas
Copy link
Contributor

Thearas commented Mar 25, 2026

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@gavinchou
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 26914 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a59d70067c52fb65a959e97a94976aeeb5aa52d1, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17654	4484	4318	4318
q2	q3	10652	820	536	536
q4	4677	370	248	248
q5	7585	1218	1035	1035
q6	176	174	147	147
q7	788	862	688	688
q8	9317	1506	1341	1341
q9	4982	4824	4758	4758
q10	6306	1921	1645	1645
q11	466	248	244	244
q12	736	580	463	463
q13	18042	2725	1958	1958
q14	229	237	217	217
q15	q16	736	756	677	677
q17	740	820	489	489
q18	6001	5476	5349	5349
q19	1201	994	646	646
q20	546	502	377	377
q21	4591	1856	1449	1449
q22	533	379	329	329
Total cold run time: 95958 ms
Total hot run time: 26914 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4824	4638	4602	4602
q2	q3	3903	4360	3840	3840
q4	896	1199	774	774
q5	4126	4456	4343	4343
q6	194	180	140	140
q7	1786	1686	1537	1537
q8	2524	2764	2580	2580
q9	7837	7438	7422	7422
q10	3742	3971	3614	3614
q11	512	446	418	418
q12	492	607	438	438
q13	2429	2881	2087	2087
q14	387	344	290	290
q15	q16	736	764	717	717
q17	1172	1341	1364	1341
q18	7061	6932	6858	6858
q19	993	941	932	932
q20	2121	2167	1988	1988
q21	3975	3499	3371	3371
q22	475	429	392	392
Total cold run time: 50185 ms
Total hot run time: 47684 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 168787 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a59d70067c52fb65a959e97a94976aeeb5aa52d1, data reload: false

query5	4329	640	488	488
query6	325	224	201	201
query7	4193	479	289	289
query8	340	246	220	220
query9	8683	2732	2691	2691
query10	528	379	342	342
query11	7015	5104	4884	4884
query12	182	123	121	121
query13	1272	459	328	328
query14	5683	3673	3440	3440
query14_1	2767	3122	2833	2833
query15	205	195	182	182
query16	1003	484	478	478
query17	930	783	619	619
query18	2440	440	338	338
query19	213	203	178	178
query20	130	123	122	122
query21	210	132	106	106
query22	13277	14003	14738	14003
query23	16723	16447	15883	15883
query23_1	15779	15637	15733	15637
query24	7200	1628	1258	1258
query24_1	1219	1237	1243	1237
query25	578	501	444	444
query26	1253	278	148	148
query27	2761	490	296	296
query28	4505	1880	1849	1849
query29	880	603	501	501
query30	299	228	192	192
query31	995	957	883	883
query32	83	74	71	71
query33	538	353	302	302
query34	886	889	522	522
query35	635	685	600	600
query36	1099	1147	969	969
query37	139	99	86	86
query38	3029	2968	2852	2852
query39	890	852	810	810
query39_1	799	797	783	783
query40	236	163	141	141
query41	66	65	64	64
query42	264	261	265	261
query43	240	254	221	221
query44	
query45	212	191	187	187
query46	883	988	611	611
query47	2118	2166	2044	2044
query48	322	322	238	238
query49	648	468	399	399
query50	700	291	214	214
query51	4176	4019	4020	4019
query52	264	271	263	263
query53	292	347	295	295
query54	313	287	289	287
query55	97	89	84	84
query56	363	346	325	325
query57	1980	1631	1645	1631
query58	321	280	278	278
query59	2855	2945	2758	2758
query60	388	326	321	321
query61	155	152	151	151
query62	627	588	542	542
query63	310	286	276	276
query64	5086	1288	1004	1004
query65	
query66	1462	467	357	357
query67	24299	24300	24373	24300
query68	
query69	421	325	277	277
query70	979	873	943	873
query71	353	318	304	304
query72	2902	2759	2468	2468
query73	547	546	312	312
query74	9602	9570	9370	9370
query75	2903	2762	2458	2458
query76	2381	1037	693	693
query77	382	403	312	312
query78	11071	11161	10536	10536
query79	1101	805	563	563
query80	1318	630	540	540
query81	548	271	222	222
query82	1273	153	119	119
query83	341	268	248	248
query84	296	118	99	99
query85	941	485	469	469
query86	436	299	311	299
query87	3124	3122	3020	3020
query88	3520	2651	2636	2636
query89	420	368	337	337
query90	2011	171	177	171
query91	171	162	139	139
query92	79	76	74	74
query93	908	840	515	515
query94	636	310	293	293
query95	587	400	330	330
query96	652	528	236	236
query97	2471	2510	2397	2397
query98	244	247	218	218
query99	997	970	929	929
Total cold run time: 250404 ms
Total hot run time: 168787 ms

@gavinchou gavinchou changed the title [feature](meta_tool) Add gen_empty_segment operation to generate empty segment files [tool](meta_tool) Add gen_empty_segment operation to generate empty segment files Mar 25, 2026
@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 25, 2026
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@doris-robot
Copy link

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.82% (19872/37619)
Line Coverage 36.33% (185678/511098)
Region Coverage 32.57% (143816/441501)
Branch Coverage 33.78% (62961/186372)

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 73.40% (27038/36836)
Line Coverage 56.86% (289740/509535)
Region Coverage 54.21% (241596/445627)
Branch Coverage 55.88% (104456/186936)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants