Commit 6ebf223
miranov25
Fix precision measurement in AliasDataFrame compression
Fixes two critical bugs in compress_columns precision measurement:
1. Integer overflow in RMSE calculation:
- uint8*uint8 arithmetic caused overflow (248*248 wraps in uint8)
- Cast to float64 before calculation to prevent overflow
- Added errstate context and robust median fallback
2. Non-finite value handling:
- NaN/inf values contaminated precision metrics
- Now filter to finite values before calculating statistics
- Track and report excluded sample count
Changes:
- Cast original/decompressed to float64 before diff calculation
- Apply finite mask: only calculate metrics on valid (finite) pairs
- Add fields: n_samples, n_total, fraction_nonfinite
- Update describe_compression to show sample counts and non-finite %
- Consistent output structure: always same 6 fields in precision_info
Impact:
- dEdxTPC RMSE now correct: 0.54 (was showing 57.6 due to overflow)
- Diagnostics match ROOT validation
- Clear reporting when data has NaN/inf values
Example output:
Precision: RMSE=0.545488, Max=2.500000, Mean=0.014717
Samples: 9,632,172/9,632,172, Non-finite: 0.00%
Related: Compression feature for TPC residuals (35% file size reduction)
ATO-6281 parent f2e537f commit 6ebf223
1 file changed
+35
-5
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
676 | 676 | | |
677 | 677 | | |
678 | 678 | | |
679 | | - | |
680 | | - | |
| 679 | + | |
| 680 | + | |
| 681 | + | |
| 682 | + | |
| 683 | + | |
| 684 | + | |
| 685 | + | |
| 686 | + | |
| 687 | + | |
| 688 | + | |
| 689 | + | |
| 690 | + | |
| 691 | + | |
| 692 | + | |
| 693 | + | |
| 694 | + | |
| 695 | + | |
| 696 | + | |
| 697 | + | |
| 698 | + | |
| 699 | + | |
| 700 | + | |
| 701 | + | |
681 | 702 | | |
682 | | - | |
683 | | - | |
684 | | - | |
| 703 | + | |
| 704 | + | |
| 705 | + | |
| 706 | + | |
| 707 | + | |
| 708 | + | |
685 | 709 | | |
686 | 710 | | |
687 | 711 | | |
| |||
883 | 907 | | |
884 | 908 | | |
885 | 909 | | |
| 910 | + | |
| 911 | + | |
| 912 | + | |
| 913 | + | |
| 914 | + | |
| 915 | + | |
0 commit comments