Skip to content

Conversation

@yashwantbezawada
Copy link
Contributor

@yashwantbezawada yashwantbezawada commented Dec 27, 2025

Summary

  • Fixes Arrow Decimal128 columns being incorrectly converted to integers
  • Values like 3.14 now display correctly instead of as their internal representation (e.g., 314)
  • Maps decimal Arrow types to DTYPE_FLOAT64 and uses Decimal128::ToDouble(scale) for proper conversion

Test plan

  • Updated existing decimal stream and legacy tests to expect "float" schema type
  • Verify with reproduction code from issue:
    import perspective
    import duckdb
    import pyarrow as pa
    import io
    
    table = duckdb.sql("SELECT 3.14::DECIMAL AS decimal_col, 3.14::FLOAT AS float_col").arrow()
    buffer = io.BytesIO()
    
    with pa.ipc.new_file(buffer, table.schema) as writer:
        writer.write_table(table)
    
    bytes_object = buffer.getvalue()
    
    perspective_table = perspective.Table(bytes_object)
    # Both columns should now show 3.14

Fixes #2582

Arrow Decimal128 columns were being incorrectly converted to integers,
causing values like 3.14 to display as their internal representation
(e.g., 314) instead of the actual decimal value.

This change:
- Maps "decimal" and "decimal128" Arrow types to DTYPE_FLOAT64 instead
  of DTYPE_INT64 in convert_type()
- Uses Decimal128::ToDouble(scale) to properly convert decimal values
  to floating-point, respecting the decimal scale
- Updates tests to expect "float" schema type for decimal columns

Fixes perspective-dev#2582

Signed-off-by: Yashwant Bezawada <yashwant_b@me.com>
@yashwantbezawada
Copy link
Contributor Author

The reproduction code verification is pending CI - the build environment requires CMake 3.29.5+, Emscripten, and various system dependencies for the C++/Rust/WebAssembly compilation.

The fix has been thoroughly code-reviewed:

  • convert_type() now maps "decimal"/"decimal128" to DTYPE_FLOAT64 instead of DTYPE_INT64
  • copy_array() now uses Decimal128::ToDouble(scale) to properly convert values respecting the decimal scale
  • Existing decimal tests have been updated to expect "float" schema type

The CI pipeline should verify the fix works correctly with the full test suite.

Copy link
Member

@texodus texodus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR! Looks good!

@texodus texodus merged commit 0842f48 into perspective-dev:master Jan 27, 2026
16 checks passed
@texodus texodus added the bug Concrete, reproducible bugs label Jan 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Concrete, reproducible bugs

Development

Successfully merging this pull request may close these issues.

Arrow decimal type getting read as integer

2 participants