HIVE-29383: Iceberg: [V3] Add support for timestamp with nanosecond precession

[ayushtkn]

What changes were proposed in this pull request?

Add support for timestamp with nanosecond precession

Why are the changes needed?

To support the new timestamp types introduced as part of iceberg v3

Does this PR introduce any user-facing change?

Yes, users can leverage timestamp_ns & timestamptz_ns data types for iceberg v3 tables

How was this patch tested?

UT

[Aggarwal-Raghav]

i didn't ran with your latest changes, i have taken 7 commits from this PR. It's higly likely that it got fixed with current set of changes.

[Aggarwal-Raghav]

On your latest changes, select * is working now in beeline ✅, but casting to string is not working ❌, you can check by adding
SELECT CAST(ts_ns AS STRING) FROM t; in timestamp_ns.q

Caused by: java.lang.RuntimeException: Hive 2 Internal error: type = nanosecond timestamp
	at org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorConverter$TextConverter.convert(PrimitiveObjectInspectorConverter.java:520)
	at org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorConverter$TextConverter.convert(PrimitiveObjectInspectorConverter.java:412)
	at org.apache.hadoop.hive.ql.udf.generic.GenericUDFToString.evaluate(GenericUDFToString.java:57)
	at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:231)
	at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:80)
	at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:68)
	at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:87)

[ayushtkn]

fixed the cast query and added in the q file

[Aggarwal-Raghav]

For classes like IcebergNanosecondTimestampObjectInspectorHive3, IcebergNanosecondTimestampWithZoneObjectInspectorHive3 , we need to override methods like getPrimitiveJavaObject, getPrimitiveWritableObject etc, they should return specific subclass like TimestampNanoTZ or TimestampNano

Any thoughts on this comments? This is what I meant:

package org.apache.iceberg.mr.hive.serde.objectinspector;

import java.time.LocalDateTime;
import org.apache.hadoop.hive.common.type.TimestampNano;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;

public class TestIcebergNanosecondTimestampObjectInspectorHive3 {

  @Test
  public void testScenario() {
    IcebergNanosecondTimestampObjectInspectorHive3 oi =
        IcebergNanosecondTimestampObjectInspectorHive3.get();
    LocalDateTime now = LocalDateTime.now();

    Object tempObject = oi.getPrimitiveJavaObject(now);

    Assertions.assertInstanceOf(
        TimestampNano.class,
        tempObject,
        "getPrimitiveJavaObject should return an instance of TimestampNano");
  }

[ERROR]   TestIcebergNanosecondTimestampObjectInspectorHive3.testScenario:37 getPrimitiveJavaObject should return an instance of TimestampNano ==> Unexpected type, expected: <org.apache.hadoop.hive.common.type.TimestampNano> but was: <org.apache.hadoop.hive.common.type.Timestamp>

[Aggarwal-Raghav]

@ayushtkn , can you please check if these changes make sense addendum.patch

[sonarqubecloud]

Quality Gate passed

Issues
68 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[Aggarwal-Raghav]

@ayushtkn , i have found another issue with the join on nanosecond timestamp (specific to MapJoin). If you agree, can we create a new sub-JIRA to fix new issues?

[ayushtkn]

Let me know the issue & I can try fix it or we can create a followup once we merge this.

Should get some review as well, if folks are ok with the approach as well. We created a bunch of empty classes. Will try ask someone next week.

I need to test it myself as well post holidays. Was just trying as a POC. The idea was to keep the delta as low as possible and leverage TIMESTAMP mostly, I think have deviated bit from my initial plan.

[Aggarwal-Raghav]

create table t1 (id int, ts nanosecond timestamp) stored BY ICEBERG STORED AS PARQUET TBLPROPERTIES('format-version'='3');
create table t2 (id int, ts nanosecond timestamp) stored BY ICEBERG STORED AS PARQUET TBLPROPERTIES('format-version'='3');

insert into t1 values(1,'2025-12-30 11:19:00.123456789');
insert into t2 values(1,'2025-12-30 11:19:00.123456789');

SET hive.auto.convert.join=true;
select * from t1 join t2 where t1.ts=t2.ts;

Caused by: java.lang.NullPointerException: Cannot invoke "java.util.List.get(int)" because "structFieldObjectInspectors" is null
	at org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.init(StandardStructObjectInspector.java:122)
	at org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.<init>(StandardStructObjectInspector.java:111)
	at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.getStandardStructObjectInspector(ObjectInspectorFactory.java:339)
	at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.getStandardStructObjectInspector(ObjectInspectorFactory.java:322)
	at org.apache.hadoop.hive.ql.exec.JoinUtil.getRowContainer(JoinUtil.java:384)
	at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.initializeOp(CommonJoinOperator.java:326)
	at org.apache.hadoop.hive.ql.exec.AbstractMapJoinOperator.initializeOp(AbstractMapJoinOperator.java:78)
	at org.apache.hadoop.hive.ql.exec.MapJoinOperator.initializeOp(MapJoinOperator.java:174)
	at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:360)
	at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:549)
	at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:503)
	at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:369)
	at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:332)

[ayushtkn]

closing, will open a new one, this approach didn't worked very well, had a word with compiler folks as well on the syntax

[Aggarwal-Raghav]

closing, will open a new one, this approach didn't worked very well, had a word with compiler folks as well on the syntax

ok

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the dev@hive.apache.org list if the patch is in need of reviews.