Skip to content

TEZ-4715: Migrate Tez PB Protocols from ProtobufRPCEngine to ProtobufRPCEngine2#493

Open
Aggarwal-Raghav wants to merge 1 commit intoapache:masterfrom
Aggarwal-Raghav:TEZ-4715
Open

TEZ-4715: Migrate Tez PB Protocols from ProtobufRPCEngine to ProtobufRPCEngine2#493
Aggarwal-Raghav wants to merge 1 commit intoapache:masterfrom
Aggarwal-Raghav:TEZ-4715

Conversation

@Aggarwal-Raghav
Copy link
Copy Markdown
Contributor

@Aggarwal-Raghav Aggarwal-Raghav commented Apr 28, 2026

  1. Migrated Protobuf imports from com.google.protobuf to shaded org.apache.hadoop.thirdparty.protobuf
  2. Added hadoop-thirdparty dependency (v1.4.0) to the root POM to ensure Hadoop 3.4.2 compatibility. This will be updated for every hadoop version upgrade. For now I have kept proto version to 3.25.5 as it was in tez and will be in-sync with hadoop-3.5.0
  3. Used maven-replacer-plugin (same as hadoop) to automated package refactoring in generated proto code.

@tez-yetus
Copy link
Copy Markdown

(!) A patch to the testing environment has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-493/1/console in case of problems.

@Aggarwal-Raghav
Copy link
Copy Markdown
Contributor Author

Jenkins is not updating the final status 😔

[2026-04-29T22:12:13.679Z] 
[2026-04-29T22:12:13.679Z] ============================================================================
[2026-04-29T22:12:13.679Z] ============================================================================
[2026-04-29T22:12:13.679Z]                          Adding comment to Github
[2026-04-29T22:12:13.679Z] ============================================================================
[2026-04-29T22:12:13.679Z] ============================================================================
[2026-04-29T22:12:13.679Z] 
[2026-04-29T22:12:13.679Z] 
[2026-04-29T22:12:16.867Z] ERROR: Failed to write github status. Token expired or missing repo:status write?
[2026-04-29T22:12:16.867Z] ERROR: Failed to write github status. Token expired or missing repo:status write?
[2026-04-29T22:12:17.431Z] ERROR: Failed to write github status. Token expired or missing repo:status write?
[2026-04-29T22:12:17.431Z] ERROR: Failed to write github status. Token expired or missing repo:status write?
[2026-04-29T22:12:17.997Z] ERROR: Failed to write github status. Token expired or missing repo:status write?
[2026-04-29T22:12:18.561Z] ERROR: Failed to write github status. Token expired or missing repo:status write?
[2026-04-29T22:12:18.561Z] ERROR: Failed to write github status. Token expired or missing repo:status write?
[2026-04-29T22:12:19.124Z] ERROR: Failed to write github status. Token expired or missing repo:status write?
[2026-04-29T22:12:19.689Z] ERROR: Failed to write github status. Token expired or missing repo:status write?
[2026-04-29T22:12:19.689Z] ERROR: Failed to write github status. Token expired or missing repo:status write?
[2026-04-29T22:12:20.253Z] ERROR: Failed to write github status. Token expired or missing repo:status write?

@Aggarwal-Raghav
Copy link
Copy Markdown
Contributor Author

Aggarwal-Raghav commented May 1, 2026

The UT failure debugging in #483 (comment) requires hadoop thirdparty making this PR dependent.

TEZ-4715 + TEZ-4708 will completely migrate tez from WritableRPCEngine + ProtobufRPCEngine => ProtobufRPCEngine2

@abstractdog abstractdog requested a review from ayushtkn May 2, 2026 05:51
@ayushtkn
Copy link
Copy Markdown
Member

ayushtkn commented May 2, 2026

@abstractdog From the code point of view, this seems right, We had to just use the RPCEngine2 & migrate to the Hadoop Thirdparty. That is rightly done here.

But I think this would be an incompatible change, beyond this point I believe Hive master won't build nor work with Tez master. Hive has to do the same activity when moving to Tez-1.0.0 & that would be incompatible for Hive as well & believe the older Hadoop versions won't work post this activity in Hive or Tez, unless there was something done later to fix it, which I don't think...

There were compat issues that is why I believe HADOOP-17046 was done later to make sure older client can work with them. Means protobuf is indeed a very known heavy Incompat change, a lot of discussion at HADOOP-13363, in case interested.

These are the tradeoffs, I just wanted to highlight. Rest all cool

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants