Skip to content

[controller][vpj][dvc] Use isFlinkVeniceViewsEnabled flag to block unnecessary write to view topic#2369

Merged
minhmo1620 merged 6 commits intolinkedin:mainfrom
minhmo1620:enable_flink_based_view
Jan 21, 2026
Merged

[controller][vpj][dvc] Use isFlinkVeniceViewsEnabled flag to block unnecessary write to view topic#2369
minhmo1620 merged 6 commits intolinkedin:mainfrom
minhmo1620:enable_flink_based_view

Conversation

@minhmo1620
Copy link
Copy Markdown
Contributor

@minhmo1620 minhmo1620 commented Jan 9, 2026

Problem Statement

When a store is marked as isFlinkVeniceViewsEnabled=true, controller should only create the view topic and broadcast START_OF_PUSH.

Currently, when there is a push job for a store, the server will ingest data into views topics, and controller will send out END_OF_PUSH and VERSION_SWAP messages.

In Flink-based views, data records and control messages (EOP/VS) will be populated through Flink job to ensure the correctness for the CM.

Solution

To block the unnecessary write to view topic, we will use store properties isFlinkVeniceViewsEnabled config, which is introduced in #2357 and #2356.

There are two main places we will block:

  1. clients/venice-push-job/src/main/java/com/linkedin/venice/hadoop/task/datawriter/AbstractPartitionWriter.java.

In this writer, we use the flag through a push job config that is only active when there is materialized view configured for the store. This push config will stop CompositeVeniceWriter creation, which will write to view topic before writing to version topic.

  1. LeaderFollowerStoreIngestionTask

In this SIT, we will not create viewWriter, hence, will not queueUpVersionTopicWritesWithViewWriters. This prevents data write in native replication process.

Code changes

  • Added new code behind a config. If so list the config names and their default values in the PR description.
  • Introduced new log lines.
    • Confirmed if logs need to be rate limited to avoid excessive logging.

Concurrency-Specific Checks

Both reviewer and PR author to verify

  • Code has no race conditions or thread safety issues.
  • Proper synchronization mechanisms (e.g., synchronized, RWLock) are used where needed.
  • No blocking calls inside critical sections that could lead to deadlocks or performance degradation.
  • Verified thread-safe collections are used (e.g., ConcurrentHashMap, CopyOnWriteArrayList).
  • Validated proper exception handling in multi-threaded code to avoid silent thread termination.

How was this PR tested?

  • New unit tests added.
  • New integration tests added.
  • Modified or extended existing tests.
  • Verified backward compatibility (if applicable).

Does this PR introduce any user-facing or breaking changes?

  • No. You can skip the rest of this section.
  • Yes. Clearly explain the behavior change and its impact.

@minhmo1620 minhmo1620 changed the title [controller][vpj] Use isFlinkVeniceViewsEnabled flag to block unncessary write to view topic [controller][vpj][dvc] Use isFlinkVeniceViewsEnabled flag to block unncessary write to view topic Jan 9, 2026
@minhmo1620 minhmo1620 marked this pull request as ready for review January 9, 2026 03:13
@minhmo1620 minhmo1620 changed the title [controller][vpj][dvc] Use isFlinkVeniceViewsEnabled flag to block unncessary write to view topic [controller][vpj][dvc] Use isFlinkVeniceViewsEnabled flag to block unnecessary write to view topic Jan 9, 2026
@minhmo1620 minhmo1620 requested a review from xunyin8 January 9, 2026 03:25
Copy link
Copy Markdown
Contributor

@xunyin8 xunyin8 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly LGTM just some minor optimizations.

@minhmo1620 minhmo1620 merged commit e922103 into linkedin:main Jan 21, 2026
50 checks passed
sushantmane pushed a commit to sushantmane/venice that referenced this pull request Jan 30, 2026
…necessary write to view topic (linkedin#2369)

This PR blocks any data and events populate to view topic if isFlinkVeniceViewsEnabled=true
when there is a new push to the store.
Controller will only create view topic and populate START_OF_PUSH per version.
misyel pushed a commit to misyel/venice that referenced this pull request Feb 2, 2026
…necessary write to view topic (linkedin#2369)

This PR blocks any data and events populate to view topic if isFlinkVeniceViewsEnabled=true
when there is a new push to the store.
Controller will only create view topic and populate START_OF_PUSH per version.
sushantmane pushed a commit to sushantmane/venice that referenced this pull request Feb 8, 2026
…necessary write to view topic (linkedin#2369)

This PR blocks any data and events populate to view topic if isFlinkVeniceViewsEnabled=true
when there is a new push to the store.
Controller will only create view topic and populate START_OF_PUSH per version.
misyel pushed a commit to misyel/venice that referenced this pull request Feb 17, 2026
…necessary write to view topic (linkedin#2369)

This PR blocks any data and events populate to view topic if isFlinkVeniceViewsEnabled=true
when there is a new push to the store.
Controller will only create view topic and populate START_OF_PUSH per version.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants