NIFI-15938 Improve TailFile startup and state recovery

[ing-mattioni]

Summary

NIFI-15938

This pull request updates TailFile to improve multi-file startup behavior and make persisted state recovery more robust.

The changes focus on startup performance, rollover state restoration, malformed persisted state handling, read-loop buffer behavior, and buffer-size validation.

Changes

• Improves multi-file startup by replacing eager file listing with streaming file discovery
• Replaces repeated state membership checks with HashSet membership
• Restores persisted tailingPostRollover state correctly
• Avoids recovery failures when rollover state exists but no rolled files remain
• Hardens recovery against malformed persisted numeric and checksum state
• Fixes buffer reuse in the read loop
• Tightens buffer-size validation to prevent zero-sized or overflow-sized buffers
• Adds regression tests for the fixed TailFile scenarios

Key fixes, more technical description:

• Improved multi-file startup by replacing eager FileUtils.listFiles and O(n²) state checks with streaming Files.find and HashSet membership.
• Fixed persisted tailingPostRollover restoration and avoided crashes when rollover state exists but no rolled files remain.
• Hardened recovery against malformed persisted numeric/checksum state.
• Fixed buffer reuse in the read loop and tightened buffer-size validation to prevent zero/overflow-sized buffers.

Tracking

Please complete the following tracking steps prior to pull request creation.

Issue Tracking

• Apache NiFi Jira issue created

Pull Request Tracking

• Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
• Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000
• Pull request contains commits signed with a registered key indicating Verified status

Pull Request Formatting

• Pull Request based on current revision of the main branch
• Pull Request refers to a feature branch with one commit containing changes

Verification

Please indicate the verification steps performed prior to pull request creation.

Build

• Build completed using ./mvnw clean install -P contrib-check
  • JDK 21
  • JDK 25

Module-level verification performed:

.\mvnw.cmd --% -pl :nifi-standard-processors -am -Dtest=TestTailFile,TestTailFileSimpleScenarios,TestTailFileGeneratedScenarios -Dsurefire.failIfNoSpecifiedTests=false test

Additional verification performed:

.\mvnw.cmd --% -pl :nifi-standard-processors -Pcontrib-check validate

Licensing

• New dependencies are compatible with the Apache License 2.0
• New dependencies are documented in applicable LICENSE and NOTICE files

No new dependencies were added.

Documentation

• Documentation formatting appears as expected in rendered files

No user-facing documentation files were changed.

[exceptionfactory]

Thanks for proposing these improvements @ing-mattioni.

On initial review, I raised some concerns about the introduction of several try-catch blocks. These seem unnecessary without further explanation.

Some of the other changes also raise implementation questions, so overall, some clarification and refinement is needed before this can go forward.

[exceptionfactory]

Suggested change

	final Set<String> filesToTailSet = new HashSet<>(filesToTail);
	final Set<String> uniqueFilesToTail = new HashSet<>(filesToTail);

[ing-mattioni]

Applied. I renamed filesToTailSet to uniqueFilesToTail for clarity.

[exceptionfactory]

Is there a particular reason for introducing this try-catch? The Processor should control the state content, avoiding potential incorrect formatting.

[ing-mattioni]

Agreed. I removed this try/catch. TailFile controls the persisted state key format, so adding defensive handling for malformed internal state is not needed here.

[exceptionfactory]

This catch also seems too broad and not necessary.

[ing-mattioni]

Agreed. I removed this broad catch as well. If the processor-controlled state cannot be restored, that should not be silently ignored by skipping the file.

[exceptionfactory]

What is the reason for creating this wrapping Set?

[ing-mattioni]

The Set is used to avoid repeated linear List.contains() checks while reconciling restored state and removing files that are no longer tailed. With many matching files, keeping this as a List would make startup/state reconciliation trend toward O(n²). The original filesToTail list is still used when adding new states, so ordering is preserved.

[ing-mattioni]

Thank you @exceptionfactory ; I've updated the PR based on the review. The added try/catch blocks have been removed, and the Set variable has been renamed to "uniqueFilesToTail". I kept the Set conversion because it avoids repeated linear membership checks during state reconciliation, which is important when TailFile starts with many matching files.