NIFI-15451: Added ability for Connectors to retrieve bundles availabl…#10756
Merged
mcgilman merged 3 commits intoapache:NIFI-15258from Jan 21, 2026
Merged
NIFI-15451: Added ability for Connectors to retrieve bundles availabl…#10756mcgilman merged 3 commits intoapache:NIFI-15258from
mcgilman merged 3 commits intoapache:NIFI-15258from
Conversation
mcgilman
requested changes
Jan 20, 2026
| } | ||
|
|
||
| @Override | ||
| public Builder componentBundleLookup(final ComponentBundleLookup bundleLookup) { |
Contributor
There was a problem hiding this comment.
I don't see this being called any place yet. A Connector calling getComponentBundleLookup would get null.
| } | ||
| } | ||
|
|
||
| public static void updateToLatestBundles(final VersionedProcessGroup processGroup, final ComponentBundleLookup componentBundleLookup) { |
Contributor
There was a problem hiding this comment.
Can we add some test coverage to these new updateToLatest methods?
Comment on lines
+424
to
+425
| final int part1 = i < parts1.length ? Integer.parseInt(parts1[i]) : 0; | ||
| final int part2 = i < parts2.length ? Integer.parseInt(parts2[i]) : 0; |
Contributor
There was a problem hiding this comment.
Do we need any additional validation to avoid NumberFormatException?
| private ComponentLog componentLog; | ||
| private SecretsManager secretsManager; | ||
| private AssetManager assetManager; | ||
| private ComponentBundleLookup componentBundleLookup; |
Contributor
There was a problem hiding this comment.
updateParameterContext above is private and unused.
| return latest; | ||
| } | ||
|
|
||
| private static int compareVersion(final String v1, final String v2) { |
Contributor
There was a problem hiding this comment.
The logic in this method doesn't work for versions with characters and trailing numbers. For instance, both 2.0.0-M1 and 2.0.0-RC1 are considered greater than 2.0.0. Maybe this isn't something we're trying to tackle but just thought I'd raise it.
…e for component types and updated VersionedFlowUtils to make use of it to easily update Versioned components
…s to how we compare component versions
markap14
added a commit
that referenced
this pull request
Jan 22, 2026
#10756) * NIFI-15451: Added ability for Connectors to retrieve bundles available for component types and updated VersionedFlowUtils to make use of it to easily update Versioned components * NIFI-15451: Added additional unit tests to VersionedFlowUtils; updates to how we compare component versions * NIFI-15451: Fixed PMD violation
markap14
added a commit
that referenced
this pull request
Feb 11, 2026
#10756) * NIFI-15451: Added ability for Connectors to retrieve bundles available for component types and updated VersionedFlowUtils to make use of it to easily update Versioned components * NIFI-15451: Added additional unit tests to VersionedFlowUtils; updates to how we compare component versions * NIFI-15451: Fixed PMD violation
pvillard31
pushed a commit
to pvillard31/nifi
that referenced
this pull request
Feb 14, 2026
apache#10756) * NIFI-15451: Added ability for Connectors to retrieve bundles available for component types and updated VersionedFlowUtils to make use of it to easily update Versioned components * NIFI-15451: Added additional unit tests to VersionedFlowUtils; updates to how we compare component versions * NIFI-15451: Fixed PMD violation
pvillard31
pushed a commit
that referenced
this pull request
Feb 14, 2026
#10756) * NIFI-15451: Added ability for Connectors to retrieve bundles available for component types and updated VersionedFlowUtils to make use of it to easily update Versioned components * NIFI-15451: Added additional unit tests to VersionedFlowUtils; updates to how we compare component versions * NIFI-15451: Fixed PMD violation
markap14
added a commit
to markap14/nifi
that referenced
this pull request
Mar 10, 2026
NIFI-15258: Fixed vulnerable lz4 dependency NIFI-15258: Fixed checkstyle violations/pmd NIFI-15258: Addressed PMD findings NIFI-15259: Added REST API for Connectors Introduced comprehensive REST endpoints for managing Connectors through the NiFi web interface and API clients. Key capabilities: - Full lifecycle management (create, read, update, delete) - Start/stop control with run status endpoints - Multi-step configuration with property groups supporting string literals, asset references, and secret references - Asynchronous configuration verification with polling support - Access to connector's managed flow and process group status - Search functionality within connector's encapsulated flow Cluster support: - Request replication across cluster nodes - Response merging for dynamic property allowable values that may differ between nodes - Consistent configuration step and property descriptor handling Infrastructure: - Audit logging for all connector configuration changes - Authorization integration with NiFi's policy framework - UI extension support for connector-specific interfaces - Frontend updates for connector access policy management NIFI-15323: Adding operate permissions to the ConnectorEntity. (apache#10625) This closes apache#10625 NIFI-15322: Require all property descriptors within a PropertyGroup / ConfigurationStep are unique (apache#10626) * NIFI-15322: Huge refactoring of Connectors to require that <Step Name, Property Name> tuple is unique. Requiring that is simple, but now that we enforce that, we can greatly simplify much of the code. * NIFI-15322: Added StepConfiguration to simplify API rather than using Map<String, ConnectorValueReference> in some places * NIFI-15322: Now that we enforce unique property names per step, introduced StepConfigurationContext to further reduce API complexity Signed-off-by: Kevin Doran <kdoran@apache.org> NIFI-15326: Adding support of configuration step documentation. (apache#10631) NIFI-15322: Adapting to new connector configuration step model. NIFI-15330: Allow GhostConnector to be updated (apache#10632) NIFI-15312: Implementation of Parameter Provider based Secrets Manager; updated mock framework to support secrets; validation cleanup (apache#10638) NIFI-15336: Created AuthorizableSecret and implemented ParameterProviderSecretProvider to provide the Parameter Provider as the associated Authorizable (apache#10644) NIFI-15258: Enable Custom UI to retrieve connector with flow contexts (apache#10652) NIFI-15352: Added ConnectorClient to toolkit-cli so that it can be us… (apache#10654) * NIFI-15352: Added ConnectorClient to toolkit-cli so that it can be used in system tests; created system tests for Connectors; bug fixes found running tests; updated Secret/SecretReference to support fully qualified names * NIFI-15352: Addressed review feedback NIFI-15343: Adding an endpoint to return available secrets to the con… (apache#10645) * NIFI-15343: Adding an endpoint to return available secrets to the connector configuration wizard. * NIFI-15343: Adding providerId to SecretDTO. * NIFI-15343: Skipping ParameterProviders that are invalid or validating when fetching secrets. * NIFI-15343: Adding fully qualified name to SecretDTO. - Fixing rebase issue. This closes apache#10645 NIFI-15315 Add support for assets in connectors (apache#10647) - Support multiple asset references - Clean up of assets in StandardConnectorDAO after apply and discard - Add endpoint for listing assets for a given connector - Add endpoint for discarding the working config of a connector - Add endpoint for retrieving asset content - Add system test for connectors that uploads an asset Signed-off-by: Kevin Doran <kdoran@apache.org> * Fix issues after rebasing, enable ClusteredConnectorAssetsIT NIFI-15361: Allowing configuration step documentation to be returned … (apache#10667) * NIFI-15361: Allowing configuration step documentation to be returned independent of any connector instances. NIFI-15369: Allow ConfigurationStep to depend on another (ConfigurationStep,Property) tuple (apache#10669) NIFI-15367: Ensure that Connectors' implicit parameter contexts are n… (apache#10665) * NIFI-15367: Ensure that Connectors' implicit parameter contexts are not registered with ParameterContextManager; added system test to verify; fixed bug in ConnectorAssetsIT where it did not wait for connector to return to its 'STOPPED' state before attempting to delete * NIFI-15367: Bug fixes; review feedback NIFI-15370: Add Connector ID to process groups and do not register create process groups if they have a connector ID. (apache#10673) NIFI-15356: Adding authorization to the StandardNiFiConnectorWebContext. (apache#10660) NIFI-15353: Adding support for rendering Connector Documentation. (apache#10683) * NIFI-15353: Adding support for rendering Connector Documentation. * NIFI-15353: Addressing review feedback. * NIFI-15353: Using mergeMap instead of switchMap to allow for concurrent doc loading. This closes apache#10683 NIFI-15429: Adding an optional query parameter for specifying which p… (apache#10732) * NIFI-15429: Adding an optional query parameter for specifying which process group should be returned within the Connectors flow. * NIFI-15429: Including managed process group id in the ConnectorDTO. * NIFI-15429: Addressing review feedback. * NIFI-15429: Updating path for fetching a Connector flow to require the desired Process Group ID. * NIFI-15429: Fixing failing unit test. NIFI-15430: Ensure that if we fail to initialize a Connector, we crea… (apache#10733) * NIFI-15430: Ensure that if we fail to initialize a Connector, we create a GhostConnector instead and ensure that we also proivde the extensionMissing flag on ConnectorNode * NIFI-15430: Added extensionMissing flag to Connector DTO * NIFI-15430: If unable to load initial flow of a Connector, make ghosted connector instead NIFI-15427: Added abiliy to drop flowfiles / drain flowfiles from a C… (apache#10730) * NIFI-15427: Added abiliy to drop flowfiles / drain flowfiles from a Connector; added some system tests to verify existing behavior; fixed existing issue with IT * NIFI-15427: Fixed Exception message in case unable to purge FlowFiles due to Connector state * NIFI-15427: Ensured thread safety of state transitions when draining/purging FlowFiles * NIFI-15427: Addressed review feedback NIFI-15433: If connector validation throws an Exception keep trying u… (apache#10736) * NIFI-15433: If connector validation throws an Exception keep trying until it completes (with a timeout between retries); some bug fixes * NIFI-15433: Addressed review feedback * NIFI-15433: Fixed failing unit tests NIFI-15434: Ensure that we start or stop connectors on flow sync (apache#10737) NIFI-15445: Removed 'disabled' state from Connectors; also fixed bug … (apache#10749) * NIFI-15445: Removed 'disabled' state from Connectors; also fixed bug to ensure that if we fail to apply an update to a Connector that we stop it. * NIFI-15445: Addressed review feedback * NIFI-15445: Fixed checkstyle NIFI-15440: Implementation of ConnectorActions (apache#10748) * NIFI-15440: Implementation of ConnectorActions * NIFI-15440: Addressed review feedback * NIFI-15440: Addressed issues that occurred after rebase * NIFI-15440: Addressed review feedback NIFI-15439: Ensure that Process Groups are accessible from the approp… (apache#10742) * NIFI-15439: Ensure that Process Groups are accessible from the appropriate endpoint but not accessible elsewhere * NIFI-15439: FlowAnalyzingRegistryClient should use an explicit Connector ID of null when retrieving Process Group from FlowManager NIFI-15468: Ensure that we properly initialize ConnectorNode even if a Ghost Connector is created (apache#10775) This closes apache#10775 NIFI-15446: When invoking ConnectorMethod, make sure to serialize/des… (apache#10750) * NIFI-15446: When invoking ConnectorMethod, make sure to serialize/deserialize the arguments and return value * NIFI-15446: Addressed review feedback NIFI-15376: Adding dependencies to the configuration step dto. (apache#10674) NIFI-15453: Adding an endpoint to return all controller services for … (apache#10770) * NIFI-15453: Adding an endpoint to return all controller services for a process group in a connector. * NIFI-15453: Addressing review feedback. This closes apache#10770 NIFI-15472: Added addAsset methods to ConnectorTestRunner (apache#10776) NIFI-15482: MockConnectorAssetManager requires SPI file to start (apache#10783) NIFI-15451: Added ability for Connectors to retrieve bundles availabl… (apache#10756) * NIFI-15451: Added ability for Connectors to retrieve bundles available for component types and updated VersionedFlowUtils to make use of it to easily update Versioned components * NIFI-15451: Added additional unit tests to VersionedFlowUtils; updates to how we compare component versions * NIFI-15451: Fixed PMD violation NIFI-15481: Updated ConnectorActions to reflect that starting is not … (apache#10782) * NIFI-15481: Updated ConnectorActions to reflect that starting is not allowed when invalid. Also updated ValidationState to ensure that we don't return any ValidationResult with a valid flag of true as a 'validation error'. * NIFI-15481: Addressed review feedback NIFI-15258: Fixes necessary after rebasing against main NIFI-15258: Addressed checkstyle violations NIFI-15461: Added ability to initiate drainage of Connector's FlowFiles and calce; added tests to verify; some bug fixes (apache#10767) NIFI-15489: Addressed the 4 bugs called out in Jira: Wrong delimiter in StandardConnectorPropertyValue.asList, ConnectorValidationContextBridge.getProperty() does not resolve property value properly; StandaloneProcessGroupLifecycle.stop() does not await controller service disabling; switched hashmap in StandaloneProcessGroupLifecycle to ConcurrentHashMap (apache#10798) - Addressed another similar bug in which an invalid connector that is expected to be running would prevent NiFi from starting NIFI-15490: If any processors or controller services are missing, make Connector invalid (apache#10799) NIFI-15485: Include provider name when providing fqn for secrets (apache#10801) NIFI-15497: Fixed unit tests (apache#10805) NIFI-15258: Fixes for system tests that needed to be updated from 2.7.0-SNAPSHOT to 2.8.0-SNAPSHOT (apache#10807) NIFI-15495 Restart Connectors that reference assets that were synchro… (apache#10806) * NIFI-15495 Restart Connectors that reference assets that were synchronized - Ensure Connectors re-resolve property values before starting - Ensure asset clean up happens only after applyUpdate fully finishes - Add connector asset properties to default nifi.properties * Fix system test * Encapsulate restart logic in new method on ConnectorNode * Fix JavaDoc NIFI-15465 Allow client-specified connector component ids (apache#10768) Add verifyCreate to ConnectorDAO and test cases NIFI-15467: Ensure that Connectors' versions are automatically resolv… (apache#10811) * NIFI-15467: Ensure that Connectors' versions are automatically resolved on startup - Allow connectors to specify Bundle Compatability when updating flow - Default to BundleCompatability.RESOLVE_BUNDLE when connectors are updating flows - Bug fixes; updated MockConnectorInitializationContext to extend from StandardConnectorInitializationContext to cut down on code duplication * NIFI-15467: Addressed review feedback NIFI-15529: Fix to StatelessGroupLifecycle to ensure that we set the desired state of Stateless Group Node before attempting to schedule; updated StandardProcessScheduler.startStatelessGroup to make it more clear if this is left out again at some point (apache#10832) NIFI-15525: Ensure that we call onStepConfigured when we restore flow; ensure that we synchronized Parmaeter Providers before Connectors (apache#10835) NIFI-15509: Enable Component reload additional classpath resources if needed (apache#10812) NIFI-15536: Change Connector API to use DescribedValue vs AllowableValue (apache#10841) NIFI-15511: Added endpoints for purging all FlowFiles for a given Con… (apache#10833) * NIFI-15511: Added endpoints for purging all FlowFiles for a given Connector; added method to ConnectorClient to call these endpoints; updated system tests to use these endpoints when tearing down flows; some bug fixes * NIFI-15511: Added WRITE permission requirement for retrieving and deleting a Connector Purge request NIFI-15514: Ensure that Parameter Contexts are assigned to all Proces… (apache#10815) * NIFI-15514: Ensure that Parameter Contexts are assigned to all Process Groups in a Connector and not just the top-level group. Ensure that all components are started when Connector starts instead of just Processors and Controller Services - When Working Context is recreated, ensure that we appropriately apply Parameter Context to newly created Process Group(s) - Ensure that when we cleanup unused assets for Connectors that we consider any assets that are referenced in either the Working or Active context instead of just the Active context - Ensure that when we stop Process Group we call all tasks in background threads instead of calling .thenRun which could potentially run in the foreground thread * NIFI-15514: Update parameter context assignment to occur during sync. * NIFI-15514: Fixes around ensuring that processors/controller services are properly configured and notified of any configuration changes when parameters change - Removed the updateParameterContexts from ProcessGroup.updateFlow, which was added in a previous commit as we went a different direction for the fix --------- Co-authored-by: Bob Paulin <bob@bobpaulin.com> NIFI-15538: When starting/stopping components allow specifying whethe… (apache#10843) * NIFI-15538: When starting/stopping components allow specifying whether or not the action should be recursive; code cleanup and simplification to use a Virtual Thread to execute code sequentially instead of chaining CompletableFutures. * NIFI-15538: Addressed review feedback --------- Co-authored-by: Mark Payne <mark.payne@snowflake.com> NIFI-15480 Added ability to drop FlowFiles selectively using a Predicate (apache#10848) * NIFI-15480: Added ability to drop FlowFiles selectively using a Predicate Added system tests to verify selective dropping of flowfiles from a Connector. Also made significant updates so that diagnostic bundles are captured when a system test fails and diagnostic bundle also contains details of each Connection's FlowFileQueue. This was necessary in aiding the debugging of the new feature but is also something that has been missing for some time. * NIFI-15480: Ensure that when we perform a selective drop on a FlowFileQueue that we pass appropriate SWAP_FILE_DELETED / SWAP_FILE_RENAMED events to the FlowFile Repository NIFI-15549: Adding endpoints for viewing and clearing state for compo… (apache#10855) * NIFI-15549: Adding endpoints for viewing and clearing state for components that are managed by a Connector. * NIFI-15549: Addressing review feedback. NIFI-15557: Allow Connectors to enable a Controller Service using overridden property values (apache#10862) NIFI-15488: Added significant number of debug log messages as well as some info/error for connector-related events (apache#10803) NIFI-15544: If a Connector requires processors or controller services that are unavailable, make Connector invalid but not ghosted (apache#10851) NIFI-15558: Sensitive Parameter References should resolve to value. (apache#10863) NIFI-15539: Updating the actions in FlowFileQueueResource to support … (apache#10846) * NIFI-15539: Updating the actions in FlowFileQueueResource to support interacting with queues in flows managed by a Connector. * NIFI-15539: Updating resource endpoint to include data policies for connectors. NIFI-15565: Code cleanup; improvements to .cursorrules to provide bet… (apache#10869) * NIFI-15565: Code cleanup; improvements to .cursorrules to provide better instructions regarding automated tests * NIFI-15565: Broke apart cursorrules file into the new format of .cursor/rules/* NIFI-15355 Connector Configuration Repository (apache#10876) * NIFI-15255 Introduce ConnectorConfigurationRepository Extension Interface * Add new property key to default nifi.properties file * Address peer review comments NIFI-15559: Enforce data permissions when purging data from a Connector. (apache#10867) This closes apache#10867 NIFI-15258: Fixes as a result of rebasing against main NIFI-15577: Including connector id in provenance event dto. (apache#10883) * NIFI-15577: Including connector id in provenance event dto. - Fixing go to routing logic in the provenance event table. * NIFI-15577: Fixing back navigation context when navigating to provenance from the flow designer. This closes apache#10883 NIFI-15581: Add support for ControllerService Mocks for Connectors. (apache#10885) NIFI-15579: When synchronizing PG when Versioned flow for stateless group node, do not synchronize parameter contexts if group is within a Connector (apache#10884) NIFI-15605 - Rebase Connectors branch with main for 2.9.0-SNAPSHOT NIFI-15606: Connector Mock War implementation. (apache#10907) * NIFI-15606: Connector Mock War implementation. * Load war files from Connector Nar * Mock Connector Web Context * Allow test runner to set http port * NIFI-15606: Connector Mock War implementation. * Code Review Feedback NIFI-15611: Fetchable Values should pull from current FlowContext (apache#10906) NIFI-15618: Bug fixes around handling of FlowFileActivity with child groups and stateless groups (apache#10912) NIFI-15598: When enabling referenced controller services in a connector, skip references from properties with unsatisfied dependencies (apache#10897) NIFI-15615: Ensure that we include necessary values in the ConnectorStatusDTO (apache#10913) NIFI-15610 Add asset management to ConnectorConfigurationProvider (apache#10909) NIFI-15621: Allow Configuration to Validate with ValidationContext (apache#10916) * Currently this is only implemented for processors in schedule periods * Allows connector to pass Validation Context to be used instead of current parameter context. NIFI-15640: NPE in the applyFlow should NOT crash the runtime. (apache#10931) * NIFI-15640: NPE in the applyFlow should NOT crash the runtime. * Leave the flow in a state of UPDATE_FAILED * With validationState of INVALID. * NIFI-15640: NPE in the applyFlow should NOT crash the runtime. * Code Review Feedback NIFI-15629 Add ConnectorRequestContext (apache#10924) Signed-off-by: David Handermann <exceptionfactory@apache.org> NIFI-15663: Update to use new ComponentHierarchyScope instead of on facade/lifecycle classes (apache#10958) NIFI-15680: Not including global parameter contexts and parameters in connector search results. (apache#10980) This closes apache#10980 NIFI-15604: Expose VersionedExternalFlow that represents the Active/Working flow from ConnectorTestRunner (apache#10901) Signed-off-by: Kevin Doran <kdoran@apache.org> NIFI-15648 - Resolve SECRET property values before fetchAllowableValues and verify in working flow context (apache#10937)
markap14
added a commit
that referenced
this pull request
Mar 10, 2026
NIFI-15258: Fixed vulnerable lz4 dependency NIFI-15258: Fixed checkstyle violations/pmd NIFI-15258: Addressed PMD findings NIFI-15259: Added REST API for Connectors Introduced comprehensive REST endpoints for managing Connectors through the NiFi web interface and API clients. Key capabilities: - Full lifecycle management (create, read, update, delete) - Start/stop control with run status endpoints - Multi-step configuration with property groups supporting string literals, asset references, and secret references - Asynchronous configuration verification with polling support - Access to connector's managed flow and process group status - Search functionality within connector's encapsulated flow Cluster support: - Request replication across cluster nodes - Response merging for dynamic property allowable values that may differ between nodes - Consistent configuration step and property descriptor handling Infrastructure: - Audit logging for all connector configuration changes - Authorization integration with NiFi's policy framework - UI extension support for connector-specific interfaces - Frontend updates for connector access policy management NIFI-15323: Adding operate permissions to the ConnectorEntity. (#10625) This closes #10625 NIFI-15322: Require all property descriptors within a PropertyGroup / ConfigurationStep are unique (#10626) * NIFI-15322: Huge refactoring of Connectors to require that <Step Name, Property Name> tuple is unique. Requiring that is simple, but now that we enforce that, we can greatly simplify much of the code. * NIFI-15322: Added StepConfiguration to simplify API rather than using Map<String, ConnectorValueReference> in some places * NIFI-15322: Now that we enforce unique property names per step, introduced StepConfigurationContext to further reduce API complexity Signed-off-by: Kevin Doran <kdoran@apache.org> NIFI-15326: Adding support of configuration step documentation. (#10631) NIFI-15322: Adapting to new connector configuration step model. NIFI-15330: Allow GhostConnector to be updated (#10632) NIFI-15312: Implementation of Parameter Provider based Secrets Manager; updated mock framework to support secrets; validation cleanup (#10638) NIFI-15336: Created AuthorizableSecret and implemented ParameterProviderSecretProvider to provide the Parameter Provider as the associated Authorizable (#10644) NIFI-15258: Enable Custom UI to retrieve connector with flow contexts (#10652) NIFI-15352: Added ConnectorClient to toolkit-cli so that it can be us… (#10654) * NIFI-15352: Added ConnectorClient to toolkit-cli so that it can be used in system tests; created system tests for Connectors; bug fixes found running tests; updated Secret/SecretReference to support fully qualified names * NIFI-15352: Addressed review feedback NIFI-15343: Adding an endpoint to return available secrets to the con… (#10645) * NIFI-15343: Adding an endpoint to return available secrets to the connector configuration wizard. * NIFI-15343: Adding providerId to SecretDTO. * NIFI-15343: Skipping ParameterProviders that are invalid or validating when fetching secrets. * NIFI-15343: Adding fully qualified name to SecretDTO. - Fixing rebase issue. This closes #10645 NIFI-15315 Add support for assets in connectors (#10647) - Support multiple asset references - Clean up of assets in StandardConnectorDAO after apply and discard - Add endpoint for listing assets for a given connector - Add endpoint for discarding the working config of a connector - Add endpoint for retrieving asset content - Add system test for connectors that uploads an asset Signed-off-by: Kevin Doran <kdoran@apache.org> * Fix issues after rebasing, enable ClusteredConnectorAssetsIT NIFI-15361: Allowing configuration step documentation to be returned … (#10667) * NIFI-15361: Allowing configuration step documentation to be returned independent of any connector instances. NIFI-15369: Allow ConfigurationStep to depend on another (ConfigurationStep,Property) tuple (#10669) NIFI-15367: Ensure that Connectors' implicit parameter contexts are n… (#10665) * NIFI-15367: Ensure that Connectors' implicit parameter contexts are not registered with ParameterContextManager; added system test to verify; fixed bug in ConnectorAssetsIT where it did not wait for connector to return to its 'STOPPED' state before attempting to delete * NIFI-15367: Bug fixes; review feedback NIFI-15370: Add Connector ID to process groups and do not register create process groups if they have a connector ID. (#10673) NIFI-15356: Adding authorization to the StandardNiFiConnectorWebContext. (#10660) NIFI-15353: Adding support for rendering Connector Documentation. (#10683) * NIFI-15353: Adding support for rendering Connector Documentation. * NIFI-15353: Addressing review feedback. * NIFI-15353: Using mergeMap instead of switchMap to allow for concurrent doc loading. This closes #10683 NIFI-15429: Adding an optional query parameter for specifying which p… (#10732) * NIFI-15429: Adding an optional query parameter for specifying which process group should be returned within the Connectors flow. * NIFI-15429: Including managed process group id in the ConnectorDTO. * NIFI-15429: Addressing review feedback. * NIFI-15429: Updating path for fetching a Connector flow to require the desired Process Group ID. * NIFI-15429: Fixing failing unit test. NIFI-15430: Ensure that if we fail to initialize a Connector, we crea… (#10733) * NIFI-15430: Ensure that if we fail to initialize a Connector, we create a GhostConnector instead and ensure that we also proivde the extensionMissing flag on ConnectorNode * NIFI-15430: Added extensionMissing flag to Connector DTO * NIFI-15430: If unable to load initial flow of a Connector, make ghosted connector instead NIFI-15427: Added abiliy to drop flowfiles / drain flowfiles from a C… (#10730) * NIFI-15427: Added abiliy to drop flowfiles / drain flowfiles from a Connector; added some system tests to verify existing behavior; fixed existing issue with IT * NIFI-15427: Fixed Exception message in case unable to purge FlowFiles due to Connector state * NIFI-15427: Ensured thread safety of state transitions when draining/purging FlowFiles * NIFI-15427: Addressed review feedback NIFI-15433: If connector validation throws an Exception keep trying u… (#10736) * NIFI-15433: If connector validation throws an Exception keep trying until it completes (with a timeout between retries); some bug fixes * NIFI-15433: Addressed review feedback * NIFI-15433: Fixed failing unit tests NIFI-15434: Ensure that we start or stop connectors on flow sync (#10737) NIFI-15445: Removed 'disabled' state from Connectors; also fixed bug … (#10749) * NIFI-15445: Removed 'disabled' state from Connectors; also fixed bug to ensure that if we fail to apply an update to a Connector that we stop it. * NIFI-15445: Addressed review feedback * NIFI-15445: Fixed checkstyle NIFI-15440: Implementation of ConnectorActions (#10748) * NIFI-15440: Implementation of ConnectorActions * NIFI-15440: Addressed review feedback * NIFI-15440: Addressed issues that occurred after rebase * NIFI-15440: Addressed review feedback NIFI-15439: Ensure that Process Groups are accessible from the approp… (#10742) * NIFI-15439: Ensure that Process Groups are accessible from the appropriate endpoint but not accessible elsewhere * NIFI-15439: FlowAnalyzingRegistryClient should use an explicit Connector ID of null when retrieving Process Group from FlowManager NIFI-15468: Ensure that we properly initialize ConnectorNode even if a Ghost Connector is created (#10775) This closes #10775 NIFI-15446: When invoking ConnectorMethod, make sure to serialize/des… (#10750) * NIFI-15446: When invoking ConnectorMethod, make sure to serialize/deserialize the arguments and return value * NIFI-15446: Addressed review feedback NIFI-15376: Adding dependencies to the configuration step dto. (#10674) NIFI-15453: Adding an endpoint to return all controller services for … (#10770) * NIFI-15453: Adding an endpoint to return all controller services for a process group in a connector. * NIFI-15453: Addressing review feedback. This closes #10770 NIFI-15472: Added addAsset methods to ConnectorTestRunner (#10776) NIFI-15482: MockConnectorAssetManager requires SPI file to start (#10783) NIFI-15451: Added ability for Connectors to retrieve bundles availabl… (#10756) * NIFI-15451: Added ability for Connectors to retrieve bundles available for component types and updated VersionedFlowUtils to make use of it to easily update Versioned components * NIFI-15451: Added additional unit tests to VersionedFlowUtils; updates to how we compare component versions * NIFI-15451: Fixed PMD violation NIFI-15481: Updated ConnectorActions to reflect that starting is not … (#10782) * NIFI-15481: Updated ConnectorActions to reflect that starting is not allowed when invalid. Also updated ValidationState to ensure that we don't return any ValidationResult with a valid flag of true as a 'validation error'. * NIFI-15481: Addressed review feedback NIFI-15258: Fixes necessary after rebasing against main NIFI-15258: Addressed checkstyle violations NIFI-15461: Added ability to initiate drainage of Connector's FlowFiles and calce; added tests to verify; some bug fixes (#10767) NIFI-15489: Addressed the 4 bugs called out in Jira: Wrong delimiter in StandardConnectorPropertyValue.asList, ConnectorValidationContextBridge.getProperty() does not resolve property value properly; StandaloneProcessGroupLifecycle.stop() does not await controller service disabling; switched hashmap in StandaloneProcessGroupLifecycle to ConcurrentHashMap (#10798) - Addressed another similar bug in which an invalid connector that is expected to be running would prevent NiFi from starting NIFI-15490: If any processors or controller services are missing, make Connector invalid (#10799) NIFI-15485: Include provider name when providing fqn for secrets (#10801) NIFI-15497: Fixed unit tests (#10805) NIFI-15258: Fixes for system tests that needed to be updated from 2.7.0-SNAPSHOT to 2.8.0-SNAPSHOT (#10807) NIFI-15495 Restart Connectors that reference assets that were synchro… (#10806) * NIFI-15495 Restart Connectors that reference assets that were synchronized - Ensure Connectors re-resolve property values before starting - Ensure asset clean up happens only after applyUpdate fully finishes - Add connector asset properties to default nifi.properties * Fix system test * Encapsulate restart logic in new method on ConnectorNode * Fix JavaDoc NIFI-15465 Allow client-specified connector component ids (#10768) Add verifyCreate to ConnectorDAO and test cases NIFI-15467: Ensure that Connectors' versions are automatically resolv… (#10811) * NIFI-15467: Ensure that Connectors' versions are automatically resolved on startup - Allow connectors to specify Bundle Compatability when updating flow - Default to BundleCompatability.RESOLVE_BUNDLE when connectors are updating flows - Bug fixes; updated MockConnectorInitializationContext to extend from StandardConnectorInitializationContext to cut down on code duplication * NIFI-15467: Addressed review feedback NIFI-15529: Fix to StatelessGroupLifecycle to ensure that we set the desired state of Stateless Group Node before attempting to schedule; updated StandardProcessScheduler.startStatelessGroup to make it more clear if this is left out again at some point (#10832) NIFI-15525: Ensure that we call onStepConfigured when we restore flow; ensure that we synchronized Parmaeter Providers before Connectors (#10835) NIFI-15509: Enable Component reload additional classpath resources if needed (#10812) NIFI-15536: Change Connector API to use DescribedValue vs AllowableValue (#10841) NIFI-15511: Added endpoints for purging all FlowFiles for a given Con… (#10833) * NIFI-15511: Added endpoints for purging all FlowFiles for a given Connector; added method to ConnectorClient to call these endpoints; updated system tests to use these endpoints when tearing down flows; some bug fixes * NIFI-15511: Added WRITE permission requirement for retrieving and deleting a Connector Purge request NIFI-15514: Ensure that Parameter Contexts are assigned to all Proces… (#10815) * NIFI-15514: Ensure that Parameter Contexts are assigned to all Process Groups in a Connector and not just the top-level group. Ensure that all components are started when Connector starts instead of just Processors and Controller Services - When Working Context is recreated, ensure that we appropriately apply Parameter Context to newly created Process Group(s) - Ensure that when we cleanup unused assets for Connectors that we consider any assets that are referenced in either the Working or Active context instead of just the Active context - Ensure that when we stop Process Group we call all tasks in background threads instead of calling .thenRun which could potentially run in the foreground thread * NIFI-15514: Update parameter context assignment to occur during sync. * NIFI-15514: Fixes around ensuring that processors/controller services are properly configured and notified of any configuration changes when parameters change - Removed the updateParameterContexts from ProcessGroup.updateFlow, which was added in a previous commit as we went a different direction for the fix --------- Co-authored-by: Bob Paulin <bob@bobpaulin.com> NIFI-15538: When starting/stopping components allow specifying whethe… (#10843) * NIFI-15538: When starting/stopping components allow specifying whether or not the action should be recursive; code cleanup and simplification to use a Virtual Thread to execute code sequentially instead of chaining CompletableFutures. * NIFI-15538: Addressed review feedback --------- Co-authored-by: Mark Payne <mark.payne@snowflake.com> NIFI-15480 Added ability to drop FlowFiles selectively using a Predicate (#10848) * NIFI-15480: Added ability to drop FlowFiles selectively using a Predicate Added system tests to verify selective dropping of flowfiles from a Connector. Also made significant updates so that diagnostic bundles are captured when a system test fails and diagnostic bundle also contains details of each Connection's FlowFileQueue. This was necessary in aiding the debugging of the new feature but is also something that has been missing for some time. * NIFI-15480: Ensure that when we perform a selective drop on a FlowFileQueue that we pass appropriate SWAP_FILE_DELETED / SWAP_FILE_RENAMED events to the FlowFile Repository NIFI-15549: Adding endpoints for viewing and clearing state for compo… (#10855) * NIFI-15549: Adding endpoints for viewing and clearing state for components that are managed by a Connector. * NIFI-15549: Addressing review feedback. NIFI-15557: Allow Connectors to enable a Controller Service using overridden property values (#10862) NIFI-15488: Added significant number of debug log messages as well as some info/error for connector-related events (#10803) NIFI-15544: If a Connector requires processors or controller services that are unavailable, make Connector invalid but not ghosted (#10851) NIFI-15558: Sensitive Parameter References should resolve to value. (#10863) NIFI-15539: Updating the actions in FlowFileQueueResource to support … (#10846) * NIFI-15539: Updating the actions in FlowFileQueueResource to support interacting with queues in flows managed by a Connector. * NIFI-15539: Updating resource endpoint to include data policies for connectors. NIFI-15565: Code cleanup; improvements to .cursorrules to provide bet… (#10869) * NIFI-15565: Code cleanup; improvements to .cursorrules to provide better instructions regarding automated tests * NIFI-15565: Broke apart cursorrules file into the new format of .cursor/rules/* NIFI-15355 Connector Configuration Repository (#10876) * NIFI-15255 Introduce ConnectorConfigurationRepository Extension Interface * Add new property key to default nifi.properties file * Address peer review comments NIFI-15559: Enforce data permissions when purging data from a Connector. (#10867) This closes #10867 NIFI-15258: Fixes as a result of rebasing against main NIFI-15577: Including connector id in provenance event dto. (#10883) * NIFI-15577: Including connector id in provenance event dto. - Fixing go to routing logic in the provenance event table. * NIFI-15577: Fixing back navigation context when navigating to provenance from the flow designer. This closes #10883 NIFI-15581: Add support for ControllerService Mocks for Connectors. (#10885) NIFI-15579: When synchronizing PG when Versioned flow for stateless group node, do not synchronize parameter contexts if group is within a Connector (#10884) NIFI-15605 - Rebase Connectors branch with main for 2.9.0-SNAPSHOT NIFI-15606: Connector Mock War implementation. (#10907) * NIFI-15606: Connector Mock War implementation. * Load war files from Connector Nar * Mock Connector Web Context * Allow test runner to set http port * NIFI-15606: Connector Mock War implementation. * Code Review Feedback NIFI-15611: Fetchable Values should pull from current FlowContext (#10906) NIFI-15618: Bug fixes around handling of FlowFileActivity with child groups and stateless groups (#10912) NIFI-15598: When enabling referenced controller services in a connector, skip references from properties with unsatisfied dependencies (#10897) NIFI-15615: Ensure that we include necessary values in the ConnectorStatusDTO (#10913) NIFI-15610 Add asset management to ConnectorConfigurationProvider (#10909) NIFI-15621: Allow Configuration to Validate with ValidationContext (#10916) * Currently this is only implemented for processors in schedule periods * Allows connector to pass Validation Context to be used instead of current parameter context. NIFI-15640: NPE in the applyFlow should NOT crash the runtime. (#10931) * NIFI-15640: NPE in the applyFlow should NOT crash the runtime. * Leave the flow in a state of UPDATE_FAILED * With validationState of INVALID. * NIFI-15640: NPE in the applyFlow should NOT crash the runtime. * Code Review Feedback NIFI-15629 Add ConnectorRequestContext (#10924) Signed-off-by: David Handermann <exceptionfactory@apache.org> NIFI-15663: Update to use new ComponentHierarchyScope instead of on facade/lifecycle classes (#10958) NIFI-15680: Not including global parameter contexts and parameters in connector search results. (#10980) This closes #10980 NIFI-15604: Expose VersionedExternalFlow that represents the Active/Working flow from ConnectorTestRunner (#10901) Signed-off-by: Kevin Doran <kdoran@apache.org> NIFI-15648 - Resolve SECRET property values before fetchAllowableValues and verify in working flow context (#10937) Co-authored-by: Bob Paulin <bob@bobpaulin.com> Co-authored-by: Bryan Bende <bbende@apache.org> Co-authored-by: Kevin Doran <kdoran@apache.org> Co-authored-by: Matt Gilman <matt.c.gilman@gmail.com> Co-authored-by: Pierre Villard <pierre.villard.fr@gmail.com> Co-authored-by: Mark Payne <markap14@hotmail.com>
12 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…e for component types and updated VersionedFlowUtils to make use of it to easily update Versioned components
Summary
NIFI-00000
Tracking
Please complete the following tracking steps prior to pull request creation.
Issue Tracking
Pull Request Tracking
NIFI-00000NIFI-00000VerifiedstatusPull Request Formatting
mainbranchVerification
Please indicate the verification steps performed prior to pull request creation.
Build
./mvnw clean install -P contrib-checkLicensing
LICENSEandNOTICEfilesDocumentation