MINIFICPP-2870 Lint/Check python code with ruff

[martinzink]

MINIFICPP-2870 Lint/Check python code with ruff

I've deliberetly done the PR in multiple commits, because that would make the review process much easier.

• ruff format this is the result of running ruff format from the root of the repo
• setting ruff rules to ignore false positives
• ruff check --fix these are the auto fixes from ruff
• the remaining non-auto fixes these are the manual fixes (to problems ruff identified)
• documentation changes
• d4c979c I needed to upgrade to python 3.10 because 3.6 didnt support bunch of now standard rules (e.g. list vs List, dict vs Dict), but 3.6 is long EoL
• rebase fix
• extend-exclude excluded and reverted changed in thirdparty

---

Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.

In order to streamline the review of the contribution we ask you to ensure the following steps have been taken:

For all changes:

• Is there a JIRA ticket associated with this PR? Is it referenced in the commit message?

• Does your PR title start with MINIFICPP-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

• Has your PR been rebased against the latest commit within the target branch (typically main)?

• Is your initial contribution a single, squashed commit?

For code changes:

• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• If applicable, have you updated the LICENSE file?
• If applicable, have you updated the NOTICE file?

For documentation related changes:

• Have you ensured that format looks appropriate for the output in which it is rendered?

Note:

Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible.

[Copilot]

Pull request overview

Note

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

This PR replaces Flake8-based Python linting with Ruff, adds Ruff configuration, and applies repo-wide Ruff formatting/autofixes while also bumping the project’s Python baseline to 3.10.

Changes:

• Add ruff.toml configuration and update CI to run ruff check and ruff format --check.
• Remove Flake8 integration (run_flake8.sh + CMake flake8 target) and update contributing docs accordingly.
• Apply Ruff formatting / lint fixes across Python sources and Behave tests; update CMake Python discovery and Python limited API version.

Reviewed changes

Copilot reviewed 159 out of 159 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
thirdparty/google-styleguide/run_linter.py	Ruff-format refactor of the cpplint runner script.
run_flake8.sh	Removes legacy Flake8 wrapper script.
ruff.toml	Adds Ruff config (builtins, ignores, per-file ignores).
minifi_rust/extensions/minifi_rs_playground/features/steps/steps.py	Import reformat and Ruff cleanups.
minifi_rust/extensions/minifi_rs_playground/features/environment.py	Type hint modernization and formatting.
extensions/standard-processors/tests/features/steps/minifi_controller_steps.py	Ruff formatting and minor refactors in step definitions.
extensions/standard-processors/tests/features/steps/minifi_c2_server_container.py	Ruff formatting + minor Python modernizations.
extensions/standard-processors/tests/features/environment.py	Formatting + minor conditional simplification.
extensions/standard-processors/tests/features/containers/tcp_client_container.py	Formatting of container init/deploy calls.
extensions/standard-processors/tests/features/containers/syslog_container.py	Modern super() usage + formatting.
extensions/standard-processors/tests/features/containers/diag_slave_container.py	Switch to module logger + formatting.
extensions/sql/tests/features/steps/steps.py	Ruff formatting; reorganized imports.
extensions/sql/tests/features/environment.py	Refactors Dockerfile templating + formatting.
extensions/sql/tests/features/containers/postgress_server_container.py	Adds module logger + formatting.
extensions/splunk/tests/features/steps/steps.py	Ruff formatting; long assertions wrapped.
extensions/splunk/tests/features/environment.py	Import ordering + formatting.
extensions/python/tests/test_python_scripts/stateful_processor.py	Formatting + minor Python 3 style updates.
extensions/python/tests/test_python_scripts/passthrough_processor_transfering_to_success.py	Formatting + string style updates.
extensions/python/tests/test_python_scripts/passthrough_processor_transfering_to_failure.py	Formatting + string style updates.
extensions/python/tests/test_python_scripts/non_transferring_processor.py	Formatting + string style updates.
extensions/python/tests/test_python_scripts/foo_bar_processor.py	Import ordering + formatting.
extensions/python/tests/features/resources/subtractutils.py	Ruff-format whitespace adjustments.
extensions/python/tests/features/resources/multiplierutils.py	Ruff-format whitespace adjustments.
extensions/python/tests/features/resources/TestStateManager.py	Adds ClassVar typing + formatting.
extensions/python/tests/features/resources/SleepForever.py	Adds ClassVar typing + formatting.
extensions/python/tests/features/resources/SingleThreadedSleepForever.py	Adds ClassVar typing + import reordering.
extensions/python/tests/features/resources/SetRecordField.py	Import grouping + minor iteration simplification.
extensions/python/tests/features/resources/RotatingForwarder.py	Formatting of long expressions.
extensions/python/tests/features/resources/RelativeImporterProcessor.py	Minor import ordering + f-string use.
extensions/python/tests/features/resources/ProcessContextInterfaceChecker.py	Adds ClassVar typing + formatting.
extensions/python/tests/features/resources/NifiStyleLogDynamicProperties.py	Adds ClassVar typing + f-string logging.
extensions/python/tests/features/resources/LogDynamicProperties.py	f-string logging updates.
extensions/python/tests/features/resources/ExpressionLanguagePropertyWithValidator.py	Adds ClassVar typing + f-string logging.
extensions/python/tests/features/resources/EvaluateExpressionLanguageChecker.py	Adds ClassVar typing + formatting.
extensions/python/tests/features/resources/CreateNothing.py	Adds ClassVar typing, formatting; removes redundant return None.
extensions/python/tests/features/resources/CreateFlowFile.py	Adds typing + formatting and minor style updates.
extensions/python/pythonprocessors/nifiapi/relationship.py	Ruff-format whitespace adjustment.
extensions/python/pythonprocessors/nifiapi/recordtransform.py	Formatting + f-string error messages.
extensions/python/pythonprocessors/nifiapi/processorbase.py	Type hint modernization + formatting.
extensions/python/pythonprocessors/nifiapi/flowfiletransform.py	Formatting + f-string error messages.
extensions/python/pythonprocessors/nifiapi/flowfilesource.py	Formatting + f-string error messages.
extensions/python/pythonprocessors/nifiapi/documentation.py	Type hint modernization (`
extensions/python/pythonprocessors/nifiapi/decorators.py	Ruff-format whitespace adjustment.
extensions/python/pythonprocessors/nifiapi/componentstate.py	Type hint modernization to built-in generics.
extensions/python/pythonprocessors/nifi_python_processors/utils/dependency_installer.py	Formatting + small simplifications.
extensions/python/pythonprocessor-examples/h2o/ConvertDsToCsv.py	Formatting + minor docstring cleanup.
extensions/python/pythonprocessor-examples/google/SentimentAnalyzer.py	Formatting + minor import grouping.
extensions/python/pythonprocessor-examples/SentimentAnalysis.py	Formatting + string style standardization.
extensions/python/pythonprocessor-examples/RemoveFlowFile.py	Removes shebang line.
extensions/python/pythonprocessor-examples/MoveContentToJson.py	Formatting + string style standardization.
extensions/python/pythonprocessor-examples/GaussianDistributionWithNumpy.py	Formatting + string style standardization.
extensions/python/pythonprocessor-examples/CountingProcessor.py	Removes redundant global + formatting.
extensions/python/pythonprocessor-examples/AddPythonAttribute.py	Removes shebang line.
extensions/python/PYTHON.md	Updates code examples to match new formatting/quoting.
extensions/python/CMakeLists.txt	Bumps Py_LIMITED_API to Python 3.10.
extensions/prometheus/tests/features/steps/steps.py	Formatting + long-line wrapping.
extensions/prometheus/tests/features/environment.py	Import ordering + formatting.
extensions/opc/tests/features/steps/steps.py	Adds module logger + formatting.
extensions/opc/tests/features/environment.py	Import ordering + string quoting standardization.
extensions/opc/tests/features/containers/opc_ua_server_container.py	Type hint modernization + formatting.
extensions/mqtt/tests/features/steps/steps.py	Formatting + long-line wrapping.
extensions/mqtt/tests/features/resources/publish_mqtt_message.py	Import ordering + formatting.
extensions/mqtt/tests/features/environment.py	Import ordering + formatting.
extensions/mqtt/tests/features/containers/mqtt_broker_container.py	Adds module logger + formatting.
extensions/lua/tests/features/steps/steps.py	Consolidates imports for Ruff formatting.
extensions/lua/tests/features/environment.py	Import ordering + quoting standardization.
extensions/llamacpp/tests/features/steps/steps.py	Consolidates imports for Ruff formatting.
extensions/llamacpp/tests/features/environment.py	Formatting + multi-line string building cleanup.
extensions/kubernetes/tests/features/steps/steps.py	Formatting + long-line wrapping.
extensions/kubernetes/tests/features/steps/minifi_as_pod_in_kubernetes_cluster.py	Adds module logger + formatting.
extensions/kubernetes/tests/features/environment.py	Import ordering + env var access formatting.
extensions/kafka/tests/features/environment.py	Import ordering + formatting.
extensions/grafana-loki/tests/features/steps/steps.py	Formatting + long-line wrapping.
extensions/grafana-loki/tests/features/resources/check_log_lines_on_grafana.py	Formatting + type hint modernization.
extensions/grafana-loki/tests/features/environment.py	Formatting + long-line wrapping.
extensions/grafana-loki/tests/features/containers/reverse_proxy_container.py	Formatting of container init/deploy calls.
extensions/grafana-loki/tests/features/containers/grafana_loki_container.py	Adds module logger + replaces .format with f-strings.
extensions/gcp/tests/features/steps/steps.py	Formatting + long-line wrapping.
extensions/gcp/tests/features/environment.py	Import ordering + formatting.
extensions/gcp/tests/features/containers/fake_gcs_server_container.py	Adds module logger + formatting.
extensions/elasticsearch/tests/features/environment.py	Import ordering + formatting.
extensions/elasticsearch/tests/features/containers/opensearch_container.py	Adds module logger + formatting.
extensions/elasticsearch/tests/features/containers/elasticsearch_container.py	Adds module logger + formatting.
extensions/elasticsearch/tests/features/containers/elastic_base_container.py	Formatting + long command wrapping.
extensions/couchbase/tests/features/steps/steps.py	Formatting + long-line wrapping.
extensions/couchbase/tests/features/environment.py	Import ordering + formatting.
extensions/civetweb/tests/features/steps/steps.py	Consolidates imports for Ruff formatting.
extensions/civetweb/tests/features/environment.py	Import grouping + formatting.
extensions/azure/tests/features/steps/steps.py	Formatting + long-line wrapping.
extensions/azure/tests/features/environment.py	Import ordering + formatting.
extensions/aws/tests/features/steps/steps.py	Formatting + long-line wrapping.
extensions/aws/tests/features/environment.py	Import ordering + formatting.
extensions/aws/tests/features/containers/s3_server_container.py	Adds module logger + fixes default-arg dict pattern.
extensions/aws/tests/features/containers/kinesis_server_container.py	Adds module logger + formatting.
examples/scripts/python/reverse_flow_file_content.py	Formatting + quoting standardization.
docker/rockylinux/Dockerfile	Updates Python devel package name (Rocky build).
cmake/GenericPython.cmake	Raises required Python version to 3.10.
bootstrap/system_dependency.py	Type hint modernization + formatting.
bootstrap/minifi_option.py	Type hint modernization + formatting and refactors.
bootstrap/main.py	Formatting + CLI option definitions wrapped.
bootstrap/cmake_parser.py	Formatting + minor string quoting changes.
behave_framework/src/minifi_behave/steps/configuration_steps.py	Import consolidation + formatting.
behave_framework/src/minifi_behave/minifi/remote_process_group.py	Formatting of YAML dict construction.
behave_framework/src/minifi_behave/minifi/remote_port.py	Formatting of YAML dict construction.
behave_framework/src/minifi_behave/minifi/processor.py	Formatting + YAML dict keys normalized.
behave_framework/src/minifi_behave/minifi/parameter_context.py	Formatting + YAML dict keys normalized.
behave_framework/src/minifi_behave/minifi/parameter.py	Formatting + YAML dict keys normalized.
behave_framework/src/minifi_behave/minifi/output_port.py	Formatting of JSON dict construction.
behave_framework/src/minifi_behave/minifi/minifi_flow_definition.py	Formatting + dict construction refactor.
behave_framework/src/minifi_behave/minifi/input_port.py	Formatting of JSON dict construction.
behave_framework/src/minifi_behave/minifi/funnel.py	Formatting of YAML dict construction.
behave_framework/src/minifi_behave/minifi/flow_definition.py	Formatting + method wrapping.
behave_framework/src/minifi_behave/minifi/controller_service.py	Formatting of YAML dict construction.
behave_framework/src/minifi_behave/core/ssl_utils.py	Import consolidation + formatting.
behave_framework/src/minifi_behave/core/minifi_test_context.py	Import ordering + formatting.
behave_framework/src/minifi_behave/core/hooks.py	Adds module logger + formatting.
behave_framework/src/minifi_behave/core/helpers.py	Adds module logger + formatting and wrapping.
behave_framework/src/minifi_behave/containers/minifi_win_container.py	Adds module logger + formatting.
behave_framework/src/minifi_behave/containers/minifi_protocol.py	One-line Protocol stubs + formatting.
behave_framework/src/minifi_behave/containers/minifi_controller.py	Adds module logger + formatting.
behave_framework/src/minifi_behave/containers/http_proxy_container.py	Formatting + complex boolean wrapping.
behave_framework/src/minifi_behave/containers/host_file.py	Ruff-format whitespace adjustment.
behave_framework/src/minifi_behave/containers/file.py	Ruff-format whitespace adjustment.
behave_framework/src/minifi_behave/containers/docker_image_builder.py	Adds module logger + formatting.
behave_framework/src/minifi_behave/containers/directory.py	Ruff-format whitespace adjustment.
behave_framework/src/minifi_behave/containers/container_protocol.py	One-line Protocol stubs + formatting.
CONTRIBUTING.md	Updates Python linting guidance to Ruff + improves shellcheck docs formatting.
CMakeLists.txt	Removes Flake8 CMake target.
.github/workflows/ci.yml	Replaces Flake8 with Ruff in linters job; updates runner label.
.github/github_scripts/github_actions_cache_cleanup_tests.py	Formatting and small refactors in unit tests.
.github/github_scripts/github_actions_cache_cleanup.py	Type hint modernization + formatting.

Comments suppressed due to low confidence (3)

extensions/python/pythonprocessors/nifiapi/recordtransform.py:1

• In this loop, result does not refer to the current partition’s result metadata; it refers to an outer-scope variable (the list-comprehension variable does not leak in Python 3). As written, all partitioned flow files can be transferred based on an unrelated/previous result. Derive the relationship from single_partition_results (e.g., the first element, or enforce/validate that all results in the partition share the same relationship) and use that variable for transfer decisions.
  thirdparty/google-styleguide/run_linter.py:1
• --includePaths is optional, but the script iterates over args.includePaths unconditionally. If -i/--includePaths is omitted, this will raise a TypeError at runtime. Make the argument required, or provide a default (e.g., empty list / current directory) and handle the empty case explicitly.
  ruff.toml:1
• These globs only match direct children of the directories and may not apply to nested packages/modules. If the intent is to ignore G010 for all Python sources under these trees, switch to recursive patterns (e.g., extensions/python/** or extensions/python/**/*.py, depending on what Ruff expects in this repo) so the rule is applied consistently.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[martinzink]

It extends the default (source: https://docs.astral.sh/ruff/configuration/)

exclude = [
    ".bzr",
    ".direnv",
    ".eggs",
    ".git",
    ".git-rewrite",
    ".hg",
    ".ipynb_checkpoints",
    ".mypy_cache",
    ".nox",
    ".pants.d",
    ".pyenv",
    ".pytest_cache",
    ".pytype",
    ".ruff_cache",
    ".svn",
    ".tox",
    ".venv",
    ".vscode",
    "__pypackages__",
    "_build",
    "buck-out",
    "build",
    "dist",
    "node_modules",
    "site-packages",
    "venv",
]