nifi: add dashboard, monitors, log pipeline, and other assets#24561
nifi: add dashboard, monitors, log pipeline, and other assets#24561steveny91 wants to merge 12 commits into
Conversation
Adds the OOTB overview dashboard, three recommended monitors (API unreachable, cluster degraded, processors in an invalid state), a log pipeline for nifi-app.log (startup/shutdown/error/audit-trail events), dataflows.yaml, three Log Explorer saved views, and two Cloud SIEM detection rules for the already-released NiFi Agent check. No check code changes.
Verified locally with `towncrier build --draft` that these render as expected under a future 1.1.0 entry.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: aae2928829
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| { | ||
| "data_source": "metrics", | ||
| "name": "query1", | ||
| "query": "min:nifi.can_connect{$host,$nifi_version}", |
There was a problem hiding this comment.
When the NiFi API is unreachable, the check emits nifi.can_connect = 0 without the nifi_version tag because it cannot fetch /flow/about (check.py emits the failure metric with only instance tags). In dashboards filtered by nifi_version, this query excludes exactly those failure samples, so the availability widget can go blank or stay stale instead of turning red for a down API. Remove $nifi_version from the can_connect query or tag failures with a cached version.
Useful? React with 👍 / 👎.
| id: "nifi" | ||
| tests: |
There was a problem hiding this comment.
The logs pipeline test fixtures in this repo use the *_tests.yaml suffix, while this new file is the only *-pipeline-tests.yaml; tooling that discovers parser tests by the established suffix will not exercise these samples, and asset packaging can treat this as another logs asset instead of a fixture. Rename it to nifi_tests.yaml next to nifi.yaml so the NiFi parser samples are picked up.
Useful? React with 👍 / 👎.
This comment has been minimized.
This comment has been minimized.
domalessi
left a comment
domalessi
left a comment
There was a problem hiding this comment.
Left some minor editorial suggestions!
Co-authored-by: domalessi <111786334+domalessi@users.noreply.github.com>
Co-authored-by: domalessi <111786334+domalessi@users.noreply.github.com>
Co-authored-by: domalessi <111786334+domalessi@users.noreply.github.com>
Co-authored-by: domalessi <111786334+domalessi@users.noreply.github.com>
Co-authored-by: domalessi <111786334+domalessi@users.noreply.github.com>
Co-authored-by: domalessi <111786334+domalessi@users.noreply.github.com>
Co-authored-by: domalessi <111786334+domalessi@users.noreply.github.com>
Co-authored-by: domalessi <111786334+domalessi@users.noreply.github.com>
Validation ReportAll 21 validations passed. Show details
|
2 participants
Summary
Adds the Datadog assets for the NiFi Agent check (already implemented and released as v1.0.0 in #23110) — this PR ships no check code changes, only assets that were pending from the original implementation:
nifi-app.log, capturing startup, shutdown, error, and audit trail events (assets/logs/nifi.yaml+ pipeline tests).dataflows.yamldeclaring the metrics and logs dataflows.metadata.csvwas reviewed against the integration's RFC and left unchanged — the RFC listsnifi.system.jvm.uptime, but that metric is not actually emitted anywhere in the shipped check (check.py/constants.py/api.py), so it was not added to avoid documenting a metric that never reports data. Worth a follow-up with whoever owns the check to either implement it or update the RFC.Known gap, out of scope for this PR:
nifi/manifest.jsondoes not exist in this repo despitepyproject.toml's sdist target listing it — this predates this PR and will need to be created separately before a Publishing Platform listing can be created.A CHANGELOG fragment will follow in a separate commit on this branch once the PR number is available (this repo's
nifi/CHANGELOG.mduses towncrier fragments underchangelog.d/, not direct edits).Test plan
ddev validate dashboards nifiddev validate metadata nifi(confirms unchanged metadata.csv still validates)nifi-app.logoutput (validated here against the RFC's documented log4j pattern and hand-traced samples; not run against a live NiFi instance)🤖 Generated with the integration-dev generate-assets / update-integration-repo skills, human-reviewed.