Skip to content

feat(util-genai): add multimodal upload usage recorder hooks#251

Open
xiongyunn wants to merge 2 commits into
alibaba:mainfrom
xiongyunn:xy/multimodal-self-monitoring
Open

feat(util-genai): add multimodal upload usage recorder hooks#251
xiongyunn wants to merge 2 commits into
alibaba:mainfrom
xiongyunn:xy/multimodal-self-monitoring

Conversation

@xiongyunn

Copy link
Copy Markdown

Description

Add a lightweight MultimodalUsageRecorder interface and default no-op implementation to opentelemetry-util-genai, and call it from FsUploader on terminal upload success/error.

Callers can register a custom recorder via set_multimodal_usage_recorder() to record upload success (provider + payload bytes) and low-cardinality failure reasons. The default recorder is no-op, so behavior is unchanged when nothing is registered.

Fixes # (issue)

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

  • tox -e py312-test-util-genai
  • Targeted tests:
    • util/opentelemetry-util-genai/tests/_multimodal_upload/test_usage_recorder.py
    • util/opentelemetry-util-genai/tests/_multimodal_upload/test_fs_uploader_usage_metrics.py

Does This PR Require a Core Repo Change?

  • Yes. - Link to PR:
  • No.

Checklist:

See contributing.md for styleguide, changelog guidelines, and more.

  • Followed the style guidelines of this project
  • Changelogs have been updated
  • Unit tests have been added
  • Documentation has been updated

@ralf0131 ralf0131 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall Assessment

Adds an optional MultimodalUsageRecorder hook to FsUploader for recording upload success/error terminal events, with a safe no-op default. Clean, backward-compatible, additive design — no existing behavior changes. Code looks good overall.

Severity Breakdown

🔴 Critical: 0 | 🟠 High: 0 | 🟡 Medium: 1 | 🔵 Low: 0 | ℹ️ Info: 1 | ✅ Praise: 1

Action Required

Severity Finding Location
🟡 Medium Recorder calls are not exception-guarded; a RuntimeError raised by a custom recorder would be caught by the upload retry clause, triggering an unnecessary retry of an already-succeeded upload fs_uploader.py:462464
ℹ️ Info raise_on_success test-fixture field is defined but never exercised by any test test_usage_recorder.py:34

Details

[Medium] Guard recorder calls against exceptionsfs_uploader.py:462
In _do_upload, record_success() is called on the success path right before return, inside the while loop whose except (OSError, IOError, RuntimeError) clause (line 464) handles storage retries. If a custom recorder raises RuntimeError from record_upload_success(), the exception is caught by this clause and the upload is retried — re-writing a file that already succeeded. The terminal_recorded flag (set at line 399 before the recorder call) correctly prevents double-counting, so this is a robustness/efficiency concern rather than a correctness bug. The default no-op recorder is safe, but a downstream metrics recorder could misbehave under resource pressure. Consider wrapping the recorder invocations (record_success, record_error, and the direct recorder.record_upload_error(...) calls in upload()) in try/except Exception with a debug-level log, so a faulty recorder cannot disrupt the upload flow — consistent with the repo's "minimal overhead / must not disrupt operation" review focus.

[Info] Untested raise_on_success fieldtest_usage_recorder.py:34
RecordingUsageRecorder.raise_on_success toggles a RuntimeError in record_upload_success, but no test sets it to True. Either add a test exercising the raise path (e.g. verifying the uploader stays stable when the recorder raises) or drop the field to keep the fixture minimal.

Positive

Excellent work on this contribution! A few things worth calling out:

  • The terminal_recorded closure is a clean, correct way to guarantee exactly one terminal event per task — no double-counting of success/error.
  • Test coverage is thorough: success, cache-hit no-op, queue-full, invalid-item, download-failed, storage-error-after-retry, and remote-URI paths are all covered.
  • provider_label_from_protocol keeps the provider label low-cardinality (oss/sls/other) — good for metrics hygiene.
  • The no-op default means zero behavior change for existing users — fully backward compatible.

CI Note

CI workflows were in action_required state (first-time contributor — workflows require maintainer approval to run). I've approved the pending workflow runs so the Lint/Test/Misc checks can execute. I'll re-check CI status on the next patrol and approve once it's green. Reviewing now as COMMENT; will APPROVE after CI passes.


Automated review by github-manager-bot

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants