docs: add noveum trace observability integration - #6819
Conversation
📝 WalkthroughWalkthroughChangesNoveum Trace documentation
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/edge/en/observability/noveum-trace.mdx`:
- Around line 43-56: Update the Crew setup example around setup_crewai_tracing
and crew.kickoff to assign the returned listener to crew.callback_function
before starting the crew, while preserving the existing listener.shutdown call
after execution.
- Around line 53-57: Update the CrewAI example around crew.kickoff() to use a
try...finally block, ensuring listener.shutdown() and noveum_trace.flush()
execute whether kickoff succeeds or raises. Preserve the existing successful
result assignment and cleanup order, with trace flushing performed before
exiting.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: c4401870-261b-4e63-a88d-f9e38f203887
📒 Files selected for processing (3)
docs/docs.jsondocs/edge/en/observability/noveum-trace.mdxdocs/edge/en/observability/overview.mdx
| listener = setup_crewai_tracing() | ||
|
|
||
| from crewai import Agent, Crew, Task | ||
| ``` | ||
| </Step> | ||
| <Step title="Run your crew and flush traces"> | ||
| Run your crew as usual, then shut down the listener and flush pending traces | ||
| before the process exits: | ||
|
|
||
| ```python | ||
| crew = Crew(agents=[...], tasks=[...]) | ||
| result = crew.kickoff() | ||
|
|
||
| listener.shutdown() |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🌐 Web query:
In the current Noveum Trace CrewAI integration documentation, verify that the listener returned by setup_crewai_tracing()must be assigned tocrew.callback_function.
💡 Result:
Yes, according to the official Noveum Trace documentation, the listener returned by setup_crewai_tracing must be assigned to the crew.callback_function attribute [1][2][3]. The standard implementation pattern for integrating Noveum Trace with CrewAI is: 1. Import the necessary function: from noveum_trace.integrations.crewai import setup_crewai_tracing [1][3] 2. Initialize the listener: listener = setup_crewai_tracing [1][2][3] 3. Assign the listener to the crew object: crew.callback_function = listener [1][2][3] This assignment ensures that the crew's execution, agents, tasks, and other events are correctly traced by the Noveum platform [1][2][4].
Citations:
- 1: https://pypi.org/project/noveum-trace/
- 2: https://github.com/Noveum/noveum-trace/blob/main/AGENTS.md
- 3: https://github.com/Noveum/noveum-trace
- 4: https://github.com/Noveum/noveum-trace/blob/main/CHANGELOG.md
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
echo "== file excerpt =="
sed -n '30,70p' docs/edge/en/observability/noveum-trace.mdx | cat -n
echo "== callback_function occurrences =="
rg -n "callback_function|setup_crewai_tracing|crew\.kickoff|listener\.shutdown" docs -S || true
echo "== local noveum references =="
rg -n "noveum|setup_crewai_tracing|crew\.callback_function" . \
--glob '!node_modules/**' --glob '!dist/**' --glob '!build/**' \
|| true
echo "== package metadata references =="
for f in pyproject.toml requirements.txt setup.py setup.cfg Poetry.lock uv.lock requirements*.txt; do
[ -e "$f" ] && { echo "--- $f"; rg -n "noveum" "$f" || true; }
doneRepository: crewAIInc/crewAI
Length of output: 50373
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
echo "== package metadata references =="
for f in pyproject.toml requirements.txt setup.py setup.cfg Poetry.lock uv.lock requirements*.txt; do
[ -e "$f" ] && { echo "--- $f"; rg -n "noveum" "$f" || true; }
done
echo "== git status summary =="
git status --shortRepository: crewAIInc/crewAI
Length of output: 244
Attach the Noveum listener before running the crew.
setup_crewai_tracing() returns a listener, but this example never assigns it to crew.callback_function, so the crew can run without Noveum receiving execution events. Assign the listener before crew.kickoff().
Proposed fix
crew = Crew(agents=[...], tasks=[...])
+ crew.callback_function = listener
result = crew.kickoff()📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| listener = setup_crewai_tracing() | |
| from crewai import Agent, Crew, Task | |
| ``` | |
| </Step> | |
| <Step title="Run your crew and flush traces"> | |
| Run your crew as usual, then shut down the listener and flush pending traces | |
| before the process exits: | |
| ```python | |
| crew = Crew(agents=[...], tasks=[...]) | |
| result = crew.kickoff() | |
| listener.shutdown() | |
| listener = setup_crewai_tracing() | |
| from crewai import Agent, Crew, Task |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/edge/en/observability/noveum-trace.mdx` around lines 43 - 56, Update the
Crew setup example around setup_crewai_tracing and crew.kickoff to assign the
returned listener to crew.callback_function before starting the crew, while
preserving the existing listener.shutdown call after execution.
Source: MCP tools
| crew = Crew(agents=[...], tasks=[...]) | ||
| result = crew.kickoff() | ||
|
|
||
| listener.shutdown() | ||
| noveum_trace.flush() |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🌐 Web query:
In the current Noveum Trace Python SDK documentation, verify the required ordering and behavior of listener.shutdown()andnoveum_trace.flush() during exception cleanup.
💡 Result:
In the Noveum Trace Python SDK, when handling exception cleanup and application teardown, you should call noveum_trace.flush() before noveum_trace.shutdown() [1][2]. The recommended behavior is to use a try...finally block to ensure that resources are properly released even if an exception occurs [1]. Specifically: 1. Perform your tracing operations within your application logic. 2. In the finally block, first call noveum_trace.flush() to ensure all pending, in-memory traces are processed and sent to the server [1][2][3]. 3. Follow this with noveum_trace.shutdown() to tear down the global SDK state, close background transports, and release system resources [1][2]. The flush() method is designed to block until pending traces are processed [2][3], whereas shutdown() performs a final flush and then terminates the client's connection and background threads [2]. Calling shutdown() without a prior flush()—or failing to call shutdown() entirely—can lead to leaked background transport states or the loss of buffered trace data [1].
Citations:
- 1: cleanup decorators Noveum/noveum-trace#56
- 2: https://github.com/Noveum/noveum-trace/blob/main/AGENTS.md
- 3: Noveum/noveum-trace@v1.5.7...v1.5.8
🌐 Web query:
site:noveum.ai/en/docs/integration-examples/crewai/overview "listener.shutdown" "noveum.trace.flush"
💡 Result:
No indexed results were found for that exact query on noveum.ai.
🌐 Web query:
noveum-trace GitHub listener.shutdown() noveum_trace.flush() finally
💡 Result:
The commands listener.shutdown() and noveum_trace.flush() serve distinct purposes within the Noveum Trace SDK ecosystem [1][2]. The noveum_trace.flush() function is a core utility used to force the immediate transmission of any buffered or pending traces to the Noveum platform [3][2]. It is commonly used at the end of scripts or long-running processes to ensure that all telemetry data is captured and sent before the program terminates [3][4]. The listener.shutdown() method is specific to certain framework integrations, most notably within the noveum-trace CrewAI integration [1][5]. When using an integration like NoveumCrewAIListener, this method is called to gracefully stop the listener, disconnect from the framework's event bus, and clean up resources [5][6]. Using them within a finally block is a recommended practice to ensure that resources are cleaned up and telemetry is flushed regardless of whether the execution succeeded or raised an exception [5]. Example pattern: import noveum_trace from noveum_trace.integrations.crewai import NoveumCrewAIListener # Initialize SDK and listener noveum_trace.init(api_key="...") listener = NoveumCrewAIListener try: # Run your application logic run_crewai_crew finally: # Ensure listener stops and pending traces are sent listener.shutdown noveum_trace.flush
Citations:
- 1: https://github.com/Noveum/noveum-trace/blob/main/CHANGELOG.md
- 2: https://github.com/noveum/noveum-trace/
- 3: https://github.com/Noveum/noveum-trace/blob/main/docs/examples/basic_usage.py
- 4: https://noveum.ai/docs/integration-examples/crewai/overview
- 5: https://github.com/Noveum/noveum-trace/blob/main/docs/examples/crewai_e2e_test.py
- 6: https://libraries.io/npm/agentsonar
🌐 Web query:
https://noveum.ai/docs/integration-examples/crewai/overview listener
💡 Result:
The term "listener" in the context of CrewAI and Noveum.ai refers to two distinct concepts depending on whether you are using the CrewAI framework natively or integrating it with Noveum.ai's observability platform [1][2][3]. 1. Noveum.ai CrewAI Integration Noveum.ai provides a specific "listener" component, the NoveumCrewAIListener, designed to automatically trace CrewAI crews, agents, tasks, and tools [1]. To use it, you must install the integration package: pip install "noveum-trace[crewai]" You can then set up tracing by assigning the listener to your crew [1]: import noveum_trace from noveum_trace.integrations.crewai import NoveumCrewAIListener, setup_crewai_tracing from crewai import Crew # Initialize Noveum Trace noveum_trace.init(project="your-project", api_key="your-api-key") # Setup and assign listener listener = setup_crewai_tracing crew = Crew(agents=[...], tasks=[...]) crew.callback_function = listener result = crew.kickoff 2. CrewAI Native "listener" (Flows) Within the CrewAI framework itself, @listen is a decorator used primarily in "Flows" [2][4]. It allows methods to react to the output of other methods within a flow [2]. This is an internal architectural feature of CrewAI for controlling execution logic, distinct from the observability listener provided by Noveum.ai [2][3]. Additionally, CrewAI features an advanced event system using BaseEventListener, which allows developers to build custom integrations or logging by reacting to internal events like task start or completion [3].
Citations:
- 1: https://pypi.org/project/noveum-trace/
- 2: https://docs.crewai.com/v1.15.3/en/concepts/flows
- 3: https://docs.crewai.com/edge/en/concepts/event-listener
- 4: https://docs.crewai.com/v1.14.7/en/concepts/flows
🌐 Web query:
site:github.com/noveum-trace crewai listener.shutdown() finally flush()
💡 Result:
No exact GitHub result matched site:github.com/noveum-trace crewai listener.shutdown() finally flush().
The closest relevant result is OpenAI’s tracing documentation, which recommends flushing buffered traces in a finally block after the trace closes:
try:
...
finally:
flush_traces()This is analogous to calling listener.shutdown() and then flushing pending data during cleanup. [1] (github.com)
Citations:
🌐 Web query:
site:github.com/noveum-trace/listener NoveumCrewAIListener shutdown
💡 Result:
The GitHub repository URL appears unavailable (404). The relevant documentation confirms that NoveumCrewAIListener provides a shutdown() method, which should be called after crew.kickoff()—typically in a finally block, especially for short-lived scripts or tests. [1]
listener = setup_crewai_tracing()
crew.callback_function = listener
try:
crew.kickoff()
finally:
listener.shutdown()[1] (noveum.ai)
Citations:
🌐 Web query:
site:github.com/noveum-trace/docs/examples/crewai_e2e_test.py listener.shutdown() flush()
💡 Result:
I couldn’t find an indexed result for that exact file or text. The closest relevant result indicates that tracing listeners should be flushed after the trace completes and before shutdown, ensuring buffered data is exported. (github.com)
Citations:
🌐 Web query:
"https://noveum.ai/en/docs/integration-examples/crewai/overview" "flush()"
💡 Result:
In the context of the Noveum.ai integration with CrewAI, the flush function is used to ensure that all batched trace spans are sent before the application process exits [1]. Because Noveum's tracing SDK uses batching for efficiency, calling flush at the end of an execution is necessary to prevent data loss [1][2]. It is important to distinguish this from the standard usage of flush=True in Python print statements, which is frequently seen in CrewAI documentation when handling streaming outputs [3][4]. In the latter case, flush=True is a parameter of the built-in print function used to force the output buffer to clear immediately, ensuring that text chunks appear in the console in real-time [3][4].
Citations:
- 1: https://noveum.ai/docs/integration-examples/crewai/overview
- 2: https://noveum.ai/en/novatrace
- 3: https://docs.crewai.com/en/learn/streaming-flow-execution
- 4: https://docs.crewai.com/en/learn/streaming-crew-execution
🌐 Web query:
"https://docs.noveum.ai/en/docs/integration-examples/crewai/overview" "finally"
💡 Result:
The Noveum.ai integration for CrewAI, managed through the Noveum Trace library, provides comprehensive observability and tracing for multi-agent workflows [1][2]. By utilizing the noveum-trace Python SDK, developers can automatically capture CrewAI execution details, including agent decisions, task completions, tool calls, and LLM interactions, without requiring significant infrastructure changes [1][3][4]. Key features of the integration include: - Hierarchical Tracing: Automatically tracks the structure of crews, agents, tasks, and flows, allowing for visualization of complex multi-agent collaborations [3][5][2]. - Comprehensive Data Capture: Records LLM interactions (tokens, costs, latency), tool invocations, memory usage, and internal agent state [1][3][2]. - Seamless Instrumentation: Integration is performed via the noveum-trace SDK using setup functions such as setup_crewai_tracing [6]. - Performance Monitoring: Provides real-time visibility into the performance of agent systems, including cost-per-trace and latency metrics [7][8][2]. For implementation, developers typically install the noveum-trace library and use the provided integration handlers [6][9][2]. Detailed documentation, including end-to-end examples and configuration constants, is available in the Noveum documentation and the associated GitHub repository [1][6][9].
Citations:
- 1: https://noveum.ai/docs/integration-examples/crewai/overview
- 2: https://noveum.ai/en/novatrace
- 3: https://noveum.ai/en/solutions/ai-agent-monitoring
- 4: https://noveum.ai/en/docs/getting-started/sdk-integration
- 5: https://noveum.ai/en/docs
- 6: https://github.com/Noveum/noveum-trace/blob/main/docs/examples/crewai_e2e_test.py
- 7: https://noveum.ai/en/docs/noveum-products/noveum-trace
- 8: https://noveum.ai/en/docs/platform/dashboard
- 9: Noveum/noveum-trace@7051b6a
Run listener cleanup and trace flushing on CrewAI errors.
If crew.kickoff() raises, lines 56-57 do not run, so failed runs can leave the listener active and lose buffered failure traces. Wrap the call in try...finally and flush traces before exit.
Proposed fix
- result = crew.kickoff()
-
- listener.shutdown()
- noveum_trace.flush()
+ try:
+ result = crew.kickoff()
+ finally:
+ listener.shutdown()
+ noveum_trace.flush()📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| crew = Crew(agents=[...], tasks=[...]) | |
| result = crew.kickoff() | |
| listener.shutdown() | |
| noveum_trace.flush() | |
| crew = Crew(agents=[...], tasks=[...]) | |
| try: | |
| result = crew.kickoff() | |
| finally: | |
| listener.shutdown() | |
| noveum_trace.flush() |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/edge/en/observability/noveum-trace.mdx` around lines 53 - 57, Update the
CrewAI example around crew.kickoff() to use a try...finally block, ensuring
listener.shutdown() and noveum_trace.flush() execute whether kickoff succeeds or
raises. Preserve the existing successful result assignment and cleanup order,
with trace flushing performed before exiting.
Source: MCP tools
1 participant
Summary
Adds Noveum Trace to CrewAI observability integrations. Noveum Trace provides a CrewAI
BaseEventListenerintegration that captures crew, task, agent, LLM, tool, memory,knowledge, MCP, flow, guardrail, and A2A delegation events. Community-maintained.
Files changed
docs/edge/en/observability/noveum-trace.mdx— new provider page (mirrors the existing observability provider pages)docs/edge/en/observability/overview.mdx— card under Monitoring & Tracing Platformsdocs/docs.json— nav entry in the English Edge Observability groupInstall
pip install "noveum-trace[crewai]"Notes
crewai >= 0.177.0(releasednoveum-trace1.5.21).capture_inputs,capture_outputs,capture_llm_messages,capture_tool_schemas, …).llm-generatedlabel (I don't have permission to add labels; happy for a maintainer to apply it).Links