Skip to content

[FIX] Resolve Prompt Studio default LLM profile and unblock deploy when LLMChallenge is off #2203

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
hari-kuriakose wants to merge 2 commits into
base: main
Choose a base branch
from
Open

[FIX] Resolve Prompt Studio default LLM profile and unblock deploy when LLMChallenge is off #2203

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a52feae
[FIX] Resolve Prompt Studio default LLM profile and unblock deploy wh…
hari-kuriakose Jul 24, 2026
5a4da0e
[TEST] Pin challenge_llm schema behaviour across the enable_challenge…
hari-kuriakose Jul 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions backend/prompt_studio/prompt_studio_core_v2/exceptions.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,8 +44,9 @@ def __init__(self, detail: str | None = None, status_code: int = 500):
class DefaultProfileError(APIException):
status_code = 500
default_detail = (
"Default LLM profile is not configured."
"Please set an LLM profile as default to continue."
"No LLM profile could be resolved for this prompt. "
"Either attach an LLM profile to the prompt, or set one as the "
"project default to continue."
Comment on lines +47 to +49

@coderabbitai coderabbitai Bot Jul 24, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Keep the error message accurate for tool-level LLM failures.

DefaultProfileError is also raised when monitor_llm or challenge_llm is unset and the project default cannot be resolved, even when the prompt already has a valid profile. In that case, “attach an LLM profile to the prompt” does not fix the failure; use a context-specific message or make those fallbacks use the already-resolved profile.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@backend/prompt_studio/prompt_studio_core_v2/exceptions.py` around lines 47 -
49, Update the DefaultProfileError message and its callers so it accurately
reflects tool-level failures from monitor_llm or challenge_llm when the prompt
already has a valid profile. Use context-specific guidance for unresolved tool
profiles, or ensure those fallbacks reuse the already-resolved prompt profile;
retain the existing guidance for genuinely missing prompt and project profiles.

)


Expand Down
10 changes: 10 additions & 0 deletions backend/prompt_studio/prompt_studio_core_v2/prompt_studio_helper.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -184,7 +184,7 @@
raise PermissionError(error_msg)

@staticmethod
def validate_profile_manager_owner_access(

Check failure on line 187 in backend/prompt_studio/prompt_studio_core_v2/prompt_studio_helper.py

View check run for this annotation

SonarQubeCloud / SonarCloud Code Analysis

Refactor this function to reduce its Cognitive Complexity from 19 to the 15 allowed. 


See more on https://sonarcloud.io/project/issues?id=Zipstack_unstract&issues=AZ-TlIdtAb0tljmIOazW&open=AZ-TlIdtAb0tljmIOazW&pullRequest=2203
profile_manager: ProfileManager,
) -> None:
"""Helper method to validate the owner's access to the profile manager.
Expand Down Expand Up @@ -696,7 +696,7 @@
]

@staticmethod
def build_fetch_response_payload(

Check failure on line 699 in backend/prompt_studio/prompt_studio_core_v2/prompt_studio_helper.py

View check run for this annotation

SonarQubeCloud / SonarCloud Code Analysis

Refactor this function to reduce its Cognitive Complexity from 16 to the 15 allowed. 


See more on https://sonarcloud.io/project/issues?id=Zipstack_unstract&issues=AZ-TlIdtAb0tljmIOazX&open=AZ-TlIdtAb0tljmIOazX&pullRequest=2203
tool: CustomTool,
doc_path: str,
doc_name: str,
Expand All @@ -721,6 +721,11 @@
profile_manager_id=profile_manager_id
)

# A prompt need not carry its own profile FK - fall back to the project
# default, matching index_document and single-pass extraction.
if not profile_manager:
profile_manager = ProfileManager.get_default_llm_profile(tool)

Comment on lines +724 to +728

@coderabbitai coderabbitai Bot Jul 24, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Propagate the selected fallback profile to callback handling.

When this path resolves a project default, cb_kwargs["profile_manager_id"] later still contains the original argument, often None. OutputManagerHelper then resolves the project default again during callback processing, so a default-profile change between execution and callback can make output bookkeeping use a different profile. Pass the resolved profile_manager.profile_id in the callback metadata.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@backend/prompt_studio/prompt_studio_core_v2/prompt_studio_helper.py` around
lines 724 - 728, Update the fallback branch around
ProfileManager.get_default_llm_profile so callback metadata records the resolved
profile rather than the original argument: after resolving the default, set
cb_kwargs["profile_manager_id"] to profile_manager.profile_id. Preserve the
existing profile selection behavior for explicitly provided profile managers.

if not profile_manager:
raise DefaultProfileError()

Expand Down Expand Up @@ -1812,6 +1817,11 @@
profile_manager_id=profile_manager_id
)

# A prompt need not carry its own profile FK - fall back to the project
# default, matching index_document and single-pass extraction.
if not profile_manager:
profile_manager = ProfileManager.get_default_llm_profile(tool)

if not profile_manager:
raise DefaultProfileError()

Expand Down
32 changes: 24 additions & 8 deletions backend/prompt_studio/prompt_studio_registry_v2/prompt_studio_registry_helper.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,14 +47,25 @@ def frame_spec(tool: CustomTool) -> Spec:
Returns:
dict: spec dict
"""
challenge_llm_property: dict[str, Any] = {
"type": "string",
"title": "Challenger LLM",
"adapterType": "LLM",
"description": "LLM to use for LLMChallenge",
"adapterIdKey": "challenge_llm_adapter_id",
}
if not tool.enable_challenge:
# With LLMChallenge off the tool instance stores challenge_llm as ""
# (no spec default to seed it). Declaring adapterType here would make
# the tool instance schema carry an enum of real adapter IDs only,
# which "" can never satisfy - so deployment validation rejects a
# tool that never used LLMChallenge in the first place. Leave the
# property a free-form string until the feature is switched on.
challenge_llm_property.pop("adapterType")
challenge_llm_property["default"] = ""

properties = {
"challenge_llm": {
"type": "string",
"title": "Challenger LLM",
"adapterType": "LLM",
"description": "LLM to use for LLMChallenge",
"adapterIdKey": "challenge_llm_adapter_id",
},
"challenge_llm": challenge_llm_property,
"enable_challenge": {
"type": "boolean",
"title": "Enable LLMChallenge",
Expand Down Expand Up @@ -87,10 +98,15 @@ def frame_spec(tool: CustomTool) -> Spec:
},
}

# challenge_llm is only meaningful when LLMChallenge is enabled. Marking
# it required unconditionally makes a tool instance that never set one
# fail validation at deployment time, long after export succeeded.
required = [JsonSchemaKey.CHALLENGE_LLM] if tool.enable_challenge else []

spec = Spec(
title=str(tool.tool_id),
description=tool.description,
required=[JsonSchemaKey.CHALLENGE_LLM],
required=required,
properties=properties,
)
return spec
Expand Down
Loading
Loading