Skip to content

Discover Foundry evaluation targets during init #457

Description

Dependency

Depends on #456, which introduces the type-first target selector, manual entry, and the No agent yet / project observability only flow.

Problem

Even with clearer target-type guidance, users must currently find and construct AgentOps identifiers themselves. This is particularly confusing for Foundry prompt and hosted agents because the portal exposes project URLs, browser URLs, agent identity URLs, version URLs, and protocol routes with different purposes.

Once agentops init has a valid Foundry project endpoint and the user selects a Foundry target type, AgentOps should use that project context to discover compatible resources and offer a selection instead of requiring internal URL knowledge.

This is separate from allowing agent-optional initialization: discovery involves Foundry/Azure APIs, authentication, RBAC, project compatibility, resource classification, pagination, and version selection. Failure to discover must never block the basic onboarding delivered by #456.

Expected behavior

After #456 asks for the target type:

  1. Use the configured Foundry project endpoint and current Azure credential to discover only resources compatible with the selected type.
  2. Present discovered resources as a selectable list containing enough identity to avoid ambiguity:
    • display name;
    • stable resource or agent name;
    • version;
    • lifecycle/status when available;
    • resolved AgentOps target type.
  3. Persist the canonical AgentOps reference derived from the selection:
    • prompt agent: <name>:<version>;
    • hosted agent: a canonical version-pinned agent resource URL that AgentOps normalizes to the Responses protocol;
    • model deployment: model:<deployment>.
  4. For an external/custom HTTP agent, skip cloud discovery and use the manual URL flow from Allow agent-optional initialization with guided target selection #456.
  5. If discovery is unavailable or unsuccessful, explain the reason and offer both manual entry and Configure later.
  6. Supplying --agent bypasses discovery and preserves direct non-interactive validation.

Discovery behavior

Prompt agents

List published prompt agents and their available versions from the connected Foundry project. Do not silently select latest when multiple versions exist.

Hosted agents

List hosted agents and versions from the connected project without requiring users to construct /agents/..., /versions/..., or /endpoint/protocols/openai/responses URLs.

Model deployments

List model deployments visible through the project/account APIs when supported. If the project type or available SDK cannot enumerate deployments, explain that limitation and fall back to manual model:<deployment> entry.

Failure states

Distinguish at least:

  • authentication unavailable;
  • insufficient RBAC;
  • unsupported project type or API capability;
  • SDK/dependency unavailable;
  • network/API failure;
  • no compatible resources found;
  • ambiguous duplicate names or multiple versions.

Discovery errors must be actionable diagnostics, not generic empty lists or fatal initialization failures.

Security and privacy

  • Use existing Azure credential and project discovery flows; do not introduce another authentication mechanism.
  • Keep SDK imports lazy.
  • Do not persist access tokens, credentials, receiver data, or full unredacted API payloads.
  • Request only read permissions required to enumerate resources.
  • Avoid exposing inaccessible resources through error details.

Acceptance criteria

  • Prompt-agent discovery lists compatible agents and explicit versions from the connected project.
  • Hosted-agent discovery lists compatible hosted agents and explicit versions.
  • Model selection lists deployments when the project/account supports enumeration.
  • Results are filtered by the target type selected in Allow agent-optional initialization with guided target selection #456.
  • Selecting a resource writes the correct canonical agent representation.
  • Pagination and duplicate names/versions are handled deterministically.
  • Every discovery failure offers manual entry and configure-later fallback.
  • Discovery failure never prevents project-only initialization.
  • External HTTP and explicit --agent flows do not perform unnecessary discovery.
  • Azure imports remain lazy and calls use existing credential conventions.
  • Tests cover successful discovery, pagination, empty results, ambiguous resources, authentication failure, insufficient RBAC, unsupported APIs, manual fallback, and configure-later fallback.
  • CLI help and init explain describe discovery, required read permissions, and fallback behavior.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions