Docs audit — observability (requests page, analytics) — 2026-08-10

[SebConejo]

Audited: the observability surface of mnfst/manifest (analytics dashboard, request drawer, spend limits/alerts, usage API) across merged PRs incl. #2374, #2456, #2485, #2502, #2508, #2535, #2609, #2642, #2666 over ~45 days, against the docs at 692a3fd. Scope: observability.mdx only.

Findings

1 · [WRONG] The self-hosted email setup tells you to configure a provider on the Limits page, but that form does not exist. Environment variables are the only way.

Page: https://manifest.build/docs/observability (Spend alerts, Email delivery, Self-hosted view)

What actually happens:

1. On a self-hosted install you want alert emails.
2. You open the harness Limits page. It shows one read-only line: the address alerts will go to. There is no provider picker, no API key, domain, or address field.
3. The only way to wire up email is environment variables: EMAIL_PROVIDER (resend, mailgun, or sendgrid), EMAIL_API_KEY, EMAIL_FROM, and EMAIL_DOMAIN for Mailgun.
4. With no provider set, alerts are skipped. Blocking rules still work.

Options:

1. Rewrite the block so the EMAIL_* env vars are the only path, and say the Limits page just shows where alerts go. Recommended.
2. Same, and state outright that there is no in-app email form, to kill the expectation.

Suggested wording (option 1), replacing the self-hosted block at observability.mdx:96-102:

<div data-deploy="selfhosted">

Set an email provider with the [`EMAIL_*` environment variables](/reference/environment-variables): `EMAIL_PROVIDER` (`resend`, `mailgun`, or `sendgrid`), `EMAIL_API_KEY`, `EMAIL_FROM`, and `EMAIL_DOMAIN` for Mailgun. The **Limits** page shows the address alerts go to.

Without an email provider, alerts are skipped. Blocking rules still work, since they don't need to send anything.

</div>

Evidence (for the agent)

• observability.mdx:98 — "Configure a provider on the Limits page: Resend, Mailgun, or SendGrid. Enter the API key, domain, and notification address, then save. You can set the same thing with the EMAIL_* environment variables."
• Code: EmailProviderSetup (the root email-config component) has zero JSX usages anywhere in packages/frontend/src; EmailProviderModal is referenced only from inside that never-mounted EmailProviderSetup (EmailProviderSetup.tsx:48), so nothing renders either. Last touched by the revert "per-user email provider configuration for notifications (#821)". pages/Limits.tsx:213 renders only the read-only <CloudEmailInfo> ("Alerts will be sent to "), for every deployment.
• Code: notifications/services/email-providers/send-email.ts:30-59 reads EMAIL_PROVIDER/EMAIL_API_KEY/EMAIL_DOMAIN/EMAIL_FROM (legacy MAILGUN_* still honored); with no provider it logs "skipping email send" and returns.
• Reference already carries the rows: reference/environment-variables.mdx:101-104 (EMAIL_PROVIDER, EMAIL_API_KEY, EMAIL_DOMAIN, EMAIL_FROM), so the fix keeps coherence.
• A self-hosted operator hunts the Limits page for a picker that is not there and never gets alert emails.

2 · [WRONG] The limit-rule "Action" row reads "Email you, block further requests, or both", as if block-only is selectable. It is not: every rule emails, and blocking is an optional toggle.

Page: https://manifest.build/docs/observability (Spend alerts, the rule-fields table)

What actually happens:

1. You open the Limits page and create a rule.
2. You set the metric, threshold, and period.
3. Every rule emails you when the threshold is crossed.
4. A single toggle, "Block requests when exceeded", optionally also blocks. There is no silent-block (block without email) choice.

Options:

1. Reword: a rule always emails, and blocking is an optional add-on toggle. Recommended.
2. Leave as-is: every combination described exists in some form; only silent-block is unreachable.

Suggested wording (option 1). Action row at observability.mdx:67:

| **Action** | Every rule emails you. Turn on blocking to also stop requests over the threshold. |

The "Configure it" step at observability.mdx:79:

    Pick the metric, threshold, and period. Every rule emails you; turn on **Block requests when exceeded** to also stop requests over the threshold.

Evidence (for the agent)

• observability.mdx:67 — "| Action | Email you, block further requests, or both |" (echoed at :79 "whether it should notify, block, or both").
• Code: components/LimitRuleModal.tsx:25 actionValue = () => (blockEnabled() ? 'both' : 'notify'); :150 the single toggle "Block requests when exceeded"; :98 copy "You'll receive an email alert when usage exceeds the threshold." The block-only enum value exists in the DTO but the UI never emits it.
• A user who wants a silent hard-block expects to select "block further requests" alone; the UI always emails.

3 · [MISSING-CASE] On the Free cloud plan the dashboard only shows 7 days of history; the 30, 90, and 365-day ranges are Pro, and the page never says so.

Page: https://manifest.build/docs/observability (The dashboard, Overview)

What actually happens:

1. A Free cloud user opens the Overview.
2. The 24h and 7d ranges work.
3. The 30-day, 90-day, and 365-day ranges are disabled.
4. Nothing on the page says this is a plan limit, so it reads like a bug.

Options:

1. Add one cloud-only line: the Free plan covers up to 7 days; longer ranges need Pro. Recommended.
2. Leave as-is: the page never promises specific ranges, so there is no direct contradiction.

Suggested wording (option 1), inserted right after the </CardGroup> at observability.mdx:44:

<div data-deploy="cloud">

On the Free plan, the Overview covers up to 7 days of history; the 30-, 90-, and 365-day ranges need Pro.

</div>

Evidence (for the agent)

• observability.mdx:32-34 — Overview card: "Spend, tokens, and request volume over time, across all harnesses or one." No range mention anywhere on the page.
• Code: pages/Overview.tsx:50 PRO_RANGES = new Set(['30d','90d','365d']); :131 usePlanRangeLock(...); :193 clamp isFreePlan() && PRO_RANGES.has(range()) back to 7d; services/plan-range-lock.ts:27-28. Same lock on the Requests log (pages/MessageLog.tsx:87,267). 24h and 7d are free.
• A Free cloud user sees greyed ranges with no explanation.

4 · [IMAGE] The "The dashboard" section names four surfaces in prose but shows no screenshot of the dashboard.

Page: https://manifest.build/docs/observability (The dashboard)

Place one screenshot directly under the four-card grid in the "The dashboard" section: the harness Overview tab, showing the KPI tiles (tokens, cost, reliability) and the spend-over-time chart, so a new reader recognizes the landing surface the four cards describe.

Evidence (for the agent)

• observability.mdx:29-44 — the ## The dashboard CardGroup (Overview / Requests / Providers / Limits) is prose only.
• Code: pages/Overview.tsx — the real rendered Overview (KPI cards + the unified chart with requests/selfheal/cost/tokens tabs + CostByModelTable + MessageTable).

---

Reply with the finding number and the chosen option
(e.g. "3 → option 2", or "3: other idea, let's discuss").
PRs are then made manually with the agent, never by the harness.
Rejected findings: reply "N → reject" and they will never be re-raised.