Skip to content

Add concurrent scraping example with structured output#112

Open
MagMueller wants to merge 1 commit intomainfrom
magnus/concurrency-docs
Open

Add concurrent scraping example with structured output#112
MagMueller wants to merge 1 commit intomainfrom
magnus/concurrency-docs

Conversation

@MagMueller
Copy link
Copy Markdown
Contributor

@MagMueller MagMueller commented Mar 31, 2026

5/10 creative test agents needed concurrency patterns but couldn't find any in the docs.

Changes

Structured output page — added a full "Scrape multiple sites concurrently" section with:

  • Python: asyncio.gather + Pydantic schema (3 sites, typed products)
  • TypeScript: Promise.all + Zod schema (same pattern)

llms.txt (both root + cloud) — added compact concurrency snippet so LLMs discover the pattern immediately.

The example scrapes 3 e-commerce sites for laptop prices concurrently, returning typed Product objects. Real, practical, copy-paste ready.

🤖 Generated with Claude Code


Summary by cubic

Added a concurrent scraping example with structured output in Python and TypeScript. Lets users run multiple client.run() sessions in parallel and return typed Product results.

  • New Features
    • Structured-output docs: new “Scrape multiple sites concurrently” example (3 e-commerce sites; typed Product list).
    • Python uses asyncio.gather with pydantic; TypeScript uses Promise.all with zod.
    • Added compact concurrency snippets to docs/llms.txt and docs/cloud/llms.txt for quick discovery.

Written for commit 981afe4. Summary will update on new commits.

5/10 creative test agents needed concurrency patterns but couldn't find
any in the docs. Added:

- Full concurrent scraping example to structured output page showing
  asyncio.gather (Python) and Promise.all (TypeScript) with typed schemas
- Compact concurrency snippet to both llms.txt files so LLMs discover
  the pattern immediately

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@mintlify
Copy link
Copy Markdown

mintlify bot commented Mar 31, 2026

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated (UTC)
browser-use 🟢 Ready View Preview Mar 31, 2026, 5:42 AM

Copy link
Copy Markdown

@cubic-dev-ai cubic-dev-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 issues found across 3 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="docs/cloud/llms.txt">

<violation number="1" location="docs/cloud/llms.txt:33">
P2: The JavaScript example references `tasks` without defining it, so the snippet will fail when copied as-is.</violation>
</file>

<file name="docs/llms.txt">

<violation number="1" location="docs/llms.txt:33">
P2: The JavaScript snippet references an undefined `tasks` variable, so the example is not runnable as written.</violation>
</file>

Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review, or fix all with cubic.

```
```javascript
const results = await Promise.all(
tasks.map((task) => client.run(task, { schema: MySchema }))
Copy link
Copy Markdown

@cubic-dev-ai cubic-dev-ai bot Mar 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: The JavaScript example references tasks without defining it, so the snippet will fail when copied as-is.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At docs/cloud/llms.txt, line 33:

<comment>The JavaScript example references `tasks` without defining it, so the snippet will fail when copied as-is.</comment>

<file context>
@@ -21,6 +21,19 @@ export BROWSER_USE_API_KEY=bu_your_key_here
+```
+```javascript
+const results = await Promise.all(
+  tasks.map((task) => client.run(task, { schema: MySchema }))
+);
+```
</file context>
Suggested change
tasks.map((task) => client.run(task, { schema: MySchema }))
["Scrape site A", "Scrape site B", "Scrape site C"].map((task) => client.run(task, { schema: MySchema }))
Fix with Cubic

```
```javascript
const results = await Promise.all(
tasks.map((task) => client.run(task, { schema: MySchema }))
Copy link
Copy Markdown

@cubic-dev-ai cubic-dev-ai bot Mar 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: The JavaScript snippet references an undefined tasks variable, so the example is not runnable as written.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At docs/llms.txt, line 33:

<comment>The JavaScript snippet references an undefined `tasks` variable, so the example is not runnable as written.</comment>

<file context>
@@ -21,6 +21,19 @@ export BROWSER_USE_API_KEY=bu_your_key_here
+```
+```javascript
+const results = await Promise.all(
+  tasks.map((task) => client.run(task, { schema: MySchema }))
+);
+```
</file context>
Suggested change
tasks.map((task) => client.run(task, { schema: MySchema }))
["Scrape site A", "Scrape site B", "Scrape site C"].map((task) => client.run(task, { schema: MySchema }))
Fix with Cubic

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant