diff --git a/README.md b/README.md
index 2d9dbb9..62a910e 100644
--- a/README.md
+++ b/README.md
@@ -89,6 +89,11 @@ src/
├── chapter_23/ # Security and Cryptography
├── chapter_24/ # Metaprogramming
├── chapter_25/ # Python Ecosystem and Best Practices
+├── chapter_26/ # Async Programming
+├── chapter_27/ # Multiprocessing and Parallelism
+├── chapter_28/ # Command-Line Interfaces
+├── chapter_29/ # Date, Time, and Scheduling
+├── chapter_30/ # XML, HTML, and Data Formats
└── ...
tests/
├── conftest.py # Shared pytest fixtures
@@ -126,6 +131,11 @@ tests/
| 23 | Security and Cryptography | hashlib, hmac, secrets, input validation, secure coding | Done |
| 24 | Metaprogramming | Dynamic attributes, class decorators, introspection, codegen | Done |
| 25 | Python Ecosystem and Best Practices | Code style, linting, testing patterns, CI/CD, project org | Done |
+| 26 | Async Programming | asyncio, async/await, tasks, gather, semaphores | Done |
+| 27 | Multiprocessing and Parallelism | multiprocessing, Pool, ProcessPoolExecutor, IPC | Done |
+| 28 | Command-Line Interfaces | argparse, subcommands, environment variables | Done |
+| 29 | Date, Time, and Scheduling | datetime, zoneinfo, timedelta, calendar, formatting | Done |
+| 30 | XML, HTML, and Data Formats | xml.etree, html.parser, configparser, struct | Done |
## Running Notebooks
diff --git a/src/chapter_26/01_coroutines_and_event_loop.ipynb b/src/chapter_26/01_coroutines_and_event_loop.ipynb
new file mode 100644
index 0000000..6f5f025
--- /dev/null
+++ b/src/chapter_26/01_coroutines_and_event_loop.ipynb
@@ -0,0 +1,559 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "a1b2c3d4",
+ "metadata": {},
+ "source": [
+ "# Chapter 26: Coroutines and the Event Loop\n",
+ "\n",
+ "This notebook covers the foundations of asynchronous programming in Python: `async`/`await` syntax, coroutines, the event loop, and `asyncio.run()`. These are the building blocks for writing concurrent, non-blocking Python code.\n",
+ "\n",
+ "## Key Concepts\n",
+ "- **Coroutines**: Functions defined with `async def` that can be suspended and resumed\n",
+ "- **`await`**: Yields control back to the event loop until a result is ready\n",
+ "- **Event loop**: The scheduler that runs coroutines and manages I/O\n",
+ "- **`asyncio.run()`**: The main entry point for running async code"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b2c3d4e5",
+ "metadata": {},
+ "source": [
+ "## Section 1: What Is a Coroutine?\n",
+ "\n",
+ "A coroutine is a function defined with `async def`. Calling it does **not** execute the body — it returns a coroutine object. The body only runs when the coroutine is awaited or scheduled on the event loop."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "c3d4e5f6",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import asyncio\n",
+ "\n",
+ "\n",
+ "# Define a coroutine with async def\n",
+ "async def greet() -> str:\n",
+ " return \"hello\"\n",
+ "\n",
+ "\n",
+ "# Calling the coroutine function returns a coroutine object\n",
+ "coro = greet()\n",
+ "print(f\"Type: {type(coro)}\")\n",
+ "print(f\"Is coroutine: {asyncio.iscoroutine(coro)}\")\n",
+ "\n",
+ "# The coroutine must be run to get the result\n",
+ "result = asyncio.run(coro)\n",
+ "print(f\"Result: {result}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "d4e5f6a7",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Coroutine objects vs coroutine functions\n",
+ "async def say_hi() -> str:\n",
+ " return \"hi\"\n",
+ "\n",
+ "\n",
+ "# The function itself is a coroutine function\n",
+ "print(f\"Is coroutine function: {asyncio.iscoroutinefunction(say_hi)}\")\n",
+ "\n",
+ "# Calling it produces a coroutine object\n",
+ "coro_obj = say_hi()\n",
+ "print(f\"Is coroutine object: {asyncio.iscoroutine(coro_obj)}\")\n",
+ "\n",
+ "# Always run the coroutine to avoid ResourceWarning\n",
+ "print(f\"Result: {asyncio.run(coro_obj)}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e5f6a7b8",
+ "metadata": {},
+ "source": [
+ "## Section 2: The `await` Keyword\n",
+ "\n",
+ "`await` suspends the current coroutine until the awaited coroutine completes. This is how coroutines chain together — one coroutine can call another using `await`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f6a7b8c9",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# await chains coroutines together\n",
+ "async def add(a: int, b: int) -> int:\n",
+ " \"\"\"A simple async addition.\"\"\"\n",
+ " return a + b\n",
+ "\n",
+ "\n",
+ "async def main() -> int:\n",
+ " \"\"\"Main coroutine that awaits another.\"\"\"\n",
+ " result = await add(3, 4)\n",
+ " return result\n",
+ "\n",
+ "\n",
+ "print(f\"3 + 4 = {asyncio.run(main())}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a7b8c9d0",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Multiple awaits in sequence\n",
+ "async def step_one() -> str:\n",
+ " return \"fetched data\"\n",
+ "\n",
+ "\n",
+ "async def step_two(data: str) -> str:\n",
+ " return f\"processed {data}\"\n",
+ "\n",
+ "\n",
+ "async def step_three(data: str) -> str:\n",
+ " return f\"saved {data}\"\n",
+ "\n",
+ "\n",
+ "async def pipeline() -> str:\n",
+ " \"\"\"Sequential async pipeline.\"\"\"\n",
+ " raw = await step_one()\n",
+ " processed = await step_two(raw)\n",
+ " result = await step_three(processed)\n",
+ " return result\n",
+ "\n",
+ "\n",
+ "print(asyncio.run(pipeline()))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b8c9d0e1",
+ "metadata": {},
+ "source": [
+ "## Section 3: `asyncio.sleep()` and Yielding Control\n",
+ "\n",
+ "`asyncio.sleep()` is the async equivalent of `time.sleep()`. Unlike `time.sleep()`, it does **not** block the event loop — it suspends the current coroutine and allows other coroutines to run."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "c9d0e1f2",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import time\n",
+ "\n",
+ "\n",
+ "async def delayed_value() -> str:\n",
+ " \"\"\"Simulate an async operation with sleep.\"\"\"\n",
+ " await asyncio.sleep(0) # Yield control, resume immediately\n",
+ " return \"done\"\n",
+ "\n",
+ "\n",
+ "result = asyncio.run(delayed_value())\n",
+ "print(f\"Result: {result}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "d0e1f2a3",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Demonstrating non-blocking sleep with timing\n",
+ "async def task_a() -> str:\n",
+ " print(\"Task A: starting\")\n",
+ " await asyncio.sleep(0.1)\n",
+ " print(\"Task A: finished\")\n",
+ " return \"A\"\n",
+ "\n",
+ "\n",
+ "async def task_b() -> str:\n",
+ " print(\"Task B: starting\")\n",
+ " await asyncio.sleep(0.05)\n",
+ " print(\"Task B: finished\")\n",
+ " return \"B\"\n",
+ "\n",
+ "\n",
+ "async def run_both() -> None:\n",
+ " \"\"\"Run both tasks concurrently — they interleave.\"\"\"\n",
+ " start = time.perf_counter()\n",
+ " results = await asyncio.gather(task_a(), task_b())\n",
+ " elapsed = time.perf_counter() - start\n",
+ " print(f\"\\nResults: {results}\")\n",
+ " print(f\"Elapsed: {elapsed:.3f}s (concurrent, not 0.15s)\")\n",
+ "\n",
+ "\n",
+ "asyncio.run(run_both())"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e1f2a3b4",
+ "metadata": {},
+ "source": [
+ "## Section 4: `asyncio.run()` — The Entry Point\n",
+ "\n",
+ "`asyncio.run()` creates an event loop, runs the given coroutine until completion, and then closes the loop. It is the recommended way to start async code from synchronous code."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f2a3b4c5",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# asyncio.run() handles the event loop lifecycle\n",
+ "async def compute_squares(numbers: list[int]) -> list[int]:\n",
+ " \"\"\"Compute squares asynchronously.\"\"\"\n",
+ " results: list[int] = []\n",
+ " for n in numbers:\n",
+ " await asyncio.sleep(0) # Simulate async work\n",
+ " results.append(n ** 2)\n",
+ " return results\n",
+ "\n",
+ "\n",
+ "# asyncio.run() creates and manages the event loop\n",
+ "squares = asyncio.run(compute_squares([1, 2, 3, 4, 5]))\n",
+ "print(f\"Squares: {squares}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a3b4c5d6",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# asyncio.run() can only be called when no event loop is already running\n",
+ "# This is important in Jupyter notebooks, which have their own event loop\n",
+ "# In scripts, asyncio.run() is the standard entry point:\n",
+ "#\n",
+ "# async def main() -> None:\n",
+ "# ...\n",
+ "#\n",
+ "# if __name__ == \"__main__\":\n",
+ "# asyncio.run(main())\n",
+ "\n",
+ "\n",
+ "async def main() -> str:\n",
+ " \"\"\"Standard async main pattern.\"\"\"\n",
+ " greeting = await greet()\n",
+ " return f\"The greeting is: {greeting}\"\n",
+ "\n",
+ "\n",
+ "print(asyncio.run(main()))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b4c5d6e7",
+ "metadata": {},
+ "source": [
+ "## Section 5: The Event Loop Under the Hood\n",
+ "\n",
+ "The event loop is the core of `asyncio`. It keeps track of all running coroutines and switches between them at `await` points. You rarely interact with the loop directly — `asyncio.run()` manages it for you."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "c5d6e7f8",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Observing the event loop from inside a coroutine\n",
+ "async def show_loop_info() -> None:\n",
+ " loop = asyncio.get_running_loop()\n",
+ " print(f\"Loop type: {type(loop).__name__}\")\n",
+ " print(f\"Loop running: {loop.is_running()}\")\n",
+ " print(f\"Loop closed: {loop.is_closed()}\")\n",
+ "\n",
+ "\n",
+ "asyncio.run(show_loop_info())"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "d6e7f8a9",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Understanding execution order with the event loop\n",
+ "async def announce(label: str, delay: float) -> str:\n",
+ " print(f\" [{label}] starting\")\n",
+ " await asyncio.sleep(delay)\n",
+ " print(f\" [{label}] done after {delay}s\")\n",
+ " return label\n",
+ "\n",
+ "\n",
+ "async def demonstrate_order() -> None:\n",
+ " \"\"\"Show that the event loop interleaves coroutines.\"\"\"\n",
+ " print(\"Launching three coroutines concurrently:\")\n",
+ " results = await asyncio.gather(\n",
+ " announce(\"slow\", 0.15),\n",
+ " announce(\"medium\", 0.10),\n",
+ " announce(\"fast\", 0.05),\n",
+ " )\n",
+ " print(f\"\\nResults (in original order): {results}\")\n",
+ "\n",
+ "\n",
+ "asyncio.run(demonstrate_order())"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e7f8a9b0",
+ "metadata": {},
+ "source": [
+ "## Section 6: Coroutines vs Regular Functions\n",
+ "\n",
+ "Understanding when to use `async def` vs `def` is important. Coroutines are best for I/O-bound work — network calls, file I/O, database queries. CPU-bound work should use threads or processes instead."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f8a9b0c1",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Regular function - blocks the thread\n",
+ "def sync_fetch(url: str) -> str:\n",
+ " \"\"\"Simulates a blocking I/O call.\"\"\"\n",
+ " time.sleep(0.05) # Blocks the entire thread\n",
+ " return f\"data from {url}\"\n",
+ "\n",
+ "\n",
+ "# Async coroutine - yields control during I/O\n",
+ "async def async_fetch(url: str) -> str:\n",
+ " \"\"\"Simulates a non-blocking I/O call.\"\"\"\n",
+ " await asyncio.sleep(0.05) # Yields control to the loop\n",
+ " return f\"data from {url}\"\n",
+ "\n",
+ "\n",
+ "# Compare sequential sync vs concurrent async\n",
+ "urls = [\"api/users\", \"api/products\", \"api/orders\"]\n",
+ "\n",
+ "# Synchronous: runs one after another\n",
+ "start = time.perf_counter()\n",
+ "sync_results = [sync_fetch(url) for url in urls]\n",
+ "sync_elapsed = time.perf_counter() - start\n",
+ "print(f\"Sync results: {sync_results}\")\n",
+ "print(f\"Sync elapsed: {sync_elapsed:.3f}s\")\n",
+ "\n",
+ "\n",
+ "# Asynchronous: runs concurrently\n",
+ "async def fetch_all(urls: list[str]) -> list[str]:\n",
+ " return await asyncio.gather(*(async_fetch(url) for url in urls))\n",
+ "\n",
+ "\n",
+ "start = time.perf_counter()\n",
+ "async_results = asyncio.run(fetch_all(urls))\n",
+ "async_elapsed = time.perf_counter() - start\n",
+ "print(f\"\\nAsync results: {async_results}\")\n",
+ "print(f\"Async elapsed: {async_elapsed:.3f}s\")\n",
+ "print(f\"\\nSpeedup: {sync_elapsed / async_elapsed:.1f}x\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a9b0c1d2",
+ "metadata": {},
+ "source": [
+ "## Section 7: Awaitable Objects\n",
+ "\n",
+ "Any object that implements the `__await__` method is awaitable. Coroutines, Tasks, and Futures are all awaitables. You can create your own awaitable objects too."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b0c1d2e3",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from collections.abc import Generator\n",
+ "\n",
+ "\n",
+ "# Custom awaitable object\n",
+ "class DelayedResult:\n",
+ " \"\"\"An awaitable that yields a result after a delay.\"\"\"\n",
+ "\n",
+ " def __init__(self, value: str, delay: float) -> None:\n",
+ " self.value = value\n",
+ " self.delay = delay\n",
+ "\n",
+ " def __await__(self) -> Generator[None, None, str]:\n",
+ " \"\"\"Make this object awaitable.\"\"\"\n",
+ " yield from asyncio.sleep(self.delay).__await__()\n",
+ " return self.value\n",
+ "\n",
+ "\n",
+ "async def use_custom_awaitable() -> None:\n",
+ " result = await DelayedResult(\"custom result\", 0.01)\n",
+ " print(f\"Got: {result}\")\n",
+ "\n",
+ "\n",
+ "asyncio.run(use_custom_awaitable())"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "c1d2e3f4",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Checking if something is awaitable\n",
+ "import inspect\n",
+ "\n",
+ "\n",
+ "async def example_coro() -> int:\n",
+ " return 42\n",
+ "\n",
+ "\n",
+ "coro = example_coro()\n",
+ "print(f\"Coroutine is awaitable: {inspect.isawaitable(coro)}\")\n",
+ "print(f\"Regular int is awaitable: {inspect.isawaitable(42)}\")\n",
+ "print(f\"String is awaitable: {inspect.isawaitable('hello')}\")\n",
+ "\n",
+ "# Clean up the coroutine\n",
+ "asyncio.run(coro)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d2e3f4a5",
+ "metadata": {},
+ "source": [
+ "## Section 8: Error Handling in Coroutines\n",
+ "\n",
+ "Exceptions in coroutines work the same as in regular functions. An exception raised inside a coroutine propagates to the caller when the coroutine is awaited."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "e3f4a5b6",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Exceptions propagate through await\n",
+ "async def risky_operation() -> str:\n",
+ " await asyncio.sleep(0)\n",
+ " raise ValueError(\"something went wrong\")\n",
+ "\n",
+ "\n",
+ "async def safe_caller() -> str:\n",
+ " \"\"\"Handle errors from awaited coroutines.\"\"\"\n",
+ " try:\n",
+ " result = await risky_operation()\n",
+ " return result\n",
+ " except ValueError as e:\n",
+ " print(f\"Caught error: {e}\")\n",
+ " return \"fallback value\"\n",
+ "\n",
+ "\n",
+ "result = asyncio.run(safe_caller())\n",
+ "print(f\"Final result: {result}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f4a5b6c7",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Handling errors with asyncio.gather using return_exceptions\n",
+ "async def succeed() -> str:\n",
+ " return \"success\"\n",
+ "\n",
+ "\n",
+ "async def fail() -> str:\n",
+ " raise RuntimeError(\"failure\")\n",
+ "\n",
+ "\n",
+ "async def mixed_results() -> None:\n",
+ " \"\"\"Gather with return_exceptions=True to capture errors.\"\"\"\n",
+ " results = await asyncio.gather(\n",
+ " succeed(),\n",
+ " fail(),\n",
+ " succeed(),\n",
+ " return_exceptions=True,\n",
+ " )\n",
+ " for i, result in enumerate(results):\n",
+ " if isinstance(result, Exception):\n",
+ " print(f\" Task {i}: ERROR - {result}\")\n",
+ " else:\n",
+ " print(f\" Task {i}: OK - {result}\")\n",
+ "\n",
+ "\n",
+ "asyncio.run(mixed_results())"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a5b6c7d8",
+ "metadata": {},
+ "source": [
+ "## Summary\n",
+ "\n",
+ "### Coroutines\n",
+ "- Defined with `async def`, return coroutine objects when called\n",
+ "- Must be awaited or scheduled on the event loop to execute\n",
+ "- Use `asyncio.iscoroutine()` and `asyncio.iscoroutinefunction()` for inspection\n",
+ "\n",
+ "### The `await` Keyword\n",
+ "- Suspends the current coroutine until the awaited result is ready\n",
+ "- Only valid inside `async def` functions\n",
+ "- Chains coroutines together in a sequential pipeline\n",
+ "\n",
+ "### The Event Loop\n",
+ "- Schedules and runs coroutines, switching at `await` points\n",
+ "- Enables concurrency without threads\n",
+ "- Access with `asyncio.get_running_loop()` from inside a coroutine\n",
+ "\n",
+ "### `asyncio.run()`\n",
+ "- Creates an event loop, runs the coroutine, and closes the loop\n",
+ "- The standard entry point for async programs\n",
+ "- Cannot be called when another event loop is already running\n",
+ "\n",
+ "### Key Patterns\n",
+ "- Use `asyncio.sleep()` instead of `time.sleep()` in async code\n",
+ "- Handle exceptions with `try`/`except` around `await` expressions\n",
+ "- Async is best for I/O-bound tasks, not CPU-bound computation"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "name": "python",
+ "version": "3.13.0"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
\ No newline at end of file
diff --git a/src/chapter_26/02_tasks_and_concurrency.ipynb b/src/chapter_26/02_tasks_and_concurrency.ipynb
new file mode 100644
index 0000000..3d264e2
--- /dev/null
+++ b/src/chapter_26/02_tasks_and_concurrency.ipynb
@@ -0,0 +1,641 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "a1000001",
+ "metadata": {},
+ "source": [
+ "# Chapter 26: Tasks and Concurrency\n",
+ "\n",
+ "This notebook covers structured concurrency in `asyncio`: creating tasks, gathering results, waiting for completion, limiting concurrency with semaphores, and handling timeouts.\n",
+ "\n",
+ "## Key Concepts\n",
+ "- **`asyncio.create_task()`**: Schedule a coroutine to run concurrently as a Task\n",
+ "- **`asyncio.gather()`**: Run multiple awaitables concurrently, collect results in order\n",
+ "- **`asyncio.wait()`**: Wait for multiple tasks with flexible completion conditions\n",
+ "- **`asyncio.Semaphore`**: Limit the number of concurrent operations\n",
+ "- **`asyncio.wait_for()` / `asyncio.timeout()`**: Cancel operations that take too long"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a1000002",
+ "metadata": {},
+ "source": [
+ "## Section 1: `asyncio.create_task()`\n",
+ "\n",
+ "`create_task()` wraps a coroutine in a `Task` object and schedules it for execution on the event loop. The task starts running as soon as the event loop gets a chance (at the next `await` point)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a1000003",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import asyncio\n",
+ "import time\n",
+ "\n",
+ "\n",
+ "async def compute(x: int) -> int:\n",
+ " \"\"\"Simulate an async computation.\"\"\"\n",
+ " await asyncio.sleep(0)\n",
+ " return x * 2\n",
+ "\n",
+ "\n",
+ "async def main() -> None:\n",
+ " # create_task schedules the coroutine for execution\n",
+ " task = asyncio.create_task(compute(5))\n",
+ " print(f\"Task type: {type(task).__name__}\")\n",
+ " print(f\"Task done before await: {task.done()}\")\n",
+ "\n",
+ " # Await the task to get its result\n",
+ " result = await task\n",
+ " print(f\"Task done after await: {task.done()}\")\n",
+ " print(f\"Result: {result}\")\n",
+ "\n",
+ "\n",
+ "asyncio.run(main())"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a1000004",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Tasks run concurrently — they don't block each other\n",
+ "async def worker(name: str, delay: float) -> str:\n",
+ " print(f\" {name}: starting (will take {delay}s)\")\n",
+ " await asyncio.sleep(delay)\n",
+ " print(f\" {name}: done\")\n",
+ " return f\"{name} result\"\n",
+ "\n",
+ "\n",
+ "async def main() -> None:\n",
+ " start = time.perf_counter()\n",
+ "\n",
+ " # Create tasks — they start running immediately\n",
+ " task1 = asyncio.create_task(worker(\"Task-1\", 0.10))\n",
+ " task2 = asyncio.create_task(worker(\"Task-2\", 0.05))\n",
+ " task3 = asyncio.create_task(worker(\"Task-3\", 0.08))\n",
+ "\n",
+ " # Await each task to get its result\n",
+ " r1 = await task1\n",
+ " r2 = await task2\n",
+ " r3 = await task3\n",
+ "\n",
+ " elapsed = time.perf_counter() - start\n",
+ " print(f\"\\nResults: {[r1, r2, r3]}\")\n",
+ " print(f\"Elapsed: {elapsed:.3f}s (concurrent, not 0.23s)\")\n",
+ "\n",
+ "\n",
+ "asyncio.run(main())"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a1000005",
+ "metadata": {},
+ "source": [
+ "## Section 2: `asyncio.gather()`\n",
+ "\n",
+ "`gather()` runs multiple awaitables concurrently and returns their results as a list. The results are always in the **same order** as the arguments, regardless of which coroutine finishes first."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a1000006",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# gather runs coroutines concurrently and collects results\n",
+ "results: list[int] = []\n",
+ "\n",
+ "\n",
+ "async def append_value(val: int) -> int:\n",
+ " await asyncio.sleep(0)\n",
+ " results.append(val)\n",
+ " return val\n",
+ "\n",
+ "\n",
+ "async def main() -> list[int]:\n",
+ " return await asyncio.gather(\n",
+ " append_value(1),\n",
+ " append_value(2),\n",
+ " append_value(3),\n",
+ " )\n",
+ "\n",
+ "\n",
+ "gathered = asyncio.run(main())\n",
+ "print(f\"Gathered results: {gathered}\")\n",
+ "print(f\"Execution order: {sorted(results)}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a1000007",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# gather preserves argument order, not completion order\n",
+ "async def delayed(val: int, delay: float) -> int:\n",
+ " \"\"\"Return a value after a delay.\"\"\"\n",
+ " await asyncio.sleep(delay)\n",
+ " return val\n",
+ "\n",
+ "\n",
+ "async def main() -> None:\n",
+ " # Task 3 finishes first, but appears last in results\n",
+ " results = await asyncio.gather(\n",
+ " delayed(1, 0.02), # Finishes last\n",
+ " delayed(2, 0.01), # Finishes second\n",
+ " delayed(3, 0.00), # Finishes first\n",
+ " )\n",
+ " print(f\"Results (argument order): {results}\")\n",
+ " # Always [1, 2, 3] regardless of completion order\n",
+ "\n",
+ "\n",
+ "asyncio.run(main())"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a1000008",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# gather with dynamic number of coroutines\n",
+ "async def fetch_page(page_num: int) -> dict[str, int | str]:\n",
+ " \"\"\"Simulate fetching a page of data.\"\"\"\n",
+ " await asyncio.sleep(0.01)\n",
+ " return {\"page\": page_num, \"status\": \"ok\"}\n",
+ "\n",
+ "\n",
+ "async def fetch_all_pages(total: int) -> list[dict[str, int | str]]:\n",
+ " \"\"\"Fetch multiple pages concurrently.\"\"\"\n",
+ " # Use unpacking to pass a generator of coroutines\n",
+ " return await asyncio.gather(\n",
+ " *(fetch_page(i) for i in range(1, total + 1))\n",
+ " )\n",
+ "\n",
+ "\n",
+ "pages = asyncio.run(fetch_all_pages(5))\n",
+ "for page in pages:\n",
+ " print(f\" Page {page['page']}: {page['status']}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a1000009",
+ "metadata": {},
+ "source": [
+ "## Section 3: Error Handling with `gather()`\n",
+ "\n",
+ "By default, if any gathered coroutine raises an exception, `gather()` re-raises it. Use `return_exceptions=True` to capture exceptions as results instead."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a1000010",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "async def maybe_fail(n: int) -> str:\n",
+ " \"\"\"Succeed on even numbers, fail on odd.\"\"\"\n",
+ " await asyncio.sleep(0)\n",
+ " if n % 2 != 0:\n",
+ " raise ValueError(f\"odd number: {n}\")\n",
+ " return f\"ok-{n}\"\n",
+ "\n",
+ "\n",
+ "async def main() -> None:\n",
+ " # return_exceptions=True captures errors instead of raising\n",
+ " results = await asyncio.gather(\n",
+ " maybe_fail(1),\n",
+ " maybe_fail(2),\n",
+ " maybe_fail(3),\n",
+ " maybe_fail(4),\n",
+ " return_exceptions=True,\n",
+ " )\n",
+ " for i, result in enumerate(results):\n",
+ " if isinstance(result, Exception):\n",
+ " print(f\" Task {i}: FAILED - {result}\")\n",
+ " else:\n",
+ " print(f\" Task {i}: OK - {result}\")\n",
+ "\n",
+ "\n",
+ "asyncio.run(main())"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a1000011",
+ "metadata": {},
+ "source": [
+ "## Section 4: `asyncio.wait()`\n",
+ "\n",
+ "`wait()` gives you more control than `gather()`. It returns two sets: `done` and `pending`. You can wait for the first task to complete, all tasks, or the first exception."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a1000012",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# wait() with FIRST_COMPLETED — process results as they arrive\n",
+ "async def timed_task(name: str, delay: float) -> str:\n",
+ " await asyncio.sleep(delay)\n",
+ " return f\"{name} ({delay}s)\"\n",
+ "\n",
+ "\n",
+ "async def main() -> None:\n",
+ " tasks: set[asyncio.Task[str]] = {\n",
+ " asyncio.create_task(timed_task(\"fast\", 0.01)),\n",
+ " asyncio.create_task(timed_task(\"medium\", 0.05)),\n",
+ " asyncio.create_task(timed_task(\"slow\", 0.10)),\n",
+ " }\n",
+ "\n",
+ " print(\"Processing tasks as they complete:\")\n",
+ " while tasks:\n",
+ " done, tasks = await asyncio.wait(\n",
+ " tasks, return_when=asyncio.FIRST_COMPLETED\n",
+ " )\n",
+ " for task in done:\n",
+ " print(f\" Completed: {task.result()}\")\n",
+ "\n",
+ "\n",
+ "asyncio.run(main())"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a1000013",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# wait() with ALL_COMPLETED (the default)\n",
+ "async def main() -> None:\n",
+ " tasks = [\n",
+ " asyncio.create_task(timed_task(\"A\", 0.03)),\n",
+ " asyncio.create_task(timed_task(\"B\", 0.01)),\n",
+ " asyncio.create_task(timed_task(\"C\", 0.02)),\n",
+ " ]\n",
+ "\n",
+ " done, pending = await asyncio.wait(tasks, return_when=asyncio.ALL_COMPLETED)\n",
+ " print(f\"Done: {len(done)}, Pending: {len(pending)}\")\n",
+ " for task in done:\n",
+ " print(f\" Result: {task.result()}\")\n",
+ "\n",
+ "\n",
+ "asyncio.run(main())"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a1000014",
+ "metadata": {},
+ "source": [
+ "## Section 5: Task Cancellation\n",
+ "\n",
+ "Tasks can be cancelled by calling `task.cancel()`. The cancelled task will raise `asyncio.CancelledError` at its next `await` point."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a1000015",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "async def long_running() -> str:\n",
+ " \"\"\"A task that takes a while.\"\"\"\n",
+ " try:\n",
+ " print(\" Long task: working...\")\n",
+ " await asyncio.sleep(10) # Will be cancelled before this finishes\n",
+ " return \"completed\"\n",
+ " except asyncio.CancelledError:\n",
+ " print(\" Long task: cancelled! Cleaning up...\")\n",
+ " raise # Re-raise to propagate cancellation\n",
+ "\n",
+ "\n",
+ "async def main() -> None:\n",
+ " task = asyncio.create_task(long_running())\n",
+ "\n",
+ " # Let the task start\n",
+ " await asyncio.sleep(0.01)\n",
+ "\n",
+ " # Cancel the task\n",
+ " task.cancel()\n",
+ "\n",
+ " try:\n",
+ " await task\n",
+ " except asyncio.CancelledError:\n",
+ " print(f\" Main: task was cancelled\")\n",
+ "\n",
+ " print(f\" Task cancelled: {task.cancelled()}\")\n",
+ "\n",
+ "\n",
+ "asyncio.run(main())"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a1000016",
+ "metadata": {},
+ "source": [
+ "## Section 6: Timeouts with `asyncio.wait_for()`\n",
+ "\n",
+ "`wait_for()` wraps a coroutine with a timeout. If the coroutine does not complete within the timeout, it is cancelled and `asyncio.TimeoutError` is raised."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a1000017",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "async def slow_operation() -> str:\n",
+ " \"\"\"An operation that takes too long.\"\"\"\n",
+ " await asyncio.sleep(5)\n",
+ " return \"finally done\"\n",
+ "\n",
+ "\n",
+ "async def main() -> None:\n",
+ " # Set a timeout of 0.1 seconds\n",
+ " try:\n",
+ " result = await asyncio.wait_for(slow_operation(), timeout=0.1)\n",
+ " print(f\"Result: {result}\")\n",
+ " except asyncio.TimeoutError:\n",
+ " print(\"Operation timed out after 0.1s\")\n",
+ "\n",
+ " # A fast operation completes within the timeout\n",
+ " async def fast_operation() -> str:\n",
+ " await asyncio.sleep(0.01)\n",
+ " return \"quick result\"\n",
+ "\n",
+ " result = await asyncio.wait_for(fast_operation(), timeout=1.0)\n",
+ " print(f\"Fast result: {result}\")\n",
+ "\n",
+ "\n",
+ "asyncio.run(main())"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a1000018",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Timeout with retry pattern\n",
+ "async def unreliable_fetch(attempt: int) -> str:\n",
+ " \"\"\"Simulates a fetch that sometimes times out.\"\"\"\n",
+ " delay = 0.5 if attempt < 3 else 0.01 # Fast on third attempt\n",
+ " await asyncio.sleep(delay)\n",
+ " return f\"data (attempt {attempt})\"\n",
+ "\n",
+ "\n",
+ "async def fetch_with_retries(\n",
+ " max_retries: int = 3,\n",
+ " timeout: float = 0.1,\n",
+ ") -> str:\n",
+ " \"\"\"Retry a fetch operation with timeout.\"\"\"\n",
+ " for attempt in range(1, max_retries + 1):\n",
+ " try:\n",
+ " result = await asyncio.wait_for(\n",
+ " unreliable_fetch(attempt), timeout=timeout\n",
+ " )\n",
+ " print(f\" Attempt {attempt}: success\")\n",
+ " return result\n",
+ " except asyncio.TimeoutError:\n",
+ " print(f\" Attempt {attempt}: timed out\")\n",
+ " raise RuntimeError(f\"Failed after {max_retries} attempts\")\n",
+ "\n",
+ "\n",
+ "result = asyncio.run(fetch_with_retries())\n",
+ "print(f\"Final result: {result}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a1000019",
+ "metadata": {},
+ "source": [
+ "## Section 7: Semaphores — Limiting Concurrency\n",
+ "\n",
+ "`asyncio.Semaphore` limits the number of coroutines that can access a resource concurrently. This is essential for rate limiting, connection pooling, and preventing resource exhaustion."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a1000020",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Semaphore limits concurrent access\n",
+ "max_concurrent: int = 0\n",
+ "current: int = 0\n",
+ "\n",
+ "\n",
+ "async def limited_task(sem: asyncio.Semaphore, task_id: int) -> None:\n",
+ " \"\"\"A task that respects a semaphore limit.\"\"\"\n",
+ " global max_concurrent, current\n",
+ " async with sem:\n",
+ " current += 1\n",
+ " if current > max_concurrent:\n",
+ " max_concurrent = current\n",
+ " print(f\" Task {task_id}: running (concurrent: {current})\")\n",
+ " await asyncio.sleep(0.01)\n",
+ " current -= 1\n",
+ "\n",
+ "\n",
+ "async def main() -> None:\n",
+ " global max_concurrent, current\n",
+ " max_concurrent = 0\n",
+ " current = 0\n",
+ "\n",
+ " # Allow at most 2 tasks to run concurrently\n",
+ " sem = asyncio.Semaphore(2)\n",
+ " await asyncio.gather(*(limited_task(sem, i) for i in range(5)))\n",
+ " print(f\"\\nMax concurrent tasks: {max_concurrent} (limit was 2)\")\n",
+ "\n",
+ "\n",
+ "asyncio.run(main())"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a1000021",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Practical example: rate-limited API fetcher\n",
+ "async def fetch_url(url: str, sem: asyncio.Semaphore) -> dict[str, str]:\n",
+ " \"\"\"Fetch a URL with rate limiting.\"\"\"\n",
+ " async with sem:\n",
+ " print(f\" Fetching {url}...\")\n",
+ " await asyncio.sleep(0.02) # Simulate network I/O\n",
+ " return {\"url\": url, \"status\": \"200\"}\n",
+ "\n",
+ "\n",
+ "async def main() -> None:\n",
+ " urls = [f\"https://api.example.com/item/{i}\" for i in range(6)]\n",
+ "\n",
+ " # Limit to 3 concurrent requests\n",
+ " sem = asyncio.Semaphore(3)\n",
+ "\n",
+ " start = time.perf_counter()\n",
+ " results = await asyncio.gather(*(fetch_url(url, sem) for url in urls))\n",
+ " elapsed = time.perf_counter() - start\n",
+ "\n",
+ " print(f\"\\nFetched {len(results)} URLs in {elapsed:.3f}s\")\n",
+ " print(f\"With concurrency limit of 3\")\n",
+ "\n",
+ "\n",
+ "asyncio.run(main())"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a1000022",
+ "metadata": {},
+ "source": [
+ "## Section 8: Task Groups (Python 3.11+)\n",
+ "\n",
+ "`asyncio.TaskGroup` provides structured concurrency — if any task in the group fails, all other tasks are cancelled. This is safer than `gather()` for error-prone operations."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a1000023",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# TaskGroup for structured concurrency\n",
+ "async def process_item(item: int) -> int:\n",
+ " \"\"\"Process a single item.\"\"\"\n",
+ " await asyncio.sleep(0.01)\n",
+ " return item * 10\n",
+ "\n",
+ "\n",
+ "async def main() -> None:\n",
+ " results: list[int] = []\n",
+ "\n",
+ " async with asyncio.TaskGroup() as tg:\n",
+ " tasks = [\n",
+ " tg.create_task(process_item(i))\n",
+ " for i in range(5)\n",
+ " ]\n",
+ "\n",
+ " # All tasks are guaranteed complete here\n",
+ " results = [t.result() for t in tasks]\n",
+ " print(f\"Results: {results}\")\n",
+ "\n",
+ "\n",
+ "asyncio.run(main())"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a1000024",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# TaskGroup handles errors by cancelling remaining tasks\n",
+ "async def might_fail(n: int) -> int:\n",
+ " await asyncio.sleep(0.01)\n",
+ " if n == 3:\n",
+ " raise ValueError(f\"Cannot process {n}\")\n",
+ " return n * 10\n",
+ "\n",
+ "\n",
+ "async def main() -> None:\n",
+ " try:\n",
+ " async with asyncio.TaskGroup() as tg:\n",
+ " tasks = [tg.create_task(might_fail(i)) for i in range(5)]\n",
+ " except* ValueError as eg:\n",
+ " print(f\"Caught {len(eg.exceptions)} error(s):\")\n",
+ " for exc in eg.exceptions:\n",
+ " print(f\" - {exc}\")\n",
+ "\n",
+ " # Check which tasks completed vs were cancelled\n",
+ " for i, t in enumerate(tasks):\n",
+ " if t.cancelled():\n",
+ " print(f\" Task {i}: cancelled\")\n",
+ " elif t.exception():\n",
+ " print(f\" Task {i}: failed\")\n",
+ " else:\n",
+ " print(f\" Task {i}: result = {t.result()}\")\n",
+ "\n",
+ "\n",
+ "asyncio.run(main())"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a1000025",
+ "metadata": {},
+ "source": [
+ "## Summary\n",
+ "\n",
+ "### `asyncio.create_task()`\n",
+ "- Wraps a coroutine in a `Task` and schedules it on the event loop\n",
+ "- Tasks start running at the next `await` point\n",
+ "- Returns a `Task` object you can await, cancel, or inspect\n",
+ "\n",
+ "### `asyncio.gather()`\n",
+ "- Runs multiple awaitables concurrently\n",
+ "- Returns results in **argument order**, not completion order\n",
+ "- Use `return_exceptions=True` to capture errors instead of raising\n",
+ "\n",
+ "### `asyncio.wait()`\n",
+ "- Returns `(done, pending)` sets of tasks\n",
+ "- Supports `FIRST_COMPLETED`, `ALL_COMPLETED`, `FIRST_EXCEPTION`\n",
+ "- Good for processing results as they arrive\n",
+ "\n",
+ "### Timeouts\n",
+ "- `asyncio.wait_for(coro, timeout)` cancels after timeout seconds\n",
+ "- Raises `asyncio.TimeoutError` on expiration\n",
+ "- Combine with retry logic for resilient operations\n",
+ "\n",
+ "### Semaphores\n",
+ "- `asyncio.Semaphore(n)` limits concurrency to `n` coroutines\n",
+ "- Use `async with sem:` to acquire and release automatically\n",
+ "- Essential for rate limiting and resource management\n",
+ "\n",
+ "### Task Groups (3.11+)\n",
+ "- `asyncio.TaskGroup` provides structured concurrency\n",
+ "- If any task fails, all others are cancelled\n",
+ "- Use `except*` to handle `ExceptionGroup` errors"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "name": "python",
+ "version": "3.13.0"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
\ No newline at end of file
diff --git a/src/chapter_26/03_async_patterns.ipynb b/src/chapter_26/03_async_patterns.ipynb
new file mode 100644
index 0000000..c83375b
--- /dev/null
+++ b/src/chapter_26/03_async_patterns.ipynb
@@ -0,0 +1,778 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "b1000001",
+ "metadata": {},
+ "source": [
+ "# Chapter 26: Async Patterns\n",
+ "\n",
+ "This notebook covers advanced async patterns: async iterators, async generators, async context managers, and the producer-consumer pattern. These patterns enable elegant, composable asynchronous code.\n",
+ "\n",
+ "## Key Concepts\n",
+ "- **Async iterators**: Objects with `__aiter__` and `__anext__` for async `for` loops\n",
+ "- **Async generators**: Functions using `async def` with `yield` for lazy async sequences\n",
+ "- **Async context managers**: Objects with `__aenter__` and `__aexit__` for `async with`\n",
+ "- **Producer-consumer**: A common concurrency pattern using `asyncio.Queue`"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b1000002",
+ "metadata": {},
+ "source": [
+ "## Section 1: Async Iterators\n",
+ "\n",
+ "An async iterator implements `__aiter__()` and `__anext__()`. It is consumed with `async for` and raises `StopAsyncIteration` when exhausted. This is the async equivalent of the regular iteration protocol."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b1000003",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import asyncio\n",
+ "\n",
+ "\n",
+ "class AsyncRange:\n",
+ " \"\"\"An async iterator that yields numbers from 0 to stop-1.\"\"\"\n",
+ "\n",
+ " def __init__(self, stop: int) -> None:\n",
+ " self.stop = stop\n",
+ " self.current = 0\n",
+ "\n",
+ " def __aiter__(self):\n",
+ " return self\n",
+ "\n",
+ " async def __anext__(self) -> int:\n",
+ " if self.current >= self.stop:\n",
+ " raise StopAsyncIteration\n",
+ " value = self.current\n",
+ " self.current += 1\n",
+ " return value\n",
+ "\n",
+ "\n",
+ "async def main() -> None:\n",
+ " # Use async for to iterate\n",
+ " print(\"Async for loop:\")\n",
+ " async for i in AsyncRange(5):\n",
+ " print(f\" {i}\")\n",
+ "\n",
+ "\n",
+ "asyncio.run(main())"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b1000004",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Async comprehension with async iterators\n",
+ "async def collect() -> list[int]:\n",
+ " \"\"\"Collect async iterator values into a list.\"\"\"\n",
+ " return [i async for i in AsyncRange(4)]\n",
+ "\n",
+ "\n",
+ "result = asyncio.run(collect())\n",
+ "print(f\"Collected: {result}\")\n",
+ "\n",
+ "\n",
+ "# Async comprehension with filtering\n",
+ "async def collect_even() -> list[int]:\n",
+ " return [i async for i in AsyncRange(10) if i % 2 == 0]\n",
+ "\n",
+ "\n",
+ "evens = asyncio.run(collect_even())\n",
+ "print(f\"Even numbers: {evens}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b1000005",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# A more realistic async iterator: paginated data fetcher\n",
+ "class AsyncPaginator:\n",
+ " \"\"\"Iterate over pages of data asynchronously.\"\"\"\n",
+ "\n",
+ " def __init__(self, total_items: int, page_size: int) -> None:\n",
+ " self.total_items = total_items\n",
+ " self.page_size = page_size\n",
+ " self.offset = 0\n",
+ "\n",
+ " def __aiter__(self):\n",
+ " return self\n",
+ "\n",
+ " async def __anext__(self) -> list[int]:\n",
+ " if self.offset >= self.total_items:\n",
+ " raise StopAsyncIteration\n",
+ " # Simulate fetching a page from a database\n",
+ " await asyncio.sleep(0)\n",
+ " end = min(self.offset + self.page_size, self.total_items)\n",
+ " page = list(range(self.offset, end))\n",
+ " self.offset = end\n",
+ " return page\n",
+ "\n",
+ "\n",
+ "async def main() -> None:\n",
+ " print(\"Paginated results (page size 3):\")\n",
+ " async for page in AsyncPaginator(total_items=10, page_size=3):\n",
+ " print(f\" Page: {page}\")\n",
+ "\n",
+ "\n",
+ "asyncio.run(main())"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b1000006",
+ "metadata": {},
+ "source": [
+ "## Section 2: Async Generators\n",
+ "\n",
+ "Async generators are the easiest way to create async iterators. They use `async def` with `yield` and can `await` inside the function body. Each `yield` suspends the generator until the next value is requested."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b1000007",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Simple async generator\n",
+ "async def async_range(stop: int):\n",
+ " \"\"\"Yield numbers 0..stop-1 with an async pause.\"\"\"\n",
+ " for i in range(stop):\n",
+ " await asyncio.sleep(0) # Simulate async work\n",
+ " yield i\n",
+ "\n",
+ "\n",
+ "async def main() -> None:\n",
+ " # Use async for to consume the generator\n",
+ " values: list[int] = []\n",
+ " async for val in async_range(5):\n",
+ " values.append(val)\n",
+ " print(f\"Values: {values}\")\n",
+ "\n",
+ " # Or use an async comprehension\n",
+ " collected = [i async for i in async_range(3)]\n",
+ " print(f\"Collected: {collected}\")\n",
+ "\n",
+ "\n",
+ "asyncio.run(main())"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b1000008",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Async generator with filtering and transformation\n",
+ "from typing import AsyncGenerator\n",
+ "\n",
+ "\n",
+ "async def fetch_items(count: int) -> AsyncGenerator[dict[str, int | str], None]:\n",
+ " \"\"\"Simulate fetching items from an async data source.\"\"\"\n",
+ " for i in range(count):\n",
+ " await asyncio.sleep(0)\n",
+ " yield {\"id\": i, \"name\": f\"item-{i}\", \"value\": i * 10}\n",
+ "\n",
+ "\n",
+ "async def filter_valuable(\n",
+ " items: AsyncGenerator[dict[str, int | str], None],\n",
+ " min_value: int,\n",
+ ") -> AsyncGenerator[dict[str, int | str], None]:\n",
+ " \"\"\"Filter items by minimum value.\"\"\"\n",
+ " async for item in items:\n",
+ " if item[\"value\"] >= min_value:\n",
+ " yield item\n",
+ "\n",
+ "\n",
+ "async def main() -> None:\n",
+ " print(\"Items with value >= 30:\")\n",
+ " async for item in filter_valuable(fetch_items(8), min_value=30):\n",
+ " print(f\" {item}\")\n",
+ "\n",
+ "\n",
+ "asyncio.run(main())"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b1000009",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Async generator pipeline: compose multiple transformations\n",
+ "async def numbers(n: int) -> AsyncGenerator[int, None]:\n",
+ " \"\"\"Generate numbers 0..n-1.\"\"\"\n",
+ " for i in range(n):\n",
+ " await asyncio.sleep(0)\n",
+ " yield i\n",
+ "\n",
+ "\n",
+ "async def squared(source: AsyncGenerator[int, None]) -> AsyncGenerator[int, None]:\n",
+ " \"\"\"Square each value.\"\"\"\n",
+ " async for val in source:\n",
+ " yield val ** 2\n",
+ "\n",
+ "\n",
+ "async def only_even(source: AsyncGenerator[int, None]) -> AsyncGenerator[int, None]:\n",
+ " \"\"\"Keep only even values.\"\"\"\n",
+ " async for val in source:\n",
+ " if val % 2 == 0:\n",
+ " yield val\n",
+ "\n",
+ "\n",
+ "async def main() -> None:\n",
+ " # Build a pipeline: numbers -> squared -> only_even\n",
+ " pipeline = only_even(squared(numbers(8)))\n",
+ " results = [val async for val in pipeline]\n",
+ " print(f\"Even squares of 0..7: {results}\")\n",
+ "\n",
+ "\n",
+ "asyncio.run(main())"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b1000010",
+ "metadata": {},
+ "source": [
+ "## Section 3: Async Context Managers\n",
+ "\n",
+ "Async context managers implement `__aenter__` and `__aexit__` and are used with `async with`. They manage resources that require async setup and teardown, like database connections or file handles."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b1000011",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Class-based async context manager\n",
+ "class AsyncResource:\n",
+ " \"\"\"A resource with async setup and teardown.\"\"\"\n",
+ "\n",
+ " async def __aenter__(self):\n",
+ " print(\" Entering: acquiring resource...\")\n",
+ " await asyncio.sleep(0) # Simulate async setup\n",
+ " return self\n",
+ "\n",
+ " async def __aexit__(self, *args: object) -> None:\n",
+ " print(\" Exiting: releasing resource...\")\n",
+ " await asyncio.sleep(0) # Simulate async teardown\n",
+ "\n",
+ " async def do_work(self) -> str:\n",
+ " return \"work done\"\n",
+ "\n",
+ "\n",
+ "async def main() -> None:\n",
+ " async with AsyncResource() as resource:\n",
+ " result = await resource.do_work()\n",
+ " print(f\" Inside: {result}\")\n",
+ " print(\" After: resource released\")\n",
+ "\n",
+ "\n",
+ "asyncio.run(main())"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b1000012",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Tracking lifecycle events\n",
+ "events: list[str] = []\n",
+ "\n",
+ "\n",
+ "class TrackedResource:\n",
+ " \"\"\"An async context manager that records lifecycle events.\"\"\"\n",
+ "\n",
+ " async def __aenter__(self):\n",
+ " events.append(\"enter\")\n",
+ " return self\n",
+ "\n",
+ " async def __aexit__(self, *args: object) -> None:\n",
+ " events.append(\"exit\")\n",
+ "\n",
+ "\n",
+ "async def main() -> None:\n",
+ " async with TrackedResource():\n",
+ " events.append(\"use\")\n",
+ "\n",
+ "\n",
+ "asyncio.run(main())\n",
+ "print(f\"Events: {events}\")\n",
+ "assert events == [\"enter\", \"use\", \"exit\"]"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b1000013",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Function-based async context manager using contextlib\n",
+ "from contextlib import asynccontextmanager\n",
+ "from typing import AsyncGenerator\n",
+ "\n",
+ "\n",
+ "@asynccontextmanager\n",
+ "async def async_timer(label: str) -> AsyncGenerator[None, None]:\n",
+ " \"\"\"Time an async block of code.\"\"\"\n",
+ " import time\n",
+ " start = time.perf_counter()\n",
+ " print(f\" [{label}] Starting...\")\n",
+ " try:\n",
+ " yield\n",
+ " finally:\n",
+ " elapsed = time.perf_counter() - start\n",
+ " print(f\" [{label}] Done in {elapsed:.4f}s\")\n",
+ "\n",
+ "\n",
+ "async def main() -> None:\n",
+ " async with async_timer(\"fetch\"):\n",
+ " await asyncio.sleep(0.05)\n",
+ "\n",
+ " async with async_timer(\"compute\"):\n",
+ " await asyncio.sleep(0.02)\n",
+ "\n",
+ "\n",
+ "asyncio.run(main())"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b1000014",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Practical example: async database connection mock\n",
+ "@asynccontextmanager\n",
+ "async def async_db_connection(\n",
+ " dsn: str,\n",
+ ") -> AsyncGenerator[\"MockConnection\", None]:\n",
+ " \"\"\"Simulate an async database connection.\"\"\"\n",
+ " conn = MockConnection(dsn)\n",
+ " await conn.connect()\n",
+ " try:\n",
+ " yield conn\n",
+ " finally:\n",
+ " await conn.close()\n",
+ "\n",
+ "\n",
+ "class MockConnection:\n",
+ " \"\"\"A mock async database connection.\"\"\"\n",
+ "\n",
+ " def __init__(self, dsn: str) -> None:\n",
+ " self.dsn = dsn\n",
+ " self.connected = False\n",
+ "\n",
+ " async def connect(self) -> None:\n",
+ " await asyncio.sleep(0)\n",
+ " self.connected = True\n",
+ " print(f\" Connected to {self.dsn}\")\n",
+ "\n",
+ " async def close(self) -> None:\n",
+ " await asyncio.sleep(0)\n",
+ " self.connected = False\n",
+ " print(f\" Disconnected from {self.dsn}\")\n",
+ "\n",
+ " async def query(self, sql: str) -> list[dict[str, str]]:\n",
+ " await asyncio.sleep(0)\n",
+ " return [{\"result\": f\"data from: {sql}\"}]\n",
+ "\n",
+ "\n",
+ "async def main() -> None:\n",
+ " async with async_db_connection(\"postgres://localhost/mydb\") as conn:\n",
+ " rows = await conn.query(\"SELECT * FROM users\")\n",
+ " print(f\" Query result: {rows}\")\n",
+ " # Connection is automatically closed here\n",
+ "\n",
+ "\n",
+ "asyncio.run(main())"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b1000015",
+ "metadata": {},
+ "source": [
+ "## Section 4: Producer-Consumer Pattern\n",
+ "\n",
+ "The producer-consumer pattern uses `asyncio.Queue` to decouple producers (which generate work) from consumers (which process it). This is a foundational pattern for building async pipelines."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b1000016",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Basic producer-consumer with asyncio.Queue\n",
+ "async def producer(\n",
+ " queue: asyncio.Queue[int],\n",
+ " count: int,\n",
+ ") -> None:\n",
+ " \"\"\"Produce items and put them on the queue.\"\"\"\n",
+ " for i in range(count):\n",
+ " await asyncio.sleep(0.01) # Simulate production time\n",
+ " await queue.put(i)\n",
+ " print(f\" Produced: {i}\")\n",
+ "\n",
+ "\n",
+ "async def consumer(\n",
+ " queue: asyncio.Queue[int],\n",
+ " name: str,\n",
+ " results: list[int],\n",
+ ") -> None:\n",
+ " \"\"\"Consume items from the queue.\"\"\"\n",
+ " while True:\n",
+ " item = await queue.get()\n",
+ " await asyncio.sleep(0.02) # Simulate processing time\n",
+ " results.append(item * 10)\n",
+ " print(f\" {name} processed: {item} -> {item * 10}\")\n",
+ " queue.task_done()\n",
+ "\n",
+ "\n",
+ "async def main() -> None:\n",
+ " queue: asyncio.Queue[int] = asyncio.Queue(maxsize=5)\n",
+ " results: list[int] = []\n",
+ "\n",
+ " # Start 2 consumers\n",
+ " consumers = [\n",
+ " asyncio.create_task(consumer(queue, f\"Worker-{i}\", results))\n",
+ " for i in range(2)\n",
+ " ]\n",
+ "\n",
+ " # Run producer\n",
+ " await producer(queue, count=6)\n",
+ "\n",
+ " # Wait for all items to be processed\n",
+ " await queue.join()\n",
+ "\n",
+ " # Cancel consumers (they loop forever)\n",
+ " for c in consumers:\n",
+ " c.cancel()\n",
+ "\n",
+ " print(f\"\\nAll results: {sorted(results)}\")\n",
+ "\n",
+ "\n",
+ "asyncio.run(main())"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b1000017",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Producer-consumer with sentinel value for graceful shutdown\n",
+ "SENTINEL: None = None\n",
+ "\n",
+ "\n",
+ "async def data_producer(\n",
+ " queue: asyncio.Queue[int | None],\n",
+ " items: list[int],\n",
+ " num_consumers: int,\n",
+ ") -> None:\n",
+ " \"\"\"Produce data, then send sentinel to each consumer.\"\"\"\n",
+ " for item in items:\n",
+ " await asyncio.sleep(0)\n",
+ " await queue.put(item)\n",
+ "\n",
+ " # Signal each consumer to stop\n",
+ " for _ in range(num_consumers):\n",
+ " await queue.put(SENTINEL)\n",
+ "\n",
+ "\n",
+ "async def data_consumer(\n",
+ " queue: asyncio.Queue[int | None],\n",
+ " name: str,\n",
+ ") -> list[int]:\n",
+ " \"\"\"Consume until sentinel is received.\"\"\"\n",
+ " processed: list[int] = []\n",
+ " while True:\n",
+ " item = await queue.get()\n",
+ " if item is SENTINEL:\n",
+ " print(f\" {name}: received shutdown signal\")\n",
+ " break\n",
+ " processed.append(item ** 2)\n",
+ " await asyncio.sleep(0)\n",
+ " return processed\n",
+ "\n",
+ "\n",
+ "async def main() -> None:\n",
+ " queue: asyncio.Queue[int | None] = asyncio.Queue()\n",
+ " num_consumers = 2\n",
+ "\n",
+ " # Start producer and consumers\n",
+ " producer_task = asyncio.create_task(\n",
+ " data_producer(queue, [1, 2, 3, 4, 5, 6], num_consumers)\n",
+ " )\n",
+ " consumer_tasks = [\n",
+ " asyncio.create_task(data_consumer(queue, f\"Consumer-{i}\"))\n",
+ " for i in range(num_consumers)\n",
+ " ]\n",
+ "\n",
+ " # Wait for all to finish\n",
+ " await producer_task\n",
+ " all_results = await asyncio.gather(*consumer_tasks)\n",
+ "\n",
+ " # Combine results from all consumers\n",
+ " combined = sorted(r for results in all_results for r in results)\n",
+ " print(f\"\\nAll squared values: {combined}\")\n",
+ "\n",
+ "\n",
+ "asyncio.run(main())"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b1000018",
+ "metadata": {},
+ "source": [
+ "## Section 5: Combining Patterns\n",
+ "\n",
+ "Real-world async code combines these patterns. Here we build a mini pipeline that uses async generators, context managers, and queues together."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b1000019",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Combining async generator + context manager + queue\n",
+ "from typing import AsyncGenerator\n",
+ "\n",
+ "\n",
+ "async def event_stream(count: int) -> AsyncGenerator[dict[str, int | str], None]:\n",
+ " \"\"\"Simulate an async event stream.\"\"\"\n",
+ " for i in range(count):\n",
+ " await asyncio.sleep(0)\n",
+ " yield {\"id\": i, \"type\": \"click\" if i % 2 == 0 else \"scroll\"}\n",
+ "\n",
+ "\n",
+ "@asynccontextmanager\n",
+ "async def event_processor(\n",
+ " name: str,\n",
+ ") -> AsyncGenerator[list[dict[str, int | str]], None]:\n",
+ " \"\"\"Context manager for collecting processed events.\"\"\"\n",
+ " processed: list[dict[str, int | str]] = []\n",
+ " print(f\" [{name}] Processor started\")\n",
+ " try:\n",
+ " yield processed\n",
+ " finally:\n",
+ " print(f\" [{name}] Processor done: {len(processed)} events\")\n",
+ "\n",
+ "\n",
+ "async def main() -> None:\n",
+ " async with event_processor(\"ClickFilter\") as results:\n",
+ " async for event in event_stream(8):\n",
+ " if event[\"type\"] == \"click\":\n",
+ " results.append(event)\n",
+ "\n",
+ " print(f\" Click events: {results}\")\n",
+ "\n",
+ "\n",
+ "asyncio.run(main())"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b1000020",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Async for with set/dict comprehensions\n",
+ "async def user_ids(count: int) -> AsyncGenerator[int, None]:\n",
+ " \"\"\"Generate user IDs with some duplicates.\"\"\"\n",
+ " for i in range(count):\n",
+ " await asyncio.sleep(0)\n",
+ " yield i % 5 # Creates duplicates\n",
+ "\n",
+ "\n",
+ "async def main() -> None:\n",
+ " # Async set comprehension — deduplicates automatically\n",
+ " unique_ids: set[int] = {uid async for uid in user_ids(10)}\n",
+ " print(f\"Unique IDs: {sorted(unique_ids)}\")\n",
+ "\n",
+ " # Async dict comprehension\n",
+ " user_map: dict[int, str] = {\n",
+ " uid: f\"user-{uid}\" async for uid in user_ids(5)\n",
+ " }\n",
+ " print(f\"User map: {user_map}\")\n",
+ "\n",
+ "\n",
+ "asyncio.run(main())"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b1000021",
+ "metadata": {},
+ "source": [
+ "## Section 6: Error Handling in Async Patterns\n",
+ "\n",
+ "Errors in async iterators and context managers follow the same patterns as their synchronous counterparts, but the `async` versions require special handling for cleanup."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b1000022",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Async context manager handles exceptions in __aexit__\n",
+ "class SafeResource:\n",
+ " \"\"\"A resource that always cleans up, even on error.\"\"\"\n",
+ "\n",
+ " def __init__(self, name: str) -> None:\n",
+ " self.name = name\n",
+ "\n",
+ " async def __aenter__(self):\n",
+ " print(f\" {self.name}: acquired\")\n",
+ " return self\n",
+ "\n",
+ " async def __aexit__(\n",
+ " self,\n",
+ " exc_type: type[BaseException] | None,\n",
+ " exc_val: BaseException | None,\n",
+ " exc_tb: object,\n",
+ " ) -> bool:\n",
+ " if exc_val:\n",
+ " print(f\" {self.name}: error occurred - {exc_val}\")\n",
+ " print(f\" {self.name}: released\")\n",
+ " return False # Don't suppress the exception\n",
+ "\n",
+ "\n",
+ "async def main() -> None:\n",
+ " # Normal usage\n",
+ " print(\"Normal case:\")\n",
+ " async with SafeResource(\"DB\") as db:\n",
+ " print(f\" Using {db.name}\")\n",
+ "\n",
+ " # Error case — resource still cleaned up\n",
+ " print(\"\\nError case:\")\n",
+ " try:\n",
+ " async with SafeResource(\"DB\") as db:\n",
+ " raise RuntimeError(\"connection lost\")\n",
+ " except RuntimeError:\n",
+ " print(\" Handled the error\")\n",
+ "\n",
+ "\n",
+ "asyncio.run(main())"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b1000023",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Async generator cleanup with try/finally\n",
+ "async def resilient_stream(\n",
+ " items: list[str],\n",
+ ") -> AsyncGenerator[str, None]:\n",
+ " \"\"\"An async generator with guaranteed cleanup.\"\"\"\n",
+ " print(\" Stream: opened\")\n",
+ " try:\n",
+ " for item in items:\n",
+ " await asyncio.sleep(0)\n",
+ " yield item\n",
+ " finally:\n",
+ " print(\" Stream: closed (cleanup done)\")\n",
+ "\n",
+ "\n",
+ "async def main() -> None:\n",
+ " # Normal: iterate to completion\n",
+ " print(\"Full iteration:\")\n",
+ " async for val in resilient_stream([\"a\", \"b\", \"c\"]):\n",
+ " print(f\" got: {val}\")\n",
+ "\n",
+ " # Early exit: break triggers cleanup\n",
+ " print(\"\\nEarly exit (break):\")\n",
+ " async for val in resilient_stream([\"x\", \"y\", \"z\"]):\n",
+ " print(f\" got: {val}\")\n",
+ " if val == \"x\":\n",
+ " break\n",
+ "\n",
+ "\n",
+ "asyncio.run(main())"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b1000024",
+ "metadata": {},
+ "source": [
+ "## Summary\n",
+ "\n",
+ "### Async Iterators\n",
+ "- Implement `__aiter__()` and `__anext__()`\n",
+ "- Raise `StopAsyncIteration` when exhausted\n",
+ "- Consumed with `async for` or async comprehensions\n",
+ "\n",
+ "### Async Generators\n",
+ "- Easiest way to create async iterators: `async def` + `yield`\n",
+ "- Can `await` inside the generator body\n",
+ "- Composable: chain generators into pipelines\n",
+ "- Support cleanup via `try`/`finally`\n",
+ "\n",
+ "### Async Context Managers\n",
+ "- Implement `__aenter__()` and `__aexit__()` for class-based\n",
+ "- Use `@asynccontextmanager` decorator for function-based\n",
+ "- Used with `async with` for resource management\n",
+ "- `__aexit__` is called even when exceptions occur\n",
+ "\n",
+ "### Producer-Consumer\n",
+ "- `asyncio.Queue` decouples producers from consumers\n",
+ "- `queue.put()` / `queue.get()` are async operations\n",
+ "- Use `queue.join()` + `task_done()` for completion tracking\n",
+ "- Sentinel values enable graceful shutdown\n",
+ "\n",
+ "### Key Patterns\n",
+ "- Use async generators for lazy, memory-efficient async sequences\n",
+ "- Use `@asynccontextmanager` for simple resource management\n",
+ "- Combine patterns for real-world async pipelines\n",
+ "- Always ensure cleanup with `try`/`finally` or context managers"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "name": "python",
+ "version": "3.13.0"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
\ No newline at end of file
diff --git a/src/chapter_26/README.md b/src/chapter_26/README.md
new file mode 100644
index 0000000..81936d5
--- /dev/null
+++ b/src/chapter_26/README.md
@@ -0,0 +1,9 @@
+# Chapter 26: Async Programming
+
+Asynchronous programming with `asyncio` — coroutines, the event loop, tasks, gathering results, semaphores, async iterators, and async context managers.
+
+## Notebooks
+
+1. **01_coroutines_and_event_loop.ipynb** — `async`/`await` syntax, coroutines, the event loop, `asyncio.run()`
+2. **02_tasks_and_concurrency.ipynb** — `asyncio.create_task()`, `gather()`, `wait()`, semaphores, timeouts
+3. **03_async_patterns.ipynb** — Async iterators, async generators, async context managers, producer-consumer
diff --git a/src/chapter_26/__init__.py b/src/chapter_26/__init__.py
new file mode 100644
index 0000000..fddb64d
--- /dev/null
+++ b/src/chapter_26/__init__.py
@@ -0,0 +1 @@
+"""Chapter 26: Async Programming."""
diff --git a/src/chapter_27/01_multiprocessing_basics.ipynb b/src/chapter_27/01_multiprocessing_basics.ipynb
new file mode 100644
index 0000000..b179481
--- /dev/null
+++ b/src/chapter_27/01_multiprocessing_basics.ipynb
@@ -0,0 +1,620 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "a1b2c3d4",
+ "metadata": {},
+ "source": [
+ "# Chapter 27: Multiprocessing Basics\n",
+ "\n",
+ "Python's `multiprocessing` module enables true parallelism by spawning separate operating\n",
+ "system processes, each with its own Python interpreter and memory space. Unlike threads,\n",
+ "processes are not constrained by the Global Interpreter Lock (GIL), making multiprocessing\n",
+ "the go-to approach for CPU-bound workloads.\n",
+ "\n",
+ "## Topics Covered\n",
+ "- **Process**: Creating and running child processes\n",
+ "- **Process lifecycle**: `start()`, `join()`, `is_alive()`, exit codes\n",
+ "- **Process identity**: PIDs and process names\n",
+ "- **Pool**: Worker pools for parallel task execution\n",
+ "- **Pool.map()**: Distributing work across processes\n",
+ "- **Pool.apply_async()**: Non-blocking task submission\n",
+ "- **Practical**: Comparing sequential vs parallel execution"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b2c3d4e5",
+ "metadata": {},
+ "source": [
+ "## The multiprocessing Module\n",
+ "\n",
+ "The `multiprocessing` module mirrors the `threading` API but uses **processes** instead of\n",
+ "threads. Each child process gets its own memory space and Python interpreter, which means:\n",
+ "\n",
+ "- No GIL contention -- true parallel execution on multiple CPU cores\n",
+ "- No shared state by default -- data must be explicitly passed or shared\n",
+ "- Higher overhead than threads (process creation is more expensive)\n",
+ "- Functions passed to processes must be **picklable** (defined at module level)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "c3d4e5f6",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import multiprocessing\n",
+ "import os\n",
+ "import time\n",
+ "\n",
+ "# Basic information about the current environment\n",
+ "print(f\"Python process PID: {os.getpid()}\")\n",
+ "print(f\"CPU count: {os.cpu_count()}\")\n",
+ "print(f\"Start method: {multiprocessing.get_start_method()}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d4e5f6a7",
+ "metadata": {},
+ "source": [
+ "## Creating a Process\n",
+ "\n",
+ "The `multiprocessing.Process` class creates a new process. You provide a **target** function\n",
+ "and optional **args** or **kwargs**. The process does not start until you call `.start()`.\n",
+ "\n",
+ "Key methods:\n",
+ "- `start()` -- spawn the child process and begin execution\n",
+ "- `join(timeout=None)` -- block until the process finishes (or timeout expires)\n",
+ "- `is_alive()` -- check if the process is still running\n",
+ "- `terminate()` / `kill()` -- forcibly stop the process"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "e5f6a7b8",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import multiprocessing\n",
+ "import os\n",
+ "\n",
+ "\n",
+ "def worker(name: str) -> None:\n",
+ " \"\"\"A simple worker function that runs in a child process.\"\"\"\n",
+ " pid: int = os.getpid()\n",
+ " parent_pid: int = os.getppid()\n",
+ " print(f\"Worker '{name}' running in PID {pid} (parent PID: {parent_pid})\")\n",
+ "\n",
+ "\n",
+ "# Create and start a process\n",
+ "p = multiprocessing.Process(target=worker, args=(\"alpha\",))\n",
+ "print(f\"Before start: is_alive={p.is_alive()}, pid={p.pid}\")\n",
+ "\n",
+ "p.start()\n",
+ "print(f\"After start: is_alive={p.is_alive()}, pid={p.pid}\")\n",
+ "\n",
+ "p.join() # Wait for the process to finish\n",
+ "print(f\"After join: is_alive={p.is_alive()}, exitcode={p.exitcode}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "f6a7b8c9",
+ "metadata": {},
+ "source": [
+ "## Process Lifecycle and Exit Codes\n",
+ "\n",
+ "Every process has a lifecycle: created, started, running, and terminated. After a process\n",
+ "finishes, you can inspect its `exitcode`:\n",
+ "\n",
+ "| Exit code | Meaning |\n",
+ "|-----------|:--------|\n",
+ "| `0` | Normal termination |\n",
+ "| `> 0` | Exception occurred (exit code 1) |\n",
+ "| `< 0` | Killed by signal `-N` (e.g., -9 = SIGKILL) |\n",
+ "| `None` | Process has not yet terminated |"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a7b8c9d0",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import multiprocessing\n",
+ "import time\n",
+ "\n",
+ "\n",
+ "def quick_task() -> None:\n",
+ " \"\"\"A task that finishes quickly.\"\"\"\n",
+ " time.sleep(0.1)\n",
+ "\n",
+ "\n",
+ "def slow_task() -> None:\n",
+ " \"\"\"A task that takes longer.\"\"\"\n",
+ " time.sleep(2.0)\n",
+ "\n",
+ "\n",
+ "# Demonstrate process lifecycle\n",
+ "p = multiprocessing.Process(target=quick_task, name=\"QuickWorker\")\n",
+ "print(f\"Created: name={p.name}, alive={p.is_alive()}, exitcode={p.exitcode}\")\n",
+ "\n",
+ "p.start()\n",
+ "print(f\"Started: name={p.name}, alive={p.is_alive()}, exitcode={p.exitcode}\")\n",
+ "\n",
+ "p.join()\n",
+ "print(f\"Finished: name={p.name}, alive={p.is_alive()}, exitcode={p.exitcode}\")\n",
+ "\n",
+ "# Demonstrate join with timeout\n",
+ "p2 = multiprocessing.Process(target=slow_task, name=\"SlowWorker\")\n",
+ "p2.start()\n",
+ "p2.join(timeout=0.5) # Wait at most 0.5 seconds\n",
+ "print(f\"\\nAfter timeout: alive={p2.is_alive()}, exitcode={p2.exitcode}\")\n",
+ "\n",
+ "p2.terminate() # Forcibly stop the process\n",
+ "p2.join() # Clean up\n",
+ "print(f\"After terminate: alive={p2.is_alive()}, exitcode={p2.exitcode}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b8c9d0e1",
+ "metadata": {},
+ "source": [
+ "## Multiple Processes and Separate Memory\n",
+ "\n",
+ "Each process has its own memory space. Modifications to variables in a child process do\n",
+ "**not** affect the parent process. This is fundamentally different from threading, where\n",
+ "threads share the same memory."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "c9d0e1f2",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import multiprocessing\n",
+ "import os\n",
+ "\n",
+ "# Use a Queue to collect results from child processes\n",
+ "# (since child processes have separate memory)\n",
+ "\n",
+ "\n",
+ "def compute_square(n: int, result_queue: multiprocessing.Queue) -> None:\n",
+ " \"\"\"Compute a square and put the result in a queue.\"\"\"\n",
+ " pid: int = os.getpid()\n",
+ " result: int = n * n\n",
+ " result_queue.put((n, result, pid))\n",
+ "\n",
+ "\n",
+ "# Create multiple processes\n",
+ "queue: multiprocessing.Queue = multiprocessing.Queue()\n",
+ "processes: list[multiprocessing.Process] = []\n",
+ "\n",
+ "for i in range(4):\n",
+ " p = multiprocessing.Process(target=compute_square, args=(i, queue))\n",
+ " processes.append(p)\n",
+ " p.start()\n",
+ "\n",
+ "# Wait for all processes to complete\n",
+ "for p in processes:\n",
+ " p.join()\n",
+ "\n",
+ "# Collect results\n",
+ "print(f\"Main process PID: {os.getpid()}\")\n",
+ "print(\"\\nResults from child processes:\")\n",
+ "while not queue.empty():\n",
+ " n, result, pid = queue.get()\n",
+ " print(f\" {n}^2 = {result} (computed by PID {pid})\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d0e1f2a3",
+ "metadata": {},
+ "source": [
+ "## Process Names and Daemon Processes\n",
+ "\n",
+ "Processes can be named for easier debugging. A **daemon** process runs in the background\n",
+ "and is automatically terminated when the main process exits.\n",
+ "\n",
+ "- Set `daemon=True` before calling `start()` to create a daemon process\n",
+ "- Daemon processes cannot spawn child processes of their own\n",
+ "- They are useful for background tasks that should not prevent program exit"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "e1f2a3b4",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import multiprocessing\n",
+ "import os\n",
+ "import time\n",
+ "\n",
+ "\n",
+ "def named_worker() -> None:\n",
+ " \"\"\"Worker that prints its own name and PID.\"\"\"\n",
+ " proc = multiprocessing.current_process()\n",
+ " print(f\" Process name={proc.name}, PID={os.getpid()}, daemon={proc.daemon}\")\n",
+ "\n",
+ "\n",
+ "# Named processes\n",
+ "p1 = multiprocessing.Process(target=named_worker, name=\"Worker-A\")\n",
+ "p2 = multiprocessing.Process(target=named_worker, name=\"Worker-B\")\n",
+ "\n",
+ "p1.start()\n",
+ "p2.start()\n",
+ "p1.join()\n",
+ "p2.join()\n",
+ "\n",
+ "# Daemon process example\n",
+ "print(\"\\nDaemon process:\")\n",
+ "p3 = multiprocessing.Process(target=named_worker, name=\"Daemon-Worker\", daemon=True)\n",
+ "p3.start()\n",
+ "p3.join(timeout=2) # Must join daemon processes explicitly if you want to wait\n",
+ "print(f\" p3 daemon={p3.daemon}, exitcode={p3.exitcode}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "f2a3b4c5",
+ "metadata": {},
+ "source": [
+ "## Pool: Managing Worker Processes\n",
+ "\n",
+ "Creating individual `Process` objects works for a few tasks, but for many tasks you want a\n",
+ "**pool** of reusable worker processes. `multiprocessing.Pool` manages a fixed number of\n",
+ "worker processes and distributes tasks to them.\n",
+ "\n",
+ "Key methods:\n",
+ "- `map(func, iterable)` -- apply `func` to every item, return ordered results (blocking)\n",
+ "- `starmap(func, iterable)` -- like `map()` but unpacks argument tuples\n",
+ "- `apply(func, args)` -- call `func` with `args` in a worker (blocking)\n",
+ "- `apply_async(func, args)` -- non-blocking version, returns `AsyncResult`\n",
+ "- `map_async(func, iterable)` -- non-blocking version of `map()`"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a3b4c5d6",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import multiprocessing\n",
+ "import os\n",
+ "\n",
+ "\n",
+ "def square(x: int) -> int:\n",
+ " \"\"\"Compute the square of a number.\"\"\"\n",
+ " return x * x\n",
+ "\n",
+ "\n",
+ "# Pool.map() distributes work across a pool of workers\n",
+ "with multiprocessing.Pool(processes=2) as pool:\n",
+ " results: list[int] = pool.map(square, [1, 2, 3, 4, 5, 6, 7, 8])\n",
+ "\n",
+ "print(f\"Input: {list(range(1, 9))}\")\n",
+ "print(f\"Squared: {results}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b4c5d6e7",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import multiprocessing\n",
+ "\n",
+ "\n",
+ "def power(base: int, exp: int) -> int:\n",
+ " \"\"\"Raise base to the given exponent.\"\"\"\n",
+ " return base ** exp\n",
+ "\n",
+ "\n",
+ "# Pool.starmap() unpacks tuples of arguments\n",
+ "args: list[tuple[int, int]] = [(2, 3), (3, 3), (4, 2), (5, 2), (10, 3)]\n",
+ "\n",
+ "with multiprocessing.Pool(processes=2) as pool:\n",
+ " results: list[int] = pool.starmap(power, args)\n",
+ "\n",
+ "for (base, exp), result in zip(args, results):\n",
+ " print(f\" {base}^{exp} = {result}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "c5d6e7f8",
+ "metadata": {},
+ "source": [
+ "## Pool.apply_async(): Non-Blocking Execution\n",
+ "\n",
+ "`apply_async()` submits a single task to the pool without blocking. It returns an\n",
+ "`AsyncResult` object that you can use to check status and retrieve results later.\n",
+ "\n",
+ "- `result.get(timeout=None)` -- block until the result is ready\n",
+ "- `result.ready()` -- check if the result is available\n",
+ "- `result.successful()` -- check if the task completed without error"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "d6e7f8a9",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import multiprocessing\n",
+ "import time\n",
+ "\n",
+ "\n",
+ "def slow_square(x: int) -> int:\n",
+ " \"\"\"Compute a square with a simulated delay.\"\"\"\n",
+ " time.sleep(0.3)\n",
+ " return x * x\n",
+ "\n",
+ "\n",
+ "with multiprocessing.Pool(processes=2) as pool:\n",
+ " # Submit tasks asynchronously\n",
+ " async_results = [\n",
+ " pool.apply_async(slow_square, args=(i,))\n",
+ " for i in range(1, 6)\n",
+ " ]\n",
+ "\n",
+ " # Check status while tasks are running\n",
+ " print(\"Submitted 5 tasks...\")\n",
+ " time.sleep(0.1)\n",
+ " for i, ar in enumerate(async_results, 1):\n",
+ " print(f\" Task {i}: ready={ar.ready()}\")\n",
+ "\n",
+ " # Collect results (blocks until ready)\n",
+ " print(\"\\nCollecting results:\")\n",
+ " for i, ar in enumerate(async_results, 1):\n",
+ " result: int = ar.get(timeout=5)\n",
+ " print(f\" {i}^2 = {result}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e7f8a9b0",
+ "metadata": {},
+ "source": [
+ "## Separate Process IDs\n",
+ "\n",
+ "A key characteristic of multiprocessing is that each worker runs in a separate OS process\n",
+ "with its own PID. This is what gives us true parallelism, but it also means that data\n",
+ "must be serialized (pickled) to pass between processes."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f8a9b0c1",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import multiprocessing\n",
+ "import os\n",
+ "\n",
+ "\n",
+ "def get_pid(_: int = 0) -> int:\n",
+ " \"\"\"Return the current process ID.\"\"\"\n",
+ " return os.getpid()\n",
+ "\n",
+ "\n",
+ "main_pid: int = os.getpid()\n",
+ "print(f\"Main process PID: {main_pid}\")\n",
+ "\n",
+ "with multiprocessing.Pool(processes=3) as pool:\n",
+ " pids: list[int] = pool.map(get_pid, range(6))\n",
+ "\n",
+ "print(f\"Worker PIDs: {pids}\")\n",
+ "unique_pids: set[int] = set(pids)\n",
+ "print(f\"Unique workers: {unique_pids}\")\n",
+ "print(f\"\\nAll worker PIDs differ from main: {all(pid != main_pid for pid in pids)}\")\n",
+ "print(f\"Number of unique workers: {len(unique_pids)}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a9b0c1d2",
+ "metadata": {},
+ "source": [
+ "## Practical: Sequential vs Parallel Execution\n",
+ "\n",
+ "Let us compare the execution time of a CPU-bound workload run sequentially versus\n",
+ "in parallel using `Pool.map()`. This demonstrates the real benefit of multiprocessing\n",
+ "for CPU-intensive tasks."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b0c1d2e3",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import multiprocessing\n",
+ "import time\n",
+ "\n",
+ "\n",
+ "def cpu_bound_task(n: int) -> int:\n",
+ " \"\"\"A CPU-bound task: sum of squares up to n.\"\"\"\n",
+ " total: int = 0\n",
+ " for i in range(n):\n",
+ " total += i * i\n",
+ " return total\n",
+ "\n",
+ "\n",
+ "workload: list[int] = [2_000_000] * 8\n",
+ "\n",
+ "# Sequential execution\n",
+ "start: float = time.perf_counter()\n",
+ "sequential_results: list[int] = [cpu_bound_task(n) for n in workload]\n",
+ "sequential_time: float = time.perf_counter() - start\n",
+ "\n",
+ "# Parallel execution\n",
+ "start = time.perf_counter()\n",
+ "with multiprocessing.Pool(processes=4) as pool:\n",
+ " parallel_results: list[int] = pool.map(cpu_bound_task, workload)\n",
+ "parallel_time: float = time.perf_counter() - start\n",
+ "\n",
+ "print(f\"Sequential time: {sequential_time:.3f}s\")\n",
+ "print(f\"Parallel time: {parallel_time:.3f}s\")\n",
+ "print(f\"Speedup: {sequential_time / parallel_time:.2f}x\")\n",
+ "print(f\"Results match: {sequential_results == parallel_results}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "c1d2e3f4",
+ "metadata": {},
+ "source": [
+ "## Error Handling in Processes\n",
+ "\n",
+ "When a child process raises an exception, the behavior depends on how you launched it:\n",
+ "\n",
+ "- **Process**: The exception is raised in the child only; the parent sees a non-zero `exitcode`\n",
+ "- **Pool.map()**: The exception is re-raised in the parent when results are collected\n",
+ "- **Pool.apply_async()**: The exception is re-raised when you call `.get()` on the result"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "d2e3f4a5",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import multiprocessing\n",
+ "\n",
+ "\n",
+ "def risky_task(x: int) -> float:\n",
+ " \"\"\"A task that might fail.\"\"\"\n",
+ " if x == 0:\n",
+ " raise ValueError(\"Cannot process zero!\")\n",
+ " return 100.0 / x\n",
+ "\n",
+ "\n",
+ "# Error with Pool.map() -- exception propagates to parent\n",
+ "print(\"Testing Pool.map() error handling:\")\n",
+ "try:\n",
+ " with multiprocessing.Pool(2) as pool:\n",
+ " pool.map(risky_task, [5, 2, 0, 1]) # 0 will cause an error\n",
+ "except ValueError as e:\n",
+ " print(f\" Caught in parent: {e}\")\n",
+ "\n",
+ "# Error with apply_async() -- exception on .get()\n",
+ "print(\"\\nTesting apply_async() error handling:\")\n",
+ "with multiprocessing.Pool(2) as pool:\n",
+ " future = pool.apply_async(risky_task, args=(0,))\n",
+ " try:\n",
+ " result = future.get(timeout=5)\n",
+ " except ValueError as e:\n",
+ " print(f\" Caught from future.get(): {e}\")\n",
+ " print(f\" future.successful(): {future.successful()}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e3f4a5b6",
+ "metadata": {},
+ "source": [
+ "## Pool with Initializer\n",
+ "\n",
+ "You can pass an `initializer` function to `Pool()` that runs once in each worker process\n",
+ "when it starts. This is useful for setting up per-worker state (database connections,\n",
+ "loading large data, etc.)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f4a5b6c7",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import multiprocessing\n",
+ "import os\n",
+ "\n",
+ "# Module-level variable that will be set by the initializer\n",
+ "worker_id: int = -1\n",
+ "\n",
+ "\n",
+ "def init_worker(base_id: int) -> None:\n",
+ " \"\"\"Initialize each worker with a unique ID.\"\"\"\n",
+ " global worker_id\n",
+ " worker_id = base_id + os.getpid() % 100\n",
+ " print(f\" Worker initialized: PID={os.getpid()}, worker_id={worker_id}\")\n",
+ "\n",
+ "\n",
+ "def task_with_state(x: int) -> str:\n",
+ " \"\"\"A task that uses worker-local state.\"\"\"\n",
+ " return f\"worker_id={worker_id}, PID={os.getpid()}, input={x}, result={x*x}\"\n",
+ "\n",
+ "\n",
+ "print(\"Initializing pool with per-worker setup:\")\n",
+ "with multiprocessing.Pool(processes=2, initializer=init_worker, initargs=(1000,)) as pool:\n",
+ " results: list[str] = pool.map(task_with_state, range(4))\n",
+ "\n",
+ "print(\"\\nResults:\")\n",
+ "for r in results:\n",
+ " print(f\" {r}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a5b6c7d8",
+ "metadata": {},
+ "source": [
+ "## Summary\n",
+ "\n",
+ "### Key Takeaways\n",
+ "\n",
+ "| Concept | API | Purpose |\n",
+ "|---------|-----|:--------|\n",
+ "| **Process** | `multiprocessing.Process` | Spawn a single child process |\n",
+ "| **Lifecycle** | `start()`, `join()`, `is_alive()` | Control process execution |\n",
+ "| **Exit codes** | `p.exitcode` | `0`=success, `>0`=error, `<0`=signal |\n",
+ "| **Daemon** | `Process(daemon=True)` | Background process, auto-killed on exit |\n",
+ "| **Pool** | `multiprocessing.Pool(n)` | Manage a pool of `n` worker processes |\n",
+ "| **map()** | `pool.map(func, iterable)` | Distribute work, return ordered results |\n",
+ "| **starmap()** | `pool.starmap(func, args)` | Like map but unpacks argument tuples |\n",
+ "| **apply_async()** | `pool.apply_async(func, args)` | Non-blocking single task submission |\n",
+ "| **Initializer** | `Pool(initializer=func)` | Run setup code once per worker |\n",
+ "\n",
+ "### Best Practices\n",
+ "- Always use `Pool` as a context manager (`with` statement) to ensure workers are cleaned up\n",
+ "- Define worker functions at module level so they can be pickled\n",
+ "- Use `join()` to wait for processes and avoid zombie processes\n",
+ "- Prefer `Pool.map()` for batch processing over manually managing `Process` objects\n",
+ "- Handle exceptions from workers -- they propagate when you collect results\n",
+ "- Choose the number of pool workers based on `os.cpu_count()` for CPU-bound tasks"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "name": "python",
+ "version": "3.13.0"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
\ No newline at end of file
diff --git a/src/chapter_27/02_shared_state_and_ipc.ipynb b/src/chapter_27/02_shared_state_and_ipc.ipynb
new file mode 100644
index 0000000..0b83bfd
--- /dev/null
+++ b/src/chapter_27/02_shared_state_and_ipc.ipynb
@@ -0,0 +1,612 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "a1b2c3d4",
+ "metadata": {},
+ "source": [
+ "# Chapter 27: Shared State and Inter-Process Communication\n",
+ "\n",
+ "Since each process has its own memory space, sharing data between processes requires\n",
+ "explicit mechanisms. The `multiprocessing` module provides several IPC (Inter-Process\n",
+ "Communication) primitives: `Queue`, `Pipe`, `Value`, `Array`, `Manager`, and\n",
+ "synchronization tools like `Lock`.\n",
+ "\n",
+ "## Topics Covered\n",
+ "- **Queue**: Thread/process-safe FIFO queue\n",
+ "- **Pipe**: Bidirectional communication channel\n",
+ "- **Value**: Shared single value in shared memory\n",
+ "- **Array**: Shared array in shared memory\n",
+ "- **Lock**: Synchronize access to shared resources\n",
+ "- **Manager**: Proxy-based shared objects (dicts, lists, etc.)\n",
+ "- **Practical**: Producer-consumer pattern with Queue"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b2c3d4e5",
+ "metadata": {},
+ "source": [
+ "## Queue: Safe Inter-Process Communication\n",
+ "\n",
+ "`multiprocessing.Queue` is a process-safe FIFO queue built on top of pipes and locks.\n",
+ "It is the most common way to pass data between processes.\n",
+ "\n",
+ "Key methods:\n",
+ "- `put(item)` -- add an item (blocks if the queue is full)\n",
+ "- `get()` -- remove and return an item (blocks if empty)\n",
+ "- `empty()` -- approximate check if the queue is empty\n",
+ "- `qsize()` -- approximate number of items"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "c3d4e5f6",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import multiprocessing\n",
+ "\n",
+ "# Basic Queue usage (within a single process for demonstration)\n",
+ "q: multiprocessing.Queue = multiprocessing.Queue()\n",
+ "\n",
+ "# Put items into the queue\n",
+ "q.put(\"hello\")\n",
+ "q.put(\"world\")\n",
+ "q.put(42)\n",
+ "\n",
+ "# Get items in FIFO order\n",
+ "print(f\"First: {q.get()}\")\n",
+ "print(f\"Second: {q.get()}\")\n",
+ "print(f\"Third: {q.get()}\")\n",
+ "print(f\"Empty: {q.empty()}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "d4e5f6a7",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import multiprocessing\n",
+ "import os\n",
+ "\n",
+ "\n",
+ "def queue_worker(q: multiprocessing.Queue, values: list[int]) -> None:\n",
+ " \"\"\"Put computed results into a queue.\"\"\"\n",
+ " pid: int = os.getpid()\n",
+ " for v in values:\n",
+ " q.put((pid, v, v * v))\n",
+ "\n",
+ "\n",
+ "# Use a Queue to collect results from multiple worker processes\n",
+ "result_queue: multiprocessing.Queue = multiprocessing.Queue()\n",
+ "\n",
+ "p1 = multiprocessing.Process(target=queue_worker, args=(result_queue, [1, 2, 3]))\n",
+ "p2 = multiprocessing.Process(target=queue_worker, args=(result_queue, [4, 5, 6]))\n",
+ "\n",
+ "p1.start()\n",
+ "p2.start()\n",
+ "p1.join()\n",
+ "p2.join()\n",
+ "\n",
+ "# Read all results\n",
+ "print(\"Results from workers:\")\n",
+ "while not result_queue.empty():\n",
+ " pid, value, squared = result_queue.get()\n",
+ " print(f\" PID {pid}: {value}^2 = {squared}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e5f6a7b8",
+ "metadata": {},
+ "source": [
+ "## Pipe: Bidirectional Communication\n",
+ "\n",
+ "`multiprocessing.Pipe()` creates a pair of connected `Connection` objects. By default,\n",
+ "the pipe is bidirectional -- each end can both `send()` and `recv()`.\n",
+ "\n",
+ "Pipes are faster than queues for simple two-process communication, but they are not\n",
+ "safe for use by more than two processes simultaneously."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f6a7b8c9",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import multiprocessing\n",
+ "\n",
+ "# Bidirectional pipe (within a single process for clarity)\n",
+ "parent_conn, child_conn = multiprocessing.Pipe()\n",
+ "\n",
+ "# Send from parent end, receive at child end\n",
+ "parent_conn.send(\"ping\")\n",
+ "print(f\"Child received: {child_conn.recv()}\")\n",
+ "\n",
+ "# Send from child end, receive at parent end\n",
+ "child_conn.send(\"pong\")\n",
+ "print(f\"Parent received: {parent_conn.recv()}\")\n",
+ "\n",
+ "# Pipes can send any picklable object\n",
+ "parent_conn.send({\"type\": \"data\", \"values\": [1, 2, 3]})\n",
+ "msg: dict = child_conn.recv()\n",
+ "print(f\"Dict received: {msg}\")\n",
+ "\n",
+ "parent_conn.close()\n",
+ "child_conn.close()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a7b8c9d0",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import multiprocessing\n",
+ "from multiprocessing.connection import Connection\n",
+ "\n",
+ "\n",
+ "def pipe_child(conn: Connection) -> None:\n",
+ " \"\"\"Child process that communicates via a Pipe.\"\"\"\n",
+ " msg: str = conn.recv()\n",
+ " conn.send(f\"echo: {msg}\")\n",
+ " conn.close()\n",
+ "\n",
+ "\n",
+ "# Two-process communication with Pipe\n",
+ "parent_conn, child_conn = multiprocessing.Pipe()\n",
+ "\n",
+ "p = multiprocessing.Process(target=pipe_child, args=(child_conn,))\n",
+ "p.start()\n",
+ "\n",
+ "# Parent sends a message and waits for the reply\n",
+ "parent_conn.send(\"hello from parent\")\n",
+ "reply: str = parent_conn.recv()\n",
+ "print(f\"Parent got reply: {reply}\")\n",
+ "\n",
+ "p.join()\n",
+ "parent_conn.close()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b8c9d0e1",
+ "metadata": {},
+ "source": [
+ "## Value and Array: Shared Memory\n",
+ "\n",
+ "`multiprocessing.Value` and `multiprocessing.Array` create shared memory objects that\n",
+ "can be accessed by multiple processes. They use ctypes type codes:\n",
+ "\n",
+ "| Type code | C type | Python type |\n",
+ "|-----------|--------|:------------|\n",
+ "| `'i'` | `int` | `int` |\n",
+ "| `'d'` | `double` | `float` |\n",
+ "| `'f'` | `float` | `float` |\n",
+ "| `'c'` | `char` | `bytes` |\n",
+ "\n",
+ "These objects live in **shared memory** (not copied), so changes are visible to all processes."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "c9d0e1f2",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import multiprocessing\n",
+ "\n",
+ "# Value: a single shared value\n",
+ "counter = multiprocessing.Value(\"i\", 0) # 'i' = signed int, initial value = 0\n",
+ "print(f\"Initial value: {counter.value}\")\n",
+ "\n",
+ "counter.value = 42\n",
+ "print(f\"After assignment: {counter.value}\")\n",
+ "\n",
+ "# Value with a float\n",
+ "temperature = multiprocessing.Value(\"d\", 98.6) # 'd' = double\n",
+ "print(f\"Temperature: {temperature.value}\")\n",
+ "\n",
+ "# Array: shared array of fixed type\n",
+ "arr = multiprocessing.Array(\"d\", [1.0, 2.0, 3.0, 4.0, 5.0])\n",
+ "print(f\"\\nArray contents: {list(arr)}\")\n",
+ "\n",
+ "arr[0] = 10.0\n",
+ "arr[4] = 50.0\n",
+ "print(f\"After modification: {list(arr)}\")\n",
+ "print(f\"Array length: {len(arr)}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "d0e1f2a3",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import multiprocessing\n",
+ "\n",
+ "\n",
+ "def increment_counter(\n",
+ " counter: multiprocessing.Value,\n",
+ " times: int,\n",
+ ") -> None:\n",
+ " \"\"\"Increment a shared counter multiple times.\"\"\"\n",
+ " for _ in range(times):\n",
+ " with counter.get_lock(): # Acquire the built-in lock\n",
+ " counter.value += 1\n",
+ "\n",
+ "\n",
+ "# Shared counter accessed by multiple processes\n",
+ "shared_counter = multiprocessing.Value(\"i\", 0)\n",
+ "\n",
+ "processes: list[multiprocessing.Process] = [\n",
+ " multiprocessing.Process(target=increment_counter, args=(shared_counter, 1000))\n",
+ " for _ in range(4)\n",
+ "]\n",
+ "\n",
+ "for p in processes:\n",
+ " p.start()\n",
+ "for p in processes:\n",
+ " p.join()\n",
+ "\n",
+ "# With proper locking, result should be exactly 4000\n",
+ "print(f\"Final counter value: {shared_counter.value}\")\n",
+ "print(f\"Expected: {4 * 1000}\")\n",
+ "print(f\"Correct: {shared_counter.value == 4000}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e1f2a3b4",
+ "metadata": {},
+ "source": [
+ "## Lock: Synchronizing Shared Access\n",
+ "\n",
+ "`multiprocessing.Lock` prevents multiple processes from accessing a shared resource\n",
+ "simultaneously. It works exactly like `threading.Lock` but across processes.\n",
+ "\n",
+ "- `lock.acquire()` / `lock.release()` -- manual lock management\n",
+ "- `with lock:` -- context manager (preferred) for automatic release\n",
+ "\n",
+ "Without a lock, concurrent modifications to shared state can produce incorrect results\n",
+ "due to **race conditions**."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f2a3b4c5",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import multiprocessing\n",
+ "\n",
+ "\n",
+ "def unsafe_increment(\n",
+ " counter: multiprocessing.Value,\n",
+ " times: int,\n",
+ ") -> None:\n",
+ " \"\"\"Increment WITHOUT a lock -- prone to race conditions.\"\"\"\n",
+ " for _ in range(times):\n",
+ " counter.value += 1 # NOT atomic!\n",
+ "\n",
+ "\n",
+ "def safe_increment(\n",
+ " counter: multiprocessing.Value,\n",
+ " lock: multiprocessing.Lock,\n",
+ " times: int,\n",
+ ") -> None:\n",
+ " \"\"\"Increment WITH a lock -- safe from race conditions.\"\"\"\n",
+ " for _ in range(times):\n",
+ " with lock:\n",
+ " counter.value += 1\n",
+ "\n",
+ "\n",
+ "# Unsafe version (may produce incorrect results)\n",
+ "unsafe_counter = multiprocessing.Value(\"i\", 0)\n",
+ "procs = [\n",
+ " multiprocessing.Process(target=unsafe_increment, args=(unsafe_counter, 10000))\n",
+ " for _ in range(4)\n",
+ "]\n",
+ "for p in procs:\n",
+ " p.start()\n",
+ "for p in procs:\n",
+ " p.join()\n",
+ "print(f\"Unsafe counter: {unsafe_counter.value} (expected 40000)\")\n",
+ "\n",
+ "# Safe version with explicit Lock\n",
+ "safe_counter = multiprocessing.Value(\"i\", 0)\n",
+ "lock = multiprocessing.Lock()\n",
+ "procs = [\n",
+ " multiprocessing.Process(target=safe_increment, args=(safe_counter, lock, 10000))\n",
+ " for _ in range(4)\n",
+ "]\n",
+ "for p in procs:\n",
+ " p.start()\n",
+ "for p in procs:\n",
+ " p.join()\n",
+ "print(f\"Safe counter: {safe_counter.value} (expected 40000)\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a3b4c5d6",
+ "metadata": {},
+ "source": [
+ "## Manager: Proxy-Based Shared Objects\n",
+ "\n",
+ "`multiprocessing.Manager()` creates a server process that hosts shared Python objects.\n",
+ "Other processes access these objects through **proxies**. This is more flexible than\n",
+ "`Value`/`Array` because it supports standard Python types:\n",
+ "\n",
+ "- `manager.list()` -- shared list\n",
+ "- `manager.dict()` -- shared dictionary\n",
+ "- `manager.Value()` -- shared value\n",
+ "- `manager.Queue()` -- shared queue\n",
+ "- `manager.Lock()` -- shared lock\n",
+ "\n",
+ "The tradeoff: Managers are **slower** than `Value`/`Array` because they use inter-process\n",
+ "communication (proxies) under the hood, but they are more convenient for complex data."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b4c5d6e7",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import multiprocessing\n",
+ "from multiprocessing.managers import DictProxy, ListProxy\n",
+ "\n",
+ "\n",
+ "def add_results(\n",
+ " shared_dict: DictProxy,\n",
+ " shared_list: ListProxy,\n",
+ " key: str,\n",
+ " values: list[int],\n",
+ ") -> None:\n",
+ " \"\"\"Add results to shared dict and list.\"\"\"\n",
+ " total: int = sum(values)\n",
+ " shared_dict[key] = total\n",
+ " shared_list.append(f\"{key}={total}\")\n",
+ "\n",
+ "\n",
+ "with multiprocessing.Manager() as manager:\n",
+ " # Create shared objects through the manager\n",
+ " shared_dict = manager.dict()\n",
+ " shared_list = manager.list()\n",
+ "\n",
+ " # Launch processes that modify shared objects\n",
+ " processes = [\n",
+ " multiprocessing.Process(\n",
+ " target=add_results,\n",
+ " args=(shared_dict, shared_list, f\"task_{i}\", list(range(i * 10))),\n",
+ " )\n",
+ " for i in range(1, 5)\n",
+ " ]\n",
+ " for p in processes:\n",
+ " p.start()\n",
+ " for p in processes:\n",
+ " p.join()\n",
+ "\n",
+ " print(\"Shared dict:\")\n",
+ " for key, value in sorted(shared_dict.items()):\n",
+ " print(f\" {key}: {value}\")\n",
+ "\n",
+ " print(f\"\\nShared list: {list(shared_list)}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "c5d6e7f8",
+ "metadata": {},
+ "source": [
+ "## Choosing the Right IPC Mechanism\n",
+ "\n",
+ "| Mechanism | Use Case | Speed | Flexibility |\n",
+ "|-----------|----------|:------|:------------|\n",
+ "| **Queue** | Multiple producers/consumers, message passing | Medium | High |\n",
+ "| **Pipe** | Two-process communication, request/response | Fast | Low |\n",
+ "| **Value** | Single shared number (int, float) | Fast | Low |\n",
+ "| **Array** | Shared fixed-size array of numbers | Fast | Low |\n",
+ "| **Manager** | Shared dicts, lists, complex objects | Slow | High |\n",
+ "| **Lock** | Protect shared resources from race conditions | -- | -- |"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d6e7f8a9",
+ "metadata": {},
+ "source": [
+ "## Practical: Producer-Consumer with Queue\n",
+ "\n",
+ "The **producer-consumer** pattern is a classic concurrency design pattern. Producers\n",
+ "generate data and place it on a queue; consumers take data from the queue and process it.\n",
+ "A special **sentinel** value (e.g., `None`) signals consumers to stop."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "e7f8a9b0",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import multiprocessing\n",
+ "import os\n",
+ "import time\n",
+ "\n",
+ "\n",
+ "def producer(\n",
+ " queue: multiprocessing.Queue,\n",
+ " items: list[int],\n",
+ " name: str,\n",
+ ") -> None:\n",
+ " \"\"\"Produce items and put them on the queue.\"\"\"\n",
+ " for item in items:\n",
+ " queue.put((name, item))\n",
+ " time.sleep(0.05) # Simulate work\n",
+ " print(f\" Producer {name} (PID {os.getpid()}) finished\")\n",
+ "\n",
+ "\n",
+ "def consumer(\n",
+ " queue: multiprocessing.Queue,\n",
+ " result_queue: multiprocessing.Queue,\n",
+ " name: str,\n",
+ ") -> None:\n",
+ " \"\"\"Consume items from the queue until sentinel is received.\"\"\"\n",
+ " processed: int = 0\n",
+ " while True:\n",
+ " item = queue.get()\n",
+ " if item is None: # Sentinel value\n",
+ " break\n",
+ " producer_name, value = item\n",
+ " result_queue.put((name, producer_name, value, value * value))\n",
+ " processed += 1\n",
+ " print(f\" Consumer {name} (PID {os.getpid()}) processed {processed} items\")\n",
+ "\n",
+ "\n",
+ "# Set up the queues\n",
+ "work_queue: multiprocessing.Queue = multiprocessing.Queue()\n",
+ "result_queue: multiprocessing.Queue = multiprocessing.Queue()\n",
+ "\n",
+ "# Start 2 producers and 2 consumers\n",
+ "producers = [\n",
+ " multiprocessing.Process(target=producer, args=(work_queue, [1, 2, 3, 4], \"P1\")),\n",
+ " multiprocessing.Process(target=producer, args=(work_queue, [5, 6, 7, 8], \"P2\")),\n",
+ "]\n",
+ "consumers = [\n",
+ " multiprocessing.Process(target=consumer, args=(work_queue, result_queue, \"C1\")),\n",
+ " multiprocessing.Process(target=consumer, args=(work_queue, result_queue, \"C2\")),\n",
+ "]\n",
+ "\n",
+ "for p in producers + consumers:\n",
+ " p.start()\n",
+ "\n",
+ "# Wait for producers to finish\n",
+ "for p in producers:\n",
+ " p.join()\n",
+ "\n",
+ "# Send sentinel values (one per consumer)\n",
+ "for _ in consumers:\n",
+ " work_queue.put(None)\n",
+ "\n",
+ "# Wait for consumers to finish\n",
+ "for c in consumers:\n",
+ " c.join()\n",
+ "\n",
+ "# Collect results\n",
+ "print(\"\\nResults:\")\n",
+ "while not result_queue.empty():\n",
+ " consumer_name, producer_name, value, squared = result_queue.get()\n",
+ " print(f\" {consumer_name} processed {producer_name}'s item: {value}^2 = {squared}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "f8a9b0c1",
+ "metadata": {},
+ "source": [
+ "## Shared Array with Worker Processes\n",
+ "\n",
+ "When multiple processes need to write results into a shared data structure, `Array`\n",
+ "combined with index-based partitioning avoids the need for locks entirely. Each process\n",
+ "writes to its own slice of the array."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a9b0c1d2",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import multiprocessing\n",
+ "\n",
+ "\n",
+ "def fill_array_slice(\n",
+ " shared_arr: multiprocessing.Array,\n",
+ " start: int,\n",
+ " end: int,\n",
+ " multiplier: float,\n",
+ ") -> None:\n",
+ " \"\"\"Fill a slice of a shared array with computed values.\"\"\"\n",
+ " for i in range(start, end):\n",
+ " shared_arr[i] = float(i) * multiplier\n",
+ "\n",
+ "\n",
+ "# Create a shared array of 12 doubles\n",
+ "size: int = 12\n",
+ "shared: multiprocessing.Array = multiprocessing.Array(\"d\", size)\n",
+ "\n",
+ "# Each process handles a different slice (no lock needed)\n",
+ "chunk: int = size // 3\n",
+ "procs = [\n",
+ " multiprocessing.Process(target=fill_array_slice, args=(shared, 0, chunk, 1.0)),\n",
+ " multiprocessing.Process(target=fill_array_slice, args=(shared, chunk, 2 * chunk, 2.0)),\n",
+ " multiprocessing.Process(target=fill_array_slice, args=(shared, 2 * chunk, size, 3.0)),\n",
+ "]\n",
+ "\n",
+ "for p in procs:\n",
+ " p.start()\n",
+ "for p in procs:\n",
+ " p.join()\n",
+ "\n",
+ "print(\"Shared array contents:\")\n",
+ "for i, val in enumerate(shared):\n",
+ " print(f\" [{i}] = {val}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b0c1d2e3",
+ "metadata": {},
+ "source": [
+ "## Summary\n",
+ "\n",
+ "### Key Takeaways\n",
+ "\n",
+ "| Concept | API | Purpose |\n",
+ "|---------|-----|:--------|\n",
+ "| **Queue** | `multiprocessing.Queue` | FIFO message passing between processes |\n",
+ "| **Pipe** | `multiprocessing.Pipe()` | Fast two-process bidirectional channel |\n",
+ "| **Value** | `multiprocessing.Value('i', 0)` | Single shared int/float in shared memory |\n",
+ "| **Array** | `multiprocessing.Array('d', [...])` | Fixed-size shared array in shared memory |\n",
+ "| **Lock** | `multiprocessing.Lock()` | Prevent race conditions on shared state |\n",
+ "| **Manager** | `multiprocessing.Manager()` | Proxy-based shared dicts, lists, etc. |\n",
+ "\n",
+ "### Best Practices\n",
+ "- Use `Queue` for general-purpose message passing between any number of processes\n",
+ "- Use `Pipe` for fast, simple two-process communication\n",
+ "- Use `Value` and `Array` for high-performance shared memory with numeric data\n",
+ "- Always use locks when multiple processes write to the same `Value` or `Array`\n",
+ "- Use `Manager` when you need to share complex Python objects (dicts, lists)\n",
+ "- Prefer partitioning work (each process writes its own slice) over shared locks\n",
+ "- Use sentinel values to signal consumers when all work is done"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "name": "python",
+ "version": "3.13.0"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
\ No newline at end of file
diff --git a/src/chapter_27/03_process_pool_executor.ipynb b/src/chapter_27/03_process_pool_executor.ipynb
new file mode 100644
index 0000000..f5d2ffb
--- /dev/null
+++ b/src/chapter_27/03_process_pool_executor.ipynb
@@ -0,0 +1,618 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "a1b2c3d4",
+ "metadata": {},
+ "source": [
+ "# Chapter 27: ProcessPoolExecutor and Futures\n",
+ "\n",
+ "The `concurrent.futures` module provides a high-level interface for parallel execution.\n",
+ "`ProcessPoolExecutor` manages a pool of worker processes and returns `Future` objects\n",
+ "that represent pending results. It is simpler and more Pythonic than the lower-level\n",
+ "`multiprocessing.Pool` API.\n",
+ "\n",
+ "## Topics Covered\n",
+ "- **ProcessPoolExecutor**: High-level process pool interface\n",
+ "- **Future**: Representing pending results\n",
+ "- **map() vs submit()**: Two ways to distribute work\n",
+ "- **as_completed()**: Processing results as they become available\n",
+ "- **CPU-bound vs I/O-bound**: Choosing the right executor\n",
+ "- **Practical**: Comparing ProcessPoolExecutor and ThreadPoolExecutor"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b2c3d4e5",
+ "metadata": {},
+ "source": [
+ "## ProcessPoolExecutor Basics\n",
+ "\n",
+ "`ProcessPoolExecutor` is part of `concurrent.futures` and provides the same API as\n",
+ "`ThreadPoolExecutor`. It uses processes instead of threads, making it ideal for\n",
+ "**CPU-bound** workloads.\n",
+ "\n",
+ "Key features:\n",
+ "- Context manager support (`with` statement)\n",
+ "- `max_workers` defaults to the number of CPUs\n",
+ "- Worker functions must be **picklable** (defined at module level)\n",
+ "- Returns `Future` objects for asynchronous result handling"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "c3d4e5f6",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import os\n",
+ "from concurrent.futures import ProcessPoolExecutor\n",
+ "\n",
+ "\n",
+ "def square(x: int) -> int:\n",
+ " \"\"\"Compute the square of a number.\"\"\"\n",
+ " return x * x\n",
+ "\n",
+ "\n",
+ "# Basic usage with executor.map()\n",
+ "print(f\"CPU count: {os.cpu_count()}\")\n",
+ "\n",
+ "with ProcessPoolExecutor(max_workers=2) as executor:\n",
+ " results: list[int] = list(executor.map(square, [1, 2, 3, 4, 5]))\n",
+ "\n",
+ "print(f\"Squares: {results}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d4e5f6a7",
+ "metadata": {},
+ "source": [
+ "## executor.map(): Ordered Parallel Mapping\n",
+ "\n",
+ "`executor.map(func, *iterables)` applies a function to every item in the iterables,\n",
+ "distributing work across the pool. It returns an **iterator** that yields results in\n",
+ "the **same order** as the input.\n",
+ "\n",
+ "Key differences from `multiprocessing.Pool.map()`:\n",
+ "- Returns a lazy iterator (not a list)\n",
+ "- Supports a `timeout` parameter\n",
+ "- Supports a `chunksize` parameter for efficiency with large iterables"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "e5f6a7b8",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import os\n",
+ "from concurrent.futures import ProcessPoolExecutor\n",
+ "\n",
+ "\n",
+ "def process_item(x: int) -> tuple[int, int, int]:\n",
+ " \"\"\"Process an item and return (input, result, pid).\"\"\"\n",
+ " return (x, x * x, os.getpid())\n",
+ "\n",
+ "\n",
+ "# map() preserves input order\n",
+ "with ProcessPoolExecutor(max_workers=2) as executor:\n",
+ " results = list(executor.map(process_item, range(8)))\n",
+ "\n",
+ "main_pid: int = os.getpid()\n",
+ "print(f\"Main PID: {main_pid}\")\n",
+ "print(\"\\nResults (order preserved):\")\n",
+ "for x, squared, pid in results:\n",
+ " print(f\" {x}^2 = {squared} (worker PID: {pid})\")\n",
+ "\n",
+ "worker_pids: set[int] = {pid for _, _, pid in results}\n",
+ "print(f\"\\nUnique worker PIDs: {worker_pids}\")\n",
+ "print(f\"All workers differ from main: {all(pid != main_pid for _, _, pid in results)}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "f6a7b8c9",
+ "metadata": {},
+ "source": [
+ "## executor.submit() and Future Objects\n",
+ "\n",
+ "`executor.submit(func, *args, **kwargs)` submits a single callable for execution and\n",
+ "returns a `Future` object immediately. The `Future` represents a computation that may\n",
+ "not have completed yet.\n",
+ "\n",
+ "Key `Future` methods:\n",
+ "- `result(timeout=None)` -- block until the result is ready, then return it\n",
+ "- `done()` -- return `True` if the computation has completed\n",
+ "- `cancelled()` -- return `True` if the task was cancelled\n",
+ "- `exception(timeout=None)` -- return the exception, if any\n",
+ "- `add_done_callback(fn)` -- register a callback to run when the future completes"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a7b8c9d0",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from concurrent.futures import Future, ProcessPoolExecutor\n",
+ "\n",
+ "\n",
+ "def square(x: int) -> int:\n",
+ " \"\"\"Compute the square of a number.\"\"\"\n",
+ " return x * x\n",
+ "\n",
+ "\n",
+ "# submit() returns a Future\n",
+ "with ProcessPoolExecutor(max_workers=1) as executor:\n",
+ " future: Future[int] = executor.submit(square, 7)\n",
+ "\n",
+ " # Inspect the future\n",
+ " print(f\"Type: {type(future).__name__}\")\n",
+ " print(f\"Done: {future.done()}\") # May or may not be done yet\n",
+ "\n",
+ " # Get the result (blocks if not yet ready)\n",
+ " result: int = future.result()\n",
+ " print(f\"Result: {result}\")\n",
+ " print(f\"Done after result(): {future.done()}\")\n",
+ " print(f\"Exception: {future.exception()}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b8c9d0e1",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from concurrent.futures import Future, ProcessPoolExecutor\n",
+ "\n",
+ "\n",
+ "def cube(x: int) -> int:\n",
+ " \"\"\"Compute the cube of a number.\"\"\"\n",
+ " return x ** 3\n",
+ "\n",
+ "\n",
+ "# Submit multiple tasks and collect futures\n",
+ "with ProcessPoolExecutor(max_workers=2) as executor:\n",
+ " futures: dict[Future[int], int] = {\n",
+ " executor.submit(cube, n): n\n",
+ " for n in range(1, 7)\n",
+ " }\n",
+ "\n",
+ " # Retrieve results from each future\n",
+ " print(\"Results from submit():\")\n",
+ " for future, input_val in futures.items():\n",
+ " result: int = future.result()\n",
+ " print(f\" {input_val}^3 = {result}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "c9d0e1f2",
+ "metadata": {},
+ "source": [
+ "## map() vs submit(): When to Use Each\n",
+ "\n",
+ "| Feature | `map()` | `submit()` |\n",
+ "|---------|---------|:-----------|\n",
+ "| Returns | Iterator of results | Individual `Future` objects |\n",
+ "| Order | Results in input order | Results in completion order (with `as_completed`) |\n",
+ "| Arguments | Single iterable per param | Full control of args/kwargs |\n",
+ "| Use case | Batch processing same operation | Different tasks or custom handling |\n",
+ "| Error handling | Exception raised during iteration | Exception stored in `Future` |\n",
+ "\n",
+ "**Rule of thumb**: Use `map()` for simple \"apply same function to many inputs\" tasks.\n",
+ "Use `submit()` when you need more control over individual tasks."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d0e1f2a3",
+ "metadata": {},
+ "source": [
+ "## as_completed(): Results in Completion Order\n",
+ "\n",
+ "`concurrent.futures.as_completed(futures)` yields futures as they **finish**, not in\n",
+ "the order they were submitted. This is useful when you want to process results as\n",
+ "soon as they are available, rather than waiting for all of them."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "e1f2a3b4",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import time\n",
+ "from concurrent.futures import Future, ProcessPoolExecutor, as_completed\n",
+ "\n",
+ "\n",
+ "def variable_work(task_id: int) -> tuple[int, float]:\n",
+ " \"\"\"Simulate work with variable duration.\"\"\"\n",
+ " # Tasks with higher IDs finish faster\n",
+ " duration: float = 0.5 - (task_id * 0.1)\n",
+ " duration = max(duration, 0.05)\n",
+ " time.sleep(duration)\n",
+ " return (task_id, duration)\n",
+ "\n",
+ "\n",
+ "with ProcessPoolExecutor(max_workers=3) as executor:\n",
+ " futures: dict[Future[tuple[int, float]], int] = {\n",
+ " executor.submit(variable_work, i): i\n",
+ " for i in range(5)\n",
+ " }\n",
+ "\n",
+ " print(\"Results in completion order:\")\n",
+ " for future in as_completed(futures):\n",
+ " task_id, duration = future.result()\n",
+ " print(f\" Task {task_id} completed (took {duration:.2f}s)\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "f2a3b4c5",
+ "metadata": {},
+ "source": [
+ "## Error Handling with Futures\n",
+ "\n",
+ "When a worker function raises an exception, it is captured by the `Future` object and\n",
+ "re-raised when you call `.result()` or `.exception()`. This makes error handling\n",
+ "straightforward."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a3b4c5d6",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from concurrent.futures import Future, ProcessPoolExecutor\n",
+ "\n",
+ "\n",
+ "def divide(a: float, b: float) -> float:\n",
+ " \"\"\"Divide a by b. Raises ZeroDivisionError if b is zero.\"\"\"\n",
+ " return a / b\n",
+ "\n",
+ "\n",
+ "with ProcessPoolExecutor(max_workers=2) as executor:\n",
+ " # Submit tasks, one of which will fail\n",
+ " tasks: list[tuple[float, float]] = [(10, 2), (20, 4), (30, 0), (40, 5)]\n",
+ " futures: list[tuple[Future[float], tuple[float, float]]] = [\n",
+ " (executor.submit(divide, a, b), (a, b))\n",
+ " for a, b in tasks\n",
+ " ]\n",
+ "\n",
+ " for future, (a, b) in futures:\n",
+ " try:\n",
+ " result: float = future.result()\n",
+ " print(f\" {a} / {b} = {result}\")\n",
+ " except ZeroDivisionError:\n",
+ " print(f\" {a} / {b} = ERROR (division by zero)\")\n",
+ " # Inspect the exception stored in the Future\n",
+ " print(f\" future.exception() = {future.exception()!r}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b4c5d6e7",
+ "metadata": {},
+ "source": [
+ "## CPU-Bound vs I/O-Bound Workloads\n",
+ "\n",
+ "Choosing between `ProcessPoolExecutor` and `ThreadPoolExecutor` depends on the nature\n",
+ "of your workload:\n",
+ "\n",
+ "| Workload | Bottleneck | Best Executor | Why |\n",
+ "|----------|-----------|:--------------|:----|\n",
+ "| **CPU-bound** | Computation | `ProcessPoolExecutor` | Bypasses GIL, true parallelism |\n",
+ "| **I/O-bound** | Network, disk | `ThreadPoolExecutor` | Threads release GIL during I/O |\n",
+ "| **Mixed** | Both | Depends | Profile first; consider async I/O |\n",
+ "\n",
+ "The GIL (Global Interpreter Lock) prevents multiple threads from executing Python bytecode\n",
+ "simultaneously, but it is released during I/O operations. Processes each have their own GIL,\n",
+ "so CPU-bound work runs truly in parallel."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "c5d6e7f8",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import time\n",
+ "from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor\n",
+ "\n",
+ "\n",
+ "def cpu_heavy(n: int) -> int:\n",
+ " \"\"\"CPU-bound: compute sum of squares.\"\"\"\n",
+ " total: int = 0\n",
+ " for i in range(n):\n",
+ " total += i * i\n",
+ " return total\n",
+ "\n",
+ "\n",
+ "workload: list[int] = [1_500_000] * 8\n",
+ "\n",
+ "# Sequential baseline\n",
+ "start: float = time.perf_counter()\n",
+ "_ = [cpu_heavy(n) for n in workload]\n",
+ "seq_time: float = time.perf_counter() - start\n",
+ "\n",
+ "# Threads (limited by GIL for CPU-bound work)\n",
+ "start = time.perf_counter()\n",
+ "with ThreadPoolExecutor(max_workers=4) as executor:\n",
+ " _ = list(executor.map(cpu_heavy, workload))\n",
+ "thread_time: float = time.perf_counter() - start\n",
+ "\n",
+ "# Processes (bypasses GIL)\n",
+ "start = time.perf_counter()\n",
+ "with ProcessPoolExecutor(max_workers=4) as executor:\n",
+ " _ = list(executor.map(cpu_heavy, workload))\n",
+ "process_time: float = time.perf_counter() - start\n",
+ "\n",
+ "print(\"CPU-bound workload comparison:\")\n",
+ "print(f\" Sequential: {seq_time:.3f}s\")\n",
+ "print(f\" ThreadPoolExecutor: {thread_time:.3f}s (speedup: {seq_time/thread_time:.2f}x)\")\n",
+ "print(f\" ProcessPoolExecutor: {process_time:.3f}s (speedup: {seq_time/process_time:.2f}x)\")\n",
+ "print(f\"\\nProcessPoolExecutor wins for CPU-bound tasks!\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "d6e7f8a9",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import time\n",
+ "from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor\n",
+ "\n",
+ "\n",
+ "def io_heavy(seconds: float) -> float:\n",
+ " \"\"\"I/O-bound: simulate waiting for a network response.\"\"\"\n",
+ " time.sleep(seconds)\n",
+ " return seconds\n",
+ "\n",
+ "\n",
+ "io_workload: list[float] = [0.2] * 8\n",
+ "\n",
+ "# Sequential baseline\n",
+ "start: float = time.perf_counter()\n",
+ "_ = [io_heavy(s) for s in io_workload]\n",
+ "seq_time: float = time.perf_counter() - start\n",
+ "\n",
+ "# Threads (efficient for I/O)\n",
+ "start = time.perf_counter()\n",
+ "with ThreadPoolExecutor(max_workers=4) as executor:\n",
+ " _ = list(executor.map(io_heavy, io_workload))\n",
+ "thread_time: float = time.perf_counter() - start\n",
+ "\n",
+ "# Processes (more overhead for I/O)\n",
+ "start = time.perf_counter()\n",
+ "with ProcessPoolExecutor(max_workers=4) as executor:\n",
+ " _ = list(executor.map(io_heavy, io_workload))\n",
+ "process_time: float = time.perf_counter() - start\n",
+ "\n",
+ "print(\"I/O-bound workload comparison:\")\n",
+ "print(f\" Sequential: {seq_time:.3f}s\")\n",
+ "print(f\" ThreadPoolExecutor: {thread_time:.3f}s (speedup: {seq_time/thread_time:.2f}x)\")\n",
+ "print(f\" ProcessPoolExecutor: {process_time:.3f}s (speedup: {seq_time/process_time:.2f}x)\")\n",
+ "print(f\"\\nBoth work for I/O, but threads have less overhead!\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e7f8a9b0",
+ "metadata": {},
+ "source": [
+ "## Callbacks on Futures\n",
+ "\n",
+ "You can attach callbacks to `Future` objects using `add_done_callback()`. The callback\n",
+ "function is called with the `Future` as its argument when the task completes. This is\n",
+ "useful for logging, chaining tasks, or updating progress."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f8a9b0c1",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from concurrent.futures import Future, ProcessPoolExecutor\n",
+ "\n",
+ "\n",
+ "def compute_factorial(n: int) -> int:\n",
+ " \"\"\"Compute n! iteratively.\"\"\"\n",
+ " result: int = 1\n",
+ " for i in range(2, n + 1):\n",
+ " result *= i\n",
+ " return result\n",
+ "\n",
+ "\n",
+ "def on_complete(future: Future[int]) -> None:\n",
+ " \"\"\"Callback when a factorial computation completes.\"\"\"\n",
+ " result: int = future.result()\n",
+ " print(f\" Callback: computation finished, result = {result}\")\n",
+ "\n",
+ "\n",
+ "with ProcessPoolExecutor(max_workers=2) as executor:\n",
+ " print(\"Submitting tasks with callbacks:\")\n",
+ " for n in [5, 8, 10, 12]:\n",
+ " future: Future[int] = executor.submit(compute_factorial, n)\n",
+ " future.add_done_callback(on_complete)\n",
+ "\n",
+ "print(\"All tasks completed.\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a9b0c1d2",
+ "metadata": {},
+ "source": [
+ "## os.cpu_count() and Choosing max_workers\n",
+ "\n",
+ "`os.cpu_count()` returns the number of logical CPUs on the system. This is the default\n",
+ "value for `max_workers` in `ProcessPoolExecutor`.\n",
+ "\n",
+ "Guidelines for choosing `max_workers`:\n",
+ "- **CPU-bound**: Use `os.cpu_count()` or slightly less (leave room for other processes)\n",
+ "- **I/O-bound**: Can use more workers than CPUs since they spend time waiting\n",
+ "- **Memory-constrained**: Each process uses its own memory, so watch usage"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b0c1d2e3",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import os\n",
+ "from concurrent.futures import ProcessPoolExecutor\n",
+ "\n",
+ "\n",
+ "def get_pid(_: int = 0) -> int:\n",
+ " \"\"\"Return the current process ID.\"\"\"\n",
+ " return os.getpid()\n",
+ "\n",
+ "\n",
+ "# os.cpu_count() tells us how many CPUs are available\n",
+ "cpus: int | None = os.cpu_count()\n",
+ "print(f\"os.cpu_count(): {cpus}\")\n",
+ "assert cpus is not None\n",
+ "assert cpus >= 1\n",
+ "\n",
+ "# Default max_workers = cpu_count\n",
+ "with ProcessPoolExecutor() as executor:\n",
+ " # _max_workers is an implementation detail, shown for educational purposes\n",
+ " actual_workers: int | None = executor._max_workers\n",
+ " print(f\"Default max_workers: {actual_workers}\")\n",
+ "\n",
+ "# Practical: verify processes run on different PIDs\n",
+ "with ProcessPoolExecutor(max_workers=4) as executor:\n",
+ " pids: list[int] = list(executor.map(get_pid, range(8)))\n",
+ " unique: set[int] = set(pids)\n",
+ " print(f\"\\n8 tasks across 4 workers:\")\n",
+ " print(f\" PIDs: {pids}\")\n",
+ " print(f\" Unique PIDs: {unique}\")\n",
+ " print(f\" All differ from main ({os.getpid()}): {all(p != os.getpid() for p in pids)}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "c1d2e3f4",
+ "metadata": {},
+ "source": [
+ "## Practical: Real-World Pattern -- Parallel Data Processing\n",
+ "\n",
+ "A common real-world pattern is to process a batch of items in parallel, collecting\n",
+ "successes and failures separately. This example demonstrates using `submit()` with\n",
+ "error handling for robust parallel processing."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "d2e3f4a5",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from concurrent.futures import Future, ProcessPoolExecutor, as_completed\n",
+ "\n",
+ "\n",
+ "def process_record(record_id: int) -> dict[str, int | str]:\n",
+ " \"\"\"Simulate processing a data record. Some records fail.\"\"\"\n",
+ " if record_id % 5 == 0:\n",
+ " raise ValueError(f\"Record {record_id} is corrupted\")\n",
+ " return {\"id\": record_id, \"status\": \"processed\", \"value\": record_id * 10}\n",
+ "\n",
+ "\n",
+ "record_ids: list[int] = list(range(1, 16))\n",
+ "successes: list[dict[str, int | str]] = []\n",
+ "failures: list[tuple[int, str]] = []\n",
+ "\n",
+ "with ProcessPoolExecutor(max_workers=3) as executor:\n",
+ " # Submit all tasks\n",
+ " future_to_id: dict[Future[dict[str, int | str]], int] = {\n",
+ " executor.submit(process_record, rid): rid\n",
+ " for rid in record_ids\n",
+ " }\n",
+ "\n",
+ " # Process results as they complete\n",
+ " for future in as_completed(future_to_id):\n",
+ " record_id: int = future_to_id[future]\n",
+ " try:\n",
+ " result: dict[str, int | str] = future.result()\n",
+ " successes.append(result)\n",
+ " except ValueError as e:\n",
+ " failures.append((record_id, str(e)))\n",
+ "\n",
+ "print(f\"Processed {len(successes)} records successfully:\")\n",
+ "for s in sorted(successes, key=lambda x: x[\"id\"]):\n",
+ " print(f\" Record {s['id']}: value={s['value']}\")\n",
+ "\n",
+ "print(f\"\\n{len(failures)} records failed:\")\n",
+ "for record_id, error in sorted(failures):\n",
+ " print(f\" Record {record_id}: {error}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e3f4a5b6",
+ "metadata": {},
+ "source": [
+ "## Summary\n",
+ "\n",
+ "### Key Takeaways\n",
+ "\n",
+ "| Concept | API | Purpose |\n",
+ "|---------|-----|:--------|\n",
+ "| **Executor** | `ProcessPoolExecutor(max_workers=n)` | High-level process pool |\n",
+ "| **map()** | `executor.map(func, iterable)` | Parallel map, results in input order |\n",
+ "| **submit()** | `executor.submit(func, *args)` | Submit single task, returns Future |\n",
+ "| **Future** | `future.result()`, `.done()`, `.exception()` | Pending result handle |\n",
+ "| **as_completed()** | `as_completed(futures)` | Yield futures in completion order |\n",
+ "| **Callbacks** | `future.add_done_callback(fn)` | Run code when a future completes |\n",
+ "| **cpu_count()** | `os.cpu_count()` | Number of logical CPUs |\n",
+ "\n",
+ "### Choosing the Right Executor\n",
+ "\n",
+ "| Scenario | Use |\n",
+ "|----------|:----|\n",
+ "| Heavy computation (math, parsing, compression) | `ProcessPoolExecutor` |\n",
+ "| Network requests, file I/O, database queries | `ThreadPoolExecutor` |\n",
+ "| Simple batch: same function, many inputs | `executor.map()` |\n",
+ "| Complex: different tasks, error handling, callbacks | `executor.submit()` |\n",
+ "\n",
+ "### Best Practices\n",
+ "- Always use executors as context managers to ensure proper cleanup\n",
+ "- Define worker functions at module level (they must be picklable)\n",
+ "- Use `as_completed()` when you want to process results as soon as they are ready\n",
+ "- Handle exceptions from `.result()` -- they are re-raised from the worker\n",
+ "- For CPU-bound tasks, `max_workers=os.cpu_count()` is a good starting point\n",
+ "- Prefer `ProcessPoolExecutor` over `multiprocessing.Pool` for new code -- it has a cleaner API"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "name": "python",
+ "version": "3.13.0"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
\ No newline at end of file
diff --git a/src/chapter_27/README.md b/src/chapter_27/README.md
new file mode 100644
index 0000000..c470c94
--- /dev/null
+++ b/src/chapter_27/README.md
@@ -0,0 +1,9 @@
+# Chapter 27: Multiprocessing and Parallelism
+
+True parallelism with `multiprocessing` — processes, pools, shared state, inter-process communication, and `ProcessPoolExecutor`.
+
+## Notebooks
+
+1. **01_multiprocessing_basics.ipynb** — `Process`, `Pool`, `map()`, process lifecycle
+2. **02_shared_state_and_ipc.ipynb** — `Queue`, `Pipe`, `Value`, `Array`, `Manager`, locks
+3. **03_process_pool_executor.ipynb** — `ProcessPoolExecutor`, `Future`, CPU-bound vs I/O-bound, `map()` vs `submit()`
diff --git a/src/chapter_27/__init__.py b/src/chapter_27/__init__.py
new file mode 100644
index 0000000..cfa6ee1
--- /dev/null
+++ b/src/chapter_27/__init__.py
@@ -0,0 +1 @@
+"""Chapter 27: Multiprocessing and Parallelism."""
diff --git a/src/chapter_28/01_argparse_fundamentals.ipynb b/src/chapter_28/01_argparse_fundamentals.ipynb
new file mode 100644
index 0000000..f8198ac
--- /dev/null
+++ b/src/chapter_28/01_argparse_fundamentals.ipynb
@@ -0,0 +1,559 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "a1b2c3d4",
+ "metadata": {},
+ "source": [
+ "# Chapter 28: Argparse Fundamentals\n",
+ "\n",
+ "This notebook covers the `argparse` module for building command-line interfaces in Python. We explore `ArgumentParser`, positional and optional arguments, type conversion, choices, defaults, and `nargs`.\n",
+ "\n",
+ "## Key Concepts\n",
+ "- **ArgumentParser**: The main entry point for defining CLI arguments\n",
+ "- **Positional arguments**: Required arguments identified by position\n",
+ "- **Optional arguments**: Flag-style arguments with `--` prefix\n",
+ "- **Type conversion**: Automatic conversion of string inputs to desired types\n",
+ "- **Choices and defaults**: Restricting and pre-filling argument values"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b2c3d4e5",
+ "metadata": {},
+ "source": [
+ "## Section 1: Creating an ArgumentParser\n",
+ "\n",
+ "`ArgumentParser` is the central class in `argparse`. It holds all the information needed to parse the command line into Python data types."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "c3d4e5f6",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import argparse\n",
+ "\n",
+ "# Create a basic parser with a description\n",
+ "parser: argparse.ArgumentParser = argparse.ArgumentParser(\n",
+ " description=\"A sample command-line tool\",\n",
+ " prog=\"mytool\", # Override the program name\n",
+ ")\n",
+ "\n",
+ "print(f\"Parser type: {type(parser).__name__}\")\n",
+ "print(f\"Program name: {parser.prog}\")\n",
+ "print(f\"Description: {parser.description}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "d4e5f6a7",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# The parser generates help text automatically\n",
+ "parser = argparse.ArgumentParser(\n",
+ " prog=\"greet\",\n",
+ " description=\"Greet someone on the command line\",\n",
+ " epilog=\"Example: greet Alice --loud\",\n",
+ ")\n",
+ "\n",
+ "# print_help() shows what --help would display\n",
+ "parser.print_help()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e5f6a7b8",
+ "metadata": {},
+ "source": [
+ "## Section 2: Positional Arguments\n",
+ "\n",
+ "Positional arguments are required by default and identified by their position on the command line rather than by a flag."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f6a7b8c9",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Positional arguments are required\n",
+ "parser = argparse.ArgumentParser(prog=\"greet\")\n",
+ "parser.add_argument(\"name\") # A simple positional argument\n",
+ "\n",
+ "# parse_args takes a list of strings (simulating sys.argv[1:])\n",
+ "args: argparse.Namespace = parser.parse_args([\"Alice\"])\n",
+ "\n",
+ "print(f\"Parsed name: {args.name}\")\n",
+ "print(f\"Namespace type: {type(args).__name__}\")\n",
+ "print(f\"As dict: {vars(args)}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a7b8c9d0",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Multiple positional arguments are parsed in order\n",
+ "parser = argparse.ArgumentParser(prog=\"copy\")\n",
+ "parser.add_argument(\"source\")\n",
+ "parser.add_argument(\"destination\")\n",
+ "\n",
+ "args = parser.parse_args([\"input.txt\", \"output.txt\"])\n",
+ "\n",
+ "print(f\"Source: {args.source}\")\n",
+ "print(f\"Destination: {args.destination}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b8c9d0e1",
+ "metadata": {},
+ "source": [
+ "## Section 3: Optional Arguments\n",
+ "\n",
+ "Optional arguments use the `--` prefix (long form) or `-` prefix (short form). They are not required unless you set `required=True`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "c9d0e1f2",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Optional arguments with -- prefix\n",
+ "parser = argparse.ArgumentParser(prog=\"server\")\n",
+ "parser.add_argument(\"--verbose\", action=\"store_true\")\n",
+ "parser.add_argument(\"--count\", type=int, default=1)\n",
+ "\n",
+ "# Providing both optional arguments\n",
+ "args = parser.parse_args([\"--verbose\", \"--count\", \"5\"])\n",
+ "print(f\"verbose: {args.verbose} (type: {type(args.verbose).__name__})\")\n",
+ "print(f\"count: {args.count} (type: {type(args.count).__name__})\")\n",
+ "\n",
+ "# Omitting optional arguments uses defaults\n",
+ "args_default = parser.parse_args([])\n",
+ "print(f\"\\nDefaults -> verbose: {args_default.verbose}, count: {args_default.count}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "d0e1f2a3",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Short and long flags together\n",
+ "parser = argparse.ArgumentParser(prog=\"tool\")\n",
+ "parser.add_argument(\"-v\", \"--verbose\", action=\"store_true\")\n",
+ "parser.add_argument(\"-o\", \"--output\", type=str, default=\"stdout\")\n",
+ "\n",
+ "# Using short flags\n",
+ "args = parser.parse_args([\"-v\", \"-o\", \"results.txt\"])\n",
+ "print(f\"verbose: {args.verbose}\")\n",
+ "print(f\"output: {args.output}\")\n",
+ "\n",
+ "# Using long flags produces the same result\n",
+ "args_long = parser.parse_args([\"--verbose\", \"--output\", \"results.txt\"])\n",
+ "print(f\"\\nSame result with long flags: {vars(args) == vars(args_long)}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e1f2a3b4",
+ "metadata": {},
+ "source": [
+ "## Section 4: Type Conversion\n",
+ "\n",
+ "By default, all arguments are parsed as strings. The `type` parameter automatically converts the input to the desired Python type."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f2a3b4c5",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Built-in type conversion\n",
+ "parser = argparse.ArgumentParser(prog=\"calc\")\n",
+ "parser.add_argument(\"value\", type=int)\n",
+ "parser.add_argument(\"--factor\", type=float, default=1.0)\n",
+ "\n",
+ "args = parser.parse_args([\"42\", \"--factor\", \"2.5\"])\n",
+ "\n",
+ "result: float = args.value * args.factor\n",
+ "print(f\"value: {args.value} (type: {type(args.value).__name__})\")\n",
+ "print(f\"factor: {args.factor} (type: {type(args.factor).__name__})\")\n",
+ "print(f\"result: {result}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a3b4c5d6",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import pathlib\n",
+ "\n",
+ "# Custom type functions for validation\n",
+ "def positive_int(value: str) -> int:\n",
+ " \"\"\"Convert string to positive integer, raising on invalid input.\"\"\"\n",
+ " ivalue: int = int(value)\n",
+ " if ivalue <= 0:\n",
+ " raise argparse.ArgumentTypeError(f\"{value} is not a positive integer\")\n",
+ " return ivalue\n",
+ "\n",
+ "parser = argparse.ArgumentParser(prog=\"worker\")\n",
+ "parser.add_argument(\"--threads\", type=positive_int, default=4)\n",
+ "parser.add_argument(\"--config\", type=pathlib.Path, default=pathlib.Path(\"config.toml\"))\n",
+ "\n",
+ "args = parser.parse_args([\"--threads\", \"8\", \"--config\", \"/etc/app/config.toml\"])\n",
+ "print(f\"threads: {args.threads} (type: {type(args.threads).__name__})\")\n",
+ "print(f\"config: {args.config} (type: {type(args.config).__name__})\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b4c5d6e7",
+ "metadata": {},
+ "source": [
+ "## Section 5: Choices and Defaults\n",
+ "\n",
+ "The `choices` parameter restricts an argument to a set of allowed values. The `default` parameter sets a fallback when the argument is omitted."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "c5d6e7f8",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# choices restricts accepted values\n",
+ "parser = argparse.ArgumentParser(prog=\"logger\")\n",
+ "parser.add_argument(\n",
+ " \"--level\",\n",
+ " choices=[\"debug\", \"info\", \"warning\", \"error\", \"critical\"],\n",
+ " default=\"info\",\n",
+ ")\n",
+ "\n",
+ "args = parser.parse_args([\"--level\", \"debug\"])\n",
+ "print(f\"Log level: {args.level}\")\n",
+ "\n",
+ "# Default value when argument is omitted\n",
+ "args_default = parser.parse_args([])\n",
+ "print(f\"Default level: {args_default.level}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "d6e7f8a9",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Demonstrating what happens with invalid choices\n",
+ "parser = argparse.ArgumentParser(prog=\"format\")\n",
+ "parser.add_argument(\"--output-format\", choices=[\"json\", \"csv\", \"xml\"])\n",
+ "\n",
+ "# Valid choice\n",
+ "args = parser.parse_args([\"--output-format\", \"json\"])\n",
+ "print(f\"Format: {args.output_format}\")\n",
+ "\n",
+ "# Invalid choice raises an error\n",
+ "try:\n",
+ " parser.parse_args([\"--output-format\", \"yaml\"])\n",
+ "except SystemExit:\n",
+ " print(\"Invalid choice 'yaml' - argparse rejects it and exits\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e7f8a9b0",
+ "metadata": {},
+ "source": [
+ "## Section 6: Nargs — Multiple Values\n",
+ "\n",
+ "The `nargs` parameter controls how many values an argument consumes:\n",
+ "- `nargs='+'` — one or more values (returns a list)\n",
+ "- `nargs='*'` — zero or more values (returns a list)\n",
+ "- `nargs='?'` — zero or one value\n",
+ "- `nargs=N` — exactly N values (returns a list)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f8a9b0c1",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# nargs='+' requires one or more values\n",
+ "parser = argparse.ArgumentParser(prog=\"process\")\n",
+ "parser.add_argument(\"files\", nargs=\"+\")\n",
+ "\n",
+ "args = parser.parse_args([\"a.txt\", \"b.txt\", \"c.txt\"])\n",
+ "print(f\"Files: {args.files}\")\n",
+ "print(f\"Type: {type(args.files).__name__}\")\n",
+ "print(f\"Count: {len(args.files)}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a9b0c1d2",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# nargs='*' accepts zero or more values\n",
+ "parser = argparse.ArgumentParser(prog=\"collect\")\n",
+ "parser.add_argument(\"--tags\", nargs=\"*\", default=[])\n",
+ "parser.add_argument(\"--ports\", nargs=3, type=int) # Exactly 3 values\n",
+ "\n",
+ "args = parser.parse_args([\"--tags\", \"dev\", \"test\", \"--ports\", \"80\", \"443\", \"8080\"])\n",
+ "print(f\"Tags: {args.tags}\")\n",
+ "print(f\"Ports: {args.ports}\")\n",
+ "\n",
+ "# Zero tags is valid with nargs='*'\n",
+ "args_empty = parser.parse_args([\"--tags\", \"--ports\", \"80\", \"443\", \"8080\"])\n",
+ "print(f\"\\nEmpty tags: {args_empty.tags}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b0c1d2e3",
+ "metadata": {},
+ "source": [
+ "## Section 7: Actions\n",
+ "\n",
+ "The `action` parameter controls how arguments are processed:\n",
+ "- `\"store\"` — default, stores the value\n",
+ "- `\"store_true\"` / `\"store_false\"` — boolean flags\n",
+ "- `\"count\"` — counts occurrences (e.g., `-vvv`)\n",
+ "- `\"append\"` — collects repeated arguments into a list"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "c1d2e3f4",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Demonstrating various actions\n",
+ "parser = argparse.ArgumentParser(prog=\"demo\")\n",
+ "\n",
+ "# store_true / store_false for boolean flags\n",
+ "parser.add_argument(\"--debug\", action=\"store_true\")\n",
+ "parser.add_argument(\"--no-cache\", action=\"store_true\")\n",
+ "\n",
+ "# count tracks how many times a flag appears\n",
+ "parser.add_argument(\"-v\", \"--verbose\", action=\"count\", default=0)\n",
+ "\n",
+ "# append collects repeated flags into a list\n",
+ "parser.add_argument(\"--include\", action=\"append\", default=[])\n",
+ "\n",
+ "args = parser.parse_args([\n",
+ " \"--debug\",\n",
+ " \"-vvv\", # Three v's = verbosity 3\n",
+ " \"--include\", \"src\",\n",
+ " \"--include\", \"tests\",\n",
+ "])\n",
+ "\n",
+ "print(f\"debug: {args.debug}\")\n",
+ "print(f\"no_cache: {args.no_cache}\")\n",
+ "print(f\"verbosity: {args.verbose}\")\n",
+ "print(f\"includes: {args.include}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d2e3f4a5",
+ "metadata": {},
+ "source": [
+ "## Section 8: Help Text and Metavar\n",
+ "\n",
+ "Good CLI tools provide clear help text. The `help` parameter adds descriptions, and `metavar` customizes the placeholder name in usage messages."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "e3f4a5b6",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Well-documented argument parser\n",
+ "parser = argparse.ArgumentParser(\n",
+ " prog=\"deploy\",\n",
+ " description=\"Deploy an application to the target environment\",\n",
+ ")\n",
+ "parser.add_argument(\n",
+ " \"app_name\",\n",
+ " help=\"Name of the application to deploy\",\n",
+ " metavar=\"APP\",\n",
+ ")\n",
+ "parser.add_argument(\n",
+ " \"--env\",\n",
+ " choices=[\"dev\", \"staging\", \"prod\"],\n",
+ " default=\"dev\",\n",
+ " help=\"Target environment (default: %(default)s)\",\n",
+ ")\n",
+ "parser.add_argument(\n",
+ " \"--replicas\",\n",
+ " type=int,\n",
+ " default=1,\n",
+ " metavar=\"N\",\n",
+ " help=\"Number of replicas to run (default: %(default)s)\",\n",
+ ")\n",
+ "parser.add_argument(\n",
+ " \"--dry-run\",\n",
+ " action=\"store_true\",\n",
+ " help=\"Show what would be done without executing\",\n",
+ ")\n",
+ "\n",
+ "parser.print_help()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f4a5b6c7",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Parsing with the documented parser\n",
+ "args = parser.parse_args([\"myapp\", \"--env\", \"staging\", \"--replicas\", \"3\", \"--dry-run\"])\n",
+ "\n",
+ "print(f\"App: {args.app_name}\")\n",
+ "print(f\"Environment: {args.env}\")\n",
+ "print(f\"Replicas: {args.replicas}\")\n",
+ "print(f\"Dry run: {args.dry_run}\")\n",
+ "\n",
+ "# Namespace can be accessed as a dictionary too\n",
+ "config: dict[str, object] = vars(args)\n",
+ "print(f\"\\nFull config: {config}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a5b6c7d8",
+ "metadata": {},
+ "source": [
+ "## Section 9: Putting It All Together\n",
+ "\n",
+ "A realistic example combining positional arguments, optional flags, types, choices, and defaults."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b6c7d8e9",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def build_parser() -> argparse.ArgumentParser:\n",
+ " \"\"\"Build a complete argument parser for a file search tool.\"\"\"\n",
+ " parser = argparse.ArgumentParser(\n",
+ " prog=\"filesearch\",\n",
+ " description=\"Search for patterns in files\",\n",
+ " )\n",
+ "\n",
+ " # Positional: the pattern to search for\n",
+ " parser.add_argument(\"pattern\", help=\"Regex pattern to search for\")\n",
+ "\n",
+ " # Positional: one or more files\n",
+ " parser.add_argument(\"files\", nargs=\"+\", help=\"Files to search\")\n",
+ "\n",
+ " # Optional flags\n",
+ " parser.add_argument(\"-i\", \"--ignore-case\", action=\"store_true\",\n",
+ " help=\"Case-insensitive matching\")\n",
+ " parser.add_argument(\"-n\", \"--line-numbers\", action=\"store_true\",\n",
+ " help=\"Show line numbers\")\n",
+ " parser.add_argument(\"--max-results\", type=positive_int, default=100,\n",
+ " metavar=\"N\", help=\"Maximum results (default: %(default)s)\")\n",
+ " parser.add_argument(\"--format\", choices=[\"text\", \"json\", \"csv\"],\n",
+ " default=\"text\", help=\"Output format (default: %(default)s)\")\n",
+ "\n",
+ " return parser\n",
+ "\n",
+ "\n",
+ "search_parser: argparse.ArgumentParser = build_parser()\n",
+ "search_parser.print_help()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "c7d8e9f0",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Simulate command-line invocation\n",
+ "args = search_parser.parse_args([\n",
+ " \"TODO\",\n",
+ " \"main.py\", \"utils.py\", \"config.py\",\n",
+ " \"--ignore-case\",\n",
+ " \"--line-numbers\",\n",
+ " \"--max-results\", \"50\",\n",
+ " \"--format\", \"json\",\n",
+ "])\n",
+ "\n",
+ "print(f\"Pattern: {args.pattern!r}\")\n",
+ "print(f\"Files: {args.files}\")\n",
+ "print(f\"Ignore case: {args.ignore_case}\")\n",
+ "print(f\"Line numbers: {args.line_numbers}\")\n",
+ "print(f\"Max results: {args.max_results}\")\n",
+ "print(f\"Format: {args.format}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d8e9f0a1",
+ "metadata": {},
+ "source": [
+ "## Summary\n",
+ "\n",
+ "### ArgumentParser Basics\n",
+ "- **`ArgumentParser()`** creates a parser with optional `prog`, `description`, and `epilog`\n",
+ "- **`add_argument()`** defines arguments with their type, default, help, and constraints\n",
+ "- **`parse_args()`** returns a `Namespace` object with parsed values\n",
+ "\n",
+ "### Argument Types\n",
+ "- **Positional**: Required by default, identified by order\n",
+ "- **Optional**: Prefixed with `--` (long) or `-` (short), not required by default\n",
+ "\n",
+ "### Key Parameters\n",
+ "- **`type`**: Converts input strings to the desired Python type\n",
+ "- **`choices`**: Restricts values to a predefined set\n",
+ "- **`default`**: Fallback value when argument is omitted\n",
+ "- **`nargs`**: Controls how many values an argument accepts (`+`, `*`, `?`, or `N`)\n",
+ "- **`action`**: Controls processing (`store_true`, `count`, `append`)\n",
+ "- **`help`** and **`metavar`**: Document the argument for users"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "name": "python",
+ "version": "3.13.0"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
\ No newline at end of file
diff --git a/src/chapter_28/02_subcommands_and_groups.ipynb b/src/chapter_28/02_subcommands_and_groups.ipynb
new file mode 100644
index 0000000..6942da2
--- /dev/null
+++ b/src/chapter_28/02_subcommands_and_groups.ipynb
@@ -0,0 +1,504 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "a1000001",
+ "metadata": {},
+ "source": [
+ "# Chapter 28: Subcommands and Groups\n",
+ "\n",
+ "This notebook covers advanced `argparse` patterns for building complex CLI tools: subparsers for git-style subcommands, mutually exclusive groups for conflicting options, and argument groups for organized help output.\n",
+ "\n",
+ "## Key Concepts\n",
+ "- **Subparsers**: Create distinct subcommands (like `git init`, `git clone`)\n",
+ "- **Mutually exclusive groups**: Prevent conflicting options from being used together\n",
+ "- **Argument groups**: Organize related arguments in help output\n",
+ "- **Defaults per subcommand**: Each subcommand can have its own arguments and defaults"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a1000002",
+ "metadata": {},
+ "source": [
+ "## Section 1: Subparsers — Git-Style Subcommands\n",
+ "\n",
+ "Many CLI tools use subcommands: `git clone`, `docker run`, `pip install`. In `argparse`, this is achieved with `add_subparsers()`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a1000003",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import argparse\n",
+ "\n",
+ "# Create the top-level parser\n",
+ "parser: argparse.ArgumentParser = argparse.ArgumentParser(prog=\"vcs\")\n",
+ "\n",
+ "# add_subparsers creates a special action for subcommands\n",
+ "# dest stores which subcommand was chosen\n",
+ "subparsers = parser.add_subparsers(dest=\"command\", help=\"Available commands\")\n",
+ "\n",
+ "# Each subcommand gets its own parser\n",
+ "init_parser = subparsers.add_parser(\"init\", help=\"Initialize a new repository\")\n",
+ "init_parser.add_argument(\"--bare\", action=\"store_true\", help=\"Create a bare repository\")\n",
+ "\n",
+ "clone_parser = subparsers.add_parser(\"clone\", help=\"Clone a repository\")\n",
+ "clone_parser.add_argument(\"url\", help=\"Repository URL to clone\")\n",
+ "clone_parser.add_argument(\"--depth\", type=int, help=\"Shallow clone depth\")\n",
+ "\n",
+ "parser.print_help()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a1000004",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Parsing the 'init' subcommand\n",
+ "args: argparse.Namespace = parser.parse_args([\"init\", \"--bare\"])\n",
+ "\n",
+ "print(f\"Command: {args.command}\")\n",
+ "print(f\"Bare: {args.bare}\")\n",
+ "print(f\"Full namespace: {vars(args)}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a1000005",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Parsing the 'clone' subcommand\n",
+ "args = parser.parse_args([\"clone\", \"https://example.com/repo\", \"--depth\", \"1\"])\n",
+ "\n",
+ "print(f\"Command: {args.command}\")\n",
+ "print(f\"URL: {args.url}\")\n",
+ "print(f\"Depth: {args.depth}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a1000006",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Subcommand-specific help\n",
+ "print(\"=== clone subcommand help ===\")\n",
+ "clone_parser.print_help()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a1000007",
+ "metadata": {},
+ "source": [
+ "## Section 2: Dispatching Subcommands\n",
+ "\n",
+ "A common pattern is using `set_defaults(func=...)` to associate each subcommand with a handler function. This avoids long if/elif chains."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a1000008",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from typing import Any\n",
+ "\n",
+ "\n",
+ "def handle_start(args: argparse.Namespace) -> str:\n",
+ " \"\"\"Handle the 'start' subcommand.\"\"\"\n",
+ " return f\"Starting service '{args.name}' on port {args.port}\"\n",
+ "\n",
+ "\n",
+ "def handle_stop(args: argparse.Namespace) -> str:\n",
+ " \"\"\"Handle the 'stop' subcommand.\"\"\"\n",
+ " force_msg: str = \" (forced)\" if args.force else \"\"\n",
+ " return f\"Stopping service '{args.name}'{force_msg}\"\n",
+ "\n",
+ "\n",
+ "def handle_status(args: argparse.Namespace) -> str:\n",
+ " \"\"\"Handle the 'status' subcommand.\"\"\"\n",
+ " return f\"Status of all services (verbose={args.verbose})\"\n",
+ "\n",
+ "\n",
+ "# Build parser with function dispatch\n",
+ "parser = argparse.ArgumentParser(prog=\"svc\")\n",
+ "sub = parser.add_subparsers(dest=\"command\")\n",
+ "\n",
+ "# 'start' subcommand\n",
+ "start_cmd = sub.add_parser(\"start\")\n",
+ "start_cmd.add_argument(\"name\")\n",
+ "start_cmd.add_argument(\"--port\", type=int, default=8080)\n",
+ "start_cmd.set_defaults(func=handle_start)\n",
+ "\n",
+ "# 'stop' subcommand\n",
+ "stop_cmd = sub.add_parser(\"stop\")\n",
+ "stop_cmd.add_argument(\"name\")\n",
+ "stop_cmd.add_argument(\"--force\", action=\"store_true\")\n",
+ "stop_cmd.set_defaults(func=handle_stop)\n",
+ "\n",
+ "# 'status' subcommand\n",
+ "status_cmd = sub.add_parser(\"status\")\n",
+ "status_cmd.add_argument(\"-v\", \"--verbose\", action=\"store_true\")\n",
+ "status_cmd.set_defaults(func=handle_status)\n",
+ "\n",
+ "# Dispatch: parse args and call the associated function\n",
+ "for cmd_line in [\"start web --port 3000\", \"stop web --force\", \"status -v\"]:\n",
+ " args = parser.parse_args(cmd_line.split())\n",
+ " result: str = args.func(args)\n",
+ " print(f\" $ svc {cmd_line}\")\n",
+ " print(f\" -> {result}\\n\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a1000009",
+ "metadata": {},
+ "source": [
+ "## Section 3: Mutually Exclusive Groups\n",
+ "\n",
+ "Mutually exclusive groups prevent two options from being used at the same time. This is useful for output format flags or conflicting modes."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a1000010",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Mutually exclusive output format flags\n",
+ "parser = argparse.ArgumentParser(prog=\"export\")\n",
+ "format_group = parser.add_mutually_exclusive_group()\n",
+ "format_group.add_argument(\"--json\", action=\"store_true\", help=\"Output as JSON\")\n",
+ "format_group.add_argument(\"--csv\", action=\"store_true\", help=\"Output as CSV\")\n",
+ "format_group.add_argument(\"--xml\", action=\"store_true\", help=\"Output as XML\")\n",
+ "\n",
+ "# Only one format flag can be used\n",
+ "args = parser.parse_args([\"--json\"])\n",
+ "print(f\"json: {args.json}, csv: {args.csv}, xml: {args.xml}\")\n",
+ "\n",
+ "args = parser.parse_args([\"--csv\"])\n",
+ "print(f\"json: {args.json}, csv: {args.csv}, xml: {args.xml}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a1000011",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Attempting to use two mutually exclusive options\n",
+ "try:\n",
+ " parser.parse_args([\"--json\", \"--csv\"])\n",
+ "except SystemExit:\n",
+ " print(\"Error: --json and --csv cannot be used together\")\n",
+ " print(\"argparse raises SystemExit when mutually exclusive args conflict\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a1000012",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# required=True forces the user to pick exactly one option\n",
+ "parser = argparse.ArgumentParser(prog=\"convert\")\n",
+ "mode_group = parser.add_mutually_exclusive_group(required=True)\n",
+ "mode_group.add_argument(\"--encode\", action=\"store_true\", help=\"Encode the input\")\n",
+ "mode_group.add_argument(\"--decode\", action=\"store_true\", help=\"Decode the input\")\n",
+ "\n",
+ "parser.add_argument(\"input_file\", help=\"File to process\")\n",
+ "\n",
+ "args = parser.parse_args([\"--encode\", \"data.bin\"])\n",
+ "print(f\"encode: {args.encode}, decode: {args.decode}, file: {args.input_file}\")\n",
+ "\n",
+ "# Omitting both from a required group is an error\n",
+ "try:\n",
+ " parser.parse_args([\"data.bin\"])\n",
+ "except SystemExit:\n",
+ " print(\"\\nError: one of --encode or --decode is required\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a1000013",
+ "metadata": {},
+ "source": [
+ "## Section 4: Determining the Selected Format\n",
+ "\n",
+ "When using mutually exclusive boolean flags, a helper function can determine which was selected."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a1000014",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def get_output_format(args: argparse.Namespace) -> str:\n",
+ " \"\"\"Determine the selected output format from mutually exclusive flags.\"\"\"\n",
+ " if args.json:\n",
+ " return \"json\"\n",
+ " elif args.csv:\n",
+ " return \"csv\"\n",
+ " elif args.xml:\n",
+ " return \"xml\"\n",
+ " return \"text\" # Default fallback\n",
+ "\n",
+ "\n",
+ "parser = argparse.ArgumentParser(prog=\"report\")\n",
+ "fmt_group = parser.add_mutually_exclusive_group()\n",
+ "fmt_group.add_argument(\"--json\", action=\"store_true\")\n",
+ "fmt_group.add_argument(\"--csv\", action=\"store_true\")\n",
+ "fmt_group.add_argument(\"--xml\", action=\"store_true\")\n",
+ "\n",
+ "# Test each format\n",
+ "for flags in [[\"--json\"], [\"--csv\"], [\"--xml\"], []]:\n",
+ " args = parser.parse_args(flags)\n",
+ " fmt: str = get_output_format(args)\n",
+ " print(f\"Flags: {flags or ['(none)']} -> format: {fmt}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a1000015",
+ "metadata": {},
+ "source": [
+ "## Section 5: Argument Groups\n",
+ "\n",
+ "Argument groups organize arguments into logical sections in the help output. Unlike mutually exclusive groups, they do not restrict which arguments can be used together."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a1000016",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Argument groups for organized help output\n",
+ "parser = argparse.ArgumentParser(prog=\"deploy\", description=\"Deploy an application\")\n",
+ "\n",
+ "# Connection arguments\n",
+ "conn_group = parser.add_argument_group(\"connection\", \"Server connection options\")\n",
+ "conn_group.add_argument(\"--host\", default=\"localhost\", help=\"Server hostname\")\n",
+ "conn_group.add_argument(\"--port\", type=int, default=22, help=\"Server port\")\n",
+ "conn_group.add_argument(\"--user\", default=\"deploy\", help=\"SSH username\")\n",
+ "\n",
+ "# Deployment arguments\n",
+ "deploy_group = parser.add_argument_group(\"deployment\", \"Deployment configuration\")\n",
+ "deploy_group.add_argument(\"--branch\", default=\"main\", help=\"Git branch to deploy\")\n",
+ "deploy_group.add_argument(\"--tag\", help=\"Specific tag to deploy\")\n",
+ "deploy_group.add_argument(\"--rollback\", action=\"store_true\", help=\"Rollback to previous version\")\n",
+ "\n",
+ "# Notification arguments\n",
+ "notify_group = parser.add_argument_group(\"notifications\", \"Notification settings\")\n",
+ "notify_group.add_argument(\"--notify-slack\", action=\"store_true\", help=\"Send Slack notification\")\n",
+ "notify_group.add_argument(\"--notify-email\", help=\"Send email notification to address\")\n",
+ "\n",
+ "parser.print_help()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a1000017",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# All groups can be used simultaneously (they are just for organization)\n",
+ "args = parser.parse_args([\n",
+ " \"--host\", \"prod.example.com\",\n",
+ " \"--port\", \"2222\",\n",
+ " \"--branch\", \"release/v2\",\n",
+ " \"--notify-slack\",\n",
+ "])\n",
+ "\n",
+ "print(f\"Host: {args.host}:{args.port}\")\n",
+ "print(f\"User: {args.user}\")\n",
+ "print(f\"Branch: {args.branch}\")\n",
+ "print(f\"Notify Slack: {args.notify_slack}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a1000018",
+ "metadata": {},
+ "source": [
+ "## Section 6: Combining Subcommands with Groups\n",
+ "\n",
+ "A real-world CLI often combines subparsers with argument groups and mutually exclusive options within each subcommand."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a1000019",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def build_db_cli() -> argparse.ArgumentParser:\n",
+ " \"\"\"Build a CLI parser for a database management tool.\"\"\"\n",
+ " parser = argparse.ArgumentParser(\n",
+ " prog=\"dbctl\",\n",
+ " description=\"Database management tool\",\n",
+ " )\n",
+ " parser.add_argument(\"-v\", \"--verbose\", action=\"store_true\")\n",
+ "\n",
+ " sub = parser.add_subparsers(dest=\"command\")\n",
+ "\n",
+ " # 'migrate' subcommand with mutually exclusive direction\n",
+ " migrate = sub.add_parser(\"migrate\", help=\"Run database migrations\")\n",
+ " direction = migrate.add_mutually_exclusive_group(required=True)\n",
+ " direction.add_argument(\"--up\", action=\"store_true\", help=\"Apply migrations\")\n",
+ " direction.add_argument(\"--down\", action=\"store_true\", help=\"Rollback migrations\")\n",
+ " migrate.add_argument(\"--steps\", type=int, default=1, help=\"Number of steps\")\n",
+ "\n",
+ " # 'dump' subcommand with argument groups\n",
+ " dump = sub.add_parser(\"dump\", help=\"Export database\")\n",
+ "\n",
+ " source_group = dump.add_argument_group(\"source\", \"Database source options\")\n",
+ " source_group.add_argument(\"--database\", required=True, help=\"Database name\")\n",
+ " source_group.add_argument(\"--tables\", nargs=\"*\", help=\"Specific tables to dump\")\n",
+ "\n",
+ " output_group = dump.add_argument_group(\"output\", \"Output options\")\n",
+ " fmt = output_group.add_mutually_exclusive_group()\n",
+ " fmt.add_argument(\"--sql\", action=\"store_true\", help=\"SQL format\")\n",
+ " fmt.add_argument(\"--csv\", action=\"store_true\", help=\"CSV format\")\n",
+ " output_group.add_argument(\"-o\", \"--output\", default=\"stdout\", help=\"Output file\")\n",
+ "\n",
+ " return parser\n",
+ "\n",
+ "\n",
+ "db_cli: argparse.ArgumentParser = build_db_cli()\n",
+ "db_cli.print_help()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a1000020",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Using the migrate subcommand\n",
+ "args = db_cli.parse_args([\"migrate\", \"--up\", \"--steps\", \"3\"])\n",
+ "print(f\"Command: {args.command}\")\n",
+ "print(f\"Direction: {'up' if args.up else 'down'}\")\n",
+ "print(f\"Steps: {args.steps}\")\n",
+ "\n",
+ "print()\n",
+ "\n",
+ "# Using the dump subcommand with groups\n",
+ "args = db_cli.parse_args([\n",
+ " \"-v\", \"dump\",\n",
+ " \"--database\", \"mydb\",\n",
+ " \"--tables\", \"users\", \"orders\",\n",
+ " \"--csv\",\n",
+ " \"-o\", \"export.csv\",\n",
+ "])\n",
+ "print(f\"Command: {args.command}\")\n",
+ "print(f\"Verbose: {args.verbose}\")\n",
+ "print(f\"Database: {args.database}\")\n",
+ "print(f\"Tables: {args.tables}\")\n",
+ "print(f\"CSV format: {args.csv}\")\n",
+ "print(f\"Output: {args.output}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a1000021",
+ "metadata": {},
+ "source": [
+ "## Section 7: Parent Parsers for Shared Arguments\n",
+ "\n",
+ "Parent parsers let you define common arguments once and share them across subcommands without repeating definitions."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a1000022",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Shared parent parser with common arguments\n",
+ "parent = argparse.ArgumentParser(add_help=False) # add_help=False avoids duplicate --help\n",
+ "parent.add_argument(\"--debug\", action=\"store_true\", help=\"Enable debug mode\")\n",
+ "parent.add_argument(\"--config\", default=\"config.toml\", help=\"Config file path\")\n",
+ "\n",
+ "# Main parser uses the parent\n",
+ "main_parser = argparse.ArgumentParser(prog=\"app\")\n",
+ "sub = main_parser.add_subparsers(dest=\"command\")\n",
+ "\n",
+ "# Both subcommands inherit --debug and --config from the parent\n",
+ "serve_cmd = sub.add_parser(\"serve\", parents=[parent], help=\"Start the server\")\n",
+ "serve_cmd.add_argument(\"--port\", type=int, default=8080)\n",
+ "\n",
+ "test_cmd = sub.add_parser(\"test\", parents=[parent], help=\"Run tests\")\n",
+ "test_cmd.add_argument(\"--coverage\", action=\"store_true\")\n",
+ "\n",
+ "# Both subcommands have --debug and --config\n",
+ "args = main_parser.parse_args([\"serve\", \"--debug\", \"--port\", \"3000\"])\n",
+ "print(f\"serve: debug={args.debug}, config={args.config}, port={args.port}\")\n",
+ "\n",
+ "args = main_parser.parse_args([\"test\", \"--debug\", \"--coverage\"])\n",
+ "print(f\"test: debug={args.debug}, config={args.config}, coverage={args.coverage}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a1000023",
+ "metadata": {},
+ "source": [
+ "## Summary\n",
+ "\n",
+ "### Subparsers\n",
+ "- **`add_subparsers(dest=...)`** creates a subcommand dispatcher\n",
+ "- **`add_parser(name)`** registers a new subcommand with its own arguments\n",
+ "- **`set_defaults(func=...)`** enables clean function dispatch per subcommand\n",
+ "\n",
+ "### Mutually Exclusive Groups\n",
+ "- **`add_mutually_exclusive_group()`** prevents conflicting flags from being used together\n",
+ "- Use `required=True` to force exactly one choice from the group\n",
+ "- Argparse raises `SystemExit` if conflicting options are provided\n",
+ "\n",
+ "### Argument Groups\n",
+ "- **`add_argument_group(title, description)`** organizes help output into sections\n",
+ "- Groups are purely organizational and do not restrict argument usage\n",
+ "- Combine groups within subcommands for well-structured CLI tools\n",
+ "\n",
+ "### Parent Parsers\n",
+ "- **`parents=[parent_parser]`** shares common arguments across subcommands\n",
+ "- Use `add_help=False` on parent parsers to avoid duplicate `--help` flags"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "name": "python",
+ "version": "3.13.0"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
\ No newline at end of file
diff --git a/src/chapter_28/03_environment_and_config.ipynb b/src/chapter_28/03_environment_and_config.ipynb
new file mode 100644
index 0000000..f6ecc49
--- /dev/null
+++ b/src/chapter_28/03_environment_and_config.ipynb
@@ -0,0 +1,683 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "b1000001",
+ "metadata": {},
+ "source": [
+ "# Chapter 28: Environment and Configuration\n",
+ "\n",
+ "This notebook covers how CLI tools interact with the operating system environment: reading environment variables with `os.environ`, inspecting command-line arguments via `sys.argv`, integrating configuration files, and using exit codes for process communication.\n",
+ "\n",
+ "## Key Concepts\n",
+ "- **`os.environ`**: A mapping of environment variables\n",
+ "- **`sys.argv`**: The raw list of command-line arguments\n",
+ "- **Config file integration**: Layering file-based config with CLI arguments\n",
+ "- **Exit codes**: Communicating success or failure to the calling process"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b1000002",
+ "metadata": {},
+ "source": [
+ "## Section 1: Environment Variables with os.environ\n",
+ "\n",
+ "`os.environ` is a dictionary-like object that provides access to environment variables. Many CLI tools use environment variables for configuration (e.g., `DATABASE_URL`, `API_KEY`)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b1000003",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import os\n",
+ "\n",
+ "# os.environ behaves like a dict[str, str]\n",
+ "print(f\"Type: {type(os.environ).__name__}\")\n",
+ "print(f\"HOME: {os.environ.get('HOME', 'not set')}\")\n",
+ "print(f\"PATH (first 80 chars): {os.environ.get('PATH', '')[:80]}...\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b1000004",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Setting and reading environment variables\n",
+ "os.environ[\"MY_APP_DEBUG\"] = \"true\"\n",
+ "os.environ[\"MY_APP_PORT\"] = \"8080\"\n",
+ "\n",
+ "debug_flag: str = os.environ[\"MY_APP_DEBUG\"]\n",
+ "port_str: str = os.environ[\"MY_APP_PORT\"]\n",
+ "\n",
+ "print(f\"MY_APP_DEBUG: {debug_flag!r} (type: {type(debug_flag).__name__})\")\n",
+ "print(f\"MY_APP_PORT: {port_str!r} (type: {type(port_str).__name__})\")\n",
+ "print()\n",
+ "print(\"Note: Environment variables are always strings.\")\n",
+ "print(f\"Port as int: {int(port_str)}\")\n",
+ "\n",
+ "# Clean up\n",
+ "del os.environ[\"MY_APP_DEBUG\"]\n",
+ "del os.environ[\"MY_APP_PORT\"]"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b1000005",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Safe access with .get() and defaults\n",
+ "# This is the recommended pattern for optional environment variables\n",
+ "database_url: str = os.environ.get(\"DATABASE_URL\", \"sqlite:///default.db\")\n",
+ "log_level: str = os.environ.get(\"LOG_LEVEL\", \"info\")\n",
+ "max_workers: int = int(os.environ.get(\"MAX_WORKERS\", \"4\"))\n",
+ "\n",
+ "print(f\"Database URL: {database_url}\")\n",
+ "print(f\"Log level: {log_level}\")\n",
+ "print(f\"Max workers: {max_workers} (type: {type(max_workers).__name__})\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b1000006",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Checking for required environment variables\n",
+ "def require_env(name: str) -> str:\n",
+ " \"\"\"Get a required environment variable or raise an error.\"\"\"\n",
+ " value: str | None = os.environ.get(name)\n",
+ " if value is None:\n",
+ " raise EnvironmentError(f\"Required environment variable '{name}' is not set\")\n",
+ " return value\n",
+ "\n",
+ "\n",
+ "# Set a variable, then require it\n",
+ "os.environ[\"API_KEY\"] = \"sk-test-12345\"\n",
+ "key: str = require_env(\"API_KEY\")\n",
+ "print(f\"API_KEY: {key}\")\n",
+ "\n",
+ "# Missing variable raises an error\n",
+ "try:\n",
+ " require_env(\"MISSING_SECRET\")\n",
+ "except EnvironmentError as e:\n",
+ " print(f\"Error: {e}\")\n",
+ "\n",
+ "# Clean up\n",
+ "del os.environ[\"API_KEY\"]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b1000007",
+ "metadata": {},
+ "source": [
+ "## Section 2: sys.argv — Raw Command-Line Arguments\n",
+ "\n",
+ "`sys.argv` is a plain list of strings representing the command-line arguments. `sys.argv[0]` is the script name, and `sys.argv[1:]` are the user-provided arguments. While `argparse` is preferred for real CLI tools, `sys.argv` is useful for understanding how argument parsing works underneath."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b1000008",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import sys\n",
+ "\n",
+ "# sys.argv is always a list with at least one element\n",
+ "print(f\"Type: {type(sys.argv).__name__}\")\n",
+ "print(f\"Length: {len(sys.argv)}\")\n",
+ "print(f\"Script name (argv[0]): {sys.argv[0]}\")\n",
+ "print(f\"Is list: {isinstance(sys.argv, list)}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b1000009",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Simulating manual argument parsing (for educational purposes)\n",
+ "# In real code, always use argparse instead\n",
+ "\n",
+ "def manual_parse(argv: list[str]) -> dict[str, str | bool]:\n",
+ " \"\"\"Parse arguments manually from a list of strings.\"\"\"\n",
+ " result: dict[str, str | bool] = {}\n",
+ " i: int = 0\n",
+ " while i < len(argv):\n",
+ " arg: str = argv[i]\n",
+ " if arg.startswith(\"--\"):\n",
+ " key: str = arg[2:].replace(\"-\", \"_\")\n",
+ " # Check if next arg is a value or another flag\n",
+ " if i + 1 < len(argv) and not argv[i + 1].startswith(\"--\"):\n",
+ " result[key] = argv[i + 1]\n",
+ " i += 2\n",
+ " else:\n",
+ " result[key] = True\n",
+ " i += 1\n",
+ " else:\n",
+ " result.setdefault(\"positional\", [])\n",
+ " result[\"positional\"].append(arg) # type: ignore[union-attr]\n",
+ " i += 1\n",
+ " return result\n",
+ "\n",
+ "\n",
+ "# Simulate parsing\n",
+ "fake_argv: list[str] = [\"app.py\", \"input.txt\", \"--verbose\", \"--output\", \"result.csv\"]\n",
+ "parsed: dict[str, str | bool] = manual_parse(fake_argv[1:]) # Skip script name\n",
+ "\n",
+ "print(f\"Input: {fake_argv}\")\n",
+ "print(f\"Parsed: {parsed}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b1000010",
+ "metadata": {},
+ "source": [
+ "## Section 3: Combining Environment Variables with argparse\n",
+ "\n",
+ "A common pattern is to check environment variables as fallback defaults for CLI arguments. This allows configuration via both the command line and environment."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b1000011",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import argparse\n",
+ "\n",
+ "\n",
+ "def build_env_aware_parser() -> argparse.ArgumentParser:\n",
+ " \"\"\"Build a parser that uses environment variables as defaults.\"\"\"\n",
+ " parser = argparse.ArgumentParser(prog=\"webapp\")\n",
+ "\n",
+ " # CLI args override env vars, which override hardcoded defaults\n",
+ " parser.add_argument(\n",
+ " \"--host\",\n",
+ " default=os.environ.get(\"APP_HOST\", \"127.0.0.1\"),\n",
+ " help=\"Server host (env: APP_HOST, default: %(default)s)\",\n",
+ " )\n",
+ " parser.add_argument(\n",
+ " \"--port\",\n",
+ " type=int,\n",
+ " default=int(os.environ.get(\"APP_PORT\", \"8000\")),\n",
+ " help=\"Server port (env: APP_PORT, default: %(default)s)\",\n",
+ " )\n",
+ " parser.add_argument(\n",
+ " \"--debug\",\n",
+ " action=\"store_true\",\n",
+ " default=os.environ.get(\"APP_DEBUG\", \"\").lower() in (\"1\", \"true\", \"yes\"),\n",
+ " help=\"Enable debug mode (env: APP_DEBUG)\",\n",
+ " )\n",
+ "\n",
+ " return parser\n",
+ "\n",
+ "\n",
+ "# Without env vars: uses hardcoded defaults\n",
+ "parser = build_env_aware_parser()\n",
+ "args = parser.parse_args([])\n",
+ "print(f\"Without env vars: host={args.host}, port={args.port}, debug={args.debug}\")\n",
+ "\n",
+ "# With env vars: uses environment as defaults\n",
+ "os.environ[\"APP_HOST\"] = \"0.0.0.0\"\n",
+ "os.environ[\"APP_PORT\"] = \"3000\"\n",
+ "os.environ[\"APP_DEBUG\"] = \"true\"\n",
+ "\n",
+ "parser = build_env_aware_parser()\n",
+ "args = parser.parse_args([])\n",
+ "print(f\"With env vars: host={args.host}, port={args.port}, debug={args.debug}\")\n",
+ "\n",
+ "# CLI args still override environment\n",
+ "args = parser.parse_args([\"--host\", \"localhost\", \"--port\", \"9000\"])\n",
+ "print(f\"CLI override: host={args.host}, port={args.port}, debug={args.debug}\")\n",
+ "\n",
+ "# Clean up\n",
+ "del os.environ[\"APP_HOST\"]\n",
+ "del os.environ[\"APP_PORT\"]\n",
+ "del os.environ[\"APP_DEBUG\"]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b1000012",
+ "metadata": {},
+ "source": [
+ "## Section 4: Config File Integration\n",
+ "\n",
+ "Many CLI tools support configuration files (TOML, JSON, INI). A robust tool layers configuration with this priority:\n",
+ "1. Command-line arguments (highest priority)\n",
+ "2. Environment variables\n",
+ "3. Config file values\n",
+ "4. Hardcoded defaults (lowest priority)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b1000013",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import json\n",
+ "import tempfile\n",
+ "from pathlib import Path\n",
+ "\n",
+ "\n",
+ "def load_config_file(path: Path) -> dict[str, object]:\n",
+ " \"\"\"Load configuration from a JSON file.\"\"\"\n",
+ " if not path.exists():\n",
+ " return {}\n",
+ " with path.open() as f:\n",
+ " config: dict[str, object] = json.load(f)\n",
+ " return config\n",
+ "\n",
+ "\n",
+ "def merge_config(\n",
+ " defaults: dict[str, object],\n",
+ " file_config: dict[str, object],\n",
+ " env_config: dict[str, object],\n",
+ " cli_args: dict[str, object],\n",
+ ") -> dict[str, object]:\n",
+ " \"\"\"Merge configuration sources with proper priority.\"\"\"\n",
+ " merged: dict[str, object] = {}\n",
+ " merged.update(defaults)\n",
+ " merged.update({k: v for k, v in file_config.items() if v is not None})\n",
+ " merged.update({k: v for k, v in env_config.items() if v is not None})\n",
+ " merged.update({k: v for k, v in cli_args.items() if v is not None})\n",
+ " return merged\n",
+ "\n",
+ "\n",
+ "# Create a temporary config file\n",
+ "config_data: dict[str, object] = {\n",
+ " \"host\": \"config-host.example.com\",\n",
+ " \"port\": 5000,\n",
+ " \"debug\": False,\n",
+ " \"log_level\": \"warning\",\n",
+ "}\n",
+ "\n",
+ "with tempfile.NamedTemporaryFile(mode=\"w\", suffix=\".json\", delete=False) as f:\n",
+ " json.dump(config_data, f)\n",
+ " config_path: Path = Path(f.name)\n",
+ "\n",
+ "# Load and display\n",
+ "file_config: dict[str, object] = load_config_file(config_path)\n",
+ "print(f\"Config file: {config_path}\")\n",
+ "print(f\"File config: {json.dumps(file_config, indent=2)}\")\n",
+ "\n",
+ "# Clean up temp file\n",
+ "config_path.unlink()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b1000014",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Demonstrate the full configuration layering\n",
+ "defaults: dict[str, object] = {\n",
+ " \"host\": \"127.0.0.1\",\n",
+ " \"port\": 8000,\n",
+ " \"debug\": False,\n",
+ " \"log_level\": \"info\",\n",
+ "}\n",
+ "\n",
+ "file_config = {\n",
+ " \"host\": \"config-host.example.com\",\n",
+ " \"port\": 5000,\n",
+ "}\n",
+ "\n",
+ "env_config: dict[str, object] = {\n",
+ " \"port\": 3000, # Overrides file config\n",
+ "}\n",
+ "\n",
+ "cli_args_config: dict[str, object] = {\n",
+ " \"debug\": True, # Overrides everything\n",
+ "}\n",
+ "\n",
+ "final: dict[str, object] = merge_config(defaults, file_config, env_config, cli_args_config)\n",
+ "\n",
+ "print(\"Configuration layering:\")\n",
+ "print(f\" Defaults: {defaults}\")\n",
+ "print(f\" File config: {file_config}\")\n",
+ "print(f\" Env config: {env_config}\")\n",
+ "print(f\" CLI args: {cli_args_config}\")\n",
+ "print(f\" Final: {final}\")\n",
+ "print()\n",
+ "print(f\" host comes from file_config: {final['host']}\")\n",
+ "print(f\" port comes from env_config: {final['port']}\")\n",
+ "print(f\" debug comes from cli_args: {final['debug']}\")\n",
+ "print(f\" log_level comes from defaults: {final['log_level']}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b1000015",
+ "metadata": {},
+ "source": [
+ "## Section 5: Exit Codes\n",
+ "\n",
+ "Exit codes communicate a program's result to the calling process (shell, CI pipeline, etc.).\n",
+ "- `0` means success\n",
+ "- Non-zero means failure (by convention, `1` is a general error, `2` is usage error)\n",
+ "\n",
+ "`sys.exit()` raises a `SystemExit` exception, which can be caught for testing."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b1000016",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import sys\n",
+ "\n",
+ "# sys.exit raises SystemExit, which can be caught\n",
+ "try:\n",
+ " sys.exit(0)\n",
+ "except SystemExit as e:\n",
+ " print(f\"Exit code: {e.code} (success)\")\n",
+ "\n",
+ "try:\n",
+ " sys.exit(1)\n",
+ "except SystemExit as e:\n",
+ " print(f\"Exit code: {e.code} (general error)\")\n",
+ "\n",
+ "try:\n",
+ " sys.exit(2)\n",
+ "except SystemExit as e:\n",
+ " print(f\"Exit code: {e.code} (usage error)\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b1000017",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# sys.exit can also accept a string message (implies exit code 1)\n",
+ "try:\n",
+ " sys.exit(\"Fatal error: configuration file not found\")\n",
+ "except SystemExit as e:\n",
+ " print(f\"Exit message: {e.code}\")\n",
+ " print(f\"Type of code: {type(e.code).__name__}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b1000018",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Common pattern: a main function that returns an exit code\n",
+ "from enum import IntEnum\n",
+ "\n",
+ "\n",
+ "class ExitCode(IntEnum):\n",
+ " \"\"\"Standard exit codes for the application.\"\"\"\n",
+ " SUCCESS = 0\n",
+ " GENERAL_ERROR = 1\n",
+ " USAGE_ERROR = 2\n",
+ " CONFIG_ERROR = 3\n",
+ " IO_ERROR = 4\n",
+ "\n",
+ "\n",
+ "def main(args: list[str]) -> ExitCode:\n",
+ " \"\"\"Application entry point that returns an exit code.\"\"\"\n",
+ " if not args:\n",
+ " print(\"Error: no arguments provided\", file=sys.stderr)\n",
+ " return ExitCode.USAGE_ERROR\n",
+ "\n",
+ " filename: str = args[0]\n",
+ " if not filename.endswith(\".txt\"):\n",
+ " print(f\"Error: unsupported file type: {filename}\", file=sys.stderr)\n",
+ " return ExitCode.GENERAL_ERROR\n",
+ "\n",
+ " print(f\"Processing {filename}...\")\n",
+ " return ExitCode.SUCCESS\n",
+ "\n",
+ "\n",
+ "# Simulate different invocations\n",
+ "for test_args in [[], [\"data.csv\"], [\"report.txt\"]]:\n",
+ " code: ExitCode = main(test_args)\n",
+ " print(f\" args={test_args} -> exit code {code} ({code.name})\\n\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b1000019",
+ "metadata": {},
+ "source": [
+ "## Section 6: Putting It All Together\n",
+ "\n",
+ "A complete example of a CLI tool that combines argparse, environment variables, config file loading, and proper exit codes."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b1000020",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import argparse\n",
+ "import json\n",
+ "import os\n",
+ "import sys\n",
+ "import tempfile\n",
+ "from pathlib import Path\n",
+ "\n",
+ "\n",
+ "def build_full_parser() -> argparse.ArgumentParser:\n",
+ " \"\"\"Build a parser for a complete CLI tool.\"\"\"\n",
+ " parser = argparse.ArgumentParser(\n",
+ " prog=\"appctl\",\n",
+ " description=\"Application control tool with layered configuration\",\n",
+ " )\n",
+ " parser.add_argument(\n",
+ " \"--config\",\n",
+ " type=Path,\n",
+ " default=Path(os.environ.get(\"APPCTL_CONFIG\", \"config.json\")),\n",
+ " help=\"Config file path (env: APPCTL_CONFIG)\",\n",
+ " )\n",
+ " parser.add_argument(\n",
+ " \"--host\",\n",
+ " default=os.environ.get(\"APPCTL_HOST\"),\n",
+ " help=\"Server host (env: APPCTL_HOST)\",\n",
+ " )\n",
+ " parser.add_argument(\n",
+ " \"--port\",\n",
+ " type=int,\n",
+ " default=int(os.environ.get(\"APPCTL_PORT\", \"0\")) or None,\n",
+ " help=\"Server port (env: APPCTL_PORT)\",\n",
+ " )\n",
+ " parser.add_argument(\n",
+ " \"-v\", \"--verbose\",\n",
+ " action=\"store_true\",\n",
+ " help=\"Enable verbose output\",\n",
+ " )\n",
+ " return parser\n",
+ "\n",
+ "\n",
+ "def run_app(argv: list[str] | None = None) -> int:\n",
+ " \"\"\"Run the application with full configuration layering.\"\"\"\n",
+ " parser: argparse.ArgumentParser = build_full_parser()\n",
+ " args: argparse.Namespace = parser.parse_args(argv)\n",
+ "\n",
+ " # Start with defaults\n",
+ " config: dict[str, object] = {\n",
+ " \"host\": \"127.0.0.1\",\n",
+ " \"port\": 8000,\n",
+ " \"verbose\": False,\n",
+ " }\n",
+ "\n",
+ " # Layer file config\n",
+ " if args.config.exists():\n",
+ " with args.config.open() as f:\n",
+ " file_cfg: dict[str, object] = json.load(f)\n",
+ " config.update(file_cfg)\n",
+ " if args.verbose:\n",
+ " print(f\"Loaded config from {args.config}\")\n",
+ "\n",
+ " # Layer CLI args (only if explicitly provided)\n",
+ " if args.host is not None:\n",
+ " config[\"host\"] = args.host\n",
+ " if args.port is not None:\n",
+ " config[\"port\"] = args.port\n",
+ " if args.verbose:\n",
+ " config[\"verbose\"] = True\n",
+ "\n",
+ " print(f\"Final config: {json.dumps(config, indent=2, default=str)}\")\n",
+ " return 0\n",
+ "\n",
+ "\n",
+ "# Create a temp config file for demonstration\n",
+ "demo_config: dict[str, object] = {\"host\": \"0.0.0.0\", \"port\": 5000}\n",
+ "with tempfile.NamedTemporaryFile(\n",
+ " mode=\"w\", suffix=\".json\", delete=False\n",
+ ") as f:\n",
+ " json.dump(demo_config, f)\n",
+ " tmp_config_path: str = f.name\n",
+ "\n",
+ "# Run with config file and CLI override\n",
+ "print(\"--- With config file and CLI port override ---\")\n",
+ "exit_code: int = run_app([\n",
+ " \"--config\", tmp_config_path,\n",
+ " \"--port\", \"9090\",\n",
+ " \"--verbose\",\n",
+ "])\n",
+ "print(f\"Exit code: {exit_code}\")\n",
+ "\n",
+ "# Clean up\n",
+ "Path(tmp_config_path).unlink()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b1000021",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Run with only defaults (no config file, no env vars, no CLI args)\n",
+ "print(\"--- With only defaults ---\")\n",
+ "exit_code = run_app([])\n",
+ "print(f\"Exit code: {exit_code}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b1000022",
+ "metadata": {},
+ "source": [
+ "## Section 7: Best Practices for CLI Configuration\n",
+ "\n",
+ "When building production CLI tools, follow these conventions for a good user experience."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b1000023",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Best practice: use a dataclass to hold validated configuration\n",
+ "from dataclasses import dataclass\n",
+ "\n",
+ "\n",
+ "@dataclass(frozen=True)\n",
+ "class AppConfig:\n",
+ " \"\"\"Validated, immutable application configuration.\"\"\"\n",
+ " host: str\n",
+ " port: int\n",
+ " debug: bool\n",
+ " log_level: str\n",
+ "\n",
+ " def __post_init__(self) -> None:\n",
+ " \"\"\"Validate configuration values.\"\"\"\n",
+ " if not 1 <= self.port <= 65535:\n",
+ " raise ValueError(f\"Port must be 1-65535, got {self.port}\")\n",
+ " valid_levels: set[str] = {\"debug\", \"info\", \"warning\", \"error\", \"critical\"}\n",
+ " if self.log_level not in valid_levels:\n",
+ " raise ValueError(f\"Invalid log level: {self.log_level}\")\n",
+ "\n",
+ "\n",
+ "# Valid configuration\n",
+ "config = AppConfig(host=\"0.0.0.0\", port=8080, debug=True, log_level=\"debug\")\n",
+ "print(f\"Valid config: {config}\")\n",
+ "\n",
+ "# Invalid port is caught immediately\n",
+ "try:\n",
+ " bad_config = AppConfig(host=\"localhost\", port=99999, debug=False, log_level=\"info\")\n",
+ "except ValueError as e:\n",
+ " print(f\"Validation error: {e}\")\n",
+ "\n",
+ "# Invalid log level\n",
+ "try:\n",
+ " bad_config = AppConfig(host=\"localhost\", port=8080, debug=False, log_level=\"verbose\")\n",
+ "except ValueError as e:\n",
+ " print(f\"Validation error: {e}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b1000024",
+ "metadata": {},
+ "source": [
+ "## Summary\n",
+ "\n",
+ "### os.environ\n",
+ "- **`os.environ[key]`** reads a variable (raises `KeyError` if missing)\n",
+ "- **`os.environ.get(key, default)`** reads safely with a fallback\n",
+ "- Environment variables are always strings; cast them explicitly\n",
+ "\n",
+ "### sys.argv\n",
+ "- **`sys.argv`** is a `list[str]` with the script name at index 0\n",
+ "- Use `argparse` instead of parsing `sys.argv` manually\n",
+ "\n",
+ "### Config File Integration\n",
+ "- Layer configuration with this priority: **CLI args > env vars > config file > defaults**\n",
+ "- Use `argparse` defaults that read from `os.environ.get(...)` for env var integration\n",
+ "- Validate final configuration with a dataclass or similar structure\n",
+ "\n",
+ "### Exit Codes\n",
+ "- **`sys.exit(0)`** signals success; non-zero signals failure\n",
+ "- **`sys.exit()`** raises `SystemExit`, which can be caught in tests\n",
+ "- Define exit codes as an `IntEnum` for clarity and consistency\n",
+ "- Return exit codes from `main()` instead of calling `sys.exit()` directly for testability"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "name": "python",
+ "version": "3.13.0"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
\ No newline at end of file
diff --git a/src/chapter_28/README.md b/src/chapter_28/README.md
new file mode 100644
index 0000000..b6a72f3
--- /dev/null
+++ b/src/chapter_28/README.md
@@ -0,0 +1,9 @@
+# Chapter 28: Command-Line Interfaces
+
+Building command-line tools with `argparse` — positional and optional arguments, subcommands, type validation, environment variables, and `sys.argv`.
+
+## Notebooks
+
+1. **01_argparse_fundamentals.ipynb** — `ArgumentParser`, positional/optional args, types, choices, defaults
+2. **02_subcommands_and_groups.ipynb** — Subparsers, mutually exclusive groups, argument groups
+3. **03_environment_and_config.ipynb** — `os.environ`, `sys.argv`, config file integration, exit codes
diff --git a/src/chapter_28/__init__.py b/src/chapter_28/__init__.py
new file mode 100644
index 0000000..4c080b7
--- /dev/null
+++ b/src/chapter_28/__init__.py
@@ -0,0 +1 @@
+"""Chapter 28: Command-Line Interfaces."""
diff --git a/src/chapter_29/01_datetime_fundamentals.ipynb b/src/chapter_29/01_datetime_fundamentals.ipynb
new file mode 100644
index 0000000..76cc310
--- /dev/null
+++ b/src/chapter_29/01_datetime_fundamentals.ipynb
@@ -0,0 +1,498 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "a1b2c3d4",
+ "metadata": {},
+ "source": [
+ "# Chapter 29: Datetime Fundamentals\n",
+ "\n",
+ "This notebook covers Python's core date and time types from the `datetime` module. You will learn how to create, inspect, combine, and perform arithmetic on dates and times using `date`, `time`, `datetime`, and `timedelta`.\n",
+ "\n",
+ "## Key Concepts\n",
+ "- **`date`**: Represents a calendar date (year, month, day)\n",
+ "- **`time`**: Represents a time of day (hour, minute, second, microsecond)\n",
+ "- **`datetime`**: Combines date and time into a single object\n",
+ "- **`timedelta`**: Represents a duration or difference between two dates/times\n",
+ "- **Comparison operators**: Dates and datetimes support ordering"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b1c2d3e4",
+ "metadata": {},
+ "source": [
+ "## Section 1: Creating Date Objects\n",
+ "\n",
+ "A `date` object represents a calendar date. It stores the year, month, and day as separate attributes."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "c1d2e3f4",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from datetime import date\n",
+ "\n",
+ "# Create a date from year, month, day\n",
+ "d: date = date(2025, 6, 15)\n",
+ "\n",
+ "print(f\"Date: {d}\")\n",
+ "print(f\"Year: {d.year}\")\n",
+ "print(f\"Month: {d.month}\")\n",
+ "print(f\"Day: {d.day}\")\n",
+ "\n",
+ "# Get today's date\n",
+ "today: date = date.today()\n",
+ "print(f\"\\nToday: {today}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "d1e2f3a4",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Useful date methods\n",
+ "d: date = date(2025, 6, 15)\n",
+ "\n",
+ "# weekday() returns 0=Monday ... 6=Sunday\n",
+ "print(f\"Weekday (0=Mon): {d.weekday()}\")\n",
+ "\n",
+ "# isoweekday() returns 1=Monday ... 7=Sunday\n",
+ "print(f\"ISO weekday (1=Mon): {d.isoweekday()}\")\n",
+ "\n",
+ "# ISO calendar returns (year, week_number, weekday)\n",
+ "iso_year, iso_week, iso_day = d.isocalendar()\n",
+ "print(f\"ISO calendar: year={iso_year}, week={iso_week}, day={iso_day}\")\n",
+ "\n",
+ "# Create from ordinal (days since Jan 1, year 1)\n",
+ "ordinal: int = d.toordinal()\n",
+ "print(f\"\\nOrdinal of {d}: {ordinal}\")\n",
+ "print(f\"Back to date: {date.fromordinal(ordinal)}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e1f2a3b4",
+ "metadata": {},
+ "source": [
+ "## Section 2: Creating Time Objects\n",
+ "\n",
+ "A `time` object represents a time of day independent of any date. It stores hours, minutes, seconds, and microseconds."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f1a2b3c4",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from datetime import time\n",
+ "\n",
+ "# Create a time object\n",
+ "t: time = time(14, 30, 45)\n",
+ "\n",
+ "print(f\"Time: {t}\")\n",
+ "print(f\"Hour: {t.hour}\")\n",
+ "print(f\"Minute: {t.minute}\")\n",
+ "print(f\"Second: {t.second}\")\n",
+ "\n",
+ "# Time with microseconds\n",
+ "precise: time = time(14, 30, 45, 123456)\n",
+ "print(f\"\\nPrecise time: {precise}\")\n",
+ "print(f\"Microsecond: {precise.microsecond}\")\n",
+ "\n",
+ "# Minimum and maximum times\n",
+ "print(f\"\\nMin time: {time.min}\")\n",
+ "print(f\"Max time: {time.max}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a2b3c4d5",
+ "metadata": {},
+ "source": [
+ "## Section 3: The `datetime` Object\n",
+ "\n",
+ "A `datetime` combines a `date` and a `time` into a single object. It is the most commonly used type for working with timestamps."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b2c3d4e5",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from datetime import datetime, date, time\n",
+ "\n",
+ "# Create a datetime\n",
+ "dt: datetime = datetime(2025, 6, 15, 14, 30)\n",
+ "\n",
+ "print(f\"Datetime: {dt}\")\n",
+ "print(f\"Year: {dt.year}\")\n",
+ "print(f\"Month: {dt.month}\")\n",
+ "print(f\"Day: {dt.day}\")\n",
+ "print(f\"Hour: {dt.hour}\")\n",
+ "print(f\"Minute: {dt.minute}\")\n",
+ "\n",
+ "# Extract date and time parts\n",
+ "d: date = dt.date()\n",
+ "t: time = dt.time()\n",
+ "print(f\"\\nDate part: {d}\")\n",
+ "print(f\"Time part: {t}\")\n",
+ "print(f\"date() == date(2025, 6, 15): {d == date(2025, 6, 15)}\")\n",
+ "print(f\"time() == time(14, 30): {t == time(14, 30)}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "c2d3e4f5",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from datetime import datetime, date, time\n",
+ "\n",
+ "# Combine a date and a time into a datetime\n",
+ "d: date = date(2025, 6, 15)\n",
+ "t: time = time(14, 30, 0)\n",
+ "combined: datetime = datetime.combine(d, t)\n",
+ "\n",
+ "print(f\"Date: {d}\")\n",
+ "print(f\"Time: {t}\")\n",
+ "print(f\"Combined: {combined}\")\n",
+ "\n",
+ "# Get the current datetime\n",
+ "now: datetime = datetime.now()\n",
+ "print(f\"\\nNow: {now}\")\n",
+ "\n",
+ "# Create from a POSIX timestamp (seconds since 1970-01-01 UTC)\n",
+ "from_ts: datetime = datetime.fromtimestamp(1750000000)\n",
+ "print(f\"\\nFrom timestamp 1750000000: {from_ts}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d2e3f4a5",
+ "metadata": {},
+ "source": [
+ "## Section 4: Replacing Components\n",
+ "\n",
+ "Date and datetime objects are immutable. The `replace()` method creates a new object with one or more fields changed."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "e2f3a4b5",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from datetime import datetime\n",
+ "\n",
+ "dt: datetime = datetime(2025, 6, 15, 14, 30)\n",
+ "\n",
+ "# Replace creates a new datetime with specified fields changed\n",
+ "new_year: datetime = dt.replace(year=2026)\n",
+ "new_month: datetime = dt.replace(month=12, day=25)\n",
+ "new_time: datetime = dt.replace(hour=9, minute=0)\n",
+ "\n",
+ "print(f\"Original: {dt}\")\n",
+ "print(f\"New year: {new_year}\")\n",
+ "print(f\"New month: {new_month}\")\n",
+ "print(f\"New time: {new_time}\")\n",
+ "\n",
+ "# Original is unchanged (immutable)\n",
+ "print(f\"\\nOriginal still: {dt}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "f2a3b4c5",
+ "metadata": {},
+ "source": [
+ "## Section 5: Timedelta -- Durations and Differences\n",
+ "\n",
+ "A `timedelta` represents a duration. It can express differences between dates or datetimes and supports arithmetic operations."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a3b4c5d6",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from datetime import timedelta\n",
+ "\n",
+ "# Create timedeltas\n",
+ "one_day: timedelta = timedelta(days=1)\n",
+ "one_week: timedelta = timedelta(weeks=1)\n",
+ "complex_delta: timedelta = timedelta(days=5, hours=3, minutes=30)\n",
+ "\n",
+ "print(f\"One day: {one_day}\")\n",
+ "print(f\"One week: {one_week}\")\n",
+ "print(f\"Complex: {complex_delta}\")\n",
+ "\n",
+ "# Internally, timedelta stores only days, seconds, and microseconds\n",
+ "td: timedelta = timedelta(hours=25, minutes=30)\n",
+ "print(f\"\\ntimedelta(hours=25, minutes=30):\")\n",
+ "print(f\" days: {td.days}\")\n",
+ "print(f\" seconds: {td.seconds}\") # 1 hour 30 min = 5400 seconds\n",
+ "print(f\" microseconds: {td.microseconds}\")\n",
+ "print(f\" total_seconds: {td.total_seconds()}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b3c4d5e6",
+ "metadata": {},
+ "source": [
+ "## Section 6: Date Arithmetic\n",
+ "\n",
+ "You can add or subtract `timedelta` objects to/from `date` and `datetime` objects. Subtracting two dates produces a `timedelta`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "c3d4e5f6",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from datetime import date, timedelta\n",
+ "\n",
+ "# Adding timedelta to a date\n",
+ "d1: date = date(2025, 1, 1)\n",
+ "d2: date = d1 + timedelta(days=30)\n",
+ "\n",
+ "print(f\"Start: {d1}\")\n",
+ "print(f\"+ 30 days: {d2}\")\n",
+ "print(f\"d2 == Jan 31: {d2 == date(2025, 1, 31)}\")\n",
+ "\n",
+ "# Subtracting dates gives a timedelta\n",
+ "diff: timedelta = date(2025, 3, 1) - date(2025, 1, 1)\n",
+ "print(f\"\\nMar 1 - Jan 1 = {diff.days} days\")\n",
+ "\n",
+ "# Subtracting a timedelta\n",
+ "yesterday: date = date.today() - timedelta(days=1)\n",
+ "print(f\"\\nYesterday: {yesterday}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "d3e4f5a6",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from datetime import datetime, timedelta\n",
+ "\n",
+ "# Datetime arithmetic works the same way\n",
+ "meeting: datetime = datetime(2025, 6, 15, 14, 30)\n",
+ "reminder: datetime = meeting - timedelta(hours=1)\n",
+ "follow_up: datetime = meeting + timedelta(days=7)\n",
+ "\n",
+ "print(f\"Meeting: {meeting}\")\n",
+ "print(f\"Reminder: {reminder}\")\n",
+ "print(f\"Follow-up: {follow_up}\")\n",
+ "\n",
+ "# Timedelta arithmetic\n",
+ "half_day: timedelta = timedelta(hours=12)\n",
+ "full_day: timedelta = half_day * 2\n",
+ "quarter_day: timedelta = half_day / 2\n",
+ "\n",
+ "print(f\"\\nHalf day: {half_day}\")\n",
+ "print(f\"Full day: {full_day}\")\n",
+ "print(f\"Quarter day: {quarter_day}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e3f4a5b6",
+ "metadata": {},
+ "source": [
+ "## Section 7: Comparing Dates and Datetimes\n",
+ "\n",
+ "Date and datetime objects support all comparison operators. This is essential for sorting, filtering, and range checks."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f3a4b5c6",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from datetime import date, datetime\n",
+ "\n",
+ "# Date comparison\n",
+ "jan: date = date(2025, 1, 1)\n",
+ "jun: date = date(2025, 6, 1)\n",
+ "jul: date = date(2025, 7, 1)\n",
+ "dec: date = date(2025, 12, 31)\n",
+ "\n",
+ "print(f\"Jun 1 < Jul 1: {jun < jul}\")\n",
+ "print(f\"Dec 31 > Jan 1: {dec > jan}\")\n",
+ "print(f\"Jun 1 == Jun 1: {jun == date(2025, 6, 1)}\")\n",
+ "\n",
+ "# Sorting dates\n",
+ "dates: list[date] = [dec, jan, jul, jun]\n",
+ "sorted_dates: list[date] = sorted(dates)\n",
+ "print(f\"\\nSorted: {sorted_dates}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a4b5c6d7",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from datetime import date, timedelta\n",
+ "\n",
+ "# Practical example: checking if a date is within a range\n",
+ "def is_within_range(\n",
+ " target: date,\n",
+ " start: date,\n",
+ " end: date,\n",
+ ") -> bool:\n",
+ " \"\"\"Check if target date falls within [start, end] inclusive.\"\"\"\n",
+ " return start <= target <= end\n",
+ "\n",
+ "today: date = date(2025, 6, 15)\n",
+ "week_start: date = today - timedelta(days=today.weekday()) # Monday\n",
+ "week_end: date = week_start + timedelta(days=6) # Sunday\n",
+ "\n",
+ "print(f\"Week: {week_start} to {week_end}\")\n",
+ "print(f\"{today} in range: {is_within_range(today, week_start, week_end)}\")\n",
+ "print(f\"{date(2025, 6, 1)} in range: {is_within_range(date(2025, 6, 1), week_start, week_end)}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b4c5d6e7",
+ "metadata": {},
+ "source": [
+ "## Section 8: Practical Patterns\n",
+ "\n",
+ "Common patterns you will encounter when working with dates and times in real applications."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "c4d5e6f7",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from datetime import date, timedelta\n",
+ "\n",
+ "# Calculate age in years\n",
+ "def calculate_age(birth_date: date, reference_date: date) -> int:\n",
+ " \"\"\"Calculate age in complete years.\"\"\"\n",
+ " age: int = reference_date.year - birth_date.year\n",
+ " # Subtract 1 if birthday hasn't occurred yet this year\n",
+ " if (reference_date.month, reference_date.day) < (birth_date.month, birth_date.day):\n",
+ " age -= 1\n",
+ " return age\n",
+ "\n",
+ "birthday: date = date(1990, 8, 20)\n",
+ "today: date = date(2025, 6, 15)\n",
+ "\n",
+ "print(f\"Born: {birthday}\")\n",
+ "print(f\"Today: {today}\")\n",
+ "print(f\"Age: {calculate_age(birthday, today)} years\")\n",
+ "\n",
+ "# Days until next birthday\n",
+ "next_birthday: date = birthday.replace(year=today.year)\n",
+ "if next_birthday < today:\n",
+ " next_birthday = next_birthday.replace(year=today.year + 1)\n",
+ "days_until: int = (next_birthday - today).days\n",
+ "print(f\"\\nNext birthday: {next_birthday}\")\n",
+ "print(f\"Days until: {days_until}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "d4e5f6a7",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from datetime import date, timedelta\n",
+ "\n",
+ "# Generate a sequence of dates\n",
+ "def date_range(\n",
+ " start: date,\n",
+ " end: date,\n",
+ " step: timedelta = timedelta(days=1),\n",
+ ") -> list[date]:\n",
+ " \"\"\"Generate dates from start to end (exclusive) with given step.\"\"\"\n",
+ " dates: list[date] = []\n",
+ " current: date = start\n",
+ " while current < end:\n",
+ " dates.append(current)\n",
+ " current += step\n",
+ " return dates\n",
+ "\n",
+ "# Daily dates for a week\n",
+ "week: list[date] = date_range(date(2025, 6, 9), date(2025, 6, 16))\n",
+ "for d in week:\n",
+ " day_name: str = d.strftime(\"%A\")\n",
+ " print(f\" {d} ({day_name})\")\n",
+ "\n",
+ "# Every other day\n",
+ "print(\"\\nEvery other day:\")\n",
+ "for d in date_range(date(2025, 6, 1), date(2025, 6, 10), timedelta(days=2)):\n",
+ " print(f\" {d}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e4f5a6b7",
+ "metadata": {},
+ "source": [
+ "## Summary\n",
+ "\n",
+ "### Core Types\n",
+ "- **`date(year, month, day)`**: Calendar date with `.year`, `.month`, `.day` attributes\n",
+ "- **`time(hour, minute, second)`**: Time of day with `.hour`, `.minute`, `.second` attributes\n",
+ "- **`datetime(year, month, day, hour, minute)`**: Combined date and time\n",
+ "- **`timedelta(days, hours, minutes, ...)`**: Duration between two points in time\n",
+ "\n",
+ "### Key Operations\n",
+ "- **Arithmetic**: `date + timedelta`, `datetime - datetime` produces `timedelta`\n",
+ "- **Comparison**: `<`, `>`, `==`, `<=`, `>=` all work on dates and datetimes\n",
+ "- **Replace**: `dt.replace(year=2026)` creates a new object with one field changed\n",
+ "- **Combine**: `datetime.combine(date, time)` merges separate date and time\n",
+ "- **Extract**: `dt.date()` and `dt.time()` pull out components\n",
+ "\n",
+ "### Important Notes\n",
+ "- `timedelta` internally stores only `days`, `seconds`, and `microseconds`\n",
+ "- `total_seconds()` returns the full duration as a float\n",
+ "- All date/time objects are **immutable** -- operations return new objects\n",
+ "- Use `date.today()` and `datetime.now()` for current date/time"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "name": "python",
+ "version": "3.13.0"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
\ No newline at end of file
diff --git a/src/chapter_29/02_timezones_and_zoneinfo.ipynb b/src/chapter_29/02_timezones_and_zoneinfo.ipynb
new file mode 100644
index 0000000..aa9a4be
--- /dev/null
+++ b/src/chapter_29/02_timezones_and_zoneinfo.ipynb
@@ -0,0 +1,444 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "a1000001",
+ "metadata": {},
+ "source": [
+ "# Chapter 29: Timezones and ZoneInfo\n",
+ "\n",
+ "This notebook covers timezone handling in Python using `timezone`, `ZoneInfo`, and the distinction between naive and aware datetimes. Correct timezone handling is critical for applications that operate across regions.\n",
+ "\n",
+ "## Key Concepts\n",
+ "- **Naive datetimes**: No timezone information attached\n",
+ "- **Aware datetimes**: Include timezone information via `tzinfo`\n",
+ "- **`timezone.utc`**: The built-in UTC timezone constant\n",
+ "- **`ZoneInfo`**: Access to the IANA timezone database (Python 3.9+)\n",
+ "- **`astimezone()`**: Convert between timezones"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a1000002",
+ "metadata": {},
+ "source": [
+ "## Section 1: Naive vs Aware Datetimes\n",
+ "\n",
+ "A **naive** datetime has no timezone info (`tzinfo is None`). An **aware** datetime carries explicit timezone information. Mixing them causes errors, so it is important to be consistent."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a1000003",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from datetime import datetime, timezone\n",
+ "\n",
+ "# Naive datetime -- no timezone\n",
+ "naive: datetime = datetime(2025, 6, 15, 12, 0)\n",
+ "print(f\"Naive datetime: {naive}\")\n",
+ "print(f\"tzinfo: {naive.tzinfo}\")\n",
+ "print(f\"Is naive: {naive.tzinfo is None}\")\n",
+ "\n",
+ "# Aware datetime -- with timezone\n",
+ "aware: datetime = datetime(2025, 6, 15, 12, 0, tzinfo=timezone.utc)\n",
+ "print(f\"\\nAware datetime: {aware}\")\n",
+ "print(f\"tzinfo: {aware.tzinfo}\")\n",
+ "print(f\"Is aware: {aware.tzinfo is not None}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a1000004",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from datetime import datetime, timezone\n",
+ "\n",
+ "# Cannot compare naive and aware datetimes\n",
+ "naive: datetime = datetime(2025, 6, 15, 12, 0)\n",
+ "aware: datetime = datetime(2025, 6, 15, 12, 0, tzinfo=timezone.utc)\n",
+ "\n",
+ "try:\n",
+ " result = naive < aware\n",
+ "except TypeError as e:\n",
+ " print(f\"Comparison error: {e}\")\n",
+ "\n",
+ "# Cannot subtract naive from aware\n",
+ "try:\n",
+ " diff = aware - naive\n",
+ "except TypeError as e:\n",
+ " print(f\"Subtraction error: {e}\")\n",
+ "\n",
+ "print(\"\\nLesson: never mix naive and aware datetimes.\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a1000005",
+ "metadata": {},
+ "source": [
+ "## Section 2: Working with UTC\n",
+ "\n",
+ "UTC (Coordinated Universal Time) is the standard reference timezone. Always store timestamps in UTC and convert to local time only for display."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a1000006",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from datetime import datetime, timezone\n",
+ "\n",
+ "# Create a UTC datetime\n",
+ "utc_now: datetime = datetime.now(timezone.utc)\n",
+ "print(f\"UTC now: {utc_now}\")\n",
+ "print(f\"tzname: {utc_now.tzname()}\")\n",
+ "\n",
+ "# Construct a specific UTC datetime\n",
+ "dt: datetime = datetime(2025, 6, 15, 12, 0, tzinfo=timezone.utc)\n",
+ "print(f\"\\nSpecific UTC: {dt}\")\n",
+ "print(f\"tzinfo == timezone.utc: {dt.tzinfo == timezone.utc}\")\n",
+ "print(f\"tzname: {dt.tzname()}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a1000007",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from datetime import datetime, timezone, timedelta\n",
+ "\n",
+ "# Fixed-offset timezones using timezone(timedelta(...))\n",
+ "utc_plus_5: timezone = timezone(timedelta(hours=5))\n",
+ "utc_minus_8: timezone = timezone(timedelta(hours=-8))\n",
+ "\n",
+ "dt_plus5: datetime = datetime(2025, 6, 15, 17, 0, tzinfo=utc_plus_5)\n",
+ "dt_minus8: datetime = datetime(2025, 6, 15, 4, 0, tzinfo=utc_minus_8)\n",
+ "\n",
+ "print(f\"UTC+5: {dt_plus5}\")\n",
+ "print(f\"UTC-8: {dt_minus8}\")\n",
+ "\n",
+ "# Both represent the same instant in time (12:00 UTC)\n",
+ "print(f\"\\nSame instant? {dt_plus5 == dt_minus8}\")\n",
+ "print(f\"Plus5 in UTC: {dt_plus5.astimezone(timezone.utc)}\")\n",
+ "print(f\"Minus8 in UTC: {dt_minus8.astimezone(timezone.utc)}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a1000008",
+ "metadata": {},
+ "source": [
+ "## Section 3: ZoneInfo -- IANA Timezones\n",
+ "\n",
+ "The `zoneinfo` module (Python 3.9+) provides access to the IANA timezone database. This gives you named timezones like `\"America/New_York\"` that automatically handle daylight saving time (DST)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a1000009",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from datetime import datetime\n",
+ "from zoneinfo import ZoneInfo\n",
+ "\n",
+ "# Create a datetime in a specific timezone\n",
+ "eastern: ZoneInfo = ZoneInfo(\"America/New_York\")\n",
+ "dt: datetime = datetime(2025, 6, 15, 12, 0, tzinfo=eastern)\n",
+ "\n",
+ "print(f\"New York: {dt}\")\n",
+ "print(f\"tzinfo: {dt.tzinfo}\")\n",
+ "print(f\"tzname: {dt.tzname()}\")\n",
+ "print(f\"UTC offset: {dt.utcoffset()}\")\n",
+ "\n",
+ "# Some common timezones\n",
+ "zones: list[str] = [\n",
+ " \"America/New_York\",\n",
+ " \"America/Chicago\",\n",
+ " \"America/Los_Angeles\",\n",
+ " \"Europe/London\",\n",
+ " \"Europe/Berlin\",\n",
+ " \"Asia/Tokyo\",\n",
+ "]\n",
+ "print(\"\\nCommon timezones:\")\n",
+ "for zone_name in zones:\n",
+ " tz: ZoneInfo = ZoneInfo(zone_name)\n",
+ " local_dt: datetime = datetime(2025, 6, 15, 12, 0, tzinfo=tz)\n",
+ " print(f\" {zone_name:25s} UTC offset: {local_dt.utcoffset()}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a1000010",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from datetime import datetime\n",
+ "from zoneinfo import ZoneInfo, available_timezones\n",
+ "\n",
+ "# List available timezones (there are hundreds)\n",
+ "all_zones: set[str] = available_timezones()\n",
+ "print(f\"Total available timezones: {len(all_zones)}\")\n",
+ "\n",
+ "# Show a sample of US timezones\n",
+ "us_zones: list[str] = sorted(tz for tz in all_zones if tz.startswith(\"US/\"))\n",
+ "print(f\"\\nUS timezones:\")\n",
+ "for tz_name in us_zones:\n",
+ " print(f\" {tz_name}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a1000011",
+ "metadata": {},
+ "source": [
+ "## Section 4: Timezone Conversion with `astimezone()`\n",
+ "\n",
+ "The `astimezone()` method converts an aware datetime to a different timezone. The underlying instant in time stays the same -- only the representation changes."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a1000012",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from datetime import datetime, timezone\n",
+ "from zoneinfo import ZoneInfo\n",
+ "\n",
+ "# Start with a UTC time\n",
+ "utc_dt: datetime = datetime(2025, 6, 15, 12, 0, tzinfo=timezone.utc)\n",
+ "print(f\"UTC: {utc_dt}\")\n",
+ "\n",
+ "# Convert to Eastern Time (UTC-4 in June due to EDT)\n",
+ "eastern: datetime = utc_dt.astimezone(ZoneInfo(\"America/New_York\"))\n",
+ "print(f\"New York: {eastern}\")\n",
+ "print(f\"Hour: {eastern.hour}\") # 8 (12 - 4)\n",
+ "print(f\"Same date: {eastern.date() == utc_dt.date()}\")\n",
+ "\n",
+ "# Convert to other timezones\n",
+ "tokyo: datetime = utc_dt.astimezone(ZoneInfo(\"Asia/Tokyo\"))\n",
+ "london: datetime = utc_dt.astimezone(ZoneInfo(\"Europe/London\"))\n",
+ "print(f\"\\nTokyo: {tokyo}\")\n",
+ "print(f\"London: {london}\")\n",
+ "\n",
+ "# All represent the same instant\n",
+ "print(f\"\\nAll equal? {utc_dt == eastern == tokyo == london}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a1000013",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from datetime import datetime, timezone\n",
+ "from zoneinfo import ZoneInfo\n",
+ "\n",
+ "# Practical: display a meeting time across multiple timezones\n",
+ "def show_meeting_times(\n",
+ " meeting_utc: datetime,\n",
+ " timezones: list[str],\n",
+ ") -> None:\n",
+ " \"\"\"Display a UTC meeting time in multiple timezones.\"\"\"\n",
+ " print(f\"Meeting time (UTC): {meeting_utc.strftime('%Y-%m-%d %H:%M %Z')}\")\n",
+ " print(\"-\" * 50)\n",
+ " for tz_name in timezones:\n",
+ " tz: ZoneInfo = ZoneInfo(tz_name)\n",
+ " local: datetime = meeting_utc.astimezone(tz)\n",
+ " print(f\" {tz_name:25s} {local.strftime('%H:%M %Z (%Y-%m-%d)')}\")\n",
+ "\n",
+ "meeting: datetime = datetime(2025, 6, 15, 15, 0, tzinfo=timezone.utc)\n",
+ "offices: list[str] = [\n",
+ " \"America/New_York\",\n",
+ " \"America/Los_Angeles\",\n",
+ " \"Europe/London\",\n",
+ " \"Europe/Berlin\",\n",
+ " \"Asia/Tokyo\",\n",
+ " \"Australia/Sydney\",\n",
+ "]\n",
+ "show_meeting_times(meeting, offices)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a1000014",
+ "metadata": {},
+ "source": [
+ "## Section 5: Daylight Saving Time\n",
+ "\n",
+ "`ZoneInfo` automatically handles DST transitions. The same timezone can have different UTC offsets depending on the date."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a1000015",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from datetime import datetime\n",
+ "from zoneinfo import ZoneInfo\n",
+ "\n",
+ "eastern: ZoneInfo = ZoneInfo(\"America/New_York\")\n",
+ "\n",
+ "# Summer (EDT -- Eastern Daylight Time, UTC-4)\n",
+ "summer: datetime = datetime(2025, 6, 15, 12, 0, tzinfo=eastern)\n",
+ "print(f\"Summer: {summer}\")\n",
+ "print(f\" tzname: {summer.tzname()}\")\n",
+ "print(f\" UTC offset: {summer.utcoffset()}\")\n",
+ "print(f\" DST offset: {summer.dst()}\")\n",
+ "\n",
+ "# Winter (EST -- Eastern Standard Time, UTC-5)\n",
+ "winter: datetime = datetime(2025, 12, 15, 12, 0, tzinfo=eastern)\n",
+ "print(f\"\\nWinter: {winter}\")\n",
+ "print(f\" tzname: {winter.tzname()}\")\n",
+ "print(f\" UTC offset: {winter.utcoffset()}\")\n",
+ "print(f\" DST offset: {winter.dst()}\")\n",
+ "\n",
+ "# The offset changes depending on date\n",
+ "print(f\"\\nSummer offset != Winter offset: {summer.utcoffset() != winter.utcoffset()}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a1000016",
+ "metadata": {},
+ "source": [
+ "## Section 6: Making Naive Datetimes Aware\n",
+ "\n",
+ "You can attach timezone info to a naive datetime using `replace()` or by passing `tzinfo` at construction time. Use `replace()` when you know the naive time is already in a specific timezone."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a1000017",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from datetime import datetime, timezone\n",
+ "from zoneinfo import ZoneInfo\n",
+ "\n",
+ "# A naive datetime (no timezone)\n",
+ "naive: datetime = datetime(2025, 6, 15, 14, 30)\n",
+ "print(f\"Naive: {naive} (tzinfo={naive.tzinfo})\")\n",
+ "\n",
+ "# Attach UTC timezone using replace\n",
+ "# This says \"this time IS in UTC\" -- it does not convert\n",
+ "as_utc: datetime = naive.replace(tzinfo=timezone.utc)\n",
+ "print(f\"As UTC: {as_utc}\")\n",
+ "\n",
+ "# Attach a named timezone\n",
+ "as_eastern: datetime = naive.replace(tzinfo=ZoneInfo(\"America/New_York\"))\n",
+ "print(f\"As Eastern: {as_eastern}\")\n",
+ "\n",
+ "# These are different instants in time!\n",
+ "print(f\"\\nAs UTC in UTC: {as_utc.astimezone(timezone.utc)}\")\n",
+ "print(f\"As Eastern in UTC: {as_eastern.astimezone(timezone.utc)}\")\n",
+ "print(f\"Same instant? {as_utc == as_eastern}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a1000018",
+ "metadata": {},
+ "source": [
+ "## Section 7: Best Practices for Timezone Handling\n",
+ "\n",
+ "Follow these guidelines to avoid common timezone bugs."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a1000019",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from datetime import datetime, timezone\n",
+ "from zoneinfo import ZoneInfo\n",
+ "\n",
+ "# BEST PRACTICE 1: Always use aware datetimes\n",
+ "# Bad: datetime.now() -- returns naive local time\n",
+ "# Good: datetime.now(tz=...) -- returns aware time\n",
+ "now_utc: datetime = datetime.now(tz=timezone.utc)\n",
+ "now_eastern: datetime = datetime.now(tz=ZoneInfo(\"America/New_York\"))\n",
+ "\n",
+ "print(f\"UTC now: {now_utc}\")\n",
+ "print(f\"Eastern now: {now_eastern}\")\n",
+ "\n",
+ "# BEST PRACTICE 2: Store in UTC, display in local\n",
+ "stored_event: datetime = datetime(2025, 6, 15, 18, 0, tzinfo=timezone.utc)\n",
+ "user_tz: ZoneInfo = ZoneInfo(\"Asia/Tokyo\")\n",
+ "displayed: datetime = stored_event.astimezone(user_tz)\n",
+ "print(f\"\\nStored (UTC): {stored_event}\")\n",
+ "print(f\"Display (Tokyo): {displayed}\")\n",
+ "\n",
+ "# BEST PRACTICE 3: Use ZoneInfo, not fixed offsets\n",
+ "# Fixed offsets do not handle DST correctly\n",
+ "print(\"\\nZoneInfo handles DST automatically:\")\n",
+ "tz: ZoneInfo = ZoneInfo(\"America/New_York\")\n",
+ "jun: datetime = datetime(2025, 6, 15, tzinfo=tz)\n",
+ "dec: datetime = datetime(2025, 12, 15, tzinfo=tz)\n",
+ "print(f\" June offset: {jun.utcoffset()}\")\n",
+ "print(f\" December offset: {dec.utcoffset()}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a1000020",
+ "metadata": {},
+ "source": [
+ "## Summary\n",
+ "\n",
+ "### Naive vs Aware\n",
+ "- **Naive**: `datetime.tzinfo is None` -- no timezone information\n",
+ "- **Aware**: `datetime.tzinfo is not None` -- explicit timezone\n",
+ "- Never mix naive and aware datetimes in comparisons or arithmetic\n",
+ "\n",
+ "### UTC\n",
+ "- Use `timezone.utc` for the built-in UTC timezone\n",
+ "- Use `datetime.now(timezone.utc)` instead of `datetime.utcnow()` (deprecated)\n",
+ "- Store timestamps in UTC; convert to local time for display only\n",
+ "\n",
+ "### ZoneInfo (Python 3.9+)\n",
+ "- `ZoneInfo(\"America/New_York\")` gives IANA timezone with DST support\n",
+ "- `available_timezones()` lists all known timezone names\n",
+ "- `astimezone()` converts between timezones, preserving the instant\n",
+ "\n",
+ "### Best Practices\n",
+ "1. Always use aware datetimes\n",
+ "2. Store in UTC, display in local time\n",
+ "3. Use `ZoneInfo` instead of fixed UTC offsets for named timezones\n",
+ "4. Use `replace(tzinfo=...)` to attach timezone info to a naive datetime"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "name": "python",
+ "version": "3.13.0"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
\ No newline at end of file
diff --git a/src/chapter_29/03_formatting_and_calendar.ipynb b/src/chapter_29/03_formatting_and_calendar.ipynb
new file mode 100644
index 0000000..b5141b4
--- /dev/null
+++ b/src/chapter_29/03_formatting_and_calendar.ipynb
@@ -0,0 +1,511 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "b1000001",
+ "metadata": {},
+ "source": [
+ "# Chapter 29: Formatting and Calendar\n",
+ "\n",
+ "This notebook covers formatting and parsing dates with `strftime`/`strptime`, working with ISO 8601 format, and using the `calendar` module for calendar-related computations.\n",
+ "\n",
+ "## Key Concepts\n",
+ "- **`strftime()`**: Format a datetime as a string using format codes\n",
+ "- **`strptime()`**: Parse a string into a datetime using format codes\n",
+ "- **ISO 8601**: The international standard for date/time representation\n",
+ "- **`isoformat()` / `fromisoformat()`**: Built-in ISO 8601 support\n",
+ "- **`calendar` module**: Leap year checks, month ranges, text calendars"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b1000002",
+ "metadata": {},
+ "source": [
+ "## Section 1: Formatting with `strftime()`\n",
+ "\n",
+ "The `strftime()` method formats a `date`, `time`, or `datetime` object into a string using **format codes** that start with `%`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b1000003",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from datetime import datetime\n",
+ "\n",
+ "dt: datetime = datetime(2025, 6, 15, 14, 30, 45)\n",
+ "\n",
+ "# Common format codes\n",
+ "print(f\"Date (YYYY-MM-DD): {dt.strftime('%Y-%m-%d')}\")\n",
+ "print(f\"Time (HH:MM): {dt.strftime('%H:%M')}\")\n",
+ "print(f\"Time (HH:MM:SS): {dt.strftime('%H:%M:%S')}\")\n",
+ "print(f\"12-hour time: {dt.strftime('%I:%M %p')}\")\n",
+ "print(f\"Full day name: {dt.strftime('%A')}\")\n",
+ "print(f\"Abbreviated day: {dt.strftime('%a')}\")\n",
+ "print(f\"Full month name: {dt.strftime('%B')}\")\n",
+ "print(f\"Abbreviated month: {dt.strftime('%b')}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b1000004",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from datetime import datetime\n",
+ "\n",
+ "dt: datetime = datetime(2025, 6, 15, 14, 30, 45)\n",
+ "\n",
+ "# Combining format codes into useful patterns\n",
+ "formats: dict[str, str] = {\n",
+ " \"ISO-like\": \"%Y-%m-%d %H:%M:%S\",\n",
+ " \"US format\": \"%m/%d/%Y\",\n",
+ " \"European\": \"%d/%m/%Y\",\n",
+ " \"Long date\": \"%B %d, %Y\",\n",
+ " \"Day and date\": \"%A, %B %d, %Y\",\n",
+ " \"Compact\": \"%Y%m%d_%H%M%S\",\n",
+ " \"Log timestamp\": \"%Y-%m-%d %H:%M:%S\",\n",
+ "}\n",
+ "\n",
+ "for name, fmt in formats.items():\n",
+ " formatted: str = dt.strftime(fmt)\n",
+ " print(f\"{name:18s} {fmt:25s} -> {formatted}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b1000005",
+ "metadata": {},
+ "source": [
+ "## Section 2: Parsing with `strptime()`\n",
+ "\n",
+ "The `strptime()` class method parses a string into a `datetime` object. The format string must match the input exactly."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b1000006",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from datetime import datetime\n",
+ "\n",
+ "# Parse ISO-like format\n",
+ "dt1: datetime = datetime.strptime(\"2025-06-15\", \"%Y-%m-%d\")\n",
+ "print(f\"Parsed: {dt1}\")\n",
+ "print(f\"Year={dt1.year}, Month={dt1.month}, Day={dt1.day}\")\n",
+ "\n",
+ "# Parse date with time\n",
+ "dt2: datetime = datetime.strptime(\"2025-06-15 14:30:00\", \"%Y-%m-%d %H:%M:%S\")\n",
+ "print(f\"\\nWith time: {dt2}\")\n",
+ "\n",
+ "# Parse US format\n",
+ "dt3: datetime = datetime.strptime(\"06/15/2025\", \"%m/%d/%Y\")\n",
+ "print(f\"US format: {dt3}\")\n",
+ "\n",
+ "# Parse long format\n",
+ "dt4: datetime = datetime.strptime(\"June 15, 2025\", \"%B %d, %Y\")\n",
+ "print(f\"Long format: {dt4}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b1000007",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from datetime import datetime\n",
+ "\n",
+ "# strptime raises ValueError on format mismatch\n",
+ "try:\n",
+ " bad: datetime = datetime.strptime(\"15/06/2025\", \"%m/%d/%Y\")\n",
+ "except ValueError as e:\n",
+ " print(f\"Parse error: {e}\")\n",
+ "\n",
+ "# The format must match exactly, including separators\n",
+ "try:\n",
+ " bad2: datetime = datetime.strptime(\"2025-06-15\", \"%Y/%m/%d\")\n",
+ "except ValueError as e:\n",
+ " print(f\"Separator mismatch: {e}\")\n",
+ "\n",
+ "print(\"\\nAlways ensure the format string matches the input pattern.\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b1000008",
+ "metadata": {},
+ "source": [
+ "## Section 3: Format Code Reference\n",
+ "\n",
+ "Here are the most commonly used `strftime`/`strptime` format codes."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b1000009",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from datetime import datetime\n",
+ "\n",
+ "dt: datetime = datetime(2025, 6, 15, 14, 30, 45)\n",
+ "\n",
+ "codes: list[tuple[str, str]] = [\n",
+ " (\"%Y\", \"4-digit year\"),\n",
+ " (\"%m\", \"Zero-padded month (01-12)\"),\n",
+ " (\"%d\", \"Zero-padded day (01-31)\"),\n",
+ " (\"%H\", \"24-hour hour (00-23)\"),\n",
+ " (\"%I\", \"12-hour hour (01-12)\"),\n",
+ " (\"%M\", \"Minute (00-59)\"),\n",
+ " (\"%S\", \"Second (00-59)\"),\n",
+ " (\"%p\", \"AM/PM\"),\n",
+ " (\"%A\", \"Full weekday name\"),\n",
+ " (\"%a\", \"Abbreviated weekday\"),\n",
+ " (\"%B\", \"Full month name\"),\n",
+ " (\"%b\", \"Abbreviated month\"),\n",
+ " (\"%j\", \"Day of year (001-366)\"),\n",
+ " (\"%U\", \"Week number (Sunday start)\"),\n",
+ " (\"%W\", \"Week number (Monday start)\"),\n",
+ " (\"%%\", \"Literal % character\"),\n",
+ "]\n",
+ "\n",
+ "print(f\"{'Code':<6} {'Description':<30} {'Result'}\")\n",
+ "print(\"-\" * 55)\n",
+ "for code, desc in codes:\n",
+ " result: str = dt.strftime(code)\n",
+ " print(f\"{code:<6} {desc:<30} {result}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b1000010",
+ "metadata": {},
+ "source": [
+ "## Section 4: ISO 8601 Format\n",
+ "\n",
+ "ISO 8601 is the international standard for date/time strings (`2025-06-15T14:30:00+00:00`). Python has built-in support via `isoformat()` and `fromisoformat()`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b1000011",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from datetime import datetime, date, time, timezone\n",
+ "\n",
+ "# isoformat() produces ISO 8601 strings\n",
+ "dt: datetime = datetime(2025, 6, 15, 14, 30, tzinfo=timezone.utc)\n",
+ "iso: str = dt.isoformat()\n",
+ "print(f\"Datetime ISO: {iso}\")\n",
+ "print(f\"Contains date: {'2025-06-15' in iso}\")\n",
+ "print(f\"Contains time: {'14:30' in iso}\")\n",
+ "\n",
+ "# Date and time also have isoformat\n",
+ "d: date = date(2025, 6, 15)\n",
+ "t: time = time(14, 30, 0)\n",
+ "print(f\"\\nDate ISO: {d.isoformat()}\")\n",
+ "print(f\"Time ISO: {t.isoformat()}\")\n",
+ "\n",
+ "# Custom separator (default is 'T')\n",
+ "print(f\"\\nWith space separator: {dt.isoformat(sep=' ')}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b1000012",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from datetime import datetime, date, timezone\n",
+ "\n",
+ "# fromisoformat() parses ISO 8601 strings\n",
+ "dt1: datetime = datetime.fromisoformat(\"2025-06-15T14:30:00\")\n",
+ "print(f\"Parsed: {dt1}\")\n",
+ "print(f\"Equal: {dt1 == datetime(2025, 6, 15, 14, 30)}\")\n",
+ "\n",
+ "# With timezone\n",
+ "dt2: datetime = datetime.fromisoformat(\"2025-06-15T14:30:00+00:00\")\n",
+ "print(f\"\\nWith tz: {dt2}\")\n",
+ "print(f\"tzinfo: {dt2.tzinfo}\")\n",
+ "\n",
+ "# Date from ISO string\n",
+ "d: date = date.fromisoformat(\"2025-06-15\")\n",
+ "print(f\"\\nDate: {d}\")\n",
+ "\n",
+ "# Round-trip: format then parse\n",
+ "original: datetime = datetime(2025, 6, 15, 14, 30, tzinfo=timezone.utc)\n",
+ "round_tripped: datetime = datetime.fromisoformat(original.isoformat())\n",
+ "print(f\"\\nOriginal: {original}\")\n",
+ "print(f\"Round-trip: {round_tripped}\")\n",
+ "print(f\"Equal: {original == round_tripped}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b1000013",
+ "metadata": {},
+ "source": [
+ "## Section 5: The `calendar` Module\n",
+ "\n",
+ "The `calendar` module provides utilities for working with calendars, including leap year checks, month lengths, and text calendar generation."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b1000014",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import calendar\n",
+ "\n",
+ "# Check for leap years\n",
+ "years: list[int] = [1900, 2000, 2024, 2025, 2100]\n",
+ "for year in years:\n",
+ " is_leap: bool = calendar.isleap(year)\n",
+ " print(f\"{year}: leap={is_leap}\")\n",
+ "\n",
+ "# Leap year rules:\n",
+ "# - Divisible by 4: leap year\n",
+ "# - BUT divisible by 100: NOT a leap year\n",
+ "# - BUT divisible by 400: leap year\n",
+ "print(f\"\\n1900: div by 100 but not 400 -> not leap: {not calendar.isleap(1900)}\")\n",
+ "print(f\"2000: div by 400 -> leap: {calendar.isleap(2000)}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b1000015",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import calendar\n",
+ "\n",
+ "# monthrange returns (weekday_of_first_day, number_of_days)\n",
+ "# weekday: 0=Monday, 6=Sunday\n",
+ "weekday, days = calendar.monthrange(2025, 2)\n",
+ "print(f\"February 2025: starts on weekday {weekday}, has {days} days\")\n",
+ "\n",
+ "# Leap year February\n",
+ "_, days_leap = calendar.monthrange(2024, 2)\n",
+ "print(f\"February 2024 (leap): {days_leap} days\")\n",
+ "\n",
+ "# Check all months in a year\n",
+ "print(f\"\\nDays per month in 2025:\")\n",
+ "for month in range(1, 13):\n",
+ " _, num_days = calendar.monthrange(2025, month)\n",
+ " month_name: str = calendar.month_name[month]\n",
+ " print(f\" {month_name:10s} {num_days} days\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b1000016",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import calendar\n",
+ "\n",
+ "# Generate a text calendar for a month\n",
+ "print(calendar.month(2025, 6))\n",
+ "\n",
+ "# monthcalendar returns weeks as lists of day numbers (0 = outside month)\n",
+ "print(\"June 2025 as nested lists (0 = not in month):\")\n",
+ "weeks: list[list[int]] = calendar.monthcalendar(2025, 6)\n",
+ "for week in weeks:\n",
+ " print(f\" {week}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b1000017",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import calendar\n",
+ "\n",
+ "# Count leap years in a range\n",
+ "leap_count: int = calendar.leapdays(2000, 2026) # 2000 to 2025 inclusive\n",
+ "print(f\"Leap years from 2000 to 2025: {leap_count}\")\n",
+ "\n",
+ "# List them\n",
+ "leap_years: list[int] = [y for y in range(2000, 2026) if calendar.isleap(y)]\n",
+ "print(f\"Leap years: {leap_years}\")\n",
+ "\n",
+ "# Day name and month name constants\n",
+ "print(f\"\\nDay names: {list(calendar.day_name)}\")\n",
+ "print(f\"Month names: {list(calendar.month_name)[1:]}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b1000018",
+ "metadata": {},
+ "source": [
+ "## Section 6: Practical Date Range Generation\n",
+ "\n",
+ "Combining `datetime`, `timedelta`, and `calendar` for generating useful date ranges."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b1000019",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import calendar\n",
+ "from datetime import date, timedelta\n",
+ "\n",
+ "\n",
+ "def weekdays_in_month(year: int, month: int) -> list[date]:\n",
+ " \"\"\"Return all weekday (Mon-Fri) dates in a given month.\"\"\"\n",
+ " _, num_days = calendar.monthrange(year, month)\n",
+ " return [\n",
+ " date(year, month, day)\n",
+ " for day in range(1, num_days + 1)\n",
+ " if date(year, month, day).weekday() < 5 # 0-4 = Mon-Fri\n",
+ " ]\n",
+ "\n",
+ "\n",
+ "# Get all weekdays in June 2025\n",
+ "workdays: list[date] = weekdays_in_month(2025, 6)\n",
+ "print(f\"Weekdays in June 2025: {len(workdays)}\")\n",
+ "for d in workdays[:5]:\n",
+ " print(f\" {d} ({d.strftime('%A')})\")\n",
+ "print(f\" ...\")\n",
+ "for d in workdays[-3:]:\n",
+ " print(f\" {d} ({d.strftime('%A')})\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b1000020",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import calendar\n",
+ "from datetime import date\n",
+ "\n",
+ "\n",
+ "def first_day_of_each_month(year: int) -> list[date]:\n",
+ " \"\"\"Return the first day of each month in a given year.\"\"\"\n",
+ " return [date(year, month, 1) for month in range(1, 13)]\n",
+ "\n",
+ "\n",
+ "def last_day_of_each_month(year: int) -> list[date]:\n",
+ " \"\"\"Return the last day of each month in a given year.\"\"\"\n",
+ " return [\n",
+ " date(year, month, calendar.monthrange(year, month)[1])\n",
+ " for month in range(1, 13)\n",
+ " ]\n",
+ "\n",
+ "\n",
+ "print(\"First day of each month in 2025:\")\n",
+ "for d in first_day_of_each_month(2025):\n",
+ " print(f\" {d} ({d.strftime('%A')})\")\n",
+ "\n",
+ "print(f\"\\nLast day of each month in 2025:\")\n",
+ "for d in last_day_of_each_month(2025):\n",
+ " print(f\" {d} ({d.strftime('%A')})\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b1000021",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from datetime import datetime, timezone\n",
+ "\n",
+ "# Practical: parse various date formats from user input\n",
+ "def parse_flexible_date(date_string: str) -> datetime:\n",
+ " \"\"\"Try multiple formats to parse a date string.\"\"\"\n",
+ " formats: list[str] = [\n",
+ " \"%Y-%m-%d\",\n",
+ " \"%Y-%m-%dT%H:%M:%S\",\n",
+ " \"%m/%d/%Y\",\n",
+ " \"%d/%m/%Y\",\n",
+ " \"%B %d, %Y\",\n",
+ " \"%b %d, %Y\",\n",
+ " ]\n",
+ " for fmt in formats:\n",
+ " try:\n",
+ " return datetime.strptime(date_string, fmt)\n",
+ " except ValueError:\n",
+ " continue\n",
+ " raise ValueError(f\"Unable to parse date: {date_string!r}\")\n",
+ "\n",
+ "\n",
+ "test_inputs: list[str] = [\n",
+ " \"2025-06-15\",\n",
+ " \"06/15/2025\",\n",
+ " \"June 15, 2025\",\n",
+ " \"Jun 15, 2025\",\n",
+ "]\n",
+ "\n",
+ "for input_str in test_inputs:\n",
+ " parsed: datetime = parse_flexible_date(input_str)\n",
+ " print(f\"{input_str:20s} -> {parsed.strftime('%Y-%m-%d')}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b1000022",
+ "metadata": {},
+ "source": [
+ "## Summary\n",
+ "\n",
+ "### Formatting (`strftime`)\n",
+ "- `dt.strftime(\"%Y-%m-%d\")` formats datetime to string\n",
+ "- Common codes: `%Y` (year), `%m` (month), `%d` (day), `%H` (hour), `%M` (minute), `%S` (second)\n",
+ "- `%A` (weekday name), `%B` (month name), `%p` (AM/PM)\n",
+ "\n",
+ "### Parsing (`strptime`)\n",
+ "- `datetime.strptime(string, format)` parses string to datetime\n",
+ "- Format string must match the input exactly\n",
+ "- Raises `ValueError` on mismatch\n",
+ "\n",
+ "### ISO 8601\n",
+ "- `dt.isoformat()` produces standard ISO 8601 strings\n",
+ "- `datetime.fromisoformat(string)` parses ISO 8601 back to datetime\n",
+ "- Round-trip safe: `fromisoformat(dt.isoformat()) == dt`\n",
+ "\n",
+ "### Calendar Module\n",
+ "- `calendar.isleap(year)` checks leap years\n",
+ "- `calendar.monthrange(year, month)` returns `(first_weekday, num_days)`\n",
+ "- `calendar.month(year, month)` generates text calendar\n",
+ "- `calendar.month_name` and `calendar.day_name` for name constants"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "name": "python",
+ "version": "3.13.0"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
\ No newline at end of file
diff --git a/src/chapter_29/README.md b/src/chapter_29/README.md
new file mode 100644
index 0000000..31a12be
--- /dev/null
+++ b/src/chapter_29/README.md
@@ -0,0 +1,9 @@
+# Chapter 29: Date, Time, and Scheduling
+
+Working with dates, times, and timezones — `datetime`, `zoneinfo`, `timedelta`, `calendar`, formatting and parsing.
+
+## Notebooks
+
+1. **01_datetime_fundamentals.ipynb** — `date`, `time`, `datetime`, `timedelta`, arithmetic
+2. **02_timezones_and_zoneinfo.ipynb** — `zoneinfo`, aware vs naive datetimes, UTC, conversions
+3. **03_formatting_and_calendar.ipynb** — `strftime`/`strptime`, ISO 8601, `calendar` module, date ranges
diff --git a/src/chapter_29/__init__.py b/src/chapter_29/__init__.py
new file mode 100644
index 0000000..925f495
--- /dev/null
+++ b/src/chapter_29/__init__.py
@@ -0,0 +1 @@
+"""Chapter 29: Date, Time, and Scheduling."""
diff --git a/src/chapter_30/01_xml_processing.ipynb b/src/chapter_30/01_xml_processing.ipynb
new file mode 100644
index 0000000..6050bc2
--- /dev/null
+++ b/src/chapter_30/01_xml_processing.ipynb
@@ -0,0 +1,613 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "a1b2c3d4",
+ "metadata": {},
+ "source": [
+ "# Chapter 30: XML Processing\n",
+ "\n",
+ "This notebook covers XML parsing and generation using Python's built-in `xml.etree.ElementTree` module. XML (eXtensible Markup Language) remains widely used in configuration files, web services (SOAP), and data exchange formats.\n",
+ "\n",
+ "## Key Concepts\n",
+ "- **Parsing XML**: Converting XML strings or files into navigable tree structures\n",
+ "- **Building XML**: Creating elements programmatically with `Element` and `SubElement`\n",
+ "- **XPath searches**: Finding elements using path expressions\n",
+ "- **Namespaces**: Working with XML namespaces to avoid name collisions\n",
+ "- **Serialization**: Converting element trees back to XML strings"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b2c3d4e5",
+ "metadata": {},
+ "source": [
+ "## Section 1: Parsing XML from Strings\n",
+ "\n",
+ "`ET.fromstring()` parses an XML string and returns the root `Element`. Each element has a tag, optional text content, attributes, and child elements."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "c3d4e5f6",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import xml.etree.ElementTree as ET\n",
+ "\n",
+ "# Parse a simple XML string\n",
+ "xml_data: str = \"- hello
- world
\"\n",
+ "root: ET.Element = ET.fromstring(xml_data)\n",
+ "\n",
+ "print(f\"Root tag: {root.tag}\")\n",
+ "print(f\"Number of children: {len(root)}\")\n",
+ "\n",
+ "# Access child elements\n",
+ "items: list[ET.Element] = root.findall(\"item\")\n",
+ "for item in items:\n",
+ " print(f\" <{item.tag}> = {item.text}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "d4e5f6a7",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Parse XML with attributes\n",
+ "xml_with_attrs: str = \"\"\"\n",
+ "\n",
+ " \n",
+ " Effective Java\n",
+ " Joshua Bloch\n",
+ " 2018\n",
+ " \n",
+ " \n",
+ " Head First Design Patterns\n",
+ " Eric Freeman\n",
+ " 2004\n",
+ " \n",
+ "\n",
+ "\"\"\"\n",
+ "\n",
+ "root = ET.fromstring(xml_with_attrs)\n",
+ "\n",
+ "for book in root.findall(\"book\"):\n",
+ " isbn: str | None = book.get(\"isbn\")\n",
+ " title: str | None = book.findtext(\"title\")\n",
+ " author: str | None = book.findtext(\"author\")\n",
+ " print(f\" [{isbn}] {title} by {author}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e5f6a7b8",
+ "metadata": {},
+ "source": [
+ "## Section 2: Parsing XML from Files\n",
+ "\n",
+ "`ET.parse()` reads an XML file and returns an `ElementTree` object. The root element is obtained via `.getroot()`. For this demo, we write a temporary file first."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f6a7b8c9",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import tempfile\n",
+ "from pathlib import Path\n",
+ "\n",
+ "# Write a sample XML file\n",
+ "sample_xml: str = \"\"\"\n",
+ "\n",
+ " \n",
+ " Alice\n",
+ " Engineering\n",
+ " \n",
+ " \n",
+ " Bob\n",
+ " Marketing\n",
+ " \n",
+ "\n",
+ "\"\"\"\n",
+ "\n",
+ "with tempfile.NamedTemporaryFile(mode=\"w\", suffix=\".xml\", delete=False) as f:\n",
+ " f.write(sample_xml)\n",
+ " temp_path: str = f.name\n",
+ "\n",
+ "# Parse the file\n",
+ "tree: ET.ElementTree = ET.parse(temp_path)\n",
+ "root = tree.getroot()\n",
+ "\n",
+ "print(f\"Root tag: {root.tag}\")\n",
+ "for emp in root.findall(\"employee\"):\n",
+ " emp_id: str | None = emp.get(\"id\")\n",
+ " name: str | None = emp.findtext(\"name\")\n",
+ " dept: str | None = emp.findtext(\"department\")\n",
+ " print(f\" Employee {emp_id}: {name} in {dept}\")\n",
+ "\n",
+ "# Clean up\n",
+ "Path(temp_path).unlink()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a7b8c9d0",
+ "metadata": {},
+ "source": [
+ "## Section 3: Building XML Programmatically\n",
+ "\n",
+ "Use `ET.Element()` for the root and `ET.SubElement()` to add children. Attributes are passed as a dictionary or keyword arguments."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b8c9d0e1",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Build a catalog XML document from scratch\n",
+ "catalog: ET.Element = ET.Element(\"catalog\")\n",
+ "\n",
+ "# Add a book with attributes and child elements\n",
+ "book: ET.Element = ET.SubElement(catalog, \"book\", attrib={\"id\": \"1\"})\n",
+ "title: ET.Element = ET.SubElement(book, \"title\")\n",
+ "title.text = \"Python\"\n",
+ "price: ET.Element = ET.SubElement(book, \"price\")\n",
+ "price.text = \"39.99\"\n",
+ "\n",
+ "# Add a second book\n",
+ "book2: ET.Element = ET.SubElement(catalog, \"book\", attrib={\"id\": \"2\"})\n",
+ "title2: ET.Element = ET.SubElement(book2, \"title\")\n",
+ "title2.text = \"Algorithms\"\n",
+ "price2: ET.Element = ET.SubElement(book2, \"price\")\n",
+ "price2.text = \"49.99\"\n",
+ "\n",
+ "# Verify the structure\n",
+ "found_book: ET.Element | None = catalog.find(\"book\")\n",
+ "print(f\"First book title: {catalog.find('book/title').text}\")\n",
+ "print(f\"First book id: {catalog.find('book').get('id')}\")\n",
+ "\n",
+ "# List all books\n",
+ "for b in catalog.findall(\"book\"):\n",
+ " print(f\" Book {b.get('id')}: {b.findtext('title')} - ${b.findtext('price')}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "c9d0e1f2",
+ "metadata": {},
+ "source": [
+ "## Section 4: Serializing XML to Strings\n",
+ "\n",
+ "`ET.tostring()` converts an element tree back to an XML byte string. Use `encoding=\"unicode\"` to get a regular string instead of bytes."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "d0e1f2a3",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Create a simple element\n",
+ "msg: ET.Element = ET.Element(\"msg\")\n",
+ "msg.text = \"hello\"\n",
+ "\n",
+ "# Serialize to bytes (default)\n",
+ "xml_bytes: bytes = ET.tostring(msg)\n",
+ "print(f\"Bytes: {xml_bytes}\")\n",
+ "print(f\"Type: {type(xml_bytes).__name__}\")\n",
+ "\n",
+ "# Serialize to unicode string\n",
+ "xml_str: str = ET.tostring(msg, encoding=\"unicode\")\n",
+ "print(f\"\\nString: {xml_str}\")\n",
+ "print(f\"Type: {type(xml_str).__name__}\")\n",
+ "\n",
+ "# Serialize the full catalog with XML declaration\n",
+ "catalog_str: str = ET.tostring(catalog, encoding=\"unicode\")\n",
+ "print(f\"\\nCatalog XML:\\n{catalog_str}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "e1f2a3b4",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Pretty-printing with indent (Python 3.9+)\n",
+ "ET.indent(catalog, space=\" \")\n",
+ "pretty_xml: str = ET.tostring(catalog, encoding=\"unicode\")\n",
+ "print(\"Pretty-printed catalog:\")\n",
+ "print(pretty_xml)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "f2a3b4c5",
+ "metadata": {},
+ "source": [
+ "## Section 5: XPath Expressions\n",
+ "\n",
+ "ElementTree supports a subset of XPath for searching the tree. Common patterns:\n",
+ "- `tag` -- direct children with given tag\n",
+ "- `.//tag` -- all descendants with given tag\n",
+ "- `*/tag` -- grandchildren with given tag\n",
+ "- `[@attrib]` -- elements with a specific attribute\n",
+ "- `[@attrib='value']` -- elements where attribute equals value"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a3b4c5d6",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "store_xml: str = \"\"\"\n",
+ "\n",
+ " \n",
+ " Novel\n",
+ " 12.99\n",
+ " \n",
+ " \n",
+ " Python\n",
+ " 39.99\n",
+ " \n",
+ " \n",
+ " Mystery\n",
+ " 9.99\n",
+ " \n",
+ "\n",
+ "\"\"\"\n",
+ "\n",
+ "store: ET.Element = ET.fromstring(store_xml)\n",
+ "\n",
+ "# Find all titles anywhere in the tree\n",
+ "all_titles: list[str] = [el.text for el in store.findall(\".//title\")]\n",
+ "print(f\"All titles: {all_titles}\")\n",
+ "\n",
+ "# Find books by attribute\n",
+ "fiction_books: list[ET.Element] = store.findall(\"book[@category='fiction']\")\n",
+ "print(f\"\\nFiction books:\")\n",
+ "for book in fiction_books:\n",
+ " print(f\" {book.findtext('title')} - ${book.findtext('price')}\")\n",
+ "\n",
+ "# Find all prices\n",
+ "prices: list[float] = [float(el.text) for el in store.findall(\".//price\")]\n",
+ "print(f\"\\nAll prices: {prices}\")\n",
+ "print(f\"Total: ${sum(prices):.2f}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b4c5d6e7",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# More XPath examples with nested structures\n",
+ "nested_xml: str = \"\"\"\n",
+ "\n",
+ " \n",
+ " \n",
+ " Alice\n",
+ " Bob\n",
+ " \n",
+ " \n",
+ " Charlie\n",
+ " \n",
+ " \n",
+ " \n",
+ " \n",
+ " Diana\n",
+ " \n",
+ " \n",
+ "\n",
+ "\"\"\"\n",
+ "\n",
+ "company: ET.Element = ET.fromstring(nested_xml)\n",
+ "\n",
+ "# All members in the entire company\n",
+ "all_members: list[str] = [m.text for m in company.findall(\".//member\")]\n",
+ "print(f\"All members: {all_members}\")\n",
+ "\n",
+ "# Only team leads\n",
+ "leads: list[str] = [m.text for m in company.findall(\".//member[@role='lead']\")]\n",
+ "print(f\"Team leads: {leads}\")\n",
+ "\n",
+ "# Members in grandchild teams (department/team/member)\n",
+ "grandchild_members: list[str] = [\n",
+ " m.text for m in company.findall(\"department/team/member\")\n",
+ "]\n",
+ "print(f\"Via path: {grandchild_members}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "c5d6e7f8",
+ "metadata": {},
+ "source": [
+ "## Section 6: Modifying XML Trees\n",
+ "\n",
+ "Elements are mutable: you can change text, attributes, add or remove children after creation."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "d6e7f8a9",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Start with a simple XML tree\n",
+ "xml_data = \"false\"\n",
+ "config: ET.Element = ET.fromstring(xml_data)\n",
+ "\n",
+ "# Modify text content\n",
+ "setting: ET.Element | None = config.find(\"setting\")\n",
+ "if setting is not None:\n",
+ " print(f\"Before: {setting.text}\")\n",
+ " setting.text = \"true\"\n",
+ " print(f\"After: {setting.text}\")\n",
+ "\n",
+ "# Modify attributes\n",
+ "if setting is not None:\n",
+ " setting.set(\"name\", \"verbose\")\n",
+ " setting.set(\"type\", \"boolean\")\n",
+ " print(f\"Attributes: {setting.attrib}\")\n",
+ "\n",
+ "# Add a new element\n",
+ "new_setting: ET.Element = ET.SubElement(config, \"setting\", name=\"timeout\")\n",
+ "new_setting.text = \"30\"\n",
+ "\n",
+ "# Remove an element\n",
+ "config.remove(setting)\n",
+ "\n",
+ "result: str = ET.tostring(config, encoding=\"unicode\")\n",
+ "print(f\"\\nFinal XML: {result}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e7f8a9b0",
+ "metadata": {},
+ "source": [
+ "## Section 7: XML Namespaces\n",
+ "\n",
+ "Namespaces prevent name collisions when combining XML from different sources. They are declared with `xmlns` and appear as prefixes in tag names."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f8a9b0c1",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# XML with namespaces\n",
+ "ns_xml: str = \"\"\"\n",
+ "\n",
+ " \n",
+ " \n",
+ " Cell 1\n",
+ " Cell 2\n",
+ " \n",
+ " \n",
+ " \n",
+ " \n",
+ " \n",
+ "\n",
+ "\"\"\"\n",
+ "\n",
+ "# Define namespace map for searching\n",
+ "namespaces: dict[str, str] = {\n",
+ " \"h\": \"http://www.w3.org/1999/xhtml\",\n",
+ " \"f\": \"http://www.w3.org/2002/xforms\",\n",
+ "}\n",
+ "\n",
+ "root = ET.fromstring(ns_xml)\n",
+ "\n",
+ "# Find elements using namespace prefix\n",
+ "table: ET.Element | None = root.find(\"h:table\", namespaces)\n",
+ "if table is not None:\n",
+ " print(f\"Found table element: {table.tag}\")\n",
+ "\n",
+ "cells: list[ET.Element] = root.findall(\".//h:td\", namespaces)\n",
+ "for cell in cells:\n",
+ " print(f\" Cell: {cell.text}\")\n",
+ "\n",
+ "# Find form elements\n",
+ "form_input: ET.Element | None = root.find(\".//f:input\", namespaces)\n",
+ "if form_input is not None:\n",
+ " print(f\"\\nForm input name: {form_input.get('name')}\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a9b0c1d2",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Register namespaces to preserve prefixes during serialization\n",
+ "ET.register_namespace(\"h\", \"http://www.w3.org/1999/xhtml\")\n",
+ "ET.register_namespace(\"f\", \"http://www.w3.org/2002/xforms\")\n",
+ "\n",
+ "# Build XML with namespaces using Clark notation {uri}local\n",
+ "XHTML: str = \"http://www.w3.org/1999/xhtml\"\n",
+ "doc: ET.Element = ET.Element(\"page\")\n",
+ "heading: ET.Element = ET.SubElement(doc, f\"{{{XHTML}}}h1\")\n",
+ "heading.text = \"Welcome\"\n",
+ "\n",
+ "result = ET.tostring(doc, encoding=\"unicode\")\n",
+ "print(f\"Namespaced XML: {result}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b0c1d2e3",
+ "metadata": {},
+ "source": [
+ "## Section 8: Iterative Parsing with iterparse\n",
+ "\n",
+ "For large XML files, `ET.iterparse()` processes elements incrementally without loading the entire document into memory."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "c1d2e3f4",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import io\n",
+ "\n",
+ "# Simulate a large XML file with a StringIO stream\n",
+ "large_xml: str = \"\"\"\n",
+ "\n",
+ " Application started\n",
+ " Low memory\n",
+ " Connection failed\n",
+ " Application stopped\n",
+ "\n",
+ "\"\"\"\n",
+ "\n",
+ "# Use iterparse to process events\n",
+ "warnings_and_errors: list[str] = []\n",
+ "\n",
+ "for event, elem in ET.iterparse(io.StringIO(large_xml), events=(\"end\",)):\n",
+ " if elem.tag == \"entry\":\n",
+ " level: str | None = elem.get(\"level\")\n",
+ " if level in (\"WARNING\", \"ERROR\"):\n",
+ " warnings_and_errors.append(f\"[{level}] {elem.text}\")\n",
+ " # Clear element to free memory in large files\n",
+ " elem.clear()\n",
+ "\n",
+ "print(\"Warnings and errors found:\")\n",
+ "for entry in warnings_and_errors:\n",
+ " print(f\" {entry}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "d2e3f4a5",
+ "metadata": {},
+ "source": [
+ "## Section 9: Practical Example -- Converting XML to Dictionaries\n",
+ "\n",
+ "A common pattern is converting XML data into Python dictionaries for easier manipulation."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "e3f4a5b6",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from typing import Any\n",
+ "\n",
+ "\n",
+ "def xml_to_dict(element: ET.Element) -> dict[str, Any]:\n",
+ " \"\"\"Recursively convert an XML element to a dictionary.\"\"\"\n",
+ " result: dict[str, Any] = {}\n",
+ "\n",
+ " # Include attributes with @ prefix\n",
+ " for key, value in element.attrib.items():\n",
+ " result[f\"@{key}\"] = value\n",
+ "\n",
+ " # Include text content\n",
+ " if element.text and element.text.strip():\n",
+ " if len(element) == 0 and not element.attrib:\n",
+ " return element.text.strip() # type: ignore[return-value]\n",
+ " result[\"#text\"] = element.text.strip()\n",
+ "\n",
+ " # Process child elements\n",
+ " for child in element:\n",
+ " child_data: dict[str, Any] | str = xml_to_dict(child)\n",
+ " if child.tag in result:\n",
+ " # Convert to list for duplicate tags\n",
+ " if not isinstance(result[child.tag], list):\n",
+ " result[child.tag] = [result[child.tag]]\n",
+ " result[child.tag].append(child_data)\n",
+ " else:\n",
+ " result[child.tag] = child_data\n",
+ "\n",
+ " return result\n",
+ "\n",
+ "\n",
+ "# Convert a sample XML to dict\n",
+ "xml_data = \"\"\"\n",
+ "\n",
+ " Alice\n",
+ " alice@example.com\n",
+ " \n",
+ " admin\n",
+ " editor\n",
+ " \n",
+ "\n",
+ "\"\"\"\n",
+ "\n",
+ "root = ET.fromstring(xml_data)\n",
+ "user_dict: dict[str, Any] = xml_to_dict(root)\n",
+ "\n",
+ "import json\n",
+ "print(json.dumps(user_dict, indent=2))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "f4a5b6c7",
+ "metadata": {},
+ "source": [
+ "## Summary\n",
+ "\n",
+ "### Parsing XML\n",
+ "- **`ET.fromstring()`**: Parse XML from a string, returns root `Element`\n",
+ "- **`ET.parse()`**: Parse XML from a file, returns `ElementTree`\n",
+ "- **`ET.iterparse()`**: Incremental parsing for large files\n",
+ "\n",
+ "### Building XML\n",
+ "- **`ET.Element(tag, attrib)`**: Create a root element\n",
+ "- **`ET.SubElement(parent, tag)`**: Add a child element\n",
+ "- Set `.text` for element content, `.attrib` for attributes\n",
+ "\n",
+ "### Searching\n",
+ "- **`find(path)`**: First matching element\n",
+ "- **`findall(path)`**: All matching elements\n",
+ "- **`findtext(path)`**: Text of first matching element\n",
+ "- XPath subset: `.//tag`, `[@attr='val']`, `tag/subtag`\n",
+ "\n",
+ "### Serialization\n",
+ "- **`ET.tostring(elem, encoding=\"unicode\")`**: Convert to string\n",
+ "- **`ET.indent(elem)`**: Pretty-print formatting (Python 3.9+)\n",
+ "\n",
+ "### Namespaces\n",
+ "- Use namespace dictionaries with `find()`/`findall()`\n",
+ "- Clark notation `{uri}local` for building namespaced elements\n",
+ "- `ET.register_namespace()` for clean serialization"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "name": "python",
+ "version": "3.13.0"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
\ No newline at end of file
diff --git a/src/chapter_30/02_html_and_config.ipynb b/src/chapter_30/02_html_and_config.ipynb
new file mode 100644
index 0000000..b3b1b23
--- /dev/null
+++ b/src/chapter_30/02_html_and_config.ipynb
@@ -0,0 +1,520 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "a1000001",
+ "metadata": {},
+ "source": [
+ "# Chapter 30: HTML Processing and Configuration Formats\n",
+ "\n",
+ "This notebook covers HTML parsing and escaping with the `html` module, and reading/writing configuration files using `configparser` and `tomllib`.\n",
+ "\n",
+ "## Key Concepts\n",
+ "- **HTML escaping**: Preventing XSS by converting special characters to entities\n",
+ "- **HTMLParser**: Event-driven parser for extracting data from HTML\n",
+ "- **configparser**: Reading and writing INI-style configuration files\n",
+ "- **tomllib**: Reading TOML configuration files (Python 3.11+)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "a1000002",
+ "metadata": {},
+ "source": [
+ "## Section 1: HTML Escaping and Unescaping\n",
+ "\n",
+ "The `html` module provides `escape()` and `unescape()` for converting between raw HTML and safe entity-encoded strings. This is essential for preventing cross-site scripting (XSS) attacks."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a1000003",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from html import escape, unescape\n",
+ "\n",
+ "# Escaping dangerous HTML content\n",
+ "dangerous: str = ''\n",
+ "safe: str = escape(dangerous)\n",
+ "\n",
+ "print(f\"Original: {dangerous}\")\n",
+ "print(f\"Escaped: {safe}\")\n",
+ "print(f\"Contains '
+ safe = escape(dangerous)
+ assert "