diff --git a/README.md b/README.md index 2d9dbb9..62a910e 100644 --- a/README.md +++ b/README.md @@ -89,6 +89,11 @@ src/ ├── chapter_23/ # Security and Cryptography ├── chapter_24/ # Metaprogramming ├── chapter_25/ # Python Ecosystem and Best Practices +├── chapter_26/ # Async Programming +├── chapter_27/ # Multiprocessing and Parallelism +├── chapter_28/ # Command-Line Interfaces +├── chapter_29/ # Date, Time, and Scheduling +├── chapter_30/ # XML, HTML, and Data Formats └── ... tests/ ├── conftest.py # Shared pytest fixtures @@ -126,6 +131,11 @@ tests/ | 23 | Security and Cryptography | hashlib, hmac, secrets, input validation, secure coding | Done | | 24 | Metaprogramming | Dynamic attributes, class decorators, introspection, codegen | Done | | 25 | Python Ecosystem and Best Practices | Code style, linting, testing patterns, CI/CD, project org | Done | +| 26 | Async Programming | asyncio, async/await, tasks, gather, semaphores | Done | +| 27 | Multiprocessing and Parallelism | multiprocessing, Pool, ProcessPoolExecutor, IPC | Done | +| 28 | Command-Line Interfaces | argparse, subcommands, environment variables | Done | +| 29 | Date, Time, and Scheduling | datetime, zoneinfo, timedelta, calendar, formatting | Done | +| 30 | XML, HTML, and Data Formats | xml.etree, html.parser, configparser, struct | Done | ## Running Notebooks diff --git a/src/chapter_26/01_coroutines_and_event_loop.ipynb b/src/chapter_26/01_coroutines_and_event_loop.ipynb new file mode 100644 index 0000000..6f5f025 --- /dev/null +++ b/src/chapter_26/01_coroutines_and_event_loop.ipynb @@ -0,0 +1,559 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "a1b2c3d4", + "metadata": {}, + "source": [ + "# Chapter 26: Coroutines and the Event Loop\n", + "\n", + "This notebook covers the foundations of asynchronous programming in Python: `async`/`await` syntax, coroutines, the event loop, and `asyncio.run()`. These are the building blocks for writing concurrent, non-blocking Python code.\n", + "\n", + "## Key Concepts\n", + "- **Coroutines**: Functions defined with `async def` that can be suspended and resumed\n", + "- **`await`**: Yields control back to the event loop until a result is ready\n", + "- **Event loop**: The scheduler that runs coroutines and manages I/O\n", + "- **`asyncio.run()`**: The main entry point for running async code" + ] + }, + { + "cell_type": "markdown", + "id": "b2c3d4e5", + "metadata": {}, + "source": [ + "## Section 1: What Is a Coroutine?\n", + "\n", + "A coroutine is a function defined with `async def`. Calling it does **not** execute the body — it returns a coroutine object. The body only runs when the coroutine is awaited or scheduled on the event loop." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c3d4e5f6", + "metadata": {}, + "outputs": [], + "source": [ + "import asyncio\n", + "\n", + "\n", + "# Define a coroutine with async def\n", + "async def greet() -> str:\n", + " return \"hello\"\n", + "\n", + "\n", + "# Calling the coroutine function returns a coroutine object\n", + "coro = greet()\n", + "print(f\"Type: {type(coro)}\")\n", + "print(f\"Is coroutine: {asyncio.iscoroutine(coro)}\")\n", + "\n", + "# The coroutine must be run to get the result\n", + "result = asyncio.run(coro)\n", + "print(f\"Result: {result}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "d4e5f6a7", + "metadata": {}, + "outputs": [], + "source": [ + "# Coroutine objects vs coroutine functions\n", + "async def say_hi() -> str:\n", + " return \"hi\"\n", + "\n", + "\n", + "# The function itself is a coroutine function\n", + "print(f\"Is coroutine function: {asyncio.iscoroutinefunction(say_hi)}\")\n", + "\n", + "# Calling it produces a coroutine object\n", + "coro_obj = say_hi()\n", + "print(f\"Is coroutine object: {asyncio.iscoroutine(coro_obj)}\")\n", + "\n", + "# Always run the coroutine to avoid ResourceWarning\n", + "print(f\"Result: {asyncio.run(coro_obj)}\")" + ] + }, + { + "cell_type": "markdown", + "id": "e5f6a7b8", + "metadata": {}, + "source": [ + "## Section 2: The `await` Keyword\n", + "\n", + "`await` suspends the current coroutine until the awaited coroutine completes. This is how coroutines chain together — one coroutine can call another using `await`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f6a7b8c9", + "metadata": {}, + "outputs": [], + "source": [ + "# await chains coroutines together\n", + "async def add(a: int, b: int) -> int:\n", + " \"\"\"A simple async addition.\"\"\"\n", + " return a + b\n", + "\n", + "\n", + "async def main() -> int:\n", + " \"\"\"Main coroutine that awaits another.\"\"\"\n", + " result = await add(3, 4)\n", + " return result\n", + "\n", + "\n", + "print(f\"3 + 4 = {asyncio.run(main())}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a7b8c9d0", + "metadata": {}, + "outputs": [], + "source": [ + "# Multiple awaits in sequence\n", + "async def step_one() -> str:\n", + " return \"fetched data\"\n", + "\n", + "\n", + "async def step_two(data: str) -> str:\n", + " return f\"processed {data}\"\n", + "\n", + "\n", + "async def step_three(data: str) -> str:\n", + " return f\"saved {data}\"\n", + "\n", + "\n", + "async def pipeline() -> str:\n", + " \"\"\"Sequential async pipeline.\"\"\"\n", + " raw = await step_one()\n", + " processed = await step_two(raw)\n", + " result = await step_three(processed)\n", + " return result\n", + "\n", + "\n", + "print(asyncio.run(pipeline()))" + ] + }, + { + "cell_type": "markdown", + "id": "b8c9d0e1", + "metadata": {}, + "source": [ + "## Section 3: `asyncio.sleep()` and Yielding Control\n", + "\n", + "`asyncio.sleep()` is the async equivalent of `time.sleep()`. Unlike `time.sleep()`, it does **not** block the event loop — it suspends the current coroutine and allows other coroutines to run." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c9d0e1f2", + "metadata": {}, + "outputs": [], + "source": [ + "import time\n", + "\n", + "\n", + "async def delayed_value() -> str:\n", + " \"\"\"Simulate an async operation with sleep.\"\"\"\n", + " await asyncio.sleep(0) # Yield control, resume immediately\n", + " return \"done\"\n", + "\n", + "\n", + "result = asyncio.run(delayed_value())\n", + "print(f\"Result: {result}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "d0e1f2a3", + "metadata": {}, + "outputs": [], + "source": [ + "# Demonstrating non-blocking sleep with timing\n", + "async def task_a() -> str:\n", + " print(\"Task A: starting\")\n", + " await asyncio.sleep(0.1)\n", + " print(\"Task A: finished\")\n", + " return \"A\"\n", + "\n", + "\n", + "async def task_b() -> str:\n", + " print(\"Task B: starting\")\n", + " await asyncio.sleep(0.05)\n", + " print(\"Task B: finished\")\n", + " return \"B\"\n", + "\n", + "\n", + "async def run_both() -> None:\n", + " \"\"\"Run both tasks concurrently — they interleave.\"\"\"\n", + " start = time.perf_counter()\n", + " results = await asyncio.gather(task_a(), task_b())\n", + " elapsed = time.perf_counter() - start\n", + " print(f\"\\nResults: {results}\")\n", + " print(f\"Elapsed: {elapsed:.3f}s (concurrent, not 0.15s)\")\n", + "\n", + "\n", + "asyncio.run(run_both())" + ] + }, + { + "cell_type": "markdown", + "id": "e1f2a3b4", + "metadata": {}, + "source": [ + "## Section 4: `asyncio.run()` — The Entry Point\n", + "\n", + "`asyncio.run()` creates an event loop, runs the given coroutine until completion, and then closes the loop. It is the recommended way to start async code from synchronous code." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f2a3b4c5", + "metadata": {}, + "outputs": [], + "source": [ + "# asyncio.run() handles the event loop lifecycle\n", + "async def compute_squares(numbers: list[int]) -> list[int]:\n", + " \"\"\"Compute squares asynchronously.\"\"\"\n", + " results: list[int] = []\n", + " for n in numbers:\n", + " await asyncio.sleep(0) # Simulate async work\n", + " results.append(n ** 2)\n", + " return results\n", + "\n", + "\n", + "# asyncio.run() creates and manages the event loop\n", + "squares = asyncio.run(compute_squares([1, 2, 3, 4, 5]))\n", + "print(f\"Squares: {squares}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a3b4c5d6", + "metadata": {}, + "outputs": [], + "source": [ + "# asyncio.run() can only be called when no event loop is already running\n", + "# This is important in Jupyter notebooks, which have their own event loop\n", + "# In scripts, asyncio.run() is the standard entry point:\n", + "#\n", + "# async def main() -> None:\n", + "# ...\n", + "#\n", + "# if __name__ == \"__main__\":\n", + "# asyncio.run(main())\n", + "\n", + "\n", + "async def main() -> str:\n", + " \"\"\"Standard async main pattern.\"\"\"\n", + " greeting = await greet()\n", + " return f\"The greeting is: {greeting}\"\n", + "\n", + "\n", + "print(asyncio.run(main()))" + ] + }, + { + "cell_type": "markdown", + "id": "b4c5d6e7", + "metadata": {}, + "source": [ + "## Section 5: The Event Loop Under the Hood\n", + "\n", + "The event loop is the core of `asyncio`. It keeps track of all running coroutines and switches between them at `await` points. You rarely interact with the loop directly — `asyncio.run()` manages it for you." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c5d6e7f8", + "metadata": {}, + "outputs": [], + "source": [ + "# Observing the event loop from inside a coroutine\n", + "async def show_loop_info() -> None:\n", + " loop = asyncio.get_running_loop()\n", + " print(f\"Loop type: {type(loop).__name__}\")\n", + " print(f\"Loop running: {loop.is_running()}\")\n", + " print(f\"Loop closed: {loop.is_closed()}\")\n", + "\n", + "\n", + "asyncio.run(show_loop_info())" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "d6e7f8a9", + "metadata": {}, + "outputs": [], + "source": [ + "# Understanding execution order with the event loop\n", + "async def announce(label: str, delay: float) -> str:\n", + " print(f\" [{label}] starting\")\n", + " await asyncio.sleep(delay)\n", + " print(f\" [{label}] done after {delay}s\")\n", + " return label\n", + "\n", + "\n", + "async def demonstrate_order() -> None:\n", + " \"\"\"Show that the event loop interleaves coroutines.\"\"\"\n", + " print(\"Launching three coroutines concurrently:\")\n", + " results = await asyncio.gather(\n", + " announce(\"slow\", 0.15),\n", + " announce(\"medium\", 0.10),\n", + " announce(\"fast\", 0.05),\n", + " )\n", + " print(f\"\\nResults (in original order): {results}\")\n", + "\n", + "\n", + "asyncio.run(demonstrate_order())" + ] + }, + { + "cell_type": "markdown", + "id": "e7f8a9b0", + "metadata": {}, + "source": [ + "## Section 6: Coroutines vs Regular Functions\n", + "\n", + "Understanding when to use `async def` vs `def` is important. Coroutines are best for I/O-bound work — network calls, file I/O, database queries. CPU-bound work should use threads or processes instead." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f8a9b0c1", + "metadata": {}, + "outputs": [], + "source": [ + "# Regular function - blocks the thread\n", + "def sync_fetch(url: str) -> str:\n", + " \"\"\"Simulates a blocking I/O call.\"\"\"\n", + " time.sleep(0.05) # Blocks the entire thread\n", + " return f\"data from {url}\"\n", + "\n", + "\n", + "# Async coroutine - yields control during I/O\n", + "async def async_fetch(url: str) -> str:\n", + " \"\"\"Simulates a non-blocking I/O call.\"\"\"\n", + " await asyncio.sleep(0.05) # Yields control to the loop\n", + " return f\"data from {url}\"\n", + "\n", + "\n", + "# Compare sequential sync vs concurrent async\n", + "urls = [\"api/users\", \"api/products\", \"api/orders\"]\n", + "\n", + "# Synchronous: runs one after another\n", + "start = time.perf_counter()\n", + "sync_results = [sync_fetch(url) for url in urls]\n", + "sync_elapsed = time.perf_counter() - start\n", + "print(f\"Sync results: {sync_results}\")\n", + "print(f\"Sync elapsed: {sync_elapsed:.3f}s\")\n", + "\n", + "\n", + "# Asynchronous: runs concurrently\n", + "async def fetch_all(urls: list[str]) -> list[str]:\n", + " return await asyncio.gather(*(async_fetch(url) for url in urls))\n", + "\n", + "\n", + "start = time.perf_counter()\n", + "async_results = asyncio.run(fetch_all(urls))\n", + "async_elapsed = time.perf_counter() - start\n", + "print(f\"\\nAsync results: {async_results}\")\n", + "print(f\"Async elapsed: {async_elapsed:.3f}s\")\n", + "print(f\"\\nSpeedup: {sync_elapsed / async_elapsed:.1f}x\")" + ] + }, + { + "cell_type": "markdown", + "id": "a9b0c1d2", + "metadata": {}, + "source": [ + "## Section 7: Awaitable Objects\n", + "\n", + "Any object that implements the `__await__` method is awaitable. Coroutines, Tasks, and Futures are all awaitables. You can create your own awaitable objects too." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b0c1d2e3", + "metadata": {}, + "outputs": [], + "source": [ + "from collections.abc import Generator\n", + "\n", + "\n", + "# Custom awaitable object\n", + "class DelayedResult:\n", + " \"\"\"An awaitable that yields a result after a delay.\"\"\"\n", + "\n", + " def __init__(self, value: str, delay: float) -> None:\n", + " self.value = value\n", + " self.delay = delay\n", + "\n", + " def __await__(self) -> Generator[None, None, str]:\n", + " \"\"\"Make this object awaitable.\"\"\"\n", + " yield from asyncio.sleep(self.delay).__await__()\n", + " return self.value\n", + "\n", + "\n", + "async def use_custom_awaitable() -> None:\n", + " result = await DelayedResult(\"custom result\", 0.01)\n", + " print(f\"Got: {result}\")\n", + "\n", + "\n", + "asyncio.run(use_custom_awaitable())" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c1d2e3f4", + "metadata": {}, + "outputs": [], + "source": [ + "# Checking if something is awaitable\n", + "import inspect\n", + "\n", + "\n", + "async def example_coro() -> int:\n", + " return 42\n", + "\n", + "\n", + "coro = example_coro()\n", + "print(f\"Coroutine is awaitable: {inspect.isawaitable(coro)}\")\n", + "print(f\"Regular int is awaitable: {inspect.isawaitable(42)}\")\n", + "print(f\"String is awaitable: {inspect.isawaitable('hello')}\")\n", + "\n", + "# Clean up the coroutine\n", + "asyncio.run(coro)" + ] + }, + { + "cell_type": "markdown", + "id": "d2e3f4a5", + "metadata": {}, + "source": [ + "## Section 8: Error Handling in Coroutines\n", + "\n", + "Exceptions in coroutines work the same as in regular functions. An exception raised inside a coroutine propagates to the caller when the coroutine is awaited." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e3f4a5b6", + "metadata": {}, + "outputs": [], + "source": [ + "# Exceptions propagate through await\n", + "async def risky_operation() -> str:\n", + " await asyncio.sleep(0)\n", + " raise ValueError(\"something went wrong\")\n", + "\n", + "\n", + "async def safe_caller() -> str:\n", + " \"\"\"Handle errors from awaited coroutines.\"\"\"\n", + " try:\n", + " result = await risky_operation()\n", + " return result\n", + " except ValueError as e:\n", + " print(f\"Caught error: {e}\")\n", + " return \"fallback value\"\n", + "\n", + "\n", + "result = asyncio.run(safe_caller())\n", + "print(f\"Final result: {result}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f4a5b6c7", + "metadata": {}, + "outputs": [], + "source": [ + "# Handling errors with asyncio.gather using return_exceptions\n", + "async def succeed() -> str:\n", + " return \"success\"\n", + "\n", + "\n", + "async def fail() -> str:\n", + " raise RuntimeError(\"failure\")\n", + "\n", + "\n", + "async def mixed_results() -> None:\n", + " \"\"\"Gather with return_exceptions=True to capture errors.\"\"\"\n", + " results = await asyncio.gather(\n", + " succeed(),\n", + " fail(),\n", + " succeed(),\n", + " return_exceptions=True,\n", + " )\n", + " for i, result in enumerate(results):\n", + " if isinstance(result, Exception):\n", + " print(f\" Task {i}: ERROR - {result}\")\n", + " else:\n", + " print(f\" Task {i}: OK - {result}\")\n", + "\n", + "\n", + "asyncio.run(mixed_results())" + ] + }, + { + "cell_type": "markdown", + "id": "a5b6c7d8", + "metadata": {}, + "source": [ + "## Summary\n", + "\n", + "### Coroutines\n", + "- Defined with `async def`, return coroutine objects when called\n", + "- Must be awaited or scheduled on the event loop to execute\n", + "- Use `asyncio.iscoroutine()` and `asyncio.iscoroutinefunction()` for inspection\n", + "\n", + "### The `await` Keyword\n", + "- Suspends the current coroutine until the awaited result is ready\n", + "- Only valid inside `async def` functions\n", + "- Chains coroutines together in a sequential pipeline\n", + "\n", + "### The Event Loop\n", + "- Schedules and runs coroutines, switching at `await` points\n", + "- Enables concurrency without threads\n", + "- Access with `asyncio.get_running_loop()` from inside a coroutine\n", + "\n", + "### `asyncio.run()`\n", + "- Creates an event loop, runs the coroutine, and closes the loop\n", + "- The standard entry point for async programs\n", + "- Cannot be called when another event loop is already running\n", + "\n", + "### Key Patterns\n", + "- Use `asyncio.sleep()` instead of `time.sleep()` in async code\n", + "- Handle exceptions with `try`/`except` around `await` expressions\n", + "- Async is best for I/O-bound tasks, not CPU-bound computation" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "version": "3.13.0" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} \ No newline at end of file diff --git a/src/chapter_26/02_tasks_and_concurrency.ipynb b/src/chapter_26/02_tasks_and_concurrency.ipynb new file mode 100644 index 0000000..3d264e2 --- /dev/null +++ b/src/chapter_26/02_tasks_and_concurrency.ipynb @@ -0,0 +1,641 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "a1000001", + "metadata": {}, + "source": [ + "# Chapter 26: Tasks and Concurrency\n", + "\n", + "This notebook covers structured concurrency in `asyncio`: creating tasks, gathering results, waiting for completion, limiting concurrency with semaphores, and handling timeouts.\n", + "\n", + "## Key Concepts\n", + "- **`asyncio.create_task()`**: Schedule a coroutine to run concurrently as a Task\n", + "- **`asyncio.gather()`**: Run multiple awaitables concurrently, collect results in order\n", + "- **`asyncio.wait()`**: Wait for multiple tasks with flexible completion conditions\n", + "- **`asyncio.Semaphore`**: Limit the number of concurrent operations\n", + "- **`asyncio.wait_for()` / `asyncio.timeout()`**: Cancel operations that take too long" + ] + }, + { + "cell_type": "markdown", + "id": "a1000002", + "metadata": {}, + "source": [ + "## Section 1: `asyncio.create_task()`\n", + "\n", + "`create_task()` wraps a coroutine in a `Task` object and schedules it for execution on the event loop. The task starts running as soon as the event loop gets a chance (at the next `await` point)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a1000003", + "metadata": {}, + "outputs": [], + "source": [ + "import asyncio\n", + "import time\n", + "\n", + "\n", + "async def compute(x: int) -> int:\n", + " \"\"\"Simulate an async computation.\"\"\"\n", + " await asyncio.sleep(0)\n", + " return x * 2\n", + "\n", + "\n", + "async def main() -> None:\n", + " # create_task schedules the coroutine for execution\n", + " task = asyncio.create_task(compute(5))\n", + " print(f\"Task type: {type(task).__name__}\")\n", + " print(f\"Task done before await: {task.done()}\")\n", + "\n", + " # Await the task to get its result\n", + " result = await task\n", + " print(f\"Task done after await: {task.done()}\")\n", + " print(f\"Result: {result}\")\n", + "\n", + "\n", + "asyncio.run(main())" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a1000004", + "metadata": {}, + "outputs": [], + "source": [ + "# Tasks run concurrently — they don't block each other\n", + "async def worker(name: str, delay: float) -> str:\n", + " print(f\" {name}: starting (will take {delay}s)\")\n", + " await asyncio.sleep(delay)\n", + " print(f\" {name}: done\")\n", + " return f\"{name} result\"\n", + "\n", + "\n", + "async def main() -> None:\n", + " start = time.perf_counter()\n", + "\n", + " # Create tasks — they start running immediately\n", + " task1 = asyncio.create_task(worker(\"Task-1\", 0.10))\n", + " task2 = asyncio.create_task(worker(\"Task-2\", 0.05))\n", + " task3 = asyncio.create_task(worker(\"Task-3\", 0.08))\n", + "\n", + " # Await each task to get its result\n", + " r1 = await task1\n", + " r2 = await task2\n", + " r3 = await task3\n", + "\n", + " elapsed = time.perf_counter() - start\n", + " print(f\"\\nResults: {[r1, r2, r3]}\")\n", + " print(f\"Elapsed: {elapsed:.3f}s (concurrent, not 0.23s)\")\n", + "\n", + "\n", + "asyncio.run(main())" + ] + }, + { + "cell_type": "markdown", + "id": "a1000005", + "metadata": {}, + "source": [ + "## Section 2: `asyncio.gather()`\n", + "\n", + "`gather()` runs multiple awaitables concurrently and returns their results as a list. The results are always in the **same order** as the arguments, regardless of which coroutine finishes first." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a1000006", + "metadata": {}, + "outputs": [], + "source": [ + "# gather runs coroutines concurrently and collects results\n", + "results: list[int] = []\n", + "\n", + "\n", + "async def append_value(val: int) -> int:\n", + " await asyncio.sleep(0)\n", + " results.append(val)\n", + " return val\n", + "\n", + "\n", + "async def main() -> list[int]:\n", + " return await asyncio.gather(\n", + " append_value(1),\n", + " append_value(2),\n", + " append_value(3),\n", + " )\n", + "\n", + "\n", + "gathered = asyncio.run(main())\n", + "print(f\"Gathered results: {gathered}\")\n", + "print(f\"Execution order: {sorted(results)}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a1000007", + "metadata": {}, + "outputs": [], + "source": [ + "# gather preserves argument order, not completion order\n", + "async def delayed(val: int, delay: float) -> int:\n", + " \"\"\"Return a value after a delay.\"\"\"\n", + " await asyncio.sleep(delay)\n", + " return val\n", + "\n", + "\n", + "async def main() -> None:\n", + " # Task 3 finishes first, but appears last in results\n", + " results = await asyncio.gather(\n", + " delayed(1, 0.02), # Finishes last\n", + " delayed(2, 0.01), # Finishes second\n", + " delayed(3, 0.00), # Finishes first\n", + " )\n", + " print(f\"Results (argument order): {results}\")\n", + " # Always [1, 2, 3] regardless of completion order\n", + "\n", + "\n", + "asyncio.run(main())" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a1000008", + "metadata": {}, + "outputs": [], + "source": [ + "# gather with dynamic number of coroutines\n", + "async def fetch_page(page_num: int) -> dict[str, int | str]:\n", + " \"\"\"Simulate fetching a page of data.\"\"\"\n", + " await asyncio.sleep(0.01)\n", + " return {\"page\": page_num, \"status\": \"ok\"}\n", + "\n", + "\n", + "async def fetch_all_pages(total: int) -> list[dict[str, int | str]]:\n", + " \"\"\"Fetch multiple pages concurrently.\"\"\"\n", + " # Use unpacking to pass a generator of coroutines\n", + " return await asyncio.gather(\n", + " *(fetch_page(i) for i in range(1, total + 1))\n", + " )\n", + "\n", + "\n", + "pages = asyncio.run(fetch_all_pages(5))\n", + "for page in pages:\n", + " print(f\" Page {page['page']}: {page['status']}\")" + ] + }, + { + "cell_type": "markdown", + "id": "a1000009", + "metadata": {}, + "source": [ + "## Section 3: Error Handling with `gather()`\n", + "\n", + "By default, if any gathered coroutine raises an exception, `gather()` re-raises it. Use `return_exceptions=True` to capture exceptions as results instead." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a1000010", + "metadata": {}, + "outputs": [], + "source": [ + "async def maybe_fail(n: int) -> str:\n", + " \"\"\"Succeed on even numbers, fail on odd.\"\"\"\n", + " await asyncio.sleep(0)\n", + " if n % 2 != 0:\n", + " raise ValueError(f\"odd number: {n}\")\n", + " return f\"ok-{n}\"\n", + "\n", + "\n", + "async def main() -> None:\n", + " # return_exceptions=True captures errors instead of raising\n", + " results = await asyncio.gather(\n", + " maybe_fail(1),\n", + " maybe_fail(2),\n", + " maybe_fail(3),\n", + " maybe_fail(4),\n", + " return_exceptions=True,\n", + " )\n", + " for i, result in enumerate(results):\n", + " if isinstance(result, Exception):\n", + " print(f\" Task {i}: FAILED - {result}\")\n", + " else:\n", + " print(f\" Task {i}: OK - {result}\")\n", + "\n", + "\n", + "asyncio.run(main())" + ] + }, + { + "cell_type": "markdown", + "id": "a1000011", + "metadata": {}, + "source": [ + "## Section 4: `asyncio.wait()`\n", + "\n", + "`wait()` gives you more control than `gather()`. It returns two sets: `done` and `pending`. You can wait for the first task to complete, all tasks, or the first exception." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a1000012", + "metadata": {}, + "outputs": [], + "source": [ + "# wait() with FIRST_COMPLETED — process results as they arrive\n", + "async def timed_task(name: str, delay: float) -> str:\n", + " await asyncio.sleep(delay)\n", + " return f\"{name} ({delay}s)\"\n", + "\n", + "\n", + "async def main() -> None:\n", + " tasks: set[asyncio.Task[str]] = {\n", + " asyncio.create_task(timed_task(\"fast\", 0.01)),\n", + " asyncio.create_task(timed_task(\"medium\", 0.05)),\n", + " asyncio.create_task(timed_task(\"slow\", 0.10)),\n", + " }\n", + "\n", + " print(\"Processing tasks as they complete:\")\n", + " while tasks:\n", + " done, tasks = await asyncio.wait(\n", + " tasks, return_when=asyncio.FIRST_COMPLETED\n", + " )\n", + " for task in done:\n", + " print(f\" Completed: {task.result()}\")\n", + "\n", + "\n", + "asyncio.run(main())" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a1000013", + "metadata": {}, + "outputs": [], + "source": [ + "# wait() with ALL_COMPLETED (the default)\n", + "async def main() -> None:\n", + " tasks = [\n", + " asyncio.create_task(timed_task(\"A\", 0.03)),\n", + " asyncio.create_task(timed_task(\"B\", 0.01)),\n", + " asyncio.create_task(timed_task(\"C\", 0.02)),\n", + " ]\n", + "\n", + " done, pending = await asyncio.wait(tasks, return_when=asyncio.ALL_COMPLETED)\n", + " print(f\"Done: {len(done)}, Pending: {len(pending)}\")\n", + " for task in done:\n", + " print(f\" Result: {task.result()}\")\n", + "\n", + "\n", + "asyncio.run(main())" + ] + }, + { + "cell_type": "markdown", + "id": "a1000014", + "metadata": {}, + "source": [ + "## Section 5: Task Cancellation\n", + "\n", + "Tasks can be cancelled by calling `task.cancel()`. The cancelled task will raise `asyncio.CancelledError` at its next `await` point." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a1000015", + "metadata": {}, + "outputs": [], + "source": [ + "async def long_running() -> str:\n", + " \"\"\"A task that takes a while.\"\"\"\n", + " try:\n", + " print(\" Long task: working...\")\n", + " await asyncio.sleep(10) # Will be cancelled before this finishes\n", + " return \"completed\"\n", + " except asyncio.CancelledError:\n", + " print(\" Long task: cancelled! Cleaning up...\")\n", + " raise # Re-raise to propagate cancellation\n", + "\n", + "\n", + "async def main() -> None:\n", + " task = asyncio.create_task(long_running())\n", + "\n", + " # Let the task start\n", + " await asyncio.sleep(0.01)\n", + "\n", + " # Cancel the task\n", + " task.cancel()\n", + "\n", + " try:\n", + " await task\n", + " except asyncio.CancelledError:\n", + " print(f\" Main: task was cancelled\")\n", + "\n", + " print(f\" Task cancelled: {task.cancelled()}\")\n", + "\n", + "\n", + "asyncio.run(main())" + ] + }, + { + "cell_type": "markdown", + "id": "a1000016", + "metadata": {}, + "source": [ + "## Section 6: Timeouts with `asyncio.wait_for()`\n", + "\n", + "`wait_for()` wraps a coroutine with a timeout. If the coroutine does not complete within the timeout, it is cancelled and `asyncio.TimeoutError` is raised." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a1000017", + "metadata": {}, + "outputs": [], + "source": [ + "async def slow_operation() -> str:\n", + " \"\"\"An operation that takes too long.\"\"\"\n", + " await asyncio.sleep(5)\n", + " return \"finally done\"\n", + "\n", + "\n", + "async def main() -> None:\n", + " # Set a timeout of 0.1 seconds\n", + " try:\n", + " result = await asyncio.wait_for(slow_operation(), timeout=0.1)\n", + " print(f\"Result: {result}\")\n", + " except asyncio.TimeoutError:\n", + " print(\"Operation timed out after 0.1s\")\n", + "\n", + " # A fast operation completes within the timeout\n", + " async def fast_operation() -> str:\n", + " await asyncio.sleep(0.01)\n", + " return \"quick result\"\n", + "\n", + " result = await asyncio.wait_for(fast_operation(), timeout=1.0)\n", + " print(f\"Fast result: {result}\")\n", + "\n", + "\n", + "asyncio.run(main())" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a1000018", + "metadata": {}, + "outputs": [], + "source": [ + "# Timeout with retry pattern\n", + "async def unreliable_fetch(attempt: int) -> str:\n", + " \"\"\"Simulates a fetch that sometimes times out.\"\"\"\n", + " delay = 0.5 if attempt < 3 else 0.01 # Fast on third attempt\n", + " await asyncio.sleep(delay)\n", + " return f\"data (attempt {attempt})\"\n", + "\n", + "\n", + "async def fetch_with_retries(\n", + " max_retries: int = 3,\n", + " timeout: float = 0.1,\n", + ") -> str:\n", + " \"\"\"Retry a fetch operation with timeout.\"\"\"\n", + " for attempt in range(1, max_retries + 1):\n", + " try:\n", + " result = await asyncio.wait_for(\n", + " unreliable_fetch(attempt), timeout=timeout\n", + " )\n", + " print(f\" Attempt {attempt}: success\")\n", + " return result\n", + " except asyncio.TimeoutError:\n", + " print(f\" Attempt {attempt}: timed out\")\n", + " raise RuntimeError(f\"Failed after {max_retries} attempts\")\n", + "\n", + "\n", + "result = asyncio.run(fetch_with_retries())\n", + "print(f\"Final result: {result}\")" + ] + }, + { + "cell_type": "markdown", + "id": "a1000019", + "metadata": {}, + "source": [ + "## Section 7: Semaphores — Limiting Concurrency\n", + "\n", + "`asyncio.Semaphore` limits the number of coroutines that can access a resource concurrently. This is essential for rate limiting, connection pooling, and preventing resource exhaustion." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a1000020", + "metadata": {}, + "outputs": [], + "source": [ + "# Semaphore limits concurrent access\n", + "max_concurrent: int = 0\n", + "current: int = 0\n", + "\n", + "\n", + "async def limited_task(sem: asyncio.Semaphore, task_id: int) -> None:\n", + " \"\"\"A task that respects a semaphore limit.\"\"\"\n", + " global max_concurrent, current\n", + " async with sem:\n", + " current += 1\n", + " if current > max_concurrent:\n", + " max_concurrent = current\n", + " print(f\" Task {task_id}: running (concurrent: {current})\")\n", + " await asyncio.sleep(0.01)\n", + " current -= 1\n", + "\n", + "\n", + "async def main() -> None:\n", + " global max_concurrent, current\n", + " max_concurrent = 0\n", + " current = 0\n", + "\n", + " # Allow at most 2 tasks to run concurrently\n", + " sem = asyncio.Semaphore(2)\n", + " await asyncio.gather(*(limited_task(sem, i) for i in range(5)))\n", + " print(f\"\\nMax concurrent tasks: {max_concurrent} (limit was 2)\")\n", + "\n", + "\n", + "asyncio.run(main())" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a1000021", + "metadata": {}, + "outputs": [], + "source": [ + "# Practical example: rate-limited API fetcher\n", + "async def fetch_url(url: str, sem: asyncio.Semaphore) -> dict[str, str]:\n", + " \"\"\"Fetch a URL with rate limiting.\"\"\"\n", + " async with sem:\n", + " print(f\" Fetching {url}...\")\n", + " await asyncio.sleep(0.02) # Simulate network I/O\n", + " return {\"url\": url, \"status\": \"200\"}\n", + "\n", + "\n", + "async def main() -> None:\n", + " urls = [f\"https://api.example.com/item/{i}\" for i in range(6)]\n", + "\n", + " # Limit to 3 concurrent requests\n", + " sem = asyncio.Semaphore(3)\n", + "\n", + " start = time.perf_counter()\n", + " results = await asyncio.gather(*(fetch_url(url, sem) for url in urls))\n", + " elapsed = time.perf_counter() - start\n", + "\n", + " print(f\"\\nFetched {len(results)} URLs in {elapsed:.3f}s\")\n", + " print(f\"With concurrency limit of 3\")\n", + "\n", + "\n", + "asyncio.run(main())" + ] + }, + { + "cell_type": "markdown", + "id": "a1000022", + "metadata": {}, + "source": [ + "## Section 8: Task Groups (Python 3.11+)\n", + "\n", + "`asyncio.TaskGroup` provides structured concurrency — if any task in the group fails, all other tasks are cancelled. This is safer than `gather()` for error-prone operations." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a1000023", + "metadata": {}, + "outputs": [], + "source": [ + "# TaskGroup for structured concurrency\n", + "async def process_item(item: int) -> int:\n", + " \"\"\"Process a single item.\"\"\"\n", + " await asyncio.sleep(0.01)\n", + " return item * 10\n", + "\n", + "\n", + "async def main() -> None:\n", + " results: list[int] = []\n", + "\n", + " async with asyncio.TaskGroup() as tg:\n", + " tasks = [\n", + " tg.create_task(process_item(i))\n", + " for i in range(5)\n", + " ]\n", + "\n", + " # All tasks are guaranteed complete here\n", + " results = [t.result() for t in tasks]\n", + " print(f\"Results: {results}\")\n", + "\n", + "\n", + "asyncio.run(main())" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a1000024", + "metadata": {}, + "outputs": [], + "source": [ + "# TaskGroup handles errors by cancelling remaining tasks\n", + "async def might_fail(n: int) -> int:\n", + " await asyncio.sleep(0.01)\n", + " if n == 3:\n", + " raise ValueError(f\"Cannot process {n}\")\n", + " return n * 10\n", + "\n", + "\n", + "async def main() -> None:\n", + " try:\n", + " async with asyncio.TaskGroup() as tg:\n", + " tasks = [tg.create_task(might_fail(i)) for i in range(5)]\n", + " except* ValueError as eg:\n", + " print(f\"Caught {len(eg.exceptions)} error(s):\")\n", + " for exc in eg.exceptions:\n", + " print(f\" - {exc}\")\n", + "\n", + " # Check which tasks completed vs were cancelled\n", + " for i, t in enumerate(tasks):\n", + " if t.cancelled():\n", + " print(f\" Task {i}: cancelled\")\n", + " elif t.exception():\n", + " print(f\" Task {i}: failed\")\n", + " else:\n", + " print(f\" Task {i}: result = {t.result()}\")\n", + "\n", + "\n", + "asyncio.run(main())" + ] + }, + { + "cell_type": "markdown", + "id": "a1000025", + "metadata": {}, + "source": [ + "## Summary\n", + "\n", + "### `asyncio.create_task()`\n", + "- Wraps a coroutine in a `Task` and schedules it on the event loop\n", + "- Tasks start running at the next `await` point\n", + "- Returns a `Task` object you can await, cancel, or inspect\n", + "\n", + "### `asyncio.gather()`\n", + "- Runs multiple awaitables concurrently\n", + "- Returns results in **argument order**, not completion order\n", + "- Use `return_exceptions=True` to capture errors instead of raising\n", + "\n", + "### `asyncio.wait()`\n", + "- Returns `(done, pending)` sets of tasks\n", + "- Supports `FIRST_COMPLETED`, `ALL_COMPLETED`, `FIRST_EXCEPTION`\n", + "- Good for processing results as they arrive\n", + "\n", + "### Timeouts\n", + "- `asyncio.wait_for(coro, timeout)` cancels after timeout seconds\n", + "- Raises `asyncio.TimeoutError` on expiration\n", + "- Combine with retry logic for resilient operations\n", + "\n", + "### Semaphores\n", + "- `asyncio.Semaphore(n)` limits concurrency to `n` coroutines\n", + "- Use `async with sem:` to acquire and release automatically\n", + "- Essential for rate limiting and resource management\n", + "\n", + "### Task Groups (3.11+)\n", + "- `asyncio.TaskGroup` provides structured concurrency\n", + "- If any task fails, all others are cancelled\n", + "- Use `except*` to handle `ExceptionGroup` errors" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "version": "3.13.0" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} \ No newline at end of file diff --git a/src/chapter_26/03_async_patterns.ipynb b/src/chapter_26/03_async_patterns.ipynb new file mode 100644 index 0000000..c83375b --- /dev/null +++ b/src/chapter_26/03_async_patterns.ipynb @@ -0,0 +1,778 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "b1000001", + "metadata": {}, + "source": [ + "# Chapter 26: Async Patterns\n", + "\n", + "This notebook covers advanced async patterns: async iterators, async generators, async context managers, and the producer-consumer pattern. These patterns enable elegant, composable asynchronous code.\n", + "\n", + "## Key Concepts\n", + "- **Async iterators**: Objects with `__aiter__` and `__anext__` for async `for` loops\n", + "- **Async generators**: Functions using `async def` with `yield` for lazy async sequences\n", + "- **Async context managers**: Objects with `__aenter__` and `__aexit__` for `async with`\n", + "- **Producer-consumer**: A common concurrency pattern using `asyncio.Queue`" + ] + }, + { + "cell_type": "markdown", + "id": "b1000002", + "metadata": {}, + "source": [ + "## Section 1: Async Iterators\n", + "\n", + "An async iterator implements `__aiter__()` and `__anext__()`. It is consumed with `async for` and raises `StopAsyncIteration` when exhausted. This is the async equivalent of the regular iteration protocol." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b1000003", + "metadata": {}, + "outputs": [], + "source": [ + "import asyncio\n", + "\n", + "\n", + "class AsyncRange:\n", + " \"\"\"An async iterator that yields numbers from 0 to stop-1.\"\"\"\n", + "\n", + " def __init__(self, stop: int) -> None:\n", + " self.stop = stop\n", + " self.current = 0\n", + "\n", + " def __aiter__(self):\n", + " return self\n", + "\n", + " async def __anext__(self) -> int:\n", + " if self.current >= self.stop:\n", + " raise StopAsyncIteration\n", + " value = self.current\n", + " self.current += 1\n", + " return value\n", + "\n", + "\n", + "async def main() -> None:\n", + " # Use async for to iterate\n", + " print(\"Async for loop:\")\n", + " async for i in AsyncRange(5):\n", + " print(f\" {i}\")\n", + "\n", + "\n", + "asyncio.run(main())" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b1000004", + "metadata": {}, + "outputs": [], + "source": [ + "# Async comprehension with async iterators\n", + "async def collect() -> list[int]:\n", + " \"\"\"Collect async iterator values into a list.\"\"\"\n", + " return [i async for i in AsyncRange(4)]\n", + "\n", + "\n", + "result = asyncio.run(collect())\n", + "print(f\"Collected: {result}\")\n", + "\n", + "\n", + "# Async comprehension with filtering\n", + "async def collect_even() -> list[int]:\n", + " return [i async for i in AsyncRange(10) if i % 2 == 0]\n", + "\n", + "\n", + "evens = asyncio.run(collect_even())\n", + "print(f\"Even numbers: {evens}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b1000005", + "metadata": {}, + "outputs": [], + "source": [ + "# A more realistic async iterator: paginated data fetcher\n", + "class AsyncPaginator:\n", + " \"\"\"Iterate over pages of data asynchronously.\"\"\"\n", + "\n", + " def __init__(self, total_items: int, page_size: int) -> None:\n", + " self.total_items = total_items\n", + " self.page_size = page_size\n", + " self.offset = 0\n", + "\n", + " def __aiter__(self):\n", + " return self\n", + "\n", + " async def __anext__(self) -> list[int]:\n", + " if self.offset >= self.total_items:\n", + " raise StopAsyncIteration\n", + " # Simulate fetching a page from a database\n", + " await asyncio.sleep(0)\n", + " end = min(self.offset + self.page_size, self.total_items)\n", + " page = list(range(self.offset, end))\n", + " self.offset = end\n", + " return page\n", + "\n", + "\n", + "async def main() -> None:\n", + " print(\"Paginated results (page size 3):\")\n", + " async for page in AsyncPaginator(total_items=10, page_size=3):\n", + " print(f\" Page: {page}\")\n", + "\n", + "\n", + "asyncio.run(main())" + ] + }, + { + "cell_type": "markdown", + "id": "b1000006", + "metadata": {}, + "source": [ + "## Section 2: Async Generators\n", + "\n", + "Async generators are the easiest way to create async iterators. They use `async def` with `yield` and can `await` inside the function body. Each `yield` suspends the generator until the next value is requested." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b1000007", + "metadata": {}, + "outputs": [], + "source": [ + "# Simple async generator\n", + "async def async_range(stop: int):\n", + " \"\"\"Yield numbers 0..stop-1 with an async pause.\"\"\"\n", + " for i in range(stop):\n", + " await asyncio.sleep(0) # Simulate async work\n", + " yield i\n", + "\n", + "\n", + "async def main() -> None:\n", + " # Use async for to consume the generator\n", + " values: list[int] = []\n", + " async for val in async_range(5):\n", + " values.append(val)\n", + " print(f\"Values: {values}\")\n", + "\n", + " # Or use an async comprehension\n", + " collected = [i async for i in async_range(3)]\n", + " print(f\"Collected: {collected}\")\n", + "\n", + "\n", + "asyncio.run(main())" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b1000008", + "metadata": {}, + "outputs": [], + "source": [ + "# Async generator with filtering and transformation\n", + "from typing import AsyncGenerator\n", + "\n", + "\n", + "async def fetch_items(count: int) -> AsyncGenerator[dict[str, int | str], None]:\n", + " \"\"\"Simulate fetching items from an async data source.\"\"\"\n", + " for i in range(count):\n", + " await asyncio.sleep(0)\n", + " yield {\"id\": i, \"name\": f\"item-{i}\", \"value\": i * 10}\n", + "\n", + "\n", + "async def filter_valuable(\n", + " items: AsyncGenerator[dict[str, int | str], None],\n", + " min_value: int,\n", + ") -> AsyncGenerator[dict[str, int | str], None]:\n", + " \"\"\"Filter items by minimum value.\"\"\"\n", + " async for item in items:\n", + " if item[\"value\"] >= min_value:\n", + " yield item\n", + "\n", + "\n", + "async def main() -> None:\n", + " print(\"Items with value >= 30:\")\n", + " async for item in filter_valuable(fetch_items(8), min_value=30):\n", + " print(f\" {item}\")\n", + "\n", + "\n", + "asyncio.run(main())" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b1000009", + "metadata": {}, + "outputs": [], + "source": [ + "# Async generator pipeline: compose multiple transformations\n", + "async def numbers(n: int) -> AsyncGenerator[int, None]:\n", + " \"\"\"Generate numbers 0..n-1.\"\"\"\n", + " for i in range(n):\n", + " await asyncio.sleep(0)\n", + " yield i\n", + "\n", + "\n", + "async def squared(source: AsyncGenerator[int, None]) -> AsyncGenerator[int, None]:\n", + " \"\"\"Square each value.\"\"\"\n", + " async for val in source:\n", + " yield val ** 2\n", + "\n", + "\n", + "async def only_even(source: AsyncGenerator[int, None]) -> AsyncGenerator[int, None]:\n", + " \"\"\"Keep only even values.\"\"\"\n", + " async for val in source:\n", + " if val % 2 == 0:\n", + " yield val\n", + "\n", + "\n", + "async def main() -> None:\n", + " # Build a pipeline: numbers -> squared -> only_even\n", + " pipeline = only_even(squared(numbers(8)))\n", + " results = [val async for val in pipeline]\n", + " print(f\"Even squares of 0..7: {results}\")\n", + "\n", + "\n", + "asyncio.run(main())" + ] + }, + { + "cell_type": "markdown", + "id": "b1000010", + "metadata": {}, + "source": [ + "## Section 3: Async Context Managers\n", + "\n", + "Async context managers implement `__aenter__` and `__aexit__` and are used with `async with`. They manage resources that require async setup and teardown, like database connections or file handles." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b1000011", + "metadata": {}, + "outputs": [], + "source": [ + "# Class-based async context manager\n", + "class AsyncResource:\n", + " \"\"\"A resource with async setup and teardown.\"\"\"\n", + "\n", + " async def __aenter__(self):\n", + " print(\" Entering: acquiring resource...\")\n", + " await asyncio.sleep(0) # Simulate async setup\n", + " return self\n", + "\n", + " async def __aexit__(self, *args: object) -> None:\n", + " print(\" Exiting: releasing resource...\")\n", + " await asyncio.sleep(0) # Simulate async teardown\n", + "\n", + " async def do_work(self) -> str:\n", + " return \"work done\"\n", + "\n", + "\n", + "async def main() -> None:\n", + " async with AsyncResource() as resource:\n", + " result = await resource.do_work()\n", + " print(f\" Inside: {result}\")\n", + " print(\" After: resource released\")\n", + "\n", + "\n", + "asyncio.run(main())" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b1000012", + "metadata": {}, + "outputs": [], + "source": [ + "# Tracking lifecycle events\n", + "events: list[str] = []\n", + "\n", + "\n", + "class TrackedResource:\n", + " \"\"\"An async context manager that records lifecycle events.\"\"\"\n", + "\n", + " async def __aenter__(self):\n", + " events.append(\"enter\")\n", + " return self\n", + "\n", + " async def __aexit__(self, *args: object) -> None:\n", + " events.append(\"exit\")\n", + "\n", + "\n", + "async def main() -> None:\n", + " async with TrackedResource():\n", + " events.append(\"use\")\n", + "\n", + "\n", + "asyncio.run(main())\n", + "print(f\"Events: {events}\")\n", + "assert events == [\"enter\", \"use\", \"exit\"]" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b1000013", + "metadata": {}, + "outputs": [], + "source": [ + "# Function-based async context manager using contextlib\n", + "from contextlib import asynccontextmanager\n", + "from typing import AsyncGenerator\n", + "\n", + "\n", + "@asynccontextmanager\n", + "async def async_timer(label: str) -> AsyncGenerator[None, None]:\n", + " \"\"\"Time an async block of code.\"\"\"\n", + " import time\n", + " start = time.perf_counter()\n", + " print(f\" [{label}] Starting...\")\n", + " try:\n", + " yield\n", + " finally:\n", + " elapsed = time.perf_counter() - start\n", + " print(f\" [{label}] Done in {elapsed:.4f}s\")\n", + "\n", + "\n", + "async def main() -> None:\n", + " async with async_timer(\"fetch\"):\n", + " await asyncio.sleep(0.05)\n", + "\n", + " async with async_timer(\"compute\"):\n", + " await asyncio.sleep(0.02)\n", + "\n", + "\n", + "asyncio.run(main())" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b1000014", + "metadata": {}, + "outputs": [], + "source": [ + "# Practical example: async database connection mock\n", + "@asynccontextmanager\n", + "async def async_db_connection(\n", + " dsn: str,\n", + ") -> AsyncGenerator[\"MockConnection\", None]:\n", + " \"\"\"Simulate an async database connection.\"\"\"\n", + " conn = MockConnection(dsn)\n", + " await conn.connect()\n", + " try:\n", + " yield conn\n", + " finally:\n", + " await conn.close()\n", + "\n", + "\n", + "class MockConnection:\n", + " \"\"\"A mock async database connection.\"\"\"\n", + "\n", + " def __init__(self, dsn: str) -> None:\n", + " self.dsn = dsn\n", + " self.connected = False\n", + "\n", + " async def connect(self) -> None:\n", + " await asyncio.sleep(0)\n", + " self.connected = True\n", + " print(f\" Connected to {self.dsn}\")\n", + "\n", + " async def close(self) -> None:\n", + " await asyncio.sleep(0)\n", + " self.connected = False\n", + " print(f\" Disconnected from {self.dsn}\")\n", + "\n", + " async def query(self, sql: str) -> list[dict[str, str]]:\n", + " await asyncio.sleep(0)\n", + " return [{\"result\": f\"data from: {sql}\"}]\n", + "\n", + "\n", + "async def main() -> None:\n", + " async with async_db_connection(\"postgres://localhost/mydb\") as conn:\n", + " rows = await conn.query(\"SELECT * FROM users\")\n", + " print(f\" Query result: {rows}\")\n", + " # Connection is automatically closed here\n", + "\n", + "\n", + "asyncio.run(main())" + ] + }, + { + "cell_type": "markdown", + "id": "b1000015", + "metadata": {}, + "source": [ + "## Section 4: Producer-Consumer Pattern\n", + "\n", + "The producer-consumer pattern uses `asyncio.Queue` to decouple producers (which generate work) from consumers (which process it). This is a foundational pattern for building async pipelines." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b1000016", + "metadata": {}, + "outputs": [], + "source": [ + "# Basic producer-consumer with asyncio.Queue\n", + "async def producer(\n", + " queue: asyncio.Queue[int],\n", + " count: int,\n", + ") -> None:\n", + " \"\"\"Produce items and put them on the queue.\"\"\"\n", + " for i in range(count):\n", + " await asyncio.sleep(0.01) # Simulate production time\n", + " await queue.put(i)\n", + " print(f\" Produced: {i}\")\n", + "\n", + "\n", + "async def consumer(\n", + " queue: asyncio.Queue[int],\n", + " name: str,\n", + " results: list[int],\n", + ") -> None:\n", + " \"\"\"Consume items from the queue.\"\"\"\n", + " while True:\n", + " item = await queue.get()\n", + " await asyncio.sleep(0.02) # Simulate processing time\n", + " results.append(item * 10)\n", + " print(f\" {name} processed: {item} -> {item * 10}\")\n", + " queue.task_done()\n", + "\n", + "\n", + "async def main() -> None:\n", + " queue: asyncio.Queue[int] = asyncio.Queue(maxsize=5)\n", + " results: list[int] = []\n", + "\n", + " # Start 2 consumers\n", + " consumers = [\n", + " asyncio.create_task(consumer(queue, f\"Worker-{i}\", results))\n", + " for i in range(2)\n", + " ]\n", + "\n", + " # Run producer\n", + " await producer(queue, count=6)\n", + "\n", + " # Wait for all items to be processed\n", + " await queue.join()\n", + "\n", + " # Cancel consumers (they loop forever)\n", + " for c in consumers:\n", + " c.cancel()\n", + "\n", + " print(f\"\\nAll results: {sorted(results)}\")\n", + "\n", + "\n", + "asyncio.run(main())" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b1000017", + "metadata": {}, + "outputs": [], + "source": [ + "# Producer-consumer with sentinel value for graceful shutdown\n", + "SENTINEL: None = None\n", + "\n", + "\n", + "async def data_producer(\n", + " queue: asyncio.Queue[int | None],\n", + " items: list[int],\n", + " num_consumers: int,\n", + ") -> None:\n", + " \"\"\"Produce data, then send sentinel to each consumer.\"\"\"\n", + " for item in items:\n", + " await asyncio.sleep(0)\n", + " await queue.put(item)\n", + "\n", + " # Signal each consumer to stop\n", + " for _ in range(num_consumers):\n", + " await queue.put(SENTINEL)\n", + "\n", + "\n", + "async def data_consumer(\n", + " queue: asyncio.Queue[int | None],\n", + " name: str,\n", + ") -> list[int]:\n", + " \"\"\"Consume until sentinel is received.\"\"\"\n", + " processed: list[int] = []\n", + " while True:\n", + " item = await queue.get()\n", + " if item is SENTINEL:\n", + " print(f\" {name}: received shutdown signal\")\n", + " break\n", + " processed.append(item ** 2)\n", + " await asyncio.sleep(0)\n", + " return processed\n", + "\n", + "\n", + "async def main() -> None:\n", + " queue: asyncio.Queue[int | None] = asyncio.Queue()\n", + " num_consumers = 2\n", + "\n", + " # Start producer and consumers\n", + " producer_task = asyncio.create_task(\n", + " data_producer(queue, [1, 2, 3, 4, 5, 6], num_consumers)\n", + " )\n", + " consumer_tasks = [\n", + " asyncio.create_task(data_consumer(queue, f\"Consumer-{i}\"))\n", + " for i in range(num_consumers)\n", + " ]\n", + "\n", + " # Wait for all to finish\n", + " await producer_task\n", + " all_results = await asyncio.gather(*consumer_tasks)\n", + "\n", + " # Combine results from all consumers\n", + " combined = sorted(r for results in all_results for r in results)\n", + " print(f\"\\nAll squared values: {combined}\")\n", + "\n", + "\n", + "asyncio.run(main())" + ] + }, + { + "cell_type": "markdown", + "id": "b1000018", + "metadata": {}, + "source": [ + "## Section 5: Combining Patterns\n", + "\n", + "Real-world async code combines these patterns. Here we build a mini pipeline that uses async generators, context managers, and queues together." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b1000019", + "metadata": {}, + "outputs": [], + "source": [ + "# Combining async generator + context manager + queue\n", + "from typing import AsyncGenerator\n", + "\n", + "\n", + "async def event_stream(count: int) -> AsyncGenerator[dict[str, int | str], None]:\n", + " \"\"\"Simulate an async event stream.\"\"\"\n", + " for i in range(count):\n", + " await asyncio.sleep(0)\n", + " yield {\"id\": i, \"type\": \"click\" if i % 2 == 0 else \"scroll\"}\n", + "\n", + "\n", + "@asynccontextmanager\n", + "async def event_processor(\n", + " name: str,\n", + ") -> AsyncGenerator[list[dict[str, int | str]], None]:\n", + " \"\"\"Context manager for collecting processed events.\"\"\"\n", + " processed: list[dict[str, int | str]] = []\n", + " print(f\" [{name}] Processor started\")\n", + " try:\n", + " yield processed\n", + " finally:\n", + " print(f\" [{name}] Processor done: {len(processed)} events\")\n", + "\n", + "\n", + "async def main() -> None:\n", + " async with event_processor(\"ClickFilter\") as results:\n", + " async for event in event_stream(8):\n", + " if event[\"type\"] == \"click\":\n", + " results.append(event)\n", + "\n", + " print(f\" Click events: {results}\")\n", + "\n", + "\n", + "asyncio.run(main())" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b1000020", + "metadata": {}, + "outputs": [], + "source": [ + "# Async for with set/dict comprehensions\n", + "async def user_ids(count: int) -> AsyncGenerator[int, None]:\n", + " \"\"\"Generate user IDs with some duplicates.\"\"\"\n", + " for i in range(count):\n", + " await asyncio.sleep(0)\n", + " yield i % 5 # Creates duplicates\n", + "\n", + "\n", + "async def main() -> None:\n", + " # Async set comprehension — deduplicates automatically\n", + " unique_ids: set[int] = {uid async for uid in user_ids(10)}\n", + " print(f\"Unique IDs: {sorted(unique_ids)}\")\n", + "\n", + " # Async dict comprehension\n", + " user_map: dict[int, str] = {\n", + " uid: f\"user-{uid}\" async for uid in user_ids(5)\n", + " }\n", + " print(f\"User map: {user_map}\")\n", + "\n", + "\n", + "asyncio.run(main())" + ] + }, + { + "cell_type": "markdown", + "id": "b1000021", + "metadata": {}, + "source": [ + "## Section 6: Error Handling in Async Patterns\n", + "\n", + "Errors in async iterators and context managers follow the same patterns as their synchronous counterparts, but the `async` versions require special handling for cleanup." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b1000022", + "metadata": {}, + "outputs": [], + "source": [ + "# Async context manager handles exceptions in __aexit__\n", + "class SafeResource:\n", + " \"\"\"A resource that always cleans up, even on error.\"\"\"\n", + "\n", + " def __init__(self, name: str) -> None:\n", + " self.name = name\n", + "\n", + " async def __aenter__(self):\n", + " print(f\" {self.name}: acquired\")\n", + " return self\n", + "\n", + " async def __aexit__(\n", + " self,\n", + " exc_type: type[BaseException] | None,\n", + " exc_val: BaseException | None,\n", + " exc_tb: object,\n", + " ) -> bool:\n", + " if exc_val:\n", + " print(f\" {self.name}: error occurred - {exc_val}\")\n", + " print(f\" {self.name}: released\")\n", + " return False # Don't suppress the exception\n", + "\n", + "\n", + "async def main() -> None:\n", + " # Normal usage\n", + " print(\"Normal case:\")\n", + " async with SafeResource(\"DB\") as db:\n", + " print(f\" Using {db.name}\")\n", + "\n", + " # Error case — resource still cleaned up\n", + " print(\"\\nError case:\")\n", + " try:\n", + " async with SafeResource(\"DB\") as db:\n", + " raise RuntimeError(\"connection lost\")\n", + " except RuntimeError:\n", + " print(\" Handled the error\")\n", + "\n", + "\n", + "asyncio.run(main())" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b1000023", + "metadata": {}, + "outputs": [], + "source": [ + "# Async generator cleanup with try/finally\n", + "async def resilient_stream(\n", + " items: list[str],\n", + ") -> AsyncGenerator[str, None]:\n", + " \"\"\"An async generator with guaranteed cleanup.\"\"\"\n", + " print(\" Stream: opened\")\n", + " try:\n", + " for item in items:\n", + " await asyncio.sleep(0)\n", + " yield item\n", + " finally:\n", + " print(\" Stream: closed (cleanup done)\")\n", + "\n", + "\n", + "async def main() -> None:\n", + " # Normal: iterate to completion\n", + " print(\"Full iteration:\")\n", + " async for val in resilient_stream([\"a\", \"b\", \"c\"]):\n", + " print(f\" got: {val}\")\n", + "\n", + " # Early exit: break triggers cleanup\n", + " print(\"\\nEarly exit (break):\")\n", + " async for val in resilient_stream([\"x\", \"y\", \"z\"]):\n", + " print(f\" got: {val}\")\n", + " if val == \"x\":\n", + " break\n", + "\n", + "\n", + "asyncio.run(main())" + ] + }, + { + "cell_type": "markdown", + "id": "b1000024", + "metadata": {}, + "source": [ + "## Summary\n", + "\n", + "### Async Iterators\n", + "- Implement `__aiter__()` and `__anext__()`\n", + "- Raise `StopAsyncIteration` when exhausted\n", + "- Consumed with `async for` or async comprehensions\n", + "\n", + "### Async Generators\n", + "- Easiest way to create async iterators: `async def` + `yield`\n", + "- Can `await` inside the generator body\n", + "- Composable: chain generators into pipelines\n", + "- Support cleanup via `try`/`finally`\n", + "\n", + "### Async Context Managers\n", + "- Implement `__aenter__()` and `__aexit__()` for class-based\n", + "- Use `@asynccontextmanager` decorator for function-based\n", + "- Used with `async with` for resource management\n", + "- `__aexit__` is called even when exceptions occur\n", + "\n", + "### Producer-Consumer\n", + "- `asyncio.Queue` decouples producers from consumers\n", + "- `queue.put()` / `queue.get()` are async operations\n", + "- Use `queue.join()` + `task_done()` for completion tracking\n", + "- Sentinel values enable graceful shutdown\n", + "\n", + "### Key Patterns\n", + "- Use async generators for lazy, memory-efficient async sequences\n", + "- Use `@asynccontextmanager` for simple resource management\n", + "- Combine patterns for real-world async pipelines\n", + "- Always ensure cleanup with `try`/`finally` or context managers" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "version": "3.13.0" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} \ No newline at end of file diff --git a/src/chapter_26/README.md b/src/chapter_26/README.md new file mode 100644 index 0000000..81936d5 --- /dev/null +++ b/src/chapter_26/README.md @@ -0,0 +1,9 @@ +# Chapter 26: Async Programming + +Asynchronous programming with `asyncio` — coroutines, the event loop, tasks, gathering results, semaphores, async iterators, and async context managers. + +## Notebooks + +1. **01_coroutines_and_event_loop.ipynb** — `async`/`await` syntax, coroutines, the event loop, `asyncio.run()` +2. **02_tasks_and_concurrency.ipynb** — `asyncio.create_task()`, `gather()`, `wait()`, semaphores, timeouts +3. **03_async_patterns.ipynb** — Async iterators, async generators, async context managers, producer-consumer diff --git a/src/chapter_26/__init__.py b/src/chapter_26/__init__.py new file mode 100644 index 0000000..fddb64d --- /dev/null +++ b/src/chapter_26/__init__.py @@ -0,0 +1 @@ +"""Chapter 26: Async Programming.""" diff --git a/src/chapter_27/01_multiprocessing_basics.ipynb b/src/chapter_27/01_multiprocessing_basics.ipynb new file mode 100644 index 0000000..b179481 --- /dev/null +++ b/src/chapter_27/01_multiprocessing_basics.ipynb @@ -0,0 +1,620 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "a1b2c3d4", + "metadata": {}, + "source": [ + "# Chapter 27: Multiprocessing Basics\n", + "\n", + "Python's `multiprocessing` module enables true parallelism by spawning separate operating\n", + "system processes, each with its own Python interpreter and memory space. Unlike threads,\n", + "processes are not constrained by the Global Interpreter Lock (GIL), making multiprocessing\n", + "the go-to approach for CPU-bound workloads.\n", + "\n", + "## Topics Covered\n", + "- **Process**: Creating and running child processes\n", + "- **Process lifecycle**: `start()`, `join()`, `is_alive()`, exit codes\n", + "- **Process identity**: PIDs and process names\n", + "- **Pool**: Worker pools for parallel task execution\n", + "- **Pool.map()**: Distributing work across processes\n", + "- **Pool.apply_async()**: Non-blocking task submission\n", + "- **Practical**: Comparing sequential vs parallel execution" + ] + }, + { + "cell_type": "markdown", + "id": "b2c3d4e5", + "metadata": {}, + "source": [ + "## The multiprocessing Module\n", + "\n", + "The `multiprocessing` module mirrors the `threading` API but uses **processes** instead of\n", + "threads. Each child process gets its own memory space and Python interpreter, which means:\n", + "\n", + "- No GIL contention -- true parallel execution on multiple CPU cores\n", + "- No shared state by default -- data must be explicitly passed or shared\n", + "- Higher overhead than threads (process creation is more expensive)\n", + "- Functions passed to processes must be **picklable** (defined at module level)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c3d4e5f6", + "metadata": {}, + "outputs": [], + "source": [ + "import multiprocessing\n", + "import os\n", + "import time\n", + "\n", + "# Basic information about the current environment\n", + "print(f\"Python process PID: {os.getpid()}\")\n", + "print(f\"CPU count: {os.cpu_count()}\")\n", + "print(f\"Start method: {multiprocessing.get_start_method()}\")" + ] + }, + { + "cell_type": "markdown", + "id": "d4e5f6a7", + "metadata": {}, + "source": [ + "## Creating a Process\n", + "\n", + "The `multiprocessing.Process` class creates a new process. You provide a **target** function\n", + "and optional **args** or **kwargs**. The process does not start until you call `.start()`.\n", + "\n", + "Key methods:\n", + "- `start()` -- spawn the child process and begin execution\n", + "- `join(timeout=None)` -- block until the process finishes (or timeout expires)\n", + "- `is_alive()` -- check if the process is still running\n", + "- `terminate()` / `kill()` -- forcibly stop the process" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e5f6a7b8", + "metadata": {}, + "outputs": [], + "source": [ + "import multiprocessing\n", + "import os\n", + "\n", + "\n", + "def worker(name: str) -> None:\n", + " \"\"\"A simple worker function that runs in a child process.\"\"\"\n", + " pid: int = os.getpid()\n", + " parent_pid: int = os.getppid()\n", + " print(f\"Worker '{name}' running in PID {pid} (parent PID: {parent_pid})\")\n", + "\n", + "\n", + "# Create and start a process\n", + "p = multiprocessing.Process(target=worker, args=(\"alpha\",))\n", + "print(f\"Before start: is_alive={p.is_alive()}, pid={p.pid}\")\n", + "\n", + "p.start()\n", + "print(f\"After start: is_alive={p.is_alive()}, pid={p.pid}\")\n", + "\n", + "p.join() # Wait for the process to finish\n", + "print(f\"After join: is_alive={p.is_alive()}, exitcode={p.exitcode}\")" + ] + }, + { + "cell_type": "markdown", + "id": "f6a7b8c9", + "metadata": {}, + "source": [ + "## Process Lifecycle and Exit Codes\n", + "\n", + "Every process has a lifecycle: created, started, running, and terminated. After a process\n", + "finishes, you can inspect its `exitcode`:\n", + "\n", + "| Exit code | Meaning |\n", + "|-----------|:--------|\n", + "| `0` | Normal termination |\n", + "| `> 0` | Exception occurred (exit code 1) |\n", + "| `< 0` | Killed by signal `-N` (e.g., -9 = SIGKILL) |\n", + "| `None` | Process has not yet terminated |" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a7b8c9d0", + "metadata": {}, + "outputs": [], + "source": [ + "import multiprocessing\n", + "import time\n", + "\n", + "\n", + "def quick_task() -> None:\n", + " \"\"\"A task that finishes quickly.\"\"\"\n", + " time.sleep(0.1)\n", + "\n", + "\n", + "def slow_task() -> None:\n", + " \"\"\"A task that takes longer.\"\"\"\n", + " time.sleep(2.0)\n", + "\n", + "\n", + "# Demonstrate process lifecycle\n", + "p = multiprocessing.Process(target=quick_task, name=\"QuickWorker\")\n", + "print(f\"Created: name={p.name}, alive={p.is_alive()}, exitcode={p.exitcode}\")\n", + "\n", + "p.start()\n", + "print(f\"Started: name={p.name}, alive={p.is_alive()}, exitcode={p.exitcode}\")\n", + "\n", + "p.join()\n", + "print(f\"Finished: name={p.name}, alive={p.is_alive()}, exitcode={p.exitcode}\")\n", + "\n", + "# Demonstrate join with timeout\n", + "p2 = multiprocessing.Process(target=slow_task, name=\"SlowWorker\")\n", + "p2.start()\n", + "p2.join(timeout=0.5) # Wait at most 0.5 seconds\n", + "print(f\"\\nAfter timeout: alive={p2.is_alive()}, exitcode={p2.exitcode}\")\n", + "\n", + "p2.terminate() # Forcibly stop the process\n", + "p2.join() # Clean up\n", + "print(f\"After terminate: alive={p2.is_alive()}, exitcode={p2.exitcode}\")" + ] + }, + { + "cell_type": "markdown", + "id": "b8c9d0e1", + "metadata": {}, + "source": [ + "## Multiple Processes and Separate Memory\n", + "\n", + "Each process has its own memory space. Modifications to variables in a child process do\n", + "**not** affect the parent process. This is fundamentally different from threading, where\n", + "threads share the same memory." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c9d0e1f2", + "metadata": {}, + "outputs": [], + "source": [ + "import multiprocessing\n", + "import os\n", + "\n", + "# Use a Queue to collect results from child processes\n", + "# (since child processes have separate memory)\n", + "\n", + "\n", + "def compute_square(n: int, result_queue: multiprocessing.Queue) -> None:\n", + " \"\"\"Compute a square and put the result in a queue.\"\"\"\n", + " pid: int = os.getpid()\n", + " result: int = n * n\n", + " result_queue.put((n, result, pid))\n", + "\n", + "\n", + "# Create multiple processes\n", + "queue: multiprocessing.Queue = multiprocessing.Queue()\n", + "processes: list[multiprocessing.Process] = []\n", + "\n", + "for i in range(4):\n", + " p = multiprocessing.Process(target=compute_square, args=(i, queue))\n", + " processes.append(p)\n", + " p.start()\n", + "\n", + "# Wait for all processes to complete\n", + "for p in processes:\n", + " p.join()\n", + "\n", + "# Collect results\n", + "print(f\"Main process PID: {os.getpid()}\")\n", + "print(\"\\nResults from child processes:\")\n", + "while not queue.empty():\n", + " n, result, pid = queue.get()\n", + " print(f\" {n}^2 = {result} (computed by PID {pid})\")" + ] + }, + { + "cell_type": "markdown", + "id": "d0e1f2a3", + "metadata": {}, + "source": [ + "## Process Names and Daemon Processes\n", + "\n", + "Processes can be named for easier debugging. A **daemon** process runs in the background\n", + "and is automatically terminated when the main process exits.\n", + "\n", + "- Set `daemon=True` before calling `start()` to create a daemon process\n", + "- Daemon processes cannot spawn child processes of their own\n", + "- They are useful for background tasks that should not prevent program exit" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e1f2a3b4", + "metadata": {}, + "outputs": [], + "source": [ + "import multiprocessing\n", + "import os\n", + "import time\n", + "\n", + "\n", + "def named_worker() -> None:\n", + " \"\"\"Worker that prints its own name and PID.\"\"\"\n", + " proc = multiprocessing.current_process()\n", + " print(f\" Process name={proc.name}, PID={os.getpid()}, daemon={proc.daemon}\")\n", + "\n", + "\n", + "# Named processes\n", + "p1 = multiprocessing.Process(target=named_worker, name=\"Worker-A\")\n", + "p2 = multiprocessing.Process(target=named_worker, name=\"Worker-B\")\n", + "\n", + "p1.start()\n", + "p2.start()\n", + "p1.join()\n", + "p2.join()\n", + "\n", + "# Daemon process example\n", + "print(\"\\nDaemon process:\")\n", + "p3 = multiprocessing.Process(target=named_worker, name=\"Daemon-Worker\", daemon=True)\n", + "p3.start()\n", + "p3.join(timeout=2) # Must join daemon processes explicitly if you want to wait\n", + "print(f\" p3 daemon={p3.daemon}, exitcode={p3.exitcode}\")" + ] + }, + { + "cell_type": "markdown", + "id": "f2a3b4c5", + "metadata": {}, + "source": [ + "## Pool: Managing Worker Processes\n", + "\n", + "Creating individual `Process` objects works for a few tasks, but for many tasks you want a\n", + "**pool** of reusable worker processes. `multiprocessing.Pool` manages a fixed number of\n", + "worker processes and distributes tasks to them.\n", + "\n", + "Key methods:\n", + "- `map(func, iterable)` -- apply `func` to every item, return ordered results (blocking)\n", + "- `starmap(func, iterable)` -- like `map()` but unpacks argument tuples\n", + "- `apply(func, args)` -- call `func` with `args` in a worker (blocking)\n", + "- `apply_async(func, args)` -- non-blocking version, returns `AsyncResult`\n", + "- `map_async(func, iterable)` -- non-blocking version of `map()`" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a3b4c5d6", + "metadata": {}, + "outputs": [], + "source": [ + "import multiprocessing\n", + "import os\n", + "\n", + "\n", + "def square(x: int) -> int:\n", + " \"\"\"Compute the square of a number.\"\"\"\n", + " return x * x\n", + "\n", + "\n", + "# Pool.map() distributes work across a pool of workers\n", + "with multiprocessing.Pool(processes=2) as pool:\n", + " results: list[int] = pool.map(square, [1, 2, 3, 4, 5, 6, 7, 8])\n", + "\n", + "print(f\"Input: {list(range(1, 9))}\")\n", + "print(f\"Squared: {results}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b4c5d6e7", + "metadata": {}, + "outputs": [], + "source": [ + "import multiprocessing\n", + "\n", + "\n", + "def power(base: int, exp: int) -> int:\n", + " \"\"\"Raise base to the given exponent.\"\"\"\n", + " return base ** exp\n", + "\n", + "\n", + "# Pool.starmap() unpacks tuples of arguments\n", + "args: list[tuple[int, int]] = [(2, 3), (3, 3), (4, 2), (5, 2), (10, 3)]\n", + "\n", + "with multiprocessing.Pool(processes=2) as pool:\n", + " results: list[int] = pool.starmap(power, args)\n", + "\n", + "for (base, exp), result in zip(args, results):\n", + " print(f\" {base}^{exp} = {result}\")" + ] + }, + { + "cell_type": "markdown", + "id": "c5d6e7f8", + "metadata": {}, + "source": [ + "## Pool.apply_async(): Non-Blocking Execution\n", + "\n", + "`apply_async()` submits a single task to the pool without blocking. It returns an\n", + "`AsyncResult` object that you can use to check status and retrieve results later.\n", + "\n", + "- `result.get(timeout=None)` -- block until the result is ready\n", + "- `result.ready()` -- check if the result is available\n", + "- `result.successful()` -- check if the task completed without error" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "d6e7f8a9", + "metadata": {}, + "outputs": [], + "source": [ + "import multiprocessing\n", + "import time\n", + "\n", + "\n", + "def slow_square(x: int) -> int:\n", + " \"\"\"Compute a square with a simulated delay.\"\"\"\n", + " time.sleep(0.3)\n", + " return x * x\n", + "\n", + "\n", + "with multiprocessing.Pool(processes=2) as pool:\n", + " # Submit tasks asynchronously\n", + " async_results = [\n", + " pool.apply_async(slow_square, args=(i,))\n", + " for i in range(1, 6)\n", + " ]\n", + "\n", + " # Check status while tasks are running\n", + " print(\"Submitted 5 tasks...\")\n", + " time.sleep(0.1)\n", + " for i, ar in enumerate(async_results, 1):\n", + " print(f\" Task {i}: ready={ar.ready()}\")\n", + "\n", + " # Collect results (blocks until ready)\n", + " print(\"\\nCollecting results:\")\n", + " for i, ar in enumerate(async_results, 1):\n", + " result: int = ar.get(timeout=5)\n", + " print(f\" {i}^2 = {result}\")" + ] + }, + { + "cell_type": "markdown", + "id": "e7f8a9b0", + "metadata": {}, + "source": [ + "## Separate Process IDs\n", + "\n", + "A key characteristic of multiprocessing is that each worker runs in a separate OS process\n", + "with its own PID. This is what gives us true parallelism, but it also means that data\n", + "must be serialized (pickled) to pass between processes." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f8a9b0c1", + "metadata": {}, + "outputs": [], + "source": [ + "import multiprocessing\n", + "import os\n", + "\n", + "\n", + "def get_pid(_: int = 0) -> int:\n", + " \"\"\"Return the current process ID.\"\"\"\n", + " return os.getpid()\n", + "\n", + "\n", + "main_pid: int = os.getpid()\n", + "print(f\"Main process PID: {main_pid}\")\n", + "\n", + "with multiprocessing.Pool(processes=3) as pool:\n", + " pids: list[int] = pool.map(get_pid, range(6))\n", + "\n", + "print(f\"Worker PIDs: {pids}\")\n", + "unique_pids: set[int] = set(pids)\n", + "print(f\"Unique workers: {unique_pids}\")\n", + "print(f\"\\nAll worker PIDs differ from main: {all(pid != main_pid for pid in pids)}\")\n", + "print(f\"Number of unique workers: {len(unique_pids)}\")" + ] + }, + { + "cell_type": "markdown", + "id": "a9b0c1d2", + "metadata": {}, + "source": [ + "## Practical: Sequential vs Parallel Execution\n", + "\n", + "Let us compare the execution time of a CPU-bound workload run sequentially versus\n", + "in parallel using `Pool.map()`. This demonstrates the real benefit of multiprocessing\n", + "for CPU-intensive tasks." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b0c1d2e3", + "metadata": {}, + "outputs": [], + "source": [ + "import multiprocessing\n", + "import time\n", + "\n", + "\n", + "def cpu_bound_task(n: int) -> int:\n", + " \"\"\"A CPU-bound task: sum of squares up to n.\"\"\"\n", + " total: int = 0\n", + " for i in range(n):\n", + " total += i * i\n", + " return total\n", + "\n", + "\n", + "workload: list[int] = [2_000_000] * 8\n", + "\n", + "# Sequential execution\n", + "start: float = time.perf_counter()\n", + "sequential_results: list[int] = [cpu_bound_task(n) for n in workload]\n", + "sequential_time: float = time.perf_counter() - start\n", + "\n", + "# Parallel execution\n", + "start = time.perf_counter()\n", + "with multiprocessing.Pool(processes=4) as pool:\n", + " parallel_results: list[int] = pool.map(cpu_bound_task, workload)\n", + "parallel_time: float = time.perf_counter() - start\n", + "\n", + "print(f\"Sequential time: {sequential_time:.3f}s\")\n", + "print(f\"Parallel time: {parallel_time:.3f}s\")\n", + "print(f\"Speedup: {sequential_time / parallel_time:.2f}x\")\n", + "print(f\"Results match: {sequential_results == parallel_results}\")" + ] + }, + { + "cell_type": "markdown", + "id": "c1d2e3f4", + "metadata": {}, + "source": [ + "## Error Handling in Processes\n", + "\n", + "When a child process raises an exception, the behavior depends on how you launched it:\n", + "\n", + "- **Process**: The exception is raised in the child only; the parent sees a non-zero `exitcode`\n", + "- **Pool.map()**: The exception is re-raised in the parent when results are collected\n", + "- **Pool.apply_async()**: The exception is re-raised when you call `.get()` on the result" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "d2e3f4a5", + "metadata": {}, + "outputs": [], + "source": [ + "import multiprocessing\n", + "\n", + "\n", + "def risky_task(x: int) -> float:\n", + " \"\"\"A task that might fail.\"\"\"\n", + " if x == 0:\n", + " raise ValueError(\"Cannot process zero!\")\n", + " return 100.0 / x\n", + "\n", + "\n", + "# Error with Pool.map() -- exception propagates to parent\n", + "print(\"Testing Pool.map() error handling:\")\n", + "try:\n", + " with multiprocessing.Pool(2) as pool:\n", + " pool.map(risky_task, [5, 2, 0, 1]) # 0 will cause an error\n", + "except ValueError as e:\n", + " print(f\" Caught in parent: {e}\")\n", + "\n", + "# Error with apply_async() -- exception on .get()\n", + "print(\"\\nTesting apply_async() error handling:\")\n", + "with multiprocessing.Pool(2) as pool:\n", + " future = pool.apply_async(risky_task, args=(0,))\n", + " try:\n", + " result = future.get(timeout=5)\n", + " except ValueError as e:\n", + " print(f\" Caught from future.get(): {e}\")\n", + " print(f\" future.successful(): {future.successful()}\")" + ] + }, + { + "cell_type": "markdown", + "id": "e3f4a5b6", + "metadata": {}, + "source": [ + "## Pool with Initializer\n", + "\n", + "You can pass an `initializer` function to `Pool()` that runs once in each worker process\n", + "when it starts. This is useful for setting up per-worker state (database connections,\n", + "loading large data, etc.)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f4a5b6c7", + "metadata": {}, + "outputs": [], + "source": [ + "import multiprocessing\n", + "import os\n", + "\n", + "# Module-level variable that will be set by the initializer\n", + "worker_id: int = -1\n", + "\n", + "\n", + "def init_worker(base_id: int) -> None:\n", + " \"\"\"Initialize each worker with a unique ID.\"\"\"\n", + " global worker_id\n", + " worker_id = base_id + os.getpid() % 100\n", + " print(f\" Worker initialized: PID={os.getpid()}, worker_id={worker_id}\")\n", + "\n", + "\n", + "def task_with_state(x: int) -> str:\n", + " \"\"\"A task that uses worker-local state.\"\"\"\n", + " return f\"worker_id={worker_id}, PID={os.getpid()}, input={x}, result={x*x}\"\n", + "\n", + "\n", + "print(\"Initializing pool with per-worker setup:\")\n", + "with multiprocessing.Pool(processes=2, initializer=init_worker, initargs=(1000,)) as pool:\n", + " results: list[str] = pool.map(task_with_state, range(4))\n", + "\n", + "print(\"\\nResults:\")\n", + "for r in results:\n", + " print(f\" {r}\")" + ] + }, + { + "cell_type": "markdown", + "id": "a5b6c7d8", + "metadata": {}, + "source": [ + "## Summary\n", + "\n", + "### Key Takeaways\n", + "\n", + "| Concept | API | Purpose |\n", + "|---------|-----|:--------|\n", + "| **Process** | `multiprocessing.Process` | Spawn a single child process |\n", + "| **Lifecycle** | `start()`, `join()`, `is_alive()` | Control process execution |\n", + "| **Exit codes** | `p.exitcode` | `0`=success, `>0`=error, `<0`=signal |\n", + "| **Daemon** | `Process(daemon=True)` | Background process, auto-killed on exit |\n", + "| **Pool** | `multiprocessing.Pool(n)` | Manage a pool of `n` worker processes |\n", + "| **map()** | `pool.map(func, iterable)` | Distribute work, return ordered results |\n", + "| **starmap()** | `pool.starmap(func, args)` | Like map but unpacks argument tuples |\n", + "| **apply_async()** | `pool.apply_async(func, args)` | Non-blocking single task submission |\n", + "| **Initializer** | `Pool(initializer=func)` | Run setup code once per worker |\n", + "\n", + "### Best Practices\n", + "- Always use `Pool` as a context manager (`with` statement) to ensure workers are cleaned up\n", + "- Define worker functions at module level so they can be pickled\n", + "- Use `join()` to wait for processes and avoid zombie processes\n", + "- Prefer `Pool.map()` for batch processing over manually managing `Process` objects\n", + "- Handle exceptions from workers -- they propagate when you collect results\n", + "- Choose the number of pool workers based on `os.cpu_count()` for CPU-bound tasks" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "version": "3.13.0" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} \ No newline at end of file diff --git a/src/chapter_27/02_shared_state_and_ipc.ipynb b/src/chapter_27/02_shared_state_and_ipc.ipynb new file mode 100644 index 0000000..0b83bfd --- /dev/null +++ b/src/chapter_27/02_shared_state_and_ipc.ipynb @@ -0,0 +1,612 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "a1b2c3d4", + "metadata": {}, + "source": [ + "# Chapter 27: Shared State and Inter-Process Communication\n", + "\n", + "Since each process has its own memory space, sharing data between processes requires\n", + "explicit mechanisms. The `multiprocessing` module provides several IPC (Inter-Process\n", + "Communication) primitives: `Queue`, `Pipe`, `Value`, `Array`, `Manager`, and\n", + "synchronization tools like `Lock`.\n", + "\n", + "## Topics Covered\n", + "- **Queue**: Thread/process-safe FIFO queue\n", + "- **Pipe**: Bidirectional communication channel\n", + "- **Value**: Shared single value in shared memory\n", + "- **Array**: Shared array in shared memory\n", + "- **Lock**: Synchronize access to shared resources\n", + "- **Manager**: Proxy-based shared objects (dicts, lists, etc.)\n", + "- **Practical**: Producer-consumer pattern with Queue" + ] + }, + { + "cell_type": "markdown", + "id": "b2c3d4e5", + "metadata": {}, + "source": [ + "## Queue: Safe Inter-Process Communication\n", + "\n", + "`multiprocessing.Queue` is a process-safe FIFO queue built on top of pipes and locks.\n", + "It is the most common way to pass data between processes.\n", + "\n", + "Key methods:\n", + "- `put(item)` -- add an item (blocks if the queue is full)\n", + "- `get()` -- remove and return an item (blocks if empty)\n", + "- `empty()` -- approximate check if the queue is empty\n", + "- `qsize()` -- approximate number of items" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c3d4e5f6", + "metadata": {}, + "outputs": [], + "source": [ + "import multiprocessing\n", + "\n", + "# Basic Queue usage (within a single process for demonstration)\n", + "q: multiprocessing.Queue = multiprocessing.Queue()\n", + "\n", + "# Put items into the queue\n", + "q.put(\"hello\")\n", + "q.put(\"world\")\n", + "q.put(42)\n", + "\n", + "# Get items in FIFO order\n", + "print(f\"First: {q.get()}\")\n", + "print(f\"Second: {q.get()}\")\n", + "print(f\"Third: {q.get()}\")\n", + "print(f\"Empty: {q.empty()}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "d4e5f6a7", + "metadata": {}, + "outputs": [], + "source": [ + "import multiprocessing\n", + "import os\n", + "\n", + "\n", + "def queue_worker(q: multiprocessing.Queue, values: list[int]) -> None:\n", + " \"\"\"Put computed results into a queue.\"\"\"\n", + " pid: int = os.getpid()\n", + " for v in values:\n", + " q.put((pid, v, v * v))\n", + "\n", + "\n", + "# Use a Queue to collect results from multiple worker processes\n", + "result_queue: multiprocessing.Queue = multiprocessing.Queue()\n", + "\n", + "p1 = multiprocessing.Process(target=queue_worker, args=(result_queue, [1, 2, 3]))\n", + "p2 = multiprocessing.Process(target=queue_worker, args=(result_queue, [4, 5, 6]))\n", + "\n", + "p1.start()\n", + "p2.start()\n", + "p1.join()\n", + "p2.join()\n", + "\n", + "# Read all results\n", + "print(\"Results from workers:\")\n", + "while not result_queue.empty():\n", + " pid, value, squared = result_queue.get()\n", + " print(f\" PID {pid}: {value}^2 = {squared}\")" + ] + }, + { + "cell_type": "markdown", + "id": "e5f6a7b8", + "metadata": {}, + "source": [ + "## Pipe: Bidirectional Communication\n", + "\n", + "`multiprocessing.Pipe()` creates a pair of connected `Connection` objects. By default,\n", + "the pipe is bidirectional -- each end can both `send()` and `recv()`.\n", + "\n", + "Pipes are faster than queues for simple two-process communication, but they are not\n", + "safe for use by more than two processes simultaneously." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f6a7b8c9", + "metadata": {}, + "outputs": [], + "source": [ + "import multiprocessing\n", + "\n", + "# Bidirectional pipe (within a single process for clarity)\n", + "parent_conn, child_conn = multiprocessing.Pipe()\n", + "\n", + "# Send from parent end, receive at child end\n", + "parent_conn.send(\"ping\")\n", + "print(f\"Child received: {child_conn.recv()}\")\n", + "\n", + "# Send from child end, receive at parent end\n", + "child_conn.send(\"pong\")\n", + "print(f\"Parent received: {parent_conn.recv()}\")\n", + "\n", + "# Pipes can send any picklable object\n", + "parent_conn.send({\"type\": \"data\", \"values\": [1, 2, 3]})\n", + "msg: dict = child_conn.recv()\n", + "print(f\"Dict received: {msg}\")\n", + "\n", + "parent_conn.close()\n", + "child_conn.close()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a7b8c9d0", + "metadata": {}, + "outputs": [], + "source": [ + "import multiprocessing\n", + "from multiprocessing.connection import Connection\n", + "\n", + "\n", + "def pipe_child(conn: Connection) -> None:\n", + " \"\"\"Child process that communicates via a Pipe.\"\"\"\n", + " msg: str = conn.recv()\n", + " conn.send(f\"echo: {msg}\")\n", + " conn.close()\n", + "\n", + "\n", + "# Two-process communication with Pipe\n", + "parent_conn, child_conn = multiprocessing.Pipe()\n", + "\n", + "p = multiprocessing.Process(target=pipe_child, args=(child_conn,))\n", + "p.start()\n", + "\n", + "# Parent sends a message and waits for the reply\n", + "parent_conn.send(\"hello from parent\")\n", + "reply: str = parent_conn.recv()\n", + "print(f\"Parent got reply: {reply}\")\n", + "\n", + "p.join()\n", + "parent_conn.close()" + ] + }, + { + "cell_type": "markdown", + "id": "b8c9d0e1", + "metadata": {}, + "source": [ + "## Value and Array: Shared Memory\n", + "\n", + "`multiprocessing.Value` and `multiprocessing.Array` create shared memory objects that\n", + "can be accessed by multiple processes. They use ctypes type codes:\n", + "\n", + "| Type code | C type | Python type |\n", + "|-----------|--------|:------------|\n", + "| `'i'` | `int` | `int` |\n", + "| `'d'` | `double` | `float` |\n", + "| `'f'` | `float` | `float` |\n", + "| `'c'` | `char` | `bytes` |\n", + "\n", + "These objects live in **shared memory** (not copied), so changes are visible to all processes." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c9d0e1f2", + "metadata": {}, + "outputs": [], + "source": [ + "import multiprocessing\n", + "\n", + "# Value: a single shared value\n", + "counter = multiprocessing.Value(\"i\", 0) # 'i' = signed int, initial value = 0\n", + "print(f\"Initial value: {counter.value}\")\n", + "\n", + "counter.value = 42\n", + "print(f\"After assignment: {counter.value}\")\n", + "\n", + "# Value with a float\n", + "temperature = multiprocessing.Value(\"d\", 98.6) # 'd' = double\n", + "print(f\"Temperature: {temperature.value}\")\n", + "\n", + "# Array: shared array of fixed type\n", + "arr = multiprocessing.Array(\"d\", [1.0, 2.0, 3.0, 4.0, 5.0])\n", + "print(f\"\\nArray contents: {list(arr)}\")\n", + "\n", + "arr[0] = 10.0\n", + "arr[4] = 50.0\n", + "print(f\"After modification: {list(arr)}\")\n", + "print(f\"Array length: {len(arr)}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "d0e1f2a3", + "metadata": {}, + "outputs": [], + "source": [ + "import multiprocessing\n", + "\n", + "\n", + "def increment_counter(\n", + " counter: multiprocessing.Value,\n", + " times: int,\n", + ") -> None:\n", + " \"\"\"Increment a shared counter multiple times.\"\"\"\n", + " for _ in range(times):\n", + " with counter.get_lock(): # Acquire the built-in lock\n", + " counter.value += 1\n", + "\n", + "\n", + "# Shared counter accessed by multiple processes\n", + "shared_counter = multiprocessing.Value(\"i\", 0)\n", + "\n", + "processes: list[multiprocessing.Process] = [\n", + " multiprocessing.Process(target=increment_counter, args=(shared_counter, 1000))\n", + " for _ in range(4)\n", + "]\n", + "\n", + "for p in processes:\n", + " p.start()\n", + "for p in processes:\n", + " p.join()\n", + "\n", + "# With proper locking, result should be exactly 4000\n", + "print(f\"Final counter value: {shared_counter.value}\")\n", + "print(f\"Expected: {4 * 1000}\")\n", + "print(f\"Correct: {shared_counter.value == 4000}\")" + ] + }, + { + "cell_type": "markdown", + "id": "e1f2a3b4", + "metadata": {}, + "source": [ + "## Lock: Synchronizing Shared Access\n", + "\n", + "`multiprocessing.Lock` prevents multiple processes from accessing a shared resource\n", + "simultaneously. It works exactly like `threading.Lock` but across processes.\n", + "\n", + "- `lock.acquire()` / `lock.release()` -- manual lock management\n", + "- `with lock:` -- context manager (preferred) for automatic release\n", + "\n", + "Without a lock, concurrent modifications to shared state can produce incorrect results\n", + "due to **race conditions**." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f2a3b4c5", + "metadata": {}, + "outputs": [], + "source": [ + "import multiprocessing\n", + "\n", + "\n", + "def unsafe_increment(\n", + " counter: multiprocessing.Value,\n", + " times: int,\n", + ") -> None:\n", + " \"\"\"Increment WITHOUT a lock -- prone to race conditions.\"\"\"\n", + " for _ in range(times):\n", + " counter.value += 1 # NOT atomic!\n", + "\n", + "\n", + "def safe_increment(\n", + " counter: multiprocessing.Value,\n", + " lock: multiprocessing.Lock,\n", + " times: int,\n", + ") -> None:\n", + " \"\"\"Increment WITH a lock -- safe from race conditions.\"\"\"\n", + " for _ in range(times):\n", + " with lock:\n", + " counter.value += 1\n", + "\n", + "\n", + "# Unsafe version (may produce incorrect results)\n", + "unsafe_counter = multiprocessing.Value(\"i\", 0)\n", + "procs = [\n", + " multiprocessing.Process(target=unsafe_increment, args=(unsafe_counter, 10000))\n", + " for _ in range(4)\n", + "]\n", + "for p in procs:\n", + " p.start()\n", + "for p in procs:\n", + " p.join()\n", + "print(f\"Unsafe counter: {unsafe_counter.value} (expected 40000)\")\n", + "\n", + "# Safe version with explicit Lock\n", + "safe_counter = multiprocessing.Value(\"i\", 0)\n", + "lock = multiprocessing.Lock()\n", + "procs = [\n", + " multiprocessing.Process(target=safe_increment, args=(safe_counter, lock, 10000))\n", + " for _ in range(4)\n", + "]\n", + "for p in procs:\n", + " p.start()\n", + "for p in procs:\n", + " p.join()\n", + "print(f\"Safe counter: {safe_counter.value} (expected 40000)\")" + ] + }, + { + "cell_type": "markdown", + "id": "a3b4c5d6", + "metadata": {}, + "source": [ + "## Manager: Proxy-Based Shared Objects\n", + "\n", + "`multiprocessing.Manager()` creates a server process that hosts shared Python objects.\n", + "Other processes access these objects through **proxies**. This is more flexible than\n", + "`Value`/`Array` because it supports standard Python types:\n", + "\n", + "- `manager.list()` -- shared list\n", + "- `manager.dict()` -- shared dictionary\n", + "- `manager.Value()` -- shared value\n", + "- `manager.Queue()` -- shared queue\n", + "- `manager.Lock()` -- shared lock\n", + "\n", + "The tradeoff: Managers are **slower** than `Value`/`Array` because they use inter-process\n", + "communication (proxies) under the hood, but they are more convenient for complex data." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b4c5d6e7", + "metadata": {}, + "outputs": [], + "source": [ + "import multiprocessing\n", + "from multiprocessing.managers import DictProxy, ListProxy\n", + "\n", + "\n", + "def add_results(\n", + " shared_dict: DictProxy,\n", + " shared_list: ListProxy,\n", + " key: str,\n", + " values: list[int],\n", + ") -> None:\n", + " \"\"\"Add results to shared dict and list.\"\"\"\n", + " total: int = sum(values)\n", + " shared_dict[key] = total\n", + " shared_list.append(f\"{key}={total}\")\n", + "\n", + "\n", + "with multiprocessing.Manager() as manager:\n", + " # Create shared objects through the manager\n", + " shared_dict = manager.dict()\n", + " shared_list = manager.list()\n", + "\n", + " # Launch processes that modify shared objects\n", + " processes = [\n", + " multiprocessing.Process(\n", + " target=add_results,\n", + " args=(shared_dict, shared_list, f\"task_{i}\", list(range(i * 10))),\n", + " )\n", + " for i in range(1, 5)\n", + " ]\n", + " for p in processes:\n", + " p.start()\n", + " for p in processes:\n", + " p.join()\n", + "\n", + " print(\"Shared dict:\")\n", + " for key, value in sorted(shared_dict.items()):\n", + " print(f\" {key}: {value}\")\n", + "\n", + " print(f\"\\nShared list: {list(shared_list)}\")" + ] + }, + { + "cell_type": "markdown", + "id": "c5d6e7f8", + "metadata": {}, + "source": [ + "## Choosing the Right IPC Mechanism\n", + "\n", + "| Mechanism | Use Case | Speed | Flexibility |\n", + "|-----------|----------|:------|:------------|\n", + "| **Queue** | Multiple producers/consumers, message passing | Medium | High |\n", + "| **Pipe** | Two-process communication, request/response | Fast | Low |\n", + "| **Value** | Single shared number (int, float) | Fast | Low |\n", + "| **Array** | Shared fixed-size array of numbers | Fast | Low |\n", + "| **Manager** | Shared dicts, lists, complex objects | Slow | High |\n", + "| **Lock** | Protect shared resources from race conditions | -- | -- |" + ] + }, + { + "cell_type": "markdown", + "id": "d6e7f8a9", + "metadata": {}, + "source": [ + "## Practical: Producer-Consumer with Queue\n", + "\n", + "The **producer-consumer** pattern is a classic concurrency design pattern. Producers\n", + "generate data and place it on a queue; consumers take data from the queue and process it.\n", + "A special **sentinel** value (e.g., `None`) signals consumers to stop." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e7f8a9b0", + "metadata": {}, + "outputs": [], + "source": [ + "import multiprocessing\n", + "import os\n", + "import time\n", + "\n", + "\n", + "def producer(\n", + " queue: multiprocessing.Queue,\n", + " items: list[int],\n", + " name: str,\n", + ") -> None:\n", + " \"\"\"Produce items and put them on the queue.\"\"\"\n", + " for item in items:\n", + " queue.put((name, item))\n", + " time.sleep(0.05) # Simulate work\n", + " print(f\" Producer {name} (PID {os.getpid()}) finished\")\n", + "\n", + "\n", + "def consumer(\n", + " queue: multiprocessing.Queue,\n", + " result_queue: multiprocessing.Queue,\n", + " name: str,\n", + ") -> None:\n", + " \"\"\"Consume items from the queue until sentinel is received.\"\"\"\n", + " processed: int = 0\n", + " while True:\n", + " item = queue.get()\n", + " if item is None: # Sentinel value\n", + " break\n", + " producer_name, value = item\n", + " result_queue.put((name, producer_name, value, value * value))\n", + " processed += 1\n", + " print(f\" Consumer {name} (PID {os.getpid()}) processed {processed} items\")\n", + "\n", + "\n", + "# Set up the queues\n", + "work_queue: multiprocessing.Queue = multiprocessing.Queue()\n", + "result_queue: multiprocessing.Queue = multiprocessing.Queue()\n", + "\n", + "# Start 2 producers and 2 consumers\n", + "producers = [\n", + " multiprocessing.Process(target=producer, args=(work_queue, [1, 2, 3, 4], \"P1\")),\n", + " multiprocessing.Process(target=producer, args=(work_queue, [5, 6, 7, 8], \"P2\")),\n", + "]\n", + "consumers = [\n", + " multiprocessing.Process(target=consumer, args=(work_queue, result_queue, \"C1\")),\n", + " multiprocessing.Process(target=consumer, args=(work_queue, result_queue, \"C2\")),\n", + "]\n", + "\n", + "for p in producers + consumers:\n", + " p.start()\n", + "\n", + "# Wait for producers to finish\n", + "for p in producers:\n", + " p.join()\n", + "\n", + "# Send sentinel values (one per consumer)\n", + "for _ in consumers:\n", + " work_queue.put(None)\n", + "\n", + "# Wait for consumers to finish\n", + "for c in consumers:\n", + " c.join()\n", + "\n", + "# Collect results\n", + "print(\"\\nResults:\")\n", + "while not result_queue.empty():\n", + " consumer_name, producer_name, value, squared = result_queue.get()\n", + " print(f\" {consumer_name} processed {producer_name}'s item: {value}^2 = {squared}\")" + ] + }, + { + "cell_type": "markdown", + "id": "f8a9b0c1", + "metadata": {}, + "source": [ + "## Shared Array with Worker Processes\n", + "\n", + "When multiple processes need to write results into a shared data structure, `Array`\n", + "combined with index-based partitioning avoids the need for locks entirely. Each process\n", + "writes to its own slice of the array." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a9b0c1d2", + "metadata": {}, + "outputs": [], + "source": [ + "import multiprocessing\n", + "\n", + "\n", + "def fill_array_slice(\n", + " shared_arr: multiprocessing.Array,\n", + " start: int,\n", + " end: int,\n", + " multiplier: float,\n", + ") -> None:\n", + " \"\"\"Fill a slice of a shared array with computed values.\"\"\"\n", + " for i in range(start, end):\n", + " shared_arr[i] = float(i) * multiplier\n", + "\n", + "\n", + "# Create a shared array of 12 doubles\n", + "size: int = 12\n", + "shared: multiprocessing.Array = multiprocessing.Array(\"d\", size)\n", + "\n", + "# Each process handles a different slice (no lock needed)\n", + "chunk: int = size // 3\n", + "procs = [\n", + " multiprocessing.Process(target=fill_array_slice, args=(shared, 0, chunk, 1.0)),\n", + " multiprocessing.Process(target=fill_array_slice, args=(shared, chunk, 2 * chunk, 2.0)),\n", + " multiprocessing.Process(target=fill_array_slice, args=(shared, 2 * chunk, size, 3.0)),\n", + "]\n", + "\n", + "for p in procs:\n", + " p.start()\n", + "for p in procs:\n", + " p.join()\n", + "\n", + "print(\"Shared array contents:\")\n", + "for i, val in enumerate(shared):\n", + " print(f\" [{i}] = {val}\")" + ] + }, + { + "cell_type": "markdown", + "id": "b0c1d2e3", + "metadata": {}, + "source": [ + "## Summary\n", + "\n", + "### Key Takeaways\n", + "\n", + "| Concept | API | Purpose |\n", + "|---------|-----|:--------|\n", + "| **Queue** | `multiprocessing.Queue` | FIFO message passing between processes |\n", + "| **Pipe** | `multiprocessing.Pipe()` | Fast two-process bidirectional channel |\n", + "| **Value** | `multiprocessing.Value('i', 0)` | Single shared int/float in shared memory |\n", + "| **Array** | `multiprocessing.Array('d', [...])` | Fixed-size shared array in shared memory |\n", + "| **Lock** | `multiprocessing.Lock()` | Prevent race conditions on shared state |\n", + "| **Manager** | `multiprocessing.Manager()` | Proxy-based shared dicts, lists, etc. |\n", + "\n", + "### Best Practices\n", + "- Use `Queue` for general-purpose message passing between any number of processes\n", + "- Use `Pipe` for fast, simple two-process communication\n", + "- Use `Value` and `Array` for high-performance shared memory with numeric data\n", + "- Always use locks when multiple processes write to the same `Value` or `Array`\n", + "- Use `Manager` when you need to share complex Python objects (dicts, lists)\n", + "- Prefer partitioning work (each process writes its own slice) over shared locks\n", + "- Use sentinel values to signal consumers when all work is done" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "version": "3.13.0" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} \ No newline at end of file diff --git a/src/chapter_27/03_process_pool_executor.ipynb b/src/chapter_27/03_process_pool_executor.ipynb new file mode 100644 index 0000000..f5d2ffb --- /dev/null +++ b/src/chapter_27/03_process_pool_executor.ipynb @@ -0,0 +1,618 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "a1b2c3d4", + "metadata": {}, + "source": [ + "# Chapter 27: ProcessPoolExecutor and Futures\n", + "\n", + "The `concurrent.futures` module provides a high-level interface for parallel execution.\n", + "`ProcessPoolExecutor` manages a pool of worker processes and returns `Future` objects\n", + "that represent pending results. It is simpler and more Pythonic than the lower-level\n", + "`multiprocessing.Pool` API.\n", + "\n", + "## Topics Covered\n", + "- **ProcessPoolExecutor**: High-level process pool interface\n", + "- **Future**: Representing pending results\n", + "- **map() vs submit()**: Two ways to distribute work\n", + "- **as_completed()**: Processing results as they become available\n", + "- **CPU-bound vs I/O-bound**: Choosing the right executor\n", + "- **Practical**: Comparing ProcessPoolExecutor and ThreadPoolExecutor" + ] + }, + { + "cell_type": "markdown", + "id": "b2c3d4e5", + "metadata": {}, + "source": [ + "## ProcessPoolExecutor Basics\n", + "\n", + "`ProcessPoolExecutor` is part of `concurrent.futures` and provides the same API as\n", + "`ThreadPoolExecutor`. It uses processes instead of threads, making it ideal for\n", + "**CPU-bound** workloads.\n", + "\n", + "Key features:\n", + "- Context manager support (`with` statement)\n", + "- `max_workers` defaults to the number of CPUs\n", + "- Worker functions must be **picklable** (defined at module level)\n", + "- Returns `Future` objects for asynchronous result handling" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c3d4e5f6", + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "from concurrent.futures import ProcessPoolExecutor\n", + "\n", + "\n", + "def square(x: int) -> int:\n", + " \"\"\"Compute the square of a number.\"\"\"\n", + " return x * x\n", + "\n", + "\n", + "# Basic usage with executor.map()\n", + "print(f\"CPU count: {os.cpu_count()}\")\n", + "\n", + "with ProcessPoolExecutor(max_workers=2) as executor:\n", + " results: list[int] = list(executor.map(square, [1, 2, 3, 4, 5]))\n", + "\n", + "print(f\"Squares: {results}\")" + ] + }, + { + "cell_type": "markdown", + "id": "d4e5f6a7", + "metadata": {}, + "source": [ + "## executor.map(): Ordered Parallel Mapping\n", + "\n", + "`executor.map(func, *iterables)` applies a function to every item in the iterables,\n", + "distributing work across the pool. It returns an **iterator** that yields results in\n", + "the **same order** as the input.\n", + "\n", + "Key differences from `multiprocessing.Pool.map()`:\n", + "- Returns a lazy iterator (not a list)\n", + "- Supports a `timeout` parameter\n", + "- Supports a `chunksize` parameter for efficiency with large iterables" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e5f6a7b8", + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "from concurrent.futures import ProcessPoolExecutor\n", + "\n", + "\n", + "def process_item(x: int) -> tuple[int, int, int]:\n", + " \"\"\"Process an item and return (input, result, pid).\"\"\"\n", + " return (x, x * x, os.getpid())\n", + "\n", + "\n", + "# map() preserves input order\n", + "with ProcessPoolExecutor(max_workers=2) as executor:\n", + " results = list(executor.map(process_item, range(8)))\n", + "\n", + "main_pid: int = os.getpid()\n", + "print(f\"Main PID: {main_pid}\")\n", + "print(\"\\nResults (order preserved):\")\n", + "for x, squared, pid in results:\n", + " print(f\" {x}^2 = {squared} (worker PID: {pid})\")\n", + "\n", + "worker_pids: set[int] = {pid for _, _, pid in results}\n", + "print(f\"\\nUnique worker PIDs: {worker_pids}\")\n", + "print(f\"All workers differ from main: {all(pid != main_pid for _, _, pid in results)}\")" + ] + }, + { + "cell_type": "markdown", + "id": "f6a7b8c9", + "metadata": {}, + "source": [ + "## executor.submit() and Future Objects\n", + "\n", + "`executor.submit(func, *args, **kwargs)` submits a single callable for execution and\n", + "returns a `Future` object immediately. The `Future` represents a computation that may\n", + "not have completed yet.\n", + "\n", + "Key `Future` methods:\n", + "- `result(timeout=None)` -- block until the result is ready, then return it\n", + "- `done()` -- return `True` if the computation has completed\n", + "- `cancelled()` -- return `True` if the task was cancelled\n", + "- `exception(timeout=None)` -- return the exception, if any\n", + "- `add_done_callback(fn)` -- register a callback to run when the future completes" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a7b8c9d0", + "metadata": {}, + "outputs": [], + "source": [ + "from concurrent.futures import Future, ProcessPoolExecutor\n", + "\n", + "\n", + "def square(x: int) -> int:\n", + " \"\"\"Compute the square of a number.\"\"\"\n", + " return x * x\n", + "\n", + "\n", + "# submit() returns a Future\n", + "with ProcessPoolExecutor(max_workers=1) as executor:\n", + " future: Future[int] = executor.submit(square, 7)\n", + "\n", + " # Inspect the future\n", + " print(f\"Type: {type(future).__name__}\")\n", + " print(f\"Done: {future.done()}\") # May or may not be done yet\n", + "\n", + " # Get the result (blocks if not yet ready)\n", + " result: int = future.result()\n", + " print(f\"Result: {result}\")\n", + " print(f\"Done after result(): {future.done()}\")\n", + " print(f\"Exception: {future.exception()}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b8c9d0e1", + "metadata": {}, + "outputs": [], + "source": [ + "from concurrent.futures import Future, ProcessPoolExecutor\n", + "\n", + "\n", + "def cube(x: int) -> int:\n", + " \"\"\"Compute the cube of a number.\"\"\"\n", + " return x ** 3\n", + "\n", + "\n", + "# Submit multiple tasks and collect futures\n", + "with ProcessPoolExecutor(max_workers=2) as executor:\n", + " futures: dict[Future[int], int] = {\n", + " executor.submit(cube, n): n\n", + " for n in range(1, 7)\n", + " }\n", + "\n", + " # Retrieve results from each future\n", + " print(\"Results from submit():\")\n", + " for future, input_val in futures.items():\n", + " result: int = future.result()\n", + " print(f\" {input_val}^3 = {result}\")" + ] + }, + { + "cell_type": "markdown", + "id": "c9d0e1f2", + "metadata": {}, + "source": [ + "## map() vs submit(): When to Use Each\n", + "\n", + "| Feature | `map()` | `submit()` |\n", + "|---------|---------|:-----------|\n", + "| Returns | Iterator of results | Individual `Future` objects |\n", + "| Order | Results in input order | Results in completion order (with `as_completed`) |\n", + "| Arguments | Single iterable per param | Full control of args/kwargs |\n", + "| Use case | Batch processing same operation | Different tasks or custom handling |\n", + "| Error handling | Exception raised during iteration | Exception stored in `Future` |\n", + "\n", + "**Rule of thumb**: Use `map()` for simple \"apply same function to many inputs\" tasks.\n", + "Use `submit()` when you need more control over individual tasks." + ] + }, + { + "cell_type": "markdown", + "id": "d0e1f2a3", + "metadata": {}, + "source": [ + "## as_completed(): Results in Completion Order\n", + "\n", + "`concurrent.futures.as_completed(futures)` yields futures as they **finish**, not in\n", + "the order they were submitted. This is useful when you want to process results as\n", + "soon as they are available, rather than waiting for all of them." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e1f2a3b4", + "metadata": {}, + "outputs": [], + "source": [ + "import time\n", + "from concurrent.futures import Future, ProcessPoolExecutor, as_completed\n", + "\n", + "\n", + "def variable_work(task_id: int) -> tuple[int, float]:\n", + " \"\"\"Simulate work with variable duration.\"\"\"\n", + " # Tasks with higher IDs finish faster\n", + " duration: float = 0.5 - (task_id * 0.1)\n", + " duration = max(duration, 0.05)\n", + " time.sleep(duration)\n", + " return (task_id, duration)\n", + "\n", + "\n", + "with ProcessPoolExecutor(max_workers=3) as executor:\n", + " futures: dict[Future[tuple[int, float]], int] = {\n", + " executor.submit(variable_work, i): i\n", + " for i in range(5)\n", + " }\n", + "\n", + " print(\"Results in completion order:\")\n", + " for future in as_completed(futures):\n", + " task_id, duration = future.result()\n", + " print(f\" Task {task_id} completed (took {duration:.2f}s)\")" + ] + }, + { + "cell_type": "markdown", + "id": "f2a3b4c5", + "metadata": {}, + "source": [ + "## Error Handling with Futures\n", + "\n", + "When a worker function raises an exception, it is captured by the `Future` object and\n", + "re-raised when you call `.result()` or `.exception()`. This makes error handling\n", + "straightforward." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a3b4c5d6", + "metadata": {}, + "outputs": [], + "source": [ + "from concurrent.futures import Future, ProcessPoolExecutor\n", + "\n", + "\n", + "def divide(a: float, b: float) -> float:\n", + " \"\"\"Divide a by b. Raises ZeroDivisionError if b is zero.\"\"\"\n", + " return a / b\n", + "\n", + "\n", + "with ProcessPoolExecutor(max_workers=2) as executor:\n", + " # Submit tasks, one of which will fail\n", + " tasks: list[tuple[float, float]] = [(10, 2), (20, 4), (30, 0), (40, 5)]\n", + " futures: list[tuple[Future[float], tuple[float, float]]] = [\n", + " (executor.submit(divide, a, b), (a, b))\n", + " for a, b in tasks\n", + " ]\n", + "\n", + " for future, (a, b) in futures:\n", + " try:\n", + " result: float = future.result()\n", + " print(f\" {a} / {b} = {result}\")\n", + " except ZeroDivisionError:\n", + " print(f\" {a} / {b} = ERROR (division by zero)\")\n", + " # Inspect the exception stored in the Future\n", + " print(f\" future.exception() = {future.exception()!r}\")" + ] + }, + { + "cell_type": "markdown", + "id": "b4c5d6e7", + "metadata": {}, + "source": [ + "## CPU-Bound vs I/O-Bound Workloads\n", + "\n", + "Choosing between `ProcessPoolExecutor` and `ThreadPoolExecutor` depends on the nature\n", + "of your workload:\n", + "\n", + "| Workload | Bottleneck | Best Executor | Why |\n", + "|----------|-----------|:--------------|:----|\n", + "| **CPU-bound** | Computation | `ProcessPoolExecutor` | Bypasses GIL, true parallelism |\n", + "| **I/O-bound** | Network, disk | `ThreadPoolExecutor` | Threads release GIL during I/O |\n", + "| **Mixed** | Both | Depends | Profile first; consider async I/O |\n", + "\n", + "The GIL (Global Interpreter Lock) prevents multiple threads from executing Python bytecode\n", + "simultaneously, but it is released during I/O operations. Processes each have their own GIL,\n", + "so CPU-bound work runs truly in parallel." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c5d6e7f8", + "metadata": {}, + "outputs": [], + "source": [ + "import time\n", + "from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor\n", + "\n", + "\n", + "def cpu_heavy(n: int) -> int:\n", + " \"\"\"CPU-bound: compute sum of squares.\"\"\"\n", + " total: int = 0\n", + " for i in range(n):\n", + " total += i * i\n", + " return total\n", + "\n", + "\n", + "workload: list[int] = [1_500_000] * 8\n", + "\n", + "# Sequential baseline\n", + "start: float = time.perf_counter()\n", + "_ = [cpu_heavy(n) for n in workload]\n", + "seq_time: float = time.perf_counter() - start\n", + "\n", + "# Threads (limited by GIL for CPU-bound work)\n", + "start = time.perf_counter()\n", + "with ThreadPoolExecutor(max_workers=4) as executor:\n", + " _ = list(executor.map(cpu_heavy, workload))\n", + "thread_time: float = time.perf_counter() - start\n", + "\n", + "# Processes (bypasses GIL)\n", + "start = time.perf_counter()\n", + "with ProcessPoolExecutor(max_workers=4) as executor:\n", + " _ = list(executor.map(cpu_heavy, workload))\n", + "process_time: float = time.perf_counter() - start\n", + "\n", + "print(\"CPU-bound workload comparison:\")\n", + "print(f\" Sequential: {seq_time:.3f}s\")\n", + "print(f\" ThreadPoolExecutor: {thread_time:.3f}s (speedup: {seq_time/thread_time:.2f}x)\")\n", + "print(f\" ProcessPoolExecutor: {process_time:.3f}s (speedup: {seq_time/process_time:.2f}x)\")\n", + "print(f\"\\nProcessPoolExecutor wins for CPU-bound tasks!\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "d6e7f8a9", + "metadata": {}, + "outputs": [], + "source": [ + "import time\n", + "from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor\n", + "\n", + "\n", + "def io_heavy(seconds: float) -> float:\n", + " \"\"\"I/O-bound: simulate waiting for a network response.\"\"\"\n", + " time.sleep(seconds)\n", + " return seconds\n", + "\n", + "\n", + "io_workload: list[float] = [0.2] * 8\n", + "\n", + "# Sequential baseline\n", + "start: float = time.perf_counter()\n", + "_ = [io_heavy(s) for s in io_workload]\n", + "seq_time: float = time.perf_counter() - start\n", + "\n", + "# Threads (efficient for I/O)\n", + "start = time.perf_counter()\n", + "with ThreadPoolExecutor(max_workers=4) as executor:\n", + " _ = list(executor.map(io_heavy, io_workload))\n", + "thread_time: float = time.perf_counter() - start\n", + "\n", + "# Processes (more overhead for I/O)\n", + "start = time.perf_counter()\n", + "with ProcessPoolExecutor(max_workers=4) as executor:\n", + " _ = list(executor.map(io_heavy, io_workload))\n", + "process_time: float = time.perf_counter() - start\n", + "\n", + "print(\"I/O-bound workload comparison:\")\n", + "print(f\" Sequential: {seq_time:.3f}s\")\n", + "print(f\" ThreadPoolExecutor: {thread_time:.3f}s (speedup: {seq_time/thread_time:.2f}x)\")\n", + "print(f\" ProcessPoolExecutor: {process_time:.3f}s (speedup: {seq_time/process_time:.2f}x)\")\n", + "print(f\"\\nBoth work for I/O, but threads have less overhead!\")" + ] + }, + { + "cell_type": "markdown", + "id": "e7f8a9b0", + "metadata": {}, + "source": [ + "## Callbacks on Futures\n", + "\n", + "You can attach callbacks to `Future` objects using `add_done_callback()`. The callback\n", + "function is called with the `Future` as its argument when the task completes. This is\n", + "useful for logging, chaining tasks, or updating progress." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f8a9b0c1", + "metadata": {}, + "outputs": [], + "source": [ + "from concurrent.futures import Future, ProcessPoolExecutor\n", + "\n", + "\n", + "def compute_factorial(n: int) -> int:\n", + " \"\"\"Compute n! iteratively.\"\"\"\n", + " result: int = 1\n", + " for i in range(2, n + 1):\n", + " result *= i\n", + " return result\n", + "\n", + "\n", + "def on_complete(future: Future[int]) -> None:\n", + " \"\"\"Callback when a factorial computation completes.\"\"\"\n", + " result: int = future.result()\n", + " print(f\" Callback: computation finished, result = {result}\")\n", + "\n", + "\n", + "with ProcessPoolExecutor(max_workers=2) as executor:\n", + " print(\"Submitting tasks with callbacks:\")\n", + " for n in [5, 8, 10, 12]:\n", + " future: Future[int] = executor.submit(compute_factorial, n)\n", + " future.add_done_callback(on_complete)\n", + "\n", + "print(\"All tasks completed.\")" + ] + }, + { + "cell_type": "markdown", + "id": "a9b0c1d2", + "metadata": {}, + "source": [ + "## os.cpu_count() and Choosing max_workers\n", + "\n", + "`os.cpu_count()` returns the number of logical CPUs on the system. This is the default\n", + "value for `max_workers` in `ProcessPoolExecutor`.\n", + "\n", + "Guidelines for choosing `max_workers`:\n", + "- **CPU-bound**: Use `os.cpu_count()` or slightly less (leave room for other processes)\n", + "- **I/O-bound**: Can use more workers than CPUs since they spend time waiting\n", + "- **Memory-constrained**: Each process uses its own memory, so watch usage" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b0c1d2e3", + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "from concurrent.futures import ProcessPoolExecutor\n", + "\n", + "\n", + "def get_pid(_: int = 0) -> int:\n", + " \"\"\"Return the current process ID.\"\"\"\n", + " return os.getpid()\n", + "\n", + "\n", + "# os.cpu_count() tells us how many CPUs are available\n", + "cpus: int | None = os.cpu_count()\n", + "print(f\"os.cpu_count(): {cpus}\")\n", + "assert cpus is not None\n", + "assert cpus >= 1\n", + "\n", + "# Default max_workers = cpu_count\n", + "with ProcessPoolExecutor() as executor:\n", + " # _max_workers is an implementation detail, shown for educational purposes\n", + " actual_workers: int | None = executor._max_workers\n", + " print(f\"Default max_workers: {actual_workers}\")\n", + "\n", + "# Practical: verify processes run on different PIDs\n", + "with ProcessPoolExecutor(max_workers=4) as executor:\n", + " pids: list[int] = list(executor.map(get_pid, range(8)))\n", + " unique: set[int] = set(pids)\n", + " print(f\"\\n8 tasks across 4 workers:\")\n", + " print(f\" PIDs: {pids}\")\n", + " print(f\" Unique PIDs: {unique}\")\n", + " print(f\" All differ from main ({os.getpid()}): {all(p != os.getpid() for p in pids)}\")" + ] + }, + { + "cell_type": "markdown", + "id": "c1d2e3f4", + "metadata": {}, + "source": [ + "## Practical: Real-World Pattern -- Parallel Data Processing\n", + "\n", + "A common real-world pattern is to process a batch of items in parallel, collecting\n", + "successes and failures separately. This example demonstrates using `submit()` with\n", + "error handling for robust parallel processing." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "d2e3f4a5", + "metadata": {}, + "outputs": [], + "source": [ + "from concurrent.futures import Future, ProcessPoolExecutor, as_completed\n", + "\n", + "\n", + "def process_record(record_id: int) -> dict[str, int | str]:\n", + " \"\"\"Simulate processing a data record. Some records fail.\"\"\"\n", + " if record_id % 5 == 0:\n", + " raise ValueError(f\"Record {record_id} is corrupted\")\n", + " return {\"id\": record_id, \"status\": \"processed\", \"value\": record_id * 10}\n", + "\n", + "\n", + "record_ids: list[int] = list(range(1, 16))\n", + "successes: list[dict[str, int | str]] = []\n", + "failures: list[tuple[int, str]] = []\n", + "\n", + "with ProcessPoolExecutor(max_workers=3) as executor:\n", + " # Submit all tasks\n", + " future_to_id: dict[Future[dict[str, int | str]], int] = {\n", + " executor.submit(process_record, rid): rid\n", + " for rid in record_ids\n", + " }\n", + "\n", + " # Process results as they complete\n", + " for future in as_completed(future_to_id):\n", + " record_id: int = future_to_id[future]\n", + " try:\n", + " result: dict[str, int | str] = future.result()\n", + " successes.append(result)\n", + " except ValueError as e:\n", + " failures.append((record_id, str(e)))\n", + "\n", + "print(f\"Processed {len(successes)} records successfully:\")\n", + "for s in sorted(successes, key=lambda x: x[\"id\"]):\n", + " print(f\" Record {s['id']}: value={s['value']}\")\n", + "\n", + "print(f\"\\n{len(failures)} records failed:\")\n", + "for record_id, error in sorted(failures):\n", + " print(f\" Record {record_id}: {error}\")" + ] + }, + { + "cell_type": "markdown", + "id": "e3f4a5b6", + "metadata": {}, + "source": [ + "## Summary\n", + "\n", + "### Key Takeaways\n", + "\n", + "| Concept | API | Purpose |\n", + "|---------|-----|:--------|\n", + "| **Executor** | `ProcessPoolExecutor(max_workers=n)` | High-level process pool |\n", + "| **map()** | `executor.map(func, iterable)` | Parallel map, results in input order |\n", + "| **submit()** | `executor.submit(func, *args)` | Submit single task, returns Future |\n", + "| **Future** | `future.result()`, `.done()`, `.exception()` | Pending result handle |\n", + "| **as_completed()** | `as_completed(futures)` | Yield futures in completion order |\n", + "| **Callbacks** | `future.add_done_callback(fn)` | Run code when a future completes |\n", + "| **cpu_count()** | `os.cpu_count()` | Number of logical CPUs |\n", + "\n", + "### Choosing the Right Executor\n", + "\n", + "| Scenario | Use |\n", + "|----------|:----|\n", + "| Heavy computation (math, parsing, compression) | `ProcessPoolExecutor` |\n", + "| Network requests, file I/O, database queries | `ThreadPoolExecutor` |\n", + "| Simple batch: same function, many inputs | `executor.map()` |\n", + "| Complex: different tasks, error handling, callbacks | `executor.submit()` |\n", + "\n", + "### Best Practices\n", + "- Always use executors as context managers to ensure proper cleanup\n", + "- Define worker functions at module level (they must be picklable)\n", + "- Use `as_completed()` when you want to process results as soon as they are ready\n", + "- Handle exceptions from `.result()` -- they are re-raised from the worker\n", + "- For CPU-bound tasks, `max_workers=os.cpu_count()` is a good starting point\n", + "- Prefer `ProcessPoolExecutor` over `multiprocessing.Pool` for new code -- it has a cleaner API" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "version": "3.13.0" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} \ No newline at end of file diff --git a/src/chapter_27/README.md b/src/chapter_27/README.md new file mode 100644 index 0000000..c470c94 --- /dev/null +++ b/src/chapter_27/README.md @@ -0,0 +1,9 @@ +# Chapter 27: Multiprocessing and Parallelism + +True parallelism with `multiprocessing` — processes, pools, shared state, inter-process communication, and `ProcessPoolExecutor`. + +## Notebooks + +1. **01_multiprocessing_basics.ipynb** — `Process`, `Pool`, `map()`, process lifecycle +2. **02_shared_state_and_ipc.ipynb** — `Queue`, `Pipe`, `Value`, `Array`, `Manager`, locks +3. **03_process_pool_executor.ipynb** — `ProcessPoolExecutor`, `Future`, CPU-bound vs I/O-bound, `map()` vs `submit()` diff --git a/src/chapter_27/__init__.py b/src/chapter_27/__init__.py new file mode 100644 index 0000000..cfa6ee1 --- /dev/null +++ b/src/chapter_27/__init__.py @@ -0,0 +1 @@ +"""Chapter 27: Multiprocessing and Parallelism.""" diff --git a/src/chapter_28/01_argparse_fundamentals.ipynb b/src/chapter_28/01_argparse_fundamentals.ipynb new file mode 100644 index 0000000..f8198ac --- /dev/null +++ b/src/chapter_28/01_argparse_fundamentals.ipynb @@ -0,0 +1,559 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "a1b2c3d4", + "metadata": {}, + "source": [ + "# Chapter 28: Argparse Fundamentals\n", + "\n", + "This notebook covers the `argparse` module for building command-line interfaces in Python. We explore `ArgumentParser`, positional and optional arguments, type conversion, choices, defaults, and `nargs`.\n", + "\n", + "## Key Concepts\n", + "- **ArgumentParser**: The main entry point for defining CLI arguments\n", + "- **Positional arguments**: Required arguments identified by position\n", + "- **Optional arguments**: Flag-style arguments with `--` prefix\n", + "- **Type conversion**: Automatic conversion of string inputs to desired types\n", + "- **Choices and defaults**: Restricting and pre-filling argument values" + ] + }, + { + "cell_type": "markdown", + "id": "b2c3d4e5", + "metadata": {}, + "source": [ + "## Section 1: Creating an ArgumentParser\n", + "\n", + "`ArgumentParser` is the central class in `argparse`. It holds all the information needed to parse the command line into Python data types." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c3d4e5f6", + "metadata": {}, + "outputs": [], + "source": [ + "import argparse\n", + "\n", + "# Create a basic parser with a description\n", + "parser: argparse.ArgumentParser = argparse.ArgumentParser(\n", + " description=\"A sample command-line tool\",\n", + " prog=\"mytool\", # Override the program name\n", + ")\n", + "\n", + "print(f\"Parser type: {type(parser).__name__}\")\n", + "print(f\"Program name: {parser.prog}\")\n", + "print(f\"Description: {parser.description}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "d4e5f6a7", + "metadata": {}, + "outputs": [], + "source": [ + "# The parser generates help text automatically\n", + "parser = argparse.ArgumentParser(\n", + " prog=\"greet\",\n", + " description=\"Greet someone on the command line\",\n", + " epilog=\"Example: greet Alice --loud\",\n", + ")\n", + "\n", + "# print_help() shows what --help would display\n", + "parser.print_help()" + ] + }, + { + "cell_type": "markdown", + "id": "e5f6a7b8", + "metadata": {}, + "source": [ + "## Section 2: Positional Arguments\n", + "\n", + "Positional arguments are required by default and identified by their position on the command line rather than by a flag." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f6a7b8c9", + "metadata": {}, + "outputs": [], + "source": [ + "# Positional arguments are required\n", + "parser = argparse.ArgumentParser(prog=\"greet\")\n", + "parser.add_argument(\"name\") # A simple positional argument\n", + "\n", + "# parse_args takes a list of strings (simulating sys.argv[1:])\n", + "args: argparse.Namespace = parser.parse_args([\"Alice\"])\n", + "\n", + "print(f\"Parsed name: {args.name}\")\n", + "print(f\"Namespace type: {type(args).__name__}\")\n", + "print(f\"As dict: {vars(args)}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a7b8c9d0", + "metadata": {}, + "outputs": [], + "source": [ + "# Multiple positional arguments are parsed in order\n", + "parser = argparse.ArgumentParser(prog=\"copy\")\n", + "parser.add_argument(\"source\")\n", + "parser.add_argument(\"destination\")\n", + "\n", + "args = parser.parse_args([\"input.txt\", \"output.txt\"])\n", + "\n", + "print(f\"Source: {args.source}\")\n", + "print(f\"Destination: {args.destination}\")" + ] + }, + { + "cell_type": "markdown", + "id": "b8c9d0e1", + "metadata": {}, + "source": [ + "## Section 3: Optional Arguments\n", + "\n", + "Optional arguments use the `--` prefix (long form) or `-` prefix (short form). They are not required unless you set `required=True`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c9d0e1f2", + "metadata": {}, + "outputs": [], + "source": [ + "# Optional arguments with -- prefix\n", + "parser = argparse.ArgumentParser(prog=\"server\")\n", + "parser.add_argument(\"--verbose\", action=\"store_true\")\n", + "parser.add_argument(\"--count\", type=int, default=1)\n", + "\n", + "# Providing both optional arguments\n", + "args = parser.parse_args([\"--verbose\", \"--count\", \"5\"])\n", + "print(f\"verbose: {args.verbose} (type: {type(args.verbose).__name__})\")\n", + "print(f\"count: {args.count} (type: {type(args.count).__name__})\")\n", + "\n", + "# Omitting optional arguments uses defaults\n", + "args_default = parser.parse_args([])\n", + "print(f\"\\nDefaults -> verbose: {args_default.verbose}, count: {args_default.count}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "d0e1f2a3", + "metadata": {}, + "outputs": [], + "source": [ + "# Short and long flags together\n", + "parser = argparse.ArgumentParser(prog=\"tool\")\n", + "parser.add_argument(\"-v\", \"--verbose\", action=\"store_true\")\n", + "parser.add_argument(\"-o\", \"--output\", type=str, default=\"stdout\")\n", + "\n", + "# Using short flags\n", + "args = parser.parse_args([\"-v\", \"-o\", \"results.txt\"])\n", + "print(f\"verbose: {args.verbose}\")\n", + "print(f\"output: {args.output}\")\n", + "\n", + "# Using long flags produces the same result\n", + "args_long = parser.parse_args([\"--verbose\", \"--output\", \"results.txt\"])\n", + "print(f\"\\nSame result with long flags: {vars(args) == vars(args_long)}\")" + ] + }, + { + "cell_type": "markdown", + "id": "e1f2a3b4", + "metadata": {}, + "source": [ + "## Section 4: Type Conversion\n", + "\n", + "By default, all arguments are parsed as strings. The `type` parameter automatically converts the input to the desired Python type." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f2a3b4c5", + "metadata": {}, + "outputs": [], + "source": [ + "# Built-in type conversion\n", + "parser = argparse.ArgumentParser(prog=\"calc\")\n", + "parser.add_argument(\"value\", type=int)\n", + "parser.add_argument(\"--factor\", type=float, default=1.0)\n", + "\n", + "args = parser.parse_args([\"42\", \"--factor\", \"2.5\"])\n", + "\n", + "result: float = args.value * args.factor\n", + "print(f\"value: {args.value} (type: {type(args.value).__name__})\")\n", + "print(f\"factor: {args.factor} (type: {type(args.factor).__name__})\")\n", + "print(f\"result: {result}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a3b4c5d6", + "metadata": {}, + "outputs": [], + "source": [ + "import pathlib\n", + "\n", + "# Custom type functions for validation\n", + "def positive_int(value: str) -> int:\n", + " \"\"\"Convert string to positive integer, raising on invalid input.\"\"\"\n", + " ivalue: int = int(value)\n", + " if ivalue <= 0:\n", + " raise argparse.ArgumentTypeError(f\"{value} is not a positive integer\")\n", + " return ivalue\n", + "\n", + "parser = argparse.ArgumentParser(prog=\"worker\")\n", + "parser.add_argument(\"--threads\", type=positive_int, default=4)\n", + "parser.add_argument(\"--config\", type=pathlib.Path, default=pathlib.Path(\"config.toml\"))\n", + "\n", + "args = parser.parse_args([\"--threads\", \"8\", \"--config\", \"/etc/app/config.toml\"])\n", + "print(f\"threads: {args.threads} (type: {type(args.threads).__name__})\")\n", + "print(f\"config: {args.config} (type: {type(args.config).__name__})\")" + ] + }, + { + "cell_type": "markdown", + "id": "b4c5d6e7", + "metadata": {}, + "source": [ + "## Section 5: Choices and Defaults\n", + "\n", + "The `choices` parameter restricts an argument to a set of allowed values. The `default` parameter sets a fallback when the argument is omitted." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c5d6e7f8", + "metadata": {}, + "outputs": [], + "source": [ + "# choices restricts accepted values\n", + "parser = argparse.ArgumentParser(prog=\"logger\")\n", + "parser.add_argument(\n", + " \"--level\",\n", + " choices=[\"debug\", \"info\", \"warning\", \"error\", \"critical\"],\n", + " default=\"info\",\n", + ")\n", + "\n", + "args = parser.parse_args([\"--level\", \"debug\"])\n", + "print(f\"Log level: {args.level}\")\n", + "\n", + "# Default value when argument is omitted\n", + "args_default = parser.parse_args([])\n", + "print(f\"Default level: {args_default.level}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "d6e7f8a9", + "metadata": {}, + "outputs": [], + "source": [ + "# Demonstrating what happens with invalid choices\n", + "parser = argparse.ArgumentParser(prog=\"format\")\n", + "parser.add_argument(\"--output-format\", choices=[\"json\", \"csv\", \"xml\"])\n", + "\n", + "# Valid choice\n", + "args = parser.parse_args([\"--output-format\", \"json\"])\n", + "print(f\"Format: {args.output_format}\")\n", + "\n", + "# Invalid choice raises an error\n", + "try:\n", + " parser.parse_args([\"--output-format\", \"yaml\"])\n", + "except SystemExit:\n", + " print(\"Invalid choice 'yaml' - argparse rejects it and exits\")" + ] + }, + { + "cell_type": "markdown", + "id": "e7f8a9b0", + "metadata": {}, + "source": [ + "## Section 6: Nargs — Multiple Values\n", + "\n", + "The `nargs` parameter controls how many values an argument consumes:\n", + "- `nargs='+'` — one or more values (returns a list)\n", + "- `nargs='*'` — zero or more values (returns a list)\n", + "- `nargs='?'` — zero or one value\n", + "- `nargs=N` — exactly N values (returns a list)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f8a9b0c1", + "metadata": {}, + "outputs": [], + "source": [ + "# nargs='+' requires one or more values\n", + "parser = argparse.ArgumentParser(prog=\"process\")\n", + "parser.add_argument(\"files\", nargs=\"+\")\n", + "\n", + "args = parser.parse_args([\"a.txt\", \"b.txt\", \"c.txt\"])\n", + "print(f\"Files: {args.files}\")\n", + "print(f\"Type: {type(args.files).__name__}\")\n", + "print(f\"Count: {len(args.files)}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a9b0c1d2", + "metadata": {}, + "outputs": [], + "source": [ + "# nargs='*' accepts zero or more values\n", + "parser = argparse.ArgumentParser(prog=\"collect\")\n", + "parser.add_argument(\"--tags\", nargs=\"*\", default=[])\n", + "parser.add_argument(\"--ports\", nargs=3, type=int) # Exactly 3 values\n", + "\n", + "args = parser.parse_args([\"--tags\", \"dev\", \"test\", \"--ports\", \"80\", \"443\", \"8080\"])\n", + "print(f\"Tags: {args.tags}\")\n", + "print(f\"Ports: {args.ports}\")\n", + "\n", + "# Zero tags is valid with nargs='*'\n", + "args_empty = parser.parse_args([\"--tags\", \"--ports\", \"80\", \"443\", \"8080\"])\n", + "print(f\"\\nEmpty tags: {args_empty.tags}\")" + ] + }, + { + "cell_type": "markdown", + "id": "b0c1d2e3", + "metadata": {}, + "source": [ + "## Section 7: Actions\n", + "\n", + "The `action` parameter controls how arguments are processed:\n", + "- `\"store\"` — default, stores the value\n", + "- `\"store_true\"` / `\"store_false\"` — boolean flags\n", + "- `\"count\"` — counts occurrences (e.g., `-vvv`)\n", + "- `\"append\"` — collects repeated arguments into a list" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c1d2e3f4", + "metadata": {}, + "outputs": [], + "source": [ + "# Demonstrating various actions\n", + "parser = argparse.ArgumentParser(prog=\"demo\")\n", + "\n", + "# store_true / store_false for boolean flags\n", + "parser.add_argument(\"--debug\", action=\"store_true\")\n", + "parser.add_argument(\"--no-cache\", action=\"store_true\")\n", + "\n", + "# count tracks how many times a flag appears\n", + "parser.add_argument(\"-v\", \"--verbose\", action=\"count\", default=0)\n", + "\n", + "# append collects repeated flags into a list\n", + "parser.add_argument(\"--include\", action=\"append\", default=[])\n", + "\n", + "args = parser.parse_args([\n", + " \"--debug\",\n", + " \"-vvv\", # Three v's = verbosity 3\n", + " \"--include\", \"src\",\n", + " \"--include\", \"tests\",\n", + "])\n", + "\n", + "print(f\"debug: {args.debug}\")\n", + "print(f\"no_cache: {args.no_cache}\")\n", + "print(f\"verbosity: {args.verbose}\")\n", + "print(f\"includes: {args.include}\")" + ] + }, + { + "cell_type": "markdown", + "id": "d2e3f4a5", + "metadata": {}, + "source": [ + "## Section 8: Help Text and Metavar\n", + "\n", + "Good CLI tools provide clear help text. The `help` parameter adds descriptions, and `metavar` customizes the placeholder name in usage messages." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e3f4a5b6", + "metadata": {}, + "outputs": [], + "source": [ + "# Well-documented argument parser\n", + "parser = argparse.ArgumentParser(\n", + " prog=\"deploy\",\n", + " description=\"Deploy an application to the target environment\",\n", + ")\n", + "parser.add_argument(\n", + " \"app_name\",\n", + " help=\"Name of the application to deploy\",\n", + " metavar=\"APP\",\n", + ")\n", + "parser.add_argument(\n", + " \"--env\",\n", + " choices=[\"dev\", \"staging\", \"prod\"],\n", + " default=\"dev\",\n", + " help=\"Target environment (default: %(default)s)\",\n", + ")\n", + "parser.add_argument(\n", + " \"--replicas\",\n", + " type=int,\n", + " default=1,\n", + " metavar=\"N\",\n", + " help=\"Number of replicas to run (default: %(default)s)\",\n", + ")\n", + "parser.add_argument(\n", + " \"--dry-run\",\n", + " action=\"store_true\",\n", + " help=\"Show what would be done without executing\",\n", + ")\n", + "\n", + "parser.print_help()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f4a5b6c7", + "metadata": {}, + "outputs": [], + "source": [ + "# Parsing with the documented parser\n", + "args = parser.parse_args([\"myapp\", \"--env\", \"staging\", \"--replicas\", \"3\", \"--dry-run\"])\n", + "\n", + "print(f\"App: {args.app_name}\")\n", + "print(f\"Environment: {args.env}\")\n", + "print(f\"Replicas: {args.replicas}\")\n", + "print(f\"Dry run: {args.dry_run}\")\n", + "\n", + "# Namespace can be accessed as a dictionary too\n", + "config: dict[str, object] = vars(args)\n", + "print(f\"\\nFull config: {config}\")" + ] + }, + { + "cell_type": "markdown", + "id": "a5b6c7d8", + "metadata": {}, + "source": [ + "## Section 9: Putting It All Together\n", + "\n", + "A realistic example combining positional arguments, optional flags, types, choices, and defaults." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b6c7d8e9", + "metadata": {}, + "outputs": [], + "source": [ + "def build_parser() -> argparse.ArgumentParser:\n", + " \"\"\"Build a complete argument parser for a file search tool.\"\"\"\n", + " parser = argparse.ArgumentParser(\n", + " prog=\"filesearch\",\n", + " description=\"Search for patterns in files\",\n", + " )\n", + "\n", + " # Positional: the pattern to search for\n", + " parser.add_argument(\"pattern\", help=\"Regex pattern to search for\")\n", + "\n", + " # Positional: one or more files\n", + " parser.add_argument(\"files\", nargs=\"+\", help=\"Files to search\")\n", + "\n", + " # Optional flags\n", + " parser.add_argument(\"-i\", \"--ignore-case\", action=\"store_true\",\n", + " help=\"Case-insensitive matching\")\n", + " parser.add_argument(\"-n\", \"--line-numbers\", action=\"store_true\",\n", + " help=\"Show line numbers\")\n", + " parser.add_argument(\"--max-results\", type=positive_int, default=100,\n", + " metavar=\"N\", help=\"Maximum results (default: %(default)s)\")\n", + " parser.add_argument(\"--format\", choices=[\"text\", \"json\", \"csv\"],\n", + " default=\"text\", help=\"Output format (default: %(default)s)\")\n", + "\n", + " return parser\n", + "\n", + "\n", + "search_parser: argparse.ArgumentParser = build_parser()\n", + "search_parser.print_help()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c7d8e9f0", + "metadata": {}, + "outputs": [], + "source": [ + "# Simulate command-line invocation\n", + "args = search_parser.parse_args([\n", + " \"TODO\",\n", + " \"main.py\", \"utils.py\", \"config.py\",\n", + " \"--ignore-case\",\n", + " \"--line-numbers\",\n", + " \"--max-results\", \"50\",\n", + " \"--format\", \"json\",\n", + "])\n", + "\n", + "print(f\"Pattern: {args.pattern!r}\")\n", + "print(f\"Files: {args.files}\")\n", + "print(f\"Ignore case: {args.ignore_case}\")\n", + "print(f\"Line numbers: {args.line_numbers}\")\n", + "print(f\"Max results: {args.max_results}\")\n", + "print(f\"Format: {args.format}\")" + ] + }, + { + "cell_type": "markdown", + "id": "d8e9f0a1", + "metadata": {}, + "source": [ + "## Summary\n", + "\n", + "### ArgumentParser Basics\n", + "- **`ArgumentParser()`** creates a parser with optional `prog`, `description`, and `epilog`\n", + "- **`add_argument()`** defines arguments with their type, default, help, and constraints\n", + "- **`parse_args()`** returns a `Namespace` object with parsed values\n", + "\n", + "### Argument Types\n", + "- **Positional**: Required by default, identified by order\n", + "- **Optional**: Prefixed with `--` (long) or `-` (short), not required by default\n", + "\n", + "### Key Parameters\n", + "- **`type`**: Converts input strings to the desired Python type\n", + "- **`choices`**: Restricts values to a predefined set\n", + "- **`default`**: Fallback value when argument is omitted\n", + "- **`nargs`**: Controls how many values an argument accepts (`+`, `*`, `?`, or `N`)\n", + "- **`action`**: Controls processing (`store_true`, `count`, `append`)\n", + "- **`help`** and **`metavar`**: Document the argument for users" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "version": "3.13.0" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} \ No newline at end of file diff --git a/src/chapter_28/02_subcommands_and_groups.ipynb b/src/chapter_28/02_subcommands_and_groups.ipynb new file mode 100644 index 0000000..6942da2 --- /dev/null +++ b/src/chapter_28/02_subcommands_and_groups.ipynb @@ -0,0 +1,504 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "a1000001", + "metadata": {}, + "source": [ + "# Chapter 28: Subcommands and Groups\n", + "\n", + "This notebook covers advanced `argparse` patterns for building complex CLI tools: subparsers for git-style subcommands, mutually exclusive groups for conflicting options, and argument groups for organized help output.\n", + "\n", + "## Key Concepts\n", + "- **Subparsers**: Create distinct subcommands (like `git init`, `git clone`)\n", + "- **Mutually exclusive groups**: Prevent conflicting options from being used together\n", + "- **Argument groups**: Organize related arguments in help output\n", + "- **Defaults per subcommand**: Each subcommand can have its own arguments and defaults" + ] + }, + { + "cell_type": "markdown", + "id": "a1000002", + "metadata": {}, + "source": [ + "## Section 1: Subparsers — Git-Style Subcommands\n", + "\n", + "Many CLI tools use subcommands: `git clone`, `docker run`, `pip install`. In `argparse`, this is achieved with `add_subparsers()`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a1000003", + "metadata": {}, + "outputs": [], + "source": [ + "import argparse\n", + "\n", + "# Create the top-level parser\n", + "parser: argparse.ArgumentParser = argparse.ArgumentParser(prog=\"vcs\")\n", + "\n", + "# add_subparsers creates a special action for subcommands\n", + "# dest stores which subcommand was chosen\n", + "subparsers = parser.add_subparsers(dest=\"command\", help=\"Available commands\")\n", + "\n", + "# Each subcommand gets its own parser\n", + "init_parser = subparsers.add_parser(\"init\", help=\"Initialize a new repository\")\n", + "init_parser.add_argument(\"--bare\", action=\"store_true\", help=\"Create a bare repository\")\n", + "\n", + "clone_parser = subparsers.add_parser(\"clone\", help=\"Clone a repository\")\n", + "clone_parser.add_argument(\"url\", help=\"Repository URL to clone\")\n", + "clone_parser.add_argument(\"--depth\", type=int, help=\"Shallow clone depth\")\n", + "\n", + "parser.print_help()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a1000004", + "metadata": {}, + "outputs": [], + "source": [ + "# Parsing the 'init' subcommand\n", + "args: argparse.Namespace = parser.parse_args([\"init\", \"--bare\"])\n", + "\n", + "print(f\"Command: {args.command}\")\n", + "print(f\"Bare: {args.bare}\")\n", + "print(f\"Full namespace: {vars(args)}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a1000005", + "metadata": {}, + "outputs": [], + "source": [ + "# Parsing the 'clone' subcommand\n", + "args = parser.parse_args([\"clone\", \"https://example.com/repo\", \"--depth\", \"1\"])\n", + "\n", + "print(f\"Command: {args.command}\")\n", + "print(f\"URL: {args.url}\")\n", + "print(f\"Depth: {args.depth}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a1000006", + "metadata": {}, + "outputs": [], + "source": [ + "# Subcommand-specific help\n", + "print(\"=== clone subcommand help ===\")\n", + "clone_parser.print_help()" + ] + }, + { + "cell_type": "markdown", + "id": "a1000007", + "metadata": {}, + "source": [ + "## Section 2: Dispatching Subcommands\n", + "\n", + "A common pattern is using `set_defaults(func=...)` to associate each subcommand with a handler function. This avoids long if/elif chains." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a1000008", + "metadata": {}, + "outputs": [], + "source": [ + "from typing import Any\n", + "\n", + "\n", + "def handle_start(args: argparse.Namespace) -> str:\n", + " \"\"\"Handle the 'start' subcommand.\"\"\"\n", + " return f\"Starting service '{args.name}' on port {args.port}\"\n", + "\n", + "\n", + "def handle_stop(args: argparse.Namespace) -> str:\n", + " \"\"\"Handle the 'stop' subcommand.\"\"\"\n", + " force_msg: str = \" (forced)\" if args.force else \"\"\n", + " return f\"Stopping service '{args.name}'{force_msg}\"\n", + "\n", + "\n", + "def handle_status(args: argparse.Namespace) -> str:\n", + " \"\"\"Handle the 'status' subcommand.\"\"\"\n", + " return f\"Status of all services (verbose={args.verbose})\"\n", + "\n", + "\n", + "# Build parser with function dispatch\n", + "parser = argparse.ArgumentParser(prog=\"svc\")\n", + "sub = parser.add_subparsers(dest=\"command\")\n", + "\n", + "# 'start' subcommand\n", + "start_cmd = sub.add_parser(\"start\")\n", + "start_cmd.add_argument(\"name\")\n", + "start_cmd.add_argument(\"--port\", type=int, default=8080)\n", + "start_cmd.set_defaults(func=handle_start)\n", + "\n", + "# 'stop' subcommand\n", + "stop_cmd = sub.add_parser(\"stop\")\n", + "stop_cmd.add_argument(\"name\")\n", + "stop_cmd.add_argument(\"--force\", action=\"store_true\")\n", + "stop_cmd.set_defaults(func=handle_stop)\n", + "\n", + "# 'status' subcommand\n", + "status_cmd = sub.add_parser(\"status\")\n", + "status_cmd.add_argument(\"-v\", \"--verbose\", action=\"store_true\")\n", + "status_cmd.set_defaults(func=handle_status)\n", + "\n", + "# Dispatch: parse args and call the associated function\n", + "for cmd_line in [\"start web --port 3000\", \"stop web --force\", \"status -v\"]:\n", + " args = parser.parse_args(cmd_line.split())\n", + " result: str = args.func(args)\n", + " print(f\" $ svc {cmd_line}\")\n", + " print(f\" -> {result}\\n\")" + ] + }, + { + "cell_type": "markdown", + "id": "a1000009", + "metadata": {}, + "source": [ + "## Section 3: Mutually Exclusive Groups\n", + "\n", + "Mutually exclusive groups prevent two options from being used at the same time. This is useful for output format flags or conflicting modes." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a1000010", + "metadata": {}, + "outputs": [], + "source": [ + "# Mutually exclusive output format flags\n", + "parser = argparse.ArgumentParser(prog=\"export\")\n", + "format_group = parser.add_mutually_exclusive_group()\n", + "format_group.add_argument(\"--json\", action=\"store_true\", help=\"Output as JSON\")\n", + "format_group.add_argument(\"--csv\", action=\"store_true\", help=\"Output as CSV\")\n", + "format_group.add_argument(\"--xml\", action=\"store_true\", help=\"Output as XML\")\n", + "\n", + "# Only one format flag can be used\n", + "args = parser.parse_args([\"--json\"])\n", + "print(f\"json: {args.json}, csv: {args.csv}, xml: {args.xml}\")\n", + "\n", + "args = parser.parse_args([\"--csv\"])\n", + "print(f\"json: {args.json}, csv: {args.csv}, xml: {args.xml}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a1000011", + "metadata": {}, + "outputs": [], + "source": [ + "# Attempting to use two mutually exclusive options\n", + "try:\n", + " parser.parse_args([\"--json\", \"--csv\"])\n", + "except SystemExit:\n", + " print(\"Error: --json and --csv cannot be used together\")\n", + " print(\"argparse raises SystemExit when mutually exclusive args conflict\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a1000012", + "metadata": {}, + "outputs": [], + "source": [ + "# required=True forces the user to pick exactly one option\n", + "parser = argparse.ArgumentParser(prog=\"convert\")\n", + "mode_group = parser.add_mutually_exclusive_group(required=True)\n", + "mode_group.add_argument(\"--encode\", action=\"store_true\", help=\"Encode the input\")\n", + "mode_group.add_argument(\"--decode\", action=\"store_true\", help=\"Decode the input\")\n", + "\n", + "parser.add_argument(\"input_file\", help=\"File to process\")\n", + "\n", + "args = parser.parse_args([\"--encode\", \"data.bin\"])\n", + "print(f\"encode: {args.encode}, decode: {args.decode}, file: {args.input_file}\")\n", + "\n", + "# Omitting both from a required group is an error\n", + "try:\n", + " parser.parse_args([\"data.bin\"])\n", + "except SystemExit:\n", + " print(\"\\nError: one of --encode or --decode is required\")" + ] + }, + { + "cell_type": "markdown", + "id": "a1000013", + "metadata": {}, + "source": [ + "## Section 4: Determining the Selected Format\n", + "\n", + "When using mutually exclusive boolean flags, a helper function can determine which was selected." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a1000014", + "metadata": {}, + "outputs": [], + "source": [ + "def get_output_format(args: argparse.Namespace) -> str:\n", + " \"\"\"Determine the selected output format from mutually exclusive flags.\"\"\"\n", + " if args.json:\n", + " return \"json\"\n", + " elif args.csv:\n", + " return \"csv\"\n", + " elif args.xml:\n", + " return \"xml\"\n", + " return \"text\" # Default fallback\n", + "\n", + "\n", + "parser = argparse.ArgumentParser(prog=\"report\")\n", + "fmt_group = parser.add_mutually_exclusive_group()\n", + "fmt_group.add_argument(\"--json\", action=\"store_true\")\n", + "fmt_group.add_argument(\"--csv\", action=\"store_true\")\n", + "fmt_group.add_argument(\"--xml\", action=\"store_true\")\n", + "\n", + "# Test each format\n", + "for flags in [[\"--json\"], [\"--csv\"], [\"--xml\"], []]:\n", + " args = parser.parse_args(flags)\n", + " fmt: str = get_output_format(args)\n", + " print(f\"Flags: {flags or ['(none)']} -> format: {fmt}\")" + ] + }, + { + "cell_type": "markdown", + "id": "a1000015", + "metadata": {}, + "source": [ + "## Section 5: Argument Groups\n", + "\n", + "Argument groups organize arguments into logical sections in the help output. Unlike mutually exclusive groups, they do not restrict which arguments can be used together." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a1000016", + "metadata": {}, + "outputs": [], + "source": [ + "# Argument groups for organized help output\n", + "parser = argparse.ArgumentParser(prog=\"deploy\", description=\"Deploy an application\")\n", + "\n", + "# Connection arguments\n", + "conn_group = parser.add_argument_group(\"connection\", \"Server connection options\")\n", + "conn_group.add_argument(\"--host\", default=\"localhost\", help=\"Server hostname\")\n", + "conn_group.add_argument(\"--port\", type=int, default=22, help=\"Server port\")\n", + "conn_group.add_argument(\"--user\", default=\"deploy\", help=\"SSH username\")\n", + "\n", + "# Deployment arguments\n", + "deploy_group = parser.add_argument_group(\"deployment\", \"Deployment configuration\")\n", + "deploy_group.add_argument(\"--branch\", default=\"main\", help=\"Git branch to deploy\")\n", + "deploy_group.add_argument(\"--tag\", help=\"Specific tag to deploy\")\n", + "deploy_group.add_argument(\"--rollback\", action=\"store_true\", help=\"Rollback to previous version\")\n", + "\n", + "# Notification arguments\n", + "notify_group = parser.add_argument_group(\"notifications\", \"Notification settings\")\n", + "notify_group.add_argument(\"--notify-slack\", action=\"store_true\", help=\"Send Slack notification\")\n", + "notify_group.add_argument(\"--notify-email\", help=\"Send email notification to address\")\n", + "\n", + "parser.print_help()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a1000017", + "metadata": {}, + "outputs": [], + "source": [ + "# All groups can be used simultaneously (they are just for organization)\n", + "args = parser.parse_args([\n", + " \"--host\", \"prod.example.com\",\n", + " \"--port\", \"2222\",\n", + " \"--branch\", \"release/v2\",\n", + " \"--notify-slack\",\n", + "])\n", + "\n", + "print(f\"Host: {args.host}:{args.port}\")\n", + "print(f\"User: {args.user}\")\n", + "print(f\"Branch: {args.branch}\")\n", + "print(f\"Notify Slack: {args.notify_slack}\")" + ] + }, + { + "cell_type": "markdown", + "id": "a1000018", + "metadata": {}, + "source": [ + "## Section 6: Combining Subcommands with Groups\n", + "\n", + "A real-world CLI often combines subparsers with argument groups and mutually exclusive options within each subcommand." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a1000019", + "metadata": {}, + "outputs": [], + "source": [ + "def build_db_cli() -> argparse.ArgumentParser:\n", + " \"\"\"Build a CLI parser for a database management tool.\"\"\"\n", + " parser = argparse.ArgumentParser(\n", + " prog=\"dbctl\",\n", + " description=\"Database management tool\",\n", + " )\n", + " parser.add_argument(\"-v\", \"--verbose\", action=\"store_true\")\n", + "\n", + " sub = parser.add_subparsers(dest=\"command\")\n", + "\n", + " # 'migrate' subcommand with mutually exclusive direction\n", + " migrate = sub.add_parser(\"migrate\", help=\"Run database migrations\")\n", + " direction = migrate.add_mutually_exclusive_group(required=True)\n", + " direction.add_argument(\"--up\", action=\"store_true\", help=\"Apply migrations\")\n", + " direction.add_argument(\"--down\", action=\"store_true\", help=\"Rollback migrations\")\n", + " migrate.add_argument(\"--steps\", type=int, default=1, help=\"Number of steps\")\n", + "\n", + " # 'dump' subcommand with argument groups\n", + " dump = sub.add_parser(\"dump\", help=\"Export database\")\n", + "\n", + " source_group = dump.add_argument_group(\"source\", \"Database source options\")\n", + " source_group.add_argument(\"--database\", required=True, help=\"Database name\")\n", + " source_group.add_argument(\"--tables\", nargs=\"*\", help=\"Specific tables to dump\")\n", + "\n", + " output_group = dump.add_argument_group(\"output\", \"Output options\")\n", + " fmt = output_group.add_mutually_exclusive_group()\n", + " fmt.add_argument(\"--sql\", action=\"store_true\", help=\"SQL format\")\n", + " fmt.add_argument(\"--csv\", action=\"store_true\", help=\"CSV format\")\n", + " output_group.add_argument(\"-o\", \"--output\", default=\"stdout\", help=\"Output file\")\n", + "\n", + " return parser\n", + "\n", + "\n", + "db_cli: argparse.ArgumentParser = build_db_cli()\n", + "db_cli.print_help()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a1000020", + "metadata": {}, + "outputs": [], + "source": [ + "# Using the migrate subcommand\n", + "args = db_cli.parse_args([\"migrate\", \"--up\", \"--steps\", \"3\"])\n", + "print(f\"Command: {args.command}\")\n", + "print(f\"Direction: {'up' if args.up else 'down'}\")\n", + "print(f\"Steps: {args.steps}\")\n", + "\n", + "print()\n", + "\n", + "# Using the dump subcommand with groups\n", + "args = db_cli.parse_args([\n", + " \"-v\", \"dump\",\n", + " \"--database\", \"mydb\",\n", + " \"--tables\", \"users\", \"orders\",\n", + " \"--csv\",\n", + " \"-o\", \"export.csv\",\n", + "])\n", + "print(f\"Command: {args.command}\")\n", + "print(f\"Verbose: {args.verbose}\")\n", + "print(f\"Database: {args.database}\")\n", + "print(f\"Tables: {args.tables}\")\n", + "print(f\"CSV format: {args.csv}\")\n", + "print(f\"Output: {args.output}\")" + ] + }, + { + "cell_type": "markdown", + "id": "a1000021", + "metadata": {}, + "source": [ + "## Section 7: Parent Parsers for Shared Arguments\n", + "\n", + "Parent parsers let you define common arguments once and share them across subcommands without repeating definitions." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a1000022", + "metadata": {}, + "outputs": [], + "source": [ + "# Shared parent parser with common arguments\n", + "parent = argparse.ArgumentParser(add_help=False) # add_help=False avoids duplicate --help\n", + "parent.add_argument(\"--debug\", action=\"store_true\", help=\"Enable debug mode\")\n", + "parent.add_argument(\"--config\", default=\"config.toml\", help=\"Config file path\")\n", + "\n", + "# Main parser uses the parent\n", + "main_parser = argparse.ArgumentParser(prog=\"app\")\n", + "sub = main_parser.add_subparsers(dest=\"command\")\n", + "\n", + "# Both subcommands inherit --debug and --config from the parent\n", + "serve_cmd = sub.add_parser(\"serve\", parents=[parent], help=\"Start the server\")\n", + "serve_cmd.add_argument(\"--port\", type=int, default=8080)\n", + "\n", + "test_cmd = sub.add_parser(\"test\", parents=[parent], help=\"Run tests\")\n", + "test_cmd.add_argument(\"--coverage\", action=\"store_true\")\n", + "\n", + "# Both subcommands have --debug and --config\n", + "args = main_parser.parse_args([\"serve\", \"--debug\", \"--port\", \"3000\"])\n", + "print(f\"serve: debug={args.debug}, config={args.config}, port={args.port}\")\n", + "\n", + "args = main_parser.parse_args([\"test\", \"--debug\", \"--coverage\"])\n", + "print(f\"test: debug={args.debug}, config={args.config}, coverage={args.coverage}\")" + ] + }, + { + "cell_type": "markdown", + "id": "a1000023", + "metadata": {}, + "source": [ + "## Summary\n", + "\n", + "### Subparsers\n", + "- **`add_subparsers(dest=...)`** creates a subcommand dispatcher\n", + "- **`add_parser(name)`** registers a new subcommand with its own arguments\n", + "- **`set_defaults(func=...)`** enables clean function dispatch per subcommand\n", + "\n", + "### Mutually Exclusive Groups\n", + "- **`add_mutually_exclusive_group()`** prevents conflicting flags from being used together\n", + "- Use `required=True` to force exactly one choice from the group\n", + "- Argparse raises `SystemExit` if conflicting options are provided\n", + "\n", + "### Argument Groups\n", + "- **`add_argument_group(title, description)`** organizes help output into sections\n", + "- Groups are purely organizational and do not restrict argument usage\n", + "- Combine groups within subcommands for well-structured CLI tools\n", + "\n", + "### Parent Parsers\n", + "- **`parents=[parent_parser]`** shares common arguments across subcommands\n", + "- Use `add_help=False` on parent parsers to avoid duplicate `--help` flags" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "version": "3.13.0" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} \ No newline at end of file diff --git a/src/chapter_28/03_environment_and_config.ipynb b/src/chapter_28/03_environment_and_config.ipynb new file mode 100644 index 0000000..f6ecc49 --- /dev/null +++ b/src/chapter_28/03_environment_and_config.ipynb @@ -0,0 +1,683 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "b1000001", + "metadata": {}, + "source": [ + "# Chapter 28: Environment and Configuration\n", + "\n", + "This notebook covers how CLI tools interact with the operating system environment: reading environment variables with `os.environ`, inspecting command-line arguments via `sys.argv`, integrating configuration files, and using exit codes for process communication.\n", + "\n", + "## Key Concepts\n", + "- **`os.environ`**: A mapping of environment variables\n", + "- **`sys.argv`**: The raw list of command-line arguments\n", + "- **Config file integration**: Layering file-based config with CLI arguments\n", + "- **Exit codes**: Communicating success or failure to the calling process" + ] + }, + { + "cell_type": "markdown", + "id": "b1000002", + "metadata": {}, + "source": [ + "## Section 1: Environment Variables with os.environ\n", + "\n", + "`os.environ` is a dictionary-like object that provides access to environment variables. Many CLI tools use environment variables for configuration (e.g., `DATABASE_URL`, `API_KEY`)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b1000003", + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "\n", + "# os.environ behaves like a dict[str, str]\n", + "print(f\"Type: {type(os.environ).__name__}\")\n", + "print(f\"HOME: {os.environ.get('HOME', 'not set')}\")\n", + "print(f\"PATH (first 80 chars): {os.environ.get('PATH', '')[:80]}...\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b1000004", + "metadata": {}, + "outputs": [], + "source": [ + "# Setting and reading environment variables\n", + "os.environ[\"MY_APP_DEBUG\"] = \"true\"\n", + "os.environ[\"MY_APP_PORT\"] = \"8080\"\n", + "\n", + "debug_flag: str = os.environ[\"MY_APP_DEBUG\"]\n", + "port_str: str = os.environ[\"MY_APP_PORT\"]\n", + "\n", + "print(f\"MY_APP_DEBUG: {debug_flag!r} (type: {type(debug_flag).__name__})\")\n", + "print(f\"MY_APP_PORT: {port_str!r} (type: {type(port_str).__name__})\")\n", + "print()\n", + "print(\"Note: Environment variables are always strings.\")\n", + "print(f\"Port as int: {int(port_str)}\")\n", + "\n", + "# Clean up\n", + "del os.environ[\"MY_APP_DEBUG\"]\n", + "del os.environ[\"MY_APP_PORT\"]" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b1000005", + "metadata": {}, + "outputs": [], + "source": [ + "# Safe access with .get() and defaults\n", + "# This is the recommended pattern for optional environment variables\n", + "database_url: str = os.environ.get(\"DATABASE_URL\", \"sqlite:///default.db\")\n", + "log_level: str = os.environ.get(\"LOG_LEVEL\", \"info\")\n", + "max_workers: int = int(os.environ.get(\"MAX_WORKERS\", \"4\"))\n", + "\n", + "print(f\"Database URL: {database_url}\")\n", + "print(f\"Log level: {log_level}\")\n", + "print(f\"Max workers: {max_workers} (type: {type(max_workers).__name__})\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b1000006", + "metadata": {}, + "outputs": [], + "source": [ + "# Checking for required environment variables\n", + "def require_env(name: str) -> str:\n", + " \"\"\"Get a required environment variable or raise an error.\"\"\"\n", + " value: str | None = os.environ.get(name)\n", + " if value is None:\n", + " raise EnvironmentError(f\"Required environment variable '{name}' is not set\")\n", + " return value\n", + "\n", + "\n", + "# Set a variable, then require it\n", + "os.environ[\"API_KEY\"] = \"sk-test-12345\"\n", + "key: str = require_env(\"API_KEY\")\n", + "print(f\"API_KEY: {key}\")\n", + "\n", + "# Missing variable raises an error\n", + "try:\n", + " require_env(\"MISSING_SECRET\")\n", + "except EnvironmentError as e:\n", + " print(f\"Error: {e}\")\n", + "\n", + "# Clean up\n", + "del os.environ[\"API_KEY\"]" + ] + }, + { + "cell_type": "markdown", + "id": "b1000007", + "metadata": {}, + "source": [ + "## Section 2: sys.argv — Raw Command-Line Arguments\n", + "\n", + "`sys.argv` is a plain list of strings representing the command-line arguments. `sys.argv[0]` is the script name, and `sys.argv[1:]` are the user-provided arguments. While `argparse` is preferred for real CLI tools, `sys.argv` is useful for understanding how argument parsing works underneath." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b1000008", + "metadata": {}, + "outputs": [], + "source": [ + "import sys\n", + "\n", + "# sys.argv is always a list with at least one element\n", + "print(f\"Type: {type(sys.argv).__name__}\")\n", + "print(f\"Length: {len(sys.argv)}\")\n", + "print(f\"Script name (argv[0]): {sys.argv[0]}\")\n", + "print(f\"Is list: {isinstance(sys.argv, list)}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b1000009", + "metadata": {}, + "outputs": [], + "source": [ + "# Simulating manual argument parsing (for educational purposes)\n", + "# In real code, always use argparse instead\n", + "\n", + "def manual_parse(argv: list[str]) -> dict[str, str | bool]:\n", + " \"\"\"Parse arguments manually from a list of strings.\"\"\"\n", + " result: dict[str, str | bool] = {}\n", + " i: int = 0\n", + " while i < len(argv):\n", + " arg: str = argv[i]\n", + " if arg.startswith(\"--\"):\n", + " key: str = arg[2:].replace(\"-\", \"_\")\n", + " # Check if next arg is a value or another flag\n", + " if i + 1 < len(argv) and not argv[i + 1].startswith(\"--\"):\n", + " result[key] = argv[i + 1]\n", + " i += 2\n", + " else:\n", + " result[key] = True\n", + " i += 1\n", + " else:\n", + " result.setdefault(\"positional\", [])\n", + " result[\"positional\"].append(arg) # type: ignore[union-attr]\n", + " i += 1\n", + " return result\n", + "\n", + "\n", + "# Simulate parsing\n", + "fake_argv: list[str] = [\"app.py\", \"input.txt\", \"--verbose\", \"--output\", \"result.csv\"]\n", + "parsed: dict[str, str | bool] = manual_parse(fake_argv[1:]) # Skip script name\n", + "\n", + "print(f\"Input: {fake_argv}\")\n", + "print(f\"Parsed: {parsed}\")" + ] + }, + { + "cell_type": "markdown", + "id": "b1000010", + "metadata": {}, + "source": [ + "## Section 3: Combining Environment Variables with argparse\n", + "\n", + "A common pattern is to check environment variables as fallback defaults for CLI arguments. This allows configuration via both the command line and environment." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b1000011", + "metadata": {}, + "outputs": [], + "source": [ + "import argparse\n", + "\n", + "\n", + "def build_env_aware_parser() -> argparse.ArgumentParser:\n", + " \"\"\"Build a parser that uses environment variables as defaults.\"\"\"\n", + " parser = argparse.ArgumentParser(prog=\"webapp\")\n", + "\n", + " # CLI args override env vars, which override hardcoded defaults\n", + " parser.add_argument(\n", + " \"--host\",\n", + " default=os.environ.get(\"APP_HOST\", \"127.0.0.1\"),\n", + " help=\"Server host (env: APP_HOST, default: %(default)s)\",\n", + " )\n", + " parser.add_argument(\n", + " \"--port\",\n", + " type=int,\n", + " default=int(os.environ.get(\"APP_PORT\", \"8000\")),\n", + " help=\"Server port (env: APP_PORT, default: %(default)s)\",\n", + " )\n", + " parser.add_argument(\n", + " \"--debug\",\n", + " action=\"store_true\",\n", + " default=os.environ.get(\"APP_DEBUG\", \"\").lower() in (\"1\", \"true\", \"yes\"),\n", + " help=\"Enable debug mode (env: APP_DEBUG)\",\n", + " )\n", + "\n", + " return parser\n", + "\n", + "\n", + "# Without env vars: uses hardcoded defaults\n", + "parser = build_env_aware_parser()\n", + "args = parser.parse_args([])\n", + "print(f\"Without env vars: host={args.host}, port={args.port}, debug={args.debug}\")\n", + "\n", + "# With env vars: uses environment as defaults\n", + "os.environ[\"APP_HOST\"] = \"0.0.0.0\"\n", + "os.environ[\"APP_PORT\"] = \"3000\"\n", + "os.environ[\"APP_DEBUG\"] = \"true\"\n", + "\n", + "parser = build_env_aware_parser()\n", + "args = parser.parse_args([])\n", + "print(f\"With env vars: host={args.host}, port={args.port}, debug={args.debug}\")\n", + "\n", + "# CLI args still override environment\n", + "args = parser.parse_args([\"--host\", \"localhost\", \"--port\", \"9000\"])\n", + "print(f\"CLI override: host={args.host}, port={args.port}, debug={args.debug}\")\n", + "\n", + "# Clean up\n", + "del os.environ[\"APP_HOST\"]\n", + "del os.environ[\"APP_PORT\"]\n", + "del os.environ[\"APP_DEBUG\"]" + ] + }, + { + "cell_type": "markdown", + "id": "b1000012", + "metadata": {}, + "source": [ + "## Section 4: Config File Integration\n", + "\n", + "Many CLI tools support configuration files (TOML, JSON, INI). A robust tool layers configuration with this priority:\n", + "1. Command-line arguments (highest priority)\n", + "2. Environment variables\n", + "3. Config file values\n", + "4. Hardcoded defaults (lowest priority)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b1000013", + "metadata": {}, + "outputs": [], + "source": [ + "import json\n", + "import tempfile\n", + "from pathlib import Path\n", + "\n", + "\n", + "def load_config_file(path: Path) -> dict[str, object]:\n", + " \"\"\"Load configuration from a JSON file.\"\"\"\n", + " if not path.exists():\n", + " return {}\n", + " with path.open() as f:\n", + " config: dict[str, object] = json.load(f)\n", + " return config\n", + "\n", + "\n", + "def merge_config(\n", + " defaults: dict[str, object],\n", + " file_config: dict[str, object],\n", + " env_config: dict[str, object],\n", + " cli_args: dict[str, object],\n", + ") -> dict[str, object]:\n", + " \"\"\"Merge configuration sources with proper priority.\"\"\"\n", + " merged: dict[str, object] = {}\n", + " merged.update(defaults)\n", + " merged.update({k: v for k, v in file_config.items() if v is not None})\n", + " merged.update({k: v for k, v in env_config.items() if v is not None})\n", + " merged.update({k: v for k, v in cli_args.items() if v is not None})\n", + " return merged\n", + "\n", + "\n", + "# Create a temporary config file\n", + "config_data: dict[str, object] = {\n", + " \"host\": \"config-host.example.com\",\n", + " \"port\": 5000,\n", + " \"debug\": False,\n", + " \"log_level\": \"warning\",\n", + "}\n", + "\n", + "with tempfile.NamedTemporaryFile(mode=\"w\", suffix=\".json\", delete=False) as f:\n", + " json.dump(config_data, f)\n", + " config_path: Path = Path(f.name)\n", + "\n", + "# Load and display\n", + "file_config: dict[str, object] = load_config_file(config_path)\n", + "print(f\"Config file: {config_path}\")\n", + "print(f\"File config: {json.dumps(file_config, indent=2)}\")\n", + "\n", + "# Clean up temp file\n", + "config_path.unlink()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b1000014", + "metadata": {}, + "outputs": [], + "source": [ + "# Demonstrate the full configuration layering\n", + "defaults: dict[str, object] = {\n", + " \"host\": \"127.0.0.1\",\n", + " \"port\": 8000,\n", + " \"debug\": False,\n", + " \"log_level\": \"info\",\n", + "}\n", + "\n", + "file_config = {\n", + " \"host\": \"config-host.example.com\",\n", + " \"port\": 5000,\n", + "}\n", + "\n", + "env_config: dict[str, object] = {\n", + " \"port\": 3000, # Overrides file config\n", + "}\n", + "\n", + "cli_args_config: dict[str, object] = {\n", + " \"debug\": True, # Overrides everything\n", + "}\n", + "\n", + "final: dict[str, object] = merge_config(defaults, file_config, env_config, cli_args_config)\n", + "\n", + "print(\"Configuration layering:\")\n", + "print(f\" Defaults: {defaults}\")\n", + "print(f\" File config: {file_config}\")\n", + "print(f\" Env config: {env_config}\")\n", + "print(f\" CLI args: {cli_args_config}\")\n", + "print(f\" Final: {final}\")\n", + "print()\n", + "print(f\" host comes from file_config: {final['host']}\")\n", + "print(f\" port comes from env_config: {final['port']}\")\n", + "print(f\" debug comes from cli_args: {final['debug']}\")\n", + "print(f\" log_level comes from defaults: {final['log_level']}\")" + ] + }, + { + "cell_type": "markdown", + "id": "b1000015", + "metadata": {}, + "source": [ + "## Section 5: Exit Codes\n", + "\n", + "Exit codes communicate a program's result to the calling process (shell, CI pipeline, etc.).\n", + "- `0` means success\n", + "- Non-zero means failure (by convention, `1` is a general error, `2` is usage error)\n", + "\n", + "`sys.exit()` raises a `SystemExit` exception, which can be caught for testing." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b1000016", + "metadata": {}, + "outputs": [], + "source": [ + "import sys\n", + "\n", + "# sys.exit raises SystemExit, which can be caught\n", + "try:\n", + " sys.exit(0)\n", + "except SystemExit as e:\n", + " print(f\"Exit code: {e.code} (success)\")\n", + "\n", + "try:\n", + " sys.exit(1)\n", + "except SystemExit as e:\n", + " print(f\"Exit code: {e.code} (general error)\")\n", + "\n", + "try:\n", + " sys.exit(2)\n", + "except SystemExit as e:\n", + " print(f\"Exit code: {e.code} (usage error)\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b1000017", + "metadata": {}, + "outputs": [], + "source": [ + "# sys.exit can also accept a string message (implies exit code 1)\n", + "try:\n", + " sys.exit(\"Fatal error: configuration file not found\")\n", + "except SystemExit as e:\n", + " print(f\"Exit message: {e.code}\")\n", + " print(f\"Type of code: {type(e.code).__name__}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b1000018", + "metadata": {}, + "outputs": [], + "source": [ + "# Common pattern: a main function that returns an exit code\n", + "from enum import IntEnum\n", + "\n", + "\n", + "class ExitCode(IntEnum):\n", + " \"\"\"Standard exit codes for the application.\"\"\"\n", + " SUCCESS = 0\n", + " GENERAL_ERROR = 1\n", + " USAGE_ERROR = 2\n", + " CONFIG_ERROR = 3\n", + " IO_ERROR = 4\n", + "\n", + "\n", + "def main(args: list[str]) -> ExitCode:\n", + " \"\"\"Application entry point that returns an exit code.\"\"\"\n", + " if not args:\n", + " print(\"Error: no arguments provided\", file=sys.stderr)\n", + " return ExitCode.USAGE_ERROR\n", + "\n", + " filename: str = args[0]\n", + " if not filename.endswith(\".txt\"):\n", + " print(f\"Error: unsupported file type: {filename}\", file=sys.stderr)\n", + " return ExitCode.GENERAL_ERROR\n", + "\n", + " print(f\"Processing {filename}...\")\n", + " return ExitCode.SUCCESS\n", + "\n", + "\n", + "# Simulate different invocations\n", + "for test_args in [[], [\"data.csv\"], [\"report.txt\"]]:\n", + " code: ExitCode = main(test_args)\n", + " print(f\" args={test_args} -> exit code {code} ({code.name})\\n\")" + ] + }, + { + "cell_type": "markdown", + "id": "b1000019", + "metadata": {}, + "source": [ + "## Section 6: Putting It All Together\n", + "\n", + "A complete example of a CLI tool that combines argparse, environment variables, config file loading, and proper exit codes." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b1000020", + "metadata": {}, + "outputs": [], + "source": [ + "import argparse\n", + "import json\n", + "import os\n", + "import sys\n", + "import tempfile\n", + "from pathlib import Path\n", + "\n", + "\n", + "def build_full_parser() -> argparse.ArgumentParser:\n", + " \"\"\"Build a parser for a complete CLI tool.\"\"\"\n", + " parser = argparse.ArgumentParser(\n", + " prog=\"appctl\",\n", + " description=\"Application control tool with layered configuration\",\n", + " )\n", + " parser.add_argument(\n", + " \"--config\",\n", + " type=Path,\n", + " default=Path(os.environ.get(\"APPCTL_CONFIG\", \"config.json\")),\n", + " help=\"Config file path (env: APPCTL_CONFIG)\",\n", + " )\n", + " parser.add_argument(\n", + " \"--host\",\n", + " default=os.environ.get(\"APPCTL_HOST\"),\n", + " help=\"Server host (env: APPCTL_HOST)\",\n", + " )\n", + " parser.add_argument(\n", + " \"--port\",\n", + " type=int,\n", + " default=int(os.environ.get(\"APPCTL_PORT\", \"0\")) or None,\n", + " help=\"Server port (env: APPCTL_PORT)\",\n", + " )\n", + " parser.add_argument(\n", + " \"-v\", \"--verbose\",\n", + " action=\"store_true\",\n", + " help=\"Enable verbose output\",\n", + " )\n", + " return parser\n", + "\n", + "\n", + "def run_app(argv: list[str] | None = None) -> int:\n", + " \"\"\"Run the application with full configuration layering.\"\"\"\n", + " parser: argparse.ArgumentParser = build_full_parser()\n", + " args: argparse.Namespace = parser.parse_args(argv)\n", + "\n", + " # Start with defaults\n", + " config: dict[str, object] = {\n", + " \"host\": \"127.0.0.1\",\n", + " \"port\": 8000,\n", + " \"verbose\": False,\n", + " }\n", + "\n", + " # Layer file config\n", + " if args.config.exists():\n", + " with args.config.open() as f:\n", + " file_cfg: dict[str, object] = json.load(f)\n", + " config.update(file_cfg)\n", + " if args.verbose:\n", + " print(f\"Loaded config from {args.config}\")\n", + "\n", + " # Layer CLI args (only if explicitly provided)\n", + " if args.host is not None:\n", + " config[\"host\"] = args.host\n", + " if args.port is not None:\n", + " config[\"port\"] = args.port\n", + " if args.verbose:\n", + " config[\"verbose\"] = True\n", + "\n", + " print(f\"Final config: {json.dumps(config, indent=2, default=str)}\")\n", + " return 0\n", + "\n", + "\n", + "# Create a temp config file for demonstration\n", + "demo_config: dict[str, object] = {\"host\": \"0.0.0.0\", \"port\": 5000}\n", + "with tempfile.NamedTemporaryFile(\n", + " mode=\"w\", suffix=\".json\", delete=False\n", + ") as f:\n", + " json.dump(demo_config, f)\n", + " tmp_config_path: str = f.name\n", + "\n", + "# Run with config file and CLI override\n", + "print(\"--- With config file and CLI port override ---\")\n", + "exit_code: int = run_app([\n", + " \"--config\", tmp_config_path,\n", + " \"--port\", \"9090\",\n", + " \"--verbose\",\n", + "])\n", + "print(f\"Exit code: {exit_code}\")\n", + "\n", + "# Clean up\n", + "Path(tmp_config_path).unlink()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b1000021", + "metadata": {}, + "outputs": [], + "source": [ + "# Run with only defaults (no config file, no env vars, no CLI args)\n", + "print(\"--- With only defaults ---\")\n", + "exit_code = run_app([])\n", + "print(f\"Exit code: {exit_code}\")" + ] + }, + { + "cell_type": "markdown", + "id": "b1000022", + "metadata": {}, + "source": [ + "## Section 7: Best Practices for CLI Configuration\n", + "\n", + "When building production CLI tools, follow these conventions for a good user experience." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b1000023", + "metadata": {}, + "outputs": [], + "source": [ + "# Best practice: use a dataclass to hold validated configuration\n", + "from dataclasses import dataclass\n", + "\n", + "\n", + "@dataclass(frozen=True)\n", + "class AppConfig:\n", + " \"\"\"Validated, immutable application configuration.\"\"\"\n", + " host: str\n", + " port: int\n", + " debug: bool\n", + " log_level: str\n", + "\n", + " def __post_init__(self) -> None:\n", + " \"\"\"Validate configuration values.\"\"\"\n", + " if not 1 <= self.port <= 65535:\n", + " raise ValueError(f\"Port must be 1-65535, got {self.port}\")\n", + " valid_levels: set[str] = {\"debug\", \"info\", \"warning\", \"error\", \"critical\"}\n", + " if self.log_level not in valid_levels:\n", + " raise ValueError(f\"Invalid log level: {self.log_level}\")\n", + "\n", + "\n", + "# Valid configuration\n", + "config = AppConfig(host=\"0.0.0.0\", port=8080, debug=True, log_level=\"debug\")\n", + "print(f\"Valid config: {config}\")\n", + "\n", + "# Invalid port is caught immediately\n", + "try:\n", + " bad_config = AppConfig(host=\"localhost\", port=99999, debug=False, log_level=\"info\")\n", + "except ValueError as e:\n", + " print(f\"Validation error: {e}\")\n", + "\n", + "# Invalid log level\n", + "try:\n", + " bad_config = AppConfig(host=\"localhost\", port=8080, debug=False, log_level=\"verbose\")\n", + "except ValueError as e:\n", + " print(f\"Validation error: {e}\")" + ] + }, + { + "cell_type": "markdown", + "id": "b1000024", + "metadata": {}, + "source": [ + "## Summary\n", + "\n", + "### os.environ\n", + "- **`os.environ[key]`** reads a variable (raises `KeyError` if missing)\n", + "- **`os.environ.get(key, default)`** reads safely with a fallback\n", + "- Environment variables are always strings; cast them explicitly\n", + "\n", + "### sys.argv\n", + "- **`sys.argv`** is a `list[str]` with the script name at index 0\n", + "- Use `argparse` instead of parsing `sys.argv` manually\n", + "\n", + "### Config File Integration\n", + "- Layer configuration with this priority: **CLI args > env vars > config file > defaults**\n", + "- Use `argparse` defaults that read from `os.environ.get(...)` for env var integration\n", + "- Validate final configuration with a dataclass or similar structure\n", + "\n", + "### Exit Codes\n", + "- **`sys.exit(0)`** signals success; non-zero signals failure\n", + "- **`sys.exit()`** raises `SystemExit`, which can be caught in tests\n", + "- Define exit codes as an `IntEnum` for clarity and consistency\n", + "- Return exit codes from `main()` instead of calling `sys.exit()` directly for testability" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "version": "3.13.0" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} \ No newline at end of file diff --git a/src/chapter_28/README.md b/src/chapter_28/README.md new file mode 100644 index 0000000..b6a72f3 --- /dev/null +++ b/src/chapter_28/README.md @@ -0,0 +1,9 @@ +# Chapter 28: Command-Line Interfaces + +Building command-line tools with `argparse` — positional and optional arguments, subcommands, type validation, environment variables, and `sys.argv`. + +## Notebooks + +1. **01_argparse_fundamentals.ipynb** — `ArgumentParser`, positional/optional args, types, choices, defaults +2. **02_subcommands_and_groups.ipynb** — Subparsers, mutually exclusive groups, argument groups +3. **03_environment_and_config.ipynb** — `os.environ`, `sys.argv`, config file integration, exit codes diff --git a/src/chapter_28/__init__.py b/src/chapter_28/__init__.py new file mode 100644 index 0000000..4c080b7 --- /dev/null +++ b/src/chapter_28/__init__.py @@ -0,0 +1 @@ +"""Chapter 28: Command-Line Interfaces.""" diff --git a/src/chapter_29/01_datetime_fundamentals.ipynb b/src/chapter_29/01_datetime_fundamentals.ipynb new file mode 100644 index 0000000..76cc310 --- /dev/null +++ b/src/chapter_29/01_datetime_fundamentals.ipynb @@ -0,0 +1,498 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "a1b2c3d4", + "metadata": {}, + "source": [ + "# Chapter 29: Datetime Fundamentals\n", + "\n", + "This notebook covers Python's core date and time types from the `datetime` module. You will learn how to create, inspect, combine, and perform arithmetic on dates and times using `date`, `time`, `datetime`, and `timedelta`.\n", + "\n", + "## Key Concepts\n", + "- **`date`**: Represents a calendar date (year, month, day)\n", + "- **`time`**: Represents a time of day (hour, minute, second, microsecond)\n", + "- **`datetime`**: Combines date and time into a single object\n", + "- **`timedelta`**: Represents a duration or difference between two dates/times\n", + "- **Comparison operators**: Dates and datetimes support ordering" + ] + }, + { + "cell_type": "markdown", + "id": "b1c2d3e4", + "metadata": {}, + "source": [ + "## Section 1: Creating Date Objects\n", + "\n", + "A `date` object represents a calendar date. It stores the year, month, and day as separate attributes." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c1d2e3f4", + "metadata": {}, + "outputs": [], + "source": [ + "from datetime import date\n", + "\n", + "# Create a date from year, month, day\n", + "d: date = date(2025, 6, 15)\n", + "\n", + "print(f\"Date: {d}\")\n", + "print(f\"Year: {d.year}\")\n", + "print(f\"Month: {d.month}\")\n", + "print(f\"Day: {d.day}\")\n", + "\n", + "# Get today's date\n", + "today: date = date.today()\n", + "print(f\"\\nToday: {today}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "d1e2f3a4", + "metadata": {}, + "outputs": [], + "source": [ + "# Useful date methods\n", + "d: date = date(2025, 6, 15)\n", + "\n", + "# weekday() returns 0=Monday ... 6=Sunday\n", + "print(f\"Weekday (0=Mon): {d.weekday()}\")\n", + "\n", + "# isoweekday() returns 1=Monday ... 7=Sunday\n", + "print(f\"ISO weekday (1=Mon): {d.isoweekday()}\")\n", + "\n", + "# ISO calendar returns (year, week_number, weekday)\n", + "iso_year, iso_week, iso_day = d.isocalendar()\n", + "print(f\"ISO calendar: year={iso_year}, week={iso_week}, day={iso_day}\")\n", + "\n", + "# Create from ordinal (days since Jan 1, year 1)\n", + "ordinal: int = d.toordinal()\n", + "print(f\"\\nOrdinal of {d}: {ordinal}\")\n", + "print(f\"Back to date: {date.fromordinal(ordinal)}\")" + ] + }, + { + "cell_type": "markdown", + "id": "e1f2a3b4", + "metadata": {}, + "source": [ + "## Section 2: Creating Time Objects\n", + "\n", + "A `time` object represents a time of day independent of any date. It stores hours, minutes, seconds, and microseconds." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f1a2b3c4", + "metadata": {}, + "outputs": [], + "source": [ + "from datetime import time\n", + "\n", + "# Create a time object\n", + "t: time = time(14, 30, 45)\n", + "\n", + "print(f\"Time: {t}\")\n", + "print(f\"Hour: {t.hour}\")\n", + "print(f\"Minute: {t.minute}\")\n", + "print(f\"Second: {t.second}\")\n", + "\n", + "# Time with microseconds\n", + "precise: time = time(14, 30, 45, 123456)\n", + "print(f\"\\nPrecise time: {precise}\")\n", + "print(f\"Microsecond: {precise.microsecond}\")\n", + "\n", + "# Minimum and maximum times\n", + "print(f\"\\nMin time: {time.min}\")\n", + "print(f\"Max time: {time.max}\")" + ] + }, + { + "cell_type": "markdown", + "id": "a2b3c4d5", + "metadata": {}, + "source": [ + "## Section 3: The `datetime` Object\n", + "\n", + "A `datetime` combines a `date` and a `time` into a single object. It is the most commonly used type for working with timestamps." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b2c3d4e5", + "metadata": {}, + "outputs": [], + "source": [ + "from datetime import datetime, date, time\n", + "\n", + "# Create a datetime\n", + "dt: datetime = datetime(2025, 6, 15, 14, 30)\n", + "\n", + "print(f\"Datetime: {dt}\")\n", + "print(f\"Year: {dt.year}\")\n", + "print(f\"Month: {dt.month}\")\n", + "print(f\"Day: {dt.day}\")\n", + "print(f\"Hour: {dt.hour}\")\n", + "print(f\"Minute: {dt.minute}\")\n", + "\n", + "# Extract date and time parts\n", + "d: date = dt.date()\n", + "t: time = dt.time()\n", + "print(f\"\\nDate part: {d}\")\n", + "print(f\"Time part: {t}\")\n", + "print(f\"date() == date(2025, 6, 15): {d == date(2025, 6, 15)}\")\n", + "print(f\"time() == time(14, 30): {t == time(14, 30)}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c2d3e4f5", + "metadata": {}, + "outputs": [], + "source": [ + "from datetime import datetime, date, time\n", + "\n", + "# Combine a date and a time into a datetime\n", + "d: date = date(2025, 6, 15)\n", + "t: time = time(14, 30, 0)\n", + "combined: datetime = datetime.combine(d, t)\n", + "\n", + "print(f\"Date: {d}\")\n", + "print(f\"Time: {t}\")\n", + "print(f\"Combined: {combined}\")\n", + "\n", + "# Get the current datetime\n", + "now: datetime = datetime.now()\n", + "print(f\"\\nNow: {now}\")\n", + "\n", + "# Create from a POSIX timestamp (seconds since 1970-01-01 UTC)\n", + "from_ts: datetime = datetime.fromtimestamp(1750000000)\n", + "print(f\"\\nFrom timestamp 1750000000: {from_ts}\")" + ] + }, + { + "cell_type": "markdown", + "id": "d2e3f4a5", + "metadata": {}, + "source": [ + "## Section 4: Replacing Components\n", + "\n", + "Date and datetime objects are immutable. The `replace()` method creates a new object with one or more fields changed." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e2f3a4b5", + "metadata": {}, + "outputs": [], + "source": [ + "from datetime import datetime\n", + "\n", + "dt: datetime = datetime(2025, 6, 15, 14, 30)\n", + "\n", + "# Replace creates a new datetime with specified fields changed\n", + "new_year: datetime = dt.replace(year=2026)\n", + "new_month: datetime = dt.replace(month=12, day=25)\n", + "new_time: datetime = dt.replace(hour=9, minute=0)\n", + "\n", + "print(f\"Original: {dt}\")\n", + "print(f\"New year: {new_year}\")\n", + "print(f\"New month: {new_month}\")\n", + "print(f\"New time: {new_time}\")\n", + "\n", + "# Original is unchanged (immutable)\n", + "print(f\"\\nOriginal still: {dt}\")" + ] + }, + { + "cell_type": "markdown", + "id": "f2a3b4c5", + "metadata": {}, + "source": [ + "## Section 5: Timedelta -- Durations and Differences\n", + "\n", + "A `timedelta` represents a duration. It can express differences between dates or datetimes and supports arithmetic operations." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a3b4c5d6", + "metadata": {}, + "outputs": [], + "source": [ + "from datetime import timedelta\n", + "\n", + "# Create timedeltas\n", + "one_day: timedelta = timedelta(days=1)\n", + "one_week: timedelta = timedelta(weeks=1)\n", + "complex_delta: timedelta = timedelta(days=5, hours=3, minutes=30)\n", + "\n", + "print(f\"One day: {one_day}\")\n", + "print(f\"One week: {one_week}\")\n", + "print(f\"Complex: {complex_delta}\")\n", + "\n", + "# Internally, timedelta stores only days, seconds, and microseconds\n", + "td: timedelta = timedelta(hours=25, minutes=30)\n", + "print(f\"\\ntimedelta(hours=25, minutes=30):\")\n", + "print(f\" days: {td.days}\")\n", + "print(f\" seconds: {td.seconds}\") # 1 hour 30 min = 5400 seconds\n", + "print(f\" microseconds: {td.microseconds}\")\n", + "print(f\" total_seconds: {td.total_seconds()}\")" + ] + }, + { + "cell_type": "markdown", + "id": "b3c4d5e6", + "metadata": {}, + "source": [ + "## Section 6: Date Arithmetic\n", + "\n", + "You can add or subtract `timedelta` objects to/from `date` and `datetime` objects. Subtracting two dates produces a `timedelta`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c3d4e5f6", + "metadata": {}, + "outputs": [], + "source": [ + "from datetime import date, timedelta\n", + "\n", + "# Adding timedelta to a date\n", + "d1: date = date(2025, 1, 1)\n", + "d2: date = d1 + timedelta(days=30)\n", + "\n", + "print(f\"Start: {d1}\")\n", + "print(f\"+ 30 days: {d2}\")\n", + "print(f\"d2 == Jan 31: {d2 == date(2025, 1, 31)}\")\n", + "\n", + "# Subtracting dates gives a timedelta\n", + "diff: timedelta = date(2025, 3, 1) - date(2025, 1, 1)\n", + "print(f\"\\nMar 1 - Jan 1 = {diff.days} days\")\n", + "\n", + "# Subtracting a timedelta\n", + "yesterday: date = date.today() - timedelta(days=1)\n", + "print(f\"\\nYesterday: {yesterday}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "d3e4f5a6", + "metadata": {}, + "outputs": [], + "source": [ + "from datetime import datetime, timedelta\n", + "\n", + "# Datetime arithmetic works the same way\n", + "meeting: datetime = datetime(2025, 6, 15, 14, 30)\n", + "reminder: datetime = meeting - timedelta(hours=1)\n", + "follow_up: datetime = meeting + timedelta(days=7)\n", + "\n", + "print(f\"Meeting: {meeting}\")\n", + "print(f\"Reminder: {reminder}\")\n", + "print(f\"Follow-up: {follow_up}\")\n", + "\n", + "# Timedelta arithmetic\n", + "half_day: timedelta = timedelta(hours=12)\n", + "full_day: timedelta = half_day * 2\n", + "quarter_day: timedelta = half_day / 2\n", + "\n", + "print(f\"\\nHalf day: {half_day}\")\n", + "print(f\"Full day: {full_day}\")\n", + "print(f\"Quarter day: {quarter_day}\")" + ] + }, + { + "cell_type": "markdown", + "id": "e3f4a5b6", + "metadata": {}, + "source": [ + "## Section 7: Comparing Dates and Datetimes\n", + "\n", + "Date and datetime objects support all comparison operators. This is essential for sorting, filtering, and range checks." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f3a4b5c6", + "metadata": {}, + "outputs": [], + "source": [ + "from datetime import date, datetime\n", + "\n", + "# Date comparison\n", + "jan: date = date(2025, 1, 1)\n", + "jun: date = date(2025, 6, 1)\n", + "jul: date = date(2025, 7, 1)\n", + "dec: date = date(2025, 12, 31)\n", + "\n", + "print(f\"Jun 1 < Jul 1: {jun < jul}\")\n", + "print(f\"Dec 31 > Jan 1: {dec > jan}\")\n", + "print(f\"Jun 1 == Jun 1: {jun == date(2025, 6, 1)}\")\n", + "\n", + "# Sorting dates\n", + "dates: list[date] = [dec, jan, jul, jun]\n", + "sorted_dates: list[date] = sorted(dates)\n", + "print(f\"\\nSorted: {sorted_dates}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a4b5c6d7", + "metadata": {}, + "outputs": [], + "source": [ + "from datetime import date, timedelta\n", + "\n", + "# Practical example: checking if a date is within a range\n", + "def is_within_range(\n", + " target: date,\n", + " start: date,\n", + " end: date,\n", + ") -> bool:\n", + " \"\"\"Check if target date falls within [start, end] inclusive.\"\"\"\n", + " return start <= target <= end\n", + "\n", + "today: date = date(2025, 6, 15)\n", + "week_start: date = today - timedelta(days=today.weekday()) # Monday\n", + "week_end: date = week_start + timedelta(days=6) # Sunday\n", + "\n", + "print(f\"Week: {week_start} to {week_end}\")\n", + "print(f\"{today} in range: {is_within_range(today, week_start, week_end)}\")\n", + "print(f\"{date(2025, 6, 1)} in range: {is_within_range(date(2025, 6, 1), week_start, week_end)}\")" + ] + }, + { + "cell_type": "markdown", + "id": "b4c5d6e7", + "metadata": {}, + "source": [ + "## Section 8: Practical Patterns\n", + "\n", + "Common patterns you will encounter when working with dates and times in real applications." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c4d5e6f7", + "metadata": {}, + "outputs": [], + "source": [ + "from datetime import date, timedelta\n", + "\n", + "# Calculate age in years\n", + "def calculate_age(birth_date: date, reference_date: date) -> int:\n", + " \"\"\"Calculate age in complete years.\"\"\"\n", + " age: int = reference_date.year - birth_date.year\n", + " # Subtract 1 if birthday hasn't occurred yet this year\n", + " if (reference_date.month, reference_date.day) < (birth_date.month, birth_date.day):\n", + " age -= 1\n", + " return age\n", + "\n", + "birthday: date = date(1990, 8, 20)\n", + "today: date = date(2025, 6, 15)\n", + "\n", + "print(f\"Born: {birthday}\")\n", + "print(f\"Today: {today}\")\n", + "print(f\"Age: {calculate_age(birthday, today)} years\")\n", + "\n", + "# Days until next birthday\n", + "next_birthday: date = birthday.replace(year=today.year)\n", + "if next_birthday < today:\n", + " next_birthday = next_birthday.replace(year=today.year + 1)\n", + "days_until: int = (next_birthday - today).days\n", + "print(f\"\\nNext birthday: {next_birthday}\")\n", + "print(f\"Days until: {days_until}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "d4e5f6a7", + "metadata": {}, + "outputs": [], + "source": [ + "from datetime import date, timedelta\n", + "\n", + "# Generate a sequence of dates\n", + "def date_range(\n", + " start: date,\n", + " end: date,\n", + " step: timedelta = timedelta(days=1),\n", + ") -> list[date]:\n", + " \"\"\"Generate dates from start to end (exclusive) with given step.\"\"\"\n", + " dates: list[date] = []\n", + " current: date = start\n", + " while current < end:\n", + " dates.append(current)\n", + " current += step\n", + " return dates\n", + "\n", + "# Daily dates for a week\n", + "week: list[date] = date_range(date(2025, 6, 9), date(2025, 6, 16))\n", + "for d in week:\n", + " day_name: str = d.strftime(\"%A\")\n", + " print(f\" {d} ({day_name})\")\n", + "\n", + "# Every other day\n", + "print(\"\\nEvery other day:\")\n", + "for d in date_range(date(2025, 6, 1), date(2025, 6, 10), timedelta(days=2)):\n", + " print(f\" {d}\")" + ] + }, + { + "cell_type": "markdown", + "id": "e4f5a6b7", + "metadata": {}, + "source": [ + "## Summary\n", + "\n", + "### Core Types\n", + "- **`date(year, month, day)`**: Calendar date with `.year`, `.month`, `.day` attributes\n", + "- **`time(hour, minute, second)`**: Time of day with `.hour`, `.minute`, `.second` attributes\n", + "- **`datetime(year, month, day, hour, minute)`**: Combined date and time\n", + "- **`timedelta(days, hours, minutes, ...)`**: Duration between two points in time\n", + "\n", + "### Key Operations\n", + "- **Arithmetic**: `date + timedelta`, `datetime - datetime` produces `timedelta`\n", + "- **Comparison**: `<`, `>`, `==`, `<=`, `>=` all work on dates and datetimes\n", + "- **Replace**: `dt.replace(year=2026)` creates a new object with one field changed\n", + "- **Combine**: `datetime.combine(date, time)` merges separate date and time\n", + "- **Extract**: `dt.date()` and `dt.time()` pull out components\n", + "\n", + "### Important Notes\n", + "- `timedelta` internally stores only `days`, `seconds`, and `microseconds`\n", + "- `total_seconds()` returns the full duration as a float\n", + "- All date/time objects are **immutable** -- operations return new objects\n", + "- Use `date.today()` and `datetime.now()` for current date/time" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "version": "3.13.0" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} \ No newline at end of file diff --git a/src/chapter_29/02_timezones_and_zoneinfo.ipynb b/src/chapter_29/02_timezones_and_zoneinfo.ipynb new file mode 100644 index 0000000..aa9a4be --- /dev/null +++ b/src/chapter_29/02_timezones_and_zoneinfo.ipynb @@ -0,0 +1,444 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "a1000001", + "metadata": {}, + "source": [ + "# Chapter 29: Timezones and ZoneInfo\n", + "\n", + "This notebook covers timezone handling in Python using `timezone`, `ZoneInfo`, and the distinction between naive and aware datetimes. Correct timezone handling is critical for applications that operate across regions.\n", + "\n", + "## Key Concepts\n", + "- **Naive datetimes**: No timezone information attached\n", + "- **Aware datetimes**: Include timezone information via `tzinfo`\n", + "- **`timezone.utc`**: The built-in UTC timezone constant\n", + "- **`ZoneInfo`**: Access to the IANA timezone database (Python 3.9+)\n", + "- **`astimezone()`**: Convert between timezones" + ] + }, + { + "cell_type": "markdown", + "id": "a1000002", + "metadata": {}, + "source": [ + "## Section 1: Naive vs Aware Datetimes\n", + "\n", + "A **naive** datetime has no timezone info (`tzinfo is None`). An **aware** datetime carries explicit timezone information. Mixing them causes errors, so it is important to be consistent." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a1000003", + "metadata": {}, + "outputs": [], + "source": [ + "from datetime import datetime, timezone\n", + "\n", + "# Naive datetime -- no timezone\n", + "naive: datetime = datetime(2025, 6, 15, 12, 0)\n", + "print(f\"Naive datetime: {naive}\")\n", + "print(f\"tzinfo: {naive.tzinfo}\")\n", + "print(f\"Is naive: {naive.tzinfo is None}\")\n", + "\n", + "# Aware datetime -- with timezone\n", + "aware: datetime = datetime(2025, 6, 15, 12, 0, tzinfo=timezone.utc)\n", + "print(f\"\\nAware datetime: {aware}\")\n", + "print(f\"tzinfo: {aware.tzinfo}\")\n", + "print(f\"Is aware: {aware.tzinfo is not None}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a1000004", + "metadata": {}, + "outputs": [], + "source": [ + "from datetime import datetime, timezone\n", + "\n", + "# Cannot compare naive and aware datetimes\n", + "naive: datetime = datetime(2025, 6, 15, 12, 0)\n", + "aware: datetime = datetime(2025, 6, 15, 12, 0, tzinfo=timezone.utc)\n", + "\n", + "try:\n", + " result = naive < aware\n", + "except TypeError as e:\n", + " print(f\"Comparison error: {e}\")\n", + "\n", + "# Cannot subtract naive from aware\n", + "try:\n", + " diff = aware - naive\n", + "except TypeError as e:\n", + " print(f\"Subtraction error: {e}\")\n", + "\n", + "print(\"\\nLesson: never mix naive and aware datetimes.\")" + ] + }, + { + "cell_type": "markdown", + "id": "a1000005", + "metadata": {}, + "source": [ + "## Section 2: Working with UTC\n", + "\n", + "UTC (Coordinated Universal Time) is the standard reference timezone. Always store timestamps in UTC and convert to local time only for display." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a1000006", + "metadata": {}, + "outputs": [], + "source": [ + "from datetime import datetime, timezone\n", + "\n", + "# Create a UTC datetime\n", + "utc_now: datetime = datetime.now(timezone.utc)\n", + "print(f\"UTC now: {utc_now}\")\n", + "print(f\"tzname: {utc_now.tzname()}\")\n", + "\n", + "# Construct a specific UTC datetime\n", + "dt: datetime = datetime(2025, 6, 15, 12, 0, tzinfo=timezone.utc)\n", + "print(f\"\\nSpecific UTC: {dt}\")\n", + "print(f\"tzinfo == timezone.utc: {dt.tzinfo == timezone.utc}\")\n", + "print(f\"tzname: {dt.tzname()}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a1000007", + "metadata": {}, + "outputs": [], + "source": [ + "from datetime import datetime, timezone, timedelta\n", + "\n", + "# Fixed-offset timezones using timezone(timedelta(...))\n", + "utc_plus_5: timezone = timezone(timedelta(hours=5))\n", + "utc_minus_8: timezone = timezone(timedelta(hours=-8))\n", + "\n", + "dt_plus5: datetime = datetime(2025, 6, 15, 17, 0, tzinfo=utc_plus_5)\n", + "dt_minus8: datetime = datetime(2025, 6, 15, 4, 0, tzinfo=utc_minus_8)\n", + "\n", + "print(f\"UTC+5: {dt_plus5}\")\n", + "print(f\"UTC-8: {dt_minus8}\")\n", + "\n", + "# Both represent the same instant in time (12:00 UTC)\n", + "print(f\"\\nSame instant? {dt_plus5 == dt_minus8}\")\n", + "print(f\"Plus5 in UTC: {dt_plus5.astimezone(timezone.utc)}\")\n", + "print(f\"Minus8 in UTC: {dt_minus8.astimezone(timezone.utc)}\")" + ] + }, + { + "cell_type": "markdown", + "id": "a1000008", + "metadata": {}, + "source": [ + "## Section 3: ZoneInfo -- IANA Timezones\n", + "\n", + "The `zoneinfo` module (Python 3.9+) provides access to the IANA timezone database. This gives you named timezones like `\"America/New_York\"` that automatically handle daylight saving time (DST)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a1000009", + "metadata": {}, + "outputs": [], + "source": [ + "from datetime import datetime\n", + "from zoneinfo import ZoneInfo\n", + "\n", + "# Create a datetime in a specific timezone\n", + "eastern: ZoneInfo = ZoneInfo(\"America/New_York\")\n", + "dt: datetime = datetime(2025, 6, 15, 12, 0, tzinfo=eastern)\n", + "\n", + "print(f\"New York: {dt}\")\n", + "print(f\"tzinfo: {dt.tzinfo}\")\n", + "print(f\"tzname: {dt.tzname()}\")\n", + "print(f\"UTC offset: {dt.utcoffset()}\")\n", + "\n", + "# Some common timezones\n", + "zones: list[str] = [\n", + " \"America/New_York\",\n", + " \"America/Chicago\",\n", + " \"America/Los_Angeles\",\n", + " \"Europe/London\",\n", + " \"Europe/Berlin\",\n", + " \"Asia/Tokyo\",\n", + "]\n", + "print(\"\\nCommon timezones:\")\n", + "for zone_name in zones:\n", + " tz: ZoneInfo = ZoneInfo(zone_name)\n", + " local_dt: datetime = datetime(2025, 6, 15, 12, 0, tzinfo=tz)\n", + " print(f\" {zone_name:25s} UTC offset: {local_dt.utcoffset()}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a1000010", + "metadata": {}, + "outputs": [], + "source": [ + "from datetime import datetime\n", + "from zoneinfo import ZoneInfo, available_timezones\n", + "\n", + "# List available timezones (there are hundreds)\n", + "all_zones: set[str] = available_timezones()\n", + "print(f\"Total available timezones: {len(all_zones)}\")\n", + "\n", + "# Show a sample of US timezones\n", + "us_zones: list[str] = sorted(tz for tz in all_zones if tz.startswith(\"US/\"))\n", + "print(f\"\\nUS timezones:\")\n", + "for tz_name in us_zones:\n", + " print(f\" {tz_name}\")" + ] + }, + { + "cell_type": "markdown", + "id": "a1000011", + "metadata": {}, + "source": [ + "## Section 4: Timezone Conversion with `astimezone()`\n", + "\n", + "The `astimezone()` method converts an aware datetime to a different timezone. The underlying instant in time stays the same -- only the representation changes." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a1000012", + "metadata": {}, + "outputs": [], + "source": [ + "from datetime import datetime, timezone\n", + "from zoneinfo import ZoneInfo\n", + "\n", + "# Start with a UTC time\n", + "utc_dt: datetime = datetime(2025, 6, 15, 12, 0, tzinfo=timezone.utc)\n", + "print(f\"UTC: {utc_dt}\")\n", + "\n", + "# Convert to Eastern Time (UTC-4 in June due to EDT)\n", + "eastern: datetime = utc_dt.astimezone(ZoneInfo(\"America/New_York\"))\n", + "print(f\"New York: {eastern}\")\n", + "print(f\"Hour: {eastern.hour}\") # 8 (12 - 4)\n", + "print(f\"Same date: {eastern.date() == utc_dt.date()}\")\n", + "\n", + "# Convert to other timezones\n", + "tokyo: datetime = utc_dt.astimezone(ZoneInfo(\"Asia/Tokyo\"))\n", + "london: datetime = utc_dt.astimezone(ZoneInfo(\"Europe/London\"))\n", + "print(f\"\\nTokyo: {tokyo}\")\n", + "print(f\"London: {london}\")\n", + "\n", + "# All represent the same instant\n", + "print(f\"\\nAll equal? {utc_dt == eastern == tokyo == london}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a1000013", + "metadata": {}, + "outputs": [], + "source": [ + "from datetime import datetime, timezone\n", + "from zoneinfo import ZoneInfo\n", + "\n", + "# Practical: display a meeting time across multiple timezones\n", + "def show_meeting_times(\n", + " meeting_utc: datetime,\n", + " timezones: list[str],\n", + ") -> None:\n", + " \"\"\"Display a UTC meeting time in multiple timezones.\"\"\"\n", + " print(f\"Meeting time (UTC): {meeting_utc.strftime('%Y-%m-%d %H:%M %Z')}\")\n", + " print(\"-\" * 50)\n", + " for tz_name in timezones:\n", + " tz: ZoneInfo = ZoneInfo(tz_name)\n", + " local: datetime = meeting_utc.astimezone(tz)\n", + " print(f\" {tz_name:25s} {local.strftime('%H:%M %Z (%Y-%m-%d)')}\")\n", + "\n", + "meeting: datetime = datetime(2025, 6, 15, 15, 0, tzinfo=timezone.utc)\n", + "offices: list[str] = [\n", + " \"America/New_York\",\n", + " \"America/Los_Angeles\",\n", + " \"Europe/London\",\n", + " \"Europe/Berlin\",\n", + " \"Asia/Tokyo\",\n", + " \"Australia/Sydney\",\n", + "]\n", + "show_meeting_times(meeting, offices)" + ] + }, + { + "cell_type": "markdown", + "id": "a1000014", + "metadata": {}, + "source": [ + "## Section 5: Daylight Saving Time\n", + "\n", + "`ZoneInfo` automatically handles DST transitions. The same timezone can have different UTC offsets depending on the date." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a1000015", + "metadata": {}, + "outputs": [], + "source": [ + "from datetime import datetime\n", + "from zoneinfo import ZoneInfo\n", + "\n", + "eastern: ZoneInfo = ZoneInfo(\"America/New_York\")\n", + "\n", + "# Summer (EDT -- Eastern Daylight Time, UTC-4)\n", + "summer: datetime = datetime(2025, 6, 15, 12, 0, tzinfo=eastern)\n", + "print(f\"Summer: {summer}\")\n", + "print(f\" tzname: {summer.tzname()}\")\n", + "print(f\" UTC offset: {summer.utcoffset()}\")\n", + "print(f\" DST offset: {summer.dst()}\")\n", + "\n", + "# Winter (EST -- Eastern Standard Time, UTC-5)\n", + "winter: datetime = datetime(2025, 12, 15, 12, 0, tzinfo=eastern)\n", + "print(f\"\\nWinter: {winter}\")\n", + "print(f\" tzname: {winter.tzname()}\")\n", + "print(f\" UTC offset: {winter.utcoffset()}\")\n", + "print(f\" DST offset: {winter.dst()}\")\n", + "\n", + "# The offset changes depending on date\n", + "print(f\"\\nSummer offset != Winter offset: {summer.utcoffset() != winter.utcoffset()}\")" + ] + }, + { + "cell_type": "markdown", + "id": "a1000016", + "metadata": {}, + "source": [ + "## Section 6: Making Naive Datetimes Aware\n", + "\n", + "You can attach timezone info to a naive datetime using `replace()` or by passing `tzinfo` at construction time. Use `replace()` when you know the naive time is already in a specific timezone." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a1000017", + "metadata": {}, + "outputs": [], + "source": [ + "from datetime import datetime, timezone\n", + "from zoneinfo import ZoneInfo\n", + "\n", + "# A naive datetime (no timezone)\n", + "naive: datetime = datetime(2025, 6, 15, 14, 30)\n", + "print(f\"Naive: {naive} (tzinfo={naive.tzinfo})\")\n", + "\n", + "# Attach UTC timezone using replace\n", + "# This says \"this time IS in UTC\" -- it does not convert\n", + "as_utc: datetime = naive.replace(tzinfo=timezone.utc)\n", + "print(f\"As UTC: {as_utc}\")\n", + "\n", + "# Attach a named timezone\n", + "as_eastern: datetime = naive.replace(tzinfo=ZoneInfo(\"America/New_York\"))\n", + "print(f\"As Eastern: {as_eastern}\")\n", + "\n", + "# These are different instants in time!\n", + "print(f\"\\nAs UTC in UTC: {as_utc.astimezone(timezone.utc)}\")\n", + "print(f\"As Eastern in UTC: {as_eastern.astimezone(timezone.utc)}\")\n", + "print(f\"Same instant? {as_utc == as_eastern}\")" + ] + }, + { + "cell_type": "markdown", + "id": "a1000018", + "metadata": {}, + "source": [ + "## Section 7: Best Practices for Timezone Handling\n", + "\n", + "Follow these guidelines to avoid common timezone bugs." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a1000019", + "metadata": {}, + "outputs": [], + "source": [ + "from datetime import datetime, timezone\n", + "from zoneinfo import ZoneInfo\n", + "\n", + "# BEST PRACTICE 1: Always use aware datetimes\n", + "# Bad: datetime.now() -- returns naive local time\n", + "# Good: datetime.now(tz=...) -- returns aware time\n", + "now_utc: datetime = datetime.now(tz=timezone.utc)\n", + "now_eastern: datetime = datetime.now(tz=ZoneInfo(\"America/New_York\"))\n", + "\n", + "print(f\"UTC now: {now_utc}\")\n", + "print(f\"Eastern now: {now_eastern}\")\n", + "\n", + "# BEST PRACTICE 2: Store in UTC, display in local\n", + "stored_event: datetime = datetime(2025, 6, 15, 18, 0, tzinfo=timezone.utc)\n", + "user_tz: ZoneInfo = ZoneInfo(\"Asia/Tokyo\")\n", + "displayed: datetime = stored_event.astimezone(user_tz)\n", + "print(f\"\\nStored (UTC): {stored_event}\")\n", + "print(f\"Display (Tokyo): {displayed}\")\n", + "\n", + "# BEST PRACTICE 3: Use ZoneInfo, not fixed offsets\n", + "# Fixed offsets do not handle DST correctly\n", + "print(\"\\nZoneInfo handles DST automatically:\")\n", + "tz: ZoneInfo = ZoneInfo(\"America/New_York\")\n", + "jun: datetime = datetime(2025, 6, 15, tzinfo=tz)\n", + "dec: datetime = datetime(2025, 12, 15, tzinfo=tz)\n", + "print(f\" June offset: {jun.utcoffset()}\")\n", + "print(f\" December offset: {dec.utcoffset()}\")" + ] + }, + { + "cell_type": "markdown", + "id": "a1000020", + "metadata": {}, + "source": [ + "## Summary\n", + "\n", + "### Naive vs Aware\n", + "- **Naive**: `datetime.tzinfo is None` -- no timezone information\n", + "- **Aware**: `datetime.tzinfo is not None` -- explicit timezone\n", + "- Never mix naive and aware datetimes in comparisons or arithmetic\n", + "\n", + "### UTC\n", + "- Use `timezone.utc` for the built-in UTC timezone\n", + "- Use `datetime.now(timezone.utc)` instead of `datetime.utcnow()` (deprecated)\n", + "- Store timestamps in UTC; convert to local time for display only\n", + "\n", + "### ZoneInfo (Python 3.9+)\n", + "- `ZoneInfo(\"America/New_York\")` gives IANA timezone with DST support\n", + "- `available_timezones()` lists all known timezone names\n", + "- `astimezone()` converts between timezones, preserving the instant\n", + "\n", + "### Best Practices\n", + "1. Always use aware datetimes\n", + "2. Store in UTC, display in local time\n", + "3. Use `ZoneInfo` instead of fixed UTC offsets for named timezones\n", + "4. Use `replace(tzinfo=...)` to attach timezone info to a naive datetime" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "version": "3.13.0" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} \ No newline at end of file diff --git a/src/chapter_29/03_formatting_and_calendar.ipynb b/src/chapter_29/03_formatting_and_calendar.ipynb new file mode 100644 index 0000000..b5141b4 --- /dev/null +++ b/src/chapter_29/03_formatting_and_calendar.ipynb @@ -0,0 +1,511 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "b1000001", + "metadata": {}, + "source": [ + "# Chapter 29: Formatting and Calendar\n", + "\n", + "This notebook covers formatting and parsing dates with `strftime`/`strptime`, working with ISO 8601 format, and using the `calendar` module for calendar-related computations.\n", + "\n", + "## Key Concepts\n", + "- **`strftime()`**: Format a datetime as a string using format codes\n", + "- **`strptime()`**: Parse a string into a datetime using format codes\n", + "- **ISO 8601**: The international standard for date/time representation\n", + "- **`isoformat()` / `fromisoformat()`**: Built-in ISO 8601 support\n", + "- **`calendar` module**: Leap year checks, month ranges, text calendars" + ] + }, + { + "cell_type": "markdown", + "id": "b1000002", + "metadata": {}, + "source": [ + "## Section 1: Formatting with `strftime()`\n", + "\n", + "The `strftime()` method formats a `date`, `time`, or `datetime` object into a string using **format codes** that start with `%`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b1000003", + "metadata": {}, + "outputs": [], + "source": [ + "from datetime import datetime\n", + "\n", + "dt: datetime = datetime(2025, 6, 15, 14, 30, 45)\n", + "\n", + "# Common format codes\n", + "print(f\"Date (YYYY-MM-DD): {dt.strftime('%Y-%m-%d')}\")\n", + "print(f\"Time (HH:MM): {dt.strftime('%H:%M')}\")\n", + "print(f\"Time (HH:MM:SS): {dt.strftime('%H:%M:%S')}\")\n", + "print(f\"12-hour time: {dt.strftime('%I:%M %p')}\")\n", + "print(f\"Full day name: {dt.strftime('%A')}\")\n", + "print(f\"Abbreviated day: {dt.strftime('%a')}\")\n", + "print(f\"Full month name: {dt.strftime('%B')}\")\n", + "print(f\"Abbreviated month: {dt.strftime('%b')}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b1000004", + "metadata": {}, + "outputs": [], + "source": [ + "from datetime import datetime\n", + "\n", + "dt: datetime = datetime(2025, 6, 15, 14, 30, 45)\n", + "\n", + "# Combining format codes into useful patterns\n", + "formats: dict[str, str] = {\n", + " \"ISO-like\": \"%Y-%m-%d %H:%M:%S\",\n", + " \"US format\": \"%m/%d/%Y\",\n", + " \"European\": \"%d/%m/%Y\",\n", + " \"Long date\": \"%B %d, %Y\",\n", + " \"Day and date\": \"%A, %B %d, %Y\",\n", + " \"Compact\": \"%Y%m%d_%H%M%S\",\n", + " \"Log timestamp\": \"%Y-%m-%d %H:%M:%S\",\n", + "}\n", + "\n", + "for name, fmt in formats.items():\n", + " formatted: str = dt.strftime(fmt)\n", + " print(f\"{name:18s} {fmt:25s} -> {formatted}\")" + ] + }, + { + "cell_type": "markdown", + "id": "b1000005", + "metadata": {}, + "source": [ + "## Section 2: Parsing with `strptime()`\n", + "\n", + "The `strptime()` class method parses a string into a `datetime` object. The format string must match the input exactly." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b1000006", + "metadata": {}, + "outputs": [], + "source": [ + "from datetime import datetime\n", + "\n", + "# Parse ISO-like format\n", + "dt1: datetime = datetime.strptime(\"2025-06-15\", \"%Y-%m-%d\")\n", + "print(f\"Parsed: {dt1}\")\n", + "print(f\"Year={dt1.year}, Month={dt1.month}, Day={dt1.day}\")\n", + "\n", + "# Parse date with time\n", + "dt2: datetime = datetime.strptime(\"2025-06-15 14:30:00\", \"%Y-%m-%d %H:%M:%S\")\n", + "print(f\"\\nWith time: {dt2}\")\n", + "\n", + "# Parse US format\n", + "dt3: datetime = datetime.strptime(\"06/15/2025\", \"%m/%d/%Y\")\n", + "print(f\"US format: {dt3}\")\n", + "\n", + "# Parse long format\n", + "dt4: datetime = datetime.strptime(\"June 15, 2025\", \"%B %d, %Y\")\n", + "print(f\"Long format: {dt4}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b1000007", + "metadata": {}, + "outputs": [], + "source": [ + "from datetime import datetime\n", + "\n", + "# strptime raises ValueError on format mismatch\n", + "try:\n", + " bad: datetime = datetime.strptime(\"15/06/2025\", \"%m/%d/%Y\")\n", + "except ValueError as e:\n", + " print(f\"Parse error: {e}\")\n", + "\n", + "# The format must match exactly, including separators\n", + "try:\n", + " bad2: datetime = datetime.strptime(\"2025-06-15\", \"%Y/%m/%d\")\n", + "except ValueError as e:\n", + " print(f\"Separator mismatch: {e}\")\n", + "\n", + "print(\"\\nAlways ensure the format string matches the input pattern.\")" + ] + }, + { + "cell_type": "markdown", + "id": "b1000008", + "metadata": {}, + "source": [ + "## Section 3: Format Code Reference\n", + "\n", + "Here are the most commonly used `strftime`/`strptime` format codes." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b1000009", + "metadata": {}, + "outputs": [], + "source": [ + "from datetime import datetime\n", + "\n", + "dt: datetime = datetime(2025, 6, 15, 14, 30, 45)\n", + "\n", + "codes: list[tuple[str, str]] = [\n", + " (\"%Y\", \"4-digit year\"),\n", + " (\"%m\", \"Zero-padded month (01-12)\"),\n", + " (\"%d\", \"Zero-padded day (01-31)\"),\n", + " (\"%H\", \"24-hour hour (00-23)\"),\n", + " (\"%I\", \"12-hour hour (01-12)\"),\n", + " (\"%M\", \"Minute (00-59)\"),\n", + " (\"%S\", \"Second (00-59)\"),\n", + " (\"%p\", \"AM/PM\"),\n", + " (\"%A\", \"Full weekday name\"),\n", + " (\"%a\", \"Abbreviated weekday\"),\n", + " (\"%B\", \"Full month name\"),\n", + " (\"%b\", \"Abbreviated month\"),\n", + " (\"%j\", \"Day of year (001-366)\"),\n", + " (\"%U\", \"Week number (Sunday start)\"),\n", + " (\"%W\", \"Week number (Monday start)\"),\n", + " (\"%%\", \"Literal % character\"),\n", + "]\n", + "\n", + "print(f\"{'Code':<6} {'Description':<30} {'Result'}\")\n", + "print(\"-\" * 55)\n", + "for code, desc in codes:\n", + " result: str = dt.strftime(code)\n", + " print(f\"{code:<6} {desc:<30} {result}\")" + ] + }, + { + "cell_type": "markdown", + "id": "b1000010", + "metadata": {}, + "source": [ + "## Section 4: ISO 8601 Format\n", + "\n", + "ISO 8601 is the international standard for date/time strings (`2025-06-15T14:30:00+00:00`). Python has built-in support via `isoformat()` and `fromisoformat()`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b1000011", + "metadata": {}, + "outputs": [], + "source": [ + "from datetime import datetime, date, time, timezone\n", + "\n", + "# isoformat() produces ISO 8601 strings\n", + "dt: datetime = datetime(2025, 6, 15, 14, 30, tzinfo=timezone.utc)\n", + "iso: str = dt.isoformat()\n", + "print(f\"Datetime ISO: {iso}\")\n", + "print(f\"Contains date: {'2025-06-15' in iso}\")\n", + "print(f\"Contains time: {'14:30' in iso}\")\n", + "\n", + "# Date and time also have isoformat\n", + "d: date = date(2025, 6, 15)\n", + "t: time = time(14, 30, 0)\n", + "print(f\"\\nDate ISO: {d.isoformat()}\")\n", + "print(f\"Time ISO: {t.isoformat()}\")\n", + "\n", + "# Custom separator (default is 'T')\n", + "print(f\"\\nWith space separator: {dt.isoformat(sep=' ')}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b1000012", + "metadata": {}, + "outputs": [], + "source": [ + "from datetime import datetime, date, timezone\n", + "\n", + "# fromisoformat() parses ISO 8601 strings\n", + "dt1: datetime = datetime.fromisoformat(\"2025-06-15T14:30:00\")\n", + "print(f\"Parsed: {dt1}\")\n", + "print(f\"Equal: {dt1 == datetime(2025, 6, 15, 14, 30)}\")\n", + "\n", + "# With timezone\n", + "dt2: datetime = datetime.fromisoformat(\"2025-06-15T14:30:00+00:00\")\n", + "print(f\"\\nWith tz: {dt2}\")\n", + "print(f\"tzinfo: {dt2.tzinfo}\")\n", + "\n", + "# Date from ISO string\n", + "d: date = date.fromisoformat(\"2025-06-15\")\n", + "print(f\"\\nDate: {d}\")\n", + "\n", + "# Round-trip: format then parse\n", + "original: datetime = datetime(2025, 6, 15, 14, 30, tzinfo=timezone.utc)\n", + "round_tripped: datetime = datetime.fromisoformat(original.isoformat())\n", + "print(f\"\\nOriginal: {original}\")\n", + "print(f\"Round-trip: {round_tripped}\")\n", + "print(f\"Equal: {original == round_tripped}\")" + ] + }, + { + "cell_type": "markdown", + "id": "b1000013", + "metadata": {}, + "source": [ + "## Section 5: The `calendar` Module\n", + "\n", + "The `calendar` module provides utilities for working with calendars, including leap year checks, month lengths, and text calendar generation." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b1000014", + "metadata": {}, + "outputs": [], + "source": [ + "import calendar\n", + "\n", + "# Check for leap years\n", + "years: list[int] = [1900, 2000, 2024, 2025, 2100]\n", + "for year in years:\n", + " is_leap: bool = calendar.isleap(year)\n", + " print(f\"{year}: leap={is_leap}\")\n", + "\n", + "# Leap year rules:\n", + "# - Divisible by 4: leap year\n", + "# - BUT divisible by 100: NOT a leap year\n", + "# - BUT divisible by 400: leap year\n", + "print(f\"\\n1900: div by 100 but not 400 -> not leap: {not calendar.isleap(1900)}\")\n", + "print(f\"2000: div by 400 -> leap: {calendar.isleap(2000)}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b1000015", + "metadata": {}, + "outputs": [], + "source": [ + "import calendar\n", + "\n", + "# monthrange returns (weekday_of_first_day, number_of_days)\n", + "# weekday: 0=Monday, 6=Sunday\n", + "weekday, days = calendar.monthrange(2025, 2)\n", + "print(f\"February 2025: starts on weekday {weekday}, has {days} days\")\n", + "\n", + "# Leap year February\n", + "_, days_leap = calendar.monthrange(2024, 2)\n", + "print(f\"February 2024 (leap): {days_leap} days\")\n", + "\n", + "# Check all months in a year\n", + "print(f\"\\nDays per month in 2025:\")\n", + "for month in range(1, 13):\n", + " _, num_days = calendar.monthrange(2025, month)\n", + " month_name: str = calendar.month_name[month]\n", + " print(f\" {month_name:10s} {num_days} days\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b1000016", + "metadata": {}, + "outputs": [], + "source": [ + "import calendar\n", + "\n", + "# Generate a text calendar for a month\n", + "print(calendar.month(2025, 6))\n", + "\n", + "# monthcalendar returns weeks as lists of day numbers (0 = outside month)\n", + "print(\"June 2025 as nested lists (0 = not in month):\")\n", + "weeks: list[list[int]] = calendar.monthcalendar(2025, 6)\n", + "for week in weeks:\n", + " print(f\" {week}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b1000017", + "metadata": {}, + "outputs": [], + "source": [ + "import calendar\n", + "\n", + "# Count leap years in a range\n", + "leap_count: int = calendar.leapdays(2000, 2026) # 2000 to 2025 inclusive\n", + "print(f\"Leap years from 2000 to 2025: {leap_count}\")\n", + "\n", + "# List them\n", + "leap_years: list[int] = [y for y in range(2000, 2026) if calendar.isleap(y)]\n", + "print(f\"Leap years: {leap_years}\")\n", + "\n", + "# Day name and month name constants\n", + "print(f\"\\nDay names: {list(calendar.day_name)}\")\n", + "print(f\"Month names: {list(calendar.month_name)[1:]}\")" + ] + }, + { + "cell_type": "markdown", + "id": "b1000018", + "metadata": {}, + "source": [ + "## Section 6: Practical Date Range Generation\n", + "\n", + "Combining `datetime`, `timedelta`, and `calendar` for generating useful date ranges." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b1000019", + "metadata": {}, + "outputs": [], + "source": [ + "import calendar\n", + "from datetime import date, timedelta\n", + "\n", + "\n", + "def weekdays_in_month(year: int, month: int) -> list[date]:\n", + " \"\"\"Return all weekday (Mon-Fri) dates in a given month.\"\"\"\n", + " _, num_days = calendar.monthrange(year, month)\n", + " return [\n", + " date(year, month, day)\n", + " for day in range(1, num_days + 1)\n", + " if date(year, month, day).weekday() < 5 # 0-4 = Mon-Fri\n", + " ]\n", + "\n", + "\n", + "# Get all weekdays in June 2025\n", + "workdays: list[date] = weekdays_in_month(2025, 6)\n", + "print(f\"Weekdays in June 2025: {len(workdays)}\")\n", + "for d in workdays[:5]:\n", + " print(f\" {d} ({d.strftime('%A')})\")\n", + "print(f\" ...\")\n", + "for d in workdays[-3:]:\n", + " print(f\" {d} ({d.strftime('%A')})\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b1000020", + "metadata": {}, + "outputs": [], + "source": [ + "import calendar\n", + "from datetime import date\n", + "\n", + "\n", + "def first_day_of_each_month(year: int) -> list[date]:\n", + " \"\"\"Return the first day of each month in a given year.\"\"\"\n", + " return [date(year, month, 1) for month in range(1, 13)]\n", + "\n", + "\n", + "def last_day_of_each_month(year: int) -> list[date]:\n", + " \"\"\"Return the last day of each month in a given year.\"\"\"\n", + " return [\n", + " date(year, month, calendar.monthrange(year, month)[1])\n", + " for month in range(1, 13)\n", + " ]\n", + "\n", + "\n", + "print(\"First day of each month in 2025:\")\n", + "for d in first_day_of_each_month(2025):\n", + " print(f\" {d} ({d.strftime('%A')})\")\n", + "\n", + "print(f\"\\nLast day of each month in 2025:\")\n", + "for d in last_day_of_each_month(2025):\n", + " print(f\" {d} ({d.strftime('%A')})\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b1000021", + "metadata": {}, + "outputs": [], + "source": [ + "from datetime import datetime, timezone\n", + "\n", + "# Practical: parse various date formats from user input\n", + "def parse_flexible_date(date_string: str) -> datetime:\n", + " \"\"\"Try multiple formats to parse a date string.\"\"\"\n", + " formats: list[str] = [\n", + " \"%Y-%m-%d\",\n", + " \"%Y-%m-%dT%H:%M:%S\",\n", + " \"%m/%d/%Y\",\n", + " \"%d/%m/%Y\",\n", + " \"%B %d, %Y\",\n", + " \"%b %d, %Y\",\n", + " ]\n", + " for fmt in formats:\n", + " try:\n", + " return datetime.strptime(date_string, fmt)\n", + " except ValueError:\n", + " continue\n", + " raise ValueError(f\"Unable to parse date: {date_string!r}\")\n", + "\n", + "\n", + "test_inputs: list[str] = [\n", + " \"2025-06-15\",\n", + " \"06/15/2025\",\n", + " \"June 15, 2025\",\n", + " \"Jun 15, 2025\",\n", + "]\n", + "\n", + "for input_str in test_inputs:\n", + " parsed: datetime = parse_flexible_date(input_str)\n", + " print(f\"{input_str:20s} -> {parsed.strftime('%Y-%m-%d')}\")" + ] + }, + { + "cell_type": "markdown", + "id": "b1000022", + "metadata": {}, + "source": [ + "## Summary\n", + "\n", + "### Formatting (`strftime`)\n", + "- `dt.strftime(\"%Y-%m-%d\")` formats datetime to string\n", + "- Common codes: `%Y` (year), `%m` (month), `%d` (day), `%H` (hour), `%M` (minute), `%S` (second)\n", + "- `%A` (weekday name), `%B` (month name), `%p` (AM/PM)\n", + "\n", + "### Parsing (`strptime`)\n", + "- `datetime.strptime(string, format)` parses string to datetime\n", + "- Format string must match the input exactly\n", + "- Raises `ValueError` on mismatch\n", + "\n", + "### ISO 8601\n", + "- `dt.isoformat()` produces standard ISO 8601 strings\n", + "- `datetime.fromisoformat(string)` parses ISO 8601 back to datetime\n", + "- Round-trip safe: `fromisoformat(dt.isoformat()) == dt`\n", + "\n", + "### Calendar Module\n", + "- `calendar.isleap(year)` checks leap years\n", + "- `calendar.monthrange(year, month)` returns `(first_weekday, num_days)`\n", + "- `calendar.month(year, month)` generates text calendar\n", + "- `calendar.month_name` and `calendar.day_name` for name constants" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "version": "3.13.0" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} \ No newline at end of file diff --git a/src/chapter_29/README.md b/src/chapter_29/README.md new file mode 100644 index 0000000..31a12be --- /dev/null +++ b/src/chapter_29/README.md @@ -0,0 +1,9 @@ +# Chapter 29: Date, Time, and Scheduling + +Working with dates, times, and timezones — `datetime`, `zoneinfo`, `timedelta`, `calendar`, formatting and parsing. + +## Notebooks + +1. **01_datetime_fundamentals.ipynb** — `date`, `time`, `datetime`, `timedelta`, arithmetic +2. **02_timezones_and_zoneinfo.ipynb** — `zoneinfo`, aware vs naive datetimes, UTC, conversions +3. **03_formatting_and_calendar.ipynb** — `strftime`/`strptime`, ISO 8601, `calendar` module, date ranges diff --git a/src/chapter_29/__init__.py b/src/chapter_29/__init__.py new file mode 100644 index 0000000..925f495 --- /dev/null +++ b/src/chapter_29/__init__.py @@ -0,0 +1 @@ +"""Chapter 29: Date, Time, and Scheduling.""" diff --git a/src/chapter_30/01_xml_processing.ipynb b/src/chapter_30/01_xml_processing.ipynb new file mode 100644 index 0000000..6050bc2 --- /dev/null +++ b/src/chapter_30/01_xml_processing.ipynb @@ -0,0 +1,613 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "a1b2c3d4", + "metadata": {}, + "source": [ + "# Chapter 30: XML Processing\n", + "\n", + "This notebook covers XML parsing and generation using Python's built-in `xml.etree.ElementTree` module. XML (eXtensible Markup Language) remains widely used in configuration files, web services (SOAP), and data exchange formats.\n", + "\n", + "## Key Concepts\n", + "- **Parsing XML**: Converting XML strings or files into navigable tree structures\n", + "- **Building XML**: Creating elements programmatically with `Element` and `SubElement`\n", + "- **XPath searches**: Finding elements using path expressions\n", + "- **Namespaces**: Working with XML namespaces to avoid name collisions\n", + "- **Serialization**: Converting element trees back to XML strings" + ] + }, + { + "cell_type": "markdown", + "id": "b2c3d4e5", + "metadata": {}, + "source": [ + "## Section 1: Parsing XML from Strings\n", + "\n", + "`ET.fromstring()` parses an XML string and returns the root `Element`. Each element has a tag, optional text content, attributes, and child elements." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c3d4e5f6", + "metadata": {}, + "outputs": [], + "source": [ + "import xml.etree.ElementTree as ET\n", + "\n", + "# Parse a simple XML string\n", + "xml_data: str = \"helloworld\"\n", + "root: ET.Element = ET.fromstring(xml_data)\n", + "\n", + "print(f\"Root tag: {root.tag}\")\n", + "print(f\"Number of children: {len(root)}\")\n", + "\n", + "# Access child elements\n", + "items: list[ET.Element] = root.findall(\"item\")\n", + "for item in items:\n", + " print(f\" <{item.tag}> = {item.text}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "d4e5f6a7", + "metadata": {}, + "outputs": [], + "source": [ + "# Parse XML with attributes\n", + "xml_with_attrs: str = \"\"\"\n", + "\n", + " \n", + " Effective Java\n", + " Joshua Bloch\n", + " 2018\n", + " \n", + " \n", + " Head First Design Patterns\n", + " Eric Freeman\n", + " 2004\n", + " \n", + "\n", + "\"\"\"\n", + "\n", + "root = ET.fromstring(xml_with_attrs)\n", + "\n", + "for book in root.findall(\"book\"):\n", + " isbn: str | None = book.get(\"isbn\")\n", + " title: str | None = book.findtext(\"title\")\n", + " author: str | None = book.findtext(\"author\")\n", + " print(f\" [{isbn}] {title} by {author}\")" + ] + }, + { + "cell_type": "markdown", + "id": "e5f6a7b8", + "metadata": {}, + "source": [ + "## Section 2: Parsing XML from Files\n", + "\n", + "`ET.parse()` reads an XML file and returns an `ElementTree` object. The root element is obtained via `.getroot()`. For this demo, we write a temporary file first." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f6a7b8c9", + "metadata": {}, + "outputs": [], + "source": [ + "import tempfile\n", + "from pathlib import Path\n", + "\n", + "# Write a sample XML file\n", + "sample_xml: str = \"\"\"\n", + "\n", + " \n", + " Alice\n", + " Engineering\n", + " \n", + " \n", + " Bob\n", + " Marketing\n", + " \n", + "\n", + "\"\"\"\n", + "\n", + "with tempfile.NamedTemporaryFile(mode=\"w\", suffix=\".xml\", delete=False) as f:\n", + " f.write(sample_xml)\n", + " temp_path: str = f.name\n", + "\n", + "# Parse the file\n", + "tree: ET.ElementTree = ET.parse(temp_path)\n", + "root = tree.getroot()\n", + "\n", + "print(f\"Root tag: {root.tag}\")\n", + "for emp in root.findall(\"employee\"):\n", + " emp_id: str | None = emp.get(\"id\")\n", + " name: str | None = emp.findtext(\"name\")\n", + " dept: str | None = emp.findtext(\"department\")\n", + " print(f\" Employee {emp_id}: {name} in {dept}\")\n", + "\n", + "# Clean up\n", + "Path(temp_path).unlink()" + ] + }, + { + "cell_type": "markdown", + "id": "a7b8c9d0", + "metadata": {}, + "source": [ + "## Section 3: Building XML Programmatically\n", + "\n", + "Use `ET.Element()` for the root and `ET.SubElement()` to add children. Attributes are passed as a dictionary or keyword arguments." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b8c9d0e1", + "metadata": {}, + "outputs": [], + "source": [ + "# Build a catalog XML document from scratch\n", + "catalog: ET.Element = ET.Element(\"catalog\")\n", + "\n", + "# Add a book with attributes and child elements\n", + "book: ET.Element = ET.SubElement(catalog, \"book\", attrib={\"id\": \"1\"})\n", + "title: ET.Element = ET.SubElement(book, \"title\")\n", + "title.text = \"Python\"\n", + "price: ET.Element = ET.SubElement(book, \"price\")\n", + "price.text = \"39.99\"\n", + "\n", + "# Add a second book\n", + "book2: ET.Element = ET.SubElement(catalog, \"book\", attrib={\"id\": \"2\"})\n", + "title2: ET.Element = ET.SubElement(book2, \"title\")\n", + "title2.text = \"Algorithms\"\n", + "price2: ET.Element = ET.SubElement(book2, \"price\")\n", + "price2.text = \"49.99\"\n", + "\n", + "# Verify the structure\n", + "found_book: ET.Element | None = catalog.find(\"book\")\n", + "print(f\"First book title: {catalog.find('book/title').text}\")\n", + "print(f\"First book id: {catalog.find('book').get('id')}\")\n", + "\n", + "# List all books\n", + "for b in catalog.findall(\"book\"):\n", + " print(f\" Book {b.get('id')}: {b.findtext('title')} - ${b.findtext('price')}\")" + ] + }, + { + "cell_type": "markdown", + "id": "c9d0e1f2", + "metadata": {}, + "source": [ + "## Section 4: Serializing XML to Strings\n", + "\n", + "`ET.tostring()` converts an element tree back to an XML byte string. Use `encoding=\"unicode\"` to get a regular string instead of bytes." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "d0e1f2a3", + "metadata": {}, + "outputs": [], + "source": [ + "# Create a simple element\n", + "msg: ET.Element = ET.Element(\"msg\")\n", + "msg.text = \"hello\"\n", + "\n", + "# Serialize to bytes (default)\n", + "xml_bytes: bytes = ET.tostring(msg)\n", + "print(f\"Bytes: {xml_bytes}\")\n", + "print(f\"Type: {type(xml_bytes).__name__}\")\n", + "\n", + "# Serialize to unicode string\n", + "xml_str: str = ET.tostring(msg, encoding=\"unicode\")\n", + "print(f\"\\nString: {xml_str}\")\n", + "print(f\"Type: {type(xml_str).__name__}\")\n", + "\n", + "# Serialize the full catalog with XML declaration\n", + "catalog_str: str = ET.tostring(catalog, encoding=\"unicode\")\n", + "print(f\"\\nCatalog XML:\\n{catalog_str}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e1f2a3b4", + "metadata": {}, + "outputs": [], + "source": [ + "# Pretty-printing with indent (Python 3.9+)\n", + "ET.indent(catalog, space=\" \")\n", + "pretty_xml: str = ET.tostring(catalog, encoding=\"unicode\")\n", + "print(\"Pretty-printed catalog:\")\n", + "print(pretty_xml)" + ] + }, + { + "cell_type": "markdown", + "id": "f2a3b4c5", + "metadata": {}, + "source": [ + "## Section 5: XPath Expressions\n", + "\n", + "ElementTree supports a subset of XPath for searching the tree. Common patterns:\n", + "- `tag` -- direct children with given tag\n", + "- `.//tag` -- all descendants with given tag\n", + "- `*/tag` -- grandchildren with given tag\n", + "- `[@attrib]` -- elements with a specific attribute\n", + "- `[@attrib='value']` -- elements where attribute equals value" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a3b4c5d6", + "metadata": {}, + "outputs": [], + "source": [ + "store_xml: str = \"\"\"\n", + "\n", + " \n", + " Novel\n", + " 12.99\n", + " \n", + " \n", + " Python\n", + " 39.99\n", + " \n", + " \n", + " Mystery\n", + " 9.99\n", + " \n", + "\n", + "\"\"\"\n", + "\n", + "store: ET.Element = ET.fromstring(store_xml)\n", + "\n", + "# Find all titles anywhere in the tree\n", + "all_titles: list[str] = [el.text for el in store.findall(\".//title\")]\n", + "print(f\"All titles: {all_titles}\")\n", + "\n", + "# Find books by attribute\n", + "fiction_books: list[ET.Element] = store.findall(\"book[@category='fiction']\")\n", + "print(f\"\\nFiction books:\")\n", + "for book in fiction_books:\n", + " print(f\" {book.findtext('title')} - ${book.findtext('price')}\")\n", + "\n", + "# Find all prices\n", + "prices: list[float] = [float(el.text) for el in store.findall(\".//price\")]\n", + "print(f\"\\nAll prices: {prices}\")\n", + "print(f\"Total: ${sum(prices):.2f}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b4c5d6e7", + "metadata": {}, + "outputs": [], + "source": [ + "# More XPath examples with nested structures\n", + "nested_xml: str = \"\"\"\n", + "\n", + " \n", + " \n", + " Alice\n", + " Bob\n", + " \n", + " \n", + " Charlie\n", + " \n", + " \n", + " \n", + " \n", + " Diana\n", + " \n", + " \n", + "\n", + "\"\"\"\n", + "\n", + "company: ET.Element = ET.fromstring(nested_xml)\n", + "\n", + "# All members in the entire company\n", + "all_members: list[str] = [m.text for m in company.findall(\".//member\")]\n", + "print(f\"All members: {all_members}\")\n", + "\n", + "# Only team leads\n", + "leads: list[str] = [m.text for m in company.findall(\".//member[@role='lead']\")]\n", + "print(f\"Team leads: {leads}\")\n", + "\n", + "# Members in grandchild teams (department/team/member)\n", + "grandchild_members: list[str] = [\n", + " m.text for m in company.findall(\"department/team/member\")\n", + "]\n", + "print(f\"Via path: {grandchild_members}\")" + ] + }, + { + "cell_type": "markdown", + "id": "c5d6e7f8", + "metadata": {}, + "source": [ + "## Section 6: Modifying XML Trees\n", + "\n", + "Elements are mutable: you can change text, attributes, add or remove children after creation." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "d6e7f8a9", + "metadata": {}, + "outputs": [], + "source": [ + "# Start with a simple XML tree\n", + "xml_data = \"false\"\n", + "config: ET.Element = ET.fromstring(xml_data)\n", + "\n", + "# Modify text content\n", + "setting: ET.Element | None = config.find(\"setting\")\n", + "if setting is not None:\n", + " print(f\"Before: {setting.text}\")\n", + " setting.text = \"true\"\n", + " print(f\"After: {setting.text}\")\n", + "\n", + "# Modify attributes\n", + "if setting is not None:\n", + " setting.set(\"name\", \"verbose\")\n", + " setting.set(\"type\", \"boolean\")\n", + " print(f\"Attributes: {setting.attrib}\")\n", + "\n", + "# Add a new element\n", + "new_setting: ET.Element = ET.SubElement(config, \"setting\", name=\"timeout\")\n", + "new_setting.text = \"30\"\n", + "\n", + "# Remove an element\n", + "config.remove(setting)\n", + "\n", + "result: str = ET.tostring(config, encoding=\"unicode\")\n", + "print(f\"\\nFinal XML: {result}\")" + ] + }, + { + "cell_type": "markdown", + "id": "e7f8a9b0", + "metadata": {}, + "source": [ + "## Section 7: XML Namespaces\n", + "\n", + "Namespaces prevent name collisions when combining XML from different sources. They are declared with `xmlns` and appear as prefixes in tag names." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f8a9b0c1", + "metadata": {}, + "outputs": [], + "source": [ + "# XML with namespaces\n", + "ns_xml: str = \"\"\"\n", + "\n", + " \n", + " \n", + " Cell 1\n", + " Cell 2\n", + " \n", + " \n", + " \n", + " \n", + " \n", + "\n", + "\"\"\"\n", + "\n", + "# Define namespace map for searching\n", + "namespaces: dict[str, str] = {\n", + " \"h\": \"http://www.w3.org/1999/xhtml\",\n", + " \"f\": \"http://www.w3.org/2002/xforms\",\n", + "}\n", + "\n", + "root = ET.fromstring(ns_xml)\n", + "\n", + "# Find elements using namespace prefix\n", + "table: ET.Element | None = root.find(\"h:table\", namespaces)\n", + "if table is not None:\n", + " print(f\"Found table element: {table.tag}\")\n", + "\n", + "cells: list[ET.Element] = root.findall(\".//h:td\", namespaces)\n", + "for cell in cells:\n", + " print(f\" Cell: {cell.text}\")\n", + "\n", + "# Find form elements\n", + "form_input: ET.Element | None = root.find(\".//f:input\", namespaces)\n", + "if form_input is not None:\n", + " print(f\"\\nForm input name: {form_input.get('name')}\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a9b0c1d2", + "metadata": {}, + "outputs": [], + "source": [ + "# Register namespaces to preserve prefixes during serialization\n", + "ET.register_namespace(\"h\", \"http://www.w3.org/1999/xhtml\")\n", + "ET.register_namespace(\"f\", \"http://www.w3.org/2002/xforms\")\n", + "\n", + "# Build XML with namespaces using Clark notation {uri}local\n", + "XHTML: str = \"http://www.w3.org/1999/xhtml\"\n", + "doc: ET.Element = ET.Element(\"page\")\n", + "heading: ET.Element = ET.SubElement(doc, f\"{{{XHTML}}}h1\")\n", + "heading.text = \"Welcome\"\n", + "\n", + "result = ET.tostring(doc, encoding=\"unicode\")\n", + "print(f\"Namespaced XML: {result}\")" + ] + }, + { + "cell_type": "markdown", + "id": "b0c1d2e3", + "metadata": {}, + "source": [ + "## Section 8: Iterative Parsing with iterparse\n", + "\n", + "For large XML files, `ET.iterparse()` processes elements incrementally without loading the entire document into memory." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c1d2e3f4", + "metadata": {}, + "outputs": [], + "source": [ + "import io\n", + "\n", + "# Simulate a large XML file with a StringIO stream\n", + "large_xml: str = \"\"\"\n", + "\n", + " Application started\n", + " Low memory\n", + " Connection failed\n", + " Application stopped\n", + "\n", + "\"\"\"\n", + "\n", + "# Use iterparse to process events\n", + "warnings_and_errors: list[str] = []\n", + "\n", + "for event, elem in ET.iterparse(io.StringIO(large_xml), events=(\"end\",)):\n", + " if elem.tag == \"entry\":\n", + " level: str | None = elem.get(\"level\")\n", + " if level in (\"WARNING\", \"ERROR\"):\n", + " warnings_and_errors.append(f\"[{level}] {elem.text}\")\n", + " # Clear element to free memory in large files\n", + " elem.clear()\n", + "\n", + "print(\"Warnings and errors found:\")\n", + "for entry in warnings_and_errors:\n", + " print(f\" {entry}\")" + ] + }, + { + "cell_type": "markdown", + "id": "d2e3f4a5", + "metadata": {}, + "source": [ + "## Section 9: Practical Example -- Converting XML to Dictionaries\n", + "\n", + "A common pattern is converting XML data into Python dictionaries for easier manipulation." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e3f4a5b6", + "metadata": {}, + "outputs": [], + "source": [ + "from typing import Any\n", + "\n", + "\n", + "def xml_to_dict(element: ET.Element) -> dict[str, Any]:\n", + " \"\"\"Recursively convert an XML element to a dictionary.\"\"\"\n", + " result: dict[str, Any] = {}\n", + "\n", + " # Include attributes with @ prefix\n", + " for key, value in element.attrib.items():\n", + " result[f\"@{key}\"] = value\n", + "\n", + " # Include text content\n", + " if element.text and element.text.strip():\n", + " if len(element) == 0 and not element.attrib:\n", + " return element.text.strip() # type: ignore[return-value]\n", + " result[\"#text\"] = element.text.strip()\n", + "\n", + " # Process child elements\n", + " for child in element:\n", + " child_data: dict[str, Any] | str = xml_to_dict(child)\n", + " if child.tag in result:\n", + " # Convert to list for duplicate tags\n", + " if not isinstance(result[child.tag], list):\n", + " result[child.tag] = [result[child.tag]]\n", + " result[child.tag].append(child_data)\n", + " else:\n", + " result[child.tag] = child_data\n", + "\n", + " return result\n", + "\n", + "\n", + "# Convert a sample XML to dict\n", + "xml_data = \"\"\"\n", + "\n", + " Alice\n", + " alice@example.com\n", + " \n", + " admin\n", + " editor\n", + " \n", + "\n", + "\"\"\"\n", + "\n", + "root = ET.fromstring(xml_data)\n", + "user_dict: dict[str, Any] = xml_to_dict(root)\n", + "\n", + "import json\n", + "print(json.dumps(user_dict, indent=2))" + ] + }, + { + "cell_type": "markdown", + "id": "f4a5b6c7", + "metadata": {}, + "source": [ + "## Summary\n", + "\n", + "### Parsing XML\n", + "- **`ET.fromstring()`**: Parse XML from a string, returns root `Element`\n", + "- **`ET.parse()`**: Parse XML from a file, returns `ElementTree`\n", + "- **`ET.iterparse()`**: Incremental parsing for large files\n", + "\n", + "### Building XML\n", + "- **`ET.Element(tag, attrib)`**: Create a root element\n", + "- **`ET.SubElement(parent, tag)`**: Add a child element\n", + "- Set `.text` for element content, `.attrib` for attributes\n", + "\n", + "### Searching\n", + "- **`find(path)`**: First matching element\n", + "- **`findall(path)`**: All matching elements\n", + "- **`findtext(path)`**: Text of first matching element\n", + "- XPath subset: `.//tag`, `[@attr='val']`, `tag/subtag`\n", + "\n", + "### Serialization\n", + "- **`ET.tostring(elem, encoding=\"unicode\")`**: Convert to string\n", + "- **`ET.indent(elem)`**: Pretty-print formatting (Python 3.9+)\n", + "\n", + "### Namespaces\n", + "- Use namespace dictionaries with `find()`/`findall()`\n", + "- Clark notation `{uri}local` for building namespaced elements\n", + "- `ET.register_namespace()` for clean serialization" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "version": "3.13.0" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} \ No newline at end of file diff --git a/src/chapter_30/02_html_and_config.ipynb b/src/chapter_30/02_html_and_config.ipynb new file mode 100644 index 0000000..b3b1b23 --- /dev/null +++ b/src/chapter_30/02_html_and_config.ipynb @@ -0,0 +1,520 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "a1000001", + "metadata": {}, + "source": [ + "# Chapter 30: HTML Processing and Configuration Formats\n", + "\n", + "This notebook covers HTML parsing and escaping with the `html` module, and reading/writing configuration files using `configparser` and `tomllib`.\n", + "\n", + "## Key Concepts\n", + "- **HTML escaping**: Preventing XSS by converting special characters to entities\n", + "- **HTMLParser**: Event-driven parser for extracting data from HTML\n", + "- **configparser**: Reading and writing INI-style configuration files\n", + "- **tomllib**: Reading TOML configuration files (Python 3.11+)" + ] + }, + { + "cell_type": "markdown", + "id": "a1000002", + "metadata": {}, + "source": [ + "## Section 1: HTML Escaping and Unescaping\n", + "\n", + "The `html` module provides `escape()` and `unescape()` for converting between raw HTML and safe entity-encoded strings. This is essential for preventing cross-site scripting (XSS) attacks." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a1000003", + "metadata": {}, + "outputs": [], + "source": [ + "from html import escape, unescape\n", + "\n", + "# Escaping dangerous HTML content\n", + "dangerous: str = ''\n", + "safe: str = escape(dangerous)\n", + "\n", + "print(f\"Original: {dangerous}\")\n", + "print(f\"Escaped: {safe}\")\n", + "print(f\"Contains ' + safe = escape(dangerous) + assert "