Validate and resolve file paths before API work

[sahilsunny]

Summary
Fix path handling for --output-file, --output-dir, and --input-file so ~/ home shortcuts work, missing output directories are created, overwrite conflicts are detected early, and input files are verified before any scrape or batch API call. This prevents wasted credits from scraping first and failing on write, and gives clear REPL-friendly errors instead of literal '~/...' path failures.

Details

• cli_utils.py:323-325 — add resolve_output_path() to expand ~ via Path.expanduser() so all path helpers share one resolution step.
• cli_utils.py:328-349 — add ensure_output_file_ready() to expand ~, mkdir parent dirs, and run overwrite checks before API work.
• cli_utils.py:352-360 — add ensure_output_dir_ready() to expand ~ and create --output-dir before batch/crawl writes.
• cli_utils.py:363-385 — add ensure_input_file_ready() to expand ~, verify the file exists/is readable, and pass stdin (-) through unchanged.
• cli_utils.py:392-403 — update confirm_overwrite() to check the expanded path so ~/Desktop/foo.png correctly detects an existing Desktop file; REPL still raises UsageError with --overwrite hint instead of blocking on click.confirm().
• cli_utils.py:591-601 — call all ensure_* helpers at the end of store_common_options(), after flag validation but before commands hit the API.
• cli_utils.py:1902-1923 — harden write_output() with expand/mkdir/open on the resolved path as a safety net when extensions are appended post-validation.
• batch.py:220-221 — expand ~ in read_input_file() as a fallback for callers outside store_common_options().
• crawl.py:432,569 — ensure crawl output dirs (explicit --output-dir and default crawl_) exist before the spider starts.
• export.py:96-101 — replace confirm_overwrite() with ensure_output_file_ready() so export validates the destination before reading input files.
• test_v132_fixes.py:600-860 — unit tests for tilde expansion, early input/output validation, and REPL scrape guards that API must not run on bad paths.
• test_repl_pty.py:320-345 — PTY regression that REPL shows overwrite error for existing ~/ output paths before scraping.
• test_v132_fixes.py:955-1001 — update batch option tests to create real input files now that store_common_options validates --input-file up front.

[kostas-jakeliunas-sb]

@sahilsunny ok I have some concerns re: this, I had suspicions re: path expansion using the tilde (~) homedir ref, nothing particular,

asked claude fable and got some stuff. I'd like P1 stuff addressed (see separate claude-generated comment), either make changes or let's discuss / I want to hear why not / why I'm paranoid :D and re: P2 - nice to have, not blocking from my PoV. But good to do, imho.

p.s. all info here: #29 (comment)

[kostas-jakeliunas-sb]

TL;DR: No path traversal found — but two P1 fixes needed before merge: (1) bare Path.expanduser() crashes on ~-prefixed filenames and silently expands ~user into other users' home dirs; (2) the new early overwrite check is bypassed when a file extension is auto-appended after validation, so existing files still get silently clobbered. Both are small fixes. P2/P3 below are non-blocking.

(Claude-generated review; verified against PR head f7c503c, expanduser behavior reproduced on Python 3.13.)

P1 — fix before merge

1. Unguarded expanduser(): crash + foreign-home writes.

• --output-file "~data.csv" (a legal filename, and exactly what a script or agent might emit) → uncaught RuntimeError: Could not determine home directory → raw traceback in CLI mode.
• --output-file "~root/x.png" → resolves to /var/root/x.png, i.e. writes into another user's home. Nobody wants ~user expansion in this tool.
• One condition fixes both: only expand paths that are exactly ~ or start with ~/ (or ~\ on Windows); treat everything else literally.

2. Extension-append bypasses the early overwrite check.

• --output-file report: the check runs on report (doesn't exist → passes), the scrape runs, the path then becomes report.html, and write_output() opens it "wb" with no overwrite check — an existing report.html is silently destroyed, --overwrite never consulted.
• This undercuts the PR's own guarantee ("overwrite conflicts are detected early"), and note the appended extension is influenced by response headers/body — i.e. by the scraped site.

P2 — agent-context hardening (non-blocking, recommended)

Default recursive mkdir -p + ~ expansion makes a single allowlisted CLI invocation a more capable filesystem-write primitive. Our own guard skill treats scraped content as hostile (prompt injection), and our agent skills actively teach --output-file/--overwrite — so "injected instruction steers one CLI call" is our documented threat model. Not a new vuln class (absolute-path writes already worked), but it widens targeting: ~ is username-independent, and mkdir -p makes previously nonexistent config trees reachable. Also, parent dirs are created before the overwrite prompt and the API call, so declined prompts / failed scrapes leave typo'd empty trees behind.

Cheap wins: (a) audit-log resolved write destinations (audit.py infra exists; the guard skill already reads that log but today only sees exec events), (b) have the guard skill flag writes outside CWD/expected output dirs, (c) create parents at write time while keeping the early writability check (preserves the don't-waste-credits goal).

P3 — quality (non-blocking)

• expand + mkdir + OSError→SystemExit logic is duplicated in ensure_output_file_ready and write_output — extract one helper; resolve_output_path is also used for input files — rename to e.g. expand_user_path.
• Tests are genuinely good (unit + PTY, Windows-aware HOME/USERPROFILE). Please add the two P1 cases (see below).

Cleared (checked, no issue)

No traversal via scraped content (batch/crawl filenames are numeric 1.<ext>, extensions whitelist-bounded — hostile URL/Content-Type can't inject / or ..); REPL survives the new SystemExit(1) paths; validation ordering is correct; --update-csv skip matches old behavior; exec_gate untouched.

---

Technical detail (for implementation)

P1.1 — resolve_output_path() at src/scrapingbee_cli/cli_utils.py:323 and read_input_file() at src/scrapingbee_cli/batch.py:220 call Path(path).expanduser() unguarded.

Repro (Python 3.13):

Path('~data.csv').expanduser()    # RuntimeError: Could not determine home directory.
Path('~root/x.png').expanduser()  # /var/root/x.png — another user's home (POSIX)
Path('~/ok.png').expanduser()     # <home>/ok.png — the only intended case

Suggested fix:

def expand_user_path(path: str) -> str:
    """Expand a leading ``~/`` (current user only); leave ``~user`` and ``~foo.csv`` literal."""
    if path == "~" or path.startswith(("~/", "~\\")):
        return str(Path(path).expanduser())
    return path

(Alternative: try/except RuntimeError → click.UsageError, but the prefix check is stricter and also stops ~user expansion.)

P1.2 — src/scrapingbee_cli/commands/scrape.py:777-786: when os.path.basename(output_path) has no dot, an extension is appended after store_common_options() validation — in the force_extension branch, the extension_for_crawl(...) branch, and the --chunk-size .ndjson branch (~line 778). write_output() (src/scrapingbee_cli/cli_utils.py:1902-1923) then does expand + mkdir + open(resolved, "wb") with no confirm_overwrite. Fix options: call confirm_overwrite(resolved, overwrite) inside write_output() whenever the final resolved path differs from the already-validated one; or pre-check candidate paths (base + appendable extensions) during early validation. Don't forget the chunking path.

Tests to add:

• --output-file "~data.csv" → clean UsageError/message, no traceback; path treated literally.
• Existing report.html + --output-file report without --overwrite → prompt (CLI) / UsageError (REPL), no silent clobber.