enhancement: Rename WARC output

[laurieburchell]

There is currently no option to rename an outputted WARC. For example, running the following command creates a file containing one record called TEST-000000.extracted.warc.gz

 cdxt --cc --crawl CC-MAIN-2025-43 --from 20251016192109 --limit 1 warc 'commoncrawl.org/get-started'

There should be the option to rename this file (-n flag?). In addition, the default name should not be TEST but something like OUTPUT.

@malteos I can't assign people in this repo but did you want to work on this?

[malteos]

There is already the --prefix argument, but this probably needs to be better documented.

[laurieburchell]

yes, the current --help text doesn't mention it:

$ cdxt --help

usage: cdxt [-h] [--version] [--verbose] [--cc] [--crawl CRAWL] [--ia] [--source SOURCE] [--wb WB] [--limit LIMIT] [--cc-mirror CC_MIRROR]
            [--cc-sort CC_SORT] [--from FROM] [--to TO] [--filter FILTER] [--get] [--closest CLOSEST]
            {iter,warc,size} ...

cdx_toolkit iterator command line tool

positional arguments:
  {iter,warc,size}
    iter                iterate printing captures
    warc                iterate over capture content, creating a warc
    size                imprecise count of how many results are available

options:
  -h, --help            show this help message and exit
  --version, -V         show program's version number and exit
  --verbose, -v         set logging level to INFO (-v) or DEBUG (-vv)
  --cc                  direct the query to the Common Crawl CDX/WARCs
  --crawl CRAWL         crawl names (comma separated) or an integer for the most recent N crawls. Implies --cc
  --ia                  direct the query to the Internet Archive CDX/wayback
  --source SOURCE       direct the query to this CDX server
  --wb WB               direct replays for content to this wayback
  --limit LIMIT
  --cc-mirror CC_MIRROR
                        use this Common Crawl index mirror
  --cc-sort CC_SORT     default mixed, alternatively: ascending
  --from FROM
  --to TO
  --filter FILTER       see CDX API documentation for usage
  --get                 use a single get instead of a paged iteration. default limit=1000
  --closest CLOSEST     get the closest capture to this timestamp. use with --get

[malteos]

It's a subcommand-specific argument (for warc). So it's hidden in cdxt warc -h:

usage: cdxt warc [-h] [--prefix PREFIX] [--subprefix SUBPREFIX] [--size SIZE] [--creator CREATOR] [--operator OPERATOR]
                 [--url-fgrep URL_FGREP] [--url-fgrepv URL_FGREPV] [--warc-download-prefix WARC_DOWNLOAD_PREFIX]
                 url

positional arguments:
  url

options:
  -h, --help            show this help message and exit
  --prefix PREFIX       prefix for the warc filename
  --subprefix SUBPREFIX
                        subprefix for the warc filename, default None
  --size SIZE           target for the warc filesize in bytes
  --creator CREATOR     creator of the warc: person, organization, service
  --operator OPERATOR   a person, if the creator is an organization
  --url-fgrep URL_FGREP
                        this pattern must be present to warc an url
  --url-fgrepv URL_FGREPV
                        this pattern must not be present to warc an url, e.g. /robots.txt
  --warc-download-prefix WARC_DOWNLOAD_PREFIX
                        prefix for downloading content, automatically set for CC

[laurieburchell]

Makes sense. The default for the file prefix should still be changed to something other than TEST though, and the default should be documented.

[wumpus]

I intentionally wanted a dumb-looking default, so that users will use --prefix.

The docs should definitely be improved to explain this more fully -- maybe also a ugly warning if --prefix wasn't specified. I don't want it to crash if --prefix is missing.

(I like half-user-friendly defaults)