feat: support tar archives for installs

[mnriem]

Summary

• add secure .tar.gz / .tgz extraction with ZIP-equivalent path, link, conflict, entry-count, and size protections
• support ZIP, .tar.gz, and .tgz across extension and preset direct URL, catalog, install, update, cleanup, and SHA-256 flows
• preserve complete workflow packages from local directories and local/remote/catalog archives with transactional directory replacement
• document workflow archive/package installation and add archive parity coverage

Validation

• .venv/bin/python -m pytest tests/test_download_security.py -q
• .venv/bin/python -m pytest tests/test_extensions.py tests/test_presets.py tests/test_workflows.py -q

Authored autonomously on behalf of @mnriem by GitHub Copilot (model: GPT-5.6 Sol).

[Copilot]

Pull request overview

Adds secure tar archive support and complete workflow package installation alongside existing ZIP flows.

Changes:

• Adds secure .tar.gz/.tgz detection and extraction.
• Extends extension, preset, and workflow install/update flows.
• Documents and tests archive parity and package preservation.

Show a summary per file

File	Description
src/specify_cli/_download_security.py	Adds archive detection and secure tar extraction.
src/specify_cli/extensions/__init__.py	Supports tar installation and catalog downloads.
src/specify_cli/extensions/_commands.py	Extends extension CLI and updates to archives.
src/specify_cli/presets/__init__.py	Supports tar installation and catalog downloads.
src/specify_cli/presets/_commands.py	Extends preset URL and catalog installation.
src/specify_cli/workflows/_commands.py	Installs complete workflow packages transactionally.
tests/test_download_security.py	Tests archive detection and tar safeguards.
tests/test_extensions.py	Tests extension tar flows.
tests/test_presets.py	Tests preset tar flows.
tests/test_workflows.py	Tests workflow package preservation.
docs/reference/workflows.md	Documents workflow package archives.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comments suppressed due to low confidence (2)

src/specify_cli/workflows/_commands.py:2183

• Catalog URLs may also be direct GitHub REST release-asset URLs, which this resolver explicitly supports, but such URLs are suffixless and normally return application/octet-stream. downloaded_archive_format then remains None, causing a ZIP/tar workflow package to take the YAML path and fail. Add byte/response-filename format detection for the ambiguous response before choosing the YAML installation branch.

            github_hosts=_github_provider_hosts(),
            redirect_validator=_reject_insecure_download_redirect,
        )
        if _resolved_workflow_url:
            workflow_url = _resolved_workflow_url

src/specify_cli/workflows/_commands.py:983

• Unlike the existing single-YAML staging path, this calls mkdtemp(..., dir=workflows_dir) before ensuring .specify/workflows exists and outside the following error handler. Older/minimal Specify projects without that directory can install a YAML workflow because _stage_workflow_file creates parents, but every archive or companion-directory install now raises FileNotFoundError. Safely create/validate workflows_dir before allocating the staging directory and report creation failures as a CLI error.

    staged_dir = Path(
        tempfile.mkdtemp(prefix=f".{definition.id}.installing-", dir=workflows_dir)
    )
    try:

• Files reviewed: 11/11 changed files
• Comments generated: 3
• Review effort level: Medium

[Copilot]

Review details

Comments suppressed due to low confidence (3)

src/specify_cli/workflows/_commands.py:1982

• Use the same URL declaration that selected downloaded_archive_format. When a declared URL such as package.zip redirects to an extensionless URL, this call currently validates only the final URL and therefore accepts tar bytes, bypassing the new suffix/content mismatch protection. Fall back to the requested URL when the final URL has no recognized archive suffix, matching the extension and preset paths.

                            source_name=final_url,

src/specify_cli/workflows/_commands.py:2246

• This extraction validates against original_workflow_url even though format selection gives a recognized redirected final_url precedence. If the catalog URL is extensionless and redirects to package.zip, tar bytes are accepted because the recognized final suffix is discarded here. Pass the same final-URL/fallback declaration used during selection so filename/content mismatches remain enforced.

                    source_name=original_workflow_url,

src/specify_cli/workflows/_commands.py:984

• The package is parsed and validated from package_dir before this copy. For local-directory installs, another process can modify workflow.yml or replace a companion between validation and copying, leaving installed bytes that were never validated while the registry records the earlier definition. Create the staged snapshot without following links, then validate and parse that staged tree before committing it.

        shutil.copytree(package_dir, staged_dir, dirs_exist_ok=True)

• Files reviewed: 11/11 changed files
• Comments generated: 1
• Review effort level: Medium

[Copilot]

Review details

Comments suppressed due to low confidence (5)

src/specify_cli/workflows/_commands.py:977

• Archive catalog installs compare raw version strings here, while the existing YAML catalog path normalizes versions with packaging.version.Version via versions_match (lines 2101-2109). Thus values accepted by the current validator but normalized as equal, such as 2.0.00 and 2.0.0, succeed for YAML yet fail for an archive. Reuse the same comparator to retain archive parity.

    if expected_version is not None and str(definition.version) != expected_version:
        console.print(
            f"[red]Error:[/red] Downloaded workflow version "
            f"({_escape_markup(str(definition.version))}) does not match the catalog "
            f"version ({_escape_markup(expected_version)})."
        )
        raise typer.Exit(1)

src/specify_cli/presets/init.py:3598

• This “backward-compatible” wrapper drops the existing force parameter. The preset bundle refresh path still calls install_from_zip(..., force=True) (src/specify_cli/bundler/services/primitives.py:211-213), so refreshing a catalog-backed preset now raises TypeError instead of reinstalling it. Preserve and forward force as the archive method does.

    def install_from_zip(
        self,
        zip_path: Path,
        speckit_version: str,
        priority: int = 10,
    ) -> PresetManifest:
        """Backward-compatible wrapper for archive installation."""
        return self.install_from_archive(zip_path, speckit_version, priority)

src/specify_cli/extensions/init.py:3978

• If format validation or the final replace fails, an OSError from this cleanup replaces the original ExtensionError (for example when a scanner locks the temporary file on Windows). Cleanup is best-effort here; suppress its failure so callers receive the actual download/validation error.

            if staging_path is not None:
                staging_path.unlink(missing_ok=True)

src/specify_cli/presets/init.py:4779

• If format validation or the final replace fails, an OSError from this cleanup replaces the original PresetError (for example when a scanner locks the temporary file on Windows). Treat temporary-file cleanup as best-effort so the real download/validation failure remains visible.

            if staging_path is not None:
                staging_path.unlink(missing_ok=True)

src/specify_cli/workflows/_commands.py:982

• mkdtemp(dir=workflows_dir) runs before any code creates workflows_dir and outside the staging error handler. A directory containing only .specify/ is accepted as a project (_require_specify_project), and the existing YAML installer creates this parent on demand, but package/archive installs now fail with an unhandled FileNotFoundError. Safely create/validate the workflows directory before staging and report creation failures through the normal install error path.

    staged_dir = Path(
        tempfile.mkdtemp(prefix=f".{definition.id}.installing-", dir=workflows_dir)
    )

• Files reviewed: 11/11 changed files
• Comments generated: 0 new
• Review effort level: Medium

[Copilot]

Review details

Comments suppressed due to low confidence (4)

src/specify_cli/presets/init.py:3598

• This backward-compatible wrapper drops the old force parameter. _PresetKindManager._do_install() still calls install_from_zip(..., force=True) during bundle refresh (src/specify_cli/bundler/services/primitives.py:211-213), so refreshing a catalog preset now raises TypeError instead of reinstalling it. Preserve and forward the existing argument.

    def install_from_zip(
        self,
        zip_path: Path,
        speckit_version: str,
        priority: int = 10,
    ) -> PresetManifest:
        """Backward-compatible wrapper for archive installation."""
        return self.install_from_archive(zip_path, speckit_version, priority)

src/specify_cli/workflows/_commands.py:1940

• Archive selection here relies only on the URL suffix or Content-Type. A valid archive served from a suffixless download endpoint with the common application/octet-stream type leaves this as None, is read under the YAML limit, and is then decoded as workflow YAML. This makes direct archive URLs such as REST release-asset endpoints fail even though detect_archive_format() supports content-based detection. Sniff the downloaded bytes while retaining the separate YAML and archive size limits.

                downloaded_archive_format = (
                    archive_format_from_name(final_url)
                    or archive_format_from_name(download_url)
                    or archive_format_from_content_type(content_type)
                )

src/specify_cli/workflows/_commands.py:2223

• Catalog archives from suffixless endpoints (for example, a GitHub REST asset URL returning application/octet-stream) are classified as YAML here and subsequently fail UTF-8/YAML parsing. Unlike the extension and preset catalog paths, this branch never detects the archive from its downloaded bytes when metadata is inconclusive. Add post-download content detection while preserving the YAML-specific size bound.

            downloaded_archive_format = (
                archive_format_from_name(final_url)
                or archive_format_from_name(original_workflow_url)
                or archive_format_from_content_type(archive_content_type)
            )

src/specify_cli/workflows/_commands.py:1059

• Rollback removes the newly installed directory before restoring backup_dir. If rmtree() fails (or the following restore rename fails), the exception escapes to the outer handler, the prior workflow remains stranded under the hidden backup name, and the original registry failure is masked. Move the failed install aside atomically, restore the backup first, and treat cleanup of the failed directory as best-effort so a registry failure cannot make the previous package disappear.

                shutil.rmtree(dest_dir)

• Files reviewed: 11/11 changed files
• Comments generated: 1
• Review effort level: Medium

[Copilot]

Review details

Comments suppressed due to low confidence (3)

src/specify_cli/presets/init.py:3596

• The backward-compatible wrapper drops the existing force parameter even though install_from_archive still supports it. External callers that used install_from_zip(..., force=True) will now fail with TypeError instead of reinstalling the preset; preserve the old parameter and forward it.

        priority: int = 10,
    ) -> PresetManifest:

src/specify_cli/workflows/_commands.py:1940

• Archive routing depends only on the URL suffix or Content-Type before examining the response body. A direct GitHub API asset URL such as /releases/assets/42 normally has no suffix and can return application/octet-stream; if that asset is ZIP/tar.gz, this remains None, the archive is read under the YAML limit, and is then parsed as UTF-8 YAML. Sniff the downloaded bytes (while retaining YAML support) so extensionless/octet-stream archive URLs use the archive path.

                downloaded_archive_format = (
                    archive_format_from_name(final_url)
                    or archive_format_from_name(download_url)
                    or archive_format_from_content_type(content_type)
                )

src/specify_cli/workflows/_commands.py:2223

• Catalog archive detection has the same metadata-only gap: a catalog URL that is a direct GitHub API asset (or any extensionless endpoint returning application/octet-stream) leaves this as None, so valid ZIP/tar.gz bytes fall through to YAML decoding instead of package extraction. Detect archive magic after download, while preserving plain-YAML API assets.

            downloaded_archive_format = (
                archive_format_from_name(final_url)
                or archive_format_from_name(original_workflow_url)
                or archive_format_from_content_type(archive_content_type)
            )

• Files reviewed: 11/11 changed files
• Comments generated: 0 new
• Review effort level: Medium

[mnriem]

Addressed the review feedback in 15255753: restored the preset force compatibility argument, added content-based detection for suffixless workflow archives while retaining YAML size limits, and made package rollback restore the prior workflow before best-effort cleanup. Added regression coverage for each path; the full local suite passes.

Posted on behalf of @mnriem by GitHub Copilot (model: GPT-5.6 Sol).

[Copilot]

Review details

Comments suppressed due to low confidence (3)

src/specify_cli/workflows/_commands.py:2362

• Archive catalog installs are covered, but the workflow-update suite still only downloads YAML. Add an archive-based update test that starts from an installed package, exercises expected_installed_version, verifies companion files are replaced, and confirms the old package survives a failed archive update. This is needed to cover the PR's stated archive update and transactional replacement behavior.

                _install_workflow_package(
                    project_root,
                    workflows_dir,
                    package_root,
                    workflow_url,
                    expected_id=workflow_id,
                    expected_version=expected_version,
                    expected_installed_version=expected_installed_version,
                    catalog_info={**info, "url": workflow_url},
                )

src/specify_cli/extensions/_commands.py:1862

• The update path now claims tar support and performs destructive remove/reinstall work, but all extension-update integration fixtures still create ZIP files; the new tar update route is therefore untested. Add a .tar.gz/.tgz update case that exercises this preflight through successful installation, and ideally verifies rollback leaves the previous extension intact on a tar validation/install failure.

                    with tempfile.TemporaryDirectory(
                        prefix="speckit-update-archive-"
                    ) as archive_tmpdir:
                        extracted_root = Path(archive_tmpdir)
                        try:
                            safe_extract_archive(archive_path, extracted_root)

src/specify_cli/workflows/_commands.py:1035

• This silently drops a package's top-level overlays/ directory, so directory and archive installs do not preserve the complete package promised by this PR and the updated workflow documentation. Project overlays live at .specify/workflows/overlays/<id>, whereas this directory would be installed under .specify/workflows/<id>/overlays, so it does not collide with the reserved project overlay root. Copy all validated package entries, or explicitly document and validate this exclusion instead of silently omitting content.

            if Path(source).resolve() == package_root and "overlays" in names:
                return {"overlays"}

• Files reviewed: 11/11 changed files
• Comments generated: 0 new
• Review effort level: Medium