E2E Test Catalogue

E2E Test Catalogue

aibox deliberately has no container-runtime bridge from its development container. Tests are placed according to the product behavior they prove, not according to the historical runner that happened to execute them.

Execution surfaces

Run the complete local suite with cd cli && cargo test. The compatibility feature e2e no longer grants access to another machine or runtime. Opt-in visual matrices run with:

./scripts/maintain.sh test-e2e-visual-status
./scripts/maintain.sh test-e2e-visual-tabs
./scripts/maintain.sh test-e2e-visual-yazi
./scripts/maintain.sh test-e2e-visual

Every process-level visual test removes inherited TMUX and TMUX_PANE, owns a socket beneath its TempDir, and cleans up only its own server. Tests must not use global tmux kill-server, broad /tmp/tmux-* removal, or global pkill; those operations can terminate the developer’s live session.

Removed companion

The former aibox-e2e-testrunner, SSH/SCP runner, nested Podman/systemd/kind image, passwordless elevation, capabilities, devices, host module mount, and relaxed security profile were removed for issue #372. No test required a separate unprivileged clean-room service. The generic Alpine pull/run test was also removed because it supplied weaker evidence than the actual generated aibox image lifecycle.

The remaining runtime-only contracts are:

• build the candidate foundation and runtime image from the release source;
• generate a downstream project with the candidate Darwin binary;
• run aibox apply and Compose up against that locally built image;
• verify identity, non-root execution, readiness, tmux/Yazi/status tooling, and the --forget-tmux-state attach behavior;
• bring down only that run’s Compose project and fail if cleanup fails;
• generate a CycloneDX SBOM and fail the vulnerability gate on high severity.

These contracts inherently need ordinary host Docker and therefore run only in the macOS release gate. Missing Docker, Apple targets, Syft, Grype, or any other mandatory prerequisite is a failure, never a passing skip.

Three expensive surfaces remain mandatory when their attested inputs change:

• grouped builds for affected download-based addons;
• a two-revision LaTeX watcher build with byte-identical preview-sidecar output;
• the infrastructure addon’s nested Podman probe, including a true rootless report.

The immutable provenance records the previous version-line tag and commit plus the exact changed-path list. The gate recomputes that diff before selecting checks. Base-image, generator, addon-loader, template, or host-gate changes select every conditional surface; an addon definition selects its group, and LaTeX or infrastructure changes select their dedicated lifecycle probe. If no comparison tag exists, all conditional checks run. Every selection and non-selection is written to evidence. A selected check cannot pass by skipping.

Host run-directory protocol

The container-side release creates immutable input under:

tmp/host-gates/aibox-release/<run-id>/
└── input/
    ├── provenance.json
    ├── checksums.sha256
    └── source.tar.gz

The owner runs the single command printed in dist/RELEASE-PROMPT.md:

./scripts/maintain.sh release-host tmp/host-gates/aibox-release/<run-id>

Append --dry-run to execute the same builds, runtime probes, cleanup, security scans, and manifest creation without invoking publication:

./scripts/maintain.sh release-host tmp/host-gates/aibox-release/<run-id> --dry-run

The completed run directory can later be published with the exact release-host-publish.sh command printed by the validator. Do not rerun the gate without --dry-run; populated runtime/ and evidence/ directories are intentionally non-resumable.

The entry point accepts that one path plus the optional fixed --dry-run flag. It canonicalizes the path, requires one direct child of the approved root, rejects symlinks, special files, hardlinks, unexpected files, unsafe permissions, bad checksums, and tag/commit mismatches, then creates runtime/ and evidence/ itself.

The entry point uses the owner-installed uv binary from the fixed Homebrew path and requires the exact Python 3.12.11 interpreter in offline mode with --no-project. Its cache and managed-Python roots are fixed beneath ~/Library/Caches/aibox-host-gates/uv and ~/Library/Application Support/aibox-host-gates/python; candidate pyproject.toml, inline metadata, uv configuration, package indexes, and inherited UV_* variables cannot affect execution. The resolved uv/Python versions and paths are recorded in evidence.

Candidate compilation, build scripts, CLI execution, and runtime smoke run with a fixed sanitized environment and a macOS sandbox that denies access to GitHub configuration, Docker configuration, SSH material, and Keychain security services. Docker receives an empty per-run configuration and no secrets or host mounts. Tool runtime state is fixed under runtime/; no candidate project configuration controls Python or package resolution.

Only after every required validation succeeds does the gate invoke the small publisher. The publisher re-verifies the release manifest and hashes, uploads exactly two Darwin archives plus their two checksum files, and pushes only the fixed foundation-version, runtime-version, and runtime-latest image tags. It does not build, test, execute candidate code, modify Git, or accept additional arguments.

Evidence contains exact commands and results, toolchain/runtime metadata, changed-path selection reasons, Darwin build and smoke records, image inspection, generated-runtime and selected conditional-check logs, SBOM, vulnerability scan, release manifest, and remote publication checks.

Failure and rerun

The gate is fail-closed and does not reuse a partially populated run directory. On failure, preserve its evidence for diagnosis. After fixing the release candidate and creating a new tag/commit, run the container-side release again to prepare a new run ID. Do not edit an old input/, delete selected evidence, or resume publication from an unverified partial run.

The owner must review the entry point before its first real invocation. Any later change to the gate or publisher requires a fresh owner review before use.