CLI distribution verification technical specification

Terminology

Bundled-Python archive: the shipped compressed package containing a relocatable Python runtime, Crystal Lattice source-shaped package trees, bundled plugin projects, manifests, and an archive launcher.
Archive launcher: bin/crystal-lattice on macOS/Linux or Scripts/crystal-lattice.cmd on Windows inside the extracted archive. It starts the packaged Python with the packaged application and runtime layout.
Installed command: the user-facing crystal-lattice command created by a public installer. It is a small forwarding script, also called a wrapper, that invokes the archive launcher from the final installation directory.
Bash installer: the standalone install.sh used on macOS, Linux, and supported Git Bash/MSYS2/Cygwin environments on Windows.
PowerShell installer: the standalone install.ps1 used natively on Windows without Git Bash, WSL, Python, or administrator privileges.
Preinstalled plugin: a packaged plugin project already installed into the archive's Python runtime and listed in preinstalled-plugins.json.
User plugin cache: the config-owned directory where the application uses packaged Python and pip to install a supported plugin that is not listed as preinstalled.

“Level 1”, “level 2”, and “level 3” were planning shorthand for the increasing number of distribution transformations. The maintained names are source-installation smoke, extracted-archive smoke, and default-installer smoke. New commands, tests, messages, and documentation use those names.

Independent smoke scenarios

The scenarios share lower-level helpers, but each user workflow is executed directly and remains independently interpretable.

Scenario	Directly executed user entry point	Fresh mutable state	Main defect boundary
Source-installation smoke	`crystal-lattice` generated by installing editable Core and application packages into a fresh venv	Venv, workspace, home, config, plugin cache, and pip cache	Source dependency installation, dynamic default-plugin installation, and application boot
Extracted-archive smoke	Archive launcher from a fresh ordinary extraction	Extraction, workspace, home, config, plugin cache, and pip cache	Archive contents, relocatability, launcher, packaged Python, preinstalled closure, and additional dynamic plugin installation
Default Bash-installer smoke	Public `install.sh`, followed by `crystal-lattice` resolved by name through PATH	Download endpoint, home, install root, command directory, workspace, config, plugin cache, and pip cache	Bash download selection, extraction, staging replacement, default command placement, wrapper forwarding, PATH discovery, relocation, and additional dynamic plugin installation
Default PowerShell-installer smoke	Public `install.ps1`, followed by `crystal-lattice` resolved by name through PowerShell and a fresh `cmd.exe`	Download endpoint, Windows user profile, install root, command directory, workspace, config, plugin cache, and pip cache	Native Windows download selection, ZIP extraction, staging replacement, per-user PATH behavior, `.cmd` forwarding, command discovery, relocation, and additional dynamic plugin installation

The extracted-archive smoke never calls an installer. Neither default-installer smoke calls or treats the extracted-archive smoke or the other installer smoke as a prerequisite. They may consume the same immutable archive built by one native CI job, but they must not share any mutable directory or cache. A failure in one smoke must not prevent the others from producing a result after archive construction and archive location have succeeded.

Each scenario is available through one root npm command. Archive-consuming commands receive an already built current-platform archive; scenario code never installs build dependencies into the repository environment.

Root orchestration may accept an archive through CRYSTAL_LATTICE_PYTHON_RUNTIME_ARCHIVE; otherwise it completes an isolated archive build before starting the consumer test. This variable is distinct from the existing CRYSTAL_LATTICE_BUNDLE_ARCHIVE input used by optimized PyInstaller tests.

The default build uses a temporary isolated host build venv. Build dependencies are installed only there, never into the repository environment running the orchestrator or a test. Unless CRYSTAL_PYTHON_RUNTIME supplies an explicit runtime directory, the build wrapper provisions an uv-managed Python 3.11 runtime under its temporary directory and passes that runtime to the package builder. Portability validation remains part of the extracted-archive smoke; an accidental dependency on the developer's active Python environment is not acceptable test setup.

Native release targets

The bundled-Python release matrix contains:

Target	Archive	Default installation paths
`darwin-amd64`	`.tar.gz`	Bash installer and manual archive
`darwin-arm64`	`.tar.gz`	Bash installer and manual archive
`linux-amd64`	`.tar.gz`	Bash installer and manual archive
`linux-arm64`	`.tar.gz`	Bash installer and manual archive
`windows-amd64`	`.zip`	PowerShell installer, Git Bash installer, and manual archive
`windows-arm64`	`.zip`	PowerShell installer and manual archive; Git Bash only when a native compatible environment is verified

An archive counts as supported only after the native runner builds it, verifies that packaged Python reports the declared architecture, and passes the direct archive smoke. Cross-compilation or emulated execution alone does not establish native support.

The release workflow must use native GitHub-hosted runners for all six targets. If a required runner is unavailable or a dependency cannot install natively, the release fails and public documentation continues identifying that target as unavailable. The workflow must not silently replace a native job with an AMD64 archive running under emulation.

Common configuration and boot contract

The source-installation smoke exercises first-run creation with:

--server --config-local --host 127.0.0.1 --port <reserved-port>

The archive-based smokes need to add the designated plugin probe, so they start with an isolated copy of the packaged default config:

--server --config <isolated-config> --host 127.0.0.1 --port <reserved-port>

The probe:

Uses a fresh working directory.
Sets both HOME and USERPROFILE to one temporary user directory so Path.home() resolves away from developer state on POSIX and Windows.
For source installation, lets --config-local copy the shipped default config into the fresh workspace and verifies that generated file.
For the archive-based smokes, copies the exact default-config directory shipped by the selected command into a fresh config directory and appends one designated packaged Python plugin to its global plugin list. The plugin must be absent from preinstalled-plugins.json.
Reads the resulting config, collects every variable referenced through ${env:...}, and removes those names from the child process environment.
Places a controlled empty .env in the working directory so standard parent-directory discovery cannot load a developer file.
Removes inherited config, plugin-cache, Python-path, and runtime-layout variables that could make the command depend on developer state.
Waits for GET /api/hello to become ready.
Requires GET /agents to equal the complete set of agent identifiers in the copied config.
Enforces the scenario-specific plugin-cache expectation.
Terminates the process on success or failure and includes the bounded log tail in early-exit and timeout errors.

Subprocess output goes to a temporary file rather than an unread pipe because dynamic plugin installation can produce enough output to block startup.

The probe makes no provider request and must not contact a model API.

Dynamic plugin-installation probe

The extracted-archive and default-installer smokes use the same designated plugin project but install it independently. The preferred existing candidate is generic-callback-app: it is a real packaged Python application plugin, is not selected by the default config, has no provider credential or external service requirement, and has no external runtime dependency.

Each smoke must:

Confirm from preinstalled-plugins.json that the candidate is not preinstalled.
Begin with an absent or empty user plugin cache and an isolated home-level pip cache.
Reference the candidate through its packaged path: source in a config copied from the packaged default.
Boot successfully and expose every default agent.
Assert candidate-specific plugin_metadata.json in that smoke's user plugin cache.
Assert that default-config plugins already covered by the preinstalled manifest were not redundantly installed into that cache.

This is a real packaged-runtime pip installation. A test-only plugin, pre-populated cache, copied installation result, or cache shared between the two smokes is not acceptable evidence.

Process-tree ownership

On POSIX, the probe starts the command in a new session. Cleanup signals the entire process group with SIGTERM, waits for bounded graceful exit, and sends SIGKILL to the group if needed.

On Windows, the probe executes archive and installed .cmd wrappers through cmd.exe with CREATE_NEW_PROCESS_GROUP. Cleanup sends CTRL_BREAK_EVENT, waits for bounded graceful exit, and then runs:

taskkill /PID <wrapper-pid> /T /F

when the tree remains alive. The /T descendant cleanup is required because the wrappers start the packaged Python as a child process.

Every cleanup path waits for the wrapper and confirms that the reserved loopback port no longer accepts connections. Failure to stop the process tree fails the smoke with the server log. Windows-specific helper coverage uses a nested .cmd that starts a child Python server before native release smoke is accepted as evidence.

Credentials and normal startup

Starting the CLI, creating its default config, serving local HTTP endpoints, and listing configured agents do not require values for the generated default config's external environment placeholders or exported shell variables.

When ${env:NAME} remains unresolved after process and .env lookup, the config loader records an env_missing warning containing the variable name and config path, substitutes the loader's normal empty value, and continues resolution. The warning itself is nonfatal.

The target warning shape is additive to the existing diagnostic:

{
  "type": "env_missing",
  "severity": "warning",
  "variable": "NAME",
  "path": "providers.example.api_key",
  "detail": "Environment variable 'NAME' is not set"
}

The resolved value is still subject to ordinary structural validation. For example, an empty plugin path may independently produce a fatal plugin_* error. This does not turn the earlier loader warning into a fatal credential or plugin-authentication check.

The probe derives the environment names to remove from the default config rather than maintaining a credential denylist. It does not add generic secret-name filtering or inspect direct environment reads inside plugins. Those nonstandard reads belong to the owning plugin's contract and are not exercised because this probe makes no provider/tool request.

The CLI loads .env values for config resolution in this precedence order:

Existing process environment.
.env beside the active config.
./.crystal/.env.
~/.crystal/.env.
The standard .env discovered from the working directory or its parents.

Earlier .env locations win over later locations. Loaded values are config-resolution inputs and do not need to be exported into the launching shell or copied into the process environment.

API-backed functionality still requires credentials according to the selected provider or tool contract. This specification neither defines a common request-time credential error nor treats provider authentication as distribution-test acceptance. Credential-free startup does not imply anonymous API access or dummy production credentials.

The distribution probes inject no dummy values. Existing startup-only setup changes only when it is directly consolidated with the shared probe. A repository-wide cleanup of provider test environments is not part of this specification.

Source-installation smoke

The source-installation smoke creates a new venv and installs:

pip install -e core/python -e application/python

The install omits development extras and repo-root plugin projects. The test confirms representative development and default-plugin distributions are not already present before booting the venv's generated crystal-lattice entry point.

A successful boot must leave plugin installation metadata in the initially empty config-owned plugin cache. This proves that default-config path: projects and their dependencies traversed the supported dynamic installation path rather than resolving from the parent repository environment.

This scenario intentionally uses package indexes for plugin dependencies. Network or dependency-resolution failure is test evidence, not a reason to reuse the developer environment.

Extracted-archive smoke

The extracted-archive smoke performs an ordinary fresh extraction without calling installer code, then retains all archive-specific checks:

required runtime files;
runtime-package manifest;
--runtime-layout-json;
resolved built-in-plugin, packaged-Python, and preinstalled-manifest paths;
macOS linked-library portability.

After structural validation, the smoke executes the extracted archive launcher directly and applies the common configuration and boot contract. The designated non-preinstalled plugin must appear in this smoke's previously empty user cache. Default-config plugins covered by preinstalled-plugins.json must continue loading from packaged Python without redundant cache installation.

The archive launcher is the server process entry point. A successful installation smoke is not a substitute for this direct boot because final installation paths, outer wrappers, or installer extraction behavior could otherwise hide a direct-archive defect.

Default Bash-installer smoke

The default Bash-installer smoke serves the checked-in public install.sh, the current native archive, and latest.txt from an isolated loopback HTTP endpoint. It executes the same download shape documented for users:

curl -fsSL <loopback-download-base>/install.sh | bash

The smoke supplies the loopback download base and an isolated HOME and USERPROFILE, but it does not set VERSION, INSTALL_ROOT, BIN_DIR, or COMMAND_NAME. This exercises latest-version lookup, target detection, archive download, the default install root, automatic command-directory selection, and the public command name.

Before installation, the smoke places stale target and staging content at the default paths under the isolated home. The installer must:

select or accept the current-platform bundled-Python archive;
replace stale staging and target directories through its staging swap;
preserve archive contents in the final install directory;
create the platform-appropriate installed command;
pass its existing layout and help checks.

After installation, the smoke resolves crystal-lattice by name through the same PATH supplied to the installer. It must not construct or invoke the installed command through a known absolute path. The resolved command then applies the common configuration and boot contract and installs the designated non-preinstalled plugin into this smoke's separate fresh cache.

The Bash installer performs installation-specific layout and help checks, but does not invoke the extracted-archive acceptance smoke. Keeping the two smokes separate prevents shared orchestration or a helper defect from incorrectly satisfying both requirements.

Default PowerShell-installer smoke

The default PowerShell-installer smoke serves the checked-in public install.ps1, the current Windows archive, and latest.txt from its own isolated loopback HTTP endpoint. It executes the native user shape:

irm <loopback-download-base>/install.ps1 | iex

The smoke supplies isolated HOME, USERPROFILE, and LOCALAPPDATA locations. It does not set VERSION, INSTALL_ROOT, BIN_DIR, or COMMAND_NAME. It starts without Git Bash, WSL, or a separately installed Python and exercises:

latest-version and native target resolution;
Windows AMD64 or ARM64 archive download;
ZIP extraction and staging replacement;
default per-user install and command paths;
persisted user PATH behavior when a fallback directory is needed;
.cmd command installation and forwarding;
layout and help checks.

The smoke resolves crystal-lattice by name in the PowerShell process that ran the installer and from a fresh cmd.exe process. The command then applies the common configuration and boot contract and installs the designated non-preinstalled plugin into state that is not shared with the Bash-installer or direct-archive smoke.

The PowerShell installer must be directly executable as a downloaded standalone script. It does not import repository modules, invoke install.sh, or delegate installation to a test-only Python implementation.

Default command-directory selection

An explicit BIN_DIR remains an advanced override for both installers. It selects a command destination but does not implicitly persist that directory to PATH.

Bash installer

Without an override, the Bash installer selects an existing directory already present on PATH so the current supported terminal can resolve crystal-lattice immediately.

Selection is deterministic:

If an existing crystal-lattice command resolves to a suitable writable PATH directory, reuse that directory for an upgrade.
Prefer conventional existing user command directories already on PATH, first ~/.local/bin and then ~/bin.
Otherwise, preserve PATH order and select the first suitable existing writable directory.
If no suitable directory exists, fail before replacing installation state and report how to provide an explicit BIN_DIR. Do not report a successful installation whose command cannot be resolved.

A suitable automatic candidate is an absolute existing directory, writable by the current user, and not an empty/relative PATH entry, the working directory, the system temporary directory itself, or a world-writable command location. Platform path normalization must preserve directories exposed by the supported terminal.

On macOS and Linux, the installer writes the executable POSIX forwarding script. On Windows, installation from Git Bash writes both the POSIX forwarding script and a .cmd forwarding command in the selected directory, so Git Bash and native Windows command lookup can reach the same installed archive. Native tests verify command-name resolution in the supported shell and through cmd.exe.

PowerShell installer

The PowerShell installer first reuses the suitable writable PATH directory of an existing bundled-Python crystal-lattice command. When no such installation exists, it creates:

%LOCALAPPDATA%\Programs\Crystal Lattice\bin

It adds that directory once to the current user's persisted PATH and to the current PowerShell process. It never changes the machine PATH, requires administrator privileges, or edits a PowerShell profile.

The installed command is a .cmd forwarding script so both PowerShell and cmd.exe use ordinary Windows command lookup. A PowerShell session executing the streamed installer can invoke it immediately. A cmd.exe process that launched a child PowerShell cannot receive environment changes from that child; a newly opened cmd.exe sees the persisted user PATH and must resolve the command.

Existing PATH entries are compared case-insensitively after Windows path normalization. Repeated installs do not duplicate the user PATH entry. Installation stops before replacing runtime state when the selected directory cannot be created, written, or persisted.

Manual archive use

Every release exposes the six immutable versioned archives. Manual installation documentation maps the operating system and architecture to the correct filename and launcher:

bin/crystal-lattice for extracted macOS/Linux archives;
Scripts\crystal-lattice.cmd for extracted Windows archives.

Direct extraction and launcher execution require neither public installer. Adding a manually extracted archive to PATH is optional and must not be presented as necessary to validate the archive itself. The existing direct archive smoke is the acceptance owner for this path.

Root npm commands and native release verification

The root distribution-smoke commands are:

npm run test:cli:source-install
npm run test:cli:archive
npm run test:cli:default-install
npm run test:cli:powershell-install

The source-installation, extracted-archive, default Bash-installer, and default PowerShell-installer acceptance implementations are standalone scenario entry points invoked through scripts/repo_run.py. They do not require pytest to orchestrate the real user workflow. Focused pytest coverage imports or invokes lower-level helpers and failure cases, but is not a second acceptance implementation.

The release workflow builds one native archive for each of darwin-amd64, darwin-arm64, linux-amd64, linux-arm64, windows-amd64, and windows-arm64. Every target invokes:

npm run test:cli:archive
npm run test:cli:default-install

Windows targets also invoke:

npm run test:cli:powershell-install

Archive-consuming commands receive that job's archive through CRYSTAL_LATTICE_PYTHON_RUNTIME_ARCHIVE. They execute in separate temporary state and separate process invocations. Each installer step still runs when another smoke fails, provided archive build and location succeeded, so every direct user path produces independent evidence.

Using one immutable archive build is not shared mutable state. Reusing a workspace, home, config, plugin cache, pip cache, install root, or command directory is prohibited.

The commands remain outside ordinary pull-request CI. Runtime may be reduced by sharing archive construction and low-level helpers, but not by replacing one direct smoke with another, skipping required plugin installation, sharing mutable state, or adding orchestration whose complexity outweighs the saved time.

The root local build and install commands use the same action-first, CLI-scoped convention:

npm run build:cli:archive
npm run build:cli:optimized
npm run install:cli:archive
npm run install:cli:dev

build:cli:archive creates the bundled-Python runtime archive. build:cli:optimized creates the separate config-specific PyInstaller archive. install:cli:archive builds and installs the current bundled-Python archive through the public installer. install:cli:dev installs the editable development command.

When CRYSTAL_LATTICE_PYTHON_RUNTIME_ARCHIVE names a matching existing archive, install:cli:archive installs it through the public installer without rebuilding. Without that override, it retains the build-then-install workflow.

The earlier unscoped build:bundle, build:optimized, install:prod:local, and install:dev:local names are removed after tracked consumers migrate. Compatibility aliases are retained only when a maintained external automation consumer is identified.

Release publication and public verification

A release publishes:

install.sh;
install.ps1;
latest.txt;
the six immutable versioned archives in the native target matrix.

Both installers are generated from their application-owned source files with the configured download base URL. Installer and version-marker files retain short cache lifetimes; versioned archives retain immutable caching. Release validation rejects a missing installer or archive before publication and rejects unexpected or duplicate release assets.

After publication and CDN invalidation, bounded endpoint checks wait for the expected version marker and installer bodies, verify all six archive URLs, and execute the public installation path on the available native verification runners. CDN propagation is handled with a bounded retry rather than a fixed unconditional sleep. A successful pre-publication loopback smoke does not substitute for this public endpoint check.

Maintainer script ownership

Application-only CLI maintainer entry points live under:

application/python/scripts/

This directory owns:

isolated managed-runtime and optimized-build wrappers;
local archive and editable-development installers;
the public standalone install.sh and install.ps1;
CLI version maintenance;
distribution-smoke dispatch and scenario entry points;
focused boot, archive-selection, and progress helpers used by those entry points.

Package-local filenames omit a redundant crystal_lattice prefix when the remaining nouns are unambiguous. Names retain concrete distinctions such as runtime_archive, optimized_archive, source_install, and default_install.

The repository root retains the cross-runtime packaging primitive cluster:

scripts/build_cloud_runtime_plugin_bundle.py
scripts/build_crystal_lattice_binary.py
scripts/build_crystal_lattice_python_runtime_package.py
scripts/build_plugin_packaging_support.py
scripts/prepare_cloud_runtime_build_context.py

Cloud-orchestrator source bundles consume this cluster, and runtime-package construction copies selected files into the shipped archive's root scripts/ directory. This explicit cross-package and shipped-runtime contract is different from application maintainer entry-point ownership.

When the runtime-package builder stages application/python, it excludes the package-local maintainer scripts/ directory and generated application cache/ directories. It then explicitly stages only the root packaging scripts required by the shipped runtime. Python package builds likewise do not include the maintainer script directory as importable package code.

Progress output for long operations

Potentially long application-owned Python entry points use a shared progress helper with the visible convention established by release:cli:

[1/5] Creating clean Python environment
[2/5] Installing Core and the CLI
      $ python -m pip install ...

The helper:

prints and flushes a numbered step before each long phase;
prints six-space-indented detail lines;
shows the command before executing a human-readable child process;
normally lets that child inherit stdout and stderr so output is visible immediately without an unread or delayed pipe;
captures output only for short machine-readable values the caller must parse.

The boot helper continues directing server output to a file so dynamic plugin installation cannot fill an unread pipe. Before waiting, the scenario prints that server-log path. While readiness remains pending, it emits bounded periodic status details without dumping the growing log. Early exit and timeout errors continue including a bounded log tail.

The public install.sh and install.ps1 files are each downloaded and executed standalone, so neither can source a repository-only logging library. Each carries equivalent small numbered-step and detail functions inline and streams download, extraction, validation, and command-installation output normally.

Progress output never prints the complete child environment or credential values. Existing process-tree termination, timeout, and port-closure behavior is unchanged.

Simplification and ownership

Each scenario has one acceptance implementation:

the source-installation entry point owns the fresh-venv user workflow;
the archive smoke owns extraction, archive validation, and direct launcher execution;
the default Bash-installer smoke owns install.sh and Bash command-name execution through PATH;
the default PowerShell-installer smoke owns install.ps1, per-user PATH behavior, and native Windows command-name execution.

The scenarios may share focused helpers for archive naming, isolated config creation, plugin-probe configuration, HTTP readiness, process-tree cleanup, and evidence inspection. A shared helper must not install on behalf of the public installer, invoke one scenario from another, reuse scenario state, or turn a test-only path into release evidence.

The public Bash and PowerShell installers are independent user-facing installation authorities. They intentionally implement platform-native shell behavior directly instead of delegating one to the other. The repository-only Python archive installer may remain for an explicit local/offline maintainer use case, but it does not provide public-installer acceptance evidence. Its overlapping staging, command, or PATH behavior must either be consolidated where bootstrap constraints allow or covered by focused parity tests; duplicated full smoke orchestration is removed.

Optimized-bundle separation

The three named smokes use only the bundled-Python runtime layout and its bin/crystal-lattice or Scripts/crystal-lattice.cmd archive-launcher shape.

PyInstaller build code, archive selection, frozen-runtime policy, and test gates remain separate. Existing optimized-bundle startup-only setup may lose dummy values if it is directly consolidated with the common boot probe, but that does not make an optimized archive an input to these scenarios or create a broader provider-test cleanup requirement.