recon_gen._dev.runner

Test layer chain runner — orchestrates the layered test chain with per-run output isolation + timing-diff drift detection.

Invoked via the ./run_tests.sh bash shim at repo root; the shim exec``s into ``python -m recon_gen._dev.runner.

Verbs:

up_to <layer> Run the chain up to and including <layer>.

Layers: unit | db | app2 | deploy | qs_api | qs_browser (pyright folds into unit via the conftest sessionstart gate). unit is variant-independent — it runs ONCE as a prelude before the matrix fans out (Y.2.gate.n), not once per cell. Equivalent forms: up_to=<layer> and up_to <layer>.

up [scope] Boot dependencies. scope = local | aws | all (default). down [scope] Tear down dependencies. scope as above. status [–cost] Show what’s currently running. sweep Clean orphan resources (tagged ManagedBy:recon-gen).

Exit codes:

0 success 1 test failure (one or more layers / variants failed) 2 needs-operator (expired creds, dirty deploy refused, missing cfg, etc.) 3 config / argument error

Substrate: pytest-as-orchestrator + this thin Python wrapper. See docs/audits/y_2_gate_b_0_runner_lang_spike.md for the design lock.

Functions

chain_through(target)	Y.2.gate.c.5 — return the slice of LAYERS from start through target.
cmd_down(args)	Tear down dependencies.
cmd_dump_last_errors(args)	Surface failing-layer assertions + missing-capture warnings from the most-recent run dir.
cmd_pyright(args)	Y.2.gate.b.14 — run pyright directly for fast type-check iteration.
cmd_status(args)	Show what's currently running.
cmd_sweep(args)	Y.2.gate.c.9 — clean orphan QuickSight resources tagged Harness:e2e.
cmd_up(args)	Boot dependencies.
cmd_up_to(args)	Run the test chain up to and including the named layer.
collect_run_outputs(run_dir, layer_results)	Y.2.gate.c.2 — write timings.json + hashes.json after the chain.
compute_drift(current, prior)	Y.2.gate.c.3 — diff per-layer durations between two timings.json blobs.
create_run_id()	Y.2.gate.c.1 — <utc-ts>-<short-sha>[-dirty].
dispatch_layer(layer, run_dir[, options, ...])	Y.2.gate.c.5 — run one layer; return its result.
find_prior_run(current_run_id[, runs_dir])	Y.2.gate.c.3 — pick the best prior run for drift comparison.
generate_db_password()	BX.248 — fresh random password for an ephemeral PG / Oracle container.
is_layer_cached_green(layer, *[, variant])	Y.2.gate.b.8.impl — True iff layer has a green cache marker for the current SHA + variant.
main([argv])
probe_dependencies(layer)	Y.2.gate.c.8 — probe every dep layer needs; return all failures.
prune_old_runs([retain, runs_dir])	Y.2.gate.c.4 — keep the most-recent retain runs; delete the rest.
report_drift(current_run_dir[, runs_dir])	Y.2.gate.c.3 — find prior run, compute drift, print report.
resolve_fuzz_seed_value()	Y.2.gate.c.6.xdist-safety — resolve the seed for this runner invocation.
write_cache_marker(layer, *, duration_seconds)	Y.2.gate.b.8.impl — record that layer passed for the current SHA + variant.

Classes

DriftEntry(layer, current_seconds, prior_seconds)	Y.2.gate.c.3 — one layer's drift vs the prior run.
LayerResult(layer, exit_code, duration_seconds)	Y.2.gate.c.5 — outcome of dispatching one layer.
ProbeFailure(kind, message)	Y.2.gate.c.8 — a single missing or broken dependency.
RunOptions([only, parallel, scenarios, ...])	Y.2.gate.c.7 — operator-supplied flags threaded through dispatch.

class recon_gen._dev.runner.DriftEntry(layer, current_seconds, prior_seconds)

Bases: object

Y.2.gate.c.3 — one layer’s drift vs the prior run.

Parameters:

• layer (str)

• current_seconds (float)

• prior_seconds (float | None)

current_seconds: float

property delta_pct: float | None

property is_drift: bool

layer: str

prior_seconds: float | None

class recon_gen._dev.runner.LayerResult(layer, exit_code, duration_seconds, skipped=False)

Bases: object

Y.2.gate.c.5 — outcome of dispatching one layer.

passed checks the exit code; duration_seconds lands in the timings.json capture (c.2). Stub layers (deploy/api/browser until cfg loading lands per Y.2.gate.h.2) report skipped=True.

Parameters:

• layer (str)

• exit_code (int)

• duration_seconds (float)

• skipped (bool)

duration_seconds: float

exit_code: int

layer: str

property passed: bool

skipped: bool = False

class recon_gen._dev.runner.ProbeFailure(kind, message)

Bases: object

Y.2.gate.c.8 — a single missing or broken dependency.

kind is a stable token (used by tests + telemetry); message is the operator-facing string (b.14.4 refusal pattern — actionable, points at what to do, never auto-invokes interactive flows).

Parameters:

• kind (str)

• message (str)

kind: str

message: str

class recon_gen._dev.runner.RunOptions(only=None, parallel=1, scenarios=None, dialects=None, targets=None, variants=None, fuzz_seeds=1, fuzz_seed_value=None, skip_cheap=False, keep_on_failure=False, trace_all=False, allow_dirty_deploy=False, coverage=False)

Bases: object

Y.2.gate.c.7 — operator-supplied flags threaded through dispatch.

Most flags are scaffolding today (consumed by future c-stage tasks):

• only — pytest -k <expr> filter (active now in c.7).

• parallel — pytest-xdist worker count (active now in c.6; default 1 = serial).

• fuzz_seed_value — the actual fuzz seed VALUE for this run (resolved at cmd_up_to entry: env-override > random-per-invocation; persists across xdist workers in this run via env passthrough — c.6.xdist-safety lock).

• trace_all — Playwright capture every test (env var passthrough; consumed by c.11).

• allow_dirty_deploy — bypass tracked-changes refusal on layer 4+ (active now per b.10).

• coverage — emit per-(variant, layer) .coverage.<variant>.<layer> data files (Y.2.gate.k.1.coverage). When set, every pytest layer (unit/db/app2/ api/browser) runs with --cov=recon_gen --cov-report= and COVERAGE_FILE pointed at <run_dir>/.coverage.<run_dir.name>.<layer>. The CI coverage aggregator job (W.8b) globs coverage-data-* artifacts and ``coverage combine``s them with no logic change. Off by default — opt in for CI; local runs don’t need it.

• scenarios / dialects / targets — variant matrix sub-flag narrowing (m.2.a). All None → compose_matrix returns the 13-cell full default. Any specified → cross-product mode where unspecified axes default per variant.DEFAULT_*.

• variants — triage escape (single/multiple <sc>_<di>_<ta> codes); mutex with the sub-flag axes. None when not pinned.

• fuzz_seeds — kept as count knob for future m.3 wiring (currently unused; fuzz cells inside compose_matrix already fan out via --scenarios=fuzz:N).

• skip_cheap — skip-if-already-green-this-SHA (active when cache lands; b.8).

• keep_on_failure — leave the variant’s ephemeral state up when the chain fails (gate.f.5; consumed in _run_one_variant’s finally — see also gate.l.2 for the lifecycle commands that clean up afterward).

Parameters:

• only (str | None)

• parallel (int)

• scenarios (str | None)

• dialects (str | None)

• targets (str | None)

• variants (str | None)

• fuzz_seeds (int)

• fuzz_seed_value (int | None)

• skip_cheap (bool)

• keep_on_failure (bool)

• trace_all (bool)

• allow_dirty_deploy (bool)

• coverage (bool)

allow_dirty_deploy: bool = False

coverage: bool = False

dialects: str | None = None

fuzz_seed_value: int | None = None

fuzz_seeds: int = 1

keep_on_failure: bool = False

only: str | None = None

parallel: int = 1

scenarios: str | None = None

skip_cheap: bool = False

targets: str | None = None

trace_all: bool = False

variants: str | None = None

recon_gen._dev.runner.chain_through(target)

Y.2.gate.c.5 — return the slice of LAYERS from start through target.

Chain semantics (b.9 LOCKED): cross-layer is sequential. up_to=db means pyright → unit → db; up_to=qs_browser means the full chain.

Return type:

list[str]

Parameters:

target (str)

recon_gen._dev.runner.cmd_down(args)

Tear down dependencies. scope = local (default).

Destructive — requires –yes (Y.2.gate.b.14.3 destructive-op opt-in). Stops the named persistent Oracle containers (PG containers are ephemeral, no action needed). The aws scope was removed in CB.11.a.2 along with RDS Aurora.

Return type:

int

Parameters:

args (Namespace)

recon_gen._dev.runner.cmd_dump_last_errors(args)

Surface failing-layer assertions + missing-capture warnings from the most-recent run dir.

Triage shortcut: instead of find runs/ → grep stdout → reconstruct pytest output, walk the latest runs/<utc-ts>-<sha>/ and dump a structured report — per failing (variant, layer) cell, with the pytest FAILED summary, the assertion text per failing test, and a pointer to (or warning about missing) AA.H.6 capture artifacts.

Surfaces:

• Per failing layer: layer name + exit code + duration + cell-level env (cmd.json fields: RECON_GEN_DEPLOYMENT_NAME, RECON_GEN_FUZZ_SEED, RECON_GEN_TEST_L2_INSTANCE).

• Per failing test: the FAILED ... summary line + the matched ____ <test_id> ____ traceback block from stdout.log (truncated at the next ____ / =====).

• Capture-artifact pointer: $RECON_GEN_RUN_DIR/qs_browser/<sanitized test_id>/ paths, with a loud warning if AA.H.6’s 6 files (screenshot.png / dom.html / console.txt / network.txt / qs_errors.txt / trace.zip) are missing — AA.H.10 wired the hook to all three QS-driver fixtures, so a missing capture is a regression worth flagging.

Use --run <run-id> to pick a specific run (e.g. 20260516T203824Z-914fc4c); default = latest by mtime. Use --variant <name> to narrow to one cell.

Exit code: always EXIT_SUCCESS — this is a triage tool, not a gate. The caller cares about the chain’s exit; this just helps them read it faster.

Return type:

int

Parameters:

args (Namespace)

recon_gen._dev.runner.cmd_pyright(args)

Y.2.gate.b.14 — run pyright directly for fast type-check iteration.

Pyright runs via the unit layer’s conftest sessionstart hook on every up_to=unit invocation, but that pulls in the full ~9s test suite. For tight type-check loops during editing, this verb shells out to .venv/bin/pyright directly.

Stays behind the runner (per b.14.2 “every sub-tool absorbed by the orchestrator”) so an always-allow rule on ./run_tests.sh* covers it — no separate Claude-Code permission for .venv/bin/pyright.

Returns FAILURE on type errors so the chain-style &&-and-continue pattern works (./run_tests.sh pyright && ./run_tests.sh up_to=db).

Return type:

int

Parameters:

args (Namespace)

recon_gen._dev.runner.cmd_status(args)

Show what’s currently running. Local-container only post-CB.12 (the AWS RDS section is gone). –cost flag retained for CLI compat but no longer surfaces hourly estimates (no cloud DB cost).

Return type:

int

Parameters:

args (Namespace)

recon_gen._dev.runner.cmd_sweep(args)

Y.2.gate.c.9 — clean orphan QuickSight resources tagged Harness:e2e.

Replaces scripts/sweep_harness_orphans.py (deletion of the standalone script is Y.2.gate.f.8). Same default: dry-run (collect + print). Pass --yes (or set RECON_GEN_RUNNER_YES=1, matching the destructive-op convention from b.14.3) to actually delete.

Tag set: Harness:e2e — production deploys don’t carry that tag (they wear ManagedBy:recon-gen + optional L2Instance:<prefix>), so this is safe against the production resource graph.

Return type:

int

Parameters:

args (Namespace)

Exit codes:

0 — clean (dry-run completed OR delete completed) 2 — needs operator (AWS creds expired / config not found)

recon_gen._dev.runner.cmd_up(args)

Boot dependencies. scope = local (default).

Local PG / Oracle / DuckDB spin on-demand inside setup_variant per matrix cell — there’s no shared “local cluster” to pre-boot. Reported for symmetry with down.

Return type:

int

Parameters:

args (Namespace)

recon_gen._dev.runner.cmd_up_to(args)

Run the test chain up to and including the named layer.

Pre-flight: probes the named layer’s required deps. On any failure, prints the operator-actionable message and exits NEEDS_OPERATOR.

For layers >= deploy, refuses on tracked-changes dirty state unless –allow-dirty-deploy (or RECON_GEN_RUNNER_YES=1) is set.

Each layer runs ONCE as a single subprocess; pytest-xdist handles in-layer parallelism; session-scoped pg_container_url / oracle_container_url fixtures (tests/conftest.py CB.17.a) provision substrate on first consumption; isolated_cfg gives each (file, worker) its own deployment_name / db_table_prefix suffix (CB.7 + CB.17.c).

Cfg discovery: RECON_GEN_CONFIG env override → _DEFAULT_RUNNER_CFG_CANDIDATES. L2 path comes from cfg.default_l2_instance (h.6) when present; missing → deploy/qs_api/qs_browser layers’ _layer_command returns None and dispatch prints dispatch-skip.

Drops env_access.json per run for env-var liveness diff.

CB.17.d (2026-06-04): replaced the legacy 13-cell variant matrix + _run_one_variant cell loop with this single-pytest-per-layer path. Per-(file, worker) isolation moved into pytest fixtures (isolated_cfg, seeded_cfg). The cell-loop architecture + its 1500+ LOC of supporting machinery deleted in the same pass.

Return type:

int

Parameters:

args (Namespace)

recon_gen._dev.runner.collect_run_outputs(run_dir, layer_results)

Y.2.gate.c.2 — write timings.json + hashes.json after the chain.

timings.json aggregates per-layer wall-clock durations + per-test timings (for layers that ran pytest, via the conftest hook). hashes.json is a placeholder — populated by future tests/code as part of c.13 (hash-lock collapses into the runs dir).

Single-source-of-drift principle (audit §7.9): both files live alongside each other under runs/<run-id>/; c.3’s drift-diff reads them together against the prior run.

Return type:

None

Parameters:

• run_dir (Path)

• layer_results (Sequence[LayerResult])

recon_gen._dev.runner.compute_drift(current, prior)

Y.2.gate.c.3 — diff per-layer durations between two timings.json blobs.

Only emits entries for layers present in current (not interested in layers that ran in prior but not now — that’s chain-narrowing, not drift).

Return type:

list[DriftEntry]

Parameters:

• current (dict[str, Any])

• prior (dict[str, Any])

recon_gen._dev.runner.create_run_id()

Y.2.gate.c.1 — <utc-ts>-<short-sha>[-dirty].

Stable, sortable, includes the dirty suffix so cross-run timing diffs don’t compare a clean run against a dirty one and claim spurious drift.

Return type:

str

recon_gen._dev.runner.dispatch_layer(layer, run_dir, options=None, *, variant_env=None, terminal_prefix='')

Y.2.gate.c.5 — run one layer; return its result.

Stub layers return a skipped=True LayerResult with exit_code=0 so the chain doesn’t break — the deferred work is c.5+ follow-up, not a runner bug. Stubs print a clear dispatch-skip line so the operator knows.

Y.2.gate.b.2.impl — variant_env (e.g., {"RECON_GEN_DEMO_DATABASE_URL": "<container-url>"}) gets merged into the subprocess env so the variant’s resources (Docker container URL etc.) are visible to pytest fixtures + cfg loaders inside the subprocess.

Per-layer subprocess capture (Y.2.gate.b.2.impl.oracle followup): every dispatch persists four artifacts under <run_dir>/<layer>/:

• cmd.json — the input: cmd argv, cwd, env-overrides (deltas from inherited os.environ — the layer-specific keys + variant env, not the noisy full environ). Written before the subprocess starts; re-written after with exit_code + duration_seconds.

• stdout.log — subprocess stdout, also teed to operator’s terminal in real time.

• stderr.log — subprocess stderr, also teed to terminal.

Streams use a per-stream daemon-thread tee so a full pipe buffer on one stream can’t deadlock the other. The operator sees live output same as before; failures leave a complete trail in the run dir for post-mortem (CI artifact upload, hands-off run review).

Return type:

LayerResult

Parameters:

• layer (str)

• run_dir (Path)

• options (RunOptions | None)

• variant_env (dict[str, str] | None)

• terminal_prefix (str)

recon_gen._dev.runner.find_prior_run(current_run_id, runs_dir=None)

Y.2.gate.c.3 — pick the best prior run for drift comparison.

Priority: (1) most-recent prior with the SAME SHA (closest signal — same code, lets us see real timing drift); (2) most-recent prior overall (good enough when no SHA match). Returns None if no prior runs exist.

Return type:

Path | None

Parameters:

• current_run_id (str)

• runs_dir (Path | None)

recon_gen._dev.runner.generate_db_password()

BX.248 — fresh random password for an ephemeral PG / Oracle container.

Returns 28 hex chars (secrets.token_hex(14)). Sized to fit inside Oracle’s 30-byte quoted-identifier limit: ALTER USER … IDENTIFIED BY “<pwd>” treats the quoted form as a quoted identifier, capping the password body at 30 bytes (CE.4-followup; pre-fix token_hex(16) = 32 chars tripped ORA-00972 “identifier is too long”, the password reset silently failed, and every login attempt counted toward FAILED_LOGIN_ATTEMPTS → ORA-28000 “account is locked”). 28 chars leaves 2 bytes of headroom and still satisfies Oracle 19c’s “alphanumeric + ≥8 chars + ≥1 letter + ≥1 digit” rule by construction (hex is mixed letter+digit).

Pre-BX.248 the runner pinned a static ORACLE_REUSE_PASSWORD constant in source — that string leaked DB credentials to anyone with repo access AND let the same source-disclosed password reach the home-firewall-exposed hotchkiss.io:5433/1522 forwards. Generating per-invocation closes that hole.

112 bits of entropy — strong enough for an ephemeral container credential.

Return type:

str

recon_gen._dev.runner.is_layer_cached_green(layer, *, variant='default')

Y.2.gate.b.8.impl — True iff layer has a green cache marker for the current SHA + variant. Used by cmd_up_to when --skip-cheap is set to short-circuit re-runs.

Return type:

bool

Parameters:

• layer (str)

• variant (str)

recon_gen._dev.runner.main(argv=None)

Return type:

int

Parameters:

argv (Sequence[str] | None)

recon_gen._dev.runner.probe_dependencies(layer)

Y.2.gate.c.8 — probe every dep layer needs; return all failures.

Probes run sequentially (cheap; few seconds total) and gather all failures so the operator sees everything missing in one pass instead of fixing one, re-running, hitting the next, etc. No state file (LOCKED §7.12) — each invocation re-probes.

Return type:

list[ProbeFailure]

Parameters:

layer (str)

recon_gen._dev.runner.prune_old_runs(retain=20, runs_dir=None)

Y.2.gate.c.4 — keep the most-recent retain runs; delete the rest.

“Most recent” = mtime (robust to dirs an operator touches). Only directories matching _RUN_ID_PATTERN are candidates — defensive: don’t accidentally nuke unrelated files an operator parked under runs/.

Returns the list of deleted paths (for tests / future telemetry). Idempotent: missing runs_dir → no-op; <retain runs → no-op.

Concurrency-safe: when multi-cell fan-out runs the unit suite in parallel and each unit subprocess itself calls runner.main(…) (e.g., test_up_to_creates_run_dir), sibling workers can race on the same runs/ dir. shutil.rmtree(old) could see a path the sibling already deleted; FileNotFoundError is benign — the work is done. stat() failures during the listing pass are similarly benign (entry vanished mid-iter); skip and move on.

#741 — tests no longer pollute the real runs/: tests/ conftest.py::pytest_configure redirects RUNS_DIR to a session tmp dir at pytest startup. So under matrix fan-out the 200+ in-process runner.main calls all prune within the session-tmp tree — no operator-runs/ contention, no need for an xdist-only short-circuit guard.

Return type:

list[Path]

Parameters:

• retain (int)

• runs_dir (Path | None)

recon_gen._dev.runner.report_drift(current_run_dir, runs_dir=None)

Y.2.gate.c.3 — find prior run, compute drift, print report.

Return type:

None

Parameters:

• current_run_dir (Path)

• runs_dir (Path | None)

Output shape:

drift: comparing against <prior_run_id> drift: pyright 1.81s (was 1.85s, -2.2%) drift: unit 15.20s (was 10.42s, +45.9%) drift: db 24.10s (was 12.30s, +96.0%) ⚠

The ⚠ marker fires on abs(delta_pct) >= DRIFT_THRESHOLD_PCT (±50%); same shape as hash-locked seed data — a sudden delta is signal, not noise.

recon_gen._dev.runner.resolve_fuzz_seed_value()

Y.2.gate.c.6.xdist-safety — resolve the seed for this runner invocation.

Priority: RECON_GEN_FUZZ_SEED env (operator pin for failure repro) > random per session (secrets.randbits(32)). Per audit §7.11 (LOCKED): default = 1 random seed per run; cumulative coverage emerges across many runs. The seed is pinned across xdist workers within a single run so parametrize collection is deterministic (otherwise each worker rolls its own seed → collection diverges → Different tests were collected error).

Return type:

int

recon_gen._dev.runner.write_cache_marker(layer, *, duration_seconds, variant='default')

Y.2.gate.b.8.impl — record that layer passed for the current SHA + variant. No-op if not in a git repo (_short_sha returns ‘nogit’) so direct pytest invocations don’t pollute the cache.

Return type:

None

Parameters:

• layer (str)

• duration_seconds (float)

• variant (str)