dsh-reliability-lab

Deterministic fault injection, Replay LLM testing, durable-session assertions, and machine-readable integration contracts for DeepSeek Harness agents.

Status: 0.6.0-alpha.0. Offline tool fixtures, the opt-in Cordis wrapper, official @deepseek-ai/dsh-llm-replay sidecars, isolated dsh --profile headless orchestration, deliberate process crash/restart, cold interrupted-turn recovery, session JSONL auditing, nested-agent graph validation, privacy-preserving side-effect ledgers, JSON evidence, and JUnit XML, observed cross-version matrices, self-contained HTML evidence timelines, and a fingerprinted Scenario v1 / Report v5 / Matrix v1 contract pack are implemented.

Source releases are published from GitHub. Prerelease npm builds use the next distribution tag only after the real-DSH publication workflow passes on Node 22 and 24:

npm install dsh-reliability-lab@next

Until the first next publication is visible in the registry, install from a GitHub release or a local checkout. No alpha release is assigned to npm's default latest tag.

Why this exists

Agent evaluations usually score answer quality. This project tests operational behavior:

• Does the agent recover after a tool timeout, executor exception, or transient LLM failure?
• Does a headless run finish inside a hard process and output bound?
• After an uncatchable process kill, does a fresh Harness boot durably close the interrupted turn?
• Is the persisted event sequence contiguous and every turn durably closed?
• Do delegated sessions preserve parent lineage and a consistent subagent depth?
• Did retries execute the same externally visible side effect more than once?
• Can the same failure be reproduced without an API key in local runs and CI?
• Can CI consume the result as both detailed JSON and standard JUnit XML?
• Which exact Harness, Node, operating-system, architecture, and scenario combinations have direct evidence—and which combinations remain untested?
• Can a reviewer inspect durable event ordering without loading a raw session payload into a tool?
• Can a community adapter discover and validate exact contract versions without copying internal validation logic?

Keyless offline quick start

The offline lane has no runtime dependencies and needs neither DSH nor an API key.

node src/cli.js validate scenarios/tool-timeout.yaml
node src/cli.js run scenarios/tool-timeout.yaml \
  --output .dsh-rel/tool-timeout.json \
  --junit .dsh-rel/tool-timeout.xml \
  --html .dsh-rel/tool-timeout.html
node src/cli.js run scenarios/session-graph-idempotency.yaml --no-write
node --test
node scripts/build.mjs

run exits with status 1 when an assertion fails. JSON reports default to .dsh-rel/results/.

Real DSH Headless quick start

The real-host lane runs the official process contract:

dsh --profile headless --patch <generated-overlay> "<task>"

Prepare a reusable profile template once. The local package contributes its bundle row; the official replay package is installed as a plain profile dependency for the generated overlay.

pnpm run build
export DSH_HOME="$PWD/.dsh-rel/headless-template"
dsh plugin --profile headless add "$PWD" @deepseek-ai/dsh-llm-replay
unset DSH_HOME

Then run the included keyless retry scenario:

DSH_REL_TESTED_VERSION="record-the-exact-installed-version"
node src/cli.js run scenarios/headless-llm-retry.yaml \
  --harness-version "$DSH_REL_TESTED_VERSION" \
  --output .dsh-rel/headless-llm-retry.json \
  --junit .dsh-rel/headless-llm-retry.xml \
  --html .dsh-rel/headless-llm-retry.html

Or run the two-boot crash-recovery scenario against the same prepared template:

node src/cli.js run scenarios/crash-recovery.yaml \
  --harness-version "$DSH_REL_TESTED_VERSION" \
  --junit .dsh-rel/crash-recovery.xml

The first boot enters an official Replay hang, waits for its ready marker, and receives a direct SIGKILL (or the platform's forceful child termination). After inspecting the still-open durable turn, the runner starts a second process against the same isolated DSH_HOME and session root. A small dsh-reliability-lab/recovery-probe plugin calls sessionPersistence.list() and then load(id) for every session, causing the Harness persistence backend itself to commit cold repair. The runner audits the on-disk log again and proves the before/after transition.

This deliberately uses load(id), not inspect(id): the upstream persistence contract defines cold load as the operation that commits synthetic closing events, while inspect is non-mutating. The probe follows the official Headless driver's lifecycle shape by awaiting Loader settlement and requesting shutdown through launcher-owned appExit.

Each invocation copies runtime.homeTemplate into a fresh run directory, creates a disposable workspace, forces raw/unpacked session persistence, disables native model adapters when Replay is enabled, and captures stdout/stderr without a shell. The report prints its runRoot, which contains the exact patch, normalized scenario, Replay override, DSH home, workspace, and session artifacts.

The current machine must provide dsh on PATH. Use runtime.command and runtime.commandArgs when testing another launcher build.

--harness-version is an evidence label, not an auto-detected package version. The runner records its source as declared; DSH_REL_HARNESS_VERSION is the lower-priority CI environment fallback and is recorded as environment. If neither is present, the report says unknown and the matrix will not call that lane verified. Pin and independently record the package or checkout used by each CI lane. The upstream project is still a developer preview, so this project intentionally does not hard-code a list of supposedly compatible releases.

Scenario v1

Tool failure

version: 1
name: recover after first tool timeout
seed: 42
harnessVersion: optional-exact-version-label

faults:
  - id: first-write-timeout
    target: tool
    tool: fs_write
    occurrence: 1
    effect:
      type: timeout
      durationMs: 1

fixture:
  calls:
    - tool: fs_write
    - tool: fs_write
      result:
        text: completed

assertions:
  - type: fault.injected
    count: 1
  - type: tool.callCount
    tool: fs_write
    equals: 2
  - type: result.errorCount
    count: 1

Replay LLM failure with preserved retry

version: 1
name: recover after transient model failure

faults:
  - id: first-rate-limit
    target: llm
    occurrence: 1
    effect:
      type: throw
      code: RATE_LIMIT
      message: injected rate limit

runtime:
  mode: headless
  task: Answer with the recorded response.
  homeTemplate: ../.dsh-rel/headless-template
  permissionMode: read-only

replay:
  fixture: ./fixtures/replay-success.jsonl
  mode: insert
  provider: deepseek-official
  model: deepseek-v4-flash

assertions:
  - type: replay.patchCount
    count: 1
  - type: process.exitCode
    equals: 0
  - type: session.eventCount
    eventType: llm/retry
    equals: 1
  - type: session.turnReasonCount
    reason: completed
    count: 1

Replay modes:

Mode	Output	Behavior
insert	Bare official ReplayEntry[]	Inserts failures at live call positions and preserves recorded calls as retry results
patch	Official { patches: [{ at, entry }] }	Replaces the recorded call at each zero-based position exactly as the upstream sidecar contract specifies

throw supports optional prefix chunks, message, and code. hang supports an optional safe relative readyFile; during a hosted run it is confined beneath the run root.

insert derives ordinary assistant/chunk rows from the fixture and accepts complete or projected (seq/time-scrubbed) envelopes. Packed chunk rows are intentionally rejected by this dependency-free lane; record the fixture with packChunks: false, or use patch mode with the upstream loader.

The parser accepts JSON plus the documented block-style YAML subset. Unsupported YAML features fail loudly rather than being interpreted ambiguously.

Two-boot crash recovery

version: 1
name: cold recovery closes a killed LLM turn

faults:
  - id: first-model-hang
    target: llm
    occurrence: 1
    effect:
      type: hang
      readyFile: markers/first-model-hang.ready

runtime:
  mode: crash-recovery
  task: Enter the deterministic model hang and wait.
  homeTemplate: ../.dsh-rel/headless-template
  crashFaultId: first-model-hang
  crashDelayMs: 250
  timeoutMs: 60000
  recoveryTimeoutMs: 30000

replay:
  fixture: ./fixtures/replay-success.jsonl
  mode: insert

assertions:
  - type: crash.readyObserved
    equals: true
  - type: crash.openTurnCount
    equals: 1
  - type: recovery.loadedSessionCount
    equals: 1
  - type: session.openTurnCount
    equals: 0
  - type: session.interruptedTurnCount
    equals: 1

crashFaultId defaults to the first LLM hang rule. crashDelayMs gives the append-only backend a short deterministic durability window after the adapter publishes its marker; it defaults to 250 ms and is bounded to 10 seconds. recoveryTimeoutMs bounds the independent second boot.

Nested-agent and side-effect audit

An audit scenario reads an existing raw session directory without launching DSH, calling a model, or invoking a tool:

version: 1
name: audit nested agents and release idempotency

audit:
  sessionRoot: ./fixtures/nested-sessions

sideEffects:
  - id: release-publish
    tool: release_publish
    source: all
    keyPaths:
      - /releaseId

assertions:
  - type: session.graph.subagentCount
    equals: 2
  - type: session.graph.maxDepth
    equals: 2
  - type: session.graph.valid
    equals: true
  - type: sideEffect.duplicateAttemptCount
    effect: release-publish
    equals: 0
  - type: sideEffect.atMostOnce
    effect: release-publish
    equals: true

The graph uses each persisted SessionHeader.id, parentSession, origin, and delegationDepth. Duplicate IDs, missing parents, lineage cycles, malformed header fields, root depths other than zero, and inconsistent subagent depths make session.graph.valid false.

A side-effect rule observes matching durable tool/call plus tool/result events and Code Mode tool/code-dispatch-start plus tool/code-dispatch events. source can be native, code, or all. keyPaths are JSON Pointers selecting the operation's idempotency key; an empty list uses the whole argument value. Selected values are canonically encoded and SHA-256 hashed. The ledger portion of the report never copies raw tool arguments or selected values.

Each ledger records succeeded, failed, and unsettled executions. atMostOnce rejects any duplicate attempt or unkeyed execution; successfulAtMostOnce permits failed retries but rejects multiple successful executions for one key. Both fail closed when arguments are malformed or a configured key path is missing. The same sideEffects block can be attached to a hosted Headless or crash-recovery scenario. Both at-most-once predicates are vacuously true when no matching call exists, so pair them with sideEffect.executionCount when at least one execution is required.

Session evidence

Headless runs generate a final overlay that sets the JSONL backend to compression: none and packChunks: false. The dependency-free auditor verifies:

• events[index].seq === index across every session;
• turn and step enclosure;
• open and interrupted turn counts;
• event-type and turn-reason counts;
• parent/child edges, subagent count, delegation depth, orphan parents, duplicate IDs, and cycles;
• native and Code Mode side-effect attempts correlated with their durable results;
• malformed or unexpectedly packed rows.

Headless scenarios receive implicit safety assertions unless they override the same type:

• process did not time out;
• process exited with 0;
• at least one session exists;
• all sequences are contiguous;
• no turn remains open;
• the persisted session graph is valid.

Crash-recovery scenarios instead receive implicit transition assertions unless overridden:

• the hang marker was observed and the crash process has no numeric exit code;
• at least one turn is open before recovery;
• the recovery process exits with 0 before its bound;
• the probe cold-loads at least one session;
• final sequences are contiguous, no turn remains open, at least one turn is interrupted, and the persisted session graph is valid.

Available hosted assertions include process.exitCode, process.timedOut, session.count, session.sequenceContiguous, session.openTurnCount, session.interruptedTurnCount, session.turnReasonCount, session.eventCount, session.graph.rootCount, session.graph.edgeCount, session.graph.subagentCount, session.graph.orphanCount, session.graph.cycleCount, session.graph.maxDepth, session.graph.depthMismatchCount, session.graph.duplicateIdCount, session.graph.valid, and session.graph.depthConsistent. Configured ledgers expose sideEffect.executionCount, sideEffect.succeededCount, sideEffect.failedCount, sideEffect.unsettledCount, sideEffect.uniqueKeyCount, sideEffect.missingKeyCount, sideEffect.invalidArgumentCount, sideEffect.duplicateAttemptCount, sideEffect.duplicateSuccessCount, sideEffect.atMostOnce, and sideEffect.successfulAtMostOnce; each requires effect: <sideEffects.id>. Two-boot runs additionally expose crash.readyObserved, crash.exitCode, crash.openTurnCount, recovery.exitCode, recovery.timedOut, and recovery.loadedSessionCount.

Audit a standalone raw artifact with:

node src/cli.js inspect-session path/to/session.jsonl

HTML evidence timeline

Generate HTML during a run or from a saved JSON report:

node src/cli.js run scenarios/session-graph-idempotency.yaml \
  --output .dsh-rel/session-audit.json \
  --html .dsh-rel/session-audit.html
node src/cli.js html .dsh-rel/session-audit.json \
  --output .dsh-rel/session-audit-review.html

The output is one offline, script-free file with inline CSS and a restrictive Content Security Policy. It shows report/runtime metadata, assertions, process phases, session graph, side-effect ledgers, infrastructure errors, and a cross-session durable-event timeline. Timeline rows project only event type, sequence, time, session lineage, turn, and step—they do not copy event payloads. The final expandable raw-JSON section still contains everything already present in the report; use the library API reportToHtml(report, { includeRawJson: false }) when that section is inappropriate.

Observed compatibility matrix

Aggregate reports produced by separately pinned CI jobs:

node src/cli.js matrix evidence/rc5/*.json evidence/rc6/*.json \
  --output .dsh-rel/compatibility/matrix.json \
  --html .dsh-rel/compatibility/matrix.html \
  --require-complete

A lane is the exact tuple harnessVersion × nodeVersion × platform × arch; scenario names become columns. Cells are passed, failed, mixed, or missing. A lane is:

• failed when any supplied cell failed or has contradictory pass/fail evidence;
• partial when at least one scenario column has no report for that lane;
• unattributed when all columns pass but the Harness version, version source, report producer, or producer version provenance is unknown;
• passed only when every supplied scenario has passing, version-attributed evidence.

verified: true therefore means complete passing coverage for exactly the supplied scenario set, not ecosystem-wide compatibility, automatic version detection, or proof about untested behavior. Missing cells remain missing and are never inferred from adjacent versions. The command exits 1 for failed lanes; --require-complete also makes partial and unattributed lanes fail CI.

Report schema v5 adds the environment provenance block and the payload-free sessions.eventTimeline projection. The matrix reader can ingest older report objects, but reports without v5 environment metadata collapse into an unknown runtime lane and cannot become verified. Compatibility matrix documents use their own independently versioned schema, currently matrixVersion: 1.

Machine-readable contract pack

Export the exact schemas and adapter protocol metadata consumed by this build, then validate JSON artifacts in CI:

node src/cli.js contracts --output .dsh-rel/contracts.json
node src/cli.js verify .dsh-rel/tool-timeout.json --kind report
node src/cli.js verify .dsh-rel/compatibility/matrix.json

The same surface is available to plugins without importing internal modules:

import {
  CONTRACT_PACK,
  createContractPack,
  validateContractDocument,
} from 'dsh-reliability-lab/contract-pack'

const { kind, document } = validateContractDocument(candidate, 'auto', 'adapter input')
console.log(kind, document, CONTRACT_PACK.fingerprint.value)

Contract pack v1 contains Draft 2020-12 JSON Schemas for Scenario v1, Evidence Report v5, and Compatibility Matrix v1; adapter protocol version, report statuses, official producer sources, capability identifiers, and compatibility policy; and a deterministic SHA-256 fingerprint over the canonical pack body. createContractPack() returns a detached copy while CONTRACT_PACK is deeply immutable.

The schemas describe the portable envelope. validateContractDocument() additionally applies Scenario defaults and cross-field rules, report provenance checks, matrix count/status consistency, and automatic kind detection. Use validate for YAML Scenario input; verify intentionally accepts JSON contract documents so adapters and CI exchange an unambiguous representation.

DSH plugin mode

The package ships cordis.patch.yml and exports a normal Cordis plugin. Tool injection is inert until explicitly enabled in a disposable test profile:

- id: dsh-reliability-lab
  config:
    enabled: true
    scenarioPath: ./scenarios/tool-timeout.yaml
    allowTools: [fs_write]
    reportRoot: ./.dsh-rel
    reportPath: ./plugin-report.json

The plugin wraps the official tools/execute waterfall and never patches Harness core code. Wildcard faults require allowWildcard: true; report paths cannot escape reportRoot.

Commands

dsh-rel contracts [--output <contract-pack.json>]
dsh-rel verify <document.json> [--kind <auto|scenario|report|matrix>]
dsh-rel validate <scenario.yaml>
dsh-rel run <scenario.yaml> [--output <report.json>] [--junit <report.xml>]
            [--html <report.html>] [--harness-version <version>]
            [--run-root <empty-directory>] [--no-write]
dsh-rel html <report.json> [--output <report.html>]
dsh-rel matrix <report.json>... [--output <matrix.json>] [--html <matrix.html>]
               [--require-complete]
dsh-rel replay-override <scenario.yaml> [--output <replay.override.json>]
dsh-rel inspect-session <session.jsonl> [--output <inspection.json>]
dsh-rel --version

Exit codes:

Code	Meaning
0	Scenario passed or validation succeeded
1	Assertions or matrix evidence failed; with --require-complete, matrix coverage is incomplete
2	Scenario or machine-readable contract validation error
3	CLI, process-launch, or runtime infrastructure error

Roadmap

• 0.1: deterministic tool timeout/throw injection, assertions, JSON report — complete
• 0.2: Replay LLM faults, isolated Headless runner, session audit, JUnit — complete
• 0.3: deliberate process kill/restart and interrupted-turn recovery across two boots — complete
• 0.4: nested-agent session graphs and side-effect/idempotency assertions — complete
• 0.5: cross-version compatibility matrix and HTML evidence timeline — complete
• 0.6: fingerprinted contract pack, runtime contract verification, and adapter protocol — current
• 1.0: stable contracts and maintained ecosystem adapters

Safety

Read SECURITY.md before running hosted fault injection. Use disposable workspaces, especially for tools with external side effects.

License

MIT