Skip to content
dsh marketplace

dsh-judge-worker

Verified

dsh-judge-worker · v0.1.0 · MIT

DSH plugin: dual-model orchestration for code tasks — a high-capability judge writes Task Contracts and reviews evidence, a cheap worker implements in a git worktree, and the plugin verifies with real commands.

Install

dsh plugin add dsh-judge-worker

Confirm the layer applied with dsh --profile default --dump-config — see the install guide.

Source

Tags

Readme

dsh-judge-worker

A DSH plugin that runs one code task through two models with different jobs:

User intent
    ↓
Judge   — a high-capability model: decides WHAT should be done, and whether it was done
    ↓  Task Contract (scope, invariants, acceptance, budget)
Worker  — a cheap coding model: implements inside a disposable git worktree
    ↓
Verifier — the plugin itself: runs the contract's commands, reads the real diff
    ↓
Judge   — reviews measured reality and decides: PASS / RETRY / RECONTRACT / SPLIT / ESCALATE
    ↓
Report  — what changed, what was measured, what it cost, where the artifacts are

The judge decides, the worker produces, the verifier measures, the plugin enforces.

The point of the plugin is not the prompt. It is that the boundaries are enforced by code: a worker cannot widen its own scope, cannot accept its own work, and cannot hide behind a claim — because the plugin runs the commands and reads the diff itself.

Install

dsh plugin --profile web add dsh-judge-worker     # or: add /path/to/this/checkout

The bundle patch mounts the plugin row; /judge-worker appears in the slash-command menu of every session. Adding a bundle to a profile changes package.json, which the live patch watcher does not reload — restart dsh web once after installing.

Configure the two models

Both models are configuration, never hard-coded. With no configuration the plugin follows the session's own model (and the deployment default for a fresh session), which is the cheapest way to try it:

# ~/.dsh/profiles/web/cordis.patch.yml
- id: judge-worker
  config:
    models:
      judge:                            # the expensive one: plans and reviews
        provider: gpt
        model: gpt-6-astra
        reasoningEffort: high
        timeoutSeconds: 900
      worker:                           # the cheap one: implements and repairs
        provider: isrc
        model: DeepSeek-V4.1-Flash
        reasoningEffort: off
    verification:
      repositoryDefaults:               # always run, a contract may add but never remove
        - npm run typecheck
        - npm test

reasoningEffort is adapter-owned and optional: it is passed through to the provider untouched, and an effort the model does not advertise is refused before any request is sent. A judge benefits from a high effort; a worker usually does not — the plugin measures the difference in the report's cost section.

A patch replaces the row's whole config, so restate every key you want to keep. Per-run overrides: --judge <model>, --worker <model>, --judge-provider <id>, --worker-provider <id>.

Use

/judge-worker fix the refresh-token flow so a client can log in again after expiry
/judge-worker --plan-only <task>      # show the Task Contract the judge would use, then stop
/judge-worker --dry-run <task>        # plan and validate the setup without changing any file
/judge-worker resume <run-id>         # continue a run that was interrupted
/judge-worker runs                    # list this repository's runs

Options: --judge <model>, --worker <model>, --judge-provider <id>, --worker-provider <id>, --max-attempts <n>, --keep-worktree, and -- to protect task text that starts with a dash.

Output:

Run: jw_20260919T141826_5y7mi1
Status: PASS
Judge calls: 2 · Worker calls: 2
Artifacts: <repo>/.dsh/judge-worker/runs/jw_20260919T141826_5y7mi1
...

What the plugin enforces (not what it asks for)

Guarantee How it is true
No contract, no execution a worker only ever starts from a stored, validated ContractRevision
A worker cannot widen its scope every write passes the scope guard; the real git diff is re-checked afterwards
A worker cannot accept work the only path to ACCEPTED is a validated Judge PASS over a passing verifier
Machine facts beat model claims the verifier runs the contract's commands and reads the diff itself
Cheap failures stay cheap a failing test routes to a worker repair, never to the Judge
Scope violations are never auto-repaired SCOPE_VIOLATION routes straight to review
No unbounded loop every loop consults a budget guard before each costly step
A crash loses nothing state is persisted after every transition; resume re-enters the state machine
The Judge never sees the transcript review input is facts and references; content is fetched on request, bounded

The worker runs as a plugin-driven tool loop, not as a subagent: the worker model emits one JSON action per step (read_file, write_file, edit_file, search, list_dir, run_command, finish) and the plugin executes it. That is what puts the scope guard and the shell policy in front of every write and every command, and it is why a worker cannot escape its sandbox by asking nicely.

What a run leaves behind

<repo>/.dsh/judge-worker/runs/<run-id>/
├── intent.json            plan.json            state.json            decision.json
├── contract-r1.json       contract-r2.json     probe-facts.md        final-report.md
├── evidence/manifest.json evidence/probe-*.json
└── attempts/<task-id>/<n>/
    ├── worker-result.json worker-transcript.json verification.json
    ├── diff.patch         diff.stat.json
    └── logs/<command>.stdout.txt  logs/<command>.stderr.txt

Worktrees live in <repo>/.dsh-worktrees/<run-id>/<task-id> and are kept by default — on failure so you can inspect what went wrong, and on success because the accepted change lives there. The report ends with the exact git apply command for every accepted task. The plugin never commits, merges, or pushes: deciding how an accepted change enters your branch is yours.

Configuration reference

Every key is optional; defaults shown.

- id: judge-worker
  config:
    models:
      judge:  { provider: <id>, model: <id>, temperature: 0.1, maxTokens: 8192, reasoningEffort: <id>, timeoutSeconds: 600 }
      worker: { provider: <id>, model: <id>, temperature: 0.1, maxTokens: 8192, reasoningEffort: <id>, timeoutSeconds: 600 }
    execution:
      maxWorkerAttempts: 3          # worker attempts per task before the judge is asked
      maxRecontracts: 2             # how often the judge may rewrite a contract
      maxTotalTasks: 12             # hard cap per run, including SPLIT-created tasks
      parallelism: 1                # MVP: one task at a time
      maxJudgeCallsPerTask: 4
      maxWorkerCallsPerTask: 5
      maxWorkerStepsPerAttempt: 24  # model steps inside one worker attempt
      maxWorkerHistoryChars: 60000  # step-history budget in the worker's context
      maxPlannerProbes: 2           # read-only probe rounds before a contract is required
      commandBackend: host          # `host` reuses the DSH shell sandbox; `local` runs directly
      integrationVerification: true # combine accepted patches and re-run the repository checks
    sandbox:
      worktreeRoot: <repo>/.dsh-worktrees
      keepWorktreeOnFailure: true
      keepWorktreeOnSuccess: true   # the accepted checkout is the deliverable; nothing is merged
      allowNetwork: false           # network-reaching commands are refused while false
      extraDenyCommands: []         # extra regular expressions refused in worker commands
      allowedCommands: []           # regular expressions overriding the built-in refusals
      protectedPaths: []            # extra paths no contract can open
    verification:
      commandTimeoutSeconds: 600
      maxLogBytes: 200000
      failOnScopeViolation: true
      runCommandsWhenScopeFails: false
      repositoryDefaults: []        # e.g. ["npm run typecheck", "npm test"]
    review:
      includeFullDiff: false        # the judge gets a bounded excerpt and can fetch more
      includeWorkerTranscript: false
      maxDiffExcerptLines: 160
    evidence: { maxFetchLines: 400, maxFetchBytes: 65536 }
    storage:  { root: <repo>/.dsh/judge-worker, writeFinalReport: true }
    prompts:  { judgePlan: "", judgeReview: "", workerExecute: "", workerRepair: "", probe: "" }
    logging:  { verbose: false }

Protected paths (VCS metadata, dependency trees, .env, keys, credential files) can never be opened by a contract, and a contract that claims the whole repository is rejected before a worker runs.

Development

npm install
npm run verify        # typecheck + 88 unit/integration tests + build
npm run build         # tsc → lib/, plus the prompt documents

The tests run the real controller against a real git repository with real command execution; only the two models are scripted (tests/helpers/scripted-model.ts). The suite covers the design's acceptance cases: a cheap repair loop that never consults the judge, a scope violation produced through the shell, a blocked worker whose contract the judge widens, budget exhaustion, resume after a crash, judge context isolation, and the plugin entry point driven through the host-facing surface.

Two of them are regressions from real production runs: a planner that returned an absolute path in scope.allow (now rejected and handed back for correction instead of aborting the run), and a probe whose absolute read paths reached the scope guard (now sanitized — and the facts it gathers now reach the next planning round).

notes/ holds the verified API references this plugin was built against (the DSH plugin surface: commands, ctx.llm, tools/presets, subagents, exec/fs/config), each with the exact declarations and file paths they were read from. They are working notes, not part of the package.

Architecture notes:

  • src/adapters/ is the only place that knows about DSH. Everything else depends on the ports in src/adapters/types.ts, which is why the pipeline is testable without a host.
  • src/contracts/, src/state/, src/sandbox/, and src/verifier/ contain no model calls.
  • src/prompts/*.md is the policy layer; each document can be replaced wholesale through prompts.

Releasing

scripts/release.sh 0.1.1     # verify → bump → commit → tag → push; CI publishes

Publishing runs in GitHub Actions (.github/workflows/release.yml) through npm trusted publishing: CI authenticates with a short-lived OIDC token, so no long-lived npm credential exists on any machine, and every published version carries a provenance attestation. The workflow refuses to publish when the tag and package.json disagree.

One-time setup on npmjs.com — package → Settings → Trusted Publisher → GitHub Actions: theonlyrnh / dsh-judge-worker / release.yml.

Publishing by hand is still possible (npm login && npm publish); prepublishOnly runs the full verification first. Either way the package is built, never committed: lib/ is git-ignored and produced by the prepare script, so a git-installed copy (dsh plugin add github:theonlyrnh/dsh-judge-worker) builds itself.

License

MIT