Skip to content

dsh-answer-review-gate

Verified

@yadsh/dsh-answer-review-gate · v0.2.9 · MIT

Independent answer review gate for DeepSeek Harness agent responses

Install

dsh plugin add @yadsh/dsh-answer-review-gate

Confirm the layer applied with dsh --profile default --dump-config — see the install guide.

Source

Published to npm without a public repository. Inspect the package contents before installing.

Tags

Readme

@yadsh/dsh-answer-review-gate

Independent answer review gate for DeepSeek Harness agents: before a candidate final answer is allowed to complete its turn, an independent reviewer checks it and can send it back for correction. The review is a lifecycle requirement enforced at the turn boundary — not a line in the system prompt.

What it does

User → primary agent → research / tools / background subagents
     → candidate answer
     → answer-review-gate → reviewer
            ├─ PASS  → the turn may finish
            └─ REVISE → findings are steered back into the primary,
                        which produces a corrected candidate that is
                        reviewed again (bounded rounds)
  • Interim orchestration turns are never reviewed: while the session's own background delegations (continuable subagents, one-shot background jobs) are still pending, a turn close is treated as a pause, not a final answer. The decision is pure runtime state — the plugin records delegated child ids from the structured delegation tool result and clears them when the runtime's subagent-settled notice arrives. No text heuristics ("waiting for…") participate in this decision.
  • A review PASS applies to the exact candidate content (SHA-256 over the normalized text). Any later edit invalidates the PASS and the new candidate is reviewed again; the receipt follows the user request, so an unchanged candidate is never reviewed twice.
  • Review rounds are bounded per user turn (maxReviewRounds, default 3). The budget belongs to the user request the candidate answers — the agent turns and revisions that request takes share one budget, and no agent turn hands out a fresh one.
  • A revision steer admits exactly one visible artifact: the corrected answer. Findings are handed back as a delimited <review_notes> block, and the steer asks for no account of the review — a demand to "state the disproof" of a rejected objection was resolved literally on a live stand, and the user read the primary's argument with its reviewer. A draft that opens by disputing the review (Опровержение вывода ревизора: …) is that leak: in a turn that has already been reviewed the gate demands the answer's shape once, and that draft is not handed to a reviewer, so it cannot be certified as verified. The honest bounds: a committed message cannot be edited at this seam, so a second such draft goes through the ordinary review path, and a request the user waived is settled before the guard runs.
  • The reviewer itself is exempt: it runs as a subagent child and subagents are never gated, so the reviewer cannot recursively review itself.

Explicit one-request waiver

When deployment policy permits it, a user can send one request without the automatic reviewer:

/no-review <request>

The command may carry the same image or file attachments as an ordinary request. It immediately shows a notice that the answer will not be independently verified, then submits <request> as the real user message. The waiver applies to that request only.

This is a structured command contract, not natural-language detection. The successful command/done.sourceEventSeq must point to the durable agent/inbox/spliced event that admitted the exact user-message id, and the same lifecycle must start with a human-issued command/run for no-review. Phrases such as “review is not needed”, including quoted or negated mentions, never bypass the gate. A failed command, unrelated message, or old command lifecycle does not bypass it either.

Reviewer backends

Both backends receive the latest human request's original durable image blocks, including requests with no text. The Host applies the reviewer's selected model capabilities and image limits; the gate does not replace images with text-only placeholders. A reviewer that cannot inspect an attachment reports its own verification gap without inferring that the primary agent could not see it. Image review through domain-expert requires a matching domain-experts build with the host-only reviewExpert entry. Older builds report expert-image-input-unavailable through the configured failure policy rather than silently reviewing only the text.

reviewer.backend: domain-expert (default)

Runs the reviewer domain configured in dsh-domain-experts by stable domain id (reviewer.domain, default answer-reviewer), so the reviewer persona, policy, tools and model are tuned in the domain-experts UI, not here. The verdict is derived from the expert's own structured protocol: findings and conflicts are the review objections; a run with no findings and no conflicts passes. The domain-experts test entry owns its lifecycle: a running child cannot be interrupted through this face. The gate checks cancellation before and after each attempt, refuses a late verdict, and starts no recovery attempt after cancellation.

reviewer.backend: subagent

Starts a native reviewer child (reviewer.provider, default spawn) with a fresh context: it receives only the user request and the candidate answer, an optional persona (reviewer.persona), a read-only tool allow-list (reviewer.allowedTools; unset means the read-only set, an explicitly empty list means no tools) and an optional model route (reviewer.route / reviewer.model / reviewer.reasoningEffort). The verdict must be delivered by the native structured_output tool; prose and fenced JSON are not captured verdicts. For correlated-failure reduction, route the reviewer to a different provider or model than the primary agent.

Capture recovery and current review status

If a reviewer ends without native structured output and without a diagnostic, the gate makes one fresh, isolated attempt on the same immutable candidate. The retry explicitly requests structured_output. Real provider errors, refusals and cancellation are not retried. Two missing captures follow the configured failure policy; reviewer prose is never promoted to PASS.

The gate records a standard producer notice containing the answer sequence, content hash, human-request sequence and current status. QA Surface renders a compact current-status badge in QA and native chat. A later result replaces the badge for that request, and applies only to its recorded answer version. Replay preserves the status; earlier model-authored messages remain history. Waived, failed and cancelled reviews are visibly different from a verified answer.

Failure policy (failMode)

What happens when the review could not complete — reviewer failure, malformed verdict, or rounds exhausted:

  • open — allow the answer; audit the failure.
  • warn (default) — allow the answer, but steer an instruction requiring the primary to explicitly qualify that independent verification did not complete.
  • closed — steer a demand to revise the answer or end with an explicit unverified-answer notice; the answer is never presented as verified.

A reviewer failure is never converted into a PASS. At most one failure steer is issued per user turn, so the loop cannot livelock. Note the honest bound: a plugin cannot hard-block a turn at this seam — closed mode forces a revision-or-disclaim step rather than silently dropping the answer.

Configuration

enabled: true
reviewer:
  backend: domain-expert
  domain: answer-reviewer
  # backend: subagent
  # provider: spawn
  # route: ""
  # model: ""
  # reasoningEffort: ""
  # persona: ""
  # allowedTools: []          # unset means the read-only set; [] means no tools
maxReviewRounds: 3
failMode: warn
trackBackgroundDelegations: true
minCandidateChars: 80
excludedAgents: []
waiver:
  enabled: true
  allowedInClosedMode: false
audit:
  enabled: true
  maxEntries: 500
Field Meaning
enabled Master switch; false registers no listeners at all.
reviewer.backend domain-expert or subagent.
reviewer.domain Reviewer domain id in dsh-domain-experts (stable id, not a display name).
reviewer.provider Subagent provider name for the subagent backend.
reviewer.route / reviewer.model / reviewer.reasoningEffort Reviewer route overrides for the subagent backend.
reviewer.persona Persona instruction for the subagent reviewer.
reviewer.allowedTools Read-only tool allow-list. Unset means the read-only set (read, read_image, glob, grep, docs_read, docs_search); an explicitly empty list means the reviewer works without tools. Destructive and writing names are removed whatever the list says.
maxReviewRounds Review/revision rounds per user turn before the failure policy applies (1–10).
failMode open / warn / closed, see above.
trackBackgroundDelegations Suppress review while the session's background work is pending.
minCandidateChars Candidates shorter than this skip review.
excludedAgents Session-id substrings that are never reviewed (reviewer children are exempt structurally).
waiver.enabled Register and honor /no-review <request> for exactly one request.
waiver.allowedInClosedMode Permit the command while failMode: closed; default false.
audit.enabled / audit.maxEntries In-memory audit ring of decisions (metadata only — never prompt or response text).

Requirements

  • DeepSeek Harness >=0.1.7-rc.2 <0.2.0
  • Node.js ^22.19.0 or >=24.0.0

Installation

dsh plugin --profile <profile> add @yadsh/dsh-answer-review-gate

The --profile flag is required.

Compatibility

  • DeepSeek Harness >=0.1.7-rc.2 <0.2.0 (see compatibility.json).
  • Uses the commands host service plus the agent/turn-stopping, tools/result and agent/inbox/inserted lifecycle seams — no DSH core changes.

Security model

  • The reviewer runs in a fresh context and receives only the review material (user request, candidate answer). It does not inherit the primary agent's hidden reasoning.
  • The domain-expert backend's tool access is whatever the configured domain grants; configure the reviewer domain read-only. The subagent backend's tools are reviewer.allowedTools minus the names src/reviewer-tools.ts excludes: an unset list gives the reviewer the read-only set (reads and searches), and a listed file_delete, write, edit, apply_patch, str_replace_editor, bash, shell, run_code, lsp, dsh_lightrag_delete or a terminal_/job_ tool is dropped before the child is started. A reviewer that reaches for a deletion stops the turn it belongs to: on a QA stand the call parked above the parent's composer for an approval a delegated child can never receive, so the boundary is held both by the allow-list and by the surface, which refuses a delegated call outright.
  • Reviewer output is untrusted model output: it is parsed and validated, a malformed verdict is a reviewer failure handled by the failure policy, and the primary remains responsible for checking reviewer evidence rather than blindly obeying it.
  • Audit records contain decision metadata only (session, turn, candidate hash, rounds, durations, verdict or structured waiver reason) — no prompt or response text. /no-review also sets recordInput: false, so its lifecycle record does not duplicate the request text; the normal user message remains the sole authoritative copy.

Scope of Phase 1

Shipped: turn-stopping gate, both reviewer backends, candidate hashing, REVISE-via-steer, round limit, recursion protection, background-delegation suppression, structured one-request waiver, in-memory audit, tests. Deferred to later phases: durable /replayable review state, evidence bundle, full metrics export, and a UI badge.

See SPEC.md for the product contract.

Development

pnpm build
pnpm lint
pnpm typecheck
pnpm test

License

MIT