Design review

documentation/ guide and reference

Guide pages

Start here

When you need it

Design review asks whether an experiment's questions, measurements, comparisons, and decision rules can support the intended inference. It examines the experiment record's design, while code review asks whether an implementation follows that design. A design review can find a flaw even when the code implements the protocol exactly.

Question and scope

A full design review evaluates the current design kernel: the experiment record's title, stable metadata, and design sections. Execution status, jobs, results, conclusions, follow-ups, artifacts, and findings are excluded from that kernel. The reviewer considers whether the protocol can distinguish the explanations it sets out to test and whether its proposed conclusions follow from the evidence it would collect.

Scientific and statistical reasoning are lenses within this review, not separate registered procedures. They can include whether the target quantity matches the question, whether measurements and comparisons identify it, whether sample size and stopping rules support the decision, and whether controls and thresholds are suitable for the stated inference. These are design questions, not a claim that a particular statistical test or universal threshold applies to every experiment.

Design-review procedures

Full design review: `review-design`

This procedure reads the complete current design kernel and assesses whether it supports the intended claims. It is the ordinary design-review procedure. Any change to the kernel invalidates prior coverage, including a change to a declared cost ceiling. When a current full pass is absent, the normal coverage path is a full review.

Design-delta review: `scan-design-delta`

This narrow procedure judges what a specific edit does to a design that was previously read in full. It does not re-adjudicate untouched sections or replace a full review of an unreviewed design. Its named basis must be a passing review-design for the same experiment record. A delta can never use another delta as its basis, so successive narrow reviews cannot move the design indefinitely from the last design anyone examined as a whole. The reviewer chooses this lane; it is not selected automatically when coverage is missing.

Experiment-yield opportunity scan: `scan-experiment-yield`

This scan looks for additional evidence a run could bank at little extra cost, such as an unrecorded measurement, an idle condition, or a slice the harness computed and discarded. It is an opportunity search, not a judgment that the design is sound or that a suggested measurement belongs in the approved scope. It does not substitute for either kind of design review.

Inputs and evidence

A full review needs the current experiment record and its design kernel. Relevant cited protocols, measurement definitions, assumptions, and analysis rules help establish what the record means and whether its claims follow. Reviewers should ground findings in the stated question, planned data, comparisons, controls, thresholds, sample size, and stopping rules. Missing or ambiguous evidence is itself a limit on what can be concluded; it is not evidence that an assumption holds.

Review the design as currently written. Execution bookkeeping and observed outcomes are outside the design kernel, so a full design pass does not inspect whether a run happened, validate its output, or establish that nobody has seen outcome data. Those require separate evidence and decisions.

Scientific fault families

Wrong target quantity. The estimand or verdict measures something other than the intended scientific effect.
Unsound decision rule. A gate cannot be met under the registered sample size, or its threshold fails to distinguish the stated explanations.
Insufficient sensitivity or invalid controls. The design cannot detect the intended effect, or a control does not test the condition it is meant to test.
Claims beyond the evidence. The comparison cannot establish the promised conclusion, or dependent inferences remain after their supporting quantities are withdrawn.

Fault discovery and fault ownership are separate. A reviewer may first notice that a threshold is unreachable, but changing the threshold changes the scientific rule and belongs to the design. If the design's threshold is valid and the script evaluates it incorrectly, the defect belongs to code review. Findings can overlap across layers; assign repair to the layer whose meaning or behavior must change.

Fault examples are an interpretive grouping from Review fault typology: what design and script reviews find (October 8, 2026), a report covering four project ledgers. They are not an exhaustive taxonomy or measured detection rates.

Outcomes and authority

A review reports a verdict and grounded findings. A pass applies to the reviewed design kernel; a later edit invalidates its full-pass coverage unless a reviewer-selected delta review covers that change against the required full basis. A pass is not a universal certification of data quality, implementation, runtime compatibility, instrument sensitivity, or scientific truth.

Review disposition and permission to execute are separate decisions. Reviews default to reporting findings; remediation must be explicitly requested. Neither a review verdict nor an edit mandate grants an amendment to a protected scientific choice, permission to run a pilot, launch authority, or approval to expand scope. Execution and amendment decisions stay with the authorized human process.

Related reviews and workflow

Code review checks the implementation against the experiment design. The general review guide covers independence, recording findings, and review gates. The autonomous design-review-and-repair workflow describes a proposed bounded process for changing and re-reviewing a design record. Ordinary design review exists; only autonomous design-review-and-repair is not implemented.