Code review checks an implementation against its scientific design. Design review checks whether that design can support its intended claims. Each can find defects that a successful run would leave hidden.
The cycle targets experiment scripts. Repairs return to fresh review; missing evidence, scientific choices, and exhausted limits require different handoffs. Each stage identifies the responsible actor, the record produced, and the authority required.
A check returns evidence to review after authorized execution. Scientific and limit stops wait for a recorded decision.
Stage in view: Bounded mandate
The researcher authorizes a defined repair, with frozen inputs, editable files, permitted checks, and limits. Studio receives that snapshot and policy.
Illustration. A scorer may be repaired to implement the registered inclusion rule; the rule itself stays fixed.
An independent panel checks the script against its design and applicable gates. Its findings determine whether the next action is a repair, an executable check, a scientific decision, or verified hand-back.
Illustration. The panel finds that the scorer includes an excluded pilot cell in the average.
A writer repairs findings within the mandate. Changed inputs return to fresh review; the model that wrote the bytes does not review its own work. The cycle repeats while there is authorized work to do.
Illustration. The writer corrects the aggregation. A fresh panel verifies that the revised script follows the original inclusion rule.
An unresolved condition can require a positive control, sensitivity check, or pilot. The controller returns a bounded job request. Once separately authorized, the runner performs the check and returns outputs and a receipt to review.
Illustration. A known-answer check has one attempt and a 10-CPU-minute cap. A passing result addresses scorer sensitivity at that tested scope, not the full study's conclusion.
A material design choice, upstream invalidation, disputed finding, or unclear authority produces an escalation packet. The researcher judges the evidence, the proposed response, its scientific cost, and the strongest credible alternative.
Illustration. Tightening a threshold after seeing results changes the analysis's selection history. The choices may be to retain it with a narrower claim, commission new evidence, or stop.
The controller stops when a round, active-time, attempt, or spending limit is reached. Continuation requires a concrete case for further work and an explicit extension of the relevant allowance.
Illustration. Two consumed rounds plus two newly authorized rounds gives a total allowance of four, with two remaining. Declining the extension leaves the workflow stopped.
Synthetic continuation account
These amounts are invented for the illustration. Experiment-run commitments are a separate account in the cost-control simulator.
When the required reviews and bounded verification pass, Studio returns review evidence and a publication plan. Import validates that evidence; applying source changes is a separate operation.
Illustration. An unchanged script preimage is insufficient if a helper or design record changed while the review ran.
Code reviews find mismatched calculations, omitted decision rules, stale artifacts, failures presented as results, unverified inputs, and checks that cannot expose a defect. The Code review guide distinguishes whole-script review, script-delta review, code audits, and change review, with their evidence and fault families.
If the registered rule requires every seed to pass but the code tests only the average, the implementation needs repair. If the rule itself cannot answer the scientific question, the fault belongs to design review, even when a code reviewer discovers it. The autonomous loop needs evidence for whether a targeted check can settle the finding or a scientific decision must return to you.
Identify the scripts, the design and evidence they depend on, the files the writer may edit, permitted verification, and limits on rounds, time, attempts, and spending. Submission freezes those inputs and the policy. Studio works on that snapshot rather than a moving copy of the laptop's tree.
A design prerequisite establishes what the script must implement. An independent script panel reviews the implementation; a writer addresses findings within the mandate; a fresh panel reviews the changed inputs. The model that wrote the bytes is excluded from reviewing those bytes. Record a repair as landed before recording it as verified. A passing test alone does not close a finding.
Mechanical waits, such as a provider quota reset, can be reconsidered automatically within the same authority. A stop requiring judgment or permission stays stopped. A threshold encoded in an editable script is still a scientific choice: file editability does not authorize changing its meaning.
Use the review guide for reviewer independence and finding records, and the experiment guide for human design and preregistration review.
A review can identify the need for a positive control, sensitivity check, or bounded pilot. Pilot eligibility says such a run could address the unresolved question. Execution additionally needs permission binding the subject, baseline, code and development inputs, relevant check, and resource caps. A changed script does not renew a consumed permission.
An ordinary verification command does not automatically supply that receipt or establish scientific sensitivity. Confirmation data and sealed reserves are outside a developmental permission. Passing a pilot does not approve the full study.
Escalate a material design change, an upstream invalidation, a disputed finding, or missing authority. If materiality is unclear, hand back rather than guess. The packet names the finding, the evidence available, the proposed change, its consequences, and the decision needed. The rest of the workflow stops unless unaffected targets are established to be independent.
Persistent review-only findings can call for a choice to simplify, commission a targeted check, reset scope, authorize a bounded number of further rounds, or accept a stated residual risk. Discuss the strongest credible alternative and what the proposed response costs scientifically. Keep the answer with its evidence so a later session can recover why the work continued or stopped.
Accepting residual risk records a decision; it does not turn a failed review into a pass. Authorizing more review does not approve a run. A scope reset needs a new mandate. Review can also end with a decision to stop the line of inquiry.
Round, active-time, attempt, and spending limits are separate constraints. After a limit stop, an explicit continuation can extend the relevant allowance while preserving accumulated usage, authorship, and evidence. A writer rotation or provider reset does not reset those counters. Unknown spend remains unknown.
Two consumed rounds plus permission for two more means four rounds in total, with two remaining. It does not grant execution authority. The cost-control simulator separately tracks incurred charges, running commitments, and planned work. Review/repair limits and experiment-run budgets are not assumed to be one enforced account.
Distinguish a limit enforced by the controller or runner from an estimate recorded in the notebook and a checkpoint requiring a person. A written ceiling is not a process kill switch. Admission needs conservative bounds for overhead and stopping delay; see the authorization and enforcement rules.
Each hand-back identifies the delegated run, its producer, and the inputs and policy it used. Import validates that evidence against the laptop's mandate. An imported record remains evidence of work performed on Studio; it does not transfer custody of a running process to the laptop.
Applying the source changes is a separate step. It checks the entire recorded input base, including design and dependencies, against the laptop's current files. A changed input creates a conflict rather than permission to overwrite it. Publication records each destination's state so interruption can be reconciled without erasing unfamiliar edits. It makes no automatic VCS commit.
Review coverage belongs to the bytes reviewed. A later edit can make dependent reviews due again, as the evidence and dependency map illustrates.
Code repair changes an implementation against a design. Design-record repair would change the record while preserving the scientific choices that belong to the researcher. A design panel would identify defects, a writer would address those within its authority, and a fresh panel would review the candidate and its changes. A defect that requires a new scientific decision would return to you with a concrete proposal.
An identified source determines the answer without changing a protected scientific choice.
Several defensible readings remain, or the edit would change how observations support a conclusion.
Further authorized repair: another bounded round. Pass with findings accounted for: hand back the reviewed record for separate publication. Unresolved decision, uncertain authority, non-convergence, or a hard limit: hand back for human action.
An amendment grant authorizes one exact edit. Design approval, data access, execution, and spending require their own authority.
Design reviews find unsuitable quantities, unsound decision rules, insufficient power, invalid controls, and claims that exceed the proposed evidence. These can remain faults when the code implements the protocol exactly. The Design review guide separates full review, design-delta review, and experiment-yield opportunity scans.
A threshold unreachable on the design's own measurement grid needs a scientific decision and amendment authority. A script that omits a valid registered condition needs code repair. The review that discovers the problem and the layer that must change are separate facts; the loop cannot infer execution eligibility from the review type alone.
Scientific eligibility depends on what the experiment's authors already know about its data and outcomes. That exposure determines whether an amendment is compatible with the preregistration status the record claims. An empty results section or a job lookup that finds nothing cannot establish that nobody inspected data or probed a model by hand. The hand-back would state the available evidence and its limits, rather than certify the experiment as never executed.
Operational eligibility concerns concurrent execution. From admission through publication, a job must not begin against the design being amended without coordination or invalidating the amendment. Checking the file's original bytes would not detect a newly admitted job. The planned workflow would therefore coordinate with execution admission and refresh its job observation before publication.
An identified authoritative source could determine a clarification: the record itself, a record it cites, or its research-spec block. The writer would cite that source. For example, a cited protocol that explicitly defines a replicate as one independently sampled item could supply a missing definition. If several defensible readings remain, including contradictory statements in the record, choosing one would require your decision.
Protection follows meaning throughout the record. Questions, predictions, estimands, decision and stop criteria, arms, controls, model selection, sample size, budget ceiling, and scope would remain frozen without an amendment grant. For example, defining a “replicate” as an item rather than a seed can change the effective sample size even if the sample-size paragraph stays untouched. A change to how observations support a conclusion would count as substantive wherever it appears.
For a substantive change, the writer could draft a replacement and explain the decision, its scientific cost, and the strongest credible alternative. Leaving the design unchanged would remain an alternative when viable. The draft would stay unapplied until you approved it. A concrete draft can save writing effort, but it also gives the model influence over which option is easiest to accept.
Approval would create an amendment grant bound to the record, workflow, design version, expected original text, and exact replacement. The controller would apply the approved bytes and verify them, rather than ask the writer to regenerate the proposal. A changed original would make the grant stale. After an interruption, the controller would reconcile the same application instead of authorizing a second one. The grant would permit only that amendment, not design approval, data access, a pilot, additional spending, or a checkpoint bypass.
A mechanical checker would compare each attempt with its immediate pre-attempt version. An unauthorized edit to a protected region, or a structure the checker could not resolve, would trigger rollback while preserving earlier authorized repairs. The rejected candidate would remain available as evidence. The exact candidate left after rollback would need validation and fresh review before publication.
The independent panel would inspect the full candidate and the actual changes for altered scientific meaning, resources, execution authority, or stopping behavior. A panel's approval could establish that a change is admissible; it could not grant permission for a frozen scientific choice. An uncertain assessment would return the work for human action.
Every finding would retain a recorded outcome. A high or blocking finding would need repair followed by re-review, or escalation; a writer's rejection would require independent adjudication. Medium and low findings could be handed back as unresolved, without being described as fixed. Success would require every required panel seat to pass the current design and all high or blocking findings to have an outcome.
Reviewers would assess convergence using the history of findings and repairs. A grounded non-convergence assessment would hand back even if a panel otherwise passed. Three rounds would be an advisory checkpoint, not an automatic stop: continuation would need a concrete account of what the next round could establish. Fewer findings alone would not suffice. The policy's spending and round limits would remain hard stops.
A changed design would make dependent script reviews due again. The hand-back would include script declarations from both before and after the edit, so deleting a declaration could not hide that dependency. A script finding that exposed an unsound design would still block design convergence; a pure implementation defect would become a script follow-up. Publication would remain a separate, base-checked application to the source record, with no automatic commit.