Evidence, claims, and data reuse

documentation/ guide and reference

Guide pages

Reference pages

An experiment record holds what was tested and observed, with the conditions needed to interpret the result. Its links show what informed the design. The claims ledger lists the assertions the project makes and the evidence offered for each.

Design choices, unexpected patterns, and proposed claims belong in the notebook's records and human review checkpoints. Recording these decisions makes them inspectable. The notebook does not establish statistical validity or enforce access to data.

The evidence network connects these records so a reader can trace the evidence behind a claim and the observations that shaped a design.

Design the record before running the experiment

Start with the question in QUESTIONS.md. In the experiment's Method, identify the population, comparison, sampling unit, data source, and analysis. Name the quantity you intend to estimate under Estimands. This is the estimand: in EXP-101, the difference in answer accuracy between the two retrieval methods on the target question population.

Write predictions and decision rules before inspecting the relevant outcomes. Include what would leave the question unanswered, not just what would support the hypothesis. Ask for human design and preregistration review before running. The experiment reference defines the registered, found, and gate values.

EXP-101: fix the intended use before collecting the evaluation results

## Estimands

### E1 Overall answer-accuracy difference
**Registration**: registered. Revised minus baseline accuracy;
positive direction and decision rule fixed before collection.

## Decision rule

- Read E1 only after the execution and scoring checks pass.
- The document-clustered 95% interval lies above zero:
  support an improvement for the specified population.
- The interval lies below zero: report worse performance.
- The interval includes zero: improvement is not established.
  Do not replace E1 with the best-looking subgroup.

Also decide what to retain. For this example, preserve question and source-document IDs, both systems' answers, scoring outcomes, document lengths, and exclusions. Keeping these observations allows paired analysis and diagnostics later. Resampling by document accounts for questions that share a source; treating every question as independent would miss that dependence.

Retaining more observations creates opportunities for later analysis. It does not add a confirmatory comparison, and the selection history of any later analysis must remain visible. Weigh storage, privacy, and instrumentation costs before collecting more. Put artifact locations and producing code revisions in the experiment record; keep large outputs outside the Markdown files.

Record what the completed run establishes

Process the primary result before exploring alternatives. Under Results, separate checks that the output exists from checks that it can answer the research question. An exit code, a content hash, and a scientific control establish different things.

Record	What it lets the next reader conclude
Artifacts and execution checks	The identified code produced these bytes; expected observations and fields are present. This establishes neither an effect nor a valid analysis.
Scientific gates	The scoring procedure, manipulation, or baseline behavior meets the preconditions for interpreting the named statistic. Check each condition that can fail separately.
Primary result	The estimate, sample size, uncertainty, and decision-rule outcome answer the registered question at its tested scope, provided the checks and analysis are sound.
Secondary or found result	An additional observation is available, with its own registration and selection history. It does not replace an inconclusive or void primary result.

Suppose EXP-101's overall interval includes zero, but a breakdown by document length looks promising. Keep the overall conclusion: the planned test did not establish improvement. Record the unexpected pattern separately as found. The same run now has a registered result and a found result; calling the whole run “confirmatory” would hide the distinction.

Disclosed case: a readout had to pass before mechanism testing

In the published mental-spaces study, a model had to distinguish a painting, a photograph within it, and reality before the researchers could interpret interventions on its internal representations. The paper reports that the nested readout failed a behavioral gate and required prompt repair before the mechanism runs. The criterion, fixed in advance, was that all three explicit frame readouts exceed 0.80 clean accuracy.

A notebook record should keep the criterion, failed check, repair, and later mechanism results distinct. A repaired measurement can enable a test; passing its gate is not itself the mechanism result. Record what evidence guided the repair so a reviewer can check whether the later evaluation remained independent of that choice.

Source: the construction result and its readout-repair footnote in One mechanism for many mental spaces. See also the evidence-class reference.

Apply the decision rule, obtain interpretation review, secure the artifacts, and update the next action. Failed and null results go through the same processing workflow as positive results. A failed scientific gate means the affected question was not tested adequately, not that its hypothesis was refuted.

Carry the data's history into the next analysis

Before scheduling another run, check whether the existing artifacts contain the needed observations. EXP-101's per-question record can support the long-document breakdown without repeating inference. A small breakdown belongs in that experiment's results or annex; a separately scoped analysis can have its own experiment record. Neither creates new observations merely by receiving a new ID.

In EXP-102, list the exact input artifacts and code under Method and Artifacts. Use Informed by to record which results led to this analysis, including plots, summary scores, and decisions received from another person or agent. A fresh session does not erase what influenced the project.

EXP-102: preserve the reason this analysis was selected

## Informed by

- [[EXP-101-retrieval-comparison]]: the overall result was
  inconclusive. Its unregistered length breakdown suggested
  a benefit on long documents; that observation selected this
  analysis and its length cutoff.

## Method

Reanalyze the paired outputs from EXP-101. No new questions
or answers are collected. Preserve the original exclusions.

## Estimands

### E1 Accuracy difference on the selected long-document subset
**Registration**: found. The subset and cutoff were chosen
from the observed results; this analysis characterizes the
pattern and does not independently confirm it.

Reanalysis can diagnose a failure, refine a hypothesis, or justify the cost of confirmation. Record all analyses and cutoffs inspected, including those not selected. An interval computed as though the subset had been fixed beforehand does not automatically account for selecting it after seeing the outcomes.

Distinguish collection from access

For an existing dataset, record when it was collected, what was accessible, and what was actually inspected before the design was fixed. Previously collected but unexamined evidence is different from evidence already used to select a method. A summary score can influence selection even when nobody sees individual examples.

The released registration convention is conservative about pre-existing data. Preserve the chronology and obtain human review of the intended use. A status label does not establish independence. Version hashes identify the artifacts used; Informed by records how their contents affected the work. This provenance, the history of the analysis and its inputs, can be accurate even when repeated feedback from the same data selected the analysis.

Arrange confirmation before opening its evidence

If the project may turn an exploratory result into a confirmatory claim, identify its confirmation source in the plan at creation. Use Confirmation reserve to state what is held back and how it will be used. A reserve is evidence whose outcomes have not influenced the selection of the claim being evaluated.

For the fictional retrieval project, the plan provides for a separate document-and-question sample drawn from the same target population. EXP-103 freezes the long-document cutoff, retrieval configurations, scoring, sample-size rule, exclusions, and analysis before collecting or inspecting that sample. EXP-101 and EXP-102 remain in its Informed by section: confirmation tests a hypothesis learned from earlier work.

Plan fragment: reserve a source and specify what opening it permits

## Confirmation reserve

**Held back:** a separately sampled set of documents and
questions from the target population. Collect it only after
EXP-103's design is frozen. Do not use its outputs for prompt,
cutoff, scoring, or exclusion choices.

**Decision rule:** EXP-103 tests the frozen long-document claim
using its prespecified paired statistic and interval procedure.
If the result does not support improvement, record that outcome.
An outcome-informed revision requires another independent
confirmation source, not another pass over this sample.

Record who controls access, where the source or sampling specification lives, and the artifact identity when it becomes available. A Markdown declaration documents the rule; access controls, a custodian, or a separate execution process must implement it. Budget enough independent sampling units for the intended inference before committing to a reserve.

A new job, a new random seed, or another analysis of the same questions is not automatically fresh confirmation. New seeds may measure stochastic variability while leaving selection on the same question set intact. Once a confirmation result guides a revision, document that use in the descendant record. Do not present the revised claim as independently tested by the evidence that helped choose it.

Independent confirmation is the default workflow here. Specialized selective-inference or restricted data-reuse protocols can support other valid uses, but require their own assumptions, access rules, and human statistical review. Notebook labels and ordinary repeated queries do not implement those guarantees.

Field details and another disclosed example: confirmation reserves in plan records.

Promote the claim at the scope its evidence supports

Before updating CLAIMS.md, read the cited estimand's result, its gates, and its Informed by history. Cite the particular quantity, such as EXP-103:E1, and state the tested population and conditions in the claim. Keep experimental completion separate from evidentiary support: a completed run may leave a claim unsupported or a question open.

Record in the example	Permitted use
EXP-101 overall result	The planned overall comparison was inconclusive. The subgroup result does not turn it into an overall improvement.
EXP-102 long-document analysis	A found pattern motivated a narrower hypothesis. Preserve that origin even if later evidence supports the hypothesis.
EXP-103 confirmation	If its checks pass and its prespecified result supports improvement, cite it for the frozen long-document claim at the tested scope. Otherwise record the failed or inconclusive confirmation.

Bring the proposed claim wording and status to human promotion review, together with estimates, uncertainty, source identities, selection history, and limitations. A result on long documents in one population does not establish an improvement on all questions or other populations. Keep cross-experiment synthesis in findings/; the claim still points to direct experiment evidence. PUBLICATION.md records which paper uses it and what remains before submission.

Use the claim ledger reference for columns and status values. A validator can resolve a citation and flag a found estimand. It cannot infer from a valid link that the data were untouched, the interval is appropriate, or the claim has the right scope.

Repair the record before commissioning more work

When review questions a claim, follow its evidence links upstream and identify what is missing. Determine whether the problem lies in the presentation, analysis, or measurement, then choose a repair that resolves it.

What the review finds	Notebook action
The wording exceeds the tested population	Narrow the claim and paper wording. Update their scope; another run is needed only if broader scope is worth pursuing.
The needed analysis is absent, but observations were retained	Analyze the preserved artifacts, recording inputs and selection history. Label the analysis appropriately; do not call it fresh evidence.
A gate or instrument cannot support the interpretation	Mark the affected result uninterpretable. Record the failure and review the measurement design before relying on another run.
A required distinction was never recorded	State what cannot be recovered. Propose collection only after weighing its cost and whether it can resolve the question.
A corrected result changes downstream conclusions	Search for the experiment ID and superseded values. Correct dependent claims and findings, and add retraction notices to affected terminal plans.

A suggested follow-up needs a decision before it can proceed. Record its rationale and cost under Follow-ups, obtain the required decision, then put adopted work in the plan or PRIORITIES.md. The proposal supplies neither evidence nor authorization. Before execution, review what observations would otherwise be lost. After processing, review which new questions the evidence makes worth asking.

At handoff, leave the experiment's conclusion and follow-ups current, make artifact copies durable, validate the notebook, and update affected indexes and claims. The next session should be able to tell which question remains open, which data it may use for exploration, and which evidence is still available for independent confirmation.

Continue with the correction and synthesis workflow or return to the workflow guide.