Experiment Records
One mutable record per experiment holds its design, its registered estimands, what informed the design, its predictions, its runs, its results, and its conclusion. The record is written before the data exists and finished after the run is processed. Rules and workflow for that record are on this page; the rest of the notebook is in the reference.
experiments / record · source
Experiment record
Use stable IDs such as
EXP-001, ACC-E1, EXP-045b.
IDs remain unique and recoverable from the filename and first heading. A revision-letter suffix
marks a follow-up that is not a separate experiment: a pilot, a re-analysis, the full-scale run
of its pilot.
# EXP-006: DAS, the minimal causal subspace mediating the space index
**Created**: 2026-06-29 (human authorized the DAS run)
**Status**: completed
**Summary**: **three-architecture POSITIVE** (Qwen2.5-3B L34 + Pythia-2.8b L19 + Falcon3-3B L13). A learned low-rank causal subspace (Qwen rank-2–4, Pythia + Falcon3 rank-1) gives full color control (flip 1.00), beating the rank-1 diff-of-means and random controls (~0.57–0.58 on Qwen/Pythia, 0.70 on Falcon3). The final causal test the hand-built patches (EXP-003) could not deliver.
**Research Questions**: [[RQ6]] (minimal causal subspace), final causal test for [[RQ3]]The status line holds one token that tools read. The one-paragraph result lives on a Summary line, where a person reads it.
| Section | Required content |
|---|---|
| Identity | Title, created date, status, and linked research questions |
| Hypothesis | A specific, falsifiable claim |
| Method | Instrument, data, conditions, metrics, script, revision, and key command |
| Estimands | One ### E# heading per measured quantity with its registration value |
| Informed by | Links to what was read or measured before the design was fixed, including any unregistered facet of a previous run |
| Preregistered predictions | Predicted ranges, reasoning, and null conditions |
| Decision rule | Actions or stop conditions for each material outcome |
| Runs | Backend, job ID, description, status, and artifacts |
| Results | Observed values, uncertainty, effect sizes, and checks |
| Prediction outcomes | Confirmed, partial, refuted, null-confirmed, or not-tested |
| Conclusion | Interpretation, tested scope, and threats to validity |
| Follow-ups and artifacts | Concrete next tests; output URI, content hash, and retrieval date |
The Pre-registered headings are accepted with or without the hyphen. Which
sections the validator requires depends on the status: a designed record needs the hypothesis,
method, predictions, and decision rule; a queued or running record adds runs; a completed
record adds results, the outcomes table, a conclusion, and artifacts.
experiments / status · schema
Status vocabulary
proposed→planned→queued→running→pilot-complete→completed abandoned is a valid exit from any state. Use in-progress for active
work that is not queue-shaped and blocked for work stopped on something the status
line names. Write the token as **Status**: completed, optionally followed by a
parenthesized qualifier. Legacy spellings
complete (for completed), done (for completed), cancelled (for abandoned), canceled (for abandoned)
validate with a warning. closed is rejected because it means abandoned
in some records and completed in others.
The vocabulary is data. The schema lists the canonical statuses, the accepted legacy spellings, and the rejected ones, and the validator reads that file rather than prose.
experiments / estimands · source
Estimands and registration
An experiment names each quantity it measures as an estimand, with a machine-readable
registration status. The heading matches ^#{2,4}\s+E\d+\b,
and the first **Registration**: line after it is required.
### E1 Derived-belief flip rate when visibility is swapped
**Registration**: registered. Thresholds 0.6 (confirms) and 0.4 (null) fixed in the Decision rule before wj3711.
### E2 Stated-belief invariance to an added visibility clause
**Registration**: registered. Threshold 0.8 fixed in the Decision rule.
### E3 Derived and stated clean accuracies
**Registration**: gate. Sanity readouts on the unmanipulated conditions.| Value | Meaning |
|---|---|
registered | The statistic, the expected direction, and the label every outcome reaches were fixed before the data existed. |
found | Nothing was fixed in advance. A claim drawing on it is a found result; confirmatory use needs a fresh registered run. |
gate | A precondition on interpreting another estimand, such as a headroom or instrument-health check. Not a result to cite on its own. |
One record usually carries more than one value. An arm registered in advance can go void
while an arm that existed only as a gate produces the run's most useful number, which is
why the status attaches to the estimand rather than to the record. A record that declares
no estimands is not flagged. A claim cites the estimand as
EXP-NNN:E#, and the validator warns when a
claim draws on a found estimand or on a gate.
experiments / results
Separate execution evidence from scientific evidence
One run can produce several kinds of evidence. Name the kind beside each recorded check or statistic. This prevents a healthy pipeline, an interpretable instrument, or an interesting side measurement from standing in for the quantity the experiment was designed to estimate.
| Evidence | What to record | What it establishes |
|---|---|---|
| Artifact authentication | Output URI, content hash, code revision, and retrieval date | Which bytes were analyzed and how they were produced |
| Execution check | Completion, schema, expected cells, missingness, and finite-value checks | The pipeline ran and produced structurally usable output |
| Scientific gate | Manipulation, headroom, capability, and control-arm values with their thresholds | Whether a named estimand can be interpreted |
| Primary scientific statistic | The preregistered estimate, sample size, uncertainty, effect size, and decision-rule outcome | The experiment's answer to its main question at the tested scope |
| Other scientific statistics | Secondary and exploratory estimates, each marked registered or found | Additional evidence whose inferential status remains visible |
Report the primary statistic first, including a missing or void result. A gate failure explains
why the linked statistic is uninterpretable; a secondary or found estimate does not replace it.
A run that only authenticates an artifact or exercises the pipeline still has a useful result.
Its conclusion should say, for example, Execution check only: artifact authenticated; no
primary scientific statistic was produced.
experiments / workflow
Register and run one experiment
Reserve a stable ID such as EXP-012. Record the instrument, data, conditions,
primary and secondary metrics, code revision, key command, preregistered predictions, and
decision rule before inspecting outcomes. Pause for design review before preregistration and
for preregistration review before execution. Attach backend job IDs and immutable artifact
locations to the same experiment record. After analysis, obtain interpretation review before
recording a synthesis or approving a follow-up.
Name each quantity the experiment measures as an estimand, E1, E2,
and so on, and say whether it is registered, found, or a
gate. Then list, under ## Informed by, what was read or measured
before the design was fixed, naming any unregistered facet of an earlier run.
## Estimands
### E1 Off-diagonal transfer (R_belief to reality, R_reality to belief) over the random floor
**Registration**: registered. The 2×2 transfer and the two labels (dissociated,
frame-agnostic) were fixed in the Method; low versus high was not given a number.
### E2 Clean readout accuracies
**Registration**: gate. Both at or above 0.8, or the model is not holding the
conflict and E1 is uninterpretable.
### E3 Bare-query default to reality
**Registration**: found. A diagnostic side-result that became evidence for
reality as the unmarked frame.
## Pre-registered predictions (a priori)
- **P1 (E1)**: dissociated at the value site, the paper's framing.
### Outcomes against pre-registered predictions
| Prediction | Verdict | Predicted | Observed |
|---|---|---|---|
| P1 (E1) | refuted | low off-diagonal transfer | off-diagonal 92–95% over floor on Qwen 7B, 14B, and 7B-Instruct; 89% Mistral-7B; 88% OLMo-2-7B |
| E2 gate | failed zero-shot, met few-shot | both readouts at or above 0.8 | reality readout 0.50 / 0.41 / 0.47 zero-shot at 7B / 14B / 32B; 0.98 to 1.00 with few-shot |One record, three registration values. The registered estimand refuted the paper's original framing and relocated the belief index to the query position; the gate failed at every scale until few-shot elicitation cleared it; the found estimand was never something the design could license a claim on. The reversal is what made the value-slot-versus-router mechanism the paper's central claim.
Example notebook · file map
Does gradient accumulation reproduce true large-batch training?
Each row summarizes the data owned by one full notebook file.
QUESTIONS.md- RQ1 asks whether optimization behavior is preserved at fixed effective batch.
plans/completed/2026-08-12-accumulation-controls.md- The plan was reviewed before execution. Phase 2 starts only after human review of the pilot gate.
EXP-001-accumulation-pilot.md- E1 registers the seed-1 loss difference with a 0.02 threshold; E2 is a finiteness gate. The pilot is
slurm/48152. EXP-002-accumulation-comparison.md- Three paired seeds test the same 0.02 margin after the pilot passes. Its
Informed bysection names the pilot and says seed 1 is reused. findings/2026-08-16-accumulation-matches-large-batch.md- Synthesizes both experiments and preserves the toy simulator's scope limit.
plans/completed/…completion report- Both goals met, each linked to its record; the limitation to the deterministic simulator; no follow-up open.
CLAIMS.md- C1 is a major claim for
gradient-accumulation-note, citesEXP-001:E1,EXP-002:E1, andEXP-002:E2, and records human promotion review; the finding supplies synthesis context.
The complete fictional notebook and its deterministic toy command are in the repository example.
experiments / finished run
Process a result
A finished process is not a finished experiment. The process is finished when it exits; the experiment is finished when its artifacts are validated, its record is written, and its decision rule has been applied. The steps are the same for a cluster job, a remote one-liner, and a script run on a laptop. Only the durability step differs.
- Validate
Confirm the run produced what it claims. Record artifact authentication, execution checks, and scientific gates separately. A run that returned numbers can still be uninterpretable.
- Record
Evaluate the primary scientific statistic first, including a missing or void result. Compare every preregistered prediction with the observed value, then label secondary and found statistics without substituting them for the primary result.
- Decide
Apply the predeclared decision rule. Failed and null results go through it too; no unregistered rescue campaign.
- Review
Ask a human to review exclusions, uncertainty, interpretation, and the proposed follow-up, then update the indexes and claim pointers the result changes.
- Secure
Job backend: retrieve and verify outputs, validate and commit the notebook, then write the processed mark last. Local run: the output is the only copy, so record the numbers before anything can overwrite them.
- Report
Follow the session's agreed interaction mode: as each result lands, with a pause after each experiment, in a review pass after the batch, or as a terminal report only.
Treat an existing processed record as an idempotency signal. Verify its evidence links rather than creating a duplicate. Failed jobs can be processed after their failure mode and useful partial artifacts are recorded. A local run has no processed flag; the experiment record's status is its equivalent.
## Decision rule (a priori)
- Baseline consequent below 0.8: uninterpretable; a capability failure, not a reversal.
- Baseline at or above 0.8, each subspace controls its own readout and cross-effects
stay low: the dissociation replicates.
- Baseline at or above 0.8 and cross-effects comparable to own-readout effects: the
dissociation reverses.
### Outcomes against pre-registered predictions
| Prediction | Verdict | Predicted | Observed |
|---|---|---|---|
| Replicates (E1, E2) | confirmed on Falcon3-3B, Qwen2.5-14B, Qwen2.5-32B | own-readout control near 1, low cross-effects, low mean cosine | own readouts 1.00 everywhere; report-to-consequent 0.49 / 0.42 / 0.04; mean cosine 0.17 / 0.14 / 0.15 |
| E3 gate on Pythia-2.8b | not-tested | baseline at or above 0.8 | baseline 0.486; capability failure |Pythia-2.8b returned numbers for every cell. Without the gate branch written in advance, its 0.51 cross-effect would read as a reversal of the project's strongest causal claim. With it, the run is a capability failure and the claim's scope is stated correctly.
experiments / siblings · source
Annexes and duplicate IDs
Breakdown tables that support the headline go in a sibling annex named
EXP-NNN-topic.annex.md. An annex carries no status and
is not indexed; the ID in its filename is the join. Split when the breakdown would crowd out
the argument, say in the experiment's results that the annex exists, and open the annex with
what the breakdown shows.
Two experiments sharing one ID are resolved by renumbering the side that never ran, by a revision-letter suffix when one is a follow-up of the other, by merging a design stub into the executed record, or, when both ran, by renumbering one side and retagging its jobs in the same change. Renumbering without retagging makes a visibly ambiguous ID resolve silently to the wrong file. A banner naming both lines is the fallback when retagging costs too much, and every file carrying such a banner must be edited when an ID it names later moves.