Experiment Records

experiments / record · source

Experiment record

Use stable IDs such as EXP-001, ACC-E1, EXP-045b. Put the unique ID in the filename and first heading. A revision-letter suffix identifies a pilot, reanalysis, or full-scale run that belongs to the same experiment.

# EXP-006: DAS, the minimal causal subspace mediating the space index

**Created**: 2026-06-29 (human authorized the DAS run)
**Status**: completed
**Summary**: **three-architecture POSITIVE** (Qwen2.5-3B L34 + Pythia-2.8b L19 + Falcon3-3B L13). A learned low-rank causal subspace (Qwen rank-2–4, Pythia + Falcon3 rank-1) gives full color control (flip 1.00), beating the rank-1 diff-of-means and random controls (~0.57–0.58 on Qwen/Pythia, 0.70 on Falcon3). The final causal test the hand-built patches (EXP-003) could not deliver.
**Research Questions**: [[RQ6]] (minimal causal subspace), final causal test for [[RQ3]]

Metadata syntax

Write experiment metadata as Markdown field lines. Put the ID in the filename and title, the lifecycle token after **Status**:, and a short account in Summary.

# EXP-101: Retrieval comparison

**Created**: 2026-09-11
**Status**: proposed
**Summary**: Design a paired comparison of retrieval methods.
**Research questions**: [[RQ1]]

Record design approval and interpretation decisions in the relevant body section, with the reviewer, date, and approved scope. A lifecycle status does not record approval.

Section headings

The Pre-registered headings are accepted with or without the hyphen. Which sections the validator requires depends on the status: a designed record needs the hypothesis, method, predictions, and decision rule; a queued or running record adds runs; a completed record adds results, the outcomes table, a conclusion, and artifacts.

## Hypothesis
## Method
## Estimands
### E1 Primary quantity
**Registration**: registered. State what was fixed and when.
## Informed by
## Preregistered predictions (a priori)
## Decision rule (a priori)
## Decisions
## Runs
| Backend | Job ID | Description | Status | Artifacts |
|---|---|---|---|---|
## Results
### Outcomes against preregistered predictions
| Prediction | Verdict | Predicted | Observed |
|---|---|---|---|
## Conclusion
## Follow-ups
## Artifacts
## Findings

The guide explains how the record develops and how its sections connect design to evidence.

experiments / status · schema

Status vocabulary

abandoned is a valid exit from any state. Use in-progress for active work outside a job queue and blocked for work awaiting a condition named on the status line. Write the token as **Status**: completed, optionally followed by a parenthesized qualifier. Legacy spellings complete (for completed), done (for completed), cancelled (for abandoned), canceled (for abandoned) validate with a warning. closed is rejected because it means abandoned in some records and completed in others.

The validator reads canonical, accepted legacy, and rejected statuses from the schema.

experiments / estimands · source

Estimands and registration

Give each estimand a heading matching ^#{2,4}\s+E\d+\b. The first **Registration**: line after it records the required registration status.

### E1 Derived-belief flip rate when visibility is swapped

**Registration**: registered. Thresholds 0.6 (confirms) and 0.4 (null) fixed in the Decision rule before wj3711.

### E2 Stated-belief invariance to an added visibility clause

**Registration**: registered. Threshold 0.8 fixed in the Decision rule.

### E3 Derived and stated clean accuracies

**Registration**: gate. Sanity readouts on the unmanipulated conditions.

The release's “before the data existed” rule is stronger than fixing an analysis before inspecting previously collected data. For an existing dataset, record collection and access chronology separately and obtain human review of its intended evidentiary use. A registration label alone does not establish independence from selection. The data reuse workflow explains what to record; the confirmation workflow explains what a fresh evidence source must preserve.

Estimands in one record can have different registration statuses. A registered arm can become uninterpretable while a gate produces useful diagnostic evidence. The validator accepts records that declare no estimands. A claim cites an estimand as EXP-NNN:E#, and the validator warns when a claim draws on a found estimand or on a gate.

experiments / decisions · source

Design decisions

The design sections are amended in place during review, so they show the option chosen and lose the one that was not. Arm order, the comparator, a threshold, a branch of the decision rule: these are the choices reviewers most often contest. Keep each in ## Decisions with the same rules as a plan's log: name the option that lost, state the cost, cite what settled it, and never edit an entry. A deviation of the executed method from the design is also an entry, dated when it was decided.

## Decisions

- **2026-08-14** — Kept the 0.02 margin rather than tightening it to 0.01 after
  [[EXP-001-accumulation-pilot]] came in at 0.004: a margin chosen after seeing
  the pilot would depend on the result it is meant to judge. Costs sensitivity
  to differences between 0.01 and 0.02. Source: pilot-gate review.
- **2026-08-14** — Reused seed 1 rather than drawing seeds 4 to 6: the paired
  comparison stays anchored to the pilot. Costs one fresh seed; E1 is a
  replication at seed 1 and a fresh test only at seeds 2 and 3. Source:
  pilot-gate review.

An entry made after results were seen links what had been seen, even when it did not bear on the choice. ## Informed by records what the designer had seen before the design was fixed; a choice made after that is recorded only in its ## Decisions entry.

experiments / results

Separate execution evidence from scientific evidence

Label the kind of evidence beside each check or statistic.

Report the primary statistic first, including a missing or void result. A gate failure explains why the linked statistic is uninterpretable; a secondary or found estimate does not replace it. If a run only authenticates an artifact or exercises the pipeline, state that limit in its conclusion: Execution check only: artifact authenticated; no primary scientific statistic was produced.

experiments / workflow

Register and run one experiment

Reserve a stable ID such as EXP-012. Record the instrument, data, conditions, primary and secondary metrics, code revision, key command, preregistered predictions, and decision rule before inspecting outcomes. Pause for design review before preregistration and for preregistration review before execution. Attach backend job IDs and immutable artifact locations to the same experiment record. After analysis, obtain interpretation review before recording a synthesis or approving a follow-up.

Name each quantity the experiment measures as an estimand, E1, E2, and so on, and say whether it is registered, found, or a gate. Then list, under ## Informed by, what was read or measured before the design was fixed, naming any unregistered facet of an earlier run.

## Estimands

### E1 Off-diagonal transfer (R_belief to reality, R_reality to belief) over the random floor

**Registration**: registered. The 2×2 transfer and the two labels (dissociated,
frame-agnostic) were fixed in the Method; low versus high was not given a number.

### E2 Clean readout accuracies

**Registration**: gate. Both at or above 0.8, or the model is not holding the
conflict and E1 is uninterpretable.

### E3 Bare-query default to reality

**Registration**: found. A diagnostic side-result that became evidence for
reality as the unmarked frame.

## Pre-registered predictions (a priori)

- **P1 (E1)**: dissociated at the value site, the paper's framing.

### Outcomes against pre-registered predictions

| Prediction | Verdict | Predicted | Observed |
|---|---|---|---|
| P1 (E1) | refuted | low off-diagonal transfer | off-diagonal 92–95% over floor on Qwen 7B, 14B, and 7B-Instruct; 89% Mistral-7B; 88% OLMo-2-7B |
| E2 gate | failed zero-shot, met few-shot | both readouts at or above 0.8 | reality readout 0.50 / 0.41 / 0.47 zero-shot at 7B / 14B / 32B; 0.98 to 1.00 with few-shot |

experiments / finished run

Process a result

Finish an experiment by validating its artifacts, recording the result, and applying its decision rule. These steps apply to cluster jobs, remote commands, and laptop scripts. How the outputs are preserved depends on where the run executes.

1. Validate

   Confirm the run produced what it claims. Record artifact authentication, execution checks, and scientific gates separately. A run that returned numbers can still be uninterpretable.

2. Record

   Evaluate the primary scientific statistic first, including a missing or void result. Compare every preregistered prediction with the observed value, then label secondary and found statistics without substituting them for the primary result.

3. Decide

   Apply the predeclared decision rule to successful, failed, and null results. Propose any rescue analysis for separate review.

4. Review

   Ask a human to review exclusions, uncertainty, interpretation, and the proposed follow-up, then update the indexes and claim pointers the result changes.

5. Account

   Record each attempt's usage in units in the experiment record, following the resource conventions. Money for the attempt goes in the spend ledger, with pending billing and its follow-up owner, before the processed mark.

6. Secure

   Job backend: retrieve and verify outputs, validate and commit the notebook, then write the processed mark last. Local run: the output is the only copy, so record the numbers before anything can overwrite them.

7. Report

   Use the agreed mode: narrated updates, a pause after each experiment in stepped mode, notebook-only reporting in unattended mode, or a terminal handoff report. A results walkthrough requires a separate request.

An existing processed record signals that the job has already been handled. Verify its evidence links and avoid duplicating the result. Failed jobs can be processed after their failure mode and useful partial artifacts are recorded. For a local run, the experiment record's status serves this purpose.

## Decision rule (a priori)

- Baseline consequent below 0.8: uninterpretable; a capability failure, not a reversal.
- Baseline at or above 0.8, each subspace controls its own readout and cross-effects
  stay low: the dissociation replicates.
- Baseline at or above 0.8 and cross-effects comparable to own-readout effects: the
  dissociation reverses.

### Outcomes against pre-registered predictions

| Prediction | Verdict | Predicted | Observed |
|---|---|---|---|
| Replicates (E1, E2) | confirmed on Falcon3-3B, Qwen2.5-14B, Qwen2.5-32B | own-readout control near 1, low cross-effects, low mean cosine | own readouts 1.00 everywhere; report-to-consequent 0.49 / 0.42 / 0.04; mean cosine 0.17 / 0.14 / 0.15 |
| E3 gate on Pythia-2.8b | not-tested | baseline at or above 0.8 | baseline 0.486; capability failure |

experiments / resources · current skill source

Resources

Keep per-attempt resource evidence beside Runs, optionally under ## Resources. Record units, never money: compute quantity and device, elapsed time, calls and tokens, and retained bytes. Money for each attempt is a row in plans/spend/LEDGER.md, keyed by the same attempt identity; ceilings are in plans/spend/AUTHORITY.json. The spend reference gives both formats.

The current validator reports a currency amount in an experiment record as an error.

experiments / conditional work

Follow-up records

Put prospective outcome branches under ## Decision rule (a priori) and concrete result-driven proposals under ## Follow-ups. A checkbox records work to consider or track; it does not record approval. A gated follow-up needs an explicit review decision before execution.

## Follow-ups

- [ ] Evaluate the long-document hypothesis on independent evidence.
  Trigger: the found length breakdown in this experiment.
  Before execution: review the proposed estimand, design, reserve,
  and resource limit; record the decision and approved scope.
  Current disposition: proposed, not authorized.
  If adopted: link the follow-up record here and cite this
  experiment under its Informed by section.

These indented fields are suggested prose labels; the validator does not check them. Link deferred work to its plan or priority entry when one exists. The guide to gated follow-ups explains their relationship to scientific gates and experiment completion.

experiments / siblings · source

Annexes and duplicate IDs

Move large breakdown tables into a sibling annex named EXP-NNN-topic.annex.md when they crowd the experiment's argument. The ID in its filename links it to the experiment. Give the annex no status or index entry. Link it from the experiment's results and open it with an explanation of what the breakdown shows.

If two experiments share an ID, renumber the one that never ran, add a revision-letter suffix to a follow-up, or merge a design stub into the executed record. If both ran, renumber one and retag its jobs in the same change. Otherwise, job links can silently resolve to the wrong experiment. When retagging is too costly, add a banner identifying both records. Update every such banner when either record's ID changes.