Methodology
How a claim is taken apart
Every review follows the same sequence, in the same order, whatever the domain. The instruments on this page are synthetic demonstrations of that sequence; they contain no laboratory data and represent no specific programme.
- All review programmes
- Eight, examined in order
- Fail-closed
- Synthetic and illustrative
The reasoning sequence
A claim is not assessed as a whole. It is decomposed into observation, measurement, assertion, model, inference, contradiction, replication, and decision — and each part is examined against what it would take for that part to be wrong. The sequence is fixed so that a favourable result at one stage cannot be used to excuse a gap at another.
Observation
A recorded event or feature, before any interpretation is attached to it.
- An artefact of the recording chain is recorded as a property of nature.
- Raw records, acquisition conditions, and a configuration in which the feature must be absent.
- We ask which recorded features survive a change of instrument pathway.
What would decide it?
The most useful output of a review is usually not a judgement but a discriminator: the cheapest observation whose outcome would separate the surviving explanations. The instrument below traces a claim to that discriminator across ten claim classes.
What would decide it?
ConceptualSelect a claim class. The instrument traces the reasoning from the claim, through the explanations that survive the present evidence, to the observation that would discriminate between them and the decision states that follow.
Reasoning chain for the claim class Anomalous signal: claim, competing explanations, existing evidence, missing discriminator, decisive observation, and possible decision states.
- 01
Current claim
A narrow feature above the modelled background indicates a new physical process.
- 02
Competing explanations
- Instrumental line from a known subsystem
- Background mis-modelling in the signal window
- Analysis-path selection producing the feature
- A genuine new process
- 03
Existing evidence
- Feature present in the primary channel across three run periods
- Amplitude stable under two of four defensible pipeline variants
- 04
Missing discriminator
UnconstrainedA configuration in which the instrumental pathway remains active while the claimed process must be absent.
- 05
Potential decisive observation
A null run with the source removed but all readout, timing and environmental pathways unchanged, analysed with the pre-specified pipeline.
- 06
Possible decision states
Unconstrained- Hold — No null configuration has been run.
- Scope — Null configuration designed but background model unvalidated.
- Proceed with conditions — Null clean and one independent channel agrees.
Each stage narrows what the evidence permits. A stage marked unconstrained is where the programme's decision is actually blocked.
Represents structure of reasoning, not a specific calculation.
The instrument does not evaluate a visitor's programme. It shows the structure the Institute applies: no decision state is reachable while more than one explanation remains consistent with the evidence at the stated confidence.
- Claim classes are generic patterns, not descriptions of any specific programme.
- Evidence entries are synthetic and stated at a coarse level.
Where uncertainty is displayed, it is qualitative: it marks which link in the chain is not yet constrained.
The most useful output of an adversarial review is often the identification of the cheapest observation that changes the decision state.
The chain renders in full without animation; stage order and content are unchanged.
Uncertainty and measurement
A measurement without a budget is a number without a meaning. Terms are combined in quadrature only where they are independent, systematic contributions are kept separate from repeatability, and the dominant contributor is named — because that is where effort changes the result.
How uncertainty changes a claim
SyntheticA synthetic measurement chain runs from instrument to interpretation. Adjust the predefined uncertainty terms and observe the combined standard uncertainty, the expanded interval, the dominant contributor, and the interpretive state that the evidence then permits.
Interactive uncertainty budget. Combined standard uncertainty 0.023, expanded interval plus or minus 0.045, dominant contributor Environmental correction, interpretive state: Marginal.
Readout noise, reducible by averaging.
Traceability of the reference used.
Not reducible by additional averaging.
Model used to convert raw to derived quantity.
Uncertainty budget — contribution chart
- Instrument0.008
- Calibration0.012
- Environmental correction0.017
- Transformation0.004
Combined standard uncertainty (quadrature): 0.023 · solid bars: systematic · dashed outline: random
- Combined standard uncertainty
- ± 0.023 arb. units
- Expanded interval (k = 2)
- ± 0.045 arb. units
- Claimed effect size
- 0.052 arb. units
- Dominant contributor
- Environmental correction (systematic)
- Effect ÷ expanded interval
- 1.15
Marginal
The claimed effect and the expanded uncertainty are comparable. A stronger constraint is required before interpretation.
Constructed for demonstration. Contains no laboratory data.
Terms are combined in quadrature, which assumes they are uncorrelated. The chain diagram marks the dominant contributor. The interpretive state is derived by comparing a fixed claimed effect of 0.052 arb. units against the expanded uncertainty at coverage factor k = 2.
- Terms are treated as uncorrelated; a correlated pair would require a covariance estimate.
- The claimed effect size is fixed at 0.052 arb. units for the demonstration.
- Coverage factor k = 2 is used for the expanded interval.
Systematic terms are shown as solid bars; random terms are outlined. Only random terms fall with additional averaging.
Metrological reasoning decides whether a claim is interpretable at all. It is not a formality applied after the physics.
Synthetic chain constructed for public methods education.
With animation disabled the sliders and readouts behave identically; no value is conveyed by transition.
Model against observation
Agreement between a model and a dataset is a statement about the model, not about nature. Admissibility is examined before accuracy: is the formulation consistent, is the problem well-posed, are the parameters identifiable, and is the numerical solution converged?
Model, observation, residual
SyntheticA synthetic observation is compared with a fitted model. The residual view exposes structure that a single goodness-of-fit statistic would conceal.
Model versus observation in raw view. The observation departs from the model progressively beyond channel 0.62, visible as a rising residual.
Solid: fitted model · thin grey: observation · shaded: model band · dotted zero line in the residual view.
Constructed for demonstration. Contains no laboratory data.
In the raw view, the model and the observation appear to agree across the peak. In the residual view, a systematic drift appears beyond channel 0.62, which no change of amplitude can absorb. The alternative model adds a second component; it improves the peak region and leaves the drift in place.
- Synthetic data generated from a fixed deterministic function plus a deterministic oscillatory term.
- The confidence band shown is the model band, not an observational error bar.
The band widens with model amplitude. It does not represent the systematic drift, which is not part of the model.
Agreement is not identity. Residual structure may contain information that a single fit statistic conceals.
Synthetic data prepared for public education. No released research is attached to this figure.
All four views are reachable without animation; switching views redraws the figure directly.
Contradiction preservation
Contradicting records are retained and continue to act on the release state. The figure below includes a suppression control, so the specific failure mode — apparent confidence rising while the physical situation is unchanged — can be seen directly.
Contradiction preservation
ConceptualTwo hypotheses, the evidence attached to each, and the discriminating test between them. Contradicting records remain in the structure and continue to act on the posterior. The suppression control shows what is lost when they are removed.
Structure showing hypothesis A and hypothesis B, 3 evidence items currently included, and the discriminating test. Contradiction suppression is off.
With contradiction retained, the posterior remains broad. The correct output is the discriminating test, not a narrower distribution.
Solid boxes: supporting records · diamond outline: contradicting records · dashed: unresolved records · hexagon: discriminating test.
Represents structure of reasoning, not a specific calculation.
When contradiction is suppressed, the posterior narrows and shifts toward hypothesis A while nothing about the physical situation has changed. This is the failure mode the Institute's contradiction register is designed to prevent.
- The posterior shown is illustrative and not attached to a released case.
- Evidence items are synthetic and stated at a coarse level.
The width of the posterior is a display of confidence under the included evidence only.
Contradiction is information, not an inconvenience to be removed.
Adding or removing evidence redraws the distribution directly; no information is carried by the transition.
Replication topology
Replication is a structure, not a count. An attempt that shares an apparatus, a pipeline, or a critical assumption with the original is dependent agreement. A null result is interpretable only where the sensitivity of the attempt was established in advance.
Replication topology
SyntheticAttempts are recorded as a topology rather than a count. The distinction between failure to reproduce and evidence of falsification is maintained explicitly, as is the distinction between independent replication and dependent agreement.
Replication tree with six nodes: original observation, independent replication, partial replication, failed replication, methodological divergence, and a superseded attempt. Currently selected: S₃ failed replication.
S₃ failed replication
Contradicted- Reference protocol, independent apparatus of a different design.
- Dispersive principle differs from the original instrument.
- No feature above stated sensitivity in the same window.
- Sensitivity established in advance and sufficient to detect the original effect.
- Failure to reproduce with demonstrated sensitivity. This is evidence against the claim — but it is not the same as falsification of the underlying physics.
Constructed for demonstration. Contains no laboratory data.
Node shape and line style carry the classification, so the topology remains readable without colour. A failed replication with demonstrated sensitivity is evidence against the claim; a failed replication without established sensitivity is uninformative.
- Synthetic case; no real study, group, or apparatus is represented.
Sensitivity of each attempt is stated in the node record, because a null result without sensitivity is not interpretable.
Failure to reproduce and evidence of falsification are never treated as synonymous.
The full topology renders statically; selection is available by keyboard through the node list.
Evidentiary notation
Every statement in an Institute record carries the state of the evidence behind it. The same notation is used on this website.
- Observed
- A recorded event or phenomenon reported by an instrument or observer, prior to interpretation.
- Measured
- A quantity produced by a traceable measurement procedure with a stated uncertainty budget.
- Replicated
- Independently reproduced under stated conditions by a group not dependent on the original analysis chain.
- Inferred
- A conclusion reached through reasoning over observations and assumptions rather than direct measurement.
- Modeled
- A quantity produced by an analytic or numerical model, conditional on the model's assumptions.
- Simulated
- A computational realisation of a model. Simulation is not observation and cannot substitute for it.
- Synthetic
- Material constructed for demonstration or education. It contains no laboratory data and describes no real programme.
- Contradicted
- Evidence exists that conflicts with the claim. Contradiction is retained as information, not removed.
- Unresolved
- The available evidence does not discriminate between competing explanations at the stated confidence.
- Superseded
- A later record replaces this one. The earlier record is preserved with its revision history.
- Decision-relevant
- This item materially changes which decision the evidence permits at the current review stage.
All figures and instruments on this page are synthetic constructions built to demonstrate reasoning structure. They contain no laboratory data and represent no specific experiment, group, or result.