Methodology

How Spyglasses Research studies are designed, run, and reported. This page is a public commitment; every published study links back to it.

Pre-registration

Hypotheses, the smallest effect size of interest (SESOI), the model, and the decision rule are written down and frozen — with a recorded git commit hash — before the study data is assembled. Only count-level data-quality checks happen pre-freeze. Anything added after freeze is labelled exploratory in the write-up. Specs are public in the experiments/ directory.

Null results are claims, not shrugs

A non-significant p-value is not evidence of absence. When we claim “X has no effect”, we mean a TOST equivalence test passed: the 90% confidence interval sits entirely inside a pre-registered SESOI band. When the interval is too wide to conclude either way, we say inconclusive — and we publish that too.

Controls, or the study stops

• Every study carries a positive control — a relationship that must show up if the data can detect anything. If it fails, the study stops; a null on the primary hypothesis would be meaningless.
• Every study carries a placebo control — a variable that cannot plausibly matter. If it “matters”, our standard errors are too small and the results are not reported as-is.

Selection effects are modelled, not ignored

Observational citation data is conditioned on retrieval: we only see sources an assistant already surfaced. Studies control for the selection criteria (e.g. semantic similarity) so that conditioning on the sample doesn’t manufacture spurious correlations, and every article states which stage of the funnel the design actually measures.

What we publish

• The full pre-registered spec and analysis code (public repository).
• Watermarked figures generated by the analysis pipeline — no hand-drawn charts.
• Anonymized datasets of derived features wherever possible, each with a datasheet and license. Customer prompts, AI responses, and customer identifiers are never included — see the data policy.
• A “What we can and cannot claim” section in every article, including the achieved equivalence bound (“we could have detected an effect of X; we found none larger than Y”).

Corrections

Mistakes get fixed in place with a dated changelog entry on the article. If a conclusion changes, the correction is announced wherever the original was.