We pre-registered 'prompt phrasing barely matters.' ChatGPT proved us wrong.

Spyglasses Research · July 23, 2026 · spec & code

Key findings

• The market leaders are phrasing-proof. Sony appeared in 90% of answers, Bose in 82%, Sennheiser in 80% — across 143 differently-worded prompts asking the same thing, every day for a week. This replicates SparkToro’s headline result on a single platform.
• But rewording the question costs real overlap. Two runs of the same prompt agree on 74% of recommended brands (Jaccard 0.74); two differently-worded prompts with the same intent agree on only 54%. The gap — about one brand swapped per answer — is twice our pre-registered “too small to matter” bound. We pre-registered the opposite expectation and are reporting the reversal.
• The retrieval layer moves even more than the answers. Rewording shifts the grounding searches ChatGPT runs (token overlap 0.51 → 0.25) and the sources it cites (domain overlap 0.43 → 0.30) — yet the citation pool is tiny and shared: six review domains absorb 78% of all citation slots in this study.
• Much of the “phrasing effect” is really sub-intent. Naming a specific dollar budget swings Bose −36 points and JBL +31; music framing lifts Sennheiser +23; travel framing lifts Anker +23 (all 90% CIs exclude zero; exploratory).
• For anyone measuring AI visibility: variety beats repetition. In simulation on our grid, 70 phrasings run once estimate a brand’s true mention rate about twice as accurately as 10 phrasings run daily for a week — the same total cost.

Why we asked

In January 2026, SparkToro ran a clever study. Instead of guessing what “the typical prompt” looks like, they surveyed 142+ real people for the prompt each would personally use to ask an AI for headphone recommendations, ran each one, and found the same top brands in 55–77% of answers despite the prompts sharing almost no wording. Their question was existential for the AI-visibility category: if AI answers are nondeterministic and everyone words their prompts differently, can a tool that tracks a sample of prompts produce data that generalizes at all?

Their design had one confound they acknowledged: each prompt ran once, spread across seven different LLMs. Phrasing variation and platform variation were mixed together, and with a single run per prompt, there was no way to measure the baseline — how much the same prompt varies when you simply ask again.

So we re-ran their instrument with the confound removed, in the other direction: one platform, repeated over time. SparkToro generously shared their de-identified survey prompts, and we ran each usable headphone prompt (143 evaluated in this study) against ChatGPT once a day for seven days, plus their second intent — 143 prompts seeking a coffee-shop brand-design agency — once, as a cross-intent control. That design separates three things their study couldn’t: run-to-run noise (same prompt, different day), phrasing effects (different prompt, same intent, same day), and intent effects (different intent entirely).

We also measured a layer they couldn’t see. ChatGPT with web search answers commercial questions by rewriting the prompt into search queries, retrieving sources, and composing over them. Our scraper exposes those grounding queries and citations, so we could test whether the pipeline between prompt and answer is stable even when wording varies. Our pre-registered prediction — the mechanistic bet the whole study was built around — was that query rewriting normalizes phrasing: differently-worded prompts would funnel into the same searches, the same sources, and therefore the same brands, with rephrasing costing no more overlap than the model’s own noise.

That is not what we found.

Methods in brief

The unit of analysis is one ChatGPT run: 1,144 runs evaluated in this study (143 headphone prompts × 7 daily waves, plus 143 coffee-agency prompts × 1 wave), collected 16–23 July 2026 via DataForSEO’s LLM scraper (en-US, web search forced, model gpt-5-5 throughout). From each answer we extracted the recommended brands (frozen alias lexicon), the cited source domains, and the grounding-query token set.

For every pair of runs we computed set overlap (Jaccard), in three pre-registered conditions: within-prompt (same prompt, different days), between-prompt (different prompts, same intent, same day), and cross-intent (headphones × coffee). The pre-registered test is equivalence (TOST logic): is the within-vs-between gap smaller than 0.10 Jaccard — roughly one brand swapped in half of pairs — the smallest difference we judged a marketer would act on? Inference is by prompt-level cluster bootstrap (90% CIs); a positive control (same-intent pairs must overlap far more than cross-intent pairs) gated the analysis, and a placebo split checked the bootstrap. Everything in this paragraph was frozen before any response data was collected: full pre-registered spec and analysis code.

The gate passed decisively (same-intent pairs cite vastly more similar domains than cross-intent pairs, Δ = 0.289 [0.257, 0.320], permutation p = 0.0002) and the placebo came back exactly null (−0.002 [−0.008, 0.007]) — the instrument works; the results below are not artifacts of a broken pipeline.

Result 1: the leaders don’t care how you ask

SparkToro found their top brands in 55–77% of answers across seven platforms. On one platform, with each prompt run seven times, we see the same shape, slightly stronger: Sony 90%, Bose 82%, Sennheiser 80%. However 143 different people worded the question, the podium barely moves. This is the part of the consistency story that fully survives our stricter design — and for the leading brands, it survives per phrasing, not just on average: Sony appears essentially every day for 87% of individual phrasings.

Result 2: but rephrasing costs real overlap — we predicted otherwise

Two runs of the same prompt on different days share a mean brand Jaccard of 0.736. Two differently-worded prompts on the same day share 0.537. The gap, Δ = 0.200 [0.162, 0.239], is double our pre-registered equivalence bound of 0.10 and the confidence interval sits entirely outside it: by the decision rule we froze before collection, this is a real phrasing effect, not equivalence and not noise. At the typical four-brand answer, it amounts to roughly one brand swapped in or out because the question was worded differently.

We want to be plain about this: we designed this study expecting to publish the opposite — “prompts vary, answers vary, but the recommendation set holds.” The equivalence machinery was built to make that null meaningful (“we could have detected a 0.10 shift and found none”). Instead it detected a shift of 0.20, and the same machinery makes the reversal just as rigorous. Every robustness check agrees: rank-sensitive overlap (Δ = 0.191), dropping the first wave (0.205), deduplicating near-identical prompts (0.203).

Where does the change land? Not on the podium — on the shortlist’s back half. Apple is the cleanest example: 22% of phrasings include it essentially every day, 31% exclude it every day, and 47% flicker. Whether a mid-tier brand appears at all is, to a striking degree, a stable property of how the question is worded — which is why the effect can be large for individual brands while the leaderboard looks serene.

Result 3: the pipeline underneath moves even more

Our mechanistic prediction was that ChatGPT’s query rewriting would normalize phrasing before retrieval. The retrieval data says otherwise:

• Grounding searches: same prompt re-run, token overlap 0.511; reworded prompt, 0.245. Δ = 0.266 [0.237, 0.295] — the largest phrasing effect in the study, and it strengthens (0.335) when restricted to runs with complete fan-out data. The user’s wording visibly leaks into the searches the model runs.
• Cited domains: 0.429 within vs 0.298 between, Δ = 0.132 [0.104, 0.160]. Cited URLs are noisier still (0.282 vs 0.119).
• Answer text (descriptive): TF-IDF cosine 0.381 within vs 0.224 between — wording churns freely at every layer.

Hold both of these at once and the picture resolves: the citation universe is an intent-level fact — six review domains take 78% of all citation slots, and cross-intent overlap is ~0.01 — but each answer cites only about two sources sampled from it. Rewording the prompt changes the grounding searches, which changes which members of the same small pool surface. Phrasing doesn’t change where ChatGPT looks; it changes what ChatGPT picks.

One thing did hold perfectly steady: time. Same-prompt overlap is flat across the week (0.74 at a one-day gap, 0.72 at six), and between-prompt domain overlap measured across different days (0.292) matches the same-day figure (0.298). Within this week, on this intent, there was no day-to-day drift for a tracker to chase — the variance lives in phrasing, not in the calendar.

Exploratory: the phrasing effect is largely sub-intent — and you can name the levers

Everything from here down is post-hoc, exploratory, and labelled as such — coded after we saw the pre-registered results, tested without multiplicity correction, on slices of 16–30 prompts. Treat it as strong hypothesis generation, not settled fact.

Reading the prompts (which are SparkToro’s survey data and stay private), the phrasings differ in legible ways: some name a dollar budget, some describe the recipient, some specify a use case or form factor. We regex-coded seven such attributes and asked whether they explain which brands swap in and out.

Naming a specific dollar amount is the single strongest lever we found: Bose drops 36 points [−50, −23], Sennheiser 17 [−32, −2], while JBL gains 31 [+19, +43]. The premium default gives way to a value shortlist — same intent, same product category, different market. Naming a gift recipient moves the same direction (Bose −19, JBL +14). Use-case wording picks favorites: music framing lifts Sennheiser +23 and Sony +15; travel framing lifts Anker +23.

Honesty about the hollow dots matters as much as the filled ones: mentioning noise-cancelling moved no brand detectably (it’s table stakes for this intent), movie-vs-music usage did nothing for Apple, and the output-format attributes (asking for a table, “give me 5 options”) were too rare in this survey to test at all. And a structural check says sub-intent is most of the phrasing story, not all of it: prompts whose attribute profiles match overlap at 0.57 — recovering roughly half to two-thirds of the 0.20 within-vs-between gap — but still short of the 0.74 same-prompt ceiling. Wording retains some irreducible influence beyond what it says.

Why this matters: “phrasing matters” sounds like bad news you can’t act on — you can’t survey every customer’s inner monologue. Sub-intent is different. Budget-framed, recipient-framed, use-case-framed queries are enumerable segments with their own leaderboards, and a brand can know exactly which segments include it and which shut it out.

Does sub-intent reach the retrieval layer?

If sub-intents really are separate markets, they shouldn’t just produce different answers — prompts sharing a sub-intent should also converge in the layer underneath: the grounding searches ChatGPT runs and the domains it cites. We checked, by splitting same-day between-prompt pairs on each attribute (both have it / one does / neither does). The answer is yes, with a mechanism split we didn’t anticipate.

Frame attributes converge — retrieval included. Two travel-framed prompts share more grounding-search tokens than travel/non-travel pairs (0.29 vs 0.16, Δ = +0.134 [+0.100, +0.170]), more cited domains (+0.093 [+0.047, +0.139]), and more brands (+0.110 [+0.039, +0.193]). Music framing converges on brands and domains too. Noise-cancelling is the neat edge case: ANC prompts run detectably more-similar searches (+0.056 [+0.012, +0.106]) yet end at indistinguishable brand sets — the retrieval funnel narrows and then lands on the same review pages everyone else gets, which is exactly what “table stakes” should look like in the data.

Valued attributes fragment. Budget, recipient, and form-factor pairs run the opposite direction: two budget-stating prompts overlap less on brands (0.37) than two budget-free prompts (0.58). The resolution is that the flag isn’t the sub-intent — the value is. Among budget prompts with a parseable amount (23 of 24, ranging 50–500 dollars), pairs in the same price bucket overlap at 0.45 versus 0.29 for pairs in different buckets (Δ = +0.167 [+0.055, +0.294]). A $100 prompt and a $400 prompt aren’t one “budget segment”; they’re two different markets. Attribute-free prompts, meanwhile, collapse onto the shared default answer — which is why “neither” pairs post the highest brand overlap in the whole analysis.

For a brand, this sharpens the segment picture from Result 3: the citation pool and grounding behavior aren’t just intent-level facts, they’re sub-intent-level facts. A travel-framed query and a budget-capped query reach into measurably different search-and-source funnels before a single brand is named. Same caveats as everything in this section: post-hoc, small slices, no multiplicity correction.

The pulse model: how to measure without fooling yourself

The practical question underneath SparkToro’s study — and underneath every AI-visibility tool, ours included — is a sampling problem: what can a finite prompt panel tell you about your brand’s true presence? Our grid (143 phrasings × 7 days) lets us answer by simulation: draw a panel of k phrasings, run it for d days, and see how far its estimate lands from the whole-grid truth.

The curve shape follows from Result 2: because a phrasing’s inclusion propensities are largely stable across days, re-running a fixed panel averages away only run-to-run noise — between a third and two-thirds of the variance (depending on the brand) is phrasing-level and never shrinks. At equal cost of 70 runs, 70 phrasings once beats 10 phrasings daily, roughly ±7 points to ±15. A small fixed panel run daily converges quickly, confidently — on its own rate, which is not necessarily the market’s.

The same simulation says the fixed panel is the right tool for the other job: since day-to-day drift was absent all week, a small consistent panel is a clean yardstick for change — exactly what repetition can measure and a one-shot sweep cannot. Two instruments, two questions. That suggests a pulse design:

• The daily panel — a small, fixed, deliberately stable set of phrasings — tracks drift: did a model update, a competitor’s content push, or your own work move anything? Its absolute level is not the point.
• The periodic sweep — a large, phrasing-diverse set, stratified across the sub-intent axes above (price-framed, recipient-framed, use-case-framed…), run once — measures your true mention rate, per segment, at a fraction of the cost of tracking those prompts continuously.
• The sweep calibrates the panel. If the sweep says 45% and your panel says 60%, your panel runs 15 points hot — a known offset you apply to every subsequent daily reading. And because sub-intent effects are near-deterministic, the sweep doubles as a discovery instrument: the phrasing segments that reliably exclude you are your content and PR target list, and the panel then tells you whether the interventions moved the needle.

Repeat the sweep when the model version changes or the panel’s drift line jumps — not on a calendar.

What we can and cannot claim

• Scope, always: one intent (headphones-as-travel-gift), one platform (ChatGPT via DataForSEO’s LLM scraper — not the logged-in consumer product), one locale (en-US), one week, web search forced. The coffee arm exists only as a control. None of this licenses claims about Gemini, AI Overviews, other intents, or personalized ChatGPT.
• The pre-registered results are the reversal of our stated expectation, reported under the decision rule we froze in advance. We could have detected a within-vs-between gap of 0.10 Jaccard; we measured 0.20 for brands, 0.13 for cited domains, 0.27 for grounding searches, each with its 90% CI wholly beyond the band.
• “Phrasing effect” ≠ “chaos.” The same data shows leader-brand presence is phrasing-robust and the citation pool is intent-level. The effect concentrates in mid-tier shortlist membership and source selection.
• Everything sub-intent and pulse-model is exploratory. Post-hoc coding, small slices, no multiplicity correction, simulation on a single observed grid. The budget→Bose/JBL flip is large enough that we’d bet on its direction; the exact magnitudes need a pre-registered follow-up.
• Run-to-run baseline is composite. We cannot separate ChatGPT’s own nondeterminism from scraper-infrastructure variation; that composite is our baseline, and it is the same baseline the phrasing effect beat.
• One collection artifact: on the final night, 74 of 143 responses returned without model-version and grounding-query metadata (answers and citations intact). All grounding analyses handle this under the pre-registered missing-data policy, and the restricted refit strengthens rather than weakens the grounding result. Logged in the spec’s deviations.

What this means if you track a brand in AI answers

• If you’re the category leader, breathe. Your presence is the most phrasing-robust fact in this dataset. Track drift; don’t chase individual answers.
• If you’re mid-tier, phrasing segments are your battleground. Somewhere between “appears for every phrasing” and “appears for none” there are enumerable sub-intents — price-framed, recipient-framed, use-case-framed — each with its own leaderboard. Find the ones that exclude you; that’s the work list.
• Get into the citation pool before worrying about any single answer. Six domains carried 78% of citation slots in this study. Presence on the RTINGS-tier sources for your category is an intent-level asset that individual answer variance can’t take away — and expecting to appear in every answer is unrealistic even for the same prompt asked twice (0.74 self-overlap).
• Judge any tracking tool — including ours — by its phrasing diversity, not its run frequency. A mention rate from a handful of fixed prompts is a precise measurement of those prompts. The pulse design above is how we’d spend a fixed budget: a small panel for drift, a broad stratified sweep for truth, and the offset between them made explicit.

Data and reproducibility

This study exists because SparkToro (Rand Fishkin) designed the original survey and shared their de-identified prompts — the rare dataset of real people’s own words for a commercial intent. Read their original study; our prompts are theirs and are not ours to publish. Researchers who want them should contact SparkToro directly.

• Dataset: Prompt-phrasing consistency of ChatGPT recommendations (derived features) (CC BY 4.0) — one row per run evaluated in this study: brands, cited domains, counts. No prompt text, no answer text, no grounding-query text. Column dictionary in the datasheet.
• Pre-registered spec: frozen at 2ee335a, including all deviations logged during collection.
• Analysis code: experiments/002-prompt-consistency/ — pipeline stages 00–05 are the pre-registered analysis; 90–92 are the labelled exploratory extensions.
• Methodology: how we design and report every study — research.spyglasses.io/methodology

Changelog

• 2026-07-23: published.
• 2026-07-23: clarified SparkToro publication date (January 2026); watermarked the pulse-model illustration.
• 2026-07-23: added exploratory retrieval-layer analysis of sub-intents (frames converge, valued attributes fragment).