Strategy and measurement
Is your AI visibility gain real, or prompt-tracking noise?
Choose prompt breadth and repeat runs for different measurement tasks, and avoid treating a daily visibility swing as proof of GEO impact.
TL;DR
- A higher visibility score is an observation; it does not by itself establish an improvement caused by your work.
- More distinct buyer questions improve coverage, while repeated runs help estimate behavior on the questions you track.
- Keep the prompt cohort stable when comparing periods, and separate branded from non-branded questions.
- Published uncertainty figures depend on their dataset and assumptions; they are not universal significance thresholds.
AI tracking reports need to answer two questions: “How broadly do buyers encounter us?” and “Has our presence changed on a defined set of questions?” The first asks how much of the market you track. The second asks what changed. Mix them up, and a noisy chart can look like a clear business result.
What the large tracking study supports
Peec AI's September 2026 analysis used roughly 120 million chats from February through August 2026 to model uncertainty. Its comparison held monthly runs constant and assumed a mix of 20% branded and 80% non-branded prompts. The figures below are that report's estimates, not measurements from your brand. Peec AI's methods and calculations.
| Design | Monthly runs | Daily 95% half-width | Monthly 95% half-width |
|---|---|---|---|
| 50 prompts, once daily | 1,500 | ±10.5 percentage points | ±6.1 percentage points |
| 10 prompts, five times daily | 1,500 | ±15.8 percentage points | ±13.2 percentage points |
This table concerns how well a score reflects the wider market. Asking a small set more often does not make it cover every buyer need.
The report also asks how large a change must be to detect it on the same set of prompts. That calculation answers a different question.
| Window | Runs | Reported change threshold |
|---|---|---|
| Daily | 50 | 12.4 percentage points |
| Weekly | 350 | 4.7 percentage points |
| Monthly | 1,500 | 2.3 percentage points |
A 12.4 percentage-point change is not a 12.4% relative change. The thresholds also should not be copied into a dashboard with a different prompt mix, dependence between runs, or model behavior.
Why a single uncertainty formula is not enough
NIST documents Wilson intervals for a binomial proportion. This gives a range for a yes-or-no result, when the sample meets the method's assumptions. It cannot fix a biased prompt set, a model change, or answers that depend on each other. NIST's proportion interval guidance is statistical context, not independent validation of Peec's dataset.
For visibility work, an observation could be “the brand is explicitly recommended in this answer.” Define that rule before counting. Do not mix a cited company page, a passing mention, and a recommendation in the same numerator. Then choose the unit: one run, an average for one prompt, or a group of buyer tasks.
A small p-value would not establish business value. A meaningful measurement still needs relevant questions and a link to the decisions your buyers make.
Worked example: a gain caused by changing the question mix
Illustrative example. A dashboard shows a brand in 20 of 50 answers one week and 30 of 50 the next. That is 40% to 60%, a gain of 20 percentage points, or 50% relative to the first score.
The team opens the exported prompt list. Ten neutral comparison questions were replaced with questions that explicitly name the brand. The numerical increase is real within each export, but it is not a like-for-like improvement.
Keep the original prompt set. Rerun it, or compare only results you have for both periods. Put new prompts in a separate view. The report should name the changed denominator, not invent a confidence interval that makes incomparable cohorts comparable.
Use this record for the next review:
Buyer task and market:
Prompt cohort/version:
Platform, model, mode, and search condition:
Period and successful/failed runs:
Outcome rule: mention / citation / recommendation
Matched-cohort result and denominator:
Content changes and possible outside causes:
What we can conclude:
What still needs evidence:
Semanta's role is to connect this record to buyer-question coverage and page changes. When the same constraint repeatedly excludes a brand, we can inspect the source evidence. When only the mix of tracked questions changed, the first repair belongs in the report.
Common questions
Are repeated runs a waste of budget?
No. They help characterize variability and monitor a fixed set over time. They cannot discover buyer needs missing from that set.
Does a stable monthly increase prove the content caused it?
No. Model changes, competitors, seasonality, and demand can move the result. Use unchanged comparison groups when feasible, document interventions, and state the remaining limits on attribution.
If daily swings keep triggering new content work, bring your prompt export and change log to discuss a more reliable GEO measurement plan with Semanta.