You can sit at number one for your core term and still lose the recommendation. You open an assistant, ask for the best tool in your category, and a competitor gets named. The price in that answer matches a docs page you replaced months ago. The rank report from this morning is green. The buying conversation is not.
An AI search visibility audit checks a different question. Did the assistant include you, cite a page that actually supports the claim, and send anyone to your site? A higher mention count can flatter you. A wrong price, repeated often, is worse than silence.
Table of Contents
- Why Rank Tracking Misses What AI Assistants Say
- How to Design Prompts and Run Each Test
- The Scorecard Metrics to Record in Every Run
- Comparing AI Providers and Telling Real Change From Noise
- Checking Whether Citations Actually Support Claims
- Ship Fixes in Priority Order and Retest
Why Rank Tracking Misses What AI Assistants Say
Say a founder still ranks first for the category term in Google. They ask ChatGPT which tool a 20-person team should buy. A rival is recommended, and the founder's pricing is quoted from an old help article. Nothing in a position report flags that.
That miss is now ordinary search behavior, not a niche test. Pew Research Center analyzed browsing from 900 U.S. adults in March 2025 and examined 68,879 Google searches. About 18% of those searches displayed an AI Overview. The feature appeared more often on harder queries: about 60% of question-based searches and about 53% of searches with 10 or more words. Across the broader browsing sample, 58% of respondents encountered at least one search page with an AI-generated summary.
Rankings miss three failures.
- Absence: the brand never appears in the generated answer.
- Omission: the answer discusses the category but never cites you.
- Inaccuracy: the summary names you and gets price, plan, or limits wrong.
For each prompt, log prompt type, answer inclusion, position or prominence, cited sources, competitor mentions, and factual accuracy. Question-style and buyer-intent prompts go first, because those formats are where summaries show up most. The audit runs beside SEO. It doesn't replace keyword tracking. Blue-link rank still matters when someone clicks through. Answer-level visibility is a second channel. Advice firms already separate those surfaces. The notes on AEO and GEO for financial advisors from Advisor Momentum are a useful read on how a regulated category treats being the cited source, not just the ranked page.
You'll leave with four working pieces: a metric scorecard, a repeatable test run, a citation check, and a fix backlog ordered by deal risk.
How to Design Prompts and Run Each Test
Pull language from sales calls, support tickets, and search query data. Don't invent the polished version your homepage uses. Prospects don't talk that way, and assistants are answering the messy version.
Build the set before you touch a provider
- Collect 40 to 60 buyer prompts in the exact wording prospects use.
- Split them into discovery, comparison, alternatives, pricing, implementation, integrations, and problem-specific sets. Each category gets its own result. A blended score hides a broken price answer inside a healthy discovery score.
- Freeze conditions. Same locale, same language, same date window, and the same logged-out or identical account state on every provider.
- Run each prompt in a fresh session. Save the answer text, your position, competitors named, sentiment, and every cited URL.
- Repeat each prompt three to five times across several days. One answer is a snapshot, not a ranking.
- Keep a fixed control cohort you never edit. That cohort is what you re-run after a change. If you rewrite the prompts, you can't tell a content fix from a new question.
Start with discovery and question-style prompts. Those are the formats where generated summaries show up most often, so they tell you fastest whether you're in the answer at all.
Before the first run, build an evidence inventory. List each buyer claim (pricing, integrations, security, limits, use cases). Assign a preferred source, such as product docs, a feature page, a reviews profile, or a partner listing, and give it a graded relevance label. Later checks compare the answer with the page you chose, not with whatever URL the assistant happened to grab.
The same discipline shows up in best practices for prompt engineering. Short, single-intent prompts are easier to score than a paragraph that asks three things at once.
Lines you can paste
- Discovery: "What is the best tool for monitoring brand mentions in AI answers for a 20-person SaaS team?"
- Comparison: "[your brand] vs [competitor] for AI answer visibility, which is better?"
- Alternatives: "Alternatives to [your brand] for tracking AI assistant answers?"
- Pricing: "How much does [your brand] cost per month for a team of 10?"
- Implementation: "How long does it take to set up [your brand] with our CRM?"
- Integrations: "Does [your brand] connect to Slack and HubSpot?"
- Problem-specific: "Why does ChatGPT describe our pricing incorrectly?"
Skip three habits that wreck the read. A tiny sample, often under 15 prompts, swings with one odd answer. Mixing countries or languages in one run makes a locale change look like a content win. Counting any mention as success treats a stale affiliate review the same as your pricing page.
If you want a walkthrough of how a logged run looks in practice, this short demo covers the logging rhythm.
The Scorecard Metrics to Record in Every Run
Each buyer prompt is a retrieval-and-attribution test, not a mention check. Ten measures come out of every run.
| Metric | How to calculate | Healthy trend | Warning sign |
|---|---|---|---|
| Presence rate | Prompts naming your brand divided by prompts run | Rising across four or more runs | Flat while a competitor rises |
| Mean position when present | Average rank of your brand, counted only in answers that name it | Stable or moving up | Consistently last in lists |
| Citation share by domain | Your-domain citations divided by all citations in the set | Growing on buyer-intent prompts | Citations come only from reviews or syndicated copies |
| Supported-claim rate | Claims the cited page backs up divided by claims made about your product | Most pricing and plan claims match the cited page | Any wrong price or plan name |
| Broken or redirected URL rate | Cited URLs returning errors or redirects divided by cited URLs | Near zero | Rising after a site migration |
| Competitor displacement rate | Prompts where a competitor replaced you since the prior run, divided by prompts run | Falling | Rising on comparison and alternatives prompts |
| Recall@5 and Recall@10 | Share of prompts where authoritative evidence appears in the top 5 or top 10 retrieved sources | Rising after content fixes | Below half and unchanged for a month |
| Hallucination rate | Answers containing a claim that contradicts your source, divided by answers | Falling | Any invented feature or integration |
| Volatility score | Share of prompts whose answer or citations changed between runs | Low enough that real changes stand out | Swings larger than the change you shipped |
| Visit attribution | AI-referred sessions by prompt category | Rising on buyer-intent categories | Rising presence with flat AI-referred sessions |
Two formulas teams get wrong
Presence rate is prompts that name you divided by prompts run. Mean position when present averages your rank only inside answers that include you. If you average in the blanks, absence pulls the mean toward the bottom and you can't tell "missing" from "mentioned last."
Citation share by domain is separate from presence. You can be named in most answers and still draw every link from a review roundup or a scraped copy of your docs. Supported-claim rate is claims the cited page actually backs, divided by claims made about the product. A high presence rate with a weak supported-claim rate is a warning, not a win.
Practical rule: Don't score a brand mention as success when the source is unsupported, syndicated, outdated, or broken. That kind of visibility doesn't hold up in a sales call.
A Columbia Journalism Review Tow Center test of 1,600 queries across eight generative search tools found incorrect answers in more than 60% of cases. Perplexity had the lowest failure rate in that test and still answered incorrectly 37% of the time. Treat one run as a snapshot. Compare trends across runs, and report a confidence interval or a bootstrap interval once the prompt count can support it. A single witty answer is not a ranking.
Store enough to recompute later: timestamp, provider, locale, prompt, full answer text, and every cited URL. If those fields are missing, you can't rebuild presence, citation share, or the broken-URL rate after the assistant UI changes. For how teams keep that log over months rather than one spreadsheet, see LLM visibility tracking.
Comparing AI Providers and Telling Real Change From Noise
A single “AI rank” combines systems that retrieve and cite information differently. Track each provider separately, using the same prompts and time window for fair comparisons. A higher mention count can still mean weaker visibility if the added answers cite inaccurate, outdated, or broken pages, or fail to send visits.
| Provider | Why test it | What to log | Known risk |
|---|---|---|---|
| OpenAI ChatGPT | Referral traffic grew 155.52% from October 2024 to February 2025, according to Statista | Mention, cited URLs, whether search ran | A self-contained answer may earn a mention without producing a click |
| Google AI Overviews | The 2025 study reports a 70% probability of change between observations | Overview present, cited links, your position | Results can shift substantially between observations |
| Perplexity | Its answers cite sources, making individual claims easier to check | Each cited URL and whether it supports the claim | In one test, it had the lowest failure rate, yet 37% of its answers were incorrect |
| Grok | Citation failures can take a distinctive form | Citation status codes and page titles | Citations may point to error pages, so log their status codes |
| Claude | Buyers may use it for long side-by-side comparisons | Mention, cited pages, claim wording | Measure it with your prompts rather than borrowing another provider's error rate |
| Microsoft Copilot | It appears in Microsoft search and workplace apps | Mention, cited pages, sentiment | Sentiment may not match the page cited |
| DeepSeek | Buyers may ask questions in another language | Mention, cited pages, language handling | Language changes can alter which sources appear |
Across 24,069 conversations and more than 366,000 citations from OpenAI, Perplexity, and Google, the study found that citations concentrate among a relatively small group of sources, while provider patterns differ. Its Google AI Overviews analysis reports an average cited-content lifespan of roughly 2.15 days and 45.5% new cited sources between observations. These results make a pooled provider score hard to interpret: report changes by provider, and inspect whether newly cited sources accurately support the answer.
When a swing is real
Keep the prompt set and testing window consistent across providers. Treat a movement as meaningful only when it holds across three consecutive runs and exceeds the provider's observed volatility. A smaller movement may reflect source reshuffling rather than an effect from your change, so don't attribute it to a fix on one observation alone.
Traffic provides a separate check. In the March 2025 browsing analysis reported by The Indian Express, users clicked a traditional result in 8% of visits when an AI Overview appeared, compared with 15% when it did not. Links inside summaries received clicks in 1% of visits. A mention can therefore raise visibility without producing a visit. Track attributed visits alongside citation accuracy and mentions, split by prompt category, to distinguish useful exposure from an answer that neither supports its claims nor sends traffic. For a side-by-side guide to comparing AI providers, use the same test criteria across each system.
Checking Whether Citations Actually Support Claims
A citation is evidence to check, not a point on the board. Score each one at three levels: whether the publisher is legitimate, whether the URL resolves to the intended page, and whether that page supports the exact sentence the assistant makes.
A 2025 review in JMIR Medical Education reports that only about 75% of cited claims in generated search results are supported. That makes citation accuracy a separate audit measure from visibility. For a SaaS brand, check whether the cited page substantiates claims about pricing, integrations, security, limits, and use cases. A brand can appear often and still give buyers poor evidence.
What each failure looks like
| Failure type | What it looks like | How to confirm |
|---|---|---|
| Retrieval omission | Your authoritative page never appears among cited sources | Check Recall@5 and Recall@10 for that prompt |
| Source mismatch | A third-party page is cited for a claim it does not contain | Open the cited page and search for the claim text |
| Stale content | Answer quotes old pricing, plan names, or retired features | Compare the quoted detail with your current page and its date |
| Context overload | Correct page is retrieved but the answer mixes in unrelated details | Compare answers with and without the noisy sources |
| Generation misinterpretation | Source supports the claim but the answer states it wrongly | Read the source sentence beside the answer sentence |
| Syndicated or wrong publisher | Cited URL is a copy, scraper, or unofficial mirror | Confirm the domain and canonical URL |
| Broken or redirected citation | Link returns an error or lands on an unrelated page | Check the HTTP status and final URL |
Use concrete claims to diagnose each failure. Retrieval omission means your pricing page is absent while an old review appears instead. Source mismatch occurs when a roundup is cited for a Slack integration it never mentions. Stale content means an outdated plan or feature is presented as current. Context overload occurs when the right document is retrieved, but details from another source distort the answer. Generation misinterpretation means a page says “Salesforce via Zapier” while the assistant claims “native Salesforce sync.” A syndicated citation points to a mirror, while a broken citation returns an error or redirects to an unrelated page.
Publisher fit matters independently of whether a sentence is technically true. A comparison blog is weak evidence for a security claim because buyers and procurement teams may not accept it as an authoritative source.
More mentions can worsen a brand's position when assistants repeatedly cite outdated documentation or affiliate reviews. Each extra appearance can reinforce the same incorrect detail. Map every important claim to its preferred evidence source in your inventory, then compare that source with what the assistant cites. For larger URL sets, a web scraping api can help check status codes and final URLs without opening pages one by one.
Prioritize source quality and claim support, not just citation volume. This guide to reliable citation analysis for AI search engines explains how to assess whether cited sources substantiate the answer. A useful audit record ties each prompt and answer sentence to the cited URL, publisher, and relevant passage. That makes it possible to distinguish a missing source from a misleading one, and to see when a citation exists but does not earn buyer trust.
Ship Fixes in Priority Order and Retest
Prioritize failures by buyer risk, not by how many prompts they affect. A wrong price in a comparison answer can cost more pipeline than a vague description in several discovery answers. Count affected prompts, but weigh each claim by whether it could mislead a buyer or block a purchase.
- Record each failed claim from the citation check: prompt, provider, quoted sentence, and cited URL.
- Rank the failures by commercial risk. Put pricing and integration errors first, followed by implementation and security. Discovery wording can wait unless it changes a buyer's understanding of the product.
- Choose the fix type. Trust covers reviews and third-party listings. Content updates the preferred source. UX makes the answer to a buyer's question easy to find on the page. Technical addresses redirects, canonicals, and crawl access.
- Ship the highest-risk fixes first and log when they go live. Start with a focused change that corrects a live price or named integration before considering a sitewide rewrite.
- Re-run the same control cohort two to four weeks later, keeping locale and account rules consistent.
- Credit a fix only when authoritative evidence becomes easier to retrieve and the answer is better supported. A more polished answer citing the same incorrect URL is not an improvement.
Separate retrieval from the answer
Use a two-stage check to identify which layer failed. First, test whether the preferred source appears near the top of retrieved results. RAG evaluation work summarized in SN Computer Science describes retrieval measures such as Recall@k, mean reciprocal rank, nDCG, and citation-source coverage, followed by checks of factual correctness, faithfulness, answer relevance, and citation quality.
For an ambiguous failure, compare three conditions: no added context, only the correct evidence, and the correct evidence mixed with noisy chunks. If the answer is correct with the evidence alone but wrong when noisy material is added, the context is interfering. If it remains wrong with only the correct evidence, the answer is misreading material you control. This comparison helps distinguish a retrieval problem from a generation problem without assuming a single test proves the cause.
Track Recall@5 and Recall@10, supported-claim percentage, hallucination rate, and source freshness separately. A single pass/fail score can hide a trade-off: retrieval may improve while citation accuracy or freshness declines. Rank alone is also insufficient, and a small prompt sample, mixed markets, or one binary scoring flag can make apparent changes unreliable.
Pair each fix with SEO recommendations suited to its cause. A trust gap calls for stronger third-party evidence, not another internal blog post. A redirect chain needs a technical correction, not a new comparison table.
Name the three prompts most likely to lose buyers and the pages they should cite. Fix those pages first. Re-run the same control cohort, then keep the change only if Recall@5 or Recall@10 improves and the cited evidence supports the answer.
MyMentions tracks prompt-level visibility, position, sentiment, competitors, and the citations behind AI answers, and connects those results with visits. Its workspace brings repeat runs and a prioritized fix backlog together, so teams can review whether visibility changes correspond to traffic.