LLM Tracking: How It Works, What to Measure and How to Choose a Tool

LLM tracking is monitoring how AI assistants such as ChatGPT, Gemini, Claude, Perplexity and Google’s AI Overviews mention, cite and describe your brand. A tracker sends the same set of prompts to each assistant on a schedule, checks every answer for your brand, competitors, links and tone, and reports how often you appear over time. It measures presence in answers, so you pair it with referral traffic and conversions to see what that presence is worth.

This guide covers how the tools collect data, which metrics hold up, how many prompts you need before a change is real, what GA4 can and cannot show, how to choose a tool and how to set tracking up. It also covers the parts most tool roundups skip: the other meaning of the term, the sample size math, debugging bad data, and a worked example that runs from visibility to revenue.

What is LLM tracking?

A large language model (LLM) is the model that writes the answer. ChatGPT, Gemini, Claude and Perplexity are apps built on top of such models, with web search and other features added. When people say “LLM tracking” in marketing, they mean tracking those apps’ answers about a category: does the assistant name you, link to you, describe you correctly, and recommend you over a rival?

It exists because classic tools do not see these answers. A rank tracker records where your URL sits for a keyword on a results page. An assistant does not show a fixed list. It writes a new answer each time, and that answer can differ by wording, location, account and model version. Your page can rank first on Google and still be missing from the AI answer to the same question.

Two different things are called LLM tracking

Before you compare tools, check which job you have. Google’s own AI Overview for this query asks whether you want brand visibility or cost and token tracking, and several vendor pages blur the two.

If you landed here to monitor an app you built, you want an observability tool. The rest of this guide is about the marketing meaning.

How LLM tracking tools work

Every tracker follows the same loop, with different choices at each step:

1. Choose inputs. Most tools ask you for prompts. Some take your SEO keywords and expand each one into several conversational prompts (LLMrefs works this way). Others start from buyer personas and generate the prompts those people would type (Gumshoe’s approach).
2. Run them. The tool sends each prompt to each assistant, in each chosen country and language, on a daily or weekly schedule.
3. Parse the answers. It detects brand and competitor names, linked citations and their domains, where in a list you appear, and the tone of the sentence around your name.
4. Aggregate. It turns many answers into rates, share of voice and trend lines, often with a proprietary score on top.

API collection vs interface collection

This is the method question to ask every vendor. Results can be pulled through a model’s API or by loading the consumer app the way a person would.

Neither method sees a logged-in user’s memory or past chats, so no tool reproduces an individual buyer’s answer exactly. Treat every tracker as a sample of typical answers, not a copy of what each person saw.

The metrics that matter

Track mention and citation rate separately. An assistant can name you without linking, or cite your page while recommending someone else. For the scoring math behind vendor dashboards, see our breakdown of the AI visibility score.

Where the top guides disagree

Is “position” in an AI answer meaningful?

Some trackers sell average position as a core metric, like a rank. Other guides argue that because assistants reorder their lists from one run to the next, a position is close to meaningless and only the share of answers that include you counts. Both have a point. Position from a single run is noise. Position averaged over many runs of the same prompt is a usable secondary signal, as long as the mention rate is your primary one. Never report a single “we rank #2 in ChatGPT” screenshot as a result.

What does each tool cost?

Review articles in the current results list different prices, prompt limits and included engines for the same products. Vendors in this category change plans often, and many charge per prompt, per answer credit or per extra engine. Use list articles to build a shortlist, then read the price and the date on the vendor’s own pricing page. Check three things: how many answers per month your prompt set needs, which engines are add-ons, and how much history you keep.

How many prompts you need before a change is real

Guides say you need a “statistically meaningful” sample. Here is what that means in numbers. A mention rate is a proportion, so its rough 95% margin of error follows the standard formula:

Here p is your mention rate and n is the number of answers behind it. The example below is illustrative, using a mention rate of 30%.

Comparing two periods is harder, because both carry error. With 400 answers in each month and a rate near 30%, the change has to be about 6 points or more (1.96 × √(2 × 0.21 ÷ 400) ≈ 0.064) before you can call it real. With 100 answers per month, a jump from 30% to 36% tells you nothing.

Then work out how many answers your setup produces. Again illustrative:

Two caveats. Repeated runs of the same prompt are not fully independent, so treat these margins as the best case. And the sample you need depends on the level you report at: a brand-wide number settles quickly, while a single prompt on a single engine never will. Report at the level your sample supports.

The three layers of LLM tracking

Most tools cover only the first layer. A complete setup watches all three, because each answers a different question.

For layer 2, the assistants publish their user agents. OpenAI’s crawler documentation separates OAI-SearchBot (used to show sites in ChatGPT search), GPTBot (used for model training) and ChatGPT-User (fetches a page when a user asks, and robots.txt rules may not apply to it). Perplexity’s crawler documentation makes the same split between PerplexityBot and Perplexity-User. Search your logs for these names: a page the search bots never fetch is unlikely to be cited.

What GA4 can and cannot tell you about LLM traffic

A common complaint from marketers: GA4 shows visits from ChatGPT or Perplexity, but not what people asked. That is how it works, and no setting changes it.

So the landing page is your best clue to intent. A visit that lands on your comparison page came from a comparison question. A visit to a deep how-to article came from a problem question. Build a custom channel group or segment for assistant referrers, then report by landing page. Our guide to tracking conversions from ChatGPT, Perplexity and Gemini walks through the setup. If you want that layer without building it, SEOConversion identifies AI assistant visits from referrers and supported UTM signals and reports their conversions and value by landing page. It does not track prompts or what assistants say, so it complements a prompt tracker rather than replacing one, and visits with no referrer stay Direct.

Worked example: from visibility to revenue by landing page

All numbers here are illustrative, for a made-up B2B software company. Its prompt tracker shows a 22% mention rate overall: strong on “best tool for” prompts, weak on “X vs Y” comparisons. Is the comparison gap worth fixing? Visibility alone cannot say. Here are its AI referral visits for one quarter, with each demo request valued at $500 and each newsletter signup at $20 (see how to calculate conversion value for setting your own figures).

The blog post gets over half the AI visits but under 7% of the value ($580 of $9,580). The comparison page earns $12.50 a visit even though the brand is weak on comparison prompts. That points the work at comparison prompts: if better coverage there added another 240 visits at the same rate, that would be about $3,000 a quarter. Raising the overall mention rate by pushing more how-to citations would mostly add $1-a-visit traffic. The prompt tracker tells you where you are missing. The landing page report tells you which gaps pay.

LLM tracking tools by category

The tools named most often across this year’s roundups fall into five groups. Within a group, the differences are mostly engine coverage, prompt limits, collection method and price.

How to choose an LLM tracking tool

Run two shortlisted tools on the same frozen prompt set for a month, and spot check 10 answers by hand in each. Keep the tool whose numbers match what you see, not the one with the best-looking score.

How to set up LLM tracking in six steps

1. Decide the question. Category visibility, answer accuracy, competitor movement or revenue from AI visits. One primary goal decides prompts, engines and tools.
2. Build and freeze a prompt set. Write 30 to 50 prompts across four intents: category (“best X for Y”), comparison (“X vs Y”), problem (“how do I fix Z”) and branded (“is X good for Y”, for accuracy). Source them from Search Console queries, sales calls, support tickets and forum threads. Save the list with a version date.
3. Choose engines and locations. Track the assistants and countries your buyers use, and read each engine on its own, because each cites different sources.
4. Add competitors and set the cadence. Add three to five rivals. Run each prompt several times per period so your rates rest on enough answers (see the sample size section).
5. Connect traffic and conversions. Report AI referral visits by landing page, with conversions and value. Without this layer you can grow visibility and never know whether it paid. The SEO conversion tracking guide covers the same landing-page method for organic search.
6. Review monthly, refresh deliberately. Act on gaps and wrong facts each month. Change the prompt set only on a schedule, such as quarterly, and note the new version so trends stay comparable.

The free route uses the same steps with a spreadsheet: one row per prompt, engine and run, with columns for mentioned, linked, position, tone and cited domains. Use a fresh or logged-out session each time.

When LLM tracking data looks wrong: debugging

Acting on what you find

• Missing from category prompts: look at the cited source domains. The review sites, comparison articles and forum threads assistants cite are where you need a presence.
• Wrong facts in branded answers: update the pages that state them, then the third-party pages the assistant cites for those claims.
• Cited but not named: make your brand and product name explicit in the passages that get quoted, not only in the logo and header.
• Never cited: confirm the search bots can fetch your pages. Allowing OAI-SearchBot while blocking GPTBot is a valid choice: per OpenAI’s documentation the two settings are independent.

For the content side of earning mentions, see how to rank in ChatGPT.

If you meant AI tracking you

Some people search this term because they want the opposite: to limit what AI services record about them. That is a privacy question. In each assistant you use, open the privacy or data settings and turn off use of your chats for model training, use temporary chats for sensitive topics, and delete history you do not need. On your phone, review which apps can access your microphone, location and contacts. Brand LLM tracking tools do not see individual users’ chats. They only see answers to prompts the tool sends itself.

Frequently asked questions