Premise: The SEO, AEO, GEO, AI Search industry as a whole frequently discusses the components of AI Search and LLM Visibility tracking, but audiences consuming these articles and posts may struggle to effectively integrate the components into a rigorous measurement and optimization framework. They have many parts from which they may not be able to assemble the whole.
This post discusses: Key questions brands should ask about how LLMs access, process, and present the brand and its assets across the entire user journey with the purpose of providing a guided, actionable framework for daily tasks and departmental goals.
From Confusion to Clarity: Better understanding LLM visibility measurement and brand appearance in AI answers
About two years into this AI journey and the leading tools all suggest that “Visibility Score” is how you should be measuring success when trying to optimize your brand’s visibility in LLMs. It sounds intuitive, but what is it actually measuring? Most commonly, it’s the rate at which your brand is mentioned in an LLM’s response across all of the responses the tool received across each run of the prompt. If you ran 100 prompts and your brand was mentioned 30 times, your visibility score would be 30%. Easy peasy, right?
It’s a bit more complicated.
First, this is only tracking against the prompts you entered. So your failure or success is highly dependent on tracking the right prompts. I have a post on that elsewhere, if you have the time and interest.
Second, and equally important, all of these tools are tracking the first response.
Why First-Response Tracking Is Helpful, But Not Adequate
It’s better than no data at all, but at this point most of us know that when we need an actual solution (not just a quick fact or fact check) it frequently requires multiple turns. And as marketers, we’re using the tools significantly more than the general public and as such have a little more familiarity with their shortcomings and nuance. We know that context is essential and when we provide that context we’re more likely to get the info we need in fewer turns. The public is being told to “converse” with LLMs and we don’t typically include a paragraph or two of context when we converse. That means lack of context and a need to provide that context over multiple turns. In a commercial conversation where an LLM user is asking the model for information or recommendations with the intent of making a purchase of some kind, that context has to be provided and refined over multiple turns. The literature says that the median number of turns is about 6, per EMarketer.
Each turn of the conversation causes the LLM to re-evaluate its recommendations against new criteria, often shifting the leading brand in its new response and sometimes introducing new questions and concerns from the user as the verbose nature of LLMs and their eagerness to bring rounded perspectives to conversations tend to cause them to highlight common online discussion points about brands and products and increasing user exposure to facets they may not have already considered. I have a whole post about the conversation tracking gap, so I won’t blather on about it again here. It is, however, crucial to be aware of and the questions I promise in the title will help you gain some insights you may not currently have.
How Should We Approach LLM Visibility Optimization?
I believe that asking a series of questions that address each of the hurdles to LLM visibility provides a clear step-by-step process to LLM visibility optimization. If you’re asking these questions, reviewing the data and taking measurements, and adjusting as necessary before moving on to the next, then you’re very likely to get where you’re trying to go.
This will naturally lead you beyond the user’s turn-one prompt and into their filtering and refinements that ultimately lead to the final set of suggested brands.
I’m listing the full set of questions here, from start to finish. I’m certain it’s not exhaustive, so if you feel there are critical items missing do please DM me on LinkedIn and let me know. For now, let’s start with these.
Essential Questions To Guide LLM Optimization
I’ll break these questions into thematic categories for ease of use and to help build the framework for optimization. Adjust as you see fit and adapt freely to your audience, business, and brand as appropriate.
Audience
Identifying answers to these questions helps you better scrutinize your Google Search Console and Bing Webmaster Tools data to find high-value phrases you can turn into prompts, as well as help you avoid tracking less valuable prompts that may skew your data.
Q: Who Is My Audience?
This is a critical starting point. You should be able to break your larger audience down into sub-segments with their own unique needs, barriers to purchase, demographics, and a any other pertinent facets.
Q: What Are Their Unique Needs, Challenges, And/Or Concerns That Shape Their Questions And Purchase Behavior?
It’s critical to answer this. Get it wrong and you’ll waste time building content and executing campaigns that either get ignored or bring you the wrong kind of prospect.
Q: Who Does My Audience Trust? Where Do They Go For Validation?
They might all go to the same place or trust the same source. It might vary across all sub-segments, or just one. For LLM optimization this important information to know and leverage where you can. In your tracking tool, look for citations containing domains that you didn’t think of (that appear consistently). Investigate those sources relative to your topic to find potential new audience insights and concerns, additional audience-brand sentiment, etc.
Q: What Does My Audience Consider Non-Starters?
What does your have little to no use for, as it pertains to the solution your brand represents? Being sure to avoid focusing on mismatched points of concern or value with your audience helps keep the data clean and clear. Keep in mind that LLMs love to bring a lot of information to their responses. They’re quite verbose. What key product, service, or solution attributes might an LLM drag in that could dissuade your audience?
Q: Where Does My Audience Spend Time Online?
Are they more Instagram, or Reddit? Do they read long form content or do they watch video? Where they spend time gives clues about the types content and content formats they consume, the culture to which they’re most likely exposed, and their interests. You can use all of these to improve the robustness of LLM responses as well as your pages cited in those responses.
Q: Where Is My Audience?
Geography is a very important component when it comes to LLMs. They all use your IP address to determine where you are, and most models attempt to provide at least some level of localization to their response. They surface local businesses, mention local landmarks, address local climate, and a host of other location-based factors that significantly shape answers. Know where your audience is and what local factors are relevant to their needs. Formulate tracking prompts that work in these elements.
Technical Access
Q: Can LLM Crawlers Access and Process My Pages/Website?
If you’re blocking bots, you’ve reached the end before you’ve begun. Validating that LLMs can crawl your site, retrieve essential content your audience needs, and that your content is parseable by LLM bots is essential. Javascript-heavy and resource intensive sites with slow response and loading times may mean that while LLMs aren’t blocked from your site, they may not be seeing your content (or all of your content).
Q: Can LLMs Accurately Pull Facts From Your Website?
Sometimes facts about brands and products live in different places on the web. Confirm that, across your most important tracked prompts with the highest intent (Mid to Bottom funnel), LLMs are providing accurate information in their response.
First-Prompt Results
Q: Does The Model Show Your Brand When User Prompts Include Your Brand or Product Name?
This is mostly a step for technical validation, but something that first-prompt measurement can actually be helpful for. If the answer is no, there’s likely a technical reason why. If you’re a new brand, it’s possible the issue isn’t technical and you just don’t have a large enough digital footprint to be considered. Brand and Content Marketing, paired with strong SEO to support RAG can help get you out of that hole and start generating LLM mentions overtime.
Q: Does Your Brand Appear for Relevant Non-Branded Prompts?
This is a big hurdle and a key one first-turn measurements can help with. If you have a very low visibility score (Under 25 to 20%, roughly) then this is the thing to be working on. Improvement here has lots of dependencies, most of which are brand-oriented or audience-oriented. How well are you connecting your brand/product to the user’s specific needs as specified in the initial prompt? The main work here is building topical authority, strengthening knowledge graph entity:attribute associations, and often some amount of branded mentions to support these two pillars. Yes, content is important, but don’t underestimate reviews and PR.
Additional note here: Paying attention to the strength of brand presence in the initial prompt response is useful, but don’t over-focus on this. Challenger and smaller brands may find that moving from monitoring the initial prompt response to evaluating conversation progression may highlight the brand in deeper turns as user needs become more clear. Keep this in mind where the initial prompt isn’t very specific about user criteria, use case, etc.
Conversation Tracking
Q: Does Your Brand Stay in the Conversation After Multiple Turns?
What we want to know here is the rate at which we’re getting filtered out over the course of conversation, measured at the audience segment level (refer to your personas and segment data to identify differentiating needs that may appear as filtering questions). You should use a matrix that captures:
- Mentioned, never dropped
- Mentioned, then dropped
- Not mentioned, then added
- Never mentioned
This article by Vignesh Kanike describes this methodology very well and in-depth.
Q: Does the Model Suggest or Recommend Your Brand?
Does this happen at any point in the conversation? Does the recommendation stick or does it fall off? The goal is to pursue the recommendation by the conversation’s end. Where the conversation ends varies and can be dependent on both model responses and your audience segment/persona.
Q: Does the Model Suggest or Recommend a Competitor?
In this instance, look to the preceding prompt and the attributes that appear in the answer. Identify the sources the LLM likely pulled this information from and look to other potential sources to see if the messaging and entity:attribute pair is repeated elsewhere (potentially reinforcing as well as serving as alternative sources).
Q: Where Your Brand or Product URLs Appear, Are These The Correct URLs For This User/Intent?
It’s great to get that link, but is it correct? If the user visits it directly from the conversation, how many more pages on the website will they need to visit to fully satisfy their need? Does it include validating information or does it’s info differ from what the LLM suggested to the user? Is it a clean URL or is it paginated, parameterized, or otherwise complicated? Ideally it’s a clean URL, otherwise you may find numerous complications from measurement to recommendation consistency. The goal is to get the user to a page where their need is fully met.
Q: What Is The First Most Likely Refinement Question?
Now that the user has an initial response to an initial question, based specifically on each of your audience segments’ needs and attributes, how are they likely to ask the LLM to refine its response. Did the results come back pricey and they are value shoppers? Did the result not meet their style, needs for their climate, the company procurement prerequisites, etc.? Look at the actual responses to first-turn questions and try to identify gaps based on your audience segment dimensions.
Pro Tip: You can build an agent to help you with this. Have it look for commonalities and gaps across all runs for a prompt. Have it review the output from the perspective of the audience segments.
Action for this one is where we encounter friction with the currently available tracking tools. Since they can only track turn 1, we have to figure out how to carry some amount of the initial LLM response and conversational context into a new prompt/conversation. THIS IS NOT PERFECT AND MAY NOT DELIVER RESPONSES USERS SEE IN THE REAL WORLD. There are workarounds and means of validation but they are time consuming and labor intensive at this point. That does not mean this is not worth the effort. I still strongly suggest compiling new prompts that attempt to see the possible contents of follow-up responses. We just need to know that we’ll be missing conversational context.
Here’s a trick you can use that sort of replicates a subsequent turn (Turn 2, Turn 3, etc. but I wouldn’t go more than 3-4 turns deep otherwise the prompt may get too large. You can manage this by tightly condensing the prompt to focus on the key elements flagged in the template below). It’s a template that focuses on carrying over the essential context from a “prior conversation” which didn’t actually happen (but the LLM and the tracking tool don’t know this).
Track this as a new prompt and be sure to tag it with it’s corresponding “turn”:
Context Carry-Over Template for Next-Turn Prompt Tracking
We were discussing [Topic]. I initially asked you: [Original Prompt]. You responded with {Entity 1}, {Entity 2}, {Entity 3}. I then refined this based on {Constraint 1}, {Constraint 2}, {Constraint 3}. Your final recommendation/conclusion was {Final Entity/Answer}, sourced from {Source/Attribute}. Now, [Follow-Up Prompt].Here’s what that would look like in a real-world situation:
We were discussing rain coat shopping. I initially asked you: what’s the best rain coat under $250. You responded with Model A from Patagonia, Model B from North Face, Model C from Arc’teryx. I then refined this based on a temperate 75°F climate, needing it to be packable, a color preference for orange, and a requirement for GoreTex material. Your final recommendation was Patagonia Model A and Columbia Model D, sourced from Patagonia.com and ColumbiaOutlet.com. Now, are there any stores near me that sell either of these jackets?
Depending on your tracking tool that “near me” bit may need to be stated as an explicit location. But otherwise, this method does a pretty reliable job delivering the URLs, brands, products, and the like that you’d typically see in the LLM response. What’s missing are simply the things the tracking tools themselves have yet to be able to replicate which are things like brand favorability based on conversation history, browser history, installed apps, etc. You’re gonna have to do your own sleuthing on those for now.
Validation
Q: For Conversations That Result In Your Brand Being Recommended, Does The Model Cite A URL?
If yes, check your data. In web analytics, how many visits did this URL get from identified LLM sources? From dark LLM sources? Check your server logs: did LLM live retrieval bots fetch the page?
If no, what did the last response say? Should the LLM be recommending a specific URL to satisfy the user intent? Do pages on your site that address all of this audience segment’s concerns in this conversation exist? Have LLM bots visited these pages? You may have to do a lot of sleuthing in this scenario, but the value is high. You have a fully qualified buyer, go get ’em.
Q: Do Visibility Metrics Align with Web Analytics Metrics?
Most brands should be able to identify the questions their audience asks the most, the questions that have the highest purchase intent, and roughly what volume looks like based on search console data and current traffic and brand mention trajectories (growing or receding?).
If your visibility is high for a high-volume, high-intent prompt but traffic is low, this warrants further digging. Did branded search increase? Did traffic increase from other owned channels? Did it increase from earned channels? Answers to these questions may identify where tracking tools diverge from reality in terms of citations (which URL they serve to the user), or even model responses.
Remember, while they are improving, LLM visibility tools are the equivalent of rodent studies in a lab. Outcomes there may not reflect reality in human trials.
Process
The above questions can help you get more out of your LLM visibility tracking platform, but they can also help guide and inform your optimizations themselves. By asking these questions we identify the steps in the process of turning bot visits into LLM visibility. There’s much more, of course, than is captured here. What’s here focuses on the touchpoints tied to tracking. Know that influencing LLM outputs isn’t just about your own website, it’s about the user journey based on the user, their digital touchpoints, and what the machines know about entities in a given solution space.
At least now, hopefully, you have a clearer way to set up and utilize the contents of your tracking tools a little better and start moving that visibility needle.