Generative Engines Are Breaking Web Analytics and Hurting Their Future
Search is moving from traditional search engines to generative engines, but traffic from many of these sites isn’t being tracked properly in analytics. It’s their fault, not yours.
I was looking at our LLM filter in Ahrefs Web Analytics and noticed some common generative engines missing from the list. They’re in our filters, but we aren’t seeing any data from them for sites.
This invisible traffic problem comes from these systems stripping the referral value. I first noticed this problem with AI Mode in Google, but it’s a common problem for generative engines.
This is most likely a mistake on their part, but in some cases may be intentional. Some of these tools probably want more market share and just made a mistake, while others may not want you to be able to measure traffic from the systems. Google has said the clicks from AI Search are higher quality, but we have no way to verify that.
If you have a website that sends traffic to other sites, you should want it to be tracked properly. In the case of generative engines, I warned that these AI bots need to send that info in order to fulfill their social contract, where they provide traffic to websites, and websites allow these bots to crawl and their data to be used.
There’s a cost to bots crawling your websites and there’s a social contract between search engines and website owners, where search engines add value by sending referral traffic to websites. This is what keeps most websites from blocking search engines like Google, even as Google seems intent on taking more of that traffic for themselves. This social contract extends to generative engines.
I think many site owners want to let these bots learn about their brand, their business, and their products and offerings. But while many people are betting that these systems are the future, they currently run the risk of not adding enough value for website owners.
The first LLM to add more value to users by showing impressions and clicks to website owners will likely have a big advantage. Companies will report on the metrics from that LLM, which will likely increase adoption and prevent more websites from blocking their bot.
The same sentiment is true for attribution. If these generative engines want to win market share, they need to be present in reporting to companies. So far, many are not doing a great job.
noreferrer attribute on the link. This would prevent the referral value from being sent.
As expected, there is no referrer shown in the Chrome Dev Tools Console. It comes back empty.
document.referrer
''
In Ahrefs Web Analytics, this is recorded as Unknown, but in Google Analytics it would be classified as Direct. Google lumps traffic from unknown sources and internal website traffic together as Direct, whereas we separate them into Unknown and Internal.
What’s interesting is that when I looked at the same type of link in a free account, it did not have the noreferrer attribute. It’s tracked properly.
For lists of links, they were also tracked properly.
The linkes to Sources in the content and at the bottom of the response are also tracked properly, and they add a URL parameter “?utm_source=chatgpt.com” to the URLs as well.
Web Search
Most of the links in Web Search mode had the referrer. I did run into an interesting example when there are multiple references. The top one had a referrer, the other 2 did not.
DeepResearch
For DeepResearch mode, in-content links were attributed properly, but the sources at the end were marked with noreferrer.
HTTP Headers
If you look at the HTTP Headers, you’ll sometimes find a Referrer-Policy header to specify what and how much information gets passed in the referrer. You can use the Ahrefs SEO Toolbar to view this information by going to the HTTP headers tab.
For ChatGPT, they’ve set a referrer-policy value of “strict-origin-when-cross-origin”. In this case, the downgrade from HTTPS to HTTP would drop the referrer. Any links to pages using HTTP wouldn’t be attributed properly.
AI Mode is marked with noreferrer.
John Mueller from Google has since confirmed it’s a bug and that they will likely fix it.
Louise Linehan mentioned that we may be underestimating AI traffic. She specifically mentioned how Copilot disappeared from our analytics tracking system. Since that time, the traffic has returned.
What I suspect is that these links were marked as noreferrer during that time period. This shows how code changes can impact your global tracking.
Everything here seemed to be tracked properly now.
That’s not the case with Copilot in Windows. I found no cases where the referrer was passed.
Minecraft is a game where you get a gaming experience of fantastic adventure with limitless possibilities like mining, building safe zones, fighting, looting your enemy’s safe zones, and much more. It has come a long way since its launch and has become one of the most popular games in the world today. Usually, people play…
We’ve heard from many of you that you want to learn more about Single Directory Components (SDCs) in Drupal. Which makes sense, since they are one of the most exciting new features in Drupal theming since Twig was added to Drupal 8. We’re excited to share that the first phase of our brand-new SDC course…
WordPress has become one of the most used and renowned CMS platforms worldwide over the years. WordPress has enough features and works almost perfectly. It is a CMS from which a user may need nothing more and nothing less. Whether you’re a beginner or an advanced-level user, this platform is meant for you! It’s best…
Turning your photographs into sketches is a great way to add a unique artistic touch to your work. But how can you achieve this without having to manually draw every image? The answer is Photoshop actions. This collection of Photoshop actions offers various options for transforming photos into beautiful sketches. They take the guesswork out…
You’re looking for this article because you think long-form content works right? Or at least that’s what you’ve heard from many marketers. You might even have seen it for yourself. You’re trying to make a simple pasta dish (like aglio e olio) but you have to scroll through a piece that’s thousands of words long,…
A product catalog does more than list what you sell, it can also help shape how people see your brand. A clean, well-structured layout can make your products easier to browse and more appealing to potential buyers. But starting from scratch in InDesign takes time. That’s where these templates come in. These pre-made layouts give…