Generative Engines Are Breaking Web Analytics and Hurting Their Future

Search is moving from traditional search engines to generative engines, but traffic from many of these sites isn’t being tracked properly in analytics. It’s their fault, not yours.

I was looking at our LLM filter in Ahrefs Web Analytics and noticed some common generative engines missing from the list. They’re in our filters, but we aren’t seeing any data from them for sites.

This invisible traffic problem comes from these systems stripping the referral value. I first noticed this problem with AI Mode in Google, but it’s a common problem for generative engines.

This is most likely a mistake on their part, but in some cases may be intentional. Some of these tools probably want more market share and just made a mistake, while others may not want you to be able to measure traffic from the systems. Google has said the clicks from AI Search are higher quality, but we have no way to verify that.

If you have a website that sends traffic to other sites, you should want it to be tracked properly. In the case of generative engines, I warned that these AI bots need to send that info in order to fulfill their social contract, where they provide traffic to websites, and websites allow these bots to crawl and their data to be used.

There’s a cost to bots crawling your websites and there’s a social contract between search engines and website owners, where search engines add value by sending referral traffic to websites. This is what keeps most websites from blocking search engines like Google, even as Google seems intent on taking more of that traffic for themselves. This social contract extends to generative engines.

I think many site owners want to let these bots learn about their brand, their business, and their products and offerings. But while many people are betting that these systems are the future, they currently run the risk of not adding enough value for website owners.

The first LLM to add more value to users by showing impressions and clicks to website owners will likely have a big advantage. Companies will report on the metrics from that LLM, which will likely increase adoption and prevent more websites from blocking their bot.

The same sentiment is true for attribution. If these generative engines want to win market share, they need to be present in reporting to companies. So far, many are not doing a great job.

noreferrer attribute on the link. This would prevent the referral value from being sent.

As expected, there is no referrer shown in the Chrome Dev Tools Console. It comes back empty.

document.referrer
''

In Ahrefs Web Analytics, this is recorded as Unknown, but in Google Analytics it would be classified as Direct. Google lumps traffic from unknown sources and internal website traffic together as Direct, whereas we separate them into Unknown and Internal.

What’s interesting is that when I looked at the same type of link in a free account, it did not have the noreferrer attribute. It’s tracked properly.

For lists of links, they were also tracked properly. 

The linkes to Sources in the content and at the bottom of the response are also tracked properly, and they add a URL parameter “?utm_source=chatgpt.com” to the URLs as well. 

Web Search

Most of the links in Web Search mode had the referrer. I did run into an interesting example when there are multiple references. The top one had a referrer, the other 2 did not.

DeepResearch

For DeepResearch mode, in-content links were attributed properly, but the sources at the end were marked with noreferrer.

HTTP Headers

If you look at the HTTP Headers, you’ll sometimes find a Referrer-Policy header to specify what and how much information gets passed in the referrer. You can use the Ahrefs SEO Toolbar to view this information by going to the HTTP headers tab.

For ChatGPT, they’ve set a referrer-policy value of “strict-origin-when-cross-origin”. In this case, the downgrade from HTTPS to HTTP would drop the referrer. Any links to pages using HTTP wouldn’t be attributed properly.

AI Mode is marked with noreferrer.

John Mueller from Google has since confirmed it’s a bug and that they will likely fix it.

Louise Linehan mentioned that we may be underestimating AI traffic. She specifically mentioned how Copilot disappeared from our analytics tracking system. Since that time, the traffic has returned.

What I suspect is that these links were marked as noreferrer during that time period. This shows how code changes can impact your global tracking.

Everything here seemed to be tracked properly now.

That’s not the case with Copilot in Windows. I found no cases where the referrer was passed.

LinkedIn or X.