Companies like OpenAI and Perplexity promised in the past that the AI search engines their models offer would provide new sources of income for publishers by directing traffic to their sites. However, according to a report shared with Forbes by content licensing platform TollBit, it has been revealed that AI search engines actually send less than 96% of traffic to news sites and blogs compared to the usual Google search. In the meantime, the practice of AI developers scraping data from websites has continued to increase, adding to the frustration many publishers feel towards these tools.
An interface of OpenAI’s ChatGPT search. Source:
OpenAI
(X/Twitter)
How AI-powered search engines are stealing the show
To understand the conflict developing between publishers and AI search engines, we need to go back to the origins. Search engine optimization became a huge deal when Yahoo! transitioned from being a pure set of directory listings in 1994 to offering search capabilities in 1995. It was the answer to internet discovery, and after Google emerged as the first effective search engine in 1997, it quickly dominated the web and became essential for web users. Google became the web’s best search engine due to its PageRank scoring mechanism, which ranked search results based on how many other websites linked to each hit. Today, with the spread of AI-powered search engines, users are opting to utilize simple AI answer agents or more elaborate “Deep Research” style AI research tools available from various AI labs to conduct online queries. This growing relationship between AI search engines and content publishers challenges the ‘social contract’ that has kept traditional search engines like Google relevant to publishers. Per this contract, publishers provided content in exchange for referral traffic from search engines, forming the basis for a symbiotic relationship that has sustained the online media ecosystem for decades.
See also
Nvidia loses $450 billion of market cap over the past 24 hours
However, now that we have AI-generated summaries that eliminate the need for users seeking direct information to visit the original sources, the dynamic has shifted significantly. This new development not only threatens the revenue models of content creators but also compels us to question the sustainability of quality journalism. Nathan Schultz, the CEO of Edtech company Chegg, believes it is time to “say no,” explaining that breaking the longstanding contract is not right. It is clear AI is here to stay, and over time, more and more people will turn to AI agents when they have questions. One expert suggests that this is because AI goes the extra mile to deliver understanding rather than just assist in discovering what you’re looking for. While this is a valuable feature, it has a devastating effect on sites that rely on referral traffic for survival. For instance, Chegg experienced a 49% decline in traffic in January year-over-year, a sharp drop from the 8% decrease in the second quarter of the previous year, when Google released AI summaries. The traffic decline has impacted Chegg so severely that it is considering going private or being acquired, according to Schultz’s statements during an earnings call.
How news publishers have been dealing with the new development
According to TollBit’s report, AI search engines send less referral traffic compared to traditional Google searches, and there has been a significant increase in web scraping by AI companies, with some scraping websites an average of 2 million times in the last quarter of 2024. Unfortunately, these scrapings do not often translate into traffic for the affected sites.
See also
Google co-founder urges employees to spend 60 hours weekly in the office to lead AGI race
Publishers are dissatisfied with this situation and have reacted primarily by taking legal actions against AI companies for intellectual property infringement. Forbes sent a cease-and-desist letter to Perplexity in June, accusing it of infringing copyright, and in October, the New York Post and Dow Jones sued the company for alleged copyright infringement and for attributing fabricated facts to media companies.
Perplexity AI shares stats about its Deep Research AI search tool’s performance. Source:
Perplexity AI
(X/Twitter)
At that time, Perplexity responded by stating that the lawsuit reflects a stance that is “fundamentally shortsighted, unnecessary, and self-defeating.” Earlier this month, a group of publishers including Condé Nast, Vox, and The Atlantic also filed a lawsuit against enterprise AI company Cohere, accusing it of allegedly scraping 4,000 copyrighted works from the internet and using them to train its suite of large language models. The issue is further complicated by the fact that AI companies do not properly identify their web crawlers, making it difficult for publishers to manage access to their content. To address these challenges, some publishers have opted for content licensing agreements with AI companies to ensure they are compensated for the use of their data. Others, like TollBit, have developed models to charge AI companies for scraping content. As things continue to evolve, legal frameworks surrounding data protection and intellectual property will become critical battlegrounds where publisher rights will be potentially defended and expanded. If this situation is allowed to continue unchecked, analysts warn that we may enter an era dominated by “AI slurry,” where high-caliber content providers are forced out of business, leading to a significant dilution in the quality of available information.
Cryptopolitan Academy: Tired of market swings? Learn how DeFi can help you build steady passive income.
Register Now