Reddit CEO Steve Huffman has announced the end of free data scraping on the platform, signalling a shift in how Reddit's data will be used and monetized. Huffman emphasized that companies, including tech giants like Microsoft, must now pay to access Reddit's vast repository of user-generated content.
In a recent interview with The Verge, Huffman specifically named Microsoft, Anthropic, and Perplexity companies using Reddit's data without permission or compensation. He stated that blocking their access has been "a real pain in the ass."
The crux of the issue lies in how these companies utilize Reddit's vast trove of user-generated content. According to Huffman, Microsoft has been using Reddit data to train its AI systems and summarize content in Bing search results "without telling us." He further alleged that this data has been sold through the Bing API to other search engines, raising concerns about the unauthorized commercialization of Reddit's content.
This move by Reddit follows recent deals struck with tech giants Google and OpenAI, allowing them to access and use Reddit's data. These agreements are the model that Huffman wants other companies to follow. The Reddit CEO emphasized that without such contracts, the platform has no control or knowledge of how its data is displayed or used.
The situation has escalated in recent months, with Reddit updating its robots.txt file in early July to block web crawlers from companies it doesn't have agreements with. This has led to a noticeable absence of Reddit results on search engines like Bing, while they remain visible on Google, which has a data agreement in place.
Huffman's stance aligns with a growing trend among content creators and platforms seeking fair compensation for their data, especially as it becomes increasingly valuable for training AI models and enhancing search capabilities. He argued that the traditional value exchange between search engines and content providers is changing, with the lines between search, summarization, and AI training becoming increasingly blurred.
The Reddit CEO pointed to OpenAI's recent announcement of SearchGPT, which will include Reddit results thanks to a deal between the two companies, as an example of the kind of arrangement he seeks with other tech firms.
In response to these developments, Microsoft's head of search, Jordi Ribas, stated on X that “Reddit has blocked Bing from crawling their site for search, favouring another search engine and impacting competition from Bing and Bing-powered engines.”
Anthropic, one of the companies named by Huffman, responded by stating that Reddit has been on their block list for web crawling since mid-May, and they have not added any URLs from Reddit to their crawler since then.
As this situation unfolds, it raises important questions about the future of data usage, content licensing, and the evolving relationship between social media platforms, search engines, and AI companies.