Reddit has commenced its assault on search engines, reportedly blocking those that don’t have a commercial relationship with the company.
Back in February this year, Google agreed to a $60 million “expanded partnership” with the mega-site which combines elements of social media, news aggregation, and messageboards. Reddit data can now be used for the advances of the Alphabet company’s AI models, while Reddit content will have a more prominent position on Google Search, making it easier to participate in the site’s conversations and communities.
One of the most notable aspects is that the US multinational tech leader now has access to the Reddit Data API which delivers real-time content.
As reported by 404 Media, Reddit results on other search engine platforms are now being blocked as the company goes on the offensive against web crawlers. That impacts the likes of Bing, DuckDuckGo, and Mojeek, with results either not showing any recent content, or not displaying the search result in full.
Microsoft spokesperson Caitlin Roulston stated, “Microsoft respects the robots.txt standard and we honor the directions provided by websites that do not want content on their pages to be used with our generative AI models,” adding that its Bing search engine stopped crawling Reddit when it updated its robots.txt file on July 1.
It’s a bold move for Reddit to effectively block some of the biggest search engines on the web, but it could be another pointer of its direction of travel since the IPO earlier this year and desire to find new ways to drive revenue.
Unable or unwilling to make enforceable promises
However, a Reddit spokesperson has provided a different explanation of the situation.
“This is not at all related to our recent partnership with Google,” stated Tim Rathschmidt in a statement to The Verge.
“We have been in discussions with multiple search engines. We have been unable to reach agreements with all of them, since some are unable or unwilling to make enforceable promises regarding their use of Reddit content, including their use for AI.”
It has been conveyed that the online results are due to changes to Reddit’s robots.txt file, aimed at “all crawlers” that do not agree to not utilize Reddit data for AI training, with the company listing Internet Archie and reddit4research as two examples of crawlers that still work.
Image credit: Via Ideogram
