Cloudflare reports that Perplexity’s AI web crawlers are allegedly evading website restrictions designed to block them. When encountering blocks, the startup reportedly disguises its crawlers by changing its user agent to impersonate browsers like Google Chrome and rotates IP addresses to circumvent blocks. This behavior was observed across thousands of domains with millions of requests daily. Despite restrictions set in robots.txt files and web application firewalls, Perplexity’s bots access sites stealthily. In response, Cloudflare has delisted Perplexity as a verified bot and implemented new blocking methods. Perplexity disputes the claims, calling the report a publicity stunt. Cloudflare CEO Matthew Prince emphasizes the threat AI poses to publishers, and the company now allows sites to charge AI companies for content crawling or block AI crawlers by default.
Back
