Inc. reports that four leading AI companies have agreed to respect a single robots.txt signal for website data scraping. However, the article notes this move does not address the fact that most internet data has already been collected by these firms.
This matters for banks, insurers and healthcare groups as it raises ongoing questions about data governance and privacy. Even with new opt-out options, previously scraped data remains in use, posing compliance and reputational risks for regulated sectors.
