I’m aware of indexing bots (spiders) and yeah some of them were just doing that but there are also malicious crawlers that actively make things worse for all of us online. And the proportion of these bad bots is growing every year. https://www.imperva.com/resources/resource-library/reports/2026-bad-bot-report/

And it’s not just AI, which is a more recent trend, as the annual Imperva reports have chronicled the rise in bad bot traffic for the past decade.