everytime i check nginx logs its more scrapers then i can count and i could not find any good open source solutions
You must log in or register to comment.
I’ve seen people suggesting and using Anubis, haven’t used it myself though.
deleted by creator
How do you know it’s “AI” scrappers?
I’ve have my server up before AI was a thing.
It’s totally normal to get thousands of bot hits and to get scraped.
I use crowdsec to mitigate it. But you will always get bot hits.
Some of them are at least honest and have it as a user agent.
Is ignoring robots.txt considered “honest”?
That’s not what I was talking about
