I don't remember exactly when but, earlier in the year, I decided I didn't want this website to be indexed by search engines anymore. A lot of it has to do w...
Same here. I would seriously doubt the scraping activity would drop, because I’ve got a few URLs/domains that have never been indexed by any search engine and are constantly hit by scrapers and bots attempting to find something (they look for .env files and .php files though there’s no PHP running in that server).
I believe the list of publicly certified domain certificates from Let’s Encrypt is the most popular source for bots now, since I’ve had new domains get hit within 1h of existing and the only “thing” that knew about them at that point was their service.
I believe the list of publicly certified domain certificates from Let’s Encrypt is the most popular source for bots now, since I’ve had new domains get hit within 1h of existing and the only “thing” that knew about them at that point was their service.
Solution could be to do wildcard certs and then put stuff you don’t want found on a subdomain.
I believe the list of publicly certified domain certificates from Let’s Encrypt is the most popular source for bots now, since I’ve had new domains get hit within 1h of existing and the only “thing” that knew about them at that point was their service.
That matches my observations (and not just from Let’s Encrypt!). People who want to do bad things scrape Certificate Transparency Logs and scan the sites very quickly after the certificates are issued.
Same here. I would seriously doubt the scraping activity would drop, because I’ve got a few URLs/domains that have never been indexed by any search engine and are constantly hit by scrapers and bots attempting to find something (they look for .env files and .php files though there’s no PHP running in that server).
I believe the list of publicly certified domain certificates from Let’s Encrypt is the most popular source for bots now, since I’ve had new domains get hit within 1h of existing and the only “thing” that knew about them at that point was their service.
score 1 · canonical
Solution could be to do wildcard certs and then put stuff you don’t want found on a subdomain.
score 2 · canonical
That might be a good incentive to use subdomains more, instead of domains. Thanks!
score 1 · canonical
You could also use a DNS-01 challenge and keep the stuff you don’t want found on a non-routable IP address.
score 1 · canonical
That matches my observations (and not just from Let’s Encrypt!). People who want to do bad things scrape Certificate Transparency Logs and scan the sites very quickly after the certificates are issued.
score 1 · canonical