Another team within the same org as my team, very different operations as they’re part of a museum that we own. They’ve been struggling with dealing with traffic like this for over a year while trying to work through a transformation programme to update all their library and collection services. But a good indication of bot traffics constant change and the high load it places on smaller teams trying to provide a service as a public good.

score 2 · canonical

Until this year, rules based on what a request says about itself, such as its address, its browser or its country, were mostly enough. Once the traffic was spread across enough addresses, they stopped working. Rules based on how a request behaves, such as whether it can run JavaScript, held only until the sender learned the test, which took less than a week each time.

First: until this year - is this an “AI”- related change, where the need to shovel more and more training content created by real humans made Wellcome Collection’s web pages crawler restrictions worth working around? I think “probably”, but the article never makes that conclusion.

Second: Wellcome Collection’s techniques for addressing overloads get circumvented by someone actively working around those techniques. Don’t the people working on circumvention ever look at their situations and decide to go along with others’ wishes? What kind of immorality is this? Why have we as a civilization failed these people by not instilling respect for others’ boundaries?

Third: are there any “mea culpa” documents about doing circumvention and switching to residential proxies and all the other work done to scrape an endless series of unwilling websites of their content? You’d think that sooner or later, conscience would plague someone into quitting and writing an insider’s account of this foul work.

score 1 · canonical