Loving the CSS on that website.
thanks for sating that. it made me go look and yes, it’s lovely.
This is a good point. Honestly, I think the tac that Anubis takes is probably the best one right now.
If each request you make to a web server is computationally expensive, that makes it harder for the bots to get that info at high speed, and thus needs to spend more power to access it. If a solution like that is deployed at a wide enough scale, it becomes prohibitively expensive to do the scraping.
Anubis is excellent, I’ve implemented it on a dozen sites now. Cuts traffic massively. We paid for it, too; awesome to be able to support a Canadian independent developer.
That’s the one
Their own wiki says it doesn’t work very well and doesn’t work if you have JavaScript disabled. So what about it do you like specifically? The character?
Honestly, the character is the worst part about it, but that can be changed with an enterprise license
The Javascript requirement is not ideal, but the reality is that the vast majority of the internet already needs it, so I don’t see it as that much of a hurdle. The other problem it has is that it requires a sort of herd immunity to achieve the goal. If a critical mass of websites don’t use it, then the AI companies can simply just pay for the increased compute costs for the sites that do.
In a world where more than half of websites that are being scraped had some kind of protection like this, it would work much better.
