Linux

It's actually not that hard. Most of these bots are using a predictable scheme of headless browsers with no js or minimal js rendering to scrape the web page. Fully deployed browser instances are demonstrably harder to scale and basically impossible to detect without behavioral pattern detection or sophisticated captchas that also cause friction to users.

The problem with bots has never rested solely on detectability. It's about:

A. How much you inconvenience the user to detect them

B. Impacting good or acceptable bots like archival, curl, custom search tools, and loads of other totally benign use cases.

It certainly is not negligble compared to static site delivery which can breezily be cached compared to on-the-fly tarpits. Even traditional static sites are getting their asses kicked sometimes by these bots. And yoy want to make that worse by having the server generate text with markov chains for each request? The point for most is reducing the sheer bandwidth and cpu cycles being eating up by these bots hitting every endpoint.

Many of these bots are designed to stop hitting endpoints when they return codes that signal they've flattened it.

Tarpits only make sense from the perspective of someone trying to cause monetary harm to an otherwise uncaring VC funded mob with nigh endless amounts of cache to burn. Chances are your middling attempt at causing them friction isn't going to, alone, actually get them to leave you.

Meanwhile you burn significant amounts of resources and traffic is still stalled for normal users. This is not the kind of method a server operator actually wanting a dependable service is deploying to try to get up and running gain. You want the bots to hit nothing even slightly expensive (read: preferably something minimal you can cache or mostly cache) and to never come back.

A compromise between these two things is what Anubis is doing. It inflicts maximum pain (on those attempting to bypass it - otheriwse it just fails) for minimal cost by creating a small seed (more trivial than even a markov chain -- it's literally just an sha256) that a client then has to solve a challenge based on. It's nice, but certainly not my preference: I like go-away because it leverages browser apis these headless agents dont use (and subsequnetly let's js-less browsers work) in this kind of field of problems. Then, if you have a record of known misbehavers (their ip ranges, etc), or some other scheme to keeo track of failed challeneges, you hit them with fake server down errors.

Markov chains and slow loading sites are costing you material just to cost them more material.

It doesn't need to be anywhere near as resource intensive as a crypto currency since it isn't used for security. The goal is not to stop bots altogether. The goal is to slow down the crawlers enough so that the server hosting the service doesn't get pegged. The bots went from being respectful of server operators to hitting pages millions of times a second. This is made much worse by the fact that git hosting services like Forgejo have many links many of which trigger the server to do computations. The idea behind Arubis is that a user really only has to do the PoW once since they aren't browsing to millions of pages. On a crawler it will try to do tons of proofs of work which will bog down the crawling rate. PoW also has the advantage of requiring the server to hold minimal state. If you try to enforce a time delay that means that the server has to track all of that.

It is also important to realize that Anubis is a act of desperation. Many projects do not want to implement it but they had no choice since their servers were getting wrecked by bots. The only other option would be Cloudflare which is much worse.