All about open source! Feel free to ask questions, and share news, and interesting stuff!
Community icon from opensource.org, but we are not affiliated with them.
I've seen this pop up on websites a lot lately. Usually it takes a few seconds to load the website but there have been occasions where it seemed to hang as it was stuck on that screen for minutes and I ended up closing my browser tab because the website just wouldn't load.
Is this a (known) issue or is it intended to be like this?
anubis is basically a bitcoin miner, with the difficulty turned way down (and obviously not resulting in any coins), so it's inherently random. if it takes minutes it does seem like something is wrong though. maybe a network error?
adding to this, some sites set the difficulty way higher then others, nerdvpn's invidious and redlib instances take about 5 seconds and some ~20k hashes, while privacyredirect's inatances are almost instant with less then 50 hashes each time
So they make the internet worse for poor people? I could get through 20k in a second, but someone with just an old laptop would take a few minutes, no?
Open source is also the AI scraper bots AND the internet itself, it is every character in the story.
I don't understand how/why this got so popular out of nowhere... the same solution has already existed for years in the form of haproxy-protection and a couple others... but nobody seems to care about those.
Probably because the creator had a blog post that got shared around at a point in time where this exact problem was resonating with users.
It's not always about being first but about marketing.
It’s not always about being first but about marketing.
And one has a cute catgirl mascot, the other a website that looks like a blockchain techbro startup.
I'm even willing to bet the amount of people that set up Anubis just to get the cute splash screen isn't insignificant.
Non paywalled link https://archive.is/VcoE1
It basically boils down to making the browser do some cpu heavy calculations before allowing access. This is no problem for a single user, but for a bot farm this would increase the amount of compute power they need 100x or more.
It inherently blocks a lot of the simpler bots by requiring JavaScript as well.
Exactly. It's called proof-of-work and was originally invented to reduce spam emails but was later used by Bitcoin to control its growth speed
it wasn't made for bitcoin originally? didn't know that!
Everytime I see anubis I get happy because I know the website has some quality information.
My archive's server uses Anubis and after initial configuration it's been pain-free. Also, I'm no longer getting multiple automated emails a day about how the server's timing out. It's great.
We went from about 3000 unique "pinky swear I'm not a bot" visitors per (iirc) half a day to 20 such visitors. Twenty is much more in-line with expectations.
This is fantastic and I appreciate that it scales well on the server side.
Ai scraping is a scourge and I would love to know the collective amount of power wasted due to the necessity of countermeasures like this and add this to the total wasted by ai.
Ooh can this work with Lemmy without affecting federation?
To be honest, I need to ask my admin about that!
We don't use anubis but we use iocaine (?), see /0 for the announcement post
Yes, it would make lemmy as unsearchable as discord. Instead of unsearchable as pinterest.
That's not true, search indexer bots should be allowed through from what I read here.
If you allow my searchxng search scraper then an AI scraper is indistinguishable.
If you mean, "google and duckduckgo are whitelisted" then lemmy will only be searchable there, those specific whitelisted hosts. And google search index is also an AI scraper bot.
"Yes", for any bits the user sees. The frontend UI can be behind Anubis without issues. The API, including both user and federation, cannot. We expect "bots" to use an API, so you can't put human verification in front of it. These "bots* also include applications that aren't aware of Anubis, or unable to pass it, like all third party Lemmy apps.
That does stop almost all generic AI scraping, though it does not prevent targeted abuse.