Share interesting Technology news and links.
Rules:
To encourage more original sources and keep this space commercial free as much as I could, the following websites are Blacklisted:
More sites will be added to the blacklist as needed.
Encouraged:
I really feel like scrapers should have been outlawed or actioned at some point.
But they bring profits to tech billionaires. No action will be taken.
No, the reason no action will be taken is because Huawei is a Chinese company. I work for a major US company that's dealing with the same problem, and the problematic scrapers are usually from China. US companies like OpenAI rarely cause serious problems because they know we can sue them if they do. There's nothing we can do legally about Chinese scrapers.
I thought Anthropic was also very abusive with their scraping?
Maybe to others, but not to us. Or if they are, they're very good at masking their traffic.
Can you not just block China?
We do, somewhat. We haven't gone as far as a blanket ban of Chinese CIDR ranges because there's a lot of risks and bureaucracy associated with a move like that. But it probably makes sense for a small company like Codeberg, since they have higher risk tolerance and can move faster.
I use a tool that downloads a website to check for new chapters of series every day, then creates an RSS feed with the contents. Would this be considered a harmful scraper?
The problem with AI scrapers and bots is their scale, thousands of requests to webpages that the internal server cannot handle, resulting in slow traffic.
Does your tool respect the site’s robots.txt?
Unfortunately, robots.txt cannot express rate limits, so it would be an overly blunt instrument for things like GP describes. HTTP 429 would be a better fit.
Crawl-delay is just that, a simple directive to add to robots.txt to set the maximum crawl frequency. It used to be widely followed by all but the worst crawlers ...
Crawl-delay
It's a nonstandard extension without consistent semantics or wide support, but I suppose it's good to know about anyway. Thanks for mentioning it.
If the site is getting slowed at times (regardless of whether it is when you scrape), you might want to not scrape at all.
Probably not a good idea to download the whole site, but then that depends upon the site.
Write TOS that state that crawlers automatically accept a service fee and then send invoices to every crawler owner.
Huawei is Chinese. There's literally zero chance a European company like Codeberg is going to successfully collect from a company in China over a TOS violation.
It's not even a company. It's a non-profit "eingetragener Verein". They have very limited resources, especially money because they purely live on membership fees and donations.
True, but it can help limit the European AI scrapers too
I really doubt it. Lawsuits are expensive, and proving responsibility is difficult, since plausible deniability is easy. All scrapers need to do is use shared IPs (e.g. cloud providers), preferably owned by a company in a different legal jurisdiction. That could be the case here: a European company could be using Huawei Cloud to mask the source of their traffic.
All scrapers need to do is use shared IPs (e.g. cloud providers),
Simple: just charge the cloud provider.
Once that gets strong enough they'll start placing terms against scraping in their TOS.
And then they just throw it in the bin because there was never a contract between you and them. What to do then? Sue Microsoft, Amazon and Google
I'm sure Codeberg, a German non-profit Verein, has time and money to do that 🤣.
They typically don't include a billing address in the User Agent when crawling 🤣
That's a technicality. The billing address can be discovered for a nominal fee as well.
I'm sure it can't, especially for foreign IP addresses, VPNs, and a ton of other situations. Even if directly connect to the internet just via your ISP, many countries in Europe (don't know about US) have laws that would require you to have very good reasons and a court order to get the info you need from the ISP - for a single(!) case.
If it would be possible to simply get the address of all digital visitors, we wouldn't have to develop all this anti scrape tech and just sue them.
Seems like such a massive waste of bandwidth since it's the same work being repeated by many different actors to piece together the same dataset bit by bit.
Ah Capitalism! Truly the king of efficiency /s
Huh, why does Anubis use SHA256? It's been optimized to all hell and back.
Ah, they're looking into it: https://github.com/TecharoHQ/anubis/issues/94
Begun, the information wars have.
The wars have been fought and lost a while ago tbh
It's being investigated at least, hopefully a solution can be found. This will probably end up in a constantly escalating battle with the AI companies. https://github.com/TecharoHQ/anubis/issues/978
Uuughhh I knew it'd always be a mouse and cat game, sincerely hope the Anubis devs figure out how to fuck up the AI crawlers again
I dont understand how challenging an AI by asking it to do some heavy computational stuff even makes sense... A computer is literally made to do computations, and AI is just a computer. 🤨
Wouldn't it make more sense to challenge the AI with a Voight-Kampff test? Ask it about baseball.
The scrapers are not actually an ai, they are just dumb scrapers there to get as much textual information as possible.
If they have to do Anubis tests, that is going to take more time to get the data they scrape. I suspect that they are probably paid per page they provide, so more time per page is less money for them.
The point is to make scraping expensive enough it isn't worth the trouble. The only reason AI scrapers are trying to get this data is because it's cheaper than the alternatives (e.g. generating synthetic data). Once it stops being cheaper, the smart scrapers will stop. The dumb scrapers don't matter because they don't have the talent to devise these kind of workarounds.
