Technology

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Hotbar 3.0)

Agreed. I use cloudflare for domain hosting because they're cheap, but I have never liked their protections.

If you use cloudflare to "protect" your site you are discriminating against browser choice and are as bad as Microsoft in 1998.

😕

Maybe is against the ToS but I've used github as CDN for free in the past... Might work for you.

I never felt it was wrong, it was around the time of the Microsoft acquisition.

Agree for static content like news and blogs. Disagree for dynamic content like games and social media. And the latter is mostly for scale (having server-side templating is expensive for rapidly changing content).

Then again, there's a case for snapshotting SM pages every so often for things like crawlers and cli browsers.

Ever been down the gemini rabbit hole? It's not perfect, but quite interesting.

I get why you're frustrated and you have every right to be. I'm going to preface what I'm going to say next by saying I work in this industry. I'm not at Cloudflare but I am at a company that provides bot protection. I analyze and block bots for a living. Again, your frustrations are warranted.

• Even if a site doesn't have sensitive information, it likely serves a captcha because of the amount of bots that do make requests that are scraping related. The volume of these requests can effectively DDoS them. If they're selling something, it can disrupt sales. So they lose money on sales and eat the load costs.

• With more and more username and password leaks, credential stuffing is getting to be a bigger issue than anyone actually realizes. There aren't really good ways of pinpointing you vs someone that has somehow stolen your credentials. Bots are increasingly more and more sophisticated. Meaning, we see bots using aged sessions which is more in line with human behavior. Most of the companies implementing captcha on login segments do so to try and protect your data and financials.

• The rise in unique, privacy based browsers is great and it's also hard to keep up with. It's been more than six months, but I've fingerprinted Pale Moon and, if I recall correctly, it has just enough red flags to be hard to discern between a human and a poorly configured bot.

Ok, enough apologetics. This is a cat and mouse game that the rest of us are being drug into. Sometimes I feel like this is a made up problem. Ultimately, I think this type of thing should be legislated. And before the bot bros jump in and say it's their right to scrape and take data it's not. Terms of use are plainly stated by these sites. They consider it stealing.

Thank you for coming to my Tedx Talk on bots.

Edit: I just want to say that allowing any user agent with "Pale Moon" or "Goanna" isn't the answer. It's trivially easy to spoof a user agent which is why I worked on fingerprinting it. Changing Pale Moon's user agent to Firefox is likely to cause you problems too. The fork they are using has different fingerprints than an up to date Firefox browser.

You can, it'll cost more and give you less, but you can.

That's the way this works.