this post was submitted on 07 Jul 2025
595 points (98.4% liked)

Open Source

38789 readers
127 users here now

All about open source! Feel free to ask questions, and share news, and interesting stuff!

Useful Links

Rules

Related Communities

Community icon from opensource.org, but we are not affiliated with them.

founded 5 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
[–] kcweller 88 points 3 days ago

Robots.txt expects that the client is respecting the rules, for instance, marking that they are a scraper.

AI scrapers don't respect this trust, and thus robots.txt is meaningless.