When the publisher brief is "grow our AI visibility" but robots.txt blocks LLMs, the reflex is: Allow โ
job done ๐ And that job was probably your last ๐ซ I mapped the nuance with a free ai robots.txt tool + decision tree here๐
https://t.co/b3Ai6m2AbU
๐ค Before I start, it's important to know that by publisher I mean any business whose content is the product: News sites, subscription libraries, research houses, review sites, course providers - anyone who sells, licenses or paywalls what it writes, whether or not they'd call themselves a publisher.
๐ก There isnโt a clean "block LLMs = bad, allow LLMs = good" answer here. The data shows a very different cost depending on how far you go (i.e. it depends - would I be an SEO without one? ๐):
๐ข Allow all LLMs to access all of your content: Maximum AI visibility, lost IP
๐ According to @Cloudflare, Anthropic crawled 38,065 pages for every visitor it sent back - and according to @pressgazette the licensing money goes to those who can say no (N
X (Twitter) ยท 2 wks
