In Tuesday’s newsletter, I promised I’d get Matt to expand on why we feel the Cloudflare AI Agent Taxonomy for robots.txt is a gamechanger, and he’s come through, despite a busy week.
You’re going to want to add this one to your reading list ASAP because understanding it is essential to your AI visibility, and your machine-first architecture.
Here’s what you can expect: Cloudflare users can now classify what types of AI agents can access site pages. Prior to this, it was a binary, “yes, they’re allowed,” or “no, they’re not” situation.
Cloudflare is implementing this on September 15, so now is a good time to decide what information you want crawled, and how you’ll classify your pages for AI agents. Matt goes into detail on how to make these decisions, and how to revise your robots.txt file, in case you need the refresher.
NOTE: Most website builders and content management systems purport to create the robots.txt for you, but we strongly recommend you look these over to ensure they’re set up correctly.
BTW, I have two quick things to ask:
If you enjoy this newsletter, please forward it to someone you think would also enjoy it, or get help from it.
If you’d like to know more about your digital estate, and how to protect it, reach out here.
Recap of This Week’s Major Discussion Topics:
AEO/SEO is the top service request in 2026 as organizations prioritize brand visibility in AI-generated answers over traditional search.
LLMS.txt files primarily aid model training rather than content comprehension, making their deployment discretionary.
Cloudflare's new bot taxonomy enables granular control over site access by classifying AI traffic, giving Penpixel Creative a powerful new lever to protect and manage your digital estate.
Deven Bhagwandin
Founder & Director of AI Search, Penpixel Creative
—
Is your website functionally invisible to the agents now commanding the web? My Machine-First Audits provide the answer. I break down your technical structure, identity signals, and transaction readiness against the specific checks required for true agent legibility.
BTW, if you know of a business that I can help, please introduce me, or refer them to https://www.penpixelcreative.com/contact.