llms.txt Generator for Cloudflare Workers & Pages
Check whether Cloudflare's default AI bot settings are blocking ChatGPT Search or Perplexity, then serve /llms.txt at the edge in under 10ms.
Cloudflare is the cleanest way to serve /llms.txt for both native Workers/Pages apps and third-party CMS platforms (Shopify, Squarespace, Beehiiv) via a dedicated /llms.txt Worker route.
Step-by-Step Cloudflare Workers & Pages Setup
- 01
Scan your Cloudflare-proxied domain above to check if Cloudflare's 'Block AI Scrapers and Crawlers' managed robots.txt is blocking OAI-SearchBot or PerplexityBot.
- 02
Copy or download the generated /llms.txt Markdown from the studio.
- 03
For Cloudflare Pages, place llms.txt inside your static public/ output folder; for any proxied site, attach the Worker snippet below to route yourdomain.com/llms.txt.
- 04
Verify Cache-Control and Content-Type: text/plain; charset=utf-8 headers with curl or a live re-scan.
const LLMS_TXT = `# Your Brand
> Concise summary of your site for AI search engines.
## Core Pages
- [Homepage](https://yourdomain.com/)
- [Pricing](https://yourdomain.com/pricing)
`;
export default {
async fetch(request) {
const { pathname } = new URL(request.url);
if (pathname === "/llms.txt") {
return new Response(LLMS_TXT, {
headers: {
"content-type": "text/plain; charset=utf-8",
"cache-control": "public, max-age=3600",
},
});
}
return fetch(request);
},
};Recommended Cloudflare Workers & Pages llms.txt Blueprint
Primary landing pages, pricing, and status endpoints.
API docs, Worker templates, and edge guides.
Security policy and legal disclosures.
Common Cloudflare Workers & Pages AI Search Pitfalls
Enabling Cloudflare's one-click 'Block AI Bots' managed robots.txt without realizing it blocks live discovery crawlers alongside training scrapers.
Serving /llms.txt with text/html instead of text/plain; charset=utf-8.
Forgetting to bypass Cloudflare Managed Challenge / Bot Fight Mode for verified AI search bots.
Cloudflare Workers & Pages llms.txt FAQs
Why does Cloudflare sometimes block AI crawlers on my domain?
Cloudflare introduced managed robots.txt and AI bot-blocking toggles in the Security → Bots dashboard. If enabled globally, it can disallow bots you actually want citing your brand. Run our scanner to see your live effective rules.
Can I use a Cloudflare Worker to add /llms.txt to Shopify or Squarespace?
Yes! Attaching a simple Worker route to yourdomain.com/llms.txt lets any Cloudflare-proxied domain serve a native HTTP 200 /llms.txt file without modifying the origin CMS.
Does serving /llms.txt from Cloudflare Workers cost anything?
Cloudflare Workers' free tier includes 100,000 requests per day—more than enough to serve /llms.txt at global edge speed for free.
Free On-Demand Scanner. Automate When You Scale.
Generate and download /llms.txt free forever with zero login. Upgrade only when you want automated weekly syncs, crawler-block alerts, or agency client reports.
Everything you need to audit any public site and ship a clean /llms.txt today.
- Unlimited zero-login live domain scans
- 0–100 AI Search Visibility Score & breakdown
- 8-bot robots.txt AI crawler inspection
- Spec-compliant /llms.txt generator (Copy & Download)
- JSON-LD, canonical & semantic HTML audit
For SaaS teams, Shopify merchants, and publishers keeping AI search visibility continuously protected.
- Up to 3 monitored production domains
- Weekly auto-regenerated /llms.txt diffs & hosted URL
- Instant email/Slack alerts if any AI search bot gets blocked
- Sitemap & JSON-LD regression tracking
- Historical 0–100 GEO score trendline
Built for SEO agencies and web studios delivering client-ready GEO audits at scale.
- 25 client websites with weekly auto-monitoring
- White-label GEO Audit PDF exports for client pitches
- Multi-client AI crawler block alert digest
- Bulk /llms.txt regeneration across Shopify, WP & Webflow
- Priority onboarding & custom webhook notifications