AI Search (GEO)
AI Crawler Manifest Builder (ai.txt)
Decide exactly which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended…) may use your content — and generate a ready-to-upload ai.txt manifest plus the matching robots.txt rules. Welcome AI search engines that cite you, block the ones you don’t want, in one click.
Loading tool…
Need reliable hosting, a domain, or a hand setting this up?
Get a free quote →About the AI Crawler Manifest Builder (ai.txt)
The AI Crawler Permission Builder generates a complete ai.txt / llms.txt style manifest plus the matching robots.txt snippet, declaring exactly how AI systems may use your website. You choose a default policy, then allow or block each well-known AI crawler individually — GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bytespider, meta-externalagent and more — and set high-level rules such as whether your content may be used for model training, whether AI answer engines may quote it, whether attribution is required, and an optional crawl-delay. The tool outputs both a human-readable policy manifest and the machine-enforced robots.txt rules crawlers actually obey.
This free tool by Hosting Cambodia helps publishers, agencies and businesses in Phnom Penh and across Cambodia take a deliberate position on AI: stay fully visible in ChatGPT Search and Perplexity answers, opt out of model training only, or block everything on staging sites. Everything is generated in your browser — fill in your site name, URL and contact email, click a few toggles, then copy or download both files and upload them to your site root.
Frequently asked questions
What is the difference between ai.txt and robots.txt?
robots.txt contains machine-enforced crawl rules that major AI crawlers actually check and obey, while ai.txt / llms.txt is an emerging plain-language manifest declaring your policy on training, citation and attribution. Publish both: the robots.txt snippet does the enforcement, the ai.txt file documents your intent.
Should I block AI crawlers from my business website?
Usually not all of them. Blocking answer bots like OAI-SearchBot and PerplexityBot removes your business from AI search results, which is where more customers are looking every month. A common middle ground is allowing search and citation bots while blocking pure training crawlers such as CCBot or Bytespider.
Where do I upload the generated files?
Upload ai.txt to your website root so it is reachable at yourdomain.com/ai.txt, and merge the robots.txt snippet into your existing robots.txt file at yourdomain.com/robots.txt. On cPanel hosting both live in the public_html folder.
Explore the full free web tools toolbox by Hosting Cambodia.