User-Agent: * Disallow: /admin/ # disable crawling django admin # AI training crawlers. The site stays readable by people, by search engines # and by AI answer engines; it is not a training corpus. User-agent: GPTBot User-agent: ClaudeBot User-agent: Claude-User User-agent: Claude-Web User-agent: anthropic-ai User-agent: CCBot User-agent: Google-Extended User-agent: Google-CloudVertexBot User-agent: Perplexity-User User-agent: Bytespider User-agent: Amazonbot User-agent: Applebot-Extended User-agent: meta-externalagent User-agent: Meta-ExternalFetcher User-agent: cohere-ai Disallow: /