User-Agent: * Allow: / # Allow major AI crawlers User-Agent: GPTBot Allow: / User-Agent: Google-Extended Allow: / User-Agent: ClaudeBot Allow: /