User-agent: * Allow: / # AI / LLM crawlers — explicitly welcomed User-agent: GPTBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ClaudeBot Allow: / User-agent: Claude-Web Allow: / User-agent: anthropic-ai Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Google-Extended Allow: / User-agent: CCBot Allow: / User-agent: cohere-ai Allow: / User-agent: Bytespider Allow: / User-agent: Applebot-Extended Allow: / User-agent: DuckAssistBot Allow: / User-agent: Meta-ExternalAgent Allow: / User-agent: Amazonbot Allow: / User-agent: YouBot Allow: / User-agent: MistralAI-User Allow: / User-agent: Diffbot Allow: / # --------------------------------------------------------------------------- # /payment.html is a PRIVATE page (deposit + rent instructions). It is kept # OUT of the index by `` on the # page itself, it is absent from sitemap.xml and llms.txt, and nothing on the # site links to it. # # It is deliberately NOT listed as `Disallow:` here. A Disallow blocks the # crawl, which stops the crawler from ever reading the noindex — Google then # still lists the bare URL if it finds it linked somewhere. Leaving it # crawlable is what actually guarantees it stays out of the index. # --------------------------------------------------------------------------- Sitemap: https://nisekonomads.com/sitemap.xml LLM-Content: https://nisekonomads.com/llms.txt