# period filtered profile pages isn't really useful to index, and with >6 million profiles, crawlers have plenty of other pages to index. User-agent: * Disallow: /profiles/*?period= Disallow: /profiles/*/playtime Disallow: /profiles/*?against Disallow: /profiles/*?comp Disallow: /profiles/*/most-killed/* Disallow: /profiles/*/archive # Much too aggressive crawling User-agent: ClaudeBot User-agent: Claude-SearchBot Disallow: /profiles/* Disallow: /infantrykillelo/* User-agent: meta-externalagent User-agent: meta-webindexer Disallow: /profiles/* Disallow: /infantrykillelo/* User-agent: Amazonbot Disallow: /profiles/* Disallow: /infantrykillelo/* # It's not useful to index these opengraph images and generating them puts unnecessary load on the server User-agent: Googlebot-Image Disallow: /profiles/*/opengraph-image/ User-agent: GPTBot Disallow: /profiles/*/opengraph-image/ Disallow: /profiles/*?period= Disallow: /profiles/*/playtime Disallow: /profiles/*?against Disallow: /profiles/*?comp Disallow: /profiles/*/most-killed/* Disallow: /profiles/*/archive User-agent: * Crawl-delay: 1