User-agent: Pinterestbot Disallow: Crawl-delay: 30 # Bingbot - limit crawl rate User-agent: bingbot Disallow: Crawl-delay: 15 # AhrefsBot - limit aggressively or consider blocking User-agent: AhrefsBot Disallow: / # (optional: allow only homepage) # Allow: / # SemrushBot - similar to Ahrefs User-agent: SemrushBot Disallow: / # DotBot (Moz) - heavy crawler, limit User-agent: DotBot Disallow: Crawl-delay: 20 # ClaudeBot - fairly aggressive, slow it down User-agent: ClaudeBot Disallow: Crawl-delay: 30 # ChatGPT user agent (if you're seeing scraping from it) User-agent: ChatGPT-User Disallow: / User-agent: * Disallow: /admin/ Disallow: /checkout/ Disallow: /widgets/ Disallow: /api/