#Allow non-bad bots to index site. User-agent: * DisAllow: #Stop the following specific bots from scanning site and limit scanning to 20seconds intervals. User-Agent: AhrefsBot Disallow: / Crawl-Delay: 20 User-Agent: SemrushBot Disallow: / Crawl-Delay: 20 User-Agent: Semrushbot-SA Disallow: / Crawl-Delay: 20 User-Agent: Semrushbot-SI Disallow: / Crawl-Delay: 20 User-Agent: Yandex Disallow: / Crawl-Delay: 20 User-Agent: BingBot Disallow: / Crawl-Delay: 20 #Need to add a site map.