How to Block Ahrefs, Moz & Majestic Bots from Crawling Your Website

SEO crawlers such as Ahrefs, Moz, and Majestic can be useful when you want to analyze backlinks, monitor competitors, or audit your own site. However, not every website owner wants third-party SEO tools crawling their pages, collecting link data, or consuming server resources. If you would rather keep these bots out, you can usually do so with a combination of robots.txt rules, server-level blocks, and firewall settings.

Why Block SEO Crawlers?

Ahrefs, Moz, and Majestic operate large crawlers that scan websites to build backlink indexes and SEO databases. These tools help marketers understand who links to whom, which pages are popular, and how competitors perform in search. From a business perspective, that data can be valuable. From a website owner’s perspective, it may also feel intrusive.

Common reasons for blocking these bots include:

• Reducing server load: Large crawlers can request many URLs, especially on sites with thousands of pages.
• Limiting competitive intelligence: Some site owners do not want competitors using third-party tools to map their backlinks or content strategy.
• Protecting private or low-value sections: Staging areas, filtered pages, and internal search results may not need to be crawled by SEO tools.
• Improving crawl budget for important bots: While third-party tools do not affect Google’s crawl budget directly, reducing unnecessary bot traffic can improve overall server performance.

Step 1: Identify the Bots You Want to Block

Before creating rules, it helps to know the common user agents used by these services. A user agent is a text string that identifies the crawler making a request to your website.

• Ahrefs: AhrefsBot
• Moz: DotBot and, historically, rogerbot
• Majestic: MJ12bot

You can confirm crawler activity by checking your server logs. Look for repeated requests from these user agents, especially if they are hitting many pages in a short period. In Apache or Nginx logs, you will usually see the requested URL, timestamp, IP address, status code, and user agent string.

Step 2: Block Them with robots.txt

The simplest method is to use your site’s robots.txt file. This file lives at the root of your domain, such as:

https://example.com/robots.txt

To ask Ahrefs, Moz, and Majestic not to crawl your site, add the following:

User-agent: AhrefsBot
Disallow: /

User-agent: DotBot
Disallow: /

User-agent: rogerbot
Disallow: /

User-agent: MJ12bot
Disallow: /

This tells each bot that it should not crawl any part of your website. Most major SEO crawlers are designed to respect robots.txt, so this is often enough for basic blocking.

If you only want to block certain sections, you can specify paths instead of the entire site:

User-agent: AhrefsBot
Disallow: /private/
Disallow: /search/
Disallow: /tag/

User-agent: MJ12bot
Disallow: /private/
Disallow: /search/
Disallow: /tag/

This approach is useful if you do not mind general crawling but want to keep thin, duplicate, or sensitive areas out of SEO databases.

The Limits of robots.txt

Although robots.txt is easy to use, it has one major limitation: it is a request, not a barrier. Well-behaved bots will follow it, but technically nothing stops a crawler from ignoring the file. Also, robots.txt does not prevent a URL from being discovered through links elsewhere; it simply tells compliant bots not to crawl the page.

For that reason, if blocking these crawlers is important for performance, privacy, or competitive reasons, use server-level blocking as well.

Step 3: Block Bots in Apache with .htaccess

If your website runs on Apache and allows .htaccess overrides, you can block requests based on user agent. Add this to your .htaccess file:

SetEnvIfNoCase User-Agent "AhrefsBot" bad_bot
SetEnvIfNoCase User-Agent "DotBot" bad_bot
SetEnvIfNoCase User-Agent "rogerbot" bad_bot
SetEnvIfNoCase User-Agent "MJ12bot" bad_bot

<RequireAll>
  Require all granted
  Require not env bad_bot
</RequireAll>

On older Apache versions, you may see syntax like this:

BrowserMatchNoCase "AhrefsBot" bad_bot
BrowserMatchNoCase "DotBot" bad_bot
BrowserMatchNoCase "rogerbot" bad_bot
BrowserMatchNoCase "MJ12bot" bad_bot

Order Allow,Deny
Allow from all
Deny from env=bad_bot

After saving changes, test your website carefully. A small syntax error in .htaccess can cause a server error. If possible, make changes during a low-traffic period and keep a backup of the original file.

Step 4: Block Bots in Nginx

If your server uses Nginx, you can block these user agents inside your server configuration. For example:

if ($http_user_agent ~* "(AhrefsBot|DotBot|rogerbot|MJ12bot)") {
    return 403;
}

This returns a 403 Forbidden response whenever the request user agent matches one of the listed bots.

A more structured approach is to use a map block:

map $http_user_agent $bad_bot {
    default 0;
    ~*AhrefsBot 1;
    ~*DotBot 1;
    ~*rogerbot 1;
    ~*MJ12bot 1;
}

server {
    if ($bad_bot) {
        return 403;
    }
}

After editing Nginx configuration, always run:

nginx -t

If the test passes, reload Nginx:

systemctl reload nginx

Step 5: Use a Firewall or CDN

If your site uses a CDN or web application firewall, such as Cloudflare or another security platform, you can create rules that block specific user agents. This is often more convenient than editing server files, especially if you manage multiple websites.

A typical firewall rule might say:

• If User Agent contains AhrefsBot, then Block
• If User Agent contains DotBot, then Block
• If User Agent contains MJ12bot, then Block

Firewall blocking has another advantage: the unwanted request may never reach your origin server. That can reduce bandwidth, CPU usage, and log noise.

Should You Block by IP Address?

Blocking by IP address can work, but it is usually harder to maintain. Crawlers may use many IP ranges, and those ranges can change over time. User-agent blocking is simpler, but it depends on the bot identifying itself honestly. For stronger protection, combine several methods: robots.txt, user-agent blocking, rate limiting, and firewall rules.

Be cautious with broad IP blocks. Accidentally blocking cloud providers, shared networks, or security services can create unexpected problems for real users.

Testing Your Blocks

After adding your rules, review your access logs over the next few days. If the bots are respecting robots.txt, you should see fewer requests. If you added server-side rules, you may see 403 responses for blocked user agents.

You should also test that normal visitors and important crawlers are unaffected. Do not block user agents such as Googlebot, Bingbot, or legitimate uptime monitoring services unless you have a very specific reason.

Final Thoughts

Blocking Ahrefs, Moz, and Majestic bots is not difficult, but it should be done thoughtfully. Start with robots.txt because it is clean, transparent, and widely respected. If you need stronger control, add server-level or firewall-based rules to prevent requests from reaching your site.

The best setup depends on your goal. If you simply want to discourage SEO tools, robots.txt may be enough. If you are protecting server resources or limiting third-party data collection, use a layered approach. With the right configuration, you can reduce unwanted crawler traffic while keeping your website accessible to users and search engines that matter most.