Let Monocle Search crawl a Cloudflare-protected site

A Cloudflare challenge warning means Cloudflare returned a challenge instead of your site's content. The challenge stops our crawler before it can read the page. A normal visit in your browser may still work.

Monocle Search uses the DoNoHarm crawler. Use the steps below to allow this crawler without removing protection for other visitors.

1. Find the feature that issued the challenge

1. Open your site in the Cloudflare dashboard.
2. Go to Security → Analytics → Events to open Security Events.
3. Select the time of the failed crawl.
4. Filter by a crawler IP from our official IP list.
5. Open a matching event.
6. Check the action, service, rule, hostname, and path.

Look for a challenge action, such as Managed Challenge. Record the feature or rule that issued it. Security Events can use sampled logs, so try a shorter time range if the event is missing.

Do not assume that every 403 response comes from a Cloudflare challenge. A Cloudflare server header alone does not confirm one either.

2. Get the current crawler IP addresses

Open our official IP list. It contains an ips array with the crawler's current public IPv4 and IPv6 addresses.

Use every address in that array. Do not copy the JSON wrapper or revision into a Cloudflare rule. Check the live list when you create or update the exception, rather than using addresses from an old message.

Do not allow traffic by user agent alone. Other visitors can copy a crawler's user-agent string.

3. Add a narrow exception

If the event names Bot Fight Mode, read the next section first. A custom Skip rule cannot bypass that product.

For features that support Skip rules:

1. Open Security rules in the Cloudflare dashboard.
2. Select Create rule → Custom rules.
3. Name the rule Allow Monocle Search crawler.
4. Set the match to source IP addresses in the current official list, including IPv4 and IPv6.
5. Add a hostname condition for the site that Monocle Search crawls.
6. Select the Skip action.
7. Select only the security features identified in Security Events, using the guidance below.
8. Keep Log matching requests enabled.
9. Place the Skip rule before the relevant challenge rules.
10. Select Deploy.

Choose the smallest applicable exception:

• Super Bot Fight Mode: select All Super Bot Fight Mode rules only if this service issued the challenge.
• WAF managed rules or rate limiting: skip only the identified feature. For a single managed rule, use a rule-specific exception where available instead of skipping all managed rules.
• A custom challenge rule: preferably edit that rule to exclude the crawler IP addresses. All remaining custom rules skips every later rule in that ruleset, so use it only if that wider exception is necessary for these crawler IPs.
• Browser Integrity Check or another supported product: select that product explicitly. Skipping WAF managed rules does not skip every other security product.

Do not select every Skip option, disable all security, or allow an entire hosting provider's network. Keep unrelated protections active. See Cloudflare's Skip options and challenge exclusion guidance.

Bot Fight Mode is different

Bot Fight Mode cannot be skipped with a WAF custom rule or Page Rule. A Skip, Bypass, or Allow action in those rules will not fix a Bot Fight Mode challenge.

Prefer Super Bot Fight Mode with a scoped Skip rule when your plan supports it. Ask your site administrator to review the plan requirements and follow Cloudflare's setup instructions. Then use the IP-based exception above and skip only All Super Bot Fight Mode rules if that service caused the challenge.

Cloudflare also documents an IP Access rule as an alternative on all plans, including Free. Bot Fight Mode does not trigger if an IP Access rule matches the request first.

An IP Access Allow rule is a broader exception than Skip. It also bypasses custom rules, rate limiting rules, and WAF managed rules for that IP. Use it only if your site administrator accepts this trade-off.

To create this exception, follow Cloudflare's IP Access rule instructions:

1. Open Security rules → Create rule → IP access rules.
2. Enter one exact address from the current official crawler IP list.
3. Set Action to Allow.
4. Set Zone to the current website only, not all websites in the account.
5. Select Create.
6. Repeat for each remaining IPv4 and IPv6 address in the list.

Do not substitute a whole IP range, ASN, country, or account-wide rule. Cloudflare notes that the dashboard option appears only when an IP Access rule already exists. If it is unavailable, ask your administrator to use the API linked in Cloudflare's instructions.

Do not turn off the site's other protections to work around Bot Fight Mode. If you cannot change the bot protection settings, contact support@monocle-search.com with the event details.

4. Crawl again and check the result

A WordPress security or password-protection plugin allowlist does not change Cloudflare. Configure each layer separately if your site uses both.

1. Open the site's Crawls page in Monocle Search.
2. Select Crawl now.
3. Wait for the crawl to complete.
4. Check that the Cloudflare warning is gone and that the expected pages appear in search.
5. For a Skip rule, review Security Events to confirm that crawler requests match the exception and no longer receive the challenge.

If another feature still challenges the crawler, identify it in Security Events before changing the exception.

IP Access Allow matches do not appear in Security Events. For that exception, use the completed crawl result and your origin's access logs to check that the crawler can reach the content.

Monocle Search sends one notification after two consecutive Cloudflare-challenged crawl attempts. Another notification becomes eligible only after an uninterrupted sequence of successful full crawls spans at least seven days. A partial, incomplete, failed, cancelled, or interrupted attempt restarts that recovery period. A completed crawl with product translation failures also restarts it.