The short answer
Decide which AI crawlers may read which pages, publish that policy, and check your server logs to see who actually visits. Crawler visits are not referral traffic or revenue.
The real question is which pages should represent your company.
In this article 7 sections
The five lessons
If you read nothing else, read these.
- Crawler access is a publishing decision.
- Access is not traffic.
- Check the rule that actually answers the request.
- Expose only what deserves to represent the company.
- Name an owner and review monthly.
Crawler access is a publishing decision
A crawler can request a page without sending a visitor, so a bot request is never a conversion. Marketing owns whether a page explains the business accurately. Web or security owns the access rule.
Cloudflare's September 15, 2026 discussion of mixed-use AI crawlers describes the tension. One crawler may serve several purposes, so a blanket allow or block is too crude. Set an explicit policy tied to content classes and an owner. AI visibility pays back only when it sits inside a growth system that measures leads.
Cloudflare, accountable mixed-use AI crawlers (September 15, 2026)
The four-part access framework
Make crawler access accountable in four stages.
- Identify: name the purpose you support, such as search discovery, user-requested retrieval or training.
- Decide: set rules by route and content class. Keep private, transactional and thin pages out of the public answer surface.
- Test: review requests, status codes, paths and rate.
- Review: one named owner reconciles the route list, policy and readout monthly.
Check the rule that actually answers the request
A robots.txt directive is a preference for cooperative crawlers, not authentication. A CDN or firewall rule can still deny a page that robots.txt allows, and a disallow rule does not secure a private dashboard. Check the page response, the robots policy and the enforcement layer separately.
Cloudflare's September 15 update gives its Training control a Disallow AI Training option that retains search access for the mixed-use crawlers it designates Accountable. Its Block setting can also block mixed-use search crawlers. Verify the effective setting for your domain. The same access question applies to agents that act on your offers.
Cloudflare: how the September 15 controls apply to mixed-use crawlers
What to expose and what to protect
Start with pages that answer a stable public question: definitions, service boundaries, methodology and evidence you stand behind. Keep customer records, proposals, dashboards and account areas behind real access controls.
List each route with its purpose, owner, update date and sensitivity. See what makes a page worth citing by AI.
Measure crawler activity without calling it traffic
Cloudflare's AI Crawl Control documentation distinguishes crawler requests from referred visits. Use logs for bot requests and analytics for human sessions. Keep the two apart.
A weekly readout has four lines: requests by crawler category, pages requested, errors or blocked requests, and human visits from separately verified sources. Record unknowns. Missing referral data is not zero. How to count the human visits is in measuring AI search traffic.
What to check this week
Ask who can change your crawler rules and which pages should represent the company. Then read your server logs to see which bots requested what.
Questions leaders ask
Does allowing an AI crawler guarantee visibility in an AI answer?
No. Access permits a request, not indexing, citation or a business outcome. Keep crawler access and commercial measurement separate.
Who should own an AI crawler policy?
Marketing owns public content accuracy and engineering or security enforces access. One named operator reconciles the policy and readout on a fixed cadence.
SOURCES
Where’s the money leaking?
Pick one. Be honest.
Cite this article
Grigorchuk, T. (2026, September 21). How Should a Business Manage AI Crawler Access and Visibility?. Megawebvision. https://megawebvision.com/insights/ai-crawler-access-and-visibility