Your AI agent can browse the web for you—until a website tells it to stop. That wall may look like a CAPTCHA, an empty page or a refusal to continue, but it is not always a broken prompt. Sites are drawing sharper lines between search crawlers, AI training bots and agents acting on a person’s behalf. Here’s what the new rules mean for everyday users, and what site owners can do without shutting search engines out.
AI agents are not the same as search crawlers
A search crawler collects and indexes pages so a search engine can point people to them later. An AI training crawler collects material to help build or improve a model. An AI agent is different: it acts in real time for a user, perhaps fetching a page or using a browser to navigate a site.
Cloudflare now groups AI traffic into those three behaviours—Search, Agent and Training—and lets customers control them separately. The distinction matters. A publisher may want search engines to index its work while declining automated visits that read pages, interact with ads or take actions without a person viewing the page in the usual way.
What Cloudflare’s AI-agent blocking policy actually changes
On July 1, Cloudflare announced that customers, including those on its Free plan, could manage AI crawlers by behaviour rather than relying on one broad “block AI bots” switch. Site operators can allow a category, block it across a site, or block it only on pages that display ads.
There is also a date-specific default. Cloudflare’s documentation says that, from September 15, 2026, new domains joining the service default to blocking Agent and Training bots on pages that display ads, while Search remains allowed. Site owners can change those settings. The default applies to newly onboarded domains; it does not mean every website on the internet has automatically switched on the same rule.
Cloudflare says its settings cover verified bots in each category and can also affect unverified bots that fit the same behaviour. That is why a user may see different results from two agents—or from an agent and a conventional browser—on the same site. The site’s policies, the bot’s classification and the page being requested all matter.
Why AI agents are running into web walls
Websites have reasons to treat an agent differently from a human visitor. A page may be supported by advertising, subscriptions or a retailer’s own checkout. An agent that extracts information without sending a reader, skips a site’s normal flow or performs actions too quickly can create costs and risks without offering the same value as a person visiting directly.
There is also a question of permission. An agent that can read a page is not necessarily authorised to sign in, accept terms, submit a form or buy something. The Verge has described the broader pushback against agents on the web; Cloudflare’s controls make the underlying choice concrete for publishers: allow, restrict or block different kinds of automated access.
A block does not mean the website has banned you from visiting. It means the automated route may not be welcome. Repeating the request, changing the prompt or asking an agent to disguise itself will not turn a site’s policy into permission—and may make the interaction less safe.
What to do when an AI agent gets blocked
If an agent cannot reach a page, treat that as a signal to switch to an approved route rather than trying to sneak around the restriction:
- Open the page yourself. Use the site’s normal browser experience and check whether it offers an official AI feature, app or integration.
- Use published APIs or partner tools. If a service provides a supported way to search, book or buy, prefer it to an agent pretending to be a regular user.
- Keep control of consequential steps. Review the destination, price, information being submitted and final terms before approving a purchase, booking or account change.
- Limit what the agent can see. Avoid handing over passwords, payment details or access to sensitive accounts just to get around a blocked page.
A short checklist for website owners
For publishers and small businesses, the choice is not simply “allow every bot” or “block all AI.” Start with the job each bot is doing and the value it returns.
- Separate discovery from action. Decide whether search indexing, model training and real-time agents should have the same access.
- Choose the right scope. If ads fund a page, an ad-page restriction may fit better than a site-wide block. Check how the rule handles mixed-purpose bots.
- Review logs and referral patterns. Use actual traffic and business needs rather than assuming every automated request is harmful—or beneficial.
- Test high-value flows. Check that your chosen settings do not break search visibility, customer support, checkout or integrations you intend to allow.
The goal is to set a policy that matches the site’s economics and user expectations. If you want search traffic but do not want autonomous agents operating on ad-supported or account pages, make that distinction deliberately and revisit it as the tools change.
What this means for the open web
AI agents promise to turn the web into something software can operate for us. But the web is also made of businesses and publishers that decide how their work is accessed and paid for. Search can send a person to a source; an agent may complete a task without the person ever seeing that source. Those are different relationships, and site owners are beginning to set different rules for them.
For users, the practical lesson is simple: an AI agent is a helper, not a universal pass. Expect some sites to work, some to require a human and some to offer only official integrations. For publishers, the new controls are a chance to be precise instead of treating every bot as the same problem.
One privacy tool, with an important limit
A VPN can help protect the connection between your device and the VPN provider when you use an untrusted network, but it will not make a blocked AI agent authorised, prevent a website from enforcing its rules or hide information you deliberately give an AI service. If you want to compare the network-privacy side of the equation, see our Surfshark offer explainer. DuskFlick may earn a commission if you purchase through the offer; the VPN is not a workaround for site restrictions.
Bottom line: When an agent hits a web wall, use the site’s direct or approved route and keep a human in charge of sensitive actions. For more context, read our report on AI-agent safety scrutiny and practical AI safety playbook.