Skip to main content
Back to AEO Agent Blog

AEO & SEO

Getting Found and Understood by AI Agents

Customers now ask AI assistants which business to choose. AI recommends the business it understands, and a growing number of sites are blocking the very crawlers (automated readers) that could recommend them. How to set your door on purpose.

· 6 min read

A small storefront at night with a glowing blue door half open. Three streams of blue light approach it: one flows through, one is reading floating answer panels, and one is stopped at a dark gate.

The question your next customer asks

A family moves to a new town and needs a dentist. They do not open a search results page. They ask an AI assistant: does this practice take our insurance, do they do Invisalign, can they see our six year old? The assistant answers for the practice it understands, and recommends that one.

That is the whole pitch at Courtyard, which builds AI knowledge bases for dental practices, and its math is hard to argue with: about 1 in 2 consumers now ask AI for local business recommendations, and they choose the business the AI understands best. Swap the dentist for your business and the question stays the same. When a customer asks an AI agent about you, what does it actually know?

Found is not the same as understood

Plenty of businesses are technically online and practically invisible to AI. Their services live in a PDF. Their prices are a photo of a printed list. Their availability sits inside a booking widget no crawler can open. Their hours changed in March and the page still says winter.

An AI agent cannot recommend what it cannot read. Faced with gaps, it does one of two things, both bad for you: it guesses, and gets your business wrong in public, or it moves on to a competitor it understands clearly. Being found used to mean ranking on a results page. Being understood means your real answers, in real text, on real pages, current enough to trust. That is a higher bar, and it is the bar that now decides who gets recommended.

Three kinds of AI visitors

Not every AI visitor is doing the same job, and treating them as one crowd is where most sites go wrong:

  • Training crawlers read your pages to teach a model. They take your work and send nothing back today.
  • AI search crawlers index your pages so assistants can find and cite you when someone asks a question you answer.
  • Agent fetchers (bots acting for a user) arrive on behalf of one specific person, right now, to check a price, compare options, or complete a task for them.
  • Three different bargains. Training is a rights question. Search is a visibility question. Agents are a customer service question. Each one deserves its own answer.

Why sites are slamming the door

The blocking wave is not paranoia. Publishers watched crawlers take their content for training with no payment and no credit, while AI answers sent back a trickle of the traffic search used to deliver. AI visitors never see the ads that pay for the content. Some agent traffic is outright abuse: price scraping, ticket hoarding, credential testing. If that is your business, a wall looks like common sense.

So the walls went up. The Register reported that Cloudflare, the network that carries a large share of the world's web traffic, changed its tollbooth from pay per crawl to pay per use, paying publishers when their content actually shows up in an AI answer, and started blocking mixed-use AI crawlers by default on pages with ads starting September 15, 2026. The message to AI companies was blunt: separate your crawlers by purpose, and pay for what you take.

The Apple twist

People assume Apple simply blocks AI agents, full stop. The real story is more useful than that. Apple publishes one of the main rulebooks everyone else uses: a crawler token called Applebot-Extended that lets any site keep its content out of Apple Intelligence training while staying fully present in search, Siri, and Spotlight.

Apple confirmed the split in writing. As GeoParrot documented, Apple revised its Applebot documentation on September 7, 2026 and spelled the split out plainly: turning on the Applebot-Extended opt-out has no effect on how a site ranks in Search, and a site that uses it stays fully present in Siri, Spotlight, and Safari. Opting out of training and opting out of being found are two different switches, and Apple built them separately on purpose. Meanwhile, data from Apple's own search crawler may be used to train its foundation models, which is exactly why the opt-out switch exists. The lesson is not that Apple fears AI. The lesson is that the biggest players treat training, search, and agents as separate doors, each with its own key.

The trap: locking out your own referrals

Here is where well-meaning sites hurt themselves. A blanket block, a copied robots.txt (rules file for bots) snippet, or a security plugin set to maximum does not read intentions. It locks the door on every AI visitor with the same key, including the search crawlers and agent fetchers that would have quoted you, recommended you, and sent you a customer who was ready to buy.

Blocking training is a defensible rights decision. Blocking retrieval is an invisibility decision, and many businesses have made it by accident, then wondered why the AI answers in their market name everyone except them. The door was never the problem. The problem is a door with one setting, slammed or wide open, when the moment calls for settings.

Set the door on purpose

The practical work is unglamorous, and it is the same work we check on our own site with our AEO scanner (tool for answer engine optimization):

  • Decide per visitor kind: training, AI search, and live agents get separate answers in your robots rules and your content delivery settings, written down where your team can find them.
  • Keep the welcome mat out for AI search crawlers and agent fetchers. They are the ones who quote and recommend you.
  • Publish your real answers as real text on real pages: services, prices, insurance, hours, policies. If it only exists in a PDF or a photo, it does not exist to an AI agent.
  • Keep the facts current. An AI that quotes your old hours teaches customers not to trust either of you.
  • Recheck after every site change. Templates, plugins, and security tools rewrite these settings silently.
  • The businesses that win the next decade of the web are not the ones that block AI or the ones that surrender to it. They are the ones an AI agent can find, understand, and quote with confidence, because someone on the team set the door on purpose. That someone should be you, before your competitor does it first.

Sources

Want Thanos working on your site?

Discuss this topic with us

Built for agencies

Run the four AI scanners and client ready reports under your own brand.

See how we work with agencies