AI-ready websites need three different fixes, not one

Published October 17, 2026

“Make the site AI-ready” is the kind of roadmap item that sounds straightforward until you actually audit what your server delivers. Sigo Seguros, a Spanish-first auto insurance agency running a static Astro marketing site plus a JavaScript quote app, did that audit and found the goal hides three separate problems. A fix for one does nothing for the other two.

Here is what they found, what they changed, and what they deliberately left alone.

Table of Contents

Three AI visitors, not one

Most AI optimization advice treats all AI visitors as the same. They’re not. Sigo’s audit identified three distinct audiences:

Crawlers and answer engines (GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot): fetch raw HTML once, never run JavaScript, read JSON-LD, sitemaps, and robots.txt
Agentic browsers (Claude in Chrome, ChatGPT agent mode, Comet, Gemini agent): run JavaScript, carry normal user agents, drive pages through the accessibility tree and DOM, sometimes use screenshots
Protocol clients (assistants calling your MCP server): skip the page entirely, read tool schemas and tool results

Keep these three apart and the work becomes specific. Mix them together and you’ll ship fixes that don’t actually fix anything.

Audience 1: crawlers that skip your JavaScript

AI crawlers fetch raw HTML and move on. Log studies confirm they don’t execute JavaScript. For Sigo, this was good news on the marketing side. Static Astro output has a real <h1>, real text, and a real <form> in source HTML. It was bad news for the quote app. A plain fetch of /application returned a 17 KB shell with a <title> and nothing else.

What they changed

A <noscript> block on every app shell. Short, locale-matched, links to a crawlable entry page and a phone number. No prices, fees, or coverage claims. The embeddable partner widget carries both languages because locale reads from a query parameter the script parses. Links use target="_top" to escape the iframe.
Structured data that actually matches reality. Their InsuranceAgency JSON-LD listed fewer states in areaServed than they serve. LLMs relay structured fields like name, phone, hours, areaServed, and sameAs more directly than classic search ranking does, so a stale field becomes a stale answer. They fixed the field, gave the organization a stable @id that every blog Article points to, and published the agency license as an identifier.
No redirect hops in published links. Every URL in agent-facing files ends with a trailing slash, and a unit test fails if a new URL doesn’t. One extra 301 per link adds up when a crawler fetches each URL only once.

On llms.txt

Sigo has one and keeps it short. They don’t treat it as an SEO lever. Adoption across the web is low, the share of AI bot requests that fetch /llms.txt is tiny, and Google has said it doesn’t use the file.

The actual value they found is what they call business-to-agent: a short authoritative contract an agent reads after a person or protocol points it there. Their file documents the one public entry parameter (?zipcode=) and states explicitly that an empty app shell on a plain fetch “is not an error condition.” That sentence matters. Without it, agents report the site as broken.

They also serve agents.md. One Cloudflare Pages gotcha: static files are typed by extension, so the Content-Type you set in code may not apply. Agents read the bytes fine. Browsers may download the file instead of rendering it.

♿ Audience 2: agentic browsers, where accessibility is the API

watercolor wireframe sketches of website layouts

Agentic browsers run JavaScript, carry normal user agents, and drive your page through the accessibility tree. Agents that work from screenshots also do measurably better on sites that pass accessibility checks. For Sigo, agent usability turned out to be the same work as accessibility debt they already owed.

Changes to the quote flow

Live regions for async steps. The quote wait screen and payment poll showed a spinner with no announced state. role="status" plus aria-busy tells an agent the page is working, not stuck.
Errors tied to fields. aria-invalid alone marks a field as bad without explaining why. Adding aria-describedby that points to the error message fixes that.
Unique accessible names. Multiple “Continuar” buttons on one screen is the classic failure for screenshot-driven agents. Where two actions share a visible label, aria-label distinguishes them.
A complete autocomplete token set. Tokens are the cheapest way for an agent to map a stored profile to your fields. Where no standard token exists, a stable name and id on the field helps.
Stable positions for the primary action and the price display. These matter more than clean CSS.

One deliberate choice: no agent detection. Sigo serves the same page to everyone. Detection is fragile because agentic browsers look human. It also approaches cloaking. Serving an agent a different page from what a person sees is a trust problem.

Audience 3: protocol clients, where discovery is the bottleneck

Sigo runs an MCP server that returns quote estimates. For months, almost nothing pointed to it. Discovery came down to a set of small, unglamorous changes:

Name the endpoint in llms.txt and agents.md, ahead of the reading list
Publish a human-readable /mcp/ page listing the endpoint and tools, with setup steps for each assistant
Serve /.well-known/mcp.json on both the main domain and the connector host (leave out the draft $schema URL if it doesn’t resolve)
Add a Link: </.well-known/mcp.json>; rel="describedby" header on the homepage and app routes, set in _headers
Register with the official MCP registry, verified by a DNS TXT record on the apex domain, plus directories like Smithery and each assistant’s own connector directory

One directory gotcha: some read your OAuth metadata and label the whole server “OAuth required” even when its public tools are anonymous. Check how you’re listed.

Why analytics can’t tell you much

Sigo pulled AI-assistant referrals from the GA4 BigQuery export. ChatGPT, Gemini, Copilot, and Perplexity together came to well under one percent of sessions. A few sessions landed directly on the app entry point that llms.txt documents.

Treat that number as a floor, not a ceiling, for two reasons. Agentic browsers carry normal user agents and run JavaScript, so they show up in GA4 as ordinary sessions. Crawler fetches never appear in client-side analytics at all because there’s no JavaScript to fire a page_view event. Only server or CDN logs see that traffic.

CDNs have also started classifying AI bots by category (training, search, agent) and some block categories by default on new zones. A spoofed user-agent curl that returns 200 proves nothing because real bots are verified by IP or signature. Only the CDN dashboard shows what verified bots actually get. Make that choice on purpose and write it down.

What they deliberately skipped

No llms-full.txt or per-page Markdown twins. The maintenance cost is real and the evidence of use isn’t there yet.
No agent checkout. Agentic commerce is settling on discovery followed by a redirect to the merchant. For a licensed brokerage, binding coverage needs a person. The pattern is: the agent quotes, the person reviews and pays on the site.
No prices, premiums, fees, or coverage figures in agent-facing files. These files get relayed without the page around them, which makes them customer-facing copy. In insurance, copy needs a state-by-state legal review.

The checklist

If you’re running a site with a mix of static and JavaScript-rendered pages, here’s the distilled version:

For crawlers: static HTML for anything you want quoted; <noscript> on JS shells; JSON-LD that matches reality; no redirect hops; a short llms.txt treated as a contract.

For agentic browsers: live regions, field-linked errors, unique accessible names, autocomplete tokens, and the same page for everyone.

For protocol clients: publish the endpoint everywhere a person or agent might look, add /.well-known/mcp.json, register with DNS verification, and audit each directory listing.

For your own measurement: accept that analytics show a floor. Pull CDN bot reports and document any out-of-git bot settings somewhere your team can find them.

The underlying work is web fundamentals: semantic HTML, accessibility, honest structured data, and clear contracts for machines. Those were worth doing before AI crawlers existed. Now there’s a new class of reader that depends on them.

Cagri Sarigoz( Founder )

I’m founder of BizStack at Cagri Sarigoz LLC and a passionate advocate for entrepreneurs.

With over 14 years in tech, marketing, and AI, including my role as Head of SEO at CitizenShipper and co-founder of TaleBot at Intale AI, I’m dedicated to sharing genuine, useful product insights and tips.

At BizStack, I aim to cut through the digital noise to provide clear, actionable advice.

And more than all else, I’m a father to a (always) little girl and a husband.

Contact me at [email protected] for assistance.