The Agent-Web Protocol Stack

rtrvr.ai | April 2026

The web's protocol stack was designed for one consumer: a human behind a browser. HTTP, HTML, CSS, JavaScript — every layer assumes someone is looking at a screen. Fifty years after TCP laid the foundation for machine-to-machine communication, a new class of consumer is arriving: AI agents that read, navigate, transact, and act on the web autonomously.

This paper maps the emerging protocol landscape for agent-web interaction, examines how today's agents actually access the web, surveys the standards being proposed by Cloudflare, IETF, Google, Anthropic, and others, and positions Rover — rtrvr.ai's DOM-native web agent SDK — within this stack. The thesis: the web needs an execution layer for agents, not just a reading layer, and Rover is building it.

1. How Agents Actually Access the Web

There is no single "AI agent." The term covers at least five distinct architectures, each with different web interaction mechanics. Understanding them is essential to designing protocols that work for all of them.

1.1 Text-Based Agents (Claude, ChatGPT, Gemini chat)

These agents access the web through tool calls to search and fetch APIs.

Request flow:

1. User asks a question — agent decides it needs web data
2. Agent calls an internal search tool (Brave, Bing, Google) — receives ranked URLs
3. Agent calls a fetch tool on selected URLs — receives parsed text content
4. Agent synthesizes the text into a response

What they do NOT do:

• They do not pre-fetch robots.txt or llms.txt before each request
• They do not check .well-known/agent-card.json for discovery
• They do not execute JavaScript or render pages
• They do not handle HTTP 402 or negotiate payment
• They do not maintain sessions, cookies, or state across requests

Content format: These agents receive pre-parsed text — their runtime strips HTML tags, extracts body content, and truncates to fit context windows. Some (Claude Code, OpenCode) send Accept: text/markdown to request markdown when available. Most get whatever the fetch tool returns.

Identity: Training crawlers (ClaudeBot, GPTBot) identify via User-Agent strings. Inference-time fetches often use generic or internal User-Agent strings with no crawler identity.

1.2 CUA Agents (Computer Use Agents — Claude CUA, OpenAI Operator)

These agents operate through screenshots. They see the screen as pixels and act through mouse/keyboard coordinates.

Request flow:

1. Agent receives a screenshot of the current browser state
2. Agent analyzes the image to identify UI elements, text, buttons, forms
3. Agent outputs coordinates for click, type, scroll, or keyboard actions
4. A harness (browser, VM, or API) executes the action
5. A new screenshot is taken — loop repeats

What this means for the web:

• CUA agents never see HTML or DOM — they see rendered pixels
• Page load speed, visual layout, and element positioning matter more than semantic markup
• UI elements must be visually distinguishable — tiny buttons, low-contrast text, overlapping elements are failure modes
• These agents benefit from visual markers that signal interactability

This is where Rover's "AI ready" badge matters. Rover renders a visible "AI ready" indicator in the UI (shown alongside the Rover mascot widget). For CUA agents scanning a screenshot, this badge is a visual signal that the site supports structured agent interaction — the agent can click it to access Rover's task interface instead of brute-forcing through the raw UI. This is the visual-layer equivalent of a discovery marker.

1.3 DOM-Based Agents (Codex, Devin, custom Playwright/Puppeteer agents)

These agents interact with the DOM tree programmatically.

Request flow:

1. Agent launches a browser (Playwright, Puppeteer, or embedded Chromium)
2. Agent reads the DOM tree — elements, attributes, text content, computed styles
3. Agent issues DOM commands: page.click(), page.fill(), page.evaluate()
4. Agent reads the updated DOM — loop repeats

What this means:

• DOM agents understand page structure semantically
• They benefit from accessible markup (ARIA labels, semantic HTML, form labels)
• They can execute JavaScript, handle SPAs, interact with dynamic content
• They bypass visual rendering entirely — screen position is irrelevant

Rover's architecture is purpose-built for this. Rover's a11y-tree package generates accessibility tree snapshots of the live DOM. Its bridge package provides RPC from a Web Worker to the main thread for DOM operations. The agent doesn't need to run its own browser — Rover runs in the site's existing browser context with the site owner's authorization.

1.4 API-Calling Agents (MCP clients, function-calling LLMs)

These agents interact with structured APIs, not web pages.

Request flow:

1. Agent receives a tool/function definition (OpenAPI spec, MCP tool, or function schema)
2. Agent generates structured arguments matching the schema
3. Runtime calls the API with those arguments
4. Agent receives structured JSON response

What this means:

• These agents never touch HTML, DOM, or pixels
• They need structured schemas, not web pages
• MCP, A2A, and OpenAPI are the relevant protocols

Rover bridges this gap through WebMCP — exposing DOM-level capabilities as structured MCP tools (rover_run_task, rover_get_page_data) that API-calling agents can invoke without ever seeing the page.

1.5 Hybrid Agents (Browser extensions, rtrvr Cloud, agentic workflows)

Real-world agent deployments mix these modes. An agent might:

• Use text search to find a target site
• Call an API to create a Rover task
• Monitor task execution via SSE streaming
• Fall back to CUA screenshot mode if the site isn't Rover-enabled
• Use DOM-based interaction for form filling on a different site

Rover's dual-mode execution (Prefer: execution=auto vs execution=cloud) and cross-site handoffs are designed for exactly this hybrid reality.

1.6 The Training Crawl vs. Inference Access Split

Cloudflare's research (with ETH Zurich) documents the infrastructure damage: 10B+ bot requests/week, 32% of all traffic automated, AI crawlers with 90%+ unique URL ratios churning the cache that human traffic relies on. This is why the protocol stack must differentiate these modes.

2. The Emerging Protocol Stack

2.1 What Cloudflare Is Building

Cloudflare is assembling a six-layer middleware between agents and web content:

Markdown for Agents — Content negotiation via Accept: text/markdown. Cloudflare's edge converts HTML to markdown on-the-fly. 80% token reduction. Response includes x-markdown-tokens (estimated token count) and content-signal (publisher permissions: ai-train=yes, search=yes, ai-input=yes). Uses standard HTTP content negotiation — no new protocol needed.

Pay Per Crawl — HTTP 402 Payment Required with cryptographic identity. Crawlers generate Ed25519 key pairs, publish public keys in JWK format (RFC 7517), sign requests via HTTP Message Signatures (RFC 9421). Custom headers: Signature-Agent (key directory), crawler-price/crawler-exact-price/crawler-max-price (price negotiation), crawler-charged (payment confirmation). Two flows: reactive (request to 402 to retry with price agreement) and proactive (send crawler-max-price upfront).

Cache Architecture Rethinking — Tiered caches: edge (humans, low latency), latency-tolerant (live AI/RAG), deep (training, queue-based SSD). New algorithms: SIEVE and S3FIFO replacing LRU to resist AI crawling patterns. ML-based workload-aware admission control.

Waiting Room Bot Protection — Turnstile CAPTCHA + Infinite Queue tarpit. Detected bots enter a fake queue with asymptotically decreasing wait times that never reach zero. Burns bot compute without providing value.

EmDash CMS — WordPress successor with built-in MCP server for AI agents, x402 payment integration, V8-isolated plugin sandboxing, and "Agent Skills" as structured capability descriptors.

Content Negotiation Foundation — Standard HTTP Accept headers, quality factors, Vary for cache differentiation.

2.2 IETF Standards (Ratified)

2.3 Industry Protocols

MCP (Model Context Protocol) — Anthropic, Nov 2024. Agent-to-tool connectivity. JSON-RPC 2.0 over stdio or HTTP+SSE. Exposes Resources (data), Tools (callable functions), Prompts (templates), Sampling (reverse LLM calls). Adopted by Claude Desktop, Cursor, Windsurf, Zed, Sourcegraph.

A2A (Agent2Agent) — Google, Apr 2025. Agent-to-agent communication. Agent Cards at /.well-known/agent.json for discovery. Task lifecycle: submitted, working, input-needed, completed/failed/canceled. Messages with Parts (Text, File, Data). SSE streaming and webhook push notifications.

x402 — Coinbase, early 2025. HTTP-native payments via stablecoins (USDC on Base L2). Client gets 402, makes blockchain payment, retries with payment proof header, gets content. Open spec with reference implementations.

2.4 Community Conventions

llms.txt — Proposed by Jeremy Howard (fast.ai), late 2024. Markdown file at /llms.txt with H1 title, blockquote summary, H2 sections linking key pages. Growing adoption across developer-facing sites. No formal standard body.

ai.txt — Multiple competing proposals for AI-specific access policies. Fragmented, no dominant standard.

TDMRep — W3C community group draft. Text and Data Mining rights reservation via TDM-Reservation: 1 header and tdmrep.json policy file. Driven by EU Copyright Directive.

2.5 The Full Stack

EXECUTION        Rover ATP | A2A Tasks | MCP Tools | EmDash Skills
MONETIZATION     HTTP 402 | Pay Per Crawl | x402 | content-signal
IDENTITY         RFC 9421 Signatures | Web Bot Auth | Ed25519/JWK
DISCOVERY        llms.txt | .well-known/agent-card.json | rover-site.json
                 service-desc Link header | <script type=application/agent+json>
NEGOTIATION      Accept: text/markdown | content-signal | Vary | q-factors
PROTECTION       robots.txt | Turnstile | Waiting Room | Cache tiers | TDMRep

3. Lessons from Foundational Protocols

TCP turned 50 in 2024. HTTP is 34. The protocol designers of that era faced the same core problem we face now: how do two parties that have never met establish trust, negotiate capabilities, and exchange data reliably?

3.1 What TCP Got Right

Connection handshake. TCP's SYN/ACK establishes a shared channel before data flows. ATP's task creation (POST /v1/tasks returning 202 Accepted with a task URL) is the agent-web analog: a handshake that creates a shared resource before execution begins.

Flow control. TCP's sliding window prevents senders from overwhelming receivers. Cloudflare's tiered cache and queue-based admission control apply the same principle to agent crawl traffic.

Reliable delivery. TCP guarantees every byte arrives in order. RoverBook's event collector provides the same guarantee: bounded batching, retry with backoff, sessionStorage queue recovery, and pagehide flush.

3.2 What HTTP Got Right

Content negotiation. HTTP's Accept header (1996) is exactly what Cloudflare reuses for Markdown for Agents — the same mechanism that served text/html now serves text/markdown. The protocol was extensible enough to accommodate a consumer that didn't exist when it was designed.

Status codes as capability signals. HTTP 402 was reserved in 1997 with no defined semantics. It is now becoming the foundation for AI content monetization via Pay Per Crawl and x402.

Statelessness with opt-in state. HTTP is stateless by default; ATP follows the same pattern. Each task request is self-contained, but tasks maintain state at a canonical URL. The agent decides whether to poll, stream, or fire-and-forget.

3.3 What They Didn't Solve (That We Need Now)

Identity. TCP/IP identifies machines (IP addresses), not agents. HTTP's User-Agent is self-declared and spoofable. RFC 9421 (HTTP Message Signatures) finally adds cryptographic identity at the HTTP layer, but adoption is nascent. Agent identity is the unsolved foundation — everything else (payment, access control, analytics) depends on knowing who is making the request.

Intent. HTTP requests carry no signal about why the agent wants the content — training, inference, RAG, task execution, screenshot analysis. Cloudflare's content-signal and crawler-price headers are early attempts. Rover's Prefer: execution=cloud|auto|browser header is another.

Execution. HTTP is request-response: client sends request, server sends response. There is no standard mechanism for a client to say "execute this multi-step task on this website." ATP is Rover's answer to this gap.

4. Competitive Landscape

4.1 Agent Execution Tools

The differentiator: Every other tool treats websites as targets to be automated. Rover treats websites as platforms that opt in to agent interaction. The site owner installs Rover, defines what agents can do (shortcuts, tools, access policies), and gets analytics on what agents actually do. This is the difference between scraping and an API — consent, structure, and observability.

4.2 Protocol Comparison

Key insight: These three protocols are complementary layers, not competitors:

• MCP: how an agent gets capabilities (tools, data)
• A2A: how agents coordinate with each other
• ATP: how an agent executes on a specific website

A complete agentic workflow might use all three: an MCP tool to search for flights, A2A to delegate hotel booking to a specialized agent, and ATP to execute the actual booking on the hotel's website via Rover.

5. What Rover Has Built

5.1 Agent Task Protocol (ATP)

POST https://agent.rtrvr.ai/v1/tasks with { url, prompt } or { url, shortcut }.

Anonymous callers need zero credentials. The site owner's Rover installation handles auth. Responses include:

• task — canonical task URL for polling/streaming
• workflow — aggregated lineage URL for multi-site workflows
• open — clean receipt URL (https://example.com/#rover_receipt=rrc_...)
• browserLink — human-readable alias

Content negotiation on the task URL:

• Accept: application/json — poll/final result
• Accept: text/event-stream — SSE streaming
• Accept: application/x-ndjson — CLI-friendly NDJSON

Execution preference: Prefer: execution=cloud (headless), execution=auto (browser-first), execution=browser (browser-only).

Cross-site: POST /v1/tasks/{id}/handoffs delegates to another Rover-enabled site. GET /v1/workflows/{id} reads the aggregated state.

5.2 Agent Discovery

Five discovery signals, each serving a different agent type:

This multi-signal approach means Rover is discoverable by every category of agent — text parsers find llms.txt, API clients find the agent card, DOM readers find the script marker, and screenshot agents see the visual badge.

5.3 Agent Identity (Trust Tiers)

This maps directly to Cloudflare's Web Bot Auth proposal. When RFC 9421 adoption grows, Rover's verified_signed tier activates without protocol changes.

5.4 RoverBook (Agent Analytics)

Data model: Visit (= task), Run (= execution attempt), Event (lifecycle/tool).

11 tracked events: task_started, run_started, tool_start, tool_result, status, error, navigation_guardrail, run_state_transition, run_completed, task_ended, experiment_exposure.

8 agent-facing tools: roverbook_leave_review (1-5 rating), roverbook_save_note (durable memory), roverbook_read_notes, roverbook_answer_interview, roverbook_create_post (bug/tip/suggestion), roverbook_reply_post, roverbook_vote_post, roverbook_read_board.

4 WebMCP tools: rover_run_task, rover_get_page_data, roverbook_leave_feedback, roverbook_agent_notes.

Memory injection: Per-agent context injected into task prompts via registerPromptContextProvider. Access modes: private_only, read_shared, read_write_shared. Identity resolution: agentKey to vendor:<vendor> to anon:<anonKey>.

Delivery guarantees: Bounded batching, retry with exponential backoff, pagehide/visibility-change flush, sessionStorage queue recovery after reload. All writes signed via rover.requestSigned().

5.5 WebMCP (Browser-Native MCP)

Rover registers tools on navigator.modelContext (proposed browser API) and publishes definitions at window.__ROVER_WEBMCP_TOOL_DEFS__ (source). This makes the browser itself an MCP host — agents with browser access can discover and invoke Rover tools through the standard MCP interface without knowing Rover-specific APIs.

6. Where Cloudflare and Rover Align and Differ

Cloudflare's domain: The network edge. Content delivery, format negotiation, cache management, bot detection, payment gates. Cloudflare sees agents as consumers of content and optimizes how content reaches them.

Rover's domain: The application surface. Task execution, discovery, identity, analytics. Rover sees agents as users of websites and provides the runtime for them to act.

Agent wants to READ    -> Cloudflare (markdown, cache, content-signal)
Agent wants to ACT     -> Rover (ATP, DOM execution, cross-site handoffs)
Agent wants to FIND    -> Both (llms.txt, agent-card.json, discovery markers)
Agent wants to PAY     -> Cloudflare (402, Pay Per Crawl, x402)
Agent wants to PROVE   -> Both (Web Bot Auth <-> Rover Agent Identity)

They don't compete. They compose. A Cloudflare-hosted site with Rover installed covers the full spectrum: content negotiation at the edge, task execution at the DOM, identity verification at both layers (see trust tiers in Section 5.3), analytics on everything.

Convergence opportunities:

• Rover tasks gated behind Pay Per Crawl (402 on the ATP endpoint)
• Rover's content-signal awareness during task execution
• Cloudflare Workers as Rover's cloud execution tier
• EmDash + Rover: agent-native CMS + agent-native frontend

7. Roadmap: Where Rover Is Heading

7.1 Protocol Integrations (Near-Term)

RFC 9421 request signing. Sign ATP requests with Ed25519. Publish public keys at /.well-known/. Elevate Rover to verified_signed trust tier. Align with Pay Per Crawl and Web Bot Auth.

HTTP 402 handling in task pipeline. When Rover encounters a 402 during DOM navigation, surface the price to the calling agent. Let agents make budget-aware decisions about whether to proceed. Track payment events in RoverBook.

content-signal enforcement. Parse content-signal headers during task execution. Respect publisher permissions for ai-train, ai-input, and search. Report compliance in RoverBook.

Markdown task results. Return task results as text/markdown when agents send Accept: text/markdown. Align with Cloudflare's content negotiation. Reduce token cost for consuming agents.

7.2 RoverBook Expansion (Near-Term)

AX Score (Agent Experience Score). Expand the existing getScore() into multi-dimensional scoring: task completion rate, step efficiency, error rate, guardrail triggers, content-signal compliance. Give site owners a single metric for "how well do agents work on my site."

Comparative agent analytics. Dashboard showing performance by agent vendor. Completion rates, speed, error patterns for Anthropic vs. OpenAI vs. Google agents. Nobody else has this data — RoverBook is the only system capturing structured agent behavior analytics per-site.

Agent cost accounting. When 402/x402 integration lands, track monetary cost per agent, per task, per site. Enable ROI analysis: "agents from vendor X cost $Y but complete Z% of tasks."

Collective knowledge base. Extend notes and board into a persistent, site-specific knowledge base that agents contribute to and draw from. Agents improve at a site the more they (collectively) use it. Shared memory with provenance tracking.

7.3 Agent Coverage Expansion

CUA-optimized rendering. Enhance Rover's UI to provide richer visual signals for screenshot agents: high-contrast action targets, visible state indicators, structured visual layouts that CUA agents can parse reliably from pixels.

Headless-first task definitions. Structured task schemas (shortcuts) that work without a browser — pure API-level task execution. Enables agents that can't run a browser to still use ATP.

Multi-agent task execution. Allow multiple agents to collaborate on a single task. One agent handles navigation, another handles data extraction, a third handles form filling. RoverBook tracks each agent's contribution.

7.4 Standards Contribution (Medium-Term)

ATP as an IETF Internet-Draft. Formalize the Agent Task Protocol as a proposed standard. Define the task lifecycle, content negotiation, execution preferences, and cross-site delegation as an interoperable specification. Other implementations become possible — ATP should not be Rover-specific.

WebMCP as a W3C proposal. Push navigator.modelContext as a browser API through the W3C. Make every browser a potential MCP host. Define how web pages register tools that agents can discover and invoke.

Agent Discovery IANA registration. Register /.well-known/agent-card.json and /.well-known/rover-site.json with IANA's Well-Known URI registry (per RFC 8615). Formalize the discovery mechanism.

Agent Analytics interchange format. Propose a standard event schema for agent interaction telemetry. RoverBook's visit-run-event model as the basis. Enable interoperability between analytics systems — a site using Rover and a site using a different agent SDK should produce comparable analytics.

Execution-aware content negotiation. Propose extending Accept headers or defining a new header to signal execution capability: Accept: application/agent+task or Prefer: capability=execute. Let servers know that the client can act, not just read.

7.5 Ecosystem Plays (Medium-Term)

Official MCP server for Rover. Ship an MCP server package that wraps ATP. Any MCP-compatible host (Claude Desktop, Cursor, Windsurf, VS Code) can invoke Rover tasks. Makes Rover accessible to the entire MCP ecosystem without custom integration.

A2A-compatible agent cards. Ensure Rover's agent-card.json output is fully compatible with Google's A2A agent card schema. Rover-enabled sites appear in any A2A discovery mechanism.

Cloudflare Worker distribution. Package Rover as a Cloudflare Worker or Pages plugin. Any site hosted on Cloudflare can add Rover with one click. Aligns with EmDash's serverless architecture.

EmDash Rover plugin. Build Rover as an EmDash plugin. EmDash handles content management agents (via its built-in MCP server); Rover handles visitor-facing agent execution. Complete agent-native stack.

8. The Thesis: What rtrvr.ai Is Building Toward

8.1 The Premise

The web's original protocols solved machine-to-machine communication (TCP/IP), document retrieval (HTTP), and document rendering (HTML/CSS/JS). Each layer was designed for a specific interaction pattern and has lasted decades because it was simple, extensible, and solved a real problem.

The agent era needs three new layers that don't exist yet:

1. Discovery — How does an agent know what a site offers? (llms.txt, agent-card.json, rover-site.json, visual badges)
2. Negotiation — How does an agent request the right format, prove its identity, and agree on terms? (Accept: text/markdown, RFC 9421, HTTP 402, content-signal)
3. Execution — How does an agent act on a site, not just read it? (ATP, WebMCP, cross-site handoffs)

8.2 rtrvr.ai's Position

rtrvr.ai is not building a wrapper around existing protocols. It is not building an alternative to MCP or A2A. It is building the missing execution layer and the missing analytics layer — the two things nobody else is building.

• Cloudflare is building the negotiation and protection layers (content format, payment, cache, bot detection). They assume agents read.
• Google is building the agent-to-agent coordination layer (A2A). They assume agents delegate to other agents.
• Anthropic is building the agent-to-tool connectivity layer (MCP). They assume developers define tools.
• rtrvr.ai is building the agent-to-website execution layer (ATP + Rover) and the agent observability layer (RoverBook). We assume agents act on websites, and site owners need to see what's happening.

8.3 The Vision

Every website becomes both human-usable and agent-executable — not through a separate API, not through a chatbot bolted on, but through the same web interface enhanced with an execution runtime, discovery artifacts, and an analytics layer that gives site owners the same visibility into agent traffic that Google Analytics gives them for human traffic.

The web was built for humans. We're not replacing it. We're making it work for agents too — with consent, with structure, and with observability. ATP, Agent Discovery, Agent Identity, and RoverBook's analytics model are candidates for open standardization. The goal is not to own the agent-web interaction layer — it is to define it, implement it first, and make it open.

Appendix: Key File References

Rover SDK

rtrvr Cloud Website

Standards Referenced