Charge the Training Bots. Leave Search Crawlers Alone.

TL;DR

Sort AI bots into two groups before you put a price on them. Leave verified search crawlers free when you want them to cite you. Charge training bots and unverified agents when the host can take the payment. On June 15, 2026, AWS showed that split on AWS WAF attached to CloudFront. Vercel does not run that switch. Write the two lines for the site you have, and leave the host where it is.

The bot that fetched the page

Picture a page you published on a Tuesday. A search bot may cite it in an answer. A training bot may copy it into a model. An unnamed agent may pull the whole page for a task you will never see. One URL, three visits.

On June 15, 2026, AWS announced AI traffic monetization for AWS WAF, its bot-control product. On a protected article, data feed, or licensed archive, AWS WAF can answer with HTTP 402, the status code that means Payment Required. The announcement says AWS WAF returns "a machine-readable HTTP 402 Payment Required response using the x402 open protocol for machine-to-machine payments." That response includes prices, accepted payment methods, and license terms. The agent presents proof of payment, WAF checks it, and the page is served in that same request. Payouts are in stablecoins, settled through Coinbase's x402 Facilitator. A stablecoin is a crypto token meant to track a dollar. Stripe account payments, and support for the Machine Payments Protocol, are listed as coming soon. They were not live in that announcement.

AWS put the split in the same post: "Publishers can apply differentiated pricing based on agent identity and intent, allow verified AI search crawlers at one price while charging a different price to unverified agents or training crawlers…" Policies can use verification status, including Web Bot Auth signatures. Web Bot Auth, here, is a signature AWS says a policy can use to check that a bot is who it claims to be. AWS says you can run a test mode before anything goes live, and that the feature is included at no additional charge beyond standard AWS WAF charges. The switch runs only where AWS WAF sits on Amazon CloudFront, Amazon's content network.^[1]AWS: "This capability is available in all edge locations where AWS WAF Web ACLs are associated with Amazon CloudFront distributions." A site on Vercel, including this one, does not have it.

I would set the verified search crawler's price at zero, and charge training bots and unverified agents, on a host that can collect. That zero is an owner choice. AWS described the split. AWS did not pick the number.

What x402 means in plain English

x402 is an open way for a machine to pay for a single request. HTTP has reserved status code 402, Payment Required, for years, and it has rarely been used. x402 puts that code to work so an agent can pay with a stablecoin inside the request, with no checkout page and no new account.

Coinbase launched x402 on May 6, 2025. The launch post says, "Coinbase is launching x402, a payment protocol that enables instant stablecoin payments directly over HTTP." The client asks for the page. The server answers 402 with the price. The client tries again with a signed payment. A facilitator, the service that checks the signature and settles the stablecoin, completes that step. The server then returns the page. Coinbase named AWS, Anthropic, Circle, and NEAR among early collaborators. Coinbase's CDP docs say the facilitator has processed more than 100 million x402 payments across Base and Solana, two crypto networks. That figure is Coinbase's count.

On April 2, 2026, the Linux Foundation said it is launching an x402 Foundation, with the protocol contributed by Coinbase. The press release names Amazon Web Services, Stripe, and Cloudflare among the companies around the effort. A name on that list means the company backed the standard. Production billing would be a separate announcement, and this press release does not include one.

Agents already pay Stacktree to publish a page

Stacktree's guide for agents shows a machine paying per request to publish a page on Stacktree, with no card and no signup. The unpaid request gets an HTTP 402. The agent signs a stablecoin payment. A facilitator settles it. Stacktree's published example for that publish is $0.50, and a persistent key in the same guide is $1. Treat those figures as Stacktree's prices for Stacktree's product.

Stacktree's explainer, updated June 27, 2026, describes the loop this way: "x402 is an open payment standard that revives HTTP 402. A server answers a request with 402 and payment terms; the agent signs a stablecoin authorization for the exact amount and retries; a facilitator settles it on-chain, on a network like Base or Solana." The same page says the Machine Payments Protocol, from Stripe and Tempo, does that job too, and that both let an agent pay per request with no account and no card.

A machine can pay per request on someone else's product. Nobody in that guide is paying you for a page on your site.

Keep verified search crawlers free

A verified AI search crawler fetches your page so an answer engine can cite it. Generative engine optimization, GEO for short, is the work of being the page those answers name. I would leave that crawler free when the page exists to be found. Zero is a price AWS allows you to set for verified search. A publisher whose product is a licensed archive may want a number above zero even for search. An owner whose pages are how clients discover the work should leave the citing crawler alone.

A fee gives that crawler a reason to skip you. This piece has no count of clients who arrived because an answer cited the page. The crawl may never show up as a person in your analytics, and you still will not get a receipt if a mention later sends someone.

A search crawler and a training crawler can look alike in an access log. The log is a list of names. A Web Bot Auth check is the signature inside AWS WAF, and a Vercel log does not run that check. Put a name on the free line when you already know it is a search crawler. Block a name only when you already know it is training a model. Leave every other name untouched. Blocking an unknown name can also block the crawler you wanted cited, and on Vercel you cannot charge that unknown bot instead.

Charge training bots and unverified agents

A training crawler reads the page to teach a model. The visit does not come back as a citation a person can click. An unverified agent is a machine you cannot identify. It might be fetching the page for someone's task, and it might be copying the archive. When a host can take a machine payment, those two should pay.

Free citation traffic is there to protect discovery and the name of the firm. A charge on training bots and unverified agents, once you can collect it, either takes a little money or turns away consumption that never sends a reader back. One high price for every bot puts that fee on the mention too. Say so on purpose if the archive itself is what you sell.

Vercel cannot collect with the AWS WAF switch described above. Hold the two prices as policy. Add a toll booth only when the host you already pay for can take the payment.

What to leave alone

Leave verified search crawlers in. A blanket block treats the citation crawler like a thief, and you lose the mention along with the scrape. Draw the line at the visit. Open weights, distillation, and free market dynamics makes the same kind of cut in another market: go after the abuse, and leave the useful case alone.

Keep two prices in the policy, and collect the second one only where the host can. Known search stays free. Known training pays, or that known name can be blocked. Unknown names stay off both lines until you can tell them apart.

Stay on the host you have. Moving the site onto CloudFront so a bot can pay is a project you would then have to keep running. The weekend build isn't the moat is about that shape of first version: easy to start, then a second job. Leave the move alone.

Takeaway

This week, write two lines in a note.

The first line is search crawlers you already know you want citing the site. Leave them free. The second line is crawlers you already know are training a model. They pay if the host can take the payment. Otherwise leave that known name for later, or block only that name.

Leave every name you cannot classify off both lines. Leave the hosting where it is.

When the public site is how people find the firm, and you want a second look at which names belong on the free line, that is a 20-minute conversation. I will look at the site you already have. I'll tell you if AI won't help. Simple conversation. No pitch, no commitment.

Book a discovery call

Footnotes

1. AWS: "This capability is available in all edge locations where AWS WAF Web ACLs are associated with Amazon CloudFront distributions." ↩