09/09/2026
Here's something worth two minutes if you run a small site. Your robots.txt decides who reads your pages. It does nothing about who acts on them. Those used to be the same job. They aren't anymore.
A new standard called WebMCP lets AI agents do real things on a site — add to a cart, fill a form, start a checkout — through tools the site itself exposes. It's already shipping in products from OpenAI, Shopify and Cloudflare. So while everyone tunes their crawler rules, a second door has quietly opened, and most small sites haven't looked at it.
Keep managing crawlers with robots.txt. Just stop assuming that file protects anything that does things. We walk through what's actually different, and what to lock down first, here: https://tpc-hosting.com/blog/robots-txt-webmcp-ai-agents-acting