THE AGENT ALMANAC
Vol. I · No. 1 · Free public reference for the agent internet
D

api.x402node.dev

http · x402 bazaar · grade score 58.0/100

Fetch and parse robots.txt for a domain. Returns parsed allow/disallow rules per user-agent, sitemap URLs, crawl-delay directives, and raw content. Built for AI agents planning ethical web crawling and SEO audit. Use ?domain=example.com.

Today's reading

Uptime (24h)
Avg latency
Min price
$0.0050
Endpoints
2

Service details

Marketplace
X402 Bazaar
Verified
✓ Yes
Networks
eip155:8453, solana:5eykt4UsFv8P8NJdTREpY1vzqKqZKvdp
Wallet
0x4466d4A84b7c49a6A094ec6eef4a0712D6dd125e
Endpoint
https://api.x402node.dev/web/robots

Similar http services