Robots.txt compliance check for polite crawlers and AI agents (GPTBot, ClaudeBot, ...). Pass a page URL and optionally your crawler's product token; returns whether the URL may be fetched, the winning Allow/Disallow rule, crawl-delay and sitemap URLs. Full RFC 9309 matching: wildcards, $ anchors, longest-match, merged groups. A 4xx robots.txt allows all; 5xx or unreachable hosts report disallowed. Cached per host for 1 hour.