Robots.txt generator and analyzer
Build a robots.txt file with a visual editor or analyze an existing one, then test URL paths against crawler rules. It runs entirely in your browser.
User-agent block 1
User-agent
Rules
Crawl-delay (seconds, optional)
Need data from a webpage as JSON?
PulpMiner turns any public page into structured JSON and can save it as a live API. See it work on a demo page, no sign-up.