You need a competitor’s prices every morning. There’s no API, copying them by hand gets old by Wednesday, and the scraper you wrote last quarter broke the day they renamed a CSS class.
Here’s why that keeps happening, and how to get the same data as JSON without maintaining a scraper at all.
The problem with web data
Product listings, job posts, property listings and news articles are published as HTML for people to read, and most sites offer no API for them. Getting that data into a program has usually meant writing a scraper.
A scraper fetches the page, parses the HTML and picks out elements with CSS selectors or XPath (two ways of pointing at parts of a page). It breaks whenever the site changes its layout or class names, which happens often, so keeping scrapers working tends to cost more than building them. A scraper is a bit like a houseplant: easy to get, surprisingly hard to keep alive.
Selector scraping and AI extraction
A traditional scraper is tied to the exact HTML structure:
const titles = document.querySelectorAll(".product-card h2.title");
const prices = document.querySelectorAll(".product-card .price-tag span");
const ratings = document.querySelectorAll(".product-card .stars-count");
// If the site renames ".price-tag span" to ".cost-display",
// prices comes back empty and nothing tells you why.AI extraction reads the page content instead of following fixed selectors. It recognizes a product name, a price or a publish date by what it is, not by where it sits in the markup, so a redesign is much less likely to break it. It isn't infallible, so check the output for pages that matter.
Extracting a page with the free tool
- Open Website to JSON and paste the URL of a public webpage.
- Select Extract JSON. PulpMiner reads the page and the AI writes its content as structured JSON, usually in a few seconds.
- Copy JSON to use it in your code, a spreadsheet or a database.
The free tool needs no sign-up. It decides the structure itself and returns the JSON once. A product listing page might come back like this:
{
"products": [
{
"name": "Nimbus Wireless Headphones",
"price": "$49.99",
"rating": 4.5,
"reviews": 1283
},
{
"name": "Harbor 7-in-1 USB-C Hub",
"price": "$29.99",
"rating": 4.2,
"reviews": 856
}
]
}Turning a page into a live API
With a free account, the Generate API screen gives you more control and lets you save the result as an endpoint:
- Render JavaScript for single-page apps and content that loads after the page.
- Instructions to tell the AI what you want, for example “get all 20 products with name, price and rating”.
- A CSS selector to read only part of the page, plus a choice of scraper and AI model.
- Edit the JSON before saving, so the endpoint returns exactly the fields you need.
- Save as API to get an endpoint, protected by your API key, that reads the page again on every call and returns fresh JSON. You can call it from code, Google Sheets, n8n or Zapier.
- Cache content so repeat calls answer faster, and Dynamic URL to turn parts of the URL into variables, so one endpoint serves every product or search page on a site.
Generating and calling endpoints uses credits, and rendering JavaScript costs extra. New accounts get 5 free credits, with no card required.
What people use it for
Price monitoring
Track competitor prices without checking each site by hand. A saved endpoint per product page gives you current prices whenever you call it, ready for a comparison sheet or dashboard.
Lead lists
Business directories and event pages list company names, roles and contact details. Extract them to JSON, then convert to CSV for your CRM. Check the site’s terms and the privacy rules that apply to personal data before you do.
Research
Researchers often combine data from government sites, academic listings and news. Extracting each source to JSON gives one format you can process with the same code.
Content aggregation
Job boards and listing sites pull from many sources into one feed. Because AI extraction doesn't depend on each site’s markup, you don't need a separate scraper per source.
Why skip the hand-written scraper
- Nothing to build. Paste a URL and get JSON, instead of writing and deploying scraping code.
- Works across layouts. The same approach handles different sites without per-site selectors.
- Less maintenance. With no selectors to break, a site redesign rarely means rewriting anything.
