How to Test CSS Selectors for Web Scraping

You write a scraper, run it, and get back an empty array. Or worse, 400 results when the page has 20 products, because your selector also caught every price in the “You might also like” carousel. Most scraping bugs are selector bugs, and the fix is to test the selector before it goes anywhere near your code.

What are CSS selectors

A CSS selector is a pattern that matches HTML elements. They were invented for styling, but they’re just as useful for finding elements in JavaScript, in end-to-end tests with tools such as Playwright, and in scrapers that pull titles and prices out of a page.

A selector can be as short as p or a long chain of classes, attributes and pseudo-classes. The longer it gets, the more it pays to try it on real HTML first.

Types of CSS selectors

Class and ID selectors

A class selector starts with a dot and an ID selector with a hash. Lots of elements can share a class. An ID should appear only once on a page.

CSS

.product-title { color: #333; }
#main-header { font-size: 24px; }

Attribute selectors

Attribute selectors match on whether an attribute exists or what its value is. They’re a lifesaver when a page has no useful class names but does have data attributes or predictable links.

CSS

/* Links whose href starts with "https" */
a[href^="https"] { color: green; }

/* Email inputs */
input[type="email"] { border: 1px solid blue; }

/* An element with a specific data-testid */
[data-testid="add-to-cart"] { cursor: pointer; }

Besides ^= (starts with), there’s $= (ends with), *= (contains) and ~= (one word in a space-separated list).

Pseudo-classes

Pseudo-classes start with a colon and match elements by state or position, such as :first-child, :last-child, :nth-child(n) and :not(). They let you pick the third row of a table or the last item in a list without a unique class.

CSS

/* Every other table row */
tr:nth-child(even) { background: #f9f9f9; }

/* The first item in each list */
li:first-child { font-weight: bold; }

/* All divs without the header class */
div:not(.header) { margin-top: 10px; }

One that bites everyone eventually: li:first-child means “an li that is the first child of its parent”, not “the first li”. If the parent starts with some other element, nothing matches. li:first-of-type is the one you probably wanted.

Combinators

Combinators describe how two elements are related:

A space (descendant) matches an element anywhere inside another, at any depth.
> (child) matches only direct children.
+ (next sibling) matches the element straight after another.
~ (subsequent sibling) matches every later sibling.

CSS

/* Paragraphs that are direct children of .article */
.article > p { line-height: 1.8; }

/* A paragraph immediately after an h2 */
h2 + p { font-size: 18px; }

/* Every paragraph after an h2 with the same parent */
h2 ~ p { color: #555; }

Understanding CSS specificity

When two rules try to set the same property on an element, specificity decides who wins. It’s counted as three numbers: how many IDs; how many classes, attributes and pseudo-classes; and how many element names and pseudo-elements. Inline styles beat any selector, and !important beats both.

So #nav .link scores (1, 1, 0) and beats .nav .link at (0, 2, 0). One ID outweighs any number of classes.

If you’re here for scraping, here’s the good news: specificity only decides which styles apply. It has no effect on what querySelectorAll() or a scraping library returns, which is simply every element the selector matches.

CSS selectors for web scraping

Most scraping tools speak CSS selectors: Cheerio in Node.js, Beautiful Soup’s select() in Python, and the browser’s own document.querySelectorAll(). A good scraping selector is specific enough to hit only what you want, and loose enough that it still works after the site’s next redesign.

JavaScript

const cheerio = require("cheerio");
const $ = cheerio.load(html);

// All product names
const names = $(".product-card > h3.title")
  .map((i, el) => $(el).text())
  .get();

// Category links in the navigation
const links = $("nav a[href^='/category/']")
  .map((i, el) => $(el).attr("href"))
  .get();

A few habits keep selectors alive longer:

Lean on stable hooks such as IDs, data-* attributes and meaningful class names, not long positional chains like div > div:nth-child(3) > span.
Avoid auto-generated class names such as css-1x2y3z. They change every time the site is rebuilt, and your scraper breaks on a Tuesday for no visible reason.
Test against the HTML your scraper actually receives. Browser developer tools show the page after its JavaScript has run, which can look quite different from the raw HTML a simple HTTP request gets.

How to use the CSS selector tester

Open the CSS selector tester. Paste HTML into the HTML box, or type a page address under Fetch HTML from a URL to load that page’s HTML.
Type a selector, or start from one of the example chips such as .item or .products > p.
Select Test selector or press Enter. You’ll see how many elements matched and the full HTML of each one. If nothing matches, or the selector is invalid, it tells you.
Tweak the selector until it matches exactly what you need. You can Copy the matched HTML too.

Selectors run in your browser on the same engine as document.querySelectorAll(), so anything your browser supports works, including :not(), :is() and :has(). HTML you paste stays in your browser. When you fetch a URL, PulpMiner’s server downloads the page and sends back its HTML.

How to test CSS selectors for web scraping