How to test XPath expressions online

Your scraper worked fine on Monday. On Tuesday the site renamed a CSS class and now every price comes back empty. An expression anchored on something sturdier, like a data attribute or the label next to the price, would have survived. XPath can do that, and testing against real HTML first is how you know it works.

What is XPath

XPath (XML Path Language) is a query language for picking things out of an XML or HTML document. You describe a path through the document tree, and it returns the elements, attributes or text at the end of that path. It was designed for XML, but it works just as well on HTML. That makes it a staple of web scraping.

Browsers support XPath 1.0 out of the box. Open DevTools in Chrome or Firefox, press Ctrl+F (Cmd+F on a Mac) in the Elements or Inspector panel, and type an expression. Matching nodes get highlighted on the page. For trying expressions against a snippet of HTML you copied from somewhere, an online tester is quicker.

Selenium, Playwright, Puppeteer, Scrapy and lxml all accept XPath, so one expression often works in your scraper, your browser tests and your data pipeline.

XPath syntax basics

Think of a document as a set of nested boxes. An expression gives directions through those boxes, and the two most important operators are the single and the double slash:

• / steps to a direct child. Starting at the root, /html/body/div selects only <div> elements that are direct children of <body>.
• // matches at any depth. //div selects every <div> in the document, however deeply it is nested.

The expressions you’ll use most:

Axes

Slashes only go down. Axes let you move in other directions: up to a parent or ancestor, or sideways to a sibling. They’re how you grab an element with no useful class or ID by starting from a neighbour that has one.

//span[@class='price']/parent::div            parent of each price
//h2[1]/following-sibling::p                   paragraphs after the first h2
//h2[3]/preceding-sibling::h2                  h2 elements before the third one
//div[@id='content']/descendant::a             links anywhere inside #content
//div[@id='content']/descendant-or-self::*     #content and everything in it
//span[@class='error']/ancestor::form          the form around an error message

Predicates

A predicate is a filter in square brackets. It keeps only the nodes that pass a test on their attributes, text or position:

//input[@type='email']                        exact attribute value
//div[contains(@class, 'product')]             attribute contains a string
//a[text()='Read more']                        exact text
//a[contains(text(), 'Read')]                  partial text
//input[@type='text' and @name='username']     both conditions
//input[@type='email' or @type='tel']          either condition
//ul[@class='menu']/li[2]                      second item of the menu
//div[not(@class)]                             divs without a class

XPath for web scraping: a worked example

Here’s a small product listing, the kind of thing you’d scrape from an online shop:

<html>
<body>
  <div id="products">
    <div class="product-card">
      <h2 class="title">Wireless Mouse</h2>
      <span class="price">$29.99</span>
      <a href="/products/wireless-mouse">View Details</a>
    </div>
    <div class="product-card">
      <h2 class="title">Mechanical Keyboard</h2>
      <span class="price">$89.99</span>
      <a href="/products/mechanical-keyboard">View Details</a>
    </div>
    <div class="product-card sold-out">
      <h2 class="title">USB-C Hub</h2>
      <span class="price">$45.00</span>
      <a href="/products/usb-c-hub">View Details</a>
    </div>
  </div>
</body>
</html>

And here’s how to pull the data out of it:

All product titles
//h2[@class='title']/text()
-> Wireless Mouse, Mechanical Keyboard, USB-C Hub

All prices
//span[@class='price']/text()
-> $29.99, $89.99, $45.00

All product links
//div[contains(@class, 'product-card')]/a/@href
-> /products/wireless-mouse, /products/mechanical-keyboard, /products/usb-c-hub

Products that are not sold out
//div[contains(@class, 'product-card') and not(contains(@class, 'sold-out'))]/h2/text()
-> Wireless Mouse, Mechanical Keyboard

The second product's price
(//div[contains(@class, 'product-card')])[2]/span[@class='price']/text()
-> $89.99

Notice the contains() on the cards. The third card has class="product-card sold-out", and @class= compares the whole attribute string. So //div[@class='product-card'] would only find the first two, and the USB-C hub would silently go missing from your data.

The parentheses in the last expression matter too. (//div)[2] is the second match in the whole document, while //div[2] is every div that is the second div inside its own parent.

XPath vs CSS selectors

If you’ve written CSS, you already know half of this. Here are the common cases side by side:

Yes, the by-class XPath is ugly. The simpler contains(@class, 'card') also matches cards and postcard. Putting a space on both sides of the class list and of the name matches whole class names only, which is what .card does in CSS.

CSS selectors are shorter for simple queries, and most JavaScript tools reach for them first. XPath pulls ahead in three areas:

1. Selecting by text. XPath can match an element by its text. Standard CSS can’t.
2. Moving up and back. XPath has parent, ancestor and preceding-sibling axes. CSS has only gained limited upward selection through :has(), and many scraping libraries do not support it.
3. Complex conditions. Predicates combine boolean logic, positions, string functions and number comparisons in one expression.

A good rule of thumb: use CSS when it’s enough, and switch to XPath when you need to match text, move upwards or filter on several conditions.

Testing XPath in the browser

In the Chrome and Firefox DevTools console, $x() runs an XPath expression against the current page and returns an array of matches:

$x("//h1"); // all <h1> elements
$x("//a/@href"); // href attribute nodes
$x("//div[contains(@class, 'product')]//span[@class='price']/text()"); // price text nodes

In page scripts, use document.evaluate():

const result = document.evaluate(
  "//h2[@class='title']",
  document,
  null,
  XPathResult.ORDERED_NODE_SNAPSHOT_TYPE,
  null,
);

for (let i = 0; i < result.snapshotLength; i++) {
  console.log(result.snapshotItem(i).textContent);
}

How to use the XPath tester

The console only works on the page you have open, after its JavaScript has run. Your scraper may see something quite different, so it pays to test against the exact HTML it will receive.

1. Open the XPath tester. Paste HTML into the HTML box, or enter a page URL and select Fetch HTML to load its source.
2. Type an expression, or click one of the examples such as //a/@href or //p/text().
3. Select Evaluate or press Enter. The tool shows how many nodes matched and, for each one, its text and its HTML. Attribute matches are shown as name="value".
4. Select Copy to copy every match as text.

The expression runs in your browser’s built-in XPath 1.0 engine, against the HTML as it is. The page’s JavaScript isn’t run, so content a site adds with JavaScript won’t be in fetched HTML. If your expression finds nothing on a page that clearly shows the data, that’s the usual reason.

The tester shows matching nodes only. Expressions that return a number or a string, such as count(//a), are reported as errors, and text nodes that are only whitespace are skipped.

XPath tips for web scraping

• Avoid absolute paths. An expression like /html/body/div[3]/div[2]/ul/li[1]/a breaks the day marketing adds a cookie banner. Anchor on attributes and use relative paths instead.
• Prefer data attributes to classes. Class names change in redesigns. Attributes such as data-product-id are often tied to application logic and change less.
• Match the stable part of generated classes. For a class such as card_a1b2c3, use starts-with(@class, 'card_') or contains(@class, 'card_').
• Normalise whitespace. normalize-space() trims the ends and collapses inner runs of whitespace, which makes text matches reliable.
• Build up step by step. Start broad, such as //div, and add one predicate at a time until only the nodes you want are left.

//p[normalize-space()='Hello World']                  text match that ignores extra whitespace
//div[starts-with(@id, 'product-')]                    IDs with a fixed prefix
//p[normalize-space()]                                 paragraphs that are not empty
//a[contains(@href, '/products/') and normalize-space()]  product links with text