extract-bench

A benchmark report · October 2026

My benchmark ranked extractors one way. Two sets of real pages ranked them two other ways.

Every AI agent that reads the web first turns HTML into text. I tested six open-source tools for that step on pages where the right answer is known, with 95% bootstrap intervals on every number and a label for every failure.

On my synthetic pages readability-lxml came first, but the top two are not significantly different once whole page types are resampled. On 40 Wikipedia articles jusText jumped from last to first. On 181 pages from 126 sites, with gold labelled by Zyte, readability-lxml and tuned trafilatura tie at 0.960 and jusText is fourth.

At a glance: rank in each slice

Figure 1. Rank by mean F1. jusText (thick red line) goes last, first, fourth. Hover a dot for its F1.
Contents

1The leaderboard

I generated 60 pages from a fixed seed: 10 page types, 6 variants each. The gold text is recorded while each page is built, so it is exact. Twenty pages are dev, used only to tune one row. Every number in this section is from the other 40.

Each tool runs on library defaults. The one exception is a single extra row, trafilatura tuned on the dev pages and then frozen. F1 is bag-of-tokens overlap with the gold main text. Leak rate is the share of planted boilerplate snippets, like nav menus, cookie banners and reader comments, that got into the output. 1I wrote the pages and I wrote the gold. That is limit 2 at the end, and the reason section 2 exists.

Pages:
  • readability-lxml
  • trafilatura (best-config)
  • trafilatura, defaults
  • converters and baseline
  • jusText
Figure 2. Mean F1 on the synthetic test pages, with 95% bootstrap intervals (10,000 resamples over pages, seed 1234). The red band is where the top two intervals overlap. The toggle drops the 4 listing pages, whose gold counts only the intro line as content, a judgment call.

readability-lxml is first. Over pages, its lead over tuned trafilatura looks real: +0.085 [+0.018, +0.166]. But the 4 test pages of a type are built from one template, so they are not independent. When I resample whole page types instead, the same gap has an interval that crosses zero. 2Most of the gap is one page type. On listings, trafilatura scores 0.21 and readability-lxml 0.88 (Figure 4).

Figure 3. readability-lxml minus trafilatura (best-config), paired bootstrap on the per-page F1 difference. The same gap is significant or not depending on what counts as the unit. A filled dot means the interval clears zero.
Click a column to sort.
Table 1. Synthetic test split, 40 pages. Latency is the median of 3 runs per page on one Windows laptop, so it is only good for ordering. Title match is n/a for tools that have no title call. The config each row ran with is under its name.
Figure 4. Mean F1 by page type. 4 test pages each, treat as rough. Darker means higher; the shading is stretched between 0.6 and 1.0, where most cells sit. jusText scores 0.00 on every recipe, listing and non-English page.

2Three slices, three rankings

A benchmark I wrote can only rank the pages I thought of. So I scored the same tools, with the same code, on two sets of real pages where I did not write the gold. They do not agree with my set, and they do not fully agree with each other.

Synthetic40 test pages, 10 types

Pages I generated from a seed: news, blog posts with comments, docs with code, product specs, forums, recipes, listings, cookie modals, div soup and non-English pages. I wrote the gold, so it is exact, and it is mine.

Wikipedia40 articles, 1 template

English articles from a seeded random draw over page ids, by a rule fixed before any tool ran. Gold is the MediaWiki plain-text extract of the same revision. Many articles, one site skin.

Zyte181 pages, 126 sites

The public article-extraction benchmark from Zyte (MIT licence), pinned at one commit. Body text labelled by their team. I used every page and retuned nothing.

  • jusText
  • readability-lxml
  • trafilatura (best-config)
  • trafilatura, defaults
  • html2text, bs4-naive, markdownify
Figure 5. Rank by mean F1 in each slice, with synthetic F1 on the left and Zyte F1 on the right. On Wikipedia the two trafilatura rows are identical (0.735). On Zyte, readability-lxml and tuned trafilatura are tied at 0.960, so their order there is a coin flip.

Wikipedia barely agrees with my order: Spearman 0.20 over 7 rows. Zyte agrees more, 0.60 over the 6 tools, and it agrees with Wikipedia at 0.83. What held everywhere: html2text, markdownify and the naive BeautifulSoup baseline stay at the bottom, below readability-lxml and trafilatura. 3With 6 or 7 tools, a rank correlation is a rough signal. Read Table 2 as direction, not as a measurement.

Spearman rank correlation6 tools7 rows

Table 2. The 7 rows count tuned trafilatura as its own row.

Why it flipped

On the synthetic set, jusText returned an empty string, with no error, on 15 of 40 pages: every non-English page, every recipe, every listing, and 3 forum threads. Each one scores 0. Wikipedia is all English articles, and there jusText was the most precise tool, at 0.936. On Zyte it was empty again on 30 of 181 pages.

So the synthetic set punished jusText for page types the real sets do not contain. Each dot below is one page, and the column of dots at zero is the story.

Figure 6. Every page's F1, per tool and slice. Dots are jittered up and down so they do not hide each other. The tall tick is the mean and the bar under it the 95% interval. Red counts are empty outputs.

The tie on many sites

On Zyte, readability-lxml minus tuned trafilatura is +0.000 [-0.011, +0.013] over pages. Pages of one site share a template, so I also resampled whole sites: [-0.013, +0.016]. A tie both ways. 4Not blind. The Zyte dataset's own README publishes results for readability, trafilatura and jusText, so those tools may have been tuned against these pages.

Figure 7. Zyte slice, readability-lxml minus trafilatura (best-config). The tuned config is the one frozen from the synthetic dev pages, not retuned here.

So the Wikipedia flip was a Wikipedia result. On many sites the synthetic top two hold up, and jusText sits in the middle.

If you benchmark extractors

  1. A tool that is last on one page mix can be first on another. Test on your own mix.
  2. Treat empty output as a failure, not a result.
  3. A benchmark you wrote yourself can only rank the pages you thought of.

3Look at one page

A score hides what a tool actually did. Pick any of the 40 synthetic test pages and two tools.

On the left is the page exactly as the tools got it. Next to it is the gold: each main paragraph marked kept or dropped, and each planted boilerplate snippet marked if it leaked. Below, every output line is marked as gold text, leaked boilerplate, or other text. 5blog_comments-2 is the demo page from the README. F1 calls trafilatura and jusText fine there (0.94), but both kept all four reader comments. That is why leak rate sits next to F1.

F1 of every tool on this page. Click one to show it as output A.

kept: gold main text dropped gold paragraph leaked boilerplate, with its kind extra: not gold, not a tracked snippet

What the browser sees

sandboxed, scripts off

Gold

written at generation time
Figure 8. Side-by-side viewer. Highlighting is approximate: each output line is matched to the gold by 4-gram overlap, line by line. The scores themselves are recomputed with the repo's own metrics.score and match results/per_doc.csv exactly.

4How they fail

A score says how much went wrong. The taxonomy says what. Each tool and page gets fixed-threshold labels, and the matrix counts the test pages with each label, out of 40.

Figure 9. Failure labels on the 40 synthetic test pages. Circle area grows with the count; a faint dot is zero. Empty output is drawn in red.

The converters keep everything. html2text, markdownify and the naive BeautifulSoup baseline leak nav or footer on 40 of 40 pages. Their recall is 1.000, and their output is about 1.7 times the real content.

readability-lxml's title is wrong on 40 of 40. Its default title() returns the whole <title> tag, site name included. Its short_title() helper matches 35 of 40. That is a side check, not part of the ranking.

jusText fails quietly. Empty output on 15 pages, tables lost on 5, code lost on 4, and no crash on any of them.

trafilatura keeps reader comments by default. Tuning on the 20 dev pages picked include_comments=False, which removed comment leaks on 4 test pages. 6This is the only tuned row. Tuning was uneven on purpose: one row, picked on dev, frozen, and named as such everywhere.

include_commentsfavor_precisionfavor_recallDev F1Dev leak rateChosen

Table 3. The trafilatura tuning grid, scored on the 20 dev pages only.

Would the right stoplist save jusText?

jusText ships a stoplist per language and defaults to English. With the matching stoplist it recovers on the German and French test pages, but stays at 0 on Italian and Hindi. The Spanish and Portuguese pages are dev pages; they recover too.

LanguagePageSplitF1, English stoplistF1, matching stoplist

Table 4. Side check, not ranked: python -m extract_bench.side_checks.

5The metric vs two LLM judges

The token metric is my choice, so I checked its pass/fail rule against two LLM judges.

The rule: GOOD when F1 is at least 0.8 and leak rate is at most 0.2, else BAD. Each judge saw the gold main text and one tool's output and answered GOOD or BAD. 140 items, 20 test pages per tool, seed 7, the same items for both judges, temperature 0, 0 failed calls. 7Kappa intervals are a percentile bootstrap over items, 2,000 resamples, seed 7.

Figure 10. Cohen's kappa with 95% intervals. The two judges agree with each other less well than qwen agrees with the metric. One judge model is not ground truth.

Table 5. Metric rule against each judge. Red cells are disagreements.

What I take from it

Where they disagree, the metric is the lenient one. The judges agree with each other on 118 of 140 items. On 9 of those the metric disagrees with both, and in all 9 the metric said GOOD. Open any of them in the viewer:

ExtractorPageF1LeakMetricBoth judges

Table 6. The 9 items. Click a page to load it in section 3.

No disagreement on the clear cases. bs4-naive, html2text and markdownify are BAD on all 60 of their items, for the metric and for both judges.

The judge is not fully stable. I re-asked gpt-oss-120b 12 of its BAD-versus-metric-GOOD items with the same prompt at temperature 0. 2 of 12 came back GOOD this time.

A lead, not a result. A stricter rule, precision and recall both at least 0.9 with leak at most 0.2, raises kappa to 0.89 with qwen and 0.59 with gpt-oss. I picked it after seeing these same items, so it needs fresh pages before it means anything.

6Limits

A benchmark that hides its limits is marketing. These are the ones I know about.

  1. Synthetic pages. Template text and my guesses at common layouts. The Wikipedia slice reorders the tools, but it is one site template. The Zyte slice agrees with the synthetic top two and puts jusText fourth. Neither real slice covers recipes, forums or listings.
  2. Same author for pages and gold on the 60 synthetic pages. I decided what counts as content, and the listing-page check shows how much one decision can move the ranking. The Wikipedia gold leaves out infobox text and tables.
  3. No JavaScript rendering. Static HTML only.
  4. Small n, few templates. 40 test pages, 4 per type, from 10 templates. Per-type cells are directions, not rankings. The honest unit of variation is closer to 10 page types than 40 pages.
  5. The metric is not a human, and neither is the judge. Bag-of-words F1 ignores order and formatting. The judges saw 140 of the 420 outputs, no human labels exist, and the two judges disagree with each other about as often as with the metric.
  6. Uneven tuning. Only trafilatura got a tuned row. jusText recovers on German and French with the right stoplist, but not on Italian or Hindi.
  7. No third-party text in git. The synthetic pages come from sentence banks written for this project, with invented company names. Wikipedia text is CC BY-SA and the Zyte pages belong to the sites that published them, so those slices ship ids, links and hashes only.
  8. One machine for latency. Latency also moves a little on every rerun. Every other number is identical run to run.

7Method and rerun

Everything runs locally: no API keys, no network at run time, no paid services. The two real-page slices fetch their pages once. Same seed, same intervals.

# Python 3.11 or newer
pip install -r requirements.txt
python -m extract_bench.build_corpus     # rebuild the 60 pages + gold
python -m extract_bench.run              # results/ and charts/, about 30 seconds
python -m extract_bench.side_checks      # jusText stoplists, readability short_title()
python -m extract_bench.demo blog_comments-2
python -m extract_bench.wikislice build  # optional, needs network
python -m extract_bench.wikislice run
python -m extract_bench.zyteslice build  # optional, needs network
python -m extract_bench.zyteslice run
python -m pytest -q

Statistics: percentile bootstrap of the mean over pages; paired bootstrap on per-page differences, where an interval that crosses 0 means not significantly different; the same test with whole page types (synthetic) or whole sites (Zyte) as the resampled unit. Every number on this page is read from site/data/site.json, exported from the repo's result files by site/scripts/export_data.py.