This week we replaced how OpenAlex decides a work’s type. Every work we ingest carries this field — article, book-chapter, dissertation, dataset, retraction, and so on (the full taxonomy is here). A new classifier now assigns every work’s type from the evidence in its own record, using an auditable set of rules, and on its first production run it changed the type of 49.6 million works — about 10% of the catalog.
(I’m Rohan Mantena, a new data/software engineer on the team, and this was our team’s project and my first big one here. Excited to share it.)
Why work types matter
Correct type classification matters to our users for a lot of reasons. A few we want to highlight:
- Finding the right research outputs: the actual papers and reviews for a literature search, the datasets to build new research on, the preprints for tracking early results.
- Excluding what isn’t relevant: most searches don’t want tables of contents, editorial boards, or the photographs and physical objects that repositories legitimately hold (paratext and other exist so you can filter them out).
- Citation metrics: Field-Weighted Citation Impact (FWCI) normalizes each work’s citations against works of the same type, field, and year. When conference papers and editorials are mislabeled as articles, they land in the wrong comparison group.
This is an inherently difficult problem. A work’s type was largely inherited from its source at ingestion, and sources genuinely differ: Crossref, DataCite, and institutional repositories each maintain their own type classification scheme, so the same kind of work arrives under different labels depending on where it came from. And even setting the schemes aside, there’s no verification of what a depositor selects — a type label is whatever someone chose from a dropdown, years or decades ago. Many sources provide no type at all. Different vocabularies plus unverified labels meant article became the default dumping ground, and our users noticed: support tickets flagging classification errors are part of what motivated this work.
So, we set out to fix it.
Execution
As with any good classifier, we needed good data and a ground truth.
A taxonomy. Before labeling anything, we had to agree on what the labels mean: is a book review a ” review”? Is a journal issue’s front matter a work? We settled on 25 canonical types (full taxonomy and definitions at openalex.org/types). Casey wrote annotator-grade definitions for each, with inclusion examples, exclusions, and the distinguishing rule against each type’s nearest neighbor. Kyle and Jason settled the contested boundary cases, like where paratext ends and other begins. Every classification decision downstream traces back to those definitions, and boundary questions get a ruling there before they get code.
A gold set. We sampled works from across the catalog and built an LLM-labeled gold set: 52,383 works, plus a 1,495-work validation set. Because LLM labels deserve scrutiny, we audited them: models from different model families independently labeled the same works, and their agreement rate bounds the labeling error — which turned out to be low. The gold set also gave us a rough measure of the problem: existing types matched it only about 70% of the time.
Building the ruleset. With solid footing, the error and rule patterns became much clearer. Repositories announce dissertations in dozens of languages (Tesis, Hochschulschrift, väitöskirja), and each vetted spelling became a rule. Correction notices start with “Erratum to:”. Conference abstracts hide in journal supplement issues. Kyle personally adjudicated over 600 disputed works, and his rulings became rules as well. Each rule states its evidence, its target type, and its guard conditions.
The effort was worth it. The output is an ordered cascade of ~160 rules, compiled into a single SQL expression that re-runs over the full corpus every day. Rules are ordered by measured trustworthiness: unambiguous structural signals fire first, softer text heuristics later, and the first match wins. If nothing speaks, the work is typed article — that default has a cost, and we come back to it below.
Why rule-based? Openness and improvability. Every classification is explainable — directly attributable to a specific rule — and can subsequently be improved. And the taxonomy stays governable: when we make a ruling about what counts as paratext vs other, that ruling maps onto specific, inspectable rules.
How we measured it
Checking the classifier against the existing types would have been the wrong test: it would simply have measured conformity to labels we already suspected. Early in the project, one improvement made our agreement score go down, and inspection showed the classifier was right and the old labels were wrong. So we evaluated against independent ground truth instead, two ways.
A blinded truth panel. We sampled 1,000 works and had three LLMs from different model families label each one. They were shown the work’s evidence but blinded to both the current OpenAlex type and the classifier’s prediction, so there was nothing to anchor on. The labelers agreed with each other strongly (κ = 0.90), and disagreements were resolved by majority. Result: the old types were right 69.3% of the time, the classifier 78.1%. And on the works whose type actually changed, the panel found 43 flips from wrong to right and 0 from right to wrong — where the classifier disagrees with the old data, it won every audited case.
An admission bar for every rule. Aggregate accuracy can hide a bad rule inside a good system, so before admission, each rule is tested on a blinded sample of the works it would actually fire on in production, against a ≥95% precision bar. Rules that miss the bar get fixed, guarded, or shelved with their evidence attached.
The results
Here is what the first production run did to the catalog:
| type | before | after | change |
| conference-paper | 2.8M | 18.2M | 6.4× |
| conference-abstract | 535k | 4.2M | 7.8× |
| book-review | 80k | 2.1M | 26× |
| editorial | 833k | 1.3M | 1.6× |
| review | 4.2M | 1.0M | -76% |
| erratum | 420k | 625k | +49% |
| retraction* | 23,142 | 45,454 | 2× |
| preprint | 10.1M | 9.3M | -8% |
| article | 264.5M | 240.3M | −9.1% |
* Retraction as a type refers to our ability to detect when a record is a notice of retraction, which is separate from when an existing work is retracted
Changes to the type list
Three vocabulary changes came along, relevant to anyone filtering by type:
- software is new — (~1M works) stay tuned for more on this soon 😉
- software-paper is new — a small type (~9,300 works) for papers describing software, distinct from software itself.
- report-component is gone — its ~800 works are now simply report.
- book-section is gone — its 2,000 works were mostly index/front-matter records, now reference-entry or paratext.
If your pipeline filters on either retired value, it will now return zero results; please update to the successor types.
What’s still hard
Our classifier isn’t perfect:
Types that leave no trace. A letter or editorial that the publisher registered as a plain journal article, with no distinguishing title, is invisible to a rules engine and stays article. Precision on these types is high; recall is the headroom. Those works are labeled no worse than before, and the fixes are input problems more than rule problems: we’ve parsed the metadata embedded in roughly 300 million landing pages, and more evidence channels are on the way.
Works with no usable metadata default to article. About a quarter of works carry no raw type at all. When nothing is indicative, article is our stated default, and when the previous label was more specific, defaulting can be a regression. We measured this tradeoff before accepting it, and chose a transparent, reproducible default over preserving labels of unknowable provenance. We monitor exactly the flows where it is induced, and recovering those works from actual evidence is on the roadmap.
Coverage varies by source. Crossref-registered works carry the richest signals; repository and legacy records are poorer, and their types will improve as more of their vocabulary passes the admission bar.
All that said, the classifier is built to close its own gaps. Every limit above is a future rule or evidence source, each one is tested against the admission bar before it ships, and because types are recomputed from current evidence daily, every fix reaches the whole catalog the next day. As we adapt and add rules, the numbers in this post keep getting better. And this approach isn’t staying confined to types: we plan to roll out the same rule-based, auditable pattern across other metadata fields over the next year — source is next in our sights.
What this means for you
- Roughly 10% of works changed type in the snapshot and API this week; the daily changefiles carry the updates. If you cache types, expect a large one-time diff.
- Counts of conference papers, errata, retractions, editorials, and book reviews are substantially higher than in any previous OpenAlex snapshot.
- If you compute or consume FWCI-style indicators, expect small shifts as tens of millions of works move into their correct comparison groups.
- If a type looks wrong, submit a ticket to support@openalex.org. Every verdict is attributed to a specific rule, so a good report (“this work is typed X, should be Y”) points straight at a rule we can fix — and fixes propagate to the whole corpus the next day. Systematic patterns are even more valuable: a single repository vocabulary we’re misreading can be thousands of works.
Your error reports made these improvements possible, so thanks again for sending us issues when you see that. Please keep ’em coming! We’re looking forward to continuing to work with the community the increase the quality and accuracy of open research information!