When affiliation errors become a research security problem

TL;DR. Research security screening checks author affiliations against lists of organizations of concern, using bibliometric data. Affiliation matching is imperfect, though, especially for companies and small institutes with short names or no ROR record, so a researcher can end up linked to an organization they have no connection to. As a first pass we cross-referenced the major watch lists to ROR and cleaned up 6,191 such over-merges across 192 listed organizations. We invite the research security community to collaborate with us to improve OpenAlex and use bibliometric data responsibly for this use case.

Research security screening is now routine. Funders, universities, and commercial compliance tools check author affiliations against government lists of organizations of concern (e.g., entities flagged for military, defence, or sanctions reasons). The lists are public and the screening is increasingly required.

One part of that screening process often relies on affiliation data from research publications. The general idea is to ask: has the research team been associated with organizations of concern directly (i.e., they worked for or were funded by them) or indirectly (through their networks of collaborators). A positive match in a bibliometric database then triggers follow-up scrutiny to determine if the connection is real and the risks associated with the connection.

Anyone who has worked with affiliation data knows that it can get messy. To demonstrate the problem, the University of British Columbia has almost 200,000 distinct text strings that have been used in publications to refer to the University, covering departments, addresses, translations, typos, text encoding issues from publishers, and more. Here are a few examples:

  • “The University of British Columbia”
  • “UBC”
  • “Univ of British Columbia, Vancouver, BC, Canada”
  • “University of Brtish Columbia, Vancouver, Canada”
  • “University of British Colombia#TAB#”

And so algorithmic approaches are used to associate these text strings with institutions in the Research Organization Registry. Research information databases each have their own processes for doing this. We make our source code openly available, and you can find our methods detailed on github. But long story short, we have linked over 150M unique text strings to 130k+ unique institutions. These algorithms generally work well, with >98% precision and >90% recall for most academic institutions. But across the full corpus of all scholarly works in OpenAlex (500M+), that’s still millions of errors.

These errors can be especially acute for companies or small research institutes because they may have very few publications to train the matching model and they often have short names that are common words. Consider the example of the company “Optical Sciences”. They have a ROR record because they publish research, but they get erroneously associated with the Optical Sciences Center of the University of Arizona which does not have its own ROR record.

Most bibliometricians are accustomed to navigating these errors in their analyses, and some even know how to fix them upstream (e.g., curating ROR records and string matching in OpenAlex). On the other hand, research security professionals come from diverse backgrounds and only sometimes have a nuanced technical understanding of bibliometric databases. That means an affiliation error that used to be a minor data-quality annoyance can now hold up a grant, trigger a foreign-influence review, or put a researcher into a compliance inquiry over a relationship that never existed.

A first pass at minimizing false positives

We took the published watch lists that screening actually runs against (Canada’s Named Research Organizations, the US Department of Defense 1260H list, the US Consolidated Screening List including the BIS Entity List, US OFAC, the US UFLPA list, the UK FCDO sanctions list, and Australia’s DFAT list) and cross-referenced each entity to ROR with a strict country-consistency check. That produced a verified crosswalk of 192 organizations covering roughly 4.24 million works.

Then, for each organization, we screened every affiliation string linked to it, about 2.5 million in all, looking for this specific failure: an unrelated company or institution absorbed into a listed entity because their names collide (often with an organization lacking a ROR record). We found 6,191 such over-merges, roughly a quarter of one percent of those strings, and we submitted manual curations to remove the linkages. The scale of the collision problem is easier to see in a few real examples than to describe.

  • A small US biotechnology firm, identified on its papers by a short name only that also refers to a broad class of medical interventions, was linked to a Russian research institute whose registered acronym is nearly identical. The two have nothing to do with each other. The firm simply had no ROR record, so the closest-looking acronym won.
  • System Equipments Div., Asahi Engineering Co., Ltd, a Japanese engineering company, was linked to a Chinese entity on the BIS Entity List whose registered name is essentially “System Equipment.”
  • ENGINEERING-PHYSICS, INC. was linked to a Russian Institute of Engineering Physics that appears on these lists, on the strength of the shared words alone.

None of these organizations has any connection to the listed entity it was linked to. In each case the two names are close enough that you can see why an algorithm reached for the wrong one because no record existed for the real organization. This work helps reduce these errors, but they still exist.

Advice to analysts doing the screening

If you’re an analyst in charge of screening research applications for potential connections with a list of organizations of concern, never assume that a bibliometric match is a definitive signal of a connection. A match can help you shorten your list of connections to confirm manually, but there are errors and you need to confirm any potential connections. You can do that easily by inspecting the raw_affiliation_string field in the OpenAlex API. For instance: https://api.openalex.org/works/w3134510327 -> raw_affiliation_string : “Optical Sciences, University of Arizona, Tucson, Arizona, USA”. You can also visit the publication’s landing page or PDF for more information directly from the source. All of that information is linked conveniently in the OpenAlex record.

You should also take efforts to identify false negatives. For instance, if the organization of concern does not have a ROR record (or doesn’t have enough information on how the organization represents itself in publications), then we won’t be able to make any matches to that institution. More broadly, I highly recommend that you connect with experts in this field on how to successfully leverage bibliometric tools in your work while respecting their limitations.

Advice to researchers submitting to screenings

Take a look in OpenAlex in advance to see if there might be issues in your author profile (my profile below). Sometimes another authors’ works might have snuck into your author profile which you can edit that there directly. And sometimes there are errors in affiliation attribution even when the work is yours (like in the examples above). When that happens, the quickest way to get it sorted out is to send details in an email to support@openalex.org. But keep in mind that funders and analysts use different databases and tools for research security compliance workflows and many of them use closed, proprietary databases you won’t be able to access.

Let’s work on this together

If you work in research security, or at a funder or institution that screens affiliations, we want to hear from you.

Screening decisions carry real weight, so the data behind them should be open to inspection. In OpenAlex every affiliation link traces back to the source publication, every correction is public and permanent, and the code that makes the matches is on GitHub (github.com/ourresearch). Nothing sits in a proprietary index that no one outside the vendor can check. When a match is wrong you can see why, contest it, and get it fixed, and that fix then helps everyone who screens after you. For a decision that can affect a researcher’s funding or reputation, that is not a small difference. Open access to the data also means scrutiny is not gated by budget: a well-resourced agency and an under-resourced one work from the same public record.

We do not think we have this solved. We know some of the limits and we are certain there are more we have not found. What we want is to work with people who do this screening every day to map those limits honestly, so nobody treats a match as more certain than it is, and to build the documentation and tooling that would make an open, auditable corpus genuinely good infrastructure for this work. A few experts collaborating with us in the open could improve it for the whole community, globally.

Get in touch at kyle@openalex.org.