One in Eight OpenAlex Abstracts Has Integrity Issues

A systematic assessment of abstract quality in a major bibliographic database reveals prevalent failure modes and calls for community annotation.

Seorin Kim; Vincent Holst; Vincent Ginis· 2026· DOI 10.48550/arXiv.2605.20168

The core problem

Scientific abstracts are increasingly used as primary data in computational metascience research, yet the quality of these abstracts in widely used bibliographic databases has not been systematically examined. OpenAlex, a prominent open bibliographic database, provides abstracts for millions of scholarly works, but the integrity of these abstracts—whether they accurately represent the underlying publications—remains unverified. This study addresses this gap by assessing the integrity of 10,000 randomly sampled English-language journal abstracts from OpenAlex. The authors aim to identify common failure modes and quantify the prevalence of integrity issues, thereby informing downstream research practices and motivating community-driven quality control.

The analysis revealed that 12% of the sampled abstracts have integrity issues. The most prevalent failure modes were insufficient content and misplaced metadata. Insufficient content refers to abstracts that are truncated, incomplete, or lack essential information, while misplaced metadata occurs when abstract text is incorrectly associated with the wrong publication or contains extraneous information. The seven failure modes identified by the authors provide a taxonomy for categorizing abstract quality problems. The prevalence rate of 12% implies that approximately one in eight abstracts in OpenAlex may be unreliable for research purposes. This finding is particularly concerning given the increasing reliance on such data in metascience. The distribution of failure modes suggests that both automated extraction errors and human errors during metadata entry contribute to the problem.

Scientific abstracts are increasingly used as primary data in computational metascience research, yet the quality of these abstracts in widely used bibliographic databases has not been systematically examined. OpenAlex, a prominent open bibliographic database, provides abstracts for millions of scholarly works, but the integrity of these abstracts—whether they accurately represent the underlying publications—remains unverified. This study addresses this gap by assessing the integrity of 10,000 randomly sampled English-language journal abstracts from OpenAlex. The authors aim to identify common failure modes and quantify the prevalence of integrity issues, thereby informing downstream research practices and motivating community-driven quality control.

The authors employed a two-stage annotation protocol combining human expert review and large language model (LLM) classification. First, a random sample of 10,000 English-language journal abstracts was drawn from OpenAlex. Human experts annotated a subset to identify and categorize integrity issues, leading to the definition of seven distinct failure modes. These annotations were then used to train and validate an LLM classifier, which was applied to the full sample. The two-stage approach ensures both depth (via expert review) and scale (via LLM). The protocol likely involved iterative refinement to ensure reliability. The classification process can be represented as a flow diagram:

The authors discuss the implications of these findings for downstream research. Integrity issues in abstracts can lead to biased or invalid results in computational metascience studies that use abstracts as primary data. For example, analyses of research trends, topic modeling, or citation networks may be distorted by incomplete or misattributed abstracts. The authors advocate for increased awareness and quality control measures. They describe a forthcoming community portal designed to support collective annotation efforts, enabling researchers to flag and correct integrity issues collaboratively. This initiative aims to improve the reliability of OpenAlex and similar databases. The study underscores the need for ongoing monitoring and the development of automated tools to detect and mitigate integrity issues. Future work should extend the analysis to other databases and non-English abstracts, and explore the root causes of the identified failure modes.

CS practitioners and researchers

Opening member content…