Journals publish different types of articles. Original research articles and reviews are the most common and are most often used in bibliometric analyses. But there are also other types, such as letters to the editor or book reviews, which are often not considered.
Both the vocabulary used to describe journal article types and the methods used to assign them vary between bibliometric databases. For example, when analysing the classification in OpenAlex, Web of Science (WoS), Scopus, PubMed and Semantic Scholar, I found that OpenAlex tends to overestimate the assignment of the document type ‘article’. OpenAlex tagged 10% of items as articles, which were labelled as editorial material in Scopus and the Web of Science.
In this blog post, I will present a classifier designed to improve the identification of journal article types in open scholarly databases like OpenAlex. The classifier uses metadata from Crossref and OpenAlex, including the number of references, citations and affiliations, to assess whether an item in a journal is a research contribution or not. To train the classifier, I used open scholarly data from PubMed, OpenAlex and Crossref. After introducing the classifier, I will compare my approach with that of OpenAlex, Scopus and the methodology employed by the CWTS to identify core publications. Both the source code of the classifier and the classified records are publicly accessible.
Data and Methods
The journal article type classifier was trained on approximately 9.5 million journal articles from PubMed, representing either research discourse or editorial discourse. PubMed was used because its classification of document types is similar to that of Scopus and Web of Science, with the advantage that PubMed data is openly accessible and reusable. The training data was limited to the publication years 2012 to 2022. In addition, articles from publishers with fewer than 5,000 publications were excluded from the training. Similar to the CWTS approach to identify core publications, the classifier also takes into account metadata retrieved from Crossref and OpenAlex. These metadata fields are:
| Metadata | Type | Retrieved from |
|---|---|---|
| has abstract? | BOOLEAN | Crossref (October 2023 snapshot) |
| title word count | INT | Crossref (October 2023 snapshot) |
| page count | INT | Crossref (October 2023 snapshot) |
| author count | INT | Crossref (October 2023 snapshot) |
| has license? | BOOLEAN | Crossref (October 2023 snapshot) |
| number of citations | INT | Crossref (October 2023 snapshot) |
| number of references | INT | Crossref (October 2023 snapshot) |
| has funding information? | BOOLEAN | Crossref (October 2023 snapshot) |
| number of affiliations | INT | OpenAlex (April 2024 snapshot) |
| has OA url? | BOOLEAN | OpenAlex (April 2024 snapshot) |
The dataset was then split into 75% training data and 25% test data. For the classifier, I used the k-nearest neighbours algorithm. Parameters for the algorithm were optimised using grid search.
The classifier is build on top of OpenAlex rule-based paratext recognition, which uses title heuristics. This means that my classifier cannot classify pre-labelled editorial material from OpenAlex as research items.
To evaluate my model, I compared the results with OpenAlex, Scopus and the CWTS Leiden Ranking Open Edition. The CWTS Leiden Ranking Open Edition is a university ranking based on a pre-computed subset of so called core publications indexed in OpenAlex to compare universities. A publication is considered a core publication if it has one or more authors, has at least one reference, is published in English and is also published in a core journal (Van Eck and Waltman 2024). A journal is considered a core journal if it has an international scope and also contains a high number of references. The CWTS approach is therefore more selective than my approach, because I did not exclude journals and non-English contributions.
For the comparison I used the September 2024 snapshot from OpenAlex, the July 2024 snapshot from Scopus and the data underlying the Leiden Ranking Open Edition 2024, which is available through Google BigQuery. Matching were carried out by DOI.
Results
Overall, the classifier categorised 12.647.946 out of 108.744.219 journal items, which were classified as articles and reviews in OpenAlex, as non-research items, representing a share of 11.6%. When restricted to the period 2012–2021, this figure drops to 3.778.393 out of 38.265.399 journal items (9.9%), presumably due to improved metadata coverage used to determine document types. Figure 1 compares the results of the classifier with OpenAlex and the CWTS approach, based on items in journals between 2012 and 2021. The left side of the figure displays all journals in OpenAlex, whereas the data from the figure on the right side is restricted to CWTS core journals. The grey line represents all articles and reviews in journals in OpenAlex. The green line indicates all articles and reviews in OpenAlex for which at least one reference and one citation were identified. The purple line below shows all items from journals in OpenAlex included as core publications in the CWTS Leiden Ranking Open Edition. The yellow line illustrates the proportion of items in OpenAlex that are recognised as research items by the classifier. About 48.3% of all journal items in OpenAlex from 2012 to 2021 were categorised as core publications by the CWTS when not restricting to core journals, demonstrating the effect of also using journal characteristics to define eligble publications. Overall, 27.72 of 203.545 journals (13.6%) in OpenAlex were considered as core journals by the CWTS in its latest Open Leiden Ranking. But also missing authors, affiliations and references had an impact on the CWTS classification. In contrast, my classifier determined about 84,5% of journal items as research items when not restricting to core journal. When restricted to core journals, it shows that the classifier is less sensitive to missing metadata or a low number of citations. In contrast, the proportion of CWTS core publications and the proportion of publications with at least one reference and one citation in OpenAlex are similar and achieve both about 71% on average.
Figure 1: Comparison of my classifier with OpenAlex and the CWTS core classification.
To check for potentially discriminatory behaviour of my classifier, I compared my approach with the CWTS approach by journal topic (see Figure 2). Again, a common corpus based on DOI matching is used. Figure 2a and 2b show all journals in OpenAlex, while the data in Figure 2c and Figure 2d are restricted to the CWTS core journals. Figures 2a and 2b show that my classifier treats publications from different disciplines in a more balanced way, while the CWTS method excludes publications from the social sciences and humanities more often, probably due to the exclusion of non-core journals. In this respect, the CWTS points out that many social science journals were not considered core journals due to the lack of a suitable number of references. When using core journals (Figure 2c and Figure 2d) the behaviour changes. Here, my classifier and the CWTS classification show a similar pattern, but with health sciences publications being excluded more often. These publications may be case reports, which usually do not contain references.
Figure 2: Comparison of the coverage of my classifier with the CWTS core classification using topics from OpenAlex.
Figure 3 compares the proportion of articles and reviews in OpenAlex and Scopus in relation to the intersection of items in OpenAlex and Scopus in journals from 2012 to 2021 using a shared corpus based on DOI matching. The results of my classifier are represented by the yellow line. OpenAlex counts more items than Scopus when restricting to the document types articles and reviews (OpenAlex: 95.9% and Scopus: 87.6%). With the help of the classifier, the retrieval of articles and reviews in OpenAlex can be improved (Classifier: 93.2%). Nevertheless, Scopus counts were lower when comparing it to my method.
Figure 3: Comparison of my classifier with OpenAlex and Scopus.
Discussion and Data Access
The comparison of my classifier with OpenAlex, Scopus and the CWTS has shown that it can help with document type assignments in open scholarly data sources such as Crossref and OpenAlex. My next step is to qualitatively check a larger sample. Meanwhile, OurResearch is also in the process of updating its classification of document types. Until then, my classifier can be used as a complementary tool for identifying suitable articles for bibliometric analysis using open scholarly data sources such as Crossref or OpenAlex.
A major limitation of my classifier is that it often identifies clinical trials and case studies as non-research articles, as these often have no references or citations. However, this could be improved by adding open data from PubMed that classifies these document types.
The classified data is available via the publicly available Google BigQuery instance provided by the SUB Göttingen. Here you can also compare it with OpenAlex, Crossref and Semantic Scholar. To query it, you can use
SELECT COUNT(DISTINCT(oal.doi)) AS n, type, label
FROM 'subugoe-collaborative.openalex.works' AS oal
JOIN 'subugoe-collaborative.resources.classification_article_reviews_september_2024' AS dt
ON oal.doi = dt.doi
GROUP BY type, label
ORDER BY n DESCThe results will be constantly updated in line with the monthly release of OpenAlex and Crossref. I would be happy to get feedback about your experiences with the classifier!