What happened? As introduced in one of the previous posts, the University of Bologna is organising a series of events dedicated to research fundamentals. On the 20th of May, a half-day conference devoted entirely to reliability, transparency, and reproducibility involved several national and international experts on the topic, who were engaged in keynote speeches and round tables. On the 9th of October, the new event of the series will be dedicated to the concept of interdisciplinarity. The event webpage includes the agenda and a registration form. The event is free and can be followed either in person or online. I hope to see you there!
What I did. At the beginning of August, we uploaded a paper describing the OpenCitations Index on arXiv, which has been accepted in Scientometrics. This work – coauthored with Ivan, Arianna and Marta – introduces the recent effort we have conducted at OpenCitations to revise the ingestion workflow of bibliographic metadata and citation data from five different sources (Crossref, DataCite, PubMed, OpenAIRE, JaLC) and integrate all their citations data into one collection, i.e. the Index, of over 2 billion unique citation links, which can be systematically accessed and queried through several services, including REST APIs and dataset dumps in various formats.
What to read. As an outcome of the first event of the series mentioned above, the University of Bologna has produced a position paper which highlights, enriched by the perspectives of international experts, the point of view and recommendation of the University on the reliability, transparency and reproducibility in research:
Credi, A., Masini, F., Diciotti, S., Forni, M., Peroni, S., Coppini, S., & Gualandi, B. (2024). Back to the Fundamentals of Research: Reliability, Transparency and Reproducibility. Position Paper [Position paper]. Alma Mater Studiorum – Università di Bologna. https://doi.org/10.6092/unibo/amsacta/7803
What happened? A few days ago, the University of Bologna organised a half-day event entirely dedicated to the concepts of reliability, transparency and reproducibility of research. It counted international experts, including Sabina Leonelli and Daniele Fanelli, who gave two exciting keynote speeches, and two roundtables with several institutional representatives (from the National Agency for The Evaluation of Universities and Research Institutes, the Ministry of University and Research, the UK Reproducibility Network, and the Jagellonian University) and with UNIBO’s researchers (from PhD students to Professors) introducing their take on reproducibility within their research and discipline. It has been a wonderful moment of discussion within our community, which will be followed by two additional events (in October and December) dedicated, respectively, to interdisciplinarity and ethics.
What I did. As a consequence of my take-home messages from the event and the slides I prepared for a lecture I gave to second-year students of the Cultural Heritage in the Digital Ecosystem PhD programme, I spent several hours on the concept of transparency, which is strongly tied to reproducibility. In several meetings and papers, I’ve seen too many times that transparency is introduced as synonymous with quality. While I understand the rationale behind such a conclusion, I believe this claim is rather simplistic – and wrong, from my perspective. I would need to spend more effort (and words) to rigorously outline the whole rationale of my thoughts – at least more than a short paragraph in a blog post. Still, the main point here is that quality is a goal to achieve (e.g. to perform qualitatively sound work), while transparency is, in fact, a tool that enables others to check if we have reached that goal. In practice, quality can exist without transparency, but transparency is needed if we want people to scrutinise quality. Thus, transparency does not build quality; it builds trust.
What to read. With all this agitation about these topics I’ve been absorbed this week, the weekly suggestion is, indeed, about research reproducibility and shows how there is not one definition that fits all but, rather, a plethora of different lenses one may have to use, depending on contextual and/or disciplinary situations:
In the scholarly ecosystem, a bibliographic citation is a conceptual directional link from a citing entity to a cited entity, used to acknowledge or ascribe credit for the contribution made by the author(s) of the cited entity. Citations are one of the core elements of scholarly communication. They enable the integration of our independent research endeavours into a global graph of relationships that can be used, for instance, to analyse how scholarly knowledge develops over time, assess scholars’ influence, and make wise decisions about research investment.
A copyrighted fact
However, as citation data, i.e. pieces of factual information aiming at identifying entities and relationships among them, are of great value to the scholarly community, it has been a “scandal” that they have not been recognised as part of the commons. Indeed, only recently we have seen some efforts – such as the Initiative for Open Citations (I4OC) – that have tried to change the behind-the-paywall status quo enforced by the companies controlling the major citation indexes used worldwide, convincing scholarly publishers to support the unrestricted availability of scholarly citation data by publishing them in suitable open infrastructures, such as Crossref and DataCite.
Of course, as for many other kinds of data, putting bibliographic and citation data behind a paywall is a thread to enabling the full reproducibility of research studies based on them (e.g. in bibliometrics, scientometrics, and science of science domains), even when such studies are published in open access articles. For instance, the results of a recent open access article published on Digital Scholarship in the Humanities, which aimed to analyse the citation behaviour of Digital Humanities (DH) research across different proprietary and open citation databases, are not fully reproducible since the majority of the databases used – namely Scopus, Web of Science, and Dimensions – do not make their bibliographic and citation data openly available.
In addition, the coverage of publications and related citations in specific disciplines, particularly those within the Social Sciences and the Humanities (SSH), is inadequatecompared tootherfields. Usually, this is due to the limited availability of born-digital publications accompanied by a wide variety of publication languages, publication types (e.g. monographs), and complex referencing practices that may limit their automatic processing and citation extraction. As a side effect, such a partial coverage may result in a considerable bias when analysing SSH disciplines compared to STEM disciplines that usually have better coverage in existing citation databases.
Reforming research assessment
All these scenarios have at least another negative effect on the area strictly concerned with the research assessment, which often uses quantitative metrics based on citation data to evaluate articles, people, and institutions. Indeed, the unavailability and partial coverage of bibliographic and citation data create an artificial barrier to the transparency of the processes used to decide the careers of scholars in terms of research, funding, and promotions.
to be open and transparent by providing machine-readable, unrestricted and reusable data and methods for calculating the metrics used in research assessment exercises, and
to leave to the research community, instead of commercial players, the control and ownership of the crucial infrastructures and tools used to retrieve, use and analyse such data within research assessment systems.
Thus, the leading guideline that can be abstracted is to follow Open Science practices even when assessing research and not only when performing research.
Some initiatives pushing for reforming the principles behind research assessment systems.
Introducing OpenCitations
Within this context, OpenCitations (full disclosure: I am one of its directors) plays an important role, acting as a key infrastructure component for global Open Science, and pushes for actively involving universities, scholarly libraries and publishers, infrastructures, governments and international organisations, research funders, developers, academic policy-makers, independent scholars and ordinary citizens. The mission of OpenCitations is to harvest and openly publish accurate and comprehensive metadata describing the world’s academic publications and the scholarly citations that link them, with the greatest possible global coverage and subject scope, encompassing both traditional and non-traditional publications, and with a breadth and depth that surpasses existing sources of such metadata, while maintaining the highest standards of accuracy and accompanying all its records with rich provenance information, and providing this information, both in human-readable form and in interoperable machine-readable Linked Open Data formats, under open licenses at zero cost and without restriction for third-party analysis and re-use.
For OpenCitations, open is the crucial value and the final purpose. It is the distinctive mark and founding principle that everything OpenCitations provide – data, services and software – is open and free and will always remain so. OpenCitations fully espouses the aims and vision of the UNESCO Recommendations on Open Science, complies with the FAIR data principles, and promotes and practises the Initiative for Open Citations recommendation that citation data, in particular, should be Structured, Separable, and Open.
The most important collection of such open citation data is COCI, the OpenCitations Index of Crossref open DOI-to-DOI citations. The last release, dated August 2022, contains more than 1.36 billion citation links between more than 75 million bibliographic entities that can be accessed programmatically using its REST API, queried via the related SPARQL endpoint, and downloaded in full as dumps in different formats (CSV, JSON, and RDF).
Collaborations between OpenCitations and other Open Science infrastructures and services.
In addition to the publication of citation data, a considerable effort has been dedicated to collaborating with other Open Science infrastructures working in the scholarly ecosystem, as summarised in the figure above. Since 2020, OpenCitations has significantly benefited from the scholarly community that resulted from the 2019 selection by the Global Sustainability Coalition for Open Science Services (SCOSS) of OpenCitations as a scholarly infrastructure worthy of financial support. The community funding permitted the appointments of people dedicated to the administration, communication, community development, and maintenance and improvement of the OpenCitations software and the computational infrastructure on which it runs. In addition, OpenCitations started its involvement with OpenAIRE and the European Open Science Cloud (EOSC), and it is collaborating with other funded projects project such as RISIS2, OutCite, OPTIMETA and B!SON.
While OpenCitations is currently providing a good set of citation data, which is already approaching parity with other commercial citation databases and that has already been used in a few studies for research purposes, there is still a margin for improvement. Currently, the citations included in the OpenCitations Indexes come mainly from Crossref data, one of the biggest open reference providers. However, Crossref does not cover all the publishers of DOI-based resources. Indeed, other DOI providers, in some cases, expose citation relations in their metadata, such as DataCite. In addition, DOI-based publications represent just a limited set of all the bibliographic entities published in the scholarly ecosystem. Other identifier schemas have been used to identify bibliographic entities – and, for some publications, there not exist identifiers at all!
Thus, to address these two issues, OpenCitations is working on expanding its coverage according to two different directions. On the one hand, OpenCitations is developing two new citation indexes of open references based on the holdings of DataCite and the National Institute of Health Open Citation Collection, which, together with COCI, will be cross-searchable through the Unifying OpenCitations REST API.
On the other hand, OpenCitations has started working to create a new database entitled OpenCitations Meta, which will provide three major benefits. First, it will permit storing in-house bibliographic metadata for the citing and cited entities involved in all OpenCitations Indexes, including author identifiers using ORCID and VIAF identifier schemes where available. Second, it will provide better query performance than the present API system, which obtains bibliographic metadata on-the-fly by live API calls to external services, such as Crossref and DataCite APIs. Finally, it will permit indexing citations involving entities lacking DOIs, by providing them OpenCitations Meta Identifiers.
This last collection, combined with automatic tools for citation extraction from digital formats, is crucial for increasing the coverage of underrepresented disciplines and fields in bibliographic databases, such as SSH publications. One of the OpenCitations’ goals is to reduce this gap in citation coverage by setting up crowdsourcing workflows for ingesting missing citation data from the scholarly community (e.g. libraries and publishers). In the future, another contribution will be to set up tools for automatic extraction of citations that can also support small and local publishers, crucial assets for SSH research, that may find difficulties in carrying out citation extraction tasks on their own since using and maintaining a tool (or paying a company addressing those tasks on behalf of the publisher) requires extra costs beyond publishers’ finances.
To conclude: OpenCitations is one piece of a puzzle that is working to change existing scholarly practices to create an open and inclusive future for science and research in which the scholarly community owns and is responsible for its own data.