Are we measuring Open Science the right way?
Over 130 participants joined us on 29 May 2026 for a ninety-minute conversation that asked a deceptively simple question: Are we measuring Open Science the right way? The session brought together four European initiatives working at the frontier of Open Science monitoring, responsible metrics, and research assessment reform, and it quickly became clear that the question itself is the right place to start.
Written by Tereza Szybisty
Setting the stage
Moderator Natalia Manola (OpenAIRE) opened the session by reframing the premise. Before asking how we measure Open Science, she invited speakers and the audience alike to ask why we measure it at all. The answer, she argued, is not trivial. We invest in Open Science because we believe it changes science and culture, and that change is precisely what indicators need to capture, not just the outputs that are easiest to count. This framing set the tone for everything that followed.
Four perspectives on the same challenge
OpenAIRE Monitor: transparency as a foundation
Ioanna Grypari (OpenAIRE) presented the OpenAIRE Monitor, now serving 77 research institutions, 25 research initiatives, 13 funders, and a growing number of national monitors. The service draws on the OpenAIRE Graph, the largest open scholarly knowledge graph in Europe. Two points from Ioanna's presentation generated lively discussion. First, data quality is not a background concern, it is the product. The Irish Monitor, for example, has used the very act of surfacing unexpected indicator values to prompt institutions to clean and improve their own data, turning measurement into a quality improvement loop. Second, the same number means different things to different readers: a 62% open access rate is a benchmark for one institution, a mandate compliance check for a funder, and a national policy progress signal for a ministry. Indicators are not neutral; their meaning is shaped by who reads them and why.
CoARA Working Group on responsible use of metrics and indicators: start with "why"
Katarzyna Nawrot (Co-chair of the CoARA Working Group on Responsible Use of Metrics and Indicators) shared findings from the CoARA working group, including a large-scale survey (learn more about the results of the survey here). Her core argument was methodological: before choosing any indicator, an institution must first ask why it is evaluating, whom or what it is evaluating, and, critically, what latent construct the indicator is supposed to capture.
The working group identifies a pattern they call the "naive non-responsible workflow": picking available indicators without first establishing what you want to measure. This leads, Katarzyna argued, to inappropriate use, not because the indicators are bad, but because the reasoning behind choosing them was never made explicit. The recommended workflow starts with values, moves to assessment goals, defines the construct to be measured, and only then selects indicators, complementing quantitative measures with qualitative assessment throughout.
The discussion touched on a broader provocation: do we measure what we treasure, or do we treasure what we measure? Katarzyna's call for indicators that guide us toward higher-quality research, rather than simply describing current practices, resonated strongly with the audience.
Open Science Monitoring Initiative (OSMI): aligning content providers
Iratxe Puebla (Co-chair of the OSMI Working Group 3) presented the Open Science Monitoring Initiative's work with scholarly content providers, publishers, repositories, and data centres. A key distinction in Iratxe's presentation: this work is explicitly not about building rankings or passing judgment on levels of openness. It is about generating evidence that allows informed decisions, for example, whether a journal should update its open access policy, or how a funder like NIH can monitor data-sharing practices across generalist repositories.
Iratxe raised a challenge that will resonate with anyone working in this space: as more platforms adopt open science monitoring, they are applying different methodologies to calculate indicators for similar practices within open science. Some level of standardisation, or at least consensual definitions, will be needed to prevent a future in which many indicators exist but none are comparable. She also called explicitly for content providers to share their indicator practices openly, so the community can audit and refine them. Learn more about the results of the OSMI Working Group 3 survey here.
EOSC Open Science Observatory: policy to practice
Tereza Szybisty (OpenAIRE, EOSC Track / EOSC Open Science Observatory) presented the Observatory as a policy intelligence platform built in close collaboration with the EOSC Steering Board and European Commission. Its purpose is to provide national policymakers, country representatives, and the research community with a data-driven picture of where Europe stands on Open Science, across policies, practices, and impacts. The monitoring framework covers eight categories: policies, practices and impacts.
One of the most striking illustrations from Tereza's presentation was the gap between policy existence and researcher coverage. The policy exists on paper; whether it reaches researchers is a different question entirely. This, she argued, is precisely why the Observatory combines quantitative data with qualitative country narratives written by National Open Science Desks - the numbers tell you that a gap exists; the narrative tells you why.
Tereza also surfaced three structural challenges facing the Observatory: comparability (countries answering the same survey questions with different understandings of basic definitions), the gap between policy coverage and live practice, and sustainability, how to maintain continuity of a monitoring mechanism as EOSC moves into its post-2027 governance phase, while simultaneously evolving the framework to match shifting policy ambitions.
Key themes from the discussion
The moderated roundtable and audience Q&A surfaced several threads worth carrying forward.
Data quality is a first-order concern. Deduplication, affiliation disambiguation, licence metadata, these are not technical footnotes but conditions for any meaningful indicator. Multiple speakers noted that the human-in-the-loop remains essential, even as algorithmic approaches improve.
Indicators are often most useful as a diagnostic tool. They show what the data behind a policy actually looks like. As Ioanna noted, this matters in its own right. A low number may point to where investment is needed, in repositories, identifiers, or curation, rather than who is falling short. It may also signal incomplete data, not necessarily poor practice. In this way, indicators do more than guide decisions; they help clarify what kind of decision is needed.
Impact measurement is the missing piece. Several speakers acknowledged that while output indicators (publications, datasets, repositories) are increasingly mature, indicators for scientific and societal impact remain underdeveloped. The PathOS Handbook of Open Science Impact Indicators was highlighted as a resource that the community should collectively work to keep alive and extend.
Openness applies to the monitoring infrastructure itself. A recurring theme was that the credibility of any monitoring initiative depends on whether its methodology, data sources, and indicator definitions are openly available for scrutiny. The EOSC Open Science Observatory makes its data freely available for reuse; the OpenAIRE Graph is publicly accessible; OSMI publishes its principles that can help us to align on Open Science monitoring.
Closing reflections
Asked what they hoped to see in two years, the speakers offered a set of complementary ambitions:
- Katarzyna called for Open Science indicators that genuinely guide quality , not just describe coverage, but point toward what matters.
- Iratxe named adoption, maturity, and openness: more content providers monitoring, with indicators that have reached sufficient quality to be useful beyond their originators.
- Ioanna asked for more granular indicators that capture the whole picture of open science: meaningful uptake rather than open-washing, consequences across different groups and contexts, and, with the help of AI, actual use and impact of scientific outputs beyond access alone.
- Tereza urged a shift from measuring what is easy to count toward measuring what actually changes , how Open Science is making research more useful, accessible, equitable, and impactful for researchers, policymakers, and society.
Across 130 participants and four initiatives, the session surfaced something that is easy to overlook in the day-to-day work of building monitors and designing frameworks: the indicators we choose are never just descriptions. They are arguments about what matters. They shape what institutions report, what researchers prioritise, and what policymakers reward. Which raises a question worth sitting with long after the webinar ended, not are we measuring Open Science the right way, but whose version of Open Science are our indicators quietly encoding?