At the same time, Google Big Query is certainly not open scholarly infrastructure, nor is it fully equitable and accessible in all parts of the world, and Google is not an organisation many of us feel able to trust. Thus, reliance on Google is not a desirable long-term solution.

There are emerging alternatives both in the cloud and for local computing, and organisations exploring these alternatives or interested in doing so. With ORION-DBs, we hope to spur on these developments and their application for opening up use of scholarly data sets.

As a first contribution, in this blog we discuss some features, or lack thereof, of Google Big Query infrastructure, what they mean for how data are made available and how public infrastructure could strive to emulate or improve these features. Considering these features separately can help to ensure conscious and justified choices are made as to which to prioritise when thinking about local server or cloud deployment.

• Having the underlying data available (stored) in publicly owned/governed infrastructure/data storage
  This not the case currently in the Google Big Query setup. However, none of the data sources are made available exclusively through ORION-DBs or any of the subsidiary providers. A condition for inclusion in ORION-DBs is that the data sets are openly available somewhere else with an open license. So at the level of the original data sources, there is no lock-in effect. For the harmonised format of the datasources (table structure after ingest) that’s different - an agreed on archival format (e.g parquet) and storage frequency and location could be of added value here.

• Having the underlying data available (stored) in Europe or other non-US localities
  Data in Google Big Query can be stored on (Google-owned) servers in Europe rather than US (other locations are also available). A point of attention is that combining data sets is only possible if they are in the same locality storage-wise. In Google Big Query, this can be solved by mirroring data sets across 2 or more localities (or have them hosted separately in 2 or more localities). This is a barrier that an open alternative might be able to circumvent entirely.

• Having the code of the infrastructure open source
  Core to our goals is helping people build new kinds of analysis and applications. Investing in new platforms will only work if people trust them to remain accessible. This is why open source is one of the Principles of Open Scholarly Infrastructure; a requirement Google Big Query clearly does not meet. Open source doesn’t guarantee continued accessibility, but it helps provide insurance against risks. BigQuery is a major product for Google and is unlikely to disappear quickly, but pricing could increase or other restrictions could make it difficult. An alternative should provide both access as well as enabling others to support that access if needed and open source (and well documented) systems substantially mitigate the risks of infrastructure disappearing from underneath people.

• Access to the infrastructure (for computing) with no or minimal gate-keeping authentication/access control
  Access to Google Big Query requires a Google account, which is in itself a barrier (dependency on US tech, personal information sharing, etc.). However, beyond this, there is no specific authentication (institutional or otherwise) required with the provider(s) of the ORION databases, minimising friction and social barriers. Risks of misuse/overuse are mitigated by a) all data sets being openly available with no or minimal re-use restrictions, b) all compute costs being borne by the user (see below) and c) the infrastructure being robust enough to not break from excessive use.

• Cost for storage (incl. pre-processing and ingest) kept separate from cost for compute
  This removes administrative burden and risk for providers (keeping their costs contained and predictable), gives users agency over their usage (no externally imposed usage limits) and could be considered an interesting sustainability model.

• Ability to combine data sources from different providers that all use the same infrastructure This removes the necessity for any provider to host ‘all’ data sets - reducing costs for providers and increasing options and flexibility for both users and providers. It also makes it possible for different providers to offer the same data set with a different structure (e.g flat or nested, fully relational or not), rather than enforcing one structure for all hosted databases. As an extension, users can add their local data sets to include in analysis (without publicly exposing them) and providers can also contribute with a single public data set.

• The infrastructure providing both access to the data and an analysis interface
  Meaning in principle, end users do not have to install specific software to use the data for analysis. At the same time, for Google Big Query, integration with other platforms/software is well documented, extending usage options to, for example, Python and other languages and workflows.

All this is not to uncritically defend our current choice for Google Big Query, but to explain what we, at least, see as some of its separate characteristics that are relevant both for data providers and users. We recommend considering all elements in this list (and potentially others) when scoping the development of public infrastructure for sharing and facilitating usage of public scholarly metadata sources.

Introduction

This tutorial shows how to measure open access by funder over time using open research information resources provided by the ORION community on Google BigQuery. We compare NIH- and European Commission-supported biomedical research literature with a focus on open access, including an estimation of articles enabled by transformative agreements. Transformative agreements have played an important role in Europe for several years, but the effect of these agreements on open access to EC grant-supported articles remains unclear. Recent discussions around the NIH open access policy and price caps for publication fees furthermore raise the question about the potential role of similar agreements in the US.

The ultimate goal in this tutorial is to re-create the following visualisation using ORION resources, with a special emphasis on funder information and different open access business models.

The area chart compares open access to biomedical research literature supported by the European Commission and the NIH between 2018 and 2024. Although both funders achieved open access to over 90% of their grant-supported output, the underlying funding models differ. Notably, the proportion of open access enabled by transformative agreements is substantially larger for research funded by the European Commission than for research funded by the NIH, reflecting the wider availability of such agreements in Europe. This comparison points to potential shifts in the US that could be driven by the increasing prevalence of open access agreements designed to accommodate funder open access policies and price caps, as suggested by publishing market observers and researchers (Clarke & Esposito, 2025; Haustein et al., 2025).

library(tidyverse)
library(bigrquery)
library(DBI)

# Connect to GBQ with billing project from SUB Göttingen,
# You have to use your own :-)
bq_con <- dbConnect(
  bigrquery::bigquery(),
  project = "subugoe-collaborative",
  dataset = "openalex",
  billing = "subugoe-collaborative"
)

pmids_nih <- read_lines("nih_pmids_2018_2024.txt")
dbWriteTable(bq_con, # Our BQ connections
             "subugoe-collaborative.resources.nih_pmids_2018_2024", # Table name
             tibble::tibble(pmids_nih = pmids_nih), # PMIDs in rectangular format with column name
             overwrite = TRUE
             )

CREATE OR REPLACE TABLE `subugoe-collaborative.resources.oa_tutorial_md_raw` AS (

WITH openaire_funders AS (
  SELECT DISTINCT
    pid.value AS pmid,
    fund.name AS funder
  FROM `sos-datasources.openaire.project_20260129` AS proj,
    UNNEST(fundings) AS fund
  INNER JOIN `sos-datasources.openaire.relation_product_project_20260129` AS pub_proj
    ON proj.id = pub_proj.target
  INNER JOIN `sos-datasources.openaire.publication_20260129` AS pub
    ON pub_proj.source = pub.id,
    UNNEST(pub.pids) AS pid
  WHERE fund.name IN ('European Commission')
    AND pid.scheme = 'pmid'
),

nih_funders AS (
  SELECT DISTINCT
    CAST(pmids_nih AS STRING) AS pmid
  FROM `subugoe-collaborative.resources.nih_pmids_2018_2024`
),

all_funders AS (
  SELECT pmid, 'European Commission' AS funder
  FROM openaire_funders
  UNION ALL
  SELECT pmid, 'NIH' AS funder
  FROM nih_funders
)

SELECT DISTINCT
    oalex.id,
    oalex.doi,
    oalex.ids.pmid,
    oalex.ids.pmcid,
    publication_year,
    publication_date,
    primary_location.source.issn_l AS issn_l,
    primary_location.source.is_in_doaj AS is_in_doaj,
    open_access.oa_status AS oa_status,
    country AS country_code,
    inst.ror,
    af.funder AS funders
  FROM `subugoe-collaborative.openalex_walden.works` AS oalex
  LEFT JOIN
    UNNEST(authorships) AS au WITH OFFSET AS pos
  LEFT JOIN
    UNNEST(au.countries) AS country
  LEFT JOIN
    UNNEST(au.institutions) AS inst
  LEFT JOIN `subugoe-collaborative.resources.document_classification_september25` AS doctype_classifier
    ON oalex.doi = doctype_classifier.doi
  LEFT JOIN all_funders AS af
    ON REGEXP_EXTRACT(oalex.ids.pmid, r'(\d+)$') = af.pmid
  WHERE
      pos = 0
    AND primary_location.source.type = "journal"
    AND is_paratext = FALSE
    AND oalex.type IN ('article','review')
    AND is_research = TRUE
    AND is_xpac = FALSE
    AND (
      NOT REGEXP_CONTAINS(oalex.biblio.issue, '^[a-zA-Z]')
      OR oalex.biblio.issue IS NULL)
    AND (
      NOT REGEXP_CONTAINS(oalex.title, '[0-9]{3} pp.'))
    AND publication_year BETWEEN 2018 AND 2025
)

CREATE OR REPLACE TABLE `subugoe-collaborative.resources.oa_tutorial_jct_articles` AS (
WITH
  -- Enrich ROR variants from associated institutions
  obtain_associated_ror_ids AS (
    SELECT
      esac_id,
      jct_inst.ror_id AS ror_jct,
      inst.ror AS ror_associated
    FROM
      `subugoe-collaborative.openbib.jct_institutions` AS jct_inst
    LEFT JOIN
      `subugoe-collaborative.openalex.institutions` AS oalex_inst
    ON
      jct_inst.ror_id = oalex_inst.ror
    LEFT JOIN
      UNNEST(oalex_inst.associated_institutions) AS inst
  ),

  create_matching_table AS (
    SELECT esac_id, 'ror_jct' AS ror_type, ror_jct AS ror
    FROM obtain_associated_ror_ids
    UNION ALL
    SELECT esac_id, 'ror_associated' AS ror_type, ror_associated AS ror
    FROM obtain_associated_ror_ids
  ),

  enriched_ror_variants AS (
    SELECT DISTINCT
      create_matching_table.*,
      DATE(jct_inst.start_date) AS start_date,
      DATE(jct_inst.end_date) AS end_date
    FROM create_matching_table
    INNER JOIN `subugoe-collaborative.openbib.jct_esac` AS jct_inst
      ON create_matching_table.esac_id = jct_inst.id
  ),

  journal_agreements AS (
    SELECT
      j.issn_l,
      e.id AS esac_id
    FROM `subugoe-collaborative.openbib.jct_journals` AS j
    INNER JOIN `subugoe-collaborative.openbib.jct_esac` AS e
      ON j.esac_id = e.id
  )

SELECT DISTINCT
  md.id,
  md.doi,
  md.issn_l AS matching_issn_l,
  md.ror AS matching_ror,
  erv.ror_type,
  erv.esac_id,
  erv.start_date,
  erv.end_date,
  md.publication_date
FROM `subugoe-collaborative.resources.oa_tutorial_md_raw` AS md
INNER JOIN journal_agreements AS ja
  ON md.issn_l = ja.issn_l
INNER JOIN enriched_ror_variants AS erv
  ON md.ror = erv.ror
  AND ja.esac_id = erv.esac_id
WHERE
  md.ror IS NOT NULL
  AND md.issn_l IS NOT NULL
  AND (PARSE_DATE('%Y-%m-%d', md.publication_date) >= erv.start_date OR erv.start_date IS NULL)
  AND (PARSE_DATE('%Y-%m-%d', md.publication_date) <= erv.end_date OR erv.end_date IS NULL)
  -- Only TA OA articles
  AND oa_status IN ('gold', 'hybrid')
ORDER BY erv.esac_id, md.publication_date
)

WITH
  oa AS (
    SELECT
      publication_year,
      funders,
      CASE
        WHEN (is_in_doaj = TRUE AND oa_status = 'gold') THEN 'gold_is_in_doaj'
        WHEN (is_in_doaj = TRUE AND oa_status = 'diamond')
          THEN 'diamond_is_in_doaj'
        WHEN oa_status = 'hybrid' THEN 'hybrid_oa'
        WHEN oa_status != 'closed' THEN 'other_oa'
        END AS oa_type,
      CASE
        WHEN jct.doi IS NOT NULL THEN TRUE
        ELSE FALSE
        END AS enabled_ta,
      COUNT(DISTINCT md.pmid) AS n_oa_articles
    FROM `subugoe-collaborative.resources.oa_tutorial_md_raw` AS md
    LEFT JOIN
      (
        SELECT DISTINCT doi
        FROM `subugoe-collaborative.resources.oa_tutorial_jct_articles`
      ) AS jct
      USING (doi)
    WHERE funders IS NOT NULL
    GROUP BY
      publication_year,
      funders,
      oa_type,
      enabled_ta
  ),
  total AS (
    SELECT
      publication_year,
      funders,
      COUNT(DISTINCT pmid) AS articles
    FROM `subugoe-collaborative.resources.oa_tutorial_md_raw`
    GROUP BY
      publication_year,
      funders
  )
SELECT
  total.publication_year,
  total.funders,
  total.articles,
  oa.n_oa_articles,
  oa.oa_type,
  oa.enabled_ta
FROM total
LEFT JOIN oa
  ON
    total.publication_year = oa.publication_year
    AND total.funders = oa.funders
WHERE total.funders IS NOT NULL AND total.publication_year < 2025
ORDER BY total.publication_year DESC

funding_oa
#> # A tibble: 109 × 6
#>    publication_year funders            articles n_oa_articles oa_type enabled_ta
#>               <int> <chr>                 <int>         <int> <chr>   <lgl>     
#>  1             2024 NIH                   89744         34829 other_… FALSE     
#>  2             2024 NIH                   89744          5054 <NA>    FALSE     
#>  3             2024 NIH                   89744          2486 diamon… FALSE     
#>  4             2024 NIH                   89744         30150 gold_i… FALSE     
#>  5             2024 NIH                   89744          1247 gold_i… TRUE      
#>  6             2024 NIH                   89744         12618 hybrid… FALSE     
#>  7             2024 NIH                   89744          3349 hybrid… TRUE      
#>  8             2024 NIH                   89744            11 other_… TRUE      
#>  9             2024 European Commissi…    26941          1346 <NA>    FALSE     
#> 10             2024 European Commissi…    26941          9466 gold_i… FALSE     
#> # ℹ 99 more rows

oa_by_year_and_type <- funding_oa |>
  filter(!is.na(oa_type)) |>
  mutate(oa_cat = case_when(
    oa_type %in% c("gold_is_in_doaj", "hybrid_oa") & enabled_ta == FALSE ~ "APC",
    oa_type %in% c("gold_is_in_doaj", "hybrid_oa") & enabled_ta == TRUE ~ "Transformative Agreement",
    oa_type == "diamond_is_in_doaj" ~ "Diamond OA",
    .default = "Other")
) |> summarise(n_oa = sum(n_oa_articles),
            .by = c(publication_year, funders, articles, oa_cat)
            ) |>
  mutate(oa_prop = n_oa / articles,
             oa_cat = factor(oa_cat, levels = c("APC", "Transformative Agreement", "Diamond OA", "Other")))

oa_by_year_and_type
#> # A tibble: 56 × 6
#>    publication_year funders             articles oa_cat             n_oa oa_prop
#>               <int> <chr>                  <int> <fct>             <int>   <dbl>
#>  1             2024 NIH                    89744 Other             34840  0.388 
#>  2             2024 NIH                    89744 Diamond OA         2486  0.0277
#>  3             2024 NIH                    89744 APC               42768  0.477 
#>  4             2024 NIH                    89744 Transformative A…  4596  0.0512
#>  5             2024 European Commission    26941 APC               12793  0.475 
#>  6             2024 European Commission    26941 Transformative A…  8863  0.329 
#>  7             2024 European Commission    26941 Other              3199  0.119 
#>  8             2024 European Commission    26941 Diamond OA          740  0.0275
#>  9             2023 NIH                    97390 Other             43955  0.451 
#> 10             2023 NIH                    97390 APC               42607  0.437 
#> # ℹ 46 more rows

oa_by_year_and_type |>
  ggplot(aes(publication_year, oa_prop, fill = fct_rev(oa_cat))) +
  geom_area(color = "black", linewidth = 0.2) +
  facet_wrap(~funders) +
  scale_y_continuous("Open Access (in%)",
    labels = scales::percent_format(), limits = c(0, 1),
    expand = c(0, 0)
  ) +
  scale_fill_manual("OA model",
    values = c(
      "APC" = "#0E6BA0",
      "Transformative Agreement" = "#30C0D2",
      "Diamond OA" = "#E2A3A4",
      "Other" = "#D8D8D8a0"
    )
  ) +
  labs(
    x = NULL,
    title = "Open Access to grant-supported biomedical research literature",
    subtitle = "Journal articles and reviews in PubMed",
    caption = "Data sources: PubMed, OpenAIRE, OpenAlex, cOAlition S Journal Checker Tool."
  ) +
  theme_minimal() +
  theme(
    panel.grid.major.x = element_blank(),
    panel.grid.minor.x = element_blank(),
    plot.title = element_text(face = "bold", size = rel(1.1), color = "grey10", margin = margin(b = 8)),
    plot.title.position = "plot",
    plot.caption = element_text(hjust = 1, size = rel(0.7), color = "grey50", margin = margin(t = 12)),
    plot.caption.position = "plot",
    legend.position = "top",
    plot.margin = margin(10, 10, 10, 10)
  ) +
  guides(fill = guide_legend(reverse = TRUE))

@online{jahn2026,
  author = {Jahn, Najko and Kramer, Bianca and Neylon, Cameron and
    Haupka, Nick},
  title = {Crosswalking {ORION} Resources to Measure Open Access to
    Biomedical Research Literature},
  date = {2026-03-13},
  url = {https://orion-dbs.community/blog/posts/nih-ec-oa/},
  doi = {10.59350/4ttp9-7sv10},
  langid = {en},
  abstract = {This tutorial shows how to combine various ORION
    resources, with a focus on funder and open access. Using the
    National Institutes of Health (NIH) and the European Commission (EC)
    as examples, it draws on data from PubMed, OpenAIRE, OpenAlex, and
    transformative agreement data from cOAlition S and ESAC. The
    analysis covers more than 800,000 biomedical research articles,
    revealing notable differences in open access funding between NIH-
    and EC-supported research.}
}

The rise of open research information resources is transforming the way we track, analyse and study research systems. Increasingly, sources like OpenAIRE, OpenAlex, Crossref, DataCite, ORCID, ROR and others are being used as the basis for making decisions, designing interventions and understanding progress in the science system. This operates both at the small scale, where access to data and evidence is easier than it has ever been, to the very large scale analysis of whole systems.

Traditionally, the capacity to do large-scale analyses was restricted to a very small set of players, like specialised research centreor companies. This kind of large scale analysis usually requires access to an actionable version of the whole dataset, particularly if the goal is combining data resources. The set of sites with access to complete copies of proprietary databases is tiny. 

Modern open data sources provide access, including access to full copies of the data, but there has been less focus on providing this access in a way that allows large scale complex querying and connecting of whole data archives - for example to compare the coverage of research outputs by OpenAlex and OpenAIRE or analyse global information on clinical trials using affiliation data from OpenAlex and clinical trials information from Pubmed. Another valuable possibility is the ability to incorporate local data enrichments from national or regional data sources to support local data needs, or improve the overall pool of data.

Google BigQuery has emerged as one powerful tool for combining and working on these large datasets at scale. Multiple groups (including the MultiObs team - continuing the work of the InSySPo team at Campinas, SUB Göttingen, Sesame Open Science and CWTS amongst others), have created ‘public’ versions of specific open datasets in the BigQuery system, which anyone can access and run their own analyses. Through these public versions, the ‘provider’ (i.e. the teams mentioned above) pays for storage, and the user freely accesses the ‘public’ versions taking responsibility for covering the costs of data querying and processing.  

Having worked independently so far, this small group came together last year to ask whether we could coordinate actions. Could it be possible to build a comprehensive open research information resource where the load of providing specific core and relevant open data sources was distributed? Rather than each separately trying to tackle the whole, potentially duplicating efforts, could we collectively create a resource that was more than the sum of its parts? 

We met with a series of key questions:

1. Can we share resources and burdens to make available key open research information resources in actionable and connectable form in the cloud?

2. Through sharing processes and systems, is it possible, over time, to build a standard for how these data sources should be made available?

3. What are the challenges that we can usefully approach collectively?

4. What are the benefits and risks of Google BigQuery as an environment and do we agree it is the best place to start?

5. What are the blockers for engagement with such an effort? What is needed for different stakeholders to make it attractive both as users and (for some) as providers?

User and use case driven

Core to our shared interest in working together was the idea of making it easier for more people to undertake large scale analysis. There are many kinds of analysis for which access to APIs is sufficient. We share a belief that large scale analysis will be useful in multiple settings, but that it has been relatively inaccessible. This inaccessibility is a hurdle to realising the promise of democratization and broader adoption of open research information in all decision making processes around science and scholarship, as proposed by the Barcelona Declaration. APIs are also expensive to run, by taking some of the heavy load use-cases away from APIs we can support providers by reducing their costs, centralising distribution, and allowing APIs to focus on the use cases they are best suited for.

There is a growing set of research projects that are exploiting this capacity for large scale and combined analysis in a range of ways. Two recent pieces of work provide examples of what is possible. One by Camilla Lindelow and Eline Vandewalle, used the combination of ORCID and OpenAlex provided by InSySPo (now MultiObs) to analyse researchers without a formal affiliation from around the world. The second example, from Cespedes and colleagues associated with the UNESCO Chair in Open Science, used a global analysis of language in OpenAlex to examine affiliation. This combines with other efforts, including comparisons of metadata coverage across sources, and combinations of data sets that exploit the capacity to do analysis at scale.

These use cases have a few things in common. They tend to be global in scope (or at least aspire to be) so they require analysis across the whole of a datasource (or a combination of datasources). They generally involve a complex form of query, requiring filtering or analysis on multiple database elements, or a combination of multiple data sources, that is difficult or impossible using the API for any given datasource. And the generated dataset is often very large in its own right - perhaps involving hundreds of millions of rows of data - and requires further reduction and analysis.

Overall, the common theme here is analyses that require entire data sources to be combinable and actionable at scale. We believe if we focus on that set of use cases we can add something valuable to the overall Open Research Information ecosystem.

Opportunities for shared systems

If people are already doing this what is the value of coordination? The first and most obvious is that with a shared cloud system we only need to pay for online storage of each dataset once and then anyone can use it (backups and versions over time are a separate issue, which we aim to address, but not as the first priority). Cloud storage costs are generally larger than the usage costs involved in running queries so sharing this load is valuable in its own right.

The second advantage is the ability to share capacities. One example of this is data preprocessing. These datasets are not “clean” in the sense that they change over time, have some internal inconsistencies, and often contain elements that raise compatibility issues with database systems. Processing hundreds of millions of lines of JSON to convert hyphens to underscores in variable names takes time and computing power (and money!). 

Systems developed within the Curtin Open Knowledge Initiative (COKI) use cloud VMs to do this on demand which scales but adds costs. The team at Göttingen are using code derived from this on their own HPC resources. The team at CWTS uses their own code to process datasource dumps on local servers so that relational databases can be integrated into their internal database system, while also exporting the results to Google BigQuery. Within the Sesame Open Science system a further evolution of the COKI code is used to process dumps on local computers. There is a clear benefit to be gained by using a common code base for necessary transformations. But also in having a community discussion on what pathways and transformations are necessary. The MultiObs team uses a quite different approach -- creating relational structures from datasource dumps -- with advantages (reduced costs, timestamps, etc.) but also disadvantages (lack of live data, need for updates, etc.) we can learn from. Different approaches and experiences, but also different sets of resources like HPC could be shared amongst an effective collaboration.

This leads to the third advantage. If we use common systems we help to develop quasi-standards that can be adopted by others. That creates an opportunity to spread the load further, as well as to increase the diversity of datasets available (again, thinking of those highly curated national datasets that are used locally but not always recombined into the global data ecosystem). If we have a clear shared approach to the data and how it is managed it makes it easier for others to contribute, and makes the whole set of resources more valuable and sustainable. In essence, the more we share the load, the less we pay for the costs of our contributions. 

A final benefit of a shared approach would be a virtuous loop in which shared systems encourage shared approaches to analysis. Common approaches can form the basis for training resources that give end-users an easy point of entry to using these data sources at scale. They will also encourage the sharing of analysis scripts and protocols creating advanced and transparent resources to support developing users.

Key to this is understanding both what has value to keep in common, but also what needs to be different to serve a diversity of use cases. We can see value in technical standards (where they are useful) and in agreements around archiving and preservation. Documentation, where it can reach common standards, will be helpful not just for users of the data, but potentially for upstream producers in understanding how the data is being used and how to optimize the provision of their data snapshots to facilitate downstream usage.

The Google-shaped elephant in the room 

A big question is why Google BigQuery? It is certainly not an open system in any meaningful sense and Google is not an organisation many of us feel able to trust. The short answer is pragmatism. There are reasons why we independently arrived at GBQ as a useful tool. Google solves a bunch of the hard problems, including authentication without the need for institutional affiliation, systems provisioning and a highly performant database system. In practice, this means datasets can be made publicly available, without the need for specific hard-or software on the side of the user, and, from a user perspective, access to datasets hosted by different providers is possible using a single system. Standing up an independent infrastructure to do this is a big job and not one we’re equipped to tackle at the moment.

That said, none of us believe that reliance on Google is a desirable long term solution, nor that it is fully equitable. There are some emerging alternatives both in the cloud and for local computing. These aren’t fully mature but they show promise. In the meantime we believe it is important to ensure we have an exit strategy. One such strategy could be a commitment to creating backups in the form of parquet files. Parquet is an interesting interoperability format for databases and can be read in by an increasing number of tools. It holds schema information and allows for database partitioning. 

Perhaps the most important argument is that with Google BigQuery and external archiving, there is at least one plausible option to explore that can provide value immediately, but also provide a potential escape route. We can save the arguments for frozen duck lakes, glaciers, torrents and MySQL for later and for those who will want to have them! But we need to think seriously about how we will work towards more independence and resilience early on in the process. 

Next steps and a call for interest

We have made a small start. Small, but useful for us.  After all, we are already using these shared data resources. We have demonstrated that without much effort or technical hassle it is possible to share the load, reduce costs and maximize benefits and accessibility. We hope by engaging a wider community we can make this more useful for more people and move us all closer to ideals of democratization of open research information, supporting adoption. How far this goes and how big a community we can create is an open question.

We have made a small start under the label of ORION-DBs, standing for Open Research Information Online Databases. There is now a website that details the datasets available, where they can be accessed and when the most recent update was. We hope this will be a useful resource for people doing ad hoc analyses, occasional use, or just one-off interest in taking a look, as well as those with bigger use cases and ongoing needs for data access. We hope a community of users and also of providers will be interested to coordinate through this platform to aid discovery, adoption and democratization of open research information.

Looking forward, we’re interested in how we can build on this base. We want to coordinate and build a shared capacity. If you have an interest in how this could be shaped, demonstrating specific use cases, or contributing additional hosted datasets, we’d love to hear from you. Coordination takes time, time requires resources. If there is sufficient interest, we will look at how we could coordinate resources and build something as lightweight as possible and as formalised as necessary.

Above all, we want to hear from those who share the vision for creating data resources that can be combined and used together and to make them as useful as possible. You can contact us through info@orion-dbs.community and depending on interest, we will set up other forums. It is through using these data sources that we identify their issues and can correct and improve them. When we do that work together, we increase the quality of all data resources faster, more sustainably and more effectively.

This post is licensed under CC BY 4.0.