Why I hope BROCCOLI adopts data mesh architecture concepts for reliable data governance

About a year ago I stumbled on something called data mesh architecture. I read a bit, nodded a lot, and it never really left my head. The idea is simple: stop treating data as something a central team digs out of pipes, and start treating it as a product. Each dataset gets an owner, a clear description of what it offers, and a promise about its quality. What won me over is that it runs on open standards. The contracts follow the Open Data Contract Standard, products can be described with the Data Product Specification, and the tooling around it is open source. No vendor lock-in, which for open research infrastructure is kind of the point.

Then yesterday I learned that the Dutch police use this approach too. They bring together data from lots of different sources, with different teams responsible for their own part, on an open source stack. That made me sit up. If it works for an organisation that big, with that many sources and owners, it can surely work for us. Because our situation looks surprisingly similar: OpenAlex, OpenAIRE, Crossref, ROR, OpenAPC, the Netherlands Research Portal and a CRIS system at every institution, each with its own quirks and its own people behind it.

So this post is my pitch to the BROCCOLI team. I really hope you pick this up, and I want to help you start on the right foot. Open Research Information (ORI) is a big field and our lake will serve many purposes. For BROCCOLI that is Open Access deals, Responsible Research Evaluation, and Open Science Monitoring. But for this blog post, let's focus on the one from the PID2Portal project: ORI Quality Monitoring. That's the dashboard that shows data stewards how complete and consistent the metadata about their institution is, where the gaps are, and what they can do about them.

The three ideas we borrow

We're not a big company with twenty domain teams, so we won't do data mesh by the book. But three of its building blocks solve problems we will definitely run into.

First, the data product. Think of it as more than a table or a pipeline. It's a bundle of a pipeline (the code that builds it), the data it produces, a contract describing what it offers, and, importantly, a team or curator who is responsible for it. The messy raw and intermediate tables stay inside. Other people only touch what the product officially offers, its output ports.

Second, the data contract. That's a YAML file describing an output port: schema, meaning of each field, quality rules, how fresh it is, who owns it. Humans can read it, machines can test against it. So the same file is both our agreement and our test suite. Nice.

Third, the data marketplace: one place where people find data products they can trust. Jochen Christ wrote a good piece on why that's not the same as a data catalog. A catalog happily indexes every table it can find, and before you know it you have thousands of mystery tables nobody feels responsible for. A marketplace only shows the finished products, each with an owner and a contract.

On top of that, data mesh talks about federated governance, which sounds heavy but just means: we agree on a few rules together, and each product owner makes sure they're met. Think of rules like "identifiers are always normalised the same way", "every output port has a contract" and "contracts only change through a pull request".

What it looks like

Here's the whole thing in one picture. I drew it with the consumers at the top, because that's where the design starts (more on that below). Data flows upwards, from the sources at the bottom to the dashboard at the top. Every dashed frame is one data product, with its own owner, pipeline, data and contract. Everything lives in the ORI DuckLake on SURF Object Store.

Medallion layers, data mesh style

You probably know the medallion architecture: staging, bronze, silver, gold. Here's how those layers map onto data products, with the names we'd use for ORI Quality Monitoring.

Staging is just the landing zone: buckets with the files exactly as we got them, plus a small manifest (source, harvest date, path, checksum). We never touch those files, which makes every later step reproducible. PDFs live here too, for later enrichment. Nobody outside the product reads from staging.

Bronze turns each source into a proper source-aligned product. Stay close to the source: flatten, cast types, deduplicate, but don't reinterpret. Think of it as a shock absorber. When OpenAlex changes its schema again, the damage stops here. DuckDB doesn't read XML natively, so a small Python step converts XML first, and PDFs stay files that a bronze table points to. And note: bronze products aren't specific to ORI Quality Monitoring. Everyone using the lake benefits from them, so they don't get the ori_quality prefix.

Silver is where ORI Quality Monitoring really starts. We take the publications of the Dutch organisations in scope (the list in nl-orgs.baseline) from all bronze sources and put them in one table, with exactly the fields the gold tables need. Every row remembers which source said what, because comparing sources is half the fun of quality monitoring. Matching is where silver earns its keep. We take the OpenAlex and OpenAIRE publications that carry the organisation's ROR, plus the publications from its own CRIS, and match them across sources: exactly on DOI where there is one, and otherwise with exact and fuzzy matching on other identifiers and titles. Every row gets a cluster ID, the method that matched it and, for fuzzy matches, a score. That is also how a DOI that is missing in the CRIS can be found in OpenAlex or OpenAIRE. And silver keeps every snapshot, not just the latest one, because the dashboard shows how quality develops over successive harvests. In sprouts-dev, the pid2portal schema with its NL subsets of OpenAlex and OpenAIRE is the closest thing to silver we have today.

Gold exists for one reason: making the dashboard feel quick. The prototype has four tabs (Coverage, Completeness, Enrichment and Accuracy), and every chart or table on them gets its own small, pre-aggregated table. Narrow, already filtered, already grouped, including a ready-made "All Types" total so the UI never has to add things up. If a widget still has to scan silver when someone clicks, the gold table isn't done yet.

One more thing about bronze: two of our sources are lakes themselves. openaire_publications reads the OpenAIRE Graph, and specifically the Dutch slice of it: the netherlands community of the Netherlands Research Portal, through the community data dump or the community filter in the Graph API (relCommunityId=netherlands). cris_<institution> harvests the institutional CRIS systems over the same OAI-PMH endpoints that OpenAIRE harvests. The OpenAIRE Graph has medallion layers of its own. It harvests the CRIS metadata as delivered (bronze), deduplicates it and unites it with everything else it knows (silver), and serves it out for specific uses, such as the Netherlands Research Portal and the KB e-Depot (gold). So the Graph's gold is our bronze: lakes feeding lakes.

The nice part is that the Dutch Repository Federation (DURF) is writing data contracts for exactly those two handoffs. C1 fixes what each CRIS delivers, as a Dutch application profile of the OpenAIRE Guidelines for CRIS Managers. C5 fixes what the Dutch slice of the Graph contains. Those two can be our bronze contracts for cris_<institution> and openaire_publications. And for once they're promises made by the producers, not just our description of what happened to arrive. When a university fixes its ROR IDs to meet C1, both the Portal and our dashboard see it.

A quick word on names. A data product called ori_quality_publications lives in a repository called data_product_ori_quality_publications. The dbt pipeline is one part of that repo, next to the contracts and the product description.

Start with the user journey, not with the data

This is the bit I care about most. It's so tempting to start at the sources and build "everything that might come in handy" upwards. I've seen where that ends: a huge silver table full of columns nobody asked for, and a dashboard that still misses what users actually need.

So let's design the other way round. In the UI design workshop with Barcelona Declaration signatories we sketched real screens and agreed on what a data steward needs to see. That became the ORI Quality Monitoring prototype, and here's a gift: its Data tab already lists the datasets behind every widget, with their columns. Those are our gold contracts, waiting to be written down. From all gold contracts together we work out the silver contract: exactly the fields and grain gold needs, plus keys to trace records back. Only then do we start building, bottom-up, with dbt.

Bronze contracts are the exception. They don't depend on the user journey, because they simply describe what we receive. For OpenAIRE and the CRIS systems you don't even have to write them: reuse the DURF contracts C5 and C1 mentioned above. For the other sources you can write them straight away, in parallel, and often get a head start by generating them from a sample file with datacontract import parquet or datacontract import json. Where the two directions meet, at silver, you do a reality check: can the sources actually deliver what the dashboard needs? If not, have that chat with the dashboard owner before anyone writes a line of SQL.

The folks at Entropy Data describe the same habit in their guide to data products with dbt: write the contract first, as your requirements, and your product stays small and focused. For BROCCOLI I'd love to turn that into a house rule: no silver or gold table without a contract that traces back to something a user needs. A new silver column needs a gold contract that uses it. A new bronze source needs a silver field it improves.

And here's the thing: none of this is specific to ORI Quality Monitoring. The BROCCOLI proposal names three use cases: monitoring publisher deals, open science monitoring and responsible research evaluation, and it explicitly leaves room for more. The recipe is the same for each one. Sit down with the users, sketch what they need to see, turn every widget or report into a gold contract, and derive the rest from there. Each use case gets its own consumer-aligned data product, for example with a publisher_deals or os_monitoring prefix, just like ori_quality here.

The nice part is how much gets reused. The source-aligned bronze products are shared by everyone, so OpenAlex only has to be harvested and cleaned once. Silver products can often be shared too, or grow a few columns when a new gold contract asks for them. A new use case mostly adds its own gold, and sometimes its own silver. The use cases are also at different stages, and contract-first helps with all of them. Publisher deal monitoring builds on UKBsis and is the most mature, so its existing reports are basically contracts waiting to be written down. Responsible research evaluation is still exploratory, and there the contract-first habit is even more valuable, because it forces the question "what do users actually need?" before we build anything. And the proposal's promise that institutions take ownership of their data quality fits nicely with data products that each have a named owner.

To make it concrete, here's a shortened gold contract for the metric cards on the Completeness tab: "Has DOI", "Has ORCID", "Has ROR" and so on. Note that the progress chart below those cards gets its own contract, because its time axis is the harvest or snapshot date, not the publication year. Different grain, different output port.

# ori-quality-gold-completeness-metrics-v1.odcs.yaml (shortened)
apiVersion: v3.1.0
kind: DataContract
id: ori-quality-gold-completeness-metrics-v1
name: Completeness metrics
version: 0.1.0
status: draft
domain: ori-quality
dataProduct: ori_quality_monitor
description:
  purpose: >
    Feeds the metric cards on the Completeness tab: for the selected organisation,
    primary source, entity and type, the share of records with a filled and valid
    value per metadata field. Based on the latest snapshot of the primary source.
  limitations: >
    Best-guess contract. This table does not exist yet in the sprouts-dev catalog.
customProperties:
  - property: dashboardTab
    value: Completeness
  - property: prototypeDataset
    value: Completeness metrics
team:
  name: ori-quality-monitoring-team
schema:
  - name: completeness_metrics
    physicalType: table
    properties:
      - name: organisation_ror_id
        logicalType: string
        required: true
        businessName: filterOrganisationRorId
        description: ROR ID of the organisation without resolver prefix.
        examples: [008xxew50]
      - name: primary_source
        logicalType: string
        required: true
        businessName: filterPrimarySource
        description: One of CRIS, OpenAlex, OpenAIRE, Crossref.
      - name: entity_table
        logicalType: string
        required: true
        businessName: filterEntityTable
        description: Entity type. Currently only Publications.
      - name: publication_type
        logicalType: string
        required: true
        businessName: filterType
        description: "'All Types' rows hold the pre-aggregated total over all types."
      - name: field
        logicalType: string
        required: true
        description: One of doi, orcid, ror, grant_doi, issn, oa_status, corresponding_author.
      - name: label
        logicalType: string
        required: true
        examples: [Has DOI]
      - name: percentage
        logicalType: number
        required: true
        description: 100 * filled / total, one decimal.
      - name: total
        logicalType: integer
        required: true
      - name: filled
        logicalType: integer
        required: true
        description: Records with a filled and valid value.
    quality:
      - type: sql
        description: filled never exceeds total.
        query: SELECT COUNT(*) FROM completeness_metrics WHERE filled > total
        mustBe: 0
servers:
  - server: development
    type: s3
    endpointUrl: https://objectstore.surf.nl
    location: s3://sprouts-dev/ori_quality/gold/completeness_metrics/*.parquet
    format: parquet

Not much in there, right? That's the point. Whoever builds the cards can read this and know exactly what they'll get. And the prototypeDataset property links it straight back to the dataset in the prototype.

The tech stack: dbt, DuckDB, DuckLake, the Data Contract CLI and Apache Ossie

The data mesh site has a nice write-up of a dbt and Snowflake stack. We keep the dbt part and swap the expensive warehouse for something open and serverless.

DuckLake is our catalog. It stores tables as Parquet files on SURF Object Store and keeps schemas, snapshots and column comments in one small catalog file. Since the whole lake is just files, we can publish it read-only as a frozen DuckLake for anyone to query. DuckDB does the actual work: it runs the SQL on your laptop, in CI, or even in a browser. dbt writes and tests the transformations, and talks to DuckDB and DuckLake through the dbt-duckdb adapter. And the Data Contract CLI lints the contracts, generates dbt model YAML from them, and tests our output ports against them. No database servers to babysit, which is exactly what we already tried out in the sprouts catalog.

A data contract nails down the shape and quality of one output port, but it doesn't say how metrics and dimensions relate to each other, or which BI tool's semantic layer gets to own that definition next. That's the gap Apache Ossie (incubating), the Open Semantic Interchange spec, is meant to close: a vendor-neutral YAML format for semantic models — fields, joins, metrics — and the ontology concepts behind them, so any BI platform, query engine or AI agent can read the same definition instead of everyone growing their own semantic layer. For BROCCOLI, the fit is natural: derive the Ossie semantic model from the gold contracts the same way we derive dbt model YAML with the Data Contract CLI, so "what a metric means" stays traceable back to the contract that defines it, not duplicated by hand in every BI tool that consumes the gold output ports.

I'd go for one dbt project per data product, each with its own contracts and CI. Inside a product, only the output_ports/ folder is public. Everything else is yours to refactor whenever you like. The silver product could look like this:

data_product_ori_quality_publications/
├── dbt_project.yml
├── ori_quality_publications.odps.yaml        # data product description, incl. owner
├── models/
│   ├── input_ports/                          # bronze products we consume, with a cached copy of their contracts
│   │   ├── openalex_works.source.yml          # sprouts-dev: openalex.works
│   │   ├── openalex_works.odcs.yaml
│   │   ├── openaire_publications.source.yml
│   │   ├── cris_publications.source.yml       # sprouts-dev: cris.publications
│   │   ├── crossref_works.source.yml          # not yet in sprouts-dev
│   │   └── nl_orgs_baseline.source.yml        # sprouts-dev: nl-orgs.baseline
│   ├── staging/                              # 1:1 cleaning per input
│   ├── intermediate/                         # matching and deduplication across sources
│   └── output_ports/v1/
│       ├── ori_quality_publications.sql
│       └── ori-quality-silver-publications-v1.odcs.yaml
└── .github/workflows/data-product.yml

Keeping a copy of each upstream contract next to its source definition means git log tells you when a source changed under your feet. When you start a model, let the CLI generate the dbt YAML (columns, types, descriptions, an enforced dbt contract) with datacontract export dbt-models, so the contract stays the single source of truth. Then write the SQL. A gold model is usually just a plain aggregation over silver:

-- data_product_ori_quality_monitor/models/output_ports/v1/completeness_metrics.sql
-- Governed by ori-quality-gold-completeness-metrics-v1.odcs.yaml
with latest as (
    -- only the latest snapshot per source; the timeline model uses all snapshots
    select * from {{ ref('ori_quality_publications') }}
    qualify snapshot_date = max(snapshot_date) over (partition by source)
),
fields as (
    select *, 'doi'   as field, 'Has DOI'   as label, coalesce(doi_valid, false)    as ok from latest
    union all
    select *, 'orcid',          'Has ORCID',          coalesce(orcids_valid, false)       from latest
    union all
    select *, 'ror',            'Has ROR',            coalesce(rors_valid, false)         from latest
    -- ... same pattern for grant_doi, issn, oa_status, corresponding_author
)
select
    organisation_abbreviation,
    organisation_ror_id,
    source                                   as primary_source,
    entity_table,
    coalesce(publication_type, 'All Types')  as publication_type,
    field,
    label,
    round(100.0 * count(*) filter (where ok) / count(*), 1) as percentage,
    count(*)                                 as total,
    count(*) filter (where ok)               as filled
from fields
group by grouping sets (
    (organisation_abbreviation, organisation_ror_id, source, entity_table, publication_type, field, label),
    (organisation_abbreviation, organisation_ror_id, source, entity_table, field, label)   -- the 'All Types' rows
)

Testing happens at two levels. dbt test checks the internals, like uniqueness and not-null, and those tests change whenever you refactor. datacontract test checks each output port the way a consumer sees it: it reads the actual Parquet through DuckDB and runs the contract's quality rules. Those only change when the promise to the consumer changes, so they're the ones that should block a merge. On every pull request we also run datacontract breaking. A breaking change to a gold contract doesn't get merged without the dashboard owner saying yes, and ideally shows up as a new v2 output port next to v1.

Let AI draft the pipelines

Here's a bonus of writing contracts first: they make a great brief for an AI coding assistant. A contract says exactly what the output must look like, which fields mean what, and which quality rules must pass. Give an assistant like Claude Code the gold or silver contract plus the contracts of its input ports, and it can draft the dbt models, the staging SQL and the tests for you. Our agentic tools already teach Claude how the ORI DuckLake is structured, so it knows which tables to join and how to unnest the nested columns. You still review every line, of course. But you don't have to take the AI's word for it: datacontract test tells you whether the draft actually keeps the promise in the contract. The contract is the spec, the AI writes the first version, the tests keep everyone honest. That makes the upfront time on contracts pay off even faster.

The infrastructure: SURF Object Store, Research Cloud and Kubernetes

All of this runs on SURF infrastructure, and the nice thing is that the stack needs surprisingly little of it. There are three pieces.

SURF Object Store is our S3-compatible storage and the home of the lake. The staging files, all Parquet files of bronze, silver and gold, and the DuckLake catalog file all live there. Anyone with DuckDB can read the published catalog straight from it, which is what the dashboard, the DuckLake Overview and the MCP server do.

SURF Research Cloud is the workbench. Data engineers spin up a workspace there to develop and try out pipelines, write contracts, run dbt and DuckDB against a development copy of the lake, and explore data in notebooks. Because everything is just files and SQL, a workspace is set up in minutes, and what works there works the same way in production.

The Kubernetes cluster is where things run for real. The pipelines of each data product run there as scheduled containers: ingest, dbt build, datacontract test, and only when everything passes, publish the new snapshot to the catalog. The cluster is also a natural home for the services around the lake, like the dashboard and the MCP server. Each data product can have its own schedule and its own container, which fits nicely with the idea that each product has its own owner.

ORI Data Marketplace - (sort of)

Here's the fun part. We don't need to buy a marketplace tool, because almost by accident we already built something in that direction. It started as a by-product of the main deliverable, but it's a nice peek into what becomes possible once the lake is open.

The DuckLake Overview (code on GitHub) is a single HTML page that runs DuckDB in your browser. Anyone can scroll through the public ORI DuckLake: schemas, datasets, columns with descriptions and a quick profile, plus a query builder with shareable links. No account, no install. And the ORI DuckLake MCP server, part of our agentic tools, gives Claude or any other MCP-capable assistant read-only SQL access to the same lake. So people can just ask questions about the data in a chat. Together with the dashboard, that's three ways into the same data: a curated view, a browsable catalog and a conversation.

Turning that into a real marketplace is a small step. We write the descriptions from each contract into the column comments of the DuckLake catalog, so the Overview and the MCP server show the same definitions as the contracts. And we publish the contracts themselves as HTML with datacontract export html, right next to the Overview. Bronze and silver contracts are just as useful for institutions and researchers who want to dig into the lake themselves. If it grows beyond that, worth knowing which of those is actually open: OpenMetadata is Apache 2.0 with no feature gate, so it's the safer default if "no vendor lock-in" matters here as much as it does for the contracts themselves. Entropy Data's UI is a nicer fit for the data-product-marketplace workflow, but its free self-hosted Community Edition holds back role-based permissions and SSO for the paid tier, so treat it as source-available rather than fully open source. Either way, exchanging the semantic layer through the emerging Apache Ossie spec instead of a proprietary format keeps the door open. Because the contracts are standard YAML, nothing gets lost along the way.

Concretely, OpenMetadata would slot into the stack in three pieces, two of which need nothing new from us. First, its dbt workflow reads the manifest.json that dbt docs generate already produces: models, descriptions, tests and the ref() dependency graph become lineage for free, no extra step in our pipeline. Second, and more to the point of this post, OpenMetadata's Data Contracts feature imports and exports ODCS directly, in merge or full-replace mode, which is exactly the format our gold and silver contracts are already written in. So a contract stays the single source of truth: written once for datacontract lint and datacontract test, then pushed into OpenMetadata as-is instead of re-typed into another tool's schema. One caveat worth knowing before depending on it: OpenMetadata's internal ODCS model still diverges from the published v3.1.0 spec on some fields, so round-trip a contract through import/export once and diff it before trusting it in production. Third, the actual gap: DuckDB and DuckLake aren't among OpenMetadata's 130+ official connectors yet, only community ones exist, so getting the lake's own table metadata in — as opposed to what dbt already tells it — means adopting a community DuckDB connector or bridging lineage through OpenLineage from dbt-duckdb, at least until that lands upstream.

I went and checked how far "adopting a community DuckDB connector" actually gets us for DuckLake specifically, since the existing community connectors talk to a bare .duckdb file, not to DuckLake's own catalog. Sketch is up at surf-ori/openmetadata-ducklake-connector: a DuckLakeSource that attaches the lake with the ducklake DuckDB extension, walks its schemas and tables, and reads each column's comment — the same field our contracts already write into via COMMENT ON COLUMN — so descriptions arrive pre-filled instead of retyped. Worth being honest about what "sketch" means here: I pinned it to the current releases — openmetadata-ingestion 2.0.2.0 and duckdb 1.5.5, both the latest on PyPI as I wrote this, against DuckLake's own spec v1.0 (production-ready since April 2026) — and installed the real openmetadata-ingestion package to check the code against it, not just against my own assumptions. That caught one real bug (an import path that doesn't exist in 2.0.2.0) and confirmed the request objects, the nested Custom Service config path, and the schema-introspection SQL all do what I expected — against a plain DuckDB database. What I couldn't check: the actual ATTACH 'ducklake:...' call. The ducklake extension downloads over plain HTTP, which the sandbox I was working in blocks outright, so the catalog-qualified queries remain unverified against a real DuckLake, and nested STRUCT/LIST columns aren't mapped yet. Try it against an actual lake before trusting it with anything.

I'd flagged OpenLineage from dbt-duckdb earlier as another way to close the "DuckDB and DuckLake aren't official connectors" gap, so I went and checked whether it's actually an alternative to the connector above or something else. It's something else. OpenMetadata's own OpenLineage connector docs say table resolution is read-only: the tables, schemas and databases have to already exist in OpenMetadata, or the incoming lineage events have nothing to attach to. So it can't discover the lake's catalog or carry over our contracts' column comments the way DuckLakeSource does — it needs that connector (or the dbt workflow) to have run first. What it adds instead is real column-level lineage: dbt-ol, OpenLineage's drop-in replacement for the dbt command, does support the duckdb adapter, and parses each model's compiled SQL (its own parser, or the sqlglot-based approach in this project) to say which output column came from which input columns — something the plain ref() DAG from manifest.json can't do, since that's model-level only. The catch is where it plugs in: not a periodic ingestion pull like the other two, but a swap in the pipeline itself, dbt-ol build instead of dbt build in the Kubernetes cron step, emitting lineage events live as each run happens.

There's a bigger catch, though, and it only showed up once I sketched the plumbing: OpenMetadata's OpenLineage connector doesn't take a plain HTTP POST the way Marquez does. Per its own docs and a recent feature request for a third option, it only consumes events from a Kafka, Kinesis or NATS JetStream topic. That's a real piece of infrastructure to run or pay for, in a stack whose whole pitch was "no database servers to babysit". I've sketched the producer side — an openlineage.yml generator pointed at a Kafka topic — as openlineage_config.py in the same connector repo, but I'd think hard about whether column-level lineage is worth a Kafka broker before standing any of this up. The dbt workflow's model-level lineage is already free.

Things I'm not sure about yet

Let's be honest about the rough edges. DuckLake is still young, and in the sprouts work we hit some instability with compound queries across schemas. So keep models simple, and check the dbt-duckdb and DuckLake release notes before pinning versions. The Data Contract CLI tests Parquet on S3 through DuckDB, but I haven't seen native DuckLake support yet, so for now each contract points at the Parquet location of its output port. Crossref isn't in sprouts-dev yet, while the dashboard counts on it. Keeping every snapshot in silver makes it grow with each harvest, so we'll need to think about partitioning early. And yes, contract-first costs some time up front. There will be moments where it feels slower than just building the table. In my experience you win that time back fast once three people work on the same pipeline.

So, where to start?

Pick one widget from the workshop screens. Write its gold contract together with whoever builds the front end. Work out which silver fields it needs, and build the thinnest possible path from one bronze source (OpenAlex is the obvious candidate) to that one chart. Run datacontract test in CI from the very first commit. Once that path works end to end, every next widget or source is repetition rather than invention.

Want to read more? The data mesh architecture site, the Data Contract CLI docs and Entropy Data's dbt guide are great places to start. And if something here doesn't fit the way you work, please tell me. That's what federated governance is for, after all.