Start with the unglamorous problem

Picture a retailer with a few thousand stores, three ERP systems inherited from acquisitions, a loyalty app, a warehouse network and a finance team that closes the books on the fifth business day of every month. Now ask a simple question: "How much did we sell last Tuesday, and how much of it was profitable?"

In a startup, somebody writes a query. In a large enterprise, that question touches six systems, four teams and at least two definitions of "profitable." Nobody is being careless. That is just what growing by addition looks like.

A data platform is the answer to that mess. Not a single product, though vendors would love you to think so, but a set of layers that carry data from where it is created to where someone can trust it. The public engineering writing I have read from big companies keeps landing on similar shapes. The details differ. The skeleton does not.

The layered skeleton

Most large platforms can be drawn as five layers with a set of shared services underneath. Sources feed ingestion, ingestion lands data in storage, processing reshapes it, and consumers (analysts, models, applications) read the result.

Two things about this picture are worth slowing down for.

First, the grey bars at the bottom. Orchestration, governance, security and cost control are not decoration. In my experience they are where the real budget goes, and they are the part most often skipped in a first version. A platform without a catalog and access policies works fine for about eight months. Then someone asks who can see customer addresses, and nobody can answer.

Second, the raw zone. Mature teams keep an untouched copy of what arrived, even when it looks ugly. It feels wasteful, but the first time a transformation has a bug (and it will), you can replay history instead of apologising to finance.

Why almost everyone landed on the lakehouse

For years the choice was framed as warehouse versus lake. Warehouses gave you fast SQL and strict structure but got expensive at scale. Lakes gave you cheap storage for anything, but they tended to turn into swamps where nobody knew which files were current.

The lakehouse idea tries to take the good half of each. Data sits in cheap object storage, and an open table format adds the things warehouses are good at: transactions, schema enforcement and the ability to go back in time. Netflix created Apache Iceberg to handle huge tables on object storage, and Uber created Apache Hudi to deal with frequent updates on its data lake. Delta Lake, from Databricks, plays a similar role. These projects came out of real pain at companies with very large data volumes, which is partly why enterprises trust them.

That said, I would not call the question settled. Plenty of Fortune 500 firms run a cloud warehouse such as Snowflake or BigQuery as their main engine and are perfectly happy. Open table formats reduce lock-in and help with cost, but they add operational work, and whether that trade pays off depends on your team size and how much you value flexibility over simplicity.

Batch, streaming, and the stubborn middle

Here is a question I like to ask about any pipeline: how stale can this number be before somebody notices? For monthly revenue, a day is fine. For card fraud, a few seconds is already too slow. Most enterprises need both speeds, and that is why you see two paths side by side.

The fast path usually runs through an event log, with Apache Kafka as the common choice (it was born at LinkedIn). Stream jobs compute things like rolling windows and joins. The slow path handles backfills and heavy joins that would be silly to run continuously.

The older Lambda approach kept two code bases producing two answers that someone had to reconcile, which anyone who has maintained duplicate business logic will remember with a wince. The newer habit is to let both paths write into the same table format, so consumers see one dataset. It is not perfect, since late-arriving events and exactly-once guarantees still start arguments in design reviews. But it beats reconciling by hand.

Bronze, silver, gold

Inside the lakehouse, many teams organise data into stages of refinement. The naming comes from the Databricks community and has become a kind of shared vocabulary.

Bronze is the raw copy, kept as received. Silver is cleaned: duplicates removed, types fixed, codes joined to reference tables. Gold is business-ready, shaped around questions people actually ask, like daily margin by region.

The names are a little cute, and "medallion architecture" made me roll my eyes at first, but the idea is sound. Each stage has a different audience and a different promise. Engineers and auditors care about bronze. Analysts live in silver. Executives and regulators should only ever see gold, because gold has passed checks.

The checks matter more than the colours. A good platform halts a pipeline when a quality rule fails, say when yesterday's order count is suddenly 90 percent lower than usual. A bad one lets the number through and a vice president finds the error in a board meeting.

The organisational question nobody can avoid

Technology is the easier half. The harder half is deciding who owns what.

The classic model is a central data team that builds everything. It gives you consistency, and also a queue, and the team slowly becomes the bottleneck it was meant to remove.

The alternative that has gotten a lot of attention is data mesh, a term Zhamak Dehghani introduced while at Thoughtworks. The core idea is that the people closest to the data, such as the finance team or the supply chain team, own it and publish it as a product with an owner, a schema and a service level. A central platform team provides the shared tools so nobody has to reinvent storage and pipelines.

I find the idea persuasive, with caveats. Mesh asks domain teams to take on work they may not have the skills or the headcount for. It also depends on strong global rules, since a hundred data products with a hundred naming conventions is just a bigger swamp. Most large companies I have read about end up somewhere in the middle: a strong central platform, federated ownership of the data itself, and a lot of negotiation about where the line sits.

Governance is where trust gets built

If I had to pick one area where enterprises differ most from smaller companies, it would be governance. Regulations push hard here. Banks answer to financial regulators, healthcare firms to privacy law, and anyone with European customers to GDPR.

In practice, that means a few habits show up again and again:

  • A catalog that tells people what data exists, who owns it and how fresh it is.
  • Lineage, so you can trace a dashboard number back to the source system, which is invaluable when someone disputes it.
  • Column-level access control and masking, so an analyst can count customers without seeing their phone numbers.

A few patterns worth stealing

After reading about enough of these platforms, a handful of lessons feel durable.

Migrate in slices, not in one go. Capital One is a well-known example of a bank that moved heavily to the public cloud and closed its own data centres, but that was a multi-year effort, not a weekend project. Pick one domain, move it, learn, then repeat.

Define metrics once. If "active customer" has three definitions, no architecture will save you. A shared semantic layer, or even a well-maintained metrics document, helps more than another tool.

Be honest about what you do not need. A mid-sized firm with one warehouse and clear ownership may be better off staying simple. Fortune 500 patterns exist because of Fortune 500 problems.

Where I would stay cautious

There is no single correct architecture here. Most of what I have described comes from public talks and blog posts, which tend to show the version that went well. The internal reality likely includes failed migrations and a few pipelines nobody dares touch. So treat the diagrams as a vocabulary rather than a blueprint, plus a checklist: where does raw data land, who owns each dataset, what stops bad data from reaching a dashboard, and how would we explain a number to an auditor?

The short version

Big companies build data platforms the same way they build anything large: in layers, with rules, and with a lot of compromise. The tooling will keep changing. Table formats, engines and vendors come and go. The questions underneath, about ownership, trust and cost, tend to stay put.

If you are starting out, I would begin with the grey bars from the first diagram. Get the catalog, the access rules and the quality checks in place early, even if the pipelines are small. It is much easier to grow a trustworthy platform than to make an untrustworthy one believable.

If you have worked on an enterprise data platform, I would genuinely like to hear what surprised you. Which part turned out harder than the diagrams suggested?