Unnamed engagement

Daily loads that stopped breaking — and say so when they don't run

Data moves raw → cleaned → trusted, with checks before anything is published.

All cases

Unnamed engagement · Data platform · Azure / Databricks

The challenge

Overnight loads failed silently. Bad or missing numbers reached reports before anyone knew, and every new source meant a rebuild.

What we shipped

A layered pipeline where data is checked at each stage before promotion, new sources plug in through configuration, and alerts reach the owner before bad numbers reach the business.

Outcome

Data is checked before it is published. A failed load reaches a person. A new source is configuration, not a rebuild.

This delivery stays unnamed. The client is not published here.

3 layersraw → cleaned → trusted before publish
1 ownerfailures page a person, not a log
0 rebuildsnew sources via config, not a rewrite

Under the hood

DatabricksDelta Lake TablesUnity CatalogAzure Data FactoryAzure MonitorDataDog

How do you find out today when a load fails?

Talk about it