Unnamed engagement · Data platform · Azure / Databricks
The challenge
Overnight loads failed silently. Bad or missing numbers reached reports before anyone knew, and every new source meant a rebuild.
What we shipped
A layered pipeline where data is checked at each stage before promotion, new sources plug in through configuration, and alerts reach the owner before bad numbers reach the business.
Outcome
Data is checked before it is published. A failed load reaches a person. A new source is configuration, not a rebuild.
This delivery stays unnamed. The client is not published here.
3 layersraw → cleaned → trusted before publish
1 ownerfailures page a person, not a log
0 rebuildsnew sources via config, not a rewrite
Under the hood
DatabricksDelta Lake TablesUnity CatalogAzure Data FactoryAzure MonitorDataDog
How do you find out today when a load fails?
Talk about it