Skip to content
Vibedata

Build something that is not there yet

Declare source freshness SLAs and make stale data fail before downstream work

  • When stakeholders notice yesterday's data before the data engineer does.
  • When a source has gone stale before and nothing in the run caught it before the marts built on top of it.

Declare a loaded-at field, a warn threshold, and an error threshold on every source, and make freshness the first step of the run so stale data fails before downstream models build on it. Covers the freshness checks themselves, not the fix for whatever made a source late.

Area
Data quality
Runs on
  • Microsoft Fabric Lakehouse
  • Microsoft Fabric Warehouse
  • MotherDuck
  • DuckDB
Built with
  • dbt
Readiness
SupportedEverything this Recipe composes runs today, without a case that proves this exact shape.

Sample This Recipe has not been materialized in the Cookbook repository yet. Its trigger, description, prompt, agent guidance and acceptance conditions, and the explanation below, are prototype drafts. Its name, job, area and readiness come from the reconciled Cookbook seed snapshot. Readiness is a separate question from this one: it says whether the capability exists, not whether the writing has been reviewed.

Use this Recipe

Use the VibeData Recipe `source-freshness-sla` at https://getvibedata.ai/cookbook/source-freshness-sla Read the Recipe and execute it in the context of the current Intent.

Recipe id source-freshness-sla · Not yet materialized in the Cookbook repository, so the pointer addresses this page.

Verified by

What has to be observably true before this Recipe is finished.

  • a stale fixture fails the freshness check
  • the freshness check runs before any downstream model in the same pipeline
  • a source that only loads on weekdays does not alert on its weekend gap
  • every source in the project carries a warn and an error threshold
Recipe promptThe task specification the agent reads. Reference only — it is not what you copy.
Deliver: Declare source freshness SLAs and make stale data fail before downstream work.

Declare a loaded-at field, a warn threshold, and an error threshold on every source, and make freshness the first step of the run so stale data fails before downstream models build on it. Covers the freshness checks themselves, not the fix for whatever made a source late.

Execute inside the current Intent. Its Domain, repository, platform, environment and attached sources are the context for this work — read them rather than asking for them.

The work is done when:
- a stale fixture fails the freshness check
- the freshness check runs before any downstream model in the same pipeline
- a source that only loads on weekdays does not alert on its weekend gap
- every source in the project carries a warn and an error threshold

Report the evidence for each condition above with the result. A condition you cannot meet is something to say, not something to work around.
Agent guidanceHow the agent approaches the work, and what it will not do.

Profile the inputs the grain, joins and measures actually depend on before proposing a model. Put the design up for review — grain first — then build in an isolated copy with tests and documentation landing beside the model rather than after it.

Composes

  • test authoring and baseline comparison
  • dbt in the project
  • isolated-copy execution and gate verification

Asks first

Semantic decisions the Intent cannot supply. Never context Studio already holds.

  • which Fabric target this work lands on, when the Domain carries both a Lakehouse and a Warehouse

Guardrails

  • Build in an isolated copy. Production is read, never written.

What you need

  • Microsoft Fabric Lakehouse, Microsoft Fabric Warehouse, MotherDuck, or DuckDB.
  • The objects under test, and a baseline you agree is correct.
  • dbt in the project, or the intent to add it.

How it goes

  1. State the outcome in one sentence, in the language the request arrived in.
  2. Let it profile the inputs the grain, the joins and the measures actually depend on.
  3. Review the design. Disagreeing about grain here costs a sentence; after the model exists it costs a rewrite.
  4. Let it build in an isolated copy, with the tests and the documentation landing beside the model rather than after it.
  5. Read the acceptance conditions against the run.

What you end up with

The deliverable, in your own repository, as data quality work a reviewer who knows the project reads as native to it. Alongside it, the evidence for every one of the acceptance conditions above — which is the part that is still there in three weeks when somebody asks.

Next to this one