Skip to content
Vibedata

Re-engineer something you own

Convert a full-refresh dbt model to incremental and show the runtime delta

  • A dbt model is correct but a full refresh is too slow or too expensive to run routinely.
  • A full-refresh model needs to become incremental without changing its required consumer-visible output.

Convert an existing full-refresh dbt model to an incremental implementation while preserving its required semantics, handling agreed late-arriving data correctly, and recording the before-and-after runtime.

Area
Transformation
Runs on
  • DuckDB
  • MotherDuck
  • Microsoft Fabric Lakehouse
  • Microsoft Fabric Warehouse
Built with
  • dbt
Readiness
SupportedEverything this Recipe composes runs today, without a case that proves this exact shape.

Use this Recipe

Use the VibeData Recipe `dbt-full-refresh-to-incremental` at https://github.com/accelerate-data/vibedata-cookbooks/tree/main/recipes/dbt-full-refresh-to-incremental Read the Recipe and execute it in the context of the current Intent.

Recipe id dbt-full-refresh-to-incremental · Canonical definition

Verified by

What has to be observably true before this Recipe is finished.

  • Incremental output matches the approved full-refresh baseline on the agreed comparison slice.
  • A repeated incremental run produces no duplicate unique keys or unintended row changes.
  • The agreed late-arriving-data case updates exactly the expected records.
  • The before-and-after runtime is recorded from observed executions.
Recipe promptThe task specification the agent reads. Reference only — it is not what you copy.
Convert the existing full-refresh dbt model to an incremental model. Preserve the required consumer-visible output and downstream contract. Determine and implement the appropriate incremental strategy from the model and source evidence, verify the unique-key and late-arriving-data behaviour, and record the runtime change against the current full-refresh baseline.
Agent guidanceHow the agent approaches the work, and what it will not do.

Inspect the existing model, dependencies, source behaviour, and current full-refresh result before changing materialization. Use evidence to choose the incremental strategy and key. Implement the smallest change that preserves the model's required semantics. Run the old and new paths in the appropriate isolated environment and collect actual comparison and runtime evidence before declaring the Recipe complete.

Composes

  • profile-source-data
  • modify-dbt-model
  • run-dbt-in-sandbox
  • compare-to-baseline
  • test-incremental-behaviour

Asks first

Semantic decisions the Intent cannot supply. Never context Studio already holds.

  • Ask for the intended late-arriving-data policy only if it cannot be inferred from approved project requirements or existing model behaviour.
  • Ask for the intended business key only if no defensible unique key can be established from repository and data evidence.

Guardrails

  • Do not invent a unique key merely to make the incremental materialization compile.
  • Do not change consumer-visible model semantics merely to make incremental processing easier.
  • Do not treat predicted performance as evidence; record observed runtime from actual executions.

What you need

  • DuckDB, MotherDuck, Microsoft Fabric Lakehouse, or Microsoft Fabric Warehouse.
  • A dbt project you can build, and read access to the models it starts from.
  • dbt in the project, or the intent to add it.

How it goes

  1. Name the thing you own, and what about its behaviour has to stay the same.
  2. Let it read the current form and write down the behaviour it is holding constant.
  3. Review the design before any SQL is written. Grain first.
  4. Let it rebuild in an isolated copy of your estate. Production is untouched throughout.
  5. Reconcile the new form against the old one, row by row.
  6. Read the acceptance conditions. They are the contract; the diff is not.

What you end up with

The deliverable, in your own repository, as transformation work a reviewer who knows the project reads as native to it. Alongside it, the evidence for every one of the acceptance conditions above — which is the part that is still there in three weeks when somebody asks.

Next to this one