Skip to content
Vibedata

Re-engineer something you own

Replace a Fivetran or Airbyte connector with a dlt pipeline and reconcile the result

  • When the connector bill changed shape, or the connector has been broken with a pending fix for weeks.
  • When a managed connector is a black box and nobody can see what it actually does to the data.

Replace a Fivetran or Airbyte connector with a dlt pipeline landing to the same bronze schema, and reconcile row counts and key sets against the incumbent before cutover. Covers the replacement pipeline and its reconciliation, not disabling the incumbent connector.

Area
Ingestion
Runs on
  • Microsoft Fabric Lakehouse
  • MotherDuck
  • DuckDB
Built with
  • dlt
Readiness
SupportedEverything this Recipe composes runs today, without a case that proves this exact shape.

Sample This Recipe has not been materialized in the Cookbook repository yet. Its trigger, description, prompt, agent guidance and acceptance conditions, and the explanation below, are prototype drafts. Its name, job, area and readiness come from the reconciled Cookbook seed snapshot. Readiness is a separate question from this one: it says whether the capability exists, not whether the writing has been reviewed.

Use this Recipe

Use the VibeData Recipe `managed-connector-to-dlt` at https://getvibedata.ai/cookbook/managed-connector-to-dlt Read the Recipe and execute it in the context of the current Intent.

Recipe id managed-connector-to-dlt · Not yet materialized in the Cookbook repository, so the pointer addresses this page.

Verified by

What has to be observably true before this Recipe is finished.

  • row counts match the incumbent connector's landed tables for the overlap period
  • key sets match between the dlt pipeline and the incumbent for the same window
  • both pipelines run in parallel for the agreed period before cutover
  • the incumbent connector is not disabled in the same change
Recipe promptThe task specification the agent reads. Reference only — it is not what you copy.
Deliver: Replace a Fivetran or Airbyte connector with a dlt pipeline and reconcile the result.

Replace a Fivetran or Airbyte connector with a dlt pipeline landing to the same bronze schema, and reconcile row counts and key sets against the incumbent before cutover. Covers the replacement pipeline and its reconciliation, not disabling the incumbent connector.

Execute inside the current Intent. Its Domain, repository, platform, environment and attached sources are the context for this work — read them rather than asking for them.

The work is done when:
- row counts match the incumbent connector's landed tables for the overlap period
- key sets match between the dlt pipeline and the incumbent for the same window
- both pipelines run in parallel for the agreed period before cutover
- the incumbent connector is not disabled in the same change

Report the evidence for each condition above with the result. A condition you cannot meet is something to say, not something to work around.
Agent guidanceHow the agent approaches the work, and what it will not do.

Write down the behaviour that has to stay constant before you touch the current form, and get that agreed. Rebuild in an isolated copy, then reconcile old against new at the grain the consumers read. The reconciliation is the deliverable; the diff is not.

Composes

  • source profiling and incremental load
  • dlt in the project
  • isolated-copy execution and gate verification

Guardrails

  • Build in an isolated copy. Production is read, never written.
  • Do not edit the object being held constant. Parity that required a change to the original is not parity.

What you need

  • Microsoft Fabric Lakehouse, MotherDuck, or DuckDB.
  • Read access to the source. Not write access to production.
  • dlt in the project, or the intent to add it.

How it goes

  1. Name the thing you own, and what about its behaviour has to stay the same.
  2. Let it read the current form and write down the behaviour it is holding constant.
  3. Review the design before any SQL is written. Grain first.
  4. Let it rebuild in an isolated copy of your estate. Production is untouched throughout.
  5. Reconcile the new form against the old one, row by row.
  6. Read the acceptance conditions. They are the contract; the diff is not.

What you end up with

The deliverable, in your own repository, as ingestion work a reviewer who knows the project reads as native to it. Alongside it, the evidence for every one of the acceptance conditions above — which is the part that is still there in three weeks when somebody asks.

Next to this one