Data Lineage in Snowflake, Databricks, dbt and Collibra: What Each One Actually Covers

All four of these give you data lineage. Each one gives it to you inside its own boundary, and each publishes the point at which it stops. Those stopping points are the whole subject of this post, because an end to end regulatory requirement lives exactly where they are.

If you are asking whether Snowflake has lineage, the answer is yes. The useful question is what happens to a field on its way in and on its way out.

The scorecard

No ranking. These are four different jobs and three of them are not competing with each other.

Snowflake

Snowflake tracks column lineage across table-like objects: tables, views, materialised views, dynamic tables, external tables and Iceberg tables. It is an Enterprise Edition feature.

Two limits matter for a regulated estate.

Column lineage is not available for external sources. Snowflake will show you object level lineage for data arriving from external ETL tools and databases, so you can see that something came from outside. You will not get the field level trace across that boundary.

It needs a direct SQL relationship. Lineage is derived from CTAS, INSERT, MERGE and similar statements. Snowflake states it cannot track data movement across disjointed queries or separate statements. A transformation split across a procedural sequence is where this shows up.

Column lineage was introduced in November 2024 and is retained for one year. Movement before that date is not available.

Databricks Unity Catalog

Unity Catalog captures lineage down to the column, automatically, for workloads that run on Databricks. In Catalog Explorer it is retained indefinitely for anything captured after 1 September 2024. The system tables hold a rolling one year window.

Its documented exclusions are the interesting part:

• Operations outside Databricks are not captured by default. Upstream sources and downstream BI tools can be registered as external assets, which records the connection rather than the column path through it.
• Resilient Distributed Datasets are not captured.
• Global temp views are not captured.
• Jobs using the spark submit task type are not captured.
• User defined functions obscure the mapping between source and target columns.
• Lineage cannot be tracked when the source or target is referenced as a path rather than by table name.

Read that list as a shape rather than as a set of gaps. Unity Catalog traces what it can see the engine do in a form it can interpret. Push logic into a UDF or a path reference and the trace thins out at precisely the step somebody wanted explained.

dbt

This is the one most often misremembered.

Column level lineage is a dbt Cloud feature on Enterprise and Enterprise plus plans, and requires access to dbt Catalog. dbt Core does not have it. dbt Core gives you a model level DAG built from ref() and source(), which is genuinely useful and is not column level.

Where it is available, dbt states three limits worth knowing before you rely on it for evidence:

• It reflects lineage from select statements only. Joins and filters are not reflected, so a field that determines whether a row is included does not appear as an input to the value.
• It cannot parse Python models.
• Column origin cannot be determined for JSON unpacking, complex lateral joins, or a hardcoded table name used instead of a ref.

At least one job in a production or staging environment has to have run dbt docs generate for any of it to exist.

Collibra

Collibra is in a different category from the other three. It is a catalog: the place lineage is stored, governed and presented, with a technical lineage capability that ingests from sources through harvesters.

That makes its coverage a function of what its harvesters can parse in your estate, which is the right way to evaluate any catalog. Atlan, DataHub, Solidatus and Informatica sit in the same category and the same question applies to each.

A catalog is not a substitute for lineage extraction. It is where extracted lineage goes. Programmes that stall usually stall because the catalog was bought and there was nothing upstream capable of filling it.

The pattern across all four

Each tool is bounded by the engine it belongs to, and each thins out at the point a transformation stops being plain SQL it can read.

That boundary is fine for most engineering work. It becomes the problem when the requirement is end to end.

Under BCBS 239, a bank is expected to reconcile a figure in a risk report back to source, at attribute level, across the whole path. The path does not stop at the warehouse edge because that is where the tooling stops. In most regulated estates the awkward stretch is upstream of all four of these: fifteen year old Pentaho or Informatica jobs, stored procedures, COBOL, written by people who have left.

A .ktr file usually does not name what it connects to. It names a variable that resolves from a properties file set by a parent job upstream. None of the four tools above claims to read that, and none of them should. It is not their job.

Where we fit, and where we do not

Dataworkz sits upstream of the catalog, not in place of it. We run agents that read legacy transformation logic, resolve the variable chains, and emit field level lineage into a catalog like Collibra. At one Fortune 200 financial services firm, lineage work that used to take weeks of analyst time now completes in minutes, for the estate in scope.

Two honest limits.

We are not a catalog. If you do not have one, you still need one. We produce the lineage; Collibra or an equivalent is where it lives and gets governed.

Reading unreadable code is a probabilistic step, not a deterministic parse. Snowflake parsing its own SQL is exact. An agent interpreting a fifteen year old job is not, which is why the output carries the source document and the reasoning behind each mapping, and why ambiguity is flagged for a human rather than resolved silently. If your estate is entirely modern SQL, you do not need us for this. Use the lineage your platform already gives you.