Data lineage is the record of where a data element came from, what happened to it along the way, and where it ended up. Column-level lineage records that path for one field rather than for a whole table. End-to-end lineage records it across every system the field passes through, not only inside one of them.
That definition is easy. Producing the record is not, and almost every conversation about lineage skips straight past the hard part.
Most lineage tooling assumes the lineage already exists and needs somewhere to live. Catalogs are built to store and visualise it. That assumption holds while your transformations are readable. It stops holding the moment the logic that moves a number from a source system to a regulatory report lives in ETL that nobody currently employed can read.
How data lineage works
Lineage is derived, not declared. Something has to read the logic that moves data and turn it into a graph of source fields, transformations and targets.
There are three places that logic can be read from.
The query layer. Parse the SQL that ran. Warehouses increasingly do this natively, and it is accurate for anything expressed as a query inside that warehouse.
The transformation code. Parse the pipeline definitions themselves: dbt models, Spark jobs, ETL job files. This is where most enterprise transformation logic actually sits.
Runtime observation. Watch what data moved where, and infer the relationship. Useful where nothing is readable, weaker on why a value changed.
Most estates need more than one, because most estates are more than one generation of tooling deep.
Table-level and column-level are not the same claim
Table-level lineage says this table feeds that table. It is straightforward to produce and no auditor has ever been satisfied by it.
Column-level lineage says this field, transformed this way, becomes that field. It is what someone means when they ask you to explain a number in a report. It is also where the difficulty concentrates, because a single column often arrives via a join, a case statement and a rename, and the rename is frequently the point at which the trail goes cold.
If a vendor says lineage without saying at what granularity, assume table-level.
Where lineage actually breaks
The failure is rarely conceptual. It is that the transformation logic is unreadable in a specific, boring way.
Take a Pentaho job. A .ktr file does not usually contain a connection string. It contains a variable. That variable resolves from a properties file, which may itself be set by a parent job three levels up, which reads an environment value that differs between environments. The file does not say what it connects to. It says how it will find out at runtime.
Multiply that across a few thousand jobs written over fifteen years by people who have left, and you have the actual state of most regulated data estates. This is why lineage projects stall. Not because nobody bought a catalog, but because the catalog has nothing to ingest.
The same shape appears wherever transformation logic predates the current team: stored procedures, COBOL copybooks, hand-rolled Informatica mappings.
What regulators actually ask for
Lineage requirements are usually attribute-level, which is the regulator’s phrase for column-level.
Under BCBS 239, the Basel Committee’s Principles for effective risk data aggregation and risk reporting, a bank is expected to be able to explain how a figure in a risk report was assembled, and to reconcile it to source. In life sciences, 21 CFR Part 11 turns on whether an electronic record and the reasoning behind it can be reconstructed. Different regimes, same underlying request: show the path from source to report, at field level, without a person reconstructing it from memory.
Two things follow from that.
The evidence has to be reproducible on demand, not compiled during an examination. A diagram assembled by an analyst the week before is not lineage. It is a drawing of what somebody believed.
And it has to survive the parts of the estate nobody wants to open. Regulators do not exempt the mainframe.
Choosing an approach
Three questions decide it.
Can your transformation logic be read by the tool you are considering? If the bulk of it is legacy ETL and the tool parses SQL only, you will produce lineage for the readable part and a gap where the risk sits.
Is the output column-level? If not, it will not answer the question that gets asked.
Is it produced or maintained? A lineage graph a human keeps up to date decays between audits. One that is regenerated from the jobs themselves does not.
Agents change the arithmetic here rather than the requirement. Reading a .ktr file, resolving its variable chain and emitting field-level lineage into a catalog is a task that reads unreadable documents and produces structured output, which is what this class of agent is for. At one Fortune 200 financial services firm, lineage work that used to take weeks of analyst time now completes in minutes, for the estate in scope. The requirement did not change. The cost of meeting it did.