Runtime Lineage: End-to-End OpenTelemetry Tracing for AI Data Systems
Most data lineage is static. Catalogs, dbt graphs, and SQL parsers read your code ahead of time and draw a table-to-table, column-to-column map of how data is supposed to flow. That is useful, until it isn’t: it describes the intended design, not what actually happened on a given run. And it stops at the column boundary, not the full path from raw source to the value something finally consumes. That gap turns into a real operational problem once AI agents enter the picture. Agents pick data sources and transformations dynamically, per request, at runtime. There is no static graph for a path that is decided live. So, when an agent on a complex data system returns a wrong or surprising answer, a design-time diagram cannot tell you what it actually did. This talk makes the case for runtime lineage: capturing the real, end-to-end path as the system executes, and treating it as an observability problem rather than a documentation one. OpenTelemetry turns out to be the natural carrier, it is already a runtime, distributed-trace standard. Using mloda (Apache-2.0), whose Extender hook wraps every transformation, a whole run becomes a single trace from source to consumed value, viewable in the Grafana Tempo / Jaeger you already operate. Live demo: watch one end-to-end run as a trace, then change the data path and watch the trace change, because it reflects what ran, not a static model.
Speaker
-
Tom KaltofenmlodaTom Kaltofen is a Berlin-based data and AI engineer and the creator of mloda, an open-source (Apache-2.0) Python framework. After years in data and ML plumbing, pipelines, feature stores, RAG, he kept hitting the same blind spot: the data layer every system depends on is the one part of the stack monitoring never reaches. mloda is his attempt to fix that, making data transformations observable and reusable instead of hand-wired.