Schema Drift when two versions should be two datasets - and lineage oh my

Data producers often create multiple versions of their data across time. Some of those changes are additive and easy to push from operational to analytical stores. Other changes are transformational breaking changes. We want to integrate these changes in a way that reduces the amount of magic required by our consumers.

All these changes should be captured in data catalogs where consumers can discover our datasets, their versions and, the datasets they feed.