Enterprise data warehouse migration to Databricks
A core engineer on an enterprise-wide Azure Synapse-to-Databricks migration spanning more than 1,700 tables across Finance, HR, Customer Direct, and Sales & Marketing. We migrated the datamarts behind revenue, labor, profitability, promotions, incentive compensation, and acquisitions reporting onto a Medallion lakehouse, without disrupting the reports the business runs on.
- 1,700+
- Tables in the program
- 50+
- Tables and procedures migrated
- ~20K
- Lines of T-SQL converted
- Led the team that set up the Unity Catalog governance framework: catalog and schema structure, naming conventions, and the standards every migration domain now follows
- Converted T-SQL stored procedures to Databricks SQL on a Bronze/Silver/Gold Delta Lake architecture, deployed with Databricks Asset Bundles
- Built a Python conversion framework on Lakebridge, with custom pre- and post-processing, that automated the translation of 25 stored procedures
- Parallel validation compares row counts and MD5 checksums on every table, with dashboards, Excel and CSV reconciliation reports, and CI exit codes that stop a failed release