Data Engineer (Outsource), KBTG
Nov, 2024 - Present
Enterprise data engineering on Databricks platform
Reviewed Data Dictionaries and sample data with System Owners before ingestion
Validated data consistency and data quality between documentation and actual datasets
Standardized metadata for Tables, Views, and RBAC on Databricks
Developed Python, PySpark, and Spark SQL for validation and quality checks
Performed data ingestion into Data Lake and executed unit tests
Managed GitHub repositories and deployed jobs via CI/CD to DEV and UAT
Conducted UAT system testing to support SIT and production releases