Description
Sr. Data Engineers (2) – Databricks Pipeline Developers
Role Overview
Build and maintain data pipelines in Databricks for ingestion, cleansing, enrichment, and storage.
Key Responsibilities
Develop PySpark Pipelines For
API ingestion
FTP file ingestion
Data cleansing
Enrichment transformations
Implement incremental & batch loads
Develop Bronze → Silver → Gold transformations
Translate legacy SQL stored procedures into Spark jobs
Optimize Delta Lake tables
Implement validation checks
Support testing & reconciliation
Required Skills
Strong PySpark / Spark SQL
Strong SQL (read & interpret legacy procedures)
Experience with API integration
Experience handling flat files (FTP ingestion)
Experience with Delta Lake
Knowledge of partitioning & optimization