About the job
We are seeking a Data Engineer to help build scalable solutions that support AI enablement across the organization. This role focuses on developing reusable platforms, frameworks, and automation capabilities within Databricks, enabling data scientists and engineering teams to efficiently develop, deploy, and manage machine learning solutions. The ideal candidate has strong Databricks and Python expertise, along with experience in cloud platforms, DevOps, and modern data engineering practices.
Responsibilities
- Design, develop, and maintain data engineering and AI enablement solutions using Databricks and Python.
- Build reusable libraries and frameworks supporting authentication, authorization, logging, reporting, and other common platform capabilities.
- Develop and automate CI/CD pipelines supporting machine learning and data engineering workflows.
- Create and maintain data ingestion, scoring, and integration pipelines.
- Integrate Databricks environments with Snowflake, APIs, cloud services, and enterprise platforms.
- Automate environment provisioning, model space creation, and deployment processes.
- Support the full machine learning lifecycle, including deployment, monitoring, governance, and retraining workflows.
- Collaborate with Data Scientists, Engineers, Actuaries, and business stakeholders to deliver scalable AI solutions.
- Participate in architecture, data modeling, performance optimization, and platform modernization initiatives.
- Promote MLOps best practices, automation, reliability, and operational excellence.
Required Skills
- Bachelor's degree in Computer Science, Data Engineering, Information Systems, Statistics, or a related field.
- 5+ years of Data Engineering or software development experience.
- Strong Python development experience.
- Hands-on experience with Databricks.
- Experience building scalable data pipelines, integrations, and cloud-based solutions.
- Experience with Snowflake or other enterprise database platforms.
- Experience with DevOps, CI/CD, and Infrastructure-as-Code tools such as Jenkins, Terraform, or CloudFormation.
- Experience with AWS services including S3, Lambda, EMR, Step Functions, RDS, DynamoDB, or related technologies.
- Knowledge of data engineering best practices and MLOps concepts.
- Experience working with SQL and modern data platforms.
- Strong communication, collaboration, and problem-solving skills.
- Ability to manage multiple projects and work effectively in a fast-paced environment.
Preferred Skills
- Master's degree in Computer Science, Data Engineering, Information Systems, Statistics, or a related field.
- Experience supporting AI, machine learning, or MLOps platforms.
- Experience with Scala, Java, or C#.
- Familiarity with Domino Data Lab.
- Experience building shared platform services and reusable engineering frameworks.
- Knowledge of insurance, risk analysis, actuarial modeling, or related domains.
- Experience translating business requirements into scalable technical solutions.
- Strong analytical and solution-design capabilities.