Working As a Data Engineer Consultant, You Will
- Design and implement data ingestion pipelines connecting external APIs and business systems with the Databricks data platform.
- Develop reliable processes for retrieving and processing historical data, including mechanisms for rate limiting, checkpointing, retries and safe reprocessing.
- Build and maintain the Bronze layer of the data platform, covering data standardisation, typing, historical tracking and document storage.
- Develop processes for linking digital documents with corresponding business records while ensuring data consistency and idempotent processing.
- Use Databricks capabilities to classify documents and extract structured information from PDFs and images, including confidence scoring and workflows for manual review.
- Contribute to the development of the Silver layer using appropriate entity modelling, CDC and incremental processing patterns.
- Implement deterministic entity resolution and source-priority logic across multiple data sources.
- Apply data quality controls within the existing framework and ensure that quality results are properly stored, monitored and available for governance purposes.
- Support the development of analytical and machine-learning data products, including feature and metric stores and batch scoring workflows.
- Work with MLflow and related processes for model registration and promotion where required.
- Improve automation, monitoring and reliability across data pipelines and contribute to CI/CD processes.
- Collaborate with other technical stakeholders to deliver agreed project milestones within a defined consulting scope.
About Chabre IT Services
Chabre IT Services is a global professional IT services provider, building long-lasting relationships with Enterprises. We specialize in the delivery of tailor-made solutions, smart outsourcing, try&hire, and success fee services. We are a smart IT boutique with unique knowledge, which will deliver your ideas into reality.
About Our Client
Our client is a global technology services organisation delivering data and digital transformation projects for enterprise customers. The project focuses on building and enhancing a modern data platform based on Databricks, with a strong emphasis on data engineering, governance, automation and analytics.
The successful candidate will join a project involving multiple data sources, API integrations, document processing and layered data architecture. The environment combines traditional data engineering with modern Databricks AI capabilities and machine-learning workflows, providing an opportunity to work across several stages of the data lifecycle.
Qualifications
- Recent, hands-on experience delivering production solutions with Databricks.
- Strong knowledge of PySpark, Delta Lake and Unity Catalog.
- Practical experience with the Databricks medallion architecture and production data pipelines.
- Experience with Spark Declarative Pipelines / Delta Live Tables and Databricks Workflows.
- Very good Python and advanced SQL skills.
- Experience integrating REST APIs into data engineering solutions, including authentication, pagination, rate limiting and incremental or checkpointed data retrieval.
- Strong understanding of data modelling, including SCD Type 2 and current-state / CDC patterns.
- Experience designing data quality controls, reconciliation processes and mechanisms for handling invalid or quarantined data.
- Knowledge of Git-based development and CI/CD for data platforms, preferably with Azure DevOps or GitHub Actions.
- Experience implementing monitoring and observability for production data pipelines.
- Ability to work within a fixed project scope, estimate work, meet milestones and take ownership of agreed deliverables.
- Strong communication skills and the ability to work independently in a consulting environment.
- Excellent English and Polish skills (min. B2 level).
We offer
- Rate up to 240,00 PLN/h + VAT
- Remote work
- Subsidy for peripherals in the amount of 500,00zł
- Working tool (MacBook Pro or Lenovo Legion 5)
- Co-financing of courses related to the position
- Benefits: MultiSport, Medicover