Tasks:
- * Build and maintain batch data pipelines
- * Develop AI-driven data quality diagnosis and asset discovery
- * Develop dbt data models and unified metrics
- * Evolve Iceberg lake storage
- * Improve CI, documentation, data quality, and alerting
- * Optimize Spark, Glue, and ClickHouse performance and cost
Perks/Benefits:
- + AI-driven data engineering tools
- + Collaborative technical environment
- + Fast feedback
- + High ownership and impact
- + Modern data stack experience
Skills/Tech stack required:
[Amazon S3] [Apache Iceberg] [Apache Spark] [AWS Glue] [Batch data] [Batch data pipelines] [CI/CD] [ClickHouse] [Cost Optimization] [Data backfills] [Data Lake] [Data Lake Architecture] [Data Modeling] [Data Observability] [Data Pipelines] [Data Quality] [Data Quality Testing] [DBT] [Go] [Idempotency] [Incremental processing] [Java] [Python] [Quality testing] [Query Optimization] [SQL]
Educational requirements:
N/A
Role(s):
[Big Data Engineer] [Data Engineer] [Engineer]