Tasks:
- * Build data cleaning and annotation quality tools
- * Build distributed data processing systems
- * Clean, standardize, and model data
- * Collect and synchronize production data
- * Design data-closure pipelines
- * Develop data management platforms
- * Develop data services and visualizations
- * Implement data versioning, lineage, and metadata management
- * Optimize large-scale data pipeline performance
- * Process batch and streaming data
Perks/Benefits:
Skills/Tech stack required:
[Apache Iceberg] [Apache Kafka] [Apache Pulsar] [Data Lakes] [Data Lineage] [Data version control] [Data Warehousing] [Distributed Systems] [Docker] [ETL] [Go] [I/O] [I/O Optimization] [Java] [Kubernetes] [Lance] [Memory Management] [Metadata Management] [MongoDB] [MySQL] [Performance optimization] [PostgreSQL] [Python] [RabbitMQ] [Redis] [Stream processing] [Version control]
Educational requirements:
[Bachelor's Degree]
Role(s):
[Data Engineer] [Data Pipeline Engineer] [Data Platform] [Data Platform Engineer] [Engineer] [Pipeline Engineer] [Platform Engineer] [Senior Data Engineer]