Tasks:
- * Build structured and unstructured data ingestion
- * Collaborate with ML research teams
- * Design distributed data pipelines
- * Develop ETL/ELT workflows
- * Implement partitioning, sharding, and caching
- * Integrate CI/CD into data workflows
- * Maintain data lineage and reproducibility
- * Manage infrastructure as code
- * Monitor pipeline reliability
- * Optimize distributed data processing
- * Preprocess image, video, and 3D training assets
- * Validate and enrich training datasets
Perks/Benefits:
Skills/Tech stack required:
[Airflow] [Apache Flink] [Apache Spark] [AWS] [Azure] [CI/CD] [Dagster] [Dask] [Data Augmentation] [Data Lakes] [Data Lineage] [Data Processing] [Data Quality] [Data Quality Validation] [Dataset versioning] [Delta Lake] [English] [ETL] [GCP] [Kubernetes] [Monitoring and observability] [Object storage] [Parquet] [Prefect] [Python] [Quality validation] [Ray] [SQL] [Terraform] [Unstructured Data] [Unstructured Data Processing]
Educational requirements:
N/A
Role(s):
[Data Engineer] [Data Infrastructure Engineer] [Distributed Systems Engineer] [Engineer] [Infrastructure Engineer] [Platform] [Platform Engineer] [Senior Data Infrastructure Engineer] [Systems Engineer]