Tasks:
- * Build failure recovery and checkpoint restart
- * Build observability and task management capabilities
- * Build video data platform
- * Collaborate with algorithm teams to accelerate training data production
- * Design data processing operators and DAG orchestration
- * Develop distributed execution for large-scale data pipelines
- * Implement task scheduling and resource management
- * Improve processing throughput and resource utilization
- * Optimize CPU, GPU, storage, and network data flow
- * Support reliable operation and troubleshooting of large-scale data jobs
Perks/Benefits:
Skills/Tech stack required:
[Apache Flink] [Apache Spark] [C++] [CUDA] [DAG scheduling] [Data Pipelines] [Distributed Systems] [Distributed task systems] [Fault Tolerance] [Go] [GPU Computing] [Kubernetes] [Performance optimization] [Python] [PyTorch] [Ray] [Resource Management] [Task Scheduling] [Task systems] [Triton] [Workflow Orchestration]
Educational requirements:
N/A
Role(s):
[Backend Engineer] [Data Platform] [Data Platform Engineer] [Distributed Systems Engineer] [Engineer] [Platform Engineer] [Systems Engineer]