Tasks:
- * Build elastic cache architectures using GPU host memory, NVMe SSDs, and parallel file systems
- * Design and develop distributed multi-tier cache acceleration systems
- * Design metadata management systems
- * Develop heat-based tiering and scheduling strategies
- * Develop large-scale distributed file cache systems for AI training clusters
- * Evaluate and deploy GPU Direct Storage, NVMe-oF, RDMA, and burst buffer technologies
- * Implement efficient metadata path lookup and caching
- * Implement intelligent data placement and lifecycle management
- * Implement TTL and LRU eviction policies
- * Optimize data prefetching
- * Optimize end to end data paths
- * Reduce GPU stalls caused by storage I/O
Perks/Benefits:
Skills/Tech stack required:
[Burst Buffer] [C#] [C++] [Consistency] [Data lifecycle] [Data Lifecycle Management] [Data loading] [Data placement] [Data prefetching] [Direct Storage] [Distributed Caching] [Distributed file systems] [Distributed Storage] [Distributed Storage Systems] [Distributed Systems] [Fault Tolerance] [File systems] [Golang] [GPU Direct] [GPU direct storage] [GPU host memory] [Lifecycle Management] [Linux Kernel] [Linux kernel storage stack] [LRU] [Metadata Management] [Multi-tier] [Multi-tier caching] [NVMe-oF] [NVME SSD] [Parallel File Systems] [PyTorch Dataloader] [PyTorch Dataset] [RDMA] [Replication] [Rust] [Sharding] [Storage stack] [Storage Systems] [Streaming Data] [Streaming data loading] [TTL] [User space] [User-space file systems]
Educational requirements:
[Bachelor's Degree]
Role(s):
[Distributed Systems Software Engineer] [Engineer] [Senior Storage Engineer] [Software Engineer] [Storage Engineer] [Systems Software Engineer]