Staff Software Engineer, Ray Data

Job summary

Responsibilities

The Ray Data team develops and maintains Ray Data, building the underlying distributed data processing infrastructure that powers modern AI workloads. We are a team of engineers passionate about solving challenging problems in distributed systems, data processing, and performance at scale.

• Design, build, and improve the core systems that power Ray Data, with a focus on performance, scalability, and reliability.
• Design and optimize distributed execution across different stages of data pipelines in heterogeneous environments.
• Build data loading and processing solutions for production training and inference workloads.
• Solve challenging problems in distributed execution, scheduling, resource management, data partitioning, fault tolerance, and performance optimization.
• Make system-level architectural decisions and reason through tradeoffs in areas such as resource allocation, execution models, batch vs.

What they're looking for

• 6+ years of experience building production-grade software, infrastructure, or developer-facing systems, with strong Python engineering experience.
• 6+ years of experience personally owning core architectural decisions within a distributed data or compute engine, rather than primarily operating or using a platform someone…
• Deep experience with distributed systems internals, such as scheduling, fault tolerance, data partitioning, distributed execution, performance optimization, or database and query…
• A track record of reasoning through system-level tradeoffs and defending architectural decisions, such as batch vs. streaming, static vs.
• Passion for solving the unsolved problems in large-scale AI infrastructure and building systems that enable the next generation of AI applications.

Tech stack

• Ray
• Python
• Azure
• GCP