Technical Lead Manager, ML Dataset Platform

Job summary

Responsibilities

With offices in Toronto, San Francisco, Dallas, and Pittsburgh, Waabi is growing quickly and looking for diverse, innovative and collaborative candidates who want to impact the world in a positive way. To learn more visit: www.waabi.ai

• Lead and grow the ML Dataset Platform team as both its technical lead and people manager: set the technical direction, define the roadmap, hire, and mentor engineers.
• Define the APIs, tools and storage formats for Waabi's ML datasets, balancing maximal reuse of data and tooling against the specific needs of each training workload.
• Build the tooling to manage datasets at scale, including versioning and efficient incremental updates without full rebuilds.
• Design the APIs that let teams express training-time transformations along the continuum between fully reproducible ahead-of-time ETL and dynamic transformation within data…
• Own the common dataloading tools used across workloads, including observability into bottlenecks and the throughput of training jobs.

What they're looking for

• Deep ML dataset experience: You have designed and built large-scale data infrastructure for ML training (storage formats, dataset versioning, ETL, dataloading) and understand the…
• Strong software engineering and implementation: You have very strong Python skills and are comfortable in PyTorch training loops; you have strong software-engineering…
• Proven technical leadership: You have led a platform or infrastructure effort used by multiple teams, defined its APIs, and driven adoption across an organization.
• People leadership: You have managed or tech-led a team of engineers, including hiring and mentoring, and can balance hands-on technical contribution with roadmap ownership and…
• Team player: You communicate clearly with researchers and engineers, build consensus across teams with different needs, and deliver in a fast-paced environment.

Tech stack

• Python
• PyTorch
• Spark
• Ray