Senior Software Engineer, Data and Orchestration

Job summary

Responsibilities

You'll help evolve Wayve's data pipeline platform. That means revamping our existing Harness system and building a new DAG-based orchestration system for our fleet's next-generation data volumes.

• Designing, building and evolving scalable data pipeline and orchestration infrastructure for Wayve's next-generation autonomous driving data platform
• Revamping the existing Harness system and developing a new DAG-based workflow platform
• Building tooling and libraries that make data ingestion, processing, inference and evaluation workflows simple, reliable and consistent across teams
• Making sure the platform runs hundreds of parallel pipeline steps efficiently, with correct outputs, no conflicts and effective resource prioritisation
• Supporting batch inference and performance optimisation workflows, so teams can run evaluations in a cost-effective and scalable way

Tech stack

• Python
• Kafka
• PostgreSQL
• Ray
• Spark