Software Engineer, Data Ingestion

Job summary

Responsibilities

Wayve is not a mature, fully-structured place with the playbook already written. Much of how we work is still being written, and if you join, you’ll help write it. That suits people who want real ownership more than people who need a settled structure from day one.

• Debug and unblock failing ingestion pipelines, stalled jobs and processing queues
• Investigate corrupt, malformed or unexpected data from a variety of sources.
• Optimise Spark jobs and batch-processing pipelines for throughput, reliability and compute efficiency.
• Design and implement more resilient pipelines so individual bad data segments do not block wider workflows.
• Improve monitoring, retries and failure recovery across multi-step ingestion workflows.

Tech stack

• Spark
• Python
• Airflow
• Databricks
• Scala
• Java
• SQL