Principal Data Engineer, Data Quality & Datasets Pipelines

Job summary

Responsibilities

Wayve is not a mature, fully-structured place with the playbook already written. Much of how we work is still being written, and if you join, you’ll help write it. That suits people who want real ownership more than people who need a settled structure from day one.

• Implement and iterate filtering, retention and offload policies across hot, cold, archive and delete storage
• Build the data movement and pipelines that our pre-ingestion selection model depends on
• Decide where and how selection models run, balancing compute cost against data value
• Build early, low-compute decision points that avoid expensive standardisation and enrichment of low-value data
• Work with the storage and data infra teams, and monitor data quality to make sure our datasets reflect what model training needs

Tech stack

• Python
• Spark