Software Engineer, Data Infrastructure - Research

Job summary

Responsibilities

We are looking for an engineer to design and implement the dataset infrastructure that powers OpenAI’s next-generation training stack. You will be responsible for building standardized dataset interfaces, scaling pipelines across thousands of GPUs, and proactively testing performance bottlenecks.

• Design and maintain standardized dataset APIs, including for multimodal (MM) data that cannot fit in memory.
• Build proactive testing and scale validation pipelines for dataset loading at GPU scale.
• Collaborate with teammates to integrate datasets seamlessly into training and inference pipelines, ensuring smooth adoption and a great user experience.
• Document and maintain dataset interfaces so they are discoverable, consistent, and easy for other teams to adopt.
• Establish safeguards and validation systems to ensure datasets remain reproducible and unchanged once standardized.