Senior Software Engineer - Data Infrastructure

Job summary

Responsibilities

We are looking for infrastructure engineers with expertise in scaling open-source data infrastructure to join the Data & ML infra group. This role will work across the entire data lifecycle (collection, ingestion, storage, querying, retrieval) and work directly with all business units across the…

• Scale infrastructure to support all deployment types (cloud, hybrid, on-prem) and across regions
• Be involved in the end to end data lifecycle, from the external-facing product to the underlying platform and infrastructure for it
• Build features to tune processing pipeline for fast data ingestion and indexing depending on customer's needs and workloads
• Enable product workflows that expose performant query interfaces and offer easy-to-use integration hooks
• Develop and deploy high-quality software using modern tooling and frameworks, especially open-source technologies

What they're looking for

• A Bachelor's degree in Computer Science, Software Engineering, or equivalent
• 3+ years of professional experience
• Experience with large-scale open source data technologies (Spark, Kafka, Hudi, Flyte, etc.)
• Experience with containerization and other modern software development workflows
• Knowledge of the open source landscape with judgment on when to choose open source versus build in-house

Tech stack

• Spark
• Kafka
• Python
• C++
• Go
• Scala
• Kubernetes