Research Engineer, Web Crawling

Job summary

Responsibilities

We're hiring a Software Engineer to build and own our web-crawling systems, from distributed collection at internet scale through filtering, deduplication, and deciding what data we keep.

• Design and scale the web crawler and ingestion infrastructure that sources Inkling's pretraining data
• Build pipelines for large-scale extraction, deduplication, and data quality filtering
• Build specialized crawlers for high-value or hard-to-reach data sources
• Work with the pretraining team to understand how changes in crawled data affect model performance
• Improve the reliability and efficiency of crawling and ingestion infrastructure at petabyte scale

What they're looking for

• 8+ years designing, building, and scaling web crawlers, scrapers, or large-scale distributed data-acquisition systems
• A track record of owning crawler or data-acquisition infrastructure at internet scale
• Strong software engineering skills in a language such as Python, Go, or Rust, with real experience in distributed systems
• Working knowledge of the practical and legal considerations of large-scale web data collection (robots.txt, rate limiting, licensing)
• Experience applying machine learning to crawl selection, extraction, or data quality classification at internet scale

Tech stack

• Python
• Go
• Rust