Research Crawling Engineer

ABOUT THE ROLE

Join a data infrastructure team building large-scale web data collection systems for AI research and model development. This role focuses on reliable, high-volume crawling and dataset preparation across diverse and changing web environments.

WHAT YOU'LL DO

• Build and maintain web crawlers across social, travel, and multilingual websites.
• Design high-throughput, fault-tolerant systems capable of collecting millions to billions of URLs per day.
• Address anti-bot systems, rate limits, and dynamic websites using browser-based and other crawling techniques.
• Develop pipelines to clean, deduplicate, filter, and normalize web data at terabyte to petabyte scale.
• Collaborate with research teams to create datasets suited to model training and research needs.
• Monitor crawl coverage, performance, and data quality, adapting systems as websites change.
• Optimize crawling infrastructure for cost, latency, and reliability across cloud or bare-metal environments.

WHAT WE'RE LOOKING FOR

• At least 2 years of experience building web crawlers or large-scale data pipelines; typically 4 or more years, with 2 or more years considered for candidates with a PhD.
• Experience operating data pipelines at terabyte scale or larger, including high-volume crawling or data acquisition.
• Strong programming skills in Go, Rust, Python, Java, or C++.
• Deep knowledge of HTTP, networking, browser behavior, distributed systems, and parallel processing.
• Hands-on experience navigating anti-bot systems, rate limits, and JavaScript-heavy websites.
• Experience with headless browsers such as Playwright or Puppeteer, proxy systems, and IP rotation.
• Background in data quality evaluation, web data processing, or cloud and bare-metal infrastructure.
• Experience with NLP pipelines or machine learning dataset curation is a plus.

COMPENSATION & BENEFITS

Annual salary range: $160,000 to $250,000 USD. Visa sponsorship is not available.

LOCATION

Remote.