Research Crawling Engineer
- Remote, United States
$160,000 to $250,000 USD / year
Applications open in a new tab.
Job details
- Pay
- $160,000 to $250,000 USD / year
- Posting language
- English
- Posted
Original job description
ABOUT THE ROLE
Join a data and infrastructure team building large-scale web data systems for AI research and model development. You will help deliver reliable, high-quality datasets by operating crawlers across complex and changing web environments.
WHAT YOU'LL DO
- Build and maintain large-scale crawlers across diverse domains, including social media, travel, and multilingual websites.
- Design high-throughput, fault-tolerant collection systems capable of processing millions to billions of URLs per day.
- Adapt crawling strategies to anti-bot measures, rate limits, and dynamic or JavaScript-heavy sites.
- Develop pipelines to clean, deduplicate, filter, and normalize web data at terabyte to petabyte scale.
- Build datasets for research and model training in collaboration with research teams.
- Monitor crawl coverage, performance, and data quality, and improve systems as web environments change.
- Optimize infrastructure for cost, latency, and reliability across cloud and bare-metal environments.
WHAT WE'RE LOOKING FOR
- Typically 4 or more years of relevant experience building web crawlers or large-scale data acquisition systems. Candidates with a PhD may qualify with 2 or more years.
- Strong experience with Go, Rust, Python, Java, or C++, and distributed systems or parallel processing.
- Experience operating data pipelines at terabyte scale and processing millions to billions of URLs per day.
- Deep understanding of HTTP, networking, and browser behavior, including experience handling anti-bot systems, rate limits, and JavaScript-heavy websites.
- Experience with headless browsers such as Playwright or Puppeteer, proxy systems, IP rotation, and request orchestration is valuable.
- Background in data quality evaluation, web data processing, NLP pipelines, or machine-learning dataset curation is a plus.
- Experience running data workloads on cloud or bare-metal infrastructure; a relevant technical degree is beneficial.
COMPENSATION & BENEFITS
Base salary: $160,000 to $250,000 USD annually. Visa sponsorship is not available.
LOCATION
Remote.