Research Crawling Engineer
- Remote, United States
$160,000 to $250,000 USD / year
Applications open in a new tab.
Job details
- Pay
- $160,000 to $250,000 USD / year
- Posting language
- English
- Posted
Original job description
ABOUT THE ROLE
Build large-scale web crawling and data acquisition systems that support AI research and model development. You will work closely with research teams to deliver high-quality web datasets for training and inference.
WHAT YOU'LL DO
- Build and operate crawlers across diverse domains, including social media, travel, and multilingual websites.
- Design high-throughput, fault-tolerant collection systems that process millions to billions of URLs per day.
- Adapt crawling approaches to anti-bot systems, rate limits, and dynamic or JavaScript-heavy websites.
- Develop large-scale pipelines for cleaning, deduplication, filtering, and normalization of web data.
- Collaborate with research teams to shape datasets for research and model training.
- Monitor crawl performance, coverage, and data quality, and respond to changes in web environments.
- Optimize infrastructure for cost, latency, and reliability across cloud and bare-metal environments.
WHAT WE'RE LOOKING FOR
- Typically 4 or more years of relevant experience building web crawlers or large-scale data acquisition systems. Candidates with a PhD may qualify with 2 or more years; hands-on crawler and pipeline experience is essential.
- Experience operating distributed data pipelines at terabyte or larger scale, ideally processing millions to billions of URLs per day.
- Strong proficiency in Go, Rust, Python, Java, or C++ and a solid understanding of distributed systems and parallel processing.
- Deep knowledge of HTTP, networking, and browser behavior, including experience handling anti-bot systems, rate limits, and dynamic sites.
- Experience with headless browsers such as Playwright or Puppeteer, Chrome DevTools Protocol, proxy systems, or IP rotation.
- Experience with web data quality, benchmarking, cleaning, deduplication, and normalization; NLP pipelines or ML dataset curation are a plus.
- Experience running crawling or data workloads on cloud platforms or bare-metal infrastructure. A degree in computer science, engineering, or a related field is a plus.
COMPENSATION & BENEFITS
Base salary is $160,000 to $250,000 annually. Visa sponsorship is not available.
LOCATION
Remote.
More about this job
Skills
- Research
- Data acquisition systems
- JavaScript
- Cleaning
- Go
- Rust
- Python
- Oracle Java
- C++
- Natural language processing
- ML
- Computer science