Research Engineer, Evals

ABOUT THE ROLE

Build evaluation systems for a web data extraction product, defining how output quality is measured across diverse websites, formats, and edge cases. You will combine research and engineering to turn evaluation results into actionable model and product improvements.

WHAT YOU'LL DO

• Design rigorous metrics for evaluating structured extraction quality across varied web content.
• Build scalable evaluation pipelines, testing harnesses, and supporting data systems.
• Generate and curate ground-truth datasets for trustworthy measurement.
• Connect evaluation findings to model and product decisions, closing the quality feedback loop.
• Make evaluation results clear and useful to the broader team.

WHAT WE'RE LOOKING FOR

• Typically 4 to 10 or more years of relevant experience, including at least 2 years working on evaluations.
• Strong Python skills and experience with machine learning, LLMs, data pipelines, web scraping, NLP, evaluation frameworks, and statistical analysis.
• Experience building evaluation systems for LLM-based structured extraction from messy, real-world web pages or documents, including field-level accuracy measurement at scale.
• Experience designing evaluation methods and metrics, building ground-truth datasets, and applying statistical rigor, including sampling, estimator bias and variance, error bounds, and judge-to-human agreement.
• Ability to combine research methodology with engineering implementation and use evaluation results to guide real model or product decisions.

COMPENSATION & BENEFITS

Base salary: $250,000 to $290,000 USD annually. Visa sponsorship is available.

LOCATION

Hybrid, based in San Francisco, California. The role requires working on-site five days a week in San Francisco or Toronto, with relocation before starting if needed.