Research Engineer, Evals

ABOUT THE ROLE

Build evaluation systems for a web data extraction product, measuring whether it reliably turns varied websites and documents into clean, structured data for AI applications. You will shape the metrics, datasets, and engineering workflows that inform model and product decisions.

WHAT YOU'LL DO

• Design rigorous metrics for output quality across diverse websites, formats, and edge cases.
• Build scalable evaluation pipelines and harnesses to measure extraction quality.
• Generate and curate trustworthy ground-truth datasets for evaluation.
• Connect evaluation results to model and product decisions, and help teams act on the findings.

WHAT WE'RE LOOKING FOR

• At least 2 years of relevant evaluation experience, with 4 to 10 or more years of overall experience.
• Experience building evaluation systems for LLM extraction of structured fields from messy, real-world web pages or documents, including field-level accuracy measurement at scale.
• Strong Python skills and experience with machine learning, LLMs, data pipelines, web scraping, NLP, evaluation frameworks, and statistical analysis.
• Ability to design evaluation methods and build the systems that support them, including gold-label datasets and reference extractions.
• Statistical rigor in quality measurement, including sampling, significance, estimator bias and variance, error bounds, and agreement between automated judges and human reviewers.
• Experience connecting evaluation results to practical model or product improvements.

COMPENSATION & BENEFITS

Base salary: $250,000 to $290,000 USD annually. Visa sponsorship is available.

LOCATION

Hybrid, based in San Francisco, California. The role requires working on-site five days per week in San Francisco or Toronto, including relocation before the start date if needed.