Research Engineer, Evals

ABOUT THE ROLE

Build evaluation systems for a web data extraction product that turns pages into structured, LLM-ready information. You will shape how quality is measured, create the infrastructure to measure it at scale, and connect evaluation results to model and product decisions.

WHAT YOU'LL DO

• Design metrics for extraction quality across diverse websites, formats, and edge cases.
• Build scalable pipelines and evaluation harnesses to measure output quality rigorously.
• Generate and curate ground-truth datasets for trustworthy evaluation.
• Close the feedback loop by translating evaluation results into model and product improvements.
• Make quality results clear and actionable for the broader team.

WHAT WE'RE LOOKING FOR

• Four or more years of relevant engineering experience, including at least two years focused on evaluation work.
• Experience building evaluations for LLM extraction of unstructured data, measuring whether fields are correctly extracted from messy, real-world pages or documents.
• Experience building or evaluating large-scale web or document extraction systems and developing evaluation metrics and systems from the ground up.
• Strong Python skills and experience with machine learning, LLMs, data pipelines, web scraping, NLP, and evaluation frameworks.
• Statistical rigor in metric design and interpretation, including sampling, significance, estimator bias and variance, error bounds, and judge-to-human agreement.
• Experience creating ground-truth datasets and using evaluation findings to inform real model or product decisions.
• A combination of research methodology and hands-on engineering, plus comfort working through ambiguous problems.

COMPENSATION & BENEFITS

Base salary: $250,000 to $290,000 USD annually. Visa sponsorship is available.

LOCATION

Hybrid, based in San Francisco, California, or Toronto, Canada. The role requires working on-site five days a week.