Full-Stack Engineer, Agent Evaluation & Reliability
- San Francisco, CA, United States
$120,000 to $180,000 USD / year
Applications open in a new tab.
Job details
- Pay
- $120,000 to $180,000 USD / year
- Posting language
- English
- Posted
Original job description
ABOUT THE ROLE
Join an AI team building backend infrastructure for agent evaluation and continuous improvement. This backend-leaning full-stack role offers ownership of architecture and product decisions while turning complex systems into useful customer-facing features.
WHAT YOU'LL DO
- Build and maintain backend systems that power agent evaluation and continuous improvement.
- Design data models for evaluation datasets, experiment runs, agent traces, knowledge versions, findings, and customer-defined quality criteria.
- Develop and support enterprise integrations, including authentication, OAuth, webhooks, incremental syncs, backfills, rate limits, retries, and monitoring.
- Own features end to end, from technical design and implementation through deployment, observability, customer feedback, and iteration.
- Work directly with customers to diagnose system failures and turn individual needs into reusable product capabilities.
- Debug distributed workflows and improve system reliability.
WHAT WE'RE LOOKING FOR
- Two to four years of experience in a full-stack or backend engineering role building and shipping production systems.
- Production experience with TypeScript and Node.js or Bun, PostgreSQL schema design, and queues or worker systems.
- Hands-on machine learning engineering experience, including model development, and familiarity with agent infrastructure, LLM evaluation, tracing, or agent memory systems.
- Experience building backend systems and third-party integrations, and owning projects from design through deployment and iteration.
- Understanding of how evaluating nondeterministic agent systems differs from conventional unit testing.
- Experience in a fast-paced startup environment and a track record of building, such as through shipped AI projects or open-source contributions.
COMPENSATION & BENEFITS
Base salary is $120,000 to $180,000 USD annually. Visa sponsorship is available, including H-1B transfers, OPT, and O-1.
LOCATION
On-site in San Francisco, California, United States.