Director of Engineering AI Evaluations & Experimentation

To get the best candidate experience, please consider applying for a maximum of 3 roles within 12 months to ensure you are not duplicating efforts. Job CategorySoftware Engineering Job Details About SalesforceSalesforce is the #1 AI CRM, where humans with agents drive customer success together. Here, ambition meets action. Tech meets trust. And innovation isn't a buzzword - it's a way of life. The world of work as we know it is changing and we're looking for Trailblazers who are passionate about bettering business and the world through AI, driving innovation, and keeping Salesforce's core values at the heart of it all. Ready to level-up your career at the company leading workforce transformation in the agentic era? You're in the right place! Agentforce is the future of AI, and you are the future of Salesforce. Overview of the RoleWe are seeking a Director of Engineering to lead our AI Agent Evaluation and Experimentation Platform team. In this role, you'll own the end-to-end evaluation and experimentation lifecycle for both agentic systems and traditional ML models. You'll be part of Salesforce's AI Engineering organization, working at the forefront of the agentic era as we build Agentforce-the future of AI-powered CRM. Your team will be responsible for building the critical infrastructure that ensures we ship high-quality, safe, and performant AI systems with confidence. Responsibilities * Define and execute the technical vision for evaluation and experimentation across AI agents and traditional ML models * Own offline evaluation, regression testing, scenario-based simulations, and multi-turn agent testing infrastructure * Build automated evaluation systems including LLM-as-Judge, rule-based scoring, and hybrid evaluation approaches * Design and operate online evaluation, observability, and continuous performance monitoring for agent behavior * Lead development of self-service evaluation and experimentation tooling for agent workflows, tool use, memory, and planning * Support experimentation for both real-time agents and batch or online traditional ML models * Integrate evaluation and experimentation pipelines into CI/CD workflows and release quality gates * Drive adoption of evaluation and experimentation best practices across engineering and AI teams * Set technical direction, review designs, and raise the bar on engineering quality * Lead and develop a senior engineering team, fostering innovation and excellence * Partner with AI research, product, security, and Responsible AI teams on evaluation and experimentation strategyThrough this role, you'll gain deep experience building large-scale AI infrastructure, shape the future of how Salesforce evaluates and ships AI systems, and make a direct impact on the quality and reliability of AI products used by millions of customers worldwide. Required Qualifications * A related technical degree required * 10+ years of engineering experience, with 5+ years leading AI/ML teams * Proven ability to lead senior engineers and engineering managers * Experience building and operating experimentation platforms for AI systems or ML products * Strong understanding of LLM-based agentic architectures and traditional ML systems * Experience designing experimentation frameworks for online and offline ML workflows * Experience building evaluation systems for models and agents, including offline tests, regression suites, online monitoring, and LLM-as-a-Judge-style approaches * Strong background in AI agents and LLM systems, including tool use, multi-step workflows, RAG, prompt and policy management, and common agent failure modes * Experience evaluating agent behavior across multi-step workflows and tool-using systems * Hands-on experience designing evaluation frameworks for AI systems * Experience with offline benchmarking, regression testing, and scenario-based evaluation * Experience wi