Machine Learning Engineer
$200000 - $300000 per year | Austin, TX | On-site | Permanent
Build the Evaluation Engine Behind Production AI Voice Agents | $200k-$300k
A bit about us:
We are a fast-growing, AI-powered communications platform transforming how businesses connect with their customers.
For over a decade, we've believed voice builds the strongest business relationships. Our platform brings voice, SMS, WhatsApp, and more than 250 integrations together in one seamless workspace, helping businesses drive revenue, resolve customer issues faster, and build lasting relationships. Our AI-powered voice agents handle routine interactions, allowing teams to focus on the conversations that matter most.
Today, our platform is trusted by more than 24,000 businesses worldwide.
We’re 600+ people from 40+ nationalities across Paris, London, Madrid, Berlin, New York, San Francisco, Seattle, Sydney, and Mexico City. We’re ambitious, product-driven, and execution-focused - built for visible impact, fast decisions, and real growth.
Come join us!
Aircall's AI suite includes an AI Voice Agent and AI Messaging Agent that autonomously handle calls, WhatsApp, and SMS, plus AI Assist, which delivers real-time coaching, call summaries, and CRM automation for sales and support teams. We are looking for someone that can build out the evaluation foundation across all of these products and other agentic products. You'll work on voice models, agent capability evals, benchmark design, LLM-as-judge systems, failure analysis, and the infrastructure that ties it together by establishing shared metrics, test sets, and tooling to measure accuracy, resolution quality, and safety consistently across products. You will set up repeatable pipelines for regression testing and benchmarking as models and features evolve so teams can ship confidently without re-inventing evaluation methodology for each product.
Base salary range:
$181,000 - $250,000 USD
Why join us?
🚀 Key moment to join Aircall in terms of growth and opportunities
💆♀️ Our people matter, work-life balance is important at Aircall
📚 Fast-learning environment, entrepreneurial and strong team spirit
🌍 45+ Nationalities: cosmopolite & multi-cultural mindset
💶 Competitive salary package & benefits
DE&I Statement:
At Aircall, we believe diversity, equity and inclusion – irrespective of origins, identity, background and orientations – are core to our journey.
We pride ourselves on promoting active inclusion within our business to foster a strong sense of belonging for all. We’re working to create a place filled with diverse people who can enrich and learn from one another. We’re committed to ensuring that everyone not only has a seat at the table but is valued and respected at it by providing equal opportunities to develop and thrive.
We will constantly challenge ourselves to make sure that we live up to our ambitions around diversity, equity and inclusion, and keep this conversation open. Above all else, we understand and acknowledge that we have work to do and much to learn.
Why join us?
🚀 An exciting time to join a fast-growing company with significant opportunities for growth and career development.
💆♀️ A people-first culture that prioritizes employee well-being and work-life balance.
📚 A fast-paced, entrepreneurial environment with continuous learning and a strong team-oriented culture.
🌍 A diverse, international team representing more than 45 nationalities and perspectives.
💶 Competitive compensation package and comprehensive benefits.
Job Details
Design and document comprehensive evaluation frameworks for AI agents across voice, chat, and messaging platforms.
Train and fine-tune voice models (TTS, ASR, and speech-to-speech) using production and synthetic data. Continuously iterate on model architecture, data composition, and training strategies to improve accuracy, naturalness, and latency.
Evaluate AI-generated solutions across training pipelines, experimentation frameworks, debugging processes, and optimization strategies.
Analyze system design decisions, identifying strengths, weaknesses, and potential failure points.
Develop annotation guidelines and workflows for human-labeled evaluation data. Calibrate LLM-as-judge systems against human reviewers to maintain reliable automated evaluations over time.
Build and maintain live quality monitoring for deployed AI agents, tracking accuracy, resolution rates, and safety signals in production while identifying model or data drift before it impacts customers.
Own the metric standards for all published AI performance metrics, including definitions, populations, data granularity, rollups, and validity windows.
Establish release gates and offline regression testing suites that every AI feature must pass before changes to prompts, models, or configurations are deployed. Measure reliability across repeated trials rather than relying solely on average pass rates.
Develop voice-specific evaluation frameworks using simulated callers across accents, languages, background noise, interruptions, DTMF inputs, and tool failures, with latency and speech recognition accuracy as key quality metrics.
Minimum Qualifications
Bachelor's degree in Computer Science, Machine Learning, Statistics, or a related field.
8+ years of overall professional experience, including at least 3 years in ML Engineering or Applied Machine Learning.
Strong experience evaluating supervised and unsupervised machine learning models, LLMs, and deep learning systems.
Hands-on experience with failure analysis and LLM evaluation.
Experience building automated evaluation systems.
Strong communication skills, with the ability to explain complex technical concepts to both technical and non-technical audiences.
Practical experience training or fine-tuning voice and speech models (TTS, ASR, or speech-to-speech), including building data pipelines and running experiments.
Preferred Qualifications
Master's or PhD in Computer Science, Machine Learning, Statistics, or a related field.
Experience evaluating LLMs or agentic systems, including LLM-as-a-judge and RAG evaluation.
Experience with synthetic data generation and prompt engineering.
Experience training or fine-tuning voice models at scale, with familiarity in synthetic data generation, model distillation, and low-latency inference optimization for production AI applications.
Jobot is an Equal Opportunity Employer. We provide an inclusive work environment that celebrates diversity and all qualified candidates receive consideration for employment without regard to race, color, sex, sexual orientation, gender identity, religion, national origin, age (40 and over), disability, military status, genetic information or any other basis protected by applicable federal, state, or local laws. Jobot also prohibits harassment of applicants or employees based on any of these protected categories. It is Jobot’s policy to comply with all applicable federal, state and local laws respecting consideration of unemployment status in making hiring decisions.
Sometimes Jobot is required to perform background checks with your authorization. Jobot will consider qualified candidates with criminal histories in a manner consistent with any applicable federal, state, or local law regarding criminal backgrounds, including but not limited to the Los Angeles Fair Chance Initiative for Hiring and the San Francisco Fair Chance Ordinance.
Information collected and processed as part of your Jobot candidate profile, and any job applications, resumes, or other information you choose to submit is subject to Jobot's Privacy Policy, as well as the Jobot California Worker Privacy Notice and Jobot Notice Regarding Automated Employment Decision Tools which are available at jobot.com/legal.
By applying for this job, you agree to receive calls, AI-generated calls, text messages, or emails from Jobot, and/or its agents and contracted partners. Frequency varies for text messages. Message and data rates may apply. Carriers are not liable for delayed or undelivered messages. You can reply STOP to cancel and HELP for help. You can access our privacy policy here: jobot.com/privacy-policy
