Title: AI/ML Quality Engineer
Location: Cupertino, CA (hybrid: Tues-Thurs in office)
Duration: 12+ month contract
Compensation: $66 - 76.00/hr. W2
Work Requirements: U.S. Citizen, GC Holders, or Authorized to Work in the U.S.
ABOUT THIS FEATURED OPPORTUNITY
The AI/ML Quality Engineer will join the Channel Sales and Operations team to help ensure the reliability and quality of the AI/ML-powered B2B chatbot and the foundational platforms supporting it. This role goes beyond traditional testing, with a focus on evaluating LLM outputs, improving model performance through structured feedback loops, and validating end-to-end system functionality, including cloud infrastructure and data pipelines.
THE OPPORTUNITY FOR YOU
- Design and execute manual and automated test cases for GenAI platforms and chatbot systems.
- Build and maintain Python-based test automation frameworks for backend services and ML pipeline validation.
- Develop and execute ML evaluation strategies, leveraging LLM-as-a-Judge methodologies and models such as Gemini and Claude to assess response accuracy, relevance, factuality, and overall quality.
- Collaborate with localization teams to conduct human evaluations, ensuring model outputs meet quality standards across different languages and markets.
- Understand and support data annotation workflows, incorporating human feedback and evaluation results into continuous model improvement and fine-tuning.
- Utilize RAGAS or similar frameworks to evaluate LLM outputs and identify opportunities for optimization.
- Conduct end-to-end testing from data ingestion to user-facing output.
- Validate system stability across GCP cloud components, including compute, storage, networking, and containers.
- Collaborate with DevOps and ML engineers to triage bugs, improve model performance, and ensure system reliability.
- Ensure test coverage spans multiple deployment environments, including Kubernetes clusters and cloud VMs.
KEY SUCCESS FACTORS
- 3+ years of QA Automation Engineering experience
- Experience with Playwright for automated application testing, including sign-in and SSO authentication flows
- Ability to design tests that validate output accuracy and system behavior across different user flows
- Experience with Python API testing using requests, Pytest, and integration into CI/CD and cloud environments.
- Experience with ML/LLM Evaluation
- Hands-on experience implementing LLM-as-a-Judge evaluation methodologies using models such as Gemini, Claude, or similar LLMs
- Experience designing evaluation criteria and assessing LLM outputs for accuracy, relevance, factuality, and quality
- Understanding of human evaluation and data annotation processes, including leveraging feedback loops to improve and fine-tune model performance
- Experience with Cloud Platforms (GCP preferred)
- Understanding of modern cloud architecture and how distributed components interconnect in production environments
NICE TO HAVES
- Experience with multilingual LLM evaluation, localization testing, and human-in-the-loop evaluation workflows
About INSPYR Solutions
Technology is our focus and quality is our commitment. As a national expert in delivering flexible technology and talent solutions, we strategically align industry and technical expertise with our clients' business objectives and cultural needs. Our solutions are tailored to each client and include a wide variety of professional services, project, and talent solutions. By always striving for excellence and focusing on the human aspect of our business, we work seamlessly with our talent and clients to match the right solutions to the right opportunities. Learn more about us at inspyrsolutions.com.INSPYR Solutions provides Equal Employment Opportunities (EEO) to all employees and applicants for employment without regard to race, color, religion, sex, national origin, age, disability, genetics, or any other protected status. INSPYR Solutions complies with all applicable laws governing nondiscrimination in employment in every location in which the company has facilities. Applicants requiring reasonable accommodation during the application or interview process should contact HR@inspyrsolutions.com for assistance.
Information collected and processed through your application with INSPYR Solutions (including any job applications you choose to submit) is subject to INSPYR Solutions' Privacy Policy and INSPYR Solutions' AI and Automated Employment Decision Tool Policy: https://www.inspyrsolutions.com/policies/. By submitting an application, you are consenting to being contacted by INSPYR Solutions through phone, email, or text.
26-160430
