AI Test Engineer x 2

The final salary or hourly wage, as applicable, paid to each candidate/applicant for this position is ultimately dependent on a variety of factors, including, but not limited to, the candidate’s/applicant’s qualifications, skills, and level of experience as well as the geographical location of the position.

Applicants must be legally authorized to work in the United States. Sponsorship not available.

Our client is seeking AI Test Engineer x 2 in Vienna, VA/Hybrid.

ROLE DESCRIPTION

We are seeking a highly motivated AI Test Engineer to help establish and scale our AI testing and evaluation capability. This role will work closely with AI engineers, platform teams, and product stakeholders to build automated testing frameworks, evaluate AI-driven solutions, and ensure the quality, reliability, and performance of AI-enabled products. The AI Test Engineer will be hands-on contributors responsible for creating automated tests, executing evaluations against AI systems, analyzing model outputs, and helping define best practices for AI quality assurance. This role requires a strong combination of software testing expertise, automation engineering, data analysis, and an understanding of modern AI and generative AI technologies.

• Help design, develop, and maintain AI testing and evaluation frameworks for generative AI, predictive AI, and AI-powered developer experience (DevEx) solutions.
• Create and execute automated test suites that validate AI model behavior, output quality, accuracy, reliability, and performance.
• Develop automated evaluation pipelines that integrate with CI/CD workflows and support continuous AI quality validation.
• Perform functional, regression, performance, and scenario-based testing of AI-enabled applications and services.
• Analyze AI-generated outputs to identify defects, inconsistencies, hallucinations, bias, performance issues, and other quality concerns.
• Build and maintain test datasets, benchmark scenarios, prompt libraries, and evaluation artifacts used to assess AI systems.
• Collaborate with software engineers, AI engineers, and platform teams to improve model quality and testing coverage.
• Define, measure, and report AI quality metrics such as accuracy, relevance, latency, consistency, robustness, and user satisfaction.
• Support root-cause analysis, troubleshooting, and validation activities for AI-related defects and incidents.
• Document test strategies, test results, evaluation methodologies, and recommendations for technical and non-technical stakeholders.
• Contribute to AI governance and responsible AI efforts by supporting evaluation activities related to reliability, fairness, transparency, and safety.
• Support delivery of strategic DevEx initiatives and AI-enabled platform capabilities across the software development lifecycle.

SKILLS & REQUIREMENTS

• Bachelor's degree in Computer Science, Information Systems, Software Engineering, Data Science, or a related technical field.
• 3+ years of experience in software testing, quality engineering, test automation, or AI/ML testing.
• Experience designing and executing automated test strategies for complex software applications.
• Strong programming and scripting skills in Python.
• Experience building automated test frameworks and test pipelines.
• Experience working with APIs, automated integration testing, and CI/CD environments.
• Knowledge of statistical analysis and evaluation techniques used to validate system performance.
• Experience using source control systems and collaborative development practices.
• Strong analytical, problem-solving, and troubleshooting skills.
• Excellent written and verbal communication skills.
• Experience working with Azure AI, GitHub Copilot, OpenAI, Hugging Face, LangChain, or comparable AI platforms.
• Familiarity with prompt engineering, benchmark dataset creation, and AI output validation techniques.
• Understanding of Responsible AI principles and AI governance frameworks.
• Experience supporting developer platforms, DevEx initiatives, or software engineering productivity tools.
• 3+ years in QA, Test Automation, SDET, Quality Engineering, or AI/ML Testing
• Strong Python programming experience
• Experience building automated test frameworks from scratch
• API testing and integration testing experience
• CI/CD pipeline experience (Jenkins, GitHub Actions, Azure DevOps, GitLab, etc.)
• Experience creating automated test strategies for complex applications
• Strong defect analysis, root cause investigation, and data validation skills
• Understanding of AI/ML, Generative AI, and LLMs
• Experience with regression, functional, performance, and end-to-end testing
• Agile environment experience
• Excellent analytical and troubleshooting skills
• Testing GenAI, LLM, Chatbot, Copilot, or RAG applications
• DeepEval, Ragas, LangSmith, or similar AI evaluation frameworks
• Prompt engineering and AI output validation
• Experience identifying hallucinations, bias, accuracy, and relevance issues
• AI observability, monitoring, and tracing
• Azure AI, OpenAI, Hugging Face, LangChain, GitHub Copilot
• Performance and reliability testing
• Responsible AI / AI Governance knowledge

Benefits/Other Compensation

Medical, Dental, Life Insurance, 401K.

Why Hays?

You will be working with a professional recruiter who has intimate knowledge of the industry and market trends. Your Hays recruiter will lead you through a thorough screening process in order to understand your skills, experience, needs, and drivers. You will also get support on resume writing, interview tips, and career planning, so when there’s a position you really want, you’re fully prepared to get it.

Nervous about an upcoming interview? Unsure how to write a new resume?

Visit the Hays Career Advice section to learn top tips to help you stand out from the crowd when job hunting.

Hays is committed to building a thriving culture of diversity that embraces people with different backgrounds, perspectives, and experiences. We believe that the more inclusive we are, the better we serve our candidates, clients, and employees. We are an equal employment opportunity employer, and we comply with all applicable laws prohibiting discrimination based on race, color, creed, sex (including pregnancy, sexual orientation, or gender identity), age, national origin or ancestry, physical or mental disability, veteran status, marital status, genetic information, HIV-positive status, as well as any other characteristic protected by federal, state, or local law. One of Hays’ guiding principles is ‘do the right thing’. We also believe that actions speak louder than words. In that regard, we train our staff on ensuring inclusivity throughout the entire recruitment process and counsel our clients on these principles. If you have any questions about Hays or any of our processes, please contact us.

In accordance with applicable federal, state, and local law protecting qualified individuals with known disabilities, Hays will attempt to reasonably accommodate those individuals unless doing so would create an undue hardship on the company. Any qualified applicant or consultant with a disability who requires an accommodation in order to perform the essential functions of the job should call or text 813.336.5570.

Drug testing may be required; please contact a recruiter for more information.

#1186884 - Kellee Jacobsen