Senior AI Tools Testing Engineer

Job Title: Senior / Lead AI Tools & Testing Engineer (Agentic AI & LLM Evaluation) Location: Dallas, TX (Hybrid) Duration: Long‑Term Experience:10 + Years

We seek a senior/lead AI engineer with hands‑on experience building, orchestrating, monitoring, and evaluating production‑grade Generative AI and Agentic AI applications. The ideal candidate is expert in Amazon Bedrock, AgentCore, LangGraph, LangChain, RAG, LLM observability, and evaluation frameworks. This role focuses on creating reliable, scalable, and production‑ready AI applications, emphasizing agent orchestration, AI testing, LLM evaluation, and continuous quality monitoring.

The candidate will collaborate with engineering, QA, and platform teams to establish best practices for building, testing, monitoring, and improving AI‑powered applications.

Key Responsibilities

1. Agentic AI & GenAI Engineering

• Design, develop, and deploy production‑grade agentic and generative AI applications.
• Build and orchestrate AI agents using LangGraph, LangChain, Amazon Bedrock, and AgentCore.
• Implement multi‑agent workflows, tool calling, state management, memory, and orchestration.
• Create scalable LLM‑powered applications and reusable AI components.

2. RAG & LLM Application Development

• Design and implement Retrieval‑Augmented Generation (RAG) solutions.
• Develop pipelines for document ingestion, chunking, embedding, retrieval, reranking, and context management.
• Optimize retrieval accuracy, response quality, latency, and cost.
• Apply grounding, hallucination reduction, and context‑relevance techniques.

3. LLM Evaluation & AI Testing

• Build evaluation frameworks for LLM‑based applications and agentic workflows.
• Automate evaluation with DeepEval and other LLM evaluation tools.
• Use LLM‑as‑a‑Judge methods to assess response quality, relevance, accuracy, faithfulness, and reasoning.
• Create evaluation datasets, test scenarios, benchmarks, and regression‑testing pipelines.
• Evaluate agent behavior, tool execution, workflow accuracy, and response consistency.
• Identify failure patterns and continuously improve AI application quality.

4. AI Observability & Monitoring

• Implement tracing, monitoring, and observability with OpenTelemetry, LangSmith, and Langfuse.
• Track LLM requests, agent execution flows, tool calls, latency, token consumption, errors, and performance.
• Establish debugging mechanisms for complex multi‑step agentic workflows.
• Develop monitoring practices to detect performance degradation and operational issues.

5. AWS AI Platform & Production Engineering

• Deploy AI applications using Amazon Bedrock and Bedrock AgentCore.
• Integrate foundation models, agent frameworks, APIs, and enterprise data sources.
• Apply production‑readiness practices covering scalability, reliability, security, and performance.
• Support CI/CD integration and automated evaluation of AI applications.
• Optimize model selection, inference costs, latency, and resource utilization.

6. Technical Leadership & Collaboration

• Provide technical guidance to engineering and QA teams.
• Establish reusable frameworks, implementation patterns, and best practices.
• Collaborate with architecture, DevOps, QA, and engineering teams.
• Participate in design discussions, code reviews, proof‑of‑concept development, and implementation.
• Mentor engineers and help teams adopt effective AI development and testing practices.

Required Technical Skills

Agentic AI & Orchestration

• LangGraph
• LangChain
• Amazon Bedrock
• Amazon Bedrock AgentCore
• AI agents, multi‑agent orchestration, tool calling, workflow management

Generative AI & RAG

• Large Language Models (LLMs)
• Retrieval‑Augmented Generation (RAG)
• Embeddings and vector search
• Prompt engineering and context management
• LLM application development

LLM Evaluation & Testing

• DeepEval
• LLM‑as‑a‑Judge techniques
• Evaluation metrics and benchmarking
• Automated evaluation and regression testing
• Assessment of hallucination, faithfulness, relevance, and response quality

AI Observability & Monitoring

• OpenTelemetry
• LangSmith
• Langfuse
• LLM tracing, debugging, and performance monitoring

Production Engineering

• Deploying and operating production‑grade GenAI applications
• API integration and backend engineering
• CI/CD and automated testing integration
• Scalability, reliability, security, and performance optimization

Preferred Qualifications

• 8+ years of software engineering or IT experience.
• Hands‑on experience developing and deploying GenAI or Agentic AI applications.
• Experience with enterprise‑scale AI implementations.
• Familiarity with AWS cloud services and enterprise AI security practices.
• Experience establishing AI quality standards and evaluation methodologies.
• Strong problem‑solving, communication, and technical collaboration skills.

What Success Looks Like

• Development and deployment of production‑grade agentic AI applications.
• Reliable orchestration and execution of AI agents and workflows.
• Effective LLM evaluation and automated AI testing frameworks.
• Improved AI application accuracy, reliability, observability, and performance.
• Reusable engineering practices for building and maintaining enterprise GenAI solutions.

Important: This is a hands‑on engineering position. Candidates with practical experience building and evaluating AI applications are preferred over those whose experience is limited to AI strategy, architecture documentation, conventional QA automation, or using AI coding assistants.

Flexible work from home options available.