Find work in the United States.

A simpler way to find your next job.

Jobs in the United States

25+ jobs

Save this search

Added 4 mins ago
BA
Platform DevOps Engineer
Booz Allen Hamilton
Colorado Springs, CO
- $61,900 to $141,000 USD / year
…and delivery consistently and continuously. As a DevSecOps engineer at Booz Allen, you'll use the latest architectural approaches and open-source frameworks and resources to overcome your…
Added 7 mins ago
Data Asset Model Senior Engineer
Mars Corporate
United States
…organizational change, we are seeking a FinTech Data Asset Model Senior Engineer to strengthen our capabilities in Azure Data Lake technologies within the FinTech Data Lake…
Added 9 mins ago
FL
Principal Software Eng./Developer*
FMR LLC dba Fidelity Investments
Boston, MA
- $134,868 to $166,000 USD / year
Delivers large-scale, multi-service Spring Boot solutions w/advanced reliability & deployment patterns. Defines RESTful API standards & builds Spring/OAuth2 components for broad reuse.…
Added 11 mins ago
Systems Engineer - Cloud Messaging & Enterprise Integration
Arcfield
Chantilly, VA
Arcfield is seeking a Senior Systems Engineer for the Merigold contract to provide SETA support for the architecture, integration, and evolution of enterprise messaging capabilities within…
Added 11 mins ago
C
AWS Cloud Architect - Networking & Observability
Cognizant
Jefferson City, MO
- $79,240 to $130,000 USD / year
+ Engineer highly available NAT Gateway and egress routing architectures.
Added 13 mins ago
Identity Operations Engineer
UMG Recordings Inc
Nashville, TN
As a key member of the UMG Operations Center, the Identity Operations Engineer operates and continuously improves UMG’s hybrid identity, access, authentication, privileged-access, and…
Added 16 mins ago
NB
Staff DevOps Engineer
Nova Biomedical
Waltham, MA
- $125,000 to $170,000 USD / year
The Staff DevOps Engineer provides senior-level technical leadership for cloud infrastructure, deployment automation, DevSecOps practices, and software delivery systems supporting regulated…
Added 16 mins ago
Staff AI Engineer
Nova Biomedical
Waltham, MA
The Staff AI Engineer will serve as a senior hands-on technical leader within Nova's Data & AI function, designing, building, and operationalizing AI-enabled applications, agents, reusable…
Added 16 mins ago
Digital Workplace Experience Engineer
Nova Biomedical
Waltham, MA
The Digital Workplace Experience Engineer is a senior individual contributor responsible for engineering the workflows, platform integrations, automation patterns, and experience…
Added 16 mins ago
Software Engineer
The Home Depot
Remote, United States
Security frameworks for user and service authentication and authorization;
Added 16 mins ago
Software Engineer Principal
The Home Depot
Remote, United States
Create meaningful architecture diagrams and other documentation needed for security reviews or other interested parties.
Added 16 mins ago
Sr. Software Engineer
The Home Depot
Remote, United States
Be involved in product and tool selection, configuration, security, resilience, performance tuning and production monitoring as well as mentoring less experienced engineers.
Added 16 mins ago
Staff Software Engineer (Platform - Access & Authorization)
Coinbase
Jefferson City, MO
Staff Software Engineer, Access & Authorization
Added 18 mins ago
MS
Lead Mainframe Hardware Engineer
MS Services Group, Inc.
Remote, United States
- $110,000 to $215,000 USD / year
…decommissioning. The candidate will also help lead resiliency, security, automation, and AI adoption efforts, partnering with engineering, operations, risk, audit, and…
Added 22 mins ago
Senior Security Engineer - Secure SDLC
Highmark Health
Jefferson City, MO
Highmark Health is seeking a Senior Security Engineer to join our Enterprise Application Security team and play a pivotal role in shaping how security is built into our software - not…
Added 26 mins ago
Senior Software Engineer - Access & Authorization (Platform)
Coinbase
Jefferson City, MO
Build secure, low-latency Go and gRPC services used across Coinbase's web, mobile, API, and partner experiences, with strong observability, failure testing, and measurable SLOs.
Added 28 mins ago
A
IT & Security Manager
Away
New York City, NY
- $130,000 to $135,000 USD / year
- You'll govern our enterprise AI platforms, including seats, access tiers, token and API security, and spend, and lead fleet-wide rollout of AI client apps with the IT Operations Engineer
Added 28 mins ago
Principal Firmware Engineer
Pelco, Inc.
Fort Collins, CO
…develop, test, and maintain software for our cutting-edge video security products. You will participate in the full software development lifecycle, from initial coding to system integration…
Added 29 mins ago
PKI Administrator
Charles Schwab
Southlake, TX
Schwab’s Security Analytics and Operations organization protects the technology and trust services that enable secure connections across the enterprise. As a Senior PKI Security Engineer,…
Added 30 mins ago
AI Portfolio Solution Architect (Sr Stf)
Lockheed Martin
Virtual Three, TX
…development environments, architecture patterns, code quality, security, testing, documentation, ownership, and release…
Added 31 mins ago
AI Systems Engineer - DevOps& Observability Manager
EY
Kansas City, MO
We are seeking an AI Systems Engineer to own the delivery, model-serving, routing, and observability layer of EY's AI-native platform. These are the systems that ship, run, and make fully…
Added 31 mins ago
AI Systems Engineer - DevOps& Observability - Senior
EY
Jefferson City, MO
We are seeking an AI Systems Engineer to own the delivery, model-serving, routing, and observability layer of EY's AI-native platform. These are the systems that ship, run, and make fully…
Added 31 mins ago
AI Systems Engineer - DevOps& Observability Manager
EY
Jefferson City, MO
We are seeking an AI Systems Engineer to own the delivery, model-serving, routing, and observability layer of EY's AI-native platform. These are the systems that ship, run, and make fully…
Added 31 mins ago
AI Systems Engineer - DevOps& Observability - Senior
EY
Saint Louis, MO
We are seeking an AI Systems Engineer to own the delivery, model-serving, routing, and observability layer of EY's AI-native platform. These are the systems that ship, run, and make fully…
Added 31 mins ago
AI Systems Engineer - DevOps& Observability Manager
EY
Saint Louis, MO
We are seeking an AI Systems Engineer to own the delivery, model-serving, routing, and observability layer of EY's AI-native platform. These are the systems that ship, run, and make fully…

Select Minds LLC

Senior AI Tools Testing Engineer

Dallas, TX, United States

$120,000 to $140,000 USD / year

Apply About employer

Applications open in a new tab.

Job details

Pay: $120,000 to $140,000 USD / year
Posting language: English
Posted: Nov 5, 2026

Original job description

Job Title: Senior / Lead AI Tools & Testing Engineer (Agentic AI & LLM Evaluation) Location: Dallas, TX (Hybrid) Duration: Long‑Term Experience:10 + Years

We seek a senior/lead AI engineer with hands‑on experience building, orchestrating, monitoring, and evaluating production‑grade Generative AI and Agentic AI applications. The ideal candidate is expert in Amazon Bedrock, AgentCore, LangGraph, LangChain, RAG, LLM observability, and evaluation frameworks. This role focuses on creating reliable, scalable, and production‑ready AI applications, emphasizing agent orchestration, AI testing, LLM evaluation, and continuous quality monitoring.

The candidate will collaborate with engineering, QA, and platform teams to establish best practices for building, testing, monitoring, and improving AI‑powered applications.

Key Responsibilities

Agentic AI & GenAI Engineering

Design, develop, and deploy production‑grade agentic and generative AI applications.
Build and orchestrate AI agents using LangGraph, LangChain, Amazon Bedrock, and AgentCore.
Implement multi‑agent workflows, tool calling, state management, memory, and orchestration.
Create scalable LLM‑powered applications and reusable AI components.

RAG & LLM Application Development

Design and implement Retrieval‑Augmented Generation (RAG) solutions.
Develop pipelines for document ingestion, chunking, embedding, retrieval, reranking, and context management.
Optimize retrieval accuracy, response quality, latency, and cost.
Apply grounding, hallucination reduction, and context‑relevance techniques.

LLM Evaluation & AI Testing

Build evaluation frameworks for LLM‑based applications and agentic workflows.
Automate evaluation with DeepEval and other LLM evaluation tools.
Use LLM‑as‑a‑Judge methods to assess response quality, relevance, accuracy, faithfulness, and reasoning.
Create evaluation datasets, test scenarios, benchmarks, and regression‑testing pipelines.
Evaluate agent behavior, tool execution, workflow accuracy, and response consistency.
Identify failure patterns and continuously improve AI application quality.

AI Observability & Monitoring

Implement tracing, monitoring, and observability with OpenTelemetry, LangSmith, and Langfuse.
Track LLM requests, agent execution flows, tool calls, latency, token consumption, errors, and performance.
Establish debugging mechanisms for complex multi‑step agentic workflows.
Develop monitoring practices to detect performance degradation and operational issues.

AWS AI Platform & Production Engineering

Deploy AI applications using Amazon Bedrock and Bedrock AgentCore.
Integrate foundation models, agent frameworks, APIs, and enterprise data sources.
Apply production‑readiness practices covering scalability, reliability, security, and performance.
Support CI/CD integration and automated evaluation of AI applications.
Optimize model selection, inference costs, latency, and resource utilization.

Technical Leadership & Collaboration

Provide technical guidance to engineering and QA teams.
Establish reusable frameworks, implementation patterns, and best practices.
Collaborate with architecture, DevOps, QA, and engineering teams.
Participate in design discussions, code reviews, proof‑of‑concept development, and implementation.
Mentor engineers and help teams adopt effective AI development and testing practices.

Required Technical Skills

Agentic AI & Orchestration

LangGraph
LangChain
Amazon Bedrock
Amazon Bedrock AgentCore
AI agents, multi‑agent orchestration, tool calling, workflow management

Generative AI & RAG

Large Language Models (LLMs)
Retrieval‑Augmented Generation (RAG)
Embeddings and vector search
Prompt engineering and context management
LLM application development

LLM Evaluation & Testing

DeepEval
LLM‑as‑a‑Judge techniques
Evaluation metrics and benchmarking
Automated evaluation and regression testing
Assessment of hallucination, faithfulness, relevance, and response quality

AI Observability & Monitoring

OpenTelemetry
LangSmith
Langfuse
LLM tracing, debugging, and performance monitoring

Production Engineering

Deploying and operating production‑grade GenAI applications
API integration and backend engineering
CI/CD and automated testing integration
Scalability, reliability, security, and performance optimization

Preferred Qualifications

8+ years of software engineering or IT experience.
Hands‑on experience developing and deploying GenAI or Agentic AI applications.
Experience with enterprise‑scale AI implementations.
Familiarity with AWS cloud services and enterprise AI security practices.
Experience establishing AI quality standards and evaluation methodologies.
Strong problem‑solving, communication, and technical collaboration skills.

What Success Looks Like

Development and deployment of production‑grade agentic AI applications.
Reliable orchestration and execution of AI agents and workflows.
Effective LLM evaluation and automated AI testing frameworks.
Improved AI application accuracy, reliability, observability, and performance.
Reusable engineering practices for building and maintaining enterprise GenAI solutions.

Important: This is a hands‑on engineering position. Candidates with practical experience building and evaluating AI applications are preferred over those whose experience is limited to AI strategy, architecture documentation, conventional QA automation, or using AI coding assistants.

Flexible work from home options available.

Apply About employer

More about this job

Skills

LangChain
Workflow
Debugging
AWS
CI/CD
Test automation
Problem solving
Communication skills
Collaboration

Report this job