Find work in the United States.

A simpler way to find your next job.

Jobs in the United States

25+ jobs

Save this search

Added 52 mins ago
Software Developer – Entry Level (Top Secret Clearance Required)
CGI
Fairfax, VA
Apply emerging technologies - such as generative AI, cloud platforms, automation tools, or DevOps practices - based on project needs and team direction.
Added 1 hour ago
SC
Principal Consultant
Softchoice Corporati
Denver, CO
- $158,720 to $198,400 USD / year
Added 1 hour ago
SC
Principal Consultant
Softchoice Corporati
Denver, CO
- $158,720 to $198,400 USD / year
Added 2 hours ago
Sr Technical Security Product Owner (DevSecOps Engineering)
Humana
Columbus, OH
You have a software engineering, DevOps, or DevSecOps background and made the deliberate move into product ownership - you can read a pull request, reason about an API contract, or sit in…
Added 3 hours ago
Software Systems Quality Engineer onsite Tucson, AZ
RTX Corporation
tucson, AZ
We have an immediate need for an onsite Senior Software Systems Quality Engineer (Compliance) to support Land & Air Defense Systems (LADS).
Added 4 hours ago
DevOps Engineer (Hybrid) - 31013
Huntington Ingalls Industries
Fort Meade, MD
Seeking a highly motivated and experienced DevOps Engineer to join our fast-paced team. The ideal candidate will have strong working knowledge in Kubernetes and Container Orchestration,…
Added 5 hours ago
DevOps Engineer (Hybrid) - 31007
Huntington Ingalls Industries
Columbia, MD
Seeking a highly motivated and experienced DevOps Engineer to join our fast-paced team. The ideal candidate will have strong working knowledge in Kubernetes and Container Orchestration,…
Added 5 hours ago
Senior Systems Engineer - FinOps
Marriott
Bethesda, MD
The engineer designs and implements automation for tagging compliance, budget guardrails, workload efficiency, and anomaly detection to ensure cost-effective, scalable data solutions.…
Added 5 hours ago
EG
Lead Software Engineer
Eliassen Group
Rockville, MD
- $75 to $83 USD / hour
Hybrid 3 days onsite / 2 days remote in Rockville, MD
Added 5 hours ago
Senior MuleSoft Engineer
Core4ce Careers
Remote, United States
Core4ce is seeking a Senior MuleSoft engineer to design, develop, and implement integration solutions using the MuleSoft Anytime Platform for the Defense Health Agency (DHA). In this role,…
Added 5 hours ago
Entry Level Software Engineer
Jacobs
Nashua, NH
Maintain a collaborative approach with Product Owners, Software Architects, DevOps, Security and UX Engineers
Added 5 hours ago
Cloud Engineer
UMB Bank
Kansas City, MO
+ AWS Solutions Architect Professional Certification, AWS DevOps Engineer Professional Certification, or any additional cloud certifications
Added 6 hours ago
EG
Senior Infrastructure Engineer
Eliassen Group
Orlando, FL
- $84 to $86 USD / hour
Hybrid 4 days on site/1 remote in Orlando, FL
Added 7 hours ago
Applications Architect (Enterprise Modernization & Cloud Migration) - (Active Top Secret/SCI Eligibility)
Akima, LLC
Alexandria, VA
Microsoft Azure certification (e.g.,Azure Developer Associate, Azure DevOps Engineer Expert, Power Platform App Maker, Power Platform Solution Architect Expert Azure Solutions Architect…
Added 8 hours ago
Principal Platform Engineer
Default Brand
Spanaway, WA
Ashley Digital is seeking a Principal Platform Engineer to own infrastructure, reliability, and delivery pipelines for our digital experience ecosystem - including our core ecommerce…
Added 8 hours ago
SO
IT System Administration - Senior/Specialist
State of Washington
Olympia, WA
- $8,522 to $11,460 USD / month
As a DevOps Engineer, you will help DCYF modernize and improve how technology is delivered by creating automation that makes systems more reliable, secure, and efficient. You will support…
Added 8 hours ago
CB
Senior Data Engineer
Commercial Bank of California
United States
- $135,000 to $170,000 USD / year
Job Type: Full-Time | Exempt | Hybrid/Remote Eligible
Added 8 hours ago
AWS Engineer
Koniag Government Services
Vienna, VA
Kadiak, LLC, a Koniag Government Services company, is seeking an AWS Engineer with a TS/SCI security clearance to support Kadiak and our government customer in Vienna, VA. This position…
Added 8 hours ago
Cloud Engineer Architect
Koniag Government Services
Washington, DC
Microsoft Azure Solutions Architect, Developer, DevOps Engineer, or similar certifications.
Added 8 hours ago
Financial Operations Engineer
Koniag Government Services
Chantilly, VA
Koniag IT Systems, LLC a Koniag Government Services company, is seeking a Financial Operations Engineer to support KITS and our government customer. The position is Hybrid (remote and the…
Added 8 hours ago
Senior DevOps Engineer
Koniag Government Services
Vienna, VA
Kadiak, LLC, a Koniag Government Services company, is seeking a Senior DevOps Engineer to support Kadiak and our government customer in Vienna, VA. This position requires an active…
Added 8 hours ago
DevOps Engineer
Koniag Government Services
Vienna, VA
Kadiak, LLC, a Koniag Government Services company, is seeking a DevOps Developer to support Kadiak and our government customer in Vienna, VA. This position requires an active Top-Secret…
Added 8 hours ago
GCP Cloud Product Owner
Koniag Government Services
Washington, DC
Google Cloud Digital Leader, Google Cloud Associate Cloud Engineer, or Google Cloud Professional Cloud Architect / DevOps Engineer.
Added 8 hours ago
AWS Cloud Product Owner
Koniag Government Services
Washington, DC
AWS Certified Cloud Practitioner, AWS Certified Solutions Architect - Associate, or AWS Certified DevOps Engineer.
Added 8 hours ago
Junior Software Engineer
Koniag Government Services
Washington, DC
Koniag IT Solutions, a Koniag Government Services company, is seeking a Junior Software Engineer with a TS/SSBI SCI eligibility a to support KITS and our government customer in Washington,…

Select Minds LLC

Senior AI Tools Testing Engineer

Dallas, TX, United States

$120,000 to $140,000 USD / year

Apply About employer

Applications open in a new tab.

Job details

Pay: $120,000 to $140,000 USD / year
Posting language: English
Posted: Nov 5, 2026

Original job description

Job Title: Senior / Lead AI Tools & Testing Engineer (Agentic AI & LLM Evaluation) Location: Dallas, TX (Hybrid) Duration: Long‑Term Experience:10 + Years

We seek a senior/lead AI engineer with hands‑on experience building, orchestrating, monitoring, and evaluating production‑grade Generative AI and Agentic AI applications. The ideal candidate is expert in Amazon Bedrock, AgentCore, LangGraph, LangChain, RAG, LLM observability, and evaluation frameworks. This role focuses on creating reliable, scalable, and production‑ready AI applications, emphasizing agent orchestration, AI testing, LLM evaluation, and continuous quality monitoring.

The candidate will collaborate with engineering, QA, and platform teams to establish best practices for building, testing, monitoring, and improving AI‑powered applications.

Key Responsibilities

Agentic AI & GenAI Engineering

Design, develop, and deploy production‑grade agentic and generative AI applications.
Build and orchestrate AI agents using LangGraph, LangChain, Amazon Bedrock, and AgentCore.
Implement multi‑agent workflows, tool calling, state management, memory, and orchestration.
Create scalable LLM‑powered applications and reusable AI components.

RAG & LLM Application Development

Design and implement Retrieval‑Augmented Generation (RAG) solutions.
Develop pipelines for document ingestion, chunking, embedding, retrieval, reranking, and context management.
Optimize retrieval accuracy, response quality, latency, and cost.
Apply grounding, hallucination reduction, and context‑relevance techniques.

LLM Evaluation & AI Testing

Build evaluation frameworks for LLM‑based applications and agentic workflows.
Automate evaluation with DeepEval and other LLM evaluation tools.
Use LLM‑as‑a‑Judge methods to assess response quality, relevance, accuracy, faithfulness, and reasoning.
Create evaluation datasets, test scenarios, benchmarks, and regression‑testing pipelines.
Evaluate agent behavior, tool execution, workflow accuracy, and response consistency.
Identify failure patterns and continuously improve AI application quality.

AI Observability & Monitoring

Implement tracing, monitoring, and observability with OpenTelemetry, LangSmith, and Langfuse.
Track LLM requests, agent execution flows, tool calls, latency, token consumption, errors, and performance.
Establish debugging mechanisms for complex multi‑step agentic workflows.
Develop monitoring practices to detect performance degradation and operational issues.

AWS AI Platform & Production Engineering

Deploy AI applications using Amazon Bedrock and Bedrock AgentCore.
Integrate foundation models, agent frameworks, APIs, and enterprise data sources.
Apply production‑readiness practices covering scalability, reliability, security, and performance.
Support CI/CD integration and automated evaluation of AI applications.
Optimize model selection, inference costs, latency, and resource utilization.

Technical Leadership & Collaboration

Provide technical guidance to engineering and QA teams.
Establish reusable frameworks, implementation patterns, and best practices.
Collaborate with architecture, DevOps, QA, and engineering teams.
Participate in design discussions, code reviews, proof‑of‑concept development, and implementation.
Mentor engineers and help teams adopt effective AI development and testing practices.

Required Technical Skills

Agentic AI & Orchestration

LangGraph
LangChain
Amazon Bedrock
Amazon Bedrock AgentCore
AI agents, multi‑agent orchestration, tool calling, workflow management

Generative AI & RAG

Large Language Models (LLMs)
Retrieval‑Augmented Generation (RAG)
Embeddings and vector search
Prompt engineering and context management
LLM application development

LLM Evaluation & Testing

DeepEval
LLM‑as‑a‑Judge techniques
Evaluation metrics and benchmarking
Automated evaluation and regression testing
Assessment of hallucination, faithfulness, relevance, and response quality

AI Observability & Monitoring

OpenTelemetry
LangSmith
Langfuse
LLM tracing, debugging, and performance monitoring

Production Engineering

Deploying and operating production‑grade GenAI applications
API integration and backend engineering
CI/CD and automated testing integration
Scalability, reliability, security, and performance optimization

Preferred Qualifications

8+ years of software engineering or IT experience.
Hands‑on experience developing and deploying GenAI or Agentic AI applications.
Experience with enterprise‑scale AI implementations.
Familiarity with AWS cloud services and enterprise AI security practices.
Experience establishing AI quality standards and evaluation methodologies.
Strong problem‑solving, communication, and technical collaboration skills.

What Success Looks Like

Development and deployment of production‑grade agentic AI applications.
Reliable orchestration and execution of AI agents and workflows.
Effective LLM evaluation and automated AI testing frameworks.
Improved AI application accuracy, reliability, observability, and performance.
Reusable engineering practices for building and maintaining enterprise GenAI solutions.

Important: This is a hands‑on engineering position. Candidates with practical experience building and evaluating AI applications are preferred over those whose experience is limited to AI strategy, architecture documentation, conventional QA automation, or using AI coding assistants.

Flexible work from home options available.

Apply About employer

More about this job

Skills

LangChain
Workflow
Debugging
AWS
CI/CD
Test automation
Problem solving
Communication skills
Collaboration

Report this job