Jobs in the United States

25+ jobs

Group jobs by employerEmail me new jobs Save this search

Added 1 min ago
C
Security Engineer
Cartesia
San Francisco, CA
- $200,000 to $240,000 USD / year
We are seeking a Security Engineer with a strong software engineering and DevOps background to enhance our engineering team. This role combines security, compliance, and DevSecOps to ensure…
Added 2 mins ago
Software Dev Engineer, Antenna Calibration, Amazon Leo
Amazon Kuiper Manufacturing Enterprises LLC
Redmond, WA
As a Software Engineer for antenna calibration, you will collaborate with systems architects, hardware engineering design teams & FW/SW design teams to bring up and test systems combining…
Added 3 mins ago
Manager Information Security
USA7 DXC Technology Services LLC
WASHINGTON, DC
Minimum of 3 years of experience in security engineering, security automation, DevSecOps, or software engineering with a security focus.
Added 3 mins ago
Manager Information Security
USA7 DXC Technology Services LLC
WASHINGTON, DC
Minimum of 3 years of experience in security engineering, security automation, DevSecOps, or software engineering with a security focus.
Added 6 mins ago
Director, Software Engineering - P&C Claims
United Services Automobile Asn
United States
We offer a flexible work environment that requires an individual to be in the office 4 days per week · This position can be based in the following location: Plano, TX
Added 11 mins ago
Azure Platform Engineer
Mosai
Remote, United States
The Azure Platform Engineer builds and supports the Azure Kubernetes Service (AKS) environments, CI/CD pipelines, infrastructure as code, and automation that form the internal platform our…
Added 13 mins ago
TT
Forward Deployed Engineer, Infrastructure
True Talent Solutions Partners
San Francisco, CA
- $200,000 to $300,000 USD / year
Join an AI infrastructure team to lead end-to-end software deployments in enterprise customer environments, from cloud VPCs to on-premises hardware. You will work directly with customer…
Added 17 mins ago
Hardware System Validation Engineer
Apple
San Jose, CA
Scripting proficiency; comfortable building and maintaining automation and/or scripting
Added 25 mins ago
A
Senior Mainframe CICS Systems Programmer
ADP
San Dimas, CA
- $93,400 to $230,000 USD / year
…performance issues to identify and resolve root causes. You drive automation of operational procedures including IPLs, CICS Cold Starts, recovery processing, and dump management. You…
Added 25 mins ago
Staff Automation Engineer
DexCom Inc
Remote, United States
…technologies spanning our suppliers, contract manufactures and automation houses. Including providing technical support for on-market production at our manufacturing sites. You will bring…
Added 26 mins ago
AE
Firmware Engineer II, C++
Advanced Energy
Redwood City, CA
- $90,000 to $150,000 USD / year
Firmware Engineer II, C+ ABOUT ADVANCED ENERGY
Added 26 mins ago
AE
Firmware Engineer I, C++
Advanced Energy
Redwood City, CA
- $90,000 to $150,000 USD / year
Firmware Engineer I, C+ ABOUT ADVANCED ENERGY
Added 26 mins ago
Hardware Test Engineer - Level 2 or 3
Northrop Grumman
Oxnard, CA
Demonstrated proficiency in test automation software (e.g., LabVIEW or similar) for product testing.
Added 26 mins ago
Embedded Software Engineer, RF Payloads (Starshield)
SpaceX
Hawthorne, CA
…earth observation, communications, and hosted payloads. As a RF software engineer on the Starshield team, you will be working on advanced development programs in support of US National…
Added 27 mins ago
Senior System Development Engineer - AWS Networking System Test, Release Engineering and System Test
Amazon
Cupertino, CA
As a Senior System Development Engineer in Networking, you'll have a unique opportunity to shape the development of our network - a fundamental contributor to the success of AWS that lets…
Added 27 mins ago
Senior Staff Software Engineer, ML, Search Ads Creatives
Google
Mountain View, CA
Share Senior Staff Software Engineer, ML, Search Ads Creatives
Added 27 mins ago
Senior Software Engineer, Mid Market Intelligence
Intuit
MOUNTAIN VIEW, CA
GBSG Mid-Market, one of Intuit's fastest-growing segments, is seeking a Senior Software Engineer to help build the industry's AI-native ERP. This is an opportunity to come in at the ground…
Added 27 mins ago
Hardware Systems Test Engineer, Networking, Google Cloud
Google
Sunnyvale, CA
hardware and software technologies that power all of Google's services. Our computational challenges are complex and unique, enabled by cutting-edge custom hardware designed and made…
Added 27 mins ago
Staff Software Engineer, Full Stack, Google Cloud Business Platform
Google
Sunnyvale, CA
Share Staff Software Engineer, Full Stack, Google Cloud Business Platform
Added 27 mins ago
Sr. Staff SWE, Provisioning Platform
Zscaler
Santa Clara, CA
The Sr. Staff Software Development Engineer will help architect and build Zscaler’s next-generation Provisioning Platform, powering SKU provisioning, entitlement management, workflow…
Added 27 mins ago
Senior Software Development Engineer, EC2 Nitro Team
Amazon
Santa Clara, CA
Ideal candidate must have strong background in Test-Driven Development and automated testing using scripting languages (Bash/Lua/JS/Python)
Added 27 mins ago
A
Senior RTL Design
Actalent
Santa Clara, CA
- $200,000 to $500,000 USD / year
We are seeking a highly skilled Senior RTL Design Engineer to join our team. You will play a pivotal role in designing, implementing, and optimizing RTL for key portions of the Velaura SoC.…
Added 27 mins ago
Senior Front-End and Agentic-UI Engineer
Evolent
Sacramento, CA
The Senior Software Engineer leads the design, development, and deployment of complex features in the web user experience for the Evolent Platform. This role builds high-performance React…
Added 27 mins ago
Core Senior .NET Software Engineer
Evolent
Sacramento, CA
The Senior Software Engineer leads the design, development, and deployment of complex software solutions, ensuring scalability and maintainability. This role architects and implements…
Added 27 mins ago
CH
Automation Testing Lead
Cayuse Holdings
Sacramento, CA
- $165,000 to $195,000 USD / year
A minimum of 7 years of experience in automation testing, software engineering, or IT systems development.

Bask Health

AI Evaluation Engineer

Remote, United States

Apply on apply.workable.com About employer

Applications open in a new tab.

Job details

Experience: 3+ years of professional experience in software engineering, AI engineering, test automation, quality engineering, or a closely related technical field.
Posting language: English
Posted: Nov 12, 2026

Original job description

About Bask Health Bask Health is a software platform that enables entrepreneurs, doctors, and healthcare businesses to launch, manage, and grow direct-to-consumer telehealth businesses. Our platform brings together patient experiences, clinical workflows, pharmacy operations, and commerce tools in one place. We're an entrepreneurial, product-focused engineering team that values ownership, technical excellence, and building software that solves real problems. Engineers work closely with product teams and company leadership, take features from idea to production, and have a direct role in shaping the platform. Engineering at Bask is AI-first and builder-led. We use tools like Cursor and Claude Code throughout development while maintaining high standards for reliability, testing, and software quality. About the Role We're looking for an AI Evaluation Engineer to help ensure that AI-powered features across Bask Health behave accurately, reliably, and consistently before and after they reach production. Bask Health supports the technology behind direct-to-consumer telehealth businesses, connecting patient experiences, merchant operations, clinical workflows, pharmacy coordination, and commerce capabilities. As we introduce more AI-powered functionality, we need dependable ways to verify that these systems work as intended under realistic conditions. You'll build and maintain evaluation workflows that test large language models, AI assistants, retrieval systems, and automated features against clearly defined product requirements. This is a hands-on engineering role that combines AI evaluation, software testing, automation, and production quality assurance. You'll develop test scenarios, evaluation datasets, automated checks, and tools that help engineering teams identify problems before they affect users. Your work may include testing AI-generated responses, validating structured outputs, evaluating tool-calling behavior, checking retrieval accuracy, and identifying regressions caused by changes to prompts, models, or application code. You'll collaborate with applied AI engineers, machine learning engineers, backend engineers, product managers, and quality-focused engineering teams to define expected behavior and translate those expectations into repeatable evaluations. You'll also help establish processes for reviewing AI failures, documenting recurring issues, and determining whether a feature meets its acceptance criteria. You'll work with engineers to reproduce problems and verify that fixes address the underlying issue. Because Bask operates in healthcare-related environments, you'll pay particular attention to privacy, permissions, unsupported medical claims, and AI behavior that could interfere with approved clinical or pharmacy workflows. Evaluation systems should help verify that AI features remain within their intended operational boundaries. This position emphasizes practical testing, thoughtful engineering, and continuous improvement. You'll contribute to evaluation standards while independently implementing and maintaining the tools and tests that support AI quality. We're looking for an engineer who enjoys finding difficult edge cases, understands how AI behavior differs from traditional deterministic software, and wants to help build products that users can depend on. What You'll Do

Design and implement evaluation workflows for AI-powered applications across Bask Health.
Build automated tests for large language model outputs and AI application behavior.
Develop evaluation datasets that represent realistic user requests and product workflows.
Create test scenarios for patient-facing, merchant-facing, and internal AI features.
Implement checks for response accuracy, relevance, completeness, and consistency.
Evaluate AI-generated content for unsupported claims and misleading information.
Test structured model outputs for schema compliance and application compatibility.
Build evaluations for AI tool calling and authorized action execution.
Validate that AI assistants respect application permissions and tenant boundaries.
Test retrieval-augmented generation systems for grounding and information relevance.
Develop regression tests that identify unexpected changes in AI behavior.
Integrate evaluation checks into engineering development and release workflows.
Create automated comparison tests for prompts, model versions, and configuration changes.
Develop scripts and utilities that simplify AI evaluation execution.
Build reporting tools that summarize evaluation results and failure patterns.
Investigate failed evaluations and help engineers reproduce underlying issues.
Work with product managers to define expected AI behavior and acceptance criteria.
Collaborate with applied AI engineers to improve prompts and application logic.
Test AI applications against unusual inputs, ambiguous requests, and difficult edge cases.
Develop evaluation cases for prompt injection and unauthorized tool-use attempts.
Support human review workflows for outputs that require subjective assessment.
Track recurring AI failures and help prioritize reliability improvements.
Monitor production evaluation signals and investigate quality regressions.
Document test coverage, evaluation procedures, and known limitations.
Improve evaluation efficiency, repeatability, and maintainability.
Participate in code reviews, engineering planning, and technical discussions.
Support controlled testing and rollout of new AI-powered features.
Own assigned AI evaluation projects from test design through implementation, reporting, and ongoing improvement.

Requirements

3+ years of professional experience in software engineering, AI engineering, test automation, quality engineering, or a closely related technical field.
Experience developing automated tests for production software applications.
Strong programming skills in Python, TypeScript, JavaScript, or comparable languages.
Familiarity with large language models and modern AI-powered applications.
Understanding of software testing methodologies and test case design.
Experience building automated testing scripts, frameworks, or validation tools.
Familiarity with API testing and backend application integrations.
Understanding of prompt engineering and common AI model failure modes.
Ability to evaluate AI-generated responses against expected requirements.
Experience working with structured data and validating application outputs.
Familiarity with regression testing and software release processes.
Experience debugging application failures and reproducing technical issues.
Understanding of authentication, authorization, and secure application behavior.
Familiarity with databases and application data models.
Experience working with Git and collaborative software development workflows.
Understanding of continuous integration and automated testing pipelines.
Familiarity with logging, monitoring, and production troubleshooting.
Ability to create clear, reproducible test cases.
Strong analytical thinking and attention to detail.
Ability to distinguish application defects from model behavior limitations.
Experience documenting technical issues and communicating findings.
Familiarity with cloud-based application environments.
Ability to collaborate effectively with engineers, product managers, and other stakeholders.
Strong problem-solving skills and a quality-focused engineering mindset.
Ability to independently own evaluation and test automation projects.

Nice to Have

Experience evaluating large language models or generative AI applications.
Familiarity with OpenAI, Anthropic, or comparable model APIs.
Experience with LangSmith, DeepEval, Ragas, promptfoo, or comparable evaluation tools.
Familiarity with LLM-as-a-judge evaluation techniques.
Experience developing AI evaluation datasets and test benchmarks.
Understanding of human evaluation and response labeling workflows.
Experience testing retrieval-augmented generation systems.
Familiarity with embeddings, semantic search, and vector databases.
Experience evaluating AI agents and tool-calling behavior.
Familiarity with prompt injection testing and AI security evaluations.
Experience implementing AI regression tests in CI/CD pipelines.
Experience building automated evaluation reports or dashboards.
Familiarity with Playwright, Jest, Vitest, Pytest, or comparable testing frameworks.
Experience testing TypeScript or Node.js applications.
Familiarity with React or Next.js application testing.
Experience working with AWS or serverless infrastructure.
Experience evaluating AI features in healthcare or other sensitive environments.
Understanding of HIPAA and healthcare data privacy considerations.
Experience developing human-in-the-loop quality review processes.
Experience using AI-assisted development tools such as Cursor or Claude Code.

Apply on apply.workable.com About employer

Report this job

Work requirements

Jobs in the United States