Principal Software Quality Engineer

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Principal Software Quality Engineer based in the United States.

In this senior technical individual-contributor role, you will define and drive quality strategy for a sophisticated distributed database platform, ensuring reliability, scalability, and resilience across complex systems. You will combine deep distributed systems expertise with advanced software testing, automation, and systems-level debugging. Working across quality engineering, client engineering, and server engineering teams, you will identify critical failure scenarios and establish robust testing frameworks. You will also lead the integration of AI agents into testing workflows, enabling intelligent test generation, automated failure analysis, and improved engineering efficiency. This position offers the opportunity to influence technical standards, mentor senior engineers, and shape quality practices across the organization. It is ideal for an experienced engineer who enjoys hands-on technical challenges, complex debugging, and building scalable quality infrastructure.

Accountabilities:

• Define and own the overall quality strategy for the distributed database platform, including test architecture, coverage models, release criteria, and risk-based prioritization.

• Drive quality across database server components, client SDKs, operators, backup tooling, and management interfaces.

• Design comprehensive testing strategies for replication, rebalancing, partitioning, sharding, consistency models, transactions, isolation, secondary indexes, query planning, compaction, backup and restore, snapshots, and cross-datacenter replication.

• Establish reliable test coverage for critical database operations and ensure quality and testability requirements are incorporated early in the software development lifecycle.

• Validate cluster correctness under production-like failure conditions, including node crashes, network partitions, clock skew, disk failures, rolling upgrades, mixed-version deployments, and long-running soak tests.

• Design and execute fault-injection scenarios involving process crashes, GDB-driven state manipulation, packet drops, latency injection, corrupted protocol data, oversized inputs, and version incompatibilities.

• Identify complex failure scenarios that conventional functional testing cannot adequately detect.

• Investigate low-level system defects involving memory boundaries, encoding and decoding inconsistencies, compression, CRC errors, and distributed consistency violations.

• Reproduce defects at the source-code level, identify potential root causes and affected functions, and provide engineering teams with minimal reproducible test cases and actionable technical analysis.

• Build and maintain scalable automated testing frameworks using Python, pytest, custom harnesses, API testing, and end-to-end automation.

• Ensure test suites operate reliably across shared testing infrastructure and CI/CD environments, including GitHub Actions and Jenkins.

• Improve test infrastructure reliability by reducing flaky tests, investigating noisy failures, and strengthening regression coverage.

• Develop testing strategies across unit, integration, API, contract, end-to-end, performance, scalability, upgrade/downgrade, and long-running regression testing.

• Lead the integration of AI agents into existing testing frameworks, designing workflows capable of planning and executing tests, interacting with tools, analyzing logs and metrics, and generating actionable defect reports.

• Develop AI-agent evaluation frameworks using golden datasets, graders, and regression suites to measure reliability, accuracy, and consistency.

• Integrate AI agents with pytest, custom test harnesses, and MCP-style tool servers.

• Enable AI-assisted test generation, edge-case discovery, flaky-test maintenance, and failure triage while maintaining appropriate human oversight.

• Participate in feature reviews, product requirements discussions, design reviews, and client specifications to establish testability requirements before implementation.

• Develop test plans for new features and collaborate with engineering teams to validate technical designs.

• Convert production incidents and pre-release defects into permanent regression tests, monitoring checks, and preventive measures.

• Lead technical debugging and defect-resolution efforts across quality engineering, client engineering, and server engineering teams.

• Mentor senior Quality Engineers and SDETs in debugging techniques, test architecture, and framework development.

• Establish technical quality standards and engineering practices that can be adopted across teams without direct people-management responsibilities.

• Drive measurable improvements in regression prevention, test infrastructure stability, defect resolution time, and AI-assisted testing effectiveness.

Requirements:

• 10–18 years of experience in Software Quality Engineering, Software Development Engineer in Test (SDET), or Systems Engineering, with significant technical impact at Staff or Principal level.

• Extensive hands-on experience testing distributed databases or storage systems, preferably NoSQL platforms such as Cassandra, ScyllaDB, MongoDB, Couchbase, Redis, DynamoDB, or HBase. Experience with distributed relational databases such as CockroachDB, TiDB, YugabyteDB, or Spanner-style systems is also relevant.

• Strong expertise in testing replication, partitioning, sharding, consistency, transactions, secondary indexes, backup and restore, and cross-datacenter replication.

• Deep understanding of distributed systems fundamentals, including replication protocols, consensus, quorum reads and writes, strong and eventual consistency, split-brain scenarios, clock skew, and failure and recovery semantics.

• Ability to read, analyze, and debug server-side code written in C, C++, Go, Rust, or Java, tracing defects to specific functions and root causes.

• Understanding of protocol and serialization edge cases, including wire formats, MessagePack, Protocol Buffers, Thrift, and compression codecs.

• Strong software engineering and automation skills in Python, with proficiency in at least one systems programming language.

• Practical experience with fault injection, GDB, chaos testing, process termination, network partitions, latency and packet-loss injection, disk faults, corrupted inputs, version skew, and boundary-value testing.

• Experience across multiple testing layers, including unit, integration, API, contract, end-to-end, performance, scalability, upgrade/downgrade, and soak testing.

• Proven ability to lead complex debugging and root-cause investigations across quality engineering, client, and server teams, producing actionable analysis and driving defects through resolution.

• Strong CI/CD expertise with GitHub Actions, Jenkins, or equivalent tools, including experience managing and improving shared test infrastructure.

• Practical experience integrating AI agents into testing workflows, including agent orchestration, tool integration, prompt design, evaluation frameworks, and integration with production-grade testing systems.

• Excellent written and verbal communication skills, with the ability to produce clear technical tickets, investigation reports, engineering discussions, and RFCs.

• Strong analytical thinking, technical judgment, and problem-solving skills, with the ability to independently investigate complex system-level issues.

• Ability to influence engineering teams, establish technical standards, and mentor experienced engineers without formal people-management responsibilities.

• Nice to have: Experience testing client SDKs across multiple programming languages, including Python, Java, Go, C, Node.js, and .NET.

• Nice to have: Knowledge of compatibility matrices, upgrade and downgrade scenarios, and behavioral consistency across client SDKs.

• Nice to have: Experience testing backup and restore, checkpointing, snapshotting, and WAL/journal correctness, including performance at terabyte scale.

• Nice to have: Expertise in query-engine and secondary-index testing, including query planning, index selection, pagination, boundary conditions, and encoding/collation correctness.

• Nice to have: Kubernetes and database operator testing experience using tools such as KUTTL, Chainsaw, or envtest.

• Nice to have: Cloud infrastructure experience across AWS, GCP, or Azure, including TLS/mTLS, IAM, secrets management, and air-gapped installations.

• Nice to have: Security-focused testing experience, including fuzzing, CVE reproduction, and identifying memory-safety issues such as out-of-bounds reads and use-after-free defects in native code.

• This is a hands-on senior technical position requiring active involvement in framework development, server-side debugging, low-level systems investigation, AI-agent evaluation, and reliable test infrastructure development.

Benefits:

• Annual Compensation: ₹40,00,000–₹70,00,000 per year, as specified in the original job posting.

• Full-time remote working arrangement.

• Opportunity to work on advanced distributed database and storage technologies.

• High-impact technical role with ownership of quality strategy and engineering standards.

• Opportunity to work with AI agents, automation frameworks, and cutting-edge testing technologies.

• Collaboration with experienced quality engineering, client engineering, and server engineering teams.

• Opportunity to influence product reliability, scalability, and engineering practices across the organization.

• Professional growth through complex systems engineering challenges and technical leadership opportunities.

How Jobgether works:

We use an AI-powered matching process to ensure your application is reviewed quickly, objectively, and fairly against the role's core requirements. Our system identifies the top-fitting candidates, and this shortlist is then shared directly with the hiring company. The final decision and next steps (interviews, assessments) are managed by their internal team.

We appreciate your interest and wish you the best!

Why Apply Through Jobgether?

Data Privacy Notice: By submitting your application, you acknowledge that Jobgether will process your personal data to evaluate your candidacy and share relevant information with the hiring employer. This processing is based on legitimate interest and pre-contractual measures under applicable data protection laws (including GDPR). You may exercise your rights (access, rectification, erasure, objection) at any time.