This position is listed on behalf of a partner company, which manages all applications and next steps. Our partner is looking for a Distributed Systems Engineer, Multi-Chip Simulation (Mid / Senior / Principal) based in the United States.
In this role, you will design and develop advanced simulation infrastructure that enables distributed AI systems to operate across multiple interconnected chips before physical hardware is available. You will work at the intersection of distributed systems, hardware architecture, and software engineering to simulate how hosts, accelerators, and communication networks interact at scale. Your mission will be to build the software architecture and simulation tools that reproduce complex multi-chip behavior, including communication protocols, data transfers, and system-level failure scenarios. Working closely with chip modeling, SDK, and runtime teams, you will help ensure that distributed software can be developed, tested, and validated throughout the hardware development lifecycle. You will also determine the appropriate simulation fidelity for different use cases, balancing accuracy and performance to support frequent testing and continuous integration. As the system evolves, you will scale simulation capabilities from a small number of chips to complete rack-level configurations. This is an opportunity to take ownership of critical engineering infrastructure and help solve some of the most challenging distributed computing problems in next-generation AI inference.
Accountabilities:
- Design, develop, and maintain multi-chip simulation environments covering host-to-chip and chip-to-chip communication layers, including drivers, links, protocols, and data transfers.
- Build simulators that accurately reproduce the interactions between multiple accelerators and the host systems coordinating them.
- Model complex communication fabrics, including transfer timing, collective operations, ordering guarantees, back-pressure, congestion, and partial system failures.
- Calibrate communication and system-level models against real hardware measurements to improve simulation accuracy and reliability.
- Own the overall software architecture of the multi-chip simulation stack, adapting it to support new chip generations, interconnect technologies, and deployment topologies.
- Integrate simulators with different levels of fidelity, including fast functional models, cycle-level simulations, and hardware emulation environments, into a unified multi-chip simulation framework.
- Scale simulation infrastructure from small multi-chip configurations to full rack-level systems while maintaining execution speeds suitable for daily continuous integration (CI) workflows.
- Identify and resolve complex distributed systems issues, including timing-dependent bugs, race conditions, communication failures, deadlocks, and synchronization problems.
- Work closely with chip modeling, SDK, and runtime engineering teams to ensure that simulation infrastructure supports software development and system validation.
- Correlate simulated behavior with hardware emulation and real hardware measurements, maintaining regression coverage and documenting assumptions, limitations, and discrepancies.
- Continuously improve simulation performance, scalability, and maintainability as system complexity increases.
- Communicate simulation findings clearly, explaining system behavior, model limitations, and recommended software or hardware improvements.
Requirements:
- At least 5 years of professional experience as a Distributed Systems Engineer or in a closely related role involving the development and operation of software coordinating multiple machines, processors, or accelerators.
- Demonstrated experience working with distributed communication mechanisms such as remote procedure calls (RPC), message passing, consensus protocols, or collective communication.
- Strong understanding of distributed systems architecture and failure modes, including race conditions, partial failures, ordering and consistency guarantees, timeouts, retries, back-pressure, and deadlocks.
- Proven ability to investigate and debug timing-dependent issues across hosts, network interface controllers (NICs), accelerators, or other interconnected systems using traces, logs, and diagnostic tools.
- Experience designing and developing simulators, emulators, or test harnesses for distributed, networked, or complex software systems is highly desirable.
- Strong understanding of simulation methodologies and the ability to select appropriate levels of fidelity depending on the problem, balancing accuracy, complexity, and execution time.
- Strong programming skills in Rust, or expert-level C/C++ proficiency combined with a willingness to work extensively in Rust.
- Proficiency in Python for developing engineering tools, automation scripts, and simulation analysis workflows.
- Experience building scalable software architectures and maintaining reliable testing and continuous integration environments.
- Excellent analytical and problem-solving skills, with the ability to understand complex interactions between hardware, software, and distributed communication layers.
- Strong communication and collaboration skills, including the ability to explain simulation results, clarify model limitations, and translate findings into actionable engineering recommendations.
- Hands-on experience with multi-GPU or multi-accelerator inference or training systems is a strong advantage.
- Familiarity with NCCL-style collective communication, tensor parallelism, pipeline parallelism, expert parallelism, and host-device runtime architectures is highly desirable.
- Experience with hardware emulation platforms such as Cadence Palladium or Synopsys ZeBu, or with FPGA-based prototyping, is an additional advantage.
- Candidates with more extensive experience may be considered for Senior or Principal-level opportunities, with technical ownership, responsibilities, and expectations tailored to their experience and seniority.
Benefits:
- Competitive annual base salary based on experience and seniority:
- Mid-level: $160,000–$220,000.
- Senior-level: $220,000–$270,000.
- Principal-level: $270,000–$350,000.
- Competitive equity opportunities as part of the overall compensation package.
- Flexibility to offer compensation above the stated salary ranges for candidates with exceptional expertise, specialized skills, or significant experience.
- Fully company-paid medical, dental, and vision insurance for employees and their dependents.
- Company-paid life and disability insurance, with additional voluntary coverage options.
- Access to supplemental hospital, critical illness, and accident insurance.
- Unlimited paid time off, encouraging employees to rest, recharge, and maintain a healthy work-life balance.
- 13 paid company holidays annually.
- Remote-first working environment, offering flexibility to work from anywhere in the United States.
- Company-provided computer and home office setup.
- 401(k) retirement plan with company matching, available from the first day of employment.
- Opportunity to own the architecture of advanced multi-chip simulation infrastructure and influence next-generation AI hardware development.
- Direct collaboration with experienced engineering teams across chip modeling, software development, SDK, and runtime engineering.
- Exposure to cutting-edge distributed computing technologies and complex AI inference architectures.
- Opportunity to build scalable engineering infrastructure that enables distributed software to be developed and validated before physical hardware becomes available.
How Jobgether works:
We use an AI-powered matching process to ensure your application is reviewed quickly, objectively, and fairly against the role's core requirements. Our system identifies the top-fitting candidates, and this shortlist is then shared directly with the hiring company. The final decision and next steps (interviews, assessments) are managed by their internal team.
We appreciate your interest and wish you the best!
Why Apply Through Jobgether?
Data Privacy Notice: By submitting your application, you acknowledge that Jobgether will process your personal data to evaluate your candidacy and share relevant information with the hiring employer. This processing is based on legitimate interest and pre-contractual measures under applicable data protection laws (including GDPR). You may exercise your rights (access, rectification, erasure, objection) at any time.