Lead InfiniBand Network Engineer

About ChronoScale ChronoScale builds and runs GPU clouds for hyperscalers, AI clouds and inference companies. Over the next year we bring online a 2,048-GPU B300 cluster and a ~25,000-GPU GB300 NVL72 build, and about 150 MW more capacity across Texas and North Dakota. NCPE builds, runs and supports all of it: DC Ops, network, SRE, NOC and platform software. Networking is split into two teams:

• Network Architecture defines the designs and standards.
• Network Engineering, this team, turns them into running networks at every site and keeps them running.

The role You join the founding core of Network Engineering. It is a senior, hands-on role in delivery, operations and automation. You:

• turn reference designs into running fabrics;
• write the automation that deploys and checks them;
• are the final escalation point when a training job slows down because of the network.

You work across every customer and site, not on a single project. You work closely with Network Architecture, DC Ops, SRE, the platform team and the NOC, and directly with customer engineering teams during onboarding. The networks you'll own Network What it is

Compute (back-end) Rail-optimized, non-blocking InfiniBand (XDR) and RoCE fabrics on NVIDIA Spectrum-X, connecting thousands of GPUs for distributed training and inference

Front-end and tenant EVPN-VXLAN fabrics under SDN control, with per-tenant VRFs from the DPU up to the firewall, and hyperscaler connections

Storage Lossless, RDMA-capable networks feeding VAST, Weka and other parallel file systems at line rate

Edge, interconnect and management 800G edge routing over long-haul waves with MACsec, IP transit, per-tenant NAT and IPsec, and the management, out-of-band and console networks that stay up when everything else is down

How we work This is the most important part of the description.

• Source of truth first. Devices, cables, addresses and VLANs live in NetBox and Infrahub. Configuration, cable schedules and customer inventory are generated from that data, never typed.
• Every change is a pull request. It is reviewed, tested in a lab (containerlab, the SDN vendor's simulator, our own virtual and hardware labs), and deployed by tooling.
• The CLI is for reading state and for break-glass, not for configuring.
• AI tools are part of the toolchain. Claude Code, Copilot, Cursor or similar draft our templates, parsers, tests and change procedures, and review diffs. You use them every day, and you are accountable for what they produce.

Responsibilities:

• Deliver new sites and phases. Turn reference designs into build plans, cabling and optics schedules and cutover steps. Coordinate them with DC Ops and vendors on site.
• Bring up fabrics.
• Zero-touch provisioning, onboarding into the SDN controller and NetBox, and image management.
• Reconcile discovered state (LLDP, serial numbers, MACs) against the cable plan until no discrepancies remain.
• Tune for AI workloads. Adaptive routing, congestion control (ECN, PFC, DCQCN), QoS, and subnet manager or UFM policy, all applied from templates, so collectives reach the expected bus bandwidth.
• Isolate tenants with EVPN-VXLAN, VRFs, InfiniBand partition keys, firewall virtual routers and ACLs, so customers share infrastructure without sharing traffic.
• Automate everything. Python, Jinja, Ansible and OpenTofu, plus API integrations with the SDN controller, NetBox, UFM and gNMI, for provisioning, config generation, compliance checks and change pipelines.
• Write the validation that proves a build, as code:

The output is the evidence customers require at acceptance.

• links and LLDP against the cable plan;
• BGP sessions against what's expected;
• optics and light levels;
• switch-pair latency;
• fabric bandwidth, NCCL all-reduce, perftest and ib_write_bw against contracted benchmarks.
• Instrument the fabric with streaming telemetry into Prometheus and Grafana. Alert on link flaps, symbol and bit errors, congestion and optic degradation before customers feel them.
• Lead incident response as the top escalation for the NOC. Run root-cause analysis with NVIDIA, Arista, Juniper and optics vendors. Write the runbooks and standard changes that let the NOC resolve the next one without you.
• Close the loop with Architecture. Bring production data and lessons from every build back to Network Architecture, and keep as-built records in the source of truth, so the tenth site comes up faster than the first.

Minimum Qualifications:

• 6+ years in network engineering, including deploying and operating production data centre or service provider networks at scale.
• Strong data centre routing and switching: BGP (including unnumbered and large-scale ECMP), EVPN-VXLAN and Clos fabrics. Deep enough to tell when generated configuration is wrong.
• At least one GPU or HPC fabric in production:

You know how to find why NCCL bandwidth is low.

• RoCE on lossless Ethernet (PFC, ECN or DCQCN, buffer tuning, Spectrum-X or similar);
• or InfiniBand (UFM or OpenSM, ibdiagnet, ibstat, perfquery).
• Multi-tenant routing: VRFs, route targets and leaking, NAT, firewall virtual routers.
• Automation you have shipped and still maintain: Python you wrote, a Git workflow, CI, and network APIs.
• Daily use of AI coding assistants on real work. Be ready to show us where they helped and where they got it wrong.
• Linux networking (iproute2, ethtool, tcpdump, iperf3) and the host side of RDMA: NICs, DPUs, drivers, MTU and queue tuning.
• Physical-layer judgement: 400G and 800G optics, DACs and AOCs, structured cabling, and reading DOM and BER data to find the bad link.
• A record of large network builds, and calm recovery during P1 incidents.

Preferred Qualifications:

• Both RoCE and InfiniBand, including fat-tree routing and UFM operations.
• GB300 or GB200 NVL72, B300 or H100/H200 clusters in production.
• BlueField DPUs in DPU mode as routers and EVPN VTEPs; DOCA, SR-IOV.
• Netris or another SDN controller; vCluster or vMetal; Tailscale run as infrastructure (subnet routers, ACLs in Git).
• NVIDIA Cumulus (NVUE) or SONiC on whitebox switches; Arista EOS; Juniper MX, SRX (Multinode HA) and EX.
• Data modelling in NetBox or Infrahub; OpenTofu providers; KQL.
• MACsec, DWDM, 400ZR and long-haul waves; IP transit and peering.
• Time in live data halls with liquid-cooled racks.

Please note that ChronoScale cannot provide employment authorization sponsorship or immigration support for this role, now or in the future. Candidates must have valid work authorization that does not require employer sponsorship.

The company has reviewed this job description to ensure that essential functions and basic duties have been included. It is intended to provide guidelines for job expectations and the employee's ability to perform the position described. It is not intended to be construed as an exhaustive list of all functions, responsibilities, skills and abilities. Additional functions and requirements may be assigned by supervisors as deemed appropriate. This document does not represent a contract of employment, and the company reserves the right to change this job description and/or assign tasks for the employee to perform, as the company may deem appropriate. This job description in no way states or implies that these are the only duties to be performed by the employee(s) incumbent in this position. Employees will be required to follow any other job-related instructions and to perform any other job-related duties requested by any person authorized to give instructions or assignments. All duties and responsibilities are essential functions and requirements and are subject to possible modification to reasonably accommodate individuals with disabilities. To perform this job successfully, the incumbents acknowledge that they possess the skills, aptitudes, and abilities to perform each duty proficiently. Some requirements may exclude individuals who pose a direct threat or significant risk to the health or safety of themselves or others. This document does not create an employment contract, implied or otherwise, other than an “at will” relationship. The company is an Equal Opportunity Employer, drug free workplace, and complies with ADA regulations as applicable.