Job Description:
The Senior Systems Engineer is the technical and team lead for Qolo's Production Support / Site Reliability Engineering (SRE) team. The role is responsible for the stability, integrity and availability of the Qolo payments platform, including issuer authorization, settlement and funds movement, scheduled jobs, reports and file transfers, and client and internal portals. This is a senior individual contributor who also leads the team day to day. The Systems Engineer III acts as Incident Commander for major incidents, serves as the L2 escalation point for team members and for machine-generated alerts, sets technical direction for monitoring, automation and runbooks, and mentors the team. The role works closely with Engineering, DevOps, Security, Product, Client Success, Onboarding, and external banks and vendors, and acts as the liaison between those teams and Infrastructure & Support leadership. Production Support / SRE keeps detection, monitoring, incident triage and incident command. Day-to-day settlement and ACH subject-matter ownership sits with Payments Operations, and code-level remediation sits with Sustaining and Feature Engineering. This role works across those boundaries and drives the requirements that reduce manual work for the team.
Essential functions Incident command and on-call · Serve as Incident Commander for Sev 1 and major Sev 2 incidents: open and run bridges, set severity and priority, manage SLA clocks, coordinate stakeholders, and send incident communications on a defined cadence until resolution. · Act as the L2 and high-level escalation point for triaging and resolving technical issues, including SQL and transaction diagnostics, manual interventions that require bank approval, and coordination with Engineering and external parties. · Lead root cause analysis: complete RCAs within the 48-hour target using blameless methods such as 5 Whys, and drive corrective actions into runbooks, alerts, automation or engineering work items. · Participate in the Production Support / SRE 24x7 on-call rotation. Help design and maintain the rotation, PagerDuty routing and escalation policies aligned to reliability domains (issuer, jobs and reports, portals, network and infrastructure), and the Engineering Areas of Responsibility (AOR) escalation model. · Own the quality of on-call handoffs, escalation criteria and incident documentation for the team. Monitoring, alerting and observability · Design, tune and maintain alerts for the issuer path, scheduled jobs, reports, file transfers and portals using Azure Monitor, Log Analytics, Application Insights and Kusto Query Language (KQL). · Improve alert quality by reducing noise, tracking actionable versus non-actionable alerts, and retiring or refining alerts that do not drive action. · Define and track service level indicators and objectives (SLIs, SLOs) and error budgets for critical paths, including availability, latency and integrity of authorization, settlement and funds movement, reports and file delivery. · Run recurring audits, including the quarterly Azure alert action group review, to confirm recipients, routing, severity and on-call coverage. · Monitor and report on system performance, and recommend capacity and configuration changes. Jobs, files, reports and funds movement operations · Oversee operation and recovery of scheduled jobs and file flows, including VisualCron jobs and Azure runbook jobs: diagnose failures, determine partial loads, back out and reprocess, and rerun safely. · Confirm that files are created, transmitted, received and posted, and follow up on file exceptions. Rerun, deduplicate, correct and re-deliver failed reports and malformed client files. · Maintain the jobs and file inventory (Qolo File Management Tracker) as the source of truth for job names, schedules, owners, dependencies and SLAs. · Support daily settlement and funds-movement operations for sponsor banks while ownership transitions to Payments Operations, and continue to own detection, triage and escalation for settlement failures. · Investigate payment incidents by building and running targeted SQL queries and by researching card network and payment portals to resolve issues. Automation and engineering · Build and maintain SQL, PowerShell and Azure-based automation (Azure Automation, GitHub workflows and runners) for jobs, audits, reruns, diagnostics and other repeatable tasks. · Convert recurring manual work into documented, restartable and monitored automation, and write clear operational requirements (alerting, restartability, rollback) for work that Engineering or DevOps must implement. · Help the team protect engineering time by tagging and reporting work by category (incident, monitoring, automation, client support, configuration, audits) and by escalating when reactive work exceeds the target cap. · Identify and prioritize the team's top recurring pain points and turn them into an automation or tooling backlog, with measurable reduction in manual effort. · Install, configure, upgrade and enhance monitoring, operating system, database and utility software that supports production operations. · Prepare, test, evaluate and implement changes that support infrastructure and operations, following Qolo change controls. Release and deployment support · Support production releases: send client and vendor deployment notices, monitor Application Insights and Azure Front Door during rollout, validate post-deployment health, and participate in rollback and hotfix steps per playbooks. · Define minimum observability, post-deployment validation and rollback requirements for services and releases, and confirm that a tested rollback or hotfix path is linked from each release record and service runbook. · Move production traffic and adjust scaling during releases as directed by the release plan. Escalated client and internal support · Provide technical support for escalations from Client Services, Client Success and Onboarding when they cannot resolve an issue: reproduce the problem, pull logs, run targeted SQL and data checks, document the outcome, and escalate to Product or Engineering when needed. · Execute program, BIN and other non-financial configuration requests in line with Qolo change controls, and help define the requirements for self-service tools that remove Production Support as middleware. · Administer and review access for production platforms and tools, and support security investigations (for example phishing and suspicious activity) in partnership with Security. Certificates, security and compliance · Maintain the inventory, renewal calendar, alerts and runbooks for certificates that affect production (TLS, mTLS, card network and bank certificates), in partnership with Security and Engineering. · Apply Azure security best practices to the resources the team administers, and support audit requests with evidence of controls, access reviews and incident records. · Operate within access-control and least-privilege policies for on-call and production changes. Policy and governance remain with the CISO. Documentation and knowledge management · Develop and maintain technical documentation: runbooks, incident playbooks, SOPs, knowledge base articles (KBIs), deployment support procedures, client notification templates and diagnostic query libraries. · Make sure every frequent alert type has a linked runbook that defines first checks, mitigation and escalation paths, and that RCA findings feed back into runbooks and SOPs. · Document completed projects clearly for the next engineer and for stakeholders. Team leadership and mentorship · Act as technical and team lead for the Production Support / SRE team: set day-to-day priorities, review work, coordinate coverage and handoffs, and keep intake disciplined. · Mentor and train team members and L1 resources, including alert triage, SQL and KQL diagnostics, runbook use and incident communication. Model "play your position" discipline and cross-training so team members are interchangeable. · Serve as liaison between the Production Support / SRE team, stakeholder teams and Infrastructure & Support leadership. Advise on current internal and external work, and report team health, workload and risks. · Proactively research and introduce new technology and best practices, and recommend the technical direction for operations and reliability projects. · Lead large or complex projects as assigned: define the plan, coordinate participants and make sure work is delivered accurately and on time. · Expand your own education, technical stack and relevant certifications. · Perform other duties as assigned.
QUALIFICATIONS
Experience · Seven or more years of successful, extensive experience in systems engineering, production support, site reliability, infrastructure operations or a related technical field. · Demonstrated experience as an incident commander or technical escalation lead for production systems. · Experience in payments, financial services or other regulated, high-availability environments is strongly preferred.
Education and Certifications · Bachelor's degree or training from an accredited university, college or trade school in a related technology field, or equivalent professional and technical experience. Advanced degrees are a plus. · Advanced certifications recommended, for example Microsoft Azure Administrator or Azure Security Engineer, ITIL, or equivalent.
Technical skills · Expert-level Azure operations and monitoring: Azure Monitor, Log Analytics, Application Insights and KQL. · Advanced SQL Server skills, including reading and writing diagnostic queries and working with stored procedures. · PowerShell scripting and automation. Experience with Azure Automation and GitHub workflows is preferred. · Windows and Linux server administration, and Azure networking and security fundamentals. · Experience with job scheduling and file-transfer tools (for example VisualCron and SFTP), and with incident and alerting platforms (for example PagerDuty). · Strong working knowledge of Qolo's solutions and service offerings, or the ability to build it quickly, including card issuing and authorization flows and money movement (ACH, settlement). · Familiarity with SRE practices: SLIs and SLOs, error budgets, blameless postmortems and progressive delivery. Other requirements · Strong time management, organizational and communication skills, with the ability to multitask in a fast-paced environment. · Ability to provide 24x7 monitoring and support through participation in an on-call rotation.
As a forward-thinking software provider, Computer Services, Inc. (CSI) helps community and regional financial institutions solve their customers’ needs through open and flexible technologies. In addition to its award-winning core banking platform, these include the latest in lending, digital banking, payments, financial crime prevention and cybersecurity. Building on its 60-year track record of personalized service, CSI is shaping the future of banking and empowering its customers to rival their competition. For more information about CSI, visit www.csiweb.com
CSI provides rewarding and challenging career opportunities for our employees. When determining your pay, we consider various factors such as your skills, qualifications, experience and location. Along with a competitive salary, this position includes eligibility for incentive awards based on both individual and business performance. We also offer a comprehensive range of benefits. To learn more about our benefits, visit: Benefits Summary
CSI is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, pregnancy, sexual orientation, gender identity, national origin, age, physical and mental disability, marital status, veteran status, or any other characteristic protected by applicable law. If you need an accommodation during the recruitment process, please email us at recruiter@csiweb.com and we will work with you to meet your accessibility needs.
For applicants residing in California, please read Privacy Notice for California Residents | CSI (csiweb.com)
Visa Sponsorship: We are unable to offer visa sponsorship for this position. Applicants must be authorized to work in the United States without the need for sponsorship now or in the future.