Please note: ONLY applications submitted via Greenhouse will be considered. APPLY USING THE LINK BELOW TO BE CONSIDERED: Job Application for Senior AI Engineer at Office of the State Auditor **SALARY RANGE (Grade 15) CSA216: $89,993.11/yr. -- $134,991.01/yr. ****(Commensurate with experience)** GENERAL STATEMENT OF DUTIES: The Senior AI Engineer designs the architecture for, builds, and deploys AI-enabled applications in OSA's Microsoft GCC environment. Working with the Innovation Assistant Director and Cloud Engineering, the role shapes OSA's AI reference architecture and turns it into production Retrieval-Augmented Generation (RAG) applications, agent workflows, the data pipelines that feed them, and cost- and performance-aware model deployments. The role builds applications on existing foundation models accessed through approved platforms such as Azure AI Foundry; it does not train or develop LLMs. The primary focus is the Audit division, delivering AI services that measurably improve audit planning, research, and analysis across OSA's statewide audit portfolio. The role also builds AI-enabled programs for OSA's administrative and operational units, including Human Resources, Administration & Finance (A&F), Occupational Schools, the Division of Local Mandates (DLM), and the Bureau of Special Investigations (BSI). OSA does not expect candidates to know every tool named in this description. Solid architecture and engineering fundamentals and hands-on experience delivering LLM applications matter most. SUPERVISION RECEIVED: Reports to the Innovation Assistant Director and works within OSA's IT Innovation group. Collaborates closely with the Manager, System and Process Improvements, Cloud Engineering (e.g., platform security, IaC, APIM/Entra), Enterprise IT, the Data Analytics Unit, data engineering staff, contractors, and business stakeholders across Audit and OSA's operational units. The Manager, System and Process Improvements maintains the backlog of Audit process-improvement opportunities, designs and measures pilots of AI-assisted audit process steps and recommends adoption; Audit and agency leadership decide whether to adopt changes to audit processes. The Innovation Assistant Director runs intake for OSA's other units, sets IT Innovation work priorities in line with the IT Governance Council, manages staff and contractors, and decides technology and build-versus-buy choices. This role owns the technical design and engineering of the solutions and does not supervise staff. DUTIES AND RESPONSIBILITIES: AI Solution Architecture • Design solution architectures for AI applications - components, integrations, data flows, security boundaries, and deployment topology - and document them in diagrams and architecture decision records. • Lead the definition and maintenance of OSA's AI reference architecture, including standard patterns for RAG, agents, API exposure, logging, and evaluation, with review and approval by the Innovation Assistant Director. • Work with Cloud Engineering to extend Azure Landing Zone patterns (CAF/WAF) for AI workloads: subscriptions, Azure Policy, VNets/NSGs, private endpoints, Key Vault, and managed identities. • Define microservice and API boundaries (REST, APIM) so AI capabilities are reused across audit and operational applications rather than rebuilt for each one. • Lead technical design reviews, and provide technical assessments of new Azure AI services, vendor tools, and build-versus-buy options to inform technology decisions by the Innovation Assistant Director. • Design for cost from the start: model and hosting choices, budgets, and tagging so AI consumption is visible and attributable by project. Software Development • Write, test, and maintain production Python and TypeScript/JavaScript code for AI-enabled applications, services, and user interfaces. • Build REST APIs and backend services that connect LLMs to OSA documents, data sources, and business systems. • Write automated unit, integration, and evaluation tests; set code review and Git/GitHub branching standards for AI projects. • Build the automated tests, evaluation harnesses, and quality metrics used in OSA's software quality assurance process for AI applications (distinct from the audit Quality Assurance function), which data engineering staff and testers use to carry out testing. • Use AI-assisted development tools such as GitHub Copilot responsibly, reviewing generated code for correctness and security. • Troubleshoot and improve the AI components of applications running in production. AI Application Engineering • Own retrieval design for RAG applications - hierarchical/semantic chunking, embedding choices, hybrid (keyword vector) search, cross-encoder reranking, and citation fidelity checks - working with data engineering staff who build ingestion, schema, and embedding pipelines. • Engineer agent and tool-calling workflows with Microsoft Agent Framework; lead migration of existing Prompt Flow orchestration and evaluation into supported pathways. • Define and instrument quality and performance measures - groundedness, relevance, citation accuracy (e.g., RAGAS), latency, TTFT, tokens/sec, concurrency, and cost-per-query - with dashboards in Azure Monitor, Log Analytics, and Application Insights, and supply usage and adoption data for the audit workflow KPIs and dashboard maintained by the Manager, System and Process Improvements. • Build the technical components of bounded pilots designed by the Manager, System and Process Improvements, and provide the data needed to measure results against established baselines. • Translate business requirements - developed with the Manager, System and Process Improvements for Audit or with the Innovation Assistant Director for other units - into technical requirements, working prototypes, and production applications. Platform, Deployment & Operations • Containerize services with Docker and deploy them on Kubernetes (AKS), including health checks, autoscaling, secrets integration, and rolling updates. • Provide ongoing support for the AI layer of deployed applications - models, prompts, retrieval, agents, and evaluations - and transition hosting, infrastructure, and general application support to Enterprise IT with runbooks and knowledge transfer. • Build and maintain infrastructure as code (Terraform or Bicep) and GitHub Actions CI/CD pipelines, and design Dev/Test/Prod and sandbox environment patterns, including promotion standards for AI services, prompts, model configurations, and evaluation assets. • Configure AI Foundry, APIM, Entra ID app registrations, and related Azure services for secure-by-default deployments. • Design backup, recovery, and business continuity approaches for AI services and document them in runbooks for Enterprise IT. Data for AI • Design ingestion and transformation pipelines (e.g., Azure Data Factory, Python, SQL, dbt), and set the patterns data engineering staff use to build and maintain them, to prepare audit, agency, and internal operational data for retrieval indexes, knowledge graphs, and analytics. • Define data interface contracts - schemas, validation rules, and reconciliation checks - and build pipelines with retries, backfills, and idempotency. Model Selection & Performance • Select and configure model deployments (e.g., standard vs provisioned throughput, regional availability in GCC) based on capability, latency, capacity, and cost for each use case. • Evaluate provider and hosting options, including the underlying accelerators (NVIDIA H100/H200/B200; AMD MI300; LPU-based services), when comparing throughput, latency, and cost. • Apply application-level optimizations: prompt/prefix caching, batch and streaming APIs, context and token budgeting, and routing between larger and smaller models. Knowledge Graphs & Taxonomy • Integrate OSA's taxonomy (categories defined with the Data Analytics Unit) and entity relationships, managed in Synaptica Graphite, into retrieval and agent workflows, and expose graph-powered APIs for Audit Research Tool, Audit Search, and TeamMate . • Model domain ontologies (entities, events, regulations) to support retrieval, entity linking, and policy-aware reasoning within agents. Security, Privacy & Governance • Design least-privilege access using Entra ID security group patterns (team vs access), RBAC, and PIM, and implement OSA AI policies in code (policy-as-code). • Implement audit-ready logging (prompts, tool calls, citations), FOIA-aware retention, and privacy controls such as DLP labels and redaction filters. • Build application features that support GAGAS-compatible protocols for documenting AI assistance, human verification, and supervisory review, as defined by the Manager, System and Process Improvements with Quality Assurance. • Protect sensitive data (PII, PHI, HIPAA-regulated healthcare data) with encryption at rest and in transit, key management, and classification-driven access. • Partner with cybersecurity on threat modeling and security reviews, and maintain audit-ready evidence of controls. Delivery & Collaboration • Share technical knowledge with developers and analysts through code reviews, documentation, and pairing on AI agent patterns, retrieval engineering, and reproducible deployments. • Build AI-enabled applications, as prioritized by the Innovation Assistant Director, for Human Resources, Administration & Finance (A&F), Occupational Schools, the Division of Local Mandates (DLM), and the Bureau of Special Investigations (BSI), reusing shared platform components. • Provide day-to-day technical direction to the AI Application Developer and data engineering staff on application, pipeline, schema, and retrieval work. • Produce architecture documentation and operational runbooks that let Enterprise IT and OSA staff operate and extend systems without vendor dependency. • Explain architecture choices and trade-offs to the IT Innovation team and project stakeholders. The Office of the State Auditor is committed to providing equal employment opportunities. Employment actions such as recruiting, hiring, training, and promoting individuals are based upon a policy of non-discrimination. Employment decisions and actions are made without regard to race, color, gender, religion, age, national origin, ancestry, sexual orientation, gender identity and expression, disability, military status, genetic information, political affiliation, or veteran’s status. MINIMUM QUALIFICATIONS: /The successful candidate will possess and/or demonstrate:/ • Bachelor’s degree in computer science, software engineering, or a related field, or equivalent experience. • 5–7 years of professional software engineering experience, including at least 2 years building production applications that use LLMs (e.g., RAG, agents, or tool-calling features). • Demonstrated design ownership of the architecture for at least one production system, including documentation of design decisions. • Strong Python; working proficiency in TypeScript/JavaScript and SQL. • Hands-on building RAG systems with hybrid search and reranking; experience with embeddings and vector stores (e.g., Azure AI Search, pgvector, Qdrant, Weaviate). • Experience designing and deploying applications on Azure (or AWS/GCP with a willingness to work in Azure), including identity, private networking, and secrets management. • Experience with Docker and Kubernetes, infrastructure as code (Terraform or Bicep), and CI/CD pipelines. • Experience designing REST APIs and service boundaries, and writing automated tests. • Experience handling sensitive data (PII/PHI) under least-privilege access and encryption controls. • Clear written communication, including architecture documentation, and the ability to explain technical trade-offs. PREFERRED QUALIFICATIONS: • Azure Government/Microsoft GCC experience, including Azure Landing Zones and Cloud Adoption Framework implementation. • Microsoft Azure certifications at the associate or expert level (e.g., AI engineering or solutions architecture). • Microsoft Agent Framework and Prompt Flow migration; LangChain/LangGraph; structured outputs and function/tool calling across providers. • Deeper Kubernetes experience on AKS: Helm charts, ingress, autoscaling, and workload identity. • Knowledge graphs, taxonomies, or ontologies (e.g., RDF, SPARQL); familiarity with Synaptica Graphite. • Data engineering with Databricks (Unity Catalog, Delta Lake) or Microsoft Fabric; data governance frameworks (e.g., DMBOK), metadata and lineage management, and MDM. • RBAC/ABAC design, Purview classification, DLP labels, Defender for Cloud, and policy-as-code. • Understanding of LLM serving trade-offs: quantization (FP8/INT4), KV-cache strategies, MoE models and expert parallelism, and open-weight serving stacks (vLLM, TensorRT-LLM, SGLang). • Experience in government, healthcare, or financial services, including shared platform or data standards across multiple agencies or business units. • Experience integrating TeamMate and external sources into AI-driven research workflows. • Experience mentoring developers or serving as technical lead on a project. • Exposure to AWS or GCP data and AI services (e.g., Vertex AI). Salary is commensurate with experience. The Office of the State Auditor is committed to providing equal employment opportunities. Employment actions such as recruiting, hiring, training, and promoting individuals are based upon a policy of non-discrimination. Employment decisions and actions are made without regard to race, color, gender, religion, age, national origin, ancestry, sexual orientation, gender identity and expression, disability, military status, genetic information, political affiliation, or veteran’s status. Job: *Information Systems and Technology Organization: *Office of the State Auditor Title: Senior AI Engineer Location: Massachusetts-Boston-1 Ashburton Place Requisition ID: 260007JK
