Overview
AI agents are changing how organizations get work done, bringing new demands for reliability, visibility, and trust to the platforms behind them. When an agent spans multiple services, engineers need to understand where work happened, what failed, and how the underlying platform behaved. That makes telemetry part of the foundation for dependable AI, not just a tool for debugging it. The CoreServices Telemetry team, part of Agent 365 Foundation, builds and operates the telemetry platform for Dataverse, CoreServices Fault Domain Islands, and Dynamics 365 Finance and Operations. The broader foundation supports Dynamics 365, Power Platform, and a growing set of Copilot and agent workloads. We own the path from signal collection through processing, storage, querying, and alerting, helping teams understand production behavior and measure service health. Our platform spans virtual machines, Service Fabric, and Kubernetes-based Core Services Containers. We bring together logs, metrics, and diagnostic streams through Geneva, Azure Data Explorer (Kusto), and data lake pipelines. We also build tools that make telemetry easier for engineers and AI assistants to discover and use. The challenge is to keep data timely, accurate, accessible, and cost-effective across regions and sovereign clouds while enforcing privacy and data residency boundaries. We are looking for a Senior Software Engineer to build the telemetry and observability foundation behind business applications and emerging agent workloads. This is a hands-on software engineering role: you will write and debug collection and processing services, improve ingestion and query performance, and ship changes safely across a distributed production fleet. You will also help build AI-assisted investigation tools that connect engineers to the right signals and reduce the work of diagnosing incidents. You will own outcomes rather than tickets. That means deciding how a pipeline behaves when a dependency slows down, how to detect missing or delayed data, and how to balance fidelity, latency, retention, and cost. You will defend those decisions with evidence and take part in the on-call Microsoft’s mission is to empower every person and every organization on the planet to achieve more. As employees we come together with a growth mindset, innovate to empower others, and collaborate to realize our shared goals. Each day we build on our values of respect, integrity, and accountability to create a culture of inclusion where everyone can thrive at work and beyond.
Responsibilities
- Design, implement, and operate telemetry collection and processing services across Dataverse, Service Fabric, Kubernetes, and Finance and Operations. Partner with teams building Copilot and agent workloads to make their platform dependencies observable.
- Own data quality and pipeline behavior: event schemas, filtering, transformations, routing, ingestion latency, completeness, and recovery from delayed or failed delivery. Make delivery guarantees and failure modes explicit.
- Build resilience through bounded buffering, backpressure, retries, circuit breakers, and dependency isolation. Validate failure recovery and capacity through automated tests, load testing, and controlled fault injection, without putting source workload availability at risk.
- Enforce privacy, security, and data residency throughout collection, routing, storage, retention, and access. Build least-privilege controls and auditable operations that respect regional, EU Data Boundary, and sovereign cloud requirements.
- Improve Azure Data Explorer ingestion and query performance, storage efficiency, and capacity management. Use evidence to balance telemetry coverage and retention against compute, storage, and operational cost, and automate safe deployments and configuration changes.
- Instrument the telemetry platform itself with actionable health signals and alerts. Drive incident mitigation, root-cause analysis, and lasting repairs; improve the service-health measurements and diagnostic tools that partner teams use to understand customer impact.
- Partner with service teams on instrumentation, onboarding, query design, and operational readiness. Build self-service tools and documentation, mentor engineers, and raise the engineering bar through design and code review.
Build and improve AI-assisted diagnostics and Model Context Protocol (MCP) integrations that help engineers discover telemetry, generate useful queries, and correlate evidence across services. Evaluate their accuracy and usefulness, respect data-access boundaries, and keep production-changing actions subject to appropriate safe
Qualifications
Required Qualifications:
- Bachelor's Degree in Computer Science or related technical field AND 4+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python
- OR equivalent experience.
Other Requirements:
Ability to meet Microsoft, customer and/or government security screening requirements are required for this role. These requirements include but are not limited to the following specialized security screenings:
- Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud background check upon hire/transfer and every two years thereafter.
Preferred Qualifications:
- Master's Degree in Computer Science or related technical field AND 6+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python
- OR Bachelor's Degree in Computer Science or related technical field AND 8+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python
- OR equivalent experience.
- Solid software engineering fundamentals, including data structures, concurrency, asynchronous processing, automated testing, and diagnosing performance and reliability problems in production.
- Practical experience with distributed data processing or telemetry pipelines, including ingestion, buffering, retries, partitioning, data quality, and the tradeoffs between throughput, latency, and cost.
- Production experience in a modern programming language such as C#, Go, Java, or C++, and the ability to learn the languages and tools used by the team.
- Demonstrated experience making services resilient under stress, validating failure and recovery behavior, and using logs, metrics, and queries to explain system behavior.
- Experience operating what you build, including on-call ownership, live-site debugging, and post-incident improvement.
- Hands-on experience with Azure Data Explorer and KQL, including ingestion, query optimization, retention policies, and managing shared analytics capacity.
- Experience with observability technologies such as Geneva, OpenTelemetry, Prometheus, or comparable collection, metrics, and monitoring platforms.
- Experience operating services on Kubernetes or Service Fabric, building stream-processing or data lake pipelines, and automating infrastructure and deployments in Azure.
- Familiarity with multi-tenant cloud platforms, privacy-aware data handling, regional data residency, or sovereign cloud operations.
- Background in capacity planning, telemetry cost optimization, or measuring service health and customer impact in large-scale cloud systems.
- Experience building AI-assisted developer tools, MCP integrations, or incident investigation automation, including evaluating output quality and applying appropriate access controls and operational safeguards.
#CAP #Agent365 #A365
Software Engineering IC4 - The typical base pay range for this role across the U.S. is USD $119,800 - $234,700 per year. There is a different range applicable to specific work locations, within the San Francisco Bay area and New York City metropolitan area, and the base pay range for this role in those locations is USD $160,200 - $261,000 per year.
Certain roles may be eligible for benefits and other compensation. Find additional benefits and pay information here: https://careers.microsoft.com/us/en/us-corporate-pay
This position will be open for a minimum of 5 days, with applications accepted on an ongoing basis until the position is filled.
Microsoft is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to age, ancestry, citizenship, color, family or medical care leave, gender identity or expression, genetic information, immigration status, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran or military status, race, ethnicity, religion, sex (including pregnancy), sexual orientation, or any other characteristic protected by applicable local laws, regulations and ordinances. If you need assistance with religious accommodations and/or a reasonable accommodation due to a disability during the application process, read more about requesting accommodations.