Overview
Imagine making AI coding tools more effective for tens of thousands of Microsoft engineers - that’s what you’ll do on the Engineering Thrive Insights team. You’ll join the team behind the data platform, metrics, and insights that measure and improve engineering at Microsoft, spanning developer satisfaction, change velocity, build quality, and AI tool impact. Your primary focus will be the AI Data Flywheel, our platform for collecting signals from engineers using AI coding tools and transforming them into actionable insights and experiments that drive continuous product improvement.
In this role, you’ll define success metrics, design and analyze experiments, develop statistical and machine-learning models, and uncover opportunities to improve AI-assisted software development. You’ll build rigorous measurement frameworks, evaluate product changes, and translate complex behavioral and product data into recommendations for engineering and product teams. You’ll deepen your expertise in experimentation, causal inference, developer productivity frameworks such as DORA and SPACE, AI product evaluation, and privacy-conscious analytics - while using the latest AI tools to help shape the next generation of developer experiences at Microsoft scale.
Responsibilities
- Leverages understanding of data science and business to examine a project and consider factors that can influence final outcomes within a technical area. Evaluates project plan for resources, risks, contingencies, requirements, assumptions, and constraints. Documents key business objectives. Effectively communicates business goals in analytical and technical terms. Consistently shares insights with stakeholders.
- Collaborates with immediate team members on defining metrics and frameworks for evaluating somewhat complex AI/large language model (LLM) systems and agentic product performance. Experiments with model before selection (e.g., LLM, prompting approach) for tasks within a product or solution Diagnose basic issues and identifies opportunities to drive improvements within AI/LLM systems within domain area. Prototypes and develops AI models using containerized environments (e.g., AI Azure Foundry, LangChain) to operationalize, build, and validate scalable tools/applications/solutions for use by the immediate team. Develops and deploys systems for basic retrieval-augmented generation workflows, assisting in the selection of data sources, retrieval systems, and generative models to ensure quality output. Assists senior members in assessing and refining LLM systems and models.
- Understands where to acquire data necessary for successful completion of the project plan. Utilizes querying, visualization, and reporting techniques to describe acquired data, including format, quantity, identities, and other surface properties. Explores data for key attributes and contributes to the development of data quality report describing results of the task, initial findings, and impact on the project by leveraging appropriate AI tools (e.g., Codex, GitHub Copilot). Collaborates with others to perform data-science experiments using established methodologies, statistics, optimization, and probability theory for general purpose software and statistical packages. Assesses different tools and techniques and selects the appropriate one. Serves as an effective partner in data preparation efforts to Solution Architects, Consultants, and Data Engineers. Adheres to Microsoft's privacy policy related to collecting data, preparing data, and using AI tools (e.g., Codex, GitHub Copilot). Identifies data integrity problems.
- Leverages knowledge of modeling and machine learning solutions (e.g., state-of-the-art statistical analysis, causal modeling, reinforcement learning) to identify the best approach to complete objectives. Understands modeling techniques (e.g., validation, hypothesis testing, optimal use of data) and machine learning in order to select the correct approach to prepare data, train and optimize the model, and evaluate the output for statistical and business significance. Understands the risks of data leakage, the bias/variance tradeoff, methodological limitations, etc. Writes all necessary scripts in the appropriate language: T-SQL, U-SQL, KQL, Python, R, etc. Constructs hypotheses, designs controlled experiments, analyzes results using statistical tests, and communicates findings to business stakeholders. Effectively communicates with diverse audiences on data-quality issues and initiatives. Understands operational considerations of model deployment, such as performance, scalability, monitoring, maintenance, integration into engineering production system, stability. Develops operational models that run at scale through partnership with data engineering teams. Understands and keeps up to date on embedding models and vector databases.
- Learns and understands the current state of the industry, including knowledge of tools, techniques, strategies, and processes that can be utilized to improve process efficiency and performance. Maintains knowledge of current trends within the discipline. Attends internal research conferences and participates in on-hands training, when appropriate. Actively contributes to the body of thought leadership and intellectual property (IP) best practices.
Qualifications
Required Qualifications:
- Doctorate in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field
- OR Master's Degree in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 1+ year(s) data-science experience (e.g., managing structured and unstructured data, applying statistical techniques and reporting results) or consulting experience
- OR Bachelor's Degree in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 2+ years data-science experience (e.g., managing structured and unstructured data, applying statistical techniques and reporting results)
- OR equivalent experience.
Other Requirements
- Ability to meet Microsoft, customer and/or government security screening requirements are required for this role. These requirements include but are not limited to the following specialized security screenings:
•
- Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud background check upon hire/transfer and every two years thereafter.
Preferred Qualifications:
- Doctorate in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 1+ year(s) data-science experience (e.g., managing structured and unstructured data, applying statistical techniques and reporting results) OR Master's Degree in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 3+ years data-science experience (e.g., managing structured and unstructured data, applying statistical techniques and reporting results) OR Bachelor's Degree in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 5+ years data-science experience (e.g., managing structured and unstructured data, applying statistical techniques and reporting results) OR equivalent experience.
- Experience with observational causal inference methods, design of controlled experiments, regression modeling, and hypothesis testing.
- Experience evaluating LLMs and/or AI agents - designing offline/online evaluations, defining quality metrics, and measuring model behavior.
- Experience designing surveys and analyzing response data
- Proficiency in Python programming
- Proficiency in analytical query languages like SQL, KQL (Kusto Query Language), or similar analytical query languages.
- Experience developing analytics on Spark-based big data platform technologies such as Databricks.
- Data storytelling - crafting and presenting compelling data stories to business decisionmakers
- Familiarity with developer productivity metrics and frameworks (e.g., DORA, SPACE, or similar).
- Demonstrated effective use of AI coding assistants in daily data science development workflow
Data Science IC3 - The typical base pay range for this role across the U.S. is USD $102,100 - $202,200 per year. There is a different range applicable to specific work locations, within the San Francisco Bay area and New York City metropolitan area, and the base pay range for this role in those locations is USD $133,800 - $219,200 per year.
Certain roles may be eligible for benefits and other compensation. Find additional benefits and pay information here: https://careers.microsoft.com/us/en/us-corporate-pay
This position will be open for a minimum of 5 days, with applications accepted on an ongoing basis until the position is filled.
Microsoft is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to age, ancestry, citizenship, color, family or medical care leave, gender identity or expression, genetic information, immigration status, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran or military status, race, ethnicity, religion, sex (including pregnancy), sexual orientation, or any other characteristic protected by applicable local laws, regulations and ordinances. If you need assistance with religious accommodations and/or a reasonable accommodation due to a disability during the application process, read more about requesting accommodations.
