Case Studies
In a consultative role, I worked with 20+ different teams at Microsoft to design research studies to address their hypotheses and guided them through every aspect of research: From developing the question to training engineers in how to talk to customers, to discussing and analyzing customer feedback. Below are three case studies.
​
Context
The Planner team wanted to gather data about their users' needs for five months before choosing an issue to focus on. As a UX PM, my goal was to help guide them towards a particular research question, towards structured interviews, and finally towards focused product changes.
​
The Planner team decided to focus on labels, a key component of the app. The PM and engineers hypothesized that users wanted more labels, but they didn't know why. We narrowed the research question down to: How many labels do customers want and why? We asked users to share scenarios about labels were used in their organization.
​
Method
-
Conducted a planning meeting with the PM (an engineer)
-
Reframed her research questions
-
Targeted Planner users in-product using telemetry
-
Wrote a discussion guide focusing on labels and open questions for users to describe their use scenario
-
Ran the research sessions with 16 engineers on the call, who took turns asking the questions
-
Led the debrief with team members to discuss customer feedback
-
Summarized the results in a report
​
Findings
-
With only 6 labels available in the app, there weren't enough labels for users to work effectively because a lack of labels stifled creativity and options to list all tasks related to a project
-
Customers wanted at least 20 labels to organize tasks effectively
-
Solution: The Planner team decided to add more labels to retain customers and enhance their options when creating tasks
Case Study 1: Planner App
Case study 2: Project App
Context
The Project App was similar to Planner. It facilitated the management of projects by organizations. Similar to the Planner team, the motivations behind the research were general, and user feedback was collected over a five-month period before the team decided on specific research questions. The questions they decided on were: What motivates users to use Planner more often than Project? Why aren't customers using the web version as often as the client version of the app?
​
Method
-
Conducted a planning meeting with the PM
-
Reframed her research questions from general questions to more specific ones
-
Targeted Project users in-product using telemetry
-
Wrote the discussion guide
-
Talked to 6 customers (numbers varied from session to session)
-
Summarized the results in a report
​
Findings
-
Users found Project more difficult to use than Planner
-
Users described the difference between the two as broad management through Project and pursuing the management of individual tasks through Planner. Project users liked Planner because it enabled them to manage individual tasks better
-
Project users preferred the client (desktop) version of the app more because it was more familiar
​
Recommendations
-
Several users expressed the need for more instructional videos and materials outlining the new features of the web-based version of Project. The recommendation was for the team to provide tutorials for web-based Project users
-
Customers shared a diversity of ideas for new features to be added to the app to enhance their work. Overall, they wanted the flexibility to copy and paste repetitive tasks, to have custom fields, and the ability for project tasks to go through an approval process
-
The recommendation for the team was to conduct additional research to prioritize new features to solve user pain points, such as the features listed above
-
As a researcher, my recommendation was for the team to select more specific research questions and narrow in on specific scenarios for the requested features above
​
​
Case Study 3: What are product teams' goals in using an internal research tool for user research?
Context
The goal of the study was to learn product teams' goals in undertaking user research using our team's internal tool. Secondarily, we wanted to learn how we could support product teams better in using our tool. I undertook an internal qualitative and quantitative research project to better understand how product teams could better utilize my team’s suite of services—all geared towards self-service research.
​
Method
-
Qualitative interviews with 10 individuals from 7 different product teams. Each interview lasted one-hour
-
An anonymous survey including 10 questions on goals, motivations, and results was circulated and received 25 responses
-
The results of the qualitative interviews were analyzed alongside the survey results
Results
-
Relatively few product teams used the research tool initially with a specific research question or hypothesis in mind. The motivations were often generalized to needing feedback before shipping
-
Qualitative interviews showed that several teams were motivated by meeting objectives and key results (OKRs) based on being customer-driven (talking to customers) rather than being motivated by specific research questions or hypotheses. This was in line with leadership goals at Microsoft
-
The study also gained quantitative data around how many teams had made product improvements following the use of our research tool. We learned that few teams had followed through to make product changes based on using our internal tool. This was due to several factors, such as a need to do additional research using usertesting.com to gather more data, due to competing priorities, and due to feedback being too generalized to make decisions
-
The primary blocker to using our research tool was that teams needed more “white gloves” support to use our tools
-
The second issue that I read in these results was that product teams did not come with a clear goal or research question, and therefore could not obtain specific feedback from customers for product changes, especially in cases where they had not followed our guidance to design the user study
​
Recommendations and Outcomes
-
I made it my goal to guide teams assertively in conducting meaningful research, in using our self-service tool to talk to more customers and to offer more assistance in using our software.
-
In my last two months at Microsoft, I doubled the number of internal participants in our service (product teams), and quadrupled the number of external participants (customers), helping the team move more quickly towards our goal of 3,000 external users of our service.
-
This involved coaching other Microsoft PMs daily in research methodology, in planning, and in preparing their team to use our service. In essence, I provided the “white gloves” treatment to internal teams to facilitate ease of use of our research service through effective planning and coaching. This also resulted in more meaningful research.