GitHub Pull Request Usability Study
Helping developers collaborate better
Overview
In collaboration with GitHub, my team and I conducted a usability study to evaluate the effectiveness and intuitiveness of their recently updated Pull Request interface, with a focus on the newly introduced UI enhancements. The study aimed to assess experienced GitHub users' learning curve, their ability to efficiently navigate the platform and complete code review tasks, and to gather valuable feedback for improving the overall developer experience.
Role
Primary Point of Contact
Usability Analyst
User Researcher
Methods
A/B Testing (remote)
System Usability Scores (SUS)
Duration
10 Weeks (Jan.2024 - Mar.2024)
Tools
Figma
Zoom
Microsoft Excel
Problem
To ensure a smooth transition for existing users, GitHub recognized the importance of testing the usability and learning curve of their upcoming updated Pull Request feature.
But firstly,
What is a “Pull Request”?
This study primarily focused on evaluating the ease of use and adaptability of the new feature for current GitHub users, aiming to identify potential challenges and gather insights to refine the user experience before the official release.
To thoroughly evaluate the usability of the updated Pull Request feature, the study aimed to address the following:
(Hover to reveal)
Methods and Procedures
To gain unbiased insights we split our participant pool into two categories during the study:
Condition 1:
Explored the existing UI first and then went to the new UI
I was the primary point of contact with the GitHub Client and was responsible for ensuring timely progress reporting at designated intervals. I also played a major role in moderating the interviews and data analysis.
Challenges in Study Design and Iteration
1.
2.
Time on Tasks
Think aloud
Creating a “realistic” GitHub Repository
Qualitative Data
Overview - Quantitative
How?
Remote A/B Testing (within-subject design)
Predefined tasks
Post-test questionnaire (SUS)
System Usability Scale Condition breakdown
Data Analysis
We performed insight analysis based on the user feedback and observations during the test. We then coded the data and came up with high-, medium-, and low-priority findings.
High Priority Insights
Finding and posting a comment
3/6
Participants faced difficulties in locating the comment widget in the new UI:
On larger screens, the visibility of comments becomes compromised, affecting usability
Avatar colors were camouflaging with the background green, contributing to the difficulty in locating the option to post a comment
Recommendation:
Maintain comments on the left side of the interface for intuitive commenting
Consider flexibility for the user in how the comment box is viewed.
Ensure high contrast between avatars and background colors to enhance accessibility.
Resolving a comment
4/6
Participants found it difficult to resolve comments:
Although, it utilizes a clear and well-labeled button for comment resolution, the button color being gray potentially indicates disabled functionality
Quantitative Data
To quantify our data, I also measured the SUS scores of the participants in both conditions and compared the scores to understand the learnability.
Findings and Recommendations
“I’d like it if I can add the comments from this side (left side) and not that very end, having it this side (left side) is so much easier” - Participant 2
Existing UI
New UI
“I think this is the button, but I expected to see the the resolve button somewhere under the comment” - Participant 3
(Hover to reveal)
What? (Data)
Task errors
Task completion
Participant feedback
System Usability Scale (SUS) scores
Condition 2:
Explored the new UI first and then went to the existing UI
Initially, our intention was to include time-on-task measurement as a success metric. However, due to the existing users' familiarity with the current UI, making comparisons with the new UI proved impractical. As an alternative, we implemented a think aloud approach during testing.
Given that our participants were software developers, it was imperative to recreate a "realistic" environment akin to their daily experiences. Consequently, I created multiple repositories with diverse code branches to mirror this environment accurately.
6 Software developers (SD)
3+ years of SD experience
+1 years of GitHub experience
Who?
Recommendation:
Enhance visibility by using a distinct color (like green) for the resolve button/icon to indicate its active state and improve user engagement
Maintaining text in the button to assist users in understanding the functionality of each interface element
Effective Reminder Communication is important.
Medium and Low Priority Insights
Existing UI
New UI
Appreciation for Merge Status Visibility.
Users valued the clear visibility of the merge status.
Contrast Challenges in Light and Dark Modes.
Issues with visibility and usability varied between light and dark modes.
Learnings and Reflections
Discoverability Issues for Suggestion Feature.
Users had trouble locating the suggestion feature when prompted.
Effective reminder communication is essential for reducing no-shows. Emphasizing the necessity of sending frequent reminder emails can significantly minimize absenteeism. Additionally, developing a robust communication strategy is crucial for enhancing participant engagement and ensuring better attendance.
The Significance of Anticipation and Flexibility in Research
The importance of anticipation and flexibility in approach is a key learning. From the outset, it is essential to incorporate crucial variables and proactively identify potential challenges to refine the study design. Remaining open to pivoting strategies based on emerging insights and challenges ensures a more adaptable and effective study process.
The Importance of Transparent Collaboration with Client
Clear collaboration with the client proved to be crucial. By constantly keeping the client in the loop, we were able to navigate the study smoothly. Working closely with the GitHub Pull Request team allowed us to better understand the complexity of the project, enabling us to develop a comprehensive and detailed study plan.
Product Context is crucial
Understanding the product context proved to be pivotal during our work on the GitHub software. Prioritizing comprehension of the pull request feature's intricacies and its various contexts was essential. Therefore, we invested time in familiarizing ourselves with the interface, a step that proved invaluable throughout the study.
View more while you are still here:
UX Research | Qualitative Analysis | Interviews | Observational Studies
Understanding educators’ perspectives on the role of LLMs in academic settings.
UX Research | User Interviews | Surveys | Mobile UI
To explore how generative AI tools can be integrated into the YT Shorts creation workflow to enhance, not replace, the human creative process.
