My Needs Assessment & Usability Testing course in my second semester at UMSI again paired us with outside clients, this time to help them with UX research projects. My team was paired with IBM, who had recently replaced their developerWorks site with a new site called IBM Developer. The new site wasn't getting as much user engagement as they'd hoped, particularly for the ‘Community’ section, and my team was tasked with learning about the needs and habits of their users and evaluating the usability and features of the website in order to come up with recommendations on how they could increase user engagement.
Project Overview
-
Client name: IBM
-
My roles: User research, comparative analysis, user testing
-
Tools/skills utilized: Interaction maps, user interviews, user surveys, heuristic evaluation, preference tests, usability testing
-
The key problems to solve: What goals did users have in mind when they visited IBM Developer? What were the features they used the most? What tasks did they have the most trouble with?
Methodology & Findings
We started by exploring the Developer site and creating an interaction map, which helped us gain a general understanding of its content and structure. Our initial impressions were that while the content was good, navigation between different sections could be confusing. (A full-sized version of our interaction map can be seen here: Interaction Map.pdf.)
Next, we conducted user interviews; due to time constraints, we were unable to get in contact with any actual IBM Developer users, so we interviewed local computer science students and full-time programmers about their usage of developer sites (and other adjacent resources) in general.
-
Question-and-answer forums were the most popular resources on developer sites, and many of our interviewees preferred using Google to locate content on specific websites rather than navigating to it organically or using in-site searches​; you can read our report on our interviews here: C2W2_Interview_Research_Report .pdf
From this data, we created three different user personas: the 'Traditional Developer', who looks for answers to specific problems as they come up and may have difficulty keeping up with new changes; the 'Explorer', who codes in more languages and enjoys learning new things about programming in their spare time; and the 'Visual Developer', who is similar to the Traditional Developer, but with an emphasis on preferring video tutorials to written articles or forum posts.
Then, to see how IBM Developer stacked up against its competitors, we performed a comparative analysis, using a scaled matrix to rate the adjacent websites our interviewees mentioned (including other developer sites as well as Stack Overflow and Github) on categories like features offered, ease of navigation, and visual consistency. You can read our comparative analysis here: Comparative Analysis.pdf.
We then sent a survey out to registered IBM Developer users, asking general questions about their needs and habits as well as more specific ones focusing on issues that had come up in our observations and interviews, and received 54 responses. Our survey report can be found here: Survey Report_C2W2.pdf.
-
Keeping with our our interview findings, the most popular resource on IBM Developer was the question-and-answer forum, which both we and many of our users felt was too hard to find, with the link being buried in the footer and labeled ‘Answers’, making it sound more like a FAQ page than a forum
​After the survey, we used Jakob Nielsen’s 10 usability design heuristics as described in Usability Inspection Methods (1994) to perform a heuristic evaluation of the website and conducted a preference test for our main proposed design changes. This involved creating some simple mockups and sending out a survey to the UMSI community asking respondents to find certain pages on the actual website and then compare it with our mockups; we got 10 responses, largely in favor of our changes.
-
Specifically, we tried renaming the ‘Answers’ link to the more self-explanatory ‘Forums’ and moved it to the main navigation bar (mockups pictured to the right), as well as re-styling the forums page to look more consistent with the rest of the site and adding a button to return to the main Developer homepage; you can read about our methods in detail and see more of our site mockups here: C2W2_Preference Test report.pdf
Finally, we conducted in-depth usability tests with four programmers and CS students we knew, having them each complete four different tasks and talk us through their thought processes.
-
Participants in our user tests thought the content of the site was very good and that most pages were well organized, but they still had trouble finding the Answers forums and got frustrated with the in-site search and its insufficient filter options to narrow down results​; more details can be found in our user testing report here: User Testing report.pdf
Project Reflection
Outcome
Our final set of recommendations for IBM was that they should make the Answers forum much more prominent on the homepage, rename it to something more indicative of a user forum, and improve the search function by adding more filters; see our video for a summary of our methods and recommendations:
Challenges
Perhaps the biggest obstacle was the somewhat rushed timeline of the course, which was particularly a problem for any stages that involved getting in contact with actual users of the website, given that we didn't have direct access to them and had to reach out through our contact at IBM. In an ideal world, we would have been able to include current users of the website in all stages of our user research, rather than only for the user survey.
Additionally, looking back now, I think we fixated far too much on the problem with the Answers forum, revisiting and re-confirming our ideas about it several times. We’d felt it was important to focus on at the time because we knew it was one of the most used features, but we ended up largely neglecting the Community section; our respondents ranked its features as lowest in importance, and in hindsight it seems obvious that we should have done more probing into why they did not care more about that content and how it could be made more relevant to them.