Gina Dobrescu and Jim Hall
“Thou shalt make thy program’s purpose and structure clear … for thy creativity is better used in solving problems than in creating beautiful new impediments to understanding.”
~ “The Ten Commandments for C Programmers” by Henry Spencer
How easily can you use your computer? Today, the graphical desktop is our primary way of doing things on our computers; we start there to run web browsers, music programs, video players, and even a command line terminal. If the desktop is too difficult to use, if it takes too many steps to do something, or if the cool functionality of the desktop is hidden so you can’t figure out how to use it, then the computer isn’t very useful to you. So it’s very important for the desktop to get it right. The desktop needs to be very easy for everyone to use.
Whenever we think of how easily you can use a program, we’re really talking about the usability of that program. Developers sometimes discount usability and think of usability as making things look nice instead of adding new, useful features. Other times, developers assume usability is too difficult, something that only experts can do. But usability is a very important part of software development.
Usability is even more important in open source software, because users often experiment with the software with no one to guide them. In such cases, the software must be intuitive and functionality must be obvious. If the software is not easy to use, if the features are not immediately accessible by a few clicks, then new users will be turned off. Open source software must have good usability if we want others to use it, and if we want more people to prefer open source software over proprietary software.
How to test usability?
Developers can use any number of methods to examine the usability of their software. From simple surveys to full usability tests, developers should simply use any method that highlights trouble spots and confusing user interfaces in the software. But it is hard to top a formal usability test: watching real people use the software, and noting what they do and where they seem confused. Developers can learn a lot about the usability of their programs just by watching a few people using the software, and making note about what they do and how they try to use it.
A formal usability test is very straightforward and doesn’t take much time. The overview is only three simple steps:
Who are your users?
Start by considering what types of people use your program. Is your program intended for general users with average knowledge? Or is your program going to be used by experts who need advanced functionality? You might write down descriptions of the people who might use your program.
Why are people using your program?
Use the descriptions of your users to better understand not only who is your intended audience, but what they will use your program for. If your program is a music player, will your users want to simply play music files on their computer, or will they also want to listen to streaming radio stations? Will they want to rip music from CDs to listen to later?
What do you want to test?
Once you have a general idea why people will want to use your program, then build a test around that. A usability test is a series of small tasks that you ask people to do. These tasks usually provide a brief context to set up a situation, then ask the tester to do something specific.
Find a few volunteers for your usability test. Most usability tests generate useful results after about five or six testers, so you don’t need many volunteers to help you. Set aside some time for them to do each task, one at a time, and watch how they use the software. What buttons and icons do they click on? What menus do they access? Observing testers in this type of usability test will quickly inform you about problem areas in the user interface, where users are likely to become confused in how to use the software. You can then improve these areas, and make the software even better.
Usability in GNOME
During a Summer 2015 internship with Outreachy, an organization that helps underrepresented groups get involved in free and open source software, we conducted a usability test of the GNOME desktop. This was a formal usability test, where we invited a dozen testers to use GNOME and GNOME applications to complete a few simple tasks.
Test volunteers were about evenly divided between men and women (slightly more men than women) predominantly at college age (12 to 25) representing all levels of computer skill but describing mostly “constant” computer use. About half of testers had not used GNOME previously (slightly more had not than had).
Participants were free to choose the language version of GNOME. Five of them chose to test GNOME in French, while the other seven used GNOME in English. The choice of language did not appear to influence the results of the usability test.
In our test, we presented each tester a set of sample tasks, one task at a time. Throughout the test, we watched each volunteer as they completed the sample tasks, and noted any problems they had with the software. We asked the testers to speak aloud during the usability test, to describe what they were looking for; if they were looking for the “Print” button, they should say, “I’m looking for the ‘Print’ button.” After each set of tasks, we took a “comment break” so participants could share their thoughts about the software and the problems they encountered.
The total test duration varied, with the shortest test at thirty minutes and the longest at an hour and a half.
To provide an overview of the usability test results, we used a heat map: a grid that indicates how easy or difficult it was for the participants to accomplish a task. Each column represents a different tester, and each row represents a specific task each tester was asked to perform. Each colored cell in the heat map represents the relative ease (or not) for the tester to use the software to accomplish a task:
Heat map of usability tests.
The heat map method presents a holistic view to the usability test. Note the “hot” rows that contain significant black, red, and orange:
Save a folder location (bookmark)
Evince + Characters
Save title location
Add a note containing a special symbol
Save file modifications
Other rows are mostly green with some red and orange, suggesting these are areas that require some improvement in usability:
Search for a fileDisplay files as list
Evince + Characters
Save file modifications
Rows that have mostly green with only a little yellow represent tasks that have good usability. These seem to perform well for testers, so do not need attention.
What went well
Participants quickly became familiar with Calendar’s interface, so they all managed to accomplish the two tasks, althoug users who encountered little difficulty during these tasks were a little bothered by some aspects of the interface.
Testers also had an easy time during the Image Viewer tasks, and encountered little difficulty accomplishing them. Unlike experienced users, GNOME beginners discovered another way to zoom into an image, by entering into the full screen mode and using the “+” and “-” top buttons in order to adjust an image.
Almost half of the participants on this usability test had never used GNOME before, but it was very interesting to observe them during the test as I discerned a common behavior pattern . For example, when they discovered the three line menu (sometimes called a “hamburger menu”) in Nautilus, they understood that they could select many options from there. Or when they wanted to launch the slideshow animation in Image Viewer, their instinct was to go directly to this menu.
In the Nautilus file manager, testers used different methods when asked to search for a file. Some clicked on the magnifying glass and then introduced the name of the file, others used the ctrl-f keyboard shortcut, and a few chose to manually click into each folder to locate the files. Regardless of the method, all testers managed to find files quickly using Nautilus.
Testers also found the Calendar application very easy to use. In the comment breaks, testers reported the program responded the way they expected. For example, they could easily create new meetings with specific start times and attendees.
Where GNOME can improve
Testers struggled through several sections of the usability test. We can learn much from observing the testers as they encountered difficulty in hidden behavior or inconsistent feedback. These provide opportunities for GNOME to improve its usability. The usability issues can be grouped into four general themes:
Functionality should be obvious.
In Nautilus, most of the testers did not manage to accomplish the fifth task (bookmark the location of a folder). One of the GNOME beginners did it by accident (drag and drop) and said he would never have thought about this method of adding a bookmark. Almost all the users expected the sidebar area to be more explicit and people expressed a desire to see titles for each category of folders (personal, network, bookmarks, etc) in this area.
People who had never used GNOME before found the Nautilus “magnifying glass” button confusing and did not think of using it when searching for a file. These testers only discovered its functionality later. For most of them, the “magnifying glass” symbol represented the zoom functionality. Also, when they were told to make the icons bigger, these testers got a little confused about the option “order by size” from the menu button.
In Evince, the only way to get to Annotations is by displaying the sidebar panel on the left side of the window, and then selecting the right option from the dropdown menu. But sometimes participants closed the sidebar panel by mistake and did not know how to get to it again, which is when most of them gave up this task. Even when they discovered this option, some of the participants could not understand how it worked, as there was no indication on how to use it.
In the Characters application, testers found it difficult to locate a specific symbol. When asked to use the check mark symbol from the “Dingbats” font in the new Characters application, some of the testers used the magnifying glass in order to search for the name of the font. They got a little confused because the search bar’s purpose is not mentioned anywhere. So after they discovered that the fonts were in the hamburger menu, they found the right symbol by browsing through all the available categories.
In general, participants encountered most of the problems in the “menus” area from the header bar (hamburger menu, magnifying glass), as they were not explicit enough for them. We want to highlight this problem, because when testers did not know what a button or other functionality meant, almost all expected a hint when they put their cursor over it, especially those testers who had never used GNOME before.
Features need to act consistently to each other.
Bookmarks caused problems in Evince, and half of the users failed to accomplish this task. In Evince, there are two ways to add a bookmark: by using the option from the hamburger menu, or the one in the sidebar pane. Experienced users who used the option from the hamburger menu also discovered the one in the sidebar panel. These testers complained about the lack of “consistency” between these two methods, as they do not function the same way. For example, a bookmark cannot be deleted from the hamburger menu, whereas it can be deleted from the side pane.
And in the Calendar application, changing the month/year of the current view caused confusion for some testers. In order to change the current month, they had to use the arrows on both sides of the “Today” button. In general, people expected these buttons to change the current day.
Activity should provide clear feedback.
In the Nautilus file manager, many testers had problems displaying files as a list. Most of the participants tried to right-click within Nautilus, or attempted to use the hamburger menu from the top-right corner of the window. A few testers discovered the Preferences menu option from the application menu, where they could modify the “Default View” parameters to a “List View” from a dropdown menu. Every participant who chose this method complained that there was no “Ok” or “Apply” button and they waited for immediate feedback (which did not happen in this case). This made them feel confused and they made comments like “Why nothing is happening?” or “This is not working.”
In order to make the icons bigger in Nautilus, testers either used the Preferences menu item (but there was no visible effect) or tried to use the slider from the top-right menu. For those who chose this method, many started to drag-and-drop the slider, but were disconcerted by the lack of “visual feedback” as they moved the cursor to the left or right. Moreover, the absence of symbols like + or – to explain the function of the slider made some of them give up.
Testers were similarly confused when trying to save the location of a folder for later use in Nautilus. You may recognize this feature as a “bookmark.” A few testers tried to drag the folder into the upper part of the left sidebar, but only managed to move the folder into another folder such as Home, Documents, or Downloads. Some accidentally dragged the folder into the Bookmarks section, which is correct, but did not realize that the action was successful.
Some testers used the hamburger menu to bookmark a folder in Nautilus, but they were not sure what happened. While their attention was in the top-right corner of the window, they did not observe what happened on the left sidebar to create the bookmark.
In the Characters app, most of the people who had never used GNOME before got lost when they had to copy the check mark special character. Testers did not understand what happened when they clicked on “Copy Character” (or if the button really worked). They complained that there was no visual feedback when clicking on this button. We think this happened because the testers were not used to this type of application, no matter the system they usually use.
Make wise use of menus.
In Evince, testers were confused when adding a note in a PDF document. Only one participant successfully completed this task, as he uses this feature at his work. After exploring the top-right corner menus, all testers hoped (again) to find a solution by trying to right click on the document. Participants seemed really lost when trying to solve this task. At a given moment, some of the testers tried almost everything, and they eventually discovered the “Annotation” feature in the dropdown menu from the sidebar panel.
Testers also struggled when they had to bookmark a specific page in Evince. Most of them missed the “add a bookmark” option from the hamburger menu, as the list of options from this menu starts with words related to file manipulation (save a copy, open a file, etc.). So whenever they opened this menu, they did not want to read all the items of this list. Most of the time, they made comments like “It’s not here, I have to look somewhere else” or “This should be obvious, I would like to know how to do it.” Most of the users who discovered this option and selected “Add a bookmark” from the list, were complaining about the fact that there was no visual effect, so they were not sure if the page was really saved somewhere.
Learning from GNOME
In open source software, developers often place highest priority on creating new features and adding new functionality. But the most feature-rich open source software program is useless if its interface is so difficult to use that no one can access the critical functions. To make their programs easier to use, open source developers must also consider the usability of the programs.
Unfortunately, the concept of usability testing sometimes runs counter the the open source software ethos, where most developers start by solving a problem that is interesting to them and then release it for others to use. But usability testing is not difficult to do. Anyone can do it. All you need is a little time and patience; watching a few testers use your software will help uncover areas that are difficult to use, so you can improve the interface and make it something that everyone will want to use.
In the absence of a formal usability test, you can take a shortcut to good usability by learning from usability testing in other programs: where those programs fare well, and where those programs need improvement. In my usability tests in GNOME, I find open source developers can get quick wins by observing these simple guidelines to user interfaces:
- Functionality should be obvious.
- Features need to act consistently to each other.
- Activity should provide clear feedback.
- Make wise use of menus.
These usability themes will make most open source programs easier to use. They provide general rules to good usability. And with good usability, everyone wins.
|Gina Dobrescu is a student who is passionate about computer science and open source technologies. She completed an internship in usability testing with GNOME Outreachy in Summer 2015.||Jim Hall, an advocate for free and open source software, and open source software usability, is the CIO for Ramsey County, Minn. He has a master’s degree from the University of Minnesota in scientific and technical communication; his master’s capstone was an examination of the usability of open source software.|