This article is a summary of a quantitative usability testing of hidden menus (such as hamburger icons) and visible menus (such as links across the top of pages), which reveals that:
Responsive design has many merits; however, one of its less fortunate effects has been an excessive popularity of mobile-first designs that translate poorly onto larger desktop screens. To be clear, “mobile-first” itself is not the issue. Rather, the issue is taking sufficiently good mobile designs and porting them to desktops where they do not work as well. Mobile-first should not equal mobile-only.
Hidden navigation, such as the hamburger menu, is one of the many patterns inspired by mobile designs. Screen space is a precious commodity on mobile. To meet the challenge of prioritizing content while still making navigation accessible, designers commonly rely on hiding the navigation under a menu — often indicated by the infamous hamburger icon. Like a cheap fast food chain, it got designers addicted to its convenience, and now serves millions each day, both on mobile devices and on desktops.
Following summarizes a study by WhatUsersDo (a company specializing in remote usability testing), which ran a study with 179 participants who completed tasks on 6 different sites, both on smartphones and desktops.
To determine the effectiveness and usability of the different menus, we ran usability tests on the desktop and mobile versions of 6 different live websites with different types of navigation designs. Here are the types of navigation that we considered:
Note that, for mobile devices, we did not fully test the visible-navigation condition. (Visible navigation is less common on mobile devices, and, due to other constraints in our study, we were not able to include this condition.) We tested the hidden and combo navigation conditions on mobile.
We wrote 2 simple tasks for each website. Some tasks could be completed without the use of the general site navigation, while other tasks required the use of navigation (or search). Here’s an example of a task:“Go to http://www.bbc.com and see if there is any information on nature or wildlife you are interested in. Once you find a particular article that is interesting, please say which one it is.”
To measure the effectiveness of these designs, we collected 5 metrics:
The first 2 metrics are related to navigation usage, while the last three refer to the quality of the user experience.
To quantify the navigation usage, we looked at (1) navigation use: whether navigation was used during a task; and (2) time to navigation: the time from the beginning of the task until the moment when the navigation was used. (We only counted time to navigation if the navigation was used.)
These metrics tell us whether the navigation is helpful and also whether it is easy to discover. If the navigation is salient, you’d expect it to be used more and sooner than if it was hard to discover.
Our tasks did not direct participants to use the navigation, search, page links, or any other interface features. Rather, people could use whatever they wanted in the interface to do the task. We assumed that if navigation is visible and seems helpful to users, they will click it; if not, they will try other things. Thus, we recognize that not clicking a menu can simply reflect that users perceived something else to be a better choice, and not that they did not notice or recognize the menu.
Our results indicate that, across both these measures, hidden navigation was significantly less discoverable than visible or combination navigation:
We ran a mixed-effects logistic regression and found that these differences were statistically significant. This result is not surprising: when (all or some of) the navigation options are visible, they are more salient to users. If nothing else, the screen footprint of visible or combo navigation is larger than that of a menu represented by just an icon or a label, so people will notice it more.
2.People were significantly more likely to use the navigation (whether hidden or partially visible) on mobile than on desktop.
There are many potential explanations for this finding:
Typing is harder and more error prone on mobile, and people may be more likely to prefer navigation instead of searching, when the navigation is available.
Although the numbers seem to indicate a device effect (that is, that participants were faster to use the hidden navigation on mobile than on desktop) or an interaction (that is, that the impact of hidden navigation was higher on desktop than on mobile), these effects were not statistically significant.
An additional few seconds spent may seem petty, and we agree that it will not be the difference between a fine and a terrible user experience. But in those few moments the user may begin to experience frustration and self-doubt. Add on a few more seconds and the user may abandon the task, or at least the menu.
The other 3 metrics that we collected focused on the quality of the user experience:
Our findings show that, across all 3 different metrics, hidden navigation significantly decreases user experience both on mobile and on desktop.
1.Both on mobile and desktop, the content discoverability was significantly lower when the navigation was hidden. This measure showed a more than 20% drop in discoverability on sites with hidden navigation, compared with sites with visible or combo navigation. In other words, visible or combo navigation made people more likely to complete the task successfully and without relying on search.
In the first section of this article, we reported that mobile users were significantly more likely to use the navigation than desktop users (see Figure 1). Here we discuss desktop–mobile differences related to the user-experience metrics.
1.Hidden navigation is more likely to reduce content discoverability on desktop than on mobile.In other words, when users had to discover content that required the use of navigation, they were more likely to fail on the desktop than on mobile. Moreover, the drop in success was greater when the navigation was hidden. This result can also be explained by desktop users’ increased propensity to use search: search is more salient (most of the time) on desktop and users may be more inclined to use it when a path to the content is not immediately apparent.
2.Desktop users were faster than mobile users (when combining all types of navigation) On average, mobile users took 12% more time to complete the tasks. This result is not surprising: due to the small screen and the narrow communication channel between the user and the device, getting the same content on mobile usually requires more interaction.
Why does hidden navigation have these effects? Throughout the article, we hinted at several answers:
Of these 5 points against hidden navigation, only the last one might decline in future years. Hypothetically, increased design standardization could also occur (which would also increase familiarity), but our experience observing the web design evolve over the decades doesn’t make us optimistic that strong design standards will emerge anytime soon. The 3 biggest points of low salience, low information scent, and extra work will always be a curse on hidden navigation that will reduce its usability.
What are the takeaways of this research for your next design project?
Desktop: There is plenty of screen real estate for displaying navigation options, and we recommend:
Mobile: For phones, there is no hard-and-fast rule. But our general recommendations are: