Predicting Traffic Impacts of New Designs

The Issue

Managers responsible for Web channels often want to know how design changes will affect
user behaviour. While traditional lab-based usability testing can help identify impediments to
successful user experience, the qualitative nature of the research makes it hard to predict how
design changes will affect actual behaviour once the site is launched.

Background

The Canada Site is the Government of Canada’s main portal for government information and services. The site is divided into three main areas (known as Gateways) directed at Canadians, Non-Canadians and Canadian Business. Within these three Gateways, subject or audience-based sections lead users to information and services based on their needs, irrespective of their knowledge of the government’s organizational structure.

The goal of the Gateways and subject/audience clusters is to ensure seamless, one-stop access to information by transcending organizational boundaries, by first aggregating, and then organizing, information in ways that resonate with the situational needs and expectations of Canadian and non-Canadian audiences.

In March 2003, the Canada Site was redesigned to reflect behaviour observed in usability testing and other research exercises. One of the main changes made to the Canada Site home page was the redesign of the Gateway entry points as buttons, with direct links to specific subject areas under the Canadians Gateway heading.

Following the launch of the new Home page in March 2003, click-through reports showed that traffic through the Canadians Gateway decreased dramatically due to the availability of the direct links to popular subject categories [1]. While the site was ultimately more usable for the majority of visitors who accessed these subject areas, the new design had the following unanticipated impacts:

• There was significant variability in traffic for individual clusters from month to month, depending on whether they were featured as a link on the Canada Site home page.
• While users could still access the Canadians Gateway Home Page either by clicking on “more topics” or the Gateway button itself, the presence of the topical links migrated attention away from the general entry points. As a result, users were no longer exposed to the range of topics and services listed on the Canadians Gateway Home Page.

To address this issue, three prototype designs were developed in the winter of 2004 for a revised version of the Canada Site Home Page. Two of these prototypes offered a pull-down menu with a list of topics that led directly to individual clusters. The third concept represented an adaptation of the site posted at the time, whereby topics were listed under the Gateway button, but were not clickable (and, therefore, did not enable the user to by-pass the Canadians Gateway Home Page). In all cases, the intent was to ensure that users were quickly directed to the specific information sought, while at the same time being exposed to the range of other topics that are available on the Canadians Gateway Home Page.

In testing the different options, the Government not only wanted to determine the usability of the prototypes, they also wanted to model the likely impact on traffic prior to re-launch. In particular, they wanted to assess which of the different options best struck and balance between usability and the Government’s desire to inform citizens of the range of services available to them.

Automated Testing and Tracking

To address this challenge, Phase 5 used a unique approach that involved online exercises, whereby we tracked behaviour and usability metrics while recruited participants were completing information retrieval exercises using the different prototypes. We recruited 130 respondents who were asked to complete three task-based exercises with one of the three prototypes. The tasks were self-completed and no human intervention occurred during the process. Rather, the respondent completed the exercises by himself or herself while software tracked their behaviour, recorded success or failure and the number of clicks and time to completion.

A relatively new and innovative approach to user testing of Web interfaces, ‘online testing’ allowed us to test the Canada Site and the Canadians Gateway prototypes with a larger audience and thereby predict with greater certainty user behaviour. It also significantly extended the geographical and demographic reach of the study.

Some of the results of the task completion exercises, in terms of the path taken through the different prototypes, are shown in the table below.

The online usability tests showed that one of the prototypes (#3) was clearly more effective at leading participants through to the Canadians Gateway. The figures in the previous table show that the Canadians Gateway button was chosen by 28% of respondents for Task 1 using prototype 3, compared to only 3% and 5% respectively for prototypes 1 and 2. This number also jumped to between 64% and 70% of respondents who used the Canadians Gateway to access information for Task 2 and 3. This jump is perhaps indicative of levels of familiarity with the design and layout of the site as respondents progressed from Task 1 to Task 2 and 3.

During follow-up in-person usability interviews, it became apparent that the symmetrical layout of prototype 3 better communicated that the gateway buttons were valid navigation options. While a number of participants used the drop-down menu within the Canadians Gateway area on Prototype 1 and 2, the layout and appearance of the gateway buttons themselves did not communicate that they were valid navigation options. As such, participants sought out other navigation options on the Canada Site.

Conclusion

Subsequent to the testing, Canada Site and Canadians Gateway managers re-launched the site, with traffic patterns emulating what was found during the testing. In addition to better understanding traffic impacts of new designs prior to launch, the Canada Site and Canadians Gateway established valuable usability benchmarks (e.g. task completion success rates, time to completion) against which future design changes can be assessed. The online task completion approach enabled them to emulate real-life impacts of site changes and, thereby, better manage towards desired outcomes.

To learn more and to find out how Phase 5 can help you implement a web traffic prediction program, please contact Doug Church at 613-241-7555, ext. 101.

1 In the month after the change, traffic through the Canadians Gateway decreased by approximately 75%.