Summary
The article describes the application of the usability test for the analysis of infor-mation architecture qualitative criteria with particular emphasis on elements of a search system. The objective of the study is the system used to find information in the popular Polish portal Merlin.pl. In terms of the quality requirements, the focus is on end-users, customers of online stores, which is consistent with the UCD approach.
The publication includes a definition of information architecture and a descrip-tion of each criterion. This paper describes the characteristics, implementadescrip-tion pro-cess and application of the usability test for the analysis of information architecture qualitative criteria. The publication contains the results of a study through a usability test on a group of selected experts.
Keywords: information architecture, online store, search system, usability test Introduction
An online store is currently one of the most popular forms of doing business on the Internet. Its effectiveness is dependent upon many external factors such as brand recognition, the volume offered and level of the offered prices. However, the popularity and willingness to use the selected store by Internet users is also affected to a large extent on internal factors of ease of use of the this store. These factors include consideration of various aspects of the discipline known as information archi-tecture. One of the aspects that facilitate network users’ access to information relevant to them is a search system, which should be provided with every major service, such as an online shop. 1. Definition of an online store
An online store is an advanced website that gives the ability to buy products over the Internet. Online shops are one of the forms of electronic commerce and work mostly in the B2C business model; so products offered by companies are directed to individual customers. This form is becom-ing each year more and more popular due to convenience and the lower cost of sales and also allows one to quickly compare prices from various suppliers. An essential part of online stores is the website where clients learn about the offers and make orders [1, 4].
One of the oldest and largest Polish online stores, operating based on a B2C business model is the service Merlin.pl. This service is an electronic agent operating in traditional supermarkets and industries. It allows one to make purchases online from over 200 000 different products in the field of books, music, movies, toys, cosmetics, clothing and sports equipment, multimedia, electronics
40
and photographic equipment [3]. The main objective of Merlin is to offer customers a wide range of products at competitive prices, which can be acquired in a safe manner and without leaving home.
Merlin began its online activities on April 1, 1999. At its inception, the Internet was primarily a bookstore, and over time the offer was enriched with new categories of goods. Currently, the store has a strong and established brand in the Polish market and abroad, and it ships to 70 countries around the world. Merlin has the reputation as a credible and reliable company, which is the result of an unwavering commitment to improving the quality of services. As many as 75 percent of polled Internet users recognize the brand Merlin.pl, and every fifth person buying on the Internet points to this name as one of their favorite stores [3]. Moreover, it has for many years been at the forefront of stores with the highest income, which in 2010 amounted to more than 111 million zł from sales alone [12]. Merlin boasts over one million registered customers: nearly 100 000 a month in orders and the daily number of visitors at the level of 100 000 people [3]. Merlin.pl ranks 9236 in the Alexa Internet ranking of the most visited websites in the world [5].
2. Characteristics of information architecture
Information architecture defines how to organize the infrastructure of the website, to find the information it carried out in the simplest way. Accordingly, it focuses on the following tasks: organ-izing information, naming the various headings, navigation, page break in functional blocks, distin-guishing the most relevant information, combined elements of a visual logic or the visualization of nesting elements.
The scope of the information architecture in websites: Ɣ classification and structure of websites,
Ɣ ways of organizing information, Ɣ ways of naming (labeling), Ɣ use of navigation systems,
Ɣ use of information searching models,
Ɣ use of thesauri, controlled vocabularies and metadata, Ɣ information architecture and graphic design for web, Ɣ methods and techniques to study users of information,
Ɣ methods, criteria and indicators for evaluating the quality of network resources. [7,8,13] Some important criteria for information architecture, to be taken into account when assessing the quality of information infrastructure business portal, include the following groups of criteria:
Ɣ appearance of the service,
Ɣ organization information on the service, Ɣ labels used on the service,
Ɣ navigation on the service,
Ɣ information searching on the service,
Ɣ information content on the service. [2, 10, 13]
Within each of these groups it is possible to extract a list of criteria, from a few to several, sub-criteria. For the purposes of this article, it was selected from one of the abovementioned groups, or “Information searching on the service”. Within this group are the following two information retrieval
Studies & Proceedings of Polish Association for Knowledge Management No. 72, 2014
systems: a simple search interface and an advanced search interface. The list of sub-criteria for these systems is as follows:
偪 Sub-criteria for a simple search interface: Ɣ ease of locating a simple search interface,
Ɣ ease of finding information about the total number of search results, Ɣ ease of use of a simple search interface,
Ɣ ease of the query finding on a page with searching results (e.g. to change it), Ɣ ease of navigating through all of the search results,
Ɣ compliance gained at the beginning (first results page) search results to the query, Ɣ resistance of a search system for grammatical errors entered by the user.
偪 Sub-criteria for an advanced search interface: Ɣ ease of locating an advanced search interface, Ɣ ease of use from advanced search options,
Ɣ ability to search within specific areas / pages / category,
Ɣ ability to search using Boolean phrases (such as AND, OR, NOT), Ɣ ability to search by changing the display of results (e.g. custom sorting). 3. Usability test as a quality evaluation method of information architecture
Of the many different methods of quality evaluation for the purpose of this article, the selected sample method has been successfully applied to the broader evaluation of the quality of websites. The article describes how its use, especially in relation to evaluating the quality of online stores’ information architecture. The chosen method is a usability test and its characteristics are presented below.
Usability test (also called Usability testing) is one of the most popular methods of utility as-sessment, based largely on the approach to user-oriented design (UCD – User-centered design). This method is used primarily to evaluate the functionality and usability of websites. Usability testing have mostly taken the form of laboratory tests in which testers perform tasks that simulate user interaction with the website and specially-adapted computer workstations. Usability testing, unlike the group of tests, is carried out usually on individual users and relies on the practical use of some of service features. During the performance of specific tasks, participants are asked to pay attention to what they see, do and feel and then describe their experience verbally or in writing. The re-searcher's task is to observe and listen to the participants and take notes on how to navigate the site. This study aims to verify the proper operation of the site and to detect areas in which users have any problems, then make recommendations for improvement [15]. The usability test data analysis, in addition to spontaneously – made by the researchers note, may also be used for computer support tools. These can include: an electronic poll form for entering user reviews, user activity recording software in the form of animations or embedded in a web browser scripts for the analysis of clicks and time spent on the site. The most common areas of activity of the analyzed websites, the useful-ness of the test include the following aspects: visibility of navigation, clarity and understanding labels, ease of navigating through the site, ease of finding needed information on the site and the performance of tasks relating to information retrieval [6].
One of the most important factors in the course of testing the functionality of network services is the number of tested individuals, which should depend on the purpose and audience of potential
42
test results. On the one hand, the more testers, the higher the cost of research; on the other, the larger the sample, the greater the probability of detecting all the problems. In practice, it is usually the first users who encounter many errors. In addition, large errors often obscure the smaller and one can just remove them and carry out another round of tests allows the discovery of most errors. Therefore, instead of one test with more users (at least eight), it is better to carry out several rounds of testing (e.g. two or three) with a smaller number of testers (e.g. three to five). As shown by Jakob Nielsen and Tom Landauer, testing web services with five users, allows one to detect 85% of the problems of a site’s functionality. Employing a larger number of users does not guarantee a linear increase in the number of errors detected [9].
Although the usability test is a time-consuming and relatively expensive method, it is flexible enough that, with minor modifications, it can be used for different types of information resources. Furthermore, the results obtained with this method can produce a lot of information that can be used to dissect the quality of those resources. In addition, the great advantage is the fact that potential users can provide such information. Conducting usability testing leads to the creation of a list of errors in the usability of the tested site and gives rise to such a re-structuring of service that its level of functionality meets the expectations of most users. Supplemented by information from the usa-bility tests can also be an analysis of the quantitative data concerning the movement of the users on the site. The most popular ways to measure are: analysis of log files (called logfiles), analysis of traffic statistics (e.g. Google Analytics), analysis of click statistics (e.g. tracking, map, list, etc.) and analysis of data from different business areas (e.g. sales, orders, costs, level of conversion, the num-ber of ads, etc.) [11, 14].
Usability test procedure consists of the following stages [6]: 1. Identification and calling the appropriate test participants. 2. Preparation of the plan and protocol testing.
3. Preparation of the laboratory for observation and data collection. 4. Conducting the test.
5. Analysis of results.
6. Presentation of final recommendations.
Application of the usability test to evaluate the quality of information architecture of an online store relies on using the abovementioned procedure in the design phase of the shop or its operation. In the first case, the testing service is subject to the prototype presented in paper form, models, graphics or HTML documents. Testing of the information architecture at this stage is so important that any early-detected abnormalities such as avoiding unnecessary amendments in the official version of the store operation. When in operation, the utility service is used to test the infor-mation architecture in two ways: locally (in a specially adapted laboratory) or remotely (on the com-puters of users who in the standard way of viewing service during the test). Remote usability testing is a good way to test a geographically distributed user groups.
Studies & Proceedings of Polish Association for Knowledge Management No. 72, 2014
4. Application of usability test to evaluate the quality of searching system
Research and analysis of the search system for the Merlin.pl online store were made by the author of the article at the beginning of 2010. As a method of research, the usability test was chosen. The study was carried out in accordance with the above-described test procedure for this method. The following stages describe the test procedure.
1. Identification and calling the appropriate test participants
To test the quality and usability of the search system in the online shop Merlin.pl, a group of eighteen experts who were students of two universities were selected: University of Szczecin (major: Computer Science and Econometrics) and West Pomeranian University of Technology (major: Computer Science). The task of the experts was to test the chosen service for the accuracy and use-fulness of the applied information architecture that is compatible with the user-centered design ap-proach.
2. Preparation of the plan and protocol testing
Research tests on the experts were conducted using an electronic poll tool, which allowed: con-ducting a poll via the Internet, automatically saving the results of completed polls in the database and later viewing the statistics of the results. Poll forms were divided into parts, which at every step of the procedure were accompanied by used definitions at the time the issues and descriptions of how to fill. Thanks to this, the poll could be delivered to the selected expert in electronic form for self-fulfillment, which greatly facilitated the processing of questionnaires and collecting results. The poll included a sample task to perform in the tested service using the integrated search system and the possibility of an expert assessment of ease of performing these tasks. The task of the experts was to make the set of poll tasks and their assessment, which formed the basis for data collection on the quality of the search system used in the tested online store.
3. Preparation of the laboratory for observation and data collection
Studies on the experts were conducted on computer workstations with access to the Internet. Some of the research was conducted in the presence of the article’s author on a specially equipped computer workstation, while the remainder was held in areas accessible only to experts (e.g. at school, at home or their place of work). The occurring variety of hardware configurations and soft-ware installed on computers did not cause any failures or problems in completing the electronic poll form.
During the study, participants used the Internet browser (to read the questionnaire and online store pages) and in the background work a program for recording the performance of their activities. Using this type of application is one of the essential factors in the collection of qualitative data about the test environment using the usability test. The results of the recording application and all the reactions of experts to the test conditions, were saved in the form of video animations, which were then used as additional information to analyze the usefulness of the search system in the Merlin shop.
44
4. Conducting the test
The main objective of the usability test by using the Merlin.pl service was the analysis of a search system. As ease of use of this system is well known and its accuracy is the determining factor in the ease of access to the information sought by users, which in turn affects their satisfaction and willingness to use the shop in the future. Therefore, subjects were given by experts to carry out the most common tasks performed by users of search engines. These tasks consisted of locating suitable site search tools, use them, read the results and final evaluation of the ease of use of the search system.
The factors analyzed in the usability testing were the following aspects: a qualitative assessment (verbal) with ease of use and the functionality of a search tool (for a given task), quantitative evalu-ation (numerical) to match a retrieval tool to the subject and purpose of the site and time to complete a task search engine on the site (based on video recordings of the program that records the activities of users).
The tasks that were given to experts in the study using a usability test of a search system are the following issues.
偪 Tasks for a simple search interface:
Ɣ area: ease of locating of a simple search interface,
Z01 task: looking for any shop subpage, locate the place of the simple search interface and then rate the ease of location;
Ɣ area: ease of finding information about the total number of search results,
Z02 task: search using a simple search interface in the shop and enter the total number of found documents (search results) in the range: > a digital camera;
Ɣ area: ease of using of a simple search interface,
Z03 task: based on the above rate: ease of using of a simple search interface and ease of finding information on the total number of documents (search results) that match the search query;
Ɣ area: ease of the query finding on a page with searching results (e.g. to change it),
Z04 task: search for products in the range: > photo accessories, then review several prod-ucts and sites obtained in the search results, and then rate the ease of query finding to the search results page;
Ɣ area: ease of navigating through all of the search results,
Z05 task: based on the abovementioned, browse a few pages of products obtained in the search results and then rate the overall ease of navigating through all of the search results; Ɣ area: compliance gained at the beginning (first results page) search results to the query,
Z06 task: search using a simple search interface in the shop products for: > Olympus E-3, then browse up to the first three search results and then evaluate the compatibility of these results to the query;
Ɣ area: resistance of a search system for grammatical errors entered by the user,
Z07 task: search using a simple search interface in the shop two different searches for the following entries: > Sony lens and: > Sony lesn, then browse up to the first three search results for both searches, compare the results with each other and then evaluate the resistance of a search system of language errors.
Studies & Proceedings of Polish Association for Knowledge Management No. 72, 2014
偪 Tasks for an advanced search interface:
Ɣ area: ease of locating of an advanced search interface,
Z08 task: looking for any shop subpage, locate the place of the advanced search interface, go to it to see the advanced options, and then rate ease of location;
Ɣ area: ease of usage from advanced search options,
Z09 task: search the entire contents in the shop products in the range: > camera, and then rate the ease of using of search options for this query;
Ɣ area: ability to search within specific areas / pages / category,
Z10 task: search within their chosen areas / pages / categories in the store products in the range: > camera (Category: Photography), and then rate the ease of using of search options for this query;
Ɣ area: ability to search using Boolean phrases (such as: AND, OR, NOT),
Z11 task: search using Boolean phrases (such as: AND, OR, NOT, etc.) in store products in the range: > not Canon lens (search for all lenses except for Canon's lens), and then rate the ease of using additional search options for this query;
Ɣ area: ability to search by changing the display of results (e.g. custom sorting),
Z12 task: search by using other options (such as custom: sorting, display, etc.) in store prod-ucts in the range: > camera (Category: Photography, sort by: price) and then rate the ease of using of other available options for this query.
5. Analysis of results
The study of the search system of the Merlin.pl store was taken into account the following indicators:
Ɣ W1 – average time of the expert spent on the shop’s page during the execution of all search tasks – measured in seconds on the basis of video material analysis;
Ɣ W2 – reliability and accuracy of performing each task by the expert, statement whether expert before the evaluation on the ease of execution of the tasks had actually done them before – measured in values from 0 (minimum reliability) to 1 (full reliability) on the basis of video material analysis and medium time of the expert residence on shop’s page (W1);
Ɣ W3 – global expert opinion (in points) from evaluation of the task – calculated as the product of a qualitative assessment of the ease of use and functionality of a searching tool (values from 1 to 4) and quantitative assessment of the fit of a searching tool to the subject and purpose of the service (values from 1 to 10) – measured in points on the basis of the results of the poll; Ɣ W4 – final evaluation of the expert and his credibility – calculated by dividing the overall
opin-ion of an expert assessment of the task to a maximum value (maximum value: 4 * 10 = 40), which may be obtained for a given task points (in percentage) multiplied by the reliability indi-cator of the expert (W2) – measured in percentage on the basis of the results of the poll and video material analysis;
Ɣ W5 – final overall evaluation of individual tasks by the experts – calculated as the sum of the overall opinion of all experts from evaluation of each task to the sum of the maximum values
46
(maximum value: 4 * 10 * 18 = 720) which may be obtained for all the tasks of points (in percentage) – measured in percentage on the basis of the results of the poll.
The results of the study of the search system in the Merlin.pl store using the usability test method using an electronic poll and a program for recording the performed activities presented in the fol-lowing table 1.
Table 1. Summary of test results on the quality of searching system in online shop Merlin.pl
Source: own computations.
6. Presentation of final recommendations
As a result of the searching system research for online store Merlin.pl obtained the following conclusions:
Ɣ Top ranked by experts, the aspect of a search system has proven to be a good location of a simple search interface (task Z01). It has been estimated at an average of more than 60%, which means that most respondents had no problems finding it.
Ɣ A similar situation occurred with regard to the use of a simple search interface and the ability to obtain basic information from its operation. And very easy to find information on the total number of search results (task Z02), the ease of using a simple search interface (task Z03) and with those obtained at the beginning of search results for the query (task Z06) was rated by respondents on average at about 50%. This may mean that respondents generally had no major problems with the operation of the search system, while some problems could cause a kind of information presentation of retrieved documents.
Studies & Proceedings of Polish Association for Knowledge Management No. 72, 2014
Ɣ The average level was also evaluated by experts, access to advanced search interface and its operation. And so ease of location of the advanced search interface (task Z08), the ease of search options (task Z09) and the ability to search within specific areas / pages / categories (task Z10) respondents evaluated at an average of 49%, which means they had no major problems with its location; furthermore, an interface allowed the implementation of the search with additional options and the user-selected parts of the service. This fact also speeds up the process of information retrieval, giving users instant access to desired products. Ɣ At a slightly lower level, the experts assessed how to navigate in the search results. And so
easy to find typed queries to the search results page (task Z04) and ease of navigating through all of the search results (task Z05) were assessed an average of around 40%. This may mean that, on the one hand, the respondents had trouble finding a form for changing the query to the shop’s page; on the other, since it is used so rarely, they are not accustomed to its pres-ence. Problems with easily navigating through the search results may be due to the fact that among these results are also often sponsored results, which often quite effectively can disrupt the view of only the interesting results from the search.
Ɣ At a similar level, approximately 40% were also evaluated by experts additional features of an advanced search system, which is the ability to search using Boolean phrases (task Z11) and finding with changing of the results display (task Z12). These results mean that the search system in the Merlin shop has advanced information retrieval capabilities for the usage of Boolean phrases and to change the default method of displaying search results by the user. The probable cause of such a relatively low assessment of the search capabilities of this sys-tem may be relatively small in popularity among users and often lack knowledge of how can they use it during the search process.
Ɣ The least among experts, because at an average of 27%, was evaluated for resistance of search system on language errors entered by the user (task Z07). This can result in underdeveloped information retrieval system or malfunction when the respondents do not fully type the pass-word correctly in the search system. Of course, in such situations the search system should propose new valid password or perform a search for similar terms.
7. Concluding remarks
Application of usability testing to evaluate the quality and usefulness of a search system in an online shop is a very good solution in the direction of the designation of areas for improvement on the target. As the user’s way of interaction with the web service is dependent on many factors, so it is important that services were the most functional, useful and intuitive to use. By using various tools to support the usability testing method allows an interdisciplinary approach to the analysis of data obtained from users, which affects the recognition of different causes of potential problems.
Carried out and described in the article, the analysis of searching system using the usability test for online store Merlin.pl shows that this method revealed many shortcomings in the test system. Of course, the results largely depend on the attitude and knowledge of expert assessors, but the same methodology of the study on the usefulness of the usability test consists in the practical interaction with the system by testing the activities available in the function is one of the best forms of verifi-cation of its usefulness.
48
Bibliography
[1] Afuah Allan, Tucci Christopher L. (2003): Biznes internetowy strategie i modele. [in] Ofi-cyna Ekonomiczna, Kraków.
[2] Dahlgaard J.J., Kristesen K., Kanji G.K. (2001): Podstawy zarzdzania jako ci. [in] PWN, Warszawa.
[3] http://merlin.pl/info/about_firm/main.html [access: 20 stycznia 2012]. [4] http://pl.wikipedia.org/wiki/Sklep_internetowy [access: 25 stycznia 2012]. [5] http://www.alexa.com/siteinfo/merlin.com.pl [access: 25 stycznia 2012].
[6] Kalbach J. (2008): Projektowanie nawigacji strony WWW. Optymalizacja funkcjonalno ci witryny. [in] Wydawnictwo Helion/O’Reilly.
[7] Mirecka Ewa (2003): Wyszukiwanie, selekcjonowanie i gromadzenie informacji Cz. 1. [in] WSiP.
[8] Nielsen Jakob, Tahir Marie (2006): Funkcjonalno stron WWW: 50 witryn bez sekretów. [in] Wydawnictwo Helion.
[9] Nielsen J., Landauer T. (2000): Why You Only Need to Test with 5 Users, Alertbox column on webpage: <http://www.useit.com> [access: 20 stycznia 2012].
[10] Nowakowski Michał (2008): Wpływ architektury informacji portali internetowych na ich popularno w ród klientów. [in] Monografia z XI konferencji z cyklu Problemy Społecze-stwa Informacyjnego, Wydawnictwo Hogben, Szczecin.
[11] Pearrow M. (2002): Funkcjonalno stron internetowych. [in] Wydawnictwo Helion, Gli-wice.
[12] Raport ecommerce 2011, Internet Standard, V edycja, padziernik 2011, <http://www.itstan-dard.pl/whitepaper/1843-Prezentujemy.raport.eCommerce.2011.html> [access: 11 stycznia 2012], pp. 63.
[13] Rosenfeld Louis, Morville Peter (2003): Architektura informacji w serwisach internetowych. [in] Wydawnictwo O’Reilly.
[14] Sapa R. (2002): Jako serwisów WWW bibliotek akademickich – usability test. [in] Biuletyn EBIB [on-line], 2002 nr 2 (31), <http://ebib.oss.wroc.pl/2002/31/sapa.php> [access: 12 stycznia 2012].
[15] Testy uyteczno ci to nie fokusy, naprawd! [in] http://www.webusability.pl, stycze 2007, <http://www.webusability.pl/2007/01/24/testy-uzytecznosci-to-nie-fokusy-naprawde> [ac-cess: 12 stycznia 2012].
Studies & Proceedings of Polish Association for Knowledge Management No. 72, 2014
ZASTOSOWANIE TESTU UĩYTECZNOĝCI DO ANALIZY SYSTEMU WYSZUKIWANIA INFORMACJI W SKLEPIE INTERNETOWYM
Streszczenie
Artykuł opisuje zastosowanie testu uĪytecznoĞci do analizy kryteriów jakoĞcio-wych architektury informacji ze szczególnym uwzglĊdnieniem elementów systemu wy-szukiwania informacji. Obiektem badania jest zastosowany system wywy-szukiwania in-formacji w popularnym polskim portalu Merlin.pl. Pod wzglĊdem wymagaĔ jakoĞciowych główny nacisk połoĪony jest na odbiorców koĔcowych, czyli klientów sklepów internetowych, co zgodne jest z podejĞciem UCD.
Słowa kluczowe: architektura informacji, sklep internetowy, system wyszukiwania informacji, test uyteczno ci
Michał Nowakowski Uniwersytet Szczeciski
Instytut Informatyki w Zarzdzaniu
Wydział Nauk Ekonomicznych i Zarzdzania ul. Mickiewicza 64, Szczecin, Poland e-mail: michal.nowakowski@wneiz.pl
