text: Managing Information Overload on the Web with Collaborative Filtering by Winnie H. Liang Submitted to the Department of Electrical Engineering and Computer Science in partial fulfillmentof the requirements for the degrees of Bachelor of Science in Computer Science and Engineering and Master of Engineering in Electrical Engineering and Computer Science at the Massachusetts Institute of Technology May 26, 1996 Copyright 1996 Winnie H. Liang. All rights reserved. The author hereby grants to MIT permission to reproduce and distribute publicly paper and electronic copies of this thesis and to grant others the right to do so. Author Department of El c-tial"E i./ng and Computer Science February 1, 1996 Certified by Barbara Liskov, Thesis Supervisor NEC Professor of Software Science and Engineering Certified by .. Product Manager of the Ilerlotes Dave Newbold, Thesis Supervisor Web Navigator team, Iris Associates Accepted by Acp \ b FrederickR. Morgenthaler Chairman, Departmeal Committee on Graduate Students .!ASSACHUSETTS ISTi .i OF TECHNOLOGY JUN 11 1996 E ARCHIVES I Managing Information Overload on the Web with Collaborative Filtering by Winnie H. Liang Submitted to the Department of Electrical Engineering and Computer Science Feb. 7, 1995 In Partial Fulfillmentof the Requirements for the Degree of Bachelor of Science in Computer Science and Engineering and Master of Engineering in Electrical Engineering and Computer Science Abstract Organized like an intricate spider's web, the World Wide Web connects many thousands of pages of information through hypertext links of URLs. A problem with these pages is that users can not easily find specific information. To help users solve this problem, this thesis manages Web pages by implementingthe collaborative filtering technique into the Lotus InterNotes Web Navigator. Collaborative filteringuses recommendations from others to help direct people to information that will be relevant and interesting to them. To determine whether the technique is successful in a workgroup setting and to direct the design of the collaborative filter, two tests were run on this collaborative filter system during the project. The results show that people do share recommendations about Web pages and that people find collaborative filtering helpful in finding and managing information on the Web. Thesis Supervisor: Barbara Liskov Title: NEC Professor of Software Science and Engineering Thesis Supervisor: Dave Newbold Title: Product Manager of InterNotes Web Navigator Team, Iris Associates Acknowledgments First of all, I thank and praise God for all the work He did in this thesis work. I could not have done this thesis without Him. All the work in this thesis is actually His work, not mine. "I can do everything through Him who gives me strength." (Bible, Philippians 4:13, NIV translation) I also want to thank my thesis advisors, Dave Newbold and Professor Barbara Liskov. I thank Dave Newbold for his guidance, support, constructive crit'ism, and for sharing his creativity, all of which contributed greatly to this thesis. I thank Professor Liskov for her support and her help in connecting me with relevant research work. Working with the InterNotes Web Navigator team on this thesis was great. All of them were always very encouraging and quick to give me URLs related to my thesis. Special thanks to Teresa Deane for sharing her technical knowledge in Lotus Script and Lotus Notes and to Barb Mathers for helping me with the testing in this thesis and for always generously giving me encouragement and help. Finally, I thank my family and friends. I am deeply indebted to my parents for their patience, support, and sacrifices. Everything my parents have given me is priceless and, in this whole lifetime, could never be repaid. I thank my sister, Cindy, for her fun-loving nature that helped me laughed away the tension that sometimes appeared while doing this thesis. I thank Angela Yin and Karen Lee, my dear sisters in Christ, for their faithful prayers and listening ears that carried me through this thesis. I thank en Cheng for his prayers, encouragement, and for helping me revise my thesis proposal and thesis. I also thank Barb Mathers, Rita Vanbever, and all my brothers and sisters in Christ from the Intercollegiate Fellowship and Chinese Bible Fellowship for their faithful prayers. Contents 1 Introduction ................................................................................................................... 1.1I Leveraging the Net's Information and Connectivity ....................................... 1.2 Use of Collaboration With Information Filtering ........................................... 9 11 12 2 Problem Statement ..................................................................................................... 14 2.1 Problems with Information on the Internet ................................................... 14 2.2 Problems with Information on the Web......................................................... 14 2.2.1 Structure of Information on the Web.............................................. 15 2.2.2 How Information on the Web is Used ............................................. 2.2.3 Summary of Information Access Problems ..................................... 2.3 Thesis Solution................................................... 2.3.1 Collaborative Filtering ................................................................... 16 18 18.............................. 18 2.3.2 General Description of Solution...................................................... 19 2.3.4 System Constraints......................................................................... 19 3 Previous Work........................................................................................................... 3.1 Common Filtering Techniques ....................................................................... 21 21 3.2 Reasons For Choosing Collaborative Filtering............................................... 25 3.2.1 The Development Environment ...................................................... 26 126.96.36.199 Description of Lotus Notes .............................................. 26 220.127.116.11.1I Organization of Notes Databases....................... 27 3.2.2 The Benefits of Placing Collaborative Filtering and Notes Together ........................................................................ 34 4 Issues in Design ......................................................................................................... 4.1 Critical Design Element is Making Recommendations ................................... 36 36 4.1.1 Maintain Low Overhead................................................................. 37 4.1.2 Support User Conventions.............................................................. 37 4.1.3 Prevent the Recreation of the Problem............................................ 38 4.1.4 Obtain Useful Information.............................................................39 S Design of the Collaborative Filter System ................................................................ 42 5.1 Brief Scenario............................................................................................... 42 5.2 Reducing Overhead ....................................................................................... 4 49 50 ................................................................. 5.3 Organizing the Recommendations 5.4 Obtaining Useful Information........................................................................53 6 Testing.........................................................................................................................58 6.1 Purpose of Testing.......................................................................................58 6.2 First Test...................................................................................................... 59 6.2.1 Description of the First Test ........................................................... 6.2.2 Results From the First Test............................................................. 6.3 Second Test.................................................................................................. 6.3.1 Description of the Second Test....................................................... 6.3.2 Results From tl'e SecondTest......................................................... 188.8.131.52 W ill People Recommend W eb Pages? .............................. 59 61 66 66 67 67 188.8.131.52 Does the Collaborative Filter Deliver Interesting Web Pages?............................................................................. 69 7 Conclusion ................................................................................................................. 71 7. Analysis of Current Work ............................................................................. 71 7.2 Future Work................................................................................................. 72 5 List of Figures 1.1 A graph from Internet World magazine showing the rapid increase in the number 9 of Intemrnethost worldwide [18]............................................................................ 1.2 A graph from Internet World magazineshowing the rapid increase in the number of Web sites for the past few years [18].............................................................. 10 2.1 Shows how page references in a Web history can be lost ..................................... 17 3.1 Screen Capture of the Web database icon on a Notes Workspace ........................ 27 3.2 Screen capture of database icons on the Notes Workspace .................................. 28 3.3a Screen capture of a Notes document in editing mode.......................................... 28 3.3b Screen capture of the same Notes document in Figure 3.2a except in viewing mode................................................................................................................... 29 3.4 A screen capture of a Notes form ........................................................................ 30 3.5 A screen capture of a Notes section within a Notes document ............................. 30 3.6a A screen capture of a Notes view..................................................... 3................... 1 3.6b A screen capture of a search bar in a Notes view ................................................. 32 3.6c A screen capture of a search dialog box in a Notes view ..................................... 33 3.6d Screen capture showing the relevance ranked results from a Notes search...........33 5.1 A Web page froin the InterNotes Web Navigator. The Recommend button is located directly on the button bar above the Web page ........................................ 43 5.2a Screen capture showing the Recommenddialog box on top of a Web page.........44 5.2b In this screen capture, the Recommend dialog box is filled in .............................. 5.3 Screen capture of the Web Navigator starting Home page ................................... 45 5.4a A screen capture of the Recommended By Reviewer view. Here all the name categories are collapsed...................................................................................... 6 44 46 5.4b The Recommended By Reviewer view collapsible categories can be expanded. Through expanding one of the names in the view, you can see the categories that one user has recommendations for and expand further to read the recommendations of the user.............................................................................. 46 5.5a A screen capture of the Recommended By Category view of the Web Navigator ......................................................................................... 47 5.5b The Recommended By Category View expanded by one of the categories..........48 5.6 A screen capture of the Recommended Top Ratings view................................... 49 5.7a The Administrator Notes Form that allows users to delete documents from the Web Navigator database periodically.................................................................. 55 5.7b Screen capture shows the time period options that users have to delete ratings....55 5.8 Recommended By Reviewer view showing the invalidation icon, which indicates that the recommendation beside it could be inaccurate ........................................ 6.1 56 This screen capture of an actual Web page in the Notes database. Directly on this page is the Recommend button which a user can mouse click on to make a rating .................................................................................................................. 7.1 63 A picture of a rating bar ...................................................................................... 73 7 List of Tables 6.1 Do you find recommended pages helpful? Would you use the recommended pages7 ................................................................................................................ 65 6.2 Was it easy or difficult to rate pages?.................................................................. 65 6.3 Would you rate pages as you browse the Web?..................................................65 6.4 This table and graph show how many ratings were made under each type of rating.................................................................................................................. 68 6.5 The total number of ratings for each testing group.............................................. 69 6.6 Did you use the Recommended/By Reviewer view .............................................. 69 6.7 Did you use the Recommended/By Category view? ............................................. 69 8 Chapter 1 Introduction "Surfing the Net" has become a household phrase in today's society. Its existence is the result of the increasing number of people accessing the Internet. The increasing growth and popularity of the Internet is overwhelming (Figures 1.1, 1.2). Especially with the quickly advancing World Wide Web technology, the Internet has suddenly become a rich source of information available to millionsof people. Figure 1.1 A graph from Internet World magazine showing the rapid increase in the number of Internet host worldwide [181. 9 Figure 1.2 A graph from Internet World magazine showing the rapid increase in the number of Web sites for the past few years [181. Although the rapid growth and popularity of the Internet suggests that more people are benefiting from the Internet, the Internet is still not very useful to people. The problem with the Internet is that users can not easily find specific information on it. Due to its recent enormous growth, the information on the Internet has become unmanageable. The development of tools to manage data has not kept up with the Internet's growth and popularity. The tools used now to access information do not help users to efficiently find information that they want. What users need is a tool that will immediately bring to them the best information from the Internet that they are looking for. In this thesis project, the concept of collaborative filtering is used to help manage the large amounts of information on the Web. Specificallyin the system, users collaborate with each other to find relevant information on the Web. 10 The following sections of this chapter describe the current research work that focused on developing tools to make the Internet more useful and thesis work that also had the same goal. Chapter 2 describes what was the exact problem that this thesis project addressed. Chapter 3 gives an overview and analysis of the techniques, including collaborative filtering, that current information systems besides the Web use to deliver customized information; and also explains why collaborative filtering was chosen for this thesis work. Chapter 4 describes the design issues and constraints of the system. Chapter 5 describes in detail the final system design. Chapter 6 presents the usage data obtained from usability experiments. Chapter 7 concludes the thesis with an analysis of the system and a discussion of future directions. 1.1 Leveraging Information and Connectivity on the Internet Current research is helping to make users more productive at everyday task by using the Internet. Some of the current Internet research is focused on leveraging the information stored on the Internet. Recently, researchers have created information filtering systems to customize information for users. With user's interests represented in profiles, some filtering systems periodically retrieve information from the Internet that suits those interest profiles [9, 12]. Other systems filter information by grouping users with similar interest profiles. They customize information by announcing to everyone in the group newly discovered information that one member of the group found interesting [5, 7, 8, 9]. Other filtering systems provide an end-user language through which users place conditions and rules on what information should be brought to their computers [3, 4, 10]. Other researchers have been trying to leverage the connectivity of the Internet by enhancing collaboration on the Web. At the Sept. '95 workshop on Web collaboration 11 sponsored by the World Wide Web consortium, members brainstormed ideas for desired collaborative capabilities for the Web and determined the standards and technologies needed to create the capabilities. Among those discussed were the following; Group annotation: the ability to allow several users to view comments and annotations directly on a Web page. Group editing: the ability to allow users to edit Web pages together. Web tour: the ability to allow users to jump to specific sites on the Web together and to talk with the others about Web pages 1.2 Use of Collaboration With Information Filtering Collaborative filtering is an increasing popular information filtering technique. Collaborative filtering systems use recommendations from many sources to filter information. People often listen to the recommendations-of others when choosing the next movie to see, book to read, or place to visit. When people finish watching a movie, they will generally give their recommendations to friends in a conversation. Similarly, people can share recommendations about Web pages that are worth looking at or about those to avoid. In giving these recommendations, people become information filters for their friends. Sharing the results of information searches is a common practice. It is one of the natural ways people find information that is relevant to them. O'Day's and Jeffries' study on library information searches is a concrete example of collaboration used in everyday information searches. ODay and Jeffries performed a study on searches conducted by library clients and noted there were four basic ways in which the clients shared their results (The study noted that these models could be applied to other information systems.) [16]: 1. Results were shared with other team members. 2. Interesting information encountered during a search was publicly broadcast. 12 3. A member of the team, acting as a consultant, shared his expertise by dealing with search requests made by others. 4. Potentially useful information was archived into a group repository. Each of these approaches is a part of the concept of collaborative filtering. However, as will be discussed in the following chapters, the collaborative filtering technique organizes the types of collaboration that are used to find information. This organization makes collaboration more powerful and useful for filtering information. In addition, the Internet provides new ways of collaboration. For example, people do not have to wait to see a person to get recommendations but can access recommendations from a group repository on the Internet where they are stored. With the Internet, recommendations can be easily shared among many co-workers who could be miles apart. 13 Chapter 2 Problem Statement 2.1 Problems with Information on the Internet A problem with the Internet today is that finding good interesting information is a difficult and time consuming process. Especially with the information super highway becoming a reality, more people have access to huge amounts of raw information. Having access to so much information appears to be beneficial, but is not because people cannot easily find what they want. Information is not structured hierarchicallyand the tools used to access information are primitive. Without the ability to quickly access the information when needed, it is as if the information does not exist. 2.2 Problems with Information on the Web The Web has this same problem. The amount ofinformation being placed and accessed on the Web has been increasing since the day it started. The Web, which started in 1990, grew from more than 4,600 servers in October 1994 to over 10,000 servers by December 1994. These servers provide retrieval access to over 1.05 milliondocuments [1]. The Lycos usage statistics collected by Carnegie MellonUniversity indicate that during the week of May 28, 1995, 2,161,152 pages were viewed through CMUs homepage. At 14 peak times, 21,000 Web pages are accessed per hour through Lycos' Web page [2]. With the Web growing daily in size and in usage, there has been an increasing need for new techniques to aid users in finding the Web information they want with minimal effort. The usability tests for this thesis showed that new Web users especially struggle with finding information. In these usability tests, a group of new Web users were asked to find a specific magazine. Users made comments such as these: "Therewas a lot of informationthrownat me..." '...So much information not broken down..." ...need something to cut out whatyou don't want and get you straight to thatpoint...." This problem is a result of the structure of the Web and the way it is currently used. 2.2.1 Structure of Information on the Web The structure of the Web is very similar to the structure of a spider's web. The Web is composed of individual accessiblepages which are linked. Information on the Web is stored by URLs (Uniform Resource Locators) which point to Web pages. There is no standard for composing URLs but like file names, the words used in the URL describe the content of the Web page. Each Web page contains text, graphics, and hypertext links which point to other Web pages. A user moves from one page to another using hypertext links. The words on the Web page associated with a hypertext link describe in a general way what the content of the linked page is about. Web pages can also be accessed directly by the URLs. The way information is added to the Web is easy but random. A page is added to the Web when it is placed onto one of the many Web servers on the Internet. Once added, the page can be linked to any other page. 15 2.2.2 How Information on the Web is Used Information on the Web is accessed and used through Web browsers in several ways. With a URL, Web browsers canjump directlyto a page. However, users usually do not search the information on the Web in this way because they do not know the URL until either they actually read the page or another user informs them of the location of the page. Often, users randomly browse the Web by starting on one page and jumping to others by hypertext links. To help users find information, search engines have been placed at various Web sites. These search engines help users find information in two ways. In the first way, either a user managing the site or the engine opens Web pages and categorizes them by content. A user scans through the categories to find needed information. In the second way, the engines search the Web for pages contains keywords selected by the user. These engines are useful but imperfect. In order to use them, users have to first locate them on the Web. Also, their catalogs do not provide users with a complete view of what information is available on the Web. Everyday, an enormous amount of information is placed on the Web undetected making it impossible to keep catalogs perfectly updated. Another problem with the catalogs is that the quality of all the Web pages under a category is not necessarily good; therefore, a user is forced to read many pages to find the good quality information. The engine search programs are also imperfect. They search for Web pages in several ways. Some look for keywords in the hypertext links or the URLs of the Web pages stored in their own catalogs. Other engines search the whole Web by opening pages depending on whether the hypertext link or the URL contains the keyword. In this method, the pages are not opened exhaustivelybut the search engine finds a certain number of pages with the keyword and stops the search to return what was found. In 16 both methods, the search engine returns a limited listing. Also, pages could be missed because the words contained in the hypertext link and the URL do not have the keyword even though a page itself does. Even when a large number of pages have been found, there is no quality control of the pages returned. Since finding a good page is rare, there are two methods in which Web browsers manage information found. The first method uses historiesof URLs. While users are browsing the Web, Web browsers keep a history of Web pages that they have accessed. If users desire to go back to a page already seen, they can use "back" and "forward" commands to scan the pages in the history. This method has some drawbacks. The backtracking process is time consuming because the scanning is performed page by page. Also, the history is created temporarily. When users quit their Web browser application, they lose the current history of pages. Also, the histories are not complete. When backtracking to a Web page, some Web page references in the history are lost if another series of Web pages are traversed from a backtracked Web page (see Figure 2.1). b.) Clue hot icas th URLthat b wseri Craxfy Mm _ S UR. c.) eba Hbsil caVlewv aURL4 huhe. URL3 werrlmo % he Sh lberjmqed Sincet b cam ULz, fr URL2, - hprbccU UI 3 a viin Mso p reita e 2.1b tSh Showlhowlpae refernces in Webhitory ca belosRL4 . Figure 2.1 Shows how page references in a Web history can be lost. 17 In the second method, users save URL references using a functionality called bookmarks. Similar to the way regular bookmarks are used with books, users add pages to a bookmark list which are used to directly jump to a specific page on the Web. For a small number of bookmarks, this capability is useful. Users do not need to remember the series of Web pages and links used to reach a desired page. As long the page is in the bookmark list, users can directly jump to the page at any time. However, bookmarks themselves add to the original search problem. If the user saves many bookmarks, searching for specific bookmarks is time consuming because the bookmarks are stored in a list. Without additional organization of the bookmarks, users with many bookmarks need to read through a long list to find a specific page. 2.2.3 Summary of Information Access Problems Even with Web search engines and all the features that Web browsers provide to access information, there are still major problems with finding information. There is no defined way for a user to find all information on the Web related to a specific subject. When users can not find something specific when browsing the Web, they can not be sure whether the information is not there or they have not found it yet. Although they may find some information for a certain topic, users do not know if there is another Web page which has better information. Without an effective way to find information, users find information only by exhaustively searching the Web page by page. 2.3 Thesis Solution 2.3.1. Collaborative filtering In this thesis, a collaborative filtering method is used in the InterNotes Web Navigator to manage the information on the Web. Collaborative filtering simply means that people 18 collaborate to help one another perform information filteringby recording their reactions to information they read [4]. These reactions, generally called annotations [4], can be used to direct other users' choices in selecting what informationto read and collect. For the Web, these reactions are valuable because there are only two ways in which users can determine the content and location of a Web page. When users encounter a page on their own, they determine the content of the page by reading it. Users can also find out the content and location of the Web page from other users who have already seen the Web page. 2.3.2 General description of Solution In this thesis, a collaborative filter is built into the Lotus InterNotes Web Navigator. Since the Lotus InterNotes Web Navigator is a Notes database application, the collaborative filter is built using Notes database capabilities. As users access Web pages, their reactions to the Web pages and information about the Web page are obtained through a Notes form (see chapter 3, Previous Work, section 3.3.1, The Development Environment, for a description of Lotus Notes.). The reaction data is stored in Notes documents within the InterNotes Web Navigator database. Users, through Notes clients, can view the reaction data using Notes views in the Web Navigator database. After reading reactions to Web documents, users can access the described Web page by simply selecting a specific reaction in the Notes view. Through the Notes views, the reactions are organized by their attributes such as the reviewer or content of the Web page. These views organize the reaction data which also help users find specific pages of interest. 2.3.4 System Constraints There are two design constraints for this system. One key constraint is the Notes application programming environment which is limited by the Notes user interface. 19 Since user's reactions and recommendations are the foundation of this system's filtering capabilities, the second constraint is that the system is to encourage users to recommend Web pages. Several design criteria have been devised from this constraint as summarized below. They are described in detail in the chapter 4 Issues in Design. * Ease of Use: To encourage the use of the filter, users should be able to know how to input their reactions to a Web page quickly. Therefore, Web and Notes user conventions will play an important part in directing the design of the collaborative filter. If these users do not know how to easily input their reactions to pages as they are browsing the Web through Notes, information filtering will not occur. * Low Computational Resources Cost: Saving and viewing the reactions should not require too many computing resources. Since the reactions are actually stored in the Web Navigator database, too much overhead in saving and storing reactions might cause other problems such as increasing the time a Web page is found and processed. * Streamlined: Already, browsing the Web takes a large amount of time and effort. Inputting a reaction to a page and finding the stored reaction should take minimal time so as not to contribute to the time and effort already needed to browse the Web. 20 Chapter 3 Previous Work 3.1 Common filtering techniques Many filtering systems have been developed recently for information systems such as email and Netnews. There are common filtering techniques used by these systems. The choice of using the collaborative filtering technique to filter information in the InterNotes Web Navigator comes from the analysis of these filtering techniques. Each of these techniques and a sample system that implements the technique will be described in the following sections. The last section will explain the reasons why the collaborative filtering technique was chosen. Collaborative filtering [1,3,4,6,7] - The collaborative filtering technique takes advantage of what other users have already read. In this technique, the users' personal reactions to information are saved through the comments and ratings users make about the information they read. Those reactions are used by other users to direct their choices in the information they read. In some of the systems, users can access the information directly from the reaction. There are several different forms of collaborative filtering. They are passive, active, and automatic collaborative filtering. Automatic filtering uses a different 21 principle than the one described here under collaborative filtering. A separate section will be used to describe automatic collaborative filtering in more detail (see Profiles section in this chapter). The first form of the collaborative filter is named passive because the filtering capabilities are passive until a user accesses the collaborativefilter system. Users rate and record their reactions to the information and the data is stored in a repository. Users go to the repository and read the ratings to help them decide what informationto read. The system is not "actively" filtering and bringing information to the user. Filtering occurs when users view the stored reactions and decide what information to read from those reactions. An example of a system that implementedthis technique is Maltz's collaborative filter for Usenet Netnews [6]. As Usenet users read Netnews documents, they vote on the content of the Netnews with a rating scheme of great, good, or bad. Through a vote distribution system Maltz implemented, users on Usenet canrteasily access the votes for specific Usenet Netnews documents. From reading the votes, users decide whether to read the document. The next form of collaborative filtering is named active because filtered information is actively being brought to the attention of a user. As users read information, they can send an annotation and an information reference to other users they think would be interested in the information they read. In this way, users become active information filters for others with similar interests. An example of this filtering technique is Ehrlich's active collaborative filter for Lotus Notes databases [1]. In her system, users can email to other users annotations and a Lotus Notes document link (an icon which opens the Notes document when doubled clicked with a mouse). Agents [9, 12] - Another common technique uses agents to filter information for users. Agents are programs that automate computer tasks such as deleting files and moving files. In these systems, agents are personalized to users information needs to help them with daily tasks. Through data obtained from the observation of user behaviors, agents learn what conditions should be met before performing any task. In information filtering 22 systems, agents are personalized to the information interests of users. Information not useful to a user is automatically deleted and information interesting to a user is automatically retrieved. Also, agents in some systems are customized to move information to directories creating an organized way for users to read the information retrieved. MIT Media Lab's evolving agents is an example of this technique. The system uses the concept of evolution to create personalized agents [12]. As the user retrieves information, the system categorizes the retrieved information to create broad and general agents where each agent retrieves one category of information. From this group of general agents, the user selects the agents that retrieve interesting and relevant information. New agents are created through the combination of the selected agents. The agent combination procedures imitate genetic processes such as mutating and crossing of genes. As new agents are created, they go through the selection process again where the users decide whether the information they retrieve is interesting or relevant. The user selection process and "mating" of agents continues to produce new and more personalized agents. In theory, more specific, customized agents are created through crossing and mutating the agents that users selected. Rules [3, 4, 10]- In several filtering systems, users create their own information filters through a set of rules. The filtering system is composed of a rules language which a user can use to set a number of conditions on what information to retrieve. Usually the goal of these systems is to represent the information and rules in a way that makes creating the information filters easy. A good example of a rule based filtering system is Malone's intelligentinformation sharing system [10]. The system is based on semi-structured message templates. Semi-structured message templates contain fields. Users input different values into the message fields. Depending on the purpose of the message, the semi-structured message templates will have different fields for input. With a filtering rules language, users can place conditions 23 based on the type of semi-structured messages and on the values found in the fields within the semi-structured messages. Users create the rules through the semi-structured message templates Making the rules is easy because users know what fields are available in each semi-structured the message type. Also, they place conditions on the values of the fields directly in the message template which users are familiar with. Profiles (Automatic collaborative filtering) [5, 7, 8, 9]- In several collaborative filtering systems, the technique is performed without the user knowing collaboration with other users is taking place. The theory behind these systems is that two people who have the same interests will find information that is interesting to the other. In order for these systems to know whether two users have similar interests, user profiles are used. Users' interests are represented in profiles and users with similar profiles are grouped together. When one of the group members finds interesting information, the system informs other people in the same profile group about the discovery. All of the systems studied use users' ratings on information as a representation for user interest profiles. A good example of the profiles technique is GroupLens which filtered Netnews [7]. In GroupLens, users rate Netnews articles. Servers called Better Bit Bureaus gather the ratings and create the user profiles. These servers find users with similar profiles through collecting the ratings for each user and grouping users who rated the largest number of articles similarly. When encountering a news article that a user has not read yet, the server piedicts the level of interest for the user by looking at how other users with similar interests rated the article. The predictions were calculatedthrough numerical equations. Virtual interest groups [2, 13]- In this filtering technique, users are allowed to make temporary virtual newsgroups, email lists, and other types of information interest groups out of the ones already available. The virtual informationinterest groups are more specific than information interest groups available. They are created based on the individual needs of users. Information filtering occurs since users are able to create specific information groups to suit their needs and interests. Also, this filtering system is flexible to the 24 changing needs and interests of the user as new groups are created and others deleted when users' interests change One system, for example, creates ephemeral interest groups [12]. In this system, specific interest groups are created from a general message system which is viewed and accessed by all the users on the network. The general message system is used to share general announcements and ideas. Users create an ephemeral interest group from a message in the general message system. The new ephemeral interest group is a temporary discussion group in which users can discuss, in more detail, the topics found in the original selected message. Only users who joined the discussion can read the messages sent to the ephemeral interest group. Once a discussion group is created, any user can join it. 3.2 Reasons for choosing collaborative filtering Each of the above techniques does a reasonable job at customizing information for a user. Therefore, how well a technique filtered informationwas not a criteria used to choose which filtering technique to place into the InterNotes Web Navigator. What was considered an important criteria in this thesis was how well the filtering technique complimented the Lotus Notes environment in which the filter is being built in. Deciding whether a technique integrates well in Notes involves several conditions. The filter can not be a separate application from Notes. Creating a filtering system that requires a separate application to be built outside of Notes creates more difficulties than benefits. For example, it can be easily ignored. Rather, the technique should be implemented using the Notes application environment. Being built into the Lotus Notes environment will reduce the overload and time that have been needed for Notes to call the separate application. Also, the user adjustment and initial learning time will be minimized. The goals set for this information filter are the following: 25 1. To make a navigation or filtering application for the Web as uncomplicated for a user as possible. 2. To help users find information on the Web that interests them as quickly as possible. 3. To allow users to view the Web a little at a time - to view only the information that is pertinent to their needs. The collaborative filtering technique best flfilled these goals and integrated well into the Notes environment. Therefore, it was the chosen filtering technique for the InterNotes Web Navigator filter. In order to understand how collaborative filtering integrates well into the development environment, the environment will be briefly described here. 3.2.1 The Development Environment The collaborative filter is being built into two environments, the Web and Lotus Notes. The InterNotes Web Navigator is a Notes database application. It allows users to view Web pages through a Lotus Notes database (Figure 3.1). Therefore, both Web and Notes featires are usable in this Navigator. Web pages become Notes documents in the Notes Web Navigator database. These Notes Web documents still behave the same as Web pages from other Navigators with hypertext links that allow users to jump from one Web page to another. The key environment is the Notes environmentwhich will be described in the following section. 184.108.40.206 Description of Lotus Notes Notes is a client-server networked application in which data are stored on databases located on servers. Notes clients who have access to servers can retrieve the database information. The databases are hierarchicallyorganized which allows databases to be customized for specific purposes. These customized databases are called Notes database applications. 26 126.96.36.199.1 The Organization of Notes Databases [15] A Notes database is a collection of documents stored in one file. In Notes, a database is represented by an icon (Figure 3.2). The informationstored within each database is organized and maintained with five basic building blocks: documents, forms, fields, sections, and views. A documents is the "record" or data in the database (Figure 3.3a and 3.3b). The information in a document may be entered by a user, a Notes formula, other applications, or another Notes database. A document may contain alphanumeric characters, several pages of text, or graphics. Figure 3.1 Screen Capture of the Web database icon on a Notes Workspace. 27 ~ ~ II !I File Edit View Create Actions · -v Help Window Figure 3.2 Screen capture of database icons on the Notes Workspace. = I I V - I UI ~~ H Ulc -wi·IMNr =- Figure 3.3a Screen capture of a Notes document in editing mode. 28 K File Window EditView Create ActionsHelp File Edit View Create Actions Window Help alrlq M.` 'NPIWRM Wnrlon.lU:BBGas U TTUI - -ia__'1 -- UM I mt OR nrr Bll . User Wire's Name Address Book ' Local Barb Fast Name: Middle Initial: Last Name: FullName: Owner: Mathers BarbMathers Winnie LiangAris ElectronicMail Infolrmation DelegationAuthority: Mal Domain: Database Mail Server: TonTrideShow. Mail File: ."..'?:!? i.. - : :: ForwardingAddress: X.400 Address: ShortName: Work Information Company: Department: Office Location: 1>~ .~. ~' '1 1, ~ Winie Liarng/Iris Iris Internotes 1f,2Jicoffic` __ Figure 3.3b Screen capture of the same Notes document in Figure 3.2a except in viecxing mode. A form is what defines the format and layout of the document (Figure 3.4). Each form can contain fields, text, graphics, and buttons that help determine how users enter the information and how the information will be displayed. A database may contain several different forms depending on its purpose. A field within a form is an defined area in the form that contains a single type of information. Forms can have an unlimited number of fields. Each field must have a name and a data type. The data type determines what kind of data is placed in a field. The data types that are allowed are text, rich text (includes graphics), numbers, keywords, and time date values. A section is a special kind of field that defines an area of a form or document. Within a section, fields and text are placed and access may be controlled. A section is marked with a solid line and can be accessed by clicking on an arrow (Figure 3.5). 29 =~1 File I Edit View 11-flIffilJ010 1N.-OA[3-I- I h rx -3 ZZ , Help 11.m - --l-WHOWN I ~~~---ENk J5,:2ti "E 17:- 1 1 - I ';-EMM Ill~l~lnrr~r 4 Create Design Text Window --- = Iw. gci-n:4 : Type User Wires Name AddreBok First Name: on Local Middle Initial: MiddlelnMal LastNae: -- I FlstName| Full Name: SAStudents' Databadse on TradeShow.; Owner: ElectronicMail Informalion Delegalion Authouity: I Localdnmn Mail Domain: MaiDornerI MadSerer| Mail Server: Mail File: I IHelv 'l O[None ~I'l ~ ~ ~ 1 Ial dI ... I i I I office Figure 3.4 A screen capture of a Notes form. Figure 3.5 A screen capture of a Notes section within a Notes document. 30 i.. LD I A view is a tabular summary of the documents in a database. Databases can have several views, each sorting, selecting, and categorizing the documents in different ways, by the field values in the documents. Each row in a view represents one document (Figure 3.6a). A column in a view may be a field from a document, a result from a Notes formula, or a combination of the two. There are many additional features in Notes that help manage database applications. Notes contains security mechanisms that regulate access to a database and its contents. Notes has a fonimu!awriting language and built in functions that help to calculate and control the display of fields, views, and forms. Macros can be written through formulas to automate II N I Figure 3.6a A screen capture of a Notes view. 31 some activity. Agents (which are small programs) can be run at regular intervals on specific data from Notes databases. Replication is available in Notes that allows the capability of synchronizing copies of several databases located on different servers. Full text search capability makes it easy for users to find a document by allowing them to search for words or phrases in a database view (Figure 3.6b, 3.6c, and 3.6d). Within the view, Notes internal full-text search engine can be easily invoked by typing search words into a search bar (Figure 3.6b) or by bringing up a search word dialog box which appears when typing within the view (Figure 3.6c). The Notes search engine returns all the documents in the view which meet the search word criteria with a relevance bar to indicate the likelihood that the documents are what the user is searching for (Figure 3.6d). Figure 3.6b A screen capture of a search bar in a Notes view. 32 Figure 3.6c A screen capture of a search dialog box in a Notes view. = -I File Edit View I Create *I I. I * Actions Window Help SE rl _; "*,47,n i 4t,11, !* :.E , ] I" ' 'IV_"~. I* .* * * . .I . |Soatchet eAdtCaicieons painting 1 ',3 "P-,, -5" j0J!r7--- h. II'- 1-i I .i I[I DOocument ac itle Re:ult Search r'ahoo I useumn: Impressionism V/ebMu;eurr,. FamousPeaintingexhibition IWebM Giorgio de Ch;nco. I'WebMuseum jWebIl4ueum Ka,dn:ky, 'Wafily li Yahoo-Arts:Exhibits IYahoo -AtsAit HitcoryArt:t: YahooSearchResults ArtontheNet:VisualArtit: |BELINDA Di LeoMFAPROJECT Webl.useum: Bienvenue! fvelcorriefromthecurator) rch:games urn:Dali.SalvadorPersistence of Memory iesSeuratHomepage archResults ts t Services IndexO-R Tate Gallery/Royal Academy/N ationalGallery 1 - .; *;; ;itf1drce2 Figure 3.6d Screen capture showing the relevance ranked rcsults from a Notes search. . ,1 3.2.2 The Benefits of Placing collaborative filtering and Notes together The Notes environment establishes a good foundation for collaborative filtering. In Notes, sharing recommendations is very easy. The Notes application's purpose is to allow users to share information easily and effectively. With the replication capabilities, up to date recommendations can be delivered easily to many users over several servers. Collaborative filtering, where sharing information is crucial for successful information filtering, benefits from being built in the Notes environment. Notes views help to organize the reaction and recommendation data. Having the template forms with fields makes searching through recommendations for specific values easier [10]. A Notes view, not only allows selection of specific field values within stored recommendations, but also, has features such as categorization and sorting which can further help organize the recommendations. Within the view, Notes internal full-text search engine can be easily used to find very specific recommendations. The Lotus Script language and macros give the database applications and Web pages program capabilities that allow users to create internalNotes programs to manipulate and use Web pages and the recommendations in new ways. Other filtering techniques do not work as well in the Notes environment and the Web environment. The virtual information group filteringtechnique is not possible. There is no way of making temporary specialized Web pages. The Web is not a discussion group therefore the virtual groups technique can not help to filter the web. A rules language is already available in Notes. Notes has its own macro and script language which can be used to add specialized information filters if the user is good at programming. Adding a new rules language specifically to filter Web pages is wasted effort and creates more overhead to Notes. Using the agent technique is difficult since monitoring user behaviors is not possible in Notes. 34 The addition of the collaborative filtering capabilitywill allow other forms of filtering in the Notes Web Navigator to be possible. For example, with collaborative filtering, a representation for user behaviors and interests is created. The user recommendations, stored in the Notes Web Navigator database, can be sorted and categorized by each user to create user interest profiles. These profiles can be used to implement other filtering techniques such as the agent filtering technique where an agent uses the profiles, represented by stored recommendations, to guide its retrieval of personalized information for a given user. 35 Chapter 4 Issues in Design 4.1 Critical Design Element is Making Recommendations For this information filtering system, most of the design work focused on strengthening the weak areas of the system. This thesis work concentrated on determining ways to improve these weak areas because the Notes application environment made it easy to implement collaborative filtering into the Web Navigator. One major weakness of the collaborative filter technique that this thesis addressed is that the system is completely useless when users do not input their reactions and recommendations to the information they read. The approach used to correct this problem was by encouraging users to recommend Web pages through a good user interface design and the usefulness of information placed in the filter. 36 4.1.1 Maintain Low Overhead The overhead needed to recommend pages will need to be kept low in order to encourage recommending. If the process of recommending adds more time to browse the Web, users will not recommend pages. Therefore, the collaborative filter will be designed so that recommending pages will not hinder users from their normal browsing activities. 4.1.2 Support User Conventions The design of the collaborative filter system will cultivate recommending Web pages by making this process intuitive and a normal Web browsing activity. The challenge in designing the system is determining how to make the particular action, recommending, a normal Web browsing activity. Since the Web Navigator combined two environments, the Web and Notes, the designing process started by studying Web and Notes user conventions in order to know what was normal behavior in these environments. An initial usability test was run on a prototype of the collaborative filter (see the Testing chapter for more details) to study user conventions. In order to separately examine Web and Notes conventions, two types of users were tested; users that were experienced with both Lotus Notes and the Web and users that were experienced only with Lotus Notes. The users with Web and Notes experience were able to give more insight on Web conventions. Users with only Notes experience provided a clear idea about Notes conventions. From this usability test, several design issues became clear. In general, Web experts tended to focus on the Web. They did not explore any features of Notes but merely jumped from one Web page to another. Notes experts explored more. They tried to determine what Notes features were available in the system. When looking for information on the Web, Notes experts tried to use Notes views, folders, and other Notes features. 37 When trying to recommend a page, the place where a recommendation is made needs to be easily seen while a user is on the Web as well as from Notes. In the prototype, the recommendation was made from selecting a menu item called "Actions-Voice Your Opinion." In general, Notes users easily found the place to make a recommendation because they looked for Notes options. Web users used more time to find how to rate and recommend pages and sometimes did not even find the functionality. Both types of users desired the process of recommending pages to be light weight. In the prototype design, a Notes form completely covered the Web page that users were recommending. Many users noted that this form was too distracting. Also, in the prototype design, making a recommendation took at least one minute. Other calculations were performed each time the recommendations were saved. Those additional calculations helped to organize the recommendations at the moment they were saved. However, performing these calculations made the process of saving the recommendations slow. Some users said that they would not recommend pages with the prototype design because it was too much trouble, clumsy, and time consuming. This testing result emphasizes more the idea that the recommending overhead needs to be kept as low as possible. 4.1.3 Prevent the Recreation of the Problem Accessing the stored recommendations can not be time consuming or difficult. If many recommendations start to accumulate, users will not be able to linearly search through each rating because it will take too much time. The collaborative filter will then contribute to the original search problem rather than solve it. The collaborative filter will be designed so users will be able to read recommendations selectively. Finding the recommendations will be organized to mimicthe way people search and find information in every day life. For example, average ratings are used for many different types of media. For books, people check the bestseller list to find what the general public 38 feels are more interesting books to read than others. Newspapers list movie ratings which help people select the best movie to see. Also, people share ratings by word of mouth. People tend to trust a friend's opinion rather than a stranger's opinion and will more likely trust those friend's ratings than any other [ 16]. When deciding on the next movie to watch, a person might ask a friend who has similar interests what good movies to watch. Also, people search for information through subject catalogs. When looking for books, people usually go to a subject catalogs to find a selected number of books under a desired topic. Having the collaborative filter organize the recommendations that mimics the way people effectively search and select information will help users easily find and select Web pages to read. 4.1.4 Obtain Useful Information Because the success of the collaborative filter relies on people's reactions to information, the filter needs to collect reaction information that is useful to users as they filter information. Designing the filter to contain useful information will hopefully attract people to using the filter. In general, the information that will be obtained from users is attributes about the Web page that are obtained when a user reads the page. This information can help users have an idea before even reaching the page what the page is about and whether they want to read the page themselves. Finding what reactions are the most useful encompasses many different design choices. For example, a rating scheme represented by either words or numbers could be used. When using either numbers or words or both in the rating scheme, there needs to be mutual understanding by all users of what each word or number represents. When using words for rating pages, selecting clear words to use is a difficult task. For example, when Maltz [6] used the rating scheme based on the words "Terrific", "Good", "Ok", and "Terrible" for rating Usenet news groups, users did not know whether to rate articles that were funny and interesting, but completely factually incorrect, as "Good" or "Terrible."[6] When using numbers to rate pages, the issue is how fine grained the numbers should be. 39 The numbers can run anywhere from 1 to 5, or to 10, or 1 to 100. A rating scheme based on a very fine-grained system, such as to 100, could be unclear and meaningless. Writing word equivalents next to numbers could help to give the numbers more meaning but the issues about using words to rate arises again. Another closely related issue is how much rating information needs to be obtained from a user as he or she is rating. A simple word or number might be enough information to tell a user whether a Web page is worth viewing. Allowing each user to append personal comments to a multiple-choice rating would confirm and refine a person's rating. In designing the rating scheme, both the format of the ratings and the amount of information to obtain will have to be carefiullyexamined. What input data to obtain from a user also involves how people find information in every day life. Since users find information by looking into alphabetized subject catalogs, obtaining content categories from users as well as the rating is beneficial. One difficulty with allowing users to input their own category for a Web page is that people categorize differently [17]. A review for a ballet performance could be under the subject, "Arts" or "Entertainment." With the categorization subjective to each user, users can have difficulty finding all the possible pages for a specific topic if relevant pages are under a category that the user did not think to check. However, the pages can be under multiple categories which is beneficial because it increases the likelihood that users would be able to find a desirable page. A method of invalidating recommendations will be devised since recommendations could start accumulating requiring unnecessary storage space. The goal of invalidation is to remove the useless recommendations and keep the useful ones. A difficult decision needs to be made which is how to determine when a recommendation is useless. Recommendations could be invalidated by time but it could be valuable even though it is old. Also, another invalidation issue is keeping the recommendations up to date with the Web page. Web page authors are continually updating and modifying their Web pages. Once a 40 page is modified, the recommendation associated with the page is possibly not valid and therefore, not useful. A method of notifyingusers that a page has changed since the time of the recommendation is necessary. Also, there is the issue about whether to delete a recommendation reference when the Web page has been modified since the recommendation date. The recommendationcould still be valid and valuable to users even after the page has been modified. For example, a Web author could have modified a page because of spelling errors. With this type of modification, the Web page has not changed its content and the recommendation is still valid. If a recommendation is deleted for a slight modification of a Web page, a valuable reference to a page could be lost. Unfortunately, when obtaining useful but personal information from individuals, the privacy of the user comes into question. With people's privacy at stake, some people are not willing to rate pages especially if their recommendations can be read by anybody [6]. To encourage as many people to rate pages as possible, some security mechanism, such as anonymous recommendations, is an important consideration to the design. 41 Chapter 5 Design of the Collaborative Filter System In this chapter, the final design of the collaborative filter is described in detail. The chapter explains how the final design dealt with each design issue. It begins with a brief scenario describing how the collaborative filter is used in a normal setting. The scenario is used to describe how the collaborative filter works. After the scenario, specific details of the design are explained to show how the issues of the design were addressed. 5.1 Brief Scenario One particular case where the collaborative filter could be used is in the following scenario. A user named Winnie Liang has recently started work in a start-up company that is planning to design a VRML (Virtual Reality Modeling Language) Web browser. She does not know that much about VRML. She decides to use the Web to learn all about VRML and the current development in the area of VRML. Also, she wants to recommend and share the pages she finds on VRML in order to help other new hires quickly learn about VRML. She feels recommendingthe Web pages she finds on VRML would be beneficial because she knows what a new hire does not know about VRML. She uses the InterNotes Web Navigator and its collaborative filter feature to accomplish this task. 42 She begins by opening the Notes Web database and browsing the Web, jumping from one page to another. When she finallyfinds a Web page that has good information about VRMI41L, she decides to recommend the Web page. She goes to the action bar and clicks on the button named, Recommend (Figure 5.1). A small dialog box appears on top of the Web page as a result (Figure 5.2a). She inputs her recommendation by clicking on one of the numbered buttons, selects "Internet" as a category for her Web page, and places her comments into the list box. After placing her inputs into the dialog box, she clicks "OK" and the system stores her inputs, the page title and URL, time and date of the recommendation, her name, and a Notes doclink to the Web page in the database (Figure 5.2b). 4slaealmnaiesianeEgnaaals . This is the VRML part of our gallery. You can click on the pictures below to view them in 3D. Here you can see examples of our work some for our clients, other just for fun. These images were done in Softimage 3D and converted to VRML using the OZ soft2vrml converter. You can also go back to our main gallery or look at how we see the future with VRML at OZ-Records. Figure 5.1 A Web page from the InterNotes Web Navigator. The Recommend button is located directly on the button bar above the Web page. 43 A 1 File Nf~0-6 Lotus Notes - OZ Gallery I http:Jwww.oz.isOZ-Interactive/vrgallery.html Help Window Edit View Create Actions Text 1.3m.1Haiti ii.1 1~o-- l.Sl I A I :.. tme :.I0OZGallery !' "- i - - - __ . w, Aw oz-:...---' "'Pox < - > _ _ . reat! _ . ! UK!_w . I |::| 1 l ~;, 'R Iv _ ks- ar [.. Categy Cornmer;s:Arts ;o 'Yofnrs: -t': .,' ,:-- . i _..... ..' .~-.~ .. - This is the VRML part of our gallery. You can click on the pictures below to view them in 3D. Here you can see examples of our work, some for our clients, other just for fun. These images were done in Softimage 3D and converted to VRML using the OZ softvrml converter. You can also go back to our main gallery or look at how we see the future with VRML at OZ-Records. !Hely '- O1[Nonel ', -ifZREM ' ., ",iI', Figure 5.2a Screen capture showing the Recommend dialog box on top of a Web page. .-: ':-' J1--_ activeivrgalley.hfil i.:,;-,",e ,..........-;.,, .. ,.. ....... ! .:;~?~lh ~ 0,... -:-'--'f98 .....-::...... 2..... -: ,....: r ourVRMLbrowsel ,':::,.' ¥2:'I: ..,....-~-., .' .!' ? ' ;:-: ':f X: " - ':; . -i!i . This is the VRML part of our gallery. You can click on the pictures below to view them in 3D. Here you can see examples of our work, some for our clients, other just for fun. These images were done in Sofimage 3D and converted to VRML using the OZ soft2vrmnl converter. You can also go back to our main gallery or look at how we see the future with VRML at OZ-Records. " " " .. .J.'." je..,, ." -.-. , .. - -j-i Figure 5.2b In this screen capture, the Recommend dialog box is filled in. 44 '1--jd4 Because she took a long time to find a good page about VRML, she wants to see if other people had found pages on VRML. Also, the user wants to see her recommendations and how they compare with others. After closingthe window containing the Web page, the user chooses Recommended Views from the InterNotes Navigator (Figure 5.3). When the list of views appear, she selects the Recommended By Reviewer view (Figure 5.4a). In that view, she clicks with her mouse her own name, Winnie Liang, and finds the Web pages that she has recommended (Figure 5.4b). While in the view, she sees the name of her boss, the founder of the company, Barbara Mathers. Mouse clicking on the founder's name, she finds several recommendations about VRML. Several of the recommendations are listed with a 5, the highest recommendation. The user selects one of those recommendations from the view and the recommended Web page opens. Figure 5.3 Screen capture of the Web Navigator starting Home page. 45 In,(WEHUMMyn file Edit View Create 0401WI-INIT-TITINHWwrim Actions Window H lp b, Barbara Mothers b Bruce Kahn 1 Cun Yong Tan ivv, 1,Dve Newbold 1 Frank Pavelski 0,Gilles Carrier 6,Kid Spots 0,Personal Finance Scott Prather 1 Stephen Stein 0,Steve Bussell b,Teresa Deane VtInnie Liang Figure 5.4a A screen capture of the Recommended By Reviewer view. Here all the name categories are collapsed. 5.4b The Recommended By Reviewer view collapsible categories can be expanded. Through expanding one of the names in the view, you can see the categories that one user has recommendations for and expand further to read the recommendations of the user. 46 Since she does not know many people in the company, the user decides to check Recommended By Category view to see if any other good pages on VRML have been found. Closing the Web page, the user is brought back to the database Recommended views. She selects Recommended By Category view (Figure 5.5a). Viewing the listed categories, the user finds the "Internet" category. After clicking on the category in the view, she finds list of Web page titles (Figure 5.5b). She finds her own new recommendations that she recently created, her boss's recommendations, and several recommendations from other co-workers within the company. * * * I ),Art Computers , -- - - - Education * * * * * * * * Finance __[.I._ _*:71vI b Intemrnet , Recreation I.Et, Sience Figure 5.5a A screen capture of the Recommended By Category view of the Web Navigator. 47 IWM " r-u:Pm'-'r-4 -32=1 1 Edit View Create Actions Window file --.. v - d;, r-Ifi.-4 tt~~~~~~ ^-f·iYS--4-W & tI I-Z '*- I -V I -- a-;- ,3 -I· * * * * ., * T1L~EO~~~.~~~~~P~~~h.l-r a. IE&iIAIMRINI,.,="-,;ii.-A I - * * * ,', I..I Help 114_-jle1 _y WO i rd - - tr-lzvAlwmimw~ mmW.-MM4e j_~~5m v Arts ·· ` g -rrryrgrR I Eli -. _ W-s-- -- _ · r% _ -- - _ -- os F _' - _,~,~':~",~ _ ' " a~a,~;T' _ _ A yaet rna e to g forr e brow:,er. ---- ., 'Ag 4 VRMLforr,HELL * biSar', te . - !,kstrV! ,:et' S . .o. r,. * -. ,. ~J' I~ntemnet i .- ; _Finance g _ C_ r- Sey ; , b Education I -. 47 / Computers UI Gai -I I,'. I~IPll F=--~P -Id I k Publications . -- ·------------ ··-- -------- I .. --- - -------· --- -·--- -·-·--- Figure5.5b The Recommended ·--- ------- ------- ·----- ·--- ----- -- ·--------------- -·-----···-·- y Category View expanded by one of the categories. Figure 5.5b The Recommended By Category View expanded by one of the categories. 48 --- · · -- The user decides to check the Top Ratings view to look at what her co-workers found the most interesting. Clicking on the Top Ratings view in the list of views brings her to the Top Ratings view (Figure 5.6). The World of Lily Wong 5 1 ~~.El......... I' Home o~S'H me"Pge Lotus Page ........... ..........4.5 3D Artist/Animator :'.;.~ 1 .,.,:<;*:-:*-:-:-:-.', 11W ===================.;:,::,*,* bj Internet Hourly News i.-t E 4 2 3 1i ~. + .....:-.-.''..-r, J~./.,;.. h : .. z.~: -,;-:. ,..-..?..-..v.-...--,% .- Figure 5.6 A screen capture of the Recommended Top Ratings view. 5.2 Reducing Overhead In several ways, the design of the collaborative filter system makes recommending efficient. The use of the button named Recommend is one of those ways (Figure 5.1). Having a recommendation performed through a simple mouse click on a button suits the Web environment well because most of the user interface of the Web is simply a point and click with the mouse. Also, the word, Recommend, clearly explains the purpose of the button. Having the button clearly shown with each Web page is important. It is clearly in 49 the sight of the user as he or she is browsing. Therefore, users directly associate the button with the Web page. Having recommendations performed through a button accommodates the user conventions of both the Web and Notes expert and the pure Notes expert. Having the button visible while browsing helps the Web experts see that the recommending function exists and they can easily try out the functionality by clicking on the button. Since Notes expert users tend to use Notes features, they will likely use the Recommend button. The lightweight dialog box where users input their ratings and recommendations was used because it least disturbed the users' browsing. The dialog box is small enough so the user can still see the Web page when the dialog box appears (Figure 5.2). The dialog box is compact and only requires users to input three valuable pieces of subjective data which are their ratings, the content of the page, and their comments. Being able to see the Web page after the dialog appears is helpful. In this design, recommendingbecomes more a part of browsing where recommending is similarto taking notes about interesting pages of a book. To make recommending fast, the overhead for making recommendations was minimized. For the final design, additional calculations other than saving the user input values were executed through Notes agents. At scheduled times, these Notes agents would run independently on Notes servers. For example, the averaging agent which averages the ratings for each Web page is run once a day. The calculated averages are used for the Top Ratings view. 5.3 Organizing the Recommendations The system has been designed to make the recommendation retrieval process easier. Notes views have features which organize the recommendations, making retrieval easier. Each Notes view, which is used to view the recommendations and their associated Web 50 page, was created to mimic the way people successfullyfind interesting or relevant information. Also, the information that is presented in each of the views has been selected because it is needed by users to make a reasonable judgment whether to read the Web page. There were three ways of finding informationthat were determined and implemented into the collaborative filter system. One search method which was implemented into the system using Notes views is finding information through a trusted expert. This search method was implemented into the system for several reasons. In O'Day's and Jeffries' study on patterns of search result sharing [16], one of the patterns observed was that a member of a team would act as a consultant sharing his search expertise with other members of the team who requested certain information [16]. This behavior is often seen in a library where people go to a librarian for help to find information under a certain topic. Also, during the first usability test run on a prototype of the system, a user noted that she would only view her friends recommendations to see what interesting URLs they had come across. This search method was implemented through the Recommended By Reviewer view (Figure 5.4a). The view categorizes the recommendations by the name of the user who made the recommendation. Under each name, the recommendations are categorized by content. Expanding the subcategory further shows the title of the Web page, the rating given to the page, and any comments that the reviewer made about the Web page (Figure 5.4b). Upon selection of a particular rating from the view, the associated Web page is retrieved for a user to look at. The organization of this view and the Notes view features help users to find the information they are interested in. A user does not need to search all the recommendations made on Web pages. Depending on who the user knows and his or her interests, the user can expand categories and view accordingly. Also, since there generally will be more pages than the number of people recommending, users will be able to easily 51 scan through the names and select those names that would be the most useful to them to find the pages that they need. Another way of searching for informationthat was implementedin the collaborative filter was finding information through subject catalogs. This search method was implemented for several reasons. When people look for information,especially in libraries, they go to alphabetized subject catalogs. Knowing what subject they are looking for or are interested in, people can go to the subject listed in the catalog and find a subset of information in the library that has the desired content. Also, in the usability test run on a prototype of this collaborative filter, when searching the Web pages for information under a specific topic, users were looking for a Notes view where they could see a list of Web pages organized by content. This search method was implemented in the Recommended\By Category view. The view first categorizes by the category that reviewers select when recommending Web pages (Figure 5.5a). In the view, these categories are alphabetized. Selecting one of these categories expands to a list of Web page titles which are each listed with a rating and comments (Figure 5.5b) and sorted in descending order of the ratings. This organization helps a user check out the best pages under a specific category. A user opens the Web page by selecting a recommendation in the view. Because there are a smaller number of categories than there are Web pages, this view helps to organize the Web pages into smaller parts that a user can view. Users generallyknow the subjects they are interested in or need so they can use this view to find Web pages that are relevant to them easily. The last method of searching that was implementedin the collaborative filter was finding information by looking at top rated information. This method was implemented for several reasons. There are many types of media that is organized by top ratings. In newspapers, they list the general ratings for recent movies so that people know what movies are worth watching and what movies are not successful. Books have bestseller 52 lists which tell people which books have been bought the most. Usually, those books that are selling the most are interesting to read. In the collaborative filter, this search method was implemented through the Top Ratings view. This view is not categorized but sorted in descending order of the average rating for each Web page. Only the first ten highest average ratings are selected into this Top Ratings view. Also listed in the view is the number of people who have rated the page and the title of the Web page. The averaging and summations are calculated regularly once a day through a Notes agent called the Averaging Agent. 5.4 Obtaining Useful Information Each field value used in the recommendation dialog box was selected carefully for usefulness. The values that are obtained from users through this collaborative filter are a personal rating for the Web page, personal comments about the page, and a category keyword that the user thinks describes the content of the Web page. The additional values that are obtained by default are the Web page title, the Web page URL, the name of the person rating the Web page, and the date the rating was made. These values help users make the decision whether to read a Web page. Also, they are characteristics of the Web page that can only be obtained when a user reads the Web page. By viewing them, users can have some information in advance about the Web page. The rating scheme runs from one to five. The scheme only runs from one to five for several reasons. One reason is that a simpler, smaller scheme seemed to be more usefiul. When running initial usability tests on a prototype of the collaborative filter which used a rating scheme running from 1 to 10, some users commented that the scheme, although good, could be shorter. Some even said a simple good and bad could be sufficient. A one to five rating seemed sufficient for this design. With this 1 to 5 scheme, users could rate a page neutral (a 3 rating), rate a page as somewhat good (a 4 rating) or somewhat bad (a 2 rating) or rate a page as a very good page (A 5 rating) or a very bad page (a 1 rating). 53 Removing the rating scheme from recommendations has been considered since many of the users in the first usability test noted that they would not rate pages xcept the very good ones (4s or 5s). However, because it was determined that average ratings help users find information, ratings were kept in the final design of the filter. Personal comments were taken as input because personal comments help to clarify the ratings, one to five. With personal comments, users can specificallyindicate what was good or not so good about a Web page. Having comments help others understand what the reviewer's opinion is about the Web page. For example, one user in the first usability test noted that he did not know whether the rating was for a page's content or layout design. Through personal comments, this users can specifywhat his rating was describing. A finite list of categories is not enough to categorize all Web pages, especially with subjective categorization. Therefore, the user can chose a category from the list given in the combo box or the user can create his or her own list. Allowing users to input their own categories will help meet the specific needs of each user and therefore, make categorization more useful for users. For example, a group could chose a specific category name to place all the recommendations about Web pages useful for a group project. The group then has a mutual category in the view to look for specific information for the group's project. A method to invalidate the recommendations has been implemented into the system. Through an administration form created in the Web Notes database, users chose whether to invalidate recommendations or not and the time in which recommendations are invalidated (Figure 5.7a, 5.7b). This administration form is seen by the database administrator who manages the Web database for a particular group. The actual invalidation and deletion is done through an agent which runs on a selected time schedule such as daily or monthly. Based on these options placed on the administration form, the agent will delete those recommendations that have been in the database longer than the selected date. 54 ~ I :'m File Edit View . mrxI* Actions Create I I Text Window Ii I 61 Help O~~~~~i~~~ii I- ! II I - II I Server Basics inteNotes Serve -Yoa SeverNameHere I I ll II I I I i II Document Managemet Prge ocue Ye name Manun database500 MB size Maxe rn 25 concurentusers: Services: rHTTP,FTP.GOPHER J Serveroggng OffJ Deleteqexped docuner. Expe docunmers dder : Expiedocumes argertharr Pxgelowhat d rnmnl database 'Yes 30Days_, Over256KB:1J :-8 size: SMTPDonair Deae i:,ate warnin RawHTMLsave r JJ ptions: Delete rYeJ Recmmendation docurmets PLuge r60 Daysj Recomendation documents alterhow mandays: Refresh docurnerts:"DISABLED-1 i-s " ·- ·.- ·- · · · · ·-- I······--·lr · · ·-· · ···- ·-··· · s-·-· -· w--- . 41, _· _ · Figure 5.7a The Administrator Notes Formthat allows users to delete documents from the Web Navigator database periodically. "" ~'.al -'- -* _~'mst ~,1 File Edit View Create &V! *A El P.-II V f 2 P--VMI - § ri :n-nzteos _ ' Actions Iext Wndow M0,rAM 0 VA KOZE W m fo;-;: .~"~~ ~:'-~' *'':'' *; 'I^ lelp ail4a u--------------_------_------ I3=GUMM I4N IEO -S54XYC,~.vWt- IO 1s_ b 4i] *1 I i I J i q M.* -- - i i i ii Ii - Puge r 60DySJ: Reconvnendation Rn docunts afterhow marwdaw. Refresh d n r DISABLED 11MM-MOMEW-11-SaCOM i -mmillirmill1illill Figure 5.7b Screen capture shows the time period options that users have to delete ratings. 55 While this method of invalidation and deleting recommendations is out of the control of users other than the database administrator, other mechanisms within the database indicate to all users whether a particular recommendation is valid. Because a recommendation could be invalid after the Web page has been modified, all the recommended views have been designed to indicate when the recommended Web pages have been modified. A Notes agent runs daily to check each Web page that has been recommended. If the page was modified after the date the recommendation was made, an icon appears in all the recommended views beside the recommendation (Figure 5.8). With this indicator, users can decide whether or not to delete the recommendation. Figure 5.8 Recommended By Reviewer view showing the invalidation icon, which indicates that the recommendation beside it could be inaccurate. 56 Having the automatic invalidation optional and the expiration time chosen depending on the needs of the users is beneficial. Users know when a recommendation is not useful to them and by trying out different time settings, users find the best time that works with their view of when a recommendation is invalid. Making automatic invalidation optional helps prevent useful recommendations from being deleted. Since recommendations are an important part of collaborative filtering, deciding when to delete recommendations is chosen carefully. Users may find that automation invalidation deletes useful, valuable ratings regardless of the expiration time chosen. Removing automatic invalidation gives users the option to delete ratings selectively. Users can specificallyfind those recommendations which are not useful and delete them. This process takes more time but users will be more sure that useful recommendations were not deleted. In this system, maintaining the users' privacy in recommendations is not an issue for three reasons. First, knowing the author of the recommendation is important to a user. By knowing the author of the recommendation, users can assess the value of the recommendation. For example, users value and will therefore use a recommendation from a friend with similar interests more than a recommendation from an unknown person. Second, since making recommendations is not a requirement, users who desire to maintain their privacy can choose not to recommend pages. Finally, while running usability tests on an early prototype which maintained privacy by giving users the option to recommend anonymously, not one of the fourteen testers chose to be anonymous. Because of these three reasons, no mechanisms were implemented to maintain the privacy of recommendations. 57 Chapter 6 Testing As the collaborative filter was being designed, two tests were run on the system. Details about the two tests are described in this section. The results of these tests, especially the first test, helped guide the design of the collaborative filtering system. 6.1 Purpose of Testing The usage of the InterNotes Web Navigator collaborative filter was monitored throughout the project. The results from this monitoring helped to determine the strengths and weaknesses of this collaborative filtering system in several areas. They are the following. 1. Ease of use 2. Accuracy of the information in suiting a user's interest 3. Efficiency in delivering personalized information 4. Integration of collaborative filtering and Web browsing The usage of the collaborative filter was observed in two different tests. In the first test, users were given no prior knowledge about the collaborative filtering system. The users 58 were simply given the system and a set of tasks to do on the system. Depending on whether the system was easy to use, the users would successfullyor unsuccessfully determine how to perform their tasks with the system. In the second test, users were given an explanation about how collaborative filtering works and how to use it in the InterNotes Web Navigator system. After they read the explanations, the users were free to use the system for their daily Web browsing activities. The results of this test helped to determine the accuracy of the system in retrieving personalized information, the efficiencyof the system in retrieving personalized information, and the quality of the integration of collaborative filtering with the InterNotes Web Navigator. In the following sections, each test is described in detail along with a summary of the results collected. 6.2 First Test 6.2.1 Description of the First Test The first test is called a usability test. This test is used to help programmers design applications that are easy for end users to use. For this thesis project, this test was performed on prototype designs of the system to help create a system that was easy to use. The general issues that were addressed through this test were the following. 1. Is the user interface intuitive? 2. How well does the collaborative filter technique work into the normal way people look for information? -How do people view Notes and the Web? 59 -Does the collaborative filtering work into that view? 3. Is the rating scheme clear? 4. Do the users want anonymous/secure/private ratings? 5. What is the best information that is useful to the users as they filter? (see Appendix 1.1 for the detailed questions that were asked.) The test was run on users who had not seen the collaborative filtering system before the testing session. Each testing session was one hour. In a testing session, the user was given the prototype system and a set of scenarios and tasks to do with the system (see Appendix 1.2 for the exact tasks used in the test). In our tests, two users worked together in the one hour session to perform the tasks. In a couple of sessions, only one person worked on the given tasks. The users were asked to think out loud as they were performing the tasks. Users' comments, their screen behaviors, and their facial expressions were recorded on videotape. These videotapes were the data for this test. After the pair of users were finished with their hour session, fifteen minutes were spent asking questions and talking with the users to get their first hand reactions to the collaborative filtering system (see Appendix 1.3 for the post test questions used). This post test information was used to further clarify users reactions and to organize the testing data. Fourteen users were given this test. Two types of users were found for this usability test. All the users who will be using the Web Navigator fall under one of these two types. There was an equal number of both types of users. One day was spent on testing the first type of user and the another day was used to test the second type. Dividing the days of testing helped us to focus on observing one type of user at a time. The first type of user is experienced with Notes but unfamiliar with the Web. Such users clearly identify the social conventions of Notes users; they help to indicate whether the filtering system is intuitive to Notes users. The second type of user was experienced with Notes and the Web. These users provided valuable information on the social conventions of Web users and helped to indicate whether the filter was intuitive to experienced Web users or not. 60 6.2.2. Results From the First Test Each of the users' comments and behaviors observed during the testing sessions was recorded on paper and organized into three categories: frequent serious problems, moderate problems, and suggested enhancements. The results were divided between the two types of users. These results can be found in the appendix 1.4 and 1.5. The following general observations directly address the issues in the design. Is the user interface intuitive? - The user interface was very visual and therefore, users were comfortable browsing in the InterNotes Web Navigator. Users were able to move around by simply mouse clicking on icons. The words used in the system were crucial to understanding the functionality of the system. In the prototype system, users recommended Web pages through clicking on a button named "Voice Your Opinion!" This phrase was chosen to catch the user's attention and to encourage ratings. However, this phrase was confusing to all the users who mistook the meaning of the button. Users did press the "Voice Your Opinion!" button but mainly out of curiosity rather than to perform the task given to them. Many of the users thought they were "voicing their opinion" to the author of the Web page. This confusion was the result of this ambiguous phrase and the confusion the users had on the integration of Lotus Notes and the Web (this confusion will be discussed in the following section). The button text that was chosen for the final design was "Recommend" which seemed to best explain the function of the button. How well does the collaborative filter fit into the normal way people look for information? -How do people view Notes and the Web? - Users had trouble with understanding the integration of Notes and the Web in the Internotes Web Navigator. Users thought that there was no integration between the Notes and Web environments. They thought that 61 once they were using the Web, Notes features were not usable. Several factors contributed to this confusion. Users were not told about the integration and how the two environments were combined and used together. Also, there were additional feature changes to Notes in this latest version of Notes, other than the Web capability. Therefore, users would try to go to the Notes menu to find a feature they previously used in Notes. When they could not find it while using the Web capability, the users thought that all Notes features were not usable with the Web. Actually, the feature was there but performed in another location in Notes. - Does the collaborative filter fit into that view? - Because of this confusion between the Web and the Notes environment, users could not to find certain views and forms from the collaborative filter system needed to do the tasks given them. The problem was clearly that users did not know that Notes features were available from the Web Navigator. Because users did not know Notes features were usable, obviously, they did not find the views where the ratings, recommendations, and links to the associated Web document were accessed. The confusion of the Web and Notes integration also caused the uncertainty where recommendations were being sent. Once the users saw the Web, they focused only on the Web and what was seen on the Web page These problems show that the action bar feature of Notes is useful in this design (Figure 6.1). Instead of going to the menu to recommend a page, users can press a button located directly on the Web page. This Notes feature becomes an easy mouse click which makes the recommend functionality similar to the Web interface which is a mouse click interface. Although users did not find the collaborative filter system, it was encouraging to find that they wished the Web was more organized, whether categorized by content or some other attribute of Web documents. It was encouraging because what they desired could be found in the collaborative filtering capabilitiesof the system. Some comments that were heard throughout the testing sessions that shows what the users desired were the following. 62 File Edit Help Wd Acti Create View -IEile Edit iew Create Actions Widow ,-tj - 0 101006 11WAMIM -JOV.1 CM ,- IT-M, 7:K- Help I. 1 Home-: Open'1lI '1 'Ia M-K oad 1 'Rd Rel wdBookmark I This is the VRML part of our gallery. You can click on the pictures below to view them in 3D. Here you can see examples of our work some for our clients, other just for fun. These images were done in Softimage 3D and converted to VRML using the OZ soft2vrnl converter. You can also go back to our main gallery or look at how we see the future with VRML at OZ-Records. -. ._.'~~~~~~~4 ;I , -..,"-N"· · _ - P I . 4, - t _ . _ 1, , ; - _ I1 :1.1--- · . -· ---------~~. -11-. I . - Joff'otice/R EM`:,.!. · -iI Figure 6.1 This screen capture of an actual Web page in the Notes database. Directly on this page is the Recommend button which a user can mouse click on to make a rating. "This is a pretty lengthy process to get to magazines..." "Just so many layers..." "There was a lot of information thrown at me..." "..it's kind of like a big open field...I didn't know where to step next..." "...need something to cut out what you don't want and get you straight to that point." "...I guess that is why you would take it [a Web page] out and put it into your views,...so that your would not have to do this [browsing] each time..." - Is the rating scheme clear? - In general, users wanted the rating scheme to be more like the Web with graphical pictures and easy to use mouse click user interface. 63 Initially, the prototype for recommending was too straightforward. There were no graphical images and the form was text based. In the final version, the system design was changed so that ratings and recommendations wer.e given through pressing buttons in a more graphical form dialog box. Inputs were left as text inputs only when necessary such as personal comments (see the Design chapter for more details). Some users questioned whether ten numbers were needed to rate pages. Many users commented that they would only rate pages which were a ten, the best rating, or a one, the worst rating. A simple good and bad rating could only be needed. Other users said three numbers would be sufficient. Other users suggest allowing people to chose their own rating scheme. They also noted that the rating one to ten was clear but that users need to customize the scheme to their needs. One user thought it was unclear what aspect of the Web page a rating was associated to. The user said the rating could be referringto the content or the overall setup of the page, or a specific sentence used in the Web page. Overall, users understood the rating scheme and many of the issues were offered as suggestions for enhancements. - Do users want anonymous/secure/privaterecommendations?- In the prototype design of the filter, users saved recommendations through two buttons. One button saved the recommendation with the reviewer's name. The other button saved the recommendation anonymously. Throughout all the sessions, users did not press the button to anonymously recommend pages. A couple of users commented that anonymous recommendations were not useful. They said they felt the goal of the recommendations was to let others know what a person thought about the Web page. Therefore, a user who recommends pages probably does not mind giving his or her name with the recommendations. If the user did not want to let others know he or she recommended a Web page, the user would choose not to recommend pages. - What is the best information that is useful to users as they filter? - Users felt that the information that was used in the collaborative filter was sufficient to help users make a 64 judgment whether to read Web page or not. Users especially thought the category field was useful because it helped to organize the recommendations by content. The answers to the post test questions were organized in order to make it easier to study them. Similar answers to three of the questions were counted and placed in tables (see Tables 6.1, 6.2, 6.3). The totals showed some interesting results, Table 6.1 shows that many users thought the collaborative filter system would be very useful to them as they browsed Table 6.2 shows that many found it was easy to recommend pages. Table 6.3 shows that there were different opinions whether users would recommend pages. Some users commented that recommending Web pages was not something that they would do. said they would. Several users said they would only recommend the best Others firmnnly pages. These questions show that whether a person recommends pages or not depends on his or her personality. yes 12 no 0 uncertain 2 Table 6.1 Do you find recommended pages helpful? Would you use the recommended pages? easy 8 difficult 0 uncertain 6 Table 6.2 Was it easy or difficult to rate pages? yes 7 no 3 yes (only best) 2 yes (if easy) 2 Table 6.3 Would you rate pages as you browse the Web? The results of the test show that the users should be told clearly how Notes and the Web Navigator was integrated. This knowledge seemed important to the success of this collaborative filter system which was built through Notes features. Also, an explanation 65 about collaborative filtering, how it works and its purpose, is needed to let people know of its existence. After I told the users about the functionality, many of them said it would be useful and that they would have used it if they knew it was there. Finding out whether the collaborative filter system truly does customize Web information was the goal of the second test which was given to users over an extended period of time. 6.3 Second Test 6.3.1 Description of the Second Test After the first usability test, there were several questions and issues still unresolved. They are the following: -Will people find collaborative filtering useful? - Does the collaborative filter bring interesting and relevant pages to the user easily? - Will people use the collaborative filter? - Will people rate pages? These unanswered questions could only be answered by observing the collaborative filter system used in a normal setting. Therefore, a second test was developed for the final design of the collaborative filter system (see the Design chapter for a detailed description of the final design). This second test was given to users for a longer period of time. The goal of this second test was to study users' behaviors as they used the collaborative filter system for every day Web browsing. Therefore, for this test, groups of users were given the system and a brief description the collaborativefilter technique and instructions on how to use the collaborative filter system in Notes (see appendix 1.6). After using the system for about a month, the testers were contacted and asked a few questions to obtain their reactions to the system (see questions in appendix 1.7). The number of 66 recommendations that were in each of the testers' Web databases were counted (results are organized in Tables 6.4 and 6.5). In addition to this data, answers to questions about using the collaborative filter were also collected from the testers. The data for this test was collected from two groups of people. One group of users was called design partners. Design partners are people from companies outside of Lotus who use Notes version 4 test design software for daily activities in exchange for their opinions and suggestions for the new Notes system. Six companies were contacted and each company had at least one person using the system. The second group of users was a small group of five Lotus employees. Data from this group was collected with caution because these users were not from the same work group within Lotus and, therefore, the benefits of the collaborative filter system could not be fully attained. 6.3.2 Results From the Second Test There were not many testers for this second usability test. Because there were so few testers in this second test, the questions that were to be answered through this test were not answered confidently and completely. However, the data from this test helped to answer with some degree of certainty several of the unresolved questions from the first usability test. 18.104.22.168 Will People Recommend Pages? Because the usefulness of the collaborative filter relies on a large number of recommendations saved in the database, it was necessary to determine whether people would or would not recommend pages as they were browsing the Web. In general, the results showed that more ratings were made when more users were using the Web Navigator (Table 6.5). The companies with only one user testing the Web Navigator did not have any ratings in their Web database. In general, the companies with 67 more users had more recommendations. This result is understandable because, in these companies, users had others to recommend pages to and had recommendations in which to find better Web pages. Without other people to communicate with, making recommendations is useless. Some interesting results were found when counting the number of recommendations by the type of rating score it had. The majority of the recommendations had rating scores 3, 4, and 5 (Table 6.4). Only nine out of eighty recommendations that were made by both the design partner testers and the Lotus testers had rating scores 2 and I1. Only one rating had a rating score. This data shows that in general, people do not rate dull Web pages. Given this result, the rating scheme needs to be redesigned. The rating scheme could be removed completely since users share only the best pages. With the rating scheme removed, users can naturally assume that all recommendations are for good Web pages. Also, another consideration is changingthe meaning of the ratings scores. Currently, a 1 rating score represents a "poor" rating and a 5 rating score represents a "great" rating. In order to support the behaviors observed in the tests, a 1 rating score can be changed to represent a "good" rating with a 5 rating score kept the same representation, "great." Type of Rating Total number of ratings 1 1 2 8 3 34 4 17 5 20 40 30 20 10 Table 6.4 This table and graph show how m~i~ratings were made under each type of rating. 68 Group # # of users # of ratings 1 5 27 2 6 2 3 4 0 4 1 0 5 5 11 6 8 40 7 5 3 (Twoode groups notlistedhavethe sWe resuls thi one.) Table 6.5 The total number of ratings for each testing group 184.108.40.206 Does the Collaborative Filter Deliver Interesting Web Pages? In general, the users found collaborative filteringuseful for retrieving interesting Web pages. Out of all the recommended views, the view that users found the most useful was the Recommended By Category view (Table 6.6 and 6.7). yes 6 no 5 Table 6.6 Did you use the Recommended/By Reviewer view? yes 8 no 3 Table 6.7 Did you use the Recommended/By Category view? 69 About half the users did not find the RecommendedBy Reviewer view useful (Table 6.6). However, one users said that this view would be more useful in a workgroup setting. He made this comment because while using the filter one day, he found a name he knew. He did not find the By Reviewer view useful until he saw this name of his colleague. Because he knew her and her interests, he viewed her recommended pages. His comments show that the By Reviewer view would be more useful in a work group environment. A common opinion among the testers was that the collaborative filter system did not have enough recommendations to make it useful!. With only a few recommendations, users sometimes did not find recommendations under categories that interest them. Therefore, several users suggested ways to encourage ratings. Many users felt that if users initially found the Recommended views helpful in findinginformation, they would start recommending pages. One user suggested havinga large number recommendations stored at various Web sites which could be downloaded into their collaborative filter system. With a large number of recommendations initiallyin the collaborative filter, users would be use the collaborative filter system and see the benefits of recommendations even before they start recommending. Although some design partner companies did not use the collaborative filter system, they desired the functionality that the collaborative filter provided. For example, one company had a group of users who stored URL references in a separate Notes database. People working in the group would go to the database to find URLs which were stored for their relevance to a particular project. Also, when users were asked what they found useful in the InterNotes Web Navigator that helped them find information, users said they liked the ability to email Web pages and URLs to others. Many also said they found good Web pages by talking with others. These observations show that sharing good Web pages is a normal behavior for Web users. What is still left to resolve is to how to effectively use recommendations to filter Web information. Collaborative filtering was designed for this purpose. However, the difficultyis knowing whether people are willing to recommend pages in the way imposed by the current collaborative filter system. 70 Chapter 7 Conclusion 7.1 Analysis of Current Work Unfortunately the two usability tests that were conducted in this thesis were unable to show whether the collaborative filter system would successfully filter Web information in a workgroup setting. The small number of testers and the lack of good workgroup testing environments prevented the filteringcapability of the collaborative filter from being fully effective in the testing sessions. In several cases, there were very few users using the same Web Navigator so they did not recommend pages. Also, many times, the testers did not have the same work interests. Since the collaborative filtering technique uses collaboration to filter information, those group of testers who did not need to collaborate with each other did not filter information with the collaborative filtering system. However, the testing results show that users will find the collaborative filtering feature in the InterNotes Web Navigator useful. The testing results show that already users find sharing recommendations help to find information. In our second test, users shared recommendations in several different ways. Users shared recommendations through emailingURL references to others and through storing relevant URL references in a separate Notes database. On a more informal level, many people share URLs in everyday 71 conversation. From these observations, it can be concluded that people will take the time to recommend pages and listen to other people's recommendations to find the information they want. A weakness of the collaborative filteringtechnique not completely resolved in this thesis is that the collaborative filtering system is useless as long as there are very few stored recommendations. Because of this weakness, the system needs an initial period of usage to successfiullyfilter and customize information. The approach used to solve this problem was through devising ways to fill the system with recommendations quickly. The amount of time needed to accumulate a large number of recommendations depends on how often people recommend pages especially when a database is empty. Therefore, the system attempted to reduce the time by encouraging users to recommend. However, before the Web Navigator was filled with recommendations, many users did not want to continue to recommend because they would recommend pages but their actions were not necessarily returned. 7.2 Future Work Because there was evidence from the results of the tests that people do share recommendations to find good Web pages, the next step is to design the collaborative filter system to make sharing recommendations more a normal part of Web browsing. For this thesis, several design choices were made to help encourage the use of the collaborative filter such as designing the filter to make recommending pages easy. Future design work and enhancements can go in several directions. For example, the collaborative filter could be extended to incorporate what users do currently to recommend Web pages. While running the tests on the collaborative filter system, Web behaviors were observed that showed different ways that users shared good Web pages. Because sharing URLs through a Notes database is new to many users,, one pair of testers suggested that the system allow users to emailtheir recommendations in addition to saving them in the Web database. They suggested that the system add the capability to email 72 recommendations because users often emailto others URL references to good Web pages. Because users already email URLs, having the email capability will hopefiully cause people to make more recommendations. From this increased usage, users will be able to quickly see the benefits of saving recommendations in the Web database. Also, depending on preference, some groups within a company rely on databases to share information and other groups rely on email to communicate. Therefore, having both methods of sharing makes the system beneficial for more users. Different approaches in making a recommendation could streamline the whole process of recommending Web pages. There are other possible enhancements that can speed up the process of recommending Web pages. Several buttons named "great," "good," and "bad" could be placed on the Web page's actions bar. Once activated, these buttons would create a "great," "good," or "bad" recommendationimmediatelywithout additional user input. Another design consideration is a rating bar that appears every time a user jumps a hypertext link. With this bar, users have the option to immediately choose a 1 to 5 recommendation on the Web page by simplyclicking along the bar (Figure 7.1). _U9 Figure 7.1 A picture of a rating bar. There is also future work that is needed to explore ways to automate information searches to the Web. With the collaborative filter system, Notes agents, Notes search features such as full text search, and the search engine resources on the Web, the Web Navigator is capable to automate many tasks involving informationsearches. For example, a Notes agent can be programmed to use a search engine on the Web to obtain URLs about 73 "ballet" and place those URL references into a Notes document that can be later read by a user looking for Web information about "ballet." Ways to use the stored recommendations needs to be explored. Having stored recommendations create new possibilities for different forms of navigation and obtaining information from the Web. Several possibilities include third-party rating services, controlled access to Web pages, and automatic collaborativefiltering. For a third-party rating service, a person would act as an information consultant by finding and recommending the best information on the Web for a certain topic. With the recommendations stored in one Notes server, users could obtain those recommendations by using Notes replication capabilitiesto replicate the recommendation to their Web Navigators. With these stored recommendations, users would not need to take the time to find good Web information on their own. To control access to Web pages, selected individuals, such as an employer, recommend Web pages appropriate for certain environments such as an office working environment. Through the Notes application environment, a capability could be implemented where the users in the office would only be allowed access to those Web pages recommended by the selected individuals. With this capability, companies can limit employees to only work related Web pages during work hours. This capability is also be helpful in creating customized workgroup desktop environments where users selectively copy one of several sets of recommendations related to different projects. Then, users could view Web pages for different projects, one project at a time. Also, the recommendations that users accumulate as they browse the Web represent users interests. With this representation, an automatic retrieval service can be implemented to deliver information from the Web customized to users' interests as defined by these recommnendations. There are many possible uses of stored recommendations. One caution when using stored recommendations is that they can be out dated quickly since Web pages are continuously changing. As stored recommendations are used, ways to assure that the recommendations are accurate will need to be designed. 74 Lastly, collaborative filtering could take new directions as Web technology continues to quickly advance. Where the Web was once used only to retrieve information, changes are being made to create an interactive Web where users can gather at various Web sites to interact with other users through mechanisms such as chat forums [19]. Especially with the development VRML (Virtual Reality Markup Language) which will allow Web developers to create 3D Web spaces, Web sites will not only be a place to obtain great information. They will become like actual places users can go on the Web to interact with other users. In the next generation of Web, users will meet other users at a Web site and literally be able to ask "What are cool places to visit on the Web?" Users may someday be able to see on their own screen the actual places on the Web and, like choosing a good restaurant, choose the next place to visit by the number of people waiting to enter the Web site. As Web technology continues to advance, collaboration on the Internet may become too much like reality. 75 Appendix 1.1 1st Usability test - Goals and Objectives Main Goals- [some might not be completelyanswered by first usability test] - Is the UI intuitive? - How well does the collaborative filter fit into the normal way people look for information? -how do people view Notes and the Web? -does the collaborative filter fit into that view? -Is the rating scheme clear? -Do the users want anonymous/secure/private ratings? -What is the best information that is useful to the users as they filter? Specific Goals for: -Form -Is the form complete? -Is the form easy to understand? (simple and compact) -Is the form's rating scheme clear? -are the word explanations clear or too specific or not helpful? -are the numbers 1 to 5 sufficient or too much? 1 to 10? or 1 to 3? -Do they understand what the anonymous field implies in the form? -what behavior do they expect? -Do the users find it surprising the form is forwarded automatically when a name is placed into the "Forward To:" field? -Do people actually want to use the "Comment" field or do they find the rating scheme of 1 to 5 sufficient? -Do they find the preview window on the bottom (which shows the actual WWW page they are rating on) helpful or distracting? -View -What views do the users focus on / find helpful / want? -views available - Recommended -by Person -by URL -Subject/Category -What information are they expecting / wanting to see in views? -just the URL name and rating? -all the fields from the form? -all the comments? -Is the averaging information useful or do the users want to see individual ratings? -When they click in the view, what are the users expecting? -the original rating form? -the actual URL page? -a rating digest? 76 Appendix 1.2 1st usability test - Testing Scenario Background on the Intemotes Web Navigator: The InterNotes Web Navigator is a Notes database application that provides Notes users with easy access to the World Wide Web. The Web Navigator database resides on a Notes server that is connected to the Intemrnet.Notes clients can access the Web through this database without having to install anything extra (such as TCP/IP) on their computers. Task 1: General browsing and checking out views (Total time: 10 minutes) You have heard a ton about the Web in the past few months and you decide to check it out for the first time. You and your partner decide to use the InterNotes Web Navigator database to see what it's all about. 1. Take turns with your partner to browse through the database to see what's out there on the Web. 2. Experiment with the graphical navigator and the different views and see which ones you like best. 3. See if you can find a view that has pages that your coworkers have rated. Check out the pages your coworkers have rated. Task 2: Doing some research on the Web (Total time: 50 minutes) You know there is quite a bit of information on the Web. You and your partner decide you want to do a little bit of research on online magazines that are available on the Web. You want to check to see if magazines like Time and PC Week and other ones are on the Web yet. You're also curious to see if you can find any magazines that have to do with your favorite hobbies. 1. Start browsing the Web for online magazines. When you find some, share your results with all your coworkers, especially these three: Barb Mathers Christine Osik Teresa Deane 77 Appendix 1.3 1st usability test - Post-test questions 1. Ask about the general usefulness of the rating pages paradigm: What was the most confusing thing in this exercise? Which views helped you the most as you wcre browsing? Can you think of what kind of views would you have liked or would have been more helpful? Could you describe them? In general, what helped you the most as you were browsing the Web? Do you find the "Recommendedpages" helpful? Would you ever visit or use the "recommended pages" view? Why or why not? Was it easy or difficult to rate pages? Why? Would you rate pages as you browse the Web? Why or why not? 2. Show them the input rating form. Explain what the purpose of the form is. Ask these questions about it? Was the rating scheme clear? If not, how would you change it? What did you expect to see when you "VoiceYour Opinion? Was this input form as expected? are there other pieces of information that you'd want to put into this form? explain are there pieces of information that you think are not important to obtain? explain 3. Show them the output person form. Explain what the purpose of the form is. Ask these questions about it: is it easy-to-read and what you expected to see when you opened it up in the Recommended By Person view? does this form contain enough or too much information with regard to rating? What information would you like to see in these forms that would help you browse the Web? 78 4. Show them the output rating digest form. Explain what the purpose of the form is. Ask these questions about it: is it easy-to-read? does it contain the information that helps you see how a page was rated? are there other pieces of informationthat you'd like to see? Which output form do you find more valuable, the digest or the person form and why? 79 Appendix 1.4 1st Usability test results - session 1 Usability Test Description This report summarizes the problems encountered during the first of the InterNotes Web Navigator Usability studies. User type: Experienced Version 3 Notes users (some with Version 4 experience); little or no Web experience. Testing goals: This first usability test for the Web Navigator focused on the collaborative filter (that is, how people in a workgroup rate Web pages and recommend those pages to coworkers). We wanted to find out these particularsabout the filter: Is the UI intuitive? How well does the collaborative filter fit into the normal way people look for information? - how do people view Notes and the Web? - does the collaborative filter fit into that view? Is the rating scheme clear? Do the users want anonymous/secure/private ratings? What is the best information that is useful to the users as they filter? Testing scenario: The testing time lasted approximately 1 hour with a 15-minute post-test interview (see appendix 1.2). We tested six subjects in pairs and one subject alone. All testers followed a scenario (see appendix 1.1). Usability Test Findings We summarized the problems the testers encountered and categorized them according to their severity as follows: Frequent, serious problems Moderate problems Suggested enhancements Serious Problems 80 The following list represents areas of the Web Navigator that occurred frequently to most or all of the test subjects and caused serious confusion. Rating form: Where does this rating page get sent? Testers didn't understand where the rating page goes. "Where does this go?" in the Notes folder or back to the Webmaster? They thought you would have to move the rating page into a personal folder manually. When testers wanted to make a Web page available to their coworkers, they thought they'd have to manually put the Web page into a public Notes folder. Or, they thought they'd have to Forward the page manually. Testers were confused between Send field and Post button. Also, they were unclear about the difference between posting publicly and anonymously. "If it's posted publicly, why would you want to mail it to someone?" What's happening when I post a rating page? Testers wanted to see a confirmation that page is being posted to folder and mailed. Testers did not find blue hotspot to open Web page. Wanted a "click here" or arrow pointing to the URL link. Another user suggested putting a rocket ship icon next to the URL link to indicate that you could launch the page from there. Once they found it, they were psyched they could go to the Web page directly from the rating page. Placement of URL link on form - it should below the black line because above the black line seems to indicate that the URL link is not a link, but part of the title. Trying to figure out how to rate the page - they first went to Forward, then Create, then Voice your Opinion. Rating views: When user clicked on document in the view, user expected to go directly to the Web page. Want a content by category instead of view by average rating. Want content on left and rating on right. Testers suggested these views: 1. URL title, content, average rating, number of people who rated. 2. Subject, URL title, rating. Testers could not find the recommended views - they didn't know how to get to the recommended views from within Web pages. Search box: 81 Have simpler box with default search engine or have options to specify other search engines. Had to click in text field to get input focus. When testers press Return key, it dismissed the Search box instead of doing a search. Then, when testers clicked a search engine, nothing happened until they pressed Return. Testers did not know what Yahoo and other search engines stood for. "I was tempted to press one of those buttons, but then I was afraid because I did not know what they were." All the sub-lists from Yahoo search were confusing; testers wanted to get a quick list of magazines rather than diving into lists and sub-lists. "This is a pretty lengthy process to get us to magazines." "Just so many layers." "Just get me some titles." Confused about the four different searches: navigator search, menu item search, and action bar search, and Yahoo search. Testers wanted to be able to do a boolean search. They were unclear if they could enter several search words or a search phrase. Navigators: Unclear that the top icons opened a Web page directly and that the bottom group opened a view. Not sure what the top group of icons represented. When navigator was replaced by a Web page, the testers felt like they had "lost it" (meaning the navigator) and didn't know how to get back to it. They liked the idea of having a more oblong navigator that would be displayed all the time on the left with the Web pages or views to the right. They didn't like having to scroll or use the vertical bar to be able to see the whole navigator or to be able to see the Web pages or views to the right. After going down several paths, testers want to be able to get Home quickly. General/Miscellaneous: Testers did not understand that they were looking at a Notes doc that had been pulled from the Web - they didn't understand the difference between a Web page and a Notes doc. This led them to think they couldn't perform normal Notes features on the Web 82 pages they were looking at because they didn't think they were regular Notes docs. "It's nice that it looks the same, but if I have trouble, who do I call?" "I'm not sure when I've actually reached the Web and left the Notes product." "Isn't that wonderful - it looks like Notes." Testers didn't understand that they could do a full-text search on the documents in the database. URL name - they wanted to have the URL name of the page displayed at the top of the Web page - they didn't want to have to scroll all the way down to the bottom of the page to see the URL. Stop action - want a stop button to be able to stop loading URL - this was an overwhelming request. Want to get to the views from the View menu as well as from the navigator. Status bar: Don't want all the text messages to come flyingby; would rather have a blue bar only that shows exactly how much of the process of loading the page has completed. Text messages are very confusing; the right number always seems to change so it seems like you'll never get the page loaded. URL links: Want to be able to run the cursor over a link to have the URL appear; then the click will take you directly to the Web page. Want to be able to click once on URL links to go to the page; the navigator only requires one click and the URL links require two inconsistent. Moderate Problems The following list represents areas that occurred to one or more of the test subjects and caused moderate confusion. Rating form: Testers were confused about what "Rate this page" and "Voice your opinion" meant on both the form and the menu item. They suggested these as alternatives: "Rate this page to your coworkers" "Share what you have found" "Rate for a coworker" "How do you rate this page for a coworker?" 83 Who can they send this rated page to? Does this have the same behavior of Notes mail? Wanted to be able to display address book from this field. Testers weren't sure if we were rating the usefulness or content of the page? Keyword fields not intuitive - testers wanted to be able to add another category in Content field. Didn't want to see all the rating numbers next to the keyword list. Rating views: Some testers did not realize they had to refresh the view in order to see the pages they just rated. Testers liked to know who rated what, but didn't want to categorize on that in the view. Thought who they specified in the Send to field would be how rated pages would be categorized in the view, instead of categorizing the pages based on who rated the page. Search box: Testers want a powerful search that behaved like Notes and would do a full-text search. Testers ignored the Search button on our Action bar and used the Yahoo search button instead. Navigators: Thought the folders on the bottom of the navigator were folders where testers could drag and drop docs into. Suggested Enhancements The following list represents suggested enhancements that the test subjects offered during their testing time. Rating form: Suggested that the Title in the view for person rating form and the average rating form should be two different colors to tell them apart quickly. Rating digest form - want average rating and then the text of the rating, then on the next line have the number of people who have rated the page. 84 Testers followed a link and ended up getting a 404 page (page not found). They did not understand that it was an error page and not the actual Web page they were trying to rate. What does happen if a user rates a 404 page? Do we save a link to the 404 page or to the actual Web page? Testers wanted to see a more graphical and spiffy looking rating page. Rating scheme - change rating numbers to graphical icons that can be clicked on instead of having to enter a keyword. Wanted less numbers, like 1 to 4 instead of I to 5. Mixed reaction on whether testers would rated pages or not. How many exactly? Mixed reaction on whether testers would rely on what their coworkers rated as good and bad. Customizability - At least one user mentioned she wanted to be able to customize the rating form per site. Testers did not know which fields were required. Rating digest - want every comment on a separate line. One user wanted to have a much smaller rating form that was on top of the window and available as you browsed rather than going to the rating form every time you wanted to rate a page. Want a top-ten button that user could click on to bring up the current list of top-ten rated pages. Add field help to the rating page. Rating views: User wanted to categorize rated pages under more than one category. Wanted to be able to allow subsequent testers to add more categories to the rated pages as they viewed them. One user commented that she wouldn't use rating pages from people she didn't know. "If I don't know a person, why would I read their comments?" One user thought the actual rating was not important, but the category was - another user thought the person who rated the page was important. Want to be able to do a keyword search on recommended pages. 85 Navigators: Want to be able to customize the display of the navigator icons to take up less real estate. View by smaller icon and view by list desired. They like the Recommended graphical icon that has the thumbs up symbol - they thought it was very intuitive. But, they wanted the text to say "Recommended by" or "Recommended pages" or "Must see pages." General/Miscellaneous: Open URL box was a dreary looking box compared to the others. User didn't know what a URL was. Open URL box - couldn't remember the correct syntax when trying to enter a URL. URL links: Want to be able to use the right mouse button to bring up a Web-related menu as in Netscape. General comments and feedback or "The good news": The following comments were collected throughout the testing. We got the overwhelming feeling during the testing that the testers absolutely loved the product and couldn't wait to start using it: "I can't wait until they make this available; it looks like I'll have a lot of fun using it." Rating view: "I really didn't care if it had a good rating or not, I just wanted to find information." "I might go to the content field if I was looking for something specific and glance quickly to spot if others had found something rather than going through the long process of browsing." "I would say content view was hit or miss since it depends whether anyone put anything there." User didn't know what the magazine was, but then opened the rating page and saw the description filled out and then understood what it was. This shows that the user-filled in content description was useful. 86 Search box: Testers liked the idea of having the search box. Liked the fact that they can enter their own search text. Navigators: Testers liked the graphical look and feel of the navigator. "I see there are a bunch of icons for exploring!" General/Miscellaneous: Most testers understood what the spinningglobe was doing; they thought if they clicked it, it would cancel the document loading. "I like that little ball in the corner over there." Action bar: Testers wanted to be able to use it - "I feel inclinedto go up to that bar and click on it." 87 Appendix 1.5 1st Usability test results - session 2 Usability Test Description This report summarizes the problems encountered during the second of the InterNotes Web Navigator Usability studies. User type: Experienced Version 3 Notes users (some with Version 4 experience); experienced Web users (had used at least one other Web browser extensively). Testing goals: This second usability test for the Web Navigator focused on the collaborative filter (that is, how people in a workgroup rate Web pages and recommend those pages to coworkers). We wanted to find out these particulars about the filter: Is the UI intuitive? How well does the collaborative filter fit into the normal way people look for information? - how do people view Notes and the Web? - does the collaborative filter fit into that view? Is the rating scheme clear? Do the users want anonymous/secure/private ratings? What is the best information that is useful to the users as they filter? Testing scenario: The testing time lasted approximately 1 hour with a 15-minute post-test interview. We tested six subjects in pairs and one subject alone. All testers followed the attached scenario. Usability Test Findings We summarized the problems the testers encountered and categorized them according to their severity as follows: Frequent, serious problems Moderate problems Suggested enhancements Serious Problems The following list represents areas of the Web Navigator that occurred frequently to most or all of the test subjects and caused serious confusion. Rating pages: 88 Testers did not understand where the rating page was getting posted. Some thought the rating was going back to the Webmaster. Testers wanted some type of verification that the page was being posted or sent or something after they filled out the form and clicked one of the Post buttons. Rating views: Testers had trouble finding the rating page they just created. Some even tried finding it using the Search icon off the Home navigator. One group of testers rated a page anonymously and then couldn't find it under the Recommended View. (Perhaps have another category of Anonymous in addition to the other people's names and put the rating pages under there.) Testers thought the rating pages would be found in the All Docs view. General Navigation: [Note: This was the biggest source of confusion to this advanced set of testers.] Testers never understood the idea of single-window instancing. They thought that each time they followed a link, the Web Navigator would open a new page. This confusion began immediately after they went to the Window menu and saw open Web pages and thought this would be the best place to navigate from Web page to Web page (which follows the Notes paradigm). Then, it was only a matter of time before they ran into the "Too many windows" error because they just kept opening new windows. We purposely placed a Previous and Next menu item on the Actions menu, but they didn't use them very much (they preferred to use the Window menu to navigate around.) This caused major frustration. The History box was confusing to them because they thought it didn't store every place they went. (In fact, it didn't because they kept opening up new instances of windows every time they navigated using the Windows menu.) Testers all wanted a stop button. Testers were not sure if they forwarded a page if the URL links were saved on that forwarded page. Almost every tester went to Parent Preview from the View menu and didn't understand what they saw. They also went to Document Link Preview and were confused about that. This happened many times to many testers. Often, testers had the Parent Preview open and would get "linked doc cannot be found" error. One tester had the Sun home page open in the top pane and the bottom pane was empty. Whenever he clicked in the top pane, the page would jump. (This was because the page was an ISMAP and he didn't realize that for some time.) Navigator: Recognized inconsistency with single-clicking here and double-clicking on URL links within Web pages. Were unsure why certain links from here took to you Yahoo while others took you directly to another Web page. 89 Search box: Every single tester typed text into the search field and then pressed Return expecting something to happen. Most of them gave up when nothing was returned and went to another path of searching. Testers were confused about what kind of search criteria they could use (boolean). Bookmarks: All testers wanted a bookmark feature. They all looked for it on the Home navigator, from the menus, and on the buttons. They got confused when they saw the word "Bookmark" off the Actions-Workflow menu because they thought this referred to Web bookmarks. Most testers thought they could use the Create-Folder or Move to Folder to create a bookmark, but none of them did this successfully. They did create a private folder and moved some pages to it, but they never were able to find their newly created folder because they couldn't see the folders. Views: Testers all wanted to see the same views that are represented in the Home navigator available also from the View menu. They were very stuck with going to views only from the View menu, so they were stuck in the few views that were available from there. General/Miscellaneous: "The document is currently empty" error happened repeatedly throughout the testing. Moderate Problems The following list represents areas that occurred to one or more of the test subjects and caused moderate confusion. Rating pages: Testers saw the "Voice your opinion" button immediately, but didn't try it. Testers didn't like the text descriptions of the rating scheme on the form - they wanted the descriptions in the keyword list. When wanting to let coworkers know of a good page, testers all wanted to forward the page instead of using the rating paradigm. Some testers went to Create-Memo instead of using the Forward action. In the Digest form, testers didn't like the wording that says: "Average rating: 3 of 7 people..." In the Digest form, testers didn't see the date and time on the Anonymous rating and wanted to see that. Wanted the rating page to look more like an e-mail message if it was getting sent to someone. Felt the send field should say "send e-mail to." 90 Wanted to see a Post button with a checkbox for Anonymous instead of having two different Post buttons. As testers were composing a rating page, they saw the link bitmap and thought they could click it right there to go to the page. (Maybe make the link bitmap hidden while composing the form to avoid confusion.) Many testers wanted access to the Address Book directly in the Send field. Testers that found an already created rating page, tried to edit it and add a recommendation to it directly. (They did have access to edit it.) Rating views: Testers wanted to have a specific place where they could store their own rating pages. They thought this would be useful if they were doing research and found pages that they wanted to keep pointers to. Didn't want to worry about their own rating pages getting purged out of the database. General Navigation: One tester thought there was a major problem with the color of links within Web pages not changing color when you've read them, plus the color of the read documents in the view not following the same convention as Netscape. Here's the color comparison: Web Navigator Red (in view) - have not read Purple (in view) - have read Blue (in Web pages) - may or may not have read Blue (at bottom of Web page) - not a link (testers tried to click this link) Netscape: Red (any link anywhere) - have read Blue (any link anywhere) - have not read All the testers found the Open URL box, but they did have to look for it (they didn't see it right away). Almost all the testers typed in the URL without the protocol and felt that we should assign http:// as the default protocol if a tester didn't type it in explicitly. Some testers wanted to see a quick example on the dialog box of how they should enter the URL (with or without the protocol). All testers noticed the blue URL links immediately. Most testers wanted to be able to pass the cursor over the URL links to get the URL popup text and then be able to single-click to go to the page. Testers understood and liked the spinning globe. Wanted to be able to click on it to stop a page from loading. Testers wanted the right mouse button to be tied to Web Navigator functions. Search box: 91 There was some confusion about what could be searched directly in the database. One tester wanted a bar for searching, but didn't realize there was a search bar from View-Search. Actually, none of the testers realized you could do a full-text search on the pages in this database. Some of them realized you could just start typing to go to a matching document in the View. [Are we going to full-text index the database automatically or is that something the system administrator will have to do?] Wanted to be able to do a search by resource name. Views: All Docs view - they thought this was a list of all the sites available from the Web Navigator. They didn't understand this was a "history of sorts. A couple of testers realized this represented "cached" docs. (Seems like they wanted another word describing the view, such as "Database History" or "Previously loaded pages" or something.) Also didn't like that there was no hierarchy or content of these pages. It really seems like if this is a history view, it should be apparent from the View name that it represents history. And, there should be some sort of Categorized view that shows cached pages by content. Almost all the testers went to View-Switch form and got the error "No form in that direction." No one knew why that happened. Files view - the error pages were in the View, but there was no title in the View. Testers clicked on them anyway, but thought it was strange they didn't have a title in the View. Also, they wondered why these error pages were in the View at all. Files by Host - testers didn't like the ordering of pages under each host; they wanted to see some sort of hierarchy of the pages under each host. General/Miscellaneous: Testers all expected to see a URL window at the top of the page as you see in other browsers. Testers did not understand the Force Reload checkbox on the Open URL dialog box. While the page was loading, testers wanted to see a wait cursor. Suggested Enhancements The following list represents suggested enhancements that the test subjects offered during their testing time. Rating pages: In the Digest form, testers didn't like the color of the form and felt it was dull looking. In the Digest form, testers wondered how you could capture what multiple people said about the page. Testers did want to have the ability to both Post the rating form to the database and send it off to another tester. They wouldn't want to eliminate one of these two functions. 92 One tester suggested having an agent that runs at night to check and verify the links on the recommended pages and then e-mail a report back to show the bad links. Testers all liked that the rating page captured the URL link. Instead of "Voice your opinion", testers suggested "Post Recommendation to xxx View" or "Post to xxx View." Most testers didn't seem to use the post rating anonymously button. Rating views: Testers wanted to be able to view the pages based on the rating they got - have a category based on each rating number. Testers chose two categories when they rated a page and the view showed both categories separated by a comma, as opposed to having the rating page show up under both categories. General Navigation: A couple of testers wanted to be able to capture the URL from the popup box and then be able to paste that URL text into a mail message to send to another tester. They felt that it would be too much of a pain to have to copy down the entire URL by hand and then retype it into a mail message. Several testers wanted to be able to see the Web page as it's being loaded (like Netscape does). Navigator: "I like this GUI page." Wondered if the end-user could customize this navigator to modify the links. Recognized the magazines here as coming from Yahoo. Status messages: Testers liked them. They didn't have any gripes about the text messages coming up before the blue box and thought they were informative messages. Thought they were "pretty comparable to what I'm used to." Search box: One tester pressed F1 hoping to get help on the search box, but it brought up regular Notes help. This tester thought it might bring up dialog box help because it looks like a dialog box. Views: Testers liked the View by Hosts. General/Miscellaneous: 93 Whenthe Web Navigator is open to a Yahoo page, there are four levels of buttons and menus. That seemed to be a bit busy and confusing. Oh, we're still in Notes. That's nice.' In referenceto the integration between the Notes menu items and the Web Navigator menu items: I can't imagine someone using this just for the Web." Testers thought we should use Netscape as a standard whenever possible. One tester suggested having two sets of defaults for the look-and-feel of the product, one for beginners and one for advanced testers. In reference to performance: 'In general it seems to be fast enough." One group of testers brought up the idea of having white lists and black lists. One wanted them, the other didn't. Both were pretty opinionated about it. Testers liked the idea of being able to ESC to close pages and admitted that he wished Netscape allowed you to do that. 94 Appendix 1.6 2nd Usability Test- Introductions and overviews for testers Overview of the Web Navigator About 4he World Wide Web The World Wide Web (Web) is an extensive hypermedia system that interconnects hypertext data from many different Internet servers. By using Web browser software, you can access and display the data in the form of Web pages and can jump from one page to another by clicking the hot links (which can be identified by underlined blue text) embedded in each page. As you browse the Web, you literallyjump from one part of the world to the other with just the click of a mouse. Web browsers communicate with Internet servers using a network protocol called HyperText Transmission Protocol (HTTP). Although HTTP is the native protocol on the Web, Web browsers also allow you to communicate.withInternet servers running other network protocols, such as File Transfer Protocol (FTP), Gopher, Wide-Area Information Servers (WAIS), and so on. Web pages each have their own unique address, known as a uniform resource locator (URL). For example, the top-level Web page (or Home page) for Lotus is: http: //www. lotus.comn/home .html The first part of the URL (http) specifies the network protocol used to access the Web page. The second part of the URL (www.lotus.com) is the Internet server address. The Internet server could also be referred to by its Internet Protocol (IP) address instead of by its host name, such as http:/22.214.171.124/home.html. Web browsers can handle both formats. The third part of the URL (home.html) is the actual name of the Web page. About the Web Navigator database 95 The InterNotes Web Navigator is a Notes Release 4 feature that allows you to browse the Web directly from your Notes environment. However, the Web Navigator is more than a Web browser - it combines the features of a Web browser with the powerful capabilities of Lotus Notes to create an easy way for you to integrate Notes and the Web. You and your coworkers access the Web through a central database called the Web Navigator database. This database resides on a Notes server connected to the Internet, referred to as the InterNotes server. InterNotes server connecting to the Internet 1 Notes clients WeD Navlgator oataoase Each time you retrieve a Web page off the Internet, the Web Navigator translates the page into a Notes document and stores it inside the database. Then, the next time you want to read that page, you can open it directly from the database instead of retrieving it from the Internet. Opening a page that is already inside the database is much faster than retrieving it off the Internet. (Your system administrator can run scheduled agents that retrieve updated Web pages from their Internet server if they have been changed.) Once the page is in the database, you can view it, copy it into a private folder, cut and paste it into another Notes document, mail it to another Notes user, and so on. All of the capabilities of Notes are available for you to use with the Web pages inside the Web Navigator database. Tip The Web Navigator database is designed for you to read existing Web pages. If you want to create your own Web page, you can use the InterNotes Web Publisher which automatically translates Notes documents into Web pages. 96 Working Together to Help Find Information on the Web I would like to take this opportunity to describe to you a feature in the Lotus Notes Web browser which can help you work with others to find information on the Web. 1. Finding information on the Web is a problem Like an intricate spider's web, the World Wide Web is a vast amount of computerized information connected by hypertext links of URLs (Uniform Resource Locators). However, like walking into a spider's web, once a user starts to access the Web's linked information, he finds himself tangled in a large sticky mess of information, not knowing where specific information is placed in the Web. The problem with the Web today is that it makes available, in an unorganized fashion, an enormous amount of information. The amount of information being placed and accessed on the Web increases daily. The Web, started in 1990, grew from more than 4,600 servers in October 1994 to over 10,000 servers by December 1994. These servers provide retrieval access to over 1.05 million documents (statistics noted in, Ehrlich, K., Maltz, D. Pointing The Way: Active Collaborative Filtering, CHI '95 Mosaic of Creativity (May 7-11, 1995, Denver, Colorado), 1995 ACM, p. 202-209). Because the Web is growing in such large proportions, it is becoming more difficult to use the Web to its full potential. Users cannot possible scan all Web pages in a day. Finding something interesting on the Web has become a long process ofjumping from one Web page to another. 2. Collaborative filtering - How does it help find information on the Web? A collaborative filter feature has been placed in the Lotus Notes Web Navigator to help users to collaborate to find interesting informationon the Web. This feature takes advantage of what people have already found on the Web. Basically, the feature uses the concept that a person does not know a Web page exists or what is contained in a page until he or she finds the page or the person hears from another person, such as a friend, about the page. This valuable information is the foundation of the collaborative filter. As users browse and find interesting information on the Web, the collaborative filter allows a user to rate the page, categorize the content of the page, and make comments about the page. This information, annotations, is then saved with a link to the page so that other users can read these annotations and, if desired, directly link to the page without having the hassle of finding the page on the Web by jumping from one page to another. 3. How collaborative filtering works in the Notes How To Rate a Web page To rate a Web page, the page needs to be opened. After you have opened it and reviewed its information: 97 1.) Press the Recommend button on the action bar 2.) Once the rating dialog box appears, chose your Rating (by pressing one of the numbered, 1 to 5, buttons), select a Category using the drop-doxwnm combo box, and type in your Comments about the page into the list box. 3.) Click on "OK" The information you just input in addition to the Web page title, URL, your name, the current date, and a doclink to the page is now saved in the Web database. How to view the stored rating information and their associated pages. To see the ratings you created as well as other people's ratings: 1.) Go back to the Home Navigator (you can usually do this back clicking on Home from various places in the Web Navigator.) 2.) Click on Recommended views icon which is located along the bottom of the Home Navigator 3.) Once you get to the Recommended views, you will see on the left, name of views. The ratings are stored in those views which are classifiedas Recommended. The three views where you can view ratings are Recommended By Category, Recommended By Reviewer, andRecommended Top Ratings. Select one of these views by clicking on one of them. 4.) To view the page associated with one of the ratings, double click on one of the available choices in the view. The Web page associated with the rating should appear. In the Recommended By Category view, you can view the annotations in a categorized view by the categories that people had chosen for the Web pages. This view is useful if you are trying to find a Web page under a specific topic. You can go to this view to see whether other people had already found Web pages for the specific topic you are looking for. In the Recommended By Reviewer view, you can view the annotations in a categorized view by the name of person who wrote the annotations. This view is useful if you know a specific person is good at findingWeb pages that you are also interested in. In the Recommended Top Ratings view, you can view the annotations in a sorted view by descending order of the average rating per page. This view is useful if you desire to see the top rated pages. 4. Possible Ways of using the collaborative filter in the Workgroup setting There are many possible workgroup tasks that could use the Notes Web collaborative filter. A group could have a specific topic that needs research on. The group could create 98 a category in the collaborative filter naming the topic that the group is researching. As each member of the group is searching the Web for pages under the desired topic, he or she could save and annotate interesting and relevant pages by using the Recommend button. Members of the group can then go under the chosen category in the Recommended By Category view to view the Web pages other members of the group has found. Another possible task is where a person needs to research and give a report to his or her group about shopping on the Web. In order to save and make available to the whole group those Web pages which are very good examples about shopping of the Web, the person giving the report could use the collaborative filter to rate and comment about specific pages. Group members could then go to the Recommended By Reviewer view and find the name of the person giving the report to see what the person had found on the Web. 99 Appendix 1.7 2nd Usability Test- Testing questions Design Partner Testing Issues 1.) Will users find this collaborative filter useful? 2.) Will users rate pages? 3.) What are some things users like about the Navigator and what changes do they want? Company Contact: How many people using the Web Navigator: Possible other people to contact in group: Specific questions: 1.) Did you rate pages? If so, how many rating have you done so far? How many pages did you rate with a 5? 4? 3? 2? 1? 2.) Did you use the Recommended/By Reviewer view? If so, did you find it useful? Why or why not? 3.) Did you use the Recommended/By Category view? If so, did you find it useful? Why or why not? 4.) Did you use the Top 10 view? If so, did you find it useful? Why or why not? 5.) If you can you remember, how many Web pages did you view through the rating views with a 5? 4? 3? 2? 100 1? 6.) Did the collaborative filter (the Recommended views) help you to find pages that -interest you (hobbies, interests, etc.)? how? -useful to you (for work)? how? 7.) What else in the Web Navigator helped you find interesting Web pages or Web pages that you needed/wanted? 8.) What would you like in the Web Navigator that you feel would help you find information on the Web? 9.) What did you like about the Notes Web Navigator? 10.) What are some suggestions you have to improve the Notes Web Navigator? 101 References [1 ] Ehrlich, K. and Maltz, D., Pointing The Way: Active Collaborative Filtering, CHI '95, Mosaic of Creativity, ACM, May 7-11 1995, p. 202-209 [2] Fischer, G. and Stevens, C., Information Access in Complex, Poorly Structured Information Spaces, Human Factors in Computing Systems, CHI '91, Conference Proceedings (New Orleans, LA), Addison-Wesley, 1991, p.63-70 [3] Terry, D.B., A Tour Through Tapestry, COOCS '93, 11/93, ACM, p. 21-30 [4] Goldberg, D., Nichols, D., Oki, B., and Terry, D., Using Collaborative Filtering to Weave an Information Tapestry, Communicationsof the ACM 35,12 (December 1992), pp. 61-70 [5] "The Webhound Personalized Document Filtering System," http://webhound.www.media.mit.edu/projects/webhound/doc/Webhound.html [6] Maltz, D.A., Distributing Information for Collaborative Filtering on Usenet Net News, MIT Department of EECS MS Thesis (May, 1994), Also available as Tech Report MIT/LCS/TR-603 [7] Resnick, P., Iacovou, N., Suchak, M., Bergstrom, P., Riedl, J., GroupLens: An Open Architecture for Collaborative Filtering of Netnews, Proceedings of CSCW '94, ACM [8] Shardanand, U., Maes, P., Social Information Filtering: Algorithms for Automating "Word of Mouth," Conference Proceedings of CHI '95, May 1995, [9] Lashkari, Y., Metral, M., Maes, P., Collaborative Interface Agents, found In Proceedings of AAAI '94, also http-version, http://agents.www.media.mit.edu/groups/agents/papers/aaai-ymp/aaai.html, postscript version ftp -//media. mit.edu/pub/agents/interface-agents/coll-agents.ps [10] Malone, T.W., Grant, K.R., Turbak, F.A., Brobst, S.A., Cohen, M.D., Intelligent Information-Sharing Systems, Communicationof the ACM 30,5, May 1987, p.3 9 0 - 402 [ 1 ] Hill, W., Stead, L., Rosenstein, M., Fumrnas,G., Recommending and Evaluating Choices in a Virtual Community of Use, CHI '95 Mosaic of Creativity, May 7-11 1995, ACM, p. 194-201 [12] Sheth, B., Maes, P., Evolving Agents For Personalized Information Filtering, IEEE, 1993, p. 3 4 5 - 3 5 2 102 [13] Brothers, L., Hollan, J., Nielsen, J., Stornetta, S., Abney, S., Furnas, G., Littman, M., Supporting Informal Communication Via Ephemeral Interest Groups, CSCW 92 Proceedings, November 1992, ACM, p. 84-91 [14] "The Lycos Home Page:Hunting Web Information", httpn://lycos.cs.cmu.edul [ 15] The Organization of Lotls Notes Databases section used in the Lotus Notes Release 3 Application Developer's Reference, chapter 1. [16] ODay, V.L., Jeffries, R Information Artisan:Patterns of Result Sharing By Information Searchers, COOCS'93 (November 1993) ACM, p.98-107 [17] Berline, L.M., Jeffries, R., O'Day, V.L., Paepcke, A., Wharton, C. Where Did You Put It? Issues in the Design and Use of a Group Memory, InterChi '93 (April 24-29, 1993), 1993 ACM, p.23-30 [18] Bournellis, Cynthia. "Internet '95." Internet World, November 1995, p. 47-52 [19] Seltzer, Richard. "Personal Touch." Internet World, November 1 95 103