Samsung USP 7,973,773
` Exhibit 1026 Page 1
`
`FUTURE DIRECTIONS FOR HUMAN-COMPUTER
`INTERACTION
`
`BEN SHNEIDERMAN
`
`Department of Computer Science, University of Maryland, College Park, MD
`20742, USA
`
`ABSTRACT
`This paper offers a set of goals for user interface development and then
`scenarios of future developments. The applications include home control,
`hypermedia, office automation, digital photography, collaborative
`meeting/classrooms, public access, professional workstations, and medical
`record-keeping. Also, predictions are made for some of the underlying
`technologies such as User Interface Management Systems, remote control,
`flexible search, display devices, and touchscreens.
`
`INTRODUCTION
`It is dangerous, but necessary, to dream about the future. Dangerous
`because misguided dreams mislead designers, necessary because without
`vision navigation is difficult. Without dreams we risk stagnation, and lose the
`chance to make a better world. This biased and partial portrait of the future is
`offered to guide designers of future interactive systems. It is organized top-
`down, from goals to applications to software/hardware. Planning for the
`future is never easy, but when innovation is the propelling force, prediction is
`especially difficult. This portrait is mainly an extrapolation of current trends,
`shaped by the high-level goals, and colored with a bit of wishful thinking.
`
`GOALS
`The goals of interactive system designers are varied: productivity increases,
`reduced error rates, easier learning, and more consistency in performance are
`often cited and are relatively easily measured. But larger goals are often implicit
`in the broader work environment:
`
`1
`
`

`
` Exhibit 1026 Page 2
`
`• increased production of high quality goods and services at low cost,
`• improved user or customer satisfaction,
`• increased safety or health,
`• improved communication or cooperation among people,
`• better educational tools,
`• reduction in disease or famine, and
`• even global goals such as world peace.
`
`These work-related and societal goals might be expanded to include personal
`goals for users:
`
`• increased sense of self-worth,
`• empowerment to deal with large institutions,
`• means to communicate facts, ideas, or feelings,
`• capacity to innovate,
`• reduced stress,
`• opportunity to explore, and
`• relaxation through entertainment.
`
`Designers may debate the relative importance of these goals, and explicit
`discussion in public forums is strongly encouraged. As a professional
`community, I believe that we will be more productive and more appreciated if we
`engage in such discussion. However, the longer debate about underlying
`values is not the focus of this paper; specific applications and directions are.
`
`APPLICATIONS
`In the past decade improved user interfaces have opened up the door to the
`widespread use of word processors for writing, spreadsheet software in
`accounting, computer assisted design for engineering, and desktop publishing
`plus graphics in organizational communications (Shneiderman, 1987). Here are
`some predictions about which applications will be similarly expanded in the next
`decade. The first group of applications have personal impact such as home
`control, public access information resources, and medical record keeping. The
`second group contains home and work related applications such as hypertext
`and digital photography. The last group of applications focus on work related
`aspects of electronic mail, collaborative meetings, and professional
`workstations.
`
`Home controls & household automation
`Internationally, many companies have logically concluded that the next big
`market will be the inclusion of richer controls in homes (Time Magazine,
`January, 1989). Simple ideas such as turning off all the lights with a single
`
`2
`
`

`
` Exhibit 1026 Page 3
`
`button or remote control of devices (either from one part of the home to another,
`from outside, or by programmed delays) are being extended in elaborate
`systems that channel sound and audio throughout the house, schedule lawn
`watering as a function of ground moisture, offer video surveillance and burgular
`alarms, and provide multiple-zone environmental controls plus detailed
`maintenance records. Demonstrations such as the Smart House project and
`installations such as those by Custom Command Systems are a testing ground
`for the next generation.
`Some futurists and marketing types promote voice controls and home robots,
`but the practical reality is more tied to traditional pushbuttons, remote controllers,
`telephone keypads, and especially touchscreens, with the latter proving to be
`the most popular. Providing users with rich feedback and a clear sense of
`control is vital in these and most other applications.
`In our own studies (Plaisant & Shneiderman, 1989) we explored four
`touchscreen designs for scheduling operations such as VCR recording or light
`switching:
`1) digital clock that is set by pressing step keys (similar to "onscreen
`programming" in
`current video cassette players),
`2) a 24-hour circular clock whose hands could be dragged with the fingers,
`3) a 12-hour circular clock (plus AM/PM toggle) whose hands could be
`dragged with the
`fingers, and
`4) a 24-hour time line where ON/OFF flags could be placed to indicate
`start/stop times.
`
`****figure 1***
`
`Figure 1:This scheduler was most successful in our usability studies. The users select a date
`by pointing on the calendar and then drag ON and OFF flags to the 24-hour time lines. The
`feedback is a red line on the claendar and the time lines. (Copyright 1988 University of
`Maryland)
`_____________________________________________________________________________
`________________________
`
`Our results indicate that (4) a 24-hour time line was easiest to understand and
`use (Figure 1). We continue to make further trials with more complex tasks such
`as editing schedules, repeated events (lights on every Friday night at 7PM), and
`long duration events. Many interesting product suggestions have emerged
`during our trials, for example an alarm clock that would ring only on weekdays,
`thus avoiding the oversight that leads to a ruined Saturday morning when the
`alarm rings at 6 AM.
`Controlling complex home equipment from a touchscreen reshapes how we
`think of homes and their residents. New questions arise, such as whether
`
`3
`
`

`
` Exhibit 1026 Page 4
`
`residents will feel safer, be happier, save money, or experience more relaxation.
`Are there new notations such as petri net variants or role/task diagrams for
`describing home automation and the social relations among residents? The
`benefits to handicapped users and the aged were often on our minds as we
`designed these systems, since they may be substantial beneficiaries of this
`technology even though initial implementations are for the healthy and wealthy.
`
`Public access information resources
`From the early days of computing, there have been innovators who have put
`computers in public access situations such as airports, hotels, libraries, banks,
`museum exhibits, or stores. Many of the early design were difficult to use, slow,
`poorly organized (too many or confusing menus), hard to read (small displays,
`poor fonts, garrish colors), error prone (imprecise touchscreens), and
`unrealiable. Much has changed and there is the possibility of a new generation
`of public access applications, if users can be convinced to forget the bad
`impressions that have already been left.
`Automatic teller machines are remarkably more successful than they were a
`decade ago because of thorough attention to the user interface by some leading
`banks. Improved designs have been proven to lead to increased usage and
`greater customer satisfaction. A similar movement is in progress with respect to
`online public access card catalogs at libraries where the awkward first
`generation of designs are being challenged by designers who believe they can
`do better. Museum and exhibit designers are developing attractive computer
`and interactive videodisc projects that have the potential to revolutionize
`museum going by making the vast knowledge and resources of curators more
`accessible to patrons (Shneiderman, Brethauer, Plaisant and Potter, 1989).
`Commercial projects such as interactive sales information on sports
`equipment, tourist destinations, vacation resorts, real estate, shoes, home
`redecorating, and clothing are now succeeding more regularly because the user
`interface designers have been able to build on a growing body of knowledge
`and experience. We will undoubtedly see many more such projects.
`
`Medical records
`It is disturbing that technology has progressed rapidly in many areas, but that
`medical record keeping is quite similar to what it was a hundred or more years
`ago. While I am a great devotee of paper and pencil approaches, there are
`substantial benefits to having records in machine readable form. First let me
`make the case for machine readable medical records and then suggest some of
`the mechanisms for creating and manipulating the information.
`If all my medical history were kept in a standardized electronic database
`form, I would be able to more easily transfer my records to a new physician or to
`a specialist during a consultation. My wife recently needed surgery and had
`difficulty getting her current records delivered on time for each consultation, and
`never could succeed in getting vital records about two previous surgeries.
`While computers are no guarantee that 20 year old records would be available,
`
`4
`
`

`
` Exhibit 1026 Page 5
`
`there does seem to be a better chance of success. Certainly, if physicians could
`see records of recent lab tests, examinations, or consultations, there is a chance
`that better medical advice could be given.
`But now, let's assume a standard electronic medical history, and explore the
`benefits and dangers. Having good records of weight or blood pressure would
`enable monitoring of abnormalities or sudden changes. Within a community, a
`doctor might be able to spot a pattern of influenza or food poisoning that might
`be otherwise undetected. Early warning approaches might alert neighboring
`communties of potential problems so that vaccination or health emergencies
`might be called for. On a societal level, researchers could analyze data from
`large numbers of people to study the effects of smoking, obesity, exercise, or diet
`on a scale that was not imaginable till now. The pioneering Framingham, MA
`study would be dwarfed by longitudinal studies of millions of individuals rather
`than thousands. Of course, there are dangers of loss of privacy and violation of
`the doctor/patient relationship, but these have been dealt with effectively in the
`banking, credit, and telephone industries and there is every reason to believe
`that adequate protection is possible. In fact, considering the lax security in many
`doctors offices that I have visited, I might prefer electronic records to increase
`privacy protection.
`Once the idea of electronic medical histories becomes acceptable, then
`other possibilities emerge. Each citizen might carry a magnetic card with a brief
`version of their medical history and links to their full record. In an emergency,
`physicians would immediately have the relevant medical information
`(pharmacological information, recent electrocardiagram, reports on recent
`illnesses, etc.) and be able to contact the patient's physician.
`Data collection could become much more thorough. For example, records of
`workouts (pulse rates, duration), medication (frequency and time of day), minor
`illnesses (headaches, colds), even bathroom scales might be recorded
`regularly. While not everyone might want such a complete record of body
`weight, blood pressure, or temperature there are many situations (hypertension
`monitoring, fertility programs) that currently require more detailed histories over
`long periods. Research on the linkage between diet, exercise, or sleep habits
`on mental or physical health would be greatly facilitated by larger clinical
`databases. This seems like a grand opportunity.
`
`Hypertext and hypermedia
`Hypertext is rapidly emerging, but the idea of reading fragments of text on
`computer screens and following links still has a long way to go to gain
`acceptance (Conklin, 1987; Marchionini and Shneiderman, 1988; Shneiderman,
`1989; Shneiderman and Kearsley, 1989). Hypertext is most appropriate when
`there are a large number of short information fragments that cross reference
`each other, and when the users need to view only a small slice at a given time.
`Catalogs, diagnostic problem-solving guides, business procedures manuals,
`organizational guides, museum exhibits, encyclopedic sources, reference
`books, and cookbooks seem likely candidates for practical commercial
`
`5
`
`

`
` Exhibit 1026 Page 6
`
`applications, but there is already lively activity among interactive fiction writers
`and visionary hypermedia designers who will tap the affective and emotional
`side (Slatin, 1988). Hypertext authors are only beginning to come to grips with
`effective organizing principles that permit users to recognize the structure and
`contents.
`There is still much work to be done in improving the user interface for
`hypertext systems. Users need to be able to conveniently find facts they seek,
`browse for exploratory purposes, and rapidly discover when the database does
`not contain relevant information. Higher speed hardware, especially for
`graphics, database search, and network transactions are needed to avoid
`delays that distract the user from their exploration. Software improvements are
`necessary to:
`• simplify and speed the authoring process,
`• allow for automatic or semi-automatic loading of existing databases,
`• permit more powerful search strategies over large databases,
`• enable network linkage across databases,
`• support aggregation mechanisms for clustering related nodes,
`• provide the capacity to extract fragments or print an entire database, and
`• offer annotation and bookmarking.
`Graphical browsing methods with a visual representation of the networks of
`nodes and links have been advocated by many hypertext proponents. While
`prototypes with a few dozen nodes seem appealing, strategies for dealing with
`larger networks are needed. Hardware to permit rapid and continuous
`zooming/panning of large networks would help, but aggregation methods may
`be more effective or at least a useful complement (Conklin and Begeman, 1988).
`The addition of still frame and full motion video creates a compeling
`hypermedia environment that designers are only beginning to exploit. Imagine
`the potential for:
`• selectively viewing sports events, entertainment, or documentary footage of
`historical events,
`• exploring the human body by examining microscopic slides of diseased
`and healthy cells,
`• discovering foreign destinations by surrogate travel,
`• revisiting historical moments such as the signing of the Declaration of
`Independence or the coronation of Queen Elizabeth, or
`• studying electronic biographies of key personalities.
`Our efforts during the past six years have been directed at creating an
`information presentation version of hypertext that has been used in museums
`exhibits, hyperbooks (Shneiderman and Kearsley, 1989) for corporate
`information distribution, training abd education, information retrieval, scientific
`conference programs, and organization orientation guides. Our system,
`Hyperties, was developed in an IBM PC version (available from Cognetics
`Corp., Princeton Jct., NJ) and a SUN workstation version (see Figure 2) for
`research purposes.
`
`6
`
`

`
` Exhibit 1026 Page 7
`
`Figure 2: The SUN 4 workstation version of Hyperties with a database about NASA's Hubble
`Space Telescope. Users point, with a mouse, at bold faced text terms or at components of
`the spacecraft to find further text of graphic information.
`
`Digital photography and image handling
`The attraction to images in office communications is an instance of the
`general appeal of visuals. The next decade should see dramatic growth in
`digital photography and elaborate methods for capturing, storing, editing,
`sending, printing, and searching image databases. Scanners are making it
`increasingly easy, cheap, and convenient to digitize images, but still-frame
`digital cameras should speed the process. Existing systems have modest
`resolution (one million pixels compared to 18 million pixels in typical snapshot)
`and are too expensive for the consumer market, but change should come
`quickly here. The attraction is instant viewing, reusable media, editing, titling,
`and electronic dissemination.
`The news services regularly send color digital images for newspaper and
`magazine usage, allowing the photographer's output to appear anywhere in the
`world within minutes by cellular phone or satellite connection. Reduced cost
`and improved quality will attract a wide following of users who will eagerly edit
`images, mix with text, and send pictures of the new baby to grandparents
`electronically. Receiving stations could be FAX machines or personal
`computers that would allow further manipulations or storage. With proper
`indexing and hypertext linking a family album or visual genealogy might be
`copied or sent electronically to neighbors, distant relations, or friends.
`
`Electronic mail and office procedures
`While text processing is already widespread and has reached a certain
`maturity, image processing is still rare, costly, and difficult. The dramatic
`emergence of FAX machines is an indictment of our failure to provide good user
`interfaces for image manipulation. If system developers had developed
`convenient mechanisms for creating, sending, browsing, storing, and searching
`images, electronic mail might have grown much more rapidly. While there are
`certainly numerous appropriate applications of FAX technology, many offices
`create documents on word processors, print them, and send them by FAX only to
`have the recipient re-enter the text because it is easier than dealing with
`electronic mail.
`It seems clear that electronic mail systems will improve and spread. Part of
`that improvement will be the more common inclusion of graphics, even color
`graphics. Further refinements will be to include spreadsheets, databases,
`hyper-documents, animations, and sound. It took more than 20 years for
`standardization of ASCII text, so it may take that long for standardization of these
`
`7
`
`

`
` Exhibit 1026 Page 8
`
`more complex structures, unless there is a strong consumer movement or
`compelling forces (government pressure or international competition) for
`industry cooperation. Voice mail will continue to expand and provide an
`appealing, more personal, easy-to-use, and accessible alternative.
`Electronic filing of correspondence and enhanced search capabilities seem
`within reach (Malone et al., 1986). There are numerous Personal Information
`Managers that are suitable bases for such development (PC Magazine reviews,
`December 13, 1988) and the relevant hardware/software is improving steadily.
`Advanced facilities such as coordinated commenting on documents by multiple
`users in parallel, even simultaneously, should become available for use when
`appropriate.
`Active routing slips, which provide for sequential review and approval would
`make a nice additional function. When the first recipient had seen the document
`and affixed an approval it could be automatically routed to the next recipient,
`with a confirmation sent to the orignator. Confirmations might be hidden or
`made visible to successive recipients. These pre-specified procedures might
`be called routing slippers, suggesting that the documents are gently walked
`through the sequence of recipients.
`
`Collaborative meeting rooms and classrooms
`Thoughtful networking of multiple personal workstations can lead to exciting
`social environments for business meetings or education (Kraemer and King,
`1988; Mantei, 1989). Many businesses and universities have training rooms
`with multiple personal computers but each person works separately or watches
`as the instructor demonstrates using the large screen projector. Networking to
`permit shared printers, file servers, or electronic mail is a step in the right
`direction, but the restructuring of social roles is a necessary preliminary to the
`development of collaborative meeting rooms or classrooms. Could we call them
`collaborooms? or maybe co-rooms?
`For example, if each participant could send their screen to the large shared
`projector, then there is the opportunity for more equal participation. In business
`meetings, participants could each show their sales projections or could add
`items to the emerging set of concerns or meeting minutes that are communally
`created and critiqued. In teaching, students could show clever or poor solutions
`for an exercise to the entire class or could add suggestions for a reading list that
`was being created. When the meeting or class was over each participant could
`copy the results to their files or print them.
`If the presenter in a collaboroom has a set of notes, they could be sent
`electronically to each attendee for copying, annotation, or printing. No more
`waiting for paper copies to be made, or worries about updating each paper copy
`when a change is made. Online votes are another nice feature of a
`collaboroom: the choices are shown on the screen, each participant simply
`selects with a mouse click, and the results are immediately displayed for all to
`see. Straw votes can be taken quickly to assess the mood and then binding
`votes can be taken for the record.
`
`8
`
`

`
` Exhibit 1026 Page 9
`
`Parallel human processing by having forty or more people work together on
`a problem is an attractive possibility. Imagine if a team of managers working in
`parallel could individually study a fraction of the reports from hundreds of
`franchise stores to spot high performing locations and try to identify the
`determinants of success. Imagine if students working in parallel could
`individually review a fraction of the 800 abstracts on "touchscreen applications
`in medical care" retrieved from a bibliographic database and select the 20-30
`relevant to the class discussion.
`Could programmers collaborate more effectively in design reviews or code
`inspections if they could easily show pieces of code to the entire group? Could
`students write poems collaboratively or engineers create Computer Assisted
`Designs for VLSI circuits cooperatively. Early successes at sites such as
`Electronic Data Systems and the University of Arizona are encouraging
`exploration in these directions.
`
`9
`
`

`
` Exhibit 1026 Page 10
`
`Professional workstations & visualization
`Professional users are likely to see rapid changes in the next few years, with
`increased screen sizes, higher resolution, multiple window facilities, faster
`response time, and improved use of color. Underlying technologies of database
`systems, networks, multi-media, sound input/output, and multiple concurrent
`processors will support exciting possibilities in engineering and design
`workstations, scientific and scholarly research tools, and exploratory
`simulations. As Ted Nelson has said: "If computers are the wave of the future,
`displays are the surfboards." We will become more aware of the truth of this
`aphorism as designers demonstrate remarkable visualizations of many
`phenomenon and domains of discourse. The bandwidth of the visual media is
`far greater than sound or other media and the perceptual skills of the human
`brain are keenly tuned to the visual (Foley and vand Dam, 1982; DeFanti and
`Brown, 1989). I conjecture that as screens get larger and higher in resolution,
`we will give up the Turing test as a meaningful idea and recognize that the goal
`of computer design is to offer users' a clearer vision rather than a mediocre
`conversation.
`Natural goals for visualization are scientific phenomenon that are
`microscopic (cell processes, molecular layouts, or crystal lattices) or
`macroscopic (geologic formations, planetary motions, or galactic collisions). But
`other goals are to show events that are not normally visible (clear air turbulence,
`computer program organization (see Figure 3), tectonic plates movements, or
`subatomic particle interactions). Helping physicists develop intuition about
`relativistic phenomenon at the speed of light or quantum theory within sub-
`atomic particles would be other goals. But novel possibilities of exploring
`emotional spaces (can we convey the depths of sadness or the heat of
`passion?) or knowledge spaces (how would we show the gulfs between
`humanities and sciences or the overlap between Eastern religious mythologies
`and Western psychologies?) might really expand our consciousness.
`
`Figure 3: A NeWS programming environment on the SUN 4 workstation. Windows with
`labelled tabs are shown on a vertical stacking bar on the left. Then a complex data structure
`is displayed on the right, but each item is selectable and can be modified, even while the
`program is executing. Control is through the hierarchical pie menus in the center.
`
`UNDERLYING TECHNOLOGY: HARDWARE/SOFTWARE
`User Interface Management Systems
`One of the most important emerging technologies is User Interface
`Management Systems and its variants such as toolkits, rapid prototyping tools,
`and user interface extensions (procedure libraries or language additions) to
`existing tools (Sibert, Hurley, and Bleser, 1988; Hartson and Hix, 1989). There is
`a longer history of forms building and menu management software, but the
`newer approaches handle graphic designs, pointing devices, and even
`
`10
`
`

`
` Exhibit 1026 Page 11
`
`animations. These tools will have a profound effect on the pace of user interface
`developments because they provide approximately an order of magnitude
`increase in productivity where they can be applied.
`UIMSs will play a role quite similar to the Data Base Management Systems
`(DBMS) that emerged 15 years ago (Carey, 1988). In the same way that data
`independence allowed information architects to deal with the logical structure of
`data (entity-relationship model, relational model, etc.) and avoid concerns about
`disk allocation or index management, now dialog or user interface
`independence will allow user interface architects to deal with the interface
`design and avoid concerns about touchscreen drivers or cursor positioning
`algorithms. This separation of function can lead to much more rapid
`development and higher quality systems.
`Along the way, we are seeing the emergence of new notations for describing
`interactions, methods for ensuring consistency, improved review processes,
`better project management, earlier and more effective usability testing, and
`simplifed maintenance procedures. This field is still new, but already it shows
`promise of becoming one of the major software classes such as operating
`systems, compilers, or database management systems. Progress is needed in
`dealing more effectively with visually-oriented direct manipulation interfaces
`(Shneiderman, 1987)
`
`Remote control - do it there, do it then
`An exciting opportunity exists in designing remote control mechanisms. The
`remoteness may be in space, that is, controlling equipment at different locations,
`or it may be remoteness in time, controlling equipment operations at future
`times. The popular hand-held remote control units for televisions, videotape
`players, and stereo systems are first steps towards more complex remote control
`devices that will allow operation of devices across telephones or high speed
`networks. Simple devices such as being able to operate slide projectors or
`computer displays in other cities will enhance the possibilities of "reaching out to
`touch someone." There will be useful home control products that will permit
`remote operation of air-conditioners, burgular alarms, or other appliances.
`Remote access to information will continue to grow rapidly so that users will
`come to expect it and even not be aware nor care that automatic bank teller
`machines, phones, and credit card machines are accessing databases that are
`geographically remote.
`On a more complex plane, NASA is pursuing telescience to allow ground-
`based scientists to operate experiments or telescopes on orbiting laboratories
`(Sheridan, 1987). In general, I expect tele-operations will become more
`productive and exciting than autonomously operating robots. Tele-operations
`offer the attraction of greater human control, more flexibility and the rewards and
`satisfaction of accomplishment. Micro-tele-operations, will flourish with doctors
`controlling small surgical devices inside the human body or engineers making
`micro-miniaturized circuits or devices. Macro-tele-operations will allow huge
`signs to be painted or remote-controlled aircraft to do skywriting or
`
`11
`
`

`
` Exhibit 1026 Page 12
`
`reconnaissance. Tele-operations will expand underwater and space
`exploration. Tele-medicine will allow specialists to provide consultations to
`remote sites by viewing X-rays, sonograms, or tissue cultures.
`Remote time control will expand with preprogrammed operation of simple
`household applicances such as turning on the dishwasher while the family is
`asleep or setting lights on while on vacation. More elaborate home controls will
`enable programming of week-long cycles of alarm clocks, heating/cooling,
`videotaping, or burgular alarm setting. Seasonal adjustments and maintenance
`to heating/cooling systems will be accomplished more reliably. Automatic
`resetting of clocks for daylight savings time will be a minor benefit. With the
`addition of sensors, sprinkler systems can be programmed to turn on only when
`the rainfall has been insufficient or heating systems can be adjusted to match
`patterns of home usage.
`Remote operations will have an increasingly large effect on manufacturing
`plants where complex procedures for assembly of electronic, optical, or
`mechanical components, spray painting, sewing, printing, or welding can be
`specified and then executed repeatedly or when needed at the local plant or
`remote sites. Benefits to offices, hospitals, or stores can also be great.
`
`Flexible search
`One of the next great advances in computer algorithms will enable greater
`flexibility in locating information. Current search paradigms are quite structured
`and rigid. Many users wish for a perfectly flexible search paradigm that finds all
`and only all of the desired items, but for the moment we can expand search
`paradigms to be a bit more flexible.
`Search algorithms currently favor two paradigms: free text search within a
`document to locate the next occurrence of a specified string (may include wild
`card characters) and index search in bibliographic databases to locate a
`collection of documents that match the search pattern (may include AND, OR,
`NOT, ADJACENT, or other operators). The flexibility of these search paradigms
`increases with each generation of system, in part in response to the increasing
`appetite of users. Still there are many situations in which search algorithms
`could be helpful but for which there is rare or awkward support. There is the
`potential for incremental but important improvements and some exciting
`products.
`
`Rainbow search: First, for searches within a document, there is the chance to
`enable rainbow searching. Most search facilities treat text as if it had only a
`single color, but most word processors already support a range of colorful
`features such as multiple fonts, font sizes, styles (italic, bold, underscore, etc.),
`and text attributes (footnotes, references, titles, headers, footers, etc.). It might
`be useful to allow users to locate the next bold-faced occurrence of George
`Washington or search only through the footnotes. This is fairly easy to
`implement, but some re-design of the user interface is necessary. In addition to
`searches, global changes such as replacement of all bold-faced items with italic
`
`12
`
`

`
` Exhibit 1026 Page 13
`
`might be useful.
`
`Search expansion: For searches across documents, increased ability to specify
`search within a component, such as the title, abstract, or conclusion might be
`helpful in some situations. Search expansions would also facilitate solution of
`some problems, for example, if the user desires to locate documents dealing
`with "New England", the system thesaurus, would inform the user that the search
`could be expanded to include "Connecticut", "Rhode Island", "Massachusetts",
`"Vermont", "New Hampshire", and "Maine". Search expansions might provide
`more specific terms (as in this example), more general terms, synonyms, or
`related terms.
`
`Sound search: Search concepts are increasingly im