`An Overview of LiveDoc
`
`James R. Miller and Thomas Bonuro
`
`One of the changes that the World Wide Web has brought to the
`computing industry is a new way of thinking about documents.
`Traditionally, documents have been seen as simple streams of
`characters, like those in a document editor. Applications that
`manage these documents may do more or less interesting things
`to the Characters, but they rarely attempt to interpret any of the
`meaning of the document. There’s obviously meaning there, but
`it only becomes apparent when read or otherwise manipulated
`by a human. In contrast, the Web has brought with it the con—
`cept of a document that has been authored in such a way that
`important bits of information are explicitly identified within the
`document, This identification exposes some of the meaning of
`the document, albeit at a fairly low level, so that various kinds of
`actions — primarily “show me this related document” — are
`ofifered to users and made easy for them to carry out.
`
`The gap that separates these two notions of documrnris the need
`for the human authoring of the Web document. More to the
`point, it’s the need for a human to identify the meaningful com-
`ponents of the document and the actions that make sense for
`those components. There is a real opportunity to advance the
`computing held here, by bringing these two worlds together: by
`enabling an ordinary document, built with any application, to
`automatically offer users access to some ofthe meaningful bits of
`its content, and by helping users any out appropriate actions on
`these objects.
`
`Bridging the Gap through Structure Detection
`
`This premise led to a collection of projects within Apple’s
`Advanced Technology Group — within the Intelligent Systems
`Program, in particular -— on the idea of structure detection. The
`work was based on the observation that, while automatically
`
`computing a high—level understanding ofan arbitrary document
`is beyond our present ability, many meaningful bits of informa-
`tion are computationally quite easy to recognize: recognizing an
`e-mail address (“fred@apple.com”) or a URI. (“http:/l
`www.apple.com”) takes little more than a contextrfree grammar,
`if not merely a regular expression parser. A firsr step to bridging
`the document gap described above is then to construct a means
`ofpassing text from a user’s document into a parser for matching
`against a collection of recognizers, each ofwhich is looking for
`some meaningful type of information. These identifications
`imply simple interpretations of the bits of information that were
`found: URLs are found by the URL grammar, email addresses
`are found by the e-mail address grammar, and so on. Then,
`actions appropriate to each kind of object can be offered, sup—
`porting users in their work on those objects and on the docu—
`ment as a whole.
`
`Our overall intent here— to examine document content, identify
`likely user actions, and provide simple ways ofsclccting and exc-
`cuting those actions — is not unlike that of the authors of other
`“intelligent” critic and advisory systems [e.g., 4, 6]. However,
`our work on structure detection differs from these systems in a
`number ofways:
`
`' Syntactically—regular information structures, and the tasks that
`follow from them, can be found in almost any user domain.
`Hence, the total number of structures and tasks for which
`structure detection assistance would be helpful is too large for
`any single person or organization to try to satisfy. Therefore,
`we have paid special attention to the importance of allowing
`application developers and even en d—users to define and
`extend the set of detectors and actions. This drove us to design
`
`SIGCHI Bulletin
`
`Volume 30, Number 2
`
`April 1998
`
`53
`
`Arendi S.A.R.L.—Ex. 2002
`Page 1 of 6
`
`Arendi S.A.R.L.-Ex. 2002
`Page 1 of 6
`
`
`
`a plug—in architecture for object recognizers and actions, as dis«
`cussed in [7]
`' We have remained open to large numbers of applications — ide—
`ally, any application available on the Macintosh platform —
`and have provided users with a consistent human interface
`across those applications.
`0 Viewing this work From an agent perspective, we have worked
`to keep the user well in control of the system’s actions. This has
`called for a clear and explicit connection between the informa—
`tion that was found and the actions taken by users on that
`information. We have also kept the grain size of the actions rel’
`atively small, so that they would be easily understandable and,
`if need be, undoable.
`
`I We Wanted to keep our approach practical enough that it could
`ship as a commercial product, rather than simply being a
`research project.
`
`Our most successful instantiation of the idea of structure
`detection (thus far) has been Apple Dam Detectors (ADD):a
`system extension for Mac OS 8 thatis now commercially avail—
`ablel. This first version ofstructure detection has been applied
`to the domain of Internet information management; finding
`structures like e-mail addresses, URLs, host names, and news—
`group names in user documents and automating actions on
`these structures, like creating a new e-mail message addressed
`to a discovered e—mail address or opening a web browser on a
`discovered URL. This capability has been implemented via
`Mac OS 8’s contextual menus: Users select a region of text and
`activate the contextual menu by pressing the keyboard’s Conv
`trol key and the mouse button. This initiates the grammatical
`analysis of the selected text, and presents a hierarchical menu
`consisting of the structures found by the grammars and, for
`each of those structures, the actions that make sense for it.
`Users can examine all the discovered structures and their asso—
`ciated actions, and invoke whichever action they choose (Fig—
`ure 1). More on the origin oprple Data Detectors, its
`implementation, and how it made the transfer From research to
`product can be found in [7]. Some other related approaches to
`the structure detection question can also be Found in [5] and
`[8].
`
`LiveDoc: Beyond Duh: Detectors
`
`Many of the goals we had set for shifting the model of docu—
`ments from passive streams of characters to manipulable collec—
`tions ofmeaningful objects were met quite nicely by Apple Data
`Detectors. Nevertheless, we saw other opportunities for struc-
`ture detection, growing out of both limitations of this first sys-
`tem and opportunities for expanding its breadth and depth.
`
`Consider, first, the user interface to ADD. Users must select a
`range of text in a document, and invoke ADD through the con—
`textual menu system. At that point, the selected text is sent to
`ADD’s parser for analysis, which produces the menu of discov—
`. ered structures and corresponding actions. This design has a
`number ofimplications on what kinds ofinteraction services can
`be ofl’ered to users:
`
`1_._..___._____.________
`as a a free
`lug-in for Mac OS 8; it 15 available via http: Happie—
`script. app e com/datadetectors.
`
`In: rmgosmltxun
`'1’“'1 JUN! Sill"! ‘jiflillwlnflhtlm’
`
`sub)“ t: It; now. 1M! votanun punt
`
`Figure 'I: The resulls 0F invoking Apple Doro Deleclors on a
`”993’
`text selection.
`
`
`
`
`
`- The basic premise of ADD is that it finds and selects manipu—
`lable bits of information for the user. However, in practice, the
`user must still find a structure that they would like to do some—
`thing with, so that they can select, for analysis by ADD, a‘
`region of text containing that structure. it’s true that ADD
`allows users to be inexact in selecting the region of text con—
`taining the structure of interest, since its grammars will find
`the desired structures in a stream of other, irrelevant characters.
`However, the tasks of discovering structures that might be
`operated upon and selecting the parts of the document around
`those structures are still imposed upon the user by ADD.
`- It is not uncommon for usefiil structures to be nested within
`
`other useful structures. The URL “http://wwwapple.com/
`defaulthtml” contains within it the host name
`
`“wwapplecom”; a similar situation holds for the host name
`embedded in an e—mail address. These multiple structures can
`make the con textual menu generated by a typical text selection
`quite long, forcing the user to choose between limiting the set
`ofactive detectors (which keeps the task offinding the desired
`structure in the contextual menu a manageable one) and hav»
`ing to search through a large number of detected structures in
`the contextual menu (which makes ADD’s services applicable
`in the largest number of situations).
`0 Listing the detected structures in the contextual menu deesn’t
`really make them manipulable, through such direct manipula—
`tion techniques as drag and drop.
`
`We encountered still other limitations resulting from our desire
`to increase the flexibility and power of ADD analyses. For
`instance, there really isn’t any semantic interpretation of the
`discovered structures in ADD: Actions are associated with
`structures through a lookup table, not through any rich seman—
`tic representation of, for instance, what a URL is, what it might
`be used for, and what constraints exist on its use. As a result,
`the set of actions that can be ofl‘ered to a user is fixed: it can nei»
`thet include nor omit actions based on the semantics of the
`
`immediate interaction context. Further, since ADD’S process—
`ing is tied to and activated by the contextual menu system, it
`
`54
`
`April l998
`
`Volume 30, Number 2
`
`SIGCHI Bulletin
`
`Arendi S.A.R.L.-Ex. 2002
`Page 2 of 6
`
`Arendi S.A.R.L.-Ex. 2002
`Page 2 of 6
`
`
`
`must complete its analysis in the very short period of time in
`which users are willing to wait For a pop—up menu to appear2 —
`about halfa second. This is typically enough time For ADD to
`run a set of ptecompiled grammars and build a menu from a
`lookup table. However, it's easy to imagine more complex anal—
`yses of documents that could not be completed in this short
`amount of time.
`
`Finally, we saw other opportunities for alternate analysis tech—
`niques, which could augment and extend the analysis model of
`ADD. ADD was built around a single parser, roughly equivalent
`to a context-free grammar (CFG); all structures found by ADD
`must be describable by a CFC. This is fine For simple structures
`such as phone numbers and e-mail addresses, but other more
`complex structures, such as a meeting announcement, are com—
`monly found in a form not describable by a CFG, and so cannot
`be recognized byADD. Some of these structures might be found
`by a more expressive grammar formalism, and others by rec0g~
`nizers that find structures by powerful but ad hoc analysis tech—
`niques instead of grammars. In addition, this textual grammar
`system cannot be easily extended to handle non—textual struc—
`tures, such as images, drawings, circuit diagrams, or other kinds
`of visual information. All of these extensions indicate the need
`for a richer analysis model than was provided in ADD.
`
`As a result, we began a follow-on project to ADD, known as
`LiveDoc, which would carry the ideas of structure detection for-
`ward to another level. In LiveDoc, the structure detection proc-
`ass is run in the background on the visible document’s text,
`whenever that document is presented or updated. The results of
`LiveDoc’s analysis are then presented by visually highlighting the
`discovered structures with a patch of color around the structure.
`Holding down a function key places the document in “LiveDoc
`mode” and presents the highlighted structures; releasing the
`function key returns the document to normal. Pointing at a
`highlight and pressing the mouse button then displays the menu
`of actions that can be applied to the structure, as shown in Fig 2.
`
`Experientially, the design of LiveDoc draws on the Web in obvi—
`ous ways: certain meaningful parts of a document are high-
`lighted, and clicking on them causes certain actions to occur.
`LiveDoc differs from the Web, of course, in that we have substi—
`tuted the automatic analysis of the document For the hand-
`authored links of web documents, so that any document in a
`LiveDoc—enabled application (more on this later) gains these
`characteristics. We are also able to assign more than one action
`to an object, something that lies outside the standard Web para-
`digm.
`
`LiveDoc’s use of background processing and automatic high—
`lighting of discovered structures offers other advantages. Struc—
`tures relevant to the user are automatically presented to the user
`while a document is in LiveDoc mode; interesting structures
`need not be searched for and highlighted manually. The prob-
`
`2. In fact, the contextual menu system enforces a timeout after this
`amount of time: contextual menu plug—ins, like ADD, that have
`not completed their analysis when this time expires are halted by
`the contextual menu manager. When this happens, a “More
`actions...” option is added to the contextual menu; selecting this
`runs all the plug—ins to completion, and brings up a dialog box
`containing the results of their analyses.
`
`
`, more «73 final Sna-
`u in mm winwg once
`l
`cgvvtuiotiomfl m 5-:
`. Raw! as. mm uallm.
`Inc mm
`my». what-tans on: raw“ ‘9“, u: mm W no
`
`mean. some new I“, {ha umbs-cl Lei-ooh In Dru-sen
`
`05th with rum-W Widow to their elm: has 1m
`Wm m «4 75a.
`
`
`Tm van to all shown: wmrwwlmlm that
`prov-m mart mum, martin and um some. this ”not
`WM. Wuhan-mum! mum to menimmnor
`
`
`919mm Haiku.
`
`
`Gar-aim Iv.
`
`
`lac
`France?”
`We
`Smith to with;
`
`
`
`
`- cum 9‘96
`
`. “W.“W
`.
`
`
`
`Figure 2: A sample interaction with LiveDoc. Note the
`highlighting oi the discovered structures, the menu oi ociions
`available or the selected structure, and lhe nested highlighting oi
`nested structures.
`
`' .
`.. aria
`
`
`lem of overly-long contextual menus is avoided, since a menu
`shows only the actions relevant to the structure it is associated
`With. Similarly, nested structures can be handled by nesting the
`mouse—sensitive regions around the structure: clicking on the
`host name part of a URL can present a menu of actions relevant
`to host names, while clicking outside the host name region
`presents actions relevant to URLs. This is shown in Figure 2: the
`host name of the e—mail address and URL (“sciorg”) is shown
`with a darker highlight than those of the e—mail address and
`URL themselves. Finally, note that the visual representation of
`these structures means that, given appropriate software support,
`they can be treated as directly accessible components of the interr
`Face: they can take part in drag and drop interactions, and in
`other forms of direct manipulation.
`
`What is described above is, of course, only a general design for
`LiveDoc. To understand how this design can be implemented,
`it’s necessary to look more closely at the systems architecture,
`and at its instantiation in several different working systems.
`
`The LiveDoc architecture: A General Description
`
`Architecturally, LiveDoc is built around the LiveDoc Manager
`(Figure 3). This component acts as an intermediary between the
`application making use of LiveDoc and the various internals of
`LiveDoc itself. In particular, the Analyzer System is made up of
`a set of detectors that analyze the content of the document
`passed to LiveDoc, a set of actions (typically, but not necessarily,
`implemented as AppleScripts) that carry out the various opera—
`tions on the discovered structures, a table that specifies the map,
`ping between detectors and actions, and an Analyzer Server that
`coordinates all these Functions.
`
`To make use of LiveDoc, applications must implement a small
`number ofcalls to the LiveDoc Manager, and a small number of
`
`SlGCHI Bulletin
`
`Volume 30, Number ’2
`
`April l998 '
`
`55
`
`Arendi S.A.R.L.-Ex. 2002
`Page 3 of 6
`
`Arendi S.A.R.L.-Ex. 2002
`Page 3 of 6
`
`
`
`Application
`Supports LiveDoc through
`ralbacks
`
`LiveDoc Mange:
`Responsible for GUL
`commumrauon Mth
`Analyzer System
`
`external functions
`
`lam
`Analyzer
`Combines gramm—
`hased detectors, hard-
`roded detectors, and
`proxy detectors talllng
`
`callback handlers to respond to calls from the LiveDoc Manager.
`The most important ofthese handlers inform the LiveDoc Man-
`ager ofchanges to the content ofthe document window, perhaps
`by the user’s adding or deleting content, or by the scrolling or
`resizing ofrhe window. The receipt ofthese calls by the LiveDrrc
`Manager signals the Analyzer Server to analyze the text provided
`by the calling application; this will typically be the text currently
`visible in the applications’ front—most window. Once the analysis
`is completed and the structures in the text have been identified,
`the LiveDoc Manager constructs the various highlights for the
`discovered structures and their corresponding menus of actions.
`LiveDoc knows where these structures appear in the text passed
`to it —— an e—mail address might appear in characters 150 through
`162 of the window’s contents — but it has no idea where in the
`
`window thOse characters physically appear, and, thus, where the
`highlights should appear. this is information held by the appli-
`cation, not by LiveDoc. Hence, LiveDoc must ask the applica—
`tion for the information about the structures it has found via a
`
`callback. Once this information is available, the highlights and
`their associated mouse—sensitive regions can be constructed.
`
`The LiveDoc Manager also controls the events that occur when
`the user presscs'the function key to enter LiveDoc mode, and
`when the mouse button is pressed while over a Lichoc item.
`The LiveDoc Manager updates the display to present the high-
`light information ovet the discovered structures when the func-
`tion key is pressed, and to remove the highlights when the
`function key is released. The LiveDoc Manager also receives the
`notification that the mouse button has been pressed over a high-
`lighted item; it then gets the list of actions appropriate to the
`selected item and presents a menu of them to the riser. If one of
`these items is selected, the action corresponding to the selection
`is run, producing the desired action.
`
`Implementation 1: LiveSimpleTex'r
`
`Our initial explorations of LiveDoc were implemented in Lisp.
`primarily to gain a rapid understanding of some of the imple—
`mentation issues we would ultimately face and to explore various
`human interface ideas. However, we soon needed to turn our
`attention to a system-level API for LiveDoc that would be both
`efficient and nOt terribly burdensome to developers who would
`have to modify their applications to take advantage of LiveDoc.
`
`To test our first implementation of the LiveDoc API, we decided
`to modify a simple text editor application, SimpleText, to be a
`LiveDoc client. Although the API was designed as a Macintosh
`toolbox manager, so that it could be incorporated into existing
`applications without requiring developers to compile and link our
`code with theirs, this initial implementation did require linking
`portions ofLiveDoc code with that ofSimpleText. (This would be
`changed in later work where the LiveDoc manager was built as a
`shared library, allowing the LiveDoc Manager to be called by mul—
`tiple LiveDoc clients.) We also needed a set ofanalyzers that could
`provide the document analysis services to the Analyzer Server. \We
`decided to use the Apple Data Detectors context—free grammar
`engine, and to additionally implement a fast string search algo-
`rithm, as described by M10 and Corasick [1]; this analyzer rapidly
`finds all instances ofstrings in a document by comparing the text
`ofthc document to a set ofdictionary entries. In doing so, we were
`able to test the multiple—analyzer part of the Analyzer Server and
`confirm its utility. As part ofthis test, we gave each analer its own
`interface affordance by varying the colors of the highlights: green
`for items found using a context free grammar (the Apple Data
`Detectors analysis engine) and pink for those items found using
`the string search.
`
`The background processing of LiveDoc raises the issue of the
`proper way in which to update the display when changes are
`made to the document. Whenever the text in the window is
`
`changed, by either the system or the user, LiveDoc must re-ana—
`lyze the text, since recognizable structures may have appeared or
`disappeared. But when should this analysis be done? It makes lit-
`tle sense to analyze a document while the user is working, since
`each change will require a re—evaluation of the text. Our current
`implementation works with a timeout, which starts a revanalysis
`when the keyboard has been inactive for a short period of time
`(about one second) after a change to the display. In this way,
`LiveDoc does not analyze the document while the user is typing,
`but resumes when there is a pause in the user’s actions. We have
`considered various algorithms that might minimize the cost of
`this analysis — perhaps only analyzing the part of the window
`that had been scrolled, for instance — but our current implemen—
`tation reanalyzcs the entire content region of a window when a
`change is detected.
`
`Overall, we believe there are some very compelling aspects ofthe
`LiveDoc interface as compared to Apple Data Detectors. As
`shown in Figure 2, LiveDoc displays its discovered structures in
`place, in the context in which they occur. It associates a menu of
`options with the object found where, in Apple Data Detectors,
`a single menu appears for all of the items found in the selection.
`Finally, LiveDoc works quietly in the background and displays
`the results of its analysis on demand, rather than performing the
`analysis on demand. Having said that, there are some user inter—
`
`56
`
`April
`
`lQQS
`
`Volume 30, Number 2
`
`SIGCHI Bulletin
`
`Arendi S.A.R.L.-Ex. 2002
`Page 4 of 6
`
`Arendi S.A.R.L.-Ex. 2002
`Page 4 of 6
`
`
`
`face issues that we have not explicitly evaluated as part of this
`work, and that would benefit from further study. The use of
`highlighting is one of these: adding the notion of a sometimes-
`visible layer to the front of the display is a considerable change to
`the graphical interface, and, while others have pursued similar
`uses of translucency (e.g., [2]), its overall utility and understand-
`ability is worthy of study.
`
`Similarly, the use of increasing degrees of saturation of the high—
`lights should be examined: are these differences clear enough to
`be discernible, and do they require a precision in mouse control
`that users both possess and are willing to apply? Note also that
`this issue of manual and visual dexterity is complicated by the
`fact that using a function key as the means of entering and leav-
`ing LiveDoc mode means that we have made the interface a two—
`handed one: one for the function key and one for the mouse,
`This3 raises disability and accessibility issues, and more general
`ease—of-use issues as well. Finally, we should note the current
`invisibility of the background document processing, and the
`need for the status of this processing to be visible to the user:
`How does a user know when the analysis is completed, and all
`the structures that can be discovered have been discovered? Some
`equivalent to a progress indicator bar, which reports how far
`along the analysis is, might be a useful addition. This may not‘be
`a problem when and if processors become fast enough to do
`these analyses very quickly, but we are hesitant to rely on sheer
`processor speed as the solution to what is really an interface
`design problem (especially since faster processors will most likely
`encourage developers to design more sophisticated, and proba-
`bly more time-expensive, analyses).
`
`Dealing will: Slatic—ness: APIs and Plug-Ins
`
`A second aspect of this project addressed some deeper questions
`about the LiveDoc architecture: the need for an API to the Live-
`
`Doc Manager (and whether it can be eliminated), the inflexibil—
`ity of the analysis architecture, and the opportunities raised by a
`greater degree of semantic representation about the discovered
`structures.
`
`LiveSirnpleText worked well as a prototype, but we were still
`concerned about requiring developers to change their applica—
`tions to gain access to LiveDoc’s capabilities. We know from
`experience that developers are justifiably reluctant to change
`their applications just to implement a new feature provided by
`the toolbox, so we experimented with some alternatives that we
`hoped would ease this restriction.
`
`Our first approach was to modify erEdit, a set of Macintosh
`toolbox routines that provide an application with minimal text
`editing and display capabilities. Although TextEdit has a limit of
`32,000 characters, too small for the needs ofmost modern word
`processing applications, most application dialog boxes that have
`editable fields use TextEdit, and some applications also use Text—
`Edit for their editors or text display fields. Ifwe could “trap” the
`calls from an application to Teerdit and pass them through
`Lichoc, we could provide LiveDoc capabilities to any applica»
`tion that uses TextEdit, without requiring any modification to
`the application. Hence, we built a Macintosh system component
`
`Like the contextual menu system, it should be noted...
`
`that trapped and patched all the TextEdit calls, and then tested
`this with a variety of applications. In doing this, we discovered
`that most applications that use TextEdit take some program—
`ming shortcuts that kept our patches from working properly We
`did however, find that the e-mail client Eudora used TextEdir
`appropriately, and we were able to provide LiveDoc functional-
`ity to an unmodified version of Eudora through this approach.
`Unfortunately, before we could deploy what we called “Eudora—
`Live", Eudora underwent an architectural change in moving
`from Eudora 2.0 to Eudora 3.0, and no longer used TextEdit.
`
`A second promising approach was to use the framework of
`OperiDoc‘i as a vehicle for LiveDoc. At the time we undertook
`this work, OpenDoc parts and part editors were just beginning
`to appear, and it appeared that we could leverage some of Open—
`Doc’s extensibility to implement Lichoc as a plug-in to the
`OpenDoc architecture, so that Opanoc text parts could
`acquire LiveDoc behavior without any source code modifia—
`tion. This was promising, but we ultimately discovered that it
`still was not possible to get information about the way in which
`text was rendered without the LiveDoc Manager consulting the
`application.
`
`These attempts to build an API—less LiveDoc ultimately failed (as
`did other attempts not discussed here, which worked with other
`parts of the Macintosh architecture. Nevertheless, we are still
`optimistic about the possibility of such approaches to system
`extension, for LiveDoc as well as other kinds of extensions. In
`retrospect, it seems that what we were trying to do was to graft
`an object-oriented software model onto a platform that was not
`object—oriented. OpenDoc had the right sense of object-orienta—
`tion, but its design happened to not: expose certain aspects of the
`system that LiveDoc required. As operating systems continue to
`evolve in an object-oriented direction, we may find that the
`kinds of extensions we sought become the norm, rather than the
`exception.
`
`Futures: Extensibility and Semantics
`
`We tried to design Live-Doc’s architecture to be as open as possi—
`ble. In doing so, when faced with a tradeoff between perform-
`ance and extensibility, we generally leaned towards extensibility.
`As implemented in the systems described here, there are two
`aspecrs to exrensibility in Lichoc: the analyzers and the kinds
`ofstructures they can discover, and the actions that can be taken
`on those strucrures. \While the analyzers we built operated on
`lexical data, much could be done by applying the LiveDoc
`model to graphical information or multi-media content. We can
`easily envision analyzers that recogniw features in drawings and
`pictures, or analyzers that “listen” for relevant structures in
`streaming audio or video. While We tried to be agnostic about
`these data types in the design of the LiveDoc architecture, it is
`inevitable that some pieces of the API would have to be
`rethought if such detectors became available.
`
`The behaviors associated with actions should also be flexible and
`extensible, and work more closely with the document structure
`than they do at present. Our initial implementation of LiveDoc
`as LiveSimpleText assumed that actions would be handled by
`WWW
`' Cf. http://www.crlabs.com.
`
`SIGCHI Bulletin
`
`Volume 30, Number 2
`
`April “998
`
`57
`
`Arendi S.A.R.L.-Ex. 2002
`Page 5 of 6
`
`Arendi S.A.R.L.-Ex. 2002
`Page 5 of 6
`
`
`
`external applications, such as a Web browser Presenting the page
`pointed to by a URL. However, other styles of interaction exist:
`Imagine a detector that finds the formula ofan organic molecule
`in a document, and an action that presents a three-dimensional
`rendering of that molecule within the context of the document
`itself, rather than in a separate application. Our API allows for
`such behavior, but more sophisticated models of documents are
`needed to pursue these possibilities in detail.
`
`Finally, LiveDoc itself knows virtually nothing of the real mean—
`ing of the items that it identifies. At best, LiveDoc might know
`that an object is a telephone number or a company name, but
`even those terms are simply labels assigned by the authors of the
`grammars, and have no extrinsic meaning. Moreover, there is no
`way to describe a composition of discovered objects in terms of
`meaningful relationships among them: all that the LiveDoc
`analysis produces is a list of discovered objects with a simple type
`identification. It’s possible to discover relatively complex objects
`with the CFG model ~ a grammar might describe a “meeting” as
`a combination ofa date, two times (presumably the starting and
`ending times ofthe meeting) and the name ofa conference room
`—— but there are still no semantics associated with this “meeting”
`object or any of its components, Hence, as noted earlier, the
`actions associated with a structure are static, rather than dynam-
`ically reflecting the nature of the user’s interaction context. Fur-
`ther, we have done nothing of significance with the idea of
`making these objects directly manipulable; they are still only tar-
`gets for a “mouse—down” interaction. These questions of direct
`manipulation and the marriage of syntax with semantics are
`important ones; we consider them in more detail in our discus-
`sion of “drop zones” [3].
`
`Summary
`
`LiveDoc was a useful experiment, in several ways. It provided a
`testbed for exploring issues in system architecture and human
`interface, and how the two can — indeed, must — work together
`to produce a useful interaction technique. In addition, LiveDoc
`played a significant role in our productizing the general notion
`of structure detection via Apple Data Detectors. Having Live—
`Doc as a demo tool showed that there was a long-term technol—
`ogy stream at the heart 0Fthe simple notion of structure
`detection. The highly visual nature of the LiveDoe interface also
`was a very ePfective marketing tool: it caught the attention of the
`product—side people who saw it, and helped open the doors to
`consideration and adoption of the basic technology.
`
`In any case, the value of the approach demonstrated by LiveDoc
`seems clear. More and more applications are providing some abil-
`ity to recognize domainirelevant structures and allow users to are
`cute relevant actions on them via a mouse click. Many email
`clients automatically recognize e—mail addresses and URLs, as do
`some newsgroup readers and even some general—purpose docu‘
`ment editors. This is fine with us: their domain—specific use of
`these techniques confirms the relevance of the general idea, and
`argues For a broader architectural approach to that general idea.
`We hope and expect that applications and the system software
`beneath them will continue to move in the direction outlined by
`LiveDoc: away from a lifeless stream of characters, and toward a
`collection of interrelated, meaningfiil, and manipulable objects.
`Having a clear system architecture and a well—conceived human
`
`interface, like that described here, will be essential to the success of
`this vision of documents, and this style of interacting with them.
`
`References
`
`1. Aho, A. V., 8: Corasick, M.]. (1975). Efficient string
`matching: An aid to bibliographic search. Communication: of
`the ACM 18(6), pp. 333—340.
`2. Bier, E. A., Stone, M. C., Fishkin, K., Buxton, W, St Bau-
`del, T. (1994). A taxonomy of see—through tools. Proceeding:
`quH1 ’94, pp. 358-364. New York: ACM Press.
`V
`3. Bonura, T., 85 Miller, J. R. (1998). Drop Zones: An exten—
`sion to LiveDoc. SIGCHI Bulletin, this volume.
`4. Cypher. A. EAGER: Programming repetitive tasks by exam-
`ple. (1991). Proc. CHI ’91, pp. 55—39. New York: ACM Press
`5. Dcy, A. K., Abowd, G. D., 85 Wood, A. (1998). CyberDesk:
`A Framework for Providing Self—Integrating Context—Aware
`Services. Proceeding: ofIntelligent User Interfizre: ’98. New
`York: ACM Press.
`6. Fischer, G., Lemke, A.