Cross-modal Interaction using XWeb
`
`Dan R. Olsen Jr., Sean Jefferies, Travis Nielsen, William Moyes, Paul Fredrickson
`Computer Science Department, Brigham Young University, Provo, UT
`{olsen, jefferie, nielsent, wmoyes, pfred}@cs.byu.edu
`
`ABSTRACT
`The XWeb project addresses the problem of interacting
`with services by means of a variety of interactive
`platforms. Interactive clients are provided on a variety of
`hardware/software platforms that can access and XWeb
`service. Creators of services need not be concerned with
`interactive techniques or devices. The cross platform
`problems of a network model of interaction, adaptation to
`screen size and supporting both speech and visual
`interfaces in the same model are addressed.
`Keywords
`interaction, network
`Cross-modal
`layout, speech interfaces.
`
`interaction, screen
`
`INTRODUCTION
`Exponential growth in computing capacity and size of the
`Internet, as well as exponential decline in costs for a given
`fixed capacity impose interactive challenges that traditional
`user interface architectures cannot meet. The exponential
`growth in capacity produces ever larger repositories of
`information with ever more diverse content. The
`exponential decline in cost pushes computation into ever-
`smaller packages into increasingly many parts of human
`activity. The prospect of cheap connectivity to virtually
`everything holds enormous potential
`for
`interactive
`systems.
`The service problem
`Information and control services will be increasingly
`diverse ranging from petabyte-sized databases down to
`microcontrollers in small appliances. This diversity poses
`two sets of problems. To justify the very large database, it
`must service a large and diverse user population. Training
`and supporting software
`installations for a diverse
`population is an almost insurmountable task. This problem
`is exacerbated by the variety of interactive platforms, such
`as lap tops, cell phones, personal digital assistants,
`interactive walls and rooms with new platforms yet to be
`invented.
`
`The converse of the large server problem is the microserver
`problem. With the advent of significant computation and
`memory in packages costing less than $50 to $100 the cost
`
`Permission to make digital or hard copies of all or part of this work for
`personal or classroom use is granted without fee provided that copies are not
`made or distributed for profit or commercial advantage and that copies bear
`this notice and the full citation on the first page. To copy otherwise, to
`republish, to post on servers or to redistribute to lists, requires prior specific
`permission and/or a fee.
`UIST ’00. San Diego, CA USA
` 2000 ACM 1-58113-212-3/00/11... $5.00
`
`gap between control and information services and their
`user interfaces becomes quite large. A primary reason for
`the difficulty in programming VCRs is that the hardware
`cost of a truly effective user interface would almost double
`the cost and size of the VCR.
`
`In addition to VCRs there are a variety of other uses for
`such microservers. Vending machines must be filled by
`people. If these machines can provide people with remote
`access to their current state, many visits can be avoided and
`costs reduced. Instruments in remote locations can be
`serviced. Large appliances and other devices can be
`instrumented for diagnostic
`information
`that can be
`remotely accessed. If, however, the user interface must be
`directly coupled to each such device, both the hardware
`and training costs will jump significantly.
`
`every
`if
`succeed
`cannot
`computation
`Pervasive
`computational device must be accompanied by its own
`interactive hardware and software. Diverse populations
`cannot be served by an architecture that requires a uniquely
`programmed solution for every combination of service and
`interactive platform. What is needed is a universal
`interactive service protocol
`to which any compliant
`interactive client can connect and access any service.
`Decoupling the user interface from the service is the only
`possible approach.
`The user problem
`Users are faced with a similar problem to that encountered
`by the service providers. There is a huge diversity in the set
`of possible control and
`information services
`that a
`particular user may find useful. Installing a unique piece of
`software for each desired service is not a good solution. It
`is already true that for most personal computer users,
`software occupies more space than any other class of
`storage. If every new service requires the installation of a
`new piece of software, the accessibility of that service is
`sharply diminished. Users have already discovered that
`new software installation is the most likely reason for
`failure of their computers.
`
`Installing unique user interfaces for each service also poses
`a serious learning barrier. The average user cannot master
`more than a few different user interfaces. If every new
`information service, appliance, entertainment device, or
`piece of automation poses a new interface to be learned,
`the result is an unusable morass. The usability problem no
`longer lies in the design of a particular interface, but rather
`in the collective mass of such interfaces. A huge barrier to
`
`CHI Letters vol 2, 2
`
`191
`
`Roku Exhibit 1020
`Roku, Inc. v. Canon Kabushiki Kaisha
`IPR2020-00357
`Page 00001
`
`

`

`pervasive computing is that users are drowning in the
`diversity.
`
`In addition to the diversity of services that a user may want
`to access there is also the diversity of situations in which
`they may want to interact. Consider our example of
`vending machine servicing. The dispatcher/manager in the
`home office will have a different set of interactive devices
`available than the service person in a truck traveling to the
`next stop. These will be different still from the set of
`appropriate user interface devices for actually servicing the
`machine. Each situation
`imposes
`its own physical
`requirements on the set of devices and set of interactive
`techniques that are possible and effective. It is not
`reasonable to require each vending machine to support all
`such possibilities. Neither is it reasonable to force the users
`in each situation to work with inappropriate user interfaces.
`A more general solution is required.
`
`An additional diversity of interactive behavior arises when
`considering people with various disabilities. It is not cost
`effective for each service to build a unique user interface
`for each unique set of disabilities. Nor is it likely that such
`users will be able to learn and/or adapt to a large diversity
`of interfaces. Again the unique user interface for each
`service approach fails.
`
`What an individual user needs is a small number of
`hardware platforms, each with its own user interface that
`has been tuned to the capabilities of that platform and the
`class of users/situations for which it is intended. Each such
`hardware/software platform must be capable of interacting
`effectively with any service. An architecture where
`interactive clients are independent from control and
`information services, can scale to the level of diversity
`brought about by Moore’s Law and the Internet.
`Learning from the WWW
`The World Wide Web meets most of the issues described
`above. HTTP/HTML provide a uniform protocol that
`separates services from their user interfaces. Users need
`only one piece of browser software for each interactive
`platform that they use. HTML browsers have been
`developed for personal digital assistants, cell phones and
`other devices in addition to the original desktop versions.
`This one piece of software accesses a huge variety of
`services.
`
`Corporate information providers are rapidly converting to
`web-based user interfaces because they are freed from the
`installation and
`training problems
`imposed by
`the
`application-based UI architecture. Devices ranging from
`digital cameras to network caching appliances provide an
`HTTP server as their only user interface. The actual user
`interface to such devices is found in any standard web
`browser. This approach is an obvious success story.
`
`interactively
`are
`and HTTP
`However, HTML
`impoverished. The level of user interface that they provide
`is equivalent to the old IBM 3270 terminals that were in
`use two decades ago. The architecture of the WWW is
`right but the interactivity is insufficient. New initiatives
`such as WebDAV and WAP address document versioning
`and cell phone interaction as incremental modifications of
`HTML/HTTP. Neither takes on the full range of interactive
`modalities.
`XWeb
`This paper describes the interactive solutions to these
`problems that have been developed as part of the XWeb
`project. XWeb is based on the architecture of the WWW
`with
`additional mechanisms
`for
`interaction
`and
`collaboration. XWeb servers provide responses to the XTP
`network interaction protocol. Server implementations are
`completely independent of the interactive platforms that
`users might use. So far we have demonstrated servers
`which provide interactive access to directory trees of XML
`documents, relational databases and home automation
`devices.
`
`Users work in the XWeb world via interactive clients that
`are
`tuned
`to
`the
`interactive capacities of particular
`interactive platforms. So far we have developed clients for
`the desk top, for speech only situations, and for pen-based
`wall displays. We are currently working on clients using
`only minimal button sets as well as multidisplay interactive
`rooms. Our strategy is to choose client situations that pose
`the greatest possible diversity of interaction.
`
`The key to the scalability of the XWeb user interface
`architecture is that services and clients can independently
`choose a variety of implementation strategies that are tuned
`to their particular needs. The vending machine status server
`can be extremely small in terms of memory and software
`complexity. Huge information repositories can be very
`complex with replication architectures and specialized
`search
`functions. All
`such
`implementations
`are
`independent of each other and of the particular mode of
`interaction that any user might chose, provided that they
`conform to the XWeb Transport Protocol (XTP). Similar
`advantages accrue to users in that they can pick a particular
`client that is suited to their needs, learn only that client, and
`yet interact with all possible XWeb services.
`
`the XWeb
`Successful development of
`architecture depends on solutions
`to
`the
`problems.
`
`interactive
`following
`
`• Defining an interactive protocol for communication
`between service and client
`• Defining user interfaces in a form that is independent
`of a particular mode of interaction
`• Adapting interaction to available input devices ranging
`from minimal button sets to interactive rooms
`
`CHI Letters vol 2, 2
`
`192
`
`IPR2020-00357 Page 00002
`
`

`

`• Adapting to variation in screen size and aspect ratio
`• Defining interfaces that can be based on speech and
`audio as well as visual display.
`
`XTP INTERACTION PROTOCOL
`The World Wide Web defines the HTTP protocol as the
`basis for communication between clients and servers. At
`the heart of this protocol is the GET message that will
`retrieve data from any HTTP server. Our XTP (XWeb
`Transport Protocol) uses that same GET method. However,
`XWeb goes beyond the publish-mostly architecture of the
`WWW. XWeb is intended to provide full interactivity.
`
`In our earlier work on user interface software architectures
`we have found that the vast majority of all interactive
`behavior is to find, browse, and modify information[9,10].
`A server is viewed as a tree of data to be searched,
`retrieved, and modified. We chose trees because of their
`ability to encode a very general set of data structures. We
`also chose trees because of the simplicity of generating
`names for objects in the tree. In this regard we have
`retained the URL ideas from the WWW. We have avoided
`graphs or directed acyclic graphs because of naming
`problems. However, our interfaces do support links from
`object to object, which essentially provides users with any
`graph representation of information.
`
`We represent all XWeb data as XML. XML is a quite
`general mechanism for representing virtually any tree-
`structured data. Note
`that XML
`is
`the
`transport
`representation for data, not necessarily
`the
`internal
`representation. We have implemented servers based on file
`directory structures, XML files, relational databases and
`Novell directory services. In the WWW, HTML is used as
`a facade for a variety of information storage formats. We
`have done the same, except that we have discarded the
`notion that all information is a formatted document.
`Retrieving information
`To XTP the server’s data looks like a tree of objects each of
`which has a tag type, named attributes (with string values)
`and zero or more child objects. Child objects can be
`referenced by ID or by index. A URL for XWeb has the
`form
`
`xweb://domainname:port/pathname
`
`much like an HTTP URL. As in HTTP the port is optional.
`Path names consist of the indices or identifiers of the child
`objects starting from the root of the site. Negative indices
`count backwards from the last child. So the last child of
`some object is at index -1. Attributes are referenced by a
`special @attname syntax. Thus using, path names it is
`possible to identify any object on the site.
`
`When referencing an object is it important to differentiate
`between just that object or the entire tree rooted at that
`object. This is a particular problem when an entire
`
`directory of objects is referenced. If the whole object is
`desired, an entire site may be downloaded, which is rarely
`appropriate. On the other hand, it is frequently useful to
`retrieve entire subtrees at once so as to interact with the
`tree as a whole. We differentiate between references to an
`entire tree and a simple skeleton description of that tree. If
`the URL ends in "/" then only the skeleton is requested. For
`implementation reasons, many types of XWeb services
`refuse
`to
`return anything more
`than
`the skeleton
`(summaries of child objects) rather than the entire subtree.
`Subobjects are then retrieved individually as needed.
`Example URLs might be
`
`•
`
`•
`
`•
`
`xweb://my.site/games/chess/3/@winner
`the winner attribute of the fourth (starts at 0)
`chess game
`xweb://automate.home /lights/livingroom/
`a skeleton description of the set of lights in
`the living room
`xweb://automate.home/lights/familyroom/-1
`all the current information about the last set of
`lights in the family room
`
`Modifying information
`All interaction with an XWeb site is defined in terms of
`changes to that site’s data tree. A CHANGE message
`consists of a URL for the site and subtree to which the
`change is to be applied. A CHANGE consists of a
`sequence of editing operations. Each operation contains
`one or more references to objects or attributes to be
`modified. Such references are relative to the root of the
`CHANGE subtree. The editing operations are:
`
`•
`set an attribute’s value
`•
`delete an attribute
`•
`change some child object to a new value
`•
`insert a new child object
`• move a subtree to another location
`•
`copy a subtree to another location
`
`This set of editing operations defines all of the ways in
`which a client may interact with an XWeb service. By
`composing these operations, any manipulation of a tree can
`be expressed. Note that these manipulations are far more
`extensive than those supported by HTML or WML.
`
`interaction on data
`the client/server
`focusing
`By
`manipulation rather than event propagation, we achieve a
`level of independence between client and service that is not
`possible otherwise. The X windows system provided for
`distributed user interfaces by propagating input events and
`drawing commands across the network. This, however,
`bound the interface to the originally concieved style and
`set of input devices. Adapting such interfaces to a different
`modality such as speech and audio
`[8]
`is quite
`cumbersome. Remote interaction in terms of events is also
`
`CHI Letters vol 2, 2
`
`193
`
`IPR2020-00357 Page 00003
`
`

`

`quite sensitive to network latency. It is unacceptable if each
`input event must make a complete round trip to the
`application before the user gets any feedback. Using
`replicated data, the user can proceed with most interactions
`without confirmation from the service. This provides for
`rapid local feedback and interaction across the network.
`
`PLATFORM INDEPENDENT INTERFACES
`A key problem is for the creator of an information service
`to define interfaces that will be effective on a variety of
`interactive platforms. Our approach to this is to define
`XViews
`that are general XML descriptions of an
`interaction, which do not specify the interactive techniques
`to be used. The primary purposes of an XView are to
`
`•
`
`•
`
`select the data elements that are to be included in the
`user interface,
`• map those data elements to specific interactors and
`ranges of possible values,
`interactors
`provide
`resources
`for
`implementing their user interfaces.
`
`to use
`
`in
`
`Atomic interactors
`Most tool kits define their basic set of atomic widgets
`around specific, generally useful interactive techniques.
`The criteria for choices are to provide flexible composition
`of widgets to meet most needs. Our design is focused on
`frequently used semantic concepts. The implementation is
`not radically different, but it makes significant differences
`in terms of separating the user interface from the service
`and preserving platform independence. Our currently
`implemented set of atomic interactors are
`
`• Numbers with multilevel units and unit conversions
`• Dates
`•
`Times
`•
`Enumeration of finite choices
`•
`Text (single or multiline)
`•
`Links to other data and/or views
`
`A normal user interface tool kit would provide check
`boxes, radio buttons, combo boxes, labels, buttons and
`scroll bars. However, each of these implies an interactive
`technique, which limits the choices for interactive client
`implementations. The semantic goal of choosing from a
`finite set is the same whether radio buttons, combo boxes,
`menus, function keys or speech are used. We define the
`enumeration
`interactor
`and
`leave
`the
`specific
`implementation to the client. The choice of interactive
`technique depends very much on the interactive devices
`available and the available presentation resources. We do
`not provide buttons because they do not model any change
`to a piece of information. Many clients use buttons but as a
`means rather than a semantic goal.
`
`We specifically identified dates and times as special
`interactors because these semantic values occur with a high
`frequency among applications. By specifying how a date
`or time is encoded in the data, we leave the client to
`develop an effective set of interactive techniques to
`manipulate that information. By stepping up to a higher
`semantic level, we provide specific client implementations
`with more interactive flexibility.
`
`Our number specification is quite sophisticated in that it
`provides an abstraction for hierarchical units as well as
`conversions among systems of units. For example, lengths
`can be specified in feet and inches as well as meters and
`centimeters. Only linear unit conversions are supported.
`Units of almost any type can be supported. By supporting
`units we have semantically packaged a concept that would
`require several widgets in most implementations. By
`retaining the semantic whole, we increase the flexibility of
`interactive choices for various client implementations.
`
`We do not provide labels in our interactor set. Each
`interactor carries with it its own descriptive information.
`Our approach is that each interactor can have a variety of
`information resources associated with
`it. The client
`
`When an XWeb client initiates an interaction, it uses a two
`part URL.
`DataURL::ViewURL
`The DataURL is a reference to some subtree of some
`XWeb site. The ViewURL is a reference to an XView
`specification that defines the interaction with the data. Any
`missing portions of the ViewURL are supplied from the
`fully
`specified DataURL. This
`simple
`referencing
`mechanism allows for multiple views of data items as well
`as the application of a view specification to numerous data
`items. We plan to further abbreviate this specification to
`allow servers to inform clients about appropriate default
`ViewURLs for a particular data item. This would allow
`clients to specify only the DataURL.
`Interactors
`The heart of the XView specification is the interactors or
`widgets. Note that our use of interactor should not be
`confused with
`the
`terminology
`introduced by Brad
`Myers[7]. The purpose of an interactor is to specify the
`possible types of values that a data item can have. It is also
`the purpose of an interactor to transform an internally
`encoded data value into an external representation that is
`appropriate for users. Interactor specifications do not
`dictate input events, interactive techniques, or layouts. An
`interactor description encodes what the desired information
`is, leaving specific mechanisms of how the users perceive
`and specify new values up
`to
`the various client
`implementations. This is key to our goal of platform
`independence.
`
`Interactors fall into two categories, atomic and aggregate.
`The set of interactors that we have provided is similar to
`the widget set that one might expect in a user interface tool
`kit except that we have specified them at a higher semantic
`level.
`
`CHI Letters vol 2, 2
`
`194
`
`IPR2020-00357 Page 00004
`
`

`

`to
`them
`then choose which of
`implementation can
`download and how they should be presented. Minimally
`each interactor must have a name. Beyond simple names,
`icons (of various sizes), abbreviations, synonyms, and
`recorded sounds can be added. Client implementations
`choose from among these resources when presenting the
`interactor to the user. Many use label widgets for the
`names. Speech clients, for example do not. The choice is in
`the implementation. Explanation and help texts can also be
`added to any interactor for presentation to the user. We do
`not use any resource inheritance mechanism such as in X
`Windows or Cascading Style Sheets [13]. Specifying an
`interactor description and then applying it to many data
`objects fills
`the need without
`the complexities of
`inheritance.
`
`By only requiring a name with all other resources being
`optional, a minimal client capability is established. We see
`future interactive clients with very limited capacities. If,
`however, such clients can manage a network connection
`and express the basic interactor values they can provide
`full interactive functionality. We see this as important to
`the development of small wearable client devices that are
`unobtrusive but fully capable. An example might be an
`infrared client with a minimal button set that is built into a
`wristwatch. In our worldview of interaction, the ability to
`scale down is as important as scaling up.
`
`Aggregate interactors
`two aggregation
`implemented
`At present we have
`interactors, which can assemble smaller pieces into a larger
`whole. These are groups and lists. A group is simply a
`finite collection of interactors, which together have some
`logical meaning. A group has descriptive resources and a
`set of child interactors. There is very limited layout
`information associated with a group other than that the
`group should be
`logically presented
`together. The
`geometric layout issues are addressed later in this paper. A
`hierarchy of groups forms the fundamental structuring
`mechanism for an XView. This structure is key to
`managing interactions where there is limited presentation
`capacity such as small screens or audio-only interfaces.
`
`A group also carries with it a summary descriptor. This is a
`pattern, which
`is used
`to assemble a brief
`textual
`description of the data contents of the group. In audio and
`small screen situations, this brief summary is invaluable in
`conserving presentation resources. The assembly of
`summaries is discussed later in the section on tree
`mapping.
`
`Ordered multi-column lists are our primary mechanism for
`handling arbitrarily large sets of data. Our implementation
`provides for client-side sorting on any of the columns.
`Most of our list views present summary information in a
`few columns with a link interactor that leads to a full view
`of the selected object.
`
`Future interactors
`There are various ways in which XWeb’s current interactor
`set falls short. There are no interactors that can manipulate
`images or sound. These media data types are important and
`must be included. It is very important that an interactor be
`provided for selection from a finite, but arbitrarily large set
`of choices. An example would be selecting a name from a
`phone book. There are many interactive techniques that
`have been developed which rely on such enumerated sets
`to provide focus to fuzzy or ambiguous inputs. Among
`them are speech recognition, text entry via phone pads[5],
`handwriting recognition, and pen-based typing [6]. XWeb
`needs such an interactor.
`
`The group and the list are an insufficient set of aggregate
`interactors. The list cannot handle very, very large data
`sets. An interactor that can handle sparse traversals of large
`data is required. We also feel that an interactor that can
`manage
`spatial
`relationships
`found
`in diagrams,
`schematics, and maps is required.
`
`In this first cut at the XWeb architecture we have focused
`on the underlying substrate and the relationships between
`clients on various platforms and various servers. Having
`laid down the fundamental architecture and explored the
`issues of cross-platform interaction, there is now greater
`justification for more general high level interactors. Such
`abstractions are not justified in the traditional architecture,
`where the user interface is tightly bound to the information,
`because the effort to use the abstraction can become as
`complicated as programming
`the solution by hand.
`However, when only a very few interactors can be
`programmed into each and every client these higher level
`abstractions become much more important.
`Tree remapping
`Defining an XView is fundamentally a process of mapping
`fragments of the data tree onto the tree of interactors that
`constitutes the user interface. Essentially the process must
`integrate content (the data) with interactive presentation.
`This is very similar to the goals of the eXtensible
`Stylesheet Language (XSL) [14]. XSL is focused on
`transforming data into a suitable presentation. This causes
`several problems. The first is that XSL is far more general
`purpose in terms of mapping arbitrary XML trees into
`other arbitrary trees. The second problem is that XSL
`mappings are many to one. In an interactive setting we
`need one to one mappings. Not only must the data be
`presented, but also when the user changes the presentation,
`that change must be transformed back into a change on the
`original data. In the case of a many to one mapping the
`reverse transformation is undetermined. In addition, it is
`hard for designers to predict the reverse behavior of a
`series of pattern matching rules. Our final constraint is that
`the mapping algorithm must be small so as not to impose
`code size problems on small interactive platforms.
`
`CHI Letters vol 2, 2
`
`195
`
`IPR2020-00357 Page 00005
`
`

`

`A second approach to defining the relationship between
`interactors and their data is to provide a programmatic
`connection such as JavaScript. Our problem with this
`solution is that designers must explicitly implement both
`directions of
`the
`transformation. This
`is
`further
`complicated by the fact that interaction is based on change
`record generation. Our last problem with explicit client-
`side programming of the interface is that it leads to
`platform specific interfaces. A declarative relationship
`between interactor and data provides more latitude for
`implementation by interactive clients.
`
`Figure 1 - Sample Interactors
`
`Example
`Assume that the following XML fragment represents the
`data for a home automation system that controls sprinklers.
`
`<sprinklers startTime="00:00">
`<days ID="days" sun="0" mon="1" tue="1"
`wed="0" thu="0" fri="0" sat="1"/>
`<zone time="30"/>
`<zone time="40"/>
`<zone time="15"/>
`<zone time="11"/>
`<zone time="22"/>
`<zone time="55"/>
`</sprinklers>
`
`What is desired is a user interface like that in figure 1.
`
`Specifying the view
`The view starts with a root interactor that is almost always
`a group interactor. The groups form a hierarchy of
`interactors. As each interactor is instantiated there is a
`binding created between that interaction description and a
`corresponding data object. Every interactor has a loc
`attribute, which specifies a path from its parent’s data
`object to its own data object. The loc may be empty, which
`indicates that the interactor references the same object as
`its parent, or it may be arbitrarily long, reaching deep into
`the data object hierarchy to reference a specific object.
`
`The skeleton view for this interaction is shown in the
`following XML
`
`<xview rootID="timer">
`<group ID="timer" loc="" >
`<time loc="" . . .> <name text="Start Time"/>
`. . . . . .
`</time>
`<group loc=""> <name text="Zone durations"/>
`<number loc="0/@time". . .> . . . . </number>
`<number loc="1/@time". . .> . . . . </number>
`. . . . . .
`</group>
`<group loc="days"> <name text="Days"/>
`<enum loc="@mon">
`<name text="Monday"/> . . . .
`</enum>
`<enum loc="@tue">
`<name text="Tuesday"/> . . . .
`</enum>
`. . . . . .
`</group>
`</group>
`</xview>
`
`The root view is the group with the ID of timer. This is
`mapped to the root object of our data, which is the object
`that encloses the <sprinklers> data object. This group
`contains a time and two other groups. Since the start time is
`found as an attribute of the <sprinkler> object, its loc is
`empty. Since all of the information for the Days group is
`found in the object whose ID="days", its loc is "days".
`Note that the second <number> in the Zone durations
`group has a
`loc of "1/@time", which
`indexes
`the
`<sprinkler>’s children (zero relative) and selects the time
`attribute as the container for this number. This recursive
`selection via path names is very easy to implement and
`easily supports the required two-way mapping between
`user and data.
`
`Data extraction composition patterns
`Some of the interactors have composite data values. For
`example a <time> consists of hours, minutes, seconds and
`
`CHI Letters vol 2, 2
`
`196
`
`IPR2020-00357 Page 00006
`
`

`

`AM/PM. A <date> has the year, month, day of the month,
`and day of the week. This information can appear in
`several formats and can be encoded in the data in various
`ways. There are also multiple ways
`in which
`the
`information can be presented to a user. In addition, there
`are the group summaries that are composed of data
`fragments from data being edited. We handle all of these
`mappings by means of a simple text matching and
`composition algorithm.
`
`All atomic data values in the data tree are text strings. A
`data value such as a date may be represented in several
`such strings (separate for day, month and year), or encoded
`into one string. The information presented to the user for
`manipulation
`is
`generally
`in
`one
`string. An
`extraction/composition pattern is a string with variable
`name references embedded in it. For example the pattern to
`extract the start time from the <sprinklers> is
`
`"$(hour_24):$(minutes)"
`
`All characters up to the colon are placed in the variable
`hour_24 and the ones after the colon are extracted into the
`variable minutes. On many interactors the variables are just
`named placeholders. In date and time there is a fixed set
`each with a special meaning. The userFormat attribute on
`the <time> specifies how the time is to be presented to the
`user. In this case the format is:
`
`"$(hour_12):$(minutes) $(AMPM)"
`
`In transforming information from the data to the user
`interface the process is to match the pattern against the
`data, filling variables as the extraction match proceedings.
`Then any variable calculations are done. In the case of
`<time>, the hour_12 (twelve hour clock) and AMPM
`values must be computed from the hour_24 variable. Once
`all variables are computed then the user data is composed
`using the user pattern. When the user edits the time, the
`reverse process is done. In the case of <time>, <date> and
`<group> summaries, the data can be in many places. The
`descriptor specifies all the places which can be sources for
`the data. Before variables are computed, all part strings are
`matched, to extract variable values from all of the places
`where they might be stored. The algorithm is simple, but
`provides for quite general encodings and
`two way
`mappings of data.
`
`Obviously the tree and data transformation capabilities of
`an XView are not sufficient for
`the most general
`transformations. We delegate such transformations to the
`server implementation. The data tree that a server presents
`is only a fa`ade over the real service. The server
`implementation