Case 1:13-cv-00919-LPS Document 307-1 Filed 03/10/21 Page 1 of 17 PageID #: 26701
`Case 1:13-cv-00919—LPS Document 307-1 Filed 03/10/21 Page 1 of 17 PageID #: 26701
`
`EXHIBIT 21
`
`EXHIBIT 21
`
`

`

`CyberDesk: A Framework for Providin Self-Inte ratin Ubiquitous Software Services
`$5 l)ocument 307-1 Flled 03/10/21 Page 2 of 17 PageID #: 26702
`Case 1:13-cv-00919-LPS Document 307-1 Filed 03/10/21 Page 2 of 17 PageID #: 26702
`Case 1: 13-cv-O 919-L
`
`
`
`Back to the Cyheflesflcmpage
`
`CyberDesk: A Framework for Providing Self-
`
`Integrating Context-Aware Services
`
`Anind K. Dey, Gregory Abowd
`ra hics
`isuah'zation
`sabih'
`
`enter
`
`1 ieorgl'a Institute at Technology
`Atlanta, GA 30332-0280 USA
`+ 1-404-894— 7512
`
`{anind, abowd}@cc.gatech. edu
`
`Andrew Wood
`chool 0
`0m uter cience
`
`The
`
`niversi
`
`0 Birmin ham
`
`Edgbaston, Birmingham, 315 2TT UK
`amw@cs. bham .ac. uk
`
`
`
`ABSTRACT
`
`Applications are often designed to take advantage of the potential for integration with each other Via shared information.
`Current approaches for integration are limited, effecting both the programmer and end-user. In this paper, we present
`
`https://web.archive. org/web/l 9991 108220548/http: /www. cc . gatech. edu/fce/cyberdesk/pubs/IUI98/1Ul98 .html[7/3 l /2020 8:42:01 AM]
`
`F OX_0 0 0 64 73
`
`

`

`CyberDesk: A Framework for Providin Self-Inte ratin Ubiquitous Software Services
`Case 1:13-cv-0d919-Lfis bocument 307-1 Flled 03/10/21 Page 3 of 17 PageID #: 26703
`Case 1:13-cv-00919-LPS Document 307-1 Filed 03/10/21 Page 3 of 17 PageID #: 26703
`CyberDesk, a framework for self-integrating software in which integration is driven by user context. It relieves the
`burden on programmers by removing the necessity to predict how software should be integrated. It also relieves the
`burden from users by removing the need to understand how different software components work together.
`
`Keywords
`
`Context-aware computing, automated software integration, dynamic mediation, ubiquitous computing
`
`INTRODUCTION
`
`Software applications often work on similar information types such as names, addresses, dates, and locations.
`Collections of applications are often designed to take advantage of the potential for integration via shared information.
`As an example, an electronic mail reader can be enhanced to automatically recognize Web addresses, allowing a reader
`to select a URL to automatically launch a Web browser on that location. Even more complex and useful integrating
`behavior is available in a number of commercial suites of applications (e. g. Microsoft Office 97, Lotus SmartSuite,
`WordPerfect Suite).
`
`We recognize the utility from the user's perspective of integrating the behavior of a number of software applications.
`With the emergence of Web-based applications and personal digital assistants (PDAs), there are even more
`opportunities to provide integration of software applications. There are some limitations, however, to the current
`approaches for providing this integration. These limitations impact both the programmer and the user.
`
`From the programmer's perspective, the integrating behavior between applications is static. That is, the behavior must be
`identified and supported when the applications are built. This means that a programmer has the impossible task of
`predicting all of the possible ways users will want a given application to work with all other applications. This usually
`results in a limited number of software applications that are made available in an integration suite.
`
`From the user's perspective, integrating behavior is limited to the applications that are bound to the particular suite being
`used. Further integration is either impossible to obtain or must be implemented by the user. In addition, the integrating
`behavior has a strong dependence on the individual applications in the suite. If a user would like to substitute a
`comparable application for one in the suite (e. g. use a different contact manager, or word processor), she does so at the
`risk of losing all integrating behavior.
`
`The project described in this paper, CyberDesk, is aimed at providing a more flexible framework for integrating
`software behavior. We aim to reduce the programming burden in identifying and defining integrating behavior, while at
`the same time retaining as much user freedom in determining how integration is to occur. The main objective of the
`ubiquitous computing project CyberDesk is to provide the infrastructure for self-integrating software in which the
`integration is driven by actions of the user. We refer to this as context-aware integration, and it is aimed at producing a
`paradigm shift in human-computer interaction that is fundamental to ubiquitous computing. Rather than settle for the
`current situation in which the user must seek out and find relevant software functionality when she wants it, we instead
`want the ubiquitous computing infrastructure to seek out the user when and where she wants it. In this paper, we will
`demonstrate how CyberDesk provides for this paradigm shift.
`
`WHAT IS CYBERDESK?
`
`CyberDesk is a component-based framework written in Java, that supports automatic integration of software
`applications. The framework is flexible, and can be easily customized and extended. The components in CyberDesk
`treat all data uniformly, as a Java object, regardless of whether the data came from a desktop application, a PDA-based
`application, or a Web-based application.
`
`The intelligence in CyberDesk's user interface comes from applications automatically providing their services to the
`user. Rather than displaying all the services to the user at all times, the interface is limited to those services that are
`
`https://web.archive. org/web/l 9991 108220548/http: /www. cc . gatech. edu/fce/cyberdesk/pubs/IUI9S/IUI98 .html[7/3 l /2020 8:42:01 AM]
`
`F OX_0 0 0 64 74
`
`

`

`CyberDesk: A Framework for Providin Self-Inte ratin Ubiquitous Software Services
`Case 1:13-cv-0d919-Lfis bocument 307-1 Filed 03/10/21 Page 4 of 17 PageID #: 26704
`Case 1:13-cv-00919-LPS Document 307-1 Filed 03/10/21 Page 4 of 17 PageID #: 26704
`relevant to the user's context. A service is an action that an application can perform or data that an application can
`provide. A user's context is any information about the user and the environment that can be used to enhance the user's
`experiences. This includes the data the user is working with, the time of day, the user's physical location, emotional
`state, social environment, objects in the room, etc. Initially, CyberDesk was only able to work with simple strings that a
`user was working with in a desktop application. Now, CyberDesk is also able to work with time and location in a
`desktop environment, networked environment, and mobile environment.
`
`Desktop applications incorporated into CyberDesk include e-mail browsers, notepads, schedulers, and contact
`managers. Network applications include phone number lookups, e-mail writing, mailing address lookups, Web searches,
`Usenet searches, e-mail address lookups, map lookups, and Web page browsing. PDA-based applications include
`contact managers and notepads. All applications make their services available to the user via a common interface. The
`services available at any particular time depend on the user's context at that time. By providing relevant suggestions and
`data to the user, the user receives useful, and possibly unexpected, help in completing their task.
`
`User Scenario
`
`To illustrate this behavior, an actual user experience follows. (All the scenarios described in the paper can be executed
`
`at h-..- .....imgc .........o_.-..r.q..-.., ......
`As seen in the figurehelow, a user is checking his e-mail, and reads an
`e-mail from a friend about some interesting research.
`
`Future Computing Environments
`
`FFIOM: Andy Wood — annu@ee.gsteeh.edu
`T0: Anind Dey — anind@oo.gatech.edu
`
`You can get more info at httpjnnrwwse.gsteohedufioe!
`
`MESSAGE:
`
`Hey Anind!
`
`Have you seen all the great stuff that IIIiregory Ahowd
`nd |Li'hris Atkeson are doing with their Future
`Computing Environments projects?
`
`Figure 1. Content of User's E-mail Message.
`
`The user is interested in the research discussed, highlights the URL in the message, and CyberDesk offers the following
`suggestions through its interface (mm): search for the selected text using AltaVista, find pages that reference this
`URL using AltaVista, and display the URL in Netscape.
`
`https://web.archive. org/web/l 9991 108220548/http: /www. cc . gatech. edu/fce/cyberdesk/pubs/IUI9S/IUI98 .html[7/3 l /2020 8:42:01 AM]
`
`F OX_0 0 0 64 75
`
`

`

`CyberDesk: A Framework for Providin Self-Inte ratin Ubiquitous Software Services
`Case 1:13-cv-O 919-Lfi8 i)ocument 307-1 Filed 03/10/21 Page 5 of 17 PageID #: 26705
`Case 1:13-cv-00919-LPS Document 307-1 Filed 03/10/21 Page 5 of 17 PageID #: 26705
`
`Future Computing Environments
`FFtOltl'l : Andy Wood — amw@cc.gatech.edu
`T0: Anind Dey — anind@cc.gatech.edu
`M ESSAGE:
`
`Hey Anind!
`Have you seen all the great etutf that Gregory Abowd
`and Chris Atkeson are doing with their Future
`Computing Environments proiects‘?
`You eon get more into at Mtg
`
`Andy.
`”IIIIIIL‘II’LKItZlfil
`
`Figure 2. User selects the URL and is offered suggestions by CyberDesk.
`Click on the screenshot to get an image with better resolution.
`
`He chooses the last option and Views the URL listed in the message (Figure 3 ).
`
`uture Computing Environments
`ROM: Andy Wood - amw @ccgalechedu
`0: Anind Dey — anind©cc.gatech.edu
`ESSAGE:
`
`ave you wen all the great stuff that Gregory Abowd
`d Chris Atkeson are doing with their Futune
`omputing Environments projects?
`
`Figure 3. CyberDesk executes the service and displays the URL.
`Click on the screenshot to get an image with better resolution.
`
`The user then selects the name of the person in charge of the research and is offered the following suggestions (Figure
`5L): search for the selected text using AltaVista, search for a phone number and mailing address using Switchboard,
`lookup the name in the contact manager. The user wants to contact this researcher so he checks to see if the name is in
`his contact manager, but it isn't.
`
`https://web.archive. org/web/l 9991 108220548/http: /www. cc . gatech. edu/fce/cyberdesk/pubs/IUI9S/IUI98 .html[7/3 l /2020 8:42:01 AM]
`
`F OX_0 0 0 64 76
`
`

`

`CyberDesk: A Framework for Providin Self-Inte ratin Ubiquitous Software Services
`Case 1:13-cv-O 919-Lfi8 i)ocument 307-1 Filed 03/10/21 Page 6 of 17 PageID #: 26706
`Case 1:13-cv-00919-LPS Document 307-1 Filed 03/10/21 Page 6 of 17 PageID #: 26706
`
`
`
`eyAnind!
`ave you seen all the great stuff that fit?
`d Chris Mkeson are doing with their Future
`omputing Environments projects?
`ou can get more info at http:innrww.cc.gatech.eduflee.i
`ndy.
`
`
`
`
`uture Computing Environments
`ROM: Andy Wood - an1w@cc.gatech.edu
`0: Anind Dey — anind@cc.gatech.edu
`ESSAGE:
`
`Figure 4. User selects a name and chooses the Contact Manager lookup service.
`Click on the screenshot to get an image with better resolution.
`
`So, he selects the phone number and mailing address lookup service (Figure 5). He then creates a new entry in the
`contact manager with this new information.
`
`Future Computing Environments
`FIOM : Andy Wood - an'rw @cc.gatech.edu
`0: Anind Dey — anind@cc.gatech.edu
`ESSAGE:
`
`ndy.
`
`ey Anind!
`
`ave you seen all the great etufl that meg _
`nd Chris Atkeson are doing with their Future
`omputing Environments projects?
`
`_
`
`_
`
`on can get more info at http:fnnrww.cc.gatech.edufloef
`
`Listingl
`
`ldelffifififilghi Hflififiaflfil
`
`About], Gregory D 3327 Glenrose Trl, Atlanta, GA 30341—5781
`Phone: (770)934-9934
`
`E]?
`
`Figure 5. User selects the phone number lookup service.
`Click on the screenshot to get an image with better resolution.
`
`ARCHITECTURE
`
`The CyberDesk system has a simple architecture, based on an event-driven model, where components act as data
`sources and/or data sinks. The system consists of four core components: the IntelliButton, the ActOn Button Bar, the
`applications, and the type converters. The IntelliButton maintains the registry of data sources and sinks. It also finds
`services or data sinks that match the input data, a task normally required of the system or service designer. The
`
`https://web.archive. org/web/l 9991 108220548/http: /www. cc . gatech. edu/fce/cyberdesk/pubs/IUI9S/IUI98 .htm1[7/3 1 /2020 8:42:01 AM]
`
`F OX_0 0 0 64 77
`
`

`

`CyberDesk: A Framework for Providin Self-Inte ratin Ubiquitous Software Services
`Case 1:13-cv-O 919-Lfi8 l)ocument 307-1 Flled 03/10/21 Page 7 of 17 PageID #: 26707
`Case 1:13-cv-00919-LPS Document 307-1 Filed 03/10/21 Page 7 of 17 PageID #: 26707
`IntelliButton displays the matches in the form of suggestions to the user, via the ActOn Button Bar. It is through the
`ActOn Button Bar that the user accesses the integrating functionality of CyberDesk. The applications are the data
`sources and sinks themselves, and are the tools the user ultimately wants to use. When the user's context changes, either
`by working with new data or by a change in time or position, the new context is passed to the CyberDesk system (Figure
`_6_). The type converters provide more powerful integrating behavior by converting this data (a string in the previous
`scenario) into other data types (e. g. name), allowing for a greater number of matches.
`
`user EDDIEII
`
`lntelliButton
`
`
`
`
`ACIOn Button
`Bar
`
`user picks
`SEWiCE
`
`Figure 6. Runtime architecture diagram.
`
`All of the components have been implemented as Java applets for simplicity of network programming. We also chose
`Java for its promise of platform independence, ability to execute within a Web browser, and obj ect-oriented nature. The
`first two features support our goal of ubiquity, and the last feature made development easier. Also, most of the
`integrated network applications are available via the Web, so the natural access method was via a Web browser. Inter-
`component communication was performed using techniques based on the CAMEO toolkit [22], a C++ toolkit built
`previously by one of the authors to facilitate the integration of application-sized components via the use of agent-like
`components. More information on CyberDesk's architecture can be found in [7].
`
`ADDING APPLICATIONS TO CYBERDESK
`
`As discussed earlier, the applications used in CyberDesk are the actual tools the user wants to use. CyberDesk provides
`an easier and faster way of accessing the functionality of these applications and the data they contain (i.e. their services).
`From the user's perspective, adding an application (or any CyberDesk component) to CyberDesk simply requires the
`addition of HTML applet tags [9] to a CyberDesk HTML page. From the programmer's perspective, adding an
`application requires more effort.
`
`Currently, CyberDesk is unable to automatically determine the services each application provides. A services
`programmer must construct a wrapper around each application. This wrapper performs two main functions: registration
`of the provided services with CyberDesk and execution of the services when called. During the registration process,
`each application registers with the IntelliButton giving a list of services it provides, both actions it can perform on
`different data types and the data types it can produce. Examples of this from the previous scenario are:
`
`c
`
`.
`
`the AltaVista wrapper declaring it can search the Web for a string and find pages that reference a given URL
`
`the Contact Manager wrapper declaring it can lookup a given name, create a new entry, and can produce string
`objects when a user selects data in the contact manager.
`
`The second portion of the wrapper deals with actually executing the services that were registered. When the user selects
`
`https://web.archive. org/web/l 9991 108220548/http: /www. cc . gatech. edu/fce/cyberdesk/pubs/IUI9S/IUI98 .html[7/3 l /2020 8:42:01 AM]
`
`F OX_0 0 0 64 78
`
`

`

`CyberDesk: A Framework for Providin Self-Inte ratin Ubiquitous Software Services
`Case 1:13-cv-0d919-Lfis bocument 307-1 Flled 03/10/21 Page 8 of 17 PageID #: 26708
`Case 1:13-cv-00919-LPS Document 307-1 Filed 03/10/21 Page 8 of 17 PageID #: 26708
`a service from the interface, a method in the wrapper is called to execute the service. This method takes the user's
`context, retrieves the relevant parameters for the service and calls a method that will execute the user-selected service.
`
`All the Web-based applications employed by CyberDesk use HTML forms. By analyzing the form, a programmer can
`fairly easily write the wrapper. These applications generally have straightforward interfaces and require a small set of
`input parameters. The parameters are passed to a URL which generates a resulting HTML page that can be displayed in
`a Web browser. For example, the AltaVista Web search service simply requires an input string and returns a list of all
`Web pages that match this string. Currently, there are 68 Web-based applications that have been integrated into
`CyberDesk. A service writer program has been written to automatically generate a wrapper for Web-based applications.
`This program is intended for use by service programmers, but is simple enough to be used by an end-user wanting to
`add a service to CyberDesk. The program takes a URL containing a form as input and presents an interface as shown in
`Figure 7.
`
`HItanstg
`
`Search for a String;
`
`Séarch for a String u5ing F'ult-‘El'u'iStE
`
`Figure 7. Service Writer interface.
`
`The service programmer selects the data type the service can act on, the values for the service parameters, and the output
`data type, if any. Upon receiving this information, the service writer program generates a wrapper for the service.
`
`All of the desktop applications added to CyberDesk have been written by other Georgia Tech students. This provided
`access to both the application APIs and the source code. The APIs were needed to determine the names of methods for
`services the applications provided and the parameters that each method required. The source code allowed the service
`programmers to add additional services to the applications and add the data selection ability shown in the original
`scenario (e. g. user selecting the URL). It should be made clear that source code was only modified to enhance the
`existing applications. If only the API were available for an application, and not the source code, a wrapper could still be
`written to take advantage of the application's existing functionality. Currently, there are 6 desktop applications that have
`integrated into CyberDesk.
`
`In addition, an automated service writer, similar to the one mentioned above, is being designed. The service writer will
`be based on the newest release of the Java language (version 1.1). It provides an automatic data selection feature,
`
`https://web.archive. org/web/l 9991 108220548/http: /www. cc . gatech. edu/fce/cyberdesk/pubs/IUI9S/IUI98 .html[7/3 l /2020 8:42:01 AM]
`
`F OX_0 0 0 64 79
`
`

`

`CyberDesk: A Framework for Providin Self-Inte ratin Ubiquitous Software Services
`Case 1:13-cv-O 919-Lfi8 Locument 307-1 Flled 03/10/21 Page 9 of 17 PageID #: 26709
`Case 1:13-cv-00919-LPS Document 307-1 Filed 03/10/21 Page 9 of 17 PageID #: 26709
`allowing for the transfer of data between (Java and non-Java) applications via a clipboard-style interface [1 l], and
`supports the use of reusable software components called JavaBeans [10]. The data transfer feature will eliminate the
`need for any application source code. The use of JavaBeans would allow the service writer to query an application and
`determine it's API at run-time, removing the need for a compile-time API and allowing the use of third party
`applications.
`
`ADDING TYPE CONVERTERS TO CYBERDESK
`
`CyberDesk applications can generate changes in the data (generally strings) the user is working with. As described
`above, CyberDesk uses type converters to convert this user context (or location or time information, as will be shown in
`an upcoming section) into other useful forms of user context. For example, in the user scenario, a StringToURL
`converter took the data selected by the user and successfully converted it to a URL. This resulted in two pieces of data
`being sent to the IntelliButton, a string and a URL. The IntelliButton sought integrating behavior for both these types,
`allowing the user to access URL-relevant services where originally they wouldn't have had the option.
`
`The type converters work in a recursive fashion. That is, the new data that is generated from a successful conversion is
`sent to the type converters. This process continues until no new data is created.
`
`Initially, applications were hardcoded to generate different data types. For example, the e-mail browser declared that it
`could generate strings when text is highlighted, but also EmailAddress objects when the "To:" or "From:" field in an e-
`mail message was selected. When EmailAddress objects were generated, they were passed through the CyberDesk
`system, as described before, to the ActOn Button Bar, which displayed services that could consume EmailAddress
`objects (e. g. Send an E—mail to this E-mail Address using Netscape). However, this required the applications themselves
`to be aware of the CyberDesk type system. It was also limiting since e-mail addresses could also appear in the
`unforrnatted body text of an e-mail message and only be recognized as a string selection.
`
`Consequently, a decision was made to use type converters. Using simple heuristics, it is possible to identify potential
`text strings that might be e-mail addresses. It would have been desirable to augment the e-mail browser with this
`capability, so that any time text was selected in it, it would try to convert the text to an e-mail address and create an
`EmailAddress object rather than just a string. But, instead of just giving this type conversion capability to the e-mail
`browser, that ability should be added to the system once, and allowed to be used in every application where e-mail
`addresses might appear. The type detection ability was removed from the individual applications and type converters, an
`independent and extensible layer in the architecture, were created.
`
`For the programmer, writing a type converter involves writing a method that accepts one data type and converts it to
`another. For the user, adding type converters to a CyberDesk session allows for the use of a wider variety of user
`context. When user context changes, (change in time, location, or data selection) type converters improve the list of
`suggested actions given by CyberDesk by providing services specific to the content of the user context, not relying
`simply on the type of user context that has changed.
`
`Currently the list of CyberDesk types include: Date, PhoneNumber, MailingAddress, Name, URL, EmailAddress,
`GPSPosition, and Time. For each data type, there is a corresponding StringTo(data type) type converter.
`
`CHAINING
`
`By making a simple extension to the application wrappers, applications can gain the same advantages as type
`converters. Most of the services provided by applications require data types as input parameters and display their results
`through a graphical interface. This was seen in the above scenario when the user searched for a phone number and
`mailing address (a Web page was displayed) and when the user looked for a name in his contact manager (a contact
`entry was displayed). Through the use of simple parsing, this data encoded in the graphical interface can be obtained. In
`the phone number search, the HTML page returned can be examined and parsed to retrieve a matching phone number.
`Similarly, if the name being looked up in the contact manager had an existing entry containing an e-mail address, the e-
`mail address could be easily retrieved. This new data is part of the user's context and is made accessible to CyberDesk.
`
`https://web.archive. org/web/l 9991 108220548/http: /www. cc . gatech. edu/fce/cyberdesk/pubs/IUI9S/IUI98 .html[7/3 l /2020 8:42:01 AM]
`
`F OX_0 0 0 64 8O
`
`

`

`CyberDesk: A Framework for Providin Self-Inte ratin Ubiquitous Software Services
`Case 1:13-cv-00§19-LP% document 307-1 Flled 03/10/21 Page 10 of 17 PageID #: 26710
`Case 1:13-cv-00919-LPS Document 307-1 Filed 03/10/21 Page 10 of 17 PageID #: 26710
`
`When applications are able to generate additional pieces of context and perform user-selected actions, they are behaving
`both as type converters and as applications, providing the advantages of both. Now, applications can suggest actions
`directly related to the change in user context and actions indirectly related to the change in user context, reducing the
`effort required by the user to find these services. This process of generating additional context for the purpose of
`increasing integrating behavior is called chaining.
`
`A sample user scenario is described below. A user is reading an appointment in her scheduler and selects the name of
`the person she is supposed to be meeting (_F_i_gu_r_e__8_). As an experienced user, she expects to be presented with a list of all
`possible services that can use a Name: search for a phone number, mailing address, look up in the contact manager,
`search name on the Web, etc. However, by using chaining, more powerful suggestions can be had. The WhoWhere Web
`application takes a name as input and returns a Web browser showing a list of possible e-mail addresses corresponding
`to that name. If we make the assumption (not always a good one) that the first e-mail address returned in the list is the
`correct one, we can now use this service to convert the name to an e-mail address. The service now creates a related
`EmailAddress object, and the user is supplied with all possible suggestions for both a Name and an EmailAddress.
`
`
`
`Figure 8. Chaining example.
`Click on the screenshot to get an image with better resolution.
`
`Chaining is potentially a very powerful tool for the user to take advantage of. It provides another dimension of
`suggestions for each data type that the user context can be converted to.
`
`COMBINING
`
`Along the same line of thought, chaining can be used along with the concept of combining to make services more
`powerful. The services previously described were designed to only operate on a single data type (at a time). With data
`being converted to multiple types via chaining, the idea is that the services should be able to take advantage of these
`multiple types. They can, through a process we call combining.
`
`Combining, in CyberDesk terms, is the ability to collect multiple data types and dynamically bind them together, as
`needed, to create meta-objects which services can use. These meta-objects can be used to perform substantially more
`powerful actions. Using the above example of a user reading an appointment in her scheduler, the user selects a name,
`and a chaining service like Fourll is used to obtain a mailing address (and create a related MailingAddress object) for
`that name. Using combining, a meta-object containing both the name and the mailing address may now be used as input
`to a phone number lookup service like Switchboard. Switchboard can find phone numbers when given simply a name as
`input, but it can perform a more accurate search when it is provided with both a name and a mailing address.
`
`https://web.archive. org/web/l 9991 108220548/http: /www. cc . gatech. edu/fce/cyberdesk/pubs/IUI9S/IUI98 .html[7/3 l /2020 8:42:01 AM]
`
`F OX_0 0 0 64 8 1
`
`

`

`CyberDesk: A Framework for Providin Self-Inte ratin Ubiquitous Software Services
`Case 1:13-cv-00§19-LP% document 307-1 Flled 03/10/21 Page 11 of 17 PageID #: 26711
`Case 1:13-cv-00919-LPS Document 307-1 Filed 03/10/21 Page 11 of 17 PageID #: 26711
`Most services will perform better when provided with pertinent, additional context to work with. CyberDesk determines
`how to bind data together based on the data it currently has (the sum total of the current user context) and on the
`services available. It will not offer a suggestion to use Switchboard with just a name as input, when it can suggest
`Switchboard with both a name and mailing address.
`
`Now that the concept of combining has been explained, a more complete example demonstrating its power is given
`below. Again, we'll use the example of the user reading an appointment in her scheduler (Figure 9). She selects the
`name of a person she is meeting tomorrow. Immediately, she is offered suggestions of actions that she can perform with
`the selected string and name. As the chaining-enhanced applications return their data, this suggested list of actions is
`augmented with actions that can use an e-mail address (via WhoWhere), phone numbers and mailing addresses (via
`Switchboard) and URLs (via AltaVista). At the same time, the IntelliButton is dynamically binding these individual
`pieces of data for services that benefit from multiple data inputs.
`
`
`
`
`
`
`
`Figure 9. Combining example - user selects a name and is offered many integrating suggestions.
`Click on the screenshot to get an image with better resolution.
`
`The user chooses to create a new entry in the contact entry.
`
`This results in a rich entry (Figure 10), containing the original name she selected, an e-mail address, a URL, a phone
`number, and a mailing address.
`
`https://web.archive. org/web/l 9991 108220548/http: /www. cc . gatech. edu/fce/cyberdesk/pubs/IUI9S/IUI98 .html[7/3 l /2020 8:42:01 AM]
`
`F OX_0 0 0 64 82
`
`

`

`CyberDesk: A Framework for Providin Self-Inte ratin Ubiquitous Software Services
`
`Case 1-13-cv-00§19-LP% document 307-1 Flled 03/10/21 Pa e 12 of 17 Pa eID #- 26712
`Case 1:13-cv-00919-LPS Document 307-1 Filed 03/10/21 Page 12 of 17 PageID #: 26712
`
`
`
`
`Figure 10. Combining example - user creates rich contact entry.
`Click on the screenshot to get an image with better resolution.
`
`Combining, like chaining, can be very powerful to the user. It does not inhibit the list of options for individual pieces of
`user context, while at the same time it combines those pieces, enhancing the available services and their results. In
`essence, chaining and combining create a context inference engine in CyberDesk.
`
`OTHER FORMS OF CONTEXT
`
`In a mobile setting, there are additional forms of context that are not necessarily available in a desktop environment.
`Examples include a user's changing location, the changing objects in the environment, and the familiarity with the
`environment. CyberDesk has integrated services which allow access to data and applications on mobile devices [17] (a
`Newton Messagepad, in particular). Now, we are looking at integrating services that are available when the user is
`mobile, to take advantage of these other forms of context.
`
`Up until now, all of the examples shown have only used changes in context based on the data the user is currently
`working with. CyberDesk can deal with other forms of context in the same way as it deals with the user's data.
`Integrated examples include significant changes in time and position. One application that has been added to CyberDesk
`is one that updates the system time every five minutes. Currently, only one service that can use time has been integrated
`into CyberDesk. This service is part of the scheduler and it acts as a reminder service for events listed in the scheduler.
`When the time input into the scheduler is within fifteen minutes of an event, the scheduler offers a suggestion to the user
`to check their scheduler. Another, but more intrusive, option would be to create a window displaying the relevant
`information to the user. Ideally, the user would be able to set the type of feedback desired, and the event windows (i.e.
`how often the time service updates and how close to an event should a user be warned).
`
`https://web.archive. org/web/l 9991 108220548/http: /www. cc . gatech. edu/fce/cyberdesk/pubs/IUI98/IUI98 .html[7/3 l /2020 8:42:01 AM]
`
`F OX_0 0 0 64 83
`
`

`

`CyberDesk: A Framework for Providin Self-Inte ratin Ubiquitous Software Services
`Case 1:13-cv-OO 19-LP35 Document 307-1 Flled 03/10/21 Page 13 of 17 PageID #: 26713
`Case 1:13-cv-00919-LPS Document 307-1 Filed 03/10/21 Page 13 of 17 PageID #: 26713
`
`Position information has also been incorporated into CyberDesk, for use in a mobile setting. The current system uses
`Global Positioning System (GPS) data and is intended for outdoor use. The application providing GPS data updates the
`system position whenever the GPS coordinates change. Again, how often the application updates will be a user-
`controlled parameter. A service has been written that accepts GPS information for a location on the Georgia Tech
`campus and returns a URL corresponding to that location. CyberDesk then suggests all the activities it can perform with
`a URL, including displaying it in a Web browser. An example of this is shown in Figure 11.
`
`College of Computing.
`
`e arrived at Georgia Teen and have been to
`following places so far
`Sludanl Cantar
`