USER INTERFACES FOR MULTIMEDIA MULTIPARTY COMMUNICATIONS
`J. R. Ensor S. R. Ahuja D. D. Seligmann
`
`AT&T Bell Laboratories
`Holmdel, NJ 07733
`
`ABSTRACT This paper, based on our experience with
`multimedia conferencing, discusses network control and
`coordination functions needed to support multimedia multiparty
`applications. In particular, the discussion focuses on user
`interfaces for these applications. Whereas conventional, single-
`medium devices (such as telephones) present their users with one
`connection to one party during a call, these new applications
`make multiple network resources visible to their users.
`Accordingly, our experience suggests that the user interfaces for
`these applications should not be built as simple additions to
`interfaces for conventional devices, but rather should reflect new
`communication models with independent controls of the network
`resources associated with the application. This paper describes
`important features of user interfaces for multimedia conferencing
`systems, illustrating how individual controls for network
`resources can be made available to the application user during a
`communication session. We also explain benefits of building
`user interfaces as collection of modules, in which some modules
`are responsible for conference conduct, and others provide
`controls for exchange of information in each medium.
`
`Introduction
`While conventional telephony is based upon a single
`connection for communication between two parties, new
`multimedia, multiparty applications are based upon multiple
`connections among several parties. Accordingly, most
`contemporary proposals for new network transport and signalling
`protocols emphasize dynamic, fine-grained control of service
`configurations. Many proposals identify information represented
`in a given medium with a connection or set of connections
`among communicating parties. By separating call and
`connection control, these designs permit changes in both the
`media and the participants involved in a call through changes in
`the connections used in the call. Other network resources are
`usually treated as support mechanisms for call management
`structures or for transmission services. For example, a video
`bridge is treated as part of a video transmission service.
`Therefore, these resources are not controlled separately; they are
`controlled as part of the connection control protocol.
`However, our experience with multimedia, multiparty
`communications indicates that other network resources deserve
`not only dynamic, but also explicit and separate control. Thus,
`bridges, billing programs, etc. need not be handled as
`components of connections, but may be treated as separate
`service nodes, each subject to independent behavior
`specifications. Applications, with access to additional network
`resources, can make more controls available to their users. For
`0-7803-0950-2/93/$3.00@1993IEEE
`
`example, a multimedia conferencing system (e.g. [Ah 881, [Wa
`901) can provide its users with separate controls for audio, video,
`and data bridges, allowing a conference participant to watch one
`person and attenuate the voice of another. However, the many
`controls which can give the application flexibility can also make
`it seem too complicated.
`This paper discusses user interfaces for one multimedia,
`multiparty application--multimedia conferencing. The
`discussion focuses on the control functions that must be provided
`by networks supporting this application and on how these
`functions can be presented to the application's user. We show
`that several sets of controls can be useful in this application,
`especially if they are presented to the user as representations of
`objects within the framework of a readily-understood model.
`Finally, we indicate how one such model not only unifies the
`control sets, but also provides a useful model of network
`services.
`
`Rapport Multimedia Conferencing System
`The Rapport multimedia desktop conferencing system
`[Ah 881 allows a group of people to hold real-time discussions in
`which they share voice, video, and program displays. Rapport
`provides call management mechanisms for its users and
`coordinates communications in various media during
`conferences. Rapport can manage conferences with a mix of
`devices, including both computers and telephones.
`Conventional telephone calls are simply two-party
`voice-only conferences. Voice is a readily-used medium for
`information exchange, as participants tell each other things; and
`it is also used as the primary medium for exchanging the inter-
`personal signals used to conduct conferences. In fact, we feel
`that real-time conferences cannot be easily conducted without
`voice.
`
`Conference participants with appropriate monitors may
`see each other through video feeds and also share displays of
`other video-based information. Video is a useful medium for
`meetings which involve negotiation, where conferees find value
`in seeing each other. Another use of video in conferences is to
`give "live" demonstration of procedure or technique, that is,
`showing how something is done or how some object looks.
`Data sharing in Rapport is accomplished by giving
`conferees the opportunity to interact in real-time with their
`existing, commonly used computer programs. The programs
`associated with Rapport conferences, which are termed
`application programs, produce identical displays on the
`computer screens of conference participants. These programs
`have been a major source of information for conference
`
`1165
`
`CISCO Exhibit 1015, pg. 1
`
`

`
`, - -
`
`Figure 1.
`Rapport Conference.
`Bob's Screen.
`
`participants, and this medium has served as the basis for
`Rapport's extensions to video teleconferencing.
`Figure I illustrates some of the sharing possible in
`Rapport by showing Bob's screen during a call with Sid. In the
`top-left region of his screen, Bob sees a live picture of Sid. Bob
`himself is represented by a still photo. Each participant's
`representation is accompanied by a listing of the media he can
`use during the call. In this conference Bob and Sid talk with
`each other using their telephones. They share programs and
`video through their computer monitors. The programs annotator
`and Artstudio have been associated with the call. The annotator
`application program allows Bob and Sid to annotate scanned or
`faxed images and documents. In this photo they have highlighted
`text as an annotation, and Bob is using his pointer to draw
`attention to a particular portion of the document. ArfSfudio
`allows the conference participants to make simple drawings on a
`full-motion video display.
`The Rapport user interface gives its user access to the
`underlying communication network. Operations are included to
`access (name) objects, for example, other people, various
`
`network resources, and ongoing conferences. Other operations
`specify how calls are initiated, suspended, resumed, and
`terminated. Finally, operations are provided to help the user
`modify ongoing conferences, by adding and deleting participants
`(members of the conference), media (used for information
`exchange), and application programs (which generate shared
`displays). Rapport's connection management level integrates
`communications in several media for each of the conferences in
`which a user participates. The communications may take place
`over a collection of networks or over a single multimedia
`network. In either case, Rapport maintains a description of
`conference
`topologies and
`required communication
`coordinations.
`
`Virtual Meeting Room Model
`People sometimes find a full display of Rapport controls
`to be daunting. Therefore, we present the controls in sets. The
`sets are unified through a single model, which we term the
`virtual meeting room. Each set of controls is associated with
`some network resource, which is described in terms of the virtual
`
`1166
`
`CISCO Exhibit 1015, pg. 2
`
`

`
`Figure 2.
`Multimedia Conference
`
`meeting room model. We have found that this model is not only
`useful for presenting controls to Rapport users, but also for
`describing network resources and their roles in services.
`
`Unifying Controls for the User
`The virtual meeting room model helps Rapport users to
`make sense of the many controls available to them. Rapport's
`single call control structure is used to establish, terminate, and
`conduct all Rapport calls, regardless of the number of conference
`participants or how many media are used. Standard telephony is
`subsumed in this model, but additional call controls become easy
`to use also. For example, n-party calls are now part of the
`normal control structure, requiring no user specification of a call
`"master" or "rendezvous" mechanism. Putting a call on hold is
`viewed as just "leaving the room."
`Similarly, Rapport users can control several media in a
`uniform manner, adding and deleting the media used in a
`conference as they wish. Application programs may be added to
`and deleted from conferences metaphorically "bringing them
`into" and "taking them out of the meeting room." Each
`application has an associated control protocol which determines
`when participants can give commands to the program. Since
`several applications can be associated with a virtual meeting
`room at once, Rapport provides a window manager for each
`room.
`
`Rapport provides specialized services to enhance
`interaction within the context of the virtual meeting room.
`Participants can signal each other or point to items of interest
`within each room, using a distinct telepointer [St 861. In Figure
`1, Sid and Bob are both using their telepointers to highlight text.
`Consider a call scenario in which Sid calls Bob.
`Regardless of the media that will be used to exchange
`information during the call, Sid places the call to "Bob." If the
`
`call is to be a voice-only conference, then Sid and Bob talk
`through a conventional telephone call. If Sid and Bob later wish
`to enhance this call by sharing programs, one or both can then
`add the programs to the conference as applications. The original
`call has expanded to include this new medium, data, but Sid's and
`Bob's user-level view of conference control remains constant.
`Similarly, Sid and Bob can add new participants to the call with a
`single control method. Each late-joining conferee automatically
`receives full and up-to-date displays in all media he or she wishes
`to use during this conference.
`As indicated in Figure 2, many connections among
`many inputloutput devices are controlled during this conference.
`Each participant controls his camera and video display, his
`microphone and the audio mixing for his speaker, his mouse and
`keyboard, and input/output filtering and distribution for his
`shared programs. Sid is also controlling video from a VCR.
`These control functions are grouped into sets. Each medium
`used in the conference has a control set, and within a medium
`individual devices are controlled as necessary. Similarly, each
`application program is associated with a set of controls.
`
`Modeling Network Services
`The virtual meeting room model is also useful for
`describing network services and the resources used to create
`them. While Figure 2 illustrates controls associated with devices
`used by conference participants, Figure 3 shows the
`interconnectivity between the conference interface and the
`various applications, managers, servers, and network resources.
`In the following scenario, Bob and Sid each control
`local input and output and video switching and bridging elements
`in the network. Bob frames the image that his camera is to
`transmit. Then he decides possible destinations for this
`transmission. The conference interface displays his choices
`
`1167
`
`CISCO Exhibit 1015, pg. 3
`
`

`
`Registry w
`f I Manager
`
`Conference
`
`Bridge
`
`CD Player
`
`Video Display
`
`Laser Disc
`
`Figure 3.
`Connections to and from the Conference Interface.
`Changing input can be accomplished by manipulation of the
`based on the information obtained from both the name server
`same icon. Each participant is represented by a static photograph
`and video bridge. Meanwhile, he specifies the source of his
`provided by the name server, the conference manager provides
`video display, and he sizes and positions this display on his
`information regarding the participant’s status (in or out of the
`computer monitor. The conference interface communicates with
`conference), the phone manager provides information regarding
`the video manager and local video display manager to
`the activity on the phone lines, and the video bridge provides the
`accomplish this. Sid performs these functions and also controls
`information to depict who the participant is looking at and
`the distribution of video from his VCR. The interface uses
`whether or not his camera is available.
`information from the name-server and video bridge to select a
`However, objects that are shared in a virtual meeting
`VCR and communicates with the video manager to route control
`room are still subject to local management. That is, Sid can turn
`commands directly to the VCR.
`down the volume of his speaker phone, move his application
`Similarly, Bob can control what audio sources are sent
`program windows, and increase the contrast on his video display.
`to the speakers in his room as well as distribute the audio from a
`Local controls are depicted by local managers.
`CD player to a set of destinations. The conference interface
`A graphical phone device, part of the conference
`queries the name server, but this time communicates with the
`interface, depicts connectivity information provided by the
`audio manager and audio bridge. Thus, Bob can additionally
`control the volume of his local speakers and as well as the CD
`conference manager, phone manager, and name server. Figure 5
`shows the graphical phone. Although the graphical phone
`player.
`resembles the physical phone on the person’s desk, it is enhanced
`with additional information and functionality. For example, the
`call appearance buttons are augmented to show the mapping to a
`particular conference. The directory service provides pictures
`and routing information for speed dialing.
`Application programs themselves can be written to
`exploit the information provided by the various connection
`
`The interface dynamically generates a comprehensible
`presentation of shared resources and connectivity. In Figure 4,
`we have labeled interface objects with both the type of
`information presented as well as its source.
`For example, the icons that represent each application
`program are color coded to indicate input control, information
`provided by the X sharing server at the application’s host site.
`
`1168
`
`CISCO Exhibit 1015, pg. 4
`
`

`
`live video of Sid
`access to:
`video-bridge
`video-manager
`
`Conference button:
`shows name and connectivity
`sources:
`conference manager
`phone manager
`shared display server
`video bridge
`can be used to enter and leave
`
`\
`
`Participant controls , \
`
`\
`\ source:
`
`\
`
`binoculars
`Sid is looking at Bob
`
`video- bridge
`
`\
`
`I
`
`I
`
`/
`
`full video option
`access to:
`local video display
`controller
`
`talking heads
`quad-box option
`access to:
`video-bridge
`conference manager
`
`application buttons
`shows:
`name
`input control
`process id
`owner
`/sources:
`/ conference manager
`-
`shared display server
`
` wav to switch inout
`
`managed by: ' / 1
`
`Bob's telepointer
`
`VMR window manager
`
`\
`
`local video controls
`
`\
`
`shared application program
`Artstudio
`
`VCR controls
`to VCR
`
`video-input buttons
`sources:
`video-bridge
`name-server
`way to select video source
`
`Figure 4.
`Rapport Interface Connectivity.
`
`servers and bridges. The Artstudio program queries the video
`bridge to determine what video sources are available, the local
`video manager to determine what local video display capabilities
`are available, and a name-server to adequately display the video
`source choices. The buttons of the application window indicate
`what video sources are available, the icons are selected based on
`the type of hardware, as described by a resource manager. Based
`on the type of hardware, and name service to indicate control
`routes, a graphical object is activated to provide specialized
`functions. In this example of Figure 6, the user has selected a
`serial line controlled VCR. When the user selected this VCR, a
`description of the VCR, retrieved from the video bridge and
`
`name-server, causes a set of VCR-specific controls to appear, as
`well a control object to be linked to both send commands and
`retrieve state.
`
`Conclusions
`Rapport and other computer-based, multimedia
`conferencing systems provide their users with electronic
`environments for sharing information. Their user interfaces often
`include controls for many of the network resources used in
`building the application. Our experience with conferencing
`systems suggests that the user interfaces for these and other
`multimedia, multiparty applications should not be built as simple
`
`1169
`
`CISCO Exhibit 1015, pg. 5
`
`

`
`Figure 5.
`Rapport Phone Controls
`
`Figure 6.
`Artstudio
`
`1170
`
`CISCO Exhibit 1015, pg. 6
`
`

`
`additions to interfaces for conventional, single-medium devices
`(such as telephones). Rather, we have found that acceptable user
`interfaces for these new applications must give first-class status
`to independent media and party controls.
`The complexity introduced by giving users functions to
`control so many items must be limited. We have found that a
`model-based interface allowed us to limit Rapport's apparent
`complexity. The virtual meeting room model has served not only
`to unify the controls in the user interface, but it has also helped
`network service designers define new service concepts.
`
`References
`
`[Ah 881
`Ahuja, S. R., Ensor, J. R., and Horn, D. N., "The Rapport
`Multimedia Conferencing System," Proc. Con. on Office
`Information Systems, Palo Alto, March 1988, pp. 1-8.
`
`[St 861
`Stefik, M., Bobrow, D., Lanning, S., and Tatar, D.,
`"WYSIWIS Revisited: Early Experiences With Multi-User
`Interfaces," Proc. Con. on Computer-Supported Cooperative
`Work, Austin, December 1986, pp. 276-290.
`
`[ Wa 901
`Watabe, K., Sakata, S., Maeno, K., Fukuoka, H., and Ohmori, T.,
`"Distributed Multiparty Desktop Conferencing System:
`MERMAID," Proc Con. on Computer-Supported Cooperative
`Work, Los Angeles, October 1990, pp.27-38.
`
`1171
`
`CISCO Exhibit 1015, pg. 7