(12) United States Patent
`Ludwig et al.
`
`USOO6237025B1
`(10) Patent No.:
`US 6,237,025 B1
`(45) Date of Patent:
`*May 22, 2001
`
`(54) MULTIMEDIA COLLABORATION SYSTEM
`(75) Inventors: Lester F. Ludwig, Hillsborough; J.
`Chris Lauwers, Menlo Park; Keith A.
`Lantz, Los Altos; Gerald J. Burnett,
`Atherton, all of CA (US); Emmett R.
`B
`Jack WY (US
`urns, JacKSon,
`(US)
`(73) Assignee: Collaboration Properties, Inc., Incline
`Village, NV (US)
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`
`(*) Notice:
`
`This patent is Subject to a terminal dis-
`claimer.
`(21) Appl. No.: 08/994,848
`(22) Filed:
`Dec. 19, 1997
`9
`Related U.S. Application Data
`(63) Continuation of application No. 08/660,461, filed on Jun. 7,
`1996, now Pat. No. 5,802,294, which is a continuation of
`application No. 08/131,523, filed on Oct. 1, 1993, now Pat.
`No. 5,689,641.
`(51) Int. Cl. ............................. G06F 13/00; G06F 15/16
`(52) U.S. Cl. ............................................. 709/204; 345/330
`(58) Field of Search ..................................... 345/326, 329,
`345/330, 331, 332, 335; 709/202, 205,
`206, 207
`
`(56)
`
`3,723.653
`
`References Cited
`U.S. PATENT DOCUMENTS
`3/1973 Tatsuzawa.
`(List continued on next page.)
`FOREIGN PATENT DOCUMENTS
`35 07 152
`8/1985 (DE).
`(List continued on next page.)
`
`OTHER PUBLICATIONS
`Rangan et al. “Software Architecture for Integration of
`Video Services in the Etherphone System,” IEEE Journal on
`Selected Area of Communications, v9, n9, Dec. 1991, pp.
`1395-1404.
`
`(List continued on next page.)
`
`Primary Examiner Dung C. Dinh
`(74) Attorney, Agent, or Firm-Sughrue, Mion, Zinn,
`Macpeak & Seas, PLLC
`(57)
`ABSTRACT
`A multimedia collaboration System that integrates Separate
`real-time and asynchronous networks-the former for real
`time audio and Video, and the latter for control Signals and
`textual, graphical and other data-in a manner that is
`interoperable acroSS different computer and network oper
`ating System platforms and which closely approximates the
`experience of face-to-face collaboration, while liberating the
`participants from the limitations of time and distance. These
`capabilities are achieved by exploiting a variety of hardware,
`software and networking technologies in a manner that
`preserves the quality and integrity of audio/video/data and
`other multimedia information, even after wide area
`transmission, and at a significantly reduced networking cost
`as compared to what would be required by presently known
`approaches. The System architecture is readily Scalable to
`the largest enterprise network environments. It accommo
`dates differing levels of collaborative capabilities available
`to individual users and permits high-quality audio and Video
`capabilities to be readily Superimposed onto existing per
`Sonal computers and WorkStations and their interconnecting
`LANs and WANs. In a particular preferred embodiment, a
`plurality of geographically dispersed multimedia LANs are
`interconnected by a WAN. The demands made on the WAN
`are significantly reduced by employing multi-hopping
`techniques, including dynamically avoiding the unnecessary
`decompression of data at intermediate hops, and exploiting
`Video mosaicing, cut-and-paste and audio mixing technolo
`gies So that Significantly fewer wide area transmission paths
`are required while maintaining the high quality of the
`transmitted audio/video.
`
`44 Claims, 34 Drawing Sheets
`
`
`
`
`
`
`
`1. $7.
`s
`
`CSCO-1016
`CISCO SYSTEMS, INC. / Page 1 of 62
`
`

`

`US 6,237,025 B1
`Page 2
`
`U.S. PATENT DOCUMENTS
`
`3/1975 Kleinerman et al. .
`3,873,771
`8/1976 Tatsuzawa.
`3,974,337
`1/1977 Verhoeckx et al. .
`4,005,265
`10/1977 Poirier et al. .
`4,054,908
`7/1980 Yumide et al. .
`4,210,927
`4/1984 Schussler et al. .
`4,441,180
`5/1984 Burke et al. .
`4,451,705
`10/1984 Brown.
`4,475,193
`5/1985 Fabris et al..
`4,516,156
`7/1985 Colton et al. .
`4,529,839
`7/1985 Colton et al..
`4,529,840
`7/1985 Colton et al..
`4,531,024
`3/1986 Scordo .
`4,574,374
`2/1987 Pressman et al. .
`4,645,872
`3/1987 Boerger et al. .
`4,650,929
`3/1987 Hayden.
`4,653,090
`8/1987 Tompkins et al..
`4,686,698
`12/1987 Tomkins et al. .
`4,710,917
`12/1987 Tompkins et al. .
`4,716,585
`1/1989 Blinken et al. .
`4,796,293
`3/1989 Cree et al..
`4,817,018
`6/1989 Cohen et al. .
`4,837,798
`7/1989 Tompkins et al..
`4,847,829
`12/1989 Ando et al..
`4,888,795
`5/1990 Hashimoto.
`4,922,523
`6/1990 Stoddard et al. .
`4,931,872
`8/1990 Hayden et al. .
`4,953,159
`10/1990 Tsugane et al. .
`4,961.211
`10/1990 Kannes.
`4,965,819
`12/1990 McGauhey et al..
`4,977.520
`1/1991 Stults et al. .
`4,987,492
`2/1991 Weber et al. .
`4,995,071
`3/1991 Kao .
`4,998.243
`3/1991 Ashida et al..
`5,003.532
`4/1991 Goodman et al. .
`5,010,399
`5/1991 Tompkins et al. .
`5,014,267
`6/1991 Baji et al..
`5,027,400
`8/1991 Lee et al. .
`5,042,062
`10/1991 Smith.
`5,056,136
`12/1991 Todd.
`5,072,442
`12/1991 Suzuki et al. .
`5,073,926
`3/1992 Blinken, Jr. et al..
`5,099,510
`7/1992 Ikeno et al. .
`5,130,399
`7/1992 Bordry et al..
`5,130,793
`7/1992 Yamaguchi.
`5,130,801
`10/1992 Hammond.
`5,155,761
`10/1992 Kassatly.
`5,157,491
`12/1992 Guichard et al. .
`5,170,427
`5,195,086 * 3/1993 Baumgartner et al. .............. 345/329
`5,195,087
`3/1993 Bennett et al. .
`5,200,989
`4/1993 Milone.
`5,202,828 * 4/1993 Vertelney et al. ................... 345/326
`5,202,957
`4/1993 Serrao .
`5,218,627
`6/1993 Corey et al..
`5,224,094
`6/1993 Maher et al. .
`5,231,492
`7/1993 Dangi et al..
`5,239,466
`8/1993 Morgan et al..
`5,253,362
`10/1993 Nolan et al..
`5,260,941
`11/1993 Wilder et al..
`5,283,637
`2/1994 Goolcharan.
`5,303,343
`4/1994 Ohya et al..
`5,315,633
`5/1994 Champa.
`5,319.795
`6/1994 Hamabe et al. .
`5,333,133
`7/1994 Andrews et al. .
`5,333,299
`7/1994 Koval et al. .
`5,335,321
`8/1994 Harney et al..
`5,345,258
`9/1994 Matsubara et al. .
`5,351.276
`9/1994 Doll, Jr. et al..
`5,353,398
`10/1994 Kitahara et al..
`5,363,507
`11/1994 Nakayama et al. .
`
`5,365,265
`5,367,629
`5,373.549
`5,374,952
`5,375,068
`5,379,374
`5,382,972
`5,384,598
`5,384,772
`5,392.223
`5,392,346
`5,404,435
`5,408.526
`5,408,662
`5,422,883
`5,432,525
`5,444,476
`5,471.318
`5,473,679
`5,475,421
`5,485,504
`5,491,.695
`5,506,954
`5,515,491
`5,526,024
`5,550,966
`5,553,222
`5,561,736
`5,565,910
`5,581,702
`5,594.495
`5,602,580
`5,608,653
`5,689,553
`5,864,844
`
`11/1994 Shibata et al..
`11/1994 Chu et al..
`12/1994 Bales et al. .
`12/1994 Flohr.
`12/1994 Palmer et al. .
`1/1995 Ishizaki et al. .
`1/1995 Kannes.
`1/1995 Rodriguez et al..
`1/1995 Marshall.
`2/1995 Caci .
`2/1995 Hassler et al..
`4/1995 Rosenbaum .
`4/1995 McFarland et al..
`4/1995 Katsurabayashi.
`6/1995 Hauris et al..
`7/1995 Maruo et al. .
`8/1995 Conway
`11/1995 Ahuja et al..
`12/1995 La Porta et al..
`12/1995 Palmer et al. .
`1/1996 Ohnsorge.
`2/1996 Meagher et al. .
`4/1996 Arshi et al..
`5/1996 Bates et al. .
`6/1996 Gaglianello et al..
`8/1996 Drake et al..
`9/1996 Milne et al..
`10/1996 Moore et al. .
`10/1996 Rowse et al..
`12/1996 McArdle.
`1/1997 Palmer et al..
`2/1997 Tseng.
`3/1997 Palmer et al. .
`11/1997 Ahuja et al. .
`1/1999 James et al. ............................. 707/4
`
`
`
`FOREIGN PATENT DOCUMENTS
`
`O 041902 12/1981 (EP).
`O 190 060
`8/1986 (EP).
`O 354 370
`2/1990 (EP).
`O 403 118
`12/1990 (EP).
`O 414 222
`2/1991 (EP).
`O 436 345
`7/1991 (EP).
`O 453 128
`10/1991 (EP).
`O 497 O22
`8/1992 (EP).
`O 516371
`12/1992 (EP).
`O 523 618
`1/1993 (EP).
`O 523 626
`1/1993 (EP).
`O 535 601
`4/1993 (EP).
`OS48597
`6/1993 (EP).
`O 561 381
`9/1993 (EP).
`O 604 O53
`6/1994 (EP).
`WO 92/21211
`11/1992 (WO).
`WO 94/24803
`10/1994 (WO).
`
`OTHER PUBLICATIONS
`
`Marskak, Ronni T. “BeyondMail for Windows-epitomiz
`ing the mail-enabled application’, Patricia Seybold's Office
`Computing Report, Sep. 1992, v15 n9. (full text copy from
`Computer Select 1992 CD).*
`Videomatic Switching: Systems and Services, C. Crawford,
`F. Milone, D. Zoppellaro. Digital Communications, 1988
`International Zurich Seminar.
`ACM Press, Conference on Organizational Computing Sys
`tems, SIGOIS Bulletin, vol. 12, No. 2-3, Nov. 5-8, 1991.
`The American Users Forum (Niu-Forum) Aug. 6-9, 1990.
`Bellcore News, “IMAL Makes Media Merging Magic,”
`5(20), Nov. 9, 1988.
`
`CSCO-1016
`CISCO SYSTEMS, INC. / Page 2 of 62
`
`

`

`US 6,237,025 B1
`Page 3
`
`Frontiers in computer communications technology, Sigcom
`87 Workshop (Aug. 11–13, 1987).
`Unix 4" Berkeley Release 1991 man pages for login,
`htmp, talk, and who. Online Internet: http://www.de
`..freebdS. org.
`Ahuja et al., “Coordination and Control of Multimedia
`Conferencing.” IEEE Communication Magazine, 30(5):
`38-42, May 1992.
`Ahuja et al., “Networking Requirements of the Rapport
`Multimedia Conferencing System,” Infocom '88, IEEE, pp.
`746-751, 1988.
`Biswas et al., “Distributed Scheduling of Meetings: A Case
`Study in Prototyping Distributed Application,” System Inte
`gration, 1992 2" International Conference.
`Cohen et al., “Audio Windows: User Interfaces for Manipu
`lating Virtual Acoustic Environments, pp. 479-480.
`Cohen et al., “Audio Windows for Binaural Telecommuni
`cation.” EIC, Tokyo (Oct. 1991).
`Cohen et al., “Design and Control of Shared Conferencing
`Environments for Audio Telecommunication,” Proceedings
`of the Second Int’l Symposium on Measurement and Con
`trol Robotics (ISMCR '92), Tsukuba Science City, Japan,
`(Nov. 15–19, 1992), pp. 405–412.
`Cohen et al., “Exocentric Control of Audio Imaging in
`Binaural Telecommunication.” IEICE Trans. Fundamentals,
`vol. E75–A, No. 2, (Feb. 1992).
`Cohen et al., “Multidimensional Audio Window Manage
`ment.” Int’l Journal of Man-Machine Studie, vol.
`34:319-336 (1991).
`Cohen et al., “Multidimensional Audio Windows: Confer
`ences, Concerts and Cocktails, Human FactorS Society
`Meeting, SF, CA, pp. 1-15, Jun. 12, 1991.
`Ensor et al., “The Rapport Multimedia Conferencing Sys
`tem-Software Overview.” Computer Workstation Confer
`ence, IEEE, pp. 52–58, 1988.
`Gopal et al., “Directories of Networks with Causally Con
`nected Users," IEEE, pp. 1060–1064, 1988.
`Horn et al., “An ISDN Multimedia Conference Bridge.”
`Tencon 90–1990 IEEE Region 10 Conference on Com
`puter and Communication, pp. 853-856, 1990.
`Kamel, “An Integrated Approach to Share Synchronous
`Groupwire Workspaces," IEEE 1993.
`Kendall et al., “Stimulating the Cue of Spatial Hearing in
`Natural Environments.” Northwestern University, Evanston,
`IL 602O1.
`Klein, Telekommunikation von AngeSichtZu AngeSicht 2323
`Telcom Report 9 (1986) Sep./Oct., No. 5, Erlangen, W.
`Germany.
`Kobayashi et al., “Development and Trial Operation of
`Video Teleconference System,” IEEE Globecom, pp.
`2060-2063, 1999.
`Lake et al." A network environment for Studying multimedia
`network architecture and control,” (1989 Globecom).
`Lantz, An Experiment in Integrated Multimedia Conferenc
`ing, Department of Computer Science, Stanford University,
`Stanford, CA 94305, Dec. 1986.
`Lantz et al., Collaboration Technology Research at Olivetti
`Research California, Aug. 1989.
`Lauwers et al., Replicated Architecture for Shared Window
`Systems: A Critique, (Olivetti Research California) Version
`of Apr. 1990.
`
`Lauwers et al., Collaboration AwareneSS in Support of
`Collaboration Transparency: Requirements for the Next
`Generation of Shared Windows Systems, (Olivetti Research
`California) Version of Apr. 1989.
`Leung et al., Optimum Connection Paths for a Class of
`Videoconferences, Department of Information Engineering,
`the Chinese University of Honk Kong, Shatin, Hong Kong.
`Maeno et al., Distributed Desktop Conferencing System
`(MERMAID) Based on Group Communication Architec
`ture, The Transactions of the Institute of Electronics, Infor
`mation and Comm. Engineers E74 (1991) Sep., No. 9,
`Tokyo, JP.
`Martens, “Principal Components Analysis and Resynthesis
`of Spectral Cues to Perceived Direction,” Proceedings of the
`1987 Int’l Computer Music Conference; Northwestern Uni
`versity, Evanston, IL 60201.
`Masaki et al., “A Desktop Teleconferencing Terminal Based
`on B-ISDN: PMTC,” NTT Review, 4(4):81–85, 1992.
`NG et al., Systems Integration ’90, (Apr. 23–26, 1990).
`Nunokawa et al., “Teleconferencing Using Stereo Voice and
`Electronic OHP, IEEE, 1988.
`Ohmori et al., “Distributed Cooperative Control for Sharing
`Applications Based on Multiparty and Multimedia Desktop
`Conferencing System,” IEEE, 1992.
`Pate, “Trends in Multimedia Applications and the Network
`Models to Support Them,” Globecom’s 90: 1990.
`Perkins, “Spider: An investigation in collaborative technolo
`gies and their effects on network performance'.
`Ramanathan et al., Optimal Communication Architectures
`for Multimedia Conferencing in Distributed Systems, Mul
`timedia Laboratory Dept. of Computer Science and Engi
`neering, University of San Diego, La Jolla, CA.
`Rangan et al., “Software architecture for integration of video
`services in the etherphone system,” IEEE J. on Selected
`Areas in Comm., 9(9): 1395–1404, Dec. 91.
`Rangan et al., “A Window-Based Editor for Digital Video
`and Audio,” System Sciences, 1992 Hawaii Int’l Confer
`ence (1992).
`Sakata, “B-ISDN Multimedia Workstation Architecture,”
`IEEE, 1993.
`Sakata et al., “Development and Evaluation of an In-House
`Multimedia Desktop Conference System,” NEC Research &
`Development, No. 98, pp. 107-117, Jul. 1990.
`Stefik et al., “Beyond the Chalkboard: Computer Support for
`Collaboration and Problem Solving,” Communications of
`the ACM, vol. 30, No. 1, Jan. 1987.
`Vinet al., Hierarchical Conferencing Architectures for Inter
`-Group Multimedia Collaboration, Multimedia Laboratory
`Department of Computer Science and Engineering Univer
`sity of California at San Diego, La Jolla.
`Vin et al., Multimedia Conferencing in the Etherphone
`Environment, Computer Magazine, Vol. 24, ISSue 10, pp.
`69-79, 1991.
`Watabe et al., “A Distributed Multiparty Desktop Confer
`encing System and Its Architecture," IEEE, 1991.
`Watabe et al., “Distributed Desktop Conferencing System
`with Multiuser Multimedia Interface.” IEEE, 1991.
`Weiss, Desk Top Video Conferencing An Important Fea
`ture of Future Visual, Siemens AG-Munich-West Ger
`many.
`Zellweger et al., “An Overview of the Etherphone System
`and Its Applications,' Computer WorkStations Conference,
`1988.
`* cited by examiner
`
`CSCO-1016
`CISCO SYSTEMS, INC. / Page 3 of 62
`
`

`

`U.S. Patent
`
`May 22, 2001
`
`Sheet 1 of 34
`
`US 6,237,025 B1
`
`CSCO-1016
`CISCO SYSTEMS, INC. / Page 4 of 62
`
`

`

`U.S. Patent
`
`May 22, 2001
`
`Sheet 2 of 34
`
`US 6,237,025 B1
`
`
`
`
`
`CSCO-1016
`CISCO SYSTEMS, INC. / Page 5 of 62
`
`

`

`U.S. Patent
`
`May 22, 2001
`
`Sheet 3 of 34
`
`US 6,237,025 B1
`
`
`
`san
`ZZ SA &
`
`9. -- -
`O
`C
`Se
`
`c S
`
`i.
`
`...)
`3.
`O
`a.
`
`- 2
`do
`CC
`cry
`?o
`C.)
`
`c - 9.
`
`g
`
`.
`
`OOOOOOO
`OOOOOOOOOO
`
`CSCO-1016
`CISCO SYSTEMS, INC. / Page 6 of 62
`
`

`

`U.S. Patent
`
`May22, 2001
`
`Sheet 4 of 34
`
`US 6,237,025 B1
`
`‘HINOOD
`
`wtAVMALVOLy|NVMHOHMS12h
`
`
`eceadel).Ch
`Feweava
`
`JOQIYG
`
`Got
`
`NVIWWHOL
`
`JYVMGHVH
`
`NVIW
`
`WAOL
`
`Qse
`
`ONIGNOdS3YHO9OLanyYYNVI
`
`
`(91)S3OUNOS3Yviva
`
`AW
`
`eC
`
`
`
`acl(=)
`
`¢aunbry
`
`09909
`
`909WAAUAS(91)saguNOSau
`
`
`
`
`
`CSCO-1016
`CISCO SYSTEMS, INC. / Page 7 of 62
`
`CSCO-1016
`CISCO SYSTEMS, INC. / Page 7 of 62
`
`

`

`U.S. Patent
`
`May 22, 2001
`
`Sheet S of 34
`
`US 6,237,025 B1
`
`LOCATION B
`
`
`
`WS'''
`
`WAN
`
`Ayw GATEWAY
`LOCATIONA wan
`GATEWAY
`
`Gaay ASW
`LOCATION D
`
`LOCATION C
`
`Aigure 4
`
`CSCO-1016
`CISCO SYSTEMS, INC. / Page 8 of 62
`
`

`

`U.S. Patent
`
`May 22, 2001
`
`Sheet 6 of 34
`
`US 6,237,025 B1
`
`
`
`:
`
`
`
`
`
`SZAVN SXS X7
`
`Aigura 2
`
`122-n -
`
`
`
`14-
`
`114-
`
`
`
`116-1
`16-in
`
`
`
`r 36
`
`MOSAICING
`CIRCUITRY
`Aigure 7
`38
`AUDIO
`MXNG
`CIRCUITRY
`Aigure 9
`39
`
`WDEO
`CIRCUTRY
`
`
`
`
`
`16a
`
`116b
`116c
`116C
`
`Aique //
`
`8a
`Ba-2
`
`:-
`
`DGTAL FRAME
`BUFFER
`17
`
`CSCO-1016
`CISCO SYSTEMS, INC. / Page 9 of 62
`
`

`

`U.S. Patent
`
`May 22, 2001
`
`Sheet 7 of 34
`
`US 6,237,025 B1
`
`
`
`camuelLeave word
`Cails on Hold:
`Lester Ludwig, Keith Lantz
`
`Resne
`Resume
`Resume
`
`Aigure 84
`
`CSCO-1016
`CISCO SYSTEMS, INC. / Page 10 of 62
`
`

`

`U.S. Patent
`
`May 22, 2001
`
`Sheet 8 of 34
`
`US 6,237,025 B1
`
`rangupted Muelleave WordStatewietoard
`
`Calls on Hold:
`
`
`
`Aigure 8A
`
`CSCO-1016
`CISCO SYSTEMS, INC. / Page 11 of 62
`
`

`

`U.S. Patent
`
`May 22, 2001
`
`Sheet 9 of 34
`
`US 6,237,025 B1
`
`
`
`s
`
`R
`
`7
`)
`
`U
`
`Rise &
`A. f
`f AF2.
`
`E). (t/
`
`N
`
`SN
`
`Nss
`
`CSCO-1016
`CISCO SYSTEMS, INC. / Page 12 of 62
`
`

`

`U.S. Patent
`
`May 22, 2001
`
`Sheet 10 of 34
`
`US 6,237,025 B1
`
`STE NO. 1
`PARTICIPANTA
`PARTICIPANT B
`AB - O UNIT DELAY ATB
`C, D = 1 UNT DELA
`
`
`
`STE NO. 1
`PARTICIPANTA
`PARTICIPANT B
`
`2
`C D
`
`
`
`
`
`
`
`
`
`
`2
`Aigure /24
`
`STE NO. 2
`PARTICIPANT A
`PARTICIPANT B
`CD - O UNIT DELAY
`cio AB = 1 UNIT DELAY
`
`STE NO. 2
`PARTICIPANT A
`PARTICIPANT B
`
`A HEARS B - C - O
`C HEARSA + 8 + D
`8 HEARSA - C - O
`D HEARSA - B - C
`A.B = 0 UNT DELAY
`C.D. s. O UNIT DELAY
`CD = 1 UNT DELAY
`AB is 1 UNT DELAY
`Aigure J2A
`
`
`
`o
`
`STE NO. 2
`PARTICIPANTD
`
`
`
`
`
`
`
`
`
`STE NO. 1
`PARTICIPANTA
`PARTICIPANT B
`PARTICIPANT C
`A.
`A,B,C.. O UNIT DELAY AIB c2 AIBD - O UNIT DELAY
`D - 1 UNIT DELAY cd
`cDA,B,C - 1 UNIT DELAY
`Aigure Ja
`
`
`
`STE NO. 1
`PARTICIPANTA
`PARTICIPANT B
`PARTICIPANT C
`
`STE NO. 2
`PARTICIPANTD
`
`A HEARS B - C - O
`B HEARSA + C + D
`C HEARSA - B - D
`AB is OUNT DELAY
`D = 1 UNT DELAY
`Aigure JA
`
`D HEARSA - B - C
`D is OUNT DELAY
`A,B,C s 1 UNT DELAY
`
`CSCO-1016
`CISCO SYSTEMS, INC. / Page 13 of 62
`
`

`

`U.S. Patent
`
`May 22, 2001
`
`Sheet 11 of 34
`
`US 6,237,025 B1
`
`c
`
`STE NO. 2 - 42.558.
`A &
`PARTICIPANT C
`PARTICIPANT B
`As
`CD
`AB is OUNT DELAY
`C.D. 1 UNT DELAY
`Aigure J44
`
`cD
`C = 0 UNT DELAY
`AB = 1 UN DELAY
`D = 2 UNTSOELAY
`
`SITE NO. 3
`PARTICIPANTD
`c 2
`cD
`D = OUNT DELAY
`AB is 1 UNT DELAY
`C is 2 UNTS DELAY
`
`STE NO. 2
`PARTICIPANT C
`C HEARS A + B -- C
`
`A + 3 + C
`C
`
`D
`STE NO. 1
`STE NO. 3
`PARTICIPANT A
`PARTICIPANT B.A. B. c. PARTICIPANT D
`A HEARS B - C - D
`D HEARSA - B - C
`B HEARSA - C - D
`AB is OUNT DELAY
`C.D. s. 1 UN DELAY
`A.
`AiGURE /45
`
`De OUNT DELAY
`AB = 1 UNT DELAY
`C = 2 UNITS DELAY
`
`C = OUNT DELAY
`AB is 1 UNT DELAY
`O = 2 UNTS DELAY
`
`2
`2B
`cD SE NO. 3
`c D
`STE NO. 1
`STE NO. 2
`PAfricIPANT AR =PAF ficIPANT BREE
`AB
`AB
`AB 2. As
`cD
`C
`CD
`As OUNT DEAY
`B is O UN DELAY
`C.D. OUNT DELAY
`B is 1 UNT DELAY
`A.C.D. 1 UNIT DELAY B s 1 UNIT DELAY
`C.D. s. 2 UNTS DELAY
`As 2 UNITS DELAY
`Aigure /2A
`
`A
`
`2
`
`B - C - D
`STE NO. 1
`STE NO. 2
`PARTICIPANTA - PARTICIPANTB
`A HEARS B - C + D
`B HEARSA - C - D
`
`C -- D
`
`B is 1 UNIT DELAY
`C.D. s. 2 UNITS DELAY
`
`A,C,D = 1 UNT DELAY
`e
`Aigure J2A
`
`STE NO. 3
`- 5.5
`PARTICIPANT C &
`C HEARS A+ B + D
`D HEARSA - B - C
`C.D. s. OUNT DELAY
`B is 1 UNT DELAY
`As 2 UNITS DELAY
`
`CSCO-1016
`CISCO SYSTEMS, INC. / Page 14 of 62
`
`

`

`U.S. Patent
`
`May 22, 2001
`
`Sheet 12 of 34
`
`US 6,237,025 B1
`
`
`
`STE NO. 2
`PARTICIPANT B
`
`cD
`
`
`
`cD
`B is OUNT DELAY
`A 1 UNT DELAY
`CDs 2 UNITS DELAY
`
`
`
`STE NO. 4
`PARTICIPANTD
`
`cD
`OUNT DELAY
`a 1 UNT DELAY
`C = 2 UNTS DELAY
`
`AB 9 = 9 NRELAY
`As 1 UNT DELAY
`CD SD3NS5ELAY
`Aigure /6
`
`CSCO-1016
`CISCO SYSTEMS, INC. / Page 15 of 62
`
`

`

`U.S. Patent
`
`May 22, 2001
`
`Sheet 13 of 34
`
`US 6,237,025 B1
`
`As OUNT DELAY
`B is 1 UNT DELAY
`C = 2 UNITS DELAY
`O - 3 UNTS DELAY
`AB
`
`C = O UNIT DELAY
`B = 0 UNT DELAY
`B, D = 1 UNIT DELAY
`A, C = 1 UNIT DELAY
`A = 2 UNITS DELAY
`D = 2 UNIT DELAY
`AB
`AIB
`2B cd Z cd
`cD
`cD
`STE NO. 3
`C-PARTICIPANrc
`Alle 2
`
`
`
`
`
`
`
`
`
`
`
`3 = 2 UN
`
`LAY
`
`D = 0 UNT DELAY
`S; NES5E AB
`3 NSBEY cD
`Aigure /74
`
`As OUNT DELAY
`B is 1 UN DELAY
`C is 2 UNTS DELAY
`D = 3 UNTS DELAY
`
`B is OUNT DELAY
`AC s 1 UNT DELAY
`Ds 2 UNT DELAY
`
`C is OUNT DELAY
`B,D s 1 UNIT DELAY
`As 2 UNTS DELAY
`
`
`
`
`
`B - C -- O
`STE NO. 2
`PARTICIPANT B
`
`
`
`
`
`STE NO. 3
`PARTICIPANT C
`
`
`
`D - O UNT DELAY
`STE NO. 4
`C = 1 UNT DELAY
`B 2 UNTS DELAY PARTICIPANTD
`A 3 UNTS DELAY
`Aigure /75
`
`CSCO-1016
`CISCO SYSTEMS, INC. / Page 16 of 62
`
`

`

`U.S. Patent
`
`May 22, 2001
`
`Sheet 14 of 34
`
`US 6,237,025 B1
`
`CAMERA 500
`de
`
`MONITOR -2OO
`old-
`
`t MICROPHONE
`M 6OO
`
`la1
`
`O. O. O. O.
`
`SPEAKER EG
`
`7OO (G
`
`CPU
`
`1 OO
`
`101
`
`130
`
`KEYBOARD 300
`
`104
`8:
`
`
`
`
`
`ADO-ON BOX 800
`
`V-OUT 803
`
`
`
`n
`V-N BO1
`
`A-N
`802
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`AV NETWORK (UTP),901
`
`DATANETWORK (UTP),902
`
`Aigure /84
`
`CSCO-1016
`CISCO SYSTEMS, INC. / Page 17 of 62
`
`

`

`U.S. Patent
`
`May 22, 2001
`
`Sheet 15 of 34
`
`US 6,237,025 B1
`
`SIDE MOUNT - 850
`SOO
`600
`
`C
`O
`N - 820
`T
`R
`O L
`S
`
`700
`
`805
`
`M ONITOR
`
`2OO
`
`u-u
`-1
`
`u-1
`
`O O O O
`
`CPU - 1 OO
`
`150
`103
`102
`
`101
`
`1292
`110 2V16E55SFAY6A552
`2NEfwOKNTEFAC26A
`
`KEYBOARD - 300
`
`M
`
`SE
`
`
`
`EEEEEEEEEEEEE
`H
`
`DATA NETWORK (UTP) /902
`
`Aigure /8A
`
`AV NETWORK
`UTP)
`90
`
`CSCO-1016
`CISCO SYSTEMS, INC. / Page 18 of 62
`
`

`

`U.S. Patent
`
`May 22, 2001
`
`Sheet 16 of 34
`
`US 6,237,025 B1
`
`
`
`811
`
`CSR IT
`
`HANDSET WO
`X POWER AMP
`PREAMP
`
`NXHEADSETVo
`POWER AMP
`POWER AMP-82
`XA-OUT
`85 PREAMP
`g-A-IN
`EO
`
`a
`
`815
`
`LOOPBACKIAVMUTE
`
`AWTRANSCEIVERS
`VIDEO OUT WIDEON
`AUDIO 6UT AUDION
`
`AUDIO/VOEO
`O
`805
`
`AVNETWORK (UTP) (80
`Aigure /9
`
`CSCO-1016
`CISCO SYSTEMS, INC. / Page 19 of 62
`
`

`

`U.S. Patent
`
`May 22, 2001
`
`Sheet 17 of 34
`
`US 6,237,025 B1
`
`WORKSTATIONSOFTWARE - 160
`
`
`
`163
`
`ROLODEX
`
`162
`
`OUICKDIAL
`
`169
`
`VIDEOPHONE
`
`
`
`
`
`
`
`OTHER
`STANDARD
`APPLICATION
`170
`
`COLLABORATION
`INTIATOR
`
`161
`
`167
`
`COMPUTER
`INTEGRATED
`TELEPHONY
`
`
`
`COMPUTER
`INTEGRATED
`
`
`
`Ailee 20
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`CSCO-1016
`CISCO SYSTEMS, INC. / Page 20 of 62
`
`

`

`U.S. Patent
`
`May 22, 2001
`
`Sheet 18 of 34
`
`US 6,237,025 B1
`
`MLAN SERVER
`SOFTWARE MODULES
`
`62
`
`SERVICE SERVER
`69
`AVNM
`
`REAL-TIME
`AUDIO VISUAL
`STORAGE SERVER
`
`OTHER
`MULTIMEDIA
`SERVICES
`
`Aft/Re 27
`(2) REQUEST CALLEES
`ADDRESS
`
`(4) RETURN CALLEE'S
`ADDRESS
`
`COLLABORATION SCALLEE
`
`AWSWTCHING
`CIRCUTRY 30
`
`Aft/Re 24
`
`
`
`CALLER
`
`
`
`
`
`
`
`CSCO-1016
`CISCO SYSTEMS, INC. / Page 21 of 62
`
`

`

`U.S. Patent
`
`May 22, 2001
`
`Sheet 19 of 34
`
`US 6,237,025 B1
`
`
`
`E.
`
`Collaboration initiator
`
`Z
`
`Aigure 22
`
`CSCO-1016
`CISCO SYSTEMS, INC. / Page 22 of 62
`
`

`

`U.S. Patent
`
`May22, 2001
`
`Sheet 20 of 34
`
`US 6,237,025 B1
`
`LeIHD
`
`AW
`
`JON3YIINODoe
`O3GIVOIGAY AYLINOYIO
`
`SNIHOLIMS
`
`AN
`
`AYLINOUID
`
`OSNIHOLIMS
`
`JOGIHE
`
`JON3Y3INOO
`
`390Iua
`
`OIGINOIGAY
`
`ONIHOLIMS
`
`AWLINDUID
`
`ONIHOLUMS
`
`AYLINDYID
`
`CSCO-1016
`CISCO SYSTEMS, INC. / Page 23 of 62
`
`CSCO-1016
`CISCO SYSTEMS, INC. / Page 23 of 62
`
`
`
`

`

`U.S. Patent
`
`May 22, 2001
`
`Sheet 21 of 34
`
`US 6,237,025 B1
`
`210a
`
`
`
`WORKSTATION
`
`A.
`
`C
`
`WORKSTATION
`
`
`
`MULTIMEDIA DOCUMENT
`AUTHORING
`
`
`
`
`
`MULTIMEDIA CAUCONF.
`RECORDING
`
`
`
`
`
`
`
`N Q NS
`
`Aigure 29
`
`CSCO-1016
`CISCO SYSTEMS, INC. / Page 24 of 62
`
`

`

`U.S. Patent
`
`May22, 2001
`
`Sheet 22 of 34
`
`US 6,237,025 B1
`
`VIGAN
`
`c0S
`
`
`
`INJQN3d30-3WILp05
`
`AAILISNASNI-SWIL
`
`VIQSW
`
`O¢7uNOIY
`
`WaAHaSNeWAISAS3¥4Sauiovs
`
`SalWaldTWNOILNIANODASVYOLS
`
`YOLIGS
`WOWAVId
`
`/olanv
`
`OJGIA
`
`
`
`INGAZJUNLONGILS“H3dAHSOlHdvuDPONIDVINIixal
`M3GHOO3YHOLIGS
`
`
`ONDINIDINIVd$4O1I03
`
`SAMIMovsWwvHSDOHd
`
`HO1IG3HOLIGS
`
`SINSW93S
`SLOHSd¥NS
`
`/orany
`OAGIA
`030N093Y
`
`3JON3YI435U
`
`SINSNOdNO9
`
`
`
`
`
`SOINdVHDS3I9VWI1X3LN3SWNDO0G
`
`VIGSINILTNW
`
`CSCO-1016
`CISCO SYSTEMS, INC. / Page 25 of 62
`
`CSCO-1016
`CISCO SYSTEMS, INC. / Page 25 of 62
`
`
`
`
`
`
`

`

`U.S. Patent
`
`May 22, 2001
`
`Sheet 23 of 34
`
`US 6,237,025 B1
`
`MLAN ('
`- - - - - - -
`
`SWITCHING
`CIRCURY
`
`
`
`REAL-TIME AUDIOWDEO
`STORAGE SERVER CD
`STORAGE
`MEDUM
`
`13b
`
`13a
`
`25
`
`30
`
`AW
`SWCING PEN
`CIRCUITRY
`
`13a
`
`otter
`STORAGE
`: SERVERS
`
`13b
`
`3b
`
`13b
`
`
`
`
`
`
`
`CLENT
`WORKSTATION
`
`CENT
`WORKSTATION
`
`CLENT
`WORKSTATION
`
`12-1
`
`12-2
`Aigure 27A
`
`12-3
`
`CSCO-1016
`CISCO SYSTEMS, INC. / Page 26 of 62
`
`

`

`U.S. Patent
`
`May 22, 2001
`
`Sheet 24 of 34
`
`US 6,237,025 B1
`
`
`
`
`
`REAL-TIME AUDIO/VIDEO
`STORAGE SERVER
`
`COMPRESS
`MODULES
`
`STORAGE
`MEDUM
`
`
`
`CLIENT
`WORKSATION
`
`N-121
`
`12-2
`Aigure s/C
`
`524
`MULTIMEDIA
`MAL SYSTEM
`
`526
`MULTIMEDIADOCUMENT
`MANAGEMENTSYSTEM
`
`STORAGE
`SERVERS
`
`12-3
`
`522
`
`
`
`CONVENTIONAL
`FILE SYSTEM
`
`502
`REAL-TIME AUDIONDEO
`STORAGE SERVER
`
`Aigure 27A
`
`CSCO-1016
`CISCO SYSTEMS, INC. / Page 27 of 62
`
`

`

`U.S. Patent
`
`May 22, 2001
`
`Sheet 25 of 34
`
`US 6,237,025 B1
`
`
`
`Aique 2
`
`SCSI CROSSBAR
`
`540
`
`l
`se
`ES5
`C2F.
`E. g.
`a <3 Šss
`
`O
`
`El DISKS 60e
`eEe
`Ege
`Es
`
`a
`
`DSKS 67e
`
`Q
`
`st
`
`Š-532 5321
`NTW NSS
`CPU
`Aigure 22
`
`SR
`SansS
`CPU
`
`CSCO-1016
`CISCO SYSTEMS, INC. / Page 28 of 62
`
`

`

`U.S. Patent
`
`May 22, 2001
`
`Sheet 26 of 34
`
`US 6,237,025 B1
`
`USERS - INFORMATION SYSTEMS:
`
`INFORMATION:
`
`
`
`O AUDIOWDEO OF AL PARTIES
`O SHARED WINDOWS
`
`O TELEPONTING/ANNOTATION
`
`
`
`SAME TIME
`SAME PACE
`
`SAME TIME
`DFFERENT PLACE
`
`DIFFERENT TIME
`SAME PLACE
`
`DIFFERENT TIME
`DIFFERENT PLACE
`
`
`
`
`
`USERS INFORMATION SYSTEMS:
`
`INFORMATION:
`
`O AUDIOWDEO OF AL PARTIES
`
`O SHARED WINDOWS
`O TELEPOINTING/ANNOTATION
`
`Aigure 24
`
`CSCO-1016
`CISCO SYSTEMS, INC. / Page 29 of 62
`
`

`

`U.S. Patent
`
`May 22, 2001
`
`Sheet 27 of 34
`
`US 6,237,025 B1
`
`
`
`S
`s
`S. N
`
`CSCO-1016
`CISCO SYSTEMS, INC. / Page 30 of 62
`
`

`

`U.S. Patent
`
`US 6,237,025 B1
`
`
`
`
`
`
`
`9, 38/25/-/
`
`CSCO-1016
`CISCO SYSTEMS, INC. / Page 31 of 62
`
`

`

`U.S. Patent
`
`May 22, 2001
`
`Sheet 29 of 34
`
`US 6,237,025 B1
`
`
`
`
`
`Zaer yw/25/-/
`
`CSCO-1016
`CISCO SYSTEMS, INC. / Page 32 of 62
`
`

`

`U.S. Patent
`
`
`
`US 6,237,025 B1
`
`CSCO-1016
`CISCO SYSTEMS, INC. / Page 33 of 62
`
`

`

`U.S. Patent
`
`
`
`US 6,237,025 B1
`
`4, 3 M/25/-/
`
`CSCO-1016
`CISCO SYSTEMS, INC. / Page 34 of 62
`
`

`

`U.S. Patent
`
`May 22, 2001
`
`Sheet 32 of 34
`
`US 6,237,025 B1
`
`
`
`RN
`
`cool
`
`RElits
`ti?s Wic
`
`-ulu
`
`CSCO-1016
`CISCO SYSTEMS, INC. / Page 35 of 62
`
`

`

`U.S. Patent
`
`May 22, 2001
`
`Sheet 33 of 34
`
`US 6,237,025 B1
`
`
`
`N
`N
`s S
`
`CSCO-1016
`CISCO SYSTEMS, INC. / Page 36 of 62
`
`

`

`U.S. Patent
`
`May 22, 2001
`
`Sheet 34 of 34
`
`US 6,237,025 B1
`
`
`
`G
`
`3.
`s
`
`CSCO-1016
`CISCO SYSTEMS, INC. / Page 37 of 62
`
`

`

`1
`MULTIMEDIA COLLABORATION SYSTEM
`
`US 6,237,025 B1
`
`2
`AS discussed below, prior art desktop Videoconferencing
`Systems provide, at best, only a partial Solution to the
`challenges of distributed collaboration in real time, prima
`rily because of their lack of high-quality Video (which is
`necessary for capturing the visual cues discussed above) and
`their limited data Sharing capabilities. Similarly, telephone
`answering machines, voice mail, fax machines and conven
`tional electronic mail Systems provide incomplete Solutions
`to the problems presented by deferred (asynchronous) col
`laboration because they are totally incapable of communi
`cating visual cues, gestures, etc and, like conventional
`Videoconferencing Systems, are generally limited in the
`richness of the data that can be exchanged.
`It has been proposed to extend traditional videoconfer
`encing capabilities from conference centers, where groups
`of participants must assemble in the same room, to the
`desktop, where individual participants may remain in their
`office or home. Such a system is disclosed in U.S. Pat. No.
`4,710,917 to Tompkins et al for Video Conferencing Net
`work issued on Dec. 1, 1987. It has also been proposed to
`augment Such video conferencing Systems with limited
`“video mail” facilities. However, Such dedicated videocon
`ferencing Systems (and extensions thereof) do not effectively
`leverage the investment in existing embedded information
`infrastructures-Such as desktop personal computers and
`workstations, local area network (LAN) and wide area
`network (WAN) environments, building wiring, etc.-to
`facilitate interactive sharing of data in the form of text,
`images, charts, graphs, recorded Video, Screen displays and
`the like. That is, they attempt to add computing capabilities
`to a Videoconferencing System, rather than adding multime
`dia and collaborative capabilities to the user's existing
`computer System. Thus, while Such systems may be useful
`in limited contexts, they do not provide the capabilities
`required for maximally effective collaboration, and are not
`cost-effective.
`Conversely, audio and Video capture and processing capa
`bilities have recently been integrated into desktop and
`portable personal computers and workstations (hereinafter
`generically referred to as “workStations'). These capabilities
`have been used primarily in desktop multimedia authoring
`systems for producing CD-ROM-based works. While such
`Systems are capable of processing, combining, and recording
`audio, video and data locally (i.e., at the desktop), they do
`not adequately Support networked collaborative
`environments, principally due to the Substantial bandwidth
`requirements for real-time transmission of high-quality,
`digitized audio and full-motion Video which preclude con
`ventional LANs from Supporting more than a few workSta
`tions. Thus, although currently available desktop multime
`dia computers frequently include Videoconferencing and
`other multimedia or collaborative capabilities within their
`advertised feature Set (see, e.g., A Reinhardt, "Video Con
`quers the Desktop,” BYTE, September 1993, pp. 64–90),
`Such Systems have not yet Solved the many problems inher
`ent in any practical implementation of a Scalable collabora
`tion System.
`BRIEF DESCRIPTION OF THE DRAWINGS
`FIG. 1 is an enterprise view of a desk-top collaboration
`System embodiment of the present invention.
`FIGS. 2A and 2B are photographs which attempt to
`illustrate, to the extent possible in a Still image, the high
`quality of the full-motion Video and related user interface
`displays that appear on typical CMW screens which may be
`generated during operation of a preferred embodiment of the
`invention.
`
`CROSS-REFERENCE TO RELATED
`APPLICATIONS
`This application is a continuation of U.S. applications Ser.
`No. 08/660,461, filed Jun. 7, 1996, now U.S. Pat. No.
`5,802,294, which is a continuation of U.S. Ser. No. 08/131,
`523, filed Oct. 1, 1993, now U.S. Pat. No. 5,689,641.
`BACKGROUND OF THE INVENTION
`The present invention relates to computer-based Systems
`for enhancing collaboration between and among individuals
`who are separated by distance and/or time (referred to herein
`as “distributed collaboration”). Principal among the inven
`tion's goals is to replicate in a desktop environment, to the
`maximum extent possible, the full range, level and intensity
`of interpersonal communication and information sharing
`which would occur if all the participants were together in the
`same room at the same time (referred to herein as “face-to
`face collaboration”).
`It is well known to behavioral Scientists that interpersonal
`communication involves a large number of Subtle and com
`plex visual cues, referred to by names like “eye contact” and
`“body language,” which provide additional information over
`and above the spoken words and explicit gestures. These
`cues are, for the most part, processed Subconsciously by the
`participants, and often control the course of a meeting.
`In addition to spoken words, demonstrative gestures and
`behavioral cues, collaboration often involves the Sharing of
`Visual information-e.g., printed material Such as articles,
`drawings, photographs, charts and graphs, as well as Video
`tapes and computer-based animations, visualizations and
`other displayS-in Such a way that the participants can
`collectively and interactively examine, discuss, annotate and
`revise the information. This combination of Spoken words,
`gestures, Visual cues and interactive data sharing Signifi
`cantly enhances the effectiveness of collaboration in a
`variety of contexts, Such as “brainstorming Sessions among
`professionals in a particular field, consultations between one
`or more experts and one or more clients, Sensitive busineSS
`or political negotiations, and the like. In distributed collabo
`ration Settings, then, where the participants cannot be in the
`Same place at the same time, the beneficial effects of
`face-to-face collaboration will be realized only to the extent
`that each of the remotely located participants can be “rec
`reated” at each site.
`To illustrate the difficulties inherent in reproducing the
`beneficial effects of face-to-face collaboration in a distrib
`uted collaboration environment, consider the case of
`decision-making in the fast-moving commodities trading
`markets, where many thousands of dollars of profit (or loss)
`may depend on an expert trader making the fight decision
`within hours, or even minutes, of receiving a request from a
`distant client. The expert requires immediate access to a
`wide range of potentially relevant information Such as