`
`[11] Patent Number:
`
`5,375,068
`
`[45] Date of Patent: Dec. 20, 1994
`Palmer et al.
`
`
`
`US005375068A
`
`[54] VIDEO TELECONFERENCING FOR
`NETWORKED WORKSTATIONS
`
`[75]
`
`Inventors: Ricky S. Palmer; Larry G. Palmer,
`both of Nashua, NH.
`
`[73] Assignee: Digital Equipment Corporation,
`Maynard, Mass.
`
`an audio data‘stream' tohthe remote Workstation such .
`that the audio‘data can be reconstructed into a continu-
`ous audio signal. A video transmitter sends video data
`”so that each frame of video data to be sent is inserted .
`into the audio data stream Without affecting the conti-
`nuity of the reconstructed audio signal at the remote
`workstation.
`
`[21] App]. No.: 893,074
`
`[22] Filed:
`
`Jun. 3, 1992
`
`Int. Cl.5 ............................................. G06K 15/00
`[51]
`[52] US. Cl. ....................................... 364/514; 370/62
`[58] Field of Search .................... 364/514; 379/96, 94,
`379/202; 370/60, 62; 395/152
`
`[56]
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`
`3,584,142
`6/1971 Schoeffler
`...... 178/68
`4,387,271
`6/1983 Artorn .........
`.. 179/2 DP
`
`4,516,156 5/1985 Fabris et a1. ........... 358/85
`
`..
`..... 364/200
`4,525,779 6/1985 Davids et a1.
`
`4,574,374 3/1986 Scordo ...................... 370/62
`
`4,645,872 2/1987 Pressman et a].
`..
`....... 379/54
`
`3/1987 Boerger et a1. ............... 358/86
`4,650,929
`
`. 379/204
`4,653,090 3/1987 Hayden ......
`
`......
`7/1987 Nelsoy et a1.
`4,679,191
`370/84
`
`.
`4,686,698
`8/1987 Tompkins et al.
`379/53
`.
`4,710,917 12/1987 Tompkins et a1.
`370/62
`
`4,734,765 3/1988 Okada et a1. .......
`. 358/102
`
`4,748,618
`5/1988 Brown et a1.
`..
`370/94
`........
`.. 358/136
`4,827,339
`5/1989 Wada et a1.
`
`4,847,829 7/1989 Tompkins et al. .
`'370/62
`
`.. 358/133
`4,849,811
`7/1989 Kleinerman
`
`4,882,743 11/1989 Mahmoud ......
`379/53
`4,888,795 12/1989 Ando et al.
`379/53
`
`.......
`4,893,326
`1/1990 Duran et a1.
`379/53
`
`1/1990 Majmudar et a1.
`4,897,866
`379/94
`
`
`. 370/94.l
`4,905,231 2/1990 ‘ Leung et a1.
`......
`............ 379/53
`4,918,718" _4fl99_0_ E99103“ a1.
`
`(List continued on next page.)
`
`workstation. The master process 'of a lecal workstation '
`causes execution of a slave process on a remote work-
`station for receiving video teleconference data from the
`local workstation. An audio data transmitter for sends
`
`46 Claims, 28 Drawing Sheets
`
`OTHER PUBLICATIONS
`
`Palmer and Palmer, “Desktop Meeting”, LAN Maga-
`zine, 6(11):111—121 (Nov. 1991).
`D. Comer, “Internetworking with TCP/IP, vol. I: Prin- V
`ciples, Protocols, and Architecture”, 2nd Edition, pp.
`1—8, 337—346, 505 (Prentice Hall: Engelwood Cliffs,
`NJ. 1991).
`
`-
`
`Primary Examiner—Emanuel T. Voeltz
`Assistant Examiner—Thomas Peeso
`Attomey, Agent, or Firm—Hamilton, Brook, Smith &
`Reynolds
`
`[57]
`
`ABSTRACI‘
`
`A video teleconferencing method and apparatus for
`computer workstations connected by a digital data net-
`work includes a transmission source portion for a local
`workstation to send audio and video teleconference
`data across the network to one or more remote worksta-
`tions, and, a receiver for the local workstation to re-
`ceive audio and video teleconference data back from
`the remote workstations. The local workstation sends
`teleconference data to each of the remote workstations .
`over a variable bandwidth digital data connection, and
`each of the remote workstations returns teleconference
`data back to the local workstation over another variable
`bandwidth digital data connection. The transmission
`source portion includes a master software process exe-
`cuting on the local workstation, and the receiver in-
`cludes a slave software process executing the remote
`
`Headphones
`with
`Microphone
`
`
`
`
`
`
`video
`
`Camera
`
`
`Headphones
`wuth
`Microphone 38
`
`
`
`To other Workstations
`
`Cisco - Exhibit 11020 {Page 1
`
`Cisco - Exhibit 1020 - Page 1
`
`
`
`M5
`
`,375,068
`Page 2
`
`4,924,311
`4,932,047
`4,935,953
`4,942,540
`4,943,994
`4,953,159
`4,953,196
`4,962,521
`4,965,819
`4,995,071
`5,003,532
`5,034,916
`
`US. PATENT DOCUMENTS
`5/1990
`6/1990
`6/1990
`7/1990
`7/1990
`8/1990
`8/1990
`10/1990
`10/1990
`2/1991
`3/1991
`7/1991
`
`.......................... 358/138
`Ohki et a].
`
`.
`...... 379/53
`Emmons et a1.
`Appel et a1. ........... 379/53
`
`.................
`Black et a1.
`364/514
`Ohtsuka et a1. ................... 379/53
`
`..... 370/62
`Hayden et a1.
`..
`
`.
`.. 379/53
`Ishikawa et a1.
`.
`...... 379/53
`Komatsu et a1.
`
`Kannes ........;.......... 379/53
`
`Weber et a1. .......... 379/53
`
`Ashida et a1.
`...... 370/62
`
`Ordish ................................. 364/900
`
`5,042,006
`5,042,062
`5,046,079
`5,046,080
`5,056,136
`5,062,136
`5,072,442
`5,079,627
`5,099,510
`5,103,444
`5,111,409
`5,1 13,431
`5,195,086
`5,200,989
`
`8/1991
`8/1991
`9/1991
`9/1991
`10/1991
`10/1991
`12/1991
`1/1992
`3/1992
`4/1992
`5/1992
`5/1992
`3/1993
`4/1993
`
`Flohrer
`..... 364/900
`Lee et a1.
`.. 379/54
`Hashimoto ..
`379/53
`
`Lee et a1.
`379/53 "
`
`Smith ..........
`389/10 '
`Gattis et a1.
`...............
`380/18
`Todd
`370/62
`
`Filo ........................
`358/85
`Blinken, Jr. et a1.
`..
`.. 379/202 ’
`
`Leung et a1. ...........
`370/60 '
`Gasper et al. ..
`.. 395/152 .
`
`Horn ...........
`379/94 :
`
`Baumgartner et a1.
`.. 379/202 -
`Milone .................................. 379/96
`
`Cisco - Exhibit 1020 - Page 2
`
`Cisco - Exhibit 1020 - Page 2
`
`
`
`US. Patent
`
`Dec. 20, 1994
`
`Sheet 1 of 28
`
`5,375,068
`
`10
`
`38
`
`video /Camera
`
`
`5O
`
`Headphones
`with
`Microphone
`
`Headphones
`With
`Microphone 38
`
`
`
`
`
`
`
`ultimode
`Fiber
`
`Video
`
`
`
`To other Workstations
`
`Cisco - Exhibit 1020 - Page 3
`
`Cisco - Exhibit 1020 - Page 3
`
`
`
`U
`
`.\.9»S.N.weon3$0.x0mm..
`
`
`
`
`
`.x34tmomnom«wwfixu'—noo_89>w‘
`um_ommm
`
`on.Owe;
`
`summnfimam_m5&8mqu_89>Aan.mm.89>_mmssomq:m_m.BD22.33%3.5.529
`
`
`._4.6528mmmmqmozofimmmazoom,
`
`
`
`
`
`O_UDOOMD@N_....IIIIII1..lllllCROM—UR,0...“.le
`_.
`
`.m85.98s85...
`
`7
`
`8
`
`So“.
`
`3,.96fl:Emfinwmz
`5S«NEoohmu8..928
`
`.ma8So...880mm-2._
`mufloEzooBooxn.x53528;.mm.
`
`
`
`MO5,x2:So....NG_n_
`
`Cisco - Exhibit 1020 - Page 4
`
`Cisco - Exhibit 1020 - Page 4
`
`
`
`
`
`US. Patent
`
`Dec. 20, 1994
`
`Sheet 3 of 28
`
`5,375,068
`
`x server (video)
`
`‘
`
`TCP/IP loco )
`
`(
`‘
`m ”NWTCPIIP
`DTP using 84
`
`50
`
`82
`
`‘Xmediousmg‘x‘ pp@MIIX
`TCP/IP or DECnei‘,(vTCP/IP (local)@0646
`
`-74
`
`v usin
`
`TCP/IP (iocgl)
`
`system calls
`
`FIG. 3
`
`®l06
`
`signal/execv
`
`
`
`@ ~e twork
`
`Machine A
`
`Machine 8
`
`FIG. 4
`
`Cisco - Exhibit 1020 - Page 5
`
`Cisco - Exhibit 1020 - Page 5
`
`
`
`US. Patent
`
`Dec. 20, 1994
`
`Sheet 4 of 28
`
`5,375,068
`
`|2CI
`
`no
`
`A calls 8, B answers
`
`FIG. 5A
`
`I2
`
`I2b
`
`:rfl02
`”20ii—T—”0 Acalls C, C answers
`“00
`I2C
`
`FIG. 58
`
`E32;
`|l20/\IIO Elgar,
`IIOO
`<
`
`”0b
`
`B cells C or A joins B and C
`
`FIG. SC
`
`Cisc‘o - Exhibit 1020 - Page 6
`
`Cisco - Exhibit 1020 - Page 6
`
`
`
`US. Patent
`
`Dec. 20, 1994
`
`Sheet 5 of 28
`
`5,375,068
`
`ANALOG VIDEO
`NTSC/SECAM/PAL
`
`. 20°
`
`DIGITIZE VIDEO IN
`FRAME BUFFER
`
`202
`
`VIDEO DATA IN
`APPLICATION BUFFER
`
`.
`
`.204
`
`VIDEO DATA IN
`NETWORK BUFFER
`
`. 206
`
`VIDEO DATA
`RAVERSES NETWORK
`
`203
`
`VIDEO DATA IN
`NETWORK BUFFER
`
`*
`
`2'0
`
`VIDEO DATA IN
`APPLICATION BUFFER
`
`. 2'2
`
`DIGITIZED VIDEO
`IN FRAME BUFFER
`
`_ 2'4
`
`DIGITIZED VIDEO
`DISPLAYED
`
`2'6
`
`FIG. 6
`
`Cisco - Exhibit 1020 - Page 7
`
`Cisco - Exhibit 1020 - Page 7
`
`
`
`US. Patent
`
`Dec. 20, 1994
`
`Sheet 6 of 28
`
`5,375,068
`
`LOCAL
`WORKSTATION A
`
`REMOTE
`WORKSTATION B
`
`
`
`INVOKE LOCAL
`DECspin
`APPLICATION
`
` 300
`
`NETWORK
`
`3.0.2_________ CREATE
`
`304
`
`PORT
`
`306
`”“_" —‘“‘“
`
`3I4
`CHEflCK fOR _____
`OK
`
`SEND
`
`"STARTHEADER"
`
`3'6
`
`‘— _ _ ’I
`
`:
`I
`
`- - — - —
`
`
`
`
`
`
`
`
`
`
`308
`
`
`
`INVOKE LOCAL
`DECspind
`APPLICATION
`
`
`
`VERIFY
`END-TO-END
`
`CONNECTION
`
`3IO
`
`3|2
`
`_ __ _.._ _
`
`l I
`
`L
`
`320
`YES
`ENTER
`NO
`
`
`CONFERENCE?
`
`
`SEND "QT"
`
`OR "ANSWER
`-MACI~IINE'l
`
`
`INVOKE LOCAL
`DEC spin
`
`APPLICATION
`
`
`
`
`
`CHECK FOR
`"OK I!
`
`
`BEGIN SENDING
`AUDIO/VIDEO
`
`DATA
`
`
`'
`
`i2: _
`
`l
`I
`l
`324 I
`I
`
`I I I
`
`_
`
`FIG. 7
`
`Cisco - Exhibit 1020 - Page 8
`
`Cisco - Exhibit 1020 - Page 8
`
`
`
`US. Patent
`
`Dec. 20, i994
`
`Sheet 7 of 28
`
`5,375,068
`
`START
`
`OK
`
`01010100 = 0x54
`
`|
`
`onoaoou = 0x53
`
`Total length = 2 bytes
`
`ouoonou = 0x43
`
`|
`
`ouoouu = Ox4F
`
`Total length = 2 bytes
`
`STARTHEADER
`
`5 reserved longwords (32 bits each)
`
`framerate desired (longword)
`
`DTP flags
`
`2 reserved longwords
`
`406
`
`4I2
`
`total number of frames
`
`-
`
`408
`
`I reserved Iongword
`
`video width
`
`video height
`
`bits per pixel (8 or 24)
`
`35 reserved longwords
`
`400
`
`402
`
`404
`
`78 longwords (used as login/ user handle)
`
`4'0
`
`FIG. IO
`
`Cisco - Exhibit 1020 - Page 9
`
`Cisco - Exhibit 1020 - Page 9
`
`
`
`US. Patent
`
`Dec. 20, 1994
`
`Sheet 8 of 28
`
`5,375,068
`
`QUIT
`
`‘
`
`OIOIOIOO = 0x54
`
`|
`
`onouooou = 0x5!
`
`Total length = 2bytes
`
`FIG. II
`
`ANSWERMACHINE
`
`onoo: IOI = 0x40
`
`|
`
`ouoooom = 0x4!
`
`Total length = 2 bytes
`
`FIG. I2
`
`CONTROL
`
`0x54
`
`I
`
`0x43
`
`450
`
`452
`
`45"
`
`456
`
`which
`
`length
`
`flags
`
`5 reserved longwords
`
`Total length is 34 bytes
`
`FIG. I3
`
`Cisco - Exhibit 1020 - Page 10
`
`Cisco - Exhibit 1020 - Page 10
`
`
`
`US. Patent
`
`Dec. 20, 1994
`
`ShEet 9 of 28
`
`5,375,068
`
`
`
`NEXT VIDEO
`FRAME
`
`AVAILABLE
`
`
`
`
` SEND
`CONTENTS OF
`
`AUDIO BUFFER
`
`352
`
`350
`
`SEND VIDEO
`FRAME
`
`354
`
`TRIM CONTE T
`
`368
`
`
`
`
`
`>I/2 sec.
`0.: Mom” 3
`
`
`TIMEOUT
`BUFFER TO
`
`LAST I12 SEC
`"OK“
`
`362-
`>2 FRAME
`
`TIMES
`IIOKII
`
`
`SEND
`
`CONTENTS OF
`
`AUDIO BUFFER
`
`
`360
`
`
`
`"OK"
`<2 FRAME
`
`cnecx FOR
`"0K"
`
`~
`
`FIG. I4
`
`Cisco - Exhibit 1020 - Page 11
`
`Cisco - Exhibit 1020 - Page 11
`
`
`
`US. Patent
`
`Dec. 20, 1994
`
`Sheet 10 of 28
`
`5,375,068
`
`AUDIOHEADER
`
`0x55
`
`|
`
`0x4!
`
`4'4
`
`length
`
`Audio Flags
`
`6 reserved longwords
`
`' FIG. l5
`
`VIDEOHEADER
`
`0x48
`
`1
`
`Ox4F
`
`420
`
`Video Width
`
`Video Height
`
`Control Flags
`
`Timing Information
`
`4 reserved longwords
`
`Total Length is 34 bytes
`
`FIG. I6
`
`4'6
`
`4'8
`
`422
`
`424
`
`426
`
`428
`
`Cisco - Exhibit 1020 - Page 12
`
`Cisco - Exhibit 1020 - Page 12
`
`
`
`US. Patent
`
`m,
`
`Mrifl'r'llLmmxoqmxo<3.2
`
`
`mNw2<mm
`
`“—0mZOECMIJOO25mm_midmmmo
`
`
`
`zo_mm_s_mz<m._.2.0mm
`
`
`
`“.0ZOCOMJJOO25mm
`
`
`
`_ms_<m_n_Pzwmmbo
`
`h
`
`Sm
`
`con:¢£24m“.mm2<muN924mm_width
`
`a_Ius:v_mcmm2<mmNH.mV*mhN._I0.?
`
`2w;15mh.0—...a
`
`w
`
`mmw09%
`
`Cisco - Exhibit 1020 - Page 13
`
`
`
`
`
`US. Patent
`
`Dec. 20, 1994
`
`Sheet 12 of 28
`
`5,375,068
`
`5|6
`
`5|8
`
`524
`
`520
`
`522 /
`
`Cisco - Exhibit 1020 - Page 14
`
`Cisco - Exhibit 1020 - Page 14
`
`
`
`US. Patent
`
`Dec. 20, 1994
`
`Sheet 13 of 28
`
`5,375,068
`
`858
`
`FIG. 19
`
`Cisco - Exhibit 1020 - Page 15
`
`Cisco - Exhibit 1020 - Page 15
`
`
`
`US. Patent 7
`
`Dec. 20, 1994
`
`Sheet 14 of 28
`
`5,375,068
`
`600
`
`620 622
`
`@540 624
`[Ed—5
`
`ha
`
`Cisco - Exhibit 1020 - Page 16
`
`Cisco - Exhibit 1020 - Page 16
`
`
`
`US. Patent
`
`Dec. 20, 1994
`
`' Sheet 15 of 28
`
`5,375,068
`
`650
`
`
`
`FIG. 21
`
`Cisco - Exhibit 1020 - Page 17
`
`Cisco - Exhibit 1020 - Page 17
`
`
`
`US. Patent
`
`Dec. 20, 1994
`
`Sheet 16 of _28
`
`5,375,068
`
`4m.X652
`
`720
`
`FIG. 22
`
`Cisco - Exhibit 1020 - Page 18
`
`Cisco - Exhibit 1020 - Page 18
`
`
`
`US. Patent
`
`Dec. 20, 1994
`
`Sheet 17 of 28
`
`5,375,068
`
`00m
`
`5324
`
`.v
`
`catatmcoemo
`
`m\
`
`u
`
`5528:...
`
`{0382
`
`3885
`
`$395
`
`¢N.oE
`
`wasth
`
`8:9.8:8o.36Av
`
`
`
`0..co_$m>E
`
`NON.+6290;.o9430.
`
`mmOE
`
`Ooh
`
`Cisco - Exhibit 1020 - Page 19
`
`Cisco - Exhibit 1020 - Page 19
`
`
`
`
`US. Patent
`
`Dec. 20, 1994
`
`I
`
`Sheet 18 of 28
`
`5,375,068
`
`°
`
`o
`
`x\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\-
`O
`
`0
`
`V2
`
`g
`
`0
`I \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\V-
`§
`
`IEEBEEEE u
`
`WorkstationA
`
`Iannaaaa
`Iaanaaau
`
`El
`
`El
`
`""uuunuunuu
`
`Imamaaaa a
`
`E]
`
`Cisco - Exhibit 1020 - Page 20
`
`Cisco - Exhibit 1020 - Page 20
`
`
`
`US. Patent
`
`Dec. 20, 1994
`
`Sheet 19‘of 28
`
`5,375,068
`
`A
`.o
`V
`(.0
`N
`.
`9
`u.
`
`.0
`
`0r
`
`0
`
`Cisco - Exhibit 1020 - Page 21
`
` m
`
`c
`o
`':
`2
`U)
`i‘
`g
`
`Cisco - Exhibit 1020 - Page 21
`
`
`
`US. Patent
`
`Dec. 20, 1994
`
`Sheet 20 of 28
`
`5,375,068
`
`290
`
`nOm®
`
`m8:29.83
`
`onmm.o_n_
`
`nOm
`
`Cisco - Exhibit 1020 - Page 22
`
`Cisco - Exhibit 1020 - Page 22
`
`
`
`
`
`US. Patent
`
`Dec. 20, 1994_
`
`Sheet 21 of 28
`
`5,375,068
`
`§ WorkstationA
`
`°
`
`O
`
`A\\\\\\\\\\\\\\\\\\\x\\\\\\\\\\\‘-
`0
`
`0
`
`¢5
`
`2
`
`n . 1
`I lI]
`
`D
`
`0
`K &.\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\V-
`
`
`
`FIG.26(d)
`
`DEEEEEEEB n Fa
`gfiananaaa
`
`El
`
`um
`
`Cisco - Exhibit 1020 - Page 23
`
`Cisco - Exhibit 1020 - Page 23
`
`
`
`US. Patent
`
`Dec.20, 1994
`
`Sheet 22 of 28
`
`5,375,068
`
` WorkstationC
`
`30c
`
`FIG.26(e)
`
`Cisco - Exhibit 1020 - Page 24
`
`Cisco - Exhibit 1020 - Page 24
`
`
`
`US. Patent
`
`Dec. 20, 1994
`
`Sheet 23 of 28
`
`5,375,068
`
`
`
`00m
`
`CvmmGE
`
`05:23:02,
`
`Cisco - Exhibit 1020 - Page 25
`
`Cisco - Exhibit 1020 - Page 25
`
`
`
`US. Patent
`
`Dec. 20, 1994_
`
`Sheet 24 of 28
`
`5,375,068
`
`“ o
`N
`
`fl 0
`
`O
`
`&\\\\\\\W\V-
`o
`
`OO..
`x \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\V_
`O
`N
`
`gfianeaaaa
`
`WorkstationA
`
`nflaamaaaa
`
`Cisco - Exhibit 1020 - Page 26
`
`Cisco - Exhibit 1020 - Page 26
`
`
`
`US. Patent
`
`Dec. 20, 1994
`
`Sheet 25 of 28
`
`5,375,068
`
`m5:235?
`
`
`gwm.9“.8m
`
`
`EEEEEEEEEHEEEEE
`
`
`
`555
`
`Cisco - Exhibit 1020 - Page 27
`
`Cisco - Exhibit 1020 - Page 27
`
`
`
`US. Patent
`
`Dec. 20, 1994
`
`Sheet 26 of 28
`
`5,375,068
`
`05:29.83
`
`
`
`m5323
`
`
`ACwm.o_n_com
`
`Cisco - Exhibit 1020 - Page 28
`
`Cisco - Exhibit 1020 - Page 28
`
`
`
`US. Patent
`
`Dec. 20, 1994
`
`Sheet 27 of _28
`
`5,375,068
`
`w
`
`m5:233;
`
`Cisco - Exhibit 1020 - Page 29
`
`Cisco - Exhibit 1020 - Page 29
`
`
`
`US. Patent
`
`Dec. 20, 1994
`
`Sheet 28 of 28
`
`5,375,068
`
`
`
`25m.0_|._
`
`3
`
`00m
`
`08:29.83
`
`Cisco - Exhibit 1020 - Page 30
`
`Cisco - Exhibit 1020 - Page 30
`
`
`
`1
`
`VIDEO TELECONFERENCING FOR
`NETWORKED WORKSTATIONS
`
`5,375,068
`
`BACKGROUND OF THE INVENTION
`
`Video communications has evolved over the years
`from a simple video telephone concept to a sophisti-
`cated network for allowing multiple users to participate
`in a video teleconference. For full featured video tele-
`conferencing, users require both an audio communica-
`tions path and a real time visual communication path
`synchronized to the audio path. Furthermore, it is desir-
`able to support full color video and telephone quality
`audio. Video teleconferencing capabilities are limited
`mainly by the bandwidth of the transmission medium
`connecting the teleconferencing terminals.
`Many computer workstations used in the office or
`laboratory environment today are connected with other
`workstations, file servers, or other resources over high-
`speed local area networks. Local area networks, in turn,
`are often connected together through high-speed speed
`gateways which connect workstations which may be
`distributed over a wide geographic area. Network wide
`protocols allow workstations to exchange packets of
`data at high rates of speed and reliability. Fixed band-
`width digital and analog video channels have been com-
`bined with computer networks to implement some
`video teleconferencing features. These include high
`bandwidth CATV/FDM type analog channels and
`fixed allocation TDM data channels for the video data.
`
`SUMMARY OF THE INVENTION
`
`Workstations today have obtained unprecedented
`computational power and utility. The powerful RISC
`type CPUs and fast, high resolution graphical displays
`have made possible multimedia workstations which
`integrate live audio and video into the programming
`environment. Graphical User Interface operating sys-
`tems (GUI) have allowed effective integration of audio
`and video into application programming.
`The present invention provides n-way video telecon-
`ferencing among networked computer workstations
`using the existing variable bandwidth digital data net-
`work for transferring synchronized audio and video
`teleconferencing data between the workstations. The
`teleconferencing apparatus and protocol of this inven-
`tion provides high quality video teleconferencing with-
`out the need for a guaranteed wide bandwidth analog
`video channel or a fixed allocation digital video chan-
`nel. Rather, the invention uses standard non-allocated
`data packets typically found on local area networks to
`transfer the audio and video teleconferencing data.
`Thus, no guaranteed bandwidth is required to carry on
`a useful video teleconference. An continuous audio data
`stream model provides continuous audio signals at the
`expense of video data when necessary, which is desir-
`able since the ear is more sensitive to a break in the
`audio data than the eye is to the loss of a frame of video
`data. A “push” data model provides a secure system by
`preventing remote workstations from activating an-
`other workstation’s video teleconferencing functions.
`In general, in one aspect, the invention features a
`video teleconferencing method and apparatus for com-
`puter workstations connected by a digital data network.
`The computer workstations include a transmission
`source means for a local workstation to send audio and
`video teleconference data across the network to one or
`more remote workstations, and, a receiver for the local
`
`5
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`45
`
`50
`
`55
`
`65
`
`2
`workstation to receive audio and video teleconference
`data back from the remote workstations. The local
`workstation sends teleconference data to each of the
`remote workstations over a variable bandwidth digital
`data connection, and each of the remote workstations
`returns teleconference data back to the local worksta-
`tion over another variable bandwidth digital data con-
`nection. The variable bandwidth digital data connec-
`tions include the data packet oriented data channels
`associated with, for example, FDDI, DECnet, and
`Ethernet local area networks. Furthermore, a wide area
`digital network, such as ISDN, can also be used with
`the video teleconferencing apparatus and method of this
`invention.
`
`In preferred embodiments, the transmission source
`means includes a master software process executing on
`the local workstation, and the receiver includes a slave
`software process executing the remote workstation.
`The master software process formats and sends video
`teleconference data to the slave process. The slave pro-
`cess receives and reconstructs the audio and video tele-
`conference data for audible and visual reproduction,
`respectively. The video data is presented as an image on
`the display of the receiving workstation, while the
`audio data is sent to either amplified speakers or head-
`phones. In other preferred embodiments, the master
`process of a local workstation causes execution of a
`slave process on a remote workstation for receiving
`video teleconference data from the local workstation.
`The slave process running on the remote workstation in
`turn causes execution of a master process on the remote
`workstation for sending video teleconference data back
`to the local workstation. The master process of the
`remote workstation in turn causes execution of a slave
`process on the local workstation for receiving the video
`teleconference data sent by the master process of the
`remote workstation. The local workstation executes a
`slave process for each master process on a remote work-
`station sending video teleconference data to the local
`workstation.
`
`In yet other preferred embodiments, the transmission
`source includes an audio data transmitter for sending an
`audio data stream to the remote workstation such that
`the audio data can be reconstructed into a continuous
`audio signal. The transmission source also includes a
`video transmitter for sending video data to the remote
`workstation so that each frame of video data to be sent
`is inserted into the audio data stream without affecting
`the continuity of the reconstructed audio signal at the
`remote workstation.
`In yet other preferred embodiments the video trans-
`mitter precludes a frame of video data from being sent
`to the remote workstation if a system overload exists.
`The audio transmitter sends the audio data stream cor-
`responding to the precluded video frame to the remote
`workstation to prevent loss of continuity of the audio
`signal during a system overload. In other preferred
`embodiments the video transmitter precludes a frame of
`video data from being sent to the remote workstation in
`response to a system failure condition. The audio trans-
`mitter accumulates audio data for a predetermined time
`interval during the system failure condition, and trans-
`mits the accumulated audio data stream to the remote
`workstation once the failure has been corrected. For
`instance, the audio transmitter may continually accumu-
`late the last ; second of audio data while the failure
`exists, trimming any audio data older than é second.
`
`Cisco - Exhibit 1020 - Page 31
`
`Cisco - Exhibit 1020 - Page 31
`
`
`
`3
`The last é second of audio data accumulated before the
`failure’s correction is sent to the remote workstation as
`soon as the failure is corrected. The most recently avail-
`able frame of video is then also sent.
`In still other preferred embodiments, timing informa-
`tion is attached to each frame of video data sent to the
`remote workstation. The timing information indicates a
`point in the continuous audio data stream which corre-
`sponds in time to the frame of video data. The receiver
`of the remote workstation includes a synchronizer for
`displaying a received frame of video when the point in
`the audio stream corresponding to the timing informa-
`tion of the received video frame is audibly reproduced
`at the remote workstation. The synchronizer counts the
`amount of audio data received in the continuous audio
`stream and compares the count to the timing informa-
`tion sent along with the most recently received video
`frame to determine when to display the frame.
`In general, in another aspect, the invention features a
`multimedia computer workstation, such as a RISC
`workstation or IBM PC, having video teleconferencing
`capabilities. The multimedia workstation of this inven-
`tion includes a network interface for establishing a vari-
`able bandwidth digital communications channel across
`a digital data network with another multimedia work-
`station. A video source provides a frame of digitized
`video data, and an audio source provides digitized audio
`data associated with the frame of video data. A data
`transmitter transmits the audio and video data through
`the network interface across the variable bandwidth
`digital communications channel to another workstation.
`A receiver receives audio and video data through the
`network interface across the variable bandwidth digital
`communications channel from another workstation.
`The workstation also includes means for displaying the
`received video data on the workstation display, and
`means for audibly reproducing the received audio data.
`In preferred embodiments, the video source includes
`a video camera, a video tape recorder, and/or a video
`laser disk player providing frames of analog video. A
`video frame grabber captures, digitizes, and stores each
`frame of analog video. The video source also includes
`digital video data stored in a file accessible by the work-
`station. A video compressor may compress the video
`data using JPEG or MPEG compression. The audio
`source includes a microphone for live audio, or pre-
`recorded audio corresponding to frames of pre-
`recorded video, from for instance a video tape recorder
`or laser disk. An audio digitizer digitizes and stores the
`audio using mu-law compression. The audio source also
`includes digital audio data stored in a file, preferably
`along with digital video data, accessible by the worksta-
`tion.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`The foregoing and other objects, features and advan-
`tages of the invention will be apparent from the follow-
`ing more particular description of preferred embodi-
`ments of the invention, as illustrated in the accompany-
`ing drawings in which like reference characters refer to
`the same parts throughout the different views. The
`drawings are not necessarily to scale, emphasis instead
`being placed upon illustrating the principles of the in-
`vention.
`
`FIG. 1 is a pictorial representation of the distributed
`computer system featuring multimedia workstations
`having video teleconferencing capabilities of this inven-
`tion.
`
`5,375,068
`
`4
`
`FIG. 2 is a block diagram showing a preferred em-
`bodiment of the multimedia workstation for implement-
`ing the video teleconferencing features of the distribu-
`tor computer system of FIG. 1.
`FIG. 3 is a block diagram showing the software envi-
`ronment for implementing a preferred embodiment of a
`DECspin video teleconferencing application for imple-
`menting the video teleconferencing features of the sys-
`tem of FIG. 1.
`I
`
`FIG. 4 is a block diagram showing the one-way con-
`nections established between two workstations to im-
`plement a two-way video teleconference.
`FIGS. 5a-5c are block diagrams illustrating the multi-
`ple one-way video teleconferencing connections estab-
`lished between three workstations joined in the same
`teleconference.
`
`FIG. 6 is a flowchart illustrating the flow of video
`during a video teleconference.
`FIG. 7 is a flowchart illustrating the teleconferencing
`protocol of this invention for establishing a video tele-
`conference connection between two workstations.
`FIG. 8 illustrates the format of the START message
`of the teleconferencing protocol of this invention.
`FIG. 9 illustrates the format of the OK message of the
`teleconferencing protocol of this invention.
`FIG. 10 illustrates the format of the START-
`HEADER message of the video teleconferencing pro-
`tocol of this invention.
`
`FIG. 11 illustrates the format of the QUIT message of
`the teleconferencing protocol of this invention.
`FIG. 12 illustrates the format of the ANSWERMA—
`CHINE message of the teleconferencing protocol of
`this invention.
`FIG. 13 illustrates the format of the CONTROL
`message of the video teleconferencing protocol of this
`invention.
`
`FIG. 14 is a flowchart showing the audio and video
`data transfer sequence of the video teleconferencing
`protocol of this invention.
`FIG. 15 shows the format of the AUDIOHEADER
`message of the video teleconferencing protocol of this
`invention,
`FIG. 16 shows the format of the VIDEOHEADER
`message of the video teleconferencing protocol of this
`invention,
`FIG. 17 shows a timing diagram of the audio and
`video data synchronization of the video teleconferenc-
`ing protocol of this invention.
`FIG. 18 shows the top level graphical user interface
`window for controlling a video teleconferencing ses-
`sion of this invention.
`
`FIG. 19 shows the graphical user interface window
`for displaying video data received from another work-
`station during a video teleconferencing session of this
`invention.
`
`FIG. 20 shows a second level graphical user call list
`interface window for establishing the video telecon-
`ferencing connections to other workstations of this
`invention.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`45
`
`50
`
`55
`
`FIG. 21 shows a second level control graphical user
`interface window for adjusting the parameters for a
`video teleconference of this invention.
`
`65
`
`FIG. 22 shows a second level monitor graphical user
`interface window for monitoring the parameters of a
`video teleconference of this invention.
`FIG. 23 shows a second level documentation graphi-
`cal user interface window for obtaining information
`
`Cisco - Exhibit 1020 - Page 32
`
`Cisco - Exhibit 1020 - Page 32
`
`
`
`5
`about the operation and features of a video teleconfer-
`ence of this invention.
`
`5,375,068
`
`FIG. 24 shows a third level documentation graphical
`user interface window for obtaining information about a
`t0pic selected from the second level user interface win-
`dow of FIG. 23.
`FIG. 25 shows a graphical user interface window of
`a ring box for announcing a video teleconference call to
`another workstation to establish a video teleconference
`of this invention.
`FIGS. 26(a)—26(k) show the dislay screens of three
`Workstations participating in a three-way video tele-
`conference of this invention.
`
`DESCRIPTION OF THE PREFERRED
`EMBODIMENT
`
`FIG. 1 shows a distributed computer system 10, hav-
`ing a plurality of multimedia workstations 12 connected
`by a high speed digital network 14, which features
`n—way video teleconferencing of this invention. Each of
`the multimedia workstations 12 is capable of producing
`live audio and video data for transmission across the
`network to another multimedia workstation. Further,
`each of the multimedia workstations is capable of soni-
`cally reproducing the audio data and visually reproduc-
`ing the video data received from another workstation.
`Two or more of the networked workstations can
`participate in an n—way video teleconference using the
`teleconferencing protocol of this invention. The tele-
`conferencing protocol of this invention allows real time
`synchronized audio and video transmission across the
`network without the use of a fixed bandwidth or dedi-
`cated time slot transmission medium. Rather, this inven-
`tion provides high performance video teleconferencing
`features using stande digital network transport level
`protocols such as Internet TCP/IP and UDP/IP, or
`DECnet TM. The physical network link 14 should be a
`high speed FDDI (Fiber Distributed Data Interface)
`fiber optic link running at 100 MB/Sec. for maximum
`performance (about 15 uncompressed black and white
`frames per second), but can also be virtually any type of
`high quality digital network link such as an Ether-
`net TM_. In the case of the FDDI network link, perfor-
`mance is typically limited by the workstation hardware
`and software, rather than by the throughput of the
`network. In addition, wide area networking (WAN)
`technologies, such as T1 and T3 digital carriers, can be
`used with the video teleconferencing protocol of this
`invention. These WAN technologies can be expected to
`achieve a video frame rate of at least about 12 Hz, as-
`suming black and white video images compressed with
`JPEG, MPEG, or another video compression tech-
`nique. The features of one preferred embodiment of this
`invention are commercially embodied in the DEC-
`spin TM product available from Digital Equipment
`Corporation, Maynard, Mass, the assignee of this pa-
`tent application.
`FIG. 2 shows a preferred embodiment of the multi-
`media workstation 12 for implementing the video tele-
`conferencing protocol of this invention. Workstation 12
`includes a high performance processor 20 connected to
`a large working memory 22 having 24 megabytes or
`more capacity, and a large hard drive having 1 giga-
`bytes or more capacity. A high performance backplane
`channel 26 connects the processor, memory, and hard
`disk to other devices in the workstation.
`
`The workstation is coupled to the digital network
`communications link 14 through a network controller
`
`5
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`65
`
`6
`28 connected between the network link 14 and the back-
`plane channel 26. The workstation is also coupled to a
`voice grade telephone line 29 through a modem 31
`connected between the backplane channel and the tele-
`phone line. Similarly, the workstation can be connected
`to other digital data conversation services, such as the
`ISDN digital telephone network.
`Multimedia workstation 12 includes a color video
`frame grabber 34 for capturing and digitizing frames of
`video from one or more analog video inputs 36. Video
`inputs 36 are each connected to a video source, such as
`a video camera 38 providing live analog video signals,
`or an auxiliary video storage device 40, such as a VCR
`or video laser disk player providing stored analog video
`signals. The analog video signals may be of any stan-
`dard types such as NTSC, SECAM, or PAL.
`The multimedia workstation 12 also includes a video
`buffer 35 which stores a frame of full color video graph-
`ics and characters generated by the workstation for
`display on a 1280x1024 pixel color monitor 30. The
`video buffer 35 (and associated buffer controller) is
`connected to the backplane channel 26 for receiving
`video data from the processor 20. The video buffer is
`also connected to a video display subsystem 36 which
`converts the stored video frame data into analog signals
`for driving the color monitor 30.
`The video frame grabber 34 stores its digitized video
`data directly into a predetermined area of the video
`buffer 35. Thus, the digitized video input to the work-
`station by the frame grabber appears directly in a prede-
`termined area on the monitor 30, without having to pass
`through processor 20 or main memory 22. Further,
`processor 20 can read back the captured video frame
`data from the video buffer, store the data in main mem-
`cry 22, and further process the video data according to
`the video teleconferencing protocol described herein.
`Frame grabber 34 digitizes and stores each frame of
`video from an analog video source and can deliver up to
`30 frames per second of digitized 640x480 true color
`(24 bits) of NTSC/SECAM/PAL video into video
`frame buffer 35. A dedicated hardware video compres-
`sion subsystem 37 can also be connected to the back-
`plane channel 26 to provide high performance video
`compression of the digitized video data.
`The audio features of the multimedia workstation 12
`are implemented using an audio controller 42 connected
`to the backplane channel 26 for interfacing audio signals
`into the workstation and reproducing audio signals out
`of the workstation. An audio distribution box 44 is con-
`nected to the audio controller for directing audio to and
`from audio devices such as a microphone 46, a head-
`phone 48, and/or a telephone receiver 50. Auxiliary
`audio devices such as a recording device, a CD player,
`or amplified