United States Patent 1191
`
`[11] Patent Number:
`
`5,375,068
`
`[45] Date of Patent: Dec. 20, 1994
`Palmer et al.
`
`
`
`US005375068A
`
`[54] VIDEO TELECONFERENCING FOR
`NETWORKED WORKSTATIONS
`
`[75]
`
`Inventors: Ricky S. Palmer; Larry G. Palmer,
`both of Nashua, N.H.
`
`[73] Assignee: Digital Equipment Corporation,
`Maynard, Mass.
`
`an audio data stream tohthe remote workstation such .
`that the audiopdata ca_r1_be reconstructed into a cont_inu-
`ous audio signal. A video transmitter sends video data H
`‘so that each frame of video data to be sent is inserted .
`into the audio data stream without affecting the‘con.ti-
`nuity of the reconstructed audio signal at the remote
`workstation.
`
`.
`
`46 Claims, 28 Drawing Sheets
`
`OTHER PUBLICATIONS
`
`Palmer and Palmer, “Desktop Meeting”, LAN Maga-
`zine, 6(l1):l11—12l (Nov. 1991).
`D. Comer, “Internetworking with TCP/IP, vol. I: Prin- V
`ciples, Protocols, and Architecture”, 2nd Edition, pp.
`1-8, 337-346, 505 (Prentice Hall: Engelwood Cliffs,
`NJ. 1991).
`
`-
`
`Primary Examiner-Emanuel T. Voeltz
`Assistant Examiner—Thomas Peeso
`Attomey, Agent, or Firm—Hamilton, Brook, Smith &
`Reynolds
`
`[57]
`
`ABSTRACI‘
`
`A video teleconferencing method and apparatus for
`computer workstations connected by a digital data net-
`work includes a transmission source portion for a local
`workstation to send audio and video teleconference
`data across the network to one or more remote worksta-
`tions, and, a receiver for the local workstation to re-
`ceive audio and video teleconference data back from
`the remote workstations. The local workstation sends
`teleconference data to each of the remote workstations ‘
`over a variable bandwidth digital data connection, and
`each of the remote workstations returns teleconference
`data back to the local workstation over another variable
`bandwidth digital data connection. The transmission
`source portion includes a master software process exe-
`cuting on the local workstation, and the receiver in-
`cludes a slave software process executing the remote
`
`[21] Appl. No.: 893,074
`
`[22] Filed:
`
`Jun. 3, 1992
`
`Int. Cl.5 ............................................. G06K 15/00
`[51]
`[52] U.S. Cl. ..................................... .. 364/514; 370/62
`[58] Field of Search .................. .. 364/514; 379/96, 94,
`379/202; 370/60, 62; 395/152
`
`[56]
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`...... 178/6.8
`6/1971 Schoeffler
`3,584,142
`.. 179/2 DP
`6/1983 Artom .........
`4,387,271
`..... .. 358/85
`....
`4,516,156 5/1985 Fabris et al.
`364/200
`..
`4,525,779 6/1985 Davids et al.
`4,574,374 3/1986 Scordo ...................... 370/62
`4,645,872 2/1987 Pressman et al.
`..
`....... 379/54
`4,650,929
`3/1987 Boerger et al. ............... 358/86
`4,653,090 3/1987 Hayden ......
`379/204
`.... ..
`4,679,191
`7/1987 Nelsoy et al.
`370/84
`4,686,698
`8/1987 Tompkins et al.
`.
`379/53
`4,710,917 12/1987 Tompkins et al. .
`370/62
`4,734,765
`3/1988 Okada et al. .......
`358/102
`4,748,618
`5/1988 Brown et al.
`..
`370/94
`4,827,339
`5/1989 Wada et a1.
`........
`358/136
`4,847,829 7/1989 Tompkins et al. .
`’370/62
`4,849,811
`7/1989 Kleinennan
`358/133
`4,882,743 11/1989 Mahmoud ......
`379/53
`4,888,795 12/1989 Ando et al.
`379/53
`.......
`4,893,326
`1/1990 Duran et al.
`379/53
`4,897,866 1/1990 Majmudar et al.
`379/94
`4,905,231
`2/1990 - Leung et a1.
`......
`. 37o/94.1
`4,91,8,7l‘8____4il99_(_J_ ___E_mmons_7et al.
`........._._............ 379/53
`
`
`
`(List continued on next page.)
`
`workstation. The master process "of a local workstation ’
`causes execution of a slave process on a remote work-
`station for receiving video teleconference data from the
`local workstation. An audio data transmitter for sends
`
`
`
`Headphones
`with
`Microphone
`
`video
`Camera
`
`
`Headphones
`with
`Microphone 38
`
`
`
`To other workstations
`
`Samsung - Exhibit 1020-- Page 1
`
`Samsung - Exhibit 1020 - Page 1
`
`

`
`5,375,068
`Page 2
`
` ;
`
`8/1991 Flohrer
`5,042,006
`
`8/1991 Lee etal.
`379/54
`5,042,062
`9/1991 Hashimoto ..
`5,046,079
`
`5,046,080 9/1991
`5,056,136 10/1991
`5,062,13610/1991
`5,072,44212/1991
`5,079,627
`1/1992
`5,099,510 3/1992
`-5-1031444 4/1992
`"$111,409
`5/1992
`5»113»431
`5/1992
`11951036 3/1993
`53001989 4/1993
`
`_
`PATENT DOCUMENTS
`_
`5/1990 011111 eta]. ..........;............... 358/138
`4,924,311
`
`6/1990 Emmonsetal..
`...... 379/53
`4,932,047
`6/1990 Appeletal. .. .. .
`...... 379/53
`4,935,953
`4,942,540 7/1990 Black eta]. .................
`364/514
`4,943,994 7/1990 Ohtsuka eta]. ................. .. 379/53
`4,953,159
`8/1990 Hayden etal. ..
`370/62
`4,953,196 8/1990 Ishikawa etal..
`.. 379/53
`4,962,521 10/1990 Komatsu eta1..
`...... 379/53
`4,965,81910/1990 Kannes ................... 379/53
`4,995,071
`2/1991 Weber et a1.‘....
`.. .... 379/53
`5,003,532
`3/1991 Ashida etal.
`.... .. 370/62
`5,034,916
`7/1991 Ordish ................................. 364/900
`
`
`
`Samsung - Exhibit 1020 - Page 2
`
`Samsung - Exhibit 1020 - Page 2
`
`

`
`U.S. Patent
`
`Dec. 20, 1994
`
`Sheet 1 of 28
`
`5,375,068
`
`IO
`
`video /Camera
`
`30
`
`38
`
`Headphones
`with
`Microphone
`
`
`
`Headphones
`with
`Microphone 38
`
`Video
`
`To other workstations
`
`Samsung - Exhibit 1020 - Page 3
`
`Samsung - Exhibit 1020 - Page 3
`
`

`
`3..>3¢m.o
`
`
`
`On.OwD_>
`
`U
`
`um_on8.\\.9S.N.weonmc«No.xom~_.
`
`
`%.84tmomnom<mm.o_>__»3I_n00_89>mX
`
`zmmmmmmam_mmmtammzqmu_89>Aan%onj89>_mm<;om<:N_m.BDs_m._.m>mm:m
`mmmméo_M,
`
`_.6528
`
`
`
`Quauomoon_iIII1IIIII.._.\.omum:.m.._P..._
`.
`
`Soats...
`
`
`.m_n8E9.ooomO2.%m_o:coco
`cumON..¢__OL#COOWmu._.._oEzooBoom”.mR.xmoafizomomémm
`
`
`5,x2:So....Nw_n_m7M3,.96am:Em>m_»:mm2
`s OD.l
`
`Samsung - Exhibit 1020 - Page 4
`
`
`

`
`U.S. Patent
`
`Dec. 20, 1994
`
`Sheet 3 of 28
`
`5,375,068
`
`66
`
`70
`ea
`.xn}ecfgc/'Igs(ilr<‘>€$<‘
`p'p:o ‘E2/><".IIIXv using
`82
`TCPIIPbra}@
`.
`_ mpusing
`TCP/IP(IocaI)
`
`mp usin
`9
`or
`
`TCPIIP
`
`84
`
`DEC
`
`1‘
`
`ne
`
`76
`TCPIIP (local)
`
`system calls
`
`78 @
`
`74
`
`I20
`
`lab
`
`
`
`'00
`
`Il2
`
`I4
`
`I02
`
`IIO
`
`'
`
`Network
`
`IQ4
`
`I06
`
`signal/execv
`
`I08
`
`
`
`
`Machine A
`
`Machine 8
`
`FIG. 4
`
`Samsung - Exhibit 1020 - Page 5
`
`Samsung - Exhibit 1020 - Page 5
`
`

`
`U.S. Patent
`
`Dec. 20, 1994
`
`Sheet 4 of 28
`
`5,375,068
`
`IIO
`
`A calls 8, B answers
`
`FIG. 5A
`
`'20
`
`—
`
`I02
`
`l2b
`
`I H I-—r’ H
`"20 j\\‘ ‘'0 A cells C, C answers
`H00
`'2C
`
`FIG. 5B
`
`I20
`
`lzb
`
`Eif-
`II2a < |o2b
`
`IIOO
`
`IIOD
`
`B cells C or A joins B and C
`
`FIG. 5C
`
`Samsung - Exhibit 1020 - Page 6
`
`Samsung - Exhibit 1020 - Page 6
`
`

`
`U.S. Patent
`
`Dec. 20, 1994
`
`Sheet 5 of 28
`
`5,375,068
`
`ANALOG VIDEO
`NTSC/SECAM/PAL
`
`200
`
`DIGITIZE VIDEO IN
`FRAME BUFFER
`
`202
`
`VIDEO DATA IN
`APPLICATION BUFFER ‘
`
`204
`
`VIDEO DATA IN
`NETWORK BUFFER
`
`206
`
`I
`
`VIDEO DATA
`RAVERSES NETWORK
`
`208
`
`VIDEO DATA IN
`NETWORK BUFFER
`
`’
`
`2IO
`
`VIDEO DATA IN
`APPLICATION BUFFER '
`
`2I2
`
`DIGITIZED VIDEO
`IN FRAME BUFFER
`
`_ 2|4
`
`DIGITIZED VIDEO
`
`DISPLAYED
`
`2I6
`
`FIG. 6
`
`Samsung - Exhibit 1020 - Page 7
`
`Samsung - Exhibit 1020 - Page 7
`
`

`
`U.S. Patent
`
`Dec. 20, 1994
`
`Sheet 6 of 28
`
`5,375,068
`
`LOCAL
`WORKSTATION A
`
`REMOTE
`WORKSTATION B
`
`
`
`INVOKE LOCAL
`DECspin
`APPLICATION
`
` 300
`
`NETWORK
`
`.332_______ _ _
`
`CREATE
`
`PORT
`
`304
`
`
`
`
`
`
`saw
`
`STARTHEADER
`
`u
`
`u
`
`CHE.-IlCK"I'-‘OR
`OK
`
`'
`
`seem SENDING
`AUDIO/VIDEO
`
`DATA
`
`CHE"CK fore
`ox
`
`_3_'_“_ __ __ _
`
`mvoxe LOCAL
`DECspind
`APPLICATION
`
`303
`
`3.0
`
`3.2
`
`3|8
`
`
`
`20
`NO
`YES
`ENTER
`
`CONFERENCE?
`
`
`
`
`SEND "QT"
`OR "ANSWER
`
`-MACHINE"
`
`I I
`
`I
`I. .._ __ ._ _
`
`I_ _ _ __ _
`
`3'6
`"" —‘ ’ —]
`
`I I
`
`_
`
`I
`
`I
`I
`324 I
`I
`
`I I L
`
`305
`""“"“ ““““
`
`
`INVOKE LOCAL
`DEC spin
`
`APPLICATION
`
`FIG. 7
`
`Samsung - Exhibit 1020 - Page 8
`
`Samsung - Exhibit 1020 - Page 8
`
`

`
`U.S. Patent
`
`Dec. 20, 1994
`
`Sheet 7 of 28
`
`5,375,068
`
`START
`
`OK
`
`OIOIOIOO = 0x54
`
`|
`
`OIOIOOII = 0x53
`
`Total length = 2 bytes
`
`ouoonou = 0x43
`
`|
`
`OlO0llll = Ox4F
`
`Total length = 2 bytes
`
`STARTHEADER
`
`5 reserved longwords (32 bits each)
`
`framerate desired (Iongword)
`
`DTP flags
`
`2 reserved longwords
`
`406
`
`4'2
`
`total number of frames
`
`-
`
`403
`
`l reserved Iongword
`
`video width
`
`video height
`
`bits per pixel (8 or 24)
`
`35 reserved longwords
`
`400
`
`402
`
`404
`
`78 longwords (used as login/ user handle)
`
`4‘°
`
`FIG. IO
`
`Samsung - Exhibit 1020 - Page 9
`
`Samsung - Exhibit 1020 - Page 9
`
`

`
`U.S. Patent
`
`Dec. 20, 1994
`
`Sheet 8 of 28
`
`5,375,068
`
`QUIT
`
`OIOIOIOO = 0x54
`
`|
`
`OlOlOOOl = Ox5l
`
`Total length = Zbytes
`
`FIG. ll
`
`ANSWERMACHINE
`
`OIOOI IOI = 0x4D
`
`|
`
`OIOOOOOI = 0x4:
`
`Total length = 2 bytes
`
`FIG. I2
`
`CONTROL
`
`0x54
`
`|
`
`0x43
`
`45°
`
`452
`
`454
`
`455
`
`which
`
`length
`
`flags
`
`5 reserved longwords
`
`Total length is 34 bytes
`
`FIG. I3
`
`Samsung - Exhibit 1020 - Page 10
`
`Samsung - Exhibit 1020 - Page 10
`
`

`
`U.S. Patent
`
`Dec.20, 1994
`
`Sheet 9 of 28
`
`5,375,068
`
`
`
`NEXT vuoeo
`FRAME
`AVAILABLE
`
`350
`
`352
`
`
`
` SEND
`CONTENTS OF
`
`AUDIO BUFFER
`
`360
`
`SEND VIDEO
`FRAME
`
`354
`
`"OK"
`
`.
`
`CHECK FOR
`
`II OKII
`
`TIME OUT
`
`
`BUFFER TO
`LAST |Al2 SEC
`
`
`
`
`
`<2 FRAME
`
`"OK"
`>2 FRAME
`352»
`
`TIMES
`IIOKII
`
`
`
`
`
`SEND
`CONTENTS OF
`AUDIO BUFFER
`
`FIG. I4
`
`Samsung - Exhibit 1020 - Page 11
`
`Samsung - Exhibit 1020 - Page 11
`
`

`
`U.S. Patent
`
`Dec. 20, 1994
`
`Sheet 10 of 28
`
`5,375,068
`
`AUDIOHEADER
`
`0x55
`
`|
`
`OX4!
`
`4'4
`
`length
`
`Audio Flags
`
`6 reserved longwords
`
` FIG. I5
`
`VIDEOHEADER
`
`0x48
`
`l
`
`Ox4F
`
`42°
`
`Video width
`
`video Height
`
`Control Flags
`
`Timing Information
`
`4 reserved longwords
`
`Total Length is 34 bytes
`
`FIG. I6
`
`4'6
`
`4'8
`
`422
`
`424
`
`425
`
`423
`
`Samsung - Exhibit 1020 - Page 12
`
`Samsung - Exhibit 1020 - Page 12
`
`

`
`mPS”U
`
`m
`
`hS
`
`
`
`:fi._$5om"6zocomjoozamm
`
`_ms_<E
`
`
`
`NNms_<Enomzo_.8w._._oozamm.ms_<E
`
`.._o
`
`
`
`zo_mm_s_mz<Ezamm
`
`
`m964Nxo<:64
`M_Tl1||._..|I||l.rl|lu._
`
`:cwms_$.._...mm_2<m.._Nms_<E_N245
`
`.._.Nus_<ENu m5EN.:o.
`mTms.»
`
`S
`
`8W.6N0.,n..5M7m39m5NWN.0....
`
`Samsung - Exhibit 1020 - Page 13
`
`
`

`
`U.S. Patent
`
`Dec. 20, 1994
`
`Sheet 12 of 28
`
`5,375,068
`
`524
`
`Samsung - Exhibit 1020 - Page 14
`
`Samsung - Exhibit 1020 - Page 14
`
`

`
`U.S. Patent
`
`Dec. 20, 1994
`
`Sheet 13 of 28
`
`5,375,068
`
`858
`
`FIG. 19
`
`Samsung - Exhibit 1020 - Page 15
`
`Samsung - Exhibit 1020 - Page 15
`
`

`
`U.S. Patent T
`
`Dec. 20, 1994
`
`Sheet 14 of 28
`
`5,375,068
`
`600
`
`@540 624
`
`620 622
`
`Samsung - Exhibit 1020 - Page 16
`
`Samsung - Exhibit 1020 - Page 16
`
`

`
`U.S. Patent
`
`Dec. 20, 1994
`
`' Sheet 15 of 28
`
`5,375,068
`
`650
`
`Samsung - Exhibit 1020 - Page 17
`
`Samsung - Exhibit 1020 - Page 17
`
`

`
`U.S. Patent
`
`Dec. 20, 1994
`
`Sheet 16 of 28
`
`5,375,068
`
`700
`
`/
`
`720
`
`FIG. 22
`
`Samsung - Exhibit 1020 - Page 18
`
`Samsung - Exhibit 1020 - Page 18
`
`

`
`P3U
`
`tH
`
`10,2
`
`moom
`
`M5324
`
`:o_8:.uo:EM:o_.6.:mcoEmo
`
`foioz
`
`8wasmm»mm»mm»sm.Nmw3o._m%.ccmmo_w
`
`W
`
`86m0.._5m7.1.39WsE.o_h_mmo:
`
`Samsung - Exhibit 1020 - Page 19
`
`
`

`
`U.S. Patent
`
`Dec.20, 1994
`
`»
`
`Sheet 18 of 28
`
`5,375,068
`
`°
`o
`
`k\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\‘-
`
`O
`
`O
`
`V2
`
`D
`
`I ~\\\\\\\\\\\\\\\\\\\\\\\\\\%\\\\\\\\Vt10
`‘%
`
`anaaaaa
`aanaaau
`
`El
`
`El
`
`ucxucznucsnu
`
`newsman n
`aunaaaa :1
`
`D
`
`Samsung - Exhibit 1020 - Page 20
`
`WorkstationA
`
`Samsung - Exhibit 1020 - Page 20
`
`

`
`U.S. Patent
`
`Dec. 20, 1994
`
`Sheet 199: 28
`
`5,375,068
`
`:5
`\.z
`(.0
`N.
`9
`LI.
`
`.0
`
`0r
`
`0
`
`Samsung - Exhibit 1020 - Page 21
`
`°°
`5
`2
`+5-’.
`3;
`;
`
`Samsung - Exhibit 1020 - Page 21
`
`

`
`tHEaP
`
`4W10..2mD
`
`R.m«nu.aMS
`
`%_m:o:2$:o>>
`
`7m3,M5m
`8m6wWEmuo_.._non
`
`S
`
`Samsung - Exhibit 1020 - Page 22
`
`

`
`U.S. Patent
`
`Dec. 20, 1994_
`
`Sheet 21 of 23
`
`5,375,068
`
`m WorkstationA
`
`O
`
`0 O
`
`:..'J
`
`256xI84 »\\\\\\\\\\\\\\\\\\\\\\\\\\\\\%VW§1
`
`O
`
`
`
`sonFIG.26(d)
`
`nfleaaamna n F:1
`gfianannaa
`
`El
`
`Samsung - Exhibit 1020 - Page 23
`
`Samsung - Exhibit 1020 - Page 23
`
`

`
`U.S. Patent
`
`Dec.20, 1994
`
`Sheet 22 of 28
`
`5,375,068
`
` WorkstationC
`
`socFIG.26(e)
`
`Samsung - Exhibit 1020 - Page 24
`
`Samsung - Exhibit 1020 - Page 24
`
`

`
`f.H8f.aP«MU
`
`Dec. 20, 1994
`
`Sheet 23 of 28
`
`5,375,068
`
`0co_S,§_._o2,
`
`38.0_:._
`
`com
`
`—
`
`QnUSmaS
`
`..mhXE
`
`0201H.
`
`—
`
`52e9aP
`
`Samsung - Exhibit 1020 - Page 25
`
`

`
`U.S. Patent
`
`Dec. 20, 1994_
`
`Sheet 24 of 28
`
`5,375,068
`
`WorkstationA
`
`O
`
`o
`o
`
`A\\\\~\\\\\\\\\\\\\V\\\\\\\\\\\\\}I
`==
`
`OO—
`O
`at \\\\\\\\\\\\N\\\\\\%$$\%\\W_
`N
`
`Qflaanaaaa
`gfiannaaaa
`
`Samsung - Exhibit 1020 - Page 26
`
`Samsung - Exhibit 1020 - Page 26
`
`

`
`U.S. Patent
`
`Dec. 20, 1994
`
`Sheet 25 of 28
`
`5,375,068
`
`
`
`
`
`EEEEHEEEEEEEEEH
`
`m8:29:25
`
`..o;m:<
`
`Samsung - Exhibit 1020 - Page 27
`
`Samsung - Exhibit 1020 - Page 27
`
`

`
`S”U
`
`f.D.maP.
`
`Dec. 20, 1994
`
`Sheet 26 of 28
`
`5,375,068
`
`U_5_.2e:§>
`
`A:8.o_.._com
`
`E3324
`
`Samsung - Exhibit 1020 - Page 28
`
`Samsung - Exhibit 1020 - Page 28
`
`

`
`U.S. Patent
`
`Dec. 20, 1994
`
`Sheet 27 of 28
`
`5,375,068
`
`n4
`
`..s
`
`mco:2mfo\<,
`
`Samsung - Exhibit 1020 - Page 29
`
`Samsung - Exhibit 1020 - Page 29
`
`

`
`mRaP&U
`
`Dec. 20, 1994
`
`Sheet 28 of 28
`
`5,375,068
`
`E
`
`08:_2e:2s
`
`3mm.0_:._
`
`com
`
`Samsung - Exhibit 1020 - Page 30
`
`Samsung - Exhibit 1020 - Page 30
`
`

`
`1
`
`VIDEO TELECONFERENCING FOR
`NETWORKED WORKSTA'I‘IONS
`
`5,375,068
`
`BACKGROUND OF THE INVENTION
`
`Video communications has evolved over the years
`from a simple video telephone concept to a sophisti-
`cated network for allowing multiple users to participate
`in a video teleconference. For full featured video tele-
`conferencing, users require both an audio communica-
`tions path and a real time visual communication path
`synchronized to the audio path. Furthermore, it is desir-
`able to support full color video and telephone quality
`audio. Video teleconferencing capabilities are limited
`mainly by the bandwidth of the transmission medium
`connecting the teleconferencing terminals.
`Many computer workstations used in the office or
`laboratory environment today are connected with other
`workstations, file servers, or other resources over high-
`speed local area networks. Local area networks, in turn,
`are often connected together through high-speed speed
`gateways which connect workstations which may be
`distributed over a wide geographic area. Network wide
`protocols allow workstations to exchange packets of
`data at high rates of speed and reliability. Fixed band-
`width digital and analog video channels have been com-
`bined with computer networks to implement some
`video teleconferencing features. These include high
`bandwidth CATV/FDM type analog channels and
`fixed allocation TDM data channels for the video data.
`
`SUMMARY OF THE INVENTION
`
`Workstations today have obtained unprecedented
`computational power and utility. The powerful RISC
`type CPUs and fast, high resolution graphical displays
`have made possible multimedia workstations which
`integrate live audio and video into the programming
`environment. Graphical User Interface operating sys-
`tems (GUI) have allowed effective integration of audio
`and video into application programming.
`The present invention provides n-way video telecon-
`ferencing among networked computer workstations
`using the existing variable bandwidth digital data net-
`work for transferring synchronized audio and video
`teleconferencing data between the workstations. The
`teleconferencing apparatus and protocol of this inven-
`tion provides high quality video teleconferencing with-
`out the need for a guaranteed wide bandwidth analog
`video channel or a fixed allocation digital video chan-
`nel. Rather, the invention uses standard non-allocated
`data packets typically found on local area networks to
`transfer the audio and video teleconferencing data.
`Thus, no guaranteed bandwidth is required to carry on
`a useful video teleconference. An continuous audio data
`stream model provides continuous audio signals at the
`expense of video data when necessary, which is desir-
`able since the ear is more sensitive to a break in the
`audio data than the eye is to the loss of a frame of video
`data. A “push” data model provides a secure system by
`preventing remote workstations from activating an-
`other workstation’s video teleconferencing functions.
`In general, in one aspect, the invention features a
`video teleconferencing method and apparatus for com-
`puter workstations connected by a digital data network.
`The computer workstations include a transmission
`source means for a local workstation to send audio and
`video teleconference data across the network to one or
`more remote workstations, and, a receiver for the local
`
`5
`
`l0
`
`15
`
`20
`
`25
`
`30
`
`35
`
`45
`
`50
`
`55
`
`65
`
`2
`workstation to receive audio and video teleconference
`data back from the remote workstations. The local
`workstation sends teleconference data to each of the
`remote workstations over a variable bandwidth digital
`data connection, and each of the remote workstations
`returns teleconference data back to the local worksta-
`tion over another variable bandwidth digital data con-
`nection. The variable bandwidth digital data connec-
`tions include the data packet oriented data channels
`associated with, for example, FDDI, DECnet, and
`Ethernet local area networks. Furthermore, a wide area
`digital network, such as ISDN, can also be used with
`the video teleconferencing apparatus and method of this
`invention.
`
`In preferred embodiments, the transmission source
`means includes a master software process executing on
`the local workstation, and the receiver includes a slave
`software process executing the remote workstation.
`The master software process formats and sends video
`teleconference data to the slave process. The slave pro-
`cess receives and reconstructs the audio and video tele-
`conference data for audible and visual reproduction,
`respectively. The video data is presented as an image on
`the display of the receiving workstation, while the
`audio data is sent to either amplified speakers or head-
`phones. In other preferred embodiments, the master
`process of a local workstation causes execution of a
`slave process on a remote workstation for receiving
`video teleconference data from the local workstation.
`The slave process running on the remote workstation in
`turn causes execution of a master process on the remote
`workstation for sending video teleconference data back
`to the local workstation. The master process of the
`remote workstation in turn causes execution of a slave
`process on the local workstation for receiving the video
`teleconference data sent by the master process of the
`remote workstation. The local workstation executes a
`slave process for each master process on a remote work-
`station sending video teleconference data to the local
`workstation.
`
`In yet other preferred embodiments, the transmission
`source includes an audio data transmitter for sending an
`audio data stream to the remote workstation such that
`the audio data can be reconstructed into a continuous
`audio signal. The transmission source also includes a
`video transmitter for sending video data to the remote
`workstation so that each frame of video data to be sent
`is inserted into the audio data stream without affecting
`the continuity of the reconstructed audio signal at the
`remote workstation.
`In yet other preferred embodiments the video trans-
`mitter precludes a frame of video data from being sent
`to the remote workstation if a system overload exists.
`The audio transmitter sends the audio data stream cor-
`responding to the precluded video frame to the remote
`workstation to prevent loss of continuity of the audio
`signal during a system overload. In other preferred
`embodiments the video transmitter precludes a frame of
`video data from being sent to the remote workstation in
`response to a system failure condition. The audio trans-
`mitter accumulates audio data for a predetermined time
`interval during the system failure condition, and trans-
`mits the accumulated audio data stream to the remote
`workstation once the failure has been corrected. For
`instance, the audio transmitter may continually accumu-
`late the last Q second of audio data while the failure
`exists, trimming any audio data older than § second.
`
`Samsung - Exhibit 1020 - Page 31
`
`Samsung - Exhibit 1020 - Page 31
`
`

`
`3
`The last é second of audio data accumulated before the
`failure’s correction is sent to the remote workstation as
`soon as the failure is corrected. The most recently avail-
`able frame of video is then also sent.
`In still other preferred embodiments, timing informa-
`tion is attached to each frame of video data sent to the
`remote workstation. The timing information indicates a
`point in the continuous audio data stream which corre-
`sponds in time to the frame of video data. The receiver
`of the remote workstation includes a synchronizer for
`displaying a received frame of video when the point in
`the audio stream corresponding to the timing informa-
`tion of the received video frame is audibly reproduced
`at the remote workstation. The synchronizer counts the
`amount of audio data received in the continuous audio
`stream and compares the count to the timing informa-
`tion sent along with the most recently received video
`frame to determine when to display the frame.
`In general, in another aspect, the invention features a
`multimedia computer workstation, such as a RISC
`workstation or IBM PC, having video teleconferencing
`capabilities. The multimedia workstation of this inven-
`tion includes a network interface for establishing a vari-
`able bandwidth digital communications channel across
`a digital data network with another multimedia work-
`station. A video source provides a frame of digitized
`video data, and an audio source provides digitized audio
`data associated with the frame of video data. A data
`transmitter transmits the audio and video data through
`the network interface across the variable bandwidth
`digital communications channel to another workstation.
`A receiver receives audio and video data through the
`network interface across the variable bandwidth digital
`communications channel from another workstation.
`The workstation also includes means for displaying the
`received video data on the workstation display, and
`means for audibly reproducing the received audio data.
`In preferred embodiments, the video source includes
`a video camera, a video tape recorder, and/or a video
`laser disk player providing frames of analog video. A
`video frame grabber captures, digitizes, and stores each
`frame of analog video. The video source also includes
`digital video data stored in a file accessible by the work-
`station. A video compressor may compress the video
`data using JPEG or MPEG compression. The audio
`source includes a microphone for live audio, or pre-
`recorded audio corresponding to frames of pre-
`recorded video, from for instance a video tape recorder
`or laser disk. An audio digitizer digitizes and stores the
`audio using mu-law compression. The audio source also
`includes digital audio data stored in a file, preferably
`along with digital video data, accessible by the worksta-
`tion.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`The foregoing and other objects, features and advan-
`tages of the invention will be apparent from the follow-
`ing more particular description of preferred embodi-
`ments of the invention, as illustrated in the accompany-
`ing drawings in which like reference characters refer to
`the same parts throughout the different views. The
`drawings are not necessarily to scale, emphasis instead
`being placed upon illustrating the principles of the in-
`vention.
`
`FIG. 1 is a pictorial representation of the distributed
`computer system featuring multimedia workstations
`having video teleconferencing capabilities of this inven-
`tion.
`
`5,375,068
`
`4
`
`FIG. 2 is a block diagram showing a preferred em-
`bodiment of the multimedia workstation for implement-
`ing the video teleconferencing features of the distribu-
`tor computer system of FIG. 1.
`FIG. 3 is a block diagram showing the software envi-
`ronment for implementing a preferred embodiment of a
`DECspin video teleconferencing application for imple-
`menting the video teleconferencing features of the sys-
`tem of FIG. 1.
`'
`
`FIG. 4 is a block diagram showing the one-way con-
`nections established between two workstations to im-
`plement a two-way video teleconference.
`FIGS. 5a—5c are block diagrams illustrating the multi-
`ple one-way video teleconferencing connections estab-
`lished between three workstations joined in the same
`teleconference.
`
`FIG. 6 is a flowchart illustrating the flow of video
`during a video teleconference.
`FIG. 7 is a flowchart illustrating the teleconferencing
`protocol of this invention for establishing a video tele-
`conference connection between two workstations.
`FIG. 8 illustrates the format of the START message
`of the teleconferencing protocol of this invention.
`FIG. 9 illustrates the format of the OK message of the
`teleconferencing protocol of this invention.
`FIG. 10 illustrates the format of the START-
`HEADER message of the video teleconferencing pro-
`tocol of this invention.
`
`FIG. 11 illustrates the format of the QUIT message of
`the teleconferencing protocol of this invention.
`FIG. 12 illustrates the format of the ANSWERMA-
`CHINE message of the teleconferencing protocol of
`this invention.
`FIG. 13 illustrates the format of the CONTROL
`message of the video teleconferencing protocol of this
`invention.
`
`FIG. 14 is a flowchart showing the audio and video
`data transfer sequence of the video teleconferencing
`protocol of this invention.
`FIG. 15 shows the format of the AUDIOHEADER
`message of the video teleconferencing protocol of this
`invention,
`FIG. 16 shows the format of the VIDEOHEADER
`message of the video teleconferencing protocol of this
`invention,
`FIG. 17 shows a timing diagram of the audio and
`video data synchronization of the video teleconferenc-
`ing protocol of this invention.
`FIG. 18 shows the top level graphical user interface
`window for controlling a video teleconferencing ses-
`sion of this invention.
`
`FIG. 19 shows the graphical user interface window
`for displaying video data received from another work-
`station during a video teleconferencing session of this
`invention.
`
`FIG. 20 shows a second level graphical user call list
`interface window for establishing the video telecon-
`ferencing connections to other workstations of this
`invention.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`45
`
`50
`
`55
`
`FIG. 21 shows a second level control graphical user
`interface window for adjusting the parameters for a
`video teleconference of this invention.
`
`65
`
`FIG. 22 shows a second level monitor graphical user
`interface window for monitoring the parameters of a
`video teleconference of this invention.
`FIG. 23 shows a second level documentation graphi-
`cal user interface window for obtaining information
`
`Samsung - Exhibit 1020 - Page 32
`
`Samsung - Exhibit 1020 - Page 32
`
`

`
`5
`about the operation and features of a video teleconfer-
`ence of this invention.
`
`5,375,068
`
`FIG. 24 shows a third level documentation graphical
`user interface window for obtaining information about a
`topic selected from the second level user interface win-
`dow of FIG. 23.
`FIG. 25 shows a graphical user interface window of
`a ring box for announcing a video teleconference call to
`another workstation to establish a video teleconference
`of this invention.
`FIGS. 26(a)—26(k) show the dislay screens of three
`Workstations participating in a three-way video tele-
`conference of this invention.
`
`DESCRIPTION OF THE PREFERRED
`EMBODIMENT
`
`FIG. 1 shows a distributed computer system 10, hav-
`ing a plurality of multimedia workstations 12 connected
`by a high speed digital network 14, which features
`n-way video teleconferencing of this invention. Each of
`the multimedia workstations 12 is capable of producing
`live audio and video data for transmission across the
`network to another multimedia workstation. Further,
`each of the multimedia workstations is capable of soni-
`cally reproducing the audio data and visually reproduc-
`ing the video data received from another workstation.
`Two or more of the networked workstations can
`participate in an n-way video teleconference using the
`teleconferencing protocol of this invention. The tele-
`conferencing protocol of this invention allows real time
`synchronized audio and video transmission across the
`network without the use of a fixed bandwidth or dedi-
`cated time slot transmission medium. Rather, this inven-
`tion provides high performance video teleconferencing
`features using standard digital network transport level
`protocols such as Internet TCP/IP and UDP/IP, or
`DECnet TM. The physical network link 14 should be a
`high speed FDDI (Fiber Distributed Data Interface)
`fiber optic link rimning at 100 MB/Sec. for maximum
`performance (about 15 uncompressed black and white
`frames per second), but can also be virtually any type of
`high quality digital network link such as an Ether-
`net TM. In the case of the FDDI network link, perfor-
`mance is typically limited by the workstation hardware
`and software, rather than by the throughput of the
`network. In addition, wide area networking (WAN)
`technologies, such as T1 and T3 digital carriers, can be
`used with the video teleconferencing protocol of this
`invention. These WAN technologies can be expected to
`achieve a video frame rate of at least about 12 Hz, as-
`suming black and white video images compressed with
`JPEG, MPEG, or another video compression tech-
`nique. The features of one preferred embodiment of this
`invention are commercially embodied in the DEC-
`spin TM product available from Digital Equipment
`Corporation, Maynard, Mass., the assignee of this pa-
`tent application.
`FIG. 2 shows a preferred embodiment of the multi-
`media workstation 12 for implementing the video tele-
`conferencing protocol of this invention. Workstation 12
`includes a high performance processor 20 connected to
`a large working memory 22 having 24 megabytes or
`more capacity, and a large hard drive having 1 giga-
`bytes or more capacity. A high performance backplane
`channel 26 connects the processor, memory, and hard
`disk to other devices in the workstation.
`
`The workstation is coupled to the digital network
`communications link 14 through a network controller
`
`5
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`65
`
`6
`28 connected between the network link 14 and the back-
`plane channel 26. The workstation is also coupled to a
`voice grade telephone line 29 through a modem 31
`connected between the backplane channel and the tele-
`phone line. Similarly, the workstation can be connected
`to other digital data conversation services, such as the
`ISDN digital telephone network.
`Multimedia workstation 12 includes a color video
`frame grabber 34 for capturing and digitizing frames of
`video from one or more analog video inputs 36. Video
`inputs 36 are each connected to a video source, such as
`a video camera 38 providing live analog video signals,
`or an auxiliary video storage device 40, such as a VCR
`or video laser disk player providing stored analog video
`signals. The analog video signals may be of any stan-
`dard types such as NTSC, SECAM, or PAL.
`The multimedia workstation 12 also includes a video
`buffer 35 which stores a frame of full color video graph-
`ics and characters generated by the workstation for
`display on a l280><1024 pixel color monitor 30. The
`video buffer 35 (and associated buffer controller) is
`connected to the backplane channel 26 for receiving
`video data from the processor 20. The video buffer is
`also connected to a video display subsystem 36 which
`converts the stored video frame data into analog signals
`for driving the color monitor 30.
`The video frame grabber 34 stores its digitized video
`data directly into a predetermined area of the video
`buffer 35. Thus, the digitized video input to the work-
`station by the frame grabber appears directly in a prede-
`termined area on the monitor 30, without having to pass
`through processor 20 or main memory 22. Further,
`processor 20 can read back the captured video frame
`data from the video buffer, store the data in main mem-
`ory 22, and further process the video data according to
`the video teleconferencing protocol described herein.
`Frame grabber 34 digitizes and stores each frame of
`video from an analog video source and can deliver up to
`30 frames per second of digitized 640x480 true color
`(24 bits) of NTSC/SECAM/PAL video into video
`frame buffer 35. A dedicated hardware video compres-
`sion subsystem 37 can also be connected to the back-
`plane channel 26 to provide high performance video
`compression of the digitized video data.
`The audio features of the multimedia workstation 12
`are implemented using an audio controller 42 connected
`to the backplane channel 26 for interfacing audio signals
`into the workstation and reproducing audio signals out
`of the workstation. An audio distribution box 44 is con-
`nected to the audio controller for directing audio to and
`from audio devices such as a microphone 46, a head-
`phone 48, and/or a telephone receiver 50. Auxiliary
`audio devices such as a recording device, a CD player,
`or amplified speakers may also be connected to the
`distribution box 44. Audio controller 42 receives audio
`input from the distribution box and digitizes the audio
`using an 8 bit mu—law encoder at 64 kbits per second or
`less to match typical telephone voice grade audio stan-
`dards (8 kHz bandwidth). For convenience, the head-
`phones 48 and microphone 46 may be combined into a
`single headset.
`In one commercial embodiment available from Digi-
`tal Equipment Corporation, multimedia workstation 12
`is built around a DECstation TM 5000/200 workstation
`in which processor 20 is an R3000 TM RISC processor
`and backplane channel 26 is a TurboChannel TM (TC)
`bu