`US005375068A
`
`United States Patent 1191
`
`[11] Patent Number:
`
`5,375,068
`
`Palmer et al.
`[45] Date of Patent: Dec. 20, 1994
`
`
`
`[54] VIDEO TELECONFERENCING FOR
`NETWORKED WORKSIAIIONS
`
`[75]
`
`Inventors:
`
`Ricky S. Palmer; Larry G. Palmer,
`both of Nashua, N.H.
`
`[73] Assignee:
`
`Digital Equipment Corporation,
`Maynard, Mass.
`
`an audio data stream to the remote workstation such .
`
`that the audio data ca_n_be reconstructed into a cont_inu- _
`ous audio signal. A video transmitter sends video data
`'so that each frame of video data to be sent is inserted
`into the audio data stream without affecting the'co7iti-
`nuity of the reconstructed audio signal at the remote '
`workstation.
`
`.
`
`[21] Appl. No.: 893,004
`
`[22] Filed:
`
`Jun. 3, 1992
`
`46 Claims, 23 Drawing Sheets
`
`OTHER PUBLICATIONS
`
`GOSK 15/00
`
`..
`
`364/514; 370/62
`364/514; 379/96, 94,
`379/202; 370/60, 62; 395/152
`
`
`
`Int. Cl.‘-'1
`[51]
`[52] us. Cl.
`[53] Field ofsearch
`
`Palmer and Palmer, "Desktop Meeting”, LAN Maga-
`zine, 6(11):11I—121 (Nov. 1991).
`D. Comer, “Internetworking with TCP/IP, vol. I: Prin-
`ciples, Protocols, and Architecture”, 2nd Edition, pp.
`1-3, 337-346, 505 (Prentice Hall: Engelwood Cliffs,
`NJ. 1991).
`
`-
`
`I
`
`Primary Examiner-—Emanuel T. Voeltz
`Assistant Exami77e71—Thomas Peeso
`Atromqa, Agent, or Firm-—Hamilton, Brook, Smith &
`Reynolds
`
`[57]
`
`ABSTRACT
`
`A video teleconferencing method and apparatus for
`computer workstations connected by a digital data net-
`work includes a transmission source portion for a local
`workstation to send audio and video teleconference
`data across the network to one or more remote worksta-
`tions, and, a receiver for the local workstation to re-
`ceive audio and video teleconference data back from
`the remote workstations. The local workstation sends
`teleconference data to each of the remote workstations v
`over a variable bandwidth digital data connection, and
`each of the remote workstations returns teleconference
`data back to the local workstation over another variable
`bandwidth digital data connection. The transmission
`source portion includes a master software process exe-
`cuting on the local workstation, and the receiver in-
`cludes a slave software process executing the remote
`
`__
`
`[56]
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`3,534,142 6/1971
`4,337,271
`6/1933
`4,516,156 5/1935
`4,525,779 6/1935
`4,574,374 53/1986
`4,645,372 2/1937
`4,650,929 3/1937
`4,653,090 3/1937
`4,679,191
`7/1937
`4,636,693
`3/1937
`4,710,917 12/1937
`4,734,765
`3/1933
`4,743,613 5/1933
`4,327,339 5/1939
`4,347,329 7/1939
`4,349,311
`7/1939
`4,332,743 11/1939
`I: 379/53
`4,333,795 12/1939 Andoetal. ..
`.... 379/53
`4,393,326
`1/1990 Duranetal.
`379/94
`4,397,366
`1/1990 Majmudaretal.
`.370/94.1
`4,905,231 2/1990 Leungetal.
`4,91_3,7_13 4/1990 _1=._n:mo:_.e_e: al. ........,............. 379153
`
`
`
`(List continued on next page.) I
`workstation The master process of a local workstation '
`causes execution of a slave process on a remote work-
`station for receiving video teleconferencedata froth the
`local workstation. An audio data transmitter for sends
`
`
`
`To other workstations
`
`Page 1 of 47
`
`LG Electronics Exhibit 1020
`
`
`
`5,375,093
`Page 2
`
`
`
`gflig
`9/1991
`5,046,079
`9/1991
`5,046,030
`5.056136 10/1991
`5,062.13610/I991
`5,072,442 12/1991
`5»079-527
`1/199?-
`5,099,510 3/ I992
`.5,103,4-44 4/1992
`5,111,409 5/I992
`5113431 5/1992
`r
`~
`$195,035 3/1993
`$399,939 4/1993
`
`.
`
`=
`,. 353/133
`__ 379/53
`...... 379/53
`.. 364/514
`...... 379/53
`370/62
`__ 379/53
`__ 379/53
`_ 379/53
`.. 379/53
`370/62
`364/900
`
`us. PATENT DocUM‘g§rI‘s
`13,924,311 5/ 1990 Ohkiuctal.
`4,932,047 6/1990 Emmons et al.
`4,935,953 6/1990 Appeletal. ....
`4,942,540 7/1990 Black etal.
`4,943,994 7/1990 Ohtsukae-tal..
`4,953,159 3/1990 I-Iaydenetal.
`4,9s3,19e
`s/1990 Ishikawa eta].
`4,952,521 1o/1990 Komatsuetal.
`4,9¢_55,s191o/1990 Kaunas .......
`4,995,071
`2/1991 Weber etal.
`5,003,532
`3/1991 Ashidaeta].
`5,034,916
`7/1991 Ordish
`
`’
`
`Page 2 of 47
`
`
`
`U.S. Patent
`
`Dec. 20, 1994
`
`Sheet 1 of 23
`
`5,375,068
`
`I0
`
`video
`comero
`
`38
`
`30
`
`- Multimode
`
`.
`
`Fiber
`
`To other workstations
`
`Page 3 of 47
`
`
`
`’ 12
`
`Headphones
`with
`
`Microphone
`
`Single Mode Fiber
`
`Headphones
`with
`Microphone 33
`
`video
`
`
`
`
`
`U.S. Patent
`
`Dec. 20, 1994
`
`Sheet 2 of :3
`
`5,375,068
`
`o\on
`
`
`_owweon9.$0.xcom.I
`
`_0..._CO.._O
`
`on.5._.
`
`3.>q._n_m_o
`_>m.5>mm=mon
`
`
`uzqm..._anII89>om_o_>_IIII_
`Emmqmo_zmwwmmmamagunmano_u_.=uomoon_IIIIIIIII|:_I\om._um..BPIJ_.5528.
`
`
`
`
`an
`
`qmmzqo
`82>I_
`
`Page 4 of 47
`
`
`
`ON.._@—_Oh.—._.._OOn—
`
`.232E.,m,__§s.zo9.3mm
`
`
`
`v_z_._Eon..
`
`_nmm39.on809.0
`
`,_W..4,_4o._.....%.aw“W...
`
`N.0_n_
`
`
`
`
`
`US. Patent
`
`Dec. 20, 1994
`
`Sheet 3 of 23
`
`5,375,068
`
`72
`
`
`70
`X server(video)
`'
`
`.-
`_
`xmxv usmg
`TCPIIP (tocal)
`
`
`system calls
`
`62
`
`
`
`
`
`TCPIIP or DEcne1
`
`66
`_ 68
`‘xrnedio using
`pipe
`32
`TCPIIP (local!)
`50
`DTP using
`-
`E romp %
`DTP using 84
`76
`
`Audio input
`
`
`
`TCP/IP (local)
`
`6‘ @
`
` 74
`
`@ 8°
`78
`
`@ %
`
`I02
`
`I04
`
`I06
`
`
`
`II2
`
`IIO
`
`Network
`
`signal/execv
`
`108
`
`Machine A
`
`Machine B
`
`Page 5 of 47
`
`FIG. 4
`
`
`
`U.S. Patent
`
`Dec. 20, 1994
`
`Sheet 4 of 23
`
`5,375,068
`
`IIO
`
`A calls B, B answers
`
`FIG. 5A
`
`%
`
`I20
`
`I02
`
`I2b
`
`Q -~\—' H
`"2°j\\ ”°Acalls C, Canswers
`H00
`'2C
`
`FIG. 5B
`
`'20
`
`I02
`
`lzb
`
`no
`
`n2
`
`U’‘\‘
`HOUR
`
`/<
`
`:3
`
`:02
`
`HOD
`
`B calls C or A joins B and C
`
`FIG. 50
`
`Page 6 of 47
`
`
`
`U.S. Patent
`
`A
`
`Dec. 20, 1994
`
`Sheet 5 of 28
`
`5,375,068
`
`ANALOG VIDEO
`NTSO/SECAM/PAL
`
`7 30°
`
`DIG!T!ZE VIDEO IN
`FRAME BUFFER
`
`3°?
`
`VIDEO DATA m
`APPLICATION BUFFER
`
`I 8A
`
`B
`
`VlDEO DATA IN
`NETwOR:< BUFFER
`
`205
`
`wDEO DATA
`RAVERSES NETWORK
`
`903
`
`VIDEO DATA IN
`NETWORK BUFFER
`
`A
`
`2'0
`
`v:DEO DATA m
`APPLICATION BUFFER
`
`.
`
`'2 N
`
`DIGITIZED VIDEO
`IN FRAME BUFFER
`
`’
`
`'2 43
`
`DIGITIZED VIDEO
`
`DISPLAYED
`
`{E2 0)
`
`Page 7 of 47
`
`FIG. 6
`
`
`
`US. Patent
`
`Dec. 20, 1994
`
`Sheet 6 of 23
`
`5,375,068
`
`LOCAL
`WORKSTATION A
`
`REMOTE
`WORKSTATION B
`
`' INVOKE LOCAL
`DECspin
`APPLICATION
`
`
`
`
`
`§
`
`NETWORK
`
`.332. _ __ _ __ _ _ _
`
`CREATE
`
`PORT
`
`304
`
`INVOKE LOCAL
`DECspind
`APPLICATION
`
`
`VERIFY
`‘r D-TO-END
`CONNECTION
`
`308
`
`
`
`310
`
`3'2
`
`3I8
`
`
`
`I
`
`I
`
`j j j j
`
`
`
`
`OR "ANSWER
`-MACHINE"
`
`
`
`
`INVOKE LOCAL
`DEC spin
`APPLICATION
`
`"STARTHEADER"
`
`
`
`cu-uacx FOR
`IIOK"
`
`322
`
`,
`
`uOKu
`
`BEGIN SENDING
`AUDIO!VIDEO
`DATA
`
`324
`
`
`
`FIG. 7
`
`Page 8 of 47
`
`
`
`US. Patent
`
`Dec. 20, mm.
`
`Sheet 7 of 23
`
`5,375,068
`
`START
`
`OK
`
`OIOIOIOO = 0x54
`
`|
`
`onouoou = 0x53
`
`Total length = 2 bytes
`
`F I G. 8
`
`oaoonou -= 01:48
`
`I
`
`oaoonu = Ox4F
`
`Total length = 2 bytes
`
`STARTHEADER
`
`5 reserved Iongwords (32 bits each)
`
`frornerate desired (longword)
`
`DTP flags
`
`2 reserved Iongwords
`
`406
`
`4'2
`
`total number of frames
`
`-
`
`403
`
`i reserved longword
`
`video width
`
`video height
`
`bits per pixe! (8 or 24)
`
`35 reserved Iongwords
`
`400
`
`402
`
`404
`
`‘I8 Iongwords (used as Ic-gin/user handle)
`
`4'0
`
`Page 9 of 47
`
`FIG. IO
`
`
`
`US. Patent
`
`Dec.20,1994
`
`Sheet 8 of 23
`
`5,375,068
`
`QUIT
`
`onoaoaoo = 0x54
`
`|
`
`o:o1ooo:=ox5:
`
`Total length = Zbytes
`
`FIG. ll
`
`ANSWERMACHINE
`
`0:00: :0: = 0x40
`
`|
`
`ouooooou = 0x4!
`
`Total length = 2 bytes
`
`FIG. I2
`
`CONTROL
`
`45°
`0x43
`I
`0x54
`452
`45‘
`456
`
`5 reserved longwords
`
`Total length is 34 bytes
`
`FIG. I3
`
`Page 10 of 47
`
`
`
`US. Patent
`
`Dec. 20, 1994
`
`Sheet 9 of 23
`
`5,375,068
`
`350
`
`352
`
`
`
` SEND
`CONTENTS OF
`
`AUDIO BUFFER
`
`NEXT VIDEO
`FRAME
`AVA l LABLE
`
`360
`
`SEND VIDEO
`FRAME
`
`354
`
`
`
`CHECK FOR
`
`ll OKII
`
`>lI2 SEC.
`
`IIOKII
`>2 FRAME
`TIMES
`
`
`
`
`
`SEND
`CONTENTS OF
`AUDIO BUFFER
`
`FIG. I4
`
`Page 11 of 47
`
`
`
`US. Patent
`
`Dec. 20, 1994
`
`Sheet 10 of 23
`
`5,375,068
`
`AUDIOHEADER
`
`414-
`
`6 reserved longwords
`
` FIG. I5
`
`VIDEOHEADER
`
`0x43
`
`|
`
`Ox4F
`
`43°
`
`4 reserved longwords
`
`Total Length is 34 bytes
`
`FIG. I6
`
`Page 12 of 47
`
`
`
`U.S. Patent
`
`Dec. 20, 1994
`
`Sheet 11 of 23
`
`5,375,068
`
`xoq
`
`m9.043.04
`
`
`
`zo_mm_s.mz$_»zamm
`
`
`
`zo_.6m_._..oozamm_ms_<mn_no
`
`Nus_<Eno
`
`
`
`n_Ozo:.om_.3oozaum
`
`
`
`_224$.kzmmmao
`
`Page 13 of 47
`
`
`
`
`U.S. Patent
`
`Dec. 20, 1994
`
`Sheet 12 of 23
`
`5,375,068
`
`524
`
`Page 14 of 47
`
`
`
`U.S. Patent
`
`Dec.20, 1994
`
`Sheet 13 of _2s
`
`5,375,068
`
`858
`
`FIG. 19
`
`Page 15 of 47
`
`
`
`U.S. Patent ~
`
`Dec. 20, 1994
`
`Sheet 14 of 23
`
`5,375,068
`
`Page 16 of 47
`
`
`
`U.S. Patent
`
`Dec. 20, 1994
`
`% Sheet 15 of 23
`
`5,375,068
`
`650
`
`Page 17 of 47
`
`
`
`U.S. Patent
`
`Dec. 20, 1994
`
`Sheet 16 of 23
`
`5,375,068
`
`700
`
`/
`
`720
`
`FIG. 22
`
`Page 18 of 47
`
`
`
`U.S. Patent
`
`Dec. 20, 1994
`
`Sheet 17 of 23
`
`5,375,068
`
`¢0-
`
`co_.5:m:oEmn_
`
`:o_8:uo._E_
`
`._..o._s.oz
`
`&ommo_o
`
`$_so.m
`
`oom
`
`Esmcd
`
`
`
`0..co_m._m>B
`
`
`
`3¢_.Cm:O.0‘
`
`mm...
`
`MN.o_n_
`
`Page 19 of 47
`
`
`
`
`U.S. Patent
`
`Dec.20,1994
`
`
`
`Sheet 18 ans
`
`5,375,068
`
`K KV'NC'R\'\VRVQxx-I:-R'i\\\\§;\Vk\\\VN‘3'N¥C'N—
`D
`I E3
`
`rI
`
`I3
`
`El
`
`
`
`Eaaaaaaa
`fiflfllflflfl
`
`El m
`
`naunnunun
`
`Emmmaama a[E
`Eaanaama n
`
`WorkstationA
`
`Page 20 of 47
`
`
`
`US. Patent
`
`Dec. 20, 1994
`
`Sheet 19_of 28
`
`5,375,068
`
`m_._o:E9_._o>>
`
`Page 21 of 47
`
`
`
`U.S. Patent
`
`Dec. 20, 1994
`
`Sheet 20 of 23
`
`5,375,068
`
`
`
`m.._o__2m._._o>>
`
`Page 22 of 47
`
`
`
`US. Patent
`
`Dec. 20, 1994,
`
`Sheet 21 of 28
`
`5,375,068
`
`'
`
`'
`
`'
`
`\:~.-'2'-i::&'\‘i::kV-..\.\\"‘-N‘.-KV::\\‘ Z
`
`:38.o_.._
`
`8m
`
`Qaaaannn
`Qflnannnaa a
`
`El
`
`0
`
`EIIIIDEIEIEI
`
`<...o__o—9_8?
`
`Page 23 of 47
`
`
`
`U.S. Patent
`
`Dec. 20,
`
`Sheet 22 of 28
`
`5,375,068
`
`E8.o_.._
`
`uom
`
`0:o:E9_._o>>
`
`Page 24 of 47
`
`
`
`U.S. Patent
`
`Dec. 20, 1994
`
`Sheet 23 of 23
`
`5,375,068
`
`0co__o..$_32,
`
`Page 25 of 47
`
`
`
`US. Patent
`
`Dec. 20, 1994_
`
`Sheet 24 of 23
`
`5,375,068
`
`WorkstationA
`
`Page 26 of 47
`
`aaaaanua
`gflnnnanaa
`
`
`
`U.S. Patent
`
`Dec. 20, 1994
`
`Sheet 25 of 23
`
`5,375,068
`
`m__._o:c..mv_.o>>
`
`Page 27 of 47
`
`
`
`U.S. Patent
`
`Dec. 20, 1994_
`
`Sheet 26 of 28
`
`5,375,068
`
`n §
`
`E E
`
`g E
`
`fi E
`
`E E
`
`EH
`
`3%.O_u._
`
`com
`
`0:o:E...$_._o2,
`
`Page 28 of 47
`
`
`
`U-S- Patent
`
`Dec. 20. 1994
`
`Sheet 27 of 23
`
`5,375,068
`
`AL
`
`mcozofifoxs
`
`Page 29 of 47
`
`
`
`US. Patent
`
`Dec. 20, 1994
`
`Sheet 28 of 28
`
`5,375,068
`
`Sowor.
`
`com
`
`0:o_.6_9_._o>>
`
`Page 30 of 47
`
`
`
`1
`
`VIDEO 'I'EI..ECONFERENClNG FOR
`NETWORKED WORKSTATIONS
`
`5,375,068
`
`BACKGROUND OF THE INVENTION
`
`5
`
`Video communications has evolved over the years
`from a simple video telephone concept to a sophisti-
`cated network for allowing multiple users to participate
`in a video teleconference For full featured video tele-
`conferencing, users require both an audio communica- 10
`tions path and a real time visual communication path
`synchronized to the audio path. Furthermore, it is desir-
`able to support full color video and telephone quality
`audio. Video teleconferencing capabilities are limited
`mainly by the bandwidth of the transmission medium 15
`connecting the teleconferencing terminals.
`Many computer workstations used in the office or
`laboratory environment today are connected with other
`workstations, fle servers, or other resources over high-
`speed local area networks. Local area networks, in turn, 29
`are often connected together through high-speed speed
`gateways which connect workstations which may be
`distributed over a wide geographic area. Network wide
`protocols allow workstations to exchange packets of
`data at high rates of speed and reliability. Fixed band- 25
`width digital and analog video channels have been com-
`bined with computer networks to implement some
`video teleconferencing features. These include high
`bandwidth CATV/FDM type analog channels and
`fixed allocation TDM data channels for the video data. 39
`
`SUMMARY OF THE INVENTION
`
`40
`
`Workstations today have obtained unprecedented
`computational power and utility. The powerful RISC
`type CPUs and fast, high resolution graphical displays 35
`have made possible multimedia workstations which
`integrate live audio and video into the programming
`environment. Graphical User Interface operating sys-
`tems (GUI) have allowed effective integration of audio
`and video into application programming.
`The present invention provides n-way video telecon-
`ferencing among networked computer workstations
`using the existing variable bandwidth digital data net-
`work for transferring synchronized audio and video
`teleconferencing data between the workstations. The 45
`teleconferencing apparatus and protocol of this inven-
`tion provides high quality video teleconferencing with-
`out the need for a guaranteed wide bandwidth analog
`video channel or a fixed allocation digital video chan-
`nel. Rather, the invention uses standard non-allocated 50
`data packets typically found on local area networks to
`transfer the audio and video teleconferencing data.
`‘Thus, no guaranteed bandwidth is required to carry on
`a useful video teleconference. An continuous audio data
`stream model provides continuous audio signals at the 55
`expense of video data when necessary, which is desir-
`able since the ear is more sensitive to a break in the
`audio data than the eye is to the loss of a frame of video
`data. A “push” data model provides a secure system by
`preventing remote workstations from activating an- 60
`other worksta.tion’s video teleconferencing functions.
`In general, in one aspect, the invention features a
`video teleconferencing method and apparatus for com-
`puter workstations connected by a digital data network.
`The computer workstations include a transmission 65
`source means for a local workstation to send audio and
`
`video teleconference data across the network to one or
`more remote workstations, and, a receiver for the local
`
`_
`
`Page 31 of 47
`
`2
`workstation to receive audio and video teleconference
`data back from the remote workstations. The local
`workstation sends teleconference data to each of the
`remote workstations over a variable bandwidth digital
`data connection, and each of the remote workstations
`returns teleconference data back to the local worksta-
`tion over another variable bandwidth digital data con-
`nection. The variable bandwidth digital data connec-
`tions include the data packet oriented data channels
`associated with, for example, FDDI, DECnet, and
`Ethernet local area networks. Furthermore, a wide area
`digital network, such as ISDN, can also be used with
`the video teleconferencing apparatus and method of this
`invention.
`
`In preferred embodiments, the transmission source
`means includes a master software process executing on
`the local workstation, and the receiver includes a slave
`software process executing the remote workstation.
`The master software process formats and sends video
`teleconference data to the slave process. The slave pro-
`cess receives and reconstructs the audio and video tele-
`conference data for audible and visual reproduction,
`respectively. The video data is presented as an image on
`the display of the receiving workstation, while the
`audio data is sent to either amplified speakers or head-
`phones. In other preferred embodiments, the master
`process of a local workstation causes execution of a
`slave process on a remote workstation for receiving
`video teleconference data from the local workstation.
`The slave process running on the remote workstation in
`turn causes execution of a master process on the remote
`workstation for sending video teleconference data back
`to the local workstation. The master process of the
`remote workstation in turn causes execution of a slave
`process on the local workstation for receiving the video
`teleconference data sent by the master process of the
`remote workstation. The local workstation executes a
`slave process for each master process on a remote work-
`station sending video teleconference data to the local
`workstation.
`
`In yet other preferred embodiments, the transmission
`source includes an audio data transmitter for sending an
`audio data stream to the remote workstation such that
`the audio data can be reconstructed into a continuous
`audio signal. The transmission source also includes a
`video transmitter for sending video data to the remote
`workstation so that each frame of video data to be sent
`is inserted into the audio data stream without affecting
`the continuity of the reconstructed audio signal at the
`remote workstation.
`In yet other preferred embodiments the video trans-
`mitter precludes a frame of video data from being sent
`to the remote workstation if a system overload exists.
`The audio transmitter sends the audio data stream cor-
`responding to the precluded video frame to the remote
`workstation to prevent loss of continuity of the audio
`signal during a system overload. In other preferred
`embodiments the video transmitter precludes a frame of
`video data from being sent to the remote workstation in
`response to a system failure condition. The audio trans-
`mitter accumulates audio data for a predetermined time
`interval during the system failure condition, and trans-
`mits the accumulated audio data stream to the remote
`workstation once the failure has been corrected. For
`instance, the audio transmitter may continually accumu-
`late the last i second of audio data while the failure
`exists, trimming any audio data older than i second.
`
`
`
`3
`The last i second of audio data accumulated before the
`failures correction is sent to the remote workstation as
`soon as the failure is corrected. The most recently avail-
`able frame of video is then also sent.
`In still other preferred embodiments, timing informa-
`tion is attached to each frame of video data sent to the
`remote workstation. The timing information indicates a
`point in the continuous audio data stream which corre-
`sponds in time to the frame of video data. The receiver
`of the remote workstation includes a synchronizer for
`displaying a received frame of video when the point in
`the audio stream corresponding to the timing informa-
`tion of the received video frame is audibly reproduced
`at the remote workstation. The synchronizer counts the
`amount of audio data received in the continuous audio
`stream and compares the count to the timing informa-
`tion sent along with the most recently received video
`frame to determine when to display the frame.
`In general, in another aspect, the invention features a
`multimedia computer workstation, such as a RISC
`workstation or IBM PC, having video teleconferencing
`capabilities. The multimedia workstation of this inven-
`tion includes a network interface for establishing a vari-
`able bandwidth digital communications channel across
`a digital data network with another multimedia work-
`station. A video source provides a frame of digitized
`video data, and an audio source provides digitized audio
`data associated with the frame of video data. A data
`
`transmitter transmits the audio and video data through
`the network interface across the variable bandwidth
`digital communications channel to another workstation.
`A receiver receives audio and video data through the
`network interface across the variable bandwidth digital
`communications channel from another workstation.
`The workstation also includes means for displaying the
`received video data on the workstation display, and
`means for audibly reproducing the received audio data.
`In preferred embodiments, the video source includes
`a. video camera, a video tape recorder, and/or a video
`laser disk player providing framm of analog video. A
`video frame grabber captures, digitizes, and stores each
`frame of analog video. The video source also includes
`digital video data stored in a file accessible by the work-
`station. A video compressor may compress the video
`data using JPEG or MPEG compression. The audio
`source includes a microphone for live audio, or pre-
`recorded audio corresponding to frames of pre-
`recorded video, from for instance a video tape recorder
`or laser disk. An audio digitizer digitizes and stores the
`audio using mu-law compression. The audio source also
`includes digital audio data stored in a file, preferably
`along with digital video data, accessible by the worksta-
`tion.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`The foregoing and other objects, features and advan-
`tages of the invention will be apparent from the follow-
`ing more particular description of preferred embodi-
`ments of the invention, as illustrated in the accompany-
`ing drawings in which like reference characters refer to
`the same parts throughout the different views. The
`drawings are not necessarily to scale, emphasis instead
`being placed upon illustrating the principles of the in-
`vention.
`
`FIG. 1 is a pictorial representation of the distributed
`computer system featuring multimedia workstations
`having video teleconferencing capabilities of this inven-
`tion.
`
`Page 32 of 47
`
`5,375,068
`
`4
`FIG. 2 is a block diagram showing a preferred em-
`bodiment of the multimedia workstation for implement-
`ing the video teleconferencing features of the distribu-
`tor computer system of FIG. 1.
`FIG. 3 is a block diagram showing the software envi-
`ronment for implementing a preferred embodiment of a
`DECspin video teleconferencing application for imple-
`menting the video teleconferencing features of the sys-
`tem of FIG. 1.
`'
`
`FIG. 4 is a block diagram showing the one-way con-
`nections established between two workstations to im-
`plement a two-way video teleconference.
`FIGS. 5a-5c are block diagrams illustrating the multi-
`ple one-way video teleconferencing connections estab-
`lished between three workstations joined in the same
`teleconference.
`
`FIG. 6 is a flowchart illustrating the flow of video
`during a video teleconference.
`FIG. 7 is a flowchart illustrating the teleconferencing
`protocol of this invention for establishing a video tele-
`conference connection between two workstations.
`FIG. 8 illustrates the format of the START message
`of the teleconferencing protocol of this invention.
`FIG. 9 illustrates the format of the OK message of the
`teleconferencing protocol of this invention.
`FIG. 10 illustrates the format of the START-
`I-IEADER message of the video teleconferencing pro-
`tocol of this invention.
`
`FIG. 11 illustrates the format ofthe QUIT message of
`the teleconferencing protocol of this invention.
`FIG. 12 illustrates the format of the ANSWERMA-
`CHINE message of the teleconferencing protocol of
`this invention.
`
`FIG. 13 illustrates the format of the CONTROL
`message of the video teleconferencing protocol of this
`invention.
`
`FIG. 14 is a flowchart showing the audio and video
`data transfer sequence of the video teleconferencing
`protocol of this invention.
`FIG. 15 shows the format of the AUDIO!-IEADER
`message of the video teleconferencing protocol of this
`invention,
`FIG. 16 shows the format of the VIDEO]-IEADER
`message of the video teleconferencing protocol of this
`invention,
`FIG. 17 shows a timing diagram of the audio and
`video data synchronization of the video teleconferenc-
`ing protocol of this invention.
`FIG. 18 shows the top level graphical user interface
`window for controlling a video teleconferencing ses-
`sion of this invention.
`FIG. 19 shows the graphical user interface window
`for displaying video data received from another work-
`station during a video teleconferencing session of this
`invention.
`
`30
`
`40
`
`45
`
`50
`
`55
`
`FIG. 20 shows a second level graphical user call list
`interface window for establishing the video telecon-
`ferencing oonnections to other workstations of this
`invention.
`
`60
`
`FIG. 21 shows a second level control graphical user
`interface window for adjusting the parameters for a
`video teleconference of this invention.
`FIG. 22 shows a second level monitor graphical user
`interface window for monitoring the parameters of a
`video teleconference of this invention.
`FIG. 23 shows a second level documentation graphi-
`cal user interface window for obtaining information
`
`
`
`5,375,068
`
`5
`about the operation and features of a video teleconfer-
`ence of this invention.
`FIG. 24 shows a third level documentation graphical
`user interface window for obtaining information about a
`topic selected from the second level user interface win-
`dow of FIG. 23.
`
`6
`28 connected between the network link 14 and the back-
`plane channel 26. The workstation is also coupled to a
`voice grade telephone line 29 through a modem 31
`connected between the backplane channel and the tele-
`phone line. Similarly, the workstation can be connected
`to other digital data conversation services, such as the
`ISDN digital telephone network.
`Multimedia workstation 12 includes a color video
`frame grabber 34 for capturing and digitizing frames of
`video from one or more analog video inputs 36. Video
`inputs 36 are each connected to a video source, such as
`a video camera 38 providing live analog video signals,
`or an auxiliary video storage device 40, such as a VCR
`or video laser disk player providing stored analog video
`signals. The analog video signals may be of any stan-
`dard types such as NTSC, SECAM, or PAL.
`The multimedia workstation 12 also includes a video
`buffer 35 which stores a frame of full color video graph-
`ics and characters generated by the workstation for
`display on a l280>< 1024 pixel color monitor 30. The
`video buffer 35 (and associated buffer controller) is
`connected to the backplane channel 26 for receiving
`video data from the processor 20. The video buffer is
`also connected to a video display subsystem 36 which
`converts the stored video frame data into analog signals
`for driving the color monitor 30.
`The video frame grabber 34 stores its digitized video
`data directly into a predetermined area of the video
`buffer 35. Thus, the digitized video input to the work-
`station by the frame grabber appears directly in a prede-
`termined area on the monitor 30, without having to pass
`through processor 20 or main memory 22. Further,
`processor 20 can read back the captured video frame
`data from the video buffer, store the data in main mem-
`ory22,andfu.rtherprocessthevideodataaccordingto
`the video teleconferencing protocol described herein.
`Frame grabber 34 digitizes and stores each frame of
`video from an analog video source and can deliver up to
`30 frames per second of digitized 640x480 true color
`(24 bits) of NTSC/SECAM/PAL video into video
`frame buffer 35. A dedicated hardware video compres-
`sion subsystem 37 can also be connected to the back-
`plane channel 26 to provide high performance video
`compression of the digitized video data.
`The audio features of the multimedia workstation 12
`are implemented using an audio controller 42 connected
`to the backplane channel 26 for interfacing audio signals
`into the workstation and reproducing audio signals out
`of the workstation. An audio distribution box 44 is con-
`nected to the audio controller for directing audio to and
`from audio devices such as a microphone 46, a head-
`phone 48, and/or a telephone receiver 5!]. Auxiliary
`audio devices such as a recording device, a CD player,
`or amplified speakers may also be connected to the
`distribution box 44. Audio controller 42 receives audio
`input from the distribution box and digitizes the audio
`using an 8 bit mu-law encoder at 64 kbits per second or
`less to match typical telephone voice grade audio stan-
`dards (8 kHz bandwidth). For convenience, the head-
`phones 48 and microphone 46 may be combined into a
`single headset
`In one commercial embodiment available from Digi-
`tal Equipment Corporation, multimedia workstation 12
`is built around a DECstation TM 5000/200 workstation
`in which processor 20 is an R3UO0TM RISC processor
`and backplane channel 26 is a Turbochannel TM (TC)
`bus. Network controller 28 is a DEC FDDIcontroller
`700'1‘M. Frame grabber 34/video buffer 35/video dis-
`
`FIG. 25 shows a graphical user interface window of
`a ring box for announcing a video teleconference call to
`another workstation to establish a video teleconference
`of this invention.
`
`FIGS. 26(a)—26(}c) show the dislay screens of three
`Workstations
`in a three-way video tele-
`conference of this invention.
`
`DESCRIPTION OF THE PREFERRED
`EMBODIMENT
`
`FIG. 1 shows a distributed computer system 10, hav-
`ing a plurality of multimedia workstations 12 connected
`by a high speed digital network 14, which features
`n-way video teleconferencing of this invention. Each of 20
`the multimedia workstations 12 is capable of producing
`live audio and video data for transmission across the
`network to another multimedia workstation. Further,
`each of the multimedia workstations is capable of soni-
`cally reproducing the audio data and visually reproduc-
`ing the video data received from another workstation.
`Two or more of the networked workstations can
`participate in an n-way video teleconference using the
`teleconferencing protocol of this invention. The tele-
`conferencing protocol of this invention allows real time
`synchronized audio and video transmission across the
`network without the use of a fixed bandwidth or dedi-
`cated time slot transmission medium. Rather, this inven-
`tion provides high performance video teleconferencing
`features using standard digital network transport level
`protocols such as Internet TCP/IP and UDP/IP. or
`DECnet TM . The physical network link 14 should be a
`high speed FDDI (Fiber Distributed Data Interface)
`fiber optic link running at 100 MB/Sec. for maximum
`performance (about 15 uncompressed black and white 40
`frames per second), but can also be virtually any type of
`high quality digital network link such as an Ether-
`net TM . In the case of the FDDI network link, perfor-
`mance" is typically limited by the workstation hardware
`and software, rather than by the throughput of the
`network. In addition, wide area networking (WAN)
`technologies, such as T1 and T3 digital carriers, can be
`used with the video teleconferencing protocol of this
`invention. These WAN technologies can be expected to
`achieve a video frame rate of at least about 12 Hz, as-
`suming black and white video images compressed with
`JPBG, MPEG, or another video compression tech-
`nique. The features of one preferred embodiment of this
`invention are commercially embodied in the DEC-
`spin TM product available from Digital Equipment
`Corporation, Maynard, Mass, the assignee of this pa-
`tent application.
`FIG. 2 shows a preferred embodiment of the multi-
`media workstation 12 for implementing the video tele-
`conferencing protocol of this invention. Workstation 12 60
`includes a high performance processor 20 connected to
`a large working memory 22 having 24 megabytes or
`more capacity, and a large hard drive having 1 giga-
`bytes or more capacity. A high performance backplane
`channel 26 connects the processor, memory, and hard
`disk to other devices in the workstation.
`
`The workstation is coupled to the digital network
`communications link 14 through a network controller
`
`Page 33 of 47
`
`
`
`7
`play subsystem 36 is a DECvideo/in TM TurboChannel
`compatible video system, and audio controller 42 is a
`DECaudio TM Turbochannel compatible audio con-
`troller.
`
`5,375,068
`
`10
`
`8
`.
`found in “ULTRIX Worksystem Software, X Window
`System Protocol: X Version 11," Digital Equipment
`Corporation order , number AA-MA98A-TE, May-
`nard, Mass. (1988, Version 2.0), the contents of which
`are incorporated herein by reference. A description of
`the Motif programming environment can be found in
`Open Software Foundation, “OSF/Motif Program-
`mer’s Guide,” Prentice Hall, Englewood Cliffs, N. J.
`07632 (1991, Revision 1.1), the contents of which are
`incorporated herein by reference.
`Through UNIX system calls the DECspin applica-
`tion establishes local TCP/IP “sockets” for communi-
`cation with various software servers providing multi-
`media services for the DECspin application. The ore-
`ation and use of UNIX “socket” system calls for creat-
`ing TCP/IP application connections is discussed in
`detail by Corner, D. E., “Internetworking With
`TCP/IP, Vol. 1: Principles, Protocols, and Architec-
`ture, Second Edition” Prentice Hall, Englewood Cliffs,
`NJ. (1991) (pp 337-346), the contents of which is incor-
`porated herein by reference. The DECspin application
`communicates with an audio server application 66
`which provides audio services using X Windows
`Xmedia commands 68 through a local TCP/IP
`“socket”. The DECspin application communicates with
`a video X Windows Xserver application 70 which pro-
`vides video services using X11 and XV extension com-
`mands 72 through another local TC!’/IP “socket”.
`DECspin communicates with a Motif based graphical
`user interface (GUI) 74 through yet another local
`TCP/IP “socket”.
`
`Alternatively, multimedia workstation 12 can be built
`around a person computer platfonn, such as an IBM ‘I'M
`compatible PC. Preferably, the PC should have a high
`speed Intel 30386'rM, 80486TM, or successor proces-
`sor. The PC should be compatible of running in the
`Microsoft Windows TM or Windows NTTM graphical
`operating system environment.
`Workstations and PCs of different performance levels
`can all participate in the same video teleconference
`using the video teleconferencing protocol of this inven-
`tion. For instance, less powerful or slower PC’s and
`workstations can simply communicate at a slower video
`rate with the more powerful workstations, without
`affecting the overall video rate between other worksta-
`tions.
`The effective video frame rate of a video teleconfer-
`ence depends to a large extent on the quantity of video
`data handled by a workstation during each video frame.
`Video compression techniques can greatly enhance the
`effective frame rate of the video teleconference espe-
`