United States Patent 1191
`
`[11] Patent Number:
`
`5,375,068
`
`[45] Date of Patent: Dec. 20, 1994
`Palmer et al.
`
`
`
`US005375068A
`
`[54] VIDEO TELECONFERENCING FOR
`NETWORKED WORKSTATIONS
`
`[75]
`
`Inventors: Ricky S. Palmer; Larry G. Palmer,
`both of Nashua, N.H.
`
`[73] Assignee: Digital Equipment Corporation,
`Maynard, Mass.
`
`an audio data stream tohthe remote workstation such .
`that the audiopdata ca_r1_be reconstructed into a cont_inu-
`ous audio signal. A video transmitter sends video data
`‘so that each frame of video data to be sent is inserted .
`into the audio data stream without affecting the‘con.ti-
`nuity of the reconstructed audio signal at the remote
`workstation.
`
`46 Claims, 28 Drawing Sheets
`
`OTHER PUBLICATIONS
`
`Palmer and Palmer, “Desktop Meeting”, LAN Maga-
`zine, 6(ll):ll1—l2l (Nov. 1991).
`D. Comer, “Internetworking with TCP/IP, vol. I: Prin- V
`ciples, Protocols, and Architecture”, 2nd Edition, pp.
`1-8, 337-346, 505 (Prentice Hall: Engelwood Cliffs,
`NJ. 1991).
`
`-
`
`Primary Examiner-Emanuel T. Voeltz
`Assistant Examiner—Thomas Peeso
`Attomey, Agent, or Firm—Hamilton, Brook, Smith &
`Reynolds
`
`[57]
`
`ABSTRACI‘
`
`A video teleconferencing method and apparatus for
`computer workstations connected by a digital data net-
`work includes a transmission source portion for a local
`workstation to send audio and video teleconference
`data across the network to one or more remote worksta-
`tions, and, a receiver for the local workstation to re-
`ceive audio and video teleconference data back from
`the remote workstations. The local workstation sends
`teleconference data to each of the remote workstations 8
`over a variable bandwidth digital data connection, and
`each of the remote workstations returns teleconference
`data back to the local workstation over another variable
`bandwidth digital data connection. The transmission
`source portion includes a master software process exe-
`cuting on the local workstation, and the receiver in-
`cludes a slave software process executing the remote
`
`[21] Appl. No.: 893,074
`
`[22] Filed:
`
`Jun. 3, 1992
`
`[51]
`Int. Cl.5 ............................................. G06K 15/00
`
`[52] U.S. Cl. .......................... .. 364/514; 370/62
`[58] Field of Search .................. .. 364/514; 379/96, 94,
`379/202; 370/60, 62; 395/152
`
`_
`
`[56]
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`3,584,142
`6/1971 Schoeffler ............................ 178/6.8
`4,387,271
`6/1983 Artorn .......
`179/2 DP
`
`4,516,156 5/1985 Fabris et al.
`..
`....... .. 358/85
`4,525,779 6/1985 Davids et al.
`364/200
`4,574,374 3/1986 Scordo ......
`..... 370/62
`4,645,872 2/1987 Pressman et al.
`379/54
`4,650,929
`3/1987 Boerger et al.
`......
`..... 358/86
`4,653,090 3/1987 Hayden ................
`379/204
`4,679,191
`7/1987 Nelsoy et al.
`.... ....
`.. ... 370/84
`4,686,698
`8/1987 Tompkins et al.
`379/53
`4,710,917 12/1987 Tompkins et al.
`370/62
`4,734,765
`3/1988 Okada et al.
`358/102
`4,748,618
`5/1988 Brown et al.
`370/94
`......
`4,827,339
`5/1989 Wada et al.
`358/136
`4,847,829 7/1989 Tompkins et al.
`...... '370/62
`4,849,811
`7/1989 Kleinerman ......
`358/133
`4,882,743 11/1989 Mahmoud
`...... 379/53
`4,888,795 12/1989 Ando et al
`379/53
`4,893,326
`1/1990 Duran et al
`379/53
`4,897,866 1/1990 Majmudar et al
`...... 379/94
`4,905,231
`2/1990 ‘ Leung et al.
`370/94.1
`4,9 l8,7l_8___ _4fl99_(_J_ ___E_mmons_7et al.
`........._._............ 379/53
`
`
`
`(List continued on next page.)
`
`workstation. The master process "of a local workstation ’
`causes execution of a slave process on a remote work-
`station for receiving video teleconference data from the
`local workstation. An audio data transmitter for sends
`
`
`
`To other workstations
`
`1
`‘Petitioners HTC and LG — Exhibit 10314, p.
`HTC and LG v. PUMA, IPR2()l5—()l5()2
`
`Headphones
`with
`Microphone
`
`Headphones
`with
`Microphone 38
`
`video
`Camera
`
`Petitioners HTC and LG - Exhibit 1034, p. 1
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`5,375,068
`Page 2
`
`5,042,006
`5,042,062
`5,046,079
`5,046,080
`5,056,136
`5,062,136
`5,072,442
`5,079,627
`5,099,510
`.5,103,444
`5,111,409
`5,1 13,431
`5,195,086
`5,200,989
`
`8/1991 Flohrer ............................... 364/900
`8/1991 Lee et al
`. .....
`... .. 379/54
`9/1991 Hashimoto
`379/53
`9/1991 Lee etal. ......
`379/53
`10/1991 Smith ............
`389/10
`10/1991 Gattis et al.
`..
`380/18
`12/1991 Todd ... ..... .....
`..... 370/62
`1/1992 Filo ........................ 358/85
`3/1992 Blinken, Jr. et al.
`379/202
`4/1992 Leung et al. .... ..
`..... 370/60
`5/1992 Gasper et al. .
`.. 395/152 .
`5/1992 Horn ........................ .. 379/94 1
`3/1993 Baumgartner et al
`379/202
`4/1993
`Milone .................................. 379/96
`
`
`
`Petitioners HTC and LG — Exhibit 1034, p. 2
`HTC and LG v. PUMA, IPR2()15—()15()2
`
`U.S. PATENT DOCUMENTS
`5/1990
`6/1990
`6/1990
`7/1990
`7/1990
`8/1990
`8/1990
`10/1990
`10/1990
`2/1991
`3/1991
`7/1991
`
`..........;............... 358/138
`Ohki et a1.
`Emmons et al.
`379/53
`Appel et a1.
`379/53
`Black et a1.
`.. 364/514
`Ohtsuka et a1.
`379/53
`Hayden et al.
`370/62
`Ishikawa et al.
`379/53
`Komatsu et a1.
`379/53
`Kannes ......
`379/53
`'
`Weber et al.
`.. 379/53
`..
`Ashida et a1.
`370/62
`Ordish ................................. 364/900
`
`
`
`4,924,311
`4,932,047
`4,935,953
`4,942,540
`4,943,994
`4,953,159
`4,953,196
`4,962,521
`4,965,819
`4,995,071
`5,003,532
`5,034,916
`
`Petitioners HTC and LG - Exhibit 1034, p. 2
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`U.S. Patent
`
`Dec. 20, 1994
`
`Sheet 1 of 28
`
`5,375,068
`
`IO
`
`video /Camera
`30
`
`38
`
`Headphones
`with
`Microphone
`
`
`
`Headphones
`with
`Microphone 38
`
`Video
`
`ultimode
`Fl ber
`
`To other workstations
`
`Petitioners HTC and LG — Exhibit 1034, p. 3
`HTC and LG V. PUMA, IPR2()15—()15()2
`
`Petitioners HTC and LG - Exhibit 1034, p. 3
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`U.S. Patent
`
`Dec. 20, 1994
`
`Sheet 2 of 23
`
`5,375,068
`
`X..>3%_o
`
`
`
`On.OwD_>
`
`N.
`
`I%Ix:<_€onmm\\.o¢weon3«No.xom~_.
`.momnom<mm.o_>__»3I_00_89>
`
`
`_O...=.=U..__O
`
`oats...
`
`
`
`o_2aoma8_IIIIIIIIII,.._mm.._.>BPII.
`anIonI89>_mmqsomqx
`mmmméo_zmmmmmmam_mmmnmsmm2<E_89>
`_.6528.
`
`s_m._.m>mm:m
`
`._5
`
`ON..¢__OL#COO
`
`.96am:Em>m_»_:mm2
`oohS«NE
`
`mu._.._oEzooBoo:xmoafiz8;.mo.5889.omooommO
`
`
`
`
`
`x2::5....N.9“.
`
`Petitioners HTC and LG — Exhibit 1034, p. 4
`HTC and LG V. PUMA, IPR2()15—()15()2
`
`Petitioners HTC and LG - Exhibit 1034, p. 4
`HTC and LG v. PUMA, IPR2015-01502
`
`
`

`
`U.S. Patent
`
`Dec. 20, 1994
`
`Sheet 3 of 28
`
`5,375,068
`
`
`
`
`
`53
`
`DTP using 84
`or
`
`TCPIIP
`
`DEC
`
`1‘
`
`ne
`
`78
`
`
`
`
`
`A
`
`X server(video)
`.-
`_
`_
`X I IIXv using
`TCPIIP (local)
`
`i
`
`system calls
`
`62
`
`72
`
`
`
`
`-
`‘xmedia using
`“P9
`rcp/19 (local)
`so
`82
`
`
`_
`
`
`DTP usmg
`
`
`
`
`
`76
`
`
`TCPIIP (local)
`
`
`
`74
`
`I00
`
`II2
`
`no
`
`Machine 8
`
`04
`
`I06
`
`signal/execv
`
`I08
`
`FIG. 4
`
`Petitioners HTC and LG — Exhibit 1034, p. 5
`HTC and LG v. PUMA, IPR2()15—()15()2
`
`Petitioners HTC and LG - Exhibit 1034, p. 5
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`U.S. Patent
`
`Dec. 20, 1994
`
`Sheet 4 of 28
`
`5,375,068
`
`I20
`
`I02
`
`fl § I2b
`
`IIO
`
`A calls 8, B answers
`
`FIG. 5A
`
`I20
`
`'02
`
`l2b
`
`-—r’ H
`"20 j\\‘ ‘'0 A cells C, C answers
`H00
`'2C
`
`FIG. 5B
`
`I20
`
`‘O2
`
`[314-
`H2O
`‘—lg—o2b
`g
`
`'2
`
`b
`
`IIOD
`
`B cells C or A joins B and C
`
`FIG. 5C
`
`Petitioners HTC and LG — Exhibit 1034, p. 6
`HTC and LG v. PUMA, IPR2()15—()15()2
`
`Petitioners HTC and LG - Exhibit 1034, p. 6
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`U.S. Patent
`
`Dec. 20, 1994
`
`Sheet 5 of 23
`
`5,375,068
`
`ANALOG VIDEO
`NTSC/SECAM/PAL
`
`_ 200
`
`DIGITIZE VIDEO m
`FRAME BUFFER
`
`202
`
`VIDEO DATA m
`APPLICATION BUFFER
`
`_ 204
`
`VIDEO DATA IN
`NETWORK BUFFER
`
` 205
`
`VIDEO DATA
`RAvERsEs NETWORK
`
`203
`
`VIDEO DATA IN
`NETWORK BUFFER
`
`A
`
`2'0
`
`VIDEO DATA m
`APPLICATION BUFFER
`
`_ 2'2
`
`DIGITIZED VIDEO
`m FRAME BUFFER
`
`2|4
`
`‘
`
`memzeo VlDEO
`DISPLAYED
`
`2'0
`
`FIG. 6
`
`Petitioners HTC and LG — Exhibit 1034, p. 7
`HTC and LG V. PUMA, IPR2()15—()15()2
`
`Petitioners HTC and LG - Exhibit 1034, p. 7
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`U.S. Patent
`
`Dec. 20, 1994
`
`Sheet 6 of 28
`
`5,375,068
`
`LOCAL
`WORKSTATION A
`
`REMOTE
`WORKSTATION B
`
`
`
`INVOKE LOCAL
`DECspin
`APPLICATION
`
` 300
`
`NETWORK
`
`.332_______ _ _
`
`CREATE
`
`PORT
`
`304
`
`305
`" "’ " “‘ “ “‘ “"
`
`mvoxe LOCAL
`DECspind
`APPLICATION
`
`
`
`
`
`
`
`303
`
`
`
`
`
`3.0
`
`3I2
`
`I I
`
`3': _
`
`"I
`
`I
`
`- —- * - —
`
`CHE"CK fore
`OK
`
`_3_'_‘_‘_ __ __ _
`
`SEND
`
`"STARTHEADER"
`
`
`-MACHINE"
`
`SEND "QT"
`OR "ANSWER
`
`
`INVOKE LOCAL
`DEC spin
`
`APPLICATION
`
`Petitioners HTC and LG — Exhibit 1034, p. 8
`HTC and LG V. PUMA, IPR2()15—()15()2
`
`CHECK FOR
`ll
`II
`
`322
`
`,
`
`
`
`seem SENDING
`AUDIO/VIDEO
`
`DATA
`
`I
`
`I I
`
`524 I
`
`I I I L
`
`FIG. 7
`
`Petitioners HTC and LG - Exhibit 1034, p. 8
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`U.S. Patent
`
`Dec. 20, 1994
`
`Sheet 7 of 28
`
`5,375,068
`
`START
`
`OK
`
`OIOIOIOO = 0x54
`
`|
`
`OIOIOOII = 0x53
`
`Total length = 2 bytes
`
`ouoonou = 0x43
`
`|
`
`OlO0llll = Ox4F
`
`Total length = 2 bytes
`
`STARTHEADER
`
`5 reserved longwords (32 bits each)
`
`framerate desired (Iongword)
`
`DTP flags
`
`2 reserved longwords
`
`406
`
`4'2
`
`total number of frames
`
`-
`
`403
`
`l reserved Iongword
`
`video width
`
`video height
`
`bits per pixel (8 or 24)
`
`35 reserved longwords
`
`400
`
`402
`
`404
`
`78 longwords (used as login/ user handle)
`
`4‘°
`
`FIG. IO
`
`Petitioners HTC and LG — Exhibit 1034, p. 9
`HTC and LG v. PUMA, IPR2()15—()15()2
`
`Petitioners HTC and LG - Exhibit 1034, p. 9
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`U.S. Patent
`
`Dec. 20, 1994
`
`Sheet 8 of 28
`
`5,375,068
`
`QUIT
`
`‘
`
`OIOIOIOO = 0x54
`
`|
`
`OlOlOOOl = Ox5l
`
`Total length = Zbytes
`
`FIG. ll
`
`ANSWERMACHINE
`
`OIOOI IOI = 0x4D
`
`|
`
`OIOOOOOI = 0x4:
`
`Total length = 2 bytes
`
`FIG. I2
`
`CONTROL
`
`0x54
`
`|
`
`0x43
`
`45°
`
`452
`
`454
`
`455
`
`which
`
`length
`
`flags
`
`5 reserved longwords
`
`Total length is 34 bytes
`
`FIG. I3
`
`Petitioners HTC and LG — Exhibit 1034, p. 1()
`HTC and LG v. PUMA, IPR2()15—()15()2
`
`Petitioners HTC and LG - Exhibit 1034, p. 10
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`U.S. Patent
`
`Dec.20, 1994
`
`Sheet 9 of 28
`
`5,375,068
`
`NEXT VIDEO
`FRAME
`AVAILABLE
`
`
`
`SEND
`CONTENTS OF
`.
`
`AUDIO BUFFER
`
`
`
`350
`
`352
`
`360
`
`SEND VIDEO
`FRAME
`
`354
`
`>2 FRAME
`TIMES
`
`
`IIOKII
`
`
`
`SEND
`CONTENTS OF
`AUDIO BUFFER
`
`FIG. I4
`
`1 1
`Petitioners HTC and LG — Exhibit 1034, p.
`HTC and LG v. PUMA, IPR2()15—()15()2
`
`
`
`
`TIME OUT
`
`
`BUFFER TO
`LAST |Al2 SEC
`
`"OK"
`<2 FRAME
`
`CHECK FOR
`
`.
`
`II OKII
`
`IIOKII
`
`Petitioners HTC and LG - Exhibit 1034, p. 11
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`U.S. Patent
`
`Dec. 20, 1994
`
`Sheet 10 of 28
`
`5,375,068
`
`AUDIOHEADER
`
`0x55
`
`|
`
`OX4!
`
`4'4
`
`length
`
`Audio Flags
`
`6 reserved longwords
`
` FIG. I5
`
`VIDEOHEADER
`
`0x48
`
`l
`
`Ox4F
`
`42°
`
`Video width
`
`video Height
`
`Control Flags
`
`Timing Information
`
`4 reserved longwords
`
`Total Length is 34 bytes
`
`FIG. I6
`
`4'6
`
`4'8
`
`422
`
`424
`
`425
`
`423
`
`Petitioners HTC and LG — Exhibit 1034, p. 12
`HTC and LG v. PUMA, IPR2()15—()15()2
`
`Petitioners HTC and LG - Exhibit 1034, p. 12
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`U.S. Patent
`
`Dec. 20, 1994
`
`Sheet 11 of 28
`
`5,375,068
`
`964~xo<364
`
`
`
`wms_<Emm_2<ENE245$245
`
`.EN.:o.
`
`
`
`N..07..
`
`Ttlllllla
`us:I_ms_<EN
`
`
`
`zo_mm_s_mz<.Ezamm
`
`
`
`zo_.8w._._oozamm_mzqmn.mo
`
`Nms_<Eno
`
`
`
`"6zoiomjoozamm
`
`
`
`_us_<m.._pzummno
`
`Petmoners
`
`HTC and LG — Exhibit 1034, p. 13
`HTC and LG V. PUMA, IPR2()15—()15()2
`
`Petitioners HTC and LG - Exhibit 1034, p. 13
`HTC and LG v. PUMA, IPR2015-01502
`
`
`
`

`
`U.S. Patent
`
`Dec. 20, 1994
`
`Sheet 12 of 28
`
`5,375,068
`
`524
`
`Petitioners HTC and LG — Exhibit 1034, p. 14
`HTC and LG v. PUMA, IPR2()15—()15()2
`
`Petitioners HTC and LG - Exhibit 1034, p. 14
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`U.S. Patent
`
`Dec. 20, 1994
`
`Sheet 13 of 28
`
`5,375,068
`
`858
`
`FIG. 19
`
`Petitioners HTC and LG — Exhibit 1034, p. 15
`HTC and LG v. PUMA, IPR2()15—()15()2
`
`Petitioners HTC and LG - Exhibit 1034, p. 15
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`U.S. Patent T
`
`Dec. 20, 1994
`
`Sheet 14 of 28
`
`5,375,068
`
`600
`
`@540 624
`
`620 622
`
`FIG. 20
`
`Petitioners HTC and LG — Exhibit 1034, p. 16
`HTC and LG v. PUMA, IPR2()15—()15()2
`
`Petitioners HTC and LG - Exhibit 1034, p. 16
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`U.S. Patent
`
`Dec.20, 1994
`
`V Sheet 15 of 23
`
`5,375,068
`
`650
`
`Petitioners HTC and LG — Exhibit 1034, p. 17
`HTC and LG v. PUMA, IPR2()15—()15()2
`
`Petitioners HTC and LG - Exhibit 1034, p. 17
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`U.S. Patent
`
`Dec. 20, 1994
`
`Sheet 16 of 28
`
`5,375,068
`
`700
`
`706
`
`Petitioners HTC and LG — Exhibit 1034, p. 18
`HTC and LG v. PUMA, IPR2()15—()15()2
`
`Petitioners HTC and LG - Exhibit 1034, p. 18
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`U.S. Patent
`
`Dec. 20, 1994
`
`Sheet 17 of 28
`
`5,375,068
`
`¢NatmmOE
`
`wm\
`
`-
`
`mw3o._m
`
`mm»mm»Non..Eofioiom>_omAv
`
`oom
`
`5324
`
`cozotmcoemo
`
`:o_.o:u9_E_
`
`fozzoz
`
`.ccmmo_w
`
`8:9.3:8otnxmAv%o._co_9.m>E
`
`om~
`
`Petitioners HTC and LG — Exhibit 1034, p. 19
`HTC and LG V. PUMA, IPR2()15—()15()2
`
`Petitioners HTC and LG - Exhibit 1034, p. 19
`HTC and LG v. PUMA, IPR2015-01502
`
`
`
`
`

`
`U.S. Patent
`
`Dec.20, 1994
`
`V
`
`Sheet 18 of 28
`
`5,375,068
`
`0 I \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\1
`
`°
`IE} °
`
`°
`
`V2
`
`D
`
`I ~\\\\\\\\\\\\\\\\\\\\\\\\\\%\\\\\\\\Vt10
`‘%
`
`E
`
`ncxucznucsnu
`
`newsman 0
`aunaaaa cs
`
`El
`
`Petitioners HTC and LG — Exhibit 1034, p. 20
`HTC and LG v. PUMA, IPR2()15—()15()2
`
`WorkstationA
`
`Petitioners HTC and LG - Exhibit 1034, p. 20
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`U.S. Patent
`
`Dec. 20, 1994
`
`Sheet 199: 28
`
`5,375,068
`
`:5
`\.z
`(.0
`N.
`9
`LI.
`
`.0
`
`0r
`
`0
`
`°°
`5
`2
`+5-’.
`3;
`;
`
`Petitioners HTC and LG — Exhibit 1034, p. 21
`HTC and LG v. PUMA, IPR2()15—()15()2
`
`Petitioners HTC and LG - Exhibit 1034, p. 21
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`U.S. Patent
`
`Dec. 20, 1994
`
`Sheet 20 of 28
`
`5,375,068
`
`mco:2mfo>>
`
`
`
`Emu.o_.._
`
`non
`
`Petitioners HTC and LG — Exhibit 1034, p. 22
`HTC and LG V. PUMA, IPR2()15—()15()2
`
`Petitioners HTC and LG - Exhibit 1034, p. 22
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`U.S. Patent
`
`Dec. 20, 1994_
`
`Sheet 21 of 23
`
`5,375,068
`
`256xI84 WorkstationA
`
`O
`
`0 O
`
`.'J
`
`»\\\\\\\\\\\\\\\\\\\\\\\\\\\\\%VW§1
`O
`
`m
`
`
`
`FIG.26(d)
`
`Eeaaaana n
`D
`:3
`gfianannaa
`
`El
`
`am
`
`Petitioners HTC and LG — Exhibit 1034, p. 23
`HTC and LG v. PUMA, IPR2()15—()15()2
`
`Petitioners HTC and LG - Exhibit 1034, p. 23
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`U.S. Patent
`
`Dec.20, 1994
`
`Sheet 22 of 28
`
`5,375,068
`
` WorkstationC
`
`socFIG.26(e)
`
`Petitioners HTC and LG — Exhibit 1034, p. 24
`HTC and LG v. PUMA, IPR2()15—()15()2
`
`Petitioners HTC and LG - Exhibit 1034, p. 24
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`U.S. Patent
`
`Dec. 20, 1994
`
`Sheet 23 of 28
`
`5,375,068
`
`0co_S,§_._o2,
`
`38.0_:._
`
`com
`
`Petitioners HTC and LG — Exhibit 1034, p. 25
`HTC and LG V. PUMA, IPR2()15—()15()2
`
`Petitioners HTC and LG - Exhibit 1034, p. 25
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`U.S. Patent
`
`Dec. 20, 1994_
`
`Sheet 24 of 28
`
`5,375,068
`
`WorkstationA
`
`O
`
`o
`o
`
`A\\\\~\\\\\\\\\\\\\V\\\\\\\\\\\\\}I
`==
`
`OO—
`at \\\\\\\\\\\\N\\\\\\%$$\%\\W_
`N
`8
`O
`
`Qflaanaaaa
`‘r-U
`-—EEflEEEE
`to
`
`Petitioners HTC and LG — Exhibit 1034, p. 26
`HTC and LG v. PUMA, IPR2()15—()15()2
`
`Petitioners HTC and LG - Exhibit 1034, p. 26
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`U.S. Patent
`
`Dec. 20, 1994
`
`Sheet 25 of 28
`
`5,375,068
`
`m8:29:25
`
`EEEEH
`
`
`
`EEEEHEEEEH
`
`..o;m:<
`
`E8.o_:._8m
`
`Petitioners HTC and LG — Exhibit 1034, p. 27
`HTC and LG V. PUMA, IPR2()15—()15()2
`
`Petitioners HTC and LG - Exhibit 1034, p. 27
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`U.S. Patent
`
`Dec.20,1994j
`
`Sheet 26 of 28
`
`5,375,068
`
`U_5_.2e:§>
`
`Sam.9...
`
`com
`
`Petitioners HTC and LG — Exhibit 1034, p. 28
`HTC and LG V. PUMA, IPR2()15—()15()2
`
`Petitioners HTC and LG - Exhibit 1034, p. 28
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`U.S. Patent
`
`Dec. 20, 1994
`
`Sheet 27 of 28
`
`5,375,068
`
`m_co:o..mfo2,
`
`Petitioners HTC and LG — Exhibit 1034, p. 29
`HTC and LG V. PUMA, IPR2()15—()15()2
`
`Petitioners HTC and LG - Exhibit 1034, p. 29
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`U.S. Patent
`
`Dec. 20, 1994
`
`Sheet 28 of 28
`
`5,375,068
`
`fl
`
`3
`
`E
`
`08:_2e:2s
`
`3mm.0_:._
`
`com
`
`Petitioners HTC and LG — Exhibit 1034, p. 30
`HTC and LG V. PUMA, IPR2()15—()15()2
`
`Petitioners HTC and LG - Exhibit 1034, p. 30
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`1
`
`VIDEO TELECONFERENCING FOR
`NETWORKED WORKSTA'I‘IONS
`
`5,375,068
`
`BACKGROUND OF THE INVENTION
`
`Video communications has evolved over the years
`from a simple video telephone concept to a sophisti-
`cated network for allowing multiple users to participate
`in a video teleconference. For full featured video tele-
`conferencing, users require both an audio communica-
`tions path and a real time visual communication path
`synchronized to the audio path. Furthermore, it is desir-
`able to support full color video and telephone quality
`audio. Video teleconferencing capabilities are limited
`mainly by the bandwidth of the transmission medium
`connecting the teleconferencing terminals.
`Many computer workstations used in the office or
`laboratory environment today are connected with other
`workstations, file servers, or other resources over high-
`speed local area networks. Local area networks, in turn,
`are often connected together through high-speed speed
`gateways which connect workstations which may be
`distributed over a wide geographic area. Network wide
`protocols allow workstations to exchange packets of
`data at high rates of speed and reliability. Fixed band-
`width digital and analog video channels have been com-
`bined with computer networks to implement some
`video teleconferencing features. These include high
`bandwidth CATV/FDM type analog channels and
`fixed allocation TDM data channels for the video data.
`
`SUMMARY OF THE INVENTION
`
`Workstations today have obtained unprecedented
`computational power and utility. The powerful RISC
`type CPUs and fast, high resolution graphical displays
`have made possible multimedia workstations which
`integrate live audio and video into the programming
`environment. Graphical User Interface operating sys-
`tems (GUI) have allowed effective integration of audio
`and video into application programming.
`The present invention provides n-way video telecon-
`ferencing among networked computer workstations
`using the existing variable bandwidth digital data net-
`work for transferring synchronized audio and video
`teleconferencing data between the workstations. The
`teleconferencing apparatus and protocol of this inven-
`tion provides high quality video teleconferencing with-
`out the need for a guaranteed wide bandwidth analog
`video channel or a fixed allocation digital video chan-
`nel. Rather, the invention uses standard non-allocated
`data packets typically found on local area networks to
`transfer the audio and video teleconferencing data.
`Thus, no guaranteed bandwidth is required to carry on
`a useful video teleconference. An continuous audio data
`stream model provides continuous audio signals at the
`expense of video data when necessary, which is desir-
`able since the ear is more sensitive to a break in the
`audio data than the eye is to the loss of a frame of video
`data. A “push” data model provides a secure system by
`preventing remote workstations from activating an-
`other workstation’s video teleconferencing functions.
`In general, in one aspect, the invention features a
`video teleconferencing method and apparatus for com-
`puter workstations connected by a digital data network.
`The computer workstations include a transmission
`source means for a local workstation to send audio and
`video teleconference data across the network to one or
`more remote workstations, and, a receiver for the local
`
`5
`
`l0
`
`15
`
`20
`
`25
`
`30
`
`35
`
`45
`
`50
`
`55
`
`65
`
`_
`
`2
`workstation to receive audio and video teleconference
`data back from the remote workstations. The local
`workstation sends teleconference data to each of the
`remote workstations over a variable bandwidth digital
`data connection, and each of the remote workstations
`returns teleconference data back to the local worksta-
`tion over another variable bandwidth digital data con-
`nection. The variable bandwidth digital data connec-
`tions include the data packet oriented data channels
`associated with, for example, FDDI, DECnet, and
`Ethernet local area networks. Furthermore, a wide area
`digital network, such as ISDN, can also be used with
`the video teleconferencing apparatus and method of this
`invention.
`
`In preferred embodiments, the transmission source
`means includes a master software process executing on
`the local workstation, and the receiver includes a slave
`software process executing the remote workstation.
`The master software process formats and sends video
`teleconference data to the slave process. The slave pro-
`cess receives and reconstructs the audio and video tele-
`conference data for audible and visual reproduction,
`respectively. The video data is presented as an image on
`the display of the receiving workstation, while the
`audio data is sent to either amplified speakers or head-
`phones. In other preferred embodiments, the master
`process of a local workstation causes execution of a
`slave process on a remote workstation for receiving
`video teleconference data from the local workstation.
`The slave process running on the remote workstation in
`turn causes execution of a master process on the remote
`workstation for sending video teleconference data back
`to the local workstation. The master process of the
`remote workstation in turn causes execution of a slave
`process on the local workstation for receiving the video
`teleconference data sent by the master process of the
`remote workstation. The local workstation executes a
`slave process for each master process on a remote work-
`station sending video teleconference data to the local
`workstation.
`
`In yet other preferred embodiments, the transmission
`source includes an audio data transmitter for sending an
`audio data stream to the remote workstation such that
`the audio data can be reconstructed into a continuous
`audio signal. The transmission source also includes a
`video transmitter for sending video data to the remote
`workstation so that each frame of video data to be sent
`is inserted into the audio data stream without affecting
`the continuity of the reconstructed audio signal at the
`remote workstation.
`In yet other preferred embodiments the video trans-
`mitter precludes a frame of video data from being sent
`to the remote workstation if a system overload exists.
`The audio transmitter sends the audio data stream cor-
`responding to the precluded video frame to the remote
`workstation to prevent loss of continuity of the audio
`signal during a system overload. In other preferred
`embodiments the video transmitter precludes a frame of
`video data from being sent to the remote workstation in
`response to a system failure condition. The audio trans-
`mitter accumulates audio data for a predetermined time
`interval during the system failure condition, and trans-
`mits the accumulated audio data stream to the remote
`workstation once the failure has been corrected. For
`instance, the audio transmitter may continually accumu-
`late the last Q second of audio data while the failure
`exists, trimming any audio data older than § second.
`
`Petitioners HTC and LG — Exhibit 1034, p. 31
`HTC and LG v. PUMA, lPR2()l5—()l5()2
`
`Petitioners HTC and LG - Exhibit 1034, p. 31
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`3
`The last é second of audio data accumulated before the
`failure’s correction is sent to the remote workstation as
`soon as the failure is corrected. The most recently avail-
`able frame of video is then also sent.
`In still other preferred embodiments, timing informa-
`tion is attached to each frame of video data sent to the
`remote workstation. The timing information indicates a
`point in the continuous audio data stream which corre-
`sponds in time to the frame of video data. The receiver
`of the remote workstation includes a synchronizer for
`displaying a received frame of video when the point in
`the audio stream corresponding to the timing informa-
`tion of the received video frame is audibly reproduced
`at the remote workstation. The synchronizer counts the
`amount of audio data received in the continuous audio
`stream and compares the count to the timing informa-
`tion sent along with the most recently received video
`frame to determine when to display the frame.
`In general, in another aspect, the invention features a
`multimedia computer workstation, such as a RISC
`workstation or IBM PC, having video teleconferencing
`capabilities. The multimedia workstation of this inven-
`tion includes a network interface for establishing a vari-
`able bandwidth digital communications channel across
`a digital data network with another multimedia work-
`station. A video source provides a frame of digitized
`video data, and an audio source provides digitized audio
`data associated with the frame of video data. A data
`transmitter transmits the audio and video data through
`the network interface across the variable bandwidth
`digital communications channel to another workstation.
`A receiver receives audio and video data through the
`network interface across the variable bandwidth digital
`communications channel from another workstation.
`The workstation also includes means for displaying the
`received video data on the workstation display, and
`means for audibly reproducing the received audio data.
`In preferred embodiments, the video source includes
`a video camera, a video tape recorder, and/or a video
`laser disk player providing frames of analog video. A
`video frame grabber captures, digitizes, and stores each
`frame of analog video. The video source also includes
`digital video data stored in a file accessible by the work-
`station. A video compressor may compress the video
`data using JPEG or MPEG compression. The audio
`source includes a microphone for live audio, or pre-
`recorded audio corresponding to frames of pre-
`recorded video, from for instance a video tape recorder
`or laser disk. An audio digitizer digitizes and stores the
`audio using mu-law compression. The audio source also
`includes digital audio data stored in a file, preferably
`along with digital video data, accessible by the worksta-
`tion.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`The foregoing and other objects, features and advan-
`tages of the invention will be apparent from the follow-
`ing more particular description of preferred embodi-
`ments of the invention, as illustrated in the accompany-
`ing drawings in which like reference characters refer to
`the same parts throughout the different views. The
`drawings are not necessarily to scale, emphasis instead
`being placed upon illustrating the principles of the in-
`vention.
`
`FIG. 1 is a pictorial representation of the distributed
`computer system featuring multimedia workstations
`having video teleconferencing capabilities of this inven-
`tion.
`
`5,375,068
`
`4
`
`FIG. 2 is a block diagram showing a preferred em-
`bodiment of the multimedia workstation for implement-
`ing the video teleconferencing features of the distribu-
`tor computer system of FIG. 1.
`FIG. 3 is a block diagram showing the software envi-
`ronment for implementing a preferred embodiment of a
`DECspin video teleconferencing application for imple-
`menting the video teleconferencing features of the sys-
`tem of FIG. 1.
`'
`
`FIG. 4 is a block diagram showing the one-way con-
`nections established between two workstations to im-
`plement a two-way video teleconference.
`FIGS. 5a—5c are block diagrams illustrating the multi-
`ple one-way video teleconferencing connections estab-
`lished between three workstations joined in the same
`teleconference.
`
`FIG. 6 is a flowchart illustrating the flow of video
`during a video teleconference.
`FIG. 7 is a flowchart illustrating the teleconferencing
`protocol of this invention for establishing a video tele-
`conference connection between two workstations.
`FIG. 8 illustrates the format of the START message
`of the teleconferencing protocol of this invention.
`FIG. 9 illustrates the format of the OK message of the
`teleconferencing protocol of this invention.
`FIG. 10 illustrates the format of the START-
`HEADER message of the video teleconferencing pro-
`tocol of this invention.
`
`FIG. 11 illustrates the format of the QUIT message of
`the teleconferencing protocol of this invention.
`FIG. 12 illustrates the format of the ANSWERMA-
`CHINE message of the teleconferencing protocol of
`this invention.
`FIG. 13 illustrates the format of the CONTROL
`message of the video teleconferencing protocol of this
`invention.
`
`FIG. 14 is a flowchart showing the audio and video
`data transfer sequence of the video teleconferencing
`protocol of this invention.
`FIG. 15 shows the format of the AUDIOHEADER
`message of the video teleconferencing protocol of this
`invention,
`FIG. 16 shows the format of the VIDEOHEADER
`message of the video teleconferencing protocol of this
`invention,
`FIG. 17 shows a timing diagram of the audio and
`video data synchronization of the video teleconferenc-
`ing protocol of this invention.
`FIG. 18 shows the top level graphical user interface
`window for controlling a video teleconferencing ses-
`sion of this invention.
`
`FIG. 19 shows the graphical user interface window
`for displaying video data received from another work-
`station during a video teleconferencing session of this
`invention.
`
`FIG. 20 shows a second level graphical user call list
`interface window for establishing the video telecon-
`ferencing connections to other workstations of this
`invention.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`45
`
`50
`
`55
`
`FIG. 21 shows a second level control graphical user
`interface window for adjusting the parameters for a
`video teleconference of this invention.
`
`65
`
`FIG. 22 shows a second level monitor graphical user
`interface window for monitoring the parameters of a
`video teleconference of this invention.
`FIG. 23 shows a second level documentation graphi-
`cal user interface window for obtaining information
`
`Petitioners HTC and LG — Exhibit 1034, p. 32
`HTC and LG v. PUMA, lPR2()l5—()l5()2
`
`Petitioners HTC and LG - Exhibit 1034, p. 32
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`5
`about the operation and features of a video teleconfer-
`ence of this invention.
`
`5,375,068
`
`FIG. 24 shows a third level documentation graphical
`user interface window for obtaining information about a
`topic selected from the second level user interface win-
`dow of FIG. 23.
`FIG. 25 shows a graphical user interface window of
`a ring box for announcing a video teleconference call to
`another workstation to establish a video teleconference
`of this invention.
`FIGS. 26(a)—26(k) show the dislay screens of three
`Workstations participating in a three-way video tele-
`conference of this invention.
`
`DESCRIPTION OF THE PREFERRED
`EMBODIMENT
`
`FIG. 1 shows a distributed computer system 10, hav-
`ing a plurality of multimedia workstations 12 connected
`by a high speed digital network 14, which features
`n-way video teleconferencing of this invention. Each of
`the multimedia workstations 12 is capable of producing
`live audio and video data for transmission across the
`network to another multimedia workstation. Further,
`each of the multimedia workstations is capable of soni-
`cally reproducing the audio data and visually reproduc-
`ing the video data received from another workstation.
`Two or more of the networked workstations can
`participate in an n-way video teleconference using the
`teleconferencing protocol of this invention. The tele-
`conferencing protocol of this invention allows real time
`synchronized audio and video transmission across the
`network without the use of a fixed bandwidth or dedi-
`cated time slot transmission medium. Rather, this inven-
`tion provides high performance video teleconferencing
`features using standard digital network transport level
`protocols such as Internet TCP/IP and UDP/IP, or
`DECnet TM. The physical network link 14 should be a
`high speed FDDI (Fiber Distributed Data Interface)
`fiber optic link rimning at 100 MB/Sec. for maximum
`performance (about 15 uncompressed black and white
`frames per second), but can also be virtually any type of
`high quality digital network link such as an Ether-
`net TM. In the case of the FDDI network link, perfor-
`mance is typically limited by the workstation hardware
`and software, rather than by the throughput of the
`network. In addition, wide area networking (WAN)
`technologies, such as T1 and T3 digital carriers, can be
`used with the video teleconferencing protocol of this
`invention. These WAN technologies can be expected to
`achieve a video frame rate of at least about 12 Hz, as-
`suming black and white video images compressed with
`JPEG, MPEG, or another video compression tech-
`nique. The features of one preferred embodiment of this
`invention are commercially embodied in the DEC-
`spin TM product available from Digital Equipment
`Corporation, Maynard, Mass., the assignee of this pa-
`tent application.
`FIG. 2 shows a preferred embodiment of the multi-
`media workstation 12 for implementing the video tele-
`conferencing protocol of this invention. Workstation 12
`includes a high performance processor 20 connected to
`a large working memory 22 having 24 megabytes or
`more capacity, and a large hard drive having 1 giga-
`bytes or more capacity. A high performance backplane
`channel 26 connects the processor, memory, and hard
`disk to other devices in the workstation.
`
`The workstatio