_
`
`UIlltBd States Patent [19]
`Tung et a1.
`
`lllllllllllllllllllllllllllIlllllllllllllllllllllllllllllllllllIIIIIIIHIII
`[11] Patent Number:
`5,434,913
`[45] Date of Patent:
`Jul. 18, 1995
`
`USO05434913A
`
`[54] AUDIO SUBSYS'I'EM FOR
`COMPUTER-BASED CONFERENCING
`SYSTEM
`
`[75] Inventors: Peter Tung, Beaverton; Ben Vrvilo,
`Portland, both of Greg.
`
`[73] Assignee: Intel Corporation, Santa Clara, Calif.
`[21] A 1 N 158 246
`pp .
`0.:
`,
`
`OTHER PUBLICATIONS
`Computer Conferencing: IBM scientists demo proto
`type of affordable computer conferencing system, Nov.
`2’ 1992_ EDGE’ on & about AT&T’ V7, 11223’ p, 22_
`Primary Examiner—.larnes L. Dwyer
`Assistant Examiner-Scott Wolinsky
`Attorney, Agent, or Firm—-Steve Mendelsohn; William
`H. Murray
`
`[22] Filed:
`
`Nov. 24, 1993
`
`[57]
`
`ABSTRACI‘
`
`[56]
`
`An audi° task ,residing °n 3,11 audio/communicating
`[51] Int. 01.6 ............................................ .. HD4M 3/56
`[52] us. (:1. .................................. .. 379/202; 395/800; Pwd °f a“ “(11° ‘Subsystem “1 a COmPuF“ °°Y%fe“?“°'
`395/162
`ing system. An audio manager and an audio applications
`[58] Field of Search ______________ u 395/800’ 162; 379/202,
`programming interface reside on a host processor of: the
`379/205 203 204
`computer conferencing system. The audio task receives
`’
`’
`local analog audio signals, generates local compressed
`audio signals corresponding to the local analog audio
`signals, and passes the local compressed audio signals to
`' a communications subsystem of the computer confer
`encing system for transmission over a communications
`link to a remote computer conferencing system. The
`audio task receives remote compressed audio signals
`from the communications subsystem and generates re
`mote decompress“ audio Signals correspmding to the
`remote compressed audio signal for local playback.
`
`References Cited
`U S PATENT DOCUMENTS
`'
`‘
`379/202
`4,475,l93 10/1984 Brown ....................... ..
`4,333,795 12/1989 AndO e} 31
`-- 379/53
`""""" " 370/62
`5,014,267 5/1991 Tompi‘ms ct a1‘ "
`snuzuktllet a1‘ """"""""
`5Z231Z492 7/1993 Dangig:al:":::............12: 358/143
`5,315,633 5/1994
`5,319,793 6/1994
`5,335,321 8/1994
`
`102
`
`16 Claims, 32 Drawing Sheets
`
`102
`
`104
`
`CANERA
`
`MICROPHONE
`
`CANERA
`
`MICROPHONE
`
`ANALOG
`VIDEO
`
`V
`
`100
`1 CONFERENCING
`SYSTEN
`A
`
`ANALOG
`AUDIO
`
`V
`'
`
`ISDN NETWORK
`
`(DIGITAL)
`1 1 0
`
`ANALOG
`VIDEO
`
`V
`
`ANALOG
`AUDIO
`
`V
`
`100
`CONFERENCING f
`SYSTEN
`B
`
`ANALOG
`VIDEO
`
`V
`
`ANALOG
`AUDIO
`
`l
`
`ANALOG
`VIDEO
`
`l
`
`ANALOG
`AUDIO
`
`V
`
`KINITOR
`
`SPEAKER
`
`NONITOR
`
`SPEAKER
`
`106
`
`108
`
`106
`
`108
`
`CISCO SYSTEMS, INC. Ex. 1131 Page 1
`
`

`
`U.S. Pafent
`
`July 18, 1995
`
`Sheet 1 of 32
`
`5,434,913
`
`wzo=¢omu_=
`
`<¢m:<u
`
`mzo=mc¢uH=
`
`<zm=<u
`
`a.D_L
`
`VOH
`
`NOH
`
`vofi
`
`Ncfi
`
`co4<z<
`
`o~a=<
`
`co.<z<
`
`omcH>
`
`oc.<z<
`
`oH==<
`
`oo.<z<
`
`omaH>
`
`Oofi
`
`ozHuzmzwmzou
`
`
`
`xxoghmzzamfi
`
`m=mHm>m
`
`ofifi
`
`2:23
`
`<=mpm>m
`
`ozHuzmmww2ou
`
`Ocfi
`
`ao4<z<
`
`oHe=<
`
`oo4<2<
`
`ema_>
`
`co4<z<
`
`oHa=<
`
`co.<z<
`
`omaH>
`
`zm¥<w¢m
`
`mo__zo=
`
`xmx<m¢m
`
`¢ohHzo=
`
`mofi
`
`oofi
`
`mo“
`
`mod
`
`clsco SYSTEMS, Inc. Ex. 1131 Pagé 2
`
`CISCO SYSTEMS, INC. Ex. 1131 Page 2
`
`
`
`

`
`US. Patent
`
`July 18, 1995
`
`Sheet 2 of 32
`
`5,434,913
`
`wcm ma <2
`
`we“
`
`52% T 22 228532 is 22>
`EKR _ _
`
`2: o H ~ No“
`
`55 65>: 5:
`
`M26525; :5 5:6
`
`NQNR
`
`“058% 5%
`
`
`
`2%.. 2: 11+ “Maw:
`
`E
`
`Fvom
`
`CISCO SYSTEMS, INC. Ex. 1131 Page 3
`
`

`
`U.S. Patent
`
`July 18, 1995
`
`Sheet 3 of 32
`
`5,434,913
`
`vow
`
`SE 3 82> 25: 82> E:
`
`luv @558 :22 .2; “8&5;
`
`
`
`won mom won Non
`
`
`
`mg mg 22> 1.1.6:“
`
`2m <2
`
`BEBE
`
`65>:
`
`m2 <2
`
`wow
`
`CISCO SYSTEMS, INC. Ex. 1131 Page 4
`
`

`
`U.S. Patent
`
`July 18, 1995
`
`Sheet 4 of 32
`
`5,434,913
`
`mwfivmw
`
`mm¥<ugm
`
`MwAV.~
`
`53>:
`
`mam<mH
`
`m~A.mw
`
`.v.o_m
`
`MwA.,V
`
`CISCO SYSTEMS, INC. Ex. 1131 Page 5
`
`CISCO SYSTEMS, INC. Ex. 1131 Page 5
`
`

`
`US. Patent
`
`July 18, 1995
`
`Sheet 5 of 32
`
`Y 5,434,913
`
`F IG . 5 .
`
`100
`
`502
`
`AUDIO/VIDEO
`CONFERENCING APPN
`
`DATA CONFERENCING
`APPLICATION
`
`5 04
`
`506'\
`
`CONFERENCING API
`
`508
`‘\-
`
`5 1 6 ‘L
`
`VIDEO API
`
`1
`
`VIDEO
`MANAGER
`
`522
`‘1 VIDEO CAPTURE
`DRIVER
`
`I‘
`
`5 10
`in
`coMM API
`
`5 1 Z
`r‘,
`A_,
`AUDIO API
`
`5 1 4
`[J
`MAvE API
`
`coMM
`MANAGER
`g
`518
`
`AUDIO
`MANAGER
`
`520
`
`5 24
`
`‘J
`
`NAvE
`DRIVER
`
`5 26,
`
`HOST INTERFACE
`
`DSP INTERFACE
`
`530\_
`
`VIDEO
`NICROCODE
`
`5 28
`
`AUDIO
`5 3 8 f TASK
`
`BOARD DSP I532
`INTERFACE
`
`5 36
`%
`HOST DEVICE
`DRIVER
`
`5 34
`H
`NAME
`TASK
`
`cGNN
`TASK
`
`1'5 40
`
`CISCO SYSTEMS, INC. Ex. 1131 Page 6
`
`

`
`US. Patent
`
`July 18, 1995
`
`Sheet 6 of 32
`
`5,434,913
`
`wON
`
`mamlm
`
`E2: 25
`
`
`a 22:62 was: 25$
`
`
`was: as: 55:5 .52 :2
`
`:85 EU
`
`
`a 22:62 as? miss.
`
`
`8:: ~55 5:52 22 :3
`
`E: w x
`
`$22 2 \ E2
`
`.@ AIL
`
`855
`
`“z: SE8 2%
`
`5 E:
`
`5 .NM
`
`22: F
`2e: -H
`5c I
`
`a E 922
`
`E8 052 55:5 2.2 v.2 5m
`
`
`
`55528.5 552 ,2 52m : _
`
`3
`
`5%
`
`. 5.0m
`
`51:2
`
`ma <2
`
`wON
`
`CISCO SYSTEMS, INC. Ex. 1131 Page 7
`
`

`
`U.S. Patent
`
`July 18, 1995
`
`Sheet 7 of 32
`
`5,434,913
`
`FIG. 7.
`
`502.504
`
`
`
`706
`
`CONFERENCING PRINITIVE DISPATCHER
`
`CONFERENCING
`
`
`
`
`
`CALLBACK
`
`7711()
`
`coMM
`
`CALLBACK
`
`coMM
`
`PRIMITIVE
`
`VIDEO
`
`AUDIO
`
`PRIMITIVE
`
`PRIMITIVE
`
`716
`
`720
`
`coMM
`
`MANAGER
`
`VIDEO
`
`MANAGER
`
`AUDIO
`
`MANAGER
`
`518
`
`516
`
`520
`
`CISCO SYSTEMS, INC. Ex. 1131 Page 8
`
`CONFERENCING APPLICATION
`
`CONFERENCING FINITE
`
`STATE MACHINE
`
`CONFERENCING
`
`PRINITIVE VALIDATOR
`
`CISCO SYSTEMS, INC. Ex. 1131 Page 8
`
`

`
`U.S. Patent
`
`July 13, 1995
`
`Sheet 8 of 32
`
`5,434,913
`
`m~m~.~»:G
`
`NMQNVQ
`
`x:<\®..E2
`.Q3951
`
`UmQumixmcb
`
`mEm\QOLQ
`
`@202:8
`
`:oU“.0m.s.m&
`
`ummoov.
`
`:00
`
`\Eumxo:
`
`238::Cqamcatm
`
`S
`
`uqmoovsA:32§.>..em
`
`:239?:EE
`
`PMam5..M_._..
`
`xc<\\C..321
`Qzmcclm
`
`be9omixmcbP
`msm\noci%Qfi.
`
`CISCO SYSTEMS, INC. Ex. 1131 Page 9
`
`
`
`
`
`
`

`
`US. Patent
`
`July 18, 1995
`
`Sheet 9 of 32
`
`5,434,913
`
`ngmbui
`
`
`
`.WSQQKBM. QQGQQ
`
`\oLummQ
`2QoLm>oomLtb
`
`$520k
`
`muLaommm
`
`A9395
`mgoLmioomLcb
`
`.uLbIQb.
`
`moLaommt
`
`\SEQBQESW
`
`\obcou
`
`@Eq?mbé?
`3.0339963
`Loicoz
`
`omkqoum.\tgmx
`
`IESQ
`Qocm.
`
`\o?cou
`
`mmbol
`
`0. .UNK
`
`CISCO SYSTEMS, INC. Ex. 1131 Page 10
`
`

`
`US. Patent
`
`July 18, 1995
`
`Sheet 10 of 32
`
`5,434,913
`
`
`
`ERFMEM. v3 v5» "3% MEQSQK
`
`33>
`
`335 §
`
`32§§
`5e
`
`QB» ESQSQ.
`t3
`
`CISCO SYSTEMS, INC. Ex. 1131 Page 11
`
`

`
`US. Patent
`
`July 18, 1995
`
`Sheet 11 of 32
`
`5,434,913
`
`F IG . 1 1 .
`
`5 1 6
`
`1 102
`
`VIDEO MANAGER
`LIBRARY
`
`[nu]
`
`SENDMESSAGE
`INTERFACE
`
`1104
`
`
`‘\- VCAPT [EXE]
`
`1 106
`
`H
`
`VPLAY [5x51
`
`m0
`
`I
`
`Rm
`
`I
`
`1 108
`
`\
`
`NETW
`[on] m
`
`REcvv
`
`8
`1 1 10
`
`xm
`
`Vf'W: VIDEO CAPTURE DRIVER
`PLAYBACK (CODEC)
`
`1.522 I
`
`CISCO SYSTEMS, INC. Ex. 1131 Page 12
`
`

`
`U.S. Patent
`
`July 18, 1995
`
`Sheet 12 of 32
`
`5,434,913
`
`FIG. 12.
`
`I1]
`
`CISCO SYSTEMS, INC. Ex. 1131 Page 13
`
`

`
`U.S. Patent
`
`July 18, 1995
`
`Sheet 13 of 32
`
`5,434,913
`
`
`
`newaxiom9.;ZQWN
`
`SNswamcm
`
`kmEmQmi\&Q
`
`
`
`Q~Qb<Bmmtm
`
`
`
`moimqQMQQU
`
`&m\_3.5
`
`.m-u~EI
`mw~.
`
`QnI¢nQQQQ¢II
`
`qmmmmmq:8
`
`Q\Q\:\
`
`MEEma.6328
`
`E3
`
`\Qv~Q\Qb<
`
`3?‘MES:
`
`NKW
`
`Em
`
`9%
`
`mm
`
`moimq
`
`m.§,.\mm
`
`Go:.3has
`
`mmmm......
`
`O00COO0000IO0O0OOO0000OOIooononoontoooooolaoouloo000000
`
`CISCO SYSTEMS, INC. Ex. 1131 Page 14
`
`CISCO SYSTEMS, INC. Ex. 1131 Page 14
`
`
`
`
`
`
`
`
`

`
`U.S. Patent
`
`July 18, 1995
`
`Sheet 14 of 32
`
`5,434,913
`
`32E«E.=..
`
`22E«E.2.‘
`
`s<§-E.m.Mb<9»«GREezmm
`
`252.3Mfibkkwb$63
`
`A2.E§Lmumxmmmw.33.:av‘
`«.?B\:€L3..cox?
`
`
`
`«:8
`
`LOQ.~:o§LmNm...%..€
`
`VN.U~n~
`
`«.xo.o\Co»Lou..:95».
`
`clsco SYSTEMS, Inc. Ex. 1131 Page 15
`
`CISCO SYSTEMS, INC. Ex. 1131 Page 15
`
`
`
`

`
`U.S. Patent
`
`July 18, 1995
`
`Sheet 15 of 32
`
`5,434,913
`
`kbE:VNR
`
`Qfiwfimk
`
`..3&b&&vG
`
`QVNKQB
`
`EmNNQWN
`
`
`
`o§£mmE.£Scam.
`
`
`
`kmS.~LQw3o«uSm
`
`
`
`osfimmstbtmQQ<
`
`
`
`.>§.ZQm3o«uE.w
`
`
`
`8:35:o..mm...Cn8u
`
`gmboumb:3
`
`
`
`93.35=o..mm.....a§u
`
`seven:8
`
`QUOUib\3\<_
`
`
`
`qmm\:o..mm...Eqa.:.w\ocumIas
`
`Qmm\
`
`:o..mm...CQq:.w\ocum.
`
`
`
`.>§.CQm3c«u3..w
`
`
`
`L.§.CQm3o«oSm.
`
`
`
`L3F.\Qm.\Lmx.$.L3Q..\Q.w\..$x.$
`
`
`
`
`
`
`
`33:3§:mxE38:32C8é..E.§s%:EQ33:
`
`%...B§\eak%ubc.§\2%
`
`
`
`L.§.CQm3u«oB,m.Lm>.CQm3c«uS.w
`
`
`
`
`
`sco SYSTEMS, Inc. Ex. 1131 Page 16
`
`QN65%82
`
`SnK
`
`85
`
`3.
`
`
`
`
`
`
`
`Lm<..CQmo.§....QL..§.CQ.8.§mq
`
`CISCO SYSTEMS, INC. Ex. 1131 Page 16
`
`
`
`
`
`
`
`
`

`
`U.S. Patent
`
`July 18, 1995
`
`Sheet 16 of 32
`
`5,434,913
`
` 1%?
`
`.§§8mm52%mmmi>52zqfl.©N.U~.RQ-I
`
`
`
`
`
`mmimqGEE
`
`89
`
`uzqamm
`
`me?
`
`Q5%:kb<9»«Q0.
`
`Qv.m&I.~mkbxQ«G
`
`kw<9SQIME
`
`mm»:1«GIWVE
`
`
`..w&.§Emm..<.§asHt§..$§§§Qxufim
`
`.m.:%8§§%.%;-xufim
`
`
`
`..--:22a§§§u.41.3mm§.§c.§d....<.§<§a§d..%Emmm
`
`9%
`
`E::365¢
`
`xm«R3.369
`
`momN
`
`9%
`
`mm:5.5.623
`
`
`
`No.935Non;vow;
`
`
`3%:x..1I
`
`
`
`
`
`zwtulmmcdim...E.3o.§.P...\_EBmam<
`
`2§uu§d:a..3%:2:EQ3.S%2g_¢8..<Q%§:8msS%2§8mmzm<atEm?
`
`CISCO SYSTEMS, INC. Ex. 1131 Page 17
`
`CISCO SYSTEMS, INC. Ex. 1131 Page 17
`
`
`
`
`
`
`
`
`
`

`
`US. Patent
`
`July 18, 1995
`
`Sheet 17 of 32
`
`5,434,913
`
`F IG . 17 .
`
`UPPER
`LAYERS
`
`HOST (moons)
`m.o
`202 \ LL
`DULDLL
`
`RDULDLL
`
`532 ‘\-
`
`DSP INTERFACE
`
`comm (0 cm) 0m com TASKS
`
`:1 cm DRIVER
`
`s CHAN DRIVERS
`
`mum/com
`BOARD (Sm)
`206
`
`CISCO SYSTEMS, INC. Ex. 1131 Page 18
`
`

`
`U.S. Patent
`
`July 18, 1995
`
`Sheet 18 of 32
`
`5,434,913
`
`~~~~~ —~
`
`-
`
`3%
`
`In
`
`N 3
`
`
`
`\_.Ll_ ___
`H
`
`g
`
`E‘
`
`E
`
`O
`
`<z
`‘-2
`Q
`
`___U_'_'______
`II
`II
`
`II
`
`Q .,
`II «v
`
`LT}
`<3
`<o§&
`2
`..
`00$
`‘W:=_u~,2,
`II
`’ 5
`03
`§ “‘
`
`"
`H g
`
`I‘
`
`H 3
`
`I!
`
`H
`
`&
`<t
`E
`1:
`II Q
`II
`
`,
`
`,
`
`cu
`
`8
`
`
`
`"
`‘$5
`,2,
`5
`Q
`
`co
`N‘
`.
`E
`R‘
`
`an
`L314
`2m2m
`§*5
`03
`~14
`QT‘!-
`EE.<
`ES‘;
`um:
`
`!
`
`1
`
`:
`*
`II
`
`1!
`’’T:_ "‘_
`
`-
`
`00
`rq
`
`‘°
`
`gé
`o§g
`903%CISCO YSTEMS, INC. Ex. 1131 Page 19
`
`CISCO SYSTEMS, INC. Ex. 1131 Page 19
`
`

`
`US. Patent
`
`July 18, 1995
`
`Sheet 19 0f 32
`
`5,434,913
`
`END sass/01v
`
`BEG 1N SESSION
`
`REJECT c0/v/v
`
`/
`/
`/
`
`\\ c0/v/v REJEC TED
`\
`
`corwv
`REQUEST
`
`REQUEST
`CONN
`
`I
`
`/
`/
`/
`CL OSE CONN
`/
`/ / co/v/v ACCEPTED
`/
`
`CONNCTED
`
`\
`\ \CHAN REJEC r50
`\
`
`ACCEPT CONN
`
`REJECT CHAN
`
`@ OPEN
`ACCEPT
`CHAN
`
`CL OSE
`CHAN
`
`CHAN
`
`1
`// CHAN
`, ACCEPTED
`
`-—— ACTIVE REQUEST 0 STATE
`“""-' RESULT OF REQUEST
`
`FIG. 19
`
`CISCO SYSTEMS, INC. Ex. 1131 Page 20
`
`

`
`U.S. Patent
`
`July 18, 1995
`
`Sheet 20 of 32
`
`5,434,913
`
`LNCOMTNG
`CALL
`
`Conn Requested
`
`
`
`OUTGOLNC
`CALL
`
`
`APP calls Tl!
`MakeCannectian
`
`
`
`RESP
`
`Conn
`
`Accepted
`
`
`
`Contra] Channel
`handshake
`
`£3L%%l9£D
`
`.I?[(;. E?()
`
`Segd Ccnnectian
`Nbtification to APP
`
`stab ished
`
`CISCO SYSTEMS, INC. Ex. 1131 Page 21
`
`
`
`Send Connection
`DLM
`Rea.
`to APP
`MbkeCbnn
`
`
`
`AWAIT LOCAL
`
`AWAIT REMONE
`RESP
`
`Call
`APP Accepts
`Progress
`
`DLM.AcceptConn
`
`
`
` AWMIT ACCEPT
`
`RESP
`
`Conn Accepted
`
`
`
`CISCO SYSTEMS, INC. Ex. 1131 Page 21
`
`

`
`U.S. Patent
`
`July 18, 1995
`
`Sheet 21 of 32
`
`5,434,913
`
`
`
`AWMlT'Alive
`
`Message
`
`Receive heartbeat
`
`CH_ESD%1L9£D
`
`(connection in CONN_ESTAB state)
`
`FIG. 21
`
`CISCO SYSTEMS, INC. Ex. 1131 Page 22
`
`(Cbnnection in CONWLAL/ME state)
`
`DLM qpen channel 0
`
`AW%lT CTL
`
`OPEN
`
`Channel open notification
`
`Send heartbeat
`
`CISCO SYSTEMS, INC. Ex. 1131 Page 22
`
`

`
`U.S. Patent
`
`July 18, 1995
`
`Sheet 22 of 32
`
`5,434,913
`
`RECEIVE
`CHANNEL
`
`SEND
`CHANNEL
`
`Chan Request CTL Msg.
`A/lac Ill and OLA! chan
`
`APP calls 7I I OpenChannel
`A/lac TI] and DUI chan
`
`DLMOpen
`
`DLMOpen
`
`C’-WLAMAI/'_OU/.0PfLRX
`
`O‘WLAWAlf_OUL09‘UX
`
`DLM Channel Opened
`Open RDLM Channel
`Send Chan Req to APP
`
`0”, Channe/ Opened
`Send Open CTL Msg.
`
`
`
`AWA/7 LOCAL RESP
`
`AWAIT REM RESP
`
`APP Accepts
`Send accept 671 4459-
`
`Receive accept CTL Alsg.
`Open RD/.M Channel
`
`Send Chan Accepted to APP
`
`FIG. 22
`
`CISCO SYSTEMS, INC. Ex. 1131 Page 23
`
`CISCO SYSTEMS, INC. Ex. 1131 Page 23
`
`

`
`U.S. Patent
`
`July 18, 1995
`
`Sheet 23 of 32
`
`5,434,913
`
`:3wumtcouQQmufix
`
`:3mmmmEmum
`
`RNQVZVE2:8
`
`
`
`
`
`MflekshzEHQVZVEEEQU
`
`:o..mmm,.w:..mum
`
`
`
`
`
`MbV.m~E:2~3369.5<k~>m.a>:3:50
`
`
`
`toxcocuamumxmamE8.‘.§:.§<\<8:o::c5L3m.§ok
`
`
`
`
`
`
`
`
`
`Ecééfiqs538.:.§<2$GEccuficmqe
`
`c:59..~monkoQUQEQMW
`
`I.l|.IuI\|.\
`11/|\I\|‘
`
`.§uc%c%o§m.s.$. ..,.:n:§Sc%8m..gem
`
`EccocgqmuuwTwtcccuuqmouvxc
`
`
`
`Lm>o:cI:EBL.£m..mmkam>o:oI:cc,uL3m..mm.<s
`
`
`
`
`
`mwe\Qsoumx..~meekEmmsoq
`
`
`
`:o.:8.C.Zo:“:96maocogucxmwlililuln
`
`
`
`
`
`
`
`Lmmatas5:092:8
`
`
`
`
`
`u...toé«noimzlllullli.28EoéE33Sammmuuzma........in
`
`
`
`
`
`
`
`mm65%
`
`YSTEMS, INC. Ex. 1131 Page 24
`
`CISCO SYSTEMS, INC. Ex. 1131 Page 24
`
`

`
`U.S. Patent
`
`July 18, 1995
`
`Sheet 24 of 32
`
`5,434,913
`
`FIG. 24.
`
`(BITS)
`
`(FIELD)
`
`32
`
`32
`
`32
`
`32
`
`32
`
`32
`
`32
`
`32
`
`32
`
`32
`
`16
`
`16
`
`0-
`
`|pData
`
`dwBufferLength
`
`dwBytesUsed
`
`dwTimeCaptured
`
`dwUser
`
`dwFIags
`
`dwReserved[O]
`
`dwReserved[1]
`
`dwReserved[2]
`
`dwReserved[3]
`
`Type
`
`Message
`
`Data ...
`
`CISCO SYSTEMS, INC. Ex. 1131 Page 25
`
`CISCO SYSTEMS, INC. Ex. 1131 Page 25
`
`

`
`U.S. Patent
`
`July 18, 1995
`
`Sheet 25 of 32
`
`5,434,913
`
`.FIG. 25.
`
`(BITS)
`
`(FIELD)
`
`Sti I lThresh(|ow)/
`Fi lterThresh(high)
`
`8 8 8
`
`8
`88
`88
`
`8 8 8 8 8
`
`8-
`
`clsco SYSTEMS, Inc. Ex. 1131 Page 26
`
`CISCO SYSTEMS, INC. Ex. 1131 Page 26
`
`

`
`U.S. Patent
`
`July 18, 1995
`
`Sheet 26 of 32
`
`5,434,913
`
`FIG. 26.
`
`(BITS)
`
`(FIELD)
`
`30
`
`1
`
`1
`
`0-
`
`Timestamp
`
`Reserved
`
`Data
`
`Mute
`
`gm...:..m=|
`
`clsco SYSTEMS, Inc. Ex. 1131 Page 27
`
`CISCO SYSTEMS, INC. Ex. 1131 Page 27
`
`

`
`U.S. Patent
`
`July 18, 1995
`
`Sheet 27 of 32
`
`5,434,913
`
`FIG. 27.
`
`(BITS)
`
`(FIELD)
`
`
`
`CISCO SYSTEMS, INC. Ex. 1131 Page 28
`
`CISCO SYSTEMS, INC. Ex. 1131 Page 28
`
`

`
`U.S. Patent
`
`July 18, 1995
`
`Sheet 28 of 32
`
`5,434,913
`
`1
`
`H
`D
`
`L
`C
`
`DA
`
`T A
`
`1
`
`LC FLAG
`
`1
`
`H
`E
`
`A
`D
`
`g
`
`1
`
`HDLD CRC
`
`I '... ’.._(1
`
`W1F"1G3
`
`FIG. 28.
`
`(BITS)
`
`8
`
`s
`
`8
`
`8
`
`1
`
`1
`
`2
`
`4
`
`0-
`
`16
`
`®
`
`clsco SYSTEMS, Inc. Ex. 1131 Page 29
`
`CISCO SYSTEMS, INC. Ex. 1131 Page 29
`
`

`
`U.S. Patent
`
`July 18, 1995
`
`Sheet 29 of 32
`
`5,434,913
`
`TII-OLM SESSION & CONNECTION ESTABLISHMENT
`
`TI 1
`
`DLM
`
`DLM
`
`TI 1
`
`Ol_M_BeginSession
`
`DLAl__HakeConnect Ion
`
`.4’?
`
`COMVLESTABL ISIE
`
`DLA{__Begin$ession
`
`DLALL ist en
`
`\ ‘ -5.
`
`C0/VALREOUESTEO
`
`DL/.I__Accep t Connect ion
`
`CONN_ES7.431. /SHED
`
`TI I —OLM CONNECTION & SESSION TEARDOMV
`
`T] I
`
`DLM
`
`DLM
`
`OLALCIoseconnect ion
`
`4‘?
`
`COMLCLOSLCOAPLETE
`
`DUL5nd59$$l'0f7
`
`\
`
`CO/W_Cl_OSE_NOT/FY
`
`Ol.l«LEnd5‘ess ion
`
`SESS_CL OSEO
`
`5ESS_CL OSEO 9
`
`FIG. 29
`
`CISCO SYSTEMS, INC. Ex. 1131 Page 30
`
`CISCO SYSTEMS, INC. Ex. 1131 Page 30
`
`

`
`U.S. Patent
`
`July 18, 1995
`
`Sheet 30 of 32
`
`5,434,913
`
`an
`05%
`
`mkbxPE\:8?Q\mEQkvfimbx.
`
`mam32
`
`Q.13¢‘Q\Q\:\
`
`QZQUIMEQ
`
`a\.Qk<1Swe<:._EmK2~
`
`
`ExamExmwxmm
`
` <2KnownE:55xmmwEIIIiIIIIIIIIIIIlIIII!1u_
`
`__
`
`7
`
`mu<.»<E2\R6
`
`,.:m<U\._
`
`95nth.a>5:<32
`
`mzmsm.\M
`
`mmommmm
`
`__
`
`_I
`
`3%miGm
`
`,m..Qh<k
`
`NMXN
`
`
`
`h.\\slmmm
`
`II|IIlIIIIIIIIIIIIIIIIll.
`
`mE«Gm.300<
`
`.2209M»3QQ\<
`
`Qmg12k%>5:<32
`
`WZMSMxm
`
`mmommmm
`
`abGmm.EC
`
`9.5:mnxm
`
`m.c.:.m
`
`NMXN
`
`_____________
`
`CISCO SYSTEMS, INC. Ex. 1131 Page 31
`
`CISCO SYSTEMS, INC. Ex. 1131 Page 31
`
`

`
`U.S. Patent
`
`July 18, 1995
`
`Sheet 31 of 32
`
`5,434,913
`
`Nm.
`
`Em
`
`md2z$c-m‘\mmm3,m.1m2QIl
`mufimbiNm
`
`x92950.51
`
`md22§oxm‘GmmEm-m>:4
`m§Q~E>:E
`
`B2%3%
`
`
`
`85%;:Exam1338:QExmd8<
`
`
`
`
`
`.§E..§Ed22§u-mmemésmm
`
`Ema.5
`
`mwmzm\omm
`
`mamQ\\Q<..w\
`
`
`
`
`
`EmmQmxmwit.3:mB<.EE2\mamo\\n1bkm.mQgQ
`
`
`
`
`
`
`
`m.§,SE2..G~>\,3m<m...:m<bx.
`
`CISCO SYSTEMS, INC. Ex. 1131 Page 32
`
`CISCO SYSTEMS, INC. Ex. 1131 Page 32
`
`

`
`U.S. Patent
`
`July 18, 1995
`
`Sheet 32 of 32
`
`5,434,913
`
`N3
`
`M24IIEs
`k\\$9:4
`
`$2233
`
`nminoommmfi
`
`
`
`QsN.:<\mmmpfimE_\QQQQV
`
`
`
`SE28.\\\....>§§
`
`otllmnoooooocoooocxmgm:»8§§EQ8HEmqfim
`
`hxdmweEé§eE§
`
`9% It%.IES:Ekbxyx
`
`«WKK\\QKWv$.Q
`§8Sm5”%§<E>=IEExams
`
`uzxmu.
`
`.328
`
`Emmfi
`
`myWW
`
`CISCO SYSTEMS, INC. Ex. 1131 Page 33
`
`CISCO SYSTEMS, INC. Ex. 1131 Page 33
`
`
`
`
`

`
`1
`
`5,434,913
`
`2
`FIG. 4 is a block diagram of the hardware configura-
`tion of the audio/comm board of the conferencing sys-
`tem of FIG. 2;
`FIG. 5 is a block diagram of the software configura-
`tion of the conferencing system of each PC system of
`FIG. 1;
`FIG. 6 is a block diagram of a preferred embodiment
`of the hardware configuration of the audio/comm
`board of FIG. 4;
`FIG. 7 is a block diagram of the conferencing inter-
`face layer between the conferencing applications of
`FIG. 5, on one side, and the com, video, and audio
`managers of FIG. 5, on the other side;
`FIG. 8 is a representation of the conferencing call
`finite state machine (FSM) for a conferencing session
`between a local conferencing system (i.e., caller) and a
`remote conferencing system (i.e., callee);
`FIG. 9 is a representation of the conferencing stream
`FSM for each conferencing system participating in a
`conferencing session;
`FIG. 10 is a representation of the video FSM for the
`local video stream and the remote video stream of a
`conferencing system during a conferencing session;
`FIG. 11 is a block diagram of the software compo-
`nents of the video manager of the conferencing system
`of FIG. 5;
`FIG. 12 is a representation of a sequence of N walk-
`ing key flames;
`FIG. 13 is a representation of the audio FSM for the
`local audio stream and the remote audio stream of a
`conferencing system during a conferencing session;
`FIG. 14 is a block diagram of the architecture of the
`audio subsystem of the conferencing system of FIG. 5;
`FIG. 15 is a block diagram of the interface between
`the audio task of FIG. 5 and the audio hardware of
`audio/comm board of FIG. 2;
`FIG. 16 is a block diagram of the interface between
`the audio task and the com task of FIG. 5;
`FIG. 17 is a block diagram of the com subsystem of
`the conferencing system of FIG. 5;
`FIG. 18 is a block diagram of the comm subsystem
`architecture for two conferencing systems of FIG. 5
`participating in a conferencing session;
`FIG. 19 is a representation of the comm subsystem
`application FSM for a conferencing session between a
`local site and a remote site;
`FIG. 20 is a representation of the comm subsystem
`connection FSM for a conferencing session between a
`local site and a remote site;
`FIG. 21 is a representation of the comm subsystem
`control channel handshake FSM for a conferencing
`session between a local site and a remote site;
`FIG. 22 is a representation of the com subsystem
`channel establishment FSM for a conferencing session
`between a local site and a remote site;
`FIG. 23 is a representation of the comm subsystem
`processing for a typical conferencing session between a
`caller and a callee;
`FIG. 24 is a representation of the structure of a video
`packet as sent to or received from the com subsystem
`of the conferencing system of FIG. 5;
`FIG. 25 is a representation of the compressed video
`bitstream for the conferencing system of FIG. 5;
`FIG. 26 is a representation of a compressed audio
`packet for the conferencing system of FIG. 5;
`FIG. 27 is a representation of the reliable transport
`comm packet structure;
`
`AUDIO SUBSYSTEM FOR COMPUTER-BASED
`CONFERENCING SYSTEM
`
`BACKGROUND OF THE INVENTION
`
`1. Field of the Invention
`The present invention relates to audio/video confer-
`encing, and, in particular, to systems for real-time audio,
`video, and data conferencing in windowed environ-
`ments on personal computer systems.
`2. Description of the Related An
`It is desirable to provide real-time audio, video, and
`data conferencing between personal computer (PC)
`systems operating in windowed environments such as
`those provided by versions of Microsoft® Windows
`operating system. There are difficulties, however, with
`providing real-time conferencing in non-real-time win-
`dowed environments.
`
`It is accordingly an object of this invention to over-
`come the disadvantages and drawbacks of the known
`art and to provide real-time audio, video, and data con-
`ferencing between PC systems operating in non-real-
`time windowed environments.
`
`It is a particular object of the present invention to
`provide real-time audio, video, and data conferencing
`between PC systems operating under a Microsoft®
`Windows operating system.
`Further objects and advantages of this invention will
`become apparent from the detailed description of a
`preferred embodiment which follows.
`SUMMARY OF THE INVENTION
`
`The present invention is an audio subsystem for a
`computer conferencing system. An audio task resides
`on an audio/communications board of the computer
`conferencing system. An audio manager and an audio
`applications programming interface reside on a host
`processor of the computer conferencing system. The
`audio task receives local analog audio signals, generates
`local compressed audio signals corresponding to the
`local analog audio signals, and passes the local com-
`pressed audio signals to a communications subsystem of
`the computer conferencing system for transmission
`over a communications link to a remote computer con-
`ferencing system. The audio task receives remote com-
`pressed audio signals from the communications subsys-
`tem and generates remote decompressed audio signals
`corresponding to the remote compressed audio signal
`for local playback.
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`Other objects, features, and advantages of the present
`invention will become more fully apparent from the
`following detailed description of the preferred embodi-
`ment,
`the appended claims, and the accompanying
`drawings in which:
`FIG. 1 is a block diagram representing real-time
`point-to-point audio, video, and data conferencing be-
`tween two PC systems, according to a preferred em-
`bodiment of the present invention;
`FIG. 2 is a block diagram of the hardware configura-
`tion of the conferencing system of each PC system of
`FIG. 1;
`FIG. 3 is a block diagram of the hardware configura-
`tion of the video board of the conferencing system of
`FIG. 2;
`
`5
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`45
`
`50
`
`55
`
`60
`
`65
`
`CISCO SYSTEMS, INC. Ex. 1131 Page 34
`
`CISCO SYSTEMS, INC. Ex. 1131 Page 34
`
`

`
`3
`FIG. 28 is a representation of the unreliable transport
`comm packet structure;
`FIG. 29 are diagrams indicating typical connection
`setup and teardown sequences;
`FIGS. 30 and 31 are diagrams of the architecture of
`the audio/comm board; and
`FIG. 32 is a diagram of the audio/comm board envi-
`ronment.
`
`DESCRIPTION OF THE PREFERRED
`EMBODIMENT(S)
`
`Point-To-Point Conferencing Network
`
`Referring now to FIG. 1, there is shown a block
`diagram representing real-time point-to-point audio,
`video, and data conferencing between two PC systems,
`according to a preferred embodiment of the present
`invention. Each PC system has a conferencing system
`100, a camera 102, a microphone 104, a monitor 106,
`and a speaker 108. The conferencing systems communi-
`cate via an integrated services digital network (ISDN)
`110. Each conferencing system 100 receives, digitizes,
`and compresses the analog video signals generated by
`camera 102 and the analog audio signals generated by
`microphone 104. The compressed digital video and
`audio signals are transmitted to the other conferencing
`system via ISDN 110, where they are decompressed
`and converted for play on monitor 106 and speaker 108,
`respectively. In addition, each conferencing system 100
`may generate and transmit data signals to the other
`conferencing system 100 for play on monitor 106. In a
`preferred embodiment, the video and data signals are
`displayed in different windows on monitor 106. Each
`conferencing system 100 may also display the locally
`generated video signals in a separate window.
`Camera 102 may be any suitable camera for generat-
`ing NSTC or PAL analog video signals. Microphone
`104 may be any suitable microphone for generating
`analog audio signals. Monitor 106 may be any suitable
`monitor for displaying video and graphics images and is
`preferably a VGA monitor. Speaker 108 may be any
`suitable device for playing analog audio signals and is
`preferably a headset.
`
`Conferencing System Hardware Configuration
`Referring now to FIG. 2, there is shown a block
`diagram of the hardware configuration of each confer-
`encing system 100 of FIG. 1, according to a preferred
`embodiment of the present invention. Each conferenc-
`ing system 100 comprises host processor 202, video
`board 204, audio/comm board 206, and industry stan-
`dard architecture (ISA) bus 208.
`Referring now to FIG. 3, there is shown a block
`diagram of the hardware configuration of video board
`204 of FIG. 2, according to a preferred embodiment of
`the present invention. Video board 204 comprises ISA
`bus interface 310, video bus 312, pixel processor 302,
`video random access memory (VRAM) device 304,
`video capture module 306, and video analog-to-digital
`(A/D) converter 308.
`Referring now to FIG. 4, there is shown a block
`diagram of the hardware configuration of audio/comm
`board 206 of FIG. 2, according to a preferred embodi-
`ment of the present invention. Audio/comm board 206
`comprises ISDN interface 402, memory 404, digital
`signal processor (DSP) 406, ISA bus interface 408, and
`audio input/output (I/O) hardware 410.
`
`5,434,913
`
`4
`
`5
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`45
`
`50
`
`55
`
`60
`
`65
`
`Conferencing System Software Configuration
`
`Referring now to FIG. 5, there is shown a block
`diagram of the software configuration each conferenc-
`ing system 100 of FIG. 1, according to a preferred
`embodiment of the present invention. Video microcode
`530 resides and runs on pixel processor 302 of video
`board 204 of FIG. 3. Com task 540 and audio task 538
`reside and run on DSP 406 of audio/comm board 206 of
`FIG. 4. All of the other software modules depicted in
`FIG. 5 reside and run on host processor 202 of FIG. 2.
`
`Video, Audio, and Data Processing
`Referring now to FIGS. 3, 4, and 5, audio/video
`conferencing application 502 running on host processor
`202 provides the top-level local control of audio and
`video conferencing between a local conferencing sys-
`tem (i.e., local site or endpoint) and a remote conferenc-
`ing system (i.e., remote site or endpoint). Audio/video
`conferencing application 502 controls local audio and
`video processing and establishes links with the remote
`site for transmitting and receiving audio and video over
`the ISDN. Similarly, data conferencing application 504,
`also running on host processor 202, provides the top-
`level local control of data conferencing between the
`local and remote sites. Conferencing applications 502
`and 504 communicate with the audio, video, and comm
`subsystems using conferencing application program-
`ming interface (API) 506, video API 508, com API
`510, and audio API 512. The functions of conferencing
`applications 502 and 504 and the APIs they use are
`described in further detail later in this specification.
`During conferencing, audio I/O hardware 410 of
`audio/comm board 206 digitizes analog audio signals
`received from microphone 104 and stores the resulting
`uncompressed digital audio to memory 404 via ISA bus
`interface 408. Audio task 538, running on DSP 406,
`controls the compression of the uncompressed audio
`and stores the resulting compressed audio back to mem-
`ory 404. Comm task 540, also running on DSP 406, then
`formats the compressed audio format for ISDN trans-
`mission and transmits the compressed ISDN-formatted
`audio to ISDN interface 402 for transmission to the
`remote site over ISDN 110.
`ISDN interface 402 also receives from ISDN 110
`compressed ISDN-formatted audio generated by the
`remote site and stores the compressed ISDN-formatted
`audio to memory 404. Comm task 540 then reconstructs
`the compressed audio format and stores the compressed
`audio back to memory 404. Audio task 538 controls the
`decompression of the compressed audio and stores the
`resulting decompressed audio back to memory 404. ISA
`bus interface then transmits the decompressed audio to
`audio I/O hardware 410, which digital-to-analog
`(D/A) converts the decompressed audio and transmits
`the resulting analog audio signals to speaker 108 for
`play.
`-
`Thus, audio capture/compression and decompres-
`sion/playback are preferably performed entirely within
`audio/comm board 206 without going through the host
`processor. As a result, audio is preferably continuously
`played during a conferencing session regardless of what
`other applications are nmning on host processor 202.
`Concurrent with the audio processing, video A/D
`converter 308 of video board 204 digitizes analog video
`signals received from camera 102 and transmits the
`resulting digitized video to video capture module 306.
`Video capture module 306 decodes the digitized video
`
`CISCO SYSTEMS, INC. Ex. 1131 Page 35
`
`CISCO SYSTEMS, INC. Ex. 1131 Page 35
`
`

`
`5,434,913
`
`5
`into YUV color components and delivers uncom-
`pressed digital video bitmaps to VRAM 304 via video
`bus 312. Video microcode 530, running on pixel proces-
`sor 302, compresses the uncompressed video bitmaps
`and stores the resulting compressed video back to
`VRAM 304. ISA bus interface 310 then transmits via
`
`ISA bus 208 the compressed video to host interface 526
`nmning on host processor 202.
`Host interface 526 passes the compressed video to
`video manager 516 via video capture driver 522. Video
`manager 516 calls audio manager 520 using audio API
`512 for synchronization information. Video manager
`516 then time-stamps the video for synchronization
`with the audio. Video manager 516 passes the time-
`stamped compressed video to communications (comm)
`manager 518 using comm application programming
`interface (API) 510. Comm manager 518 passes the
`compressed video through digital signal processing
`(DSP) interface 528 to ISA bus interface 408 of audio/—
`comm board 206, which stores the compressed video to
`memory 404. Comm task 540 then formats the com-
`pressed video for ISDN transmission and transmits the
`ISDN-formatted compressed video to ISDN interface
`402 for transmission to the remote site over ISDN 110.
`ISDN interface 402 also receives from ISDN 110
`
`ISDN-formatted compressed video generated by the
`remote site system and stores the ISDN-formatted com-
`pressed video to memory 404. Comm task 540 recon-
`structs the compressed video format and stores the
`resulting compressed video back to memory 404. ISA
`bus interface then transmits the compressed video to
`com manager 518 via ISA bus 208 and DSP interface
`528. Comm manager 518 passes the compressed video
`to video manager 516 using comm API 510. Video
`manager 516 decompresses the compressed video and
`transmits the decompressed video to the graphics de-
`vice interface (GDI) (not shown) of Microsoft ® Win-
`dows for eventual display in a video window on moni-
`tor 106.
`
`For data conferencing, concurrent with audio and
`video conferencing, data conferencing application 504
`generates and passes data to com manager 518 using
`conferencing API 506 and com API 5 10. Comm
`manager 518 passes the data through board DSP inter-
`face 532 to ISA bus interface 408, which stores the data
`to memory 404. Comm task 540 formats the data for
`ISDN transmission and stores the ISDN-formatted data
`back to memory 404. ISDN interface 402 then transmits
`the ISDN-formatted data to the remote site over ISDN
`110.
`ISDN interface 402 also receives from ISDN 110
`ISDN-formatted data generated by the remote site and
`stores the ISDN-formatted data to memory 404. Comm
`task 540 reconstructs the data format and stores the
`resulting data back to memory 404. ISA bus interface
`408 then transmits the data to com manager 518, via
`ISA bus 208 and DSP interface 528. Comm manager
`518 passes the data to data conferencing application 504
`using comm API 510 and conferencing API 506. Data
`conferencing application 504 processes the data and
`transmits the processed data to Microsoft ® Windows
`GDI (not shown) for display in a data window on moni-
`tor 106.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`45
`
`50
`
`55
`
`Preferred Hardware Configuration for Conferencing
`System
`Referring again to FIG. 2, host processor 202 may be
`any suitable general-purpose processor and is preferably
`
`65
`
`6
`an Intel ® processor such as an Intel ® 486 micro-
`processor. Host processor 202 preferably has at least 8
`megabytes of host memory. Bus 208 may be any suitable
`digital communications bus and is preferably an Indus-
`try Standard Architecture (ISA) PC bus.
`Referring again to FIG. 3, video A/D converter 308
`of video board 204 may be any standard hardware for
`digitizing and decoding analog video signals that are
`preferably NTSC or PAL standard video signals. Video
`capture module 306 may be any suitable device for
`capturing digital video color component bitmaps and is
`preferably an Intel ® ActionMedia ® II Capture Mod-
`ule. Video capture module 306 preferably captures
`video as subsampled 4:1:l YUV bitmaps (i.e., YUV9 or
`YVU9). Memory 304 may be any suitable computer
`memory device for storing data during video processing
`such as a random access memory (RAM) device and is
`preferably a video RAM (VRAM) device with at least
`1 megabyte of data storage capacity. Pixel processor
`302 may be any suitable processor for compressing
`video data and is preferably an Intel ® pixel processor
`such as an Intel ® i750® Pixel Processor. Video bus
`312 may be any suitable digital communications bus and
`is preferably an Intel ® DVI ® bus. ISA bus interface
`310 may be any suitable interface between ISA bus 208
`and video bus 312, and preferably comprises three In-
`tel ® ActionMedia ® Gate Arrays and ISA configura-
`tion jumpers.
`Referring now to FIG. 6, there is shown a block
`diagram of a preferred embodiment of the hardware
`configuration of audio/comm board 206 of FIG. 4. This
`preferred embodiment comprises:
`Two 4—wire S-bus RJ-45 ISDN interface connectors,
`one for output to ISDN 110 and one for input from
`ISDN 110. Part of ISDN interface 402 of FIG. 4.
`
`Standard bypass relay allowing incoming calls