United States Patent [19]
`Smith et ai.
`
`111111
`
`1111111111111111111111111111111111111111111111111111111111111
`US006128649A
`[11] Patent Number:
`[45] Date of Patent:
`
`6,128,649
`Oct. 3, 2000
`
`[54] DYNAMIC SELECTION OF MEDIA
`STREAMS FOR DISPLAY
`
`XP002068243 & JP 08317366 A (Nippon Denki Eng KK)
`Nov. 29, 1996.
`
`[75]
`
`Inventors: Keith M. Smith; Russell W. Pretty,
`both of Ottawa, Canada
`
`[73] Assignee: Nortel Networks Limited, Montreal,
`Canada
`
`[21] Appl. No.: 08/867,624
`
`[22]
`
`Filed:
`
`Jun. 2, 1997
`
`[51]
`[52]
`[58]
`
`[56]
`
`Int. CI? ....................................................... H04N 7/14
`U.S. CI. ................................ 709/217; 348/15; 348/17
`Field of Search .................. 348/14-20; 395/200.47,
`395/200.34; 379/202; 370/260, 235, 468
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`4,531,024
`5,003,532
`5,374,952
`5,382,972
`5,392,223
`5,473,367
`5,491,797
`5,544,313
`5,574,934
`5,615,338
`5,657,096
`5,682,460
`5,737,011
`5,760,838
`5,768,280
`5,768,527
`5,778,187
`
`7/1985 Colton et al. ............................... 179/2
`3/1991 Ashida et al.
`............................ 370/62
`12/1994 Flohr ......................................... 348/12
`1/1995 Kannes ...................................... 348/15
`2/1995 Caci ... ... ... ... ... .... ... ... ... ... ... ...... 364/514
`12/1995 Bales et al. .... ... ... ... ... .... ... ... ..... 348/13
`2/1996 Thompson et al. ................ 395/200.34
`8/1996 Schachnai et al.
`................ 395/200.01
`11/1996 Mirashrafi et al. .. ... ... ... .... ... ... 395/800
`3/1997 Poole et al. .......................... 395/200.4
`8/1997 Lukacs .. ... ... ... .... ... ... ... ... ... .... .... 348/15
`10/1997 Hyziak et al. ..................... 395/200.47
`4/1998 Lukacs .. ... ... ... .... ... ... ... ... ... .... .... 348/15
`6/1998 Adams et al.
`.......................... 348/460
`6/1998 Way ........................................ 370/486
`6/1998 Zhu et al.
`.......................... 395/200.61
`7/1998 Monteiro et al. .................. 395/200.61
`
`FOREIGN PATENT DOCUMENTS
`
`0581 101 A1
`0724362 A1
`0669765 A2
`WO 96/14711
`
`7/1993
`1/1995
`2/1995
`11/1995
`
`European Pat. Off .......... H04N 7/14
`European Pat. Off .......... H04N 7/15
`European Pat. Off .......... H04N 7/24
`WIPO .............................. H04N 7/14
`
`OlliER PUBLICATIONS
`
`Database WPI, Section EI, Week 9707, Derwent Publica(cid:173)
`tions Ltd., London, GB; Class WOl,AN 97-071568,
`
`Herng-Yow Chen, Ja-Ling Wu, MultiSync: A Synchroni(cid:173)
`zation Model for Multimedia Systems, IEEE Journal on
`Selected Areas in Communications, vol. 14, No.1, Jan.
`1996.
`
`Cosmos Nicolaou, An Architecture for Real-Time Multime(cid:173)
`dia Communication Systems, IEEE Journal on Selected
`Areas in Communications, vol. 8, No.3, Apr. 1990.
`
`Xiaobao Chen, End-to-End Synchronization Control Sup(cid:173)
`port for Multiservice Applications, Computer laboratory,
`University of Cambridge, Nov. 1995.
`
`Ralf Steinmetz, Gerald Blakowski, A Media Synchroniza(cid:173)
`tion Survey: Reference Model, Specification, and Case
`Studies, IEEE Journal on Selected Areas in Communica(cid:173)
`tions, vol. 14, No.1, Jan. 1996.
`
`Primary Examiner-Andrew I. Faile
`Assistant Examiner-Reuben Brown
`Attorney, Agent, or Firm-C. W. Junkin
`
`[57]
`
`ABSTRACT
`
`One or more streams from a number of real-time video
`streams available to be transmitted across a communications
`network, are selected for display on respective terminals of
`a first, and at least one other user. First, a policy of the first
`user for making the selection, is determined, the policy
`containing conditions to be evaluated, selectable by the first
`user. Then, which streams to select for the first user is
`determined dynamically by evaluating the current condi(cid:173)
`tions according to the first user's policy. Only the selected
`streams are passed for display on the terminal of the first
`user, independently of selections made for passing to the
`other users. Dynamic selection from multiple streams
`enables the user to concentrate on the content not the form
`of presentation. Individual control of what is displayed,
`independent of what other users see, enables tailoring to
`video conferences to be handled more easily.
`
`19 Claims, 28 Drawing Sheets
`
`D
`I ----1 LOCAL NETWORK CONGESTION
`
`ALARMS
`
`-----i WORKSTATION RESOURCE~
`
`OO~~""
`[Q] ~~"D'~'~"cw
`
`~-~ SAFERESQURCESIATE
`
`STATE
`
`* CRITICAL RESOURCE
`
`CISCO Exhibit 1001, pg. 1
`
`

`
`u.s. Patent
`
`Oct. 3, 2000
`
`Sheet 1 of 28
`
`6,128,649
`
`.. -... -~
`
`MCU
`SELECT/MUX
`
`MEDIA
`STREAMS
`FROM
`USERS
`
`CONTROL
`
`MIXEDI
`SELECTED
`STREAM
`
`" " "
`
`NETWORK
`(POINT TO POINT)
`
`~2
`
`H
`
`~ ,
`
`n
`
`.4~
`
`Jl
`
`~ r
`
`CONT ROL
`
`~r
`
`USER
`
`USER
`
`USER
`
`-
`
`)
`3
`
`)
`3
`
`-7
`
`3
`
`PRIOR ART
`
`FIG. 1
`
`CISCO Exhibit 1001, pg. 2
`
`

`
`u.s. Patent
`
`Oct. 3, 2000
`
`Sheet 2 of 28
`
`6,128,649
`
`LAN NETWORK
`(WITH MULTICAST CAPABILITY)
`
`~4
`
`h
`
`USER
`
`USER
`
`"
`
`USER
`
`3
`
`3
`
`PRIOR ART
`
`FIG. 2
`
`10 '\
`
`USER
`
`-....
`
`.. ....
`
`11,\
`
`OTHER USER AWARENESS
`INFORMATION
`
`VIDEO SELECTION REQUEST
`
`-.. NETWORK
`
`PACKET
`
`SELECTED VIDEO
`
`PRIOR ART
`
`FIG. 3
`
`-~
`
`3 .......--....., OTHER
`USERS
`
`CISCO Exhibit 1001, pg. 3
`
`

`
`u.s. Patent
`
`Oct. 3, 2000
`
`Sheet 3 of 28
`
`6,128,649
`
`10 '\
`
`USER
`
`12\
`
`OTHER USER AWARENESS
`INFORMATION
`
`~ -
`
`SELECTION
`POLICY
`....
`
`--
`
`DYNAMIC
`SELECTION
`CONTROLLER
`13~
`
`SELECTION
`REQUESTS_
`
`..
`
`NETWORK
`
`--
`
`SELECTED MEDIA
`
`+
`3 ......---... OTHER
`USERS
`
`FIG. 4
`
`CISCO Exhibit 1001, pg. 4
`
`

`
`u.s. Patent
`
`Oct. 3, 2000
`
`Sheet 4 of 28
`
`6,128,649
`
`20 -----
`
`MEDIA IN
`E.G. VIDEO RECEPTION
`PROCESSES AND AUDIO
`RECEPTION, NETWORK
`MONITORING
`
`-
`
`22 -----
`
`11r
`
`COMMAND
`AND
`CONTROL
`
`--...
`
`1r
`
`WORKSTATION
`RESOURCE
`MONITOR
`
`MEDIA OUT
`~~-.t E.G. VIDEO & AUDIO
`TRANSMISSION
`
`... .
`
`NETWORK
`24
`
`FIG.5a
`
`CISCO Exhibit 1001, pg. 5
`
`

`
`u.s. Patent
`
`Oct. 3, 2000
`
`Sheet 5 of 28
`
`6,128,649
`
`26
`27
`D-STREAM PROCESSING 28 r 0-
`
`VIDEO RECEPTION
`INTEGRITY MONITOR
`
`I
`
`
`Cp
`
`r -
`
`J MEDIA
`
`-
`
`IGMP, RSVP
`
`SYNC
`,-,---------------------
`NETWORK FEEDBACK
`MO N ITOR (N FM)
`
`'----+------l~. I
`:. P
`'------+-------1 .. ~ C CONGESTION ANALYSIS 31
`MONITOR (CAM)
`
`30 \0lil-___ - -1
`--
`
`RTCP
`
`NETWORK
`
`l'
`
`AUDIO RECEPTION
`
`ACTIVITY MONITOR
`
`32
`33 :::::
`I
`:::::
`P T-STREAM PROCESSING 34 --
`--
`C
`NETWORK SIGNALING 35
`
`,.
`-
`
`RTP/IP MEDIA STREAM(S)
`
`IGMP, RSVP
`
`..
`--
`
`COMMAND & CONTROL 36
`
`I CONFERENCE AWARENESS 37 - AWARENESS MESSAGES ..
`-
`--
`38 0
`
`P
`C
`
`-
`...
`
`1~
`
`..
`
`MOM
`
`GUI
`
`39
`
`NETWORKSIGNALING 29 rlr-' :I'll I I
`
`I-
`
`I- __ RTP/IP MEDIA STREAM(S)
`-
`I
`/
`
`/
`
`--
`
`I
`
`CISCO Exhibit 1001, pg. 6
`
`

`
`u.s. Patent
`
`Oct. 3, 2000
`
`Sheet 6 of 28
`
`6,128,649
`
`STREAM
`
`PROCESSING -
`-~ - T-STREAM
`--- ACTIVITY
`-
`
`I
`
`J
`
`MONITOR
`(TAM)
`
`MEDIA
`STREAMS
`
`I
`
`I
`D-STREAM
`INTEGRITY
`MONITOR
`(DIM)
`
`46,,\
`
`33)
`
`27)
`
`WORKSTATION
`RESOURCE
`MONITOR
`(WRM)
`
`I;:
`
`LOCAL \
`... CONGESTION
`
`ONGESTIONf
`ANALYSIS
`MONITOR
`(CAM)
`
`-
`MOM
`FOREIGN
`'------,----./ L'" CONGESTION,.-___ -'-------,
`/ ""\
`NETWORK
`38
`FEEDBACK
`MONITOR
`(NFM)
`
`/
`30
`
`39~
`
`Q \ "
`
`GUI
`(VIEWING
`POLICY)
`
`FEEDBACK
`MESSAGES
`CONNECTION MEMBERSHIP
`SIGNALING
`
`CONFERENCE AWARENESS MESSAGES
`
`..
`
`.. ...
`... .. ... .. -
`
`FIG. 6
`
`/
`24
`NETWORK
`
`--.. ---
`
`CISCO Exhibit 1001, pg. 7
`
`

`
`u.s. Patent
`
`Oct. 3, 2000
`
`Sheet 7 of 28
`
`6,128,649
`
`60 \
`
`GUI EVENT
`
`61 \
`.. UPDATE STATE
`
`JIll"
`
`62\
`
`LOCK/UNLOCK
`~
`STREAM
`
`63\
`
`"
`RESOLVE
`VIOLATION
`IFANY
`
`64\
`
`-..
`
`UPDATE
`DISPLAY
`
`FIG. 7
`
`CISCO Exhibit 1001, pg. 8
`
`

`
`u.s. Patent
`
`Oct. 3, 2000
`
`Sheet 8 of 28
`
`6,128,649
`
`70
`
`71
`
`GUI EVENT
`
`RESPOND TO EVENT
`PERFORM
`UPDATE
`LOCK/UNLOCK SEITINGS
`
`SETTINGS
`
`LOCK/UNLOCK
`
`72
`
`77
`
`VISUAL
`MANAGE
`
`DETECT VIOLATION
`CONSTRAINT #1
`(REAL ESTATE/STREAMS)
`
`NO VIOLATION
`
`VIOLATED!
`
`73
`
`CONFORMANCE
`
`UNDISPLAY SILENT
`D_STREAM(S) UNTIL
`CONFORMANCE WITH C1
`
`NON
`CONFORMANCE
`74
`
`UNDISPLAY LEAST ACTIVE
`D_STREAM(S) TILL
`CONFORMANCE WITH C1
`
`CONFORMANCE
`
`NON
`CONFORMANCE
`
`75
`
`FIG. 8
`
`CISCO Exhibit 1001, pg. 9
`
`

`
`u.s. Patent
`
`Oct. 3, 2000
`
`Sheet 9 of 28
`
`6,128,649
`
`100\
`
`NO
`
`DETECT VIOLATIONS 14~t--------,
`
`YES
`
`'r
`
`RESOLVE VIOLATIONS
`
`102~
`
`\ .
`
`DISPLAY ADDITIONAL
`STREAMS IF
`RESOURCES PERMIT
`
`-
`
`UPDATE STATE AND DISPLAY
`
`FIG. 9
`
`CISCO Exhibit 1001, pg. 10
`
`

`
`u.s. Patent
`
`Oct. 3, 2000
`
`Sheet 10 of 28
`
`6,128,649
`
`1 4 - - - - - - - - - - - , FIG. 10
`
`CHECK
`FLOCK
`
`SET
`>--------------;~
`
`CLEAR
`
`108
`VIOLATED! r---------'----~
`UNDISPLAY ALL
`I------'---a.t D STREAMS
`VIOLATING
`CONSTRAINT #3
`
`111
`
`110
`
`VISUAL
`MANAGE
`
`106
`
`107
`
`DETECT VIOLATION
`CONSTR. #3 = CRITICAL
`(FOREIGN NET. CON G.)
`
`112
`
`NO VIOLATION
`
`DETECT VIOLATION
`CONSTR. #4 = CRITICAL
`(NET. LOCAL CONG.)
`
`VIOLATED!
`
`UNDISPLAY THE
`ONE D_STREAM
`r----I~ WITH THE MOST
`SILENCE
`
`DONE
`
`116
`
`NO VIOLATION
`
`114
`
`IF NONE ARE
`SILENT
`
`DETECT VIOLATION
`CONSTR. #2 = CRITICAL
`(BANDWIDTHITARIFFS) VIOLATED!
`
`117
`
`NO VIOLATION
`
`DETECT VIOLATION
`CONSTR. #5 = CRITICAL 1---__ - - '
`(COMP. RESOURCES)
`VIOLATED!
`
`118
`
`NO VIOLATION
`
`DETECT
`UNDER-UTILIZATION
`CONSTR. #1
`CONSTR. #2 = SAFE
`CONSTR. #4 = SAFE
`CONSTR. #5 = SAFE
`
`ALL SAFE
`
`120
`
`SOME VIOLATION
`
`UNDISPLAY THE
`ONE D_STREAM
`WITH THE LEAST
`ACTIVITY
`
`115
`
`IF NONE ARE
`ACTIVE
`
`119
`
`DISPLAY THE MOST
`ACTIVE D _STREAM
`(F _C3 MUST BE
`CLEAR)
`
`DONE
`
`IF NONE ARE ACTIVE
`~--~--~ DONE
`DISPLAY THE LEAST
`SILENT D STREAM
`(F _C3 MUST BE
`CLEAR)
`
`IF NONE ARE
`SILENT
`
`CISCO Exhibit 1001, pg. 11
`
`

`
`u.s. Patent
`
`Oct. 3, 2000
`
`Sheet 11 of 28
`
`6,128,649
`
`140 \
`
`START TRIGGERED BY
`NEW T-STREAM ACTIVITY
`
`141\
`
`"
`
`DETECT
`CONSTRAINT
`VIOLATION
`
`NO
`
`142\
`1r
`
`DISPLAY NEW
`STREAM
`
`...
`--
`
`(
`
`143
`
`YES
`
`..
`....
`
`TEST MONITOR
`TRANSMISSION
`WITHOUT DISPLAY
`
`OK
`
`NOTO K
`
`•
`
`1 44
`DON'T DISPLAY
`NEW STREAM ~
`
`FIG. 11
`
`CISCO Exhibit 1001, pg. 12
`
`

`
`u.s. Patent
`
`Oct. 3, 2000
`
`Sheet 12 of 28
`
`6,128,649
`
`NEW T_STREAM
`ACTIVITY EVENT r-----~
`
`CHECK
`F _LOCK
`F C3
`
`NOT CLEAR
`
`140
`
`VIOLATED
`
`148
`
`DETECT VIOLATION
`CONSTRAINT #1
`(REAL ESTATE/
`NUM. STREAMS)
`
`NO VIOLATION
`
`149
`
`NOT SAFE
`
`DETECT
`UNDER-UTILIZA TION
`CONSTR.#2 = SAFE
`CONSTR.#4 = SAFE
`CONSTR.#5 = SAFE
`
`157
`
`DISPLAY NEW
`D_STREAM
`
`ALL
`SAFE
`
`153
`
`~~~ __ ~~~FOUND
`SEARCH FOR MOST
`SILENT D STREAM
`TO REPLACE
`(WITH> RESOURCES)
`
`~------~
`
`UNDISPLA Y SILENT
`I---'-~ D_STREAM, DISPLAY
`NEWD STREAM
`
`151
`
`NO
`CRITICAL
`155
`(WARNINGS
`_____ ----..L...-.-.,.POSSI BLE) ~-----'------....
`VERIFY NO
`TEST MODE
`CRITICAL STATE
`1. INVISIBLE DISPLAY OF
`IN
`NEW ACTIVE 0 STREAM
`CONS TRA INT#2
`2. MONITOR {C2,C4,C5}
`CONS TRA IN T#4
`FOR SHORT TERM
`CONS TRA INT#5
`CRITICAL VIOLATION
`
`NO
`CRITICAL
`(COMMIT TO
`DISPLAY)
`
`154
`
`CRITICAL
`VIOLATION
`
`FIG. 12
`
`CRITICAL
`VIOLATION
`
`REMOVE NEW
`D_STREAM
`
`VISUAL
`MANAGE
`
`156
`
`152
`
`CISCO Exhibit 1001, pg. 13
`
`

`
`u.s. Patent
`
`Oct. 3, 2000
`
`Sheet 13 of 28
`
`6,128,649
`
`200 \
`
`START ON
`T-STREAM SILENCE EVENT
`
`201\
`
`'f
`
`REPLACE WITH MOST ACTIVE
`UNDISPLAYED D-STREAM
`
`FIG. 13
`
`SEARCH FOR MOST
`ACTIVE D_STREAM TO
`DISPLAY
`(WITH ~ RESOURCES,
`F _C3 CLEAR)
`
`206
`
`FOUND!
`
`205
`
`NONE
`FOUND
`
`UNDISPLAY SILENT STREAM,
`DISPLAY ACTIVE STREAM
`
`207
`
`VISUAL
`MANAGE
`
`FIG. 14
`
`NEW T _STREAM
`SILENCE EVENT
`
`CISCO Exhibit 1001, pg. 14
`
`

`
`u.s. Patent
`
`Oct. 3, 2000
`
`Sheet 14 of 28
`
`6,128,649
`
`240\
`
`START ON
`O-STREAM CLOSE EVENT
`
`241\
`
`"
`REPLACE WITH MOST ACTIVE
`UNDISPLAYED D-STREAM
`
`FIG. 15
`
`CISCO Exhibit 1001, pg. 15
`
`

`
`u.s. Patent
`
`Oct. 3, 2000
`
`Sheet 15 of 28
`
`6,128,649
`
`240
`
`DISPLAY-LOCKED
`
`D_STREAM
`CLOSE EVENT
`
`NOT CURRENTLY
`DISPLAYED
`
`241
`
`SIGNAL
`
`CURRENTLY
`DISPLAYED
`
`SEND GUI EVENT
`(UNLOCK
`D_STREAM)
`
`UNDISPLA Y CLOSE
`STREAM
`
`242
`
`243
`
`NOT A LOCKED
`STREAM
`
`SEARCH FOR MOST ACTWE
`D STREAM TO DISPLAY
`(WITH :s. RESOURCES,
`F _C3 CLEAR)
`
`FOUND!
`
`246
`
`NONE
`FOUND
`
`SEARCH FOR LEAST
`SILENT
`D_STREAM TO DISPLAY
`(WITH .5 RESOURCES,
`F _C3 CLEAR)
`
`FIG. 16
`
`244
`
`DISPLAY
`NEW STREAM
`
`245
`
`VISUAL
`MANAGE
`
`8
`
`CISCO Exhibit 1001, pg. 16
`
`

`
`u.s. Patent
`
`Oct. 3, 2000
`
`Sheet 16 of 28
`
`6,128,649
`
`FIG. 17a(i)
`
`CYCLE
`EVERY
`TIMESLICE
`
`250
`
`251 ""
`\
`ACTIVITY
`~--~~----~
`~------~----~
`DETECT
`INCREMENT
`T _STREAM ACTIVITY
`TAM_DEGREE
`IN TIMESLICE
`(MAX OF TAM_MAX)
`
`254
`
`NO ACTIVITY
`
`252
`
`DECREMENT
`TAM_DEGREE
`(MIN OF TAM_MIN)
`
`NEW ACTIVITYEVENT? NEW
`(TAM_HI_THR
`OVERRUN)
`
`SIGNAL
`
`253
`
`SEND ACTIVITY
`EVENT
`UPDATE
`TAM_STATE
`
`NONE
`
`256
`
`t-N_E_W---..t SE N D SI LENC E EVENT t----1.-t
`UPDATE TAM_STATE
`
`255
`
`NEW SILENCE
`EVENT?
`(TAM_LOW_THR
`UNDERRUN)
`
`NONE
`
`SIGNAL FIG. 17a(ii)
`
`CISCO Exhibit 1001, pg. 17
`
`

`
`u.s. Patent
`
`Oct. 3, 2000
`
`Sheet 17 of 28
`
`6,128,649
`
`COUNTER NAME: DIM_DEGREE = {DIM_MIN, DIM_MAX}
`COUNTER STATE: DIM_STATE = {INTEGRITY I LOI}
`COUNTER VARIABLE: DIM_LOSS_FACTOR (OLF)
`
`DIM MIN
`
`DIM LOW THR
`
`DIM HI THR
`
`(
`
`DIM MAX
`
`TIMESLICE
`
`1111111111111111111111111111111111111111111111111111111111111111111111
`I~
`~I
`INTEGRITY
`GRACE
`LOSS OF INTEGRITY (LOI)
`.1_
`.1'"
`(DIMJNT*DLF)DIM_GRACE*DLF)
`(DIM_LOI*DLF)
`
`NEW LOI EVENtl
`I~ NEW INTEGRITY EVENT
`FIG. 17b{i)
`
`CYCLE
`EVERY
`TIMESLICE
`
`260
`
`261\
`
`DETECT
`D_STREAM LOI
`IN TIMESLICE
`
`LOI
`
`INCREMENT
`DIM_DEGREE BY DLF
`(MAX OF DIM_MAX)
`
`SIGNAL
`
`264
`
`NO LOI
`
`262
`
`263
`
`DECREMENT
`DIM_DEGREE
`(MAX OF DIM_MIN)
`
`NEW LOI EVENT?
`(DIM_HI_ THR
`OVERRUN)
`
`NEW SEND LOI EVENT
`UPDATE
`DIM_STATE (LOI)
`
`NONE
`
`266
`
`NEW SEND INTEGRITY EVENT
`UPDATE DIM_STATE 1---_
`(INTEGRITY)
`
`265
`
`NEW INTEGRITY
`EVENT?
`(DIM_LOW_ THR
`UNDERRUN)
`
`NONE
`
`SIGNAL FIG. 17b{ii)
`
`CISCO Exhibit 1001, pg. 18
`
`

`
`u.s. Patent
`
`Oct. 3, 2000
`
`Sheet 18 of 28
`
`6,128,649
`
`MONITOR FOR WORKSTATION RESOURCE i
`MONITOR STATE: WRM_STATE_i = {SAFE I WARNING I CRITICAL}
`MONITOR VARIABLE: WRM_NUM_SAMPLES_i (WNSi)
`MONITOR VARIABLE: WRM DEGREE i
`-
`-
`
`50%
`
`WRM WARN
`-
`
`WRM CRIT
`-
`
`100%
`
`1111111111111111111111111111111111111111111111111111111111111111111111
`
`I'"
`
`SAFE
`
`.I~ WARNING .I~ CRITICAL
`
`FIG. 17c(i)
`
`.1
`
`CYCLE
`EVERY
`TIMESLICE
`
`270
`
`SAMPLE
`UTILIZATION
`OF RESOURCE i
`
`271
`
`AVERAGE
`OVER LAST
`WNSi SAMPLES
`
`272
`
`CHANGE WNSi
`VALUE
`
`SAFE
`EVENT
`
`WARNING CRITICAL
`EVENT
`EVENT
`
`273
`
`UPDATE
`WRM STATE i
`-
`-
`
`IF STATE CHANGED,
`SEND EVENT!
`
`FIG. 17c(ii)
`
`CISCO Exhibit 1001, pg. 19
`
`

`
`u.s. Patent
`
`Oct. 3, 2000
`
`Sheet 19 of 28
`
`6,128,649
`
`300\
`
`DETERMINE LOI
`STREAMS FOR
`WHICH NFM
`SHOWS FOREIGN
`CONGESTION
`
`IF SOME
`
`DETERMINE
`CRITICAL LOCAL
`CONGESTION
`
`YES
`
`... ..
`
`302~ IF NONE
`\ ,r
`
`DETERMINE MAJOR
`CORRELATION
`OF DEGREE OF LOI
`BETWEEN
`STREAMS
`
`YES
`r---
`
`EXPANDED IN
`FIG. 17d(ii)
`
`EXPANDED IN
`FIG. 17d(iii)
`
`303~
`\ 1~
`
`DETERMINE MINOR
`CORRELATION OF
`DEGREE OF LOI
`BETWEEN
`STREAMS
`
`u
`
`304\
`
`(305
`
`r
`
`... --
`
`ASSUME REMAINING
`LOI IS CAUSED BY
`FOREIGN
`CONGESTION
`
`r ... ..
`
`OUTPUT
`TOMDM
`
`FIG. 17d(i)
`
`CISCO Exhibit 1001, pg. 20
`
`

`
`u.s. Patent
`
`Oct. 3, 2000
`
`Sheet 20 of 28
`
`6,128,649
`
`CYCLE
`EVERY
`TIMESLICE
`
`BUILD CAM_LOI_SET
`SET IS ALL DIM STREAMS
`SHOWING LOI
`(E.G. DIM_STATE = LOI)
`
`e-----I~
`
`BUILD CAM_F _SET
`MOVE ANY NFM STATE =
`FOREIGN STREAMS ALSO
`FOUND IN CAM_LOI_SET
`I NTO CAM F SET
`
`310
`
`312
`
`IF CAM_F _SET IS EMPTY
`SET CAM F STATE =
`SAFE, ELSE SET
`CAM_F _STATE =
`CRITICAL
`
`EMPTY
`
`NOT EMPTY
`
`DETECT CRITICAL
`LOCAL CONGESTION
`DOES ~ CAM_ CRIT _PER
`% OF ALL STREAMS
`SHOW LOI?
`
`314
`
`YES
`
`311
`
`SIGNAL
`
`BUILD CAM_L_SET
`MOVE ALL STREAM IDs
`FROM CAM LOI SET TO
`CAM_L_SET, SET
`CAM L STATE = CRITICAL
`
`SEND NEW SETS/STATES
`TO MOM
`(CAM_L_SET, CAM_F _SET,
`CAM_L_STATE,
`CAM_F _STATE)
`
`315
`
`NO
`
`TO FIG. 17d(iii)
`
`313
`
`FIG. 17d{ii)
`
`CISCO Exhibit 1001, pg. 21
`
`

`
`u.s. Patent
`
`Oct. 3, 2000
`
`Sheet 21 of 28
`
`6,128,649
`
`FROM FIG. 17d(ii)
`
`316 \
`
`+
`
`CALCULATE MEAN (f-L)
`OF DIM DEGREE FOR
`STREAMS IN SET
`CAM_LOI_SET
`
`DETECT MAJOR
`CORRELATED LOI
`DOES ~ CAM_MAJ_PER% OF
`ALL STREAMS IN
`CAM LOI SET RESIDE WITHIN
`CAM MAJ STD OF
`-
`-
`THE MEAN( f-L)?
`
`318 \
`
`YES
`
`"
`BUILD CAM_L_SET
`MOVE ALL CORRELATED
`STREAM IDs FROM
`CAM LOI SET TO
`CAM_LSET, SET
`CAM_L_STATE = CRITICAL
`
`319 ~
`
`NO
`
`• ...
`
`DETECT MINOR
`CORRELATED LOI
`DOES ~ CAM_MINOR_PER%
`OF ALL STREAMS IN
`CAM LOI SET RESIDE WITHIN
`CAM MINOR STD OF
`-
`-
`THE MEAN(f-L)?
`,~--------~------~~
`NO
`
`YES
`
`320 \
`
`r-~~~
`
`BUILD CAM_L_SET
`MOVE ALL CORRELATED
`STREAM IDs FROM
`CAM LOI SET TO
`CAM_L_SET, SET
`CAM L STATE = WARNING
`
`SIGNAL
`
`~
`&- SEND NEW SETSI
`
`STATES TO MOM
`(CAM_L_SET,
`CAM_F_SET,
`CAM_L_STATE,
`CAM_F _STATE)
`
`COMPLETE CAM_F _SET
`MOVE REMAINDER OF
`STREAMS IDs IN
`CAM_LOI_SET INTO
`CAM_F _SET SET
`CAM F STATE
`
`322~
`
`321~
`FIG. 17d(iii)
`
`CISCO Exhibit 1001, pg. 22
`
`

`
`u.s. Patent
`
`Oct. 3, 2000
`
`Sheet 22 of 28
`
`6,128,649
`
`NETWORK FEEDBACK MONITOR (NFM)
`
`CYCLE
`EVERY
`TIMESLICE
`
`NETWORK
`FEEDBACK
`(STREAM i)
`
`330
`
`PROCESS NETWORK
`FEEDBACK
`MESSAGES
`
`333
`
`f-N_0----.l SET NFM STATE i
`-
`TO l()CAL
`
`331
`
`DETECT LOSS
`EVENT
`( > NFM_THRj)
`
`LOSS
`
`332
`
`ARE AT LEAST
`NCPi % OF ALL
`RECEIVERS
`DETECTING LOSS
`EVENT?
`
`336
`
`NO LOSS
`
`335
`
`YES
`
`SET NFM_STATEJ
`TO NONE
`
`SET NFM_STATE_i
`TO FOREIGN
`
`334
`
`IF NEW EVENT
`STATE,
`SEND SIGNAL
`
`SIGNAL
`{NONE I LOCAL I FOREIGN}
`
`FIG. 17e
`
`CISCO Exhibit 1001, pg. 23
`
`

`
`u.s. Patent
`
`Oct. 3, 2000
`
`Sheet 23 of 28
`
`6,128,649
`
`350
`
`RECEIVE FUNCTIONS
`LISTEN FOR PERIODIC
`KEEP _ALlVE_MSG MESSAGES.
`REQUEST AND RECEIVE INFORM MSG
`MESSAGES.
`UPDATE THE LOCAL USER STATE
`DATABASE WITH NEW CHANGES.
`INFORM THE GUI OF CHANGES TO THE
`USER STATE DATABASE.
`
`GUI
`MOM
`
`351
`
`CONFERENCE
`USER STATE
`DATABASE
`
`NETWORK
`
`352
`
`TRANSMIT FUNCTIONS
`SEND AN INITIAL INFORM_MSG TO ALL
`CONFEREES.
`SEND PERIODIC KEEP _ALlVE_MSG TO
`ALL CONFEREES.
`DETECT GUI SETTINGS CHANGE.
`RESEND KEEP _ALlVE_MSG TO ALL
`CONFEREES ON SETTINGS CHANGE.
`RESPOND TO INFORM_MSG MESSAGE
`REQUESTS.
`SEND LEAVE_MSG TO ALL CONFEREES
`WHEN TERMINATING.
`
`FIG. 18a
`
`CISCO Exhibit 1001, pg. 24
`
`

`
`u.s. Patent
`
`Oct. 3, 2000
`
`Sheet 24 of 28
`
`6,128,649
`
`365
`
`WAIT FOR
`INFORM_MSG
`REQUEST
`
`366
`
`SEND
`INFORM_MSG
`MESSAGE
`(UNICAST)
`
`LOOP
`
`360
`
`BROADCAST
`INFORM_MSG
`MESSAGE
`
`NEWGUI
`SETTINGS
`
`SLEEP UNTIL
`INTERVAL TIMEOUT
`OR DETECTION OF
`CHANGE IN GUI
`SETTINGS
`
`362
`
`361
`
`BROADCAST
`KEEP _AUVE_MSG
`MESSAGE
`
`363
`
`TERMINATE
`SIGNAL
`
`364
`
`TERMINATE
`
`RECEIVED?
`
`CONTINUE
`
`LOOP
`
`BROADCAST
`LEAVE_MSG
`MESSAGE
`
`FIG.18b
`
`CISCO Exhibit 1001, pg. 25
`
`

`
`u.s. Patent
`
`Oct. 3, 2000
`
`Sheet 25 of 28
`
`6,128,649
`
`A
`
`370\
`
`~Ir
`
`WAIT FOR ARRIVAL OF
`BROADCAST
`
`..
`.. KEEP _ALlVE_MSG OR
`
`INFORM_MSG
`MESSAGES
`
`371\
`
`11r
`
`NEED TO REQUEST AN
`INFORM_MSG
`MESSAGE?
`
`374\
`
`NEEDED
`
`-..
`
`SEND INFORM_MSG
`REQUEST
`
`372\
`
`~r
`
`NO NEED
`
`375\
`
`CONFERENCE USER -
`--
`
`UPDATE
`
`STA TE OA TABASE
`
`l'
`
`WAIT FOR ARRIVAL
`OF UNICAST
`INFORM_MSG
`RESPONSE
`
`373~ 1.
`
`SIGNAL GUI AND MOM
`OF DATABASE UPDATE
`
`LOOP
`
`~DATABASE
`
`UPDATED!
`
`FIG.18c
`
`CISCO Exhibit 1001, pg. 26
`
`

`
`u.s. Patent
`
`Oct. 3, 2000
`
`Sheet 26 of 28
`
`6,128,649
`
`Video Conference Control Panel
`
`I File
`
`Settings
`
`View
`
`Help
`
`CHRISTINE
`
`GARY
`
`THINH
`
`VIDEO SERVER
`
`ERIC
`
`~~ ~
`[Q] 00
`
`~~ ~~
`~~)
`[Q]
`[Q]
`[Q]
`D
`I
`* FOREIGN NETWORK CONGESTION
`----1 LOCAL NETWORK CONGESTION
`
`----1 NETWORK BWITARIFF LIMITS
`
`ALARMS
`
`----1 WORKSTATION RESOURCES
`----1 DESKTOP REAL-ESTATE LIMITS
`----1 USER-LOCKED SETTINGS
`
`ALARM REPORT:
`I
`FOREIGN NETWORK CONGESTION DETECTED ON VIDEO STREAM ERIC
`
`I
`
`LEGEND
`
`DISPLAY VIDEO:
`
`DISPLAYED (LOCKED)
`
`UNDISPLAY VIDEO:
`
`\'tQJl
`
`SHOULD NEVER BE
`DISPLAYED
`
`TRIGGERED VIDEO:
`
`o SHOULD BE CONTINUOUSLY
`[Q] OFFERED VIDEO MEDIA
`00 OFFERED VIDEO MEDIA
`[Q] OFFERED VIDEO MEDIA SHOULD
`----1 SAFE RESOURCE STATE
`
`{ ) ONLY BE DISPLAYED UPON THE
`DETECTION OF AUDIO ACTIVITY
`
`PLAY AUDIO:
`OFFERED AUDIO
`MEDIA SHOULD BE
`PLA YED(LOCKED)
`
`MUTE AUDIO:
`OFFERED AUDIO MEDIA
`SHOULD BE MUTED
`
`D
`
`SLIDE PRESENTATION:
`SLIDE PRESENTATION
`SHOULD BE VIEWED
`
`CRITICAL RESOURCE
`STATE
`
`*
`
`FIG. 19
`
`CISCO Exhibit 1001, pg. 27
`
`

`
`u.s. Patent
`
`Oct. 3, 2000
`
`Sheet 27 of 28
`
`6,128,649
`
`10,\
`
`OTHER USER AWARENESS
`INFORMATION
`
`..
`
`NETWORK
`
`USER
`
`SELECTION
`DYNAMIC
`POLICY
`r---------------+-.~ SELECTION
`CONTROLLER
`
`..
`.....
`
`SELECTED MEDIA
`
`13-.--J
`
`.h
`
`SELECTION
`REQUESTS
`
`• ,r
`3 --------- OTH E R
`USERS
`
`FIG. 20
`
`CISCO Exhibit 1001, pg. 28
`
`

`
`u.s. Patent
`
`Oct. 3, 2000
`
`Sheet 28 of 28
`
`6,128,649
`
`10 '\
`
`-.....
`
`12~
`
`OTHER USER AWARENESS
`INFORMATION
`
`USER
`
`SELECTION
`...
`POLICY
`...
`
`DYNAMIC
`SELECTION
`CONTROLLER
`13~
`
`NETWORK
`
`SELECTION
`REQUESTS ...
`...
`
`SELECTED MEDIA
`
`.-
`
`-
`
`.....
`
`MEDIA
`
`-
`NETWORK -.....
`14~
`
`Jil
`
`,.
`
`OTHER
`USERS
`
`3
`
`FIG. 21
`
`CISCO Exhibit 1001, pg. 29
`
`

`
`6,128,649
`
`1
`DYNAMIC SELECTION OF MEDIA
`STREAMS FOR DISPLAY
`
`FIELD OF THE INVENTION
`
`The invention relates to methods and apparatus for select(cid:173)
`ing for display one or more from a number of real-time
`media streams and to corresponding software, network
`nodes and terminals.
`
`BACKGROUND TO THE INVENTION
`
`Today's computing and network architectures readily
`support the transfer of text and still graphics or images.
`However, support for real-time media processing and net(cid:173)
`working has, until very recently, been realised entirely with
`overlay networks and service specific terminal equipment
`for displaying such media to the user. Voice telephony for
`example, is the most pervasive media specific network.
`Broadcast video and cable TV also use a dedicated trans(cid:173)
`mission and switching infrastructure. In the same vein, high
`quality video conferencing requires leased lines and expen(cid:173)
`sive dedicated equipment.
`The most commercially successful segment of the video
`conferencing market has been for so-called "Px64" systems
`based on the ITU's H.320 series standards. Such systems
`aggregate from 2 to 30 DSO (64 kbps) channels over
`switched or leased-line Time Division Multiplexing (TDM)
`networks into a wideband channel (128 kbps-2 Mbps) to
`transport audio, video, data and control in a point-to-point
`manner. Multipoint conferencing is achieved through a
`centralised Multicast Control Unit (MCU) as shown in FIG.
`1 which typically mixes audio and multicasts the single
`current speaker to all sites. An alternative, a distributed
`switching and mixing arrangement will be described with
`reference to FIG. 2.
`More rapid growth of this market has been hampered by
`both high equipment costs and high service costs. In addition
`to the cost of video COder/DECoder (CODEC) hardware,
`equipment costs are exacerbated in current networks by the 40
`need for Inverse Multiplexers (I-MUXs) to aggregate
`switched DSO circuits due to the absence of wideband
`channel switching, and MCUs due to the absence of multi(cid:173)
`cast switching. Service costs have been kept high due to
`bandwidth based tariffs needed to protect revenues from
`voice telephony and so high bandwidth, high quality video
`conferencing is still a luxury rarely afforded.
`Further limitations exist at the users terminal. Real-time
`media imposes high processing requirements, particularly if
`the media stream needs to be decompressed for display.
`Usually the resolution and size of display monitors is
`restricted.
`FIG. 1 shows in schematic form a known video confer(cid:173)
`encing arrangement. Using multicast control unit (MCU) for
`multiplexing or mixing and distributing all video streams
`transmitted by the network 2 to and from users 3 gives a
`centralised topology. This is suitable for use with a point to
`point network such as the telephone network. The expensive
`dedicated video mixing or selecting equipment need be
`provided in only one place while making use of the switch(cid:173)
`ing capability already provided in the network 2. One of the
`users 3, a chairman, has facilities to control the MCU, 1.
`In operation, each user sends its own video and audio to
`the MCU. The chairman controls the MCU to select one of
`the incoming video streams, or add the video streams
`together in separate windows of the single output video
`streams. The input audio streams would all be mixed
`
`2
`together, or the audio streams with most activity could be
`selected for mixing and outputting. The MCU duplicates its
`output video and audio streams and sends them to each of
`the users. Such arrangements may be limited in bandwidth
`5 or number of users by the capabilities of the MCU, or by the
`bandwidth of the telephone network connections.
`In another known conferencing arrangement illustrated in
`FIG. 2, a LAN network 4 with multicast capability connects
`users 3. This obviates the need for a dedicated MCU. Users
`10 3 can control which other users they see.
`FIG. 3 illustrates the information which may pass
`between a user 10 and a network 11 such as a packet
`network, connecting other users 3 for video conferencing.
`Awareness information of which of a users are connected to
`15 the network is passed from the network to the user 10. In
`response to this information, a user can choose manually
`which other users he wants to see. Video selection request
`information is then passed from the user to the network. The
`network has the capability to take the request and switch
`20 appropriate video streams from other users 3 to the user 10.
`Network restrictions have often precluded sending
`streams from all users in a video conference to all other
`users. Accordingly, the centralised switching approach
`25 shown in FIG. 1 involves either selecting one of the streams
`from users 1, 2, 3, for display, or creating a single image
`stream comprising a composite display of two or more
`reduced size images or windows.
`U.S. Pat. No. 4,531,024 (Colton) describes a way of
`30 resolving how to select a single video stream to be trans(cid:173)
`mitted to all other conference locations. The selection is
`made automatically by centralised detection of either one
`and only one "talker" or one and only one video graphics
`transmission request. Manual override is possible at each
`35 location, to select manually the video to be viewed.
`U.S. Pat. No. 5,003,532 (Ashida) shows a video confer(cid:173)
`ence system having a centralised image selector operating
`according to requests from users or according to speaker
`detection.
`U.S. Pat. No. 5,382,972 (Karres) describes a conferencing
`system which creates a composite signal with voice sensitive
`switching means for moving the component streams to
`different regions on the screen, and different sizes of picture,
`according to who is talking. A master user has an override
`45 control.
`A further development is shown in U.S. Pat. No. 5,473,
`367, in which any conferee can assume the chairing role, and
`manipulate manually the picture which will be viewed by
`all. Additionally, each conferee can choose their own picture
`50 content, or take the chair view.
`U.S. Pat. No. 5,615,338 shows a system in which a central
`controller controls the transmission from each user terminal
`directly, and selects two users to transmit to all other users
`55 according to user requests and a predetermined priority
`scheme.
`U.S. Pat. No. 5,392,223 shows a communications proces(cid:173)
`sor for linking a group of workstations, to a network for
`video conferencing. A workstation initiates a request for
`60 service including type of service and destination.
`Bandwidth, resolution and transmission rate are adjustable.
`Artificial intelligence software is used in the processor,
`which reacts to the instantaneous loadings, and indicates to
`the user what is possible if the request for service can't be
`65 fulfilled.
`Another example of a decentralised videoconferencing
`network is shown in U.S. Pat. No. 5,374,952, using a
`
`CISCO Exhibit 1001, pg. 30
`
`

`
`6,128,649
`
`3
`broadband LAN. Television signals from each user are
`transmitted simultaneously at different frequencies. Each
`user's computer monitors the status of channel allocations
`and generates the channel selecting control signals. Such
`dedicated LANs have inherent broadcasting capability
`which implies the ability to multicast, i.e. send to a select
`group of users.
`None of the above systems is scaleable to handle large
`conferences because of human cognitive limitations in view(cid:173)
`ing a screen with too many windows displayed
`simultaneously, or with manually selecting between too
`many available windows. Additionally, the users terminal
`may have limited processing power and limited display area,
`and the network resources may limit how many streams can
`be sent to him anyway.
`
`4
`tion of how to make the selection dependent on the activity.
`This enables the selection to include the streams most likely
`to be of interest and to drop streams of less interest.
`Preferably, where two or more of the media streams
`5 originate from a single source, the indication of how to make
`the selection dependent on the activity comprises making the
`selection of one of the co-originating streams dependent on
`activity on another of the co-originating streams. This
`enables the user to be presented automatically with streams
`10 which are likely to be of interest because they are associated
`with a stream showing activity.
`Advantageously, the step of determining the selected
`streams is carried out at the first user's terminal. This enables
`the method to be used with less specialised equipment in the
`15 network.
`Preferably, the method further comprises the step of
`indicating to the network which are the selected streams,
`wherein the step of passing the selected streams comprises
`transmitting across the network only the selected streams.
`20 This enables reduced bandwidth requirements in the net(cid:173)
`work and at the user network interface.
`Advantageously, the media streams comprise video
`streams. As video has high bandwidth, it is more important
`to utilise and share resources more efficiently, bearing in
`mind human perception and resource limitations.
`Preferably, the network is a multicast capable network. If
`the network is multicast capable, less specialised equipment
`is necessary in the network and at the users terminal.
`Advantageously, the first user also transmits at least one
`real-time media stream to take part in a conference between
`multiple users across the communications network. The
`dynamic nature of conference type communication makes
`for heavier burdens on network and computing resources
`and on human cognitive abilities, and so better management
`of the selection is more advantageous.
`According to another aspect of the invention, there is
`provided software, stored on computer readable media, for
`carrying out the method.
`According to another aspect of the invention, there is
`provided a method of selecting for display one or more
`streams from a number of real-time media streams available
`to be transmitted across a communications network for
`display on respective terminals of a first, and at least one
`other user, the method comprising the steps of: determining
`a policy of each of the users for making their respective
`selection, the policy comprising an indication of how the
`selection should be made, the policy being selectable by the
`first user; determining a condition of at least one of the
`communications network and the terminal; determining
`dynamically which streams to select for each user according
`to the condition and according to the respective policy; and,
`causing only the selected streams to be passed for display on
`the terminal of each user, independently of selections made
`for passing to the other users.
`According to another aspect of the invention, there is
`provided apparatus for selecting for display, one or more
`real-time media streams available to be transmitted across a
`communications network for display on terminals of a first
`60 user and other users, the apparatus comprising: means for
`determining a policy of the first user for making the
`selection, the policy comprising an indication of how the
`selection should be made, the policy being selectable by the
`first user; means for determining a condition of at least one
`65 of the communications network and the terminal; means for
`determining dynamically which streams to select for the first
`user according to the condition and according to the first
`
`40
`
`SUMMARY OF THE INVENTION
`It is an object of the invention to provide improved
`method and apparatus.
`According to the invention, there is provided a method of
`selecting for display one or more streams from a number of
`real-time media streams available to be transmitted across