Ulllted States Patent
`
`[19]
`
`[11] Patent Number:
`
`5,953,506
`
`Kalra et al.
`
`[45] Date of Patent:
`
`Sep. 14, 1999
`
`US005953506A
`
`[54] METHOD AND APPARATUS THAT
`PROVIDES A SCALABLE MEDIA DELIVERY
`SYSTEM
`
`W0 96/14711
`WO 96/23280
`WO 97/15149
`
`5/1996 Wll’0 .
`8/1996 WIPO ,
`4/1997 w1po .
`
`[75]
`
`lnventors: Devendra Kalra, Fremont;
`Karnamadakala Krishnamohan, San
`Jose; Venkatasubbarao Ramamoorthy,
`Pleasanton; Jeyendran Balakrishnan,
`Sunnyvale; Timothy J. Burr, San Jose;
`Kowsik Guruswamya Mountain View’
`31]] Of Cahf,
`
`[73] Assignee: Adaptive Media Technologies,
`.
`Sunnyvale, Calif.
`
`[21] APPL N07 08/768414
`[22]
`Filed:
`Dec. 17, 1996
`[51]
`Int. Cl.6 ............................. .. G06F 15/16; H04L 7/30
`[52] U..S. Cl.
`.................... .. 395/200.61; 345/428; 348/420
`[58] Field of Search ..................... .. 395/200.61; 370/431,
`370/437; 345/302, 428; 348/17, 18, 19,
`20> 384> 387> 390> 420
`
`OTHER PUBLICATIONS
`
`Moura et al. “Retrieving quality video across heterogeneous
`networks”, IEEE Personal Communications, Feb. 1996, pp.
`44-54‘
`.
`.
`.
`“
`.
`.
`IBM Technical Disclosure Bulletin Mi1lti—Rate Video
`Traiisiiiissioii Sclieiiie”, vol. 38 No. 12, Dec. 1995. pp.
`59-62.
`
`Doenges, P.K. et al., “Audio/video and synthetic graphics/
`audio for mixed media,” Signal Processing: Image Commu-
`nication, vol. 9, No. 4, May 1997.
`Arikawa, M. et al., “Dynamic LoD for QoS Management in
`the Next Generation VRML,” Proceedings of the Intl. Conf.
`0” M"”‘”’“d‘“ C"’”P’”’”g “ml Sy""’”’~‘> J‘'“ 17> 1996*
`(List Continued on next page.)
`
`primary Examme,,_Dung C. Dinh
`Attorney, Agent, or Firm—Pillsbury Madison & Sutro LLP
`
`[56]
`
`References Cited
`
`[57]
`
`ABSTRACT
`
`U.S. PATENT DOCUMENTS
`
`The present invention provides an apparatus and method for
`
`4,672,444
`5,053,726
`5,196,933
`
`.......................... 348/441
`N 343/412
`.. 348/419
`
`..
`
`6/1987 Bergen et al.
`.
`11/1991 Kondo 6] a],
`3/1993 Henot .......... ..
`11:11“ fit all-~~
`,
`,
`,
`’ ause et a .
`5,659,691
`8/1997 Durward et al.
`345/329
`5,675,721
`10/1997 Freedman et al.
`N 345/502
`5,699,361
`12/1997 Ding et al.
`370/431
`5,737,495
`4/1993 Adams et fl1~
`------------------- -- 395/200-49
`FOREIGN PATENT DOCUMENTS
`
`..
`
`
`
`..
`
`?’“°°d1“g; 5‘F’““g> transmfiung and d°°°dmg, Fnummedla
`information in the form of scalable, streamed digital data. A
`base stream containing basic informational content and
`subsequent streams containing additive informational con-
`,
`,
`tent are initially created from standard digital multimedia
`data by a transcoder. Client computers, each of which may
`have different configurations and capabilities are capable of
`accessing a stream server that contains the scalable streamed
`digital data. Each different client. computer, ‘therefore, may
`access different stream combinations according to a profile
`associated with each different client computer. Thus, the
`streams accessed from the server are tailored to match the
`profile of each client computer so that the best combination
`of streams can be provided to maximize the resolution of the
`3D, audio and video components.
`
`80 Claims, 39 Drawing Sheets
`
`22
`
`
`
`1/1994 European Pat Off ~
`0 377 3271“
`4/1994 European Pat Off '
`0 393 013 A2
`7/1995
`European Pat. Off.
`.
`0 661 326 A2
`7/1995
`European Pat. Off.
`.
`0 661 385 A1
`0 687 112 A2 12/1995
`European Pat. Off.
`.
`0 739 140 A2 10/1996
`European Pat. Off.
`.
`0 751 685 A1
`1/1997 European Pat. Off.
`.
`WO 94/11993
`5/1994 WIPO .
`
`Adaptive Digital
`S"'9a"“5 F°"ma‘
`
`“Ab
`Base Stream
`
`14A]
`1st
` 20
`Additive Stream
`2nd
`Additive Stream
`3rd
`Additive Stream
`1
`i
`i
`
`
`
`
`
`Stream
`Multimedia
`Devices
`Management
`Module
`
`
`
`RPX Exhibit 1115
`RPX Exhibit 11 15 RPX V. DAE
`RPX v. DAE
`
`Nth
`
`Additive Stream
`14AN
`
`
`
`16~/
`
`

`
`5,953,506
`Page 2
`
`OTHER PUBLICATIONS
`
`Kudumakis, P.E. et al., “Wavelet packet Based Scalable
`Audio Coding,” 1996 IEEE Intl. Symposium on Circuits and
`Systems (ISCAS) Circuits and Systems Connecting the
`World, Atlanta, May 12-15, 1996; vol. 2, May 1996,Inst. of
`Electrical and Electronics Engineers, pp. 41-44.
`“Transmission of non-telephone signals; information tech-
`nology—generic coding of moving pictures and associated
`audio information: video” ITU—T Telecommunication Stan-
`
`dardization Sector ofITU, Jul. 1995.
`Riegel, T.; “Coding of combined natural and computer
`rendered image sequences,” Proc. of SPIE vol. 2451, 1995,
`pp. 207-211.
`Broll, W., et al.; “VRML: Today and Tomorrow,” Computers
`& Graphics vol. 20, No. 3, May 1996, pp. 427-434.
`
`Funkhouser, T.A. and Sequin, C.H., “Adaptive Display
`Algorithm for Interactive Frame Rates during Visualization
`of Complex Virtual Environments,” Computer Graphics
`Proceedings, Annual Conf. Series 1993, pp. 247-254.
`Hoppe, H., Progressive Meshes, Computer Graphics Pro-
`ceedings, Annual Conference Series, 1996, pp. 99-108.
`Popovic, J .; and Hoppe, H., Progressive Simplicial Com-
`plexes, Computer Graphics (SIGGRAPH ’97 Proceedings),
`pp. 217-224.
`Clarke, R.J., Standards for Image Sequence Coding from
`Digital Compression of Still Images and Video, 1995, pp.
`285-299.
`
`Kuan Hui Tan/Mohammad Ohanbari, Layered Image Cod-
`ing Using the DCT Pyramid from IEEE Transactions On
`Image Processing, Apr. 1995, pp. 512-516.
`
`

`
`U.S. Patent
`
`Sep.14,1999
`
`Sheet 1 of 39
`
`5,953,506
`
`gay. 1
`
`12
`
`14
`
`Streams Format
`
`T’a”S°°°"”
`
`Adaptive Digital
`
`Stan_dard.Digital
`Multimedia Data
`
`14A1
`
` 14Ab
` 22
`
`Stream
`Management
`
`Module
`
`
`1st
`Additive Stream
`2nd
`Additive Stream
`
`3rd
`Additive Stream
`
`I
`i
`l
`l
`
`l
`
`I
`I
`I
`I
`
`Nth
`Additive Stream
`
`
`
`Streams Format
`
`Adaptive Digital
`
`16
`
`
`
`
`Multimedia
`Devices
`
`
`
`
`
`

`
`U.S. Patent
`
`Sep.14,1999
`
`Sheet 2 of 39
`
`5,953,506
`
`Ty. 299
`
`Visual
`
`Multimedia
`
`
`
`Animation
`3D & Video
`
`Audio
`Rock & Roll
`Classical
`
`Stream
`
`
`MaK',%%‘:T;e”t
`
`Easy Listening —
`English
`French
`German
`
`
`
`Text
`
`English
`French
`German
`
`Dynamic Computational
`& User Profiles
`
`20
`
`Adaptive
`Streams
`
`Netscape
`Microsoft
`
`9
`
`fig. 3
`
`Application
`
`APDU (Telnet)
`
`Presentation
`
`PPDU
`
`Session
`
`SPDU (HTTP/RTP)
`
`Transport
`
`TPDU (TCP)
`
`Network
`
`Packet (IP)
`
`Eth
`t C d
`paCe|<r2;3Dri3<;r:
`
`,
`Data Link
`
`Frame (LLC)
`
`Physical
`
`Bit (Ethernet)
`
`

`
`U.S. Patent
`
`Sep.14,1999
`
`Sheet 3 of 39
`
`5,953,506
`
`24
`
`|e———j Video Sequencee
`
` I, P or B
`
`Picture
`
`
`
`32, 8 x 8 Array of Data
`
`

`
`U.S. Patent
`
`Sep.14,1999
`
`Sheet 4 of 39
`
`5,953,506
`
`fig, 5
`
`J40
`
`MPEG Coded Video Stream (Input Stream)
`
`
`
`Transcode
`
` E
`
`2(3)
`
`2(4)
`
`2(5)
`
`2(1)
`
`3(2)
`
`2(0)
`
`2(0) Stream
`
`
`2(1) Stream
`
`.
`
`O
`
`
`
`
`
`WIIEII
`—Ill
`E.-.E.|I|I|I|I
`‘-- |
`
`11
`
`18
`
`27
`
`38
`
`
`
`
`
`
`
`
`
`
`
`
`
` 2 5 Stream
`
` 2(6) Stream
`Hmwm
`
`7
`
`2(2) Stream
`
`2 3 Stream
`/‘x
`‘a
`
`2(4) Stream
`
`/\
`
`\/
`
`———Z
`
`
`
`Enmaway
`
`

`
`U.S. Patent
`
`Sep. 14, 1999
`
`Sheet 5 of 39
`
`5,953,506
`
`$
`if’
`1g. 6
`
`EEEE
`
`FIPIMMFJ
`liflfllll
`Vlfilflfl
`
`WEEK!
`
`
`
`
`
`gay. 66
`
`

`
`U.S. Patent
`
`Sep.14,1999
`
`Sheet 6 of 39
`
`5,953,506
`
`9:?’
`
`Picture Start
`
`42
`
`Sequence End
`
`
`
`
`
`64
`
`
`
`Drop Frame Code
`
`58, Picture
`Header
`
` 48
`
`
`54, Group
`
`Code
`
`Picture Number =
`Temporal Reference
`
`Next Picture Pointer
`
`
`
`III|l|—|II|||II|||-I|||II 29
`
`V
`
`CorrectionStartCode
`
`
`IIIIIIIIIIIIIIII-I|||II 29>
`
`I-1
`
`III|||II|||II|||II||||I 20>
`
`
`
`Sequence Start
`
`Picture Start
`
`50
`40
`
`
`
`
`Picture Start
`
`
`
`
`

`
`tHEtaP&U
`
`Sep. 14, 1999
`
`Sheet 7 of 39
`
`60S,359’5
`
`NI8__m
`
`FI8__m
`
`8__m
`
`Eommz
`
`8__m
`
`Bnmmz
`
`o::o_n_
`
`Eummz
`
`9.20
`
`Bnmwz
`
`mocmzcww
`
`Eummr
`
`E
`
`EEE
`
`8__m
`
`tflm
`
`ES
`
`8__w
`
`cam
`
`93o_n_
`
`new
`
`Qseo
`
`
`
`tflmmocmsumw.9»
`
`new
`
`
`

`
`U.S. Patent
`
`Sep.14,1999
`
`Sheet 8 of 39
`
`5,953,506
`
`fig. 7c
`
`Sequence_header_codc
`Ox OOOOOIB3
`(32 bits)
`
`I42
`
`Group_start__code
`Ox 000001B8 (32bits)
`
`Horizontal_size (12 bits)
`
`Time__code (25 bits)
`
`I46
`
`|48
`
`'50
`
`Cl
`
`B 0:e“g;:lk E1 bit;
`en
`m "
`
`1 b't
`
`1
`
`d
`tart
`P_ m
`_co e
`1c
`re_s
`Ox 00000100
`(32 bits)
`
`.
`
`(12 bits)
`Vertical_s1ze
`Pel_aspect_ratio (4 bits)
`
`.
`
`Picture__rate
`(4 bits)
`.
`.
`B“—‘a“°'
`_
`(13 bf“)
`Marker_b1t = 1
`(1 bit)
`VBV_buffer_sizc (10bitS)
`Constr._par_flag (1 bit)
`_
`Load_1ntra'r°D97
`
`on C3 0-‘ _.:.\/
`
`Load_non_intra_q (1 bit)
`
`|44A
`
`152A
`Temporal_rcf
`(10 bits)
`N tP_ “Jr P _ t
`(l6b_t)
`
`ex
`
`IC
`
`e
`
`0111 er
`
`1 S
`
`I5-4A
`
`Drop flame code (8 bits)
`o
`/\L» O‘ ....FrIn
`
`Picture_cod_
`
`VBV_delay
`
`(16 bits)
`
`Full_pe1_fr_vector (1 bit
`
`\.IvV
`I'S
`|58A
`
`Forward_f_code (3 bits)
`
`5%
`
`Full_pel__bk_vector (lbit
`
`Backward f code (3 bits)
`
`Slice__start_code
`Ox 00000101 (32 bits)
`
`I56
`
`uantizer_scale (5 bits)
`
`
`
`Do all macro blocks
`
`in the
`
`
`
`Slice
`
`
`
`
`
`
`
`Sequence__end_code
`Ox 000001B7 (32 bits)
`
`

`
`U.S. Patent
`
`Sep.14,1999
`
`Sheet 9 of 39
`
`5,953,506
`
`fig. 7:0
`
`Sequence_start_code
`
`Ox 000002Bx (32 bits)
`
`
`
`
`
`
`Picture_start_code
`
`Ox 000OO20x (32 bits)
`
`Temporal_ref (10 bits)
`
`Next Picture Pointer (16 bits)
`
`Drop Frame Code (8 bits)
`
`S|ice_start_code
`
`0x 00000101 (32 bits)
`
`Modified Zig-Zag scan of
`all macroblocks in the slice
`
`
`
`
`
`
`
`
`
`
`
`1 54A
`
`
`
`
`
`for the Sigma Stream Z(x)
`
`End of Block Code (2 bits)
`
`Correction Start Code
`
`Ox 0000030x (32 bits)
`
`Correction Stream
`
`
`
`
`
`Sequence_end_code
`0x 00O002Cx (32 bits)
`
`
`
`

`
`U.S. Patent
`
`Sep. 14,1999
`
`Sheet 10 of 39
`
`5,953,506
`
`fig. 8541
`
`Slice
`40
`
`
`'
`
`
`
`:§§fllIIflEIlII
`
`
`aallllamllll
`Eawlllgaallll
`
`
`
`Macroblocks
`
`...6 Total
`Times
`
`

`
`U.S. Patent
`
`Sep. 14,1999
`
`Sheet 11 of 39
`
`5,953,506
`
`U
`
`fig. 9% 100
`
`MPEG coded
`video bitstream
`
`Conventional
`MPEG decoder
`
`102
`
`4 : 2 : 0
`
`Format
`
`||——-
`II
`
`Cr
`
`Cb
`
`(pixel domain frames)
`
`104
`Video sequence {\_l (n)}
`occuring at frame rate
`of F frames per sec.
`(F = 24 fps or 30 fps)
`
`1060
`
`This filter enhances quality
`106B\ , - - - - - - - - * — - - - - --
`T 4
`Temporal filler
`'
`:
`Frame store
`I
`faghgfb :
`1102 Hon
`
`1100
`
`1101
`
`and
`decimation
`
`Temporal filter
`
`
`
`Quality
`enhancement
`for F/4 and
`below
`
`DEED
`Cl’
`Cb
`
`1200
`
`Video
`sequence at
`frame rate F/4
`frames per sec.
`(1/4)
`{\_/ W}
`
`WEE
`C
`Cb
`
`r
`
`1203
`
`Temporally
`subsampled
`output
`Video
`Sequence at
`fframe rate F/2
`rames per Sec‘
`(1/2)
`{M (n)}
`
`
`
`
`1223
`
`AMT specific
`1220 MPEG encoder
`
`1248
`
`Spatial scaling
`
`Eu/4)
`
`
`
`
`
`AMT specific
`MPEG encoder
`
`Spatial scaling
`
`,
`
`_:(1/2)
`
`Full
`
`temporal
`resolution
`video
`
`sequence
`
`F,
`,
`frames are '
`passed
`'
`through
`'
`the filter
`:
`5ePara1e'V |
`I
`:
`
`Temporal filter
`to enhance
`
`quality for all
`rates between
`
`
`
`(F, F/2)
`
`120A
`
`{y (n)}
`
`AMT specific
`MPEG encoder
`
`122A
`
`Spatial scaling
`transcoder
`
`2(1)
`
`

`
`U.S. Patent
`
`Sep. 14,1999
`
`Sheet 12 of 39
`
`5,953,506
`
`fig.
`
`Data in
`
`140
`
`
`
`142
`
`144
`
`Write adaptive stream sequence start code
`
`Write MPEG standard sequence
`header type information
`
`146
`
`Write adaptive stream group start code (New I)
`
`148
`
`150
`
`152
`
`154
`
`156
`
`Write MPEG standard headers group
`
`Write adaptive stream picture start code
`
`Write MPEG picture headers
`
`Write adaptive stream picture headers
`(see Fig. 9C )
`
`Write adaptive stream slice start code
`
`Write MPEG slice headers
`
`Do & write all macroblocks in slice
`
`Write correction code
`
`Sequence
`end?
`
`
`
`
`
`
`Z0 Transcoder
`Ft
`
`
`sequence start
`code?
`
`
`
`
`
`
`

`
`U.S. Patent
`
`Sep. 14,1999
`
`Sheet 13 of 39
`
`5,953,506
`
`fig. 9:32
`
`)2 0- 27 Transcoder
`
`Fi
`
`sequence start
`code?
`
`180
`
`
`
`182
`
`Findp
`
`start code?
`
`Write adaptive stream picture start code
`
`184
`
`188
`
`190
`
`
`
`Write adaptive stream slice start code
`
`Do & write all macroblocks in slice
`
`192
`
`194
`
`sequence?
`
`No
`
`N0 YSS
`
`

`
`U.S. Patent
`
`Sep. 14,1999
`
`Sheet 14 of 39
`
`5,953,506
`
`
`
`
`
`228
`
`Are I and P
`
`previously
`selected?
`
`200
`
`
`
`Update
`pointer for
`next frame
`
`Ref I frame
`
`previously
`selected?
`
`

`
`U.S. Patent
`
`Sep. 14,1999
`
`Sheet 15 of 39
`
`5,953,506
`
`S_5__8_n_
`
`E£m>w
`
`::m_
`
`£95..
`
`Buoomo
`
`9%
`
`S.3
`
`afioo
`
`39:8
`
`cam
`
`£33
`
`Euooma
`
`Ew\_m
`
`595..Si
`
`.300
`
`
`momm:_E8=_dom
`wv_8_m
`
`8_§_§n_
`
`mmm
`
`
`
`
`
`
`
`
`

`
`U.S. Patent
`
`Sep. 14,1999
`
`Sheet 16 of 39
`
`5,953,506
`
`Do all Macro Blocks in the Slice
`
`0
`
`fig. 11:?!
`
`N
`
`Macroblock_address_escape (11 bits)
`yes
`f address_escape
`0
`Ob 0000 0001000
`
`
`
`Macroblock_address_inc (1 - 11 bits)
`
`
`
`Macroblock_type
`
`( 1- 61bits)
`
`_
`uantlzer Scale
`
`_
`(5 blts)
`
`If macroblock quant
`‘
`
`No
`
` O
`Ifmacroblock_motion_forward
`|Motion_hor_fr_code ( 1 — 11 bits)
`
`Q3
`
`forward_f != l &&
`If
`motior1_hor_fr_code != 0
`
`Motion_hor_fr_r
`
`( l-6 bits)
`
`
`
`Motion_vert_forward_r
`
`(1-1 1 bits)
`
`If forward_f != 1 &&
`m0ti0n_vert_fi"_code != 0
`
`Motion_ver1_fr_r (1-6 bits)
`
`CD
`
`

`
`U.S. Patent
`
`Sep. 14,1999
`
`Sheet 17 of 39
`
`5,953,506
`
`37'
`0 15°11
`
`$
`
`Ifmacroblock_motion_backward
`
`IMotion_hor__bk_code ( 1 — 11 bits)
`
`If backward_f != 1 &&
`motion_hor_bk_codc != 0
`
`Motion_hor_bk_r
`
`( 1-6 bits)
`
`
`
`Motion_vert_backward_r
`
`(1-1 1 bits)
`
`Z60
`
`If backward_f I= 1 &&
`
`motion_vert_bk_code !=0
`
`I
`
`Motion_vert_bk_r (1-6 bits)
`
`If macroblock_pattern
`
`( 3- 9 bits)
`
`Coded_b19ck_pattem
`
`
`
`
`Do all Macroblocks in the slice
`
`Block Layer E
`
`

`
`U.S. Patent
`
`Sep. 14,1999
`
`Sheet 18 of 39
`
`5,953,506
`
`Block Layer
`
`For I=0to5
` i{f(pattem_code[I])
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
` 262
`dct_dc_differential (1-8 bits)
`
`
` dct_dc_size_chrominance (2-8 bits)
`
`
`
`
`dct_dc_differential (1-8 bits)
`
`if(macroblock_intra)
`{
`
`if(I < 4)
`{
`.
`
`dct_dc_size_luminance (2-7 bits)
`if(dc_size_luminance != 0)
`
`{ }
`
`if(dc_size_chrominance != O)
`
`{ }
`
`
`
`
`fig. 11c
`
` Block layer definition for the base Sigma
`Stream 2(0)
`
`
`} { } e
`
`nd_of_block (2 bits)
`
`
`dct_coeff_first (2-28 bits)
`
`

`
`U.S. Patent
`
`Sep. 14,1999
`
`Sheet 19 of 39
`
`5,953,506
`
`6
`
`'1. 15¢
`
`.
`
`400
`
`Client Computer
`5001
`
`#1
`
`
`
`
`
`
`Adaptive
`Stream
`
`Client
`
`l ! |
`
`:
`
`Adaptive
`Stream
`Server
`
`Client
`
`Adaptive
`Stream
`Server
`
`Adaptive
`Stream
`
`500 2
`
`500 3
`
`
`
`Client Computer
`#2
`
`Client Computer
`#3
`
`Client Computer
`#N
`
`500 N
`
`Stream Server
`
`

`
`U.S. Patent
`
`Sep. 14,1999
`
`Sheet 20 of 39
`
`5,953,506
`
`Profile
`
`Determine streams to transmit
`(See Fig. 15B)
`
`Transmit
`
`(See Fig. 15C)
`
`gag. 15:21
`
`
`
`
`
`556
`
`End of
`session?
`
`558
`
`Yes
`
`552A
`
`552B
`
`552C
`
`
`
`fig. 15991
`
`
`
`
`
`
`
`552
`
`554
`
`

`
`tHEtaP5U
`
`Sep. 14, 1999
`
`Sheet 21 of 39
`
`5,953,506
`
`Qwwk90mmmsemQNm
`
`com?EmNww
`
`moo
`
`
`
`newmmvommcom
`
`oomoi
`
`Em
`
`oz.
`
`mo...SmEN8.
`
`emumam
`
`8.»
`
`OS»
`
`SmcomSmom.
`
`oomA—m
`
`8v
`
`mmmh_.Nom:NE
`
`oovSm
`
`mow::
`
`_N_.9:Bm:mi.N:E.
`
`—N
`
`8am
`
`8_:&
`
`85F
`
`3.5.,
`
`3.5.
`
`35
`
`8%m2w
`
`moo
`
`5.
`
`
`
`mm.‘omm
`
`com
`
`comoi
`
`omv
`
`Nov
`
`SmEm
`
`omw
`
`ownmam
`
`OS»
`
`8m
`
`oomEm
`
`om.
`
`cooNB
`
`.3mmm
`tmom:
`
`No.
`
`E
`
`2:E
`
`oowSmcom9:
`
`mi.N:
`
`E
`
`Fm
`
`8.8
`
`mévm
`
`8.5_
`
`8.5_
`
`8.2
`
`3.6
`
`mawkma
`Evmama
`0:.99®:.__.m._n_
`
`amamw.33
`
`SNSMN
`>Em.9:9“.3
`
`
`
`mawk.mamm
`3mamaw:99wEm.u.
`
`oom.m5
`
`N_.w
`
`mom
`
`newN?ommcom
`
`Smoi
`
`Em
`
`omv
`
`mowcomoum©m_.
`
`ownmam
`
`Sq
`
`O;
`
`ommcomo_.Now;
`
`So.\.E
`
`9:,
`
`5.‘
`
`mumM—Nom:mo.
`
`oov85
`
`SN
`
`com
`
`mm:m:N:on
`
`8.5m
`
`83m
`
`8.5_
`
`3.5_
`
`3.5F
`
`mom5P
`
`mm:
`
`52:3
`
`9:3.5
`
`«mamH.5
`
`25tm
`
`EvmamCm
`
`oovown
`
`commm:mi.
`
`Smoi
`
`one
`
`omv
`
`mow8m
`
`Em
`
`ownmam
`
`om.‘
`
`OIn
`
`ownoom
`
`EN
`
`8Em5
`
`N_.m
`
`moo
`
`NewN?
`
`Bm
`
`oom
`
`85m
`
`Sim
`
`3.5_
`
`35_
`
`85_
`
`03.RFQ2
`
`mom3:9:am
`
`mawkmaEmw:
`Qvmama99mEm.E
`
`umamm.m§
`
`
`
`
`
`

`
`U.S. Patent
`
`Sep. 14,1999
`
`Sheet 22 of 39
`
`5,953,506
`
`5
`0
`fhg. 1 C
`
`554A
`
`Send adaptive stream I.D. & headers
`
`5543
`
`Send group 2 picture code sequence
`
`554“
`
`554D
` Send each of X1 -7
`
`incrementally as appropriate for
`short code sequence
`
`
`
`
`Profile
`9
`”‘’date‘
`
`No
`
`Yes
`
`G0rt)‘:off<;\(;ieW
`(See Fig. 15A)
`
`

`
`U.S. Patent
`
`Sep. 14,1999
`
`Sheet 23 of 39
`
`5,953,506
`
`gay. 1 am
`
`Determine CPU
`8/8
`7/8
`6/8
`
`I i
`
`'
`
`602A
`
`5023
`
`Set video
`
`preference
`
`See attached
`
`graph
`(See Fig. 16A2)
`
`504
`
`606
`
`508
`
`610
`
`Establish
`connection
`
`Send
`profile
`
`Make
`selection
`
`Data
`
`612
`
`No%
`
`614
`
`Yes
`
`End
`
`

`
`U.S. Patent
`
`Sep. 14,1999
`
`Sheet 24 of 39
`
`5,953,506
`
`99;. 1 5,92
`
`
`
`10
`-
`
`s 0.5
`
`VlDEO__ONLY
`BE'l'l'EFt_VlDEO
`o
`f()
`DEFAULT
`‘at {if BETTER_AUDlO
`E
`PREFAV<
`
`m
`
`AUDlO_ONLY
`
`28
`
`64
`
`128
`
`256
`
`512
`
`1024
`
`1536
`
`BWNET
`(kbps)
`
`0
`
`fig. 16543
`
`1.0
`
`0-5
`
`CPURA
`RPREFAV
`
`
`Better Audio
`
`Default
`
`Better Video
`
`1.0
`
`2.0
`
`3.0
`
`4.0
`
`5.0
`
`486/66
`
`Pentium/166
`
`PentiumPro/200
`
`ADR: Audio decode time
`
`ratio (LCD platform relative
`to other platforms)
`
`

`
`U.S. Patent
`
`Sep. 14,1999
`
`Sheet 25 of 39
`
`5,953,506
`
`99
`95'
`15. 1 6
`
`520
`
`Receive stream |.D. & headers
`
`
`
`Receive group code sequence
`
`Receive each of 21 -27
`(incrementally)
`
`622
`
`523
`
`530
`
`632
`
`Send data for picture to adaptive
`stream decoder (See Fig. 16C)
`
`Send MPEG result to MPEG
`decoder
`
`Go to Step 602
`at |=ig_ 15A1
`and make profile
`
`634
`
`Yes
`
`_
`New profile?
`
`

`
`U.S. Patent
`
`Sep. 14,1999
`
`Sheet 26 of 39
`
`5,953,506
`
`gay. 1 ac
`
`Receive group start code MPEG
`group headers
`
`
`
`650
`
`552
`
`554
`
`555
`
`558
`
`660
`
`652
`
`668
`
`Receive MPEG picture headers
`
`Receive slice start code
`
`Receive MPEG slice headers
`
`Receive all macroblocks in slice,
`reconstruct DCT coefficients
`
`New slice
`start code?
`
`
`
`664
`
`666
`
`Must be new group
`transmit reconstructed IBP
`
`

`
`U.S. Patent
`
`Sep. 14,1999
`
`Sheet 27 of 39
`
`5,953,506
`
`99;. 1 7
`
` Bare-bones SG
`
`Remaining SG
`
`710
`
`712
`
`
`
`Compute
`
`
`Object
`Graph
`
`Multires
`
`Optimized
`Scene
`
`
`
`
`Encodings
`
`
`
`
`
`Geometric
`Multires
`Encoding
`
`
`
`
`
`714A
`
`|
`
`I
`
`:
`=
`|: I
`El
`:
`D j ;
`|::3 '
`I Etc
`i_ _ _ _ _ _ _ _ _
`_ _ _ _ _ _ _ _
`
`Global and
`
`
`Other '70“
`Adaptive
`
`data
`
`
`
`
`
`
`
`Tristrip &
`Compress
` Base
`
`
`
`Compressed Meshes
`
`
`
`Base Meshes
`on disk
`
`
`
`Compressed Encodings
`
`VSp|it Lists
`on disk
`
`714C
`
`
`
`Material
`Multires
`Encoding
`
`
`
`TextL_ire
`on disk
`
`Mate_ria|
`on disk
`
`:1
`
`I
`
`'
`=
`:
`.
`
`I ;
`
`I
`
`

`
`taP&U
`
`m.&
`
`60S,359,5
`
`
`
`
`
`.mM$4E
`
`_mno_wNutzmmwwom:Eou_:3_.§%.Wqoohfimw
`
`
`
`
`
`ooow.m.«mo_Omo_£m_>23you8%ownsamooheceu_E_.__moomsé
`
`.O
`
`o_£m_>:o:SmSanommmoE68000.0m.“<moHmoco:o_EmSmDomen9%Bfixohommmo.\..Smw
`
`
`
`132w\u.:=m,a.m-=o20womohfimfi3:322oMNofifimobm
`
`MNONSMU
`
`

`
`Sep. 14, 1999
`
`Sheet 29 of 39
`
`5,953,506
`
` U.S.Patent
`
`ES.3
`
`
`
`
`
`
`mmn:uxmH¢mamwumpmzwmmUozImmmqIAwmvnQmnwWcmom#nu"nmvmmmmmmnmuma
`
`mcuoommlmumolamnoamHmvmmmmmmnmumonu”mumolamnoaw
`
`
`
`
`
`
`
`muHm>gmwzxmz
`
`
`
`
`
`uuonmUmcmflmcsnu"mmvozlmmoqlwmw
`
`
`
`
`
`umocmlvmcoflmcsHunmamflnmumzw
`
`
`
`
`
`uuozmUmflwflwfldnu”mmH:uxme¢
`
`
`
`
`
`ummmumaCHmuHw>mo#mapmmamflommmv#QHUmcowmcsnu”muHm>nmmzlxmz
`
`
`
`
`
`
`
`Amecanmma
`
`
`
`
`
`u:Hom3mH>owcacoHummH>mzOMCHUHHO3nu“munoommmumoHmnoaw
`
`
`
`
`
`
`
`
`
`I#wpnmHq_munwHq¢nuwmummucmfiq
`
`
`
`Ammauovumm:Hm>.mu>Qnu"mwnmflqw
`
`
`
`
`
`
`
`
`
`ugmflqq2m>nu”uswflq
`
`

`
`tHEtaD1&nu
`
`Sep.14,1999
`
`Sheet30 0f39
`
`.3
`
`iv.39’
`
`cnu5,
`
`
`
`
`
`
`
`
`
`ommmcwuvaflsoozflamemsouwvcwunaflnuuflHamIoeImflxmnu"mumoIaHmo
`
`
`
`
`
`Amuaa.wuoH.xuHo.HamoImmmqmoov"mmsHm>I.muflnNIn”uflHmmIoeImflxm
`
`III.owmmHamowmoq_ommmcmauHflnoInuflzImsouo
`
`
`
`
`.u...mQHHICQHUHHEUwwvocOCHOI"mm5Hm>
`
`
`£u.H.-3UwumwoommmHamoQua:Umumfloommmmmuocuajuwo
`
`
`
`mumoHUImmuaUmnu”mumsHUIoflwflommmmaaemm
`
`ImaawowMommcflficsomlunulmumoHUE|m®HB|UM
`
`A3m®ESHO>
`
`
`
`
`
`HHwu#MmumoHHwo_uh"mumowmueIum
`
`mummmwneUMnu"mumomm
`
`
`
`
`
`Iucflnmcmflmcmuuwmaamow
`
`
`
`m:am>uaamwnu”ommmCOHUHflEUozaamoQsouw
`
`
`
`
`
`
`
`
`
`bH..o_umsHm>.mpHnmIn"W5Hm>IuHHQm
`
`
`
`
`
`Amvmpflmum>ummmmmmuozcmHnHflao¢Mu“ommmamHuHHnoInufl3HHmoImsouw
`
`
`
`
`
`
`
`
`IIImuflnmHuhcmgmaflgux
`
`
`
`ImmvozcmHuHflgo¢AvVum>ummmmV_mpHn¢ImmuozcmuuHflgo¢V
`
`
`
`
`
`mmmonQ:mmflommmopmuxnuxmcmnu®NHHHUD.mm»H..cmuUHHLumo#mapmwflummmopmuflnvuxmcmapmm:
`
`
`
`
`umsm.02uoHmm:Hm>
`
`IIAfimpflnm
`
`
`
`panHnancmnuHflcu#IuomImu>mIuxmzImNHaflua
`
`.cmunHHgu
`
`
`
`
`
`cmuuaflgo#IHomImu>mIuxmzImNflaflpanu”momgmIHHmoImmmq
`
`
`
`
`
`
`
`OCHUCSODHN_mmHuwonHI.wwuuUxlnovIumm5Hm>Mmwxflnu”wmmelmm
`
`
`
`
`
`
`
`mam;.3
`
`mumsmmHmummmmmIn”ocfivoocmIAmmvmcficoflufluummamflummm
`
`
`
`
`
`
`IHmumoHumoflmflommmmmmemmgmQ>Hmmnu”nmvmmmmm
`
`

`
`tHEtaP5U
`
`Sep. 14,1999
`
`Sheet 31 of 39
`
`5,953,506
`
`E
`
`w
`
`
`
`EEwcmmb_E_xo.n_
`
`82ama___n_m_>
`
`_u::o.mxomm
`
`
`
`
`
`Emcmmmzmcamgomcwm.wu:__>Ogomcwmocm_n_
`
`
`
`
`
`
`
`E$9:26:SmbwnfiSwcm
`
`ELowcwm0_.:_._.Fm
`
`
`Eco;mmEmama2mc_t2m:«mmmom“.vwxmcc_
`
`
`
`
`
`
`
`mzwcamn_mu:__>odcoo.xom
`
`
`
`
`
`$9:mvo:Smzmnfim.Bm_oEmE_
`
`
`
`a__oo__§<
`
`
`
`._Em._o9wE_:o:mEm_.O
`
`
`
`.o..m._on_._wE_:o_.._mon_
`
`H E.co_m___oo
`
`
`—FE:Boo:..om:mn_mEm:V
`
`—I
`
`
`
`9m:_EooOm::xm.._.
`
`=-
`
`.53.§
`
`
`
`Em_.__aco_62_o
`
`:33E_on_
`
`Em:Sam
`
`::o..mcm.P
`Eozflom.co:m_m:9.r_m_mommmu:_oc:
`
`
`
`
`
`
`
`
`
`
`
`
`8:.:o:mm_>mz$9:2.0:Homzwomv>:mEom0J
`
`
`
`
`
`
`
`
`
`

`
`U.S. Patent
`
`Sep. 14,1999
`
`Sheet 32 of 39
`
`5,953,506
`
`929. 19
`
`

`
`Sep. 14, 1999
`
`Sheet 33 of 39
`
`5,953,506
`
`Exam?%OS
`
`ommflmxE0:
`
`EmomfiE02
`
`.8:_omSam
`
`NN §
`
`98?80.30
`
` U.S.Patent
`
`mmB
`
`
`
`

`
`U.S. Patent
`
`Sep. 14,1999
`
`Sheet 34 of 39
`
`5,953,506
`
`
`Open SSF File to get Server name
`and file name. Call Server with file
`
`
`
`name and profile. Set up the 1/0 to
`receive data from the server.
`
`
`Receive global scene data and the
`k-d tree. Set up internal data
`structures
`
`
`
`
`750
`
`752
`
`754
`
`
`
`Merge any data received from
`
`Draw current frame. While
`drawing, accumulate Visibility
`information and performance
`
`
`
`statistics
`server into render data
`
`
`
`
`Communicate the stats to the level
`of detail module to compute new
`
`information required and old
`
`information not needed any more
`
` Transmit to server the messages to
`send/stop desired/undesired data
`
`

`
`U.S. Patent
`
`Sep. 14,1999
`
`Sheet 35 of 39
`
`5,953,506
`
`Eq. 24
`
`802
`
`
`
`I __
`5
`
`
`
`Data
`Dictionary
`
`Insert data into object
`------------------------------------ --.
`'
`
`MU'tiP‘9_X€‘d
`Streaming
`Data
`
`....
`
`
`
`
`
`<------------- -- Data Structure Access
`
`T Program Flow
`
`
` Stream
`Management
`Module
`
`
`808, lmager
`
`ttttttttt at . =
`
`
`814i
`
`
`
`eeeeeee
`
`
`

`
`tHEtaP&U
`
`Sep. 14, 1999
`
`Sheet 36 of 39
`
`60S,359’5
`
`
`
`_w_._3m~=dmmfi
`
`germ.:s5e<
`
`
`
`uocooaou.§:8n_m
`
`>3Q.=.5x®,F
`
`E
`
`
`
`DHDHXOF®>_a_Ufl<
`
`2.30%RS...Allg
`
`25:22
`
`xomm23¢
`
`.5>.5mSowO:
`
`
`
`65:8...E.<ia=o.zu_v2:
`
`..3Eu<252:uEEbBQEa
`
`
`
`
`

`
`U.S. Patent
`
`Sep. 14,1999
`
`Sheet 37 of 39
`
`5,953,506
`
`fig. 26
`
`Statistics
`
`LOD Evaluator
`
`+ Add/remove vertices
`
`+ Change rendering modes
`+ Add remove texture and
`
`material detail
`
`+ Increase, decrease viewport
`size
`
`+ Change relative priority of
`objects
`+ Suspend / Resume request
`
`Determination of control
`
`messages for Server
`
`Update of current
`frame data
`
`

`
`U
`
`Heta
`
`S
`
`M,
`
`wMwM...MS
`
`S’
`
`3
`
`M
`
`pNs”N.
`
`mmeanAuo
`
`t2%..vc=
`
`
`
`33.nvvnvEwanE+P.2_2mu€33._
`
`
`
`88$mmobm889$“E9:Q04uaDOA%:§?2_o_m-=H«EU3:-.1.ago?
`
`5,2
`
`%__
`
`............................................................o
`
`
`
`a._:m_unnE_o&§>Boa8_§m_QtinoDOu_u:82ommm
`
`
`

`
`U.S. Patent
`
`Sep. 14,1999
`
`Sheet 39 of 39
`
`5,953,506
`
`905
`
`_ . - - -
`
`_ _ . . . - - - - - - ' ' ’ ' ' "
`
` "
`
`s.?O3
`
`-.--I-..-..’------
`
`. - . - ' ‘ -.
`
`

`
`5,953,506
`
`1
`METHOD AND APPARATUS THAT
`PROVIDES A SCALABLE MEDIA DELIVERY
`SYSTEM
`
`FIELD OF THE INVENTION
`
`The present invention relates to apparatus and methods
`for providing a scalable media delivery system capable of
`encoding, transmitting and decoding multimedia informa-
`tion in the form of streamed digital data.
`
`BACKGROUND OF THE RELATED ART
`
`Many standardized formats exist for creating digital sig-
`nals that allow for images and sounds to be recorded, stored,
`transmitted and played back. Such formats include the
`MPEG format for digital video, VRML format for 3-D
`graphics and MPEG and WAV formats for digital audio.
`Each of these formats is capable of storing sufficient infor-
`mation with respect to a particular image or sound that
`allows for extremely accurate reproduction of the image or
`sound.
`
`Despite the fact that these formats allow for conceptually
`distortion free reproduction of images and sounds, limita-
`tions in the computational power of computer systems and
`network bandwidth limitations prevent reproductions that
`are as accurate as desired while meeting real time con-
`straints. For more compact and distortion free reproduction,
`larger quantities of data and/or faster processing is typically
`required. Accordingly, the digital information that is typi-
`cally encoded in a given format provides less than optimum
`resolution so as not to exceed the computational power for
`decoding available in an “average” computer system and the
`network bandwidth limitations. Unfortunately, however,
`computing systems having computational power and avail-
`able bandwidth that is greater than “average” cannot use the
`extra computational power they contain and available band-
`width to reproduce images and sound with even greater
`performance and clarity, since the originally encoded signal
`contains no further information with which to obtain this
`greater resolution.
`Conversely, if the digital information that is encoded in a
`given format that provides optimum resolution when being
`decoded by a high end computer system, other “average”
`computer systems are unable to decode all of this additional
`digital information in real time and, therefore, will be unable
`to reproduce any sound or image at all.
`Accordingly, there is a need to for a method and apparatus
`that allows for the high-end computer systems to decode as
`much digital information as possible so that they can repro-
`duce images or sounds with optimum resolution for high
`performance computer systems at the available bandwidth
`and also provide for “average” or low-end computer systems
`that receive lesser amounts of information corresponding to
`their performance capabilities, as well as taking into con-
`sidcration bandwidth limitations. Thus, for all of these
`systems, there is the need to receive digital information that
`is matched to the computational power available.
`Further, there is the need for servers to be able, in real
`time,
`to determine the amount of digital
`information to
`transmit and then transmit this digital information while
`rriinimizing the computational power required to perform
`such operation.
`SUMMARY OF THE INVENTION
`
`therefore, an object of the present invention to
`is,
`It
`provide a method and apparatus for reproducing sounds
`
`10
`
`15
`
`20
`
`30
`
`35
`
`40
`
`45
`
`50
`
`60
`
`65
`
`2
`and/or images with a resolution that is optimized to the
`capabilities of the client computer that is decoding previ-
`ously encoded sounds and/or images.
`It is also an object of the present invention to provide a
`method and apparatus for encoding digital data representing
`sounds and/or images as base streams and additive streams
`of digital data.
`It is another object of the present invention to provide a
`method and apparatus for transmitting base streams and a
`desired number of additive streams of digital data from a
`stream server
`to a client computer based on a profile
`obtained from the client computer.
`It is a further object of the present invention to provide a
`method and apparatus for decoding base streams and addi-
`tive streams of digital data to allow for accurate reproduc-
`tion of sounds and images.
`It is a further object of the present invention to provide a
`method and apparatus that allows for variation in resolution
`of different media forms so that the quality of a media form
`such as sound can be increased at the expense of the quality
`of another media form, such as picture image, according to
`the desires of the user.
`
`It is a further object of the present invention to provide a
`method and apparatus that allows minimal processing by the
`server to achieve the objects recited above.
`In order to obtain the objects recited above, among others,
`the present invention provides an apparatus and method for
`encoding, storing,
`transmitting and decoding multimedia
`information in the form of scalable, streamed digital data. A
`base stream containing basic informational content and
`subsequent streams containing additive informational con-
`tent are initially created from standard digital multimedia
`data by a transcoder. Client computers, each of which may
`have different configurations and capabilities are capable of
`accessing a stream server that contains the scalable streamed
`digital data. Each different client computer, therefore, may
`access different stream combinations according to a profile
`associated with each dilferent client computer. Thus, the
`streams accessed from the server are tailored to match the
`
`profile of each client computer so that the best combination
`of streams can be provided to maximize the resolution of the
`3D, audio and video components. Since different stream
`combinations can be accessed, this advantageously allows
`for the various combinations of content and resolution that
`
`are tailored to match that of the specific client computer. If
`desired, however,
`the profilc can be further adapted to
`increase the resolution of certain characteristics, such as
`sound, at the expense of other characteristics, such as video.
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIGS. 1 is a block diagram of a transcoder according to
`the present invention for converting standard digital multi-
`media data into digital streams using a transcoder according
`to the present invention;
`FIG. 2A is a block diagram illustrating a stream manage-
`ment module according to the present invention that selects
`base and additive streams for use by a multimedia device;
`FIG. 2B an example of digital streams being used with a
`multimedia device through a network having many client
`devices according to the present invention;
`FIG. 3 illustrates the use of digital streams according to
`the present
`invention in application and/or presentation
`layers for media based on the OSI reference transport model;
`FIG. 4 illustrates properties of conventional digital audio/
`video format;
`
`

`
`5,953,506
`
`3
`FIG. 5 illustrates a block diagram of an MPEG coded
`video stream being transcoded into an adaptive layered
`stream according to the present invention;
`FIGS. 6A—C illustrates embodiments of adaptive layered
`streams derived from a block of MPEG data;
`FIGS. 7A—D illustrate various adaptive layered streams
`according to a preferred embodiment of the present inven-
`tion;
`FIGS. 8A and 8B illustrate the contents of a slice of
`MPEG data and illustrate the preferred embodiments
`method of obtaining adaptive layered streams for each
`macroblock of MPEG data;
`FIGS. 9A, 9B1A, 9B1B, 9B2 and 9C provide a flow chart
`that illustrates a method of creating a base adaptive layered
`stream from a stream of MPEG data according to the present
`invention;
`FIG. 10 provides a flow chart that illustrates a method of
`creating correction codes according to the present invention;
`FIGS. 11A—11C filrther illustrates step 160 in FIG. 9B1B;
`FIG. 12 illustrates an overview of an adaptive stream
`management apparatus including an adaptive stream server
`and an adaptive stream configured computer according to
`the preferred embodiment of the present invention;
`FIGS. 13-14 illustrate a sequence that can be used to
`establish communication between an adaptive stream server
`and an adaptive stream configured computer according to
`the preferred embodiment of the present invention;
`FIG. 14 illustrates a more detailed block diagram of
`components of the adaptive stream server according to the
`preferred embodiment of the present invention illustrated in
`FIG. 3;
`FIGS. 15A and 15B1 illustrates block diagrams of a
`sequence of steps used at the client computer according to
`the present invention;
`FIG. 15C illustrates a transmit sequence at the server
`according to the present invention;
`FIGS. 16A1—A3, 16B, 16C1—16C2 illustrate sequences of
`operations at the client computer according to the present
`invention;
`FIG. 17 illustrates a flow chart of the 3-D transcoder
`
`according to the present invention;
`FIGS. 18A—C illustrate types of graphics adaptive data
`according to the present invention;
`FIGS. 19-21 illustrate a scene, bounded scene and result-
`ing K-D tree according to the present invention;
`FIG. 22 illustrates portions of a dictionary according to
`the present invention according to the present invention;
`FIG. 23 illustrates overall operation of the graphics
`stream processing according to the present invention;
`FIG. 24 illustrates a client computer architecture and
`program flow according to the present invention;
`FIG. 25 illustrates decoder operation according to the
`present invention;
`FIG. 26 illustrates the level of detail evaluation according
`to the present invention;
`FIG. 27 illustrates the level of detail function according to
`the present invention
`FIG. 28 illustrates 3D decoder controlling video
`sequences and spatial resolution in dependence upon dis-
`tance from the camera according to the present invention.
`DETAILED DESCRIPTION OF THE
`PREFERRED EMBODIMENT
`
`4
`into what will be called adaptive (or scalable) digital
`streams, such as adaptive digital stream I4, which are
`created so that subsets of the digital data that allow for
`distortion free reproduction of images and sounds at differ-
`ent resolutions, depending on factors discussed further here-
`inafter. Operation of transcoder 10 will be explained
`hereinafter, but is initially mentioned to clarify that the
`present invention can operate upon standard digital multi-
`media data that is stored in one of a variety of formats,
`MPEG, YUV, and BMP formats for digital video, VRML
`form at for 3-D graphics and MPEG, WAV and AIFF formats
`for digital audio, as well as be implemented from a multi-
`media signals that are not digitized.
`FIG. 2A illustrates that the adaptive digital streams 14
`according to the present
`invention can be identified as
`having various components, specifically that of a base
`stream l4Ab, a first additive stream 14A1, a second additive
`stream 14A2, a third additive stream 14A3 to an nth additive
`stream 14A". Adaptive streams I6 and 18 are illus