WO 97/30351
`
`POCT/ELST0885
`
`a decoder for decompressing the encoded database on the client computer into a
`sequence of frames for real time display.
`There is also provided in accordance with another preferred embodiment of the
`present invention a system for transmitting model based data representations of three
`dimensional images over plural transmission links having limited bandwidth, said system
`inchiding:
`
`a digital data source storing model based data representations of three dimensional
`images:
`
`an image processor for rendering views of said model based dats representations
`into raster bitmap format;
`a digital data receiver receiving said digital data in said raster bitmap format over a
`one of the plural transmission links having limited bandwidth: and
`a digital data transmitter operative to transmit the digital data in said raster bitmap
`format to said receiver over a transmission fink having a limited bandwidth in plural
`blocks which are sequentially transmitted at a rate determined by the limited bandwidth,
`each block being an incomplete collection of data which includes parts of multiple
`frames, each frame being viewable in a selectable order by said receiver even when less
`than all of the plural blocks have been received, receipt of subsequent blocks by the
`receiver being used to cumulatively improve the quality of the digital data viewed bythe
`receiver,
`
`Further in accordance with a preferred embodiment of the present invention the
`model based data representations comprise VRML representations.
`Still further in accordance with a preferred embodiment ofthe present invention
`ithe model based data representations comprise CAD-CAM representations.
`Additionally in accordance with a preferred embodiment of the present invention
`the image processor is operative to render only views which are selected by a user,
`There is also provided in accordance with another preferred embodiment of the
`present invention a method for transmitting model based data representations of three
`dimensional images over plural transmission links having limited bandwidth, said system
`including:
`storing model based data representations of three dimensional images,
`
`25
`
`SUBSTITUTE SHEET (RULE 26}
`
`IPR2022-01227
`IPR2022-01227
`EXHIBIT 1003 - PAGE 01451
`EXHIBIT 1003 - PAGE 01451
`
`

`

`WO 97/3055
`
`PCT/ILS70055
`
`rendering views of the model based data representations into raster bitmap format,
`receiving the digital data in said raster bitmap format over a one of said plural
`transmission links having limited bandwidth; and
`
`the receiver over a
`to
`transmit the digital data in said raster bitmap format
`transmission link having a limited bandwidth in plural blocks which are sequentially
`transmitted at a rate determined by the limited bandwidth, each block being an
`incomplete collection of data which includes parts of multiple frames, each frame being
`viewable in a selectable order by the receiver even when less than all of the plural blocks
`have been received,
`receipt of subsequent blocks by the receiver being used to
`cumulatively improve the quality of the digital data viewed by the receiver.
`Further in accordance with a preferred embodiment of the present invention the
`model based data representations comprise VRML representations, and CAD-CAM
`representations.
`
`Stil further in accordance with a preferred embodiment of the present invention
`the image processor is operative to render only views which are selected by a user.
`The following definitions are employed throughout the specification and claims:
`RESOLUTION--The relationship between the number of digital samples per unit
`of an original and the number of digital samples per unit in a rendered version thereof.
`Specifically, when dealing with images, resolution refers to the relationship between the
`number of pixels per unit area of an original image or scene and the number ofpixels per
`unit area in a displayed image. Specifically, when dealing with audio, resolution refers to
`the relationship between the number of samples per unit time of an original sound and the
`number of samples per unit time in a played sound.
`
`QUALITY--The degree to which a rendered version of an original is faithful to the
`original. Specifically, when dealing with images, quality refers to the degree to which the
`displayed image ts faithful to the original image or scene. Normally this is expressed as
`the degree to which the approximation ofpixel values in the displayed image approaches
`the correct pixel values in the original image or scene. Specifically, when dealing with
`audio, quality refers to the degree to which a played sound is faithful to the original
`sound.
`
`25
`
`SUBSTITUTE SHEET (RULE 26)
`
`IPR2022-01227
`IPR2022-01227
`EXHIBIT 1003 - PAGE 01452
`EXHIBIT 1003 - PAGE 01452
`
`

`

`WO 97/30551
`
`POTELST/O0055
`
`FRAME--A portion of an original which can be independently and interactively
`manipulated. Specifically, when dealing with images, frame refers to a portion of an
`image or of a collection of images which can be independently and interactively
`manipulated. Specifically, when dealing with audio, frame refers to a portion of a sound
`which is delimited im time and can be independently and interactively manipulated.
`BLOCK--A sequentially transmitted collection of partial data which is used to
`build multiple frames. The frames are built up of one or more sequentially transmitted
`blocks, whose contents are accumulated. Specifically, when dealing with images, the
`block contains image data. Specifically when dealing with audio,
`the block contains
`audio data.
`
`PARTIAL FRAME--The part of a frame which is contained in a given block.
`TILE--A window sized pixel array ofa predetermined given size forming part ofan
`unage. For example, tiles partition an image into a plurality of arrays, each of which
`contains an identical numberofpixels.
`
`27
`
`SUBSTITUTE SHEET (RULE 26}
`
`IPR2022-01227
`IPR2022-01227
`EXHIBIT 1003 - PAGE 01453
`EXHIBIT 1003 - PAGE 01453
`
`

`

`WO F355
`
`PCTALLS7/0038
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`The present invention will be understood and appreciated more fully from the
`
`following detailed description, taken in conjunction with the drawings in which:
`
`Fig.
`
`| and Fig. 2 are simplified block diagrams illustrating a system for scalable
`
`representation of multimedia
`
`data
`
`for progressive
`
`asynchronous
`
`transmission,
`
`constructed and operative in accordance with a preferred embodiment of the present
`
`invention;
`
`Fig. 3A is a simplified schematic diagram of the database structure of the present
`
`invention which inchides three databases, embodied within the client-server system of
`
`the present invention,
`
`Fig. 3B is a simplified diagram of a database structure particularly useful in the
`
`client database of Fig. 3A, illustrating its two-dimensional nature;
`
`Fig. 4 is an illustration of the operation of a preferred embodiment of the present
`
`invention:
`
`Fig. 5 is a simplified schematic diagram of a production tool for converting a
`
`digital multimedia file into a progressive scalable database representation for storage on a
`
`server computer in accordance with a preferred embodiment of the present invention,
`
`Fig. 6 is a simplified schematic diagram of the structure of a block within the server
`
`database, partitioned into frames which can be accessed randomly in accordance with a
`
`preferred embodiment of the present invention,
`
`Fig. 7 is a simplified schematic diagram of a decoder for receiving and integrating
`
`data blocks from a scalable database, to form a version of a digital multimedia object for
`
`playback in accordance with a preferred embodiment of the present invention,
`
`Fig. 8 is a simplified schematic diagram of a scalable progressive database for a
`
`video clip in which the first data blocks are used for previewing the video in accordance
`
`with a preferred embodiment of the present invention;
`
`Fig, 9 and Fig. 10 are simplified schematic diagrams of a system for incorporating a
`
`scalable progressive database into a time-based video sequence of frames indexed by two
`
`time scales: a macro and micro seale, in accordance with a preferred embodiment of the
`
`present invention;
`
`28
`
`SUBSTITUTE SHEET (RULE 28)
`
`IPR2022-01227
`IPR2022-01227
`EXHIBIT 1003 - PAGE 01454
`EXHIBIT 1003 - PAGE 01454
`
`

`

`WE O7/3055k
`
`POCTELS7@G083
`
`Fig. 11 is a simplified block diagram of a proxy system used to cache in a central
`hub multimedia data which is transmitted from servers to clients in accordance with a
`
`preferred embodiment of the present invention.
`Fig. 12 is a simplified block diagram of a system for generating a scalable database
`from digital media data, by running a compressorin a feedback loop in accordance with a
`preferred embodiment of the present invention;
`Fig. 13 is a simplified block diagram of a decoder for the database generated by the
`system ofFig. 12 in accordance with a preferred embodiment ofthe present mvention:
`Fig. 14 is a simplified diagram illustrating a scalable progressive database useful for
`a large still image in accordance with a preferred embodiment ofthe present invention;
`Fig. 15 is a simplified diagram illustrating a virtual reality system constructed and
`operative in accordance with a preferred embodiment ofthe present invention:
`Fig. 16 is a simplified fowchari illustrating operation ofthe system ofFig. 15: and
`Fig. 17 is an illustration of the operation of a preferred embodimentof the present
`invention illustrated in Figs. 15 and 16, permission to reproduce Fig. 17? was granted by
`Tecnomatix, Ltd.
`
`29
`
`SUBSTITUTE SHEET (RULE 26)
`
`IPR2022-01227
`IPR2022-01227
`EXHIBIT 1003 - PAGE-01455
`EXHIBIT 1003 - PAGE 01455
`
`

`

`WO 97/30551
`
`BCTLY7/00055
`
`DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS
`
`invention may be better understood
`The preferred embodiments of the present
`given the following technical and theoretical explanation.
`The present invention provides a novel method for representing multimedia data.
`The invention provides a scalable representation, so that the data can be asynchronously
`transmitted to clients having different bandwidth connections, played on-line almost
`immediately after the transmission begins, interactively controlled, and also progressively
`upgraded as it is replayed.
`Although the present invention is described hereinbelow with particular reference
`to image data, it is to be appreciated that it is applicable also to non-image data, such as
`audio data.
`
`When addressing bandwidth limitations, it is natural to think in terms of data rates,
`or velocities. Video players, for example, play at standard rates such as thirty frames per
`second (fps), and require the imagesfor display to be available at this rate. If the images
`are already stored on a local hard disk, then all that is necessary is disk access, which is
`very fast. On the other hand, if the images are streamed in from a server, then in order
`for on-line playback to be possible before a
`full download is finished,
`the rate of
`transmission must be great enough to supply the framesat thirty fps. This does not mean
`that the network link has to transmit the data equivalent to thirty full frames every
`second. Due to compression,
`it suffices if the network transmits thirty compressed
`frames every second.
`
`For example, if the compression achieved is 10:1, then it suffices to transmit at a
`rate of three fps, provided that the client CPU can decompress thirty compressed frames
`into full frames every second.
`In fact, compression is the mediator between the video
`player and the bandwidth. The player does not slow down when bandwidth is low;
`rather, the compression ratio has to be greater. Should a bottleneck arise, and a frameis
`not available when the player needs it,
`then the player simply skips that frame, but
`continues to expect frames at the thirty fps rate. The video can be preset at the outset
`for lower rates than thirty fps, but not much lower, since slow video playback breaks the
`continuity between frames, and thus loses the effect of motion.
`
`30
`
`SUBSTITUTE SHEET (RULE 26)
`
`IPR2022-01227
`IPR2022-01227
`EXHIBIT 1003 - PAGE 01456
`EXHIBIT 1003 - PAGE 01456
`
`

`

`WO 87/30351
`
`PCT/TLS7/80085
`
`Thus, realizing that higher compression means lower quality, it can be appreciated
`that users having high bandwidth connections can receive high quality video for on-line
`playback, and users having low bandwidth connections receive low quality video.
`In a
`non-scalable setting, it is thus necessary to prepare different compressed media files for
`each playback rate and bandwidth combination. For example, four different versions of
`the media could exist for (a) 24 fps playback, 14.4 Kbs bandwidth, (b) 24 fps playback,
`28.8 Kos bandwidth, (c) 30 fps playback, 14.4 Kbs bandwidth, (d) 30 fps, 28.8 Kbs
`bandwidth. As described below, using the present invention a single media file can be
`used to accommodateall four of these combinations.
`
`in the present invention, the media data is comprised of mt frames F;, Fo, .... Fy. A
`frame can be, for example, an individual frame of a movie sequence, a piece of a
`panoramic view, an individual segment of an audio signal, or even a sub-sampled version
`of a large still image.
`It can also be a group of such frames, such as for example, a group
`of inter-frames between key frames in a video segment, in a case where an H.263 codec
`is being used.
`In broad terms, frames are units of interactivity. For example, in object
`movies where interactivity means frame advance, a frame unit is an individualstill image,
`whereas in gazing applications where interactivity means zooming in and out, the frame
`units are multi-resolution tiles.
`
`.., B,
`The representation encodes the media data into n data blocks B,, B,
`preferably of roughly equal size. Each encoded data block 8; contains m compressed
`frame units F/, F/, .... F,/. Thus it is appreciated that the database is arranged in two
`dimensions, corresponding to blocks and frames. The dimension used for blocks is for
`achieving progressiveness, and the dimension used for
`frames
`is
`for achieving
`interactivity. The frame data can be transmitted in a selective order, but the blocks must
`be transmitted in sequence, since they build cumulatively. This is an essential feature of
`the subject invention.
`
`Data block 8; is used to deliver the media at the lowest bandwidth, say/; Kbs; data
`blocks 8, and 8; when integrated together, are used to deliver the media at bandwidth I
`- 2f; Kbs; and in general, for 1 < k < », data blocks 8; 8; ... By when integrated
`together, are used to deliver the media at bandwidth Je» kfi Kos. Each higher bandwidth
`version delivers a higher quality rendition of the media.
`In this way, the representation
`
`34
`
`SUBSTITUTE SHEET (RULE 26}
`
`IPR2022-01227
`IPR2022-01227
`EXHIBIT 1003 - PAGE 01457
`EXHIBIT 1003 - PAGE 01457
`
`

`

`WO 97/303851
`
`PCTALS7/00055
`
`can accommodate multiple clients connecting to the network with different bandwidths.
`Moreover, a client connecting with bandwidth #; who has downloaded data block &, and
`played the medis, can continue downloading block 8B;
`in background,
`since the
`bandwidth has been freed, and thereby achieve the same quality as bandwidth ( the next
`time the media is replayed.
`Within each block the frames can be accessed randomly and delivered selectively,
`so that the user can vary the quality level among the frames. For example, a viewer who
`wanis to gaze at frame #3 may instruct the database to send frame #3 data from the frst
`ten blocks, but only ane block of data for all of the other frames. The viewer selection is
`carried out interactively, through the use of keyboard presses and mouse clicks, as the
`media is being played. Whereas for some applicationsit may be most natural to transmit
`the entire blocks in sequence, for other applications it may be more effective to first
`deliver as much data as possible for specific frames at the expense of lowering the quality
`of other frames. The two dimensionality of the database, together with its interactive
`accessibility, gives the user complete control over the transmission sequence.
`The scalable representation that is the subject of the present invention is embodied
`in a production tool which enables the producer to control the bandwidth parameters f,,
`or equivalently, the qualities of the media versions obtained by integrating blocks B,, B,,
`. By.
`In general,
`it
`is not necessary that the blocks be of equal size, nor that the
`frequencies f, be given by ky), although this is the preferred embodiment.
`The
`production tool also enables the producer to control the final quality of the highest
`bandwidth version, or equivalently,
`the total number,
`nw, of data blocks
`in the
`representation.
`
`in contrast, if the media data representation is not scalable, but is encoded instead
`for a specific bandwidth f then only clients with bandwidth J oF greater can play the
`media on-line as it is being downloaded. A client with a lower bandwidth than J would
`have to download the entire data stream to memory in order to begin playback, which
`can take a great deal of time on account of the large file sizes typically used in
`mulumedia production. A client with a higher bandwidth connection thanJ would not be
`able to take advantage ofit to receive higher quality media. Moreover, there would be
`no means of upgrading media quality, even for clients with high bandwidth connections,
`
`32
`
`SUBSTITUTE SHEET (RULE 26)
`
`IPR2022-01227
`IPR2022-01227
`EXHIBIT 1003 - PAGE 01458
`EXHIBIT 1003 - PAGE 01458
`
`

`

`WO 97/30551
`
`PCTALS7/00055
`
`other than to transmit an entirely new data stream from the server side, and discard the
`previously downloaded data.
`
`scalable audio and video
`inter alia,
`Applications of the invention include,
`transmission, video previewing, progressively rendered object movies and panoramas,
`large still images, efficient proxy or multi-casting unit (MCU) management for web and
`other hubs, and VRML transmission, as described hereinafter in greater detail.
`Scaleable audio transmission: Digital audio data can be progressively encoded into
`a scalable database for asynchronous transmission at different bandwidths. A client
`connected with a low bandwidth line can receive a low quality version of the audio,
`which can be played back on-line at the low bandwidth as the data streams in. After the
`audio is played, additional data blocks can continue to be received and integrated with
`the previous blocks, so that the audio is upgraded to higher quality for replay.
`
`scaleable video transmission’
`Similar to the description above for the audio
`transmission, digital video data can be encodedinto a scalable database for asynchronous
`delivery and progressive quality upgrade.
`For certain time-based video systems, there is disclosed a novel way to incorporate
`the scalable database so that progressiveness and immediate playback can be achieved,
`even in a single-play mode. Specifically,
`this applies to video systems with two time
`scales, such as is present in the Apple QUICKTIME” movie player. The first time scale
`(hereinafter referred to as the “major scale”) is used to advance from one frame to the
`next, based on major units of time. The second time scale (hereinafter referred to as the
`“minor scale”) is a sub-division of the major scale into smaller time units, and is used to
`incorporate small changes or fluctuations into the frame being displayed. For example,
`the major scale can be advancing through a movie ofa bird flying and the minorscale can
`be adding fluttering to the bird’s wings. The advantage of such a two-scale player is that
`the decoder, which does the intensive processing to supply the frames, need only run at
`the slowerrate, ¢.g. 3 frames per second (fps), governed by the major scale, whereas the
`viewer, doing the less intensive processing,
`is playing at the faster rate, e.g. 30 fps,
`governed by the minorscale.
`The subject invention can be incorporated into a system having two time scales as
`described hereinabove by using the minor scale in a way different from the way that was
`
`33
`
`SUBSTITUTE SHEET (RULE 26)
`
`IPR2022-01227
`IPR2022-01227
`EXHIBIT 1003 - PAGE 01459
`EXHIBIT 1003 - PAGE 01459
`
`

`

`WO S97/30851
`
`PCTELS7/00055
`
`Instead of being used to introduce fluctuations, it is used to display
`originally intended.
`progressively rendered versions of a frame. At each minor time unit. the player displays
`the latest version available of the frame indexed by the major scale. For example,
`suppose there are ten minor time units within the major time unit during which frame #4
`is to be displayed. Az the first minor time unit, the player imitially displays the version of
`frame #4 which it has available from the first data blocks already processed. As
`additional blocks of data are accumulated and higher quality versions of frame #4
`become available, the player displays those frames at successive minor time units. This
`continues for ten minor time units, until the next major time unit, at which time frame #5
`is to be displayed. The cycle then repeats, and the version of frame #5 which is already
`available is initially displayed at the first minor time unit. Thus it can be seen that
`progressiveness can be achieved by interjecting into the minor scale, the versions of the
`frames obtained by accumulating successive blocks.
`For the user to be able to view the video immediately, without waiting for the
`entire file to download, the production tool must store the encoded video in the order of
`successive blocks. Each partial frame must be handled as if it were an entire frame. That
`is, the production tool must treat the movie as if there were a total of mn distinct
`frames being encoded. On the other hand, each frameis sent only once to the codec for
`encoding, and is returned as a series of encoded partial frames. Thus it is necessary to
`post-process the encoded data file, to rearrange the data items from a frame dominated
`order to a block dominated order.
`This rearrangement process is referred to as
`“flattening” in the art.
`The player in turn, however, must know that althoughit is receiving what appears
`to be m - # data items, there are really only a total of m frames.
`It must decode and
`accumulate every successive sequence ofm data items with the previous ones, to update
`the frames. The combined effect of the flattening on the production side and the player’s
`interpretation on the client side enables seamless integration of the scalable progressive
`database within a non-progressive video interface.
`That
`is,
`the incorporation of
`progressive blocks does not require any modifications to the existing interface,
`Video previewing: When encoding digital video data into a scalable database, the
`first data blocks cam be used to generate a preview of the video, restricted to selected
`
`34
`
`SUBSTITUTE SHEET (RULE 26)
`
`IPR2022-01227
`IPR2022-01227
`EXHIBIT 1003 - PAGE 01460
`EXHIBIT 1003 - PAGE 01460
`
`

`

`WO 97/30851
`
`POTALSTAOS|
`
`frames. The preview can be played back by the client almost immediately after the
`streaming begins. Moreover, additional data blocks received are integrated with the data
`blocks from the preview, to form full view versions of the video.
`Obiect_movies.
`Advertising agencies are using object movies to produce
`interactive 3-D virtual reality presentations of merchandise on the Internet. The user can
`rotate and zoom the 3-D object, and examineit from different viewing angles. Using the
`methodology of the current invention, object movies can be progressively encoded so
`that the viewer can download and begin playing them almost
`immediately after the
`streaming begins.
`Initially the movie will scale to a quality commensurate with the
`bandwidth of the user’s network connection, but as the data blocks are received and the
`user interacts with the movie, additional data blocks are delivered and integrated with the
`previous blocks, resulting in a higher and higher quality movie. An important feature of
`the invention is that, regardless ofbandwidth, the user can begin playback and interaction
`almost immediately, and does not need to wait for the complete download, as the first
`version of the movie delivered scales itself to the native bandwidth. As playback
`continues additional data streams in the background and the movie version is upgraded
`to higher and higher quality.
`
`Panoramas: Panoramas are very large images which the user cannot view in their
`entirety, but rather sees within a restricted viewing window. By panning in various
`directions, and zooming in and out,
`the user navigates through the panorama. The
`continuous change in viewing window gives the effect of movement within a scene.
`Similar to the description above for object movies, panoramas can be progressively
`encoded so that the viewer can download and begin navigating through them almost
`immediately after the streaming begins.
`Initially the panorama scales to the cHent
`bandwidth, and afier the first data blocks are received, additional data blocks are
`streamed in background while the panorama is playing, to provide higher and higher
`image quality.
`Large still images: Althoughlarge high quality still images are not compased of
`frames in the conventional sense, the large sizes of the image data files makes the present
`invention an effective means for interactive on-line viewing. The frames can be small
`image tiles within the full image at different resolutions, the smaller tiles having higher
`
`33
`
`SUBSTITUTE SHEET (RULE 26)
`
`IPR2022-01227
`IPR2022-01227
`EXHIBIT 1003 - PAGE 01461
`EXHIBIT 1003 - PAGE 01461
`
`

`

`WO 97/3055
`
`PCTIELS7/6055
`
`resolution than the larger ones. The locations of the tiles can be marked as hot spots.
`When the viewerclicks on a hot spot within a specific tile, the database delivers that tile
`
`at a higher resolution, giving the effect of a zoom in. Within the higher resclution tile
`
`there can be more hot spots, and the zooming can continue through the database.
`As a simple example, the first frame within a data block may contain the full image
`sub-sampled by 4:1, for example, in each dimension. The next set of frames within the
`
`image sub-
`data block may contain (some subset of) the four quadrants of the full
`sampled by 2:1 in each dimension. The next set of frames within the data block may
`contain (some subset of} the sixteen quadrants within the above four quadrants at the
`orginal resolution. A viewer could see the 4:1 reduction ofthe original image (the first
`block), click on one of the quadrants and then see that quadrant at a 2:1 reduction (a
`frame from the second set), and click further on one of its quadrants and then see it at
`
`full scale (a frame from the third set).
` ge
`hubs within networks, used as large caches for data being delivered from servers to
`clients. Similarly MCUs are large storage devices used as caches for data being delivered
`from broadcasting stations to viewers, such as cable TV. As data is streamed from
`
`: Proxies are large storage devices, located as
`
`servers to chents or from broadcasting stations to viewers upon request, the proxy or
`MCUstores the data in a central hub so that
`it
`is available for delivery at a high
`bandwidth if requested again by any of the clients connected to the hub.
`It plays a similar
`
`role to paging files on a local computer disk, but on a much larger scale and for a much
`
`larger clientele.
`
`The scalable representation of the subject invention is particularly well suited for
`
`proxies and MCUs which operate in asynchronous environments.
`
`Server/client
`
`connections and broadcast transmissions can be of many different bandwidths, and so the
`
`proxy or MCU can be accumulating versions of the same multimedia data corresponding
`to different qualities. Without scalability these versions are all
`independent of one
`another, and cannot be combined to achieve quality levels other than those originally
`preset or combined to save space. Using the scalable representation of the subject
`invention,
`the proxy or MCU can be optimized to cache the various progressive
`
`building blocks. This affords great flexibility in beg able to create versions of different
`
`36
`
`SUBSTITUTE SHEET (RULE 26)
`
`IPR2022-01227
`IPR2022-01227
`EXHIBIT 1003 - PAGE 01462
`EXHIBIT 1003 - PAGE 01462
`
`

`

`WO OT/30551
`
`PCTELST/A0SS
`
`quality levels, and reduces the Space requirements.
`As a simple example, a first user with a low bandwidth connection f; to the
`
`server, who demands a multimedia file, downloads data black 8,, which is then also
`
`saved to cache on the proxy, and the user receives a low quality version (quality level 1)
`
`of the media. A second user with a higher bandwidth connection / to the server, who
`
`demands the same multimedia file, can download data blocks 2, and B; from the server,
`
`and can access block 8, directly from the proxy. The three data blocks are integrated
`and the second user receives a very high quality version (quality level 3} of the media.
`
`Data blocks 8, and 8; would then also be stored on the proxy. A third user with a direct
`
`connection to the proxy of bandwidth who now demands the same multimedia file, can
`receive the high quality version (quality level 2) comprising of blocks 8, and B, directly
`from the proxy. Without a scalable representation it would not be possible for the
`second user, with a bandwidth connection of ff,
`to receive a quality level 3 version
`corresponding to the higher bandwidth f, nor for the proxy to deliver to the third user a
`
`version at a quality level different from those directly available in its cache. Moreover,
`without a scalable representation the proxy memory required to cache the f, f) and ji
`versions would equal the size of six data blocks, rather than three.
`It can thus be seen
`
`that the proxy or MCU inherits the scalability from the servers, giving it a great deal
`more flexibility in its media delivery to the clients than would be possible in a non-
`
`scalable environment.
`
`VRML transmission: Virtual reality modeling language (VRML)is a descriptive
`language for representing and rendering three-dimensional objects. The objects are
`modeled as collections of polygonal elements, the description of which forms a VRML
`database. Depending on the viewing parameters, the individual elements are processed
`and the desired view of the object is rendered into a raster bitmap for display. The
`VRML representation is rich enough to encapsulate all possible views of the object.
`In
`fact, there is an infinity of possible variations in viewing parameters. A user interacts
`with the VRML object by adjusting viewing parameters,
`through mouse clicks and
`keyboard presses. VRML was first popularized by Silicon Graphics. Their top Iris
`workstations, for example, can render on the order of a million polygonal elements per
`second. VRML images are characterized by their sharp photo-realistic attributes.
`
`37
`
`SUBSTITUTE SHEET (RULE 28)
`
`IPR2022-01227
`IPR2022-01227
`EXHIBIT 1003 - PAGE 01463
`EXHIBIT 1003 - PAGE 01463
`
`

`

`WO 97/30581
`
`POT/ELS7/06855
`
`The present invention can be applied to efficiently deliver VRML imagery over a
`server/client network, for on-line interaction. Within the VRML environment itself two
`problems arise when transmitting VRML databases over a network for on-line
`imteraction.
`First,
`there is
`the bandwidth limitation, which inhibits the rate of
`transmission. Second, there is the intense processing on the client side, necessary to
`render the many polygonal elements into a bitmap for viewing,
`It would be preferable to
`have the server (typically a more powerful computer) do the rendering, but then it would
`be necessary to store rendered bitmaps of every possible set of viewing conditions ~ a
`feat that would require on the order ofterabytes of disk space.
`The present
`invention can be used to mitigate the problem by allowing the
`rendering to be done on the server computer without requiring enormous memory, and
`yet enable the client to freely interact with the VRML object in an on-line interactive
`setting. This is one of many examples involving real-time encoding. The invention
`operates by receiving the viewing parameters from the user, rendering the corresponding
`image on the server into a raster bitmap image, encoding the bitmap into progressive
`partial frames and inserting them into a two-dimensional server database. The encoded
`data within the server database is continually streamed from serverto client, enabling the
`client to begin viewing a low quality image as soon as the first partial frame data arrives.
`As the user navigates through the VRML, additional bitmaps are rendered, encoded and
`inserted into the server database. Whenever the user re-traces steps, so that the viewing
`parameters are the same as those selected at some previous stage, the server does not
`need to render the same bitmaps again. Rather,
`the streaming simply continues in
`background, and the quality of the image on the client side is enhanced as additional
`partial