United States Patent (19)
`Poulton et al.
`
`||||||IIII
`US005481669A
`11
`Patent Number:
`5,481,669
`45) Date of Patent:
`Jan. 2, 1996
`
`54 ARCHITECTURE AND APPARATUS FOR
`IMAGE GENERATION UTILIZING
`ENHANCED MEMORY DEVICES
`
`75 Inventors: John W. Poulton; Steven E. Molnar,
`John G. Eyles, all of Chapel Hill, N.C.
`73) Assignee: The University of North Carolina at
`Chapel Hill, Chapel Hill, N.C.
`
`21 Appl. No.: 383,969
`22 Filed:
`Feb. 6, 1995
`Related U.S. Application Data
`
`63 Continuation of Ser. No. 975,821, Nov. 13, 1992, Pat. No.
`5,388,206.
`int. Cl. ............................................ G06F 12/00
`(51
`52 U.S. Cl. ........................... 395/164; 395/163; 395/162
`58) Field of Search ..................................... 395/162-164,
`395/127,425, 650, 118, 140, 141; 34.5/185,
`189, 201
`
`(56)
`
`References Cited
`U.S. PATENT DOCUMENTS
`4,590,465 5/1986 Fuchs ...................................... 395/118
`4,648,045 3/1987 Demetrescu ............................ 3957163
`4,783,649 1/1988 Fuchs et al. ............................ 34.5/189
`4,827,445 5/1989 Fuchs .............
`345/189
`5.388,206 2/1995 Poulton et al. .............. 3957163
`FOREIGN PATENT DOCUMENTS
`
`WO9209947 6/1992 WIPO ............................... G06F 2/14
`
`OTHER PUBLICATIONS
`Fussell et al., A VLSI-Oriented Architecture for Real-Time
`Raster Display of Shaded Polygons (Preliminary Report);
`Graphics Interface '82, pp. 373-380, 1982.
`Shaw et al., A VLSI Architecture for Image Composition, pp.
`185-199, 1988.
`Demetrescu, High Speed Image Rasterization. Using Scan
`Line Access Memories; 1985 Chapel Hill Conference on
`VLSI, pp. 221-243, 1989.
`Gharachorloo et al., Subnanosecond Pixel Rendering with
`
`
`
`Million Transistor Chips; Computer Graphics, vol. 22, No.
`4, Aug. 1988.
`Evans & Sutherland, Design Systems Divisions, Technical
`Report.
`Molnar et al., Pixel Flow: High-Speed Rendering Using
`Image Composition; Computer Graphics, vol. 26, No. 2, Jul.
`1992.
`Molnar, Image-Composition Architectures for Real-Time
`Image Generation, Dissertation submitted to Univ. of
`N.C.-Chapel Hill, 1991.
`Molnar, Supercomputing Power for Interactive Visualizai
`ton; Report of Research Progress, Mar. 1991-Nov. 1991;
`Univ. of N.C.-Chapel Hill, 1991.
`Poulton, Breaking the Frame-Buffer Bottleneck with
`Logic-Enhanced Memories, IEEE Computer Graphics and
`Applications, Nov. 1992, pp. 65-74.
`Primary Examiner-Raymond J. Bayerl
`Assistant Examiner-Kee M. Tung
`Attorney, Agent, or Firm-Bell, Seltzer, Park & Gibson
`57)
`ABSTRACT
`A system for image generation comprising a plurality of
`renderers, each having a geometry processor and a rasterizer,
`that operate in parallel to compute pixel values for a set of
`primitive objects that comprise the image to be rendered.
`The geometry processor transforms graphics primitive
`objects from their native object coordinates to screen coor
`dinates. The rasterizer consists of an array of enhanced
`memory devices having a processor and memory for each
`pixel in a region of a screen. The processors and their
`associated memories operate in SIMD fashion on screen
`space primitive descriptions to compute and store pixel
`values for an entire such region. The enhanced memory
`devices further comprise compositors for combining their
`pixel values, for example, based on a visibility test, with
`those from a corresponding memory device of another
`rasterizer. The image generation system may further com
`prise shaders, which compute pixel colors based on pixel
`values computed in the plurality of renderers, and video
`frame buffers, which store these pixel colors and refresh a
`raster display device. The shaders and frame buffers consist
`of a renderer with additional memory devices that store
`texture and image data. Also disclosed are enhanced
`memory devices and rasterizers for use in the present image
`generation system.
`
`15 Claims, 17 Drawing Sheets
`
`MEDIATEK, Ex. 1025, Page 1
`IPR2018-00101
`
`

`

`U.S. Patent
`
`Jan. 2, 1996
`
`Sheet 1 of 17
`
`5,481,669
`
`FG,
`
`
`
`PROCESSOR
`
`3O
`
`FRAME
`BUFFER
`
`R
`
`R
`
`
`
`
`
`s/
`
`IMAGE COMPOSITION
`NETWORK
`
`MEDIATEK, Ex. 1025, Page 2
`IPR2018-00101
`
`

`

`U.S. Patent
`
`Jan. 2, 1996
`
`Sheet 2 of 17
`
`5,481,669
`
`| Z | O | –
`
`OZ
`
`}}OSSIBOOÀjoj
`
`|SO H
`
`
`
`MEDIATEK, Ex. 1025, Page 3
`IPR2018-00101
`
`

`

`U.S. Patent
`
`Jan. 2, 1996
`
`Sheet 3 of 17
`
`5,481,669
`
`
`
`MEDIATEK, Ex. 1025, Page 4
`IPR2018-00101
`
`

`

`U.S. Patent
`
`Jan. 2, 1996
`
`Sheet 4 of 17
`
`5,481,669
`
`FIG, 4C
`
`RENDERING
`COMMANDS
`
`-- - - - - - - - - - • • • • • •
`
`[] [] [] [] [] [] [] []
`[] [] [] [] [] [] [] []
`
`[][][][][][][][]
`
`DJ [] [] [] [] []\[] []
`[] [] [] [] [] [V] [] []
`
`[][][][][]\[] [] []
`
`
`
`RENDERER
`BOARD
`O
`
`O
`
`28X28-PXEL
`SIMD ARRAY
`(64 EMC's)
`
`MEDIATEK, Ex. 1025, Page 5
`IPR2018-00101
`
`

`

`U.S. Patent
`
`Jan. 2, 1996
`
`Sheet 5 of 17
`
`5,481,669
`
`FIG, 4b.
`
`SHADING
`COMMANDS
`
`3O
`
`GEOMETRY
`PROCESSOR
`OO
`
`SHADER/
`COMPOSTOR
`1 120
`
`SHADER
`BOARD
`5
`
`
`
`O
`
`
`
`
`
`
`
`
`
`
`
`28x28-PXEL
`SIMD ARRAY
`C64 EMC's)
`
`MEDIATEK, Ex. 1025, Page 6
`IPR2018-00101
`
`

`

`U.S. Patent
`
`Jan. 2, 1996
`
`Sheet 6 of 17
`
`5,481,669
`
`FIG. 4C
`
`WDEO
`FRAMEBUFFER
`COMMANDS
`
`-------------------------
`
`30
`
`GEOMETRY
`PROCESSOR
`OO
`
`FRAME BUFFER/
`COMPOSITOR
`- 12O
`
`a
`
`M
`
`
`
`FRAME
`BUFFER
`BOARD
`30
`
`
`
`O
`
`
`
`28x28-PXE
`SIMD ARRAY
`(64 EMC's)
`
`O
`D D
`ODO D D
`EEEEE
`O
`
`O
`
`7O
`
`MEDIATEK, Ex. 1025, Page 7
`IPR2018-00101
`
`

`

`U.S. Patent
`
`Jan. 2, 1996
`
`Sheet 7 of 17
`
`5,481,669
`
`F.G. 5
`
`PXEL
`ALU
`A,B,C
`MEMORY
`MICRO-
`DATA
`INPUT INSTRUCTION ADDRESS
`
`
`
`4. BSCE
`OF LOCAL
`DAA PORT
`
`COM
`POSITOR
`BUFFER
`
`LOCAL
`DATA
`BUFFER
`
`NEAR
`EXPRESSION
`EVALUATOR
`(OUTPUTS
`Ax+By+C)
`
`2%PSL
`
`COMPOSTOR
`
`4-BITSUCE OF
`IMAGE COMPOSION
`NEWORK
`
`MEDIATEK, Ex. 1025, Page 8
`IPR2018-00101
`
`

`

`U.S. Patent
`
`Jan. 2, 1996
`
`Sheet 8 of 17
`
`5,481,669
`
`F.G. 6
`
`RENDERER
`
`RENDERER
`
`ENDERER
`R
`BOARD
`
`O
`
`1
`
`6
`ATE y
`
`UES
`
`
`
`
`
`
`
`
`
`
`
`
`
`MEDIATEK, Ex. 1025, Page 9
`IPR2018-00101
`
`

`

`U.S. Patent
`
`Jan. 2, 1996
`
`Sheet 9 of 17
`
`5,481,669
`
`_LVWBO-l
`
`-HEANOO
`
`NOIS
`
`SON Í LHOS
`CÌNÝ NOË IVW
`-HO-HSNVH ?
`
`
`
`NO|| VHENES)
`
`BALLIWI?ld
`
`X\H LE WOE5)
`
`SHOSSHOOdd
`
`00 ||
`
`09
`
`SHEOJVHS — —)
`
`G |
`
`
`
`NO]] |SOCHWOO
`
`|-— —)
`
`G€
`
`XHONAI BN
`
`
`
`SHEZI HELLSWH
`
`OG ?.
`
`
`
`MEDIATEK, Ex. 1025, Page 10
`IPR2018-00101
`
`

`

`U.S. Patent
`
`Jan. 2, 1996
`
`Sheet 10 of 17
`
`5,481,669
`
`XferGO
`
`XferReddy
`
`
`
`Reddyn
`
`
`
`GOOut
`
`ReddyOut
`
`READY/GO
`CONTROLLER GOn
`
`F. G. 8.
`
`XferGO
`
`
`
`XferReddy
`
`ME
`
`FRAME
`BUFFER
`
`RENDERER
`
`RENDERER
`
`MASTER
`RENDERER
`
`MEDIATEK, Ex. 1025, Page 11
`IPR2018-00101
`
`

`

`U.S. Patent
`
`Jan. 2, 1996
`
`Sheet 11 of 17
`
`5,481,669
`
`5 O2
`
`
`
`
`
`
`
`
`
`VRAM
`MEMORY
`SERAL
`PORT
`DMA ADRESSES
`ENGINE
`
`
`
`
`
`
`
`|GC
`COMMANDS
`
`FC 1 O.
`
`MEDIATEK, Ex. 1025, Page 12
`IPR2018-00101
`
`

`

`U.S. Patent
`
`Jan. 2, 1996
`
`Sheet 12 of 17
`
`5,481,669
`
`
`
`T]\/NO|| ?O•
`
`JOWEJ W | .....!.!.!.!.!.!.!...I..E..,-
`
`• • • • • • • • • • • • • • • • • • • • & • • • • • • • •
`
`. . . . . . . . . . . . . . . . . . . . . . . . . . - - - - - - - - - - - • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • •
`
`L-G?I
`
`THOEHTOEHT|---THITHTHITHLH H H H
`
`FHETHITHT|---THITHTHITHTHOHLHL] | ............................ . . . . . . . . , • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • •|
`
`
`
`
`
`............................ . . . . . . . . . . . . . . … • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • ** ***|
`
`THTHOEHT|---THITHTHOEHLH H H Jo B?nGOW < OG
`
`(NO|OBÈH TEXld
`gŽIX9Ó I)
`SOWIE #79
`
`MEDIATEK, Ex. 1025, Page 13
`IPR2018-00101
`
`

`

`U.S. Patent
`
`Jan. 2, 1996
`
`Sheet 13 of 17
`
`5,481,669
`
`OL
`
`SOWE
`
`CWNE
`
`HEONETY?DES
`
`
`
`OS9
`
`
`
`
`
`SVUL
`
`HE ONE(YOES
`|
`
`HETTOHLNO O LH
`
`
`
`
`
`MEDIATEK, Ex. 1025, Page 14
`IPR2018-00101
`
`

`

`U.S. Patent
`
`Jan. 2, 1996
`
`Sheet 14 of 17
`
`5,481,669
`
`HO-H
`
`
`
`SESSE HOJCIV
`
`TIVNO|]_c}O
`
`„KHOWEW
`
`BOH V LVC]
`
`TVNOI_LCHO
`
`ÅHOWNEW
`
`
`
`
`
`S(nº O|SVL ETIFICJOW-HELNÍ
`
`
`
`O
`
`(EOTIS
`LIE)
`
`GOTIS
`LIE)
`
`(9 HO ENO) ? ETIOCJOWN
`
`
`
`A = ??8 OWE
`
`MEDIATEK, Ex. 1025, Page 15
`IPR2018-00101
`
`

`

`U.S. Patent
`
`Jan. 2, 1996
`
`Sheet 15 of 17
`
`5,481,669
`
`
`
`LOETIES
`
`
`
`ST18. d5)O|SVL-HELNI
`
`ST18
`
`LEOd «+ No.
`
`º(SÅVM HLOG)
`
`MEDIATEK, Ex. 1025, Page 16
`IPR2018-00101
`
`

`

`U.S. Patent
`
`Jan. 2, 1996
`
`Sheet 16 of 17
`
`5,481,669
`
`ShOld
`
`SLIS2}AINO
`
`SSSYCCV
`
`<O:LL><O:L>YA
`
`AewcASxc
`
`WVvdGWvud
`
`<\
`
`YW
`
`LHDISAOSNO SSSYu00VYOdGSCASN
`
`SSINGOW
`
`VLVG
`
`<O:L><O:L>
`
`SndOISWL
`
`c&
`
`oO}
`uJ
`
`1o
`Ww
`
`OWA
`
`WO u
`
`w
`
`OWS
`
`OWS
`
`WO W
`
`wW
`
`OWS
`
`MEDIATEK,Ex. 1025, Page 17
`IPR2018-00101
`
`MEDIATEK, Ex. 1025, Page 17
`IPR2018-00101
`
`
`

`

`U.S. Patent
`
`Jan. 2, 1996
`
`Sheet 17 of 17
`
`5,481,669
`
`LHSIA4OANO SSSYCCVYOd
`
`SSINGOW
`
`VLVG
`
`91Old
`
`GACGASNSLIGctAINO
`
`<O:LL><O°L>/SSauday-
`<O0:L><0:L>
`ChSeAb||CLSHL
`WGAWVHA
`tt|fdLf
`f_\tt
`SNdOISVL
`
`ax
`
`VA
`
`ce
`
`OWS
`
`OWS
`
`OWS
`
`OWS
`
`OWS
`
`OWS
`
`OWS
`
`OWA
`
`MEDIATEK,Ex. 1025, Page 18
`IPR2018-00101
`
`MEDIATEK, Ex. 1025, Page 18
`IPR2018-00101
`
`
`
`

`

`5,481,669
`
`1
`ARCHITECTURE AND APPARATUS FOR
`IMAGE GENERATION UTILIZING
`ENHANCED MEMORY DEVICES
`
`This application is a continuation of issued prior appli
`cation Ser. No. 07/975,821, filed Nov. 13, 1992, and will
`issue as U.S. Pat. No. 5,388,206 on Feb. 7, 1995, the
`disclosure of which is incorporated by reference herein in its
`entirety.
`
`FIELD OF THE INVENTION
`The present invention relates to image generation systems
`in general and in particular to image generation systems
`using image composition techniques. The present invention
`further relates to memory structures and hardware designs
`for implementing image generation systems using image
`composition techniques.
`
`2
`It is yet another objective of the present invention to
`provide an image generation system which is scalable over
`a wide variety of performance levels and to arbitrarily high
`levels by combining an arbitrary number of renderers. It is
`also an objective of the present invention to provide for a
`simplified programming model without any need for primi
`tive redistribution. An additional objective of the present
`invention is to provide a high bandwidth image composition
`network suitable for use with antialiasing algorithms.
`
`SUMMARY OF THE INVENTION
`A first aspect of the present invention comprises an image
`generation system, comprising a primitive processing means
`for generating primitive screen data and a plurality of
`rasterizers associated with the primitive processing means
`for computing pixel values from the primitive screen data.
`The rasterizer comprises an enhanced memory device cor
`responding to a selected set of screen coordinates. The
`enhanced memory device has for each of the selected screen
`coordinates processing means for computing pixel values to
`provide a computed pixel value, storage means associated
`with each of the processing means for storing data, and a
`compositor buffer associated with each of the processing
`means for storing the computed pixel value. The enhanced
`memory device also has input means for receiving computed
`pixel values from a corresponding enhanced memory device
`associated with a different one of the plurality of geometry
`processors, compositor means for compositing the stored
`computed pixel value and the pixel value received by the
`input means to determine a composited pixel value, and
`output means operably associated with the compositor
`means for outputting the composited pixel value.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`BACKGROUND OF THE INVENTION
`Graphics applications such as flight and vehicle simula
`tion, computer-aided design, scientific visualization, and
`virtual reality demand high-quality rendering, high polygon
`rates, and high frame rates. Existing commercial systems
`render at peak rates up to 2 million polygons per second
`(e.g., Silicon Graphics' SkyWriter and Hewlett-Packard's
`VRX). If antialiasing or realistic shading or texturing is
`required, however, their performance falls by an order of
`magnitude.
`To support demanding applications, future graphics sys
`tems must generate high-resolution images of datasets con
`taining hundreds of thousands or millions of primitives, with
`realistic rendering techniques such as Phong shading, anti
`aliasing, and texturing, at high frame rates (230 Hz) and
`with low latency.
`Attempts to achieve high performance levels encounter
`two bottlenecks: inadequate floating-point performance for
`geometry processing and insufficient memory bandwidth to
`the frame buffer. For example, to render a scene with
`100,000 polygons updated at 30 Hz, geometry processing
`requires approximately 350 million floating-point operations
`per second, and rasterization requires approximately 750
`million integer operations and 450 million frame-buffer
`accesses. Parallel solutions are mandatory.
`Some current systems use pixel-parallelism for rasteriza
`tion; frame-buffer memory is divided into several inter
`leaved partitions, each with its own rasterization processor.
`This multiplies the effective frame-buffer bandwidth by the
`number of partitions, but does not reduce the number of
`primitives each processor must handle, since most primi
`tives contribute to most partitions. Because of this limita
`tion, and the bandwidth limitations of commercial VRAMs,
`this approach does not scale much beyond today's rates of
`a few million polygons per second.
`Accordingly, it is an objective of the present invention to
`provide a high performance image generation system.
`It is a further objective of the image generation system
`according to the present invention to support a wide variety
`of rendering algorithms and primitive types, from Gouraud
`shaded polygons, to Phong-shaded volume data, to directly
`rendered Constructive-Solid-Geometry objects.
`It is a further objective of the present invention to provide
`an architecture which is suitable for a variety of image
`generation functions including rasterizing/rendering, shad
`ing, texturizing and image buffering.
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`FIG. 1 is a block diagram illustrating one embodiment of
`the image generation system of the present invention.
`FIG. 2 is a block diagram illustrating a second embodi
`ment of the image generation system of the present inven
`tion including shader boards.
`FIG. 3 is a block diagram illustrating an embodiment of
`a renderer according to the present invention.
`FIG. 4a is a block diagram illustrating an embodiment of
`a renderer according to the present invention.
`FIG. 4b is a block diagram illustrating an embodiment of
`a shader according to the present invention.
`FIG. 4c is a block diagram illustrating an embodiment of
`a frame buffer according to the present invention.
`FIG. 5 is a block diagram illustrating an embodiment of
`an enhanced memory device according to the present inven
`tion.
`FIG. 6 is a pictorial diagram illustrating an embodiment
`of the image generation system of the present invention
`including shader boards in which the host processor is
`connected to each renderer and shader over a separate fast
`serial link.
`FIG. 7 is a block diagram illustrating the steps for
`rendering polygons and the system components on which
`they are executed for an embodiment of the image genera
`tion system of the present invention.
`FIG. 8 is a block diagram illustrating an embodiment of
`the control logic for an image composition network accord
`ing to the present invention.
`
`MEDIATEK, Ex. 1025, Page 19
`IPR2018-00101
`
`

`

`5,481,669
`
`10
`
`15
`
`20
`
`25
`
`30
`
`3
`FIG. 9 is a block diagram illustrating an embodiment of
`the control flow of an image composition network according
`to the present invention showing ready and go chains with
`master and slave controllers at various stages during a
`transfer cycle.
`FIG. 10 is a block diagram illustrating an embodiment of
`a command queue providing communication buffering
`between a geometry processor and a rasterizer according to
`the present invention.
`FIG. 11 is a block diagram illustrating an embodiment of
`a rasterizer according to the present invention.
`FIG. 12 is a block diagram illustrating an embodiment of
`an image generation controller according to the present
`invention.
`FIG. 13 is a block diagram illustrating an embodiment of
`the connection between the texture ASICs and the enhanced
`memory devices according to the present invention.
`FIG. 14 is a block diagram illustrating an embodiment of
`a texture ASIC chip according to the present invention.
`FIG. 15 is a block diagram illustrating an embodiment of
`a shader board module according to the present invention.
`FIG. 16 is a block diagram illustrating an embodiment of
`one module of a video board rasterizer according to the
`present invention.
`DETAILED DESCRIPTION OF THE
`PREFERRED EMBODIMENTS
`The image generation system of the present invention
`exploits the advantages of image-composition to provide for
`real-time 3D graphics algorithms and applications. The
`image generation system of the present invention may be
`described with reference to FIGS. 1 through 16 but is not
`limited to the embodiments shown in those figures. The
`present invention will first be described generally and then
`a particular embodiment will be described for illustrative
`purposes.
`As seen in FIG. 1, at its highest level the image generation
`system of the present invention is comprised of a plurality of
`renderers 10 acting in parallel to produce a final image. The
`renderers 10 receive primitives of a screen image from a host
`processor 20 over a host interface 25. Pixel values are then
`determined by the renderers 10 and the visibility of a
`particular pixel calculated by a given renderer determined
`45
`through a compositing process and stored in the frame buffer
`30 for display on the video display 40. The linear array of
`renderers results in the final image being produced at the
`output of the last renderer,
`The primitives received by the renderers are geometric
`shapes which may be combined to create an image. Primi
`tives may contain shape, depth, color, shading and other
`information about the image known to those of skill in this
`art. The primitives may be distributed throughout the plu
`rality of renderers 10 in any of a number of schemes known
`to those of skill in this art. The final image is then created by
`putting together the pixel values calculated by the plurality
`of renderers. Image composition techniques are especially
`suited for use with the present image generation system as
`a means for recombining the pixel values from the plurality
`of renderers.
`When image composition is used, primitives for the entire
`screen or a segment of the screen are distributed over the
`plurality of processors. Rather than having a single renderer
`calculate pixel values for all primitives for a portion of the
`screen, each of the renderers only calculates pixel values for
`a portion of the primitives for a region of the screen. A
`
`50
`
`35
`
`40
`
`55
`
`60
`
`65
`
`4
`region of the screen may encompass the entire screen or a
`portion thereof. Therefore, all of the renderers calculate
`pixel values for the same portion of the screen and then these
`pixel values from each of the renderers are combined to
`produce a final image for that portion of the screen. The
`regions of the screen are processed sequentially until the
`final image is produced. This combination of pixel values for
`a region of the screen from the plurality of renderers is
`referred to as compositing.
`As shown in FIG. 2, the image generation system may
`further include shaders 15 which provide for texturing and
`shading of the image after composition by the renderers 10
`and before storage in the frame buffer 30. The shaders may
`also be used to perform antialiasing operations. The shaders
`receive shading instructions from the host processor 20 and
`the composited pixel values from the last renderer in the
`linear array of renderers 10. Deferred shading algorithms,
`such as Phong shading and procedural and image-based
`textures, are implemented on separate hardware shaders 15
`that reside just ahead of the frame buffer 30. Regions of
`pixels, containing attributes such as intrinsic color, surface
`normals, and texture coordinates are rasterized and compos
`ited along the image composition network 35 of the render
`ers 10 and loaded into the shaders 15 from the image
`composition network 35. The image composition network
`35 may optionally be structured as a ring network to allow
`any board to communicate with any other board. It will be
`apparent to one of skill in the art that other means for
`structuring the image composition network would also pro
`vide this capability. Shaders 15 operate on entire regions in
`parallel, to convert raw pixel attributes into final RGB
`values, blend multiple samples together for antialiasing, and
`forward final color values to the frame buffer 30 over the
`image composition network 35.
`FIG.3 is a block diagram of a renderer 10 according to the
`present invention. Each of the plurality of renderers may be
`identical. The renderers 10 are comprised of a geometry
`processor 100 for receiving primitives and translating the
`primitives to screen coordinates to provide primitive screen
`data. This primitive screen data is then transferred to a
`rasterizer 120 associated with the geometry processor for
`computing pixel values from the primitive screen data.
`Alternatively, this primitive screen data could be provided
`by the host processor 20 or by other primitive processing
`means to the renderer 10. The renderer would then not
`include a geometry processor 100.
`The rasterizer 120 is made up of an image generation
`controller 110 and at least one enhanced memory device
`150, but preferably a plurality of enhanced memory devices
`forming an enhanced memory device array 125. The image
`generation controller 110 receives the primitive screen data
`and provides it to the enhanced memory devices 150. The
`image generation controller 110 may also adjust the values
`received from the geometry processor 100 before passing
`the values on to the enhanced memory devices 150 for use
`in antialiasing. Each of these enhanced memory devices 150
`corresponds to a selected set of screen coordinates such that
`any pixel coordinate in the region of the screen being
`generated has a corresponding enhanced memory device.
`Each of the enhanced memory devices 150 has a pixel
`processor 151 or other processing means, for each of the set
`of coordinates associated with that particular enhanced
`memory device for computing pixel values. The enhanced
`memory devices have storage means 152 associated with
`each of the pixel processors 151 for storing data for use by
`these pixel processors. Each of the enhanced memory
`devices 150 further has a compositor buffer 153 or other
`
`MEDIATEK, Ex. 1025, Page 20
`IPR2018-00101
`
`

`

`5
`storage means associated with each of said pixel processors
`for storing the computed pixel values for each screen
`coordinate. The enhanced memory devices also have a
`compositor input 155 or other input means, for receiving
`computed pixel values from the corresponding enhanced
`memory devices associated with the other renderers and a
`compositor output 156 or other output means, for outputting
`the computed pixel values. The enhanced memory devices
`also have compositor circuitry 154 or other compositing
`means for compositing the computed pixel value stored in
`the compositor buffer 153 and the pixel value received at the
`compositor input 155, and providing the composited pixel
`values to the compositor output 156.
`By adding the capability to load the values received at the
`compositor input 155 into the enhanced memory device 150,
`the renderer 10 may be utilized for shading functions. In this
`case, the shader 15 would receive shading information from
`the host processor for use by the geometry processor 100 and
`rasterizer 120. As described below, further modifications to
`the basic renderer design may be made to allow for textur
`ling.
`Of particular advantage to the image generation system of
`the present invention is the modularity of the structure of the
`renderers, shaders and frame buffer. The single instruction
`multiple data (SIMD) rasterizer 120 used in the renderer 10
`is an ideal processor for deferred shading, because shading
`calculations can be performed for all pixels simultaneously.
`Therefore, the shaders 15 can simply be designated render
`ers, with a slight enhancement of the compositor circuitry
`30
`154 on the enhanced memory devices to allow bidirectional
`data transfers between the image composition network 35
`and the enhanced memory devices 150. Shaders can be
`augmented with additional hardware to allow them to com
`pute image-based textures in addition to procedural textures.
`35
`As seen in FIG. 4, the shader 15, shown in FIG. 4b, and
`frame buffer 30, shown in FIG.4c, are substantially the same
`as the renderer 10, shown in FIG. 4a. As seen in FIG. 4b, the
`addition of a local port 160 to the enhanced memory devices
`150 allows for texturing as described above. A local buffer
`161 stores data for access by the pixel processors 151 and
`the local port 160. The local port 160 provides access to
`local external memory 165 or other external memory means
`which may be used for storing texturing information. Thus,
`by the addition of the local port 160, the local buffer 161 and
`the local external memory 165, the renderer 10 may be
`utilized as a shader 15 and perform texturing algorithms.
`Similarly, through the addition of the local buffer 161, the
`local port 160 and the local external memory 165, all that
`need be added to the shader 15 to operate as a frame buffer
`30 is the inclusion of video circuitry 170. As shown in FIG.
`4c, as a frame buffer 30, the local external memory 165 acts
`as the frame buffer memory and the video circuitry 170 reads
`the pixel image data from the memory. In the frame buffer
`30, the local memory 165 is, in one embodiment, a separate
`double-buffered VRAM frame buffer.
`In one embodiment of the present invention, the primi
`tives received from the host processor 20 are transformed by
`the geometry processor into primitive screen data repre
`sented as coefficients of mathematical expressions f(x,y)
`representing the primitive. In particular, the primitives may
`be linearized into a series of equations of the form f(x,y)=
`Ax+By--C, where x and y are screen pixel coordinates and
`A, B and C are coefficients which define a plane contained
`in a particular primitive. The use of linear expressions of the
`above form to render primitives is described in U.S. Pat. No.
`4,590,465 at column 4, line 47 through column 6, line 8, the
`
`50
`
`40
`
`45
`
`55
`
`60
`
`65
`
`5,481,669
`
`10
`
`5
`
`20
`
`25
`
`6
`disclosure of which is incorporated herein by reference as if
`set forth fully. While linearization has been described, other
`mathematical expressions such as quadratic, or other poly
`nomial representations in screen coordinates may be uti
`lized.
`In one embodiment of the present invention, the geometry
`processor 100 provides the A, B and C coefficients to the
`enhanced memory devices 150 through the image generation
`controller 110. The image generation controller 110 controls
`the enhanced memory device array 125. It converts floating
`point A, B, and C coefficients into byte-serial, fixed-point
`form; it sequences enhanced memory device operations by
`broadcasting command information such as primitive data,
`control, and address information to the enhanced memory
`device array 125; and it controls the compositor ports 157 on
`the enhanced memory devices 150.
`The image generation controller 110 may also be used for
`anti-aliasing through the use of super-sampling. Super
`sampling involves calculating the pixel values at sub-pixel
`locations surrounding a pixel and then calculating the actual
`pixel value for a given screen coordinate by combination of
`these sub-pixel values. The image generation controller 110
`can modify the coefficients to reflect the sub-pixel coordi
`nates and then retransmit these coefficients to the enhanced
`memory devices 150 without obtaining additional coeffi
`cients from the geometry processor 100. The image genera
`tion controller 110 contains a subpixel offset register that
`allows for the multiple samples of the supersampling filter
`kernel to be computed from the same set of rasterization
`commands. This allows for increased system performance
`when supersampling, because additional samples are raster
`ized without increasing the load on the geometry processor.
`FIG. 5 is a block diagram of a particular embodiment of
`the enhanced memory device 150 which is utilized in the
`image generation system of the present invention. It will be
`appreciated by those of skill in the art that the enhanced
`memory device of the present invention may be fabricated
`on a single integrated circuit or chip using fabrication
`techniques known to one of skill in the art.
`As seen in FIG. 5, in this embodiment the pixel processors
`151 each share a linear expression evaluator 200 which
`computes values of the primitive screen data as the bilinear
`expression Ax+By+C defining a plane that represents a
`portion of a particular primitive at every pixel processor 151
`in parallel. Each pixel processor 151 also has a small local
`ALU210 that performs arithmetic and logical operations on
`the segment of local memory 220 which acts as the storage
`means 152 associated with that pixel processor and on the
`local value of the bilinear expression. Operation of the pixel
`processors 151 is SIMD (single-instruction-multiple-data),
`and all processors operate on data items at the same address.
`Each pixel processor 151 includes an enable register which
`qualifies writes to memory, so that a subset of the processors
`can be disabled for certain operations (e.g. painting a
`scan-converted polygon). Also included are the local buffer
`161 for providing information for use by pixel processors
`151 and receiving information from and outputting infor
`mation to the local port 160. The compositor buffer 153
`stores pixel values to be composited. The compositor buffer
`153 provides its data to the compositor circuitry 154 for
`compositing and output. The compositor circuitry 154
`receives data from the compositor input 155 and outputs
`data on the compositor output 156.
`As seen in FIG. 6, one embodiment of the image genera
`tion system of the present invention may be composed of
`one or more card cages, each containing multiple circuit
`boards and connected to a host processor 20. The boards in
`each card cage are attached to a common backplane, which
`contains an image composition network 35 that extends to
`
`MEDIATEK, Ex. 1025, Page 21
`IPR2018-00101
`
`

`

`5,481,669
`
`10
`
`15
`
`20
`
`7
`each system board. Each board has provisions for connect
`ing to the host processor 20. The backplane further distrib
`utes power and clocks to the system boards. Multiple card
`cages can be connected by placing them side-by-side and
`connecting their backplanes together with special bridge
`boards (not shown) and connectors.
`The image generation system of the present invention
`contains three main board types:
`Renderers 10, which are one-board graphics computers
`capable of rendering well over one million Z-buffered tri
`angles per second
`Shaders 15, which are one-board graphics computers
`capable of computing shading models for pixels in parallel
`and texturing; and
`Frame buffer 30, which buffers and displays composited
`pixels.
`In a typical application, the host processor 20 is the
`overall system master. It loads code and data onto the
`various system boards, and sends the display primitives to
`the renderers 10.
`The system rasterizes one region of the screen at a time.
`This means that transformed primitives must be sorted into
`bins for each screen region before rasterization can take
`place. These bins are stored in VRAM memory 502 of the
`geometry processor.
`After the primitives have been sorted, they are converted
`into image generation controller instructions, and rasterized
`one bin at a time. The image generation controller 110 and
`enhanced memory device array 125 rasterize the