UNITED STATES PATENT AND TRADEMARK OFFICE
`
`BEFORE THE PATENT TRIAL AND APPEAL BOARD
`
`REALTEK SEMICONDUCTOR CORP.,
`Petitioner,
`
`V.
`ATI TECHNOLOGIES ULC
`Patent Owner.
`
`IPR2023-00922
`Case No.
`U.S. Patent No. 8,760,454
`
`DECLARATION OF DR. WILLIAM MANGIONE-SMITH
`REGARDING U.S. PATENT NO. 8,760,454
`
`I declare under penalty of perjury under the laws of the United States of America
`
`that the folowing is true and correct.
`
`September6, 2023
`
`RespectfullyŞubmited
`
`Dr. WilliamMangione-Smith
`
`ATI Ex. 2001
`IPR2023-00922
`Page 1 of 268
`
`

`

`
`
`TABLE OF CONTENTS
`
`I. 
`II. 
`
`Page
`INTRODUCTION ........................................................................................... 1 
`SUMMARY OF CONCLUSIONS ................................................................. 2 
`A. 
`Summary of Conclusions Regarding Reduction to Practice,
`Conception, and Diligence .................................................................... 2 
`Summary of Conclusions Regarding Validity ...................................... 3 
`B. 
`III.  EXPERT QUALIFICATIONS ........................................................................ 3 
`IV.  HIGH LEVEL GRAPHICS PROCESSING OVERVIEW ............................. 7 
`A.  Graphics Processors Use Vertex and Pixel Data to Create
`Display Images ...................................................................................... 7 
`Graphics Processors Use Instruction “Threads” To Transform
`Vertex and Pixel Data.......................................................................... 10 
`Conventional Graphics Processors Executed Vertex And Pixel
`Threads Using Separate Vertex Shaders And Pixel Shaders .............. 11 
`V.  OVERVIEW OF THE ’454 PATENT .......................................................... 14 
`The ’454 Patent’s Unified Shader ....................................................... 14 
` 
`B. 
`The Unified Shader Can Simultaneously Execute Vertex And
`Pixel Threads And Switch Quickly Between Threads At
`Various Degrees Of Completion ......................................................... 15 
`The Invention of The ’454 Patent Determines What Data To
`Process By Evaluating Dynamic Storage Capacity ............................ 17 
`The Invention Of The ’454 Patent Triggers Execution By
`Transmitting Data Rather Than Instructions ....................................... 18 
`The ’454 Patent’s Challenged Claims ................................................. 19 
`All Challenged Claims Require A Unified Shader ................... 19 
` 
`Challenged Claims 1 And 3-11 Require Performing Operations
` 
`Based On Dynamically-Monitored Data Storage Capacity ...... 19 
`Challenged Claim 2 Is Directed To A Unified Shader Processor
`That Executes Pixel Or Vertex Thread Instructions In Response
`To Receiving Selected Data From A Register .......................... 20 
`VI.  RELEVANT LEGAL STANDARDS ........................................................... 21 
`Conception and Reduction to Practice ................................................ 21 
` 
`Claim Construction ............................................................................. 23 
` 
`Obviousness ......................................................................................... 23 
` 
`
`B. 
`
`C. 
`
`C. 
`
`D. 
`
`E. 
`
` 
`
`i
`
`ATI Ex. 2001
`IPR2023-00922
`Page 2 of 268
`
`

`

`
`
`VII.  GENERAL DESIGN AND DEVELOPMENT PROCESS AT AMD ......... 25 
`VIII.  CONCEPTION .............................................................................................. 29 
`’454 Patent ........................................................................................... 31 
` 
`1. 
`Claim 1 ...................................................................................... 31 
`Claim 2 ...................................................................................... 42 
` 
`Claim 3 ...................................................................................... 59 
` 
`Claim 4 ...................................................................................... 66 
` 
`Claim 5 ...................................................................................... 74 
` 
`Claim 6 ...................................................................................... 84 
` 
`Claim 7 ...................................................................................... 86 
` 
`Claim 8 ...................................................................................... 91 
` 
`Claim 9 ...................................................................................... 94 
` 
`  Claim 10 .................................................................................... 98 
`  Claim 11 .................................................................................. 101 
`IX.  DILIGENCE IN REDUCING TO PRACTICE .......................................... 108 
`X. 
`CONSTRUCTIVE REDUCTION TO PRACTICE .................................... 152 
`’454 Patent ......................................................................................... 153 
` 
`Claim 1 .................................................................................... 153 
` 
`Claim 2 .................................................................................... 159 
` 
`Claim 3 .................................................................................... 170 
` 
`Claim 4 .................................................................................... 180 
` 
`Claim 5 .................................................................................... 190 
` 
`Claim 6 .................................................................................... 195 
` 
`Claim 7 .................................................................................... 197 
` 
`Claim 8 .................................................................................... 198 
` 
`Claim 9 .................................................................................... 199 
` 
`  Claim 10 .................................................................................. 200 
`  Claim 11 .................................................................................. 201 
`XI.  NO ABANDONMENT, SUPPRESSION, OR CONCEALMENT OF
`R400 ............................................................................................................. 207 
`XII.  ASSERTED PRIOR ART ........................................................................... 208 
`A. 
`Lindholm ’685 ................................................................................... 209 
`B. 
`Amanatides ........................................................................................ 211 
`C. 
`Selzer ................................................................................................. 212 
`XIII.  EVALUATION OF PETITIONER’S PROPOSED GROUNDS ............... 214 
`
`ii
`
`ATI Ex. 2001
`IPR2023-00922
`Page 3 of 268
`
`

`

`
`
`2. 
`
`3. 
`
`B. 
`
`A.  Ground 1: Lindholm ’685 and Lindholm ’913 Do Not Render
`the ’454 Patent Obvious .................................................................... 214 
`1. 
`No Unified Shader, Because No Ability to Switch Between
`Unfinished Vertex And Pixel Threads (All Claims) ............... 214 
`No Determination of Data to Process Based On Evaluation of
`Data Storage Space (Claims 1, 3-11) ...................................... 215 
`No Execution of Instructions “In Response to” Receiving
`Selected Data (Claim 2) .......................................................... 217 
`Ground 2: Amanatides and Kohn Do Not Render the ’454
`Patent Obvious .................................................................................. 218 
`1. 
`No Unified Shader, Because No Ability to Switch Between
`Unfinished Vertex And Pixel Threads (All Claims) ............... 219 
`No Determination of Data to Process Based on Evaluation of
`Data Storage Space (Claims 1, 3-11) ...................................... 220 
`No “Selected Data” (Claims 2, 5) ........................................... 222 
` 
`Ground 3: Selzer and Fiske Do Not Render the ’454 Patent
`Obvious ............................................................................................. 223 
`No Unified Shader, Because No Ability to Switch Between
` 
`Unfinished Vertex And Pixel Threads (All Claims) ............... 224 
`No Determination of Data to Process Based On Evaluation of
`Data Storage Space (Claims 1, 3-11) ...................................... 226 
`No Execution of Instructions “In Response to” Receiving
`Selected Data (Claim 2) .......................................................... 227 
`XIV.  CONCLUSION ............................................................................................ 228 
`
`C. 
`
` 
`
` 
`
` 
`
`iii
`
`ATI Ex. 2001
`IPR2023-00922
`Page 4 of 268
`
`

`

`
`
`
`
`TABLE OF ABBREVIATIONS
`
`Abbreviation
`
`Full Description
`
`AMD
`ATI
`GPU
`
`Advanced Micro Devices, Inc.
`ATI Technologies, ULC
`Graphics processing unit
`
`
`i
`
`ATI Ex. 2001
`IPR2023-00922
`Page 5 of 268
`
`

`

`
`
`TABLE OF EXHIBITS
`
`1008
`
`1009
`
`Exhibit # Reference Name
`1001
`U.S. Patent 8,760,454 to Morein et al.
`1002
`Prosecution History of U.S. Patent 8,760,454
`1003
`Declaration of Hanspeter Pfister, Ph.D.
`1004
`Curriculum Vitae of Hanspeter Pfister, Ph.D.
`1005
`U.S. Patent 7,038,685 to Lindholm et al. (“Lindholm ’685”)
`1006
`U.S. Patent No. 7,015,913 to Lindholm et al. (“Lindholm ’913”)
`1007
`John Amanatides and Edward Szurkowski, A Simple, Flexible,
`Parallel Graphics Architecture, In Proceedings of Graphics
`Interface at 155-160 (Canadian Information Processing Society
`1993) published in Proc. Graphics Interface ’93 in May 1993
`(“Amanatides”)
`Les Kohn and Neal Margulis, Introducing the Intel i860 64-bit
`Microprocessor, IEEE, Volume 9, Issue 4, pages 15-30, August
`1989 (“Kohn”)
`Harald Selzer, Dynamic Load Balancing within a High
`Performance Graphics System, In Proceedings of Rendering,
`Visualization and Rasterization Hardware (Eurographics' 91
`Workshop) at 37-53 (Springer-Verlag 1993) published in 1993
`(“Selzer”) [Library of Congress]
`Stuart Fiske and William J. Dally, Thread prioritization: A Thread
`Scheduling Mechanism for Multiple-Context Parallel Processors,
`In Proceedings of First Symposium on High-Performance
`Computer Architecture, 1995 at 210-221 (IEEE 1995) published in
`1995 (“Fiske”)
`Harald Selzer, Dynamic Load Balancing within a High
`Performance Graphics System, In Proceedings of Rendering,
`Visualization and Rasterization Hardware (Eurographics' 91
`Workshop) at 37-53 (Springer-Verlag 1993) published in 1993
`(“Selzer”) [University of California, Berkeley, Library]
`EX 2001 Declaration of William Mangione-Smith and CV
`EX 2002
`IPR2015-00325, Declaration of Dr Wolfe, Sept. 9, 2015
`EX 2003
`IPR2015-00325, Declaration of Calvin Watson, Sept. 9, 2015
`EX 2004
`IPR2015-00325, Declaration of Lefebvre, Sept. 9, 2015
`
`1010
`
`1016
`
`ii
`
`ATI Ex. 2001
`IPR2023-00922
`Page 6 of 268
`
`

`

`
`
`EX 2007
`
`EX 2006
`
`Exhibit # Reference Name
`EX 2005
`IPR2015-00325,-00326, and-00330, Deposition Transcript of
`Calvin Watson, November 4, 2015
`IPR2015-00325, -00326, and -00330, Deposition Transcript of
`Dr. Wolfe, Nov. 10, 2015
`IPR2015-00325,-00326, and -00330, Deposition Transcript of
`Laurent Lefebvre, Nov.13, 2015
`EX 2008 Gruber et al, R400 Shader Processor 2001, Oct. 9, 2015
`EX 2009 R400 Document Library Folder History, November 1, 2000
`through April 8, 2005
`EX 2010 R400 Sequencer Specification (Version 0 4), August 14, 2001
`EX 2011 R400 Sequencer Specification (Version 2 0), September 24, 2001
`EX 2012 R400 Shader Processor (Version 0 1), Jan. 23, 2001
`EX 2013 R400 Top Level Specification (Version 0 2), March 11, 2001
`EX 2014 R400 Program Logs
`EX 2015 Microsoft Site Visit (Feb. 2003)
`EX 2016
`Log of Exhibits and File Location
`EX 2017 R400 Program Review Documentation (Feb. 2003)
`EX 2018 R400 MM Software Status (Feb. 2003)
`EX 2019 R400 I/O Presentation (Feb. 2003)
`EX 2020 R400 I/O Presentation (Feb. 2003)
`EX 2021 R400 August Program Review
`EX 2022
`Executive Review - R400 (Oct. 2002)
`EX 2023 R400 Area Estimate
`EX 2024 R400 Executive Review (Sept. 2002)
`EX 2025 GFIXIP 9x SX Micro-Architecture Specification
`EX 2026 WD/IA VGT Micro-Architecture Specification
`EX 2027 GX9 SPI Specification
`EX 2028 R400 Top Level Specification (Version 0.2)
`EX 2029 R400 Folder History Log
`EX 2030 R400 Folder History Log
`EX 2031 R400 Folder History Log
`EX 2032 R400 Review PowerPoint
`
`iii
`
`ATI Ex. 2001
`IPR2023-00922
`Page 7 of 268
`
`

`

`
`
`Exhibit # Reference Name
`EX 2033 R400 Review PowerPoint
`EX 2034 R400 Development and Documentation
`EX 2035 Xenos GFX Change History July 2003 - Dec. 2003 Log
`EX 2036 R400 GFX Change History March 2003 - Dec. 2003
`EX 2037 R400 Primitive Assembly
`EX 2038 R400 Primitive Assembly
`EX 2039 Development Documentation
`EX 2040 Development Documentation
`EX 2041 Development Documentation
`EX 2042 R400 Performance Verification
`EX 2043 Development Documentation
`EX 2044 R400 PAD Program Review (Dec. 2002)
`EX 2045 R400 EMU Test Regress History Log
`EX 2046 R400 EMU Test Regress Statistics Log
`EX 2047 R400 Program Review (Dec. 2002)
`EX 2048 R400 Program Review (Dec. 2002)
`EX 2049 Device Development Progress
`EX 2050 Device Development Progress
`EX 2051 Device Development Progress
`EX 2052 Device Development Progress
`EX 2053 R400 Review Status PowerPoint
`EX 2054
`TV Specification
`EX 2055 R400 Program Review PowerPoint and Test Results
`EX 2056 Development Documentation
`EX 2057 R400 PowerPoint
`EX 2058 R400 Program Review PowerPoint
`EX 2059 R400 Program Review
`EX 2060 Block Development Progress
`EX 2061 R400 Program Review (Dec. 11, 2002)
`EX 2062 R400 Program Review (Dec. 12, 2002)
`EX 2063 Development Documentation
`EX 2064 Development Documentation
`
`iv
`
`ATI Ex. 2001
`IPR2023-00922
`Page 8 of 268
`
`

`

`
`
`Exhibit # Reference Name
`EX 2065 Development Documentation
`EX 2066 R400 Technical Documentation
`EX 2067 R400 Technical Documentation
`EX 2068 R400 GFX Change History March 2003 to December 2003 Log
`EX 2069
`Ikos 2002 Spreadsheet
`EX 2070 R400 - Program Review (Oct. 2002)
`EX 2071
`Sqsp Regress Report
`EX 2072 Xenos Sq Change Log
`EX 2073 R400 Regress Testing
`EX 2074 Virtual Logic 3.1 User's Guide
`EX 2075 Virtual Logic 3.5.5 User's Guide
`EX 2076
`IKOS Virtual Logic 2.1
`EX 2077
`IKON Screenshots
`EX 2078
`IKON Screenshots
`EX 2079 GFX9 User's Guide
`EX 2080 US Patent No. 6,897,871
`EX 2081
`Samsung Exynos 5430 Octa SoC
`EX 2082
`Samsung Exynos 3 Quad 3470 Processor Database
`EX 2083 HW Emulator First Triangle
`EX 2084 HW Simulator First Triangle
`EX 2085 R400 Regress Testing Logs
`EX 2086 R400 Regress Testing Logs
`EX 2087 R400 IKOS Status
`EX 2088 R400 Program Review
`EX 2089 R400 Program Review Documentation
`EX 2090
`PA Check-in History Log
`EX 2091 RB Check-in History Log
`EX 2092
`SC Check-in History Log
`EX 2093
`SPI Check-in History Log
`EX 2094
`SP Check-in History Log
`EX 2095
`SQ Check-in History Log
`EX 2096
`SX Check-in History Log
`
`v
`
`ATI Ex. 2001
`IPR2023-00922
`Page 9 of 268
`
`

`

`
`
`
`
`Exhibit # Reference Name
`EX 2097 VGT Check-in History Log
`EX 2098 R400 File Logs 10/1/2002 – 4/17/2003
`EX 2099 Netlist Area Sheets
`EX 2100
`IKOS Schematic
`EX 2101
`Stephen Morein Depo Transcript (AMD v. LG) May 25, 2017
`EX 2102
`Laurent Lefebvre Depo Transcript (337-TA-1044) June 28, 2017
`EX 2103
`IPR2015-00325, Declaration of Laurent Lefebvre
`EX 2104 Development Documentation
`EX 2105 Development Documentation
`EX 2106 Development Documentation
`EX 2107 Development Documentation
`EX 2108 Development Documentation
`EX 2109 Development Documentation
`EX 2110
`IPR2015-00325, EX 2018 - EX. 2056
`EX 2111
`IPR2015-00325, EX 2057 - EX 2071
`EX 2112 Block Change Logs
`EX 2113 Andrew Gruber Depo Transcript (AMD v. LG) July 27, 2017
`EX 2114
`IPR2015-00326, EX 2001 - EX2002
`EX 2115
`IPR2015-00326, EX 2003
`EX 2116
`IPR2015-00326, EX 2004
`EX 2117
`IPR2015-00326, EX 2005
`EX 2118
`IPR2015-00326, EX 2006
`EX 2119
`IPR2015-00326, EX 2007 - EX 2072
`EX 2120
`IPR2015-00326, EX 2073 - EX 2118
`
`vi
`
`ATI Ex. 2001
`IPR2023-00922
`Page 10 of 268
`
`

`

`PROTECTIVE ORDER
`
`I.
`
`INTRODUCTION
`1.
`I have been retained by ATI Technologies UCL (“ATI” or “Patent
`
`Owner”), as an independent expert in this proceeding before the Patent Trial and
`
`Appeal Board (“PTAB” or “Board”), to examine whether United States Patent No.
`
`8,760,454 (“the ’454 Patent”) was conceived and diligently reduced to practice prior
`
`to the Lindholm ’685 and ’913 patents, whose effective dates are June 30, 2003 and
`
`June 27, 2003, respectively. I understand that Realtek is requesting that the Board
`
`institute an Inter Partes Review (“IPR”) proceeding of the ’454 Patent (Ex. 1001) in
`
`IPR2023-00922.
`
`2.
`
`I am providing this testimony for consideration by the Patent Trial and
`
`Appeal Board in this proceeding. I have personal knowledge of the facts and
`
`opinions set forth in this declaration and all of the opinions and conclusions found
`
`in this declaration are my own.
`
`3.
`
`I am being compensated at my ordinary hourly rate for my efforts with
`
`regards to this matter. My compensation does not depend in any way on the outcome
`
`of this proceeding or the particular opinions I express, or the testimony I give.
`
`4.
`
`This declaration contains my conclusions and a summary of my
`
`analysis including a summary of my conclusions, an overview of my qualifications
`
`as an expert, an overview of the materials I have considered in arriving at my
`
`conclusions, an overview of the terminology and legal principles that I applied in
`
`ATI Ex. 2001
`IPR2023-00922
`Page 11 of 268
`
`

`

`
`
`my analysis, an overview of the technical background of the subject matter, an
`
`overview of the ’454 Patent, and an analysis of when the inventions of the ’454
`
`Patent were conceived and diligently reduced to practice.
`
`5.
`
`This declaration is based on information currently available to me. I
`
`intend to continue my investigation and study, which may include a review of
`
`documents and information that may yet be produced, as well as deposition
`
`testimony from depositions for which transcripts are not yet available or that may
`
`yet be taken in this proceeding. Therefore, I expressly reserve the right to expand or
`
`modify my opinions as my investigation and study continue, and to supplement my
`
`opinions in response to any additional information that becomes available to me, any
`
`matters raised by Petitioner (or its experts) or the Board (or its experts), or in light
`
`of any relevant opinions or orders from the Patent Trial and Appeal Board or other
`
`authoritative body. Moreover, I reserve the right to provide rebuttal testimony
`
`regarding any analyses and opinions raised in opposition to my declaration.
`
`II.
`
`SUMMARY OF CONCLUSIONS
`A.
`Summary of Conclusions Regarding Reduction to Practice,
`Conception, and Diligence
`It is my opinion that the historical ATI design documentation
`
`6.
`
`demonstrates that at least by at least August 24, 2001, the inventors had conceived
`
`of the inventions claimed in claims 1-11 of the ’454 Patent. It is also my opinion that
`
`the inventors diligently reduced the inventions of the ’454 Patent to practice. It is
`
`2
`
`ATI Ex. 2001
`IPR2023-00922
`Page 12 of 268
`
`

`

`
`
`my opinion that the filing of US Patent Application 10/718,318 application, a parent
`
`application to the application which resulted in the ’454 Patent, on November 20,
`
`2003 qualifies as a constructive reduction to practice of claims 1-11 of the ’454
`
`Patent. US Patent Application 10/718,318 resulted in U.S. Patent No. 6,897,871 (the
`
`“’871 Patent”). The specification of the ’871 Patent discloses each and every
`
`element of claims 1-11 of the ’454 Patent.
`
`B.
`7.
`
`Summary of Conclusions Regarding Validity
`It is my opinion that the ’454 Patent is not obvious in view of: (i)
`
`Lindholm ’685 and Lindholm ’913; (ii) Amanatides and Kohn; and (iii) Selzer and
`
`Fiske.
`
`III. EXPERT QUALIFICATIONS
`8. My technical background and experience covers most aspects of
`
`computer system design, including low level circuitry, computer architecture,
`
`computer networking, graphics, application software, client-server application, Web
`
`technology, and system software (e.g., operating systems and compilers). I am a
`
`member of the Institute of Electrical and Electronics Engineers and the Association
`
`for Computing Machinery, which are the two most significant professional
`
`organizations in my profession. I have been employed as a design engineer, research
`
`engineer, professor and technical expert. Over my professional career, I have been
`
`an active inventor with 112 issued U.S. patents, 200 published and pending U.S.
`
`3
`
`ATI Ex. 2001
`IPR2023-00922
`Page 13 of 268
`
`

`

`
`
`patent applications and many unpublished U.S. patent applications.
`
`9.
`
`From 1984 until 1991, I attended the University of Michigan in Ann
`
`Arbor, Michigan, where I was awarded the degrees of Bachelor of Science and
`
`Engineering, Master of Science and Engineering, and Doctorate of Philosophy. My
`
`doctoral research focused on high performance computing systems including
`
`computer architecture, applications and operating system software, and compiler
`
`technology. One of my responsibilities during my graduate studies included
`
`teaching senior undergraduate students who were about to enter the profession.
`
`10. After graduating from the University of Michigan, I was employed by
`
`Motorola in Schaumburg, Illinois. While at Motorola, I was part of a team designing
`
`and manufacturing the first commercial battery-powered product capable of
`
`delivering Internet email over a wireless (i.e., radio frequency) link and one of the
`
`first personal digital assistants. I also served as the lead architect on the second-
`
`generation of this device with control over the entire system design including the
`
`memory subsystem architecture, embedded processor, ASIC, power system, and
`
`analog circuitry. Part of my responsibilities at Motorola involved the specification,
`
`design, and testing of system control Application-Specific Integrated Circuits
`
`(“ASICs”). I conducted the initial research and advanced design that resulted in the
`
`Motorola M*Core embedded microprocessor. M*Core was designed to provide the
`
`high performance of desktop microprocessors with
`
`the
`
`low power of
`
`4
`
`ATI Ex. 2001
`IPR2023-00922
`Page 14 of 268
`
`

`

`
`
`contemporaneous embedded processors. The M*Core received widespread use in
`
`many communications products including various cellular handsets, advanced
`
`pagers, and embedded infrastructure. While at Motorola I was the sole inventor on
`
`one U.S. patent.
`
`11. From 1995 until 2005, I was employed by the University of California
`
`at Los Angeles (“UCLA”) as a professor of Electrical Engineering. I was the director
`
`of the laboratory for Compiler and Architecture Research in Embedded Systems
`
`(“CARES”) and served as the field chair for Embedded Computing Systems. The
`
`CARES research team focused on research, engineering and design challenges in the
`
`context of battery-powered and multi-media mobile computing devices. One of the
`
`key developments of my lab was the Mediabench software tool, which is widely
`
`used to design and evaluate multi-media embedded devices. Key elements of
`
`Mediabench include software that is essential for modern digital wireless
`
`communications. My primary responsibility, in addition to classroom teaching,
`
`involved directing the research and training of graduate students. I was a tenured
`
`member of the faculty, and had responsibilities for teaching as well as scholarly
`
`research. While at UCLA I was a named inventor on three U.S. patent applications,
`
`one of which issued as a patent. My colleagues at UCLA were some of the leading
`
`scientists and engineers in the world with a long list of innovations from computer
`
`network security devices to the nicotine patch. The graduate student researchers in
`
`5
`
`ATI Ex. 2001
`IPR2023-00922
`Page 15 of 268
`
`

`

`
`
`my laboratory came from a diverse set of backgrounds, all with undergraduate
`
`degrees in computer engineering, electrical engineering or computer science, many
`
`with multiple years of experience working as professional engineers in areas such as
`
`software development, computer system design and ASIC circuit design.
`
`12. From 2005 until 2009, I was employed at Intellectual Ventures in
`
`Bellevue, Washington. My responsibilities at Intellectual Ventures included
`
`business development, technology assessment, market forecasting, university
`
`outreach, collaborative inventing, intellectual property licensing support, and
`
`intellectual property asset pricing. My colleagues and co-inventors at Intellectual
`
`Ventures included the former lead intellectual property strategist at Intel, Intel’s
`
`former lead IP council, Microsoft’s former chief software architect, the founder of
`
`Microsoft research, the designer of the Mach operating system, the architect of the
`
`U.S. Defense Department’s Strategic Defense Initiative, the founder of Thinking
`
`Machines (a seminal parallel processing computer system), and Bill Gates. I had
`
`responsibility for hiring and managing over 15 staff members including multiple
`
`Ph.Ds. with degrees in electrical engineering and decades of experience in product
`
`design and engineering.
`
`13. A summary of some of my qualifications for forming the opinions in
`
`this declaration are as follows: I have more than 30 years of experience as a computer
`
`architect, computer system designer, educator, and as an executive in the PC and
`
`6
`
`ATI Ex. 2001
`IPR2023-00922
`Page 16 of 268
`
`

`

`
`
`electronics business. I am also a member of several professional associations, such
`
`as the ACM, IEEE and have been intimately involved in professional research
`
`through the International Symposium on Microarchitecture (Program Chair for 26th
`
`and General Chair for 36th), IEEE Transactions on Computers (Associate Editor),
`
`ACM Transactions on Embedded Computing Systems (Associate Editor), and IEEE
`
`Computer (Associate Editor). I also have been on the program committees for ISCA,
`
`MICRO, ISLPED, Network Processors Workshop, FPL, Complexity-Effective
`
`Design, RAW, Workshop on Mediaprocessors, and DSP, FPT, and INTERACT.
`
`14. For further details regarding my employment and academic history,
`
`please refer to my curriculum vitae Appendix A.
`
`IV. HIGH LEVEL GRAPHICS PROCESSING OVERVIEW
`15. The ’454 Patent discloses novel architectures for graphics processing
`
`unit circuitry. Consumer products are often used to generate and display graphics on
`
`an output device such as a built-in screen or an external monitor. When complex and
`
`realistic graphics are desired, there is often additional specialized circuitry, in the
`
`form of a chip, which is added to the consumer product to assist it with the complex
`
`processing that it must perform to render the graphics to the screen. This specialized
`
`circuitry is known as a graphics processing unit or “GPU.”
`
`A. Graphics Processors Use Vertex and Pixel Data to Create Display
`Images
`16. Graphics processors (GPUs) are designed to convert a three-
`
`7
`
`ATI Ex. 2001
`IPR2023-00922
`Page 17 of 268
`
`

`

`
`
`dimensional object into an image for display on a two-dimensional screen. As part
`
`of this process, a three-dimensional object is rendered as a collection of simple
`
`shapes, called primitives. For example, in the figure below, a three-dimensional
`
`character has been rendered as a collection of triangular primitives.
`
`
`17. Each triangular primitive is defined by the positions of its three corner
`
`points, i.e., its “vertices.” After a three-dimensional object is rendered as a group of
`
`primitives, subsequent processing steps include converting the vertices of each
`
`primitive from three-dimensional coordinates to two-dimensional coordinates. To
`
`illustrate, in the example shown below, a triangular primitive is being used to render
`
`a portion of a sphere and has vertices with three-dimensional position coordinates
`
`along the X-, Y-, and Z- axes. These three-dimensional vertices coordinates are
`
`8
`
`ATI Ex. 2001
`IPR2023-00922
`Page 18 of 268
`
`

`

`
`
`converted into two-dimensional coordinates (along only the X- and Y- axes) for
`
`display on a two-dimensional screen.
`
`
`
`18. Subsequent processing steps include rendering each primitive as two-
`
`dimensional collection of dots on a screen called “pixels.” As part of this process,
`
`the graphics processor uses the two-dimensional vertices coordinates to determine
`
`which pixels fill a particular primitive. In the illustration below, the pixels that fill
`
`the primitive defined by the XY coordinates are depicted in blue.
`
`19. Further processing includes identifying the correct color and texture for
`
`
`
`9
`
`ATI Ex. 2001
`IPR2023-00922
`Page 19 of 268
`
`

`

`
`
`each pixel filling the primitive.
`
`
`
`B. Graphics Processors Use Instruction “Threads” To Transform
`Vertex and Pixel Data
`20. Graphics processors use various collections of instructions to transform
`
`a three-dimensional object into a two-dimensional display. Each collection of
`
`instructions is called a “thread.” Each thread contains a sequence of instructions to
`
`be performed one after another. Each instruction consists of an “OpCode” and an
`
`“Operand.” The OpCode indicates which calculation operation is required by the
`
`instruction, such as an arithmetic or logical operation. The Operand indicates the
`
`location of the graphics data upon which the calculation operation is to be performed.
`
`
`
`10
`
`ATI Ex. 2001
`IPR2023-00922
`Page 20 of 268
`
`

`

`
`
`21. Some threads comprise instructions that manipulate and transform
`
`vertex data. These threads are known as “vertex threads,” and can include
`
`instructions that, for example, work together to transform a triangular primitive’s
`
`vertex coordinates from three-dimensional to two-dimensional (as discussed above),
`
`to translate or scale the primitive, to remove any portion of the triangular primitive
`
`that may be off the screen, or to cull primitives that do not project onto the screen.
`
`22. Other threads comprise instructions that manipulate and transform pixel
`
`data. These threads are known as “pixel threads,” and can include instructions that,
`
`for example, work together to transform pixel data by assigning color value to the
`
`pixels, determining lighting effects, determining texture coordinate values, and
`
`blending colors.
`
`C. Conventional Graphics Processors Executed Vertex And Pixel
`Threads Using Separate Vertex Shaders And Pixel Shaders
`23. Conventional graphics processors featured several circuits, each
`
`configured to perform a single type of transformation. These circuits are known as
`
`“shaders.”
`
`24. Conventionally, each shader was dedicated to executing a particular
`
`type of thread to transform a particular type of data. For example, a “vertex shader”
`
`would execute only vertex threads to transform vertex data, such as, for example,
`
`transforming the three-dimensional coordinates of a primitive’s vertices in object
`
`space into two-dimensional coordinates for a display screen. EX. 1001 at 1:63-2:7.
`
`11
`
`ATI Ex. 2001
`IPR2023-00922
`Page 21 of 268
`
`

`

`
`
`But to execute pixel threads to transform pixel data, another, separate shader was
`
`needed. This type of shader is known as a “pixel shader,” and was used, for example,
`
`determine and apply the color for each pixel making up a rendered object. Id. at 2:8-
`
`18.
`
`25. Thus, to execute the position and color threads and operations necessary
`
`to transform a three-dimensional object to a two-dimensional display, a conventional
`
`graphics processor needed at least two shaders, one a vertex shader and the other a
`
`pixel shader. Id. at 2:15-18. As described in the ’454 patent, this requirement resulted
`
`in drawbacks in both size and performance. First, the need for multiple specialized
`
`shaders increased the size of graphics processors:
`
`Conventional graphics processors require the use of both a vertex
`shader and a pixel shader in order to generate an object. Because both
`types of shaders are required, known graphics processors are relatively
`large in size, with most of the real estate being taken up by the vertex
`and pixel shaders.
`
`Id. at 2:20-29.
`
`26. Second, sequencing the pixel shader after the vertex shader meant that
`
`one shader was dependent on – and had to wait for – the output of the other before
`
`it could continue processing, leading to idle downtime that slowed down the graphics
`
`processor and resulted in relatively poor performance:
`
`In addition to the real estate penalty associate with conventional
`graphics processors, there is also a corresponding performance pena