`
`
`
`
`
`
`
`
`
`
`
`UNITED STATES PATENT AND TRADEMARK OFFICE
`__________________________________
`
`BEFORE THE PATENT TRIAL AND APPEAL BOARD
`__________________________________
`
`AMAZON.COM, INC.,
`Petitioner,
`
`v.
`
`CUSTOMPLAY, LLC,
`Patent Owner.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Case No. TBD
`U.S. Patent No. 9,124,950
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`DECLARATION OF ALAN C. BOVIK IN SUPPORT OF
`PETITION OF AMAZON.COM, INC. FOR
`INTER PARTES REVIEW OF U.S. PATENT NO. 9,124,950
`
`
`
`AMAZON EX. 1002
`Amazon v. CustomPlay
`US Patent No. 9,124,950
`
`

`

`Amazon.com v. CustomPlay
`IPR Petition – U.S. Patent No. 9,124,950
`TABLE OF CONTENTS
`
`
`I.
`
`BACKGROUND ............................................................................................. 2 
`A. 
`Experience and Qualifications .............................................................. 2 
`B. 
`Applicable Legal Standards .................................................................. 7 
`C. 
`Person of Ordinary Skill in the Art ..................................................... 10 
`II. BACKGROUND OF THE TECHNOLOGY ................................................ 12 
`III. THE ’950 PATENT ....................................................................................... 17 
`A. 
`Summary of the ’950 Patent ................................................................ 17 
`1. 
`The Specification ...................................................................... 17 
`2. 
`The Claims ................................................................................ 17 
`The Priority Date of the ’950 Patent ................................................... 19 
`B. 
`Claim Construction ............................................................................. 19 
`C. 
`IV. CLAIMS 2, 4, 6, 14, 16, AND 19 OF THE ’950 PATENT
`WOULD HAVE BEEN OBVIOUS .............................................................. 19 
`A. 
`Claims 6 and 19 Would Have Been Obvious in View
`of Rangan and the Knowledge of a Person of Ordinary
`Skill in the Art. .................................................................................... 20 
`1. 
`Claim 6 ...................................................................................... 22 
`a. 
`Retrieving a Video Frame Identifier Responsive
`to a Play Location ........................................................... 22 
`Displaying An Initial Indication That Information
`is Available ..................................................................... 25 
`Retrieving a Subsequent Video Frame Identifier ........... 27 
`Displaying Subsequent Indication
`Contemporaneously with Initial Indication .................... 27 
`
`c. 
`d. 
`
`b. 
`
`-i-
`
`

`

`f. 
`
`B. 
`
`C. 
`
`Amazon.com v. CustomPlay
`IPR Petition – U.S. Patent No. 9,124,950
`Receiving a Request Responsive to the
`e. 
`Initial Indication .............................................................. 30 
`Displaying Information Associated with the
`Initial Indication .............................................................. 32 
`Claim 19 .................................................................................... 34 
`2. 
`Claims 2, 14, and 16 Would Have Been Obvious in View
`of Rangan and Rakib. .......................................................................... 34 
`1. 
`Claim 2 ...................................................................................... 34 
`a. 
`Receiving a Request for Additional Information ........... 34 
`b. 
`Displaying Additional Information ................................ 41 
`Claims 14 and 16....................................................................... 42 
`2. 
`Claim 4 Would Have Been Obvious in View of Rangan,
`Rakib, and Abecassis. .......................................................................... 43 
`1. 
`Claim 4 ...................................................................................... 43 
`a. 
`Pausing the Playing in Response to the Request
`for Additional Information ............................................. 43 
`Resuming Playing At a Beginning of a Video
`Clip That is Responsive to the Request for
`Additional Information ................................................... 47 
`Claim Chart ............................................................................... 51 
`2. 
`Claims 2, 6, 14, 16, and 19 Would Have Been Obvious
`in View of Armstrong and Livesey. .................................................... 71 
`1. 
`Claim 6 ...................................................................................... 74 
`a. 
`Retrieving a Video Frame Identifier Responsive
`to a Play Location ........................................................... 74 
`Displaying An Initial Indication That Information
`is Available ..................................................................... 76 
`
`D. 
`
`b. 
`
`b. 
`
`-ii-
`
`

`

`e. 
`
`f. 
`
`2. 
`
`Amazon.com v. CustomPlay
`IPR Petition – U.S. Patent No. 9,124,950
`Retrieving a Subsequent Video Frame Identifier ........... 78 
`c. 
`d. 
`Displaying Subsequent Indication
`Contemporaneously with Initial Indication .................... 78 
`Receiving a Request Responsive to the
`Initial Indication .............................................................. 87 
`Displaying Information Associated with the
`Initial Indication .............................................................. 88 
`Claim 2 ...................................................................................... 90 
`a. 
`Receiving a Request for Additional Information ........... 90 
`b. 
`Displaying Additional Information ................................ 91 
`Claim 14 .................................................................................... 92 
`3. 
`Claim 16 .................................................................................... 92 
`4. 
`Claim 19 .................................................................................... 92 
`5. 
`Claim 4 Would Have Been Obvious in View of Armstrong,
`Livesey, and Abecassis. ...................................................................... 93 
`1. 
`Claim 4 ...................................................................................... 93 
`a. 
`Pausing the Playing in Response to the Request
`for Additional Information ............................................. 93 
`Resuming Playing At a Beginning of a Video Clip
`That is Responsive to the Request for Additional
`Information ..................................................................... 96 
`Claim Chart ............................................................................... 99 
`2. 
`Claims 2, 4, 6, 14, 16, and 19 Would Have Been Obvious
`in View of Rakib and Livesey. ..........................................................117 
`1. 
`Claim 6 ....................................................................................118 
`a. 
`Retrieving a Video Frame Identifier Responsive
`to a Play Location .........................................................119 
`-iii-
`
`E. 
`
`b. 
`
`F. 
`
`

`

`c. 
`d. 
`
`Amazon.com v. CustomPlay
`IPR Petition – U.S. Patent No. 9,124,950
`Displaying an Initial Indication That Information
`b. 
`is Available ...................................................................125 
`Retrieving a Subsequent Video Frame Identifier .........127 
`Displaying Subsequent Indication
`Contemporaneously with Initial Indication ..................129 
`Receiving a Request Responsive to the
`Initial Indication ............................................................135 
`Displaying Information Associated with
`the Initial Indication ......................................................137 
`Claim 2 ....................................................................................140 
`a. 
`Receiving a Request for Additional Information .........140 
`b. 
`Displaying Additional Information ..............................141 
`Claim 4 ....................................................................................142 
`a. 
`Pausing the Playing in Response to the Request
`for Additional Information ...........................................142 
`Resuming Playing At a Beginning of a Video
`Clip That is Responsive to the Request for
`Additional Information .................................................145 
`Claim 14 ..................................................................................147 
`4. 
`Claim 16 ..................................................................................147 
`5. 
`Claim 19 ..................................................................................147 
`6. 
`Claim Chart .............................................................................148 
`7. 
`V. CONCLUSION ............................................................................................169 
`
`2. 
`
`3. 
`
`e. 
`
`f. 
`
`b. 
`
`-iv-
`
`

`

`Amazon.com v. CustomPlay
`IPR Petition – U.S. Patent No. 9,124,950
`I, Alan C. Bovik, do hereby declare:
`
`1.
`
`I am making this Declaration at the request of Amazon.com, Inc.
`
`(“Amazon”).
`
`2.
`
`I am being compensated for my work in this matter and I am being re-
`
`imbursed at cost for my expenses. My compensation in no way depends upon the
`
`outcome of this proceeding.
`
`3.
`
`In preparing this Declaration, I considered the following materials:
`
`Exhibit No.
`1001
`
`Description
`U.S. Patent No. 9,124,950 (“the ’950 patent”)
`
`1004
`
`1005
`
`1010
`
`1011
`
`1013
`
`1014
`
`1021
`
`1022
`
`1023
`
`1024
`
`U.S. Patent Publication No. 2007/0250901 (“McIntire”)
`
`U.S. Patent No. 5,696,905 (“Reimer”)
`
`Prosecution File History for the ’950 Patent
`
`Nevenka Dimitrova et al., Media Augmentation and Personaliza-
`tion Through Multimedia Processing and Information Extrac-
`tion, in Personalized Digital Television 203 (L.Ardissono et al.
`eds., 2004) (“Dimitrova”)
`
`U.S. Patent No. 8,875,212 (“Rakib”)
`
`U.S. Patent No. 6,154,771 (“Rangan”)
`
`U.S. Patent Publication No. 2007/0003223 (“Armstrong”)
`
`U.S. Patent Publication No. 2008/0253739 (“Livesey”)
`
`U.S. Patent No. 6,965,890 (“Dey”)
`
`U.S. Patent No. 6,038,367 (“Abecassis”)
`
`-1-
`
`

`

`Amazon.com v. CustomPlay
`IPR Petition – U.S. Patent No. 9,124,950
`
`Exhibit No.
`1025
`
`Description
`U.S. Patent No. 4,694,490 (“Harvey”)
`
`1026
`
`Excerpts from Interactive Video: Algorithms and Technologies
`(R. I. Hammoud ed., 2006) (“Hammoud”)
`
`
`
`I. BACKGROUND
`A. Experience and Qualifications
`
`4. My experience and qualifications are summarized in my curriculum
`
`vitae, a copy of which is provided as Exhibit 1020.
`
`5.
`
`I hold a Ph.D. in Electrical and Computer Engineering from the Uni-
`
`versity of Illinois, Urbana-Champaign (awarded in 1984). I also hold a Master’s
`
`degree in Electrical and Computer Engineering from the University of Illinois, Ur-
`
`bana-Champaign (awarded in 1982).
`
`6.
`
`I am a tenured full Professor and I hold the Cockrell Family Regents
`
`Endowed Chair at the University of Texas at Austin. My appointments are in the
`
`Department of Electrical and Computer Engineering, the Department of Computer
`
`Sciences, and the Department of Biomedical Engineering. I am also the Director
`
`of the Laboratory for Image and Video Engineering (“LIVE”).
`
`7. My research is in the general area of digital television, digital camer-
`
`as, image and video processing, computational neuroscience, and modeling of bio-
`
`logical visual perception. I have published over 800 technical articles in these areas
`
`-2-
`
`

`

`Amazon.com v. CustomPlay
`IPR Petition – U.S. Patent No. 9,124,950
`and hold seven U.S. patents. I am also the author of The Handbook of Image and
`
`Video Processing, Second Edition (Elsevier Academic Press, 2005); Modern Im-
`
`age Quality Assessment (Morgan & Claypool, 2006); The Essential Guide to Im-
`
`age Processing (Elsevier Academic Press, 2009); and The Essential Guide to Video
`
`Processing (Elsevier Academic Press, 2009); and numerous other publications.
`
`8.
`
`I received the 2017 Edwin H. Land Medal from the Optical Society of
`
`America in September 2017 with the following citation: For substantially shaping
`
`the direction and advancement of modern perceptual picture quality computation,
`
`and for energetically engaging industry to transform his ideas into global practice.
`
`I received a Primetime Emmy Award for Outstanding Achievement in Engineering
`
`Development, for the Academy of Television Arts and Sciences, in October 2015,
`
`for the widespread use of my video quality prediction and monitoring models and
`
`algorithms that are widely used throughout the global broadcast, cable, satellite and
`
`internet television industries.
`
`9.
`
`Among other awards and honors, I have received the 2013 Institute of
`
`Electrical and Electronics Engineers (“IEEE”) Signal Processing Society’s “Socie-
`
`ty Award,” which is the highest honor accorded by that technical society (“for fun-
`
`damental contributions to digital image processing theory, technology, leadership
`
`and education”). In 2005, I received the Technical Achievement Award of the
`
`IEEE Signal Processing Society, which is the highest technical honor given by the
`
`-3-
`
`

`

`Amazon.com v. CustomPlay
`IPR Petition – U.S. Patent No. 9,124,950
`Society, for “broad and lasting contributions to the field of digital image pro-
`
`cessing”; and in 2008 I received the Education Award of the IEEE Signal Pro-
`
`cessing Society, which is the highest education honor given by the Society, for
`
`“broad and lasting contributions to image processing, including popular and im-
`
`portant image processing books, innovative on-line courseware, and for the crea-
`
`tion of the leading research and educational journal and conference in the image
`
`processing field.”
`
`10. My technical articles have been widely recognized as well, including
`
`the 2009 IEEE Signal Processing Society Best Journal Paper Award for the paper
`
`“Image quality assessment: From error visibility to structural similarity,” published
`
`in IEEE Transactions on Image Processing, volume 13, number 4, April 2004; this
`
`same paper received the 2017 IEEE Signal Processing Society Sustained Impact
`
`Paper Award as the most impactful paper published over a period of at least ten
`
`years; the 2013 Best Magazine Paper Award for the paper “Mean squared error:
`
`Love it or leave it?? A new look at signal fidelity measures,” published in IEEE
`
`Transactions on Image Processing, volume 26, number 1, January 2009; the IEEE
`
`Circuits and Systems Society Best Journal Paper Prize for the paper “Video quality
`
`assessment by reduced reference spatio-temporal entropic differencing,” published
`
`in IEEE Transactions on Circuits and Systems for Video Technology, vol. 23, no.
`
`4, pp. 684-694, April 2013.
`
`-4-
`
`

`

`Amazon.com v. CustomPlay
`IPR Petition – U.S. Patent No. 9,124,950
`11.
`I received the Google Scholar Classic Paper citation twice in 2017, for
`
`the paper “Image information and visual quality,” published in the IEEE Transac-
`
`tions on Image Processing, vol. 15, no. 2, pp. 430-444, February 2006 (the main
`
`algorithm developed in the paper, called the Visual Information Fidelity (“VIF”)
`
`Index, is a core picture quality prediction engine used to quality-assess all encodes
`
`streamed globally by Netflix), and for “An evaluation of recent full reference im-
`
`age quality assessment algorithms,” published in the IEEE Transactions on Image
`
`Processing, vol. 15, no. 11, pp. 3440-3451, November 2006 (the picture quality da-
`
`tabase and human study described in the paper, the LIVE Image Quality Database,
`
`has been the standard development tool for picture quality research since its first
`
`introduction in 2003). Google Scholar Classic Papers are very highly-cited papers
`
`that have stood the test of time, and are among the ten most-cited articles in their
`
`area of research over the ten years since their publication.
`
`12.
`
`I have also been honored by other technical organizations, including
`
`the Society for Photo-optical and Instrumentation Engineers (“SPIE”), from which
`
`I received the Technology Achievement Award (2013) “For Broad and Lasting
`
`Contributions to the Field of Perception-Based Image Processing,” and the Society
`
`for Imaging Science and Technology, which accorded me Honorary Membership,
`
`which is the highest recognition by that Society given to a single individual, “for
`
`his impact in shaping the direction and advancement of the field of perceptual im-
`
`-5-
`
`

`

`Amazon.com v. CustomPlay
`IPR Petition – U.S. Patent No. 9,124,950
`age processing.” I was also elected as a Fellow of the IEEE “for contributions to
`
`nonlinear image processing” in 1995, a Fellow of the Optical Society of America
`
`(“OSA”) for “fundamental research contributions to and technical leadership in
`
`digital image and video processing” in 2006, and as a Fellow of SPIE for “pioneer-
`
`ing technical, leadership, and educational contributions to the field of image pro-
`
`cessing” in 2007.
`
`13.
`
`I have many years of experience in the design of digital video sys-
`
`tems. For example, among other relevant research, I have worked with the Nation-
`
`al Aeronautics and Space Administration (“NASA”) to develop high compression
`
`image sequence coding and animated vision technology, on various military pro-
`
`jects for the Air Force Office of Scientific Research, Phillips Air Force Base, the
`
`Army Research Office, and the Department of Defense. These projects have fo-
`
`cused on developing local spatio-temporal analysis in vision systems, scalable pro-
`
`cessing of multi-sensor and multi-spectral imagery, image processing and data
`
`compression tools for satellite imaging, AM-FM analysis of images and video, the
`
`scientific foundations of image representation and analysis, computer vision sys-
`
`tems for automatic target recognition and automatic recognition of human activi-
`
`ties, vehicle structure recovery from a moving air platform, passive optical model-
`
`ing, and detection of speculated masses and architectural distortions in digitized
`
`mammograms. My research has also recently been funded by Netflix, Qualcomm,
`
`-6-
`
`

`

`Amazon.com v. CustomPlay
`IPR Petition – U.S. Patent No. 9,124,950
`Facebook, Texas Instruments, Intel, Cisco, and the National Institute of Standards
`
`and Technology (“NIST”) for research on image and video quality assessment. I
`
`have also received numerous grants from the National Science Foundation for re-
`
`search on image and video processing and on computational vision.
`
`B. Applicable Legal Standards
`
`14.
`
` I have been asked to provide my opinion as to whether the claims of
`
`the ’950 patent would have been obvious to a person of ordinary skill in the art at
`
`the time of the alleged invention, in view of the prior art.
`
`15.
`
`I am an engineer by training and profession. The opinions I am ex-
`
`pressing in this report involve the application of my training and technical
`
`knowledge and experience to the evaluation of certain prior art with respect to the
`
`’950 patent.
`
`16. Although I have been involved as a technical expert in patent matters
`
`before, I am not an expert in patent law. Therefore, the attorneys from Knobbe,
`
`Martens, Olson & Bear, LLP have provided me with guidance as to the applicable
`
`patent law in this matter. The paragraphs below express my understanding of how
`
`I must apply current principles related to patent validity to my analysis.
`
`17.
`
`It is my understanding that in determining whether a patent claim is
`
`obvious in view of the prior art, the Patent Office construes the claim by giving the
`
`claim its broadest reasonable interpretation consistent with the specification. For
`
`-7-
`
`

`

`Amazon.com v. CustomPlay
`IPR Petition – U.S. Patent No. 9,124,950
`the purposes of this review, and to the extent necessary, I have construed each
`
`claim term in accordance with its plain and ordinary meaning under the required
`
`broadest reasonable interpretation. My opinions would not change, however, if the
`
`claims were given their ordinary and customary meaning ascribed to them by a
`
`person of ordinary skill in the art at the time of the invention.
`
`18.
`
`It is my understanding that a claim is “obvious” if the claimed subject
`
`matter as a whole would have been obvious to a person of ordinary skill in the art
`
`at the time of the alleged invention. I also understand that an obviousness analysis
`
`takes into account the scope and content of the prior art, the differences between
`
`the claimed subject matter and the prior art, and the level of ordinary skill in the art
`
`at the time of the invention.
`
`19.
`
`In determining the scope and content of the prior art, it is my under-
`
`standing that a reference is considered appropriate prior art if it falls within the
`
`field of the inventor’s endeavor. In addition, a reference is prior art if it is reason-
`
`ably pertinent to the particular problem with which the inventor was involved. A
`
`reference is reasonably pertinent if it logically would have commended itself to an
`
`inventor’s attention in considering his problem. If a reference relates to the same
`
`problem as the claimed invention, that supports use of the reference as prior art in
`
`an obviousness analysis.
`
`-8-
`
`

`

`Amazon.com v. CustomPlay
`IPR Petition – U.S. Patent No. 9,124,950
`20. To assess the differences between prior art and the claimed subject
`
`matter, it is my understanding that the law requires the claimed invention to be
`
`considered as a whole. This “as a whole” assessment requires showing that one of
`
`ordinary skill in the art at the time of invention, confronted by the same problems
`
`as the inventor and with no knowledge of the claimed invention, would have se-
`
`lected the elements from the prior art and combined them in the claimed manner.
`
`21.
`
`It is my understanding that something is “inherent in,” and therefore
`
`taught by, the prior art, if it necessarily flows from the explicit disclosure of the
`
`prior art. I understand that the fact that a certain result or characteristic may be
`
`present in the prior art is not sufficient to establish inherency. However, if the re-
`
`sult or characteristic is necessarily present based upon the explicit disclosure in the
`
`prior art, it is inherent in the prior art and is therefore disclosed.
`
`22.
`
`It is my further understanding that the law recognizes several ration-
`
`ales for combining references or modifying a reference to show obviousness of
`
`claimed subject matter. Some of these rationales include: combining prior art ele-
`
`ments according to known methods to yield predictable results; simple substitution
`
`of one known element for another to obtain predictable results; a predictable use of
`
`prior art elements according to their established functions; applying a known tech-
`
`nique to a known device (method or product) ready for improvement to yield pre-
`
`dictable results; choosing from a finite number of identified, predictable solutions,
`
`-9-
`
`

`

`Amazon.com v. CustomPlay
`IPR Petition – U.S. Patent No. 9,124,950
`with a reasonable expectation of success; and some teaching, suggestion, or moti-
`
`vation in the prior art that would have led one of ordinary skill to modify the prior
`
`art reference or to combine prior art reference teachings to arrive at the claimed in-
`
`vention.
`
`23.
`
`I also understand that an obviousness analysis must consider whether
`
`there are additional factors that would indicate that the invention would not have
`
`been obvious. These factors include whether there was: (i) a long-felt need in the
`
`industry; (ii) any unexpected results; (iii) skepticism of the invention; (iv) a teach-
`
`ing away from the invention; (v) commercial success; (vi) praise by others for the
`
`invention; and (vii) copying by other companies. I am not aware of any evidence
`
`that would suggest that the claims of the ’950 patent would have been non-
`
`obvious.
`
`C.
`
`24.
`
`Person of Ordinary Skill in the Art
`
`It is my understanding that when interpreting the claims of the ’950
`
`patent, I must do so based on the perspective of a person of ordinary skill in the art
`
`at the relevant priority date. As discussed below, I understand the relevant priority
`
`date to be March 26, 2012.
`
`25. The ’950 patent describes the use of well-known technologies for
`
`providing supplemental information with videos. Based on my review of the spec-
`
`ification and claims of the ’950 patent, it is my opinion that a person of ordinary
`
`-10-
`
`

`

`Amazon.com v. CustomPlay
`IPR Petition – U.S. Patent No. 9,124,950
`skill in the art would be a person having at least a bachelor’s degree in Electrical
`
`Engineering, Computer Engineering, or Computer Science. Additionally, such a
`
`person would have at least three years of experience in the design of digital video
`
`systems.
`
`26.
`
`I am able to make this assessment because in the 1990s and 2000s, I
`
`was familiar with the design and operation of numerous systems for displaying
`
`supplemental information with videos. With my graduate students, I conducted re-
`
`search on visual eye tracking, whereby the scan paths and fixation points of human
`
`viewers of videos were recorded as annotations that would later be analyzed and/or
`
`viewed as supplemental information with or superimposed on each video. I also
`
`have conducted extensive human studies whereby human subjects would view nu-
`
`merous videos and would record their subjective impressions of the video quality
`
`over time (or overall). This supplemental video information could then be ac-
`
`cessed for each video showing graphs, traces, or numerical representations of video
`
`quality. The engineers and scientists that I worked with during the 1990s and
`
`2000s had the requisite knowledge to make and use systems as described in the
`
`claims of the ’950 patent. Because I have worked with and supervised engineers in
`
`the field of video metadata, I know very well what their capabilities were in the
`
`1990s and 2000s, how those engineers would interpret and understand the claims
`
`-11-
`
`

`

`Amazon.com v. CustomPlay
`IPR Petition – U.S. Patent No. 9,124,950
`of the ’950 patent, and how they would understand the disclosures in the prior art
`
`discussed herein.
`
`27.
`
`In my opinion, as set forth in more detail below, a person having ordi-
`
`nary skill in the art at the time of the invention would have considered the devices
`
`and methods claimed in the ’950 patent to be obvious in view of the prior art.
`
`II. BACKGROUND OF THE TECHNOLOGY
`
`28.
`
`In a video context, supplemental information, also referred to as “con-
`
`tent information,” “extrinsic data,” or “video metadata,” is used to display certain
`
`information that describes content within a video presented to a user. Including
`
`supplemental information with, for example, a movie enhances the viewing experi-
`
`ence by providing on-demand access to background information without requiring
`
`a user to perform an external search. An example of supplemental information is
`
`biographical information of actors appearing in video content, thereby answering
`
`the age-old question: “who is that guy?” Supplemental information is also used to
`
`provide other useful information, such as music playing in a scene (“what is this
`
`song?”), filming location (“where is this street/building?”), trivia, and mistakes
`
`made during filming (such as inconsistencies in wardrobe or display of buildings
`
`that do not exist in the purported location of the movie).
`
`29. Development of video systems that could display supplemental infor-
`
`mation began in the mid-1990s when the use of digital content became widespread.
`
`-12-
`
`

`

`Amazon.com v. CustomPlay
`IPR Petition – U.S. Patent No. 9,124,950
`This time period coincided with the rise in popularity of DVDs, digital cable tele-
`
`vision, and digital satellite television. Before digital, most content was distributed
`
`through analog signals via cable (i.e., cable television) or video home system
`
`(“VHS”) tapes. The digitization of content enabled individual portions of a video
`
`to be coded with supplemental information. In addition, the use of digital trans-
`
`mission networks and display devices enabled the supplemental information itself
`
`to be easily sent and stored as one or more digital files.
`
`30. There are many examples of systems for providing supplemental in-
`
`formation during a video that have been known for decades. E.g., Ex. 1025 (Har-
`
`vey) (1987), 20:16-68 (viewers of a TV program may request supplemental infor-
`
`mation such as a recipe being discussed during cooking show); Ex. 1014 (Rangan),
`
`2:20-5:26 (discussing hypervideo systems dating back to 1991). For example, in
`
`1995, Reimer disclosed:
`
`presenting [a] movie to the user, and then receiving from the user a
`query pertaining to the movie. The invention determines the frame of
`the movie that was being presented to the user when the user issued
`the query …. The invention identifies, as specified by the query,
`portions of the movie related information relating to the frame, and
`retrieves those portions of the move related information. These
`retrieved portions of the movie related information are presented to
`the user.
`
`Ex. 1005 (Reimer), 3:32-42.
`
`-13-
`
`

`

`Amazon.com v. CustomPlay
`IPR Petition – U.S. Patent No. 9,124,950
`31. The presence of supplemental information was often signaled by an
`
`indication or alert. Rangan, for example, discloses a “hotspot” such as “a grid, or a
`
`coloration, or some other visually perceptible ‘clue’” to indicate the presence of a
`
`hyperlink. Ex. 1014 (Rangan), 15:23-33. Rakib disclosed various types of indica-
`
`tions, including a “small picture-in-picture of icons or small photos of recognized
`
`objects, putting an arrow or marker over the metadata linked objects or by other
`
`means.” Ex. 1013 (Rakib) ¶[0082].
`
`32. Because supplemental information was provided for items on the
`
`screen at any given time, the indications were often mapped to individual frames of
`
`the video. Ex. 1005 (Reimer), 3:32-42; Ex. 1014 (Rangan), 15:41-47 (“The over-
`
`laying of the hotspots is preferably done for individual hypervideo frames[.]”); Ex.
`
`1013 (Rakib) ¶[0113] (information tied to individual frame number). However,
`
`this approach was problematic because the indications and hyperlinks could disap-
`
`pear rapidly as the video frame changed. As Rangan explained:
`
`The objects and events, and their associated hyperlinks, may also be
`exceedingly transitory. The whole purpose of interactive video is
`compromised if the only way that a subscriber/user/viewer may be
`assured of ‘catching’ and exercising a hyperlink is to remain tensely
`poised on the unfolding video, ready to both make near instantaneous
`judgements and to physically ‘spring like a cat’ when the opportunity
`to profitably hyperlink presents itself. This much tension can ruin the
`hypervideo viewing experience.
`
`-14-
`
`

`

`Amazon.com v. CustomPlay
`IPR Petition – U.S. Patent No. 9,124,950
`Ex. 1014 (Rangan), 16:1-67, 19:58-20:19; see also Ex. 1023 (Dey), 3:16-22 (rec-
`
`ognizing that “it may take a certain amount of time for the user to decide that he is
`
`interested in obtaining additional material, and a certain amount of time to maneu-
`
`ver the mouse … to indicate interest”), 7:30-35; Ex. 1011 (Dimitrova) at 212
`
`(“Displaying only the actors currently on screen would often require users to scan
`
`back in the movie, because, by the time they realized they wanted the information
`
`and grabbed the remote control, the shot with the actor they wanted might have
`
`ended”); Ex. 1004 (McIntire) ¶¶[140], [0272] (“the [user request] may be received
`
`at time t0, but the viewer may be reacting to an article appearing in the media
`
`stream at time t0-n (i.e., the viewer’s reaction time is slower than the progression of
`
`the media stream)”).
`
`33. The prior art solved this problem long before the priority date of the
`
`’950 patent. Rangan overcame this problem by displaying thumbnails 74 of past
`
`frames with hotspots to allow a user to click a hotspot after the frame had passed:
`
`-15-
`
`

`

`Amazon.com v. CustomPlay
`IPR Petition – U.S. Patent No. 9,124,950
`
`
`
`Ex. 1014 (Rangan), 10:20-44, 15:61-16:67. Armstrong, by contrast, maintained a
`
`menu on the screen while the video played and, rather than removing indices from
`
`the menu once they were no longer on the screen, simply added indices to new ob-
`
`jects as they appeared in the scene. Ex. 1021 (Armstrong) ¶[0023]. Other systems
`
`left the alert (e.g., icon) on the screen for an extended period of time to allow the
`
`user time to select the icon and access the associated information. Ex. 1022
`
`(Livesey) ¶[0012]. Each of these solutions included displaying indications from
`
`multiple frames contemporaneously.
`
`-16-
`
`

`

`Amazon.com v. CustomPlay
`IPR Petition – U.S. Patent No. 9,1