Paper No.
`
`UNITED STATES PATENT AND TRADEMARK OFFICE
`
`
`BEFORE THE PATENT TRIAL AND APPEAL BOARD
`
`
`SONY CORPORATION
`
`Petitioner
`
`V.
`
`Patent of YISSUM RESEARCH DEVELOPMENT COMPANY OF THE
`
`HEBREW UNIVERSITY OF JERUSALEM
`
`Patent Owner
`
`
`Case lPR2013-00219 (SCM)1
`Patent 7,477,284
`Title: SYSTEM AND METHOD FOR CAPTURING AND VIEWING
`
`STEREOSCOPIC PANORAMIC IMAGES
`
`
`DECLARATION OF [RF AN ESSA PH.D.
`
`1 The IPR2013-00327 proceeding has beenjoined with this proceeding.
`
`
`
`UNITED STATES PATENT AND TRADEMARK OFFICE
`
`
`BEFORE THE PATENT TRIAL AND APPEAL BOARD
`
`
`SONY CORPORATION
`
`Petitioner
`
`V.
`
`Patent of YISSUM RESEARCH DEVELOPMENT COMPANY OF THE
`
`HEBREW UNIVERSITY OF JERUSALEM
`
`Patent Owner
`
`
`Case lPR2013-00219 (SCM)1
`Patent 7,477,284
`Title: SYSTEM AND METHOD FOR CAPTURING AND VIEWING
`
`STEREOSCOPIC PANORAMIC IMAGES
`
`
`DECLARATION OF [RF AN ESSA PH.D.
`
`1 The IPR2013-00327 proceeding has beenjoined with this proceeding.
`
`
`
`I, Dr. lrfan Essa, declare:
`
`1.
`
`I am making this declaration at the request of Yissum Research
`
`Development Company of the Hebrew University of Jerusalem, in the matter of the
`
`Inter Partes Review Case lPR2013-00219, of US. Patent No 7,477,284 (“the ”284
`
`Patent,” SONY-1001) to Peleg.
`
`2.
`
`I am being compensated for my work in this matter. My compensation
`
`in no way depends upon the outcome of this proceeding.
`
`3.
`
`In the preparation of this declaration, l have studied:
`
`(1)
`
`The Sony’s Petitions in lPR2013-00219 and lPR2013-00327;
`
`(2)
`
`The Board’s Decisions in lPR2013-00219 and lPR2013-00327;
`
`(3)
`
`The Declaration of Dr. Trevor Darrell, SON Y-1013;
`
`(4)
`
`The ’284 Patent, SONY-1001;
`
`(5) US. Patent No. 6,665,003 (“the ’003 Patent), SONY-1002;
`
`(6)
`
`The English translation of Kawakita et al., (“Kawakita”),
`
`SONY-1004;
`
`(7)
`
`The English translation of Asahi et al., (“Asahi”), SON Y-101 0.
`
`
`
`4.
`
`In forming the opinions expressed below, I have considered:
`
`(1)
`
`The documents listed above;
`
`(2)
`
`The relevant legal standards, including the standard for
`
`obviousness provided in KSR International Co. v. Teleflex, Inc., 550
`
`US. 398 (2007) and any additional authoritative documents as cited in
`
`the body of this declaration; and
`
`(3) My knowledge and experience based upon my work in this area
`
`as described below.
`
`
`Highlighted Qualifications and Professional Experience
`
`5.
`
`I am a Professor in the School of Interactive Computing (IC) of the
`
`College of Computing (CoC), and Adjunct Professor in the School of Electrical
`
`and Computer Engineering, Georgia Institute of Technology (GA Tech), in
`
`Atlanta, Georgia, USA. Currently, I am also serving as the Director/Assistant Dean
`
`of Off Campus Initiatives for the College of Computing at Georgia Institute of
`
`Technology.
`
`6.
`
`I received my PhD. degree from the Massachusetts Institute of
`
`Technology (“MIT”) — Cambridge in 1995.
`
`7.
`
`I received my MS. degree from MIT — Cambridge in 1990.
`
`
`
`8.
`
`I received my BS. degree from Illinois Institute of Technology —
`
`Chicago in 1988.
`
`9.
`
`At Georgia Institute of Technology, I have taught graduate level
`
`courses in the field of computer vision and related technologies, including CS 7321
`
`Low-level Computer Vision, CS 7322 High-level Computer Vision CS
`
`4480/8803dfx Digital Video Special Effects (SB), CS 8803PHO Advanced
`
`Computational Photography.
`
`10.
`
`I have conducted extensive research in the areas of Computer Vision,
`
`Computer Graphics, Computational Perception, Robotics and Computer
`
`Animation, Machine Learning, and Social Computing, with potential impact on
`
`Video Analysis and Production (e.g., Computational Photography & Video, Image-
`
`based Modeling and Rendering, etc.) Human Computer Interaction, Artificial
`
`Intelligence and Computational Journalism research.
`
`11.
`
`I have authored a high number of publications in peer reviewed
`
`journals, books, and conferences in the field of computer vision and related
`
`technologies.
`
`12. A true and correct copy of my curriculum vitae (“CV”) is attached to
`
`this declaration. (See YRD—2009.)
`
`13.
`
`In my opinion, the level of ordinary skill in the art for the ”284 Patent
`
`4
`
`' I
`
`I
`
`
`
`is at least a Master of Science degree in electrical engineering, computer
`
`engineering, computer science, or physics combined with relevant graduate level
`
`experience in this technology or relevant industry experience.
`
`14.
`
`Based upon my experience and education, I consider myself to be a
`
`person of ordinary skill in the art. Unless otherwise stated, my testimony below
`
`refers to the knowledge of one of ordinary skill in the generation and display of
`
`stereoscopic panoramic images technology during the 1998-1999 time period,
`
`including the priority date of the ”284 Patent.
`
`Recent Testimony
`
`15.
`
`l have never provided testimony in a patent related legal proceeding.
`
`
`Relevant Legal Standards
`
`16.
`
`l have been asked to provide my opinions regarding whether the
`
`claims of the ”284 Patent are anticipated or would have been obvious to a person
`
`having ordinary skill in the art at the time of the alleged invention, in light of the
`
`cited references.
`
`17.
`
`It is my understanding that, to anticipate a claim under 35 U.S.C. §
`
`102, a reference must teach every element of the claim either expressly or
`
`inherently. It is my understanding that an element is inherent if it is necessarily
`
`present in the cited reference.
`
`It is my understanding that a claimed invention is
`
`5
`
`' I
`
`
`
`unpatentable under 35 U.S.C. § 103 if the differences between the invention and
`
`the prior art are such that the subject matter as a whole would have been obvious at
`
`the time the invention was made to a person having ordinary skill in the art to
`
`which the subject matter pertains. I also understand that the obviousness analysis
`
`takes into account factual inquiries including the level of ordinary skill in the art,
`
`the scope and content of the prior art, and the differences between the prior art and
`
`the claimed subject matter.
`
`18.
`
`It is my understanding that the Supreme Court has recognized several
`
`rationales for combining references or modifying a reference to show obviousness
`
`of claimed subject matter. (See e.g., KSR International Co. v. Teleflex, Inc, 550
`
`US. 398 (2007)). Some of these rationales include the following: combining prior
`
`art elements according to known methods to yield predictable results; simple
`
`substitution of one known element for another to obtain predictable results; use of
`
`a known technique to improve a similar device (method, or product) in the same
`
`way; applying a known technique to a known device (method, or product) ready
`
`for improvement to yield predictable results; choosing from a finite number of
`
`identified, predictable solutions, with a reasonable expectation of success; and
`
`some teaching, suggestion, or motivation in the prior art that would have led one of
`
`ordinary skill to modify the prior art reference or to combine prior art reference
`
`teachings to arrive at the claimed invention.
`6
`
`' I
`
`I
`
`
`
`
`Background of the ’284 Patent
`
`19.
`
`The ’284 Patent is a continuation in part and incorporates by
`
`reference, amongst other disclosures, the disclosure of US. Patent Application No.
`
`09/396,248, filed September 16, 1999, issued as the ’003 Patent. The ”284 Patent
`
`describes and claims generating and displaying mosaic images that provide a
`
`perception of depth to a person. As discussed below in greater detail, a perception
`
`of depth is the visual perception of differential distances among objects in a
`
`person’s line of sight. (See e.g., YRD-2003).
`
`In other words, one object in an
`
`image will be perceived by the viewer as being closer as compared to another
`
`object in the image.
`
`20.
`
`The ’284 Patent notes that the arrangement for generating mosaic
`
`images is similar to the arrangements described in the ’003 Patent. (Sony-1001 at
`
`3:26-60; see also 9:16-19.) In that regard, the ‘003 Patent, in connection with Figs.
`
`2 and 3, describes utilizing a rotating camera system to generate obtain a series of
`
`left (26L) and right (26R) strips which are subsequently mosaiced together to
`
`generate the left and right mosaic image pairs that provide a perception of depth to
`
`a person.
`
`
`
`CAMERA
`13
`
`15’
`
`«~11
`
`H10
`
`SCENEE?
`
`27R 27::
`l
`‘
`
`a
`
`‘
`
`‘
`
`23"—
`
`r
`f
`
`I
`,
`wee;
`|
`J
`|
`r
`
`i
`
`.
`
`
`
`
`MOTOR
`
`CONTROL
`
`‘I?
`
`
`
`PANORAMA
`
`
`M 0 ERIC IMAG E
`
`
`GEN ERATOR
`
`12
`
`
`
`21.
`
`To create mosaic images that provide a perception of depth to a
`
`person, the ”003 Patent notes that the series of image strips must be generated and
`
`mosaiced from the perspective of human eyes:
`
`It will be apparent from FIG. 1A that each the succession of images
`
`as seen by the observer ’5 two eyes as he or she rotates, can be
`
`separated into separate sets of images, with one set of images being
`
`associated with each eye...
`
`...to facilitate the viewing of a stereoscopic panoramic image of the
`
`scene by a viewer, the images as would be received by each of the
`
`observer ’5 eyes can be separately recorded and viewed by, or
`
`otherwise displayed to, the respective eyes of the viewer.”
`
`(SONY-1002 at 3:8-31; see also 2:55-59, emphasis added).
`
`8
`
`' I
`
`
`
`It will be appreciated that the lefi‘ and right panoramic images 311.
`
`and 3 IR conform to what an observer would see through his or her
`
`left and right eyes, respectively, as they revolve through the left and
`
`right viewing circles 5L and SR described above in connection with
`
`FIG. 13.
`
`(SONY-1002 at 6:42-47, emphasis added).
`
`A person can see stereoscopically because his or her eyes are
`
`displaced horizontally (when standing) which, will provide a
`
`perception of depth when viewing a scene, which would not be
`
`present otherwise.
`
`Stereoscopic images comprise two images
`
`recorded of a scene from slightly displaced positions, which when
`
`viewed simultaneously by the
`
`respective
`
`eyes, provides
`
`a
`
`perception of depth.
`
`(SONY-1002 at 6:42-47, emphasis added).
`
`P'P
`
`rP
`
`Yip
`
`'l
`
`l'
`
`/,"
`3L_.:'
`5
`
`'
`
`21.x.
`
`‘
`‘.‘
`'r3R
`
`‘
`
`2R
`
`5
`
`F! G. IA
`
`3b;
`
`.'
`
`.‘I
`,5
`
`.
`
`
`
`3mm»,
`
`/.
`'_,‘3|_(2_}
`" data)
`
`
`
`leafy“
`43%)
`
`{5}
`
`lib—SR
`
`‘(3ij '5.
`.
`l‘
`
`3mm..--
`“
`
`
`."-3R{2)
`
`.‘
`
`"73"”) we)?” _‘
`.
`1'
`""3R13)
`"laL
`I“
`l'.
`“-.-3R{4]
`739(5) \
`.-
`
`Fl G. If?
`
`22.
`
`The mosaic image pairs, which were specifically created to conform
`
`9
`
`' I
`
`
`
`to what an observer would see through his or her eyes, can then be displayed to or
`
`viewed simultaneously by the left and right eyes of a person to provide a
`
`perception of depth.
`
`In one example, the ”003 Patent discloses using polarized
`
`glasses “to view the panoramic image stereoscopically.” (See SONY-1002, 11:60-
`
`65).
`
`An Exemplary Stereoscopic Image Pair that Provides a Perception of Depth
`
`23.
`
`To assist the Board in understanding the mechanism of how a
`
`stereoscopic image pair provides a perception of depth to a person, I provide the
`
`following discussion, with reference to exhibit YRD—2007, which is reproduced
`
`below for convenience.
`
`24.
`
`I note that Exhibit YRD-2007 was obtained from the following public
`
`URL: http:l/upload.wikimedia.org/wikipedia/commons/d/d7/Art_lnstitute_of
`
`Chicago_Lion_Statue_%2Sanagly ph_stereo%29.j pg.
`
`10
`
`
`
`
`
`25.
`
`It is my opinion that exhibit YRD-2007 provides an exemplary
`
`representation of one way to display stereoscopically an image pair by way of an
`
`anaglyph (i.e., red/cyan stereo image) which can be viewed with red/cyan glasses
`
`by a person. When viewed with a pair of red/cyan glasses, the images of exhibit
`
`YRD-2007 are a stereoscopic image pair that can be stereoscopically fused by a
`
`11
`
`
`
`person so as to provide a perception of depth where a lion is perceived in the
`
`foreground and a building in perceived the background.
`
`26.
`
`A person of ordinary skill in the art understands the term
`
`“stereoscopically fused” or “stereo fusion” refer to a process whereby
`
`corresponding points in two scenes are brought together horizontally by the human
`
`visual system so as to create a sense of depth.
`
`In the above example, when viewed
`
`with red/cyan glasses, the points of the red image and the cyan image are brought
`
`together in a person’s mind.
`
`I note that if the red/cyan images are not generated
`
`from the perspective of human eyes and not properly aligned horizontally, then
`
`when a person viewed the images with the red/cyan glasses, double (overlap)
`
`images would appear and make stereoscopic fusion impossible. (See also
`
`paragraphs 36- 37 below, which discusses how the Kawakita observers perceived
`
`overlapping images and therefore could not view the images stereoscopically).
`
`In
`
`my opinion, if a person perceives double images (overlapping images) while
`
`viewing an anaglyph stereo image and wearing appropriate red/cyan glasses (or
`
`some other appropriate viewing mechanism) then stereo fusion is not possible, and
`
`consequently, the person would not perceive depth solely from the stimulus
`
`provided by that image pair.
`
`27.
`
`I further note that in addition to requiring that the images be properly
`
`12
`
`
`
`aligned to result in stereo fusion (as discussed above at paragraph 26), to provide a
`
`perception of depth to a person there must be differential distances between objects
`
`or surfaces or other elements of the scene. To clarify, a perception of depth
`
`requires the perception of differential distances among objects in the scene.
`
`(See
`
`also, YRD—2003.) For example, with reference to the image of Exhibit YRD-2007,
`
`I note that there is a perception of depth, when viewing the image, because the lion
`
`is in the foreground and the building is in the background. However, if these
`
`objects in the scene were at the same distance from the camera, a person viewing
`
`the images would not have a sense of depth because there would be no depth
`
`differences.
`
`I note that if all objects in the scene are roughly at the same distance
`
`in a scene, a person would not be able to distinguish if the objects are near or far
`
`with respect to each other.
`
`28.
`
`In summary, for a stereoscopic image pair to provide a perception of
`
`depth, the image pair must (1) be capable of being stereoscopically fused 0.8-,
`
`specifically generated from the perspective of human eyes and properly aligned
`
`horizontally so as to not result in double images or overlap) when viewed by a
`
`person and (2) the objects in the scene must be at differential distances from the
`
`camera.
`
`13
`
`
`
`Kawakita
`
`29.
`
`The Kawakita reference (“Kawakitafi’ exhibit SON Y-1004), is
`
`directed to a technique for generating images for stereoscopic viewing. (SON Y-
`
`1004 at 13-14). Kawakita discloses a tripod that is rotated manually about an axis
`
`to capture vertical strips of images. (SONY-1004 at 14). The setup is illustrated
`
`below in Kawakita’s Fig. 1.
`
`//,,_.___~_x
`
`I.
`
`C
`
`i
`
`\
`\
`
`\
`
`\
`"I.
`|
`if
`
`/
`
`\
`
`x
`
`Fig. 1: Image Capture Technique Using a Video Camera
`
`30. Optical flow computation between images is used to determine the
`
`size of vertical strips, and these strips are mosaiced together to form a panorama.
`
`(SON Y-1004 at 14—15). Optical flow is different for objects closer to the camera
`
`then for objects farther from the camera. (SONY-1004 at 14-15). Specifically, as
`
`the camera rotates around the vertical axis, objects closer to the camera appear to
`
`move faster (in this case horizontally) than objects further away. (SONY-1004 at
`
`14—15).
`
`14
`
`
`
`31.
`
`The following example is provided to illustrate how optical flow
`
`would generate strips with different widths:
`
`Far background: strip = 1 pixel
`
`
`
`
`
`
`Close background: strip = 10 pixels
`
`
`
`-
`
`,.
`I i
`
`.
`
`\_
`
`.-'
`
`.-'
`.-'
`‘-.
`Amovin camera
`I
`
`Left
`
`strip
`
`Right
`
`strip
`
` Left Eye Mosaic
`
`
`Right Eye Mosaic
`
`mm
`
`32.
`
`In the above example, consider an arm including a camera attached to
`
`a tripod. Consider this camera arm to be very long (so as to define a radius of
`
`moving arm that is large enough to approximate a straight line camera path as
`
`opposed to a curved arc path). The arm including the camera is moved across a
`
`scene having objects at different distances. As illustrated in the above example,
`
`when moving the camera across the scene (to the right), the far background as
`
`viewed by the right eye strip would have a single (1) pixel width while the close
`15
`
`
`
`background as viewed by the left eye strip would have a ten (10) pixel width. This
`
`difference in pixel width for the left and right eye mosaics would create mosaics
`
`with different dimensions and would necessitate adjusting the images. A real
`
`world example would be looking outside the side window in a car as the car moves
`
`forward and observing that close objects (e. g, road signs, trees, etc...) appear to
`
`move faster than far away objects (e. g, mountains, buildings, etc. . .).
`
`33.
`
`If the distance from the camera to the objects varies greatly, as
`
`illustrated in the above example, Kawakita’s method of using optical flow to
`
`determine the strip width would result in inconsistent stereo pair images that have
`
`to be adjusted (as noted by Kawakita parallax adjustment). Said differently, in
`
`Kawakita, the distance between the objects and the camera results in strips with
`
`different widths. Conversely, if the distance of the objects does not vary greatly,
`
`then the objects would be roughly at the same distance and the strip widths would
`
`be roughly the same. However, in instances where the distance of the objects to
`
`the camera is roughly the same, no perception of depth would be possible.
`
`I note
`
`that the distance is not an absolute number because it is based on accuracy of the
`
`imaging apparatus.
`
`34. Kawakita explains that in one scenario, stereoscopic viewing is
`
`possible if the image was captured from a sufficient distance and if the distance
`
`16
`
`
`
`from the camera to the objects does not vary greatly. Kawakita states:
`
`When the left and right panoramic images obtained using the
`
`foregoing procedure are viewed binocular stereoscopically, a
`
`stereoscopic view is possible that
`
`faithfully reproduces
`
`the
`
`positional relationships. However, if the camera was placed at a
`
`comparatively close distance, or if the distance from the camera to
`
`the objects varies greatly, the positions representing the left and the
`
`right panoramic images must be adjusted.
`
`(SONY-1004 at 16-17).
`
`35.
`
`Said differently, in this scenario, stereoscopic viewing is possible if
`
`(1) the images are captured from a sufficient distance and (2) the distance from the
`
`camera to the objects is roughly the same. To clarify, and as I noted above at
`
`paragraph 33, Kawakita’s optical flow image strip determination results in images
`
`having roughly the same width (and therefore not requiring parallax adjustment)
`
`only if the objects are roughly at the same distance from the camera.
`
`It is my
`
`opinion that although this scenario of Kawakita would result in images that can be
`
`stereoscopically fused, a person would not perceive depth because the objects in
`
`the scene would be perceived as being at the same distance as they are not at
`
`differential distances from each other.
`
`36. Kawakita further explains that in embodiments where the distance
`
`from the camera to the objects varies greatly, the objects appear to overlap (double
`
`l7
`
`
`
`images) or have some other fault making faithful stereoscopic viewing impossible,
`
`unless adjustment is performed to the images:
`
`When the left and right panoramic images obtained using the
`
`foregoing procedure are viewed binocular stereoscopically, a
`
`stereoscopic view is possible that
`
`faithfully reproduces
`
`the
`
`positional relationships. However, if the camera was placed at a
`
`comparativer close distance, or if the distance from the camera to
`
`the objects varies greatly, the positions representing the left and the
`
`right panoramic images must be adjusted.
`
`...However, if the panoramic image is viewed stereoscopically as-
`
`is, since 6 is the parallax angle calculated from parallel light beams,
`
`using a captured image in which objects were placed at a close
`
`distance, object A and object B are displayed as infinitely distant.
`
`Therefore, objects appear to overlap or some other fault, making
`
`faithful stereoscopic viewing impossible.
`
`(SONY-1004 at 16-17, emphasis added).
`
`37.
`
`In my opinion, when Kawakita states that “objects appear to overlap
`
`making faithful stereoscopic viewing impossible,” Kawakita is not attempting
`
`to distinguish between good and bad stereoscopic viewing. Rather, Kawakita is
`
`stating that it is impossible for the human mind to stereoscopically fuse the images
`
`together because of the overlap, which may also be referred to as double images
`
`(similar to seeing objects when looking cross-eyed). As I noted above, if a human
`
`18
`
`
`
`perceives overlap (or there are double images) when looking at a stereo image pair
`
`as the sole stimulus, the human mind cannot stereoscopically fuse corresponding
`
`points of the images together and consequently there is no perception of depth.
`
`Here, Kawakita’s unadjusted images have “objects [that] appear to overlap” and
`
`consequently, the images cannot be stereoscopically fused without additional
`
`parallax adjustment.
`
`38.
`
`Regarding the parallax adjustment process, Kawakita discloses a
`
`manual adjustment for a specific portion of the image while observers are actually
`
`looking at the images:
`
`A field test was conducted applying these techniques to panoramic
`
`images of an elevator hallway in which the distance to objects
`
`varies greatly. First, while actually looking at
`
`the panoramic
`
`images, alignment was performed in several sight line directions so
`
`faithful stereoscopic viewing would be possible, and the depth
`
`parallax angle in each sight line direction was recorded.
`
`(SONY-1004, at 18, emphasis added).
`
`39.
`
`In my opinion, Kawakita teaches displaying and adjusting only a
`
`portion of the image that is in the sight line of an observer and not the whole
`
`image. (SONY-1004, at 18).
`
`I further note that because only part of the image
`
`(not the whole panoramic imager) is adjusted, only that part of the image can
`
`19
`
`
`
`provide a perception of depth to a person. Further, if no subsequent adjustment is
`
`made, when a viewer shifts to a different line of site, stereoscopic viewing of that
`
`portion of the scene is not possible because of overlap (or double images), and
`
`consequently there is no perception of depth.
`
`40.
`
`A common day illustration is displaying a standard television (“TV”)
`
`image on a wide screen TV, with black bars on the left and right side. Such an
`
`example is illustrated below:
`
`
`
`41.
`
`It is readily apparent that the above image, although displayed on a
`
`wide screen, is not a wide screen image of the scene. Likewise, Kawakita’s
`
`partially parallax adjusted stereoscopic images do not provide a perception of
`
`depth of the scene— rather they provide a perception of depth of a partial scene.
`
`42.
`
`Therefore, in my opinion, although Kawakita’s partially adjusted
`
`images provide a perception of depth of a partial scene, the partially adjusted
`
`20
`
`
`
`images do not disclose the claimed mosaic images that provide a perception of
`
`depth of the scene.
`
`
`Asahi
`
`43.
`
`The Asahi reference (“‘Asahi,” exhibit SONY-1010), is directed to
`
`calculating the heights of objects to make contour maps of a terrain. (SONY-1010,
`
`at 11 0003-0004).
`
`In Asahi, a helicopter moves through the air and captures video
`
`images, along with GPS, gyro, and other metadata known for the camera
`
`location/orientation. (SONY-1010, at 11 0014-0015). As illustrated in Fig. 19,
`
`reproduced below, single pixel width image lines are obtained from the images a
`
`base length (B) away from each other and mosaiced together to generate forward
`
`and rearward view images.
`
`[FIGAQ]
`
`i
`
`B ghase length!
`
`l
`I c1
`
`I
`
`l
`
`imaging plane
`“focal length)
`Iensfocus
`
`H (height)
`
`forward view image
`
`rearward view image
`
`21
`
`
`
`44.
`
`The forward and rearward mosaic images are subsequently adjusted to
`
`remove vertical parallax relative to each other by rotation, as shown below in Fig.
`
`15. (SON Y-lOlO, at i 0050—0052).
`
`[FIG.15]
`
`imaging diramion
`
`imaging direction
`
`i I
`
`III III
`
`
`
`vertical parallax removal image
`
`matching image
`
`45. After the forward and rearward view images have been adjusted, they
`
`are utilized in calculating height (h). The formula for calculating height is
`
`h=dxH/B, where h is the height (meters) to be determined, d is the parallax
`
`difference (meters), B is the base length (meters), and H is the imaging altitude
`
`(meters).
`
`(SONY-1010 at 1] 0064).
`
`46.
`
`A terrain is likely to have various hills and valleys, which can vary in
`
`height in the range of tens or hundreds of meters, and a helicopter needs to fly over
`
`these varying terrains at a safe altitude. Therefore, the base distance B between the
`
`forward and rearward images (which is related to the flying altitude H, as defined
`
`by the equation h=dxH/B) should be very large, likely in the tens or hundreds of
`
`meters.
`
`22
`
`
`
`47. Notably, Asahi is only concerned with calculating height and does not
`
`disclose generating its images in accordance with the separation and from the
`
`perspective of human eyes to thereby provide a perception of depth — this would
`
`require generating the images for human vision and alignment between the images
`
`in the horizontal direction (not vertical).
`
`1 note that for Asahi to calculate the
`
`height (h) of objects and to create the Digital Elevation Model (DEM) of the scene,
`
`it is not necessary to generate images capable of providing a perception of depth to
`
`a person because computer algorithms can calculate height by searching between
`
`images with a much larger range of horizontal disparities than an individual
`
`human.
`
`48.
`
`In my opinion, although Asahi uses the term “stereoscopic viewing,”
`
`this term does not refer to displaying images to a person to thereby provide a
`
`perception of depth.
`
`It is my opinion that the term "stereoscopic image" is a broad
`
`term, which generally refers to a pair of images that view a scene from two
`
`different viewpoints. The term “stereoscopic image” would have different
`
`meanings depending on the context that the term is used. For example, when
`
`speaking with a colleague that is working on images generated to be suitable for
`
`human viewing, I would understand the term as referring to human perception of
`
`depth and when speaking with a colleague that is performing depth calculation of
`
`objects, I would understand that the term “stereoscopic image” solely refers to an
`23
`
`
`
`image for computer vision which is used to calculate the depth of the objects in the
`
`scene.
`
`49.
`
`In the context of Asahi, a person of ordinary skill would understand
`
`the term “stereoscopic viewing” solely to refer to an image for computer vision
`
`which is used to calculate the height of points in the scene.
`
`I also note that other
`
`terms used by Asahi (126., “stereo images,
`
`stereoscopic images,” and “stereo
`
`99 8“.
`
`matching,”) likewise in the context of Asahi would be understood to solely refer to
`
`computer vision method used for calculating the height of points in the scene. For
`
`example, regarding claim 11, it is my opinion that it only discloses calculating
`
`height of objects in a scene and when the claim uses the terms “stereoscopic
`
`viewing” it is solely referring to a computer viewing the images to perform the
`
`height calculation. As another example, regarding claim 24, it is my opinion that it
`
`only discloses generating two images, which can be used for height calculation and
`
`does not disclose displaying these images to provide a perception of depth. Thus,
`
`it is my opinion that Asahi does not disclose a display that receives mosaic images
`
`and displays the images to a person to thereby provide a perception of depth of the
`
`scene.
`
`50.
`
`In my opinion Asahi’s images are not mosaic images that are
`
`generated so as to provide a perception of depth to a person. Rather, Asahi only
`
`24
`
`
`
`generates it images to perform stereo matching computations (See SONY-1010 at 1]
`
`0060) and Asahi never discloses that its images are provided for human viewing to
`
`provide a perception of depth (see paragraph 49). As I have already discussed
`
`above, to construct images suitable for human viewing, the images must be
`
`specifically generated in accordance with the separation and from the perspective
`
`of human eyes.
`
`It is my opinion that Asahi never teaches that its images are
`
`created for human consumption.
`
`51. Moreover, I note that Asahi fails to explain how any modification or
`
`adjustment would be performed to its continuous images (making them suitable for
`
`human viewing), because it is not necessary for Asahi in order to perform height
`
`calculation to construct images capable of being displayed to a person so as to
`
`provide a perception of depth (see paragraph 47). Indeed, Asahi only discusses
`
`vertical parallax adjustment and fails to disclose horizontal adjustment, which
`
`would be necessary for human viewing.
`
`In my opinion, creating images suitable
`
`for human viewing so as to provide a perception of depth is beyond the scope of
`
`Asahi, which is specifically directed to creating images suitable for a computer to
`
`calculate height to create a DEM.
`
`52. Additionally, it is my opinion that the images of Asahi have defects,
`
`which are indicative of images that cannot be viewed by a person to provide a
`
`25
`
`
`
`perception of depth. For example, Asahi at Fig. 11, reproduced below, illustrates
`
`an image representing the scene as a letter “F” having local defects (e_g_, waves)
`
`resulting from flight turbulence/deviation, imaging limitations, and instrumentation
`
`inaccuracy, etc....
`
`I note that the forward view image would have different defects
`
`than the rearward view image because they are captured at different points in time
`
`(See SONY-1010 at Fig. 19, where forward view images are captured a base (B)
`
`away from rearward view images). The resulting local defects are not rectified by
`
`the vertical parallax removal adjustment, which only rotates the image as shown
`
`below in Fig. 15.
`
`I note that Asahi’s computer can still determine the height of a
`
`point in the scene (for example the point of the upper left comer of the “F”),
`
`notwithstanding the defects.
`
`[F1611]
`
`(—/
`
`,._-/
`
`“‘9 #1
`
`0(1.Y1.21.m‘.t‘.lt1l
`
`line
`
`I, 2
`
`{Xe-szzallflz..2. K2]
`
`fine 33
`
`{X3.Y3.Za “343"3)
`
`:
`
`Ime
`
`g j
`
`“I'Yl .zi.u1l.hu(|l
`
`.
`.
`.
`COI‘IlInuOuS “1053": Image
`
`nne #n
`
`finannmnamxn)
`
`[P1615]
`
`magma direction
`
`Imagng CIYECIIOI'I
`
`
`
`uerlicalparallar removal mage
`
`matching Image
`
`53.
`
`In my opinion, because the noted image defects in the rearward and
`
`forward view images would be different, the defects would prevent stereo fusion
`
`26
`
`
`
`and consequently the images would fail to provide a perception of depth to a
`
`human.
`
`54.
`
`In summary, it is my opinion that Asahi’s use of the terms “stereo
`
`93 CI”.
`
`39 (.6
`
`images,
`
`stereoscopic images,
`
`stereo matching,” and “stereoscopic viewing,”
`
`solely referrer to a computer viewing the images and calculating a height of an
`
`object and do not refer to displaying the images to a person so as to provide a
`
`perception of depth. Further, it is my opinion that because the forward and
`
`rearward images of Asahi are not generated in accordance with the separation and
`
`from the perspective of human eyes and because the images have defects, Asahi’s
`
`forward and rearward images would not provide a perception of depth to a person.
`
`55.
`
`Therefore, in my opinion, Asahi does not disclose generating a
`
`plurality of mosaics that provide a sense of depth of the scene and does not
`
`disclose a display that receives a plurality of the mosaics and displays them so as to
`
`provide a sense of depth of the scene, as claimed.
`
`27
`
`
`
`Declaration
`
`56.
`
`I declare that all statements made within this declaration are based on
`
`my own knowledge are true and that all statements made on information and belief
`
`are believed to be true, and further, that these statements were made with the
`
`knowledge that willful false statements and the like so made are punishable by fine
`
`or imprisonment, or both, under Section 1001 of Title 18 of the United States
`
`Code.
`
`
`Executed: December 18, 2013
`
`.-"
`
`I'x
`
`3'?"
`
`.y
`
`1'I
`.
`I '
`u
`,- "
`4,. fl
`, ,.
`
`
`/
`
`By:
`
`firfan Essa PhD.
`
`28
`
`
Accessing this document will incur an additional charge of $.
After purchase, you can access this document again without charge.
Accept $ Charge
Still Working On It
This document is taking longer than usual to download. This can happen if we need to contact the court directly to obtain the document and their servers are running slowly.
Give it another minute or two to complete, and then try the refresh button.
A few More Minutes ... Still Working
It can take up to 5 minutes for us to download a document if the court servers are running slowly.
Thank you for your continued patience.
This document could not be displayed.
We could not find this document within its docket. Please go back to the docket page and check the link. If that does not work, go back to the docket and refresh it to pull the newest information.
Your account does not support viewing this document.
You need a Paid Account to view this document. Click here to change your account type.
Your account does not support viewing this document.
Set your membership
status to view this document.
With a Docket Alarm membership, you'll
get a whole lot more, including:
- Up-to-date information for this case.
- Email alerts whenever there is an update.
- Full text search for other cases.
- Get email alerts whenever a new case matches your search.
One Moment Please
The filing “” is large (MB) and is being downloaded.
Please refresh this page in a few minutes to see if the filing has been downloaded. The filing will also be emailed to you when the download completes.
Your document is on its way!
If you do not receive the document in five minutes, contact support at support@docketalarm.com.
Sealed Document
We are unable to display this document, it may be under a court ordered seal.
If you have proper credentials to access the file, you may proceed directly to the court's system using your government issued username and password.
Access Government Site
