Case 1:12-cv-00398-GMS Document 27-1 Filed 02/21/13 Page 1 of 25 PageID #: 320
`Case 1:12—cv—00398—GMS Document 27-1 Filed 02/21/13 Page 1 of 25 Page|D #: 320
`
`EXHIBIT A
`
`EXHIBIT A
`
`
`
`
`
`
`

`
`Case 1:12-cv-00398-GMS Document 27-1 Filed 02/21/13 Page 2 of 25 PageID #: 321
`
`UNITED STATES DEPARTMENT OF COMMERCE
`United States Patent and Trademark Office
`
`February 14, 2012
`
`THIS IS TO CERTIFY THAT ANNEXED HERETO IS A TRUE COPY FROM
`THE RECORDS OF THIS OFFICE OF:
`
`U.S. PATENT: 6,665,003
`ISSUE DATE: December 16,2003
`
`By Authority of the
`Under Secretary of Commerce for Intellectual Property
`and Director of the United States Patent and Trademark Office
`
`T. LAWRENCE
`Certifying Officer
`
`HET00000001
`
`

`
`Case 1:12-cv-00398-GMS Document 27-1 Filed 02/21/13 Page 3 of 25 PageID #: 322
`
`c12) United States Patent
`Peleg et al.
`
`111111
`
`1111111111111111111111111111111111111111111111111111111111111
`US006665003Bl
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 6,665,003 Bl
`Dec. 16, 2003
`
`(54) SYSTEM AND METHOD FOR GENERATING
`AND DISPlAYING PANORAMIC IMAGES
`AND MOVIES
`
`(75)
`
`Inventors: Shmuel Peleg, Mevaseret Zion (IL);
`Moshe Benezra, Jerusalem (IL);
`RobertS. Rosenschein, Jerusalem (IL)
`
`(73) Assignee: Issum Research Development
`Company of the Hebrew University of
`Jerusalem, Jerusalem (IL)
`
`( *) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days. ·
`
`(21) Appl. No.: 09/396,248
`
`(22) Filed:
`
`Sep. 16, 1999
`
`(60)
`
`(51)
`(52)
`(58)
`
`(56)
`
`Related U.S. Application Data
`Provisional application No. 60/100,721, filed on Sep. 17,
`1998, provisional application No. 60/102,720, filed on Sep.
`29, 1998, provisional application No. 60/113,962, filed on
`Dec. 28, 1998, provisional application No. 60/116,138, filed
`on Jan. 12, 1999, provisional application No. 60/116,139,
`filed on Jan. 12, 1999, provisional application No. 60/123,
`080, filed on Mar. 4, 1999, and provisional application No.
`60/129,987, filed on Apr. 18, 1999.
`Int. Cl.7
`•.•••. .......... ••.•..••••..•••••...••.• ....... ••. H04N 7/00
`U.S. CI • ............... .......................................... 348/36
`Field of Search ...................... ........ 348/42, 47, 51,
`348/53, 64, 39; 382/154, 284, 285, 294;
`349/15, 62, 74, 194
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`• 4/1996 Mayhew ............. ......... 348/47
`5,510,831 A
`7/1997 Burt et al. .................. 382/284
`5,649,032 A
`5,686,975 A * 11/1997 Lipton ................... ...... 349/15
`5,721,585 A
`2/1998 Keast ....................... ... 348/36
`5,768,447 A
`6/1998 Irani .......................... 382/305
`5,850,352 A
`• 12/1998 Moezzi et al. ...... ........ 364/514
`5,929,951 A
`• 7/1999 Sasakura et al. .............. 349/62
`5,963,664 A
`• 10/1999 Kumar et al. ............ ... 382/154
`6,064,353 A
`• 5/2000 Hoshi ............................ 345/7
`6,141,036 A * 10/2000 Katayama et al. ............ 348/47
`6,157,747 A * 12!2000 Szeliski et al. ............. 382/284
`6,252,707 B1 * 6/2001 Kleinberger et al. ........ 359/465
`............. 348/59
`6,275,254 B1 • 8/2001 Beeteson et al.
`6,278,480 B1 • 8/2001 Kurahashi et al. ............ 348/59
`6,301,447 B1
`10/2001 Jackson ...................... 396/329
`6,331,871 B1 * 12!2001 Taylor ........................ 348/159
`2002/0005921 A1 • 1/2002 Sasakura et al. ...... ........ 349/62
`* cited by examiner
`Primary Examiner-Vu Le
`Assistant Examiner-Behrooz Senfi
`(74) Attorney, Agent, or Firm--Fenster & Company
`
`(57)
`
`ABSTRACT
`
`Systems are disclosed for generating left and right pan(cid:173)
`oramic mosaic images for use in facilitating panoramic
`stereoscopic viewing of a scene, and for displaying stereo(cid:173)
`scopic panoramic images to a viewer. The left and right
`panoramic image generators generate the left and right
`panoramic mosaic images from a series of images recorded
`or otherwise generated corresponding to respective angular
`or other positions, each image having a respective left and
`right image portion, and mosaics portions of those images
`together to form the respective left and right panoramic
`images. The display system displays a stereoscopic pan(cid:173)
`oramic image to a viewer by displaying left and right
`panoramic images such that each is viewed by a respective
`one of the viewer's eyes.
`
`5,049,987 A * 9/1991 Hoppenstein ................ 348/48
`5,130,794 A * 7/1992 Ritchey ...... ................. 348/39
`
`100 Claims, 9 Drawing Sheets
`
`+--50
`
`Copy provided by USPTO from the PIRS Image Database on 02/13/2012
`
`HET00000002
`
`

`
`Case 1:12-cv-00398-GMS Document 27-1 Filed 02/21/13 Page 4 of 25 PageID #: 323
`
`U.S. Patent
`
`Dec. 16, 2003
`
`Sheet 1 of 9
`
`US 6,665,003 Bl
`
`HET00000003
`
`0....
`
`~
`--
`
`- - - - -
`
`------
`
`--
`~
`----
`(\')
`J-------
`\
`- '
`,'
`z
`'
`0:::
`('f)/
`\/
`'
`
`...--.,
`
`I
`
`a..
`
`~-------------,---­---
`
`...J
`(\')
`
`N'
`.........
`\
`...J
`\ (\')
`\(
`
`\
`
`\
`
`\
`
`\
`
`\
`
`\
`
`------
`
`\
`
`\ -\
`' -', ...J
`
`(0
`
`'C")
`\,
`'
`
`~------------------
`------------------
`
`Copy provided by USPTO from the PIRS Image Database on 02/13/2012
`
`

`
`U.S. Patent
`
`Dec. 16, 2003
`
`Sheet 2 of 9
`
`US 6,665,003 Bl
`
`HET00000004
`
`Case 1:12-cv-00398-GMS Document 27-1 Filed 02/21/13 Page 5 of 25 PageID #: 324
`
`~
`WM
`~ ......
`<(
`(.)
`
`' 1.0
`
`0 ..-
`J
`
`..(cid:173),....
`J
`
`Copy provided by USPTO from the PIRS Image Database on 02/13/2012
`
`

`
`0
`0
`"1:1
`'<
`"1:1
`0
`<
`i:i:
`CD
`Q.
`IT
`'<
`c:
`C/)
`"tl
`-1
`
`0 -... 0
`3
`s: CD
`"tl
`ii
`C/)
`3
`
`AI
`IQ
`CD
`c
`!.
`il
`
`Ill
`CD
`0
`:I
`0
`~
`~ 0 .....
`
`~
`
`Case 1:12-cv-00398-GMS Document 27-1 Filed 02/21/13 Page 6 of 25 PageID #: 325
`
`. . . .
`
`. . . .
`
`30R(1)
`
`30L(2)
`
`30R(N)
`
`LEFT PANORAMIC IMAGE 31L
`
`30L(N)
`
`32L(N)
`
`• • • •
`
`FIG. 4
`
`d •
`rJ1 •
`~ = ......
`~ = ......
`
`~
`~
`~
`~
`,..0\
`N
`Q
`~
`
`00. =(cid:173)~ ::a.
`0 ....,
`
`~
`
`~
`
`e
`
`rJ1
`~
`'0-,
`~
`01
`
`~ = w
`== ~
`
`HET00000005
`
`

`
`Case 1:12-cv-00398-GMS Document 27-1 Filed 02/21/13 Page 7 of 25 PageID #: 326
`
`U.S. Patent
`
`Dec. 16, 2003
`
`Sheet 4 of 9
`
`US 6,665,003 Bl
`
`!'(oo. CAMERA RIG 11 IS POSITIONED SUCH THAT THE
`AXIS OF MAST 141S PLACED AT THE CENTER OF
`ROTATION FOR THE STEREOSCOPIC PANORAMA, AND
`THE MOTOR CONTROL 17 INITIALLY ENABLES THE
`MOTOR 16 TO POSITION THE CAMERA 13 AT A STARTING
`POINT AT WHICH THE FIRST IMAGE 30(1) WILL BE
`RECORDED
`
`'
`
`,,
`
`'
`
`.....t
`
`101. MOTOR CONTROL 17 NOTIFIES THE PANORAMA
`MOSAIC IMAGE GENERATOR 12 THAT THE CAMERA 131S
`APPROPRIATELY POSITIONED
`
`~
`
`102. PANORAMA MOSAIC IMAGE GENERATOR 12
`CONTROLS THE CAMERA 13 TO RECORD AN IMAGE ON
`
`THE IMAGE RECORDING MEDIUM 21 •
`
`I..
`I""
`
`""
`
`I.Jo3. PANORAMA MOSAIC IMAGE GENERATOR 12
`DETERMINES WHETHER ALL OF THE IMAGES 30{1)
`THROUGH 30(N) TO BE USED IN GENERATING THE LEFT
`YES ,_
`AND RIGHT PANORAMIC IMAGES 31 LAND 31 R FOR THE
`STEREOSCOPIC PANORAMA IMAGE PAIR HAVE BEEN
`RECORDED
`
`'
`
`.....111
`
`Nt
`
`104. PANORAMA MOSAIC IMAGE GENERATOR 12
`ENABLES THE MOTOR CONTROL 17 TO, IN TURN,
`CONTROL THE MOTOR 16 TO ADVANCE A STEP
`
`105. PANORAMA MOSAIC IMAGE GENERATOR 12
`IDENTIFIES THE LEFT AND RIGHT IMAGE PORTIONS
`30L(n) AND 30R(n) FOR THE RESPECTIVE IMAGES 30(n)
`
`06. PANORAMA MOSAIC IMAGE GENERATOR 12
`MOSAICS SUCCESSIVE LEFT IMAGE PORTIONS 30L(n) TO
`FORM THE LEFT PANORAMIC IMAGE 31 LAND THE
`SUCCESSIVE RIGHT IMAGE PORTIONS 30R(n) TO FORM
`THE RIGHT PANORAMIC IMAGE 31 R
`
`FIG. 5
`
`Copy provided by USPTO from the PIRS Image Database on 02/13/2012
`
`HET00000006
`
`

`
`Case 1:12-cv-00398-GMS Document 27-1 Filed 02/21/13 Page 8 of 25 PageID #: 327
`
`I
`I
`
`I
`I
`
`~~·-/
`,;z:
`'Y
`. .
`
`' ,
`
`'
`
`'
`
`' ,
`
`/ / I
`
`I
`I
`I
`
`'
`
`/
`, '
`
`/
`
`'
`'
`'
`'
`'
`'
`'
`'
`'
`'
`'
`J::~/"1\ ~-1::~/~\
`
`I
`
`I
`
`/
`
`' '
`
`-------------
`' ' ' '
`' '
`' ' ' /
`
`/
`/ /
`l ~
`
`52(3)
`
`~50
`
`'
`' ' '
`
`52(N)
`
`' '
`' ' '
`',,, 51 (7)
`' ' '
`
`0
`
`0"
`'<
`
`0 ,
`'< , .. 0 <
`ii:
`" 1:1.
`c: en ,
`d
`~
`3
`:r
`" ,
`~
`3
`~
`" c
`Ill
`fli
`0"
`Ill ::
`
`0
`:I
`0
`~ ......
`w
`i\3
`0 ......
`
`1\)
`
`' ' ' ' '
`
`FIG. 6
`
`PANORAMA
`MOSAIC I MAGE
`GENERATOR
`12
`
`00
`•
`
`c •
`~ =
`~ = .......
`
`.......
`
`~
`~
`~
`~
`~0\
`N c
`~
`
`Ill =(cid:173)~
`~ .....
`tl1 = ....
`
`'.c
`
`~
`1.11
`="
`~
`
`~ = = ~
`
`== ~
`
`HET00000007
`
`

`
`0
`0
`"C
`'<
`"C ...
`0 < a:
`(1) c.
`C"
`'<
`c:
`(/)
`
`"C d -0
`
`3
`;.
`
`(1)
`"C
`
`~
`3
`
`Ill
`IQ
`(1)
`
`~ Ill
`
`C"
`Ill
`Ill
`(1)
`0
`:I
`0
`~ .....
`~ .....
`
`~
`
`Case 1:12-cv-00398-GMS Document 27-1 Filed 02/21/13 Page 9 of 25 PageID #: 328
`
`62~ 1 COli.)'\"''""'
`
`7
`
`'
`
`i
`
`'
`
`(\
`
`65(2)(L)
`
`FIG. 7A
`
`1 cA~~RA P.+---rBL __ ~;------ _
`
`FIG. 7
`
`80 _.71
`
`FIG. 8
`
`-----
`
`I
`I
`I
`I
`I
`I
`
`I L84
`
`I
`
`I
`
`- - - - -
`
`· ls3
`
`~ •
`00
`•
`~
`~ ......
`~ = ......
`
`~
`(I)
`~
`~
`~0\
`N c
`~
`
`rJ:l =(cid:173)(!)
`....
`
`(I)
`
`0\
`0 ......,
`1,0
`
`cj
`rJ).
`.9'
`c::l'\
`~
`~
`~ w
`t:=
`~
`
`HET00000008
`
`

`
`U.S. Patent
`
`Dec. 16,2003
`
`Sheet 7 of 9
`
`US 6,665,003 Bl
`
`HET00000009
`
`Case 1:12-cv-00398-GMS Document 27-1 Filed 02/21/13 Page 10 of 25 PageID #: 329
`
`0
`0
`......
`
`\
`
`0
`N
`......
`
`\
`
`'-'
`
`"'-i
`"'-i
`
`~
`~
`
`N
`>:_JN
`0..-
`:)o::w
`a_i-....J
`cnz==>
`-oo
`Ouo
`~
`
`Copy provided by USPTO from the PIRS Image Database on 02/13/2012
`
`5
`
`D..
`U)
`0
`1-
`I
`0
`~
`
`0:::
`C')
`N ......
`
`

`
`U.S. Patent
`
`Dec. 16, 2003
`
`Sheet 8 of 9
`
`US 6,665,003 Bl
`
`HET00000010
`
`Case 1:12-cv-00398-GMS Document 27-1 Filed 02/21/13 Page 11 of 25 PageID #: 330
`
`(.)
`Cf)
`
`w
`z w
`
`(.)
`
`1-:::l!W
`I(2<!>
`<!>a~
`c:::z:::l!
`~
`
`l-(2w
`(!)
`LL
`wa<(
`...Jz~
`
`(.)
`::2:
`
`~
`
`~
`~
`~
`~
`~
`
`Copy provided by USPTO from the PIRS Image Database on 02/13/2012
`
`

`
`U.S. Patent
`
`Dec. 16, 2003
`
`Sheet 9 of 9
`
`US 6,665,003 Bl
`
`HET00000011
`
`Case 1:12-cv-00398-GMS Document 27-1 Filed 02/21/13 Page 12 of 25 PageID #: 331
`
`:::r
`~'0
`
`C\1
`
`("')
`
`N'
`N'
`....
`,l,
`
`"¢
`
`~
`~
`~
`~
`~
`
`0:::
`("')
`"¢
`
`~
`
`0
`"¢
`,.-
`~
`
`,.-
`
`-JjVJ N'
`
`c;)
`
`c;)
`
`"¢
`
`~
`
`~
`("')
`"¢
`
`.... '411~ S"
`
`~
`
`....
`....
`
`"¢
`
`"¢ UlJ ~
`
`("')
`"¢
`
`,.-
`C'\1
`"¢
`
`-
`
`"¢
`"¢
`"!1
`
`- --- ·· --···-· ···
`
`I{)
`"¢
`
`~ ~
`
`~
`~
`~
`~
`
`··---- -
`
`Copy provided by USPTO from the PIRS Image Database on 02/13/2012
`
`cY
`rC:J~
`:::r
`
`z
`[E'
`
`("')
`
`("')
`"¢
`
`N'
`J,
`
`-z
`
`
`
`"¢
`
`\
`
`0
`"¢
`,.-
`
`

`
`Case 1:12-cv-00398-GMS Document 27-1 Filed 02/21/13 Page 13 of 25 PageID #: 332
`
`US 6,665,003 Bl
`
`1
`SYSTEM AND METHOD FOR GENERATING
`AND DISPlAYING PANORAMIC IMAGES
`AND MOVIES
`
`This application claims the benefit of Provisional appli(cid:173)
`cation Ser. No. 60/100,721 filed Sep. 17, 1998; Provisional
`application Ser. No. 60/102,720 filed Sep. 29, 1998; Provi(cid:173)
`sional application Ser. No. 60/113,962 filed Dec. 28, 1998;
`Provisional application Ser. No. 60/116,138 filed Jan. 12,
`1999; Provisional application Ser. No. 60/116,139 filed Jan.
`12, 1999; Provisional application Ser. No. 60/123,080 filed
`Mar. 4, 1999; Provisional application Ser. No. 60/129,987
`filed Apr. 18, 1999.
`
`FIELD OF THE INVENTION
`
`25
`
`The invention relates generally to the field of recording,
`generating and playing back or displaying images, and more
`particularly to the generation and displaying of panoramic
`images stereoscopically.
`Panoramic images are images of a scene having a wide
`field of view, up to a full 360°. Panoramic images may be
`recorded using a wide angled lens, a mirror, or the like,
`providing a wide field of view. Panoramic images having a
`wider field of view can be generated by, for example,
`recording a plurality of images around a particular point and,
`using conventional mosaicing techniques, generating a
`single mosaic image. Panoramic images may also be gen(cid:173)
`erated of simulated scenes using conventional computer
`graphics techniques.
`A problem arises in connection with viewing panoramic
`images stereoscopically. A person can see stereoscopically
`because his or her eyes are displaced horizontally (when
`standing) which, will provide a perception of depth when
`viewing a scene, which would not be present otherwise. 35
`Stereoscopic images comprise two images recorded of a
`scene recorded from slightly displaced positions, which,
`when viewed simultaneously by the respective eyes, pro(cid:173)
`vides a perception of depth. Although currently there are
`arrangements for generating and displaying stereoscopically 40
`non-panoramic images, currently there are no such arrange(cid:173)
`ments for generating and displaying stereoscopically pan-
`oramic images.
`
`2
`invention may be better understood by referring to the
`following description taken in conjunction with the accom(cid:173)
`panying drawings, in which:
`FIGS. 1A and lB are useful in understanding operations
`5 performed by arrangements for generating and displaying
`stereoscopic panorama images constructed in accordance
`with the invention;
`FIG. 2 schematically depicts a stereoscopic panorama
`recording and generating system constructed in accordance
`10 with one embodiment of the invention;
`FIG. 3 schematically depicts a camera used in the stereo(cid:173)
`scopic panorama recording and generating system depicted
`in FIG. 2;
`FIG. 4 depicts generation of left and right panorama
`15 images from images recorded by the camera depicted in
`FIG. lB, useful in understanding the operation of the
`panoramic image generator of the stereoscopic panorama
`recording and generating system depicted in FIG. 2;
`FIG. 5 depicts a flow chart describing operations per-
`20 formed by the stereoscopic panorama recording and gener(cid:173)
`ating system;
`FIG. 6 schematically depicts a stereoscopic panorama
`recording and generating system constructed in accordance
`a second embodiment of the invention;
`FIGS. 7 and 7A schematically depicts a stereoscopic
`panorama recording and generating system constructed in
`accordance with a third embodiment of the invention;
`FIG. 8 schematically depicts a stereoscopic panorama
`30 recording and generating system constructed in accordance
`with a fourth embodiment of the invention;
`FIG. 9 schematically depicts a stereoscopic panorama
`recording and generating system constructed in accordance
`with a fifth embodiment of the invention;
`FIG. 10 is useful in understanding generation of a stereo(cid:173)
`scopic panorama image pair using computer graphics tech(cid:173)
`niques;
`FIG. 11 schematically depicts a first arrangement for
`displaying a stereoscopic panorama image to a viewer; and
`FIGS. 12A and 12B together schematically depict a
`second arrangement for displaying a stereoscopic panorama
`image to a viewer.
`DETAILED DESCRIPTION OF AN
`ILLUSTRATIVE EMBODIMENT
`The invention provides systems and methods for gener(cid:173)
`ating stereoscopic panoramic images of a scene, and for
`displaying the images to a one or more viewers in a manner
`so that the viewer(s) can view the panoramic images ste-
`50 reoscopically. Before describing the inventive systems and
`methods, it would be helpful to first describe what a stereo(cid:173)
`scopic panoramic image is and generally how the various
`systems and methods described herein generate the stereo(cid:173)
`scopic panoramic images and facilitate their display. This
`55 will be done in connection with FIGS. lA and lB. With
`reference initially to FIG. 1A that FIG. schematically depicts
`an observer, and, particularly, eyes represented by dots 2L
`and 2R (generally identified by reference numeral "2L/R")
`standing vertically and observing a point P in a scene. The
`60 observer sees point P by means of rays of light reflected from
`the point and directed toward the eyes 2L and 2R along
`respective rays represented by dashed arrows 3L and 3R. It
`will be appreciated that, since the rays 3L and 3R are not
`parallel, the observer will be able to observe a depth in
`65 connection with the region of the scene at and near point P.
`The observer typically can see only a small portion of the
`360° panorama around himself or herself. To see more of the
`
`45
`
`SUMMARY OF THE INVENTION
`The invention provides a new and improved system and
`method of generating and displaying stereoscopic pan(cid:173)
`oramic images.
`In brief summary, in one aspect the invention provides a
`system for generating left and right panoramic mosaic
`images for use in facilitating panoramic stereoscopic view(cid:173)
`ing of a scene. The left and right panoramic image genera(cid:173)
`tors generate the left and right panoramic mosaic images
`from a series of images recorded or otherwise generated
`corresponding to respective angular or other positions, each
`image having a respective left and right image portion, and
`mosaics portions of those images together to form the
`respective left and right panoramic images.
`In another aspect, the invention provides a system for
`displaying a stereoscopic panoramic image to a viewer, by
`displaying left and right panoramic images such that each is
`viewed by a respective one of the viewer's eyes.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`This invention is pointed out with particularity in the
`appended claims. The above and further advantages of this
`
`Copy provided by USPTO from the PIRS Image Database on 02/13/2012
`
`HET00000012
`
`

`
`Case 1:12-cv-00398-GMS Document 27-1 Filed 02/21/13 Page 14 of 25 PageID #: 333
`
`US 6,665,003 Bl
`
`3
`panorama, the observer will rotate his or her head in, for
`example, the direction indicated by the arrow identified by
`reference numeral 4. Rotation of the head will allow the
`observer to view other points (not shown) in the scene, along
`rays (also not shown) that rotate with him or her. If the
`observer rotates around a full 360°, each eye will revolve
`around the same viewing circle 5.
`It will be apparent from FIG. 1A that each the succession
`of images as seen by the observer's two eyes as he or she
`rotates, can be separated into separate sets of images, with
`one set of images being associated with each eye. This will
`be described in connection with FIG. lB. FIG. lB. depicts
`the viewing circle 5 divided into separate viewing circles 5L
`and 5R (generally 5L!R) for the respective left and right
`eyes, with point P being shown in the same position as in
`FIG. lA, with respect to each viewing circle 5L/R, and the
`associated ray 3L(1) and 3R(1), which correspond to rays 3L
`and 3R depicted in FIG. lA. Each viewing circle 5L!R also
`depicts other rays, identified by reference numerals
`3L(2), . . . , 3L(N) (generally identified by reference numeral
`3L(n)) and 3R(2), ... , 3R(N) (generally identified by
`reference numeral 3R(n)) that represent images that the
`respective left and right eyes of the observer will receive of
`the various points in the scene as he or she rotates in the
`direction represented by arrows 4L and 4R.
`Further in connection with FIG. lB, to facilitate the
`viewing of a stereoscopic panoramic image of the scene by
`a viewer, the images as would be received by each of the
`observer's eyes can be separately recorded and viewed by, or
`otherwise displayed to, the respective eyes of the viewer.
`Thus, if, for example, images are recorded around a circle
`corresponding to viewing circle 5L at successive points, in
`successive direction depicted by rays 3L(1), ... 3L(N), and
`the images mosaiced together, and further images are
`recorded around a circle corresponding to viewing circle 5R
`at successive points, in successive direction depicted by rays
`3R(1), . . . 3R(N), and if those images are suitably aligned
`(such that the point of intersection of the rays 3L(n) and
`3R(n) are viewed in the same relative location) and dis(cid:173)
`played to respective eyes of a viewer, the viewer can see a 40
`stereoscopic panoramic image of the scene.
`In a similar manner, stereoscopic panoramic images can
`be generated using computer graphics techniques. However,
`instead of the regular perspective projection used in con(cid:173)
`ventional image rendering, the panoramic image for the left
`eye will be rendered using rays tangent to a circle such as
`viewing circle 5L, and the panoramic image for the right eye
`will be rendered using rays tangent to a circle such as
`viewing circle SR.
`The invention provides various arrangements for facili(cid:173)
`tating the recording of images from which left and right
`panoramic images can be generated, which, in turn, can be
`displayed to a viewer. Generally, left and right panoramic
`images can be generated by rendering from a model, or by
`recording a plurality of non-panoramic images and mosaic(cid:173)
`ing together portions thereof to generate the left and right
`panoramic images. While in FIG. lA, the eyes are depicted
`as viewing in a direction tangent to the viewing circle 5, it
`will be appreciated that in some embodiments the camera
`may instead be generally, although not precisely, perpen(cid:173)
`dicular to the viewing circle.
`FIG. 2 schematically depicts a stereoscopic panorama
`recording and generating system 10 constructed in accor(cid:173)
`dance with one embodiment of the invention. With reference 65
`to FIG. 2, system 10 includes a camera rig 11 and a
`panorama mosaic image generator 12. The camera rig 11
`
`4
`includes an image recording device such as a camera 13
`mounted on a support comprising an upwardly (in the case
`of horizontally-recorded panoramas) extending mast 14 and
`an arm 15 rigidly affixed thereto and extending therefrom.
`5 The camera 13 may record images using any of a number of
`types of image recording media, including, for example,
`film, charge-coupled devices (CCD's) or the like. The cam(cid:173)
`era 13 is rigidly mounted on the arm 15 so that the images
`recorded by the camera 13 will be recorded thereby a
`10 predetermined distance from the mast 14. A motor 16 is
`configured to rotate the mast 14 around a vertical axis,
`thereby to, in turn, cause the camera 13 to revolve along a
`curved path centered on the axis around which mast 14
`rotates. The movement of the motor 16 is controlled by a
`15 motor control 17 which, in tum, is controlled by the pan(cid:173)
`orama mosaic image generator 12, as will be described
`below. Since the camera 13 is rigidly affixed to the arm 15,
`the camera 13 will point in a direction defined by the arm 15.
`In one embodiment, the camera 13 is a still camera, as
`20 opposed to a motion picture camera, and in that embodiment
`the motor 16 will preferably be a stepping motor to enable
`the camera 13 to revolve step-by-step, with the camera 13
`being stopped at each step to allow the camera 13 to record
`an image. In that case, the angle between steps will be such
`25 as to facilitate mosaicing images recorded at each step into
`a panoramic image, as will be described below in connection
`with FIGS. 2 and 3. At some point, each of the images
`recorded at successive steps will be provided to the pan(cid:173)
`orama mosaic image generator 12 for processing. The pan-
`30 orama mosaic image generator 12, in turn, receives the
`images recorded at the successive steps and mosaics por(cid:173)
`tions of the images together to generate therefrom a stereo(cid:173)
`scopic panoramic image pair comprising left and right
`panoramic images. The left and right panoramic images
`35 comprising the stereoscopic panoramic image pair may be
`displayed to or viewed by a user as will be described below
`in connection with FIG. 3 to provide the user with a
`stereoscopic image of the scene as recorded by the camera
`13.
`As noted above, the panoramic image is generated by
`mosaicing together portions of images recorded with the
`camera 13 at various angular positions around the center of
`rotation of the mast 14. Before proceeding further, it would
`be helpful to describe details of the camera 13 and how the
`45 panoramic image generator 12 mosaics portions of the
`images together to form a panoramic image pair that, when
`viewed simultaneously by an observer's left and right eyes,
`provides a unitary stereoscopic panoramic image of the
`scene surrounding the location of the mast 14. FIG. 3 is a top
`50 plan view depicting details of the interior of camera 13, as
`seen from the top in FIG. 2, and FIG. 4 depicts details of
`images recorded by the camera and how portions of those
`images are mosaiced together to form the stereoscopic
`panoramic image pair. With reference initially to FIG. 3,
`55 camera 13 includes a housing 24 having a forward aperture
`20, a rear image recording medium 21, a screen 22 proxi(cid:173)
`mate to and forward of the image recording medium 21, and
`a shutter 23. The image recording medium 21, which defines
`an image plane for the camera 13, may comprise any
`60 convenient image recording medium, including film, a CCD
`array, or the like. The camera 13 may also include a lens (not
`shown) in the aperture 20 to facilitate focusing of images on
`the image plane. Alternatively, the camera 13 may comprise
`a pinhole camera, in which case no lens will be provided.
`The shutter 23 is provided to selectively allow light
`reflected from portions of a scene 27 at which the camera 13
`is directed is allowed to enter the camera and project upon
`
`Copy provided by USPTO from the PIRS Image Database on 02/13/2012
`
`HET00000013
`
`

`
`Case 1:12-cv-00398-GMS Document 27-1 Filed 02/21/13 Page 15 of 25 PageID #: 334
`
`US 6,665,003 Bl
`
`10
`
`5
`the screen 22 and portions of the image recording medium
`21 for recording thereby. The scene 27 forms part of a
`panoramic scene which is centered on the axis of rotation of
`the mast 14. Preferably, the shutter 23 will be closed while
`the motor 16 moves the camera 13 from one step to the next,
`thereby to block light from the scene 27 from entering the
`camera 13 and exposing the image recording medium 21
`during such movement, which might otherwise cause blur in
`the recorded image. After the motor 16 has reached a next
`step and the camera 13 has stopped moving, the shutter 23
`can be opened to allow light from the scene 27 to enter the
`camera and be directed rearwardly toward the screen 22 and
`image recording medium 21. After the image recording
`medium 21 has been appropriately exposed, the shutter can
`be closed, and the motor 16 energized to rotate the mast 14,
`thereby step the camera 13 to a new orientation, at which 15
`point these operations can be repeated. Preferably, if the
`image recording medium 21 is film, the film will be
`advanced before the shutter 23 is again opened so as to avoid
`double-exposure; similarly, if the image recording medium
`21 is a CCD device, the image information can be retrieved 20
`and stored and the CCD's refreshed as necessary to avoid
`double exposure. The image recorded by the camera 13 at
`the respective steps will be independent of each other, so as
`to avoid multiple exposures.
`The screen 22 is configured to generally cover portions of 25
`the image recording medium 21, except for at least two
`vertical slits 25L and 25R, which allow light from respective
`directions and portions 27L and 27R of the scene 27 to fall
`on proximate regions 26L and 26R of the image recording
`medium 21 and be recorded thereby. As is shown by the 30
`dashed lines 28L and 28R, the slit 25R is positioned to allow
`region 26R of the image recording medium 21 to record the
`left-hand portion 27R of the scene 27, and the slit 25L is
`positioned to allow region 26L of the image recording
`medium 21 to record the right-hand portion 27L of the scene 35
`27. With reference to FIG. lA, it will be appreciated that the
`portion 27R of the scene 27 as recorded by the region 26R
`of the image recording medium 21 is from a direction which
`corresponds to the direction that an observer's right eye
`would be viewing that portion 27R of the scene if he or she 40
`were looking directly at the portion 27R of the scene.
`Similarly, the portion 27L of the scene 21 as recorded by the
`region 26L of the image recording medium is from a
`direction which corresponds to the direction that the observ(cid:173)
`er's left eye would be viewing that portion 27L of the scene 45
`if he or she were looking directly at the portion 27L of the
`scene. It will further be appreciated that, preferably the
`placement of the camera 13 on the arm 15 will be such that,
`as the camera 13 is rotated, the regions 26L and 26R revolve
`through a circle, namely, the image circle, and the rays 50
`shown by dashed lines 28L and 28R are tangent to an inner
`viewing circle similar to that described above in connection
`with FIGS. 1A and lB. Thus, as the motor 16 steps the
`camera 13 so as to enable the region 26L of the image
`recording medium 21 to record what is depicted in FIG. 3 as 55
`portion 27R of the scene 27, an observer who contempora(cid:173)
`neously views that image with the left eye, and with the right
`eye the image previously recorded in the region 27R of the
`scene 27, would see a stereoscopic image of that portion 27R
`of the scene 27. Similarly, when the motor 16 steps the 60
`camera 13 so as to enable the region 26R of the image
`recording medium 21 to record what is depicted in FIG. 3 as
`region 27L of the scene 27, an observer who contempora(cid:173)
`neously views that image with the right eye, and with the left
`eye the image previously recorded in the region 27L of the 65
`scene 27, would see a stereoscopic image of that portion 27L
`of the scene 27.
`
`6
`In the embodiment depicted in FIG. 3, the screen 22 of
`camera 13 also is provided with a center slit 25C, which,
`when shutter 22 opens to allow light from the scene 27 to
`enter the camera 13, allows a center region 26C of the image
`5 recording medium 21 to record what is depicted in FIG. 3 as
`region 27C of the scene 27. It will be appreciated that the
`center region 26C is a direct view of the region, and the ray
`28C approximately corresponds to the optical center of the
`camera.
`As noted above, the images recorded by the camera 13 at
`successive steps around the center of rotation of the mast 14
`can be mosaiced together by the panorama mosaic image
`generator 12 to provide left and right panorama images. The
`left and right panoramic images, when viewed simulta(cid:173)
`neously by an observer, in particular by the observer's left
`and right eyes, results in a stereoscopic panoramic image of
`the panoramic scene centered on the center of rotation of
`mast 14. The manner in which the left and right panoramic
`images are generated will be described in detail in connec(cid:173)
`tion with FIG. 4. With reference to FIG. 4, each of the
`images 30(1) through 30(N) (generally identified by refer-
`ence numeral 30(n)) recorded at "N" (where "N" is an
`integer) successive steps around the center of rotation of the
`mast 14 includes a left image portion 30L(n) and a right
`portion 30R(n). The left image portion 30L(n) of each image
`30(n), in turn, corresponds to the left region 26L as recorded
`on the image recording medium 21 at the "n-th" step, and the
`right image portion 30R(n) corresponds to the right region
`26R as recorded on the image recording medium 21 at the
`same "n-th" step. The panorama mosaic image generator 12
`receives the successive images 30(1), 30(2), . . . 30(N) and
`mosaics the left image portions 30L(1), 30L(2), .. . 30L(N)
`therefrom together to form the left panoramic image 31L, as
`indicated by the arrows 32L(1), 32L(2), . . . 32L(N).
`Similarly, the panorama mosaic image generator 12 mosaics
`the right image portions 30R(1), 30R(2), . . . 30R(N)
`therefrom together to form the right panoramic image 31R,
`as indicated by the arrows 32R(1), 32R(2), . . . 32R(N). The
`panorama mosaic image generator 12 can generate the left
`and right panoramic images 31L and 31R using any con(cid:173)
`ventional technique for mosaicing images or portions of
`images together. It will be appreciated that the left and right
`panoramic images 31L and 31R conform to what an
`observer would see through his or her left and right eyes,
`respectively, as they revolve through the left and right
`viewing circles 5L and 5R described above in connection
`with FIG. lB. The panoramic mosaic image generator 12
`can generate the left and right panoramic images 31L and
`31R as respective image strips, or it may form the images as
`respective continuous loops by mosaicing together their
`respective left and right ends. In addition, the images may be
`formed on or stored in any convenient medium, such as
`paper or film, in digital form in electronic or magnetic data
`storage, or other media as will be appreciated by those
`skilled in the art.
`It will further be appreciated that the widths of the left and
`right image portions 30L(n) and 30R(n) of the respective
`images 30(n), which are generally related to the widths of
`the left and right slits 25L and