`Peleg et ai.
`
`111111
`
`1111111111111111111111111111111111111111111111111111111111111
`US006665003Bl
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 6,665,003 BI
`Dec. 16,2003
`
`(54) SYSTEM AND METHOD FOR GENERATING
`AND DISPLAYING PANORAMIC IMAGES
`AND MOVIES
`
`(75)
`
`Inventors: Shmuel Peleg, Mevaseret Zion (IL);
`Moshe Benezra, Jerusalem (IL);
`Robert S. Rosenschein, Jerusalem (IL)
`
`(73) Assignee: Issum Research Development
`Company of the Hebrew University of
`Jerusalem, Jerusalem (IL)
`
`( *) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.c. 154(b) by 0 days.
`
`(21) Appl. No.: 09/396,248
`
`(22) Filed:
`
`Sep. 16, 1999
`
`Related U.S. Application Data
`(60) Provisional application No. 60/100,721, filed on Sep. 17,
`1998, provisional application No. 60/102,720, filed on Sep.
`29, 1998, provisional application No. 60/113,962, filed on
`Dec. 28, 1998, provisional application No. 60/116,138, filed
`on Jan. 12, 1999, provisional application No. 60/116,139,
`filed on Jan. 12, 1999, provisional application No. 60/123,
`080, filed on Mar. 4, 1999, and provisional application No.
`60/129,987, filed on Apr. 18, 1999.
`
`Int. CI? .................................................. H04N 7/00
`(51)
`(52) U.S. CI. ......................................................... 348/36
`(58) Field of Search .............................. 348/42, 47, 51,
`348/53, 64, 39; 382/154, 284, 285, 294;
`349/15, 62, 74, 194
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`5,049,987 A * 9/1991 Hoppenstein ................ 348/48
`5,130,794 A * 7/1992 Ritchey ....................... 348/39
`
`5,510,831 A * 4/1996 Mayhew ...................... 348/47
`5,649,032 A
`7/1997 Burt et al.
`.................. 382/284
`5,686,975 A * 11/1997 Lipton ......................... 349/15
`5,721,585 A
`2/1998 Keast .......................... 348/36
`5,768,447 A
`6/1998 Irani .......................... 382/305
`5,850,352 A * 12/1998 Moezzi et al. .............. 364/514
`5,929,951 A * 7/1999 Sasakura et al. .............. 349/62
`5,963,664 A * 10/1999 Kumar et al. ............... 382/154
`6,064,353 A * 5/2000 Hoshi ............................ 345/7
`6,141,036 A * 10/2000 Katayama et al. ............ 348/47
`6,157,747 A * 12/2000 Szeliski et al.
`............. 382/284
`6,252,707 B1 * 6/2001 Kleinberger et al. ........ 359/465
`6,275,254 B1 * 8/2001 Beeteson et al.
`............. 348/59
`6,278,480 B1 * 8/2001 Kurahashi et al. ............ 348/59
`6,301,447 B1
`10/2001 Jackson ...................... 396/329
`6,331,871 B1 * 12/2001 Taylor ........................ 348/159
`2002/0005921 A1 * 1/2002 Sasakura et al. .............. 349/62
`* cited by examiner
`Primary Examiner-Vu Le
`Assistant Examiner-Behrooz Senfi
`(74) Attorney, Agent, or Firm-Fenster & Company
`
`(57)
`
`ABSTRACT
`
`Systems are disclosed for generating left and right pan(cid:173)
`oramic mosaic images for use in facilitating panoramic
`stereoscopic viewing of a scene, and for displaying stereo(cid:173)
`scopic panoramic images to a viewer. The left and right
`panoramic image generators generate the left and right
`panoramic mosaic images from a series of images recorded
`or otherwise generated corresponding to respective angular
`or other positions, each image having a respective left and
`right image portion, and mosaics portions of those images
`together to form the respective left and right panoramic
`images. The display system displays a stereoscopic pan(cid:173)
`oramic image to a viewer by displaying left and right
`panoramic images such that each is viewed by a respective
`one of the viewer's eyes.
`
`100 Claims, 9 Drawing Sheets
`
`~50
`
`1
`
`Petition for Inter Partes Review of
`U.S. Pat. No. 7,477,284
`IPR2013‐00327
`EXHIBIT
`Sony‐
`
`
`
`..-p
`t,
`
`t
`t
`t
`t
`t
`t
`t
`t
`t
`t
`t
`t
`t
`t
`t
`t
`
`I
`\
`\
`\
`\
`\
`\
`\
`\
`,
`\
`\
`\
`\
`,
`\
`
`\4
`~t
`,
`t
`\
`t
`,
`t
`\
`t
`3L-,'
`'('3R
`
`t
`
`2R
`
`F JG.IA
`FIG.1A
`
`e-P
`
`\
`\
`\
`\
`\
`\
`
`\ ,
`
`\
`\
`\
`\
`
`\-3R
`
`\
`\
`\
`\
`
`I
`
`/
`
`/
`
`/
`
`I
`
`I
`
`/'~-3R(2)
`
`I
`
`I
`
`I
`
`---
`-- '-3R(3)
`"
`\
`"
`\-3R(5),~,~~(4 )
`
`\
`
`d •
`rJl
`•
`
`
`
`
`
`cm.5.8.::8.a.88952Ham.cm939:8E
`
`.... p
`
`t
`t
`t
`t
`t
`t
`t
`t
`t
`t
`
`3L'"" "
`
`~
`
`///~3L(2)
`
`,'-3L(5)
`
`t
`
`F IG.IB
`FIG.1B
`
`
`
`\
`
`\
`
`\
`
`3L(N)·\
`
`\
`
`\
`
`\
`
`\
`
`\
`
`~
`~
`~
`~
`~
`
`~~~3~(6)
`
`\
`
`\
`
`\\~L(4) 3R(6}-..//
`
`I
`
`\
`
`\
`
`\
`
`\
`
`I
`
`/
`
`/
`
`/
`
`t
`t
`t
`t
`
`2
`
`
`
`
`~ ______ ~ CAMERA
`13
`
`FIG.3
`
`d •
`rJl
`•
`
`27R,,27C,
`SCENE 27
`
`I
`I
`
`\
`\
`
`\
`
`\
`\
`\
`\
`
`: .. 28C I
`
`I
`
`I
`I
`I
`I
`
`28R~\
`
`\
`\
`\
`
`1~28L
`
`I
`I
`
`24
`
`k- 13
`
`I
`I
`I
`I
`
`\
`\
`\
`
`I
`I
`I
`I
`
`\
`\
`
`\
`\
`
`22
`
`26R
`
`~11
`
`~10
`
`MOTOR
`16
`
`MOTOR
`CONTROL
`17
`
`PANORAMA
`MOSAIC IMAGE
`GENERATOR
`12
`
`FIG.2
`
`3
`
`
`
`d •
`rJl
`•
`
`. . . .
`
`30R(1)
`
`30L(2)
`
`30R(N)
`
`LEFT PANORAMIC IMAGE 31 L
`
`30L(N)
`
`32L(N)
`
`. . . .
`
`FIG.4
`
`4
`
`
`
`u.s. Patent
`
`Dec. 16,2003
`
`Sheet 4 of 9
`
`US 6,665,003 BI
`
`!'f00. CAMERA RIG 11 IS POSITIONED SUCH THAT THE
`AXIS OF MAST 141S PLACED AT THE CENTER OF
`ROTATION FOR THE STEREOSCOPIC PANORAMA, AND
`THE MOTOR CONTROL 17 INITIALLY ENABLES THE
`MOTOR 16 TO POSITION THE CAMERA 13 AT A STARTING
`POINT AT WHICH THE FIRST IMAGE 30(1) WILL BE
`RECORDED
`"-
`
`..0lIl
`
`"-
`
`101. MOTOR CONTROL 17 NOTIFIES THE PANORAMA
`MOSAIC IMAGE GENERATOR 12 THAT THE CAMERA 131S
`APPROPRIATELY POSITIONED
`
`~
`
`102. PANORAMA MOSAIC IMAGE GENERATOR 12
`CONTROLS THE CAMERA 13 TO RECORD AN IMAGE ON
`
`THE IMAGE RECORDING MEDIUM 21 •
`
`YES , -
`
`I'"
`
`........
`
`..,oj
`
`'103. PANORAMA MOSAIC IMAGE GENERATOR 12
`DETERMINES WHETHER ALL OF THE IMAGES 30(1)
`THROUGH 30(N) TO BE USED IN GENERATING THE LEFT
`AND RIGHT PANORAMIC IMAGES 31 LAND 31 R FOR THE
`STEREOSCOPIC PANORAMA IMAGE PAIR HAVE BEEN
`RECORDED
`"-
`
`t
`
`104. PANORAMA MOSAIC IMAGE GENERATOR 12
`ENABLES THE MOTOR CONTROL 17 TO, IN TURN,
`CONTROL THE MOTOR 16 TO ADVANCE A STEP
`
`105. PANORAMA MOSAIC IMAGE GENERATOR 12
`IDENTIFIES THE LEFT AND RIGHT IMAGE PORTIONS
`30L(n) AND 30R(n) FOR THE RESPECTIVE IMAGES 30(n)
`
`106. PANORAMA MOSAIC IMAGE GENERATOR 12
`MOSAICS SUCCESSIVE LEFT IMAGE PORTIONS 30L(n) TO
`FORM THE LEFT PANORAMIC IMAGE 31L AND THE
`SUCCESSIVE RIGHT IMAGE PORTIONS 30R(n) TO FORM
`THE RIGHT PANORAMIC IMAGE 31 R
`
`FIG.5
`
`5
`
`
`
`, , , ,
`
`52(N)
`
`/
`
`/ , ,
`
`/
`
`/
`
`/
`
`/
`
`/
`
`/
`
`/
`
`)~.//
`Y
`..
`///:
`: ",
`
`I
`
`I ,
`I
`I
`
`.....
`
`"
`
`;'
`/"
`. . . . . , .
`.....,.
`
`I . . . . .
`I
`"
`I
`I
`I
`I
`1
`1
`
`I
`I
`
`/~
`
`,.
`,,/
`,.
`
`I
`I
`I
`
`I
`I
`I
`I
`I
`I
`I
`
`•
`rJl
`•
`
`52(2)
`
`, , ,
`
`-E-- 50
`
`52(3)
`
`------------;
`
`/
`
`/
`
`/
`
`/
`
`/
`
`/
`
`/
`
`/
`
`/
`
`/
`
`/
`
`............ _, , /
`,
`51 (3)"'"
`
`,
`, , ,
`"
`-------------y-
`,
`
`52(6)
`
`,
`
`/ ,
`
`52(5)
`
`1
`1
`1
`1
`1
`1
`1
`1
`1
`1
`1
`1
`,I
`
`,
`
`,
`
`"
`
`,
`
`"
`
`, , , , ,
`
`52(4)
`
`FIG. 6
`
`PANORAMA
`MOSAIC IMAGE
`GENERATOR
`12
`
`6
`
`
`
`------,u~'\_'_'_F~
`
`61 (s)(L)
`
`621~
`
`
`65(2)(L)
`65(2)(L)'J
`64(2) \
`64(2)
`
`F [G. 7A
`
`I 4"1) \
`
`I 61(2) 61(4
`
`
`65(1)(
`
`64(1)’
`
`65(1)(RP
`66(1
`
`64(N)(R)
`
`
`
`
`4(6)(R)
`
`I CA~~RA pm_[85m_~;m ___
`
`F [G. 7
`
`80.-/1
`
`F [G. 8
`
`I
`I
`I
`I
`I
`I
`I
`
`~84
`
`I
`
`- 'e;;
`
`d •
`rJl
`•
`
`
`
`mama'S'n
`
`€00Z‘91'93(1
`
`6J09133118
`
`
`
`IEI900‘999‘9Sfl
`
`7
`
`
`
`~100
`
`~104
`
`d •
`rJl
`•
`
`F [G. 9
`
`~120
`
`F [G.ll
`
`DISPLAY
`CONTROL
`MODULE 122
`
`124
`
`8
`
`
`
`u.s. Patent
`
`US. Patent
`
`Dec. 16,2003
`D
`0
`
`9f000whS
`Sheet 8 of 9
`
`US 6,665,003 BI
`US 6,665,003 B1
`
`EmA‘
`2,mo<s=n/Fvmeg.mo_s_<m02<n_ao_2<«_02<n_wEm:Ema
`
`§~G~K
`
`~
`~
`~
`~
`~
`
`W
`Z
`W
`u
`en
`
`mzmom
`
`U
`f-~W
`I«(9
`(9O::::«E
`-O~
`O::::z_
`ct
`
`m0<§_
`
`
`
`03Hmm2<¢Agg<moz<amPlug
`
`u
`:2: w
`mo<2_
`t=~(9E
`wOo::(
`.....lz:2:
`0:
`
`9
`
`
`
`
`
`
`d •
`rJl
`•
`
`1:::'140
`
`143RD A3L(2) V <-142(2)
`
`F JG.l2A
`
`C::P1)
`
`143 'D
`
`142(1)-"
`
`141
`
`140~
`
`F JG.l2B
`
`143R(b
`
`d (3 )
`
`'\.142(3)
`
`i44
`
`•
`
`(145
`
`[J II ~142(N)
`
`143L(N)J
`
`v'43R(N)
`
`~
`
`'JJ.
`
`=-~
`.....
`\C
`0 ....,
`\C
`
`10
`
`
`
`US 6,665,003 Bl
`
`1
`SYSTEM AND METHOD FOR GENERATING
`AND DISPLAYING PANORAMIC IMAGES
`AND MOVIES
`
`This application claims the benefit of Provisional appli(cid:173)
`cation Ser. No. 60/100,721 filed Sep. 17, 1998; Provisional
`application Ser. No. 60/102,720 filed Sep. 29, 1998; Provi(cid:173)
`sional application Ser. No. 60/113,962 filed Dec. 28, 1998;
`Provisional application Ser. No. 60/116,138 filed Jan. 12,
`1999; Provisional application Ser. No. 60/116,139 filed Jan.
`12, 1999; Provisional application Ser. No. 60/123,080 filed
`Mar. 4, 1999; Provisional application Ser. No. 60/129,987
`filed Apr. 18, 1999.
`
`FIELD OF THE INVENTION
`
`The invention relates generally to the field of recording,
`generating and playing back or displaying images, and more
`particularly to the generation and displaying of panoramic
`images stereoscopically.
`Panoramic images are images of a scene having a wide
`field of view, up to a full 360°. Panoramic images may be
`recorded using a wide angled lens, a mirror, or the like,
`providing a wide field of view. Panoramic images having a
`wider field of view can be generated by, for example, 25
`recording a plurality of images around a particular point and,
`using conventional mosaicing techniques, generating a
`single mosaic image. Panoramic images may also be gen(cid:173)
`erated of simulated scenes using conventional computer
`graphics techniques.
`A problem arises in connection with viewing panoramic
`images stereoscopically. A person can see stereoscopically
`because his or her eyes are displaced horizontally (when
`standing) which, will provide a perception of depth when
`viewing a scene, which would not be present otherwise. 35
`Stereoscopic images comprise two images recorded of a
`scene recorded from slightly displaced positions, which,
`when viewed simultaneously by the respective eyes, pro(cid:173)
`vides a perception of depth. Although currently there are
`arrangements for generating and displaying stereoscopically 40
`non-panoramic images, currently there are no such arrange(cid:173)
`ments for generating and displaying stereoscopically pan-
`oramic images.
`
`2
`invention may be better understood by referring to the
`following description taken in conjunction with the accom(cid:173)
`panying drawings, in which:
`FIGS. lA and lB are useful in understanding operations
`5 performed by arrangements for generating and displaying
`stereoscopic panorama images constructed in accordance
`with the invention;
`FIG. 2 schematically depicts a stereoscopic panorama
`recording and generating system constructed in accordance
`10 with one embodiment of the invention;
`FIG. 3 schematically depicts a camera used in the stereo(cid:173)
`scopic panorama recording and generating system depicted
`in FIG. 2;
`FIG. 4 depicts generation of left and right panorama
`15 images from images recorded by the camera depicted in
`FIG. lB, useful in understanding the operation of the
`panoramic image generator of the stereoscopic panorama
`recording and generating system depicted in FIG. 2;
`FIG. 5 depicts a flow chart describing operations per-
`20 formed by the stereoscopic panorama recording and gener-
`ating system;
`FIG. 6 schematically depicts a stereoscopic panorama
`recording and generating system constructed in accordance
`a second embodiment of the invention;
`FIGS. 7 and 7A schematically depicts a stereoscopic
`panorama recording and generating system constructed in
`accordance with a third embodiment of the invention;
`FIG. 8 schematically depicts a stereoscopic panorama
`30 recording and generating system constructed in accordance
`with a fourth embodiment of the invention;
`FIG. 9 schematically depicts a stereoscopic panorama
`recording and generating system constructed in accordance
`with a fifth embodiment of the invention;
`FIG. 10 is useful in understanding generation of a stereo(cid:173)
`scopic panorama image pair using computer graphics tech(cid:173)
`niques;
`FIG. 11 schematically depicts a first arrangement for
`displaying a stereoscopic panorama image to a viewer; and
`FIGS. l2A and l2B together schematically depict a
`second arrangement for displaying a stereoscopic panorama
`image to a viewer.
`
`45
`
`DETAILED DESCRIPTION OF AN
`ILLUSTRATIVE EMBODIMENT
`The invention provides systems and methods for gener(cid:173)
`ating stereoscopic panoramic images of a scene, and for
`displaying the images to a one or more viewers in a manner
`so that the viewer(s) can view the panoramic images ste-
`50 reoscopically. Before describing the inventive systems and
`methods, it would be helpful to first describe what a stereo(cid:173)
`scopic panoramic image is and generally how the various
`systems and methods described herein generate the stereo(cid:173)
`scopic panoramic images and facilitate their display. This
`55 will be done in connection with FIGS. lA and lB. With
`reference initially to FIG. lA that FIG. schematically depicts
`an observer, and, particularly, eyes represented by dots 2L
`and 2R (generally identified by reference numeral "2L/R")
`standing vertically and observing a point P in a scene. The
`60 observer sees point P by means of rays of light reflected from
`the point and directed toward the eyes 2L and 2R along
`respective rays represented by dashed arrows 3L and 3R. It
`will be appreciated that, since the rays 3L and 3R are not
`parallel, the observer will be able to observe a depth in
`65 connection with the region of the scene at and near point P.
`The observer typically can see only a small portion of the
`360° panorama around himself or herself. To see more of the
`
`SUMMARY OF THE INVENTION
`
`The invention provides a new and improved system and
`method of generating and displaying stereoscopic pan(cid:173)
`oramic images.
`In brief summary, in one aspect the invention provides a
`system for generating left and right panoramic mosaic
`images for use in facilitating panoramic stereoscopic view(cid:173)
`ing of a scene. The left and right panoramic image genera(cid:173)
`tors generate the left and right panoramic mosaic images
`from a series of images recorded or otherwise generated
`corresponding to respective angular or other positions, each
`image having a respective left and right image portion, and
`mosaics portions of those images together to form the
`respective left and right panoramic images.
`In another aspect, the invention provides a system for
`displaying a stereoscopic panoramic image to a viewer, by
`displaying left and right panoramic images such that each is
`viewed by a respective one of the viewer's eyes.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`This invention is pointed out with particularity in the
`appended claims. The above and further advantages of this
`
`11
`
`
`
`US 6,665,003 Bl
`
`3
`panorama, the observer will rotate his or her head in, for
`example, the direction indicated by the arrow identified by
`reference numeral 4. Rotation of the head will allow the
`observer to view other points (not shown) in the scene, along
`rays (also not shown) that rotate with him or her. If the
`, each eye will revolve
`observer rotates around a full 360 0
`around the same viewing circle S.
`It will be apparent from FIG. 1A that each the succession
`of images as seen by the observer's two eyes as he or she
`rotates, can be separated into separate sets of images, with
`one set of images being associated with each eye. This will
`be described in connection with FIG. lB. FIG. lB. depicts
`the viewing circle S divided into separate viewing circles SL
`and SR (generally SUR) for the respective left and right
`eyes, with point P being shown in the same position as in
`FIG. lA, with respect to each viewing circle SUR, and the
`associated ray 3L(1) and 3R(1), which correspond to rays 3L
`and 3R depicted in FIG. 1A. Each viewing circle SUR also
`depicts other rays, identified by reference numerals
`3L(2), ... , 3L(N) (generally identified by reference numeral
`3L(n)) and 3R(2), ... , 3R(N) (generally identified by
`reference numeral 3R(n)) that represent images that the
`respective left and right eyes of the observer will receive of
`the various points in the scene as he or she rotates in the
`direction represented by arrows 4L and 4R.
`Further in connection with FIG. 1B, to facilitate the
`viewing of a stereoscopic panoramic image of the scene by
`a viewer, the images as would be received by each of the
`observer's eyes can be separately recorded and viewed by, or
`otherwise displayed to, the respective eyes of the viewer.
`Thus, if, for example, images are recorded around a circle
`corresponding to viewing circle SL at successive points, in
`successive direction depicted by rays 3L(1), ... 3L(N), and
`the images mosaiced together, and further images are
`recorded around a circle corresponding to viewing circle SR
`at successive points, in successive direction depicted by rays
`3R(1), ... 3R(N), and if those images are suitably aligned
`(such that the point of intersection of the rays 3L(n) and
`3R(n) are viewed in the same relative location) and dis(cid:173)
`played to respective eyes of a viewer, the viewer can see a 40
`stereoscopic panoramic image of the scene.
`In a similar manner, stereoscopic panoramic images can
`be generated using computer graphics techniques. However,
`instead of the regular perspective projection used in con(cid:173)
`ventional image rendering, the panoramic image for the left
`eye will be rendered using rays tangent to a circle such as
`viewing circle SL, and the panoramic image for the right eye
`will be rendered using rays tangent to a circle such as
`viewing circle SR.
`The invention provides various arrangements for facili(cid:173)
`tating the recording of images from which left and right
`panoramic images can be generated, which, in turn, can be
`displayed to a viewer. Generally, left and right panoramic
`images can be generated by rendering from a model, or by
`recording a plurality of non-panoramic images and mosaic(cid:173)
`ing together portions thereof to generate the left and right
`panoramic images. While in FIG. lA, the eyes are depicted
`as viewing in a direction tangent to the viewing circle S, it
`will be appreciated that in some embodiments the camera
`may instead be generally, although not precisely, perpen(cid:173)
`dicular to the viewing circle.
`FIG. 2 schematically depicts a stereoscopic panorama
`recording and generating system 10 constructed in accor(cid:173)
`dance with one embodiment of the invention. With reference 65
`to FIG. 2, system 10 includes a camera rig 11 and a
`panorama mosaic image generator 12. The camera rig 11
`
`4
`includes an image recording device such as a camera 13
`mounted on a support comprising an upwardly (in the case
`of horizontally-recorded panoramas) extending mast 14 and
`an arm IS rigidly affixed thereto and extending therefrom.
`5 The camera 13 may record images using any of a number of
`types of image recording media, including, for example,
`film, charge-coupled devices (CCD's) or the like. The cam(cid:173)
`era 13 is rigidly mounted on the arm IS so that the images
`recorded by the camera 13 will be recorded thereby a
`10 predetermined distance from the mast 14. A motor 16 is
`configured to rotate the mast 14 around a vertical axis,
`thereby to, in turn, cause the camera 13 to revolve along a
`curved path centered on the axis around which mast 14
`rotates. The movement of the motor 16 is controlled by a
`15 motor control 17 which, in turn, is controlled by the pan(cid:173)
`orama mosaic image generator 12, as will be described
`below. Since the camera 13 is rigidly affixed to the arm IS,
`the camera 13 will point in a direction defined by the arm IS.
`In one embodiment, the camera 13 is a still camera, as
`20 opposed to a motion picture camera, and in that embodiment
`the motor 16 will preferably be a stepping motor to enable
`the camera 13 to revolve step-by-step, with the camera 13
`being stopped at each step to allow the camera 13 to record
`an image. In that case, the angle between steps will be such
`25 as to facilitate mosaicing images recorded at each step into
`a panoramic image, as will be described below in connection
`with FIGS. 2 and 3. At some point, each of the images
`recorded at successive steps will be provided to the pan(cid:173)
`orama mosaic image generator 12 for processing. The pan-
`30 orama mosaic image generator 12, in turn, receives the
`images recorded at the successive steps and mosaics por(cid:173)
`tions of the images together to generate therefrom a stereo(cid:173)
`scopic panoramic image pair comprising left and right
`panoramic images. The left and right panoramic images
`35 comprising the stereoscopic panoramic image pair may be
`displayed to or viewed by a user as will be described below
`in connection with FIG. 3 to provide the user with a
`stereoscopic image of the scene as recorded by the camera
`13.
`As noted above, the panoramic image is generated by
`mosaicing together portions of images recorded with the
`camera 13 at various angular positions around the center of
`rotation of the mast 14. Before proceeding further, it would
`be helpful to describe details of the camera 13 and how the
`45 panoramic image generator 12 mosaics portions of the
`images together to form a panoramic image pair that, when
`viewed simultaneously by an observer's left and right eyes,
`provides a unitary stereoscopic panoramic image of the
`scene surrounding the location of the mast 14. FIG. 3 is a top
`50 plan view depicting details of the interior of camera 13, as
`seen from the top in FIG. 2, and FIG. 4 depicts details of
`images recorded by the camera and how portions of those
`images are mosaiced together to form the stereoscopic
`panoramic image pair. With reference initially to FIG. 3,
`55 camera 13 includes a housing 24 having a forward aperture
`20, a rear image recording medium 21, a screen 22 proxi(cid:173)
`mate to and forward of the image recording medium 21, and
`a shutter 23. The image recording medium 21, which defines
`an image plane for the camera 13, may comprise any
`60 convenient image recording medium, including film, a CCD
`array, or the like. The camera 13 may also include a lens (not
`shown) in the aperture 20 to facilitate focusing of images on
`the image plane. Alternatively, the camera 13 may comprise
`a pinhole camera, in which case no lens will be provided.
`The shutter 23 is provided to selectively allow light
`reflected from portions of a scene 27 at which the camera 13
`is directed is allowed to enter the camera and project upon
`
`12
`
`
`
`US 6,665,003 Bl
`
`5
`
`5
`the screen 22 and portions of the image recording medium
`21 for recording thereby. The scene 27 forms part of a
`panoramic scene which is centered on the axis of rotation of
`the mast 14. Preferably, the shutter 23 will be closed while
`the motor 16 moves the camera 13 from one step to the next,
`thereby to block light from the scene 27 from entering the
`camera 13 and exposing the image recording medium 21
`during such movement, which might otherwise cause blur in
`the recorded image. After the motor 16 has reached a next
`step and the camera 13 has stopped moving, the shutter 23 10
`can be opened to allow light from the scene 27 to enter the
`camera and be directed rearwardly toward the screen 22 and
`image recording medium 21. After the image recording
`medium 21 has been appropriately exposed, the shutter can
`be closed, and the motor 16 energized to rotate the mast 14,
`thereby step the camera 13 to a new orientation, at which 15
`point these operations can be repeated. Preferably, if the
`image recording medium 21 is film, the film will be
`advanced before the shutter 23 is again opened so as to avoid
`double-exposure; similarly, if the image recording medium
`21 is a CCD device, the image information can be retrieved 20
`and stored and the CCD's refreshed as necessary to avoid
`double exposure. The image recorded by the camera 13 at
`the respective steps will be independent of each other, so as
`to avoid multiple exposures.
`The screen 22 is configured to generally cover portions of 25
`the image recording medium 21, except for at least two
`vertical slits 2SL and 2SR, which allow light from respective
`directions and portions 27L and 27R of the scene 27 to fall
`on proximate regions 26L and 26R of the image recording
`medium 21 and be recorded thereby. As is shown by the 30
`dashed lines 28L and 28R, the slit 2SR is positioned to allow
`region 26R of the image recording medium 21 to record the
`left-hand portion 27R of the scene 27, and the slit 2SL is
`positioned to allow region 26L of the image recording
`medium 21 to record the right-hand portion 27L of the scene 35
`27. With reference to FIG. lA, it will be appreciated that the
`portion 27R of the scene 27 as recorded by the region 26R
`of the image recording medium 21 is from a direction which
`corresponds to the direction that an observer's right eye
`would be viewing that portion 27R of the scene if he or she 40
`were looking directly at the portion 27R of the scene.
`Similarly, the portion 27L of the scene 21 as recorded by the
`region 26L of the image recording medium is from a
`direction which corresponds to the direction that the observ(cid:173)
`er's left eye would be viewing that portion 27L of the scene 45
`if he or she were looking directly at the portion 27L of the
`scene. It will further be appreciated that, preferably the
`placement of the camera 13 on the arm IS will be such that,
`as the camera 13 is rotated, the regions 26L and 26R revolve
`through a circle, namely, the image circle, and the rays 50
`shown by dashed lines 28L and 28R are tangent to an inner
`viewing circle similar to that described above in connection
`with FIGS. 1A and lB. Thus, as the motor 16 steps the
`camera 13 so as to enable the region 26L of the image
`recording medium 21 to record what is depicted in FIG. 3 as 55
`portion 27R of the scene 27, an observer who contempora(cid:173)
`neously views that image with the left eye, and with the right
`eye the image previously recorded in the region 27R of the
`scene 27, would see a stereoscopic image of that portion 27R
`of the scene 27. Similarly, when the motor 16 steps the 60
`camera 13 so as to enable the region 26R of the image
`recording medium 21 to record what is depicted in FIG. 3 as
`region 27L of the scene 27, an observer who contempora(cid:173)
`neously views that image with the right eye, and with the left
`eye the image previously recorded in the region 27L of the 65
`scene 27, would see a stereoscopic image of that portion 27L
`of the scene 27.
`
`6
`In the embodiment depicted in FIG. 3, the screen 22 of
`camera 13 also is provided with a center slit 2SC, which,
`when shutter 22 opens to allow light from the scene 27 to
`enter the camera 13, allows a center region 26C of the image
`recording medium 21 to record what is depicted in FIG. 3 as
`region 27C of the scene 27. It will be appreciated that the
`center region 26C is a direct view of the region, and the ray
`28C approximately corresponds to the optical center of the
`camera.
`As noted above, the images recorded by the camera 13 at
`successive steps around the center of rotation of the mast 14
`can be mosaiced together by the panorama mosaic image
`generator 12 to provide left and right panorama images. The
`left and right panoramic images, when viewed simulta-
`neously by an observer, in particular by the observer's left
`and right eyes, results in a stereoscopic panoramic image of
`the panoramic scene centered on the center of rotation of
`mast 14. The manner in which the left and right panoramic
`images are generated will be described in detail in connec-
`tion with FIG. 4. With reference to FIG. 4, each of the
`images 30(1) through 30(N) (generally identified by refer-
`ence numeral 30(n)) recorded at "N" (where "N" is an
`integer) successive steps around the center of rotation of the
`mast 14 includes a left image portion 30L(n) and a right
`portion 30R(n). The left image portion 30L(n) of each image
`30(n), in turn, corresponds to the left region 26L as recorded
`on the image recording medium 21 at the "n-th" step, and the
`right image portion 30R(n) corresponds to the right region
`26R as recorded on the image recording medium 21 at the
`same "n-th" step. The panorama mosaic image generator 12
`receives the successive images 30(1), 30(2), ... 30(N) and
`mosaics the left image portions 30L(1), 30L(2), ... 30L(N)
`therefrom together to form the left panoramic image 31L, as
`indicated by the arrows 32L(1), 32L(2), . . . 32L(N).
`Similarly, the panorama mosaic image generator 12 mosaics
`the right image portions 30R(1), 30R(2), . . . 30R(N)
`therefrom together to form the right panoramic image 31R,
`as indicated by the arrows 32R(1), 32R(2), ... 32R(N). The
`panorama mosaic image generator 12 can generate the left
`and right panoramic images 31L and 31R using any con(cid:173)
`ventional technique for mosaicing images or portions of
`images together. It will be appreciated that the left and right
`panoramic images 31L and 31R conform to what an
`observer would see through his or her left and right eyes,
`respectively, as they revolve through the left and right
`viewing circles SL and SR described above in connection
`with FIG. lB. The panoramic mosaic image generator 12
`can generate the left and right panoramic images 31L and
`31R as respective image strips, or it may form the images as
`respective continuous loops by mosaicing together their
`respective left and right ends. In addition, the images may be
`formed on or stored in any convenient medium, such as
`paper or film, in digital form in electronic or magnetic data
`storage, or other media as will be appreciated by those
`skilled in the art.
`It will further be appreciated that the widths of the left and
`right image portions 30L(n) and 30R(n) of the respective
`images 30(n), which are generally related to the widths of
`the left and right slits 2SL and 2SR in the screen 22 (FIG. 3),
`will generally determine the angle to which the motor 16
`will be constrained for successive steps. The angle between
`successive steps will be such as to ensure that the successive
`image portions 30L(1), 30L(2), ... 30L(N) can be mosaiced
`together to provide a single and continuous left panoramic
`image 31L and the successive images 30R(1), 30R(2), ...
`30R(N) can be mosaiced together to provide a single and
`continuous right panoramic image 31R.
`
`13
`
`
`
`US 6,665,003 Bl
`
`7
`As noted above, in one embodiment, the screen 22 (FIG.
`3) also has a center slit 25C to facilitate recording of a
`portion of the scene 27 in a center region 26C of the image
`recording medium, the portion being intermediate the por(cid:173)
`tions recorded on the left and right regions 26L and 26R. The 5
`center image portions as recorded on the respective images
`30(n) are identified in FIG. 2 by reference numeraI30C(n).
`In that case, the panorama mosaic image generator 12 can
`also generate a center panoramic image from center image
`portions 30C(n) of the successive images 30(n). While the 10
`center panoramic image so generated (which is not depicted
`in FIG. 4, may be used for stereo viewing together with one
`of the other two panoramic images 31L and 31R, it is
`preferable to use the left and right panoramic images 31L
`and 31R since the symmetry between the two images reduce 15
`distortion and increase stereoscopic disparity.
`The distance of the camera 13 from the mast 14, as well
`as the separation between the left and right slits 25L and
`24R, are selected such that for normal stereoscopy the radius
`of the viewing circle to which all rays 28R, 28L are tangent, 20
`approximates the distance between a person's eyes, which,
`in turn, approximately corresponds to the diameter of the
`viewing circle described above in connection with FIG. 1A.
`It will be appreciated that, the diameter of the viewing circle
`can be enlarged or reduced for exaggerated or reduced 25
`stereo.
`With this background, operations performed by the ste(cid:173)
`reoscopic panorama recording and generating system 10 in
`connection with generating the left and right panoramic
`images 31L and 31R will be described in connection with
`the flow chart in FIG. 5. With reference to FIG. 5, after the
`camera rig 11 has been positioned such that the axis of mast
`14 is placed at the center of rotation for the stereoscopic
`panorama, the motor control 17 initially enables the motor
`16 to position the by camera 13 at a starting point at which
`the first image 30(1) will be recorded (step 100). After the
`camera 13 is appropriately positioned, the motor control 17
`notifies the panorama mosaic image generator 12 (step 101).
`Thereafter, the stereoscopic panorama recording and gen(cid:173)
`erating system 10, in a number of iterations, rec