(12) United States Patent
`Peleg et al.
`
`I IIIII
`
`1111111111111111111111111111111111111111111111111111111111111
`US006831677B2
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 6,831,677 B2
`Dec.14,2004
`
`(54) SYSTEM AND METHOD FOR
`FACILITATING THE ADJUSTMENT OF
`DISPARITY IN A STEREOSCOPIC
`PANORAMIC IMAGE PAIR
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`6,665,003 B1 * 12/2003 Peleg et a!. ................... 348/36
`
`(75)
`
`Inventors: Shmuel Peleg, Mevaseret Zion (IL);
`Moshe Ben-Ezra, Jerusalem (IL); Yael
`Pritch, Jerusalem (IL)
`
`* cited by examiner
`
`(73) Assignee: Yissum Research Development
`Company of the Hebrew University of
`Jerusalem, Jerusalem (IL)
`
`( *)
`
`Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 640 days.
`
`(21)
`
`Appl. No.: 09/792,638
`
`(22)
`
`Filed:
`
`Feb.24,2001
`
`(65)
`
`Prior Publication Data
`
`US 2001/0038413 A1 Nov. 8, 2001
`
`(60)
`
`(51)
`(52)
`(58)
`
`Related U.S. Application Data
`Provisional application No. 60/184,589, filed on Feb. 24,
`2000.
`
`Int. Cl? .................................................. H04N 7/00
`U.S. Cl. ............................................ 348/36; 348/46
`Field of Search .............................. 348/36, 42, 43,
`348/44, 46, 47; 382/154, 294; H04N 7/00
`
`Primary Examiner-Young Lee
`(74) Attorney, Agent, or Firm---Fenster & Company
`
`(57)
`
`ABSTRACT
`
`A system for generating a stereoscopic panoramic image
`pair having left and right panoramic mosaic images for use
`in facilitating stereoscopic viewing of a panoramic scene,
`the system comprising a panoramic image generator and a
`strip separation value generation module. The panoramic
`image generator is configured to mosaic together respective
`left and right image strips from respective ones of a plurality
`of images to form the respective left and right panoramic
`images. The panoramic image generator is configured to use
`strip separation values for the respective images to deter(cid:173)
`mine the separation of the respective left and right image
`strips in generating the respective left and right panoramic
`images. The strip separation value generation module is
`configured to generate, for respective ones of said images,
`the strip separation values so as to provide at least one
`selected disparity for objects in the panoramic scene.
`
`25 Claims, 5 Drawing Sheets
`
`100. CONSTRUCT A STEREOSCOPIC PANORAMIC IMAGE
`PAIR FROM THE IMAGES 10(n) USING A FIXED VALUE FOR
`THE DISTANCE "v" BETWEEN THE CENTER OF THE
`IMAGE PLANE 30 AND THE STRIPS 10R(n) AND 10L(n)
`THAT ARE USED IN THE RESPECTIVE LEFT AND RIGHT
`PANORAMIC IMAGES
`
`~
`
`101. ALIGN THE LEFT AND RIGHT PANORAMIC IMAGES
`11 LAND 11 R COMPRISING THE STEREOSCOPIC
`PANORAMIC IMAGE PAIR SO THAT OBJECTS THAT ARE
`LOCATED AT A VERY LARGE, PREFERABLY INFINITE
`DISTANCE FROM THE CENTER OF ROTATION "0" OF THE
`CAMERA WILL HAVE NO DISPARITY
`
`~
`
`102. FOR EACH COLUMN IN THE RESPECTIVE LEFT AND )
`RIGHT PANORAMIC IMAGES, DETERMINE THE DISPARITY
`AS BETWEEN THE ALIGNED LEFT AND RIGHT
`PANORAMIC IMAGES
`
`~
`
`103. DETERMINE A UNITARY DISPARITY VALUE FOR
`EACH COLUMN
`
`l
`
`104. A FILTER MAY BE APPLIED TO THE UNITARY
`DISPARITY VALUES TO SMOOTH THEM OUT
`
`~
`
`105. USING IMAGES 10(n), CONSTRUCT A NEW
`STEREOSCOPIC PANORAMIC IMAGE PAIR IN WHICH THE
`VALUE OF "v," THE DISTANCE BETWEEN THE CENTER OF
`THE IMAGE PLANE AND THE RESPECTIVE STRIPS USED
`IN GENERATING THE LEFT AND RIGHT PANORAMIC
`MOSAIC IMAGES, IS VARIED TO PROVIDE
`PREDETERMINED EFFECTS
`
`1
`
`Petition for Inter Partes Review 
`of U.S. Pat. No. 6,665,003
`IPR2013‐00218
`EXHIBIT
`Sony‐
`
`Petition for Inter Partes Review
`of U.S. Pat. No. 7,477,284
`IPR2013‐00219
`EXHIBIT
`Sony‐
`
`

`

`e-P
`
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`1'-3R
`
`I
`I
`I
`
`~R
`'-/3R(N)
`\
`' ', ...... ...
`', ...
`3R(7)'\._-~
`-----
`4~
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`d •
`\Jl
`•
`~
`~ ......
`~ = ......
`
`~
`~
`!"l
`"""'
`~,J;;..
`N c c
`
`,J;;..
`
`'JJ. =(cid:173)~
`~ ......
`"""' 0 ......,
`
`Ul
`
`e
`rJ'l
`-..a-..
`00
`~
`1--"
`0..,
`""-l
`""-l
`~
`N
`
`I
`
`I
`
`I
`
`I
`
`~~3R(2)
`
`I
`
`I
`
`1 1
`
`I
`I
`
`I
`
` ~--c-3R(3)
`\
`',,~3R(4)
`'r-3R(5) ' ... ,
`
`'
`
`' ,
`
`\.~3L(4) 3R(6~/
`' ' \
`' \
`
`\
`
`I
`I
`
`,~3L(5)
`
`I
`
`I
`
`I
`
`I
`
`I
`
`'
`
`F IG.JB
`
`F IG.JA
`
`,~
`
`~ " "
`"
`"
`,~'3L(6)
`
`... p
`
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`3L"": I
`
`I
`I
`
`~ I
`
`~
`~~
`
`I
`
`:
`
`/~3L(2)
`
`I
`~~
`I
`"
`II~
`I
`I
`
`' ' '
`3L(N0,
`' \
`
`' '
`'
`
`~ Ill
`
`Ill
`I l l
`I 1 I
`I
`I
`
`
`:
`
`11
`
`\ alpha
`1/2 alpha~1/2 alpha
`3L--;'
`( " 3R
`
`I
`
`I
`
`I
`
`I
`
`I
`I
`I
`
`1
`
`I
`
`I
`
`14
`~I
`~ :
`I~
`1/2 alpha_;;:
`:
`\ )-1/2 alpha
`6L --I: ~ \ '-6R
`2L' I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`11 / I ' I I
`
`2R
`
`2
`
`

`

`U.S. Patent
`
`Dec.14,2004
`
`Sheet 2 of 5
`
`US 6,831,677 B2
`
`0
`N
`
`0:::
`
`I
`
`I
`
`\
`
`ro
`..c c..
`ro
`~
`~~~~~~~~
`_. :: : : ---_1,_
`---\_
`----:0-\
`(
`::r
`,____
`\
`\ ' - - . .
`a..
`\
`-~~~~---- -c
`
`I
`
`I \
`
`0
`N
`
`L!)
`N
`
`a::
`L!)
`N
`
`.....J
`L!)
`N
`
`----
`
`0
`
`~
`~
`~
`~
`
`3
`
`

`

`• • • •
`
`12R(N)
`
`IMAGE
`10(2)
`• • • •
`
`1 OR(1)
`
`10R(N)
`
`LEFT PANORAMIC IMAGE 11 L
`
`10L(N)
`
`12L(N)
`
`FIG. 2B
`
`d •
`\Jl
`•
`~
`~ ......
`~ = ......
`
`~
`~
`!"l
`'"""'
`~,J;;..
`N c c
`
`,J;;..
`
`'JJ. =(cid:173)~
`~ .....
`
`~
`0 ......,
`Ul
`
`e
`rJ'l
`-..a-..
`00
`~
`1--"
`0..,
`""-l
`""-l
`~
`N
`
`4
`
`

`

`U.S. Patent
`US. Patent
`
`Dec.14,2004
`Dec. 14, 2004
`
`Sheet 4 of 5
`Sheet 4 0f 5
`
`US 6,831,677 B2
`US 6,831,677 B2
`
`Q)
`~
`()
`Q)
`
`C') cu
`E
`\
`
`
`
`/ImageCircle
`
` FIG.3
`
`1/2alph
`
`P
`
`5
`
`

`

`U.S. Patent
`
`Dec.14,2004
`
`Sheet 5 of 5
`
`US 6,831,677 B2
`
`/
`~
`100. CONSTRUCT A STEREOSCOPIC PANORAMIC IMAGE
`PAIR FROM THE IMAGES 10(n) USING A FIXED VALUE FOR
`THE DISTANCE "v" BETWEEN THE CENTER OF THE
`IMAGE PLANE 30 AND THE STRIPS 10R(n) AND 10L(n)
`THAT ARE USED IN THE RESPECTIVE LEFT AND RIGHT
`PANORAMIC IMAGES
`
`.....
`
`/
`
`~r
`
`'
`101. ALIGN THE LEFT AND RIGHT PANORAMIC IMAGES '
`
`11 LAND 11 R COMPRISING THE STEREOSCOPIC
`PANORAMIC IMAGE PAIR SO THAT OBJECTS THAT ARE
`LOCATED AT A VERY LARGE, PREFERABLY INFINITE
`DISTANCE FROM THE CENTER OF ROTATION "0" OF THE
`CAMERA WILL HAVE NO DISPARITY
`.....
`
`'
`
`r
`102. FOR EACH COLUMN IN THE RESPECTIVE LEFT AND
`RIGHT PANORAMIC IMAGES, DETERMINE THE DISPARITY
`AS BETWEEN THE ALIGNED LEFT AND RIGHT
`PANORAMIC IMAGES
`
`,,.
`
`103. DETERMINE A UNITARY DISPARITY VALUE FOR
`EACH COLUMN
`
`~r
`
`104. A FILTER MAY BE APPLIED TO THE UNITARY
`DISPARITY VALUES TO SMOOTH THEM OUT
`
`r
`'-los. USING IMAGES 1 O(n), CONSTRUCT A NEW
`STEREOSCOPIC PANORAMIC IMAGE PAIR IN WHICH THE
`VALUE OF "v," THE DISTANCE BETWEEN THE CENTER OF
`THE IMAGE PLANE AND THE RESPECTIVE STRIPS USED
`IN GENERATING THE LEFT AND RIGHT PANORAMIC
`MOSAIC IMAGES, IS VARIED TO PROVIDE
`(REDETERMINED EFFECTS
`
`'
`
`.....
`
`FIG. 4
`
`6
`
`

`

`US 6,831,677 B2
`
`10
`
`25
`
`2
`by the right eye. In the system described in the Peleg
`application, each panoramic image in the stereoscopic pan(cid:173)
`oramic image pair is generated from images recorded by a
`single camera rotated around an axis located behind the
`5 camera's center of projection. The camera is rotated around
`the rotational axis in a series of steps, and an image is
`recorded at each step. Strips from successive images are
`mosaiced together to form mosaic panoramic images.
`Generally, two strips will be obtained from each image for
`use in generating the mosaic panoramic images, one to the
`right of the center of the image and the other to the left of
`the center of the image, with the strips being generally
`parallel to the axis of rotation of the camera. As described in
`the Peleg application, strips that are mosaiced together for
`the right panoramic image will be the strips that are to the
`15 left of the center of the image (as the image is viewed), and
`strips that are mosaiced together for the left panoramic
`image will be the strips that are to the right of the center of
`the image. The widths of the strips, and the angle between
`steps at which the camera records images, is selected to
`20 ensure that each pair of strips, that is, the pair of strips to the
`right of the center of the image, and the pair of strips to the
`left of the center of the image, can be mosaiced together to
`form respective right and left mosaic panoramic images of
`the stereoscopic panoramic image pair.
`As noted above, two strips will be obtained from each
`image for use in generating the mosaic panoramic images,
`one to the right of the center of the image and the other to
`the left of the center of the image. Since the light rays enter
`the camera through a lens or pinhole, the light rays that are
`30 recorded to define the strip to the right of the center will be
`reflected off surfaces generally directed toward the right (as
`seen by the camera) of in the scene, and the light rays that
`are recorded to define the strip to the left of the center will
`be reflected off surfaces generally directed to the left (as seen
`35 by the camera) of in the scene. It will be appreciated that, in
`an image, the left and right strips will comprise images of
`different objects, but when the strips are mosaiced together
`to form the respective left and right panoramic images, the
`left and right panoramic images will contain images of the
`40 same objects, although from different, that is, left and right,
`viewpoints. When the portions of the left and right mosaic
`panoramic images that are images of the same portion of the
`scene are simultaneously displayed to the respective left and
`right eyes of a viewer, the viewer will be able to observe
`45 stereoscopic depth in the same manner as if he or she had
`viewed the scene with his or her eyes. It will be appreciated
`that the separation between the locations in the camera's
`image plane at which the left and right strips are recorded
`corresponds to the baseline that can provide the disparity
`50 required to view depth stereoscopically. It will be appreci(cid:173)
`ated that, for relatively distant objects, the disparity may be
`insufficient to allow for stereo fusion. Alternatively, depend(cid:173)
`ing on the variation of distance of objects in the scene from
`the camera, the disparity may vary widely.
`
`1
`SYSTEM AND METHOD FOR
`FACILITATING THE ADJUSTMENT OF
`DISPARITY IN A STEREOSCOPIC
`PANORAMIC IMAGE PAIR
`
`INCORPORATION BY REFERENCE
`
`This application claims the benefit of provisional appli(cid:173)
`cation Ser. No. 60/184,589 filed Feb. 24, 2000.
`U.S. patent application Ser. No. 09/396,248, filed Sep. 16,
`1999, in the names of Shmuel Peleg, et al., entitled "System
`and Method for Generating and Displaying Panoramic
`Images and Movies," assigned to the assignee of the present
`application, incorporated herein by reference.
`
`FIELD OF THE INVENTION
`
`The invention relates generally to the field of recording
`and generating images, and more particularly to the genera(cid:173)
`tion and displaying of panoramic images stereoscopically.
`The invention specifically provides a system and method for
`generating a stereoscopic panoramic image pair, comprising
`respective left and right panoramic images, in which dis(cid:173)
`parity is controlled to provide selected effects.
`
`BACKGROUND OF THE INVENTION
`
`When a person views a scene with his or her eyes, the
`disparity, or angular difference in viewing directions by his
`or her eyes for each point in the scene, is interpreted by the
`brain as depth, or distance, to an object in the scene through
`a process referred to as "stereo fusion." The disparity is a
`function of both the distance of the object from the viewer
`and a baseline corresponding to, for example, the distance
`between the viewer's eyes, which results in an angle that the
`brain interprets as distance. Depth can be perceived if the
`disparity is in a particular angular range. If an object is too
`far away from a person, the disparity would be too small for
`depth to be perceived. The maximum stereoscopic range,
`however, can be extended by use of aids such as binoculars,
`which, in addition to magnifying, can also serve to increase
`the baseline. On the other hand, if the object is too close to
`the viewer, the disparity would be too large for stereo fusion.
`Depth can also be perceived from images that have been
`recorded of the scene. In order to observe depth, two images
`are required of the same portion of the scene, one of which
`will be viewed by the left eye and the other of which will be
`viewed by the right eye. A pair of such images, which will
`be referred to as a stereoscopic image pair, comprises two
`images, which have been recorded from two different view(cid:173)
`points. Since the images comprising a stereoscopic image
`pair are recorded from two different viewpoints, they will
`provide a disparity, in a manner similar to the way in which
`the eyes provide disparity if they were viewing the scene
`directly. The amount of the disparity will depend on the
`distance of the points in the scene from the position at which 55
`the images were recorded and the distance between the
`viewpoints at which the images were recorded.
`U.S. patent application Ser. No. 09/396,248, filed Sep. 16,
`1999, in the names of Shmuel Peleg, et al., entitled "System
`and Method for Generating and Displaying Panoramic 60
`Images and Movies" (hereinafter referred to as "the Peleg
`application") describes a system and method for generating
`images comprising a stereoscopic panoramic image pair,
`which can be viewed to provide a stereoscopic panoramic
`view of a scene. A stereoscopic panoramic image pair 65
`comprises two panoramic images, one of which can be
`viewed by the left eye and the other of which can be viewed
`
`SUMMARY OF THE INVENTION
`The invention provides a new and improved system and
`method for generating a stereoscopic panoramic image pair,
`comprising respective left and right panoramic images, in
`which disparity is controlled to provide selected effects,
`which may include, for example, providing a minimum or
`maximum disparity for all objects in the scene to allow for
`stereo fusion therefor, adjusting disparity so to a selected
`range, or other effects that will be apparent to those skilled
`in the art.
`In brief summary, the invention provides, in one aspect, a
`system for generating a stereoscopic panoramic image pair
`
`7
`
`

`

`US 6,831,677 B2
`
`10
`
`3
`comprising left and right panoramic mosaic images for use
`in facilitating stereoscopic viewing of a panoramic scene,
`the system comprising a panoramic image generator and a
`strip separation value generation module. The panoramic
`image generator is configured to mosaic together respective 5
`left and right image strips from respective ones of a plurality
`of images to form the respective left and right panoramic
`images. The panoramic image generator is configured to use
`strip separation values for the respective images to deter(cid:173)
`mine the separation of the respective left and right image
`strips in generating the respective left and right panoramic
`images. The strip separation value generation module is
`configured to generate, for respective ones of said images,
`the strip separation values so as to provide at least one
`selected disparity for objects in the panoramic scene.
`In another aspect, the invention provides a method gen- 15
`erating a stereoscopic panoramic image pair comprising left
`and right panoramic mosaic images for use in facilitating
`stereoscopic viewing of a panoramic scene. In an initial
`panoramic image generation step, respective left and right
`image strips from respective ones of a plurality of images are 20
`mosaiced together to form the respective left and right
`panoramic images. During the initial panoramic image gen(cid:173)
`eration step a predetermined strip separation value being
`used to determine the separation of the respective left and
`right image strips in generating the respective left and right 25
`panoramic images. During an initial disparity value genera(cid:173)
`tion step, respective initial left and right panoramic images
`are processed to generate initial disparity values for projec(cid:173)
`tions of at least some points in the scene. During a separation
`value generation step, final strip separation values are gen- 30
`erated from the initial disparity values. During a final initial
`panoramic image generation step, respective left and right
`image strips from respective ones of said plurality of images
`are mosaiced together to form the respective left and right
`panoramic images. During the final panoramic image gen- 35
`eration step, the final strip separation values are used to
`determine the separation of the respective left and right
`image strips in generating the respective left and right
`panoramic images.
`In yet another aspect, the invention provides a computer
`program product for use in connection with a programmable
`device to facilitate generation of a stereoscopic panoramic
`image pair comprising left and right panoramic mosaic
`images for use in facilitating stereoscopic viewing of a
`panoramic scene, the computer program product comprising
`a device-readable medium having encoded thereon a pan(cid:173)
`oramic image generation module, an initial disparity value
`generation module, a separation value generation module,
`and a control module. The panoramic image generation
`module is configured to enable said programmable device to
`mosaic together respective left and right image strips from
`respective ones of a plurality of images to form the respec(cid:173)
`tive left and right panoramic images, the panoramic image
`generation module being enabled to use a selected strip
`separation value to determine the separation of the respec(cid:173)
`tive left and right image strips in generating the respective
`left and right panoramic images. The initial disparity value
`generation module is configured to enable said program(cid:173)
`mable device to process respective initial left and right
`panoramic images to generate initial disparity values for 60
`projections of at least some points in the scene. The sepa(cid:173)
`ration value generation module is configured to enable said
`programmable device to generate final strip separation val(cid:173)
`ues from the initial disparity values. The control module is
`configured to
`(i) enable programmable device to utilize said panoramic
`image generation module, in the process initially using
`
`4
`a predetermined strip separation value in generating
`initial left and right panoramic images for use by said
`initial disparity value generation module, and
`(ii) after the final strip generation values have been
`generated, enable the programmable device to utilize
`said panoramic image generation module, in the pro(cid:173)
`cess using the final strip separation value in generating
`final left and right panoramic images.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`This invention is pointed out with particularity in the
`appended claims. The above and further advantages of this
`invention may be better understood by referring to the
`following description taken in conjunction with the accom(cid:173)
`panying drawings, in which:
`FIGS. lA and lB are useful in understanding operations
`performed by arrangements for generating and displaying
`stereoscopic panorama image pairs constructed in accor(cid:173)
`dance with the invention;
`FIGS. 2A through 2C are useful in understanding gen(cid:173)
`eration of a stereoscopic panorama image pair;
`FIG. 3 is useful in understanding the relationship between
`the disparity for a point at an arbitrary location relative to an
`image recording medium and other variables related to the
`processes of acquiring an image of point on the image
`recording medium; and
`FIG. 4 depicts a flow chart of operations performed in
`connection with generating a stereoscopic panorama image
`pair in which disparity is automatically controlled to provide
`selected effects.
`
`DETAILED DESCRIPTION OF AN
`ILLUSTRATIVE EMBODIMENT
`
`The invention provides a system and method for auto(cid:173)
`matically controlling the disparity in a stereoscopic pan(cid:173)
`oramic image pair. Before describing the inventive system
`and method, it would be helpful to first describe what a
`40 stereoscopic panoramic image is and generally how the
`system and method described herein controls disparity in
`generating the stereoscopic panoramic image pair. This will
`be done initially in connection with FIGS. lA and lB. With
`reference initially to FIG. lA, that FIG. schematically
`45 depicts an observer, as seen from above, standing vertically
`and observing a point P directly ahead in a scene. The
`observer's eyes are represented by dots 2L and 2R (generally
`identified by reference numeral "2L/R"). The observer sees
`point P by means of rays of light reflected from the point and
`50 directed toward the eyes 2L and 2R along respective rays
`represented by dashed arrows 3L and 3R. The angle "alpha"
`between rays 3L and 3R corresponds to the aforementioned
`disparity, and as long as the distance between point P, on the
`one hand, and the eyes 2L and 2R, on the other hand, is such
`55 that the angle "alpha" is within a large enough range, the
`observer will be able to observe depth, or distance, from the
`observation point to the point P.
`It will be appreciated that, for each eye, since the point P
`is directly ahead of the observer, the angle with which the
`ray 3L, 3R strikes the respective eye, relative to a direction
`straight ahead (as represented by the dashed lines 6L, 6R) of
`the respective eye, will be 'h alpha. For points to the left of
`point P (as shown in FIG. lA), the angle, relative to the
`direction straight ahead, with which the ray 3L strikes the
`65 left eye 2L will be reduced, until it is straight ahead, after
`which the angle will increase. In addition, for points to the
`left of point P, the angle with which the ray 3R strikes the
`
`8
`
`

`

`US 6,831,677 B2
`
`5
`
`5
`right eye 2R will be increased. Contrariwise, for points to the
`right of point P (as shown in FIG. lA), the angle, relative to
`the direction straight ahead, with which the ray 3R strikes
`the right eye 2R will be reduced, until it is straight ahead,
`after which the angle will increase. In addition, for points to
`the right of point P, the angle with which the ray 3L strikes
`the left eye 2L will be increased. In any case, since the
`distance between observer's is fixed, for points farther away
`from the observer, the disparity that he or she will observe
`will be less than these angles whereas, for points closer to
`the observer, the disparity will be greater than these angles.
`The observer typically can see only a small portion of the
`360° panorama around himself or herself. To see more of the
`panorama, the observer will rotate his or her head in, for
`example, the direction indicated by the arrow identified by
`reference numeral 4. Rotation of the head will allow the
`observer to view other points (not shown) in the scene, along
`rays (also not shown) that rotate with him or her. If the
`observer rotates around a full 360°, each eye will revolve
`around the same viewing circle 5.
`It will be apparent from FIG. lA that each the succession
`of images as seen by the observer's two eyes as he or she
`rotates, can be separated into separate sets of images, with
`one set of images being associated with each eye. This will
`be described in connection with FIG. lB. FIG. lB, depicts
`the viewing circle 5 divided into separate viewing circles 5L
`and 5R (generally 5L!R) for the respective left and right
`eyes, with point P being shown in the same position as in
`FIG. lA, with respect to each viewing circle 5L!R, and the
`associated ray 3L(l) and 3R(l), which correspond to rays 3L 30
`and 3R depicted in FIG. lA. Each viewing circle 5L!R also
`depicts other rays, identified by reference numerals
`3L(2), ... , 3L(N) (generally identified by reference numeral
`3L(n)) and 3R(2), ... , 3R(N) (generally identified by
`reference numeral3R(n)) that represent images of points in
`the scene directly ahead of the observer at the respective
`angular position that would strike the respective left and
`right eyes of the observer as he or she rotates in the direction
`represented by arrows 4L and 4R. As with rays 3L and 3R
`(FIG. lA), the angle between rays 3L(n) and 3R(n) repre(cid:173)
`sents the disparity alpha between the viewing directions of
`the left and right panoramic images. In addition, at each
`angular position, the angle with which other points to the left
`or right of the point represented by the respective ray would
`vary in the same manner as described above in connection
`with FIG. lA.
`Further in connection with FIG. lB, to facilitate the
`viewing of a stereoscopic panoramic image of the scene by
`a viewer, the images can be separately recorded and viewed
`by, or otherwise displayed to, the respective eyes of the
`viewer. Thus, if, for example, images are recorded around a
`circle corresponding to viewing circle 5L at successive
`points, in successive direction depicted by rays 3L(l), ...
`3L(N), and the images mosaiced together, and further
`images are recorded around a circle corresponding to view(cid:173)
`ing circle 5R at successive points, in successive direction
`depicted by rays 3R(l), ... 3R(N), and if those images are
`suitably aligned (such that the point of intersection of the
`rays 3L(n) and 3R(n) are viewed in the same relative
`location) and displayed to respective eyes of a viewer, the
`viewer can see a stereoscopic panoramic image of the scene.
`In a similar manner, stereoscopic panoramic image pairs
`can be generated using computer graphics techniques.
`However, instead of the regular perspective projection used
`in conventional image rendering, the panoramic image for
`the left eye will be rendered using rays tangent to a circle
`such as viewing circle 5L, and the panoramic image for the
`
`6
`right eye will be rendered using rays tangent to a circle such
`as viewing circle SR.
`As noted above, when a person views a scene with his or
`her eyes, the disparity is interpreted by the brain as depth. As
`further noted above, disparity is a function of both the
`distance of the point from the viewer and a baseline. In this
`case, the disparity baseline corresponds to the diameter of
`the viewing circle 5. If the diameter of the viewing circle is
`increased, depth can be observed for points P at greater
`10 distances from the eyes. On the other hand, if the diameter
`of the viewing circle is decreased, the distance at which
`depth can be observed will be reduced. Similarly, when
`recording images for a stereoscopic panoramic image pair, if
`the diameters of the viewing circles 5L and 5R are increased
`15 or decreased, the distance at which an observer who
`observes the stereoscopic panoramic image pair will be able
`to observe depth can be increased or decreased. If the
`diameter of the viewing circles 5L and 5R is increased to
`allow the observer to observe depth at greater distances, that
`20 can have the result of exaggerating depth of objects rela(cid:173)
`tively near the camera. Since, in different portions of a scene
`for which a panoramic image is to be generated, objects may
`be located at significantly different distances from the
`camera, it may be desirable to increase the disparity in
`25 regions of the panorama at which objects are farther from the
`camera, and/or decrease the disparity in regions at which
`objects are nearer to the camera. The invention provides an
`arrangement that provides for automatic control of the
`disparity to accomplish that.
`Before describing a system in accordance with the
`invention, it would be useful to describe how panoramic
`mosaic images comprising a stereoscopic panoramic image
`pair are generated. This will be done in connection with
`FIGS. 2A through 2C. FIG. 2A is a top plan view depicting
`35 details of the interior of camera 20, as seen from the top.
`With reference initially to FIG. 2A, camera 20 includes a
`housing 21 having a forward aperture 22, a rear image
`recording medium 23, and a shutter 24. The image recording
`medium 23, which defines an image plane for the camera 20,
`40 may comprise any convenient image recording medium,
`including film, a CCD array, or the like. The camera 20 may
`also include a lens (not shown) in the aperture 22 to facilitate
`focusing of images on the image plane. Alternatively, the
`camera 20 may comprise a pinhole camera, in which case no
`45 lens will be provided.
`The shutter 24 is provided to selectively allow light
`reflected from portions of a scene at which the camera 20 is
`directed is allowed to enter the camera and project upon the
`screen 22 and portions of the image recording medium 23
`50 for recording thereby. Three points from the scene, namely
`points PR, P c and Pv are depicted in FIG. 2A. The scene
`forms part of a panoramic scene which is centered on an axis
`defined by a center of rotation "0" for the camera 20.
`Preferably, the shutter 24 will be closed while the motor 16
`55 moves the camera 20 from one step to the next, thereby to
`block light from the scene from entering the camera 20 and
`exposing the image recording medium 23 during such
`movement, which might otherwise cause blur in the
`recorded image. After the camera 20 has been moved to the
`60 next step stopped moving, the shutter 24 can be opened to
`allow light from the scene to enter the camera and be
`directed rearwardly toward the image recording medium 23.
`After the image recording medium 23 has been appropri(cid:173)
`ately exposed, the shutter can be closed, and the camera 20
`65 stepped to a new orientation, at which point these operations
`can be repeated. Preferably, if the image recording medium
`23 is film, the film will be advanced before the shutter 24 is
`
`9
`
`

`

`US 6,831,677 B2
`
`7
`again opened so as to avoid double-exposure; similarly, if
`the image recording medium 23 is a CCD device, the image
`information can be retrieved and stored and the CCD's
`refreshed as necessary to avoid double exposure. The image
`recorded by the camera 20 at the respective steps will be 5
`independent of each other, so as to avoid multiple exposures.
`As is conventional, light rays different portions of the
`scene 26 will be directed to different portions of the image
`recording medium 23 for recording thereby. For example,
`light rays 26R, 26C and 26L from respective points PR, Pc 10
`and P L of the scene project onto respective points 25R, 25C
`and 25L, as shown in FIG. 2A, of the image recording
`medium 23 for recording thereby. Comparing FIG. 2A with
`FIG. lA, it will be appreciated that the point PR as recorded
`by the point 25R of the image recording medium 23 is from 15
`a direction that corresponds to the direction that an observ(cid:173)
`er's right eye would be viewing that point PRof the scene if
`he or she were looking directly at the point P R of the scene,
`for some angular position of his or her head. Similarly, the
`point P L of the scene as recorded by the point 25L of the 20
`image recording medium is from a direction that corre(cid:173)
`sponds to the direction that the observer's left eye would be
`viewing that point PL of the scene if he or she were looking
`directly at the portion 26L of the scene, for some angular
`position of his or her head. It will further be appreciated that, 25
`preferably the placement of the camera 20 relative to the
`center of rotation "0" will be such that, as the camera 20 is
`rotated, the points on the plane of image recording medium
`23 will revolve through a circle, namely, an image circle,
`and the which maintain the same angular position as rays 30
`26R and 26L will be tangent to an inner viewing circle
`similar to that described above in connection with FIGS. lA
`and lB. Both the image circle and the viewing circle will be
`centered on the center of rotation "0." Since the ray 26C is
`orthogonal to the image plane of the image recording 35
`medium 23, that ray will not be tangent to a viewing circle.
`The generation of left and right panoramic mosaic images
`from the images recorded by the camera 20 will be described
`in connection with FIG. 2B. With reference to FIG. 2B, that
`FIG. depicts a series of images 10(1) through lO(N) 40
`(generally identified by reference numerallO(n)) recorded at
`"N" (where "N" is an integer) successive steps around the
`center of rotation "0." Each image includes a left image
`strip lOL(n) and a right image strip lOR(n) located a distance
`"v" from the center of the image. The left image strips 45
`lOL(l), 10L(2), ... lOL(N) from successive images 10(1),
`10(2), ... lO(N) can be mosaiced together to form the left
`panoramic image llL, as indicated by the arrows 12L(l),
`12L(2), .