111111
`
`1111111111111111111111111111111111111111111111111111111111111
`US007477284B2
`
`(12) United States Patent
`Peleg et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 7,477,284 B2
`Jan. 13,2009
`
`(54) SYSTEM AND METHOD FOR CAPTURING
`AND VIEWING STEREOSCOPIC
`PANORAMIC IMAGES
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`(75)
`
`Inventors: Shmuel Peleg, Mevaseret Zion (IL);
`Moshe Ben-Ezra, Jerusalem (IL); Yael
`Pritch, Jerusalem (IL)
`(73) Assignee: Yissum Research Development
`Company of the Hebrew University of
`Jerusalem, Jerusalem (IL)
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.c. 154(b) by 894 days.
`(21) Appl. No.: 09/861,859
`(22) Filed:
`Apr. 19, 2001
`Prior Publication Data
`(65)
`
`( *) Notice:
`
`Nov. 11,2004
`US 2004/0223051 Al
`Jul. 21, 2005
`US 2005/0157166 A9
`Related U.S. Application Data
`
`(60) Provisional application No. 60/230,562, filed on Aug.
`30, 2000, provisional application No. 60/217,152,
`filed on Jul. 6, 2000, provisional application No.
`601198,381, filed on Apr. 19,2000, provisional appli(cid:173)
`cation No. 60/113,962, filed on Dec. 28, 1998, provi(cid:173)
`sional application No. 60/102,720, filed on Sep. 29,
`1998, provisional application No. 60/100,721, filed on
`Sep. 17, 1998.
`Int. Cl.
`(2006.01)
`H04N 13/00
`(52) U.S. Cl. ........................................................ 348/53
`(58) Field of Classification Search . ... ... ... ..... .... 348/36,
`348/42,51, 158,218.1,98,44,46,48,49,
`348/52,53; 3821154,232,294,299,284,
`382/268; 352/57,69-70; 396/324; H04N 13/00
`See application file for complete search history.
`
`(51)
`
`4,134,644 A
`5,455,689 A *
`5,594,498 A *
`5,649,032 A *
`5,657,073 A
`5,721,585 A
`5,764,871 A *
`5,825,466 A
`5,963,664 A *
`5,986,668 A *
`5,991,444 A *
`
`111979 Marks et al ................. 350/132
`1011995 Taylor et al ................. 358/450
`111997 Fraley ........................ 348/158
`7/1997 Burt et al .................... 3821284
`8/1997 Henley et al.
`2/1998 Keast et al .................... 348/36
`6/1998 Fogel ......................... 345/427
`1011998 Lo et al ........................ 355/33
`1011999 Kumar et al.
`............... 382/154
`1111999 Szeliski et al. .............. 345/634
`1111999 Burt et al .................... 3821232
`
`(Continued)
`
`FOREIGN PATENT DOCUMENTS
`
`EP
`
`0361297
`
`4/1990
`
`(Continued)
`
`OTHER PUBLICATIONS
`
`Peleg, Schmuel et aI., "Stereo Panorama with a Single Camera",
`IEEE, 1999, pp. 395-401.
`
`(Continued)
`
`Primary Examiner-Tung Vo
`(74) Attorney, Agent, or Firm-Browdy and Neimark,
`P.L.L.c.
`
`(57)
`
`ABSTRACT
`
`Method and apparatus for generating images of a scene from
`image data of the scene and displaying the images to provide
`a sense of depth. In some embodiments of the method and
`apparatus the generated images are mosaics.
`
`55 Claims, 6 Drawing Sheets
`
`51a~
`
`MOSAIC
`IMAGE "a"
`
`~--~~~--------------~
`
`:
`
`51 blL ______ --"--________ I_M_AG_E_'_'b_" __ ---"
`
`MOSAIC
`
`51d~
`
`518l
`
`d1d2 d3
`
`I~
`~;
`
`ele2~
`
`MOSAIC
`IMAGE "d"
`
`MOSAIC
`IMAGE"e"
`
`1
`
`Petition for Inter Partes Review
`of U.S. Pat. No. 7,477,284
`IPR2013‐00219
`EXHIBIT
`Sony‐
`
`

`

`US 7,477,284 B2
`Page 2
`
`u.s. PATENT DOCUMENTS
`6,075,905 A *
`6/2000 Herman et al. .............. 382/284
`6,078,701 A *
`6/2000 Hsu et al.
`................... 382/294
`6,083,353 A *
`7/2000 Alexander, Jr.
`............. 202/158
`10/2000 Nalwa
`6,141,145 A
`6,275,206 Bl *
`8/2001 Tsai et al. ..................... 345/88
`6,278,480 Bl*
`8/2001 Kurahashi et al. ............. 348/59
`6,377,294 B2 *
`4/2002 Toyofuku et al.
`............. 348/36
`6,434,280 Bl*
`8/2002 Peleg et al.
`................. 382/299
`6,435,969 Bl*
`8/2002 Tanakaetal. ................. 463/44
`6,512,892 Bl*
`112003 Montgomery et al. ....... 396/326
`6,522,787 Bl*
`2/2003 Kumar et al.
`............... 382/268
`6,532,036 Bl*
`3/2003 Peleg et al.
`................... 348/36
`6,532,298 Bl*
`3/2003 Cambier et al. ............. 382/117
`6,535,650 Bl*
`3/2003 Poulo et al. ................. 382/284
`6,643,413 Bl* 1112003 Shumetal. ................. 382/284
`6,665,003 Bl* 12/2003 Peleg et al.
`................... 348/36
`6,717,608 Bl*
`4/2004 Mancuso et al. .............. 348/36
`6,750,860 Bl*
`6/2004 Shumetal. ................. 345/419
`6,831,677 B2 * 12/2004 Peleg et al.
`................... 348/36
`6,850,210 Bl*
`2/2005 Lipton et al. ................... 345/6
`6,956,573 Bl* 10/2005 Bergen et al. ............... 345/473
`200110010546 Al *
`8/2001 Chen .......................... 348/218
`200110020976 Al
`9/2001 Peleg et al.
`
`1112001 Peleg et al.
`200110038413 Al
`200110043264 Al * 1112001 Sinclairetal. ................ 348/36
`
`FOREIGN PATENT DOCUMENTS
`
`EP
`EP
`IL
`JP
`JP
`JP
`JP
`WO
`WO
`
`0989436 A2
`0989436
`WO 98/34195
`11 146425
`11146425
`11 164326
`11164326
`WO 99117555
`WO 99118725
`
`3/2000
`3/2000
`* 8/1998
`5/1999
`5/1999
`6/1999
`6/1999
`4/1999
`4/1999
`
`OTHER PUBLICATIONS
`
`Ishiguro, H. et al.; "Correspondence, Omni-Directional Stereo";
`IEEE Transactions on Patern Analysis and Machine Intelligence, vol.
`14, Feb. 1992;pp.257-262.
`Gluckman, J. et al.; "Real-Time Omnidirectional and Panoramic
`Stereo"; Proc. OfImage Understanding Workshop, IUW98;1998; 5
`Pages; Retrieved from the Internet: <http://cis.poly.edu/-gluckman/
`papers/uw98a.pdf> .
`* cited by examiner
`
`2
`
`

`

`u.s. Patent
`
`Jao.13,2009
`
`Sheet 1 of 6
`
`US 7,477,284 B2
`
`FIG.l
`11C)
`118)
`Stereo data Stereo cia
`source
`source
`
`11A
`
`souroe
`
`11N
`
`source
`
`10~
`
`Distribution
`Channel
`
`(13
`
`Viewing
`device
`12A~
`
`Vie-wing
`device
`
`VieViTing
`device
`
`VieViling
`device
`12M
`
`FIG.2
`
`/23
`I
`
`,
`
`3
`
`I
`
`i L---20
`
`j
`
`l
`DO
`I
`100
`------------------------------
`
`I
`
`2 _I
`
`Y,-
`2
`
`L.-- 24
`/22
`L
`4 1001
`DO
`DO
`DDI
`
`I
`~
`
`3
`
`

`

`u.s. Patent
`
`Jao.13,2009
`
`Sheet 2 of 6
`
`US 7,477,284 B2
`
`FIG.3
`30
`
`34
`
`Capture
`u.mt
`
`I
`
`Dlsplay, n Lo(!al
`
`(mono/stere(i) !
`r - ' 5 ___
`
`Locol
`. Display
`i (rn¢no!stere¢)
`
`33A
`
`338
`
`'------3-.-J _ - -Jj F2
`Prooessing
`mut
`
`OPR CTRL
`PANEL 22
`
`'----__ :~~.~_,I -_J
`
`Local
`memory
`mtit
`
`I
`~31
`I
`
`i
`
`FIG., 4
`---------------------------------, -'"
`./' 40",-
`I
`~i Receiver
`11
`
`\\
`
`\
`
`b
`
`r--,
`
`i
`
`'j
`
`Decoder
`
`!
`
`,
`
`(Vi'ireless or
`non -wir1!l eS!1)
`
`;
`
`L
`I
`i
`j
`
`d
`
`Direction
`Control
`unit
`
`~
`
`1 /42
`
`Display
`
`L....---_uru_'t -j
`
`--------------------------------
`
`4
`
`

`

`u.s. Patent
`
`Jao.13,2009
`
`Sheet 3 of 6
`
`US 7,477,284 B2
`
`FIG.5
`
`50(1)
`
`51a
`
`f
`
`51b
`-
`
`51c
`
`51d
`
`51e
`
`,
`"
`f!
`
`<.
`~~
`>l1
`
`i ~ ~ ~
`f~ ---
`W.
`j
`.
`}j I
`l'
`iii
`rn
`I I iii I
`
`50(2)
`•••
`
`50(3)
`
`I , I
`
`.!<
`
`II , if
`I,
`
`it:
`
`~:j :#
`
`I
`
`MOSAIC
`IMAGE na"
`
`MOSAIC
`IMAGE nb"
`
`MOSAIC
`IMAGE "e"
`
`MOSAIC
`IMAGE lTd ll
`
`MOSAIC
`IMAGE "e"
`
`5
`
`

`

`u.s. Patent
`
`Jao.13,2009
`
`Sheet 4 of 6
`
`US 7,477,284 B2
`
`6761 (2)
`62(3)(A)------ ' 7Le
`62(2)(B)----, -~:-----B...-.
`62(1)(C)--~_-'1 l
`
`62(3)(B)--- .
`
`62(3)(C)---
`
`62(2)(A)--
`
`62(2)(C)--
`
`62(1)(A)--'
`
`62(1)(B)--;
`
`FIG. 6
`
`l'lticu,arLens C ~
`"
`
`63(B)
`
`1m
`A
`~r-"""""'~
`. ,
`
`60--'
`
`61
`
`61(1)
`
`63(A)
`
`100 --::t
`
`FIG. 7
`
`6
`
`

`

`u.s. Patent
`
`Jao.13,2009
`
`Sheet 5 of 6
`
`US 7,477,284 B2
`
`FIG.8
`
`F /G.10
`
`15
`
`110
`
`116A--=
`9
`~ 116S1
`
`131
`
`35
`,
`136A~ / =--136C
`133
`130~ 1368
`
`1448
`
`142
`
`F IG.11
`
`'-----,---\--,-..-..J 45
`146A-= 143
`
`140~ 14681
`
`~154C 2548
`
`..... 154A
`152A
`
`0'
`
`1520
`
`, /'
`
`151
`
`55
`
`156A-= r =--156C
`
`1568
`
`~ 53
`
`150~
`
`F /G.12
`
`164C
`
`164A
`
`65
`166A---cs 9 =--166C
`163
`166B1
`100~ F I G.13
`
`7
`
`

`

`u.s. Patent
`
`Jan. 13,2009
`
`Sheet 6 of 6
`
`US 7,477,284 B2
`
`FIG.9
`
`V124B
`_ ..... --'--;:1 24C 122A~tr_' _1_2_2_8 _-'--1_24_A ____
`o
`
`120~
`
`170~
`
`171K
`
`F IG.14
`
`8
`
`

`

`US 7,477,284 B2
`
`1
`SYSTEM AND METHOD FOR CAPTURING
`AND VIEWING STEREOSCOPIC
`PANORAMIC IMAGES
`
`RELATED APPLICATIONS
`
`2
`in the art. In one technique, described in loshua Gluckman, et
`aI., "Real-Time Omnidirectional And Panoramic Stereo,"
`DARPA Image Understanding Workshop, 1998, two onmidi(cid:173)
`rectional cameras, vertically displaced along a common axis,
`record panoramic images of the surrounding scene. Since the
`cameras are displaced, the pair of images recorded by the
`cameras, when considered in combination, will provide depth
`information for objects in the scene surrounding the cameras.
`However, since the displacement is vertical, the recorded
`10 images are inappropriate for human stereo panoramic percep(cid:173)
`tion.
`
`SUMMARY OF THE INVENTION
`
`This application is a continuation-in-part of U.S. applica(cid:173)
`tion Ser. No. 091792,638 filed On Feb. 24, 2001, U.S. appli(cid:173)
`cation Ser. No. 091726,198 filed on Nov. 29, 2000 now U.S.
`Pat. No. 6,795,109, and U.S. application Ser. No. 09/396,248
`filed on Sep. 16, 1999, now U.S. Pat. No. 6,665,003. U.S.
`application Ser. No. 091726,198 is a continuation-in-part of
`U.S. application Ser. No. 09/396,248. This application also
`claims the benefit under 119(e) of U.S. provisional applica(cid:173)
`tion 601198,381 filed onApr. 19,2000, U.S. provisional appli - 15
`cation 60/217,152 filed luI. 6, 2000 and U.S. provisional
`application 60/230,562 filed on Aug. 30, 2000.
`
`INCORPORATION BY REFERENCE
`
`U.S. patent application Ser. No. 09/396,248, filed Sep. 16,
`1999, in the names of Shmuel Peleg, et aI., entitled "System
`and Method for Generating and Displaying Panoramic
`Images and Movies,"(hereinafter referred to as "the Peleg I
`patent application") assigned to the assignee of the present
`application, incorporated herein by reference.
`U.S. patent application Ser. No. 091726,198, filed Nov. 29,
`2000, in the names of Shmuel Peleg, et aI., entitled "Stereo
`Panoramic Camera Arrangements For Recording Panoramic
`Images Useful InA Stereo Panoramic Image Pair," (herein- 30
`after referred to as "the Peleg II patent application") assigned
`to the assignee of the present application, incorporated herein
`by reference.
`U.S. patent application Ser. No. 091792,638, filed Feb. 24,
`2001, in the names of Shmuel Peleg, et aI., entitled "System 35
`And Method For Facilitating The Adjustment Of Disparity In
`A Stereoscopic Panoramic Image Pair,"(hereinafter referred
`to as "the Peleg III patent application") assigned to the
`assignee of the present application, incorporated herein by
`reference.
`
`FIELD OF THE INVENTION
`
`The invention provides a new and improved system and
`method for generating and displaying a stereoscopic pan(cid:173)
`oramic image set, comprising respective at least two pan(cid:173)
`oramic images of a scene, each having a different viewing
`direction, for contemporaneous viewing by respective left
`20 and right eyes of a viewer to provide an apparent stereoscopic
`image of the scene to the viewer.
`In brief summary, the invention provides an arrangement
`for recording images for use in generating and utilizing
`images comprising a stereoscopic image set. The arrange-
`25 ment includes at least one stereoscopic data source, at least
`one utilization device and a distribution charmel. The at least
`one stereoscopic data source records images from which a
`stereoscopic image set can be generated. The utilization
`device is provided to facilitate viewing, printing, or otherwise
`utilizing the stereoscopic image set. The distribution chaunel
`facilitates transferring information between the stereoscopic
`data source and the utilization device, the information includ(cid:173)
`ing the images as recorded by the stereoscopic data source or
`the stereoscopic image set itself. If a stereoscopic image set is
`to comprise a plurality of mosaic images, the mosaic images
`may be generated by the stereoscopic data source, the image
`utilization device and/or the distribution charmel.
`A number of stereoscopic data sources of diverse configu(cid:173)
`rations are disclosed, including fixed and moving mirrors,
`40 prisms, and lenses, which may be used in the stereoscopic
`data source in the stereoscopic image arrangement.
`
`The invention relates generally to the field of recording and
`generating images, and more particularly to the generation, 45
`displaying and printing of panoramic images stereoscopi(cid:173)
`cally. The invention specifically provides a system and
`method for generating and displaying a stereoscopic pan(cid:173)
`oramic image set, comprising respective at least two pan(cid:173)
`oramic images of a scene, each having a different viewing 50
`direction, for contemporaneous viewing by respective left
`and right eyes of a viewer to provide an apparent stereoscopic
`image of the scene to the viewer.
`
`BACKGROUND OF THE INVENTION
`
`55
`
`Panoramic images are images of a scene having a wide
`• Panoramic images may be
`field of view, up to a full 3600
`recorded using a wide angled lens, a mirror, or the like,
`providing a wide field of view. Panoramic images having a 60
`wider field of view can be generated by, for example, record(cid:173)
`ing a plurality of images around a particular point and, using
`conventional mosaicing techniques, generating a single
`mosaic image. Panoramic images may also be generated of
`simulated scenes using conventional computer graphics tech- 65
`niques. Stereoscopic panoramic images can also be generated
`from images using various techniques known to those skilled
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`This invention is pointed out with particularity in the
`appended claims. The above and further advantages of this
`invention may be better understood by referring to the fol(cid:173)
`lowing description taken in conjunction with the accompany-
`ing drawings, in which:
`FIG. 1 schematically depicts a stereoscopic panoramic
`image arrangement for recording, generating and displaying
`stereoscopic panoramic images, constructed in accordance
`with the invention;
`FIG. 2 schematically depicts an exterior plan view of an
`illustrative stereoscopic data source for use in connection
`with the arrangement depicted in FIG. 1;
`FIG. 3 depicts a functional block diagram of the stereo(cid:173)
`scopic data source depicted in FIG. 2;
`FIG. 4 depicts a functional block diagram of an illustrative
`viewing device depicted in FIG. 3;
`FIG. 5 is useful in understanding the operations performed
`by the stereoscopic panoramic image arrangement depicted
`in FIG. 1 in connection with generating a stereoscopic pan(cid:173)
`oramic set;
`FIG. 6 is useful in understanding the arrangement for gen(cid:173)
`erating and displaying lenticular prints; and
`
`9
`
`

`

`US 7,477,284 B2
`
`3
`FIGS. 7 through 14 depict illustrative image recording
`arrangements that may be used in connection with the stereo(cid:173)
`scopic data source described in connection with FIGS. 2 and
`3.
`
`DETAILED DESCRIPTION OF AN
`ILLUSTRATIVE EMBODIMENT
`
`FIG. 1 schematically depicts a stereoscopic panoramic
`image arrangement 10 for recording, generating and display(cid:173)
`ing stereoscopic panoramic images, constructed in accor(cid:173)
`dance with the invention. With reference to FIG. 1, the ste(cid:173)
`reoscopic panoramic image arrangement 10 includes one or
`more stereoscopic data sources lIA through lIN (generally
`identified by reference numeral lIn ) and one or more viewing
`devices 12A through 12M (generally identified by reference
`numeral 12m). Generally, each stereoscopic data source lIn
`records images and generates therefrom at least one stereo(cid:173)
`scopic panoramic image set comprising a set of panoramic
`images. Each stereoscopic data source lIn can also transmit
`respective stereoscopic panoramic image sets that it generates
`through a distribution channel 13 to one or more of the view(cid:173)
`ing devices 12m for viewing, generation of lenticular prints,
`or other disposition as will be apparent to those skilled in the
`art.
`An illustrative stereoscopic data source lIn will be
`described below in connection with FIGS. 2 and 3. Generally,
`the stereoscopic data source lIn will include an image
`recording arrangement, an image processing arrangement
`and a commnnication arrangement. The image recording
`arrangement records one or more images from which a ste(cid:173)
`reoscopic panoramic image set is generated. The image
`recording arrangement may be similar to those described in
`the Peleg I or Peleg II patent applications. If the image record(cid:173)
`ing arrangement is similar to that described in the Peleg I
`patent application, or to those described in the Peleg II patent
`application for which the arrangement is rotated and/or trans(cid:173)
`lated to facilitate recording of images from which a pan(cid:173)
`oramic image is generated, at least the image recording
`arrangement records a series of images as it is rotated around
`an axis comprising a center of rotation, translated along a
`path, or any combination of rotation and translation. As the
`image recording arrangement is rotated and/or translated, it
`records a series of images from which the image processing
`arrangement generates at least one set of panoramic images
`comprising a stereoscopic panoramic image set as described
`in the Peleg I patent application, or as described below in
`connection with FIG. 5. Generally, in generating a stereo(cid:173)
`scopic panoramic image set, the image recording arrange(cid:173)
`ment will record a series of images. For each panoramic
`image in a stereoscopic panoramic image set, the image pro(cid:173)
`cessing arrangement will generate the panoramic image by
`mosaicing together strips from successive ones of the images.
`The strips obtained from the respective images may all have
`the same displacement from the center of the respective
`images, as described in the Peleg I patent applications, or they
`may have different displacements, as described in the Peleg
`III patent application. Alternatively, as noted above, the pan(cid:173)
`oramic images may be generated as described below in con(cid:173)
`nection with FIG. 5.
`Alternatively, the image recording arrangement may be
`similar to those described in the Peleg II patent application.
`For several of the image recording arrangements that are
`described in the Peleg II patent application, the respective
`arrangements may also be rotated and/or translated to facili(cid:173)
`tate recording of a series of images from which strips may be
`mosaiced together to generate a set of panoramic images
`
`4
`comprising a stereoscopic panoramic image set. On the other
`hand, for others of the image recording arrangement
`described in the Peleg II patent application, the respective
`arrangements may be those including omni-cameras, which
`arrangements can directly record panoramic images for a
`stereoscopic panoramic image set without requiring mosaic(cid:173)
`ing.
`The communication arrangement facilitates transmission
`of the stereoscopic panoramic image set to the distribution
`10 channel 13 for distribution to one or more of the viewing
`devices 12m. Generally, information defining the stereo(cid:173)
`scopic panoramic image set transferred by the commnnica(cid:173)
`tion arrangement will be in digital form, and the communi(cid:173)
`cation arrangement may include any arrangement that
`15 facilitates transfer of digital data between two devices, which
`may be at the same location or at different locations. Illustra(cid:173)
`tive commnnication arrangements include, for example, a
`direct connection, such as a wire, cable or optical fiber con(cid:173)
`nection, a wireless connection, any other arrangement for
`20 facilitating the transfer of information in digital form, or any
`combination thereof. A direct connection may include, for
`example, a direct network connection, an indirect connection
`to a network through, for example, a computer, a connection
`through the public switched telephony network, and the like.
`25 A wireless connection may include, for example, a radio
`connection, a cellular telephone connection, an infrared con(cid:173)
`nection, and the like
`An illustrative viewing device will be described below in
`connection with FIG. 4. Generally, the viewing device will
`30 include an a communication arrangement, an image storage
`arrangement and a display, printer or other device for gener(cid:173)
`ating images for viewing by a viewer. The viewing device's
`communication arrangement facilitates reception of the ste(cid:173)
`reoscopic panoramic image set from the distribution channel
`35 13. Generally, information defining the stereoscopic pan(cid:173)
`oramic image set received by the commnnication arrange(cid:173)
`ment will be in digital form, and the communication arrange(cid:173)
`ment may include a direct connection, such as a wire, cable or
`optical fiber connection, a wireless connection, such as a
`40 cellular telephone connection, or any other arrangement for
`facilitating the transfer of information in digital form.
`The image storage arrangement stores the digital informa(cid:173)
`tion comprising a stereoscopic panoramic image set as
`received by the viewing device's communication arrange-
`45 ment from the distribution channel 13 for later display.
`Depending on the amonnt of information that can be stored in
`the image storage arrangement and the amount of information
`comprising an stereoscopic panoramic image set, the image
`storage arrangement may have the capacity to store informa-
`50 tion comprising only one stereoscopic panoramic image set,
`or it may have the capacity to store information comprising a
`plurality of stereoscopic panoramic image sets.
`The display, if provided, will display at least a portion of
`one or more of the images comprising a stereoscopic pan-
`55 oramic image set. It will be appreciated that, if the display
`displays at least a portion of one of the images, the image,
`when viewed, will not be stereoscopic. On the other hand, if
`the display displays at least a portion of both images using, for
`example, a lenticular lens, as will be described below in
`60 connection with FIG. 6, the images, when viewed, will be
`stereoscopic. If a printer is provided, the printer can generate
`hardcopy prints of one or more of the images comprising a
`stereoscopic panoramic image set, and may generate, for
`example, lenticular prints, as will be described below in con-
`65 nection with FIG. 6, which can be viewed by a viewer using
`a lenticular lens to provide a stereoscopic image of the scene
`represented by the stereoscopic panoramic image set.
`
`10
`
`

`

`US 7,477,284 B2
`
`5
`It will be appreciated that both each stereoscopic data
`source lIn and viewing device 12m may also include control
`arrangements for facilitating control thereof by a respective
`operator. For example, if the stereoscopic data source lIn
`includes an image recording arrangement that is to be rotated
`and/or translated, it can include a control to enable the image
`recording arrangement to rotate and/or translate. In addition,
`the stereoscopic data source lIn can include controls to allow
`an operator to actuate the image recording arrangement to
`facilitate recording of images of a scene, to enable generation
`by the image processing arrangement of a stereoscopic pan(cid:173)
`oramic image set for the scene, and to enable the communi(cid:173)
`cation arrangement to transmit an stereoscopic panoramic
`image set to the distribution channel 13 for transfer to one or
`more of the viewing devices.
`Similarly, the respective viewing device 12m can include a
`control to facilitate receiving of stereoscopic panoramic
`image sets from the distribution channel 13 and storage by the
`image storage arrangement. In addition, the respective view(cid:173)
`ing device 12m can include a control to facilitate selection of
`a stereoscopic panoramic image set for display, selection of
`display mode, and selection of a portion of the stereoscopic
`panoramic image set to be displayed. The viewing device 12m
`may be able to selectively display, as well as at least portions
`of a stereoscopic panoramic image set, at least portions of
`individual panoramic images of the stereoscopic panoramic
`image set as respective display modes, and the selection of the
`display mode can facilitate display of one or more of the
`panoramic images of the selected stereoscopic panoramic
`image set. In addition, the viewing device 12m can include a
`control to facilitate selection of a respective portion of an
`stereoscopic panoramic image set that is to be displayed, or
`the portion of an individual image from the stereoscopic
`panoramic image set that is to be displayed of the viewing
`device 12m provides such display modes.
`As noted above, a viewing device 12m can, instead of or in
`addition to providing a display for displaying an stereoscopic
`panoramic image set with a lenticular lens to provide for
`stereoscopic viewing, provide a printer or similar device for
`generating a hardcopy print of at least a portion of one or both
`of the images comprising a stereoscopic panoramic image set,
`or a hardcopy print of at least a portion of the stereoscopic
`panoramic image set that, when viewed through a lenticular
`lens, would provide a stereoscopic panorama. A viewing
`device 12m that provides a printer or similar device can
`include controls for enabling it to generate such hardcopy
`prints.
`With this background, an illustrative stereoscopic data
`source lIn will be described in connection with FIGS. 2 and
`3. FIG. 2 depicts an exterior plan view of the illustrative 50
`stereoscopic data source lIn, and FIG. 3 depicts a functional
`block diagram of the illustrative stereoscopic data source.
`Generally, the illustrative stereoscopic data source lIn is a
`portable device that can be, for example, hand-held by an
`operator, mounted on a vehicle, or the like, and rotated and/or 55
`translated to facilitate recording of the series of images from
`which the stereoscopic panoramic image set is generated. In
`addition, the transfers the digital infonnation comprising the
`stereoscopic panoramic image set over a cellular telephone
`commnnication link. With reference to FIG. 2, the illustrative 60
`stereoscopic data source lIn includes a housing 20 that
`houses and supports a video camera 21, an operator control
`panel 22, a display 23 and an antenna 24. The video camera 21
`provides the image recording arrangement described above,
`and the operator control panel 22 comprises a plurality of 65
`controls that may be actuated to enable the video camera 21 to
`perform predetennined operations. The controls can be
`
`6
`implemented in a number of ways, including, for example,
`pushbuttons that may be depressed by the operator to actuate
`the respective control.
`After the operator energizes the stereoscopic data source
`lIn by actuating a respective control on the operator control
`panel 22, he or she can directs the video camera 21 in a
`particular direction. The display 23 can display an image
`indicating what the video camera 21 can record. The operator
`can enable the video camera by actuating a control on the
`10 operator control panel 22. As the operator actuates the image
`recording control, and rotates and/or translates the stereo(cid:173)
`scopic data source lIn, he or she can enable the video camera
`21 to record a series of images from which a stereoscopic
`panoramic image set can be generated. After the stereoscopic
`15 data source lIn has recorded a series of images, the operator
`can release the image recording control, at which point the
`video camera 21 can stop recording of images.
`After the stereoscopic data source lIn has recorded a series
`of images, the operator can actuate another control on the
`20 operator control panel to enable the stereoscopic data source
`lIn to generate the panoramic images comprising the stereo(cid:173)
`scopic panoramic image set. After at least one of the pan(cid:173)
`oramic images has been generated, the operator can actuate
`another control on the operator control panel 22 to enable the
`25 stereoscopic data source lIn to display at least a portion the
`generated panoramic image in the display 23, with the portion
`being selectable by use of the same or another control on the
`operator control panel 22. Similarly, after at least two pan(cid:173)
`oramic images of the stereoscopic panoramic image set have
`30 been generated, the operator can actuate a control on the
`operator control panel 22 to enable the stereoscopic data
`source lIn to display them in a manner so that they can be
`viewed stereoscopically. The stereoscopic data source lIn
`may display the stereoscopic panoramic image set in a man-
`35 ner such that any a number of kinds of appliances may be
`required to facilitate stereoscopic viewing, including, for
`example, a lenticular lens, glasses with polarized lenses or
`lenses of different color, or other appliances as will be appre(cid:173)
`ciated by those skilled in the art, with the stereoscopic data
`40 source lIn displaying the stereoscopic panoramic image set
`in a corresponding manner.
`In addition, after the stereoscopic data source lIn has
`generated the panoramic images comprising the stereoscopic
`panoramic image set, it can transmit them to the distribution
`45 channel for distribution to respective ones of the viewing
`devices 12m. As noted above, the illustrative stereoscopic
`data source lIn makes use of a cellular telephone connection
`to transferinfonnation to the distribution channel 13. Accord-
`ingly, the operator can actuate one or more other controls on
`the operator control panel 22 to enable the stereoscopic data
`source lIn to establish a cellular telephone call to the distri-
`bution channel 13 through a cellular provider (not shown). In
`that case, the stereoscopic data source lIn can establish a
`cellular link through the antenna 24.
`FIG. 3 depicts a functional block diagram of the stereo(cid:173)
`scopic data source lIn described above in connection with
`FIG. 2. With reference to FIG. 3, the stereoscopic data source
`lIn includes an image capture unit 30, a local memory unit
`31, a processing unit 32, one or more local displays 33A,
`33B, ... , and a communication unit 34, as well as the operator
`control panel 22. The image capture unit 30, local memory
`unit 31 and processing unit 32 together form the video camera
`21 described above in connection with FIG. 2, and the local
`displays 33A, 33B, ... fonn the display 23 described above in
`connection with FIG. 2. The capture unit 30 will include, for
`example, an image sensor, aperture, lenses, and/or the like to
`facilitate capturing or acquiring of the respective images. The
`
`11
`
`

`

`US 7,477,284 B2
`
`7
`image sensor may be any of a number of conventional image
`sensors, including, for example, CCD (charge coupled
`devices), film, and the like.
`Generally, after the operator energizes the stereoscopic
`data source lIn by actuating a respective control on the
`operator control panel 22, the processing unit can control the
`capture unit 30, in particular the image sensor, to begin
`receiving images, and send them to a local display, for
`example, local display 33A, for display to the operator. When
`the operator actuates the image recording control on the
`operator control panel 22, the processing unit 32, in addition
`to sending images from the image sensor to the local display
`33A for display to the operator, will send the images to the
`local memory unit 31 for storage. In addition, when the opera(cid:173)
`tor actuates the stereoscopic panoramic image set generation
`control on the operator control panel 22, the processing unit
`32 will generate the panoramic images comprising the ste(cid:173)
`reoscopic panoramic image set and can display them in the
`local displays 33A, 33B, ....
`Furthermore, when the operator actuates the stereoscopic
`panoramic image set transmission control on the operator
`control panel 22, the processing unit 32 enables the pan(cid:173)
`oramic images comprising the stereoscopic panoramic image
`set to be transmitted through the communication unit 34 to the
`distribution channel 13. In that operation, since, as noted
`above, the stereoscopic data source lIn makes use of a cel(cid:173)
`lular telephone links to facilitate transfer of stereoscopic pan(cid:173)
`oramic image sets to the distribution chaunel13, the process(cid:173)
`ing unit 32 can initiate a cellular telephone call to the
`distribution chaunel13 and, after the distribution channel 13
`responds, cooperate with the distribution channel 13 to facili(cid:173)
`tate transmission of the stereoscopic panoramic image set to
`the distribution channel 13. The processing unit 32 can use a
`predetermined telephone number in initiating the cellular
`telephone call, or it can make use of a telephone number
`provided by the operator through the operator control panel
`22. Accordingly, the operator control panel