`U5006844990B2
`
`(12) United States Patent
`(10) Patent N0.:
`US 6,844,990 B2
`Artonne et al.
`(45) Date of Patent:
`Jan. 18, 2005
`
`(54) METHOD FOR CAPTURING AND
`DISPLAYING A VARIABLE RESOLUTION
`DIGITAL PANORAMIC IMAGE
`
`(56)
`
`References Cited
`U .S. PATENT DOCUMENTS
`
`(75)
`
`Inventors: Jean-Claude Artonne, Montreal (CA);
`Christophe Moustier, Marseilles (FR);
`Benjamin Blane, Montreal (CA)
`
`(73) Assignee: 6115187 Canada Inc., Saint Laurent
`(CA)
`
`( * ) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`
`(21)
`
`(22)
`
`(65)
`
`Appl. N0.: 10/706,513
`
`Filed:
`
`Nov. 12, 2003
`Prior Publication Data
`
`US 2004/0136092 Al Jul. [5, 2004
`
`Related US. Application Data
`
`(63) Continuation of application No. PCT/FR02/01588, filed on
`May 10, 2002.
`
`Foreign Application Priority Data
`(30)
`May 11. 2001
`(FR)
`............................................ 01 06261
`
`Int. Cl.7 .......................... G02B 13/06; (30213 13/18
`(51)
`(52) US. Cl.
`........................................ 359/725; 359/718
`(58) Field of Search ................................. 359/718, 719,
`359/725, 728
`
`4/1976 Fisher et al.
`3953111 A
`Ishii et a1.
`3/1999
`5,880,896 A
`Inoue
`2/2000
`6,031,670 A
`6,333,826 B1 * 12,/2001 Charles ...................... 359/725
`6,449,103 B1 +
`9/2002 Charles ...................... 359/725
`FOREIGN PATENT DOCUMENTS
`0 695 085 A1
`1 004 915 A1
`WO 00/42470 A1
`
`1/1996
`5/2000
`7/2000
`
`EP
`EP
`W0
`
`* cited by examiner
`Primary ExaminerfiScott J . Sugarm an
`(74) Attorney, Agent, or FirmfiAkin Gump Strauss Hauer
`& Feld, LIP
`
`(57)
`
`ABSTRACT
`
`A method for capturing a digital panoramic image includes
`projecting a panorama onto an image sensor by means of a
`panoramic objective lens. The panoramic objective lens has
`a distribution function of the image points that is not linear
`relative to the field angle of the object points of the
`panorama, such that at least one zone of the image obtained
`is expanded While at least another zone of the image is
`compressed. When a panoramic image obtained is then
`displayed, correcting the non-linearity of the initial image is
`required and is performed by means of a reciprocal function
`of the non-linear distribution function of the objective lens
`or by means of the non—linear distribution function.
`
`26 Claims, 11 Drawing Sheets
`
`lng
`
`
`lmgl
`
`IDl
`
`LGE Exhibit 1001
`LGE v. ImmerVision
`
`Page 1 of 27
`
`

`

`US. Patent
`
`Jan. 18, 2005
`
`Sheet 1 0f 11
`
`US 6,844,990 B2
`
`
`
`LGE Exhibit 1001
`LGE v. ImmerVision
`
`Page 2 of 27
`
`

`

`US. Patent
`
`Jan. 18, 2005
`
`Sheet 2 0f 11
`
`US 6,844,990 B2
`
`
`
`0'7
`
`ch
`
`10 20 30 40 50 60 70 80 90
`
`
`I-w-fiwa
`T
`T
`"'1
`-_‘l
`I
`I
`
`O
`
`Fig. 4B
`
`LGE Exhibit 1001
`LGE v. ImmerVision
`
`Page 3 of 27
`
`

`

`US. Patent
`
`Jan. 18, 2005
`
`Sheet 3 0f 11
`
`US 6,844,990 B2
`
`PRIOR ART
`
`E93
`
`
`
`_l?_'_i_g. 7B
`
`10' 20' 30' 40' 50' 60' 70‘ 80' 90'
`
`a
`
`LGE Exhibit 1001
`LGE v. ImmerVision
`
`Page 4 of 27
`
`

`

`US. Patent
`
`Jan. 18, 2005
`
`Sheet 4 0f 11
`
`US 6,844,990 B2
`
`dr
`Pdl(dr=0.777)
`
`
`
`Pd(dr=0.3)
`
`Fd2
`
`0.3
`
`
`
`70°
`
`90°
`
`
`
`
`
`30°
`
`70"
`
`90°
`
`LGE Exhibit 1001
`LGE v. ImmerVision
`
`Page 5 of 27
`
`

`

`US. Patent
`
`Jan. 18, 2005
`
`Sheet 5 0f 11
`
`US 6,844,990 B2
`
`
`
`LGE Exhibit 1001
`LGE v. ImmerVision
`
`Page 6 of 27
`
`

`

`US. Patent
`
`Jan. 18, 2005
`
`Sheet 6 0f 11
`
`US 6,844,990 B2
`
`
`
`
`
`LGE Exhibit 1001
`LGE v. ImmerVision
`
`Page 7 of 27
`
`

`

`US. Patent
`
`Jan. 18, 2005
`
`Sheet 7 0f 11
`
`US 6,844,990 B2
`
`S] — Acquisition
`
`— Taking a panoramic image by means of a still digital camera
`or a digital video camera equipped with a panoramic lens
`having a non-linear distribution function F d
`
`82 — Transfer of the image file into a computer
`
`- Transfer of
`microcomputer
`- Storage in the auxiliary storage (optional)
`
`image
`
`file
`
`(image
`
`the
`
`
`
`disk)
`
`into a
`
`53 -Linearisation of the image disk
`
`— Transfer of the image points of the initial image disk into a
`second virtual image disk comprising more image points than
`the initial image disk, by means of the fimction Fd'
`Obtaining a linear image disk
`
`
`S4 — Digitisation
`
`— Transfer of the image points of the second image disk into a
`system of axes OXYZ in spherical coordinates Obtaining a
`panoramic image in a hemisphere
`
`SS — Interactive display
`
`- Determination of the image points of an image sector to be
`displayed
`— Display of the image sector on a display window
`- Detection of the user’s actions on a screen pointer or any
`other control means,
`- Detection of the user’s actions on keys
`enlargement,
`- Modification of the sector displayed (sliding the image
`sector displayed on the surface of the hemisphere and/or
`shrinking/expanding the image sector displayed)
`
`image
`
`for
`
`LGE Exhibit 1001
`LGE v. ImmerVision
`
`Page 8 of 27
`
`

`

`US. Patent
`
`Jan. 18, 2005
`
`Sheet 8 0f 11
`
`US 6,844,990 B2
`
`mm, W)
`
`
`
`13(an P)” DZ)
`(i, j)
`
`
`
`DW
`
`HS
`
`LGE Exhibit 1001
`LGE v. ImmerVision
`
`Page 9 of 27
`
`

`

`US. Patent
`
`Jan. 18, 2005
`
`Sheet 9 of 11
`
`US 6,844,990 B2
`
`Fig. 14
`
`81 — Acquisition
`
`- Taking a panoramic image by means of a still digital camera
`or a digital video camera equipped with a panoramic lens
`
`52 - Transfer of the image file into a computer
`
`having a non-linear distribution function Fd
`
`file
`
`(image
`
`disk)
`
`into
`
`a
`
`
`
`- Transfer of
`microcomputer
`- Storage in the auxiliary storage (Optional)
`
`the
`
`image
`
`S3' — Interactive display with implicit correction of the
`non—linearity of the initial image
`
`A - Determination of the colour of the points E(i, j) of an
`image sector to be displayed using the points p(pu, pv) of the
`image disk:
`
`I- determination of the coordinates Ex, By, 52 in the
`coordinate system OXYZ of each point E(i, j) of the sector to
`be displayed,
`2- determination of the coordinates Px, Py, P2 of points P of
`the hemisphere corresponding to the points E(i, j),
`3-~ calculation of the coordinates, in the coordinate system
`O'UV of the image disk, of the points p(pu, pv) corresponding
`to the points P of the hemisphere, by means of the function
`Fd,
`
`B - Presentation of the image sector in a display window,
`C - Detection of the user’s actions on a screen pointer or any
`other control means,
`D - Detection of the user’s actions on enlargement keys,
`E - Modification of the image sector displayed (moving
`and/or shrinking/expanding the image sector)
`
`LGE Exhibit 1001
`LGE v. ImmerVision
`
`Page 10 of 27
`
`

`

`US. Patent
`
`Jan. 18, 2005
`
`Sheet 10 01'11
`
`US 6,844,990 B2
`
`
`
`339. 1'7
`
`LGE Exhibit 1001
`LGE v. ImmerVision
`
`Page 11 of 27
`
`

`

`US. Patent
`
`Jan. 18, 2005
`
`Sheet 11 0f 11
`
`US 6,844,990 B2
`
`
`
`LGE Exhibit 1001
`LGE v. ImmerVision
`
`Page 12 of 27
`
`

`

`US 6,844,990 B2
`
`1
`METHOD FOR CAPTURING AND
`DISPLAYING A VARIABLE RESOLUTION
`DIGITAL PANORAMIC IMAGE
`
`CROSS—REFERENCE TO RELATED
`APPLICATIONS
`
`This application is a continuation of International Appli-
`cation No. PCT/FR02’01588, filed May 10, 2002 the dis—
`closure of which is incorporated herein by reference.
`BACKGROUND OF THE INVENTION
`
`10
`
`The present invention relates to obtaining digital pan-
`oramic images and displaying panoramic images on com-
`puter screens.
`FIG. 1 represents a classical device allowing a digital
`panoramic image to be produced and presented on a com-
`puter screen. The device comprises a digital camera 1
`equipped with a panoramic objective lens 2 of the “fish-eye”
`type, having an angular aperture on the order of 180°. The ,
`camera 1 is connected to a computer 5, such as a micro-
`computer for example, equipped with a screen 6. The
`connection to the microcotnputer 5 may be permanent,
`when, for example, the camera 1 is a digital video camera,
`or temporary, when, for example,
`the camera 1 is a still
`digital camera equipped with an image memory, the con-
`nection then being carried out at the time the image files are
`to be transferred into the microcomputer.
`FIG. 2 schematically represents the appearance of a
`panoramic image 3 obtained by means of the panoramic
`objective lens 2. The round appearance of the image is
`characteristic of the axial symmetry of panoramic objective
`lenses and the image has dark edges 4 that will subsequently
`be removed. This digital panoramic image is delivered by
`U4Ur
`the camera 1 in the form of a computer file containing image ,
`points coded RGBA arranged in a two-dimensional table,
`“R” being the red pixel of an image point, “G” the green
`pixel, “B” the blue pixel, and “A” the Alpha parameter or
`transparency. The parameters R, G, B, A are generally being
`coded on 8 bits.
`
`30
`
`40
`
`2
`central point 0' of the system of axes O'X‘Y'Z, which defines
`with the center 0" of the image sector 12, a direction O‘O”
`called “viewing direction”.
`In order to avoid the image sector displayed 12 having
`geometrical distortions unpleasant
`for the observer,
`the
`classical panoramic objective lenses must have a distribution
`function of the image points according to the field angle of
`the object points of a panorama that is as linear as possible.
`Therefore, if two points A', B', situated on the same meridian
`of the hemisphere 11, and the corresponding points A, B on
`the image disk 10 are considered,
`the ratio between the
`angles (A’O‘Z) and (B‘O'Z) must be equal
`to the ratio
`between the distances 0A and OB on the image disk.
`Due to this property of linearity of a classical panoramic
`objective lens, image points corresponding to object points
`having an identical field angle form concentric circles C10,
`C20 .
`.
`. C90 on the image disk 10, as represented in FIG,
`4A. Classically, “field angle of an object point” means the
`angle of an incident light ray passing through the object
`point considered and through the center of the panorama
`photographed, relative to the optical axis of the objective
`lens. The field angle of an object point can be between 0 and
`90° for an objective lens having an aperture of 180°,
`Therefore,
`the circle C10 is formed by the image points
`corresponding to object points having a field angle of 10°,
`the circle C20 is formed by image points corresponding to
`object points having a field angle of 20°, etc., the circle C90
`being formed by the image points having a field angle of
`90°.
`
`FIG. 4B represents the shape of the distribution function
`ch of a classical panoramic objective lens, which deter-
`mines the relative distanee Ch of an image point in relation
`to the center of the image disk according to the field angle
`ax of the corresponding object point. The relative distance dr
`is between 0 and 1 and is equal to the distance of the image
`point in relation to the center of the image divided by the
`radius of the image disk. The ideal form of the function ch
`is a straight line of gradient K:
`
`dr=ch(ot)=Kor
`
`The image file is transferred into the [microcomputer 5
`which transforms the initial image into a three-dimensional
`digital image, then presents the user with a sector of the
`three-dimensional image in a display window 7 occupying
`all or part of the screen 6.
`FIG. 3 schematically shows classical steps of transform—
`ing the two-dimensional panoramic image into a panoramic
`image offering a realistic perspective effect. After removing
`he black edges of the image, the microcomputer has a set of
`image points forming an image disk 10 of center 0 and axes
`OX and OY. The image points of the image disk are
`ransferred into a three—dimensional space defined by an
`orthogonal coordinate system of axes O‘X'Y'Z, the axis O'Z
`veing perpendicular to the plane of the image disk. The
`ransfer is performed by a mathematical function imple-
`mented by an algorithm executed by the microcomputer, and
`eads to obtaining a set of image points referenced in the
`coordinate system O‘X'Y'Z. These image points are for
`example coded in spherical coordinates RGBA((|),0), (I) being
`he latitude and 6 the longitude of an image point. The angles
`(1) and 0 are coded in 4 to 8 bytes (IEEE standard). These
`image points form a hemisphere 11 when the panoramic
`objective lens used has an aperture of 180°, otherwise a
`Jor‘tion of a hemisphere. The microcomputer thus has a
`virtual image in the shape of a hemisphere one sector 12 of
`which, corresponding to the display window 7, is presented
`on the screen (FIG. 1) considering that the observer is on the
`
`
`
`in which the constant K is equal to 0.111 degree'1 (1,’90°).
`This technique of displaying a digital panoramic image
`sector on a computer screen has various advantages, par—
`ticularly the possibility of “exploring” the panoramic image
`by sliding the image sector presented on the screen to the
`left, the right, upwards or downwards, until the limits of the
`panoramic image are reached. This technique also allows
`complete rotations of the image to be carried out when two
`complementary digital images have been taken and supplied
`to the microcomputer, the latter thus reconstituting a corn-
`plete panoramic sphere by assembling two hemispheres.
`Another advantage provided by presenting a panoramic
`image on screen is to enable the observer to make enlarge—
`ments or zooms on parts of the image. The zooms are
`performed digitally, by shrinking the image sector displayed
`and expanding the distribution of the image points on the
`pixels of the screen.
`Various examples of interactive panoramic images can be
`found on the Web. Reference could be made in particular to
`the central site “http://www.panoguide.com” (“The Guide to
`Panoramas and Panoramic Photography”) which gives a
`full overview of all the products available to the public to
`produce these images. Software progams allowing digital
`panoramic photographs to be transformed into interactive
`panoramic images are offered to the public in the form of
`downloadable programs or CD—ROMs available in stores.
`
`LAm
`
`60
`
`LGE Exhibit 1001
`LGE v. ImmerVision
`
`Page 13 of 27
`
`

`

`US 6,844,990 B2
`
`
`
`3
`
`
`Despite the various advantages that this technique for
`
`displaying digital images 0 ers,
`the digital enlargements
`have the disadvantage of being limited by the resolution of
`the image sensor used when aking the initial image and the
`resolution of an image sensor is generally much lower than
`that of a classical photograph. Therefore, when the enlarge-
`ment increases, the granulosity of the image appears as the
`limits of the resolution of the image sensor are being
`reached.
`is well known to
`it
`To overcome this disadvantage,
`proceed with pixel interpolations so as to delay the appari—
`tion of the blocks of color which betray the limits of the
`resolution of the sensor. However,
`this method only
`improves the appearance of the enlarged image sector and
`does not in any way increase the definition. Another obvious
`solution is to provide an image sensor with a high resolution,
`higher than the resolution required to present an image
`sector without enlargement, so that there is a remaining
`margin of definition for zooms. However, this solution is
`expensive as the cost price of an image sensor rapidly rises
`with the number of pixels per unit of area.
`Some attempts have been made to improve the quality of
`the enlargements, by changing the optical properties of the
`panoramic objective lenses themselves. Thus, US. Pat. No.
`5,710,661 teaches capturing a panoramic image with two
`overlocking objective lenses using a set of mirrors. Afirst set
`of mirrors provides an overall view, and a mobile central
`
`mirror provides a detailed view on a determined zone of the
`
`
`panorama. However, this solution does not 0 er the same
`flexibility as digital zooms, particularly when the image is
`not displayed in real time, as the observer no longer has the
`possibility of choosing the image portion tha he wants to
`enlarge once the photograph has been taken.
`BRIEF SUMMARY OF TH]: INVENTION
`
`
`
`10
`
`30
`
`40
`
`U4en
`invention comprises a method ,
`the present
`Therefore,
`
`
`allowing the physical limits of image sensors to be circum—
`
`
`vented and the definition 0 ‘ered by digital enlargements
`concerning certain parts of a digital panoramic image to be
`improved, without the need to increase the number of pixels
`per unit of area of an image sensor or to provide an
`overlooking optical enlargement system in a panoramic
`objective lens.
`The present invention is based on the observation that, in
`several applications, only certain zones of a panoramic
`image are of a practical
`interest and are likely to be
`expanded by the observer by me ans of a digital zoom. Thus,
`in applications such as video surveillance,
`videoconferencing, visio—conferencing, a panoramic camera
`can be installed against a wall or on the ceiling and there is
`generally no reason to make enlargements on the zones of
`the panoramic image corresponding to the wall or
`the
`ceiling. Similarly, as part of a videoconference performed by
`means of a panoramic camera, the most interesting zone is
`generally situated at a specific place situated towards the
`center of the image (in the case of individual use) or on the
`edges of the image (in the case of collective use or visio-
`conferencing). Furthermore, when used for recreation and
`leisure, most panoramic images comprise parts that are less
`interesting than others, such as the parts representing the sky
`or a ceiling for example, the most useful part generally being
`in the vicinity of the center of the image.
`Therefore, the present invention is based on the premise
`that a panoramic image has some zones that are not very
`useful and that can tolerate a reasonable definition to the
`benefit of other zones of the image.
`On the basis of this premise,
`the idea of the present
`invention is to produce panoramic photographs by means of
`
`LAm
`
`60
`
`4
`a panoramic objective lens that is not linear, which expands
`certain zones of the image and compresses other zones ofthc
`image. The technical ellect obtained is that the expanded
`zones of the image cover a number of pixels of the image
`sensor that is higher than if they were not expanded, and thus
`benefit from a better definition. By choosing an objective
`lens that expands the most useful zones of an image (which
`depend on the intended application), the definition is excel-
`lent in these zones and the definition is mediocre in the zones
`of lesser importance.
`Thus, the present invention proposes a method for cap—
`turing a digital panoramic image, by projecting a panorama
`onto an image sensor by means of a panoramic objective
`lens, in which the panoramic objective lens has an image
`point distribution function that is not linear relative to the
`field angle of object points of the panorama, the distribution
`function having a maximum divergence of at least 110%
`compared to a linear distribution function, such that the
`panoramic image obtained has at
`least one substantially
`expanded zone and at least one substantially compressed
`zone.
`
`According to one embodiment, the objective lens has a
`non-linear distribution function that is symmetrical relative
`to the optical axis of the objective lens, the position of an
`image point relative to the center of the image varying
`according to the field angle of the corresponding object
`point.
`According to one embodiment, the objective lens expands
`the center of the image and compresses the edges of the
`image.
`According to one embodiment, the objective lens expands
`the edges of the image and compresses the center of the
`image.
`According to one embodiment, the objective lens c0111-
`presses the center of the image and the edges of the image,
`and expands an intermediate zone of the image located
`between the center and the edges of the image.
`According to one embodiment, the objective lens com-
`prises a set of lenses forming an apodizer.
`According to one embodiment, the set of lenses forming
`an apodizer comprises at least one aspherical lens.
`According to one embodiment, the set of lenses forming
`an apodizer comprises at least one diffractive lens.
`According to one embodiment, the objective lens com—
`prises a set of mirrors comprising at least one distorting
`mirror.
`
`The present invention also relates to a method for dis—
`playing an initial panoramic image obtained in accordance
`with the method described above, comprising a step of
`correcting the non-linearity of the initial image, performed
`by means of a reciprocal function of the non-linear distri-
`bution function of the objective lens or by means of the
`non-linear distribution function.
`
`the step of correcting
`According to one embodiment,
`comprises a step of transforming the initial image into a
`corrected digital
`image comprising a number of image
`points higher than the number ofpixels that the image sensor
`comprises.
`According to one embodiment, the method comprises a
`step of calculating the size of the corrected image, by means
`of the reciprocal function of the distribution function, so that
`the resolution of the corrected image is equivalent to the
`most expanded zone of the initial
`image, and a step of
`scanning each image point of the corrected image, searching
`for the position of a twin point of the image point on the
`
`LGE Exhibit 1001
`LGE v. ImmerVision
`
`Page 14 of 27
`
`

`

`US 6,844,990 B2
`
`6
`BRIEF DESCRIPTION OF THE SEVERAL
`VIEWS OF THE DRAWINGS
`
`5
`initial image and allocating the color of the twin point to the
`image point of the corrected image.
`According to one embodiment, the initial image and the
`corrected image comprise an image disk.
`According to one embodiment, the method comprises a
`step of transferring the image points of the corrected image
`into a three-dimensional space and a step of presenting one
`sector of the three—dimensional image obtained on a display
`means.
`
`10
`
`According to one embodiment, the method comprises a
`step of determining the color of image points of a display
`window, by projecting the image points of the display
`window onto the initial image by means of the non-linear
`distribution function, and allocating to each image point of
`the display window the color of an image point that is the
`closest on the initial image.
`the projection of the
`According to one embodiment,
`image points of the display window onto the initial image
`comprises a step of projecting the image points of the
`display window onto a sphere or a sphere portion, a step of '
`determining the angle in relation to the center of the sphere
`or the sphere portion of each projected image point, and a
`step of projecting onto the initial image each image point
`arojected onto the sphere or the sphere portion, the projec—
`ion being performed by means of the non-linear distribution
`unction considering the field angle that each point to be
`rojeeted has in relation to the center of the sphere or the
`sphere portion.
`The present invention also relates to a panoramic objec-
`ive lens comprising optical means for projecting a pan-
`orama into an image plane of the objective lens, the pan-
`oramic objective lens having an image point distribution
`tmction that is not linear relative to the field angle of obj ect
`joints of the panorama, the distribution function having a
`
` maximum divergence of at least 110% compared to a linear
`
`
`
`distribution function, such that a panoramic image obtained
`3y means of the objective lens comprises at
`least one
`substantially expanded zone and at least one substantially
`compressed zone.
`According to one embodiment, the panoramic objective
`ens has a non—linear distribution function that is symmetri—
`cal relative to the optical axis of the objective lens,
`the
`aosition of an image point relative to the center of an image
`obtained varying according to the field angle of the corre-
`sponding object point,
`According to one embodiment, he panoramic o3jective
`ens expands the center of an image and compresses the
`edges of the image.
`According to one embodiment, he panoramic oajcctivc
`ens expands the edges of an image and compresses the
`center of the image.
`According to one embodiment, he panoramic oajcctivc
`ens compresses the center of an image and the edges of the
`image, and expands an intermediate zone of the image
`ocated between the center and the edges of the image.
`According to one embodiment, he panoramic oojective
`ens comprises a set of lenses forming an apodizer.
`
`According to one embodiment, tie set Of lenses orming
`an a odizer com rises at least one as herical lens.
`p
`.
`p
`.
`p
`.
`According to one embodiment, tie set of lenses orming
`an apodizer comprises at least one diffractive lens.
`According to one embodiment, he panoramic oajective
`lens comprises polymethacrylate lenses.
`According to one embodiment, he panoramic owjective
`lens comprises a set of mirrors comprising at least one
`distorting mirror.
`
`The foregoing summary, as well as the following detailed
`description of preferred embodiments of the invention, will
`be better understood when read in conjunction with the
`appended drawings, For the purpose of illustrating the
`invention, there are shown in the drawings embodiments
`which are presently preferred.
`It should be understood,
`however, that the invention is not limited to the precise
`arrangements and instrumentalities shown.
`In the drawings:
`FIG. 1 described above represents a system for displaying
`a digital panoramic image on a screen;
`FIG. 2 described above represents a panoramic image
`before it is processed by a computer;
`FIG. 3 described above shows a classical method for
`transforming a two-dimensional panoramic image into a
`three-dimensional digital panoramic image;
`FIGS. 4A and 4B described above show the linearity of a
`classical panoramic objective lens;
`FIGS. 5 and 6 show one aspect of the method according
`to the present invention and respectively represent a distri-
`bution of image points obtained with a classical panoramic
`objective lens and a distribution of image points obtained
`with a non-linear panoramic objective lens according to the
`present invention;
`FIGS. 7A and 7B show a first example of non-linearity of
`a panoramic objective lens according to the present inven-
`tion;
`FIG. 8 shows a second example of non-linearity of a
`panoramic objective lens according to the present invention;
`FIG. 9 shows a
`third example of non-linearity of a
`panoramic objective lens according to the present invention;
`FIG. 10 represents a system for displaying a digital
`panoramic image by means of which a method for correcting
`the panoramic image according to the present invention is
`implemented;
`FIG. 11 schematically shows a first embodiment of the
`correction method according to the present invention;
`FIG. 12 is a flow chart describing a method for displaying
`a panoramic image incorporating the lirst correction method
`according to the present invention;
`FIG. 13 schematically shows a second embodiment of the
`correction method according to the present invention;
`FIG. 14 is a flow chart describing a method for displaying
`a panoramic image incorporating the second correction
`method according to the present invention;
`FIG. 15 is a cross-section of a first embodiment of a
`non-linear panoramic objective lens according to the present
`invention;
`FIG. 16 is an exploded cross—section of a system of lenses
`present in the panoramic objective lens in FIG. 15;
`FIG. 17 is a side view of a lens present in the panoramic
`objective lens in FIG. 15; and
`FIG 18 is the diagram of a second embodiment of a
`.
`'
`.
`.
`.
`.
`50 non-linear panoramlc objective lens accordmg to the present
`invention
`’
`DETAILED DESCRIPTION OF THE
`INVENTION
`AiCompression/Expansion of an Initial Image
`FIG. 5 schematically represents a classical system for taking
`panoramic shots, comprising a panoramic objective lens 15
`
`30
`
`U4U!
`'
`
`40
`
`LAm
`
`LGE Exhibit 1001
`LGE v. ImmerVision
`
`Page 15 of 27
`
`

`

`US 6,844,990 B2
`
`7
`of optical axis OZ and a digital image sensor 17 arranged in
`the image plane of the objective lens 15. Here, four object
`points a, b, c, (1 will be considered that belong to a panorama
`PM located opposite the objective lens and respectively
`having angles of incidence (XI, (1.2, —(1.2, —(1.1. As explained
`in the preamble, the field angle of an object point is the angle
`that an incident light ray passing through the object point
`considered and through the center of the panorama PM,
`marked by a point “p” on FIG. 5, has relative to the optical
`axis OZ of the objective lens. In this example, the angle 0.1
`is equal to two times the angle (12. On the image sensor 17,
`image points a‘, b', c‘, d‘ corresponding to the object points
`a, b, c, d are located at distances from the center of the image
`respectively equal to d1, d2, —d2, —d1. As the distribution of
`the image points according to the field angle of the object
`points is linear With a classical panoramic objective lens, the
`distances d1 and d2 are linked by the following relation:
`d1/o1=d2/a2
`
`10
`
`As the angle otl is here equal to 2&2, it follows that:
`d1=2d2
`
`8
`and compressing another part of the image, the part to be
`expanded and the part to be compressed can be chosen
`according to the intended application, by producing several
`types of non-linear objective lenses and by choosing an
`objective lens suited to the intended application. Depending
`on the intended application,
`the most useful part of a
`panoramic image may be located in the center of the image,
`on the edge of the image, in an intermediate zone situated
`between the center and the edge of the image, etc.
`FIGS. 7A—7B, 8 and 9 show three examples of non-linear
`distribution filnetions according to the present invention.
`The distribution function shown in FIGS. 7A and 7B
`corresponds to the example in FIG. 6, that is a panoramic
`objective lens that expands the image in the center. FIG. 7A
`represents equidistant concentric circles C10, C20, .
`.
`. , C90
`present on an image disk, each circle being formed by image
`points corresponding to object points having the same field
`angle. The circle C10 is formed by the image points corre—
`sponding to object points having a field angle of 10°, the
`circle C20 is formed by image points corresponding to
`object points having a field angle of 20°, etc. By comparing
`FIG. 7A with FIG. 4A described in the preamble, it appears
`that the circles C10 and C20 are further from the center of
`the image and further from each other than the circles C10
`and C20 obtained With a classical objective lens, While the
`circles C30 to C90 are closer to each other. This panoramic
`image thus has an expanded zone in the center and a
`compressed zone on the edge of the image disk.
`FIG. 413 represents the curve of the corresponding distri-
`bution function Fd1. The classical
`linear distribution
`function, expressed by ch=K(x and in the form of a straight
`line of gradient K, is also represented as a guide mark (the
`constant K being equal to 1.60 for an objective lens having an
`aperture of 180°, i.e., a gradient of 0.111 per degree). The
`field angle or of the object points is represented on the X—axis
`and is between 0 and 90°. The relative distance dr of an
`image point in relation to the center of the image disk is
`represented on the Y-axis and is between 0 and 1. The curve
`of the function Fdl has a higher gradient than the straight
`line ch for angles 0. of between 0 and 20°, then a lesser
`gradient after 20° and 11p to 90°. Ahigh gradient means an
`expansion of the image and a low gradient means a com-
`pression of the image.
`As demonstrated in this example, the curve Fd1 has a
`point of maximum divergence Pd at the angle a=20°. “Point
`of maximum divergence” refers to the image point Pd((x) at
`which the greatest gap in relative distance dr in relation to
`a corresponding point Pdl(o.) on the linear distribution
`straight line Ka can be observed. In this example, the point
`Pd(ot=20°) has a relative distance dr equal to 0.5 relative to
`the center of the image while the corresponding point
`Pdl(a=20°) on the linear curve ch has a relative distance dr
`of 0.222. The maximum divergence DIVmax of the distri-
`bution function Fd] according to the present invention can
`be calculated by a formula of the type:
`
`DIVmax %=[[dr(Pd)—dr(PdZ)]/[dr(Pd1 )]]*100
`
`i.e.:
`
`DIVmax %=[[dr(PzI)—K*EL(P(I)]/[ ”u(P¢I)]]*100
`
`In which dr(Pd) is the relative distance in relation to the
`center of the point of maximum divergence Pd, dr(Pdl) is the
`relative distance in relation to the center of the correspond-
`ing point on the linear distribution straight line ch, o.(Pd)
`being the abscissa of the point Pd, i.e., the field angle of the
`corresponding object point.
`
`LGE Exhibit 1001
`LGE v. ImmerVision
`
`Page 16 of 27
`
` aresent invention, the image sensor 17 being arranged in the '
`
`As is well known by those skilled in the art, the term
`“linearity” here refers to a ratio of proportionality between
`he distance of an image point measured relative to the
`center of the image and the field angle of the corresponding
`object point. The notion of “linearity” in the field of pan-
`oramic objective lenses is therefore difierent from that
`wrevailing in the field of paraxial optics (in the vicinity of the
`optical axis) when the conditions of Gauss are met.
`FIG. 6 represents a system for taking shots of the same
`ype as above, but in which the classical panoramic objective
`ens 15 is replaced by an objective lens 18 ac