US006128145A
`6,128,145
`(114) Patent Number:
`United States Patent 55
`Nagaoka
`[45] Date of Patent:
`Oct. 3, 2000
`
`
`[54]
`
`IMAGE PICK-UP DEVICE, IMAGE DISPLAY
`DEVICE AND INFORMATION RECORDING
`MEDIUM COMPRISING A FISHEYE LENS
`
`[75]
`
`Inventor: Tooru Nagaoka, Nagano-Pref, Japan
`
`[73] Assignees: FIT Corporation, Nagano-Pref.; Rios
`Corporation; Advanet, Inc., both of
`Okayama-Pref., all of Japan
`
`[21] Appl. No.: 09/300,972
`[22]
`Filed:
`Apr. 28, 1999
`
`Foreign Application Priority Data
`[30]
`’
`Nov. 25, 1998
`[JP]
`Japan oo.cteseeeeceeee 10-350751
`7
`[S51]
`Tint. C1ceceeeseseseecnnecneeeneeeneess G02B 13/04
`;
`[52] US. CD. ieeeesssessenecnseeneesnes 359/749; 359/672
`.
`[58] Field of Search 0.0. 359/749, 750-753,
`359/680-0682, 672-675
`
`[56]
`
`References Cited
`U.S. PATENT DOCUMENTS
`~~
`8/1983 Hayashida .......cece 359/662
`4,400,063
`
`6/1992 Heggetal. ....
`5,121,099
`3/1996 Jamieson ....... ee eeeereeeeeeeeeeeee 359/355
`5,502,592
`Primary Examiner—Georgia Epps
`Assistant Examiner—Jordan M. Schwartz
`Attorney, Agent, or Firm—McAulay Nissen Goldberg &
`Kiel, LLP
`[57]
`..
`. -
`An imageis picked up by a camera comprising a fisheye lens
`having a relationship of h=n-f-tan(6/m), wherein h is the
`height of an image of a subject at a certain point, f is the
`focal distance of the fisheyelens, @ is a field angle, m has a
`value of 1.62mS3, and n has a value of m-0.4Sn=m+0.4,
`and the image data of which is output from the camera, is
`converted into a plane image by an image data processing
`unit, and this converted image is then output to a monitor
`unit. Preferably, n and m both equal 2.
`
`ABSTRACT
`
`14 Claims, 9 Drawing Sheets
`
`
`
`DIO Di2 pi6
`
`LGE Exhibit 1004
`LGE v. ImmerVision
`Page 1 of 15
`
`LGE Exhibit 1004
`LGE v. ImmerVision
`Page 1 of 15
`
`

`

`U.S. Patent
`
`Oct. 3, 2000
`
`Sheet 1 of 9
`
`6,128,145
`
`
`
`
`
`CAMERA
`
`
`
`
`
`IMAGE DATA
`
`PROCESSING
`UNIT
`
`ZS
`MONITOR UNIT
`
`FIG.1
`
`LGE Exhibit 1004
`LGE v. ImmerVision
`Page 2 of 15
`
`LGE Exhibit 1004
`LGE v. ImmerVision
`Page 2 of 15
`
`

`

`U.S. Patent
`
`Oct. 3, 2000
`
`Sheet 2 of 9
`
`6,128,145
`
`ce
`&
`
`Sa
`tm co
`ats
`~
`Cian
`n
`x pS ze
`
`w
`eE-
`—
`6 2 FL
`—_
`Te
`—_
`Ae 4a = 8
`ne OA
`a
`=
`N
`x
`ga
`
`—
`a
`
`|
`
`=
`A
`iN
`Nt
`
`Q
`
`y
`
`—_
`=
`
`—_
`a
`
`tt
`
`a
`
`©
`ae
`x
`2-8
`
`Xi
`
`=r
`a
`
`
`
`SF BS
`
`N
`
`oF
`
`
`
`Sle\ yO
`—S
`=NN
`
`
`
`LGE Exhibit 1004
`LGE v. ImmerVision
`Page 3 of 15
`
`LGE Exhibit 1004
`LGE v. ImmerVision
`Page 3 of 15
`
`

`

`U.S. Patent
`
`Oct. 3, 2000
`
`Sheet 3 of 9
`
`6,128,145
`
`
`
`(91aueL-3e~Fgyey
`
` 06OsO0209OSOFOF02Ol guls+40I(2/Q)UIS-4Z02
`
`(2/9)uel«42a
`G-sG2|4os
`(2)@3TONYGTalsVSOldg<———
`
`(€/9uel-t¢
`
`(wu)
`
`LGE Exhibit 1004
`LGE v. ImmerVision
`Page 4 of 15
`
`LGE Exhibit 1004
`LGE v. ImmerVision
`Page 4 of 15
`
`

`

`U.S. Patent
`
`Oct. 3, 2000
`
`Sheet 4 of 9
`
`6,128,145
`
`[ 94 tan(@/1.6)
`
`[ (=C1)
`
`60 50
`
`2.44 tan(Q/2)
`
`1.24 tan(@/1.6)
`
`1.6f tan(@/2)
`
`LGE Exhibit 1004
`LGE v. ImmerVision
`Page 5 of 15
`
`0
`
`10
`
`29
`
`30
`
`40
`
`50
`
`60
`
`70
`
`80 go
`
`FIG.3B
`
`LGE Exhibit 1004
`LGE v. ImmerVision
`Page 5 of 15
`
`

`

`U.S. Patent
`
`Oct. 3, 2000
`
`Sheet 5 of 9
`
`6,128,145
`
`
`h = 2f - sin (@/2)ee
`
`FIG.4C
`
`FIG.4D
`
`LGE Exhibit 1004
`LGE v. ImmerVision
`Page6 of 15
`
`LGE Exhibit 1004
`LGE v. ImmerVision
`Page 6 of 15
`
`

`

`U.S. Patent
`
`Oct. 3, 2000
`
`Sheet 6 of 9
`
`6,128,145
`
`
`
` 2
`
`FIG.6
`
`LGE Exhibit 1004
`LGE v. ImmerVision
`Page7 of 15
`
`LGE Exhibit 1004
`LGE v. ImmerVision
`Page 7 of 15
`
`

`

`U.S. Patent
`
`Oct. 3, 2000
`
`Sheet 7 of 9
`
`6,128,145
`
`S1
`
`CAMERA
`
`S52
`
`53
`
`S4 IMAGE DATA
`PROCESSING
`UNIT
`
`FIG.7
`
`LGE Exhibit 1004
`LGE v. ImmerVision
`Page8 of 15
`
`LGE Exhibit 1004
`LGE v. ImmerVision
`Page 8 of 15
`
`

`

`U.S. Patent
`
`Oct. 3, 2000
`
`Sheet 8 of 9
`
`6,128,145
`
`51
`
`52
`
`FIG.BA
`
`FIG.8B
`
`75
`
`71
`79
`
`72
`
`73°74
`FIG.9A
`
`#2783
`
`LGE Exhibit 1004
`LGE v. ImmerVision
`Page 9 of 15
`
`LGE Exhibit 1004
`LGE v. ImmerVision
`Page 9 of 15
`
`

`

`U.S. Patent
`
`Oct. 3, 2000
`
`Sheet 9 of 9
`
`6,128,145
`
`
`
`LGE Exhibit 1004
`LGE v. ImmerVision
`Page 10 of 15
`
`LGE Exhibit 1004
`LGE v. ImmerVision
`Page 10 of 15
`
`

`

`6,128,145
`
`1
`
`IMAGE PICK-UP DEVICE, IMAGE DISPLAY
`DEVICE AND INFORMATION RECORDING
`MEDIUM COMPRISING A FISHEYE LENS
`
`BACKGROUND OF THE INVENTION
`
`2
`plane image when imagesofall the directions ofthe field of
`view aroundthe optical axis are picked upata field angle of
`at least 90° with respect to the optical axis and are converted
`into plane images.
`Various other objects, advantages and features of the
`present invention will becomereadily apparent to those of
`ordinary skill in the art, and the novel features will be
`particularly pointed out in the appended claims.
`SUMMARYOF THE INVENTION
`
`To attain the above object, according to a first aspect of
`the present invention, there is provided an image pick-up
`device comprising a fisheye lens for picking up an image of
`all the directions of the field of view around the optical axis
`of the fisheye lens at a field angle of at least 90° in each
`direction with respect to the optical axis, wherein the fisheye
`lens has a relationship of h=nf-tan(6/m) (wherein h is the
`height of an image of a subject at a certain point obtained by
`the fisheye lens, f is the focal distance of the fisheye lens,
`1.6=m3, m-0.4£n=m+0.4, and 6 is a field angle).
`According to a second aspect of the present invention, the
`fisheye lens is constructed by a master lens provided on an
`existing image pick-up device and by an attachmentlens to
`be attached to the master lens.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`The present invention relates to an image pick-up device
`comprising a fisheye lens, an image display device and an
`information recording medium, all of which can obtain a
`high-quality converted image when an image picked up by
`the fisheye lens is converted into a plane image.
`A monitoring system using a camera which enables
`product examination at a plant or construction work at a
`construction site to be monitored from a remote place has
`recently been developed. In this monitoring system, depend-
`ing on what is monitored, capability of monitoring a wide
`range at a limited number of camerasis desired. To realize
`this, the development of a monitoring system comprising a
`fisheye lens which can pick up an imageofall the directions
`of the field of view around the optical axis at a field angle
`of at least 90° in each direction with respect to the optical
`axis is under way.
`Use of this fisheye lens makes it possible to obtain an
`image of all the space with a single camera. That is, the
`space is regarded as a single sphere, a camerais installed at
`Further, according to a third aspect of the present
`the center of the sphere, an image of half of the sphere is
`invention, there is provided an image display device com-
`picked up by the fisheye lens, the camera is turned at an
`prising an image data processing unit for converting an
`angle of 180° from that position, an image of the other half
`image obtained by the image pick-up device of the first or
`of the sphere in the opposite direction is picked up, and the
`second aspect of the present invention into a plane image
`two images are combined together to obtain an imageofall
`and a display unit for displaying the converted plane image.
`the directions of the field of view in the space of 360°, that
`is, the sphere. This image is converted into a plane image.
`According to a fourth aspect of the present invention,
`there is provided an information recording medium that
`As the monitoring system comprisingafisheye lensofthe
`records a program havingat least the step of converting an
`prior art, there is a system disclosed by Japanese Patent
`image obtained by a fisheye lens having a relationship of
`Application Laid-open No. Hei6-501585 (to be referred to
`h=nf-tan(6/m) (wherein h is the height of an image of a
`as “prior art” hereinafter), for example. Although this prior
`subject at a certain point, f is the focal distance ofthe fisheye
`art makesit possible to pick up an imageofall the directions
`lens, 6 is a field angle, 1.6=m<3. and m-0.4£n=m+0.4)
`of the field of view,the lens used in the priorart is a fisheye
`into a plane image, the step of displaying a predetermined
`lens having a relationship of h=f-6 (wherein h is the height
`portion of the converted plane image on a display unit and
`of an image of a subject at a certain point obtained by the
`the step of changing continuously the predetermined portion
`fisheye lens, f is the focal distance of the fisheye lens and 8
`with instruction means.
`is a field angle). This is obvious from the fact that Nikon’s
`8-mm f/2.8 lens is used as the fisheye lens in the above
`Japanese Patent Application Laid-open No. Hei6-501585.
`Conventional fisheye lenses generally have a relationship of
`h=f-0 and Nikon’s 8-mm f/2.8 fisheye lens has the above
`relationship of h=f-6.
`The method of picking up an image by a fisheye lens
`having this relationship of h=f-@ and converting the image
`into a plane imageis called “equidistant projection”. Since
`an image picked up bya fisheye lens having the above
`characteristics has a small volume of image data on its
`peripheral portion (field angle of around 90° with respectto
`the optical axis of the fisheye lens), when the image is
`converted into a plane image,
`there are many missing
`portions of image data on the peripheral portion of the image
`and the missing portions must be interpolated. In addition,
`the image picked up bythe fisheye lens having the above
`characteristics involves such a problem that the peripheral
`portion of the imageis distorted.
`An object of the present invention is to provide an image
`pick-up device comprising a fisheye lens, an image display
`device and an information recording medium, which mini-
`mize missing portions of image data by extracting a large
`volume of image data at a field angle of around 90° with
`respect to the optical axis of the fisheye lens to reduce
`interpolating of the missing portions and can obtain a natural
`
`45
`
`50
`
`55
`
`60
`
`65
`
`Oneof the fisheye lens used in the present invention has
`the relationship of h=2f-tan(@/2). Compared with an ordi-
`nary fisheye lens having a relationship of h=f-0, an imageat
`a peripheral portion (field angle of around 90° with respect
`to the optical axis of the fisheye lens) is enlarged and
`missing portions of image data on the peripheral portion can
`be minimized with the fisheye lens in accordance with the
`present invention. With this, when a picked-up imageis to
`be converted into a plane image, the interpolating of image
`data can be reduced, thereby making it possible to obtain a
`more natural plane image.
`The fisheye lens according to the present invention may
`be constructed by attaching an attachmentlens to a master
`lens provided on an existing camera so that the fisheye lens
`can be attached to almost all
`the existing cameras.
`In
`addition, only the attachment
`lens is newly produced,
`thereby making it possible to reduce costs.
`Further, the image display device for displaying a plane
`image converted from an image picked up by the image
`pick-up device having the above fisheye lens on a display
`unit makes the displayed image easy to be seen, thereby
`improving the value of the device. When the information
`recording medium recording the above steps is read by a
`computer and the program is executed, a more natural plane
`image can be displayed on the display unit and the displayed
`
`LGE Exhibit 1004
`LGE v. ImmerVision
`Page 11 of 15
`
`LGE Exhibit 1004
`LGE v. ImmerVision
`Page 11 of 15
`
`

`

`6,128,145
`
`45
`
`3
`portion can be freely shifted within the range of the image
`picked up by the image pick-up device.
`
`4
`image data processing unit 3 for processing image data from
`the camera 2, and a monitor unit 4 for displaying an image
`processed by the image data processing unit 3. The image
`BRIEF DESCRIPTION OF THE DRAWINGS
`data processing unit 3 has a CPU, memory means and the
`like and performs various processing using image data
`The following detailed description, given by way of ° output
`from the camera 2.
`In the case of the present
`example and not intended to limit the present invention
`invention, the image data processing unit 3 also converts an
`solely thereto, will best be appreciated in conjunction with
`image picked up bythe fisheye lens 1 into a plane image.
`the accompanying drawings, wherein like reference numer-
`As shown in FIG. 2,
`the fisheye lens 1 used in this
`als denote like elements and parts, in which:
`1g embodiment roughly consists of a lens unit (called master
`FIG. 1 is a schematic structural diagram of an image lens unit) 10 provided on the camera 2 andalens unit (called
`
`processing system using an image pick-up device compris-
`attachment lens unit) 20 that can be attached to and detached
`ing a fisheye lens according to the present invention;
`from the master lens unit 10. The fisheye lens 1 of the
`FIG.2 are a structural diagram of the fisheye lens shown
`present
`invention functions as a fisheye lens when the
`:
`.
`.
`:
`attachmentlens unit 20 is attached to the master lens unit 10.
`in FIG. 1 and a corresponding diagram schematically show-
`.
`.
`ing lens intervals (lens intervals and lens thicknesses);
`coaltacences pt 3 coreve aptlens late
`FIG. 3(A)is a diagram showing the relationship between
`25. The master lens unit 10 consists ofa fifth lens 11, a sixth
`field angle 8 and image height h with respect to fisheye
`lens 12, a seventh lens 13, an eighth lens 14, a ninth lens 15
`lenses having relationships of h=f-, h=2f-sin (6/2), h=f-sin
`and a diaphragm 26 interposed between the sixth lens 12 and
`0, h=f-tan 0, h=3f-tan (0/3), h=2f-(tan 0/1.6) and one of the 20 jn. seventh lens 13.
`fisheye lenses of the present invention having a relationship
`The curvature R (diameter of the curved surface of the
`of h=2ftan(0/2);
`.
`.
`.
`.
`lens) of each lens and interval D (lens thickness or lens
`FIG.3(B) is a diagram showing therelationship between
`interval) in this embodimentare as follows. That is, begin-
`field angle 8 and image height h with respect to fisheye
`ning with the curvature R1 of the left curved surface of the
`lenses having relationships of h=2f-tan(6/1.6), h=2.4f-tan(8/ 75 first lens 21 on the leftmost side of FIG. 2, in turn, the
`2), h=3.4ftan(0/3), h=1.2f-tan(6/1.6), h=1.6f-tan(@/2) and
`curvatures R1 and R2 ofthe first lens 21 are 40.0 mm and
`h=f-0.
`9.0 mm,the curvatures R3 and R4 of the second lens 22 are
`FIGS. 4(A) to 4(D) are viewsillustrating, in concentric
`-26.0 mm and 80.0 mm,the curvatures R5 and R6 of the
`circles each centering around the optical axis of each fisheye
`third lens 23 are -36.0 mm and —20.0 mm,and the curva-
`lens shown in FIG. 3, changes of image heights when the °° tures R7 and R8 of the fourth lens 24 are -81.0 mm and
`field angle is changed in 10° with respectto the optical axis
`-27.0 mm,respectively.
`of each fisheye lens;
`Further, the curvatures R9 and R10 ofthe fifth lens 11 are
`FIG. 5 is a diagram for explaining a method of polar-
`14.0 mm and 68.0 mm,the curvatures R11 and R12 ofthe
`coordinate converting a hemispherical image obtained by a ,, sixth lens 12 are 9.0 mm and 3.0 mm,the curvatures R13 and
`fisheye lens;
`R14 of the seventh lens 13 are 0.0 mm and -8.0 mm, the
`HIG.6 is a diagram for explaining a method of obtaining
`curvatures R15 and R16 ofthe eighth lens 14 are 10.0 mm
`the position of an image formation point on the surface of
`and -6.0 mm, and the curvatures R17 and R18 of the ninth
`CCD image pick-up elements in the polar coordinate con-
`lens 15 are 11.0 mm and -9.0 mm,respectively.
`version of FIG. 5;
`49 Meanwhile, the thickness D1 of the first lens 21 on the
`FIG.7 is a flow chart for explaining the steps of process-
`leftmost side of FIG. 2 is 1.2 mm,the interval D2 between
`
`D4 between the second lens 22 andthe third lens 23 is 14.0
`FIGS. 8(A) and 8(B) are diagrams showing another
`mm,
`the thickness D5 of the third lens 23 is 2.0 mm,
`the
`application example of the image pick-up device of the
`presentinvention, wherein FIG. 8(A) is a schematic diagram 4 interval D6 betweenthe third lens 23 and the fourth lens 24
`showingthe side thereof and FIG. 8(B) is a diagram when
`is 3.0 mm, and the thickness D7 of the fourth lens 24 is 5.0
`seen from a direction indicated by an arrow B in FIG.8(A);
`mm.
`,
`.
`FIGS. 9(A) and 9(B) are diagrams showing still another
`Further, the interval D8 between the fourth lens 24 and the
`application example of the image pick-up device of the . ffh Jens 11 is 7.0 mm,the thickness D9ofthe fifth lens I1
`presentinvention, wherein FIG. 9(A) is a schematic diagram
`is 2.0 mm,the interval D10 betweenthefifth lens 11 and the
`showing the side thereof and FIG. 9(B) is a diagram when
`sixth lens 12 is 0.3 mm,andthe thickness D11 of the sixth
`
`seen fromadirection indicated by an arrow B in FIG. 9(A); lens 12 is 0.8 mm The seventh lens 13,the eighth lens 14 and
`and
`the ninth lens 15 can be movedin the direction of the optical
`FIG. 10 is a diagram showing another example of an 5; axis to change magnification, and the intervals between
`image processing system utilizing the image pick-up device
`adjacent lenses to be described hereinafter are maximum
`comprising a fisheye lens of the present invention.
`values thereof. The interval D12 between the diaphragm 26
`and the seventh lens 13 is 4.0 mm,the thickness D13 of the
`seventh lens 13 is 1.0 mm,the interval D14 of the seventh
`DETAILED DESCRIPTION OF CERTAIN
`69 lens 13 andthe eighth lens 14 is 1.0 mm,andthe thickness
`PREFERRED EMBODIMENTS
`D15of the eighth lens 14 is 4.0 mm.
`Preferred embodiments of the present invention will be
`The interval D16 between the eighth lens 14 and the ninth
`described hereinafter with reference to FIGS. 1 to 10.
`lens 15 is 2.0 mm,and the thickness D17 of the ninth lens
`FIG. 1 schematically shows an image processing system
`is 4.0 mm.Parallel plates 16 and 17 are arranged on the right
`utilizing an image pick-up device comprising a fisheye lens
`of the present
`invention. This image processing system 65 side in FIG. 2 ofthe ninth lens 15.
`comprises a camera (such as a video camera) 2 that is an
`In this arrangement, light incident upon thefirst lens 21
`image pick-up device equipped with a fisheye lens 1, an
`passes through the first to fourth lenses 21 to 24, further
`
`LGE Exhibit 1004
`LGE v. ImmerVision
`Page 12 of 15
`
`LGE Exhibit 1004
`LGE v. ImmerVision
`Page 12 of 15
`
`

`

`6,128,145
`
`6
`further an increase in image height h tends to be smaller as
`the field angle becomescloser to 90° whenthe fisheye lenses
`having the relationships of h=2f-sin(6/2) and h=f-sin@ are
`used.
`
`5
`through the fifth to ninth lenses 11 to 15 and is input into
`CCD image pick-up elements 30 in the camera 2. In this
`attachmentlens unit 20, parallel rays input into the first lens
`21 are output from the fourth lens 24 as parallel rays.
`An increase in image height becomeslarger toward the
`Therefore, this attachment lens unit 20 can be attached to 5
`peripheral portion (field angle of 90°) whenthe fisheye lens
`almost all the cameras. The width of the parallel pencil of
`_—=‘aving therelationship of h=ftan 0 is used and more image
`rays output from the fourth lens 24 of the attachment lens
`data can be obtained. However, at a field angle 8 of 90°, tan
`unit 20 (shown by “w’”in the figure) is set to 4 orless the
`9 becomes infinite. Since the fisheye lens is required to
`effective diameter of the master lens 10 of the camera to
`which the attachmentlens unit 20 is attached. In FIG.2, the 10 obtain an mage of all the directions of the field of view
`spherical surface 40 at the front of the first lens 21 represents
`around the optical axis at a field angle ofat least 90° with
`a virtual subject surface of the picked up image.
`respectto the optical axis, it can be said that the fisheye lens
`:
`”
`an
`.
`having the relationship of h=f-tan 0 is not suitable.
`As described above, the present invention is characterized
`Therefore,
`the fisheye lenses having relationships of
`in that a desired fisheye lens 1 constructed by the master lens
`h=2f-tan(@/2), h=f-6, h=2f-sin(6/2) and h=f-sin@ may be
`unit 10 and the attachmentlens unit 20 and hasarelationship 15 yoeq. FIGS. 4(A) to 4(D) show,in concentric circles each
`of h=2ftan(8/2) (wherein h is the height of an image of a
`centering around the optical axis of eachfisheye lens, image
`subject at a certain position, f is the focal distance of the
`heights h whenthe field angle 0 is changed in 10° with
`fisheye lens and @ is a field angle).It is noted that although
`respect to the optical axis of each fisheye lens. FIG. 4(A)
`the preferred embodiment of the present invention has the
`shows the image height
`in the fisheye lens having the
`above-indicated relationship,
`the present
`invention also 29 relationship of h=2f-tan(6/2), FIG. 4(B) shows the image
`embodies fisheye lenses having the relationship of h=n-f-tan
`height in the fisheye lens having the relationship of h=f:0,
`(8/m), where m has the value of 1.65m53, and nhas the
`FIG. 4(C) showsthe image height in the fisheye lens having
`value of m-0.4=n=m+0.4. Also,
`the present
`invention
`the relationship of h=2f-sin(6/2), and FIG. 4(D) showsthe
`contemplates such a relationship when m equals n. Also,the
`image height in the fisheye lens having the relationship of
`relationship h=1.2 ftan(8/m), m21.6 also is embodied by °° p-f-sind. In FIGS. 4(A) to 4(D), ho represents the height of
`the present invention. However, for purposes of discussion
`an image Monearthe optical axis of each fisheye lens and
`herein, m and n both equalto 2.
`he represents the height of an image Meat a field angle of
`The fisheye lens that has been generally used in the prior
`around 90°.
`art has a relationship of h=f-0 as described above. These 39 Asis understood from FIGS. 4(A) to 4(D), imageheight
`functions are used to map a spherical image as a polar-
`at a field angle of around 90° when the fisheye lens having
`coordinate converted image. The relationship, other than
`the relationship of h=2f-sin(@/2) orthe fisheye lens having
`that, may be h=2f-sin(/2), h=2f=-sin 0 or h=f-tan 0.
`the relationship of h=f-sin® is used is smaller than image
`FIG. 3(A) is a diagram showing relationships between
`height near the optical axis and only a small volume of
`field angle @ and image height h whenfisheye lenses having 35 imagedata can be obtained. The imageheighthe of an image
`relationships of h=2f-tan(@/2), h=f-6, h=f-sin(6/2), h=f-sin @
`Meat a peripheral portion of the fisheye lens that has been
`and h=f-tan 6, etc. are used. Here, 8=90° showsa field angle
`generally used and hasthe relationship of h=f-6 is the same
`with respect to the optical axis (the field angle of the optical
`as the image height ho of an image Monearthe optical axis
`axis is 0°).
`and the imageis distorted.
`In FIG. 3(A), a curve C1 showstherelationship between 4,
`From these facts, it can be said that the fisheye lenses
`field angle @ and image height h whenthe fisheye lens of the
`having relationships of h=2f-sin(6/2) and h=f-sin@ are not
`present invention having the relationship of h=2f-tan(6/2) is
`preferred in view that how large volume of data can be
`used, and a curve C2 showsthe relationship between field
`obtainedat a field angle of 90° or therearound. Even with the
`angle @ and image height h whena fisheye lens having the
`fisheye lens that has been generally used and has the
`relationship of h=f-0 is used. A curve C3 showsthe rela- 4;
`relationship of h=f-0 is not satisfactory.
`tionship between field angle 8 and image height h when a
`In contrast to that, the image height he of the image Me
`fisheye lens having the relationship of h=2f-sin(6/2) is used,
`at a peripheral portion of the fisheye lens 1 having the
`a curve C4 showsthe relationship between field angle 6 and
`relationship of h=2f-tan(6@/2) in accordance with the present
`image height h whena fisheye lens havingthe relationship
`invention is enlarged and larger than the imageheightho of
`of h=2f-sin® is used, and a curve C5 showsthe relationship <,
`the image Mo nearthe optical axis, a larger volume of image
`
`between field angle 6 and image height h whenafisheye lens data can be obtained in comparison with the conventional
`having the relationship of h=ftanO is used. A curve Cl’
`fisheye lens, and the obtained imageis not distorted.
`
`shows the relationship of h=3f-tan (0/3) and a curve C" Whenasingle spherical image obtained by combining
`showsthe relationship of h=2f-tan (0/1.6).
`two hemispherical imagesofall the directionsof the field of
`FIG. 3(B), is a diagram showing the relationship between 55 view around the optical axis of the fisheye lens 1, which are
`field angle 6 and image height when fisheye lenses having
`picked up at a field angle of 90° with respect to the optical
`relationships of h=2f-tan (6/1.6), h=2.4f-tan(6/2), h=3.4f-tan
`axis is converted into a plane image by the image data
`(8/3), h=1.2f-tan (6/1.6), h=1.6f-tan (6/2) and h=f-6. All of
`processing unit 3,
`it
`is necessary to interpolate missing
`these fisheye lenses, except h=f-8, are embodied by the
`image data on the peripheral portion (field angle of around
`present invention.
`60 90° with respectto the optical axis) of the image. According
`As is evident from FIG. 3, an increase in image height h
`to the present invention, since an image at the peripheral
`at a field angle 6 of about 90°is largest when the fisheye lens
`portion is enlarged and a large volume of data on the
`having the relationship of h=f-tan 6 is used and is second
`peripheral portion can be extracted, the volume of image
`largest when the fisheye lens having the relationship of
`data to be interpolated can be greatly reduced, when com-
`h=2f-tan(6/2) is used. Changes in image height h with 65 pared with the conventional system.
`respect to changesin field angle 8 become linear when the
`An imageofall the directions of the field of view around
`the optical axis is picked up at a field angle of at least 90°
`fisheye lens having the relationship of h=f-@ is used and
`
`LGE Exhibit 1004
`LGE v. ImmerVision
`Page 13 of 15
`
`LGE Exhibit 1004
`LGE v. ImmerVision
`Page 13 of 15
`
`

`

`6,128,145
`
`8
`The above two images are then combined together and the
`combined image is converted into a plane image by the
`image data processing unit 3 (step S3). At this point, an area
`corresponding to a connection portion between these hemi-
`spheres must be corrected. Since the polar-coordinate con-
`verted image obtained by the fisheye lens 1 has a large
`volumeof information on a peripheral portion, the process-
`ing of combining these imagesis easy. Thereafter, a prede-
`termined portion of the thus obtained plane image is
`extracted and displayed on the monitor unit 4 (step S4).
`A usershifts the screen with instruction means such as a
`
`10
`
`7
`to the optical axis and is polar-coordinate
`with respect
`converted into a plane image in the following manner.
`An X, Y and Z coordinate system as shown in FIG. 5 is
`imagined in subject space. At this point, the optical axis of
`the fisheye lens 1 is made Z axis. The coordinates of a
`certain point p are represented as (X1, Y1, Z1) and the
`elevation angle of the point p from the origin O of the
`coordinates with respect to the XZ plane is represented by @.
`The elevation angle of the point p from the position of Z1 on
`the Z axis with respect to the XZ plane is represented by g.
`When an x and y coordinate system having the optical
`axis (Z axis) as an origin o is imagined on the surface of
`CCD image pick-up elements 30 as shownin FIG. 6 and the
`focal distance of the fisheye lens 1 is represented by f, the
`image formation point (p') for the point p is located as shown
`in FIG. 6. In FIG. 6, 7 is added to @ because an image formed
`at the point p' is inverted vertically and horizontally with
`respect to the image of the subject surface (point p). The
`optical axis in FIG. 6 is present in a direction perpendicular
`to the paper from the origin o of the x and y coordinates.
`The position of the point p' is expressed as polar coordi-
`nates with a length (h) between the origin o and the point p'
`and an angle g+ formed by op' and the x axis. When the
`polar coordinates are expressed on the x and y rectangular
`coordinates, the position (x1, y1) on the x and y rectangular
`coordinates are expressed as follows.
`
`15
`
`20
`
`25
`
`xl=h-cos (+m)
`
`yl=h-sin (+0)
`
`()
`
`2)
`
`30
`
`In addition, the image height h of the point p' is represented
`by h=2f-tan(6/2), hence, when h=2f-tan(6/2) is substituted
`into the above expressions (1) and (2), the coordinates (x1,
`yl) of the image formation point p' on the surface of the
`CCD image pick-up elements 30 are as follows.
`
`mouse whenthe user likes to change the displayed prede-
`termined portion. This shifting can be made continuously in
`any direction of 360° around the portion displayed on the
`monitor unit 4 (step S5).
`The above steps are for picking up an imageof a sphere
`in all the directions of 360°. When only a single hemisphere
`is photographed, the same steps are taken. However, step S2
`is unnecessary and the processing of combining two images
`in step S3 is also unnecessary.
`In the present invention, since the volume of information
`on the peripheral portion of an image obtained bythe fisheye
`lens 1 is large, that is, an image at the peripheral portion is
`enlarged,it is convenient whenthe present invention is used
`for the examination of a product. For example, when the
`inner surface 52 of a cylindrical body 51 is photographedat
`the condition that the optical axis of the fisheye lens 1 of the
`present invention is aligned with the central axis of the
`cylindrical body 51 as shown in FIGS. 8(A) and 8(B), a
`peripheral portion of an image can be extracted as an image
`having a larger volumeof information than a central portion
`in the present invention. Therefore, it is easy to find that a
`35 scratch has been generated in the inner surface 52.
`Consequently, this can be used for the examination of a
`pipe-like body such as a water pipe or gas pipe and further
`for the monitoring of a crack that has been generated in the
`wall surface of a tunnel orthe like.
`.
`oo
`It can also be used for the examination of the connection
`condition of a small part such as an IC. That is, when a part
`71 is fixed to a substrate 74 by a solder 73 onits both sides
`72 and 72 shown in FIGS. 9(A) and 9(B), the soldering state
`|. of the part must be checked from its side direction in the
`prior art. However, when anotherpart 75 is existent in a side
`direction, the solder 73 on the part 75 side cannot be seen
`through a camera, thereby making automatic examination
`difficult. On the other hand, even whenthe fisheye lens 1 is
`Thus,the position of the point p' on the surface of CCD 50 installed right over the part 71 as shownin FIG. 9(A), the
`side direction of the part 71 can be sufficiently photographed
`image pick-up elements 30 can be obtained for the point p
`by the camera 2 comprising the fisheye lens 1 of the present
`on the surface of the subject.
`invention, thereby enabling automatic examination with the
`Thereafter, a description is subsequently given, with ref-
`camera 2.
`erence to FIG. 7, of steps required when the sphere (all
`directions) is photographed by the camera 2 comprising the
`fisheye lens 1 and an image thereof is displayed on the
`monitor unit 4 that is the display unit.
`First, a hemisphere in one direction is photographed by
`the camera 2 comprising the fisheye lens 1 (step S1).
`Thereby, an image of the hemisphere is formed on the
`surface of the CCD image pick-up elements 30 as a polar-
`coordinate converted image. Thereafter,
`the camera 2 is
`turned at an angle of 180° to photograph the other hemi-
`sphere in the opposite direction (step S2). Thereby, an image
`of the other hemisphere is formed on the surface of the CCD
`image pick-up elements 30 as a polar-coordinate converted
`image.
`
`w1=2ftan (6/2)-cos (9+)
`y=2f-tan (0/2)-sin (6+)
`
`As a result, they are expressed as follows.
`
`x=-2ftan (0/2)-cos #
`y=-2ftan (6/2)-sin
`.
`In the above expressions, 0 and ¢ are defined as follows.
`6=tan? (vX12*¥1"/71)
`
`(5)
`(6)
`
`3)
`(4)
`
`40
`
`g=tan? (Y1/X1)
`
`55
`
`60
`
`65
`
`Although each foregoing embodimentis an example of a
`preferred embodimentof the invention,it is to be understood
`that the invention is not limited thereto and that various
`
`changes and modifications may be made in the invention
`without departing from the spirit and scope thereof. For
`example, the fisheye lens may be constructed by the attach-
`ment
`lens unit 20 alone without the master lens 10, or
`contrariwise may be constructed by an integrated unit of the
`master lens unit 10 and the attachmentlens unit 20. Also, the
`construction and numerical values of the fisheye lens 1
`shown in the above embodiment are just examples and a
`fisheye lens having other construction and numerical values
`may be used.
`
`LGE Exhibit 1004
`LGE v. ImmerVision
`Page 14 of 15
`
`LGE Exhibit 1004
`LGE v. ImmerVision
`Page 14 of 15
`
`

`

`6,128,145
`
`9
`Further, as the system comprising the fisheye lens 1 of the
`present invention, an image processing system 81 shown in
`FIG. 10 may be used. This image processing system 81 is