`6,128,145
`[11] Patent Number:
`(15
`United States Patent
`Nagaoka
`[45] Date of Patent:
`Oct. 3, 2000
`
`
`[54]
`
`IMAGE PICK-UP DEVICE, IMAGE DISPLAY
`DEVICE AND INFORMATION RECORDING
`MEDIUM COMPRISING A FISHEYE LENS
`
`[56]
`
`[75]
`
`Inventor: Tooru Nagaoka, Nagano-Pref, Japan
`
`[73] Assignees: FIT Corporation, Nagano-Pref.; Rios
`Corporation; Advanet, Inc., both of
`Okayama-Pref., all of Japan
`
`[21] Appl. No.: 09/300,972
`
`References Cited
`US. PATENT DOCUMENTS
`~
`8/1983 Hayashida oes 359/662
`
`6/1992 Hegg et al.
`.
`340/461
`3/1996)
`Jamieson oe. cece eeeeeeneceeeeeee 359/355
`
`4,400,063
`5,121,099
`5,502,592
`
`Primary Examiner—Georgia Epps
`Assistant Examiner—Jordan M. Schwartz
`Attorney, Agent, or Firm—McAulay Nissen Goldberg &
`Kiel, LLP
`
`[57]
`
`ABSTRACT
`
`Nov. 25,1998
`
`[JP]
`
`Japan oo. eeeeeeeeees 10-350751
`
`[52] U.S. Cheeee eeesessnteenenneens 359/749; 359/672
`
`Apr. 28, 1999
`Filed:
`[22]
`An image is picked up by a camera comprisingafisheye lens
`[30]
`Foreign Application Priority Data
`having a relationship of h=n-f-tan(@/m), wherein h is the
`height of an image of a subject at a certain point, f is the
`focal distance of the fisheye lens, 0 is a field angle, m has a
`value of 1.62mS3, andnhas a value of m-0.4SnSm+0.4,
`[SD]
`Tint. C07 eee ceseeenseeccecsecsnnmmseneeees G02B 13/04
`and the image data of which is output from the camera, is
`converted into a plane image by an image data processing
`unit, and this converted image is then output to a monitor
`unit. Preferably, n and m both equal 2.
`
`[58] Field of Search... 359/749, 750-753,
`359/680-682, 672-675
`
`14 Claims, 9 Drawing Sheets
`
`DIO Di2 pi6
`
`APPLE 1010
`
`APPLE 1010
`
`1
`
`
`
`U.S. Patent
`
`Oct. 3, 2000
`
`Sheet 1 of 9
`
`6,128,145
`
`
`
`
`
`CAMERA
`
`
`
`
`
`IMAGE DATA
`
`PROCESSING
`UNIT
`
`ZS
`MONITOR UNIT
`
`FIG.1
`
`2
`
`
`
`U.S. Patent
`
`Oct. 3, 2000
`
`Sheet 2 of 9
`
`6,128,145
`
`11
`
`D15
`
`DIODi2pies
`
`DI7
`
`
`
`
`
`
`
`
`
`5 iR ee]=
`
`aR1013R14
`
`
`
`3
`
`
`
`U.S. Patent
`
`Oct. 3, 2000
`
`Sheet 3 of 9
`
`6,128,145
`
`
`
`Se
`
`oe
`
`Q)UIs4z0¢ (2)@STONYGIAldVeOldg—____—____-_ (2/
`
`
`(€/9)uel-4¢qa)
`8Gz
`
`19
`
`
`
`LHDIHJDVWI
`
`4
`
`
`
`U.S. Patent
`
`Oct. 3, 2000
`
`Sheet 4 of 9
`
`6,128,145
`
`[94 tan(@/1.6)
`
`[ (=C1)
`
`60 50
`
`2.4f tan(@/2)
`
`1.2¢ tan(@/1.6)
`
`1.6f tan(@/2)
`
`0
`
`10
`
`29
`
`30
`
`40
`
`50
`
`60
`
`70
`
`80 90
`
`FIG.3B
`
`5
`
`
`
`U.S. Patent
`
`Oct. 3, 2000
`
`Sheet 5 of 9
`
`6,128,145
`
`
`
`h = 2f - sin (@/2)
`
`h=f-sing
`
`ho
`
`he
`
`FIGAC
`
`“FIG.AD
`
`6
`
`
`
`U.S. Patent
`
`Oct. 3, 2000
`
`Sheet 6 of 9
`
`6,128,145
`
`o(X1, Y1, 21)
`
`
`
`
`
`tant
`Avot
`
`D
`h=2{ tans
`
`
`FIG.6
`
`7
`
`
`
`U.S. Patent
`
`Oct. 3, 2000
`
`Sheet 7 of 9
`
`6,128,145
`
`S1 .
`
`CAMERA
`
`S52
`
`S4 IMAGE DATA
`PROCESSING
`UNIT
`
`FIG.7
`
`8
`
`
`
`U.S. Patent
`
`Oct. 3, 2000
`
`Sheet 8 of 9
`
`6,128,145
`
`51
`
`o2
`
`FIG.BA
`
`FIG.8B
`
`B +
`
`71
`79
`
`72
`
`(
`
`75
`
`73°
`
`#74
`FIG.9A
`
`£473
`
`9
`
`
`
`U.S. Patent
`
`Oct. 3, 2000
`
`Sheet 9 of 9
`
`6,128,145
`
`
`
`10
`
`
`
`6,128,145
`
`1
`IMAGE PICK-UP DEVICE, IMAGE DISPLAY
`DEVICE AND INFORMATION RECORDING
`MEDIUM COMPRISING A FISHEYE LENS
`
`BACKGROUND OF THE INVENTION
`
`The present invention relates to an image pick-up device
`comprising a fisheye lens, an image display device and an
`information recording medium, all of which can obtain a
`high-quality converted image when an image picked up by
`the fisheye lens is converted into a plane image.
`A monitoring system using a camera which enables
`product examination at a plant or construction work at a
`construction site to be monitored from a remote place has
`recently been developed. In this monitoring system, depend-
`ing on what is monitored, capability of monitoring a wide
`range at a limited number of camerasis desired. ‘lo realize
`this, the development of a monitoring system comprising a
`fisheye lens which can pick up an imageofall the directions
`of the field of view around the optical axis at a field angle
`of at least 90° in each direction with respect to the optical
`axis is under way.
`Use of this fisheye lens makes it possible to obtain an
`image of all the space with a single camera. That is, the
`space is regarded as a single sphere, a camerais installed at
`the center of the sphere, an image of half of the sphere is
`picked up by the fisheye lens, the camera is turned at an
`angle of 180° fromthat position, an image of the other half
`of the sphere in the opposite direction is picked up, and the
`two images are combined together to obtain an imageofall
`the directions of the field of view in the space of 360°, that
`is, the sphere. This image is converted into a plane image.
`As the monitoring system comprisinga fisheye lens of the
`prior art, there is a system disclosed by Japanese Patent
`Application Laid-open No. Hei6-501585 (to be referred to
`as “prior art” hereinafter), for example. Although this prior
`art makesil possible to pick up an imageof all the directions
`of the field of view,the lens usedin the priorart is a fisheye
`lens having a relationship of h=f-6 (wherein h is the height
`of an image of a subject at a certain point obtained by the
`fisheye lens, f is the focal distance of the fisheye lens and 0
`is a field angle). This is obvious from the fact that Nikon’s
`8-mm f/2.8 lens is used as the fisheye lens in the above
`Japanese Patent Application Laid-open No. Hei6-501585.
`Conventionalfisheye lenses generally have a relationship of
`h=f-@ and Nikon’s 8-mm f/2.8 fisheye lens has the above
`relationship of h=f-0.
`The method of picking up an image by a fisheye lens
`having this relationship of h=f-6 and converting the image
`into a plane imageis called “equidistant projection”. Since
`an image picked up by a fisheye lens having the above
`characteristics has a small volume of image data on its
`peripheral portion (field angle of around 90° with respect to
`the optical axis of the fisheye lens), when the image is
`converted into a plane image,
`there are many missing
`portions of imagedata on the peripheral portion of the image
`and the missing portions must be interpolated. In addition,
`the image picked up by the fishcyc lens having the above
`characteristics involves such a problem that the peripheral
`portion of the imageis distorted.
`An object of the present invention is to provide an image
`pick-up device comprising a fisheye lens, an image display
`device and an information recording medium, which mini-
`mize missing portions of image data by extracting a large
`volume of image data at a field angle of around 90° with
`respect to the optical axis of the fisheye lens to reduce
`interpolating of the missing portions and can obtain a natural
`
`10
`
`15
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`plane image when imagesof all the directions of the field of
`view aroundthe optical axis are picked upat a field angle of
`at least 90° with respect to the optical axis and are converted
`into plane images.
`Various other objects, advantages and features of the
`present invention will become readily apparent to those of
`ordinary skill
`in the art, and the novel features will be
`particularly pointed out in the appended claims.
`SUMMARYOF THE INVENTION
`
`To attain the above object, according to a first aspect of
`the present invention, there is provided an image pick-up
`device comprising a fisheye lens for picking up an image of
`all the directions of the field of view around the optical axis
`of the fisheye lens at a field angle of at least 90° in each
`direction with respect to the optical axis, wherein thefisheye
`lens has a relationship of h=nf-tan(0/m) (wherein h is the
`height of an image of a subject at a certain point obtained by
`the fisheye lens, f is the focal distance of the fisheye lens,
`1.62mS83, m-0.4£n2£m+0.4, and 0 is a field angle).
`According to a second aspect of the present invention, the
`fisheye lens is constructed by a master lens provided on an
`existing image pick-up device and by an attachmentlens to
`be attached to the master lens.
`
`Further, according to a third aspect of the present
`invention, there is provided an image display device com-
`prising an image data processing unit for converting an
`image obtained by the image pick-up device of the first or
`second aspect of the present invention into a plane image
`and a display unit for displaying the converted plane image.
`According to a fourth aspect of the present invention,
`there is provided an information recording medium that
`records a program having at least the step of converting an
`image obtained by a fisheye lens having a relationship of
`h=nf-tan(6/m) (wherein h is the height of an image of a
`subject at a certain point, f is the focal distance of the fisheye
`lens, 8 is a field angle, 1.62m<3. and m-0.4SnSm+0.4)
`into a plane image, the step of displaying a predetermined
`portion of the converted plane image on a display unit and
`the step of changing continuously the predetermined portion
`with instruction means.
`
`Oneof the fisheye lens used in the present invention has
`the relationship of h=2f-tan(6/2). Compared with an ordi-
`nary fisheye lens having a relationship of h=f-0, an image at
`a peripheral portion(field angle of around 90° with respect
`to the optical axis of the fisheye lens) is enlarged and
`missing portions of image data on the peripheral portion can
`be minimized with the fisheye lens in accordance with the
`present invention. With this, when a picked-up imageis to
`be converted into a plane image, the interpolating of image
`data can be reduced, thereby making it possible to obtain a
`more natural plane image.
`The fisheye lens according to the present invention may
`be constructed by attaching an attachment lens to a master
`lens provided on an existing camera so that the fisheye lens
`can be attached to almost all
`the existing cameras.
`In
`addition, only the attachment
`lens is newly produced,
`thereby making it possible to reduce costs.
`Further, the image display device for displaying a plane
`image converted from an image picked up by the image
`pick-up device having the above fisheye lens on a display
`unit makes the displayed image easy to be seen, thereby
`improving, the value of the device. When the information
`recording medium recording the above steps is read by a
`computer and the program is executed, a more natural plane
`image can be displayed on the display unit and the displayed
`
`11
`
`11
`
`
`
`6,128,145
`
`3
`portion can be freely shifted within the range of the image
`picked up by the image pick-up device.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`The following detailed description, given by way of
`example and not intended to limit the present
`invention
`solely thereto, will best be appreciated in conjunction with
`the accompanying drawings, wherein like reference numer-
`als denote like elements and parts, in which:
`FIG. 1 is a schematic structural diagram of an image
`processing system using an image pick-up device compris-
`ing a fisheye lens according to the present invention;
`TIG. 2 are a structural diagram of the fisheye lens shown
`in FIG. 1 and a corresponding diagram schematically show-
`ing lens intervals (lens intervals and lens thicknesses);
`FIG. 3(A) is a diagram showingtherelationship between
`field angle @ and image height h with respect to fisheye
`lenses having relationships of h=f-0, h=2f-sin (6/2), h=f-sin
`0, h=f-tan 0, h=3f-tan (6/3), h=2f-(tan 6/1.6) and one of the
`fisheye lenses of the present invention having a relationship
`of h=2f-tan(6/2);
`FIG. 3(B) is a diagram showingthe relationship between
`field angle 6 and image height h with respect to fisheye
`lenses having relationships of h=2f-tan(0/1.6), h=2.4f-tan(0/
`2), h=3.4ftan(6/3), h=1.2f-tan(6/1.6), h=1.6f-tan(8/2) and
`h=f-0.
`
`FIGS. 4(A) to 4(D) are viewsillustrating, in concentric
`circles each centering aroundthe optical axis of each fisheye
`lens shown in FIG. 3, changes of image heights when the
`field angle is changed in 10° with respect to the optical axis
`of each fisheye lens;
`FIG. 5 is a diagram for explaining a method of polar-
`coordinate converting a hemispherical image obtained by a
`fisheye lens;
`FIG. 6 is a diagram for explaining a method of obtaining
`the position of an image formation point on the surface of
`CCD image pick-up elements in the polar coordinate con-
`version of FIG. 5;
`FIG. 7 is a flow chart for explaining the steps of process-
`ing an image using the image processing system of FIG. 1;
`FIGS. 8(A) and 8(B) are diagrams showing another
`application example of the image pick-up device of the
`present invention, wherein FIG. 8(A) is a schematic diagram
`showing the side thereof and FIG. 8(B) is a diagram when
`seen from a direction indicated by an arrow B in FIG. 8(A);
`FIGS. 9(A) and 9(B) are diagrams showingstill another
`application example of the image pick-up device of the
`present invention, wherein FIG. 9(A) is a schematic diagram
`showing the side thereof and FIG. 9(B) is a diagram when
`seen from a direction indicated by an arrow B in FIG. 9(A);
`and
`
`FIG. 10 is a diagram showing another example of an
`image processing systemutilizing the image pick-up device
`comprising a fisheye lens of the present invention.
`DETAILED DESCRIPTION OF CERTAIN
`PREFERRED EMBODIMENTS
`
`Preferred embodiments of the present invention will be
`described hereinafter with reference to FIGS. 1 to 10.
`
`FIG. 1 schematically shows an image processing system
`utilizing an image pick-up device comprising a fisheye lens
`of the present
`invention. This image processing system
`comprises a camera (such as a video camera) 2 that is an
`image pick-up device equipped with a fisheye lens 1, an
`
`5
`
`10
`
`15
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`4
`image data processing, unit 3 for processing image data from
`the camera 2, and a monitor unit 4 for displaying an image
`processed by the image data processing unit 3. The image
`data processing unit 3 has a CPU, memory meansand the
`like and performs various processing using image data
`output from the camera 2.
`In the case of the present
`invention, the image data processing unit 3 also converts an
`image picked up bythe fisheye lens 1 into a plane image.
`As shown in FIG. 2,
`the fisheye lens 1 used in this
`embodiment roughly consists of a lens unit (called master
`lensunit) 10 provided on the camera 2 and a lensunit (called
`attachment Icns unit) 20 that can be attached to and detached
`from the master lens unit 10. The fisheye lens 1 of the
`present
`invention functions as a fisheye lens when the
`attachment lens unit 20 is attached to the master lens unit 10.
`The attachment lens unit 20 consists of a first lens 21, a
`second lens 22, a third lens 23, a fourth lens 24 and a plate
`25. The master lens unit 10 consists of a fifth lens 11, a sixth
`lens 12, a seventh lens 13, an eighth lens 14, a ninth lens 15
`and a diaphragm 26 interposed betweenthe sixth lens 12 and
`the seventh lens 13.
`
`The curvature R (diameter of the curved surface of the
`lens) of each lens and interval D (lens thickness or lens
`interval) in this embodimentare as follows. Thatis, begin-
`ning with the curvature R1 ofthe left curved surface of the
`first lens 21 on the leftmost side of FIG. 2, in turn, the
`curvatures R1 and R2 of the first lens 21 are 40.0 mm and
`9.0 mm,the curvatures R3 and R4 of the second lens 22 are
`-26.0 mm and 80.0 mm, the curvatures R5 and R6 of the
`third lens 23 are -36.0 mm and —20.0 mm,and the curva-
`tures R7 and R8 of the fourth lens 24 are -81.0 mm and
`
`-27.0 mm,respectively.
`Further, the curvatures R9 and R10ofthe fifth lens 11 are
`14.0 mm and 68.0 mm, the curvatures R11 and R12 of the
`sixth lens 12 are 9.0 mm and 3.0 mm,the curvatures R13 and
`R14 of the seventh lens 13 are 0.0 mm and -8.0 mm, the
`curvatures R15 and R16 of the eighth lens 14 are 10.0 mm
`and -6.0 mm, and the curvatures R17 and R18of the ninth
`lens 15 are 11.0 mm and -9.0 mn, respectively.
`Meanwhile, the thickness D1 of the first lens 21 on the
`leftmost side of FIG. 2 is 1.2 mm, the interval D2 between
`the first lens 21 and the second lens 22 is 10.00 mm,and the
`thickness D3 of the second lens 22 is 1.2 mm. Theinterval
`D4 between the second lens 22 and the third lens 23 is 14.0
`mm, the thickness D5 of the third lens 23 is 2.0 mm, the
`interval D6 betweenthe third lens 23 and the fourth lens 24
`is 3.0 mm,and the thickness D7 of the fourth lens 24 is 5.0
`mm.
`
`Further, the interval D8 betweenthe fourth lens 24 and the
`fifth lens 11 is 7.0 mm, the thickness D9 ofthe fifth lens 11
`is 2.0 mm, the interval D10 betweenthe fifth lens 11 and the
`sixth lens 12 is 0.3 mm, and the thickness D11 of the sixth
`lens 12 is 0.8 mm The seventh lens 13, the eighth lens 14 and
`the ninth lens 15 can be moved inthe direction of the optical
`axis to change magnification, and the intervals between
`adjacent lenses to be described hereinafter are maximum
`values thereof. The interval D12 between the diaphragm 26
`and the seventh lens 13 is 4.0 mm,the thickness D13 of the
`seventh lens 13 is 1.0 mm, the interval D14 of the seventh
`lens 13 and the eighth lens 14 is 1.0 mm,andthe thickness
`D15 of the eighth lens 14 is 4.0 mm.
`The interval D16 between the eighth lens 14 and the ninth
`lens 15 is 2.0 mm, and the thickness D17 of the ninth lens
`is 4.0 mm. Parallel plates 16 and 17are arranged on the right
`side in FIG. 2 of the ninth lens 15.
`
`In this arrangement, light incident upon the first lens 21
`passes through the first to fourth lenses 21 to 24, further
`
`12
`
`12
`
`
`
`6,128,145
`
`6
`further an increase in image height h tends to be smaller as
`the field angle becomescloser to 90° whenthefisheye lenses
`having the relationships of h=2f-sin(@/2) and h=f-sin@ are
`used.
`
`An increase in image height becomes larger toward the
`peripheral portion (field angle of 90°) whenthe fisheye lens
`having the relationship of h=f-tan 6 is used and more image
`data can be obtained. However,at a ficld angle 0 of 90°, tan
`6 becomes infinite. Since the fisheye lens is required to
`obtain an image of all the directions of the field of view
`around the optical axis at a field angle of at least 90° with
`respect to the optical axis, it can be said that the fishcyc lens
`having the relationship of h=f-tan 0 is not suitable.
`Therefore,
`the fisheye lenses having relationships of
`h=2f-tan(6/2), h=f-0, h=2f-sin(@/2) and h=f-sind may be
`used. FIGS. 4(A) to 4(D) show, in concentric circles each
`centering around the optical axis of each fisheye lens, image
`heights h when the field angle @ is changed in 10° with
`respect to the optical axis of each fisheye lens. FIG. 4(A)
`shows the image height
`in the fisheye lens having the
`relationship of h=2f-tan(@/2), FIG. 4(B) shows the image
`height in the fisheye lens having the relationship of h=f-0,
`FIG. 4(C) showsthe image height in the fisheye lens having
`the relationship of h=2f-sin(6/2), and FIG. 4(D) showsthe
`image height in the fisheye lens having the relationship of
`h=f-sin8. In FIGS. 4(A) to 4(D), ho represents the height of
`an image Mo near the optical axis of each fisheye lens and
`he represents the height of an image Meat a field angle of
`around 90°.
`
`5
`through the fifth to ninth lenses 11 to 15 and is input into
`CCD image pick-up elements 30 in the camera 2. In this
`attachmentlens unit 20, parallel rays input into the first lens
`21 are output from the fourth lens 24 as parallel rays.
`Therefore, this attachment lens unit 20 can be attached to
`almost all the cameras. The width of the parallel pencil of
`rays output from the fourth lens 24 of the attachment lens
`unit 20 (shown by “w”in the figure) is set to % or less the
`effective diameter of the master lens 10 of the camera to
`whichthe attachmentlens unit 20 is attached. In FIG. 2, the
`spherical surface 40 atthe front of the first lens 21 represents
`a virtual subject surface of the picked up image.
`As described above, the present invention is characterized
`in that a desired fisheye lens 1 constructed by the master lens
`unit 10 and the attachmentlens unit 20 and has a relationship
`of h=2f-tan(6/2) (wherein h is the height of an image of a
`subject at a certain position, f is the focal distance of the
`fisheye lens and 6 is a field angle). It is noted that although
`the preferred embodimentof the present invention has the
`above-indicated relationship,
`the present
`invention also
`embodies fisheye lenses having the relationship of h=n-f-tan
`(8/m), where m has the value of 1.6=m=3, and n has the
`value of m-0.42n=m+0.4. Also,
`the present
`invention
`contemplates such a relationship when m equals n. Also, the
`relationship h=1.2 ftan(@/m), m21.6 also is embodied by
`the present invention. However, for purposes of discussion
`herein, m and n both equalto 2.
`The fisheye lens that has been generally used in the prior
`art has a relationship of h=f-6 as described above. These
`As is understood from FIGS. 4(A) to 4(D), image height
`functions are used to map a spherical image as a polar-
`at a field angle of around 90° whenthe fisheye lens having
`coordinate converted image. The relationship, other than
`the relationship of h=2f-sin(6/2) or the fisheye lens having
`that, may be h=2f-sin(0/2), h=2f=-sin 0 or h=f-tan 0.
`the relationship of h=f-sinO is used is smaller than image
`FIG. 3(A) is a diagram showing relationships between
`height near the optical axis and only a small volume of
`image data can be obtained. The image height he of an image
`field angle 9 and image height h whenfisheye lenses having
`relationships of h=2f-tan(6/2), h=f-8, h=f-sin(6/2), h=f-sin 6
`Meat a peripheral portion of the fisheye lens that has been
`and h=f-tan 9, etc. are used. Here, 9=90° showsa field angle
`generally used and hasthe relationship of h=f-0 is the same
`with respectto the optical axis (the field angle of the optical
`as the image height ho of an image Monearthe optical axis
`axis is 0°).
`and the image is distorted.
`In FIG. 3(A), a curve C1 showsthe relationship between
`From these facts,
`it can be said that the fisheye lenses
`field angle 8 and image height h whenthefisheye lens of the
`having relationships of h=2f-sin(@/2) and h=f-sin@ are not
`present invention having the relationship of h=2f-tan(6/2) is
`preferred in view that how large volume of data can be
`obtainedat a field angle of 90° or therearound. Even with the
`used, and a curve C2 showsthe relationship between field
`
`angle 6 and image height h whenafisheye lens having the fisheye lens that has been generally used and has the
`45
`relationship of h=f-6 is used. A curve C3 showsthe rela-
`relationship of h=f-@ is not satisfactory.
`tionship betweenficld angle @ and image height h when a
`In contrast to that, the image height he of the image Me
`fisheye lens having the relationship of h=2f-sin(6/2) is used,
`at a peripheral portion of the fisheye lens 1 having the
`a curve C4 showsthe relationship betweenfield angle 6 and
`relationship of h=2f-tan(6/2) in accordance with the present
`image height h when a fisheye lens having the relationship
`invention is enlarged and larger than the image height ho of
`of h=2f-sin0 is used, and a curve C5 showsthe relationship
`the image Moneartheoptical axis, a larger volume of image
`betweenfield angle 8 and image height h whena fisheye lens
`data can be obtained in comparison with the conventional
`having the relationship of h=f-tan6 is used. A curve Cl'
`fisheye lens, and the obtained imageis not distorted.
`shows the relationship of h=3f-tan (0/3) and a curve C"
`Whena single spherical image obtained by combining
`showsthe relationship of h=2f-tan (6/1.6).
`two hemispherical imagesofall the directionsof the field of
`FIG. 3(B),is a diagram showingthe relationship between
`view around the optical axis of the fisheye lens 1, which are
`field angle 0 and image height whenfisheye lenses having
`picked up at a field angle of 90° with respect to the optical
`relationships of h=2f-tan (6/1.6), h=2.4f-tan(6/2), h=3.4f-tan
`axis is converted into a plane image by the image data
`(6/3), h=1.2f-tan (6/1.6), h=1.6f-tan (6/2) and h=f-6. All of
`processing unit 3, it is necessary to interpolate missing
`these fisheye lenses, except h=f-0, are embodied by the
`image data on the peripheral portion (field angle of around
`present invention.
`90° with respect to the optical axis) of the image. According
`to the present invention, since an image at the peripheral
`As is evident from FIG. 3, an increase in image height h
`portion is enlarged and a large volume of data on the
`at a field angle 6 of about 90° is largest whenthe fisheye lens
`peripheral portion can be extracted, the volume of image
`having the relationship of h=f-tan 6 is used and is second
`data to be interpolated can be greatly reduced, when com-
`largest when the fisheye lens having the relationship of
`pared with the conventional system.
`h=2f-tan(@/2)
`is used. Changes in image height h with
`respect to changes in field angle @ become linear when the
`An imageofall the directions ofthe field of view around
`the optical axis is picked up at a field angle of at least 90°
`fisheye lens having the relationship of h=f-0 is used and
`
`5
`
`10
`
`15
`
`30
`
`35
`
`40
`
`50
`
`55
`
`60
`
`65
`
`13
`
`13
`
`
`
`6,128,145
`
`7
`with respect to the optical axis and is polar-coordinate
`converted into a plane image in the following manner.
`An X, Y and Z coordinate system as shown in FIG. 5 is
`imagined in subject space. At this point, the optical axis of
`the fisheye lens 1 is made Z axis. The coordinates of a
`certain point p are represented as (X1, Y1, Z1) and the
`elevation angle of the point p from the origin O of the
`coordinates with respect to the XZ plane is represented by @.
`The elevation angle of the point p from the position of Z1 on
`the Z axis with respect to the XZ plane is represented by @.
`When an x and y coordinate system having the optical
`axis (Z axis) as an origin o is imagined on the surface of
`CCD image pick-up elements 30 as shown in FIG. 6 and the
`focal distance of the fisheye lens 1 is represented byf, the
`image formation point(p') for the point p is located as shown
`in FIG. 6. In FIG. 6, 7 is added to g because an image formed
`at the point p' is inverted vertically and horizontally with
`respect to the image of the subject surface (point p). The
`optical axis in FIG. 6 is present in a direction perpendicular
`to the paper from the origin o of the x and y coordinates.
`The position of the point p' is expressed as polar coordi-
`nates with a length (h) between the origin o and the point p’
`and an angle ¢+2 formed by op' and the x axis. When the
`polar coordinates are expressed on the x and y rectangular
`coordinates, the position (x1, y1) on the x and y rectangular
`coordinates are expressed as follows.
`
`10
`
`15
`
`x1=h-cos (+n)
`
`yl=h-sin (6+)
`
`cD)
`
`2)
`
`30
`
`In addition, the image height h of the point p' is represented
`by h=2f-tan(6/2), hence, when h=2f-tan(6/2) is substituted
`into the above expressions (1) and (2), the coordinates (x1,
`yl) of the image formation point p' on the surface of the
`CCD image pick-up elements 30 are as follows.
`
`x1=2ftan (0/2)-cos (d+n)
`
`y=2ftan (8/2)sin (64m)
`
`As a result, they are expressed as follows.
`x=-2ftan (0/2)-cos ¢
`
`y=-2ftan (0/2)-sin
`
`3)
`
`(4)
`
`(5)
`
`(6)
`
`In the above expressions, 6 and ¢ are defined as follows.
`@stan? (VX1*¥"/Z1)
`
`g=tan™ (Y1/X1)
`
`Thus, the position of the point p' on the surface of CCD
`image pick-up clements 30 can be obtained for the point p
`on the surface of the subject.
`Thereafter, a description is subsequently given, with ref-
`erence to FIG. 7, of steps required when the sphere (all
`directions) is photographed by the camera 2 comprising the
`fisheye lens 1 and an image thereof is displayed on the
`monitor unit 4 that is the display unit.
`First, a hemisphere in one direction is photographed by
`the camera 2 comprising the fisheye lens 1 (step S1).
`Thereby, an image of the hemisphere is formed on the
`surface of the CCD image pick-up elements 30 as a polar-
`coordinate converted image. Thereafter,
`the camera 2 is
`turned at an angle of 180° to photograph the other hemi-
`sphere in the opposite direction (step $2). Thereby, an image
`ot the other hemisphere is formed on the surface of the CCD
`image pick-up elements 30 as a polar-coordinate converted
`image.
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`8
`The above two imagesare then combined together and the
`combined image is converted into a plane image by the
`image data processing unit 3 (step $3). At this point, an area
`corresponding to a connection portion between these hemi-
`spheres must be corrected. Since the polar-coordinate con-
`verted image obtained by the fisheye lens 1 has a large
`volume of information on a peripheral portion, the process-
`ing of combining these imagesis easy. Thereafter, a prede-
`termined portion of the thus obtained plane image is
`extracted and displayed on the monitor unit 4 (step S4).
`A user shifts the screen with instruction means such as a
`
`mouse when the user likes to change the displayed prede-
`termined portion. This shifting can be made continuously in
`any direction of 360° around the portion displayed on the
`monitor unit 4 (step $5).
`The above steps are for picking up an image of a sphere
`in all the directions of 360°. When only a single hemisphere
`is photographed, the samesteps are taken. However, step S2
`is unnecessary and the processing of combining two images
`in step S3 is also unnecessary.
`In the present invention, since the volume of information
`on the peripheral portion of an image obtainedbythe fisheye
`lens 1 is large, that is, an image at the peripheral portion is
`enlarged, it is convenient when the present invention is used
`for the examination of a product. For example, when the
`inner surface 52 of a cylindrical body 51 is photographedat
`the condition that the optical axis of the fisheye lens 1 ofthe
`present invention is aligned with the central axis of the
`cylindrical body 51 as shown in FIGS. 8(A) and 8(B), a
`peripheral portion of an image can be extracted as an image
`having a larger volume of information than a central portion
`in the present invention. Therefore, it is easy to find that a
`scratch has been generated in the inner surface 52.
`Consequently,
`this can be used for the examination of a
`pipe-like body such as a water pipe or gas pipe and further
`for the monitoring of a crack that has been generated in the
`wall surface of a tunnel or the like.
`It can also be used for the examination of the connection
`
`condition of a small part such as an IC. That is, when a part
`71 is fixed to a substrate 74 by a solder 73 on its both sides
`72 and 72 shown in FIGS. 9(A)and 9(B), the soldering state
`of the part must be checked from its side direction in the
`prior art. However, when another part 75 is existent in a side
`direction, the solder 73 on the part 75 side cannot be seen
`through a camera, thereby making automatic examination
`difficult. On the other hand, even whenthe fisheye lens 1 is
`installed right over the part 71 as shown in FIG. 9(A), the
`side direction of the part 71 can be sufficiently photographed
`by the camera 2 comprising the fisheye lens 1 of the present
`invention, thereby enabling automatic examination with the
`camera 2.
`
`Although each foregoing embodimentis an example of a
`preferred embodimentofthe invention,it is to be understood
`that the invention is not limited thereto and that various
`
`changes and modifications may be made in the invention
`without departing from the spirit and scope thereof. For
`example, the fisheye lens may be constructed by the attach-
`ment
`lens unit 20 alone without the master lens 10, or
`contrariwise may be constructed by an integrated unit of the
`master lens unit 10 and the attachmentlens unit 20. Also, the
`construction and numerical values of the fisheye lens 1
`shown in the above embodiment are just examples and a
`fisheye lens having other construction and numerical values
`may be used.
`
`14
`
`14
`
`
`
`6,128,145
`
`10
`
`9
`Whatis claimed ts:
`Further, as the system comprising, the fisheye lens 1 of the
`1. An image pick-up device, comprising:
`present invention, an image processing system 81 shown in
`FIG. 10 may be used. This image processing system 81 is
`a fisheye lens for picking up an imageof all directions of
`mainly constructed with a camera 2 equipped with a fisheye
`a field of view around an optical axis of said fisheye
`lens atafield angle of at least 90° in each direction with
`lens 1 and an image data processing unit/monitor 5 con-
`nected to the camera 2 by a cable. The image data processing
`respect to the optical axis,
`unit/monitor 5 is a personal computer equipped with a
`the fisheye lens having a relationship of h=n-f-tan(6/m), h
`monitor, and a key board 5a and a mouse 5b are connected
`being a height of an image of a subject at a predeter-
`to the computer as instruction means.
`mined point obtained bythe fisheyelens, f being a focal
`Also, this image data processing unit/monitor 5 has a hard
`distance of the fisheye lens, @ being a field angle, m
`disk (not shown) in that
`the contents of an information
`having a value of 1.6Sm<3, and n having a value of
`m-0.4<n=m+0.4.
`recording medium (floppy disk) 6 recording a program for
`executing the steps S3, S4 and S5 shownin FIG. 7 are to be
`2. The image pick-up device of claim 1, wherein m andn
`installed. By installing this program in the image data
`are substantially equal.
`processing