United States Patent [191
`Ueno et a1.
`
`US005293427A
`[11] Patent Number:
`[45] Date of Patent:
`
`5,293,427
`Mar. 8, 1994
`
`[54] EYE POSITION DETECTING SYSTEM AND
`METHOD THEREFOR
`
`FOREIGN PATENT DOCUMENTS
`
`
`
`[75] Inventors: Hiroshi Ueno, Yokohama; Kazuhiko Yoshida, Yokosuka, both of Japan
`
`
`
`?g: 61-77705 4/1986 Japan .>
`
`.
`_
`.
`[73] Assignee. Nissan Motor Company, Ltd.,
`Yokohama, Japan
`[211 Appl- N<>-= 805,143
`[22] Filed:
`Dem 11, 1991
`
`[331cc 14 15;)"; Ania?” Pmnty D.“ 2402512
`
`.
`
`H .
`
`.
`
`.
`
`p """""""""""" "
`’
`'
`[51] Int. Cl.5 ........................... .. G06K 9/00
`[52] US. Cl. ......................................... .. 382/1; 382/6;
`382/48
`[58] Field of Search ..................... .. 351/208, 209, 210;
`382/1, 2, 48, 28, 6; 340/575, 576
`R f n
`Cited
`U S PA’SEeQCSOCI-IMENTS
`-
`-
`4,625,329 11/1986 Ishikawa et al. ...................... .. 382/1
`4,8l2,033 6/1989 Ishikawa .......... ..
`351/208
`5,008,946 4/1991 Ando ..................................... .. 382/2
`
`56
`[
`1
`
`Pri
`
`Exa '
`
`61-77706 4/1986 Japan .
`6443491 1/1989 hpan _
`J
`h
`—-
`M
`2112231 Em'i'éiiiriipcm‘?‘m
`Attorney, Agent, or Firm-Lowe, Price, LeBlanc &
`Becker
`[57]
`
`ABSTRACT
`
`The invention detects the position of an eye in image of
`a driver’s face quickly and with high reliability, even if
`the density of the driver’s face image is not uniform on
`the right and left sides due to direct sunlight. Only one
`eye detection area is obtained on the dark side of the
`face image when the density of the face image is not
`uniform on the right and left sides. One eye window is
`degected the (1:1: elye detection1 area, aréd an iris center
`t e eye win ow to iscriminate an
`wit '
`is etect
`inattentive drive’ wh° is “Zing
`
`9 Claims, 15 Drawing Sheets
`
`\- 5
`
`TIMING
`C K T
`
`TV CAMERA
`
`A D
`CONVERTOR
`
`/( 7
`
`/( 9
`
`_ IMAGE
`MEMORY
`i
`EYE wmoow /10
`DETECT SEC
`1
`
`1 3
`l R I S
`DETECT SEC /
`i
`INATTEN DRIVEVHS
`DIS CR SEC
`
`Page 1 of 26
`
`SAMSUNG EXHIBIT 1008
`Samsung v. Image Processing Techs.
`
`

`

`US. Patent
`
`Mar. 8, 1994
`
`Sheet 1 of 15
`
`5,293,427
`
`F I G. I
`
`II
`
`A-D
`CONVERTOR
`
`/(3
`TV CAMERA
`
`IMAGE
`MEMORY
`I
`EYE WINDOW
`DETECT SEC
`I
`
`IRIS
`DETECT SEC
`I
`INATTEN DRIVE
`DISCR SEC
`
`SAMSUNG EXHIBIT 1008
`Page 2 of 26
`
`

`

`US. Patent
`
`Mar. 8, 1994
`
`Sheet 2 of 15
`
`5,293,427
`
`FIG.2
`'10
`FROM 9
`I /
`IMAGE INPUT MEANS
`
`II
`EYE DETECT AREA
`DECIDE MEANS
`
`II
`EYE WINDOW
`DETECT MEANS
`I
`TO 13
`
`FIG.2A
`
`FROM 101
`102A
`FROM 101
`I / 102B 1
`‘ \ I
`DENSITY DETECT
`BINARIZING MEANS
`MEANS
`
`1028 /
`
`I
`DENSITY CHANGE
`POINT DETECT MEANS
`
`I
`EYE DETECT AREA
`DECIDE MEANS
`I
`TO 103
`
`I
`
`COMPARE MEANS
`
`I
`EYE DE TECT AREA
`DECIDE MEANS
`I
`TO 103
`
`SAMSUNG EXHIBIT 1008
`Page 3 of 26
`
`

`

`US. Patent
`
`Mar. 8, 1994
`
`Sheet 3 of 15
`
`5,293,427
`
`FIG.3
`
`(
`
`START
`
`I
`
`II
`ACTIVATE STROBE TO
`/SI
`TAKE FACE IMAGE
`
`I
`STORE EYE IMAGE
`FOR EACH FRAME
`
`I
`READ EYE IMAGE DATA
`/S3
`
`I
`DEC I DE EYE
`DETECT AREA
`
`/(54
`
`I
`
`DETECT EYE WINDOW fss
`
`I
`DETECT IRIS CENTER /S6
`
`DISCR I M I NATE
`INATTENTIVE DR IVE J57
`
`SAMSUNG EXHIBIT 1008
`Page 4 of 26
`
`

`

`US. Patent
`
`Mar. 8, 1994
`
`Sheet 4 of 15
`
`5,293,427
`
`FIG.3A
`
`FROIM S3
`
`DETERMINE
`THRESHOLD
`
`f S4A-2
`,
`BINARIZE EYE
`IMAGE DATA
`
`/f. S
`y
`DETECT DENSITY
`CHANGE POINTS
`f" S 4A-4
`II
`DECIDE EYE
`DETECT AREA
`
`I
`
`FIG.3B
`
`FROM 53
`l f 548 -1
`DETECT IMAGE
`DENSITIES
`
`COMPARE
`DENSITIES
`"
`j”
`DECIDE EYE
`DETECT AREA
`
`I
`
`SAMSUNG EXHIBIT 1008
`Page 5 of 26
`
`

`

`US. Patent
`
`Mar. 8, 1994
`
`Sheet 5 Or 15
`
`5,293,427
`
`FIG.4
`
`(520,500)
`
`SAMSUNG EXHIBIT 1008
`Page 6 of 26
`
`

`

`U.S. Patent
`
`Mar.8, 1994
`
`Sheet 6 of 15
`
`5,293,427
`
`STEP S4A-4
`
`oo
`
`FIG.
`
`S
`
`STEP _S4A-3
`
`S101
`
`S106~
`
`119s
`
`(X-5,Y) =4u(X,YO
`3111
`
`Y=200
`$105
`$102
`~>—ES—<“ 7-500>
`$124
`<-20F>S04||_X20
`
`YES|X=X+1 ==
`S1255
`YES SexeaNO
`X=X+1
`E107
`Xed= 4 Xed/n S109\
`XB = AXB/n
`i Co
`|| Xst= 4Xst/n
`| NO
`“TYEs
`AJ(X,Y)=
`AM(XY)"AJ (X,Y)
`| AJ(X,Y) +3 (X,Y)
`~Jmo(X,¥)+J (X,Y)
`wn;
`ae oO Co \
`nnwnR/\BJ— Q oO we)nHul
`
`UnoO
`0 NOThtget>S112, {YES
`
`Ri S113.
`“yes
`maxJ=J(X-5,Y)
`YES
`S114
`7
`
`Zz oO S122.
`
`
`
`NO
`
`|
`
`$123
`AXB=AXB+(x-5)
`AXst=AXst +Xst
`AXed=AXed+Xed
`
`S115
`
`|NO
`
`Ht
`
`YES
`Xst =X
`
`- S117
`
`NO
`
`Lffg=1 Xed=X|
`
`S118.
`
`TYES
`
`SAMSUNG EXHIBIT 1008
`Page 7 of 26
`
`SAMSUNG EXHIBIT 1008
`Page 7 of 26
`
`

`

`US. Patent
`
`‘ Mar. 8, 1994
`
`Sheet 7 of 15
`
`5,293,427
`
`F I G. 6
`
`5 “DENSITY CHANGE
`5
`PO
`
`SAMSUNG EXHIBIT 1008
`Page 8 of 26
`
`

`

`U.S. Patent
`
`Mar. 8, 1994
`
`Sheet 8 of 15
`
`5,293,427
`
`STEP $4B-2,3
`
`stig-N|
`——_—_____—
`
`NO S414 !
`/
`[stig=R]
`[sttg=L
`
`ooo
`S420
`
`YES JR=AJRI250||
`.
`JL=AJL/250|| 2402
`
`
`
`41] ge
`
`SR so"
`
`T
`
`941|X=0|RETURN
`
`VESeeNOeve] 200
`
`i
`
`Y=V +4
`
`S408
`
`$415
`
`S416
`
`6$42)
`
`£9423
`Yes
`AJ(X,Y) = 4) (X,Y)
`-jJmo(X,Y)+J(X, Y)
`
`J(X-5, Y)=JCX,Y)M0
`($425
`
`irs
`yes_(9427
`maxJ=J(X-5, Y)
`
`fcon= icon +1
`
`,
`($429
`
`S30
`
`i S
`
`bo
`
`AMSUNG EXHIBIT 1008
`Page 9 of 26
`
`SAMSUNG EXHIBIT 1008
`Page 9 of 26
`
`

`

`U.S. Patent
`
`Mar. 8, 1994
`
`Sheet 9 of 15
`
`5,293,427
`
`FIG.8
`STEP 5 (X)
`
`~
`
`START
`ra
`
`YES
`
`XL=250 XR=250
`$202
`
`Rot
`
`S203
`
`$200
`
`$201
`
`No
`
`
`XR=XRM
`
`YES
`
`$210
`
`NO ;5211
`
`-$206
`
`S207
`
`No
`
`.
`
`S214
`
`$215
`
`2
`|
`
`WXCL= WXCL-1<KL>XLM TEP
`KLXL
`He,Sa
`
`
`O
`
`NO
`
`$220
`
`OKL=0 OKR=0
`
`|
`
`||
`
`fveyer(|9922
`7 Din
`$223
`RETRIEVE ENDS?
`¥=500)
`YES
`
`NO
`
`SAMSUNG EXHIBIT 1008
`Page 10 of 26
`
`SAMSUNG EXHIBIT 1008
`Page 10 of 26
`
`

`

`U.S. Patent
`
`Mar. 8, 1994
`
`Sheet 10 of 15
`
`5,293,427
`
`FIG. 9A
`
`STEP 5(Y)
`
`- $301
`
`
`BY 1MAX=0
`
`BY 2MAX=0
`
`| XCHECK = X2-10 (Xx1#10)
`_YCHECK= YL
`
`$313 ~
`
`{YES
`
`
`peeSS
`ae
`
`SAMSUNG EXHIBIT 1008
`Page 11 of 26
`
`SAMSUNG EXHIBIT 1008
`Page 11 of 26
`
`

`

`U.S. Patent
`
`Mar, 8, 1994
`
`Sheet 11 of 15
`
`5,293,427
`
`F1G.9B
`STEP 5(Y)
`
`@
`
`
`
`$330
`
`
`
`
`
`
`
`:
`
`| i
`
`!
`
`
`
`9340,
`
`“Tres
`
` @
`
`XC= X2
`(XC=XX1)
`
`$333
`
`$335
`
`YES $9936
`BLACKX =
`BLACKX +1
`
`XC=XC +1
`(XC=XC-1)
`
`$337 |
`!
`
`|
`
`¢ $338
`
`SAMSUNG EXHIBIT 1008
`Page 12 of 26
`
`SAMSUNG EXHIBIT 1008
`Page 12 of 26
`
`

`

`US. Patent
`
`Mar. 8, 1994
`
`Sheet 12 of 15
`
`5,293,427
`
`FIG.1O
`
`RETRIEVE START LINE
`(X=250)
`
`) 0.
`mw
`
`(0,500) Y
`
`FIG.11
`
`SAMSUNG EXHIBIT 1008
`Page 13 of 26
`
`

`

`US. Patent
`
`Mar. 8, 1994
`
`Sheet 13 of 15
`
`5,293,427
`
`SAMSUNG EXHIBIT 1008
`Page 14 of 26
`
`

`

`US. Patent \
`
`Mar. 8, 1994
`
`Sheet 14 of 15
`
`5,293,427
`
`FIG.13
`STEP 56
`
`LOOP FDR
`$11?
`
`y = 1
`
`$902
`
`X
`
`£
`Y
`N
`
`Y_E_S
`R=Rmin
`+
`A = 0
`=i
`P = o
`“
`
`5903
`
`$904 '
`/
`59,05
`/
`
`fSQDG
`
`SAMSUNG EXHIBIT 1008
`Page 15 of 26
`
`

`

`US. Patent
`
`Mar. 8, 1994
`
`Sheet 15 of 15
`
`5,293,427
`
`FIG.14
`
`IMAGE SIG INPUT
`TIMING
`
`'
`1
`l
`
`?
`I
`I
`I
`
`I
`
`I
`I
`'
`
`i
`I
`I
`
`I
`l
`l
`
`(TIME)
`
`CLIOSE cL'osE cLosE
`cLdsE cLbsE CLIIOSE CLOSE
`O "U -________ ITI Z
`OFI’EN
`
`I
`
`1
`DIOZE
`
`|
`I
`|
`l
`l
`l
`
`a
`DOZE DOZE
`
`SAMSUNG EXHIBIT 1008
`Page 16 of 26
`
`

`

`1
`
`EYE POSITION DETECTING SYSTEM AND
`METHOD THEREFOR
`
`5
`
`20
`
`BACKGROUND OF THE INVENTION
`1. Field of the Invention
`The present invention relates to an eye position de
`tecting system, and more speci?cally to an eye position
`detecting system suitable for use in an automotive vehi
`cle, to detect driver’s inattentive driving due to a doze
`or looking aside.
`2. Description of the Prior Art
`The background of the present invention will be
`explained with respect to its application to the system
`used with an automotive vehicle.
`As the ?rst type of prior-art driver’s eye position
`detecting systems for automotive vehicles, driver’s eye
`position recognizing systems have been proposed as
`disclosed in Japanese Unexamined Published (Kokai)
`v Pat. Appli. Nos. 60-158303, 60-158304, 61-77705 and
`61-17706, in which an upper half of a driver face is
`irradiated with infrared rays emitted from two infrared
`stroboscopes; the re?ected light is detected by two
`image sensors; two dark areas within the detected
`25
`bright face image are extracted as two singular points;
`and the two extracted singular points are recognized as
`two eyes.
`_
`Further, as the second type of prior-art systems for
`automotive vehicles, an electric apparatus energizing
`control system has been proposed as disclosed in Japa
`nese Unexamined Published (Kokai) Pat. Appli. No.
`64-13491, in which the change in driver’s eye conditions
`or the shape of driver’s mouth speaking conditions is
`recognized to turn-on or off various electric apparatus
`such as buzzers, meters, automatic doors, relays, moni
`tor TVs, etc.
`In these prior-art eye position detecting systems, two
`eye existing areas are ?rst decided before detecting eye
`positions. That is, a head top position of a driver face
`from a ?xed position is detected in a face image: the
`detected face is divided into a few (e.g., three) areas;
`appropriate threshold values are determined for each
`areas respectively; and the lightness can be obtained as
`binary image signals to decide the eye existing areas.
`In the ?rst-type prior-art eye position detecting sys
`tem, however, since the two dark areas within a closed
`bright area in a driver’s face image are recognized as
`eye positions of singular points, when a half side of the
`driver’s face is irradiated with direct sunlight, since the
`optical density (lightness) gradation on the driver face is
`excessively high and therefore is not uniformly de
`tected, it is impossible to distinguish dark areas from
`bright areas so that eye positions cannot be detected.
`Further, in the second-type prior-art eye position
`detecting system, when the external light conditions
`vary complicatedly during vehicle travelling, there
`exist various problems in that the driver’s head position
`cannot be detected within the face image (being out of 60
`the detected image) so that the eye positions cannot be
`detected, and therefore the threshold levels must be
`adjusted many times according to the light density gra
`dation in order to obtain binary image signals, thus
`resulting in slow response speed.
`In summary, in the prior-art eye position detecting
`system, when the driver face is irradiated with direct
`sunlight, it has been difficult to accurately detect the
`
`5,293,427
`2
`driver eye position at high response speed and with high
`reliability.
`SUMMARY OF THE INVENTION
`With these problems in mind, therefore, it is the ob
`ject of the present invention to provide an eyeball posi
`tion detecting system which can accurately detect a
`driver’s eye existing area (eye window) even if the driv
`er’s face is exposed directly to the sun so that the optical
`density (brightness) gradation is not uniform on the
`right and left sides of the face image.
`To achieve the above-mentioned object, the present
`invention provides an eye position detecting system for
`detecting an eye position in a face image taken by a
`camera, comprising: (a) inputting means for inputting
`face image signals as light density gradation data; (b)
`deciding means for obtaining a one eye detection area
`on a dark side in the face image when the inputted light
`density gradation data are not uniform right and left
`beyond a predetermined density level; and (c) detecting
`a means for detecting one eye window in the one eye
`detection area in the face image when not uniform.
`In the ?rst modi?cation, the deciding means com
`prises: (a) binarizing means for binarizing the inputted
`face image signals on the basis of a predetermined
`threshold level; (b) detecting means for detecting den
`sity change points in a vertical direction on the basis of
`the binarized image signals to check whether the light
`density is not uniform right and left in the face image;
`and (c) obtaining means for deciding only one eye de
`tection area on one dark side in the face image when the
`number of the detected density change points exceeds a
`predetermined value. In the second modi?cation, the
`deciding means comprises: (a) density detecting means
`for detecting light density on the basis of the image
`signals on both sides in the face image, separately; (b)
`comparing means for comparing the detected light den
`sities on both the sides; and (c) obtaining means for
`deciding only one eye detection area on one dark side in
`the face image when a difference in light density be
`tween both sides in the face image exceeds a predeter
`mined level.
`When the eye position detecting system is applied to
`an inattentive driver preventing system, the system
`further comprises iris detecting means for detecting an
`iris center within the one eye window and discriminat
`ing means for discriminating on the basis of lightness
`change at the detected iris.
`Further, to achieve the above-mentioned object, the
`present invention provides a method of detecting an eye
`position in a face image, comprising the steps of: (a)
`irradiating the face with infrared rays; (b) taking face
`images by a camera in synchronism with the infrared
`rays; (c) obtaining a one eye detection area on a dark
`side in the face image when the face image is not uni
`form in light density on right and left sides beyond a
`predetermined density level; (d) detecting one eye win- '
`dow in the decided one eye detection area in the face
`image; (e) detecting an iris center within the detected
`eye window to discriminate an inattentive driver on the
`basis of change in lightness at the detected iris.
`In the first modification, the step of obtaining one eye
`detection area comprises the steps of: (a) determining a
`threshold level; (b) binarizing the face image signal on
`the basis of the determined threshold level; (0) detecting
`density change points in a vertical direction on the basis
`of the face image signals to check whether face light
`density is not uniform on right and left sides in the face
`
`35
`
`45
`
`55
`
`65
`
`SAMSUNG EXHIBIT 1008
`Page 17 of 26
`
`

`

`5,293,427
`4
`3
`image; and (d) deciding only one eye detection area on
`FIG. 11 is another illustration for assistance in ex
`plaining the processing of detecting the horizontal and
`one dark side in the face image when the number of the
`density change points exceeds a predetermined value.
`vertical positions of an eye window;
`FIG. 12 is an illustration for assistance in explaining
`In the second modi?cation, the step of obtaining one
`the principle of detecting an iris center;
`eye detection area comprises the steps of: (a) detecting
`FIG. 13 is a ?owchart for assistance in explaining the
`light density on the basis of the face image signals on
`procedure of detecting an iris; and
`both sides in the face image, separately; (b) comparing
`FIG. 14 is an illustration for assistance in explaining
`the detected both-side light densities; and (c) obtaining
`the discrimination of a doze.
`only one eye detection area on one dark side in the face
`image when a difference in light density between both
`sides in the face image exceeds a predetermined level.
`In the eye position detecting system according to the
`present invention, when the light density in the driver
`face image is not uniform between the right and left side
`due to direct sunlight, only one eye detection area is
`decided on the dark side of the face image, and one eye
`window is detected in the decided eye detection area.
`Further, an iris center is detected within the detected
`eye window to detect an inattentive driver due to doz
`ing forv instance. Since it is unnecessary to obtain an eye
`detection area or to detect an eye window on the exces
`sively bright side in the face image, the image signal
`retrieval area can be reduced and further the eye posi
`tion detection precision can be improved.
`
`25
`
`DETAILED DESCRIPTION OF THE
`PREFERRED EMBODIMENTS
`Embodiments of the eye position detecting system
`suitable for use in an automotive vehicle according to
`the present invention will be described hereinbelow
`with reference to the attached drawings.
`In FIG. 1, the system comprises an infrared strobe 1.
`a TV camera 3, a timing circuit 5, an A-D convertor 7,
`an image memory 9, an eye existing area (window)
`detecting circuit 10, an iris detecting circuit 13 and an
`inattentive drive discriminating circuit 15. The infrared
`strobe 1 is disposed within an instrument panel (not
`shown) to transmit infrared rays upon a driver face
`looking correctly frontward. The TV camera 3 takes
`images of driver face irradiated with the infrared rays
`emitted from the infrared strobe 1. The timing circuit 5
`matches the timing at which the infrared rays are emit
`ted from the strobe 1 with that at which a driver face
`image is taken by the TV camera 3. That is, when a
`strobe activating command is outputted from the timing
`circuit 5 to the infrared strobe 1, the strobe 1 generates
`infrared rays to irradiate the driver face with the infra
`red rays. Simultaneously, when an image taking com
`mand is outputted from the timing circuit 5 to the TV
`camera 3, the camera 3 takes an image of the driver face
`irradiated with the infrared rays. The image taken by
`the TV camera 3 is composed of 520 horizontal (X)
`direction picture elements (pixels) and 500 vertical (Y)
`direction picture elements (pixels) as shownin FIG. 4
`and the ?eld angle of the image is so adjusted that the
`whole face can appear full in the face image.
`The A-D convertor 7 converts analog image signals
`obtained by the TV camera 3 into digital image signals,
`and the image memory 9 stores these digital image sig
`nals. The eye existing area (window) detecting section
`10 detects an area or areas within which two eyes exist
`on the basis of the image signals stored in the image
`memory 9. The iris detecting section 13 detects an iris
`position by processing the image signals within the area
`detected by the eye window detecting section 10. The
`inattentive drive discriminating section 15 discriminates
`the presence or absence of inattentive driving (e.g., a
`doze, looking aside) on the basis of the result detected
`by the iris detecting section 13.
`With reference to FIG. 2, the eye window detecting
`section 10 comprises image inputting means 10 for in
`putting image signals stored in the image memory 9,
`frame by frame as light density gradation data; eye
`detection area (dark face side) deciding means 102 for
`checking whether the inputted light density gradation
`data in the face image is uniform right and left, and
`deciding one eye detection area on a dark side in the
`face image when not uniform right and left beyond a
`predetermined level; and eye window detecting means
`103 for detecting one eye window in the decided one
`eye detection area in the face image when not uniform.
`In the ?rst modi?cation of the embodiment, as shown
`in FIG. 2A, the eye detection area deciding means 102A
`
`35
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`FIG. 1 is a schematic block diagram showing an
`embodiment of the eye position detecting system ac
`cording to the present invention;
`30
`FIG. 2 is a block diagram showing the eye window
`detecting section shown in FIG. 1;
`FIG. 2A is a detailed block diagram showing a ?rst
`modi?cation of the eye detection area deciding means
`shown in FIG. 2;
`FIG. 2B is a detailed block diagram showing a second
`modi?cation of the eye detection area deciding means
`shown in FIG. 2;
`FIG. 3 is a ?owchart for assistance in explaining the
`operation of the detecting system shown in FIG. 2;
`40
`FIG. 3A is a ?owchart for assistance in explaining a
`?rst modi?cation of the step of deciding eye detection
`area shown in FIG. 3;
`FIG. 3B is a ?owchart for assistance in explaining a
`second modi?cation of the step of deciding eye detec
`tion area shown in FIG. 3;
`FIG. 4 is an illustration for assistance in explaining a
`uniform lightness face image including two eye win
`dows;
`FIG. 5 is a ?owchart for assistance in explaining the
`procedure of the ?rst modi?cation of the step of decid
`ing eye detection area shown in FIG. 3A;
`FIG. 6 is an illustration for assistance in explaining a
`non-uniform lightness face image including one eye
`window;
`FIG. 7 is a ?owchart for assistance in explaining the
`procedure of the second modi?cation of the step of
`deciding eye detection area shown in FIG. 3B;
`FIG. 8 is a ?owchart for assistance in explaining the
`procedure of the step of deciding the horizontal posi
`tions of an eye window;
`FIGS. 9A and 9B are ?owcharts for assistance in
`explaining the procedure of the step of deciding the
`vertical positions of an eye window under consideration
`of the presence or absence of eyeglasses;
`FIG. 10 is an illustration for assistance in explaining
`the procedure of detecting a face width beginning from
`a central line;
`
`45
`
`55
`
`65
`
`SAMSUNG EXHIBIT 1008
`Page 18 of 26
`
`

`

`15
`
`20
`
`25
`
`5,293,427
`5
`6
`comprises binarizing means 102A-1 for binarizing the
`basis of the face image signal on both sides in the face
`inputted face image signal on the basis of a predeter
`image, separately (in step S4B-1). The detected light
`mined threshold level to emphasize the lightness and
`densities on both sides are compared to determine
`darkness of the face image; density change point detect
`which side is darker in the face image (in step S4B-2).
`ing means 102A-2 for detecting density change points in
`Further, only one eye detection area is obtained on one
`a vertical direction on the basis of the binarized image
`dark side in the face image when a difference in light
`signals to check whether the light density is not uniform
`density between both sides in the face image exceeds a
`right and left in the face image; and eye detecting area
`_ predetermined level (in step S4B-3), as shown in FIG. 6.
`deciding means 102A-3 for deciding only one eye detec
`FIG. 5 is a detailed ?owchart for assistance in ex
`plaining the steps of detecting density change points in
`tion area on one dark side in the face image when the
`number of the detected density change points exceeds a
`the face image (step S4A-3) and deciding one eye detec
`tion area (in step S4A-4) of the ?rst modi?cation. In this
`predetermined value.
`‘
`?owchart, control roughly executes the following
`In the second modi?cation of the embodiment, 8
`shown in FIG. 2B, the eye detection area deciding
`steps: (a) retrieving a predetermined number of bina
`means 102B comprises density detecting means 102B-1
`rized image signals at pixels in a horizontal direction
`for detecting light density on the basis of the image
`within a lower half area of the face image; (b) calculat
`signals on both sides in the face image, separately; com
`ing an average density on the basis of the retrieved
`paring means 102B-1 for comparing the detected both
`image signals; (c) checking whether the calculated aver
`side light density to check whether the face image is
`age density is beyond a predetermined value to deter
`uniform right and left; and eye detection area deciding
`mine a continuous bright portion along the horizontal
`means 102B-3 for deciding only one eye detection area
`direction; (d) checking whether the calculated average
`on one dark side in the face image when a difference in
`density is below another predetermined value to detect
`light density between both sides in the face image ex
`a density change point in the horizontal direction; (e)
`ceeds a predetermined level.
`repeating the above steps (a) to (d) along a ‘vertical
`The entire operation of the system will be explained
`direction; (t) counting the number of density change
`hereinbelow with reference to the flowchart shown in
`points along the vertical direction; (g) if the counted
`FIG. 3.
`number of the density change points is less than a prede
`Images of driver’s face area are taken by the TV
`termined value, determining that the light density is
`camera 3 in synchronism with infrared rays generated
`uniform on right and left sides of the face image to
`by the infrared strobe 1 (in step S1). The image signals
`decide two eye detection areas on both sides in the face
`are converted into digital image signals via the A-D
`image; and (h) if the counted number of the density
`converter 7 and then stored in the image memory 9
`change points is the predetermined value or more, de
`frame by frame (in step S2). In this conversion process,
`termining that the light density is not uniform right and
`the digital video signals are converted into 256 grada
`left in the face image to decide one eye detection area
`tion data from 0 to 255 where 0 denotes the darkest
`on the side where the continuous bright portion has not
`gradation and 255 denotes the whitest gradation. The
`been checked.
`image data stored in the image memory 9 are read and
`In more detail, control ?rst sets the vertical scanning
`inputted to the eye window detecting section 10 (in step
`coordinate value to y=200 (in step S101) and the hori
`S3) in order to decide one eye detection area on the
`zontal scanning coordinate value to X=0 (in step $102).
`darker side in the face image when the face image is not
`The reason why the vertical scanning coordinate value
`uniform in light density right and left beyond a prede
`is set to Y=200 is to securely detect density change
`termined density level (in step $4).
`points at only the lower half of the face. The horizontal
`scanning coordinate value X is incremented pixel by
`One eye window is detected in the decided one eye
`detection area in the face image (in step S5). An iris
`pixel (in step S104) until X reaches 520 to cover the
`center is detected within the detected eye window (in
`retrieve range (x=520) (in step S103).
`Further, whenever the image signals along the hori
`step S6) to discriminate an inattentive drive on the basis
`of change in lightness at the detected iris (in step S7).
`zontal line have been detected, the vertical scanning
`As shown in FIG. 3A, in the ?rst modi?cation of the
`coordinate value Y is incremented pixel by pixel (in step
`step of deciding one eye detection area (dark face side)
`S106) until Y reaches 500 to cover the retrieve range
`(step in S4), a threshold level is determined in accor
`(Y =500) (in step S105).
`dance with an appropriate image processing such as
`The controller checks whether the accumulated re
`discrimination analysis method, differential histogram
`tn'eved pixels in the X direction numbers 11 pixels or
`method, etc. (in step S4A-1). The face image signal is
`more (in step S107). If less than 11 pixels, control accu
`binarized on the basis of the determined threshold level
`mulates ten pixels (in step S108), where AJ(AJ(X,Y), AJ
`to emphasize the difference in lightness in the face
`(X-1,Y), . . .) denote an accumulated pixels. If 11 pixels
`image (in step S4A-2). Density change points are de
`or more (in step S107), an old pixel density J m (X, Y)
`tected in a vertical direction on the basis of the face
`(retrieved before ten pixels are retrieved) is omitted and
`image signals to check whether face light density is not
`a newly retrieved pixel density J (X, Y) is added (in step
`uniform right and left in the face image (in step S4A-3).
`S109). The new ten-pixel density thus obtained is aver
`When the number of the density change points exceeds
`aged to obtain an average density AI (X, Y)/l0 and is
`a predetermined value, since this indicates that the face
`set again as a current density J (X-S, Y) at a pixel 5
`image is not uniform on right and left sides in light
`pixels before the current pixel (in step S110).
`density, only one eye detection area is determined on
`The controller checks whether the newly set pixel
`one dark side in the face image (in step S4A-4), as
`density is 240 gradation or more (very bright) (in step
`shown in FIG. 6.
`65
`S111). If 240 gradation or more, the controller com
`As shown in FIG. 3B, in the second modi?cation of
`pares the maximum value max J of the densities so far
`the step of deciding one eye detection area (dark face
`stored with the current pixel density J (X-S, Y) (in step
`side) (in step S4), the light density is detected on the
`S112). If I (X-5, Y) is larger than max J, J (X-S, Y) is set
`
`40
`
`45
`
`55
`
`SAMSUNG EXHIBIT 1008
`Page 19 of 26
`
`

`

`15
`
`30
`
`5,293,427
`7
`to a new maximum pixel density max J (in step 113) and
`counts up a continuation counter 1m, indicative of a
`continuous bright portion (in step S114). The controller
`further checks whether the continuation counter 1m, is
`l (in step S115). If 1m,,=l, the coordinate value X at
`this moment is stored as a bright continuation start point
`X“ (in step S116), as shown in FIG. 6. If lmnis not 1 (in
`step S115), the controller checks whether the continua~
`tion number of the bright portion is 10 or more (in step
`S117). If 10 or more, a continuation flag L?ag is set and
`the coordinate value X at this time is stored as a bright
`continuation end point X“; (in step $118), as shown in
`FIG. 6.
`If the pixel density is 240 gradation or less in step
`$111, the continuation counter 1m is cleared (in step
`S119). Further, the controller checks whether the con
`tinuation ?ag Lflag is set (in step $120). If not set, since
`this indicates that the continuous portion is not yet
`detected, the controller proceeds to step 103 to retrieve
`the successive pixel. If set in step S120, the controller
`checks whether a difference between the maximum
`pixel density max J and the current pixel density J (X-S,
`Y) is 30 gradation or more (becomes darker) (in step
`S121). If less than 30 gradation (not darker), the control
`ler retrieves the succeeding pixel (in step $103). If gra
`dation is 30 or more (darker) in step S121, the controller
`determines the current pixel density as a density change
`point, counts up the density change point counter n (in
`step S122), accumulates the density change point coor
`dinate value X, the bright continuation start point X,,
`and the bright continuation and point X,,;, and stores
`these as AXB, AX,‘ and AX“; (in step $123). Under these
`conditions, since the density change point at the current
`retrieve line X has been detected, the controller pro
`35
`ceeds to step S105 to increment the retrieve line Y. The
`above-mentioned processing is repeated until Y=500
`(th lowermost point of the face image). That is, if
`Y=500 (in step S105), the controller proceeds to the
`steps of deciding an eye detection area (face side) (in
`steps S124 and after).
`The controller ?rst checks whether the density
`change point it is 20 or more (in step S124). If less than
`20, the controller sets the eye retrieve range ?ag S?g to
`N (in step S128). This indicates that the image density is
`uniform, so that the eye detection area is on both the
`sides of the face image, respectively. If n is 20 or more
`(in step S124), since the imagev is not uniform, control
`calculates an average coordinate value XB of the den
`sity change point, an average coordinate value X“ of the
`bright continuous portion start point and an average
`coordinate value X“; of the bright continuous portion
`end point by dividing these accumulated values AXB,
`AX," and Axed by detection number n of the density
`change points (in step S125).
`Thereafter, the controller calculates the continuous
`portion start point position LL and the same end point
`position LR with the X=250 as the central point of the
`image (in step $126). The controller compares the start
`point position LL with the end point position LR (in
`step $127). If LL> LR, since this indicates that the
`continuous portion is located on the left side, the con
`troller sets the eye retrieve range ?ag S?g to L (in step
`$129) to determine that the eye detection area is located
`on the right side of the density change point. If not
`LL>LR, since this indicates that the continuous por
`tion is located on the right side, the controller sets the
`eyeball retrieve range ?ag S?g to R (in step S130) to
`
`8
`determine that the eye detection area is located on the
`left side of the density change point.
`FIG. 6 shows an example of the face image obtained
`when the left half of the face image is directly irradiated
`with sunlight. Therefore, the density change points are
`denoted by white circles roughly at the center of the
`face image and the bright continuous portion is located
`on the left cheek as denoted by a start point X,‘ and an
`end point X04.
`FIG. 7 is a detailed ?owchart for assistance in ex
`plaining the steps of detecting image densities (step
`S4B-1), comparing the detected densities (step S4B-2)
`and deciding the eye detection area (step S4B-3) of the
`second modi?cation. In this ?owchart, the controller
`roughly executes the following steps: (a) retrieving a
`predetermined number of image signals at pixels in a
`horizontal direction within a lower halfarea of the face
`image beginning from the center to the right and left,
`respectively; (b) calculating an average density on the
`basis of the retrieved image signals on the right and left
`sides, respectively; (c) repeating the above steps (a) and
`(b) in a pre