Ulllted States Patent [19]
`Suenaga et al.
`
`USOOS 80572OA
`[11] Patent Number:
`[45] Date of Patent:
`
`5,805,720
`Sep. 8, 1998
`
`[54] FACIAL IMAGE PROCESSING SYSTEM
`
`A1 7/1995 Germany .
`404174309 6/1992 Japan ................................... .. 382/117
`
`
`
`Inventors: Nobumasa Suenaga; Kenji ogawa, both of Tokyo, Japan
`
`
`
`Japan ................................... .. 6-32154 2/1994 Japan .
`
`[73] Assignee: Mitsubishi Denki Kabushiki Kaisha,
`Toky0> Japan
`
`[21] Appl. No.: 612,815
`
`[22]
`
`Mar‘ 11’ 1996
`Flled:
`Foreign Application Priority Data
`
`Jul. 28, 1995
`
`[JP]
`
`Japan .................................. .. 7-193647
`
`[51] Int. Cl.6 ..................................................... .. G06K 9/00
`[52] US. Cl. ......................... .. 382/117; 351/209; 340/575
`[58] Field of Search .......................... .. 382/117—118, 103,
`382/104; 351/209_211; 340/575, 576; 128/745;
`180/272; 364/423098, 424051
`
`[56]
`
`References Cited
`
`U'S' PATENT DOCUMENTS
`6/1993 Ueno etal. ........................... .. 351/210
`5,218,387
`3/1994 Ueno et al. ........................... .. 382/117
`5,293,427
`6/1995 Rothberg et al.
`351/209
`5,422,690
`345/156
`5,455,601 10/1995 OZaki ------------- -
`5,570,698 11/1996 Liang et a1. .......................... .. 340/575
`FOREIGN PATENT DOCUMENTS
`
`0552770 7/1993 European Pat- Off- ------------- -- 382/118
`44 41 332 A1 11/1993 Germany-
`195 O9 689
`A1 3/1994 Germany .
`196 13 614
`A1 5/1995 Germany .
`196 03 287
`
`OTHER PUBLICATIONS
`Ueno et al, Development of drowsiness detection system,
`Vehicle Navigation & Information Systems, pp. 15—20, Jul.
`1994_
`
`Primary Examiner—Leo H. Boudreau
`Assistant Examiner—Bhavesh Mehta
`Attorney, Agent, or Firm—Sughrue, Mion, Zinn, Macpeak
`& Seas, PLLC
`
`[57]
`
`ABSTRACT
`
`Afaeial image preeessihg System Of the present invention is
`provided With an evaluation function calculation device
`Which calculates shape functions (for example, an upper
`boundary and an approximate line representing a predeter
`mined range of the upper boundary) representing the shape,
`namely, the feature of the entire eye image from an eye
`presence area and further calculates an evaluation function
`(namely, the gradient of the ePPreXiIhete 1ihe) from the
`shape function (namely, the approximate line). The facial
`image processing system is further provided With an opened
`or-closed condition judgment device Which determines a
`threshold Value from the Values K of the time_varying
`evaluation function and makes a judgment on Whether the
`eye is opened or closed. Thereby, even in the case that a
`change in the number of continuous black piXels arranged in
`Y-direction When closing the eye is small oWing to an
`eyebroW, it can be accurately judged on Whether the eye is
`Opened or closed'
`
`32 Claims, 65 Drawing Sheets
`
`CAMERA
`
`IMAGE INPUT
`
`V11
`(V12
`
`EYE EXTRACTION ~14
`MEANS
`
`EVALUATION FUNCTION
`~15
`CALCULATION MEANS
`
`OPENED-OR-CLOSED
`CONDITION
`JUDGMENT MEANS
`
`Page 1 of 80
`
`SAMSUNG EXHIBIT 1007
`Samsung v. Image Processing Techs.
`
`

`

`U.S. Patent
`
`Sep. 8, 1998
`
`Sheet 1 0f 65
`
`5,805,720
`
`FIG. 1
`
`CAMERA ~11
`
`IMAGE INPUT
`MEANS
`
`12
`
`?lNARIZATION ~13
`
`MEANS
`
`1 4
`EYE EXTRACTION
`MEANS W
`
`15
`/ EVALUATION FUNCTION
`CALCULATION MEANS “T
`
`/ OPENED-OR-CLOSED
`CONDITION
`JUDGMENT MEANS
`
`16
`“T
`
`SAMSUNG EXHIBIT 1007
`Page 2 of 80
`
`

`

`U.S. Patent
`
`éOld
`
` OLalVL
`
`SWIL
`
`
`
`
`
`3AaONISOTONAHMJA3dONINSdONAHM
`Woee,en,
`
`Sep. 8, 1998
`
`Sheet 2 of 65
`
`5,805,720
`
`WAS
`
`WnsWns
`
`NOILOAYIG-A
`
`SAMSUNG EXHIBIT 1007
`Page 3 of 80
`
`SAMSUNG EXHIBIT 1007
`Page 3 of 80
`
`
`

`

`U.S. Patent
`
`Sep. 8,1998
`
`Sheet 3 0f 65
`
`5,805,720
`
`
`
`NOLLONfH NOILVH'IVAH :10 >| El?‘IVA
`
`FIG. 3
`
`TIME
`
`SAMSUNG EXHIBIT 1007
`Page 4 of 80
`
`

`

`Sep. 8, 1998
`
`Sheet 4 of 65
`
`{i_——
`
`enei
`
`aD
`
`
`
`JAZONINSdON3IHM
`
`eed
`
`WIL
`
`U.S. Patent
`
`vyOla
`
`
`
`4AaONISOTON3HM
`
`
`aaiWL
`
`5,805,720
`
`Xx¥oz+17xXuyo2+174Xuyoc+11000
`
`osonpion
`
`SAMSUNG EXHIBIT 1007
`Page 5 of 80
`
`SAMSUNG EXHIBIT 1007
`Page 5 of 80
`
`
`

`

`U.S. Patent
`
`Sep. 8,1998
`
`Sheet 5 0f 65
`
`5,805,720
`
`
`
`NOLLONFH NOliV?‘lVAEl :10 >! EIFI'IVA
`
`FIG. 5
`
`TIME
`
`SAMSUNG EXHIBIT 1007
`Page 6 of 80
`
`

`

`U.S. Patent
`
`Sep. 8,1998
`
`Sheet 6 0f 65
`
`5,805,720
`
`
`
`mi @2506 zmI>>
`
`w\ (_ u
`
`x m 8: 4
`
`U o
`
`m
`
`@E
`
`UP mk <P
`
`1w )1 JT
`
`1 u n " ME;
`
`L k L
`
`“6 to
`
`
`
`ME oziwmo zmI>>
`
`SAMSUNG EXHIBIT 1007
`Page 7 of 80
`
`

`

`U.S. Patent
`
`Sep. 8,1998
`
`Sheet 7 0f 65
`
`5,805,720
`
`FIG. 7
`
`TA
`
`TB
`
`TC
`
`TIME
`
`SAMSUNG EXHIBIT 1007
`Page 8 of 80
`
`

`

`0a|
`;6;
`SOTONSHM
`
`U.S. Patent
`
`OL
`
`8‘Olas
`
`giVi
`
`JAWON!aw
`
`
`
`Sep. 8, 1998
`
`etnt
`
`JAaONINSdONSHM
`
`eed
`
`Sheet 8 of 65
`
`OF!WnswnsWNS
`
`5,805,720
`
`
`
`xXYHO2—-HO2+171XHOc-HO2+17XHOZ—-H0Z+17;.:00
`
`SAMSUNG EXHIBIT 1007
`Page 9 of 80
`
`SAMSUNG EXHIBIT 1007
`Page 9 of 80
`
`
`

`

`U.S. Patent
`
`Sep. 8,1998
`
`Sheet 9 0f 65
`
`5,805,720
`
`FIG. 9
`
`TB
`
`TIME
`
`SAMSUNG EXHIBIT 1007
`Page 10 of 80
`
`

`

`U.S. Patent
`
`OlSls
`
`Sep. 8, 1998
`
`JA3ONISOTO
`
`al
`
`NAHM OL
`
`
`
`3A3ONINSdONSHM
`
`V1
`
`AWIL
`
`Sheet 10 of 65
`
`5,805,720
`
`8
`
`o>
`
`on
`
`dn
`
`XY02-YO¢+7171
`
`XWYO2-Y02+771
`
`SAMSUNG EXHIBIT 1007
`Page 11 of 80
`
`SAMSUNG EXHIBIT 1007
`Page 11 of 80
`
`
`
`
`

`

`U.S. Patent
`
`Sep.8,1998
`
`Sheet 11 0f 65
`
`5,805,720
`
`FIG. 11
`
`K
`
`_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ -
`
`_ _ _ _ _ _ _ _ _ _ _
`
`r _ _
`
`TC
`
`TB
`
`TIME
`
`SAMSUNG EXHIBIT 1007
`Page 12 of 80
`
`

`

`Sep.8,1998
`Sep. 8, 1998
`
`U.S. Patent
`U.S. Patent
`
`NF
`
`cLOld
`
`
`OLglVL
`
`SINIL
`
`FAgONISOTONSHMJAIONINSdONSHM
`
`
`
`
`
`Weeee
`
`ee
`
`5,805,720
`5,805,720
`
`X
`
`xXYYO2—-HOZ+711X¥YH02-HO2+11XYYOc—H0Z+7
`
`Sheet 12 0f 65
`Sheet 12 of 65
`
`|__|=,488
`
`!!!!!»'13!/0!66
`
`SAMSUNG EXHIBIT 1007
`Page 13 of 80
`
`SAMSUNG EXHIBIT 1007
`Page 13 of 80
`
`
`

`

`U.S. Patent
`
`Sep.8,1998
`
`Sheet 13 0f 65
`
`5,805,720
`
`
`
`NOILONFH NOILVH'IVAH :10 N HITIVA K 0:)
`
`FIG. 13
`
`TIME
`
`SAMSUNG EXHIBIT 1007
`Page 14 of 80
`
`

`

`U.S. Patent
`
`Sep.8,1998
`Sheet 14 0f 65
`FIG. 14
`
`5,805,720
`
`LEFT EYE
`
`RIGHT EYE
`
`SUM
`
`T
`
`LEFT EYE
`
`RIGHT EYE
`
`SUM
`
`RR—20 RR X
`
`RIGHT EYE
`
`@8 Q8
`
`9"
`
`a
`
`SAMSUNG EXHIBIT 1007
`Page 15 of 80
`
`

`

`U.S. Patent
`
`Sep.8,1998
`
`Sheet 15 0f 65
`
`5,805,720
`
`
`
`NOLLON?zI NOILVHWVAH :lO >I EIFTIVA
`
`FIG. 15
`
`TIME
`
`SAMSUNG EXHIBIT 1007
`Page 16 of 80
`
`

`

`U.S. Patent
`
`Sep.8,1998
`Sheet 16 0f 65
`FIG. 16
`
`5,805,720
`
`LEFT EYE
`
`RIGHT EYE
`
`UF
`
`LEFT EYE
`
`RIGHT EYE
`
`TB-
`
`UF
`
`RR-‘20 RR x
`
`RIGHT EYE
`
`To- I @8 EM;
`UF
`
`9..
`
`I
`I
`l
`l
`I
`I
`l
`I
`I
`l
`I
`
`X
`
`LL—20 RR X
`
`SAMSUNG EXHIBIT 1007
`Page 17 of 80
`
`

`

`U.S. Patent
`
`Sep.8,1998
`
`Sheet 17 0f 65
`
`5,805,720
`
`FIG. 17
`
`TC
`
`TB
`
`TIME
`
`SAMSUNG EXHIBIT 1007
`Page 18 of 80
`
`

`

`U.S. Patent
`
`Sep.8,1998
`Sheet 18 0f 65
`FIG. 18
`
`5,805,720
`
`LEFT EYE
`
`RIGHT EYE
`
`E
`
`F I'll
`
`R
`
`n T
`
`. . I I . I . 1 I l . . I .R W
`
`- R m
`
`Ln E E - - _
`
`E ........ I
`....m_ Y E m
`9 R H m , m
`m G .9
`
`X 8 X
`
`m ....... I ,UE L
`
`w E w E 2
`
`C X C X
`
`m T
`
`Y 9 w 0
`+ w + n 9 - :H
`E |L._ E L
`
`L L I I . I 1 I I I I :L
`
`u o
`
`r L WL 1 C
`
`F. EquE C HOW...h
`Mm llll
`
`F
`
`SAMSUNG EXHIBIT 1007
`Page 19 of 80
`
`

`

`U.S. Patent
`
`Sep.8,1998
`
`Sheet 19 0f 65
`
`5,805,720
`
`7‘ cu
`
`
`
`NOllON?zl NOILVH'IVAH :10 )I EIFTIVA
`
`FIG. 19
`
`TIME
`
`SAMSUNG EXHIBIT 1007
`Page 20 of 80
`
`

`

`U.S. Patent
`
`Sep. 8, 1998
`
`Sheet 20 of 65
`
`
`WHENOPENINGEYE|WHENCLOSINGEYE
`
`
`00LL+10CRXLL+10CR
`
` TATBTCTIME————_p_Fees
`
`5,805,720
`
`
`SUMSUM
`
`
`SAMSUNG EXHIBIT 1007
`Page 21 of 80
`
`SAMSUNG EXHIBIT 1007
`Page 21 of 80
`
`

`

`U.S. Patent
`
`Sep. 8, 1998
`
`Sheet 21 of 65
`
`5,805,720
`
`FIG. 21
`
`“A oo
`
`
`
`NOILONNSNOILVNIVASJO>ANTIVA
`
`TIME
`
`SAMSUNG EXHIBIT 1007
`Page 22 of 80
`
`SAMSUNG EXHIBIT 1007
`Page 22 of 80
`
`

`

`U.S. Patent
`
`Sep. 8, 1998
`
`Sheet 22 of 65
`
`5,805,720
`
`FIG.22
`
`TC
`
`TA
`
`WHENCLOSINGEYE
`
`
`
`LL+10C
`LL+i10C
`TIME
`WHENOPENINGEYE
`
`
`SAMSUNG EXHIBIT 1007
`Page 23 of 80
`
`SAMSUNG EXHIBIT 1007
`Page 23 of 80
`
`

`

`U.S. Patent
`
`Sep. 8, 1998
`
`Sheet 23 of 65
`
`5,805,720
`
`FIG. 23
`
`A wo
`
`
`
`NOILONN4NOILLWATIVASdO>ANIVA
`
`TIME
`
`SAMSUNG EXHIBIT 1007
`Page 24 of 80
`
`SAMSUNG EXHIBIT 1007
`Page 24 of 80
`
`

`

`U.S. Patent
`
`Sep. 8, 1998
`
`Sheet 24 of 65
`
`5,805,720
`
`
`
`Lu
`
`u ©zw Oo
`>L
`
`O
`
`2L
`
`u x=
`
`OF
`
`l
`
`am
`-
`
`
`
`WHENOPENINGEYE
`
`TA
`
`TIME
`
`FIG.24
`
`
`
`x o
`
`c
`
`LL+10C
`
`SAMSUNG EXHIBIT 1007
`Page 25 of 80
`
`SAMSUNG EXHIBIT 1007
`Page 25 of 80
`
`

`

`U.S. Patent
`
`Sep. 8, 1998
`
`Sheet 25 of 65
`
`5,805,720
`
`FIG. 25
`
`
`
`
`
`NOLLONN4SNOLLVNTWAS
`
`4O»ANTIVA
`
`Aw wo
`
`
`
`
`TA
`
`TB
`
`TC
`
`TIME
`
`SAMSUNG EXHIBIT 1007
`Page 26 of 80
`
`SAMSUNG EXHIBIT 1007
`Page 26 of 80
`
`

`

`U.S. Patent
`
`92Olds
`
`al
`
` OL
`
`VL
`
`AWIL
`
`Sep. 8, 1998
`
`
`
`JAdSNISOTSNAHM
`
`
`
`4A3ONINAdONAHM
`
`Sheet 26 of 65
`
`5,805,720
`
`XYOlL-Y9O01+71
`
`XYHOI-Y901+T1
`
`xXYOl-YO0L+11
`
`WNS
`
`WNS
`
`WNS
`
`SAMSUNG EXHIBIT 1007
`Page 27 of 80
`
`SAMSUNG EXHIBIT 1007
`Page 27 of 80
`
`
`
`
`
`

`

`U.S. Patent
`
`Sep. 8, 1998
`
`Sheet 27 of 65
`
`5,805,720
`
`FIG. 27
`
`A ve)
`
`
`
`NOILONNANOLLWNIWASdO>ANIA
`
`TIME
`
`SAMSUNG EXHIBIT 1007
`Page 28 of 80
`
`SAMSUNG EXHIBIT 1007
`Page 28 of 80
`
`

`

`U.S. Patent
`
`8gOld
`
`NAHM OL
`
`al
`
`VL
`
`IWIL
`
`Sep. 8, 1998
`
`Sheet 28 of 65
`
`5,805,720
`
`3A3ONISOTO
`
`JA3ONINSdONSHM
`
`An
`
`)
`
`KT
`
`|
`apLY
`
`XYOlL-YdDOTT1
`
`X4OL-HDOI+171
`
`SAMSUNG EXHIBIT 1007
`Page 29 of 80
`
`SAMSUNG EXHIBIT 1007
`Page 29 of 80
`
`
`
`
`

`

`U.S. Patent
`
`Sep. 8, 1998
`
`Sheet 29 of 65
`
`5,805,720
`
`FIG. 29
`
`NOILVATWAAJO>ANIWA
`
`NOLLONN4A
`
`TIME
`
`SAMSUNG EXHIBIT 1007
`Page 30 of 80
`
`SAMSUNG EXHIBIT 1007
`Page 30 of 80
`
`

`

`nnreIWIL
`
`’ne
`
`
`
`JAaONISOTONaHM
`
`OL
`
`CE
`
`O€‘Old
`
`al
`
`VL
`
`
`
`JAdONINSdONAHM
`
`U.S. Patent
`
`Sep. 8, 1998
`
`Sheet 30 of 65
`
`5,805,720
`
`'tt‘1i1|1tI11{1'
`
`Torr cre aa
`O
`
`XxHOI-HYOOl+T1
`
`XYoOl-Y9O0lt+11
`
`X¥OL-HOOL+11
`
`SAMSUNG EXHIBIT 1007
`Page 31 of 80
`
`SAMSUNG EXHIBIT 1007
`Page 31 of 80
`
`
`
`
`

`

`U.S. Patent
`
`Sep. 8, 1998
`
`Sheet 31 of 65
`
`5,805,720
`
`FIG. 31
`
`NOILWNIWASJO>ANIWA
`
`NOILONNA
`
`TIME
`
`SAMSUNG EXHIBIT 1007
`Page 32 of 80
`
`SAMSUNG EXHIBIT 1007
`Page 32 of 80
`
`

`

`U.S. Patent
`
`Sep. 8, 1998
`
`Sheet 32 of 65
`
`5,805,720
`
`TIME
`
`TA
`
`TCy
`
`
`
`FIG. 32
`
`WHEN
`OPENING
`EYE
`
`aa:
`
`LEFT EYE
`
`RIGHT EYE
`
`LEFT EYE
`
`RIGHT EYE
`
`WHEN
`CLO
`SING
`EYE
`
`SU
`
`L
`
`SAMSUNG EXHIBIT 1007
`Page 33 of 80
`
`SAMSUNG EXHIBIT 1007
`Page 33 of 80
`
`
`
`
`

`

`U.S. Patent
`
`Sep. 8, 1998
`
`Sheet 33 of 65
`
`5,805,720
`
`FIG. 33
`
`NOILWNIWAAdO>ANIWA
`
`NOILONNS
`
`TIME
`
`SAMSUNG EXHIBIT 1007
`Page 34 of 80
`
`SAMSUNG EXHIBIT 1007
`Page 34 of 80
`
`

`

`U.S. Patent
`
`Sep. 8, 1998
`
`Sheet 34 of 65
`
`5,805,720
`
`FIG. 34
`
`LEFT EYE
`
`RIGHT EYE
`
`TIME
`
`TA+
`
`PrN
`OPE
`EYE
`
`
`
`
`
`
`UF
`
`WHEN
`CLOSING
`EYE
`
`TC+
`
`SAMSUNG EXHIBIT 1007
`Page 35 of 80
`
`SAMSUNG EXHIBIT 1007
`Page 35 of 80
`
`

`

`U.S. Patent
`
`Sep. 8, 1998
`
`Sheet 35 of 65
`
`5,805,720
`
`
`
`
`
`NOILONN4SNOILVANTWASJO>ANTWA
`
`FIG. 35
`
`~wo
`
`TIME
`
`SAMSUNG EXHIBIT 1007
`Page 36 of 80
`
`SAMSUNG EXHIBIT 1007
`Page 36 of 80
`
`

`

`U.S. Patent
`
`Sep. 8, 1998
`
`Sheet 36 of 65
`
`5,805,720
`
`FIG. 36
`
`TIME
`TA}
`
`LEFT EYE
`OPENING ~~:
`
`WHEN
`
`RIGHT EYE
`
`~8
`
`EYE TB
`
`LEFT EYE
`
`RIGHT EYE
`
`
`
`WHEN
`
`8
`
`TC+ cLosiNc}—mge7”|~~8 ~~
`
`EYE
`
`CF
`
`SAMSUNG EXHIBIT 1007
`Page 37 of 80
`
`SAMSUNG EXHIBIT 1007
`Page 37 of 80
`
`

`

`U.S. Patent
`
`Sep. 8, 1998
`
`Sheet 37 of 65
`
`5,805,720
`
`FIG. 37
`
`~wo
`
`NOILVANTVAS
`
`NOILONN4
`
`dO4ANTIVA
`
`TIME
`
`SAMSUNG EXHIBIT 1007
`Page 38 of 80
`
`SAMSUNG EXHIBIT 1007
`Page 38 of 80
`
`

`

`U.S. Patent
`
`Sep. 8, 1998
`
`Sheet 38 of 65
`
`5,805,720
`
`FIG. 38
`
`POS
`
`
`
`SAMSUNG EXHIBIT 1007
`Page 39 of 80
`
`SAMSUNG EXHIBIT 1007
`Page 39 of 80
`
`

`

`U.S. Patent
`
`Sep. 8, 1998
`
`Sheet 39 of 65
`
`5,805,720
`
`
`
`NOILONNANOILVNIVASJO>ANTWA
`
`FIG. 39
`
`A wo
`
`TIME
`
`SAMSUNG EXHIBIT 1007
`Page 40 of 80
`
`SAMSUNG EXHIBIT 1007
`Page 40 of 80
`
`

`

`U.S. Patent
`
`Sep. 8, 1998
`
`Sheet 40 of 65
`
`5,805,720
`
`FIG. 40
`
`TIME
`
`SAMSUNG EXHIBIT 1007
`Page 41 of 80
`
`SAMSUNG EXHIBIT 1007
`Page 41 of 80
`
`

`

`U.S. Patent
`
`Sep. 8, 1998
`
`Sheet 41 of 65
`
`5,805,720
`
`
`
`
`
`NOILONNANOILWATWASJO>ANIWA
`
`FIG. 41
`
`~A wo
`
`TIME
`
`SAMSUNG EXHIBIT 1007
`Page 42 of 80
`
`SAMSUNG EXHIBIT 1007
`Page 42 of 80
`
`

`

`U.S. Patent
`
`Sep. 8, 1998
`
`Sheet 42 of 65
`
`5,805,720
`
`FIG. 42
`
`
`
` POS
`
`aeYeeeeewrea
`
`SAMSUNG EXHIBIT 1007
`Page 43 of 80
`
`SAMSUNG EXHIBIT 1007
`Page 43 of 80
`
`

`

`U.S. Patent
`
`Sep. 8, 1998
`
`Sheet 43 of 65
`
`5,805,720
`
`FIG. 43
`
`TWASAO>ANTVA
`NOILONNANOLLVA
`
`
`TIME
`
`SAMSUNG EXHIBIT 1007
`Page 44 of 80
`
`SAMSUNG EXHIBIT 1007
`Page 44 of 80
`
`

`

`U.S. Patent
`
`Sep. 8, 1998
`
`Sheet 44 of 65
`
`5,805,720
`
`WHENCOLOSING EYE
`
`So
`
`>
`
`>”
`
`?
`
`23
`
`ws uw
`
`266
`"TN
`
`tu
`
`>
`
`Ee
`
`eee x
`5
`oO
`if
`
`;
`
`
`
`oO
`
`5
`
`“”
`
`SAMSUNG EXHIBIT 1007
`Page 45 of 80
`
`©-
`
`<}
`
`z
`
`SAMSUNG EXHIBIT 1007
`Page 45 of 80
`
`

`

`U.S. Patent
`
`Sep. 8, 1998
`
`Sheet 45 of 65
`
`5,805,720
`
`
`
`
`
`NOLLONNANOILVATWASdO>ANIVA
`
`FIG. 45
`
`“~ wo
`
`TIME
`
`SAMSUNG EXHIBIT 1007
`Page 46 of 80
`
`SAMSUNG EXHIBIT 1007
`Page 46 of 80
`
`

`

`U.S. Patent
`
`Sep. 8, 1998
`
`Sheet 46 of 65
`
`5,805,720
`
`ke
`
`9 L
`
`L
`
`WHENCLOSINGEYE O
`
`asw
`cy Sot
`
`O
`
`oO
`
`Ly
`=
`ke
`
`So
`
`5
`
`SAMSUNG EXHIBIT 1007
`Page 47 of 80
`
`SAMSUNG EXHIBIT 1007
`Page 47 of 80
`
`

`

`U.S. Patent
`
`Sep. 8, 1998
`
`Sheet 47 of 65
`
`5,805,720
`
`ke
`
`WHENCLOSING EYE ©
`
`23
`Ww = Ww
`Lan
`ou+o
`
`4+
`-
`
`Lu
`
`2k
`
`e
`
`en
`+
`
`LL
`
`SAMSUNG EXHIBIT 1007
`Page 48 of 80
`
`SAMSUNG EXHIBIT 1007
`Page 48 of 80
`
`

`

`U.S. Patent
`
`Sep. 8, 1998
`
`Sheet 48 of 65
`
`5,805,720
`
`FIG. 48
`
`
`
`TA+WHEN OPENING P
`
`TIME
`
`OS
`
`EYE
`
`TBI-
`
`TCE WHEN CLOSING
`EYE
`
`
`
`
`C+20
`C-20
`
`SAMSUNG EXHIBIT 1007
`Page 49 of 80
`
`SAMSUNG EXHIBIT 1007
`Page 49 of 80
`
`

`

`U.S. Patent
`
`Sep. 8, 1998
`
`Sheet 49 of 65
`
`5,805,720
`
`
`
`NOILONN4NOILVNIVAS4O1ANTWA
`
`FIG. 49
`
`Az wo
`
`TIME
`
`SAMSUNG EXHIBIT 1007
`Page 50 of 80
`
`SAMSUNG EXHIBIT 1007
`Page 50 of 80
`
`

`

`Sep. 8, 1998
`
`Sheet 50 of 65
`
`5,805,720
`
`FIG. 50
`
`WHEN
`TA+OPENING
`EYE
`
`U.S. Patent
`
`TIME
`
`TB
`
`EYE
`
`WHEN
`TC+CLOSING
`
`C-20
`
`C+20
`
`SAMSUNG EXHIBIT 1007
`Page 51 of 80
`
`SAMSUNG EXHIBIT 1007
`Page 51 of 80
`
`

`

`U.S. Patent
`
`Sep. 8, 1998
`
`Sheet 51 of 65
`
`5,805,720
`
`
`
`
`
`NOILONN4ANOILWNTWASdO>ANTIVA
`
`FIG. 51
`
`“~ wo
`
`TIME
`
`SAMSUNG EXHIBIT 1007
`Page 52 of 80
`
`SAMSUNG EXHIBIT 1007
`Page 52 of 80
`
`

`

`U.S. Patent
`
`Sep. 8, 1998
`
`Sheet 52 of 65
`
`5,805,720
`
`OPENINGEYE
`
`SAMSUNG EXHIBIT 1007
`Page 53 of 80
`
`
`
`WHENCLOSINGEYE
`
`WHEN
`
`OE
`
`mo
`E
`
`Lu
`
`=—
`
`kK
`
`FIG.52
`
`SAMSUNG EXHIBIT 1007
`Page 53 of 80
`
`

`

`U.S. Patent
`
`Sep. 8, 1998
`
`Sheet 53 of 65
`
`5,805,720
`
`FIG. 53
`
`TWASJO>ANTWA
`
`NOILONN4NOILVN
`
`TIME
`
`SAMSUNG EXHIBIT 1007
`Page 54 of 80
`
`SAMSUNG EXHIBIT 1007
`Page 54 of 80
`
`

`

`Sep. 8, 1998
`
`Sheet 54 of 65
`
`5,805,720
`
`OPENINGEYE TIME
`
`Liu
`
`>L
`
`u ©2w O—
`
`do
`2p
`Lu
`
`r=
`
`WHEN
`
`U.S. Patent
`
`FIG.54
`
`ol
`rl
`
`ma
`ke
`
`TA
`
`SAMSUNG EXHIBIT 1007
`Page 55 of 80
`
`SAMSUNG EXHIBIT 1007
`Page 55 of 80
`
`

`

`U.S. Patent
`
`Sep. 8, 1998
`
`Sheet 55 of 65
`
`5,805,720
`
`FIG. 55
`
`A oo
`
`
`
`NOILONNANOILVNTIVAS4O>ANIWA
`
`TIME
`
`SAMSUNG EXHIBIT 1007
`Page 56 of 80
`
`SAMSUNG EXHIBIT 1007
`Page 56 of 80
`
`

`

`U.S. Patent
`
`Sep. 8, 1998
`
`Sheet 56 of 65
`
`5,805,720
`
`FIG. 56
`
`TIME
`LEFT EYE
`RIGHT EYE
`WHEN
`OPENING ———_ >
`A EYE
`8
`
`,
`
`<
`
`TCosinc}7|~~8 (ee5
`
`WHEN
`
`EYE
`
`SAMSUNG EXHIBIT 1007
`Page 57 of 80
`
`SAMSUNG EXHIBIT 1007
`Page 57 of 80
`
`

`

`U.S. Patent
`
`Sep. 8, 1998
`
`Sheet 57 of 65
`
`5,805,720
`
`FIG. 57
`
`TWASdO»ANTIVA
`
`
`
`NOILONNANOILVN
`
`TIME
`
`SAMSUNG EXHIBIT 1007
`Page 58 of 80
`
`SAMSUNG EXHIBIT 1007
`Page 58 of 80
`
`

`

`U.S. Patent
`
`Sep. 8, 1998
`
`Sheet 58 of 65
`
`5,805,720
`
`TIME
`
`WHEN
`TA+ OPENING
`EYE
`
`EYE
`
`WHEN
`TC+ CLOSING
`
`SAMSUNG EXHIBIT 1007
`Page 59 of 80
`
`SAMSUNG EXHIBIT 1007
`Page 59 of 80
`
`

`

`U.S. Patent
`
`Sep. 8, 1998
`
`Sheet 59 of 65
`
`5,805,720
`
`
`
`NOILONNANOILVATVAAdO>ANIVA
`
`FIG. 59
`
`“A w
`
`TIME
`
`SAMSUNG EXHIBIT 1007
`Page 60 of 80
`
`SAMSUNG EXHIBIT 1007
`Page 60 of 80
`
`

`

`U.S. Patent
`
`Sep. 8, 1998
`
`Sheet 60 of 65
`
`5,805,720
`
`FIG. 60
`
`CAMERA
`
`IMAGE INPUT
`MEANS
`
`BINARIZATION
`MEANS
`
`EVALUATION FUNTION
`CALCULATION MEANS
`
`
`
` 15A
`
`
`
`EYE EXTRACTION
`MEANS
`
` 14A
`
`SAMSUNG EXHIBIT 1007
`Page 61 of 80
`
`SAMSUNG EXHIBIT 1007
`Page 61 of 80
`
`

`

`U.S. Patent
`
`Sep. 8, 1998
`
`Sheet 61 of 65
`
`5,805,720
`
`FIG. 61
`
`
`
`SAMSUNG EXHIBIT 1007
`Page 62 of 80
`
`SAMSUNG EXHIBIT 1007
`Page 62 of 80
`
`

`

`U.S. Patent
`
`Sep. 8, 1998
`
`Sheet 62 of 65
`
`5,805,720
`
`
`
`IMAGE INPUT
`MEANS
`
`BINARIZATION
`MEANS
`
`
`
`EVALUATION FUNCTION
`CALCULATION MEANS
`
`15A
`
` 14B
`
`
`
`EYE EXTRACTION/OPENED-
`OR-CLOSED-CONDITION
`- JUDGMENT MEANS
`
`SAMSUNG EXHIBIT 1007
`Page 63 of 80
`
`SAMSUNG EXHIBIT 1007
`Page 63 of 80
`
`

`

`U.S. Patent
`
`Sep. 8, 1998
`
`Sheet 63 of 65
`
`5,805,720
`
`FIG. 63
`PRIOR ART
`
`IMAGE INPUT
`MEANS
`
`BINARIZATION
`MEANS
`
`EYEBALL PRESENCE
`AREA SETTING
`MEANS
`
`EYEBALL DETECTING
`MEANS
`
`
`
`
`
`
`
`OPENED-OR-CLOSED EYE
`DETECTING MEANS
`
`CONDITION-OF-DRIVER
`JUDGING MEANS
`
`SAMSUNG EXHIBIT 1007
`Page 64 of 80
`
`SAMSUNG EXHIBIT 1007
`Page 64 of 80
`
`

`

`U.S. Patent
`
`Sep. 8, 1998
`
`Sheet 64 of 65
`
`5,805,720
`
`FIG. 64
`PRIOR ART
`
`SCANNING
`DIRECTION
`
`
`
`
`50 DOTS.
`
`(X1,YB)
`
`(X2,YB)
`
`SAMSUNG EXHIBIT 1007
`Page 65 of 80
`
`SAMSUNG EXHIBIT 1007
`Page 65 of 80
`
`

`

`U.S. Patent
`
`Sep. 8, 1998
`
`Sheet 65 of 65
`
`5,805,720
`
`NAHM LYVYOldd
`
`8o9Sls
`
`
`
`VG9‘Sid
`
`LYVYOldd
`
`
`
`NOILOSYICONINNVOS
`
`
`
`JAdONISOTONSHM
`
`
`
`JAaSNINAdO
`
`SAMSUNG EXHIBIT 1007
`Page 66of 80
`
`SAMSUNG EXHIBIT 1007
`Page 66 of 80
`
`

`

`5,805,720
`
`1
`FACIAL IMAGE PROCESSING SYSTEM
`
`BACKGROUND OF THE INVENTION
`
`1. Field of the Invention
`
`The present invention relates to a facial image processing
`system for detecting various conditions, for example, an
`unclearly awakening condition and a dozing or drowsy
`condition of an automobile driver or the like from the
`
`opened and closed conditions of his eyes.
`2. Description of the Related Art
`Hereinafter, a conventional system will be described by
`referring to FIGS. 63 to 65B. FIG. 63 is a diagram for
`illustrating the conceptional structure of the conventional
`system for detecting the condition of a driver, which is
`disclosed in, for instance, the Japanese Patent Laid-Open
`No. 6-32154.
`
`the conventional system is
`As illustrated in FIG. 63,
`provided with:
`image input means 1 for inputting data
`representing an image of the face of a driver and for
`performing an analog-to-digital (A/D) conversion on the
`input data; binarization means 2 for binarizing the facial
`image inputted from (and converted by) the image input
`means 1; eyeball presence area setting means 3 for deter-
`mining where an eyeball presence area is in the binarized
`image; eyeball detecting means 4 for detecting an eyeball in
`the eyeball presence area on the basis of continuous black
`pixels (or picture elements) placed in the longitudinal direc-
`tion (namely, the Y-direction) thereof; opened/closed eye
`detecting means 5 for detecting the opened or closed eye of
`the driver on the basis of the numberof the continuous black
`pixels of the detected eyeball; and condition-of-driver judg-
`ing means6 for judging the condition of the driver according
`to the pattern of the opened or closed eye detected by the
`opened/closed eye detecting means 5.
`FIG. 64 illustrates a binary image indicating an eyeball
`presence area 7 to be processed by the opened/closed eye
`detecting means 5.
`An operation of the conventional system is performed as
`follows. As illustrated in FIG. 63, an imageof the face of a
`driver, which is taken by using the image input means1, is
`binarized by the binarization means 2 and is thus converted
`into a binary image.
`In the binary image,
`the eyeball
`presencearea 7 is established by the eyeball presence setting
`means 3. The eyeball of the driver is detected in the area 7
`by the eyeball detecting means 4. Further, the opened or
`closed state of the eye is detected by the opened/closed eye
`detecting means 5. The patterns of the opened and closed
`eyes are inputted to the condition-of-driver judging means 6
`whereupon the dozing condition or the like of the driver is
`judged.
`Further, the operation of the opened/closed eye detecting
`means 5 will be described hereinbelow in detail. Namely,
`when detecting an opened or closed eye in the binary image
`in which, for example, a left eye is preset as illustrated in
`FIG. 64, the opened/closed eye detecting means 5 searches
`(namely, scans) an area, in which abscissas (namely, coor-
`dinates in the lateral direction) of pixels range from X1 to
`X2 and ordinates (namely, coordinates in the Y- or longitu-
`dinal direction) thereof range from YB (namely,the ordinate
`of the search starting point) to (YB-50), for black pixels.
`The longitudinal scanning of 1 line or column ofpixels is
`performed on all columnsarrangedin the lateral direction of
`the eyeball presence area 7 repeatedly and sequentially.
`Upon completion of the scanning of the columns over the
`full width of the entire eyeball presence area 7, the maxi-
`
`10
`
`15
`
`20
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`mum number of continuous black pixels is obtained as the
`size in the longitudinal direction (namely, in the Y-direction)
`of the imaged eyeball of the left eye. Thus, a threshold value
`for detecting the opened/closed condition of an eye is
`predetermined onthe basis of data concerning the maximum
`numberof continuousblack pixels in the cases of the opened
`and closed conditions thereof. Thereafter, the detection of
`the opened/closed conditions of an eye can be achieved by
`comparing the maximum number of continuous black
`pixels, which is obtained by the aforementioned scanning
`operation, with the predetermined threshold value.
`As above described, the conventional system employs the
`technique of finding the maximum numberof continuous
`black pixels arranged in the longitudinal direction (namely,
`the Y-direction) of an eye for judging whether the eye of a
`driver is opened or closed. However, sometimes, the binary
`image of the area to be searched becomesasillustrated in
`FIGS. 65A or 65B, owing to the presence of an eyebrow or
`a shade. In such a case, there is no large difference between
`the maximum number L1 of continuous black pixels
`arranged in the Y-direction in the case of the opened
`condition of the eye and that L2 of continuousblack pixels
`arranged in the same direction in the case of the closed
`condition thereof. Thus,
`the conventional system has
`encountered the problem that an error of the judgment on
`whether the eye is opened or closed.
`The present invention is accomplishedto solve the afore-
`mentioned problem of the conventional system.
`SUMMARYOF THE INVENTION
`
`It is, accordingly, an object of the present invention to
`provide a facial image processing system which can accu-
`rately judge whether an eye is openedorclosed, even in the
`cases that a change in the numberof continuousblack pixels
`arranged in the Y-direction, which occurs at the time of
`closing the eye, is small and that a change in the maximum
`number of continuous black pixels arranged in the
`Y-direction results from a change in the upward or down-
`ward direction of a face or from a change in the leftward or
`rightward inclination of the face, by directing attention to a
`change in the shape of the eye, which is caused by closing
`the eye.
`To achieve the foregoing object, in accordance with the
`present invention, there is provided a facial image process-
`ing system that comprises: a camera for taking a facial
`image; image input means for inputting a video signal sent
`from the camera; binarization meansfor binarizing the facial
`image inputted from the image input means; eye extraction
`meansfor extracting an eye presence area, which includes an
`eye, from the binarized facial image; evaluation function
`calculation means for calculating a shape function repre-
`senting a shape feature of the eye from the eye presence area
`and for calculating an evaluation function according to the
`shape function; and opened/closed condition judgment
`means for judging according to the evaluation function
`whether the eye is opened or closed.
`Further, in the case of an embodimentof the facial image
`processing system of the present invention, the evaluation
`function calculation means obtains a first-order shape func-
`tion from the binary image of the eye presence area and
`calculates a second-order shape function according to the
`first-order shape function and further calculates an evalua-
`tion (or performance) function according to the second-order
`shape function.
`Moreover, in the case of another embodimentofthe facial
`image processing system of the present invention, the evalu-
`
`SAMSUNG EXHIBIT 1007
`Page 67 of 80
`
`SAMSUNG EXHIBIT 1007
`Page 67 of 80
`
`

`

`5,805,720
`
`3
`ation function calculation means obtains first and second
`shape functions from the binary image of the eye presence
`area and calculates an evaluation function according to the
`first and second shape functions.
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`Other features, objects and advantages of the present
`invention will become apparent from the following descrip-
`tion of preferred embodiments with reference to the draw-
`ings in which like reference characters designate like or
`corresponding parts throughout several views, and in which:
`FIG. 1 is a diagram for illustrating the configuration of
`Embodiment 1 of the present invention;
`FIG. 2 is a diagram forillustrating a method for detecting
`the opened or closed condition of an eye, which is employed
`by Embodiment 1 of the present invention;
`FIG. 3 is a graph for illustrating how an evaluation
`function employed by Embodiment 1 of the present inven-
`tion varies with time;
`FIG. 4 is a diagram forillustrating a method for detecting
`the opened or closed condition of an eye, which is employed
`by Embodiment 2 of the present invention;
`FIG. 5 is a graph for illustrating how an evaluation
`function employed by Embodiment 2 of the present inven-
`tion varies with time;
`FIG. 6 is a diagram forillustrating a method for detecting
`the opened or closed condition of an eye, which is employed
`by Embodiment 3 of the present invention;
`FIG. 7 is a graph for illustrating how an evaluation
`function employed by Embodiment3 of the present inven-
`tion varies with time;
`FIG. 8 is a diagram forillustrating a method for detecting
`the opened or closed condition of an eye, which is employed
`by Embodiment4 of the present invention;
`FIG. 9 is a graph for illustrating how an evaluation
`function employed by Embodiment4 of the present inven-
`tion varies with time;
`FIG. 10 is a diagram for illustrating a method for detect-
`ing the opened or closed condition of an eye, which is
`employed by Embodiment5 of the present invention;
`FIG. 11 is a graph for illustrating how an evaluation
`function employed by Embodiment 5 of the present inven-
`tion varies with time;
`FIG. 12 is a diagram for illustrating a method for detect-
`ing the opened or closed condition of an eye, which is
`employed by Embodiment6 of the present invention;
`FIG. 13 is a graph for illustrating how an evaluation
`function employed by Embodiment6 of the present inven-
`tion varies with time;
`FIG. 14 is a diagram for illustrating a method for detect-
`ing the opened or closed condition of an eye, which is
`employed by Embodiment7 of the present invention;
`FIG. 15 is a graph for illustrating how an evaluation
`function employed by Embodiment 7 of the present inven-
`tion varies with time;
`FIG. 16 is a diagram for illustrating a method for detect-
`ing the opened or closed condition of an eye, which is
`employed by Embodiment8 of the present invention;
`FIG. 17 is a graph for illustrating how an evaluation
`function employed by Embodiment8 of the present inven-
`tion varies with time;
`FIG. 18 is a diagram for illustrating a method for detect-
`ing the opened or closed condition of an eye, which is
`employed by Embodiment9 of the present invention;
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`4
`FIG. 19 is a graph for illustrating how an evaluation
`function employed by Embodiment 9 of the present inven-
`tion varies with time;
`FIG. 20 is a diagram forillustrating a method for detect-
`ing the opened or closed condition of an eye, which is
`employed by Embodiment 10 of the present invention;
`FIG. 21 is a graph for illustrating how an evaluation
`function employed by Embodiment10 of the present inven-
`tion varies with time;
`FIG. 22 is a diagram forillustrating a method for detect-
`ing the opened or closed condition of an eye, which is
`employed by Embodiment 11 of the present invention;
`FIG. 23 is a graph for illustrating how an evaluation
`function employed by Embodiment 11 of the present inven-
`tion varies with time;
`FIG. 24 is a diagram forillustrating a method for detect-
`ing the opened or closed condition of an eye, which is
`employed by Embodiment 12 of the present invention;
`FIG. 25 is a graph for illustrating how an evaluation
`function employed by Embodiment 12 of the present inven-
`tion varies with time;
`FIG. 26 is a diagram forillustrating a method for detect-
`ing the opened or closed condition of an eye, which is
`employed by Embodiment 13 of the present invention;
`FIG. 27 is a graph for illustrating how an evaluation
`function employed by Embodiment13 of the present inven-
`tion varies with time;
`FIG. 28 is a diagram forillustrating a method for detect-
`ing the opened or closed condition of an eye, which is
`employed by Embodiment 14 of the present invention;
`FIG. 29 is a graph for illustrating how an evaluation
`function employed by Embodiment 14 of the present inven-
`tion varies with time;
`FIG. 30 is a diagram forillustrating a method for detect-
`ing the opened or closed condition of an eye, which is
`employed by Embodiment 15 of the present invention;
`FIG. 31 is a graph for illustrating how an evaluation
`function employed by Embodiment 15 of the present inven-
`tion varies with time;
`FIG. 32 is a diagram forillustrating a method for detect-
`ing the opened or closed condition of an eye, which is
`employed by Embodiment 16 of the present invention;
`FIG. 33 is a graph for illustrating how an evaluation
`function employed by Embodiment 16 of the present inven-
`tion varies with time;
`FIG. 34 is a diagram forillustrating a method for detect-
`ing the opened or closed condition of an eye, which is
`employed by Embodiment17 of the present invention;
`FIG. 35 is a graph for illustrating how an evaluation
`function employed by Embodiment 17 of the present inven-
`tion varies with time;
`FIG. 36 is a diagram forillustrating a method for detect-
`ing the opened or closed condition of an eye, which is
`employed by Embodiment 18 of the present invention;
`FIG. 37 is a graph for illustrating how an evaluation
`function employed by Embodiment18 of the present inven-
`tion varies with time;
`FIG. 38 is a diagram forillustrating a method for detect-
`ing the opened or closed condition of an eye, which is
`employed by Embodiment 19 of the present invention;
`FIG. 39 is a graph for illustrating how an evaluation
`function employed by Embodiment 19 of the present inven-
`tion varies with time;
`
`SAMSUNG EXHIBIT 1007
`Page 68of 80
`
`SAMSUNG EXHIBIT 1007
`Page 68 of 80
`
`

`

`5,805,720
`
`5
`FIG. 40 is a diagram for illustrating a method for detect-
`ing the opened or closed condition of an eye, which is
`employed by Embodiment 20 of the present invention;
`FIG. 41 is a graph for illustrating how an evaluation
`function employed by Embodiment20 of the present inven-
`tion varies with time;
`FIG. 42 is a diagram for illustrating a method for detect-
`ing the opened or closed condition of an eye, which is
`employed by Embodiment 21 of the present invention;
`FIG. 43 is a graph for illustrating how an evaluation
`function employed by Embodiment 21 of the present inven-
`tion varies with time;
`FIG. 44 is a diagram for illustrating a method for detect-
`ing the opened or closed condition of an eye, which is
`employed by Embodiment 22 of the present invention;
`FIG. 45 is a graph for illustrating how an evaluation
`function employed by Embodiment 22 of the present inven-
`tion varies with time;
`FIG. 46 is a diagram for illustrating a method for detect-
`ing the opened or closed condition of an eye, which is
`employed by Embodiment 23 of the present invention;
`FIG. 47 is a diagram for illustrating a method for detect-
`ing the opened or closed condition of an eye, which is
`employed by Embodiment 24 of the present invention;
`FIG. 48 is a diagram for illustrating a method for detect-
`ing the opened or closed condition of an eye, which is
`employed by Embodiment 25 of the present invention;
`FIG. 49 is a graph for illustrating how an evaluation
`function employed by Embodiment25 of the present inven-
`tion varies with time;
`FIG. 50 is a diagram for illustrating a method for detect-
`ing the opened or closed condition of an eye, which is
`employed by Embodiment 26 of the present invention;
`FIG. 51 is a graph for illustrating how an evaluation
`function employed by Embodiment26 of the present inven-
`tion varies with time;
`FIG. 52 is a diagram for illustrating a method for detect-
`ing the opened or closed condition of an eye, which is
`employed by Embodiment 27 of the present invention;
`FIG. 53 is a graph for illustrating how an evaluation
`function employed by Embodiment 27 of the present inven-
`tion varies with time;
`FIG. 54 is a diagram for illustrating a method for detect-
`ing the opened or closed condition of an eye, which is
`employed by Embodiment 28 of the present invention;
`FIG. 55 is a graph for illustrating how an evaluation
`function employed by Embodiment 28 of the present inven-
`tion varies with time;
`FIG. 56 is a diagram for illustrating a method for