United States Patent [191
`Ando
`
`[11] Patent Number:
`[45] Date of Patent:
`
`5,008,946
`Apr. 16, 1991
`
`[54] SYSTEM FOR RECOGNIZING IMAGE
`[75] Inventor: Mitsuhiro Ando, Tokyo, Japan
`[73] Assignee: Aisin Seiki K.K., Tokyo, Japan
`[21] Appl. No.: 242,441
`[22] Filed:
`Sep. 9, 1988
`[30]
`Foreign Application Priority Data
`Sep. 9, 1987 [JP]
`Japan .............................. .. 62-225862
`
`[51] Int. Cl.5 ............................................. .. G06K 9/00
`[52] US. Cl. ........................................ .. 382/2; 382/23;
`.
`382/24; 434/43; 180/271; 180/167
`[58] Field of Search ......................... .. 382/2, 1, 23, 24;
`340/825.03; 358/103; 434/62, 43, 44; 244/222,
`76 R, 194, 195, 3.11, 3.14; 414/901; 180/167,
`272, 271
`
`[56]
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`3,638,188 l/l972 Pincoffs et a1. ..................... .. 382/23
`4,281,734 8/1981 Johnston ........................... .. 180/167
`4,479,784 10/1984 Mallinson et a1. .............. .. 434/43
`4,625,329 11/1986 Ishikawa et a1. .................. .. 180/271
`
`OTHER PUBLICATIONS
`Nakasi Honda et al., Multivariate Data Representation
`
`and Analysis by Face Pattern Using Facial Expression
`Characteristics, 6-1985, 85-93.
`
`.
`Primary Examiner—Michael Razavi
`Attorney, Agent, or Firm-Sughrue, Mion, Zinn,
`Macpeak & Seas
`
`ABSTRACT
`[57]
`There is disclosed a system which permits an automo
`bile driver to control electrical devices installed on an
`automobile by moving his or her pupils and mouth
`intentionally. The system includes a TV camera, a light,
`a ?rst microprocessor which controls the electrical
`devices according to the changes in the shape of the
`driver’s mouth, a second microprocessor for perform
`ing~ various arithmetic operations, and memories. Refer
`ence values have been previously assigned to various
`elements of the driver’sface and stored in one of the
`memories. The second microprocessor normalizes the
`distances between the elements of the face with the
`distance between the pupils of the eyes and compares
`the normalized distances with the reference values to
`calculate the degrees of similarity of the elements.
`
`1 Claim, 22 Drawing Sheets
`
`Page 1 of 44
`
`SAMSUNG EXHIBIT 1009
`Samsung v. Image Processing Techs
`
`

`

`U.S. Patent
`
`Apr. 16, 1991
`
`Sheet 1 of 22
`
`5,008,946
`
`8¢
`
`62
`
`O¢
`
`lN
`
`N
`
`£2
`
`be
`
`Ge
`
`9¢
`
`éN
`
`02
`
`Ie
`
`6
`
`bt
`
`‘NOD43ZZN
`
`‘NOD3SINYD:
`
`‘NODY3dIM
`‘NODO10VY
`435LS04I30
`“1QNOD_UIV
`dWV10V3H
`‘TOULNOD
`TOULNOD
`TOULNOD
`TOULNOD|
`
`‘tLUVOID
`440/NOaeNMOC|eth|440/NO
`340/NO>—-NMOC|440/NO|JOVIUILNIaaNMOT|noi)|
`a-NMOG|bang|440/NO
`340/NO
`440/NO
`340/NO
`NMOT|
`(NOLVOVUS)
`JOVINILNI
`
`JwVu4
`
`ANOWWN
`
`THWad
`
`TaNWavd
`
`JOWIYILNI
`
`q
`
`/V
`
`LH9I1
`
`YSTIOULNOD
`
`Sng
`
`‘TOULNOD
`
`Nd)
`
`WId3as
`
`JOVINILNI
`
`YITIOULNOD
`
`LHOI)
`
`SAMSUNG EXHIBIT 1009
`Page 2 of 44
`
`£
`
`i
`
`IWS
`
`AMOWSW
`
`(AUVNIG)
`
`Ol
`
`DYOld
`
`SAMSUNG EXHIBIT 1009
`Page 2 of 44
`
`
`
`
`
`
`
`
`
`
`
`
`

`

`US. Patent
`
`Apr. 16, 1991
`
`Sheet 2 of 22
`
`5,008,946
`
`FIG. 4
`
`{P
`
`uem ADJUST. ~ILc
`
`MODIFY BRIGHT. ~10
`N0
`9
`HIGHEST DUTY
`CYCLE?
`
`n
`\
`LIGHT 4 OFF
`1
`WRITE DATA
`T0 MEMORY 13 \12
`t
`
`WRITE DATA ~5
`T0 MEMORY 13
`i
`CALCUL. BRIGHT.
`
`~6
`
`SAMSUNG EXHIBIT 1009
`Page 3 of 44
`
`

`

`US. Patent
`
`Apr. 16, 1991
`
`Sheet 3 0f 22
`
`5,008,946
`
`FIG. 2
`
`POWER ON
`i
`INITIAL.
`
`INPUT MODE
`
`LIGHT ADJUST.
`
`ERROR PROCESS. I
`
`DETECT. 0F HEAD /* HDD
`
`ERPI
`
`RED
`
`FOREHEAD DETECT. / BRD
`4--—-I A
`RIGHT EYE DETECT.
`_—-——l *
`SET wmoow we
`f
`LEFT EYE DETECT.
`'
`I
`EYEBROWS DETECT.
`<——-I i
`MOUTH DETECT.
`
`SET wmoow wm
`'
`NOSE DETECTION f NOD
`..____.___.I
`I
`FACE DETECTION
`_________I
`READ IMAGE DATA r40
`DIGITIZE DATA
`
`R
`
`ERROR pnoczssz .
`\
`ERPZ
`
`44
`
`42 .
`SECLOSED? NO
`YES
`SEARCH FOR PUPILS
`‘
`
`'
`
`Y
`SEARCH FOR MOUTH
`I
`DECISIONZ A?
`i
`CLEAR IPDE coum
`
`OUTPUT
`
`54
`
`49”“ DECISION 1
`1
`» 50* CLEAR IPDE coum.
`l__— I
`
`SAMSUNG EXHIBIT 1009
`Page 4 of 44
`
`

`

`US. Patent
`
`Apr. 16, 1991
`
`Sheet 4 0f 22
`
`5,008,946
`
`FIG. 50
`
`HDD~ DETECTION OF HEAD
`3
`13-» DETERM. TH FOR
`HEAD DET CTION
`I
`FLAG<—0
`i
`'5 *1 HEAD DETECTION
`
`16
`
`YES
`
`FIG.- 5b
`
`DETERM. THE FOR
`HEAD DETE TION
`?
`,, 55~ DETERM. MASK H10
`1
`CAL. GRAD.
`HISTOGRAM
`Y
`57~ CLEAR REGISTER 8mm
`1
`CALMEAN
`CONCEN. M1
`7
`CLEAR REGISTER
`
`58”
`
`CAL. PIXEL NUMBER
`m, MEAN CONCEN. M1,
`PIXEL NUMBER '42,
`MEAN CONCEN. M2
`i
`
`+ MHz-M1
`
`NO
`
`YES
`STORE BIN REG. Bm~63
`STOREiIN REG. THh ~64
`——-1
`STORE mm REG.i ~55
`
`SAMSUNG EXHIBIT 1009
`Page 5 of 44
`
`

`

`U.S. Patent
`
`Apr. 16, 1991
`
`Sheet 5 of 22
`
`5,008,946
`
`FIG. 5¢=[LoeecrornewJ}
`
`DETERM. MASK 2Hy,
`
`[~°”
`
`68—STORE Xin IN REG. LN
`=~
`=
`S
`=
`S
`
`COUNT NUMBER
`BNP OF PIXELS
`LN+1 IN REG. LN
`EXCEEDING THh
`
`70
`
`
`
`Fahy Or
`ai
`Xmax INREG. LN
`
`BNP>50 ?
`
`=<
`
`2 S
`
`=S
`
`5
`in&o
`
`
`
`COUNT NUMBER |f-79|&
`
`
`BNP OF PIXELS
`-
`rs
`EXCEEDING TH
`
`76
`
`BNP>50 ?
`
`--/9
`Wane
`LN IN REG. ALX
`
`
`=ALX-
`80
`
`RETIN | AW=ALX-ARX +1 |]
`
`YES
`
`17
`
`YES
`
`3B
`
`s
`
`Ymin IN REG. LN
`
`L—81
`
`
`
`LeenaearlDETECT.OFUPPERENDATY
`
`fos = + = =m> rm=
`
`LNs!
`
`COUNT NUMBER
`
`EXCEEDING TH,
`83
`
`
` BNP OF PIXELS
`
`
`RETURN
`
`FLAG-=—1
`
`[87
`
`LN IN REG. ATY
`
`YES
`
`SAMSUNG EXHIBIT 1009
`Page6 of 44
`
`SAMSUNG EXHIBIT 1009
`Page 6 of 44
`
`

`

`US. Patent
`
`Apr. 16, 1991
`
`Sheet 6 of 22
`
`5,008,946
`
`FIG. 6
`
`FIG. 7b
`
`ERROR PROCESS] ~ERP1
`?
`IPDF ~—IPDF+ 1
`
`'7
`
`20~ DETERM. THf FOR
`FOREHEAD DETECTION
`$
`DETERMTH; FOR N
`RIGHT HALF 5ETERM.
`88
`To TH \ BY POINTS (ARX, ATY),
`h
`(ARY+AW/2, 255)
`
`'
`
`V
`DETERM. THf FOR ~89
`DETERM.
`LEFT HALF DéTERM.
`SIMILARLY
`BY POINTS (ARX+AW/2,
`To T“h \ m), (ALX, 255)
`
`FIG. 70
`
`ABRD
`
`DETECTION
`OF FOREHEAD
`§>
`05mm. THf FOR ~20
`FOREHEAD omcnon
`1
`FLAG -—0 ~21
`i
`DETECTION
`0F FOREHEAD
`
`,__,22
`
`TH gm -—TH fL
`
`THfm<— mfr'
`
`_
`
`V
`ASSIGN THfr, THfm, TH?
`T0 REGIONS 1, 2, 3. REGION
`11s DETERM. BY POINTS
`(ARX, ATY), (ARX+AW/3,
`255). REGION 2 1s DETERM.
`BY POINTS (ARX+AW/3,
`ATY), (ARX+2AW/3, 255).
`REGION 1 1s DETERM. BY
`POINTS (ARX+2AW/3, ATY),
`(ALX, 255)
`
`r
`RETURN
`
`)
`
`(
`
`23
`
`'
`
`£381 "0 @
`YES
`
`SAMSUNG EXHIBIT 1009
`Page 7 of 44
`
`

`

`US. Patent
`
`Apr. 16, 1991
`
`sheet 7 of 22
`
`‘5,008,946
`
`FIG. 7c
`
`~22
`
`DETECTION
`0F FOREHEAD
`9
`REG. LM<-ATY ~94
`
`REG. XP<—ARX
`REG. CK ‘— 0
`REG. RL<— o
`
`96
`
`DETERM. THfr, ~101
`THfm, THrL FOR XP
`y
`READGRFADDAPTA M102
`Dd 0R X
`
`REG. LN<—LN+I ~99
`
`100
`
`REG. HW<—MXL
`REG. HTY<- LN
`FLAG<—1
`7‘
`RETURN )
`
`(
`
`109
`/
`REG. HLX<—XP-1
`
`no
`
`N0
`
`/
`REG. RL<-RL+1
`
`\
`
`REG. XP<-XP+I
`——j '07
`
`111
`YES J
`REG. MXL<—RL
`
`‘
`
`r
`
`REG. CK<—0
`
`V
`
`4
`>1;
`5
`§
`L g
`:E 2
`‘<5
`F5
`s
`
`SAMSUNG EXHIBIT 1009
`Page 8 of 44
`
`

`

`US. Patent
`
`Apr. 16, 1991
`
`Sheet 8 of 22
`
`5,008,946
`
`FIG. 8a
`
`DETECTION OF ,\
`RIGHT EYE
`RED
`?
`24
`
`.05 N
`4
`
`DETERMINE TH
`FOR PUPIL oErEci
`v
`FLAG <— o
`v
`PUPIL DETECT.
`
`25
`
`26
`
`21
`
`\
`
`MODIFY
`
`gm
`
`BUZZER OFF ~31
`~32
`
`°
`
`IPDF '
`'
`CAL. CENTER
`(Ecx,EcY) 0F PUPIL
`v
`CAL‘ FEATURE
`
`~33
`
`FIG. 8b
`
`25~ DETERMOFTHe
`FOR PUPIL DETECT.
`c;
`11 ~ DECIDE REGION sd
`3
`DETERM. BY POINTS
`(HRX,HTY),
`(RRx+Rw/2, HTY+ HW)
`v
`114 ~ CAL. DIFFER. GRAD.
`HISTOG. FOR REGION 5d
`Y
`H5~ REG. i<—i(0FMmux)
`
`116~ coum NUMBER
`us
`5 AND NUMBER
`/
`Sb OFPIXEL_S
`REEF-Rum")
`EXCEEDING I
`1
`'
`I17
`
`REG. THe <—- i
`
`\
`RETURN
`
`>
`
`(
`
`SAMSUNG EXHIBIT 1009
`Page 9 of 44
`
`

`

`US. Patent
`
`Apr. 16, 1991
`
`Sheet 9 of 22
`
`5,008,946
`
`F IG. 8 C
`
`DETECTION
`
`OF PUPIL
`‘3
`REG. CK<—O
`REG. LN‘- HTY+1
`
`~
`
`27
`
`120
`
`FIG. 8d
`
`coum BLACK PIXELS \ Eggugg
`
`N12,
`
`THE
`‘22
`
`3°\ MODIFY THRESHOLD
`1431
`<3
`K1<— 01K]
`REG. K2<— 01x2
`
`REG. MXL<—BPN
`REG. WG-0
`REG. TP<—LN I127
`REG. BP-—LN
`
`ELAe—-—2
`7
`( RETURN )
`
`'4'
`/
`
`I
`
`128
`
`YES LN>HTY+HW?
`N0
`REG. LN<-LN+1 ’‘29
`i
`130
`coum BLACK PIXELS
`
`133
`REG MXL<-BPN
`-
`\
`
`FLAG <—-2
`REG. EBY‘- BP
`REG. ETY<-TP
`REG’ EYEw<-w
`/ I
`138
`
`‘—'—1 L42
`FLAG <— 1
`REG. EBY<- BP
`REG. ETY——TP
`
`SAMSUNG EXHIBIT 1009
`Page 10 of 44
`
`

`

`US. Patent
`
`Apr. 16, 1991
`
`Sheet 10 of 22
`
`5,008,946
`
`FIG. 9a
`
`N
`FAD
`
`CHECK 0N
`FACE 051m.
`?
`CALCUL. P104200 “F 1
`1
`CAL. SPACE EW ~F2
`BETWEEN PUPILS
`T
`NORMAUZE
`Flo-F200 WITH aw
`0
`CAL. F'
`(i=1-é0)
`1
`F28=G21-F21*G22-F22 ~F5
`t
`F29= e23- F23*G24 - F24 ~F6
`
`NF;
`
`N
`F4
`
`@
`
`SAMSUNG EXHIBIT 1009
`Page 11 of 44
`
`

`

`U.S. Patent
`
`Apr. 16, 1991
`
`Sheet 11 of 22
`
`5,008,946
`
`
`
`SAMSUNG EXHIBIT 1009
`Page 12 of 44
`
`SAMSUNG EXHIBIT 1009
`Page 12 of 44
`
`

`

`US. Patent
`
`Apr. 16, 1991
`
`Sheet 12 of 22
`
`5,008,946
`
`F60
`
`_
`EGZ-FZJ- 1?
`
`YES
`
`NO
`
`F64
`
`F65
`
`11o
`
`_
`
`YES
`
`61 F1<o? @ YES
`
`1
`RETURN
`
`>
`
`(
`
`SAMSUNG EXHIBIT 1009
`Page 13 of 44
`
`

`

`US. Patent
`
`Apr. 16, 1991
`
`Sheet 13 of 22 ‘
`
`5,008,946
`
`FIG. 100
`
`FIG. 10b
`
`SEARCH FOR ~44
`PUPILS
`9
`
`oovmwmo SCAN ~144
`ON VERTICAL UNE
`
`@
`
`<9
`
`SCAN FROM UPPER LEFT ~159
`END T0 LOWER mom END
`
`152
`\
`HORIZONTAL
`5W
`
`153
`
`N0
`
`'
`
`'
`
`SCAN FROM LOWER mam ~1s1
`END TO DETECT POINT a
`v
`_ MLX+MRX ~
`“ex-T ‘62
`9
`wcww ~‘63
`2
`|_________
`——'—y
`SCAN FROM UPPER RIGHT ~164
`END T0 LOWER LEFT END
`165
`
`POINTA
`DETECTED?
`I
`SCAN FROM LOWER LEFT vwe
`END T0 DETECT POINT B
`v
`wcxdml? ~16?
`2
`'
`
`WCY : MTYZMBY P168
`
`RESET wmoow we ~15'
`
`UPWARD SCAN
`T0 DETECT MBY
`I
`-MJBLBY ~14?
`MLX
`WCY-
`z
`DETECTED?
`{
`has
`HORIZ. SCAN T0 ~14s
`154~ HORIZ. SCAN TO
`DETECT MLX
`DETECT am
`v
`T
`~ _MLX+MRX HORIZ. SCAN T0 ~
`WC)"
`2
`DETECT MRY
`‘55
`149
`i
`i
`156” DQWNWARD SCAN
`WCX: MLX+ MRX
`TO DETECT m
`2
`v
`Go
`157~ UPWARD 5cm T0
`0mm HEY
`;
`MTY+MBY
`15e~
`“Ch-T"
`L______..
`\
`
`Q5
`
`@5
`
`SAMSUNG EXHIBIT 1009
`Page 14 of 44
`
`

`

`US. Patent
`
`Apr. 16, 1991
`
`Sheet 14 of 22
`
`5,008,946
`
`DECISION 1
`
`CALCUL. EH
`1
`CALCUL. EW
`
`FIG. 110
`
`1
`0011111. 10 REG.
`
`1
`RESET 100111101 ~180
`1
`CLEAR 110911110 ~1a1
`
`173w COUNT. T0 REG.
`1
`174 *4 RESET T COUNTER
`1
`175~ SET CLOS. FLAG
`
`011 9011111
`1
`
`OFF 9011111
`1
`
`194
`\
`01151011111
`
`YES
`
`192
`
`"0
`
`RADIO
`011 1
`193
`1E5
`\
`OFF 51011111
`
`50'
`
`191
`
`1921
`YES
`
`111101
`POSITION 1
`110
`
`AIR 0010.
`POSITION?
`110
`F
`
`SAMSUNG EXHIBIT 1009
`Page 15 of 44
`
`

`

`US. Patent
`
`Apr. 16, 1991
`
`Sheet 15 of 22
`
`5,008,946
`
`FIG. ND
`
`201
`\
`
`AIR CONDI. c
`FOR NAME REG.
`
`RADIO com.
`FOR NAME REG.
`1
`
`,
`
`CLEAR REG. 1-3 “253
`
`wmoow com
`FOR NAME REG
`I
`
`.
`
`6')
`
`7
`CLEAR REG. 1-3
`272 ~
`
`SAMSUNG EXHIBIT 1009
`Page 16 of 44
`
`

`

`US. Patent
`
`Apr. 16, 1991
`
`Sheet 16 of 22
`
`5,008,946
`
`FIG. 12a
`
`DECISION 2
`I
`DETECT RIGHT
`END OF LIPS
`1
`DETECT LEFT
`END OF LIPS
`
`DETECT LATERAL
`WIDTH MW
`1
`SCAN VERTICAL LINE
`CROSSING CENTER
`T
`DETEC T UPPER LOWER
`ENDS 0F LIPS
`if
`DETECT VERTICAL
`WIDTH NH
`
`FOR NAME REG.
`I
`CLEAR REG. I-3
`
`SAMSUNG EXHIBIT 1009
`Page 17 of 44
`
`

`

`US. Patent
`
`Apr. 16, 1991
`
`Sheet 17 of 22
`
`5,008,946
`
`FIG. 12b
`
`CR
`FOR
`
`com.
`EREG. “239
`
`7
`CLEAR REG.1—3 ~240
`
`.
`.
`
`DOOR I.
`FOR NAME
`i
`CLEAR REG. 1-3
`
`251"“ SUNROOF CONT.
`FOR NAME REG.
`T
`252'” CLEAR REG. 1-3
`
`SAMSUNG EXHIBIT 1009
`Page 18 of 44
`
`

`

`US. Patent
`
`Apr. 16, 1991
`
`Sheet 18 of 22
`
`5,008,946
`
`FIG. 130
`
`FIG. 13b
`
`FIG. 13c
`
`FIG. 13d
`
`SAMSUNG EXHIBIT 1009
`Page 19 of 44
`
`

`

`US. Patent
`
`Apr. 16, 1991
`
`Sheet 19 of 22
`
`FIG. I39
`
`EYEN
`
`ETY
`
`-
`
`é; EBY
`
`FIG. 13g
`
`1 Sb
`
`In)
`
`4
`
`HRX
`
`IbI
`
`ETY
`
`EBY
`
`ETY
`
`EBY
`
`FIG. I3i
`BRX
`BLX
`
`§ § xgirm-Hw/g
`
`""* ‘~" "ETY+EYEN
`
`rv
`
`MGY"
`
`HLX
`
`FIG. 13f
`,ZIJSEBY
`
`ETY
`
`Jill/Ill”
`
`FIG.
`
`\ x
`I
`
`.
`
`I
`
`ETY
`
`VIEW
`
`ETY-EMW
`
`'
`
`1,, ‘"1 ,,
`
`/ A
`
`SAMSUNG EXHIBIT 1009
`Page 20 of 44
`
`

`

`U.S. Patent
`
`Apr. 16, 1991
`
`Sheet 20 of 22
`
`5,008,946
`
`029
`
`619819
`
`64
`
`M4~FOL1S101]OL1SI0|}OLUSIGF--F4SJ
`
`350Nfaacana1a|
`19919G1969go}291g
`
`bdCl4ld
`
`Loads||4033930|]¥F8"193dSv|]40sayz||Y28¥g4~|F204
`
`
`
`
`Oliva|[AxaTaWODOliv|[ALIXTaNOD
`FONVISIC)|FONVISIO9uy19el219ust=Old]gy
`
`124—~[NO1ISOa|]avHS|[NOUISOd]3avHS|]NOLLISOd]}3aVHS|_24NOLLISOd]|
`
`
`
`HLAOWOL)JAIL1I91|}3A3LY||3SONO1|]HLAOWOL
`
`JONVISIC]}OL“SIG}}OL“1SIG}]
`igi]HANOW)73SONTTia|]dhotyLees90£96¢9829
`Le)929G2929£c9029Ic)
`gi|Sed|bed|ged|peed
`
`viOld
`
`OldGj
`
`Old
`
`oy[624
`
`
`
`aga|[aa
`
`SAMSUNG EXHIBIT 1009
`Page 21 of 44
`
`SAMSUNG EXHIBIT 1009
`Page 21 of 44
`
`
`
`
`

`

`U.S. Patent
`
`Apr. 16, 1991
`
`Sheet 21 of 22
`
`5,008,946
`
`FIG. 15d
`
`WOX
`
`MTY
`
`WCY
`MBY
`
`--
`
`~~wey
`
`FIG. 154
`
`WINDOW We
`
`|
`
`FIG. 15¢
`
`WCX
`
`MRK
`
`MLX
`
`FIG. 15d
`
`MTY
`WCY
`MBY
`
`
`
`pointe
`
`MAX wy
`
`SAMSUNG EXHIBIT 1009
`Page 22 of 44
`
`__
`
`__
`
`MRX
`
`—MLX
`
`FIG. 15e
`
`POINT
`
`A MRX
`
`WCX
`
`.
`
`MBY
`
`POINT
`
`SAMSUNG EXHIBIT 1009
`Page 22 of 44
`
`

`

`U.S. Patent
`
`Apr. 16, 1991
`
`Sheet 22 of 22
`
`5,008,946
`
`EX
`
`S 0
`
`SAMSUNG EXHIBIT 1009
`Page 23 of 44
`
`FIG. 15g
`
`NWe
`
`EHEW
`
`tf
`
`cH
`
`EW
`
`FIG. 164
`
`WINDOW
`
`Hm /
`
`SAMSUNG EXHIBIT 1009
`Page 23 of 44
`
`

`

`5,008,946
`
`1
`
`SYSTEM FOR RECOGNIZING IMAGE
`
`FIELD OF THE INVENTION
`
`The present invention relates to a system which de-
`tects an image andalso the elements of the image and,
`more particularly,
`to a system which recognizes an
`image, such as an object or person to turn on or off
`desired electrical devices, to increase or decrease the
`output powerofthe devices, or to otherwise control the
`devices by responding to the motion or operation of the
`image in a noncontact manner, it being noted that the
`present invention is not limited to these applications.
`BACKGROUND OF THE INVENTION
`
`10
`
`Theprior art techniquesofthis kind are used in auto-
`matic doors employing photosensors, footboards,etc.,
`warning devices for informing a person of entry or
`intrusion, and metal sensors. Any of these devices
`makes use of a noncontact sensor, such as a photosen-
`sor, microswitch, electrostatic field-type proximity
`switch, or electromagnetic sensor, or a mechanical
`switch, and detects opening or closure of an electrical
`contact, making or breaking of an electromagnetic
`wave path, a changein anelectric field, or a change in
`a magnetic field which is caused when an object or a
`person makes contact with, approaches, or passes.
`through, the device to turn on or off a desired electrical
`device, such as a buzzer, meter, automatic door, relay,
`monitor television , or an electrically controlled ma-
`chine.
`This electrical device cannot be controlled, e.g.,
`turned on andoff, unless an object or personis close to
`the device and makesrelatively large movement. Since
`a changein the state of a minute portion of an object or
`human body cannot be detected by a sensor, an input
`device consisting principally of keyswitches has been
`heretofore most frequently used to energize various
`electrical devices. As an example, various electrical
`devices are installed on an automobile, and various
`keyswitches, volumes, etc. are disposed corresponding
`to those electrical devices. However,
`if the driver
`stretches his or her arm or twists around to operate a
`switch or volume control, then the driving is endan-
`gered. Also, it is easy to meticulously operate a switch
`or volume control, because the driver cannot keep his
`or her eyesoff the front view for a relatively long time
`to watch a device. Accordingly,
`it may be contem-
`plated to install a speech recognition apparatus which
`recognizes the driver’s speech and controls various
`electrical devices. Unfortunately, a large amount of
`noise takesplace inside the automobile and so the recog-
`nition involves noticeable error.
`In order to automatically control or energize various
`electrical devices according to the changein the state of 55
`a small portion within a broad region and to permit the
`driver to control various electrical devices relatively
`precisely in a noncontact manner without requiring
`great care or large motion, the present inventor has
`developed an apparatus that turns on and off devices
`installed on a vehicle in response to the motion of driv-
`er’s eyes and mouth, as disclosed in Japanese Patent
`application No. 272793/1985.
`This apparatus makes use of image pattern recogni-
`tion techniques. In particular, this apparatus uses a cam-
`era means for converting an image, or information in
`the form oflight, into an electrical signal and a position-
`detecting means that detects the position of certain
`
`40
`
`45
`
`50
`
`60
`
`65
`
`2
`portions of the image. In operation, the apparatus takes
`a picture of an object or person, such as an automobile
`driver, and detects the positions of the certain portions
`of the picture, such as the driver’s eyes and mouth.
`Since the brightness inside the automobile varies, an
`illuminating meansforilluminating the driver, a bright-
`ness-setting meansfor setting the brightness of the illu-
`minating means, and a brightness control means are
`provided. The brightness control means detects the
`brightness on the driver’s face and adjusts the setting of
`the brightness-setting means to change the brightness.
`Thus, the brightness on the driver’s face is maintained
`constant to prevent the image processing from produc-
`ing error due to variations in the brightness.
`The position of the driver’s face may be changed by
`vibration of the automobile body or may vary because
`of his or her unintentional minute motion or a changein
`the posture. Also, the eyes and mouth maybeintention-
`ally moved to controlelectrical devices in a noncontact
`manneras described later. To precisely extract informa-
`tion about the eyes and mouth from image information
`in response to the changes in the positions of the face,
`eyes, and mouth, the apparatus further includes a stor-
`age meansfor storing the detected positions, a window
`setting means for setting a region narrower than the
`image produced by the camera means accordingto the
`stored positions, a means for setting the region covered
`by a position-detecting means to the narrower region
`after a certain period of time elapses since the detected
`positions are stored in the storage means, and an updat-
`ing means: for updating the positions of the aforemen-
`tioned certain portions within the narrower region
`which are stored in the storage means. Oncethe posi-
`tions of the certain portions, i.e., the eyes and mouth,
`are detected,
`the scan made to detect
`the eyes and
`mouth is limited to the narrower region and so they can
`be detected quickly. Further, the accuracy with which
`the detection is made is enhanced. Consequently, the
`apparatus follows the eyes and mouth quickly and pre-
`cisely.
`further equipped with a state
`This apparatus is
`change-detecting means for detecting the states of the
`eyes and mouth at successive instants of time to detect
`the changes in thestates. Also, the apparatus includes an
`output-setting means which supplies a control signal or
`electric power to an electrical device according to the
`changesin the states. Specifically, when the states of the
`monitored eyes and mouth are found to change in a
`predetermined manner, i.e, it is ready to activate the
`electrical device, electric power is supplied to the de-
`vice according to the change.
`The apparatus enables the driver to control the elec-
`trical device by moving his or her eyes or mouth while
`assuming a posture adequate to drive the automobile.
`Therefore, the electrical device installed on the automo-
`bile can be quite easily operated. This contributes to a
`comfortable and safe drive. As an example, when the
`driver utters a word to indicate something, the electri-
`cal device is controlled according to the shape of the
`mouth. If the driver utters no word but moves the
`mouth intentionally as if to utter a word, then the elec-
`trical device is controlled according to the shape of the
`mouth. Since the operation of the device is not affected
`by utterance, the detection involvesnoerrorin spite of
`noise produced inside the passenger’s compartment.
`Also,if the radio set is played, or if a passenger is speak-
`
`SAMSUNG EXHIBIT 1009
`Page 24 of 44
`
`SAMSUNG EXHIBIT 1009
`Page 24 of 44
`
`

`

`5,008,946
`
`3
`is unlikely that the electrical device is
`it
`ing loudly,
`caused to malfunction.
`The concept of the aforementioned apparatus can be
`similarly applied to the case. where a person other than
`an automobile driver is monitored. For example, a simi-
`lar apparatus allows a patient with an advanced disease
`to operate, stop, or control the surrounding medical
`instruments or assisting instruments with his or her eyes
`and mouth.
`The apparatus can also monitor a machine to detect
`abnormality and protect the machine. A certain part or
`portion of the machine is checked for trouble. If this
`part or portion operates abnormally, the operation of
`the machine is stopped, or a warning deviceis operated.
`In this way, the above described apparatus can be also
`employed with similar utility to monitor an object other
`than a person.
`Further, the invention can be utilized to monitor a
`broad region such as a natural sight, especially to moni-
`tor animals or vehicles moving in the region. For in-
`stance, a gate in a safari park can be opened and closed
`according to the movement of a vehicle or fierce ani-
`mals. For a manufacturing plant a belt conveyor line
`can be monitored to check the parts or products on the
`conveyor. When they move in a given direction, a
`safety device is operated, or equipment for the next
`manufacturing step is run. In this way, the aforemen-
`tioned apparatus can be used with similar utility in the
`same manneras the foregoing.
`The apparatus described above can detect the driv-
`er’s head, face, and pupils with high accuracy and yield
`the foregoing advantages when the monitored object,
`such as the face of the automobile driver, has a rela-
`tively uniform brightness, typically encountered when
`no car is running in the opposite direction at night and
`substantially only the interior light illuminates the face,
`thus permitting the monitoring. However, when the
`driver’s face or head is illuminated with intense light
`emanating from the headlampseither on a car running
`in the opposite direction or on a succeeding car even at
`night, or when the sunlightis intense in the daytime, the
`external light stronger than the light emitted from the
`interior light is reflected or intercepted by the driver’s
`face or head.Inthis situation the brightness on the face
`frequently becomes nonuniform. Thatis, intenselight is
`reflected from only a portion of the face; the remaining
`portion is in shadow and darker. As an example, when
`the automobile is running in fine weather under the sun
`located to the right of the automobile, the surroundings
`of the right eye are very bright, while the surroundings
`of the left eye are quite dark. In this nonuniform illumi-
`nation, the accuracy with which the driver’s pupils are
`detected deteriorates, because the apparatus uses only
`one threshold value in digitizing the whole obtained
`image. Also, the shape of the driver’s mouth is detected
`with decreased accuracy.
`Accordingly, the present inventor has developed an
`improvement over the aforementioned known appara-
`tus to detect elements, such as the pupils or the mouth
`or both, of a monitored object, such as the driver’s face,
`with increased accuracy,as disclosed in Japanese Patent
`application No. 169325/1987. The improved apparatus
`arithmetically obtains a first gradation histogram for
`each of small neighboring regions, for example the right
`half and the left half, within a desired portion such as a
`humanface included in the monitored image. Then, a
`threshold value for each region is determined, based on
`the histogram. Information about the gradation of the
`
`5
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`4
`image is digitized, and a characteristic index (HTY)
`which indicates the boundary between the hair and the
`forehead, for example,
`is determined. This boundary
`extends through the neighboring regions on the moni-
`tored face. Opposite sides of the boundarydiffer in gray
`level. A second gradation histogram is created from
`information about the gradation of an imageofa set
`region Sg based on the determined characteristic index
`(HTY). The set region Sg contains the eyes. Then, a
`threshold value (TH.) is determined according to this
`histogram to digitize the gradation of the image of the
`region (Sz). Thus, the positions of certain small portion
`or portions, such as pupils, within the region (S,) are
`detected. The certain small portion can be a mouth
`instead of pupils.
`Determination of a threshold value from a gradation
`histogram and digitization an analog signal are known
`in the field of object recognition image processing.
`These techniques are adequate to separate an object
`located in front of the background from the background
`of the image when the concentration of the image var-
`ies. Accordingly, this improved apparatus can precisely
`detect
`the characteristic index which indicates the
`upper end of the forehead. This digitization is adequate
`to detect a characteristic index (HTY) indicating the
`boundary between the background, or hair, and the
`main portion, or forehead, in each divided region even
`if the monitored object is not uniformly illuminated or
`the brightness of the light source itself varies. Hence,
`the index (HTY) can be detected with accuracy. The
`index (HTY) represents a reference position on the
`detected object, or face.
`The region (Sg) surrounding the eyesis set according
`to the characteristic index (HTY). A threshold valueis
`set according to a gradation histogram obtained from
`this region (Sg). Then, an analog signal is transformed
`into binary codes, using the threshold value. These
`techniques are adequate to define the given region (Sq)
`containing the certain small regions, or pupils, of the
`detected object, and to separate the pupils whose gray
`levels suddenly change in the region (Sg), from the
`background, or the surroundings of the pupils if the
`object is illuminated asymmetrically or the brightness of
`the light sourceitself varies. Consequently, the certain
`small portions, or the pupils, can be detected accurately.
`Also, the small portions can be mouthorlips.
`In this manner, the improved apparatus is capable of
`detecting given portions of an object accurately if the
`object is illuminated asymmetrically or the brightness of
`the light sourceitself varies.
`If the driver sitting on the driver’s seat of an automo-
`bile shifts the seat forward or rearward to adjust the
`posture for driving, the distance between the camera
`means and the subject, or face, changes. At this time, an
`automatic focusing device prevents the obtained image
`from getting blurred. However,
`the possibility that
`elements of the imageare incorrectly detected, e.g., the
`nostrils are regarded as the mouth, increases.
`SUMMARY OF THE INVENTION
`
`65
`
`It is an object of the present invention to provide a
`system capable of detecting elements of an image with
`increased accuracy.
`The above object is achieved in accordance with the
`invention by a system comprising: a camera which con-
`verts optical information obtained from the image into
`an electrical signal; a position-detecting circuit for de-
`tecting three or more elements of the image and their
`
`SAMSUNG EXHIBIT 1009
`Page 25 of 44
`
`SAMSUNG EXHIBIT 1009
`Page 25 of 44
`
`

`

`5,008,946
`
`15
`
`20
`
`25
`
`40
`
`45
`
`6
`FIG. 2 is a flowchart schematically illustrating a
`sequence of operations performed by the microproces-
`sor 6 shownin FIG. 1a;
`FIGS. 3, 4, 5a, 58, 5c, 6, 7a, 7b, Te, 8a, 8b, 8c, 8d, 9a,
`9b, 9c, 9d, 10a, 10b, 11a, 11d, 12a, 124, and 12c are flow-
`charts particularly illustrating operations performed by
`the microprocessors 6 and 8 shownin FIG.1a;
`FIGS. 13a, 135, 13c, 13d, 13¢, 13f 13g, and 134A, 133,
`13j are plan viewsofall or some of images taken by the
`camera 3 shownin FIGS.1a and 10;
`FIG. 14 is a diagram showingthe relations. between
`the degrees of similarity F|-F 14 and the degreesofcerti-
`tude F2|-F32 calculated in the routine (FAD)illustrated
`in FIG. 9a for checking detection ofa face;
`FIGS. 15a, 156, 15¢, 15d, 15e, 15f and 15g are plan
`views of window regions W, formed for searching for
`pupils and detected pupils;
`FIG. 16a is a plan view of a window region Wm
`formed for searching for a mouth and a detected mouth;
`and
`FIG. 166 is a plan view of a mouth, for showing
`various shapes of the mouth taken to pronounce vowels.
`DETAILED DESCRIPTION OF THE
`INVENTION
`
`5
`positions according to the electrical signal; a distance
`detecting circuit for detecting the distances between the
`detected elements; a normalizing circuit for normalizing
`data about
`the detected distances with the distance
`between given two of the detected elements; a storage
`circuit which hold reference values previously assigned
`to the elements of the image; a similarity degree calcu-
`lating circuit which compares the normalized data
`about the distances with the reference values and pro-
`duces data about the degrees ofsimilarity to the ele-
`ments of a reference image; and a determining circuit
`which determines whether the image has been success-
`fully detected, from the data about the degrees of simi-
`larity of the detected elements.
`The position detecting circuit detects the positions of
`three or more elements, such as the right pupil, theleft
`pupil, the nostrils, and the mouth, of an image such as a
`human face. The distance-detecting circuit detects the
`distances between the elements. The distances are nor-
`malized with the distance between certain elements.
`Therefore, the normalized data indicating the distances
`between the elements are substantially independent of
`the distance between the camera means and the image.
`The similarity degree-detecting circuit compares the
`normalized data with reference values which are stored
`Referring to FIG. 1a, there is shown a system em-
`in the storage circuit and have been previously assigned
`bodying the concept of the present invention. This sys-
`to the elements of the image to produce data about the
`tem is installed on an automobile and acts to turn on or
`degrees of similarity of the detected elements to the
`off, increase or decrease the power, or otherwise con-
`elements of the reference image. The degreesofsimilar-
`trol electrical devices installed on the automobile, ac-
`ity indicate the degrees to which the positions of the
`cording to intentional movementof the pupils and the
`elements of the optical image formed by the camera
`mouth of the driver’s face.
`bear resemblanceto the positions of the elements of the
`The system includes a TV camera 3 andalight 4 that
`reference image, or the normalized data about the dis-
`illuminates at least the driver’s face. The camera 3 and
`tances between the elements. As the degrees of similar-
`the light 4 are combined into a unit and mounted on the
`ity of the elements increase, the optical image formed
`instrumental panel 2 so as to be movable vertically and
`by the camera means approachesthe reference image.
`horizontally, as shown in FIG. 15. Indicated by numeral
`The determining circuit determines whether the ele-
`1 in FIG. 10is the steering wheel turned by the driver
`ments have been detected successfully, based on the
`to steer the vehicle.
`data about the degrees of similarity. That is, if a high
`Referring again to FIG.1a, the light 4 is turned on
`degree of similarity is obtained, thenit is found that the
`and poweredby a light controller 19, which also con-
`image formed by the camera approximates the reference
`trols the brightness of the light. The light 4 consists of
`image. Conversely, if a low degree of similarity is ob-
`an incandescent lampusinga filamentas a light source.
`tained, then the image formed by the camerais judged
`The controller 19 rapidly switches on and off a direct
`to be different from the reference image.
`voltage, using a thyristor chopper and applies it to the
`Accordingly, where the elements such as the pupils
`light 4. When an OFFsignalis applied to the controller
`and the mouth of an automobile driver’s face, for exam-
`19 to turn off the lighi, the controller turns off the thy-
`ple, are detected to turn on andoff or otherwise control
`ristor chopper. When an ONsignal is applied to the
`electrical devices installed on an automobile according
`controller 19, it switches on and off the chopper at a
`to the shapes ofthe elements and the pattern of change
`normal duty cycle. Whena signal indicating an increase
`in the shapes, the decision to determine whether the
`in the brightness arrives at the controller, it increases
`image formed by the camerais the face or not can be
`the duty cycle by one step. Whena signal indicating a
`madeprecisely. Therefore, the electrical devices can be
`decrease in the brightness is fed to the controller,
`it
`controlled with reduced error. Especially, where the
`distance between the camera and the driver’s face
`reduces the duty cycle by onestep. If the brightnessis
`increased to its maximum level or reduced to its mini-
`change as encountered when the driver movestheseat
`mum level, the brightness is no longer changed. A mi-
`forward or rearward, the pupils and the mouth can be
`croprocessor 6 supplies the aforementioned signals: to
`detected correctly. The electrical devices can be con-
`the light controller 19 via an interface 14 to turn on or
`trolled precisely according to the positions of the ele-
`off the light or to increase or decrease the brightness.
`ments and t