Page 1 of 404
`
`SAMSUNG EXHIBIT 1004
`Samsung v. Image Processing Techs.
`
`

`

`
`V
`.
`«r»
`fAPPLICATIoN
`6‘
`_
`_
`,.
`N
`llllilllllIIINIIIIILIIIJIMQQILILIIININIlilllll
`‘9‘?
`{
`56:33“
`;
`‘
`
`INITIALS_‘___..__H._
`
`g
`
`_ 4,;
`
` {_ \
`
`'
`
`‘.
`
`.-
`
`.
`
`-
`
`.
`
`»
`
`>
`
`'
`
`.
`,
`Date‘ Received
`(Incl. 3}of M.)
`bate Mailed '»
`
`N
`
`
`IN
`“
`
`_
`
`7
`CONTENTS
`Date Received ’
`(IncI. c. 61 M.)
`»
`Datecl’INai'led
`
`'
`
`.
`
`I;
`
`I
`
`-
`
`I
`
`'
`
`>
`=
`
`‘
`
` \
`
`-. a ..- .22-.“
`
`—7-O
`
`.______"____.____._ ,___._.-
`
`ig‘fg’OD—a/g.______._f_________ _______
`\‘\)\fi\‘DM‘W\!’;£:gI
`. MKEIITIMI‘
`Ji- 0‘;
`50._________'___E ______"‘_
`N10.WEN '
`£2" A
`.51.____;__j_;__ _____.__
`Age: Q‘flflllm51/1, fit“ 52.
`“1 II 1 .122’1:
`,1(131%:04M“
`
`
`*
`
`16.
`
`1
`
`.
`
`57.
`
`,
`
`-
`
`..
`
`56.
`
`58.“
`
`59.
`60.
`61.
`62.
`
`63'.
`
`64.
`65.
`66.
`67.
`
`
`
`17.-
`
`__
`
`.
`
`N
`
`N
`
`N 18.‘~_____,_;._._
`19.
`N 20.
`l 21.
`
`I
`
`‘
`
`f
`
`‘
`
`22.____________ _____.
`
`’
`
`' 23.
`24_
`,
`25.
`‘ 26.é__'____
`
`.
`
`‘1
`
`‘
`
`-
`
`.
`
`N
`
`>
`
`'
`N
`
`_
`27. ___________¥
`28.__.+——¢—% __..__._'
`”M _.._
`N3ov_,____.___.'.—.._ _______
`
`'
`58.
`69. _.._____...____._.———
`70.
`‘.
`‘ =
`71.
`
`f
`
`’
`
`.
`
`33-___2__._..'_.
`a 34.W 1
`
`'
`
`,
`
`36. ______________
`N
`N 37-_________
`38.
`‘
`
`.
`
`'
`
`y-
`
`.
`
`f
`
`.
`
`1
`
`'
`
`74.
`75_
`
`77_
`7a.
`79.
`
`.7 '_
`
`'
`
`,N
`
`’,
`
`.
`
`39.
`*— .:80.___________.2 __.__1..__
`? 4o.______________; ._____-_
`31.
`82.
`
`
`
`I
`
`\
`_
`b
`“Egan ;..
`
`2
`
`(LEFT OUTSIDE)
`
`SAMSUNG EXHIBIT 1004‘
`Page 2‘ of 404
`
`"
`
`SAMSUNG EXHIBIT 1004
`Page 2 of 404
`
`

`

`‘. .7 71“,; w‘fiifii,~...tm.
`
`‘
`
`
`
`
`
`
`
`INTERFERENCE SEARC-HED
`
`
`
`SEARCH NOTES
`- (INCLUDING SEARCH STRAIEGYI‘
`
`
`
`(RIGHT OUTSIDE)
`
`’SAMSUNG EXHIBIT 1004 I
`
`Page 3 01‘4""
`
`SAMSUNG EXHIBIT 1004
`Page 3 of 404
`
`

`

`
`
`' ISSUE SLIP. STAPLE AREA (for additional cross references)
`
`INDEX OF CLAIMS
`
`v
`
`_.
`
`..............
`.........................
`
`........ Rejected
`........ Allowed
`(ThrOugh numeral.)
`Canceled
`........ Restricted
`
`'
`
`N .........................
`........................
`|
`A ........................
`O ......................
`
`......... Non-elected
`......... Interference
`
`......... Appeal .
`......... Obj‘ected
`
`Claim
`
`Date
`
`
`
`g E
`
`
`
`"J
`
`‘54
`
`era
`. @197;
`[OD
`
`
`
`[J
`
`
`
`
`NI
`
`_ WIIIIIIIIIIII
`4IiIIIIIIIII
`
`EIIIIIIIIIIIII
`“IIIIIIII
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`EIEIIIIIIIII
`W@MEIIIIIIII
`
`III-IIIIIIIII
`
`IIIIIIIIIIII
`IIIIIIIIIIII
`IIIIIIIIIIII
`IIIIIIIIIIII
`III-IIIIIIII
`IIIIIIIIIIII
`
`47IIIIIIIIII
`
`48IIIIIIIIII
`
`IIIIIIIIIIII
`
`IIIIIIIIIII
`
`C_aim“ !
`i
`
`
`
`
`
`Final§OriginalH
`
`
`
`
`
`
`
`
`I
`
`
`
`:aA—r““7“”:a"v”—‘_‘2:7;-
`
`IWIIIII II
`III-III II
`IIIIIIIIII
`'IMIIIIIIII
`Iflllllllll
`‘
`lull-ll...
`IMIIIIIIIIIII
`
`
`
`
`IMIIIIIIIIII;
`.lfllllllllllly
`
`
`<IMIIIIIIIIIII
`¥IWIIIIIIIIIII
`
`
`IIIIIIIIIIIII
`
`IIIIIIIIIIII
`
`
`IIIIIIIIIIII
`
`
`IEIIIIIIIIII\
`
`
`IIIIIIIIIIIII
`
`IIIIIIIIIIII‘
`
`IIIIIIIIIIIII
`
`
`
`
`IEIIIIIIIIIIy.
`
`
`IfiIIIIIIIIII_
`
`IIIIIIIIIIIII
`
`III-IIIIIIII‘:
`IMIIIIIIIIIII
`
`_; (a
`IIIIIIIIIIII
`IIIIIIIIIII
`
`_x
`IIIIIIIIIII
`
`EIIIIIIIIII
`
`IIIIIIIIIIII
`
`IEIIIIIIIIII
`
`IIIIIIIIIIII,
`EIIIIIIIIII‘
`
`IEIIIIIIIIII‘
`
`
`
`
`
`
`
`
`
`
`
`xii—I!-00N
`
`
`
`
`
`
`
`ClaimD—te
`
`
`
`inalriginali
`
`
`
`IIIIIIIIIIII
`
`.IEIIIIIIIIII
`
`
`IIIIIIIIIIII
`
`
`
`IIIIIIIIIIII
`IIIIIIIIIIII
`
`
`III-IIIIIIII
`IEIIIIIIIIII
`
`III-IIIIIIII,
`
`’IIIIIIIIIIII
`IEIIIIIIIIII
`
`IIIIIIIIIIII?
`IEIIIIIIIIII
`
`IIIIIIIIIIII
`
`IEIIIIIIIIII
`
`'IEIIIIIIIIII
`
`
`
`
`
`
`
`
`
`
`
`
`IEIIIIIIIIII
`IEIIIIIIIIII
`
`
`IIIIIIIIIIII‘
`
`IflIIIIIIIIII
`IIIIIIIIIIII
`IIIIIIIIIIII
`IIIIIIIIIIII
`
`IIIIIIII
`
`
`
`
`
`
`Io
`
`n
`
`03OD
`
`I I
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`MIII
`IIIIIIIIIIII
`IEIIIIIIIIII
`IEIIIIIIIIII
`IIIIIIIIIIII
`IEIIIIIIIIII
`IIIIIIIIIIII
`IEIIIIIIIIII
`IIIIIIIIIIII
`III-IIIIIIII
`
`
`IIIIIIIIIIII
`III-IIIIIIII
`
`
`III-IIIIIIII
`IEIIIIIIIIII
`
`
`III-IIIIIIII
`IEIIIIIIIIII
`
`
`IMIIIIIIIIII
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`If more than 150 claims or 10 actions
`
`staple additional sheet here
`
`(LEFT INSIDE)
`
`SAMSUNG EXHIBIT 1004
`
`Page 4 of 404
`
`SAMSUNG EXHIBIT 1004
`Page 4 of 404
`
`

`

`r
`
`I
`
`
`
`
`UNITED STATES PATENT AND TRADEMARK OFFICE
`COMMISSIONER FOR PATENTS
`UNITED STATES PATENT AND TRADEMARK OFFICE
`WASHINGTON. DC. 20231
`www.uspto.gov
`
`1’
`
`IIIIIIIIIIIIIIIIIIIIIII|IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII
`Bib Data Sheet
`
`
`20046H-00060
`
`CLASS
`340
`
`.
`
`GROUP ART UNIT
`_2632
`
`ATTORNEY
`DOCKET No_
`
`SERIAL NUMBER
`09/600,390
`
`.
`
`‘ PPLICANTS
`
`FILING DATE
`02/09/2001
`RULE
`
`Patrick Pirim, Paris, FRANCE;
`Thomas Binford,’ Cupertino, CA ;
`
`.
`
`
`* CONTINUING DATA ******************** > *** / M
`
`THIS APPLICATION IS A 371
`PCT/EP99/00300 01/15/1999
`
`* FOREIGN APPLICATIONS ** **************~/[\M
`1998
`FRANCE 98/00378 01/1
`PCT/EP98/05383 08/25/1998
`IF REQUIRED, FOREIGN FILING LICENSE
`GRANTED '* 02/23/2001
`oreign Priorityclaimed Z yesD
`35usc119_d
`dt
`D"0E!
`meI
`‘3 m" ”°”s Eves
`no
`erified and
`19%;
`,fdl wane
`Examiner'sSi-nature
`‘
`DDRESS
`1
`
`..
`
`Metafier
`-~
`L
`Initials
`
`- F
`
`SHEETS
`STATE OR
`COUNTRY DRAWING
`
`"
`
`'
`
`-_
`
`.,
`
`TOTAL
`CLAIMS
`39
`
`INDEPENDENT
`CLAIMS
`1,6
`'
`
`Babak s Sani
`Townsend and Townsend and Crew
`
`Two Embarcadero Center 8th Floor
`San Francisco ,CA 94111
`
`TITLE
`
`_
`
`Method and apparatus for detection of drowsiness
`
`FILING FEE FEES:Authorityhasbeengivenin Paper
`
`AJECEIVED No.
`
`for following:
`
`to charge/credit DEPOSIT ACCOUNT time)
`Cl 1.18 Fees ( Issue)
`|D Other
`ID Credit
`
`Cl1.18 Fees____(________Filing)
`
`[C1 All Fees
`1
`
`
`‘9 117 Fees(PFOCGSSINQ EXI- 0f:|
`
`I
`I
`
`SAMSUNG EXHIBIT 1004
`
`Page 5 of 404
`
`SAMSUNG EXHIBIT 1004
`Page 5 of 404
`
`

`

`Q-
`
`,
`
`V
`
`. m
`
`,/
`
`/
`
`i
`
`{1,
`\
`‘
`,.\ f
`
`t
`
`"
`
`,
`
`...l
`
`/
`
`/
`
`,,
`
`‘
`
`09/600390
`
`patent application serial no.
`
`Department of Commerce
`Patent and Trademark om“
`fee record
`
`
`
`
`SAMSUNG EXHIBIT 1004
`
`Page 6 of 404
`
`SAMSUNG EXHIBIT 1004
`Page 6 of 404
`
`

`

`~"15-oi-eooo
`
`'
`
`’
`
`:
`
`:1
`
`EP o%%o
`.uu j. 09/6.opa90
`IE. 00..
`.0 I. PCTEP'9;IQQ3:UGOO.
`
`048 I PCT 361
`2000
`
`534Rec’dPCT/PltWIZJULZUOU
`WW
`
`Inventors:
`
`Dr. Patrick Piri‘m
`Dr. Thomas Binford
`
`5
`
`BACKGROUND OF THE INVENTION
`
`
`1.
`
`Field of the Invention.
`The present invention relates generally 'to an image processing system, and
`more particularly to. the use of a generic image processing system to detect
`
`drowsiness.
`
`10
`
`15
`
`A
`
`1.Description ofthe Related Art.
`Itis well known that a significant number ofhighway accidents result from
`drivers becoming drowsy or falling asleep, which results in many deaths and
`injuries. Drowsiness is also a problemin other fields, such as for airline pilots and
`power plant operators, in which great damage may result from failure to stay alert.
`A number of difi‘erent physical criteria may be used to establish when a
`person is drowsy, including a change in the duration'and interval of eye blinking.
`Normally, the duration of blinking is about 100m 200 ms when awake'and about
`
`
`
`
`
`
`
`25
`
`three categories: '1) devices that dete movement of the head of the driver e.g.,
`tilting; ii) devices that detect a phys'
`logical change in the driver, e.g., altered
`heartbeat or breathing, and iii) devi
`s thatdetect a physical result of the driver
`falling asleep, e.g., a reduced grip o the steering wheel. None of these devices is
`
`believed to have met with cornrnerc'
`
`success.
`
`AMENDED SHEET
`2
`
`. SAMSU.N.,,G..EXHIBIT 1004
`Page 7 of 404
`
`SAMSUNG EXHIBIT 1004
`Page 7 of 404
`
`

`

`‘ is-oi-‘aooo
`
`9
`
`9
`
`'
`
`EP 009900300
`
`:
`
`Ia
`
`.:.
`
`'..' PCTAEP-99'IQQ300'..'
`048 JPCT 361
`
`January 11, 2000
`
`
`
`an area comprising the driver’s ey s,
`
`
`
`o the detection ofthe tim gap between the closing of the driver eyelids and
`
`10
`
`their full opening and
`
`0 a treatment in a mem ry 11 and a processor 22 in combination with unit
`
`10? to calculate a ratio of slow link apparition.
`
`15
`
`The object of the international patent application published WO-97/01246
`is a security system comprising a video camera placed within the rear-view mirror
`of a car and a video screen remotely disposed for the analysis ofwhat is happening
`in the car and around it, as well as of what happened due to the recording of the
`output video signal of the camera This is in ‘fact a concealed camera (within the
`
`rear-view mirror), so that it is imperceptible to vandals and thieves and which
`
`observes a large scope including the inside of the car and its surroundings, the
`
`20
`
`record allowing one to know later. what has happened in this scope (page 6, lines
`
`13 to 19), this is not a detector whose efi‘ective angle is strictly limited to the car
`driver face in order to detect its eventual drowsiness and to make him awake.
`
`p ommonly—owned PCT
`98/05383 disclose a 'g
`
`
`
`pplication Serial Nos. PCT/FR97/013 54 and
`eric image processing system that operates to
`
`.AMENDE/DD SHEET »
`91>
`
`'
`‘
`_ SAMSUNGJEJHIBIT1004
`Page 8 of 404
`
`__._._
`
`SAMSUNG EXHIBIT 1004
`Page 8 of 404
`
`

`

`“Rte/PCTRecfdté«101.2006.
`
`35‘“.r
`
`99, 6 0,0 3 9 0 '
`
`wo 99/36893
`
`'
`
`,
`
`‘
`
`PCT/EP99/00300
`
`'METHOD AND APPARATUS FOR DETECTION OF DROWSINESS
`
`BACKGROUND OF THE INVENTION
`
`1.
`
`Field of the Invention.
`
`The present invention relates generally to an image processing system,
`and more particularly to the use of a generic image processing system to detect
`drowsiness.
`
`1.Description of the Related Art.
`
`It is well known that a significant number of highway accidents result from
`
`drivers becoming drowsy or falling asleep, which results in many deaths and injuries.
`Drowsiness is also a problem'in other fields, such as for airline pilots and power plant
`
`operators, in which great damage may result from failure to stay alert.
`
`A number of different physical criteria may be used to establish when a person is
`
`drowsy, including a change in the duration and interval of eye blinking. Normally, the
`
`duration of blinking is about 100 to 200 ms When awake and about 500 to 800 ms when
`
`drowsy. The time interval between successiveblinks is generally constant while awake,
`
`but varies within a relatively broad range when drowsy.
`
`Numerous devices have been proposed to detect drowsiness of drivers. Such
`devices are
`shown,
`for
`example,
`in US Patent Nos.
`5,841,354; 5,873,99;
`
`5,469,143;
`5,682,144;
`5,689,241;5,684,461;
`4,928,090; 4,555,697; 4,485,375; and 4,259,665.
`
`5,195,606;
`5,353,013;
`5,402,109;
`In general, these devices fall into
`
`three categories: i) devices that detect movement of the head of the driver, e.g., tiltinggii)
`devices that detect a physiolOgical change in the driver, e.g., altered heartbeat or
`breathing, and iii)ldevices that detect {a physical result of the driver falling asleep, e.g., a
`
`reduced grip on the steering wheel. None of these devices is believed to have met With
`
`commercial success.
`
`I
`
`Commonly-owned PCT Application
`
`Serial Nos. PCT/FR97/Ol354
`
`and
`
`PCT/EP98/05383 disclosea generic image processing system that operates to localize
`
`(mayv9745%sz 10
`
`15
`
`20
`
`25
`
`30
`
`SUBSTITUTE SHEET
`
`E 2
`
`a -_. (,RUL, QAMSUNQaEXHIBITWM.
`Page 9 of 404
`
`'
`
`'
`
`SAMSUNG EXHIBIT 1004
`Page 9 of 404
`
`

`

`WO 99136893
`
`'
`
`-
`
`‘
`
`PCTlEP99/00300
`
`
`
`objects in relative movement in an ima e and to detennine the speed and direction of the
`objects in real-time. Each pixel of an mage is smoothed using its own time constant. A
`binary value corresponding to the e stence of a significant variation in the amplitude of I
`
`the sm00thed pixel from the pri r frame, and the amplitude of the variation, are
`determined, and the time constant or the pixel is updated. For each particular pixel, two
`
`matrices are formed that include
`
`
`
`subset of the pixels spatially related to the particular
`
`pixel. The first matrix contain the binary values of the subset of pixels. The second
`
`In the first matrix,
`the variatidn of the subset of pixels.
`matrix contains the amplitude
`xels- along an oriented direction relative to the particular
`it is determined whether the
`pixel have binary values repr sentative of significant variation, and, for'such pixels, it is
`
`
`
`10
`
`determined in the second m trix whether the amplitude of these pixels varies in a known
`manner indicating move ent
`in the oriented direction;
`In domains that
`include
`
`luminance, hue, saturatio speed, oriented direction, time constant, and x and y position,
`
`a histogram is, formed
`
`
`
`the values in the first and second matrices falling in user
`
`there is an area having t
`
`selected combinations 0 such domains. Using the histograms, it is determined whether
`1 characteristics of the selected combinations of domains.
`
`It would be desirable to apply such a generic image processing system to detect
`
`the drowsiness of a person.
`SUMMARY or THE INVENTION
`
`\
`
`20
`
`The present invention is a process of detecting a driver falling asleep in which an
`image-of the face of the driver is acquired. Pixels of the image having characteristics
`
`corresponding to characteristics of at least one eye of the driver are selected and a
`histogram is formed of the selected pixels. The histogram is analyzed overtime to
`
`identify each opening and closing of the eye, and from the eye opening and closing
`
`25
`
`information, characteristics indicative of a driver falling asleep are determined.
`
`In one embodiment, a sub-area of the image comprising the eye is detennined
`
`prior to the step of selecting pixels of the, image. having characteristics corresponding to
`
`characteristics of an eye.
`
`In this embodiment, the step of selecting pixels of the image
`
`having characteristics of an eye involves selecting pixels within the suboarea of the image.
`
`30
`
`The step of identifying a sub-area of the image preferably involves identifying the head of
`
`SUBSTITUTE SWEET (RVFEngmsuNG EXHIBIT 1004,
`Page‘WiO 6f‘4m04
`
`SAMSUNG EXHIBIT 1004
`Page 10 of 404
`
`

`

`WO 99/36893
`
`-
`
`PCT/EP99/00300
`
`the driver, or a facial characteristic of the driver, such as the driver's nostrils, and then
`
`identifying the sub-area of the image usingan anthropomorphic model. The head of the
`
`driver may be identified by} selecting pixels of the image having characteristics ‘
`
`5
`
`corresponding to edges of the head of the driver. Histograms of the selected pixels of
`the edges of the driver's head are projected onto orthogonal axes. These histograms are
`then analyzed to identify the edges of the driver's head.
`
`
` selected pixels of the facial characteristic
`
`
`
`
`.
`If desired, the step of
`histograms are then analyzed to identify t e facial characteristic.
`identifying the facial characteristic in th image involves searching sub-images of the
`image until the facial characteristic is to nd.
`In the case in which the facial characteristic
`is the nostrils of the driver, a histogra
`is formed of pixels having low luminance levels
`
` to detect the nostrils. To confirm det ction of the nostrils, the histograms of the nostril
`
`hether the spacing between the nostrils is within a
`pixels may be analyzed to determine
`desired range and Whether the dim sions of the nostrils fall within a desired range.
`In
`order to confirm the identification f the facial characteristic, an anthropomorphic model
`and the location of the facial ch racteristic are used to select a sub-area of the image
`containing a second facial ch acteristic.
`Pixels of the image having characteristics
`corresponding to the second acial characteristic are selected and a histograms of the
`selected pixels of the sec nd facial characteristic are analyzed to confirm the
`
`1
`
`15
`
`20
`
` —
`
`identification of the first faci lcharacteristic.
`_
`closings of the eyes of the driver, the step of
`
` n order to determine openings at
`
`
`In this embodi ent, the step analyzing the histogram over time to
`shadowing of the eye.
`identify each opening and closing 0 the eye involves analyzing the shape of the eye
`shadowing to determine openings
`closings of the eye. The histograms of shadowed
`pixels are preferably projected onto orthogonal axes, and the step of analyzing the shape
`of the eye shadowing involves» anal
`’ng the width and height ofthe shadowing.
`
`
`
`30
`
`SUBSTITUTEéE
`ET (RULE 262‘
`,
`..
`.. S ,MSLl,l}l.§wE.Xl-.I.J.BJLJ.QQA
`Page 11 of 404
`
`SAMSUNG EXHIBIT 1004
`Page 11 of 404
`
`

`

`WO 99/36893
`
`'
`
`4
`
`. PCTlEP99/00300
`
`
`alternative method of de rmining openings and closings of the eyes of the
`involves selecting pixels 0
`the image having characteristics of movement
`
`In thi embodiment, the step analyzing the histogram over
`corresponding to blinking.
`
`time to identify each opening an clOsing of the eye involves analyzing the number of '
`pixels in movement Correspon ng to blinking over time. The characteristics of a
`blinking eye are preferably se cted from the group consisting of i) DP—fl,
`ii) CO
`
`5
`
`indicative of a blinking eyelid,
`ii) velocity indicative of a blinking eyelid, and iv) up and
`down movement indicative o a blinking eyelid.
`
`lling asleep includes a sensor for acquiring
`
`An apparatus for detecting a driver
`
`of the face of the driver, a co troller, and a histogram formation unit for
`
`forming a histogram on pixels having sele ted characteristics. The controller controls the
`
`
`$55.10
`ani
`
`
`e
`‘3’
`:
`Q
`
`histogram formation unit
`to select pixels of the image having characteristics
`corresponding to characteristics of at 1 ast one eye of the driver and to form a histogram
`of the selected pixels. The centroller analyzes the histogram over time to identify each
`opening and closing of the eye, and
`terrnines from the opening and closing information
`on the eye, characteristics indicativ of the driver falling asleep.
`
`
`
`15
`
`5w
`
`,.-;,;
`it;
`
`20
`
`25
`
`In one embodiment, the controller interacts with the histogram formation unit to
`
`identify a sub-area of the image comprising the eye, and the controller controls the
`histogram formation unit
`to select pixels of the image having characteristics
`corresponding to characteristics of the eye only within the sub-area of the image.
`In
`
`order to select the suboarea of the image, the controller interacts withthe histogram
`
`formation unit to identify the head of the driver in the image, or a facial characteristic of
`
`the driver, such as the driver's nostrils. The controller then identifies the sub-area of the '
`
`image using an anthropomorphic model.
`To identify the head of the driver,
`the
`histogram formation unit selects pixels of the image having characteristics corresponding
`to edges of the head of the driver and forms histograins of the selected pixels projected
`
`onto orthogonal axes. To identify a facial characteristic of the driver, the histogram
`
`formation unit selects pixels of the image having characteristics corresponding to‘the
`
`facial characteristic and forms histograms of the selected pixels projected onto
`orthogonal axes. The controller then analyzes the histograms of the selected pixels to
`
`30
`
`SUBSTITUTESigET(RULE26)
`
`,
`
`,
`
`.SAMSUNQEXHJBIT.,1,004
`
`Page 12 of 404
`
`SAMSUNG EXHIBIT 1004
`Page 12 of 404
`
`

`

`WO 99/36893
`
`.'
`
`‘
`
`.
`
`PCT/EP99/00300
`
`identify the edges ofthe head ofthe driver or the facial characteristic, as the case may be.
`If the facial characteristic is the nostrils of the driver, the histogram formation unit selects
`
`pixels of the image having low luminance levels corresponding to the luminance level of
`the nostrils. The controller may also analyze the histograms of the nostril pixels to
`determine whether the'spacing between the nostrils-is within a desired range and whether
`dimensions of the nostrils fall within a'desired range.
`If desired, the controller may
`interact with the histogram formation unit to search sub-images of the image to identify
`
`the facial characteristic.
`
`10
`
`15
`
`In order to verify identification of the facial characteristic, the controller uses an
`anthropomorphic model and the location of the facial characteristic to cause the
`
`histogram formation unit to select a sub-area of the image containing a“ second facial
`
`characteristic. The histogram formation unit selects pixels of the image in the sub-area
`
`having characteristics corresponding to the second facial characteristic and forms a
`
`histogram of such pixels. The controller then analyzes the histogram of the selected
`
`pixels corresponding to the secnnd facial characteristic to identify the second facial
`
`characteristic and to thereby confirm the identification ofthe first facial characteristic.
`
`In one embodiment, the histogram formation unit selects pixels of the image
`
`having low luminance levels corresponding to shadowing of the eyes, and the controller
`
`then analyzes the. shape of the eye shadowing to identify shapes corresponding to
`openings and closings of the eye. The histogram formation unit preferably forms
`
`20
`
`25
`
`histograms of the shadowed pixels of the eye projected onto orthogonal axes, and the
`
`controller analyzes the width and height of the shadowing to determine openings and
`closings of the 'eye.
`‘
`In an alternative embodiment, the histogram fonnation unit selects pixels of the
`image in movement corresponding to blinking and the controller analyzes the number of
`pixels in movement over time to determine openings and closings of the eye. The
`characteristics of movement corresponding to blinking are preferably selected from the
`group consisting of i) DP=l, ii) C0 indicative of a blinking eyelid, iii) velocity indicative
`
`of a blinking eyelid, and iv) up and down movement indicative of a blinking eyelid.
`
`“g.
`SUBSTITUTE SHEET {RULE 2m
`
`‘
`
`‘
`
`SAMSUNG‘EXHIBIT 1004
`
`Page 13 of 404
`
`SAMSUNG EXHIBIT 1004
`Page 13 of 404
`
`

`

`m“xfiv1]
`
`1"ItIII
`
`
`liué'a@333All.a...a.
`
`
`3:161,
`
`WO 99/36893
`I
`
`—
`
`PCTIEP99/00300
`
`6
`
`If desired, the sensor may be integrally constructed with the controller and the
`
`histogram formation unit. The apparatus may comprise an alarm, which the controller
`operates upon detection of the driver falling asleep, and may comprise an illumination
`
`source, such as a source of IR radiation, with the sensor being adapted to view the driver
`when illuminated by the illumination source.
`
`A rear-view mirror assembly comprises a rear-view mirror and the described
`
`apparatus for detecting driver drowsiness mounted to the rear-view mirror.
`
`In one
`
`embodiment, a bracket attaches the apparatus to the rear-view mirror.
`
`In an alternative
`
`embodiment,
`
`the rear-view mirror comprises a housing having an open side and an
`
`interior. The rear-view mirror is mounted to the open side of the housing, and is see-
`through from the interior of the housing to the exterior of the housing. The drowsiness
`
`detection apparatus is mounted interior to the housing with the sensor directed toward
`the rear-view mirror.
`If desired, a joint attaches the apparatus to the rear-view mirror
`
`assembly, with the joint being adapted to maintain the apparatus in a position facing the
`driver during adjustment of the mirror assembly by the driver. The rear-view mirror
`assembly may include a source of illumination directed toward the driver, with the sensor
`
`adapted to view the driver when illuminated by the source of illumination. The rear-view
`mirror assembly may also include an alarm, with the controller operating the alarm upon
`
`detection of the. driver falling asleep. Also disclosed is a vehicle comprising the
`
`10
`
`15
`
`20
`
`drowsiness detection device.
`
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`E“
`
`25
`
`Figxl‘iis a diagrammatic illustration of the system according to the invention.
`Fig. 215‘ a block diagram of the temporal and spatial processing units of the
`inventionat"'§fl
`I
`Fig31s a block diagram of.the temporal processing unit of the invention.
`Fig: 41s ablock diagram ofthe spatial processingunit ofthe invention.
`Fig.5isa diagram showing the processing of pixels
`in accordance with the
`invention.
`I
`I
`I
`
`Fig. 6 illtistrates the numerical values of the Freeman code used to detemiine
`
`30
`
`movement dii’ection in accordance with the invention.
`
`SUBSTITUTE SHEET (RULE 26)SAMSUNGEXHIBIT1004
`Page 14 of 404
`
`SAMSUNG EXHIBIT 1004
`Page 14 of 404
`
`

`

`WO 99/36893
`
`1
`
`PCTlEP99/00300
`
`f/
`Fig. Zillustrates nested matrices as processed by the temporal processing unit.
`Fig. {Sh/illustrates hexagonal matrices as processed by the temporal processing
`/
`Fig. 9 illustfates reverse-L matrices as processed by the temporal processing unit.
`
`unit.
`
`Fig.1 illustrates angular sector shaped matrices as processed by the temporal
`processing unit /
`.
`.
`spatial prozsing units, and the histogram formation units.
`Fig. 11 IS a block diagramshowing the relationship between the temporal and
`Fig.121’1s/a block diagram showing the interrelationship between the various
`
`histogram formation units
`Fig 13shows the formation of a two-dimensional histogram of a moving area
`
`from two one-dimensional histograms.
`Fig.11s/a block‘ diagram6fan individual histogram formation unit.
`Figs. 15Aand ISB’illustrate the use of a histogram formation unit to find the
`orientation of a line relative to an analysisaxis.
`Fig. 16illustrates a one-dimensional histogram.
`Fig. /7illustrates the use of semi-graphic sub-matrices to selected desired areas
`of an image
`/”'
`Fig.
`118" is a side view illustrating a rear view mirror in combination with the
`drowsiness detection system ofthe invention.
`1..
`Fig. 19ISatop view illustrating operation ofa rear view mirror
`Fig. 26is a schematic illustrating operation of a rear view mirror.
`
`Fig 2
`
`is a cross-sectional top view illustrating a rear view mirror assembly
`
`incorporating the drowsiness detection system ofthe invention.
`Fig. Ms a partial cross-sectional top view illustrating a joint supporting the
`drowsiness detection, system oftheinvention in the mirror assembly ofFig. 21.
`Fig..«2‘3fyisfla top view illustrating" the relationship between the rear view mirror
`assembly of Fig. 21./and a driver.
`I
`Fig. M/lfistrates detection ofthe edges ofthe head ofa person using the system
`
`
`
`10
`
`15
`
`20
`
`25
`
`30_
`
`of the invention.
`
`SUBSTITUTE SHEET (RULE 2§AMSUNG EXHIBIT 1004
`Page 15 Of 404
`
`SAMSUNG EXHIBIT 1004
`Page 15 of 404
`
`

`

`WO 99/36893
`
`_ PCTIEP99/00300
`
`. Fig.1
`illustrates masking outside of the edges ofthe head of a person.
`Fig. fidfistraies masking outside ofthe eyes ofa person.
`Fig. Mistrates detection of the eyes of a person using the system of the
`
`invention.
`
`l
`
`5
`
`10
`
`15
`
`.
`
`Figkzgfillustrates successive blinks in a three-dimensional orthogonal coordinate
`~
`M";
`,
`
`system.
`
`Figs. 2%nd/ygillu5trate conversion of peaks and valleysof eye movement
`histograms to infofr/rpation indicativeiofblinking.
`Fig: 394% a flow diagram illustrating the use of the system of the invention to‘
`detect drowsines
`Fig. }/Ii/ll:strates the use ofsub~images to search a complete image.
`
`a
`f
`Fig. 32/illustrates the use of the system of the invention to detect nostnls and to
`track eye madame.
`Fig.fl} illustrates the use of the system ofthe invention to detect an open eye.
`Fig. Qilflflus’trates the use ofthe system ofthe invention to detect a closed eye.
`,3
`//
`.
`FWOW diagram ofan alternative method ofdetecting drowsiness.
`
`illustrates use of the system to detect a pupil.
`
`F'g. 5,
`
`DETAILED DESCRIPTION OF THE INVENTION
`
`The present invention discloses an application of the generic image processing
`
`20
`
`system disclosed in commonly-owned PCT Application Serial Nos. PCT/FR97/Ol354
`
`and PCT/EP98/05383, the contents of which are incorporated herein by reference for
`
`detection of various criteria'associated withthe human eye, and especially to detection
`
`
`
`
`
`a
`
`/
`
`n
`
`is similar to that described in the aforementioned
`
`that a driver is falling asleep while driving a vehicle.
`
`
`The apparatus of the inventi
`23%Mpc ‘pplication Serial Nos. .Pcr 1197/0354 and PCT/EP98/05383, which will be
`
`
`described herein for purposes of cl
`
`'ty. Referring to Figs.
`
`
`1 and 10, the generic image
`
`with a histogram formation unit
`
`a.
`
`Spatial and temporal processing unit 11 includes
`
`processing system 22 includes a s atialand temporal processing unit ll in combination
`
`'deo signal S originating from a video camera or other
`
`an input 12 that receives a digital
`
`30
`
`imaging device 13 which moni ors a scene 13a.
`
`Imaging device 13 is preferably a
`
`
`
`SUBSTITUTE SEE? (RWPQ‘MSUNG EXHJBJtiom
`
`Page 16 of 404
`
`SAMSUNG EXHIBIT 1004
`Page 16 of 404
`
`

`

`WO 99/36893
`
`_
`
`‘
`
`PCT/EP99/00300
`
`
`
`conventional CMOS-type CCD amera, which for purposes of the presently-described
`
`invention is mounted on a vehicl
`
`facing the driver.
`
`It will be appreciated that when used
`
`in non-vehicular applications,
`
`
`
`e camera may be mounted in any desired fashion to
`
`It is also foreseen that any other appropriate
`
`detect the specific criteria of nterest.
`
`igital output, or an analog output that is converted by an
`
`5
`
`sensor, e.g., ultrasound, IR,
`
`dar, etc., may be used as the imaging device,
`
`Imaging
`
`device 13 may have a direct
`
`A/D converter into digital si nal S.
`
`Imaging device 13 may also be integral with generic
`
`image processing system 2
`
`if desired.
`
`
`
`signal S may be
`
`it
`
`is preferably composed of a
`
`
`progressive signal,
`cession of pairs of interlaced rames, TR, and TR'I and TR; and TR'z, each consisting
`
`of a succession of horizontal canned lines, e.g., l“, ll_2,...,ll.17 in TR}, and 2,; in TRz.
`
`sion of pixels or image-points PI, e.g., an, an and am for
`Each line consists of a succ
`line 11.17 ; alu and am for line 111. Signal S(PI) represents
`line I”; aim and allnif f
`
`signal S composed of pix ls PI.
`
`
`
`
`
`
`S(PI) includes a frame sync onization signal (ST) at the beginning of each
`
`synchronization signal SL) at the beginning ofeach line, and a blanking
`
`
`
`gnal (BL). Thus, S(PI) includes a succession frames, which are representative of the
`time domain, and within each fr
`
`
`e, a series of lines and pixels, which are representative
`
`
`
`
`
`
`
`of the spatial domain.
`
`frames" shall refer to successive frames of the
`I“...
`e
`
`R, or even frames such as TR'I), and "successwe
`
`‘t 11 generatesoutputs ZH and SR 14 to a
`
`
`
`30
`
`output from the system is input digit
`
`V video signal S, which is delayed (SR) to make it
`
`SUBSTITUTE SHEET (RULE 2QAMSUNG EXHIBIT 1004
`
`SAMSUNG EXHIBIT 1004
`Page 17 of 404
`
`

`

`WO 99/36893
`
`’
`
`PCTIEP99/00300
`
`10
`
`
`for the frame, taking into account the calculation time
`synchronous with the output
`for the data in composite si
`al ZH (one frame) The delayed signal SRis used to
`
`display the image received by» amera 13 on a monitor or television screen 10, which may
`also be used to display the1 formation contained1n composite signal ZH. Composite i
`signal ZH may also be trans
`'tted to a separate processing assembly 10a in which fiirther
`processing of the signal may e accomplished.
`
`includes a first
`1‘]
`Referring to Fig. 2, spatial and temporal processing unit
`assembly 11a, which consists of a temporal processing unit 15 having an associated
`memory 16, a spatial processing unit 17 having a delay unit 18 and sequencing unit 19,
`and a pixel clock 20, which generates a clock signal HP, and which serves‘as a clock for
`temporal processing unit 15 and sequencingunit 19. Clock pulses HP are generated by
`clock 20 at the pixel rate ofthe image, which is preferably 13.5 MHZ.
`
`Fig. 3 shows the operation of temporal processing unit 15, the function of which
`
`5
`
`S
`
`is to smooth the video signal and generate a number of outputs that are utilized by spatial
`processing unit 17. During processing,
`temporal processing unit 15 retrieves from
`
`15
`
`memory 16 the smoothed pixel values LI of the digital video signal from the immediately
`
`prior frame, and the values of a smoothing time constant CI for each pixel. As used
`
`herein, L0 and CO shall be used to denote the pixel values (L) and time constants (C)
`
`20
`
`stored in "memory 16 from temporal processing unit 15, and LI and CI shall denote the
`pixel values (L) and time constants (C) respectively for such values retrieved from
`memory 16' for use by temporal processing unit 15. Temporal processing ui'iit 15
`
`generates a binary output signal DP for each pixel, which identifies whether the pixel has
`
`undergone significant variation, and a digital signal CO, which represents the updated
`
`1::
`
`“Wit2“3223!
`43.131my
`
`
`rim}:lifi’ézKim}::5"
`13':was.ad“1m==
`
`i
`
`6“]:
`
`
`
`El"3L5152?:72mg11:2211333
`
`
`
`
`
`1)):ra
`(Emit
`
`limit,1?“
`«an:
`
`.
`calculated value of time constant C.
`
`‘
`erring to Fig. 3, tempora processing unit 15 includes a first block 15a which
`
`signal S. For each pixel PI, the temporal processing
`5 the pixels PI of input vide
`
`unit retrieves from memory 16 a moothed value LI of this pixel from the immediately
`preceding corresponding frame,
`
`hich was calculated by temporal processing unit 15
`
`
`
`during processing of the imme iately prior frame and stored in memory 16 as L0.
`
`30
`
`Temporal processing unit 15 ca ulates the absolute value AB of the difference between
`
`
`
`SUBSTITUTE SHEET{RULE 2GsAMSLJNG EXHIBIT 1QQ4 ,
`-'
`~
`_,
`‘
`Page 18 of 404
`
`SAMSUNG EXHIBIT 1004
`Page 18 of 404
`
`

`

`WO 99/36893
`
`-
`
`PCTlEP99/00300
`
`11
`
`e pixel position (for example an, of In in TR; and
`
`
`
`each pixel value PI and L1 for the s
`
`
`of!” in TR2:
`
`f-= lPI-LI
`
`Temporal processing unit 15 is controlled by clock signal HP from clock 20 in
`
`5
`
`order to maintain synchronization with the incoming pixel stream. Test block 15b of
`
`temporal processing unit 15 receives signal AB and a threshold .value SE. Threshold SE
`
`may be constant, but preferably varie