Page 1 of 404
`
`SAMSUNG EXHIBIT 1004
`Samsung v. Image Processing Techs.
`
`

`

`Se
`
`,
`
`:
`
`|
`
`Daie Received
`(Lach. C. of M.)
`or
`Date Mailed
`
`a DateReceived
`
`os
`
`.
`
`or
`
`88
`47.
`se 8,
`18.
`_
`|
`60.
`19
`| A.
`"20.
`ee
`22.
`8B.
`
`.
`
`©
`
`24,
`
`me
`
`:
`
`23,$f Se.
`95. oe ws.
`
`65.
`
`. NTs|
`‘enti
`“CONTENTS
`7
`(Incl. C. of M.)
`Date Mailed ©
`wehnbiarnGece4SA QPOs 0
`asdYMu)fe 52.
`
`40Ad Oe(A| / 2-10Pine 51. : ee
`
`
`sia tA bo
`
`
`Page 2of 404
`
`' 26.
`
`-
`
`Te 67.0
`
`-_—_—
`
`. 68. oe
`O87.
`| gg
`|
`|
`28.
`| 2 To.
`29.0
`30
`
`_
`
`: Ht
`|
`33.
`8 eerTeneEe
`SX 38:
`| a 76a
`
`36.00
`(37
`38.
`39.
`
`|
`
`_
`
`77
`78 ee
`*
`79.
`|
`| 8
`
`AO 81.:
`
`
`
`82.
`
`(LEFT OUTSIDE)
`
`SAMSUNG EXHIBIT 1004
`
`SAMSUNG EXHIBIT 1004
`Page 2 of 404
`
`

`

`
`
`
`
`
`
`SEARCH NOTES
`
`_(INCLUDING SEARCH STRATEGY)
`
`RH
`
`(RIGHT OUTSIDE)
`
`SAMSUNG EXHIBIT 1004 |
`
`Page 3 of4042se
`
`SAMSUNG EXHIBIT 1004
`Page 3 of 404
`
`

`

`
`
`- ISSUE SLIP STAPLE AREA(for additional cross references)
`
`
`
`0.1.P.E. CLASSIFIER ©
`
`FORMALITY REVIEW
`
`
`RESPONSE FORMALITY REVIEW] ——~=SsdSCSCSCSSC*d
`
`
`
`
`INDEX OF CLAIMS
`sveessssssessesseesssssneees Rejected \sessesssansetsssueeesssees Non-elected
`VO
`Hs ibiscesssescesesseeesesceesees Allowed
`|
`iesestesesssenecapeseeseeeesees Interference
`— (Through numeral)... Canceled
`I veescsvlesecsscesesesesesesenees Appeal .
`i ceesectseseesseneeseseeses Restricted
`0...evaeeeeseseeenesorseeeene Objected
`
`
`
`
`
`
`
`
`
`
`47 coe|
`
`
`
`Final 2|Originat||+=||||||=||||||
`
`TEESEEEE
`
`
`
`ZOMECEE
`
`
`
`
`asltti|tTTe
`‘
`OU!|||||yy
`
`
`
`[ish|TTPTTETt
`eeait|||tttT
`Jatt|tttedete
`LfNTPPeTpt
`holyTTTTTTTPealteaiqietTtetT
`|feoffTTtETtT
`
`
`|feyhePT
`|fez,TtTTTT
`
`
`
`HTiEPyPfestFPPrTTTT
`|fest|TTTTTTT
`7Bf
`
`SeoPtyPPEET
`
`
`
`
`
`EEEEEEEE
`a
`|v2}PPTTETTe
`
`
`
`|frst|PETTET
`
`
`pedPP
`tafe|ft
`LhePettePetTtT
`|heol|TTTETTTT
`
`
`EpetfpPtPt
`ra
`|_|
`
`
`
`
`|foolTTTTTT
`
`|fstTTtTTT
`
`
`
`ftpyete
`|fiedTTTeTI
`
`
`
`PfedTtTTTTTTT
`
`
`
`PpaTTTETTTT
`
`fedTT
`
`
`AE
`P||tTytSEE
`
`
`ceo
`
`
`PiaTEEETTT
`Lie}TPee
`
`BA)ff|jos]|ftTt
`eee+H
`
`
`SCEEEE
`SaEEEEE
`|43;||tTTt
`
`
`|[es]TTTTtTTTt
`|fastTtTETT
`
`
`ge}|TtTTTT
`|fae}|TTTEETTTTT
`ft)|PfLfftTTtT|
`
`
`Scere
`
`
`|fodTTTTttttt
`
`Claim
`
`
`
`
`
`
`
`
`
`
`If more than 150 claims or 10 actions
`staple additional sheet here
`
`(LEFT INSIDE)
`
`SAMSUNG EXHIBIT 1004
`Page 4 of 404
`
`SAMSUNG EXHIBIT 1004
`Page 4 of 404
`
`

`

`
`
`UNITED STATES PATENT AND TRADEMARK OFFICE
`
`«“
`
`x3
`
`—
`
`:
`
`:
`
`:
`COMMISSIONER FOR PATENTS
`UNITED STATES PATENT AND TRADEMARK OFFICE
`WASHINGTON, D.C, 20231
`www. uspto.gov
`
`.
`
`QVEVN
`Bib Data Sheet
`
`;
`
`FILING DATE
`
`
`
`
`
`
`
`
`SERIAL NUMBER |—_92/09/2001 CLASS GROUP ARTUNIT| DocKETNO.
`
`
`
`
`_ 2632
`20046H-00060
`09/600,320
`340
` RULE
`
`APPLICANTS
`Patrick Pirim, Paris, FRANCE;
`ThomasBinford, Cupertino, CA;
`
`
`:
`.
`, ot
`* CONTINUING DATA KREKKEKRERERRERREERERR y pit / y
`PCT/EP99/00300 01/15/1999
`THIS APPLICATION IS A 371
`
`
`/
`
`* FOREIGN APPLICATIONS (* vemerneeeeTRA
`
`FRANCE 98/00378 01/15/1998
`PCT/EP98/05383 08/25/1998
`IF REQUIRED, FOREIGN FILING LICENSE .
`-
`ISRANTED ** 02/23/2001
`
`INDEPENDENT
`5 USC
`119
`(acd)
`condi
`10 yes
`no
`:
`STATE OR|
`SHEETS
`Foreign Priority‘claimed i LJ
`
`
`
`
`3 COUNTRY|DRAWING119 (a-d) conditions Ja yes C) no C) Met after CLAIMS
`met
`—f wane:
`6
`erified and
`ae
`:
`
`Acknowledged
`Examiner's Signature
`Initials
`.
`ADDRESS
`,
`
`Babak $ Sani
`Townsend and Townsend and Crew
`Two Embarcadero Center 8th Floor
`san Francisco CA 94111
`TITLE
`
`|
`
`~
`
`a
`
`a
`4
`
`Method and apparatus for detection of drowsiness
`
`] | |
`= All Fees
`O11.16Fees(Filing)
`FILING FEE |FEES:Authority has been given in Paper
`O 1.17 Fees ( Processing Ext. of
`
`4!ECEIVED—jNo. to charge/credit DEPOSIT ACCOUNT time ) : ———
`
`
`forfollowing:
`C] 1.18 Fees ( Issue )
`|
`C] other
`E Credit
`
`|
`
`|
`
`SAMSUNG EXHIBIT 1004
`Page 5 of 404
`
`SAMSUNG EXHIBIT 1004
`Page 5 of 404
`
`

`

`09/600390 —
`
`patent application sertal no.
`
`Department of Commerce
`Patent and Trademark Office
`fee record
`
`
`SAMSUNG EXHIBIT 1004
`Page6 of 404
`
`
`
`SAMSUNG EXHIBIT 1004
`Page 6 of 404
`
`

`

`“45.01.2000
`
`EPAbe
`sue ve, 09/6.00390
`: 2 pchmvodoube..!
`ote eee’
`°
`taniaJMDa
`534Rec'd PCT/PITtA JUL2000
`
`METHODANDAPPARATUSFORDETECTIONOFDROWSINESS
`Inventors:
`Dr.Patrick Pirim
`Dr. Thomas Binford
`
`:
`
`‘i
`
`5
`
`BACKGROUND OF THE INVENTION
`
`
`
`
`
`
`
`1.
`
`15
`
`20
`
`Field of the Invention.
`
`The present invention relates generally to an image processing system, and
`more particularly to the use of a generic image processing system to detect
`drowsiness.
`.
`.
`10
`1.Description ofthe Related Art.
`It is well knownthata significant number ofhighway accidents result from
`drivers becoming drowsy or falling asleep, which results in many deaths and
`injuries. Drowsinessis also a problem in other fields, such as for airline pilots and
`powerplant operators, in which great damage may result from failure to stay alert.
`A number ofdifferent physical criteria may be used to establish when a
`‘person is drowsy, including a change in the durationand interval of eye blinking.
`Normally, the duration ofblinking is about 100to 200 ms when awakeand about
`500 to. 800 ms when drowsy. The time interval between successive blinks is
`generally constant while awake, but varies within a relatively broad range when
`drowsy.
`
`Numerous devices have been proposed todetect drowsiness of drivers.
`vices are shown, for example, im U.S. Patent Nos. 5,841,354; 5,813,99;
`5,689,241; 5,684,461; 5,682,144; 5,46p,143; 5,402,109; 5,353,013; 5,195,606;
`
`4,928,090; 4,555,697; 4,485,375; and 4/259,665. In general, these devices fall into
`25
`three categories: i) devices that detect/ movement of the head ofthe driver, ¢.g.,
`tilting: ii) devices that detect a phys} logical change in the driver, ¢.g., altered
`heartbeat or breathing, andiii) devices thatdetect a physical result of the driver
`falling asleep, e.g., a reduced grip om
`the steering wheel. Noneofthese devicesis
`believed to have met with commercyal
`success.
`
`te
`
`
`
`AMENDED SHEET
`
`_ SAMSUNGEXHIBIT. 1004
`Page 7 of 404
`
`SAMSUNG EXHIBIT 1004
`Page 7 of 404
`
`

`

`"45-04-2000
`
`|
`
`|
`
`:
`
`EP 009900300
`
`©
`
`tn
`
`gfe eee
`
`ee PETIEP-99Q0300 oo”
`048 JPCT 361
`January 11, 2000
`
`
`
`
`
`Ww
`
`10
`
`15
`
`20
`
`evaluation of drowsinesslevel of a drver using a video cameraplaced near the feet
`of the driver and a processing unit fiir the camera image with a software detecting
`the blinks of the eyes determining the time gap between the beginning and the end
`of the blink. More particularly, a
`upit 10 ofthe processorrealizes:
`
`e a memorization of the vifleo image andits treatment, so as to determine
`
`an area comprising the driver’s eyps,
`e the detection ofthe tim¢ gap betweenthe closing ofthe driver eyelids and
`‘their full opening and
`°
`« a treatment in a mem@ry 11 and a processor 22 in combination with unit
`10to calculate a ratio of slow blink apparition.
`|
`The object of the intemnational patent application published WO-97/01246
`is a security system comprising a video camera placed within the rear-view mizror
`ofacar and a video screen remotely disposed for the analysis ofwhatis happening
`in the car and aroundit, as well as ofwhat happened due to the recording ofthe
`output video signal of the camera. This is in fact a concealed camera (within the
`rear-view mirror), so that it is imperceptible to vandals and thieves and which
`observes a large scope including the inside of the car andits surroundings, the
`record allowing one to know later. what has happened in this scope (page6, lings
`13 to 19), this is not a detector whoseeffective angle is strictly limited to the car
`driver face in orderto detect its eventual drowsiness and to make him awake.
`
`Commonly-owned PCT
`Application Serial Nos. PCT/FR97/01354 and
`
`98/05383 disclose a generic image processing system that operates to
`
`AMENDED SHEET
`
`|
`
`5S_ SAMSUNG EXHIBIT.1004
`Page 8 of 404
`
`—_—
`
`SAMSUNG EXHIBIT 1004
`Page 8 of 404
`
`

`

`_ PrOMCTRec'd.4 JUL2000
`
`09/600390
`
`WO99/36893
`
`we
`
`PCT/EP99/00300
`
`METHOD AND APPARATUS FOR DETECTION OF DROWSINESS
`
`Keplicebybat ans So
`
`_ in
`
`20
`
`25
`
`30
`
`BACKGROUND OF THE INVENTION
`
`1.
`
`Field ofthe Invention.
`The present invention relates generally to an image processing system,
`and more particularly to the use of a generic image processing system to detect
`drowsiness.
`.
`
`1 Description of the Related Art.
`It is well known that a significant number of highway accidents result from
`drivers becoming drowsy or falling asleep, which results in many deaths and injuries.
`Drowsinessis also a problemin other fields, such as for airline pilots and power plant
`operators, in which great damage mayresult from failure to stay alert.
`A numberofdifferent physical criteria may be used to establish when a personis
`
`drowsy, including a change in the duration and interval of eye blinking. Normally, the
`duration of blinking is about 100 to 200 ms when awake and about 500 to 800 ms when
`drowsy. The timeinterval between successiveblinks is generally constant while awake,
`but varies within a. relatively broad range when drowsy.
`Numerous devices have been proposed to detect drowsiness of drivers. Such
`devices are
`shown,
`for
`example,
`in U.S. Patent Nos.
`5,841,354; 5,813,99;
`5,689,241;5,684,461;
`5,682,144;
`5,469,143;
`5,402,109;
`5,353,013;
`5,195,606;
`4,928,090: 4,555,697; 4,485,375; and 4,259,665.
`In general, these devices fall into
`three categories: i) devices that detect movement ofthe head ofthe driver, e.g., tilting;ii)
`devices that detect a physiological change in the driver, e.g., altered heartbeat or
`breathing, and iii)devices that detect F physical result ofthe driver falling asleep, ¢.g., a
`reduced grip on the steering wheel. None ofthese devices is believed to have met with
`commercial success.
`|
`
`and
`Serial Nos. PCT/FR97/01354
`Commonly-owned PCT Application
`PCT/EP98/05383 disclose a generic image processing system that operates to localize
`
`SUBSTITUTE SHEET
`
`eH SHEET (RULE 2@\MSUNGEXHIBIT1004_
`a
`Page 9 of 404
`
`E 2
`
`SAMSUNG EXHIBIT 1004
`Page 9 of 404
`
`

`

`WO 99/36893
`
`,
`
`Oo
`
`PCT/EP99/00300
`
`
`
`20
`
`objects in relative movementin an image and to determine the speed and direction of the
`objects in real-time. Each pixel ofan {mage is smoothed using its own time constant. A
`binary value corresponding to the exfstence ofa significant variation in the amplitude of
`the smoothed pixel from the prigr frame, and the amplitude of the variation, are
`determined, and the time constant
`for the pixel is updated. For each particular pixel, two
`
`matrices are formed that include&subset of the pixels spatially related to the particular
`pixel. The first matrix containg the binary values of the subset of pixels. The second
`matrix contains the amplitude
`the variation of the subsetof pixels.
`In the first matrix,
`
`it is determined whether the
`pixels along an oriented direction relative to the particular
`pixel have binary values reprpsentative of significant variation, and, forsuch pixels, it is
`determined in the second mgtrix whether the amplitude of these pixels varies in a known
`mannerindicating move ent
`in the oriented direction.
`In domains that
`include
`
`luminance, hue, saturation{
`speed, oriented direction, time constant, and x and y position,
`a histogram is formedofthe values in the first and second matrices falling in user
`
`selected combinations offsuch domains. Using the histograms, it is determined whether
`
`there is an area having tHe characteristics ofthe selected combinations of domains.
`It. would be desirable to apply such a generic image processing system to detect
`the drowsiness of a person.
`,
`SUMMARYOF THE INVENTION
`Thepresent invention is a process of detecting a driver falling asleep in which an
`image ofthe face of the driver is acquired. Pixels of the image having characteristics
`corresponding to characteristics of at least one eye of the driver are selected and a
`histogram is formed of the selected pixels. The histogram is analyzed over time to
`identify each opening and closing of the eye, and from the eye opening and closing
`information, characteristics indicative of a driver falling asleep are determined.
`In one embodiment, a sub-area of the image comprising the eye is determined
`priorto the step of selecting pixels of the imagehaving characteristics corresponding to
`characteristics of an eye.
`In this embodiment, the step of selecting pixels of the image
`having characteristics of an eye involvesselecting pixels within the sub-area of the image.
`The step ofidentifying a sub-area ofthe image preferably involves identifying the head of
`
`25
`
`30
`
`4
`
`SUBSTITUTE SHEET (RULE2@,MSUNG EXHIBIT 1004
`Page10 of404
`
`SAMSUNG EXHIBIT 1004
`Page 10 of 404
`
`

`

`WO 99/36893
`
`PCT/EP99/00300
`
`3
`
`the driver, or a facial characteristic of the driver, such as the driver's nostrils, and then
`identifying the sub-area of the image using an anthropomorphic model. The head of the
`driver may be identified by selecting pixels of the image having characteristics ©
`corresponding to edges of the head ofthe driver. Histograms of the selected pixels of
`the edgesofthe driver's head are projected onto orthogonal axes. These histogramsare
`then analyzed to identify the edges ofthe driver's head.
`
`The facial characteristic of the driver
`nay be identified by selecting pixels of the
`
`Aving characteristics correspondingtf the facial characteristic. Histogramsof the
`
`selected pixels of the facial characteristic
`fare projected onto orthogonal axes. These
`
`histograms are then analyzedto identify the facial characteristic.
`If desired, the step of
`
`identifying the facial characteristic in thé
`image involves searching sub-images of the
` to detect the nostrils. To confirm detéction ofthe nostrils, the histograms ofthe nostril
`image until the facial characteristic is foynd.
`In the case in which the facial characteristic
`is the nostrils of the driver, a histogra
`is formedof pixels having low luminance levels
`
`
`5
`
`15
`
`20
`
`bd
`
`pixels may be analyzed to determine whether the spacing between the nostrils is within a
`desired range and whether the dimeAsionsofthe nostrils fall within a desired range.
`In
`orderto confirm theidentification of the facial characteristic, an anthropomorphic model
`and the location of the facial chdracteristic are used to select a sub-area of the image
`containing a second facial chafacteristic.
`Pixels of the image having characteristics
`corresponding to the second facial characteristic are selected and a histograms of the
`selected pixels of the secpnd facial characteristic are analyzed to confirm the
`identification ofthe first facial characteristic.
`
`—
`
`.
`
`In this embodiment, the step analyzing the histogram over time to
`shadowing ofthe eye.
`identify each opening and closing of
`the eye involves analyzing the shape of the eye
`shadowing to determine openingsandclosings ofthe eye. The histograms of shadowed
`pixels are preferably projected onto/orthogonal axes, and the step of analyzing the shape
`ofthe eye shadowing involvesanalyzing the width and height ofthe shadowing.
`
`30
`
`
`SUBSTITUTESenn(RULE26)
`
`ne __......SAMSUNG EXHIBIT.1004.
`Page 11 of 404
`
`SAMSUNG EXHIBIT 1004
`Page 11 of 404
`
`

`

`WO 99/36893
`
`. PCT/EP99/00300
`
`
`
`alternative method of dejermining openings and closings of the eyes of the
`An
`
`involves selecting pixels of
`the image having characteristics of movement
`
`corresponding to blinking.
`In this/embodiment, the step analyzing the histogram over
`
`time to identify each opening ang
`closing of the eye involves analyzing the number of -
`pixels in movement corresponding to blinking over time. The characteristics of a
`blinking eye are preferably sefected from the group consisting ofi) DP=1,
`ii) CO
`indicative of a blinking eyelid, Aii) velocity indicative of a blinking eyelid, and iv) up and
`down movementindicative offa blinking eyelid.
`
`
`An apparatus for detecting a driver
`falling asleep includes a sensor for acquiring
`of the face of the driver, a coftroller, and a histogram formation unit for
`an i
`
`forming a histogram on pixels having selefted characteristics. The controller controls the
`
`histogram formation unit
`to select
`/pixels of the image having characteristics
`corresponding to characteristics ofat Idast one eye ofthe driver and to form a histogram
`of the selected pixels. The controller/analyzes the histogram overtime to identify each
`determines from the opening andclosing information
`opening and closing ofthe eye, and
`
`ofthe driverfalling asleep.
`on the eye, characteristics indicative
`In one embodiment, the controller interacts with the histogram formation unit to
`identify a sub-area of the image comprising the eye, and the controller controls the
`histogram formation unit
`to select pixels of the image having characteristics
`corresponding to characteristics of the eye only within the sub-area of the image.
`In
`order to select the sub-area of the image, the controller interacts with the histogram
`formation unit to identify the head of the driver in the image, or a facial characteristic of
`the driver, such as the driver's nostrils. The controller then identifies the sub-area of the —
`
`15
`
`fo,
`
`20
`
`the
`To identify the head of the driver,
`image using an anthropomorphic model.
`histogram formation unit selects pixels of the image having characteristics corresponding
`to edges of the head ofthe driver and forms histograms ofthe selected pixels projected
`onto orthogonal axes. To identify a facial characteristic of the driver, the histogram
`formation unit selects pixels of the image having characteristics corresponding to the
`facial characteristic and forms histograms of the selected pixels projected onto
`orthogonal axes. The controller then analyzes the histograms of the selected pixels to
`
`25
`
`30
`
`SUBSTITUTEY,(RULE26)
`
`SAMSUNGEXHIBIT. 1004
`Page 12 of 404
`
`SAMSUNG EXHIBIT 1004
`Page 12 of 404
`
`

`

`WO 99/36893
`
`"
`
`.
`
`PCT/EP99/00300
`
`identify the edges ofthe head ofthe driver or the facial characteristic, as the case maybe.
`If the facial characteristic is the nostrils of the driver, the histogram formation unit selects
`pixels of the image having low luminancelevels corresponding to the luminancelevel of.
`the nostrils. The controller may also analyze the histograms of the nostril pixels to
`determine whether thespacing between the nostrilsis within a desired range and whether
`dimensions of the nostrils fall within a.desired range.
`If desired, the controller may
`interact with the histogram formation unit to search sub-images of the image to identify
`the facial characteristic.
`
`In order to verify identification of the facial characteristic, the controller uses an
`anthropomorphic model and the location of the facial characteristic to cause the
`histogram formation unit to select a sub-area of the image containing a second facial
`characteristic. The histogram formation unit selects pixels of the image in the sub-area
`having characteristics corresponding to the second facial characteristic and forms a
`histogram of such pixels. The controller then analyzes the histogram of the selected
`pixels corresponding to the second facial characteristic to identify the second facial
`characteristic and to thereby confirm the identification ofthe first facial characteristic.
`In one embodiment, the histogram formation unit selects pixels of the image
`having low luminance levels corresponding to shadowing ofthe eyes, and the controller
`then analyzes the shape of the eye shadowing to identify shapes corresponding to
`openings and closings of the eye. The histogram formation unit preferably forms
`histograms of the shadowed pixels of the eye projected onto orthogonal axes, and the
`controller analyzes the width and height of the shadowing to determine openings and
`closings ofthe eye.
`.
`In an alternative embodiment, the histogram formation unit selects pixels of the
`image in movement corresponding to blinking and the controller analyzes the number of
`pixels in movement over time to determine openings and closings of the eye. The
`characteristics of movement correspondingto blinking are preferably selected from the
`group consisting of i) DP=1, ii) CO indicative of a blinking eyelid, iii) velocity indicative
`ofa blinking eyelid, and iv) up and down movementindicative ofa blinking eyelid.
`
` 10
`
`20
`
`25
`
`SUBSTITUTE SHEET(RULE26)
`om
`SAMSUNGEXHIBIT 1004
`Page 13 of 404
`
`SAMSUNG EXHIBIT 1004
`Page 13 of 404
`
`

`

`
`
`
`
` HA.
`ayagepyresdtHORdineaaf
`ts
`
`WO 99/36893
`‘
`
`-
`
`PCT/EP99/00300
`
`6
`
`If desired, the sensor may beintegrally constructed with the controller and the
`histogram formation unit. The apparatus may comprise an alarm, which the controller
`operates upon detection of the driver falling asleep, and may comprise an illumination
`source, such as a source ofIR radiation, with the sensor being adapted to view the driver
`
`whenilluminated by the illumination source.
`A rear-view mirror assembly comprises a rear-view mirror and the described
`apparatus for detecting driver drowsiness mounted to the rear-view mirror.
`In one
`embodiment, a bracket attaches the apparatus to the rear-view mirror.
`In an alternative
`embodiment,
`the rear-view mirror comprises a housing having an open side and an
`interior. The rear-view mirror is mounted to the open side of the housing, and is see-
`through from theinterior of the housing to the exterior of the housing. The drowsiness
`detection apparatus is mounted interior to the housing with the sensor directed toward
`the rear-view mirror.
`If desired, a joint attaches the apparatus to the rear-view mirror
`assembly, with the joint being adapted to maintain the apparatus in a position facing the
`driver during adjustment of the mirror assembly by the driver. The rear-view mirror
`assembly mayinclude asourceofillumination directed toward the driver, with the sensor
`adapted to view the driver whenilluminated by the source of illumination. The rear-view
`mirror assembly mayalso include an alarm, with the controller operating the alarm upon
`detection of the. driver falling asleep. Also disclosed is a vehicle comprising the
`drowsiness detection device.
`=
`BRIEF DESCRIPTION OF THE DRAWINGS
`Fig.Yisa diagrammaticillustration ofthe system according to the invention.
`Fig.2: is a block diagram of the temporal and spatial processing units of the
`invention”
`Fig.3-is a block diagram ofthe temporal processing unit of the invention.
`Fig.4iiSaa blockdiagram ofthe spatial processingunit ofthe invention.
`Fig.5ds‘a diagram showing the processing of pixels
`in accordance with the
`invention.
`.
`Fig. 6 illdstrates the numerical values of the Freeman code used to determine
`movement difection in accordance with the invention.
`
`
`
`10
`
`15
`
`20
`
`25
`
`30
`
`SUBSTITUTE SHEETT(RULE 26)_SAMSUNG.EXHIBIT4004
`Page 14 of 404
`
`SAMSUNG EXHIBIT 1004
`Page 14 of 404
`
`

`

`WO 99/36893
`
`:
`
`PCT/EP99/00300
`
`/
`
`unit.
`
`Fig. 7illustrates nested matrices as processed by the temporal processing unit.
`Fig. 8illustrates hexagonal matrices as processed by the temporal processing
`ic Loy
`.
`Fig. 9 iflustfates reverse-L matrices as processed by the temporal processing unit.
`Fig. 19 illustrates angular sector shaped matrices as processed by the temporal
`processing unit, 2 ,
`spatialreeunits, andthehistogramformationunits.
`Fig. 11/is a block diagram showing the relationship between the temporal and
`Fig. 12%a block diagram showing the interrelationship between the various
`histogram formation units.
`Fig. 13°shows the formation of a two-dimensional histogram of a moving area
`from two one-dimensional histograms.
`Fig.
`| sa blockdiagramofan individual histogram formation unit.
`Figs. 15Aand 15B“fllustrate the use of a histogram formation unit to find the
`orientation ofa linerelative to an analysisaxis.
`Fig. 16.illastrates a one-dimensional histogram.
`Fig.aillustrates the use ofsemi-graphic sub-matrices to selected desired areas
`ofan image: So
`Fig. 18is a side view illustrating a rear view mirror in combination with the
`drowsiness detection system ofthe invention.
`.
`Fig. 193isatopview illustrating operation ofa rear view mirror.
`Fig. 20s a schematicillustrating operation of a rear view mirror.
`Fig. 214s a cross-sectional top view illustrating a rear view mirror assembly
`incorporating the drowsiness detection system ofthe invention.
`Fig. 22isis a partial cross-sectional top viewillustrating a joint supporting the
`drowsiness detectionsystem ofthe invention in the mirror assembly ofFig. 21.
`Fig..231is a top view illustrating the relationship between the rear view mirror
`assembly ofFig. 2Land a driver.
`Fig. 24 Atfustrates detection of the edges of the head of a person using the system
`
`
`
`20
`
`25
`
`30
`
`of the invention.
`
`SUBSTITUTE SHEET (RULE 2@AMSUNG EXHIBIT 1004
`“Page 15 of 404
`
`SAMSUNG EXHIBIT 1004
`Page 15 of 404
`
`

`

`WO 99/36893
`
`_ PCT/EP99/00300
`
`system.
`
`_ Fig.,
`illustrates masking outside ofthe edges ofthe headofa person.
`Fig. 26ise masking outside ofthe eyes ofa person.
`Fig. 27Aflustrates detection of the eyes of a person using the system of the
`invention.
`Fig.28illustrates successive blinks in a three-dimensional orthogonal coordinate
`oe
`Figs. 296and296illustrate conversion ofpeaks and valleysofeye movement
`histogramsto information indicativeofblinking.
`Fig. 304s a flow diagram illustrating the use of the system of the invention to
`detect drowsiness
`.
`_ 0
`Fig.syusrtheuseofsub-imagesto search a completeimage.
`Fig. 32itfustrates the use of the system ofthe invention to detect nostrils and to
`
`track eye mbvemefit
`Fig.33illustrates the use ofthe system ofthe invention to detect an openeye.
`Fig.HMiusiratesthe use ofthe system ofthe invention to detect a closed eye.
`neCasesuseofthesystemtodetectapupil.
`
`
`Fig,,25
`
`is
`
`a-fiow diagram ofan alternative method of detecting drowsiness.
`
`DETAILED DESCRIPTION OF THE INVENTION
`
`The present invention discloses an application of the generic image processing
`system disclosed in commonly-owned PCT Application Serial Nos. PCT/FR97/01354
`and PCT/EP98/05383, the contents of which are incorporated herein by referénce for
`detection of various criteria associated with. the human eye, and especially to detection
`that a driveris falling asleep while driving a vehicle.
`
`is similar to that described in the aforementioned
`The apparatus oftheinventi
`<Pplication Serial Nos. -PCT/FR97/01354 and PCT/EP98/05383, which will be
`described herein for purposes ofcldri
`ty. Referring to Figs.
`1 and 10, the generic image
`
`processing system 22 includes a spatial and temporal processing unit 11 in combination
`with a histogram formation unit
`
`i Spatial and temporal processing unit 11 includes
`an input 12 that receives a digital/video signal S originating from a video camera or other
`imaging device. 13 which monifors a scene 13a.
`Imaging device 13 is preferably a
`
`
`
`
`
`
`
`30
`
`SUBSTITUTE SHEET (RULE ¥ MSUNG EXHIBIT 1004
`Page 16 of 404
`
`SAMSUNG EXHIBIT 1004
`Page 16 of 404
`
`

`

`WO 99/36893
`
`.
`
`‘
`
`PCT/EP99/00300
`
`
`conventional CMOS-type CCD qamera, which for purposes of the presently-described
`inventionis mounted on a vehiclg facing the driver.
`It will be appreciated that when used
`
`in non-vehicular applications,
`fhe camera may be mounted in any desired fashion to
`detect the specific criteria of {nterest.
`It is also foreseen that any other appropriate
`
`sensor, e.g., ultrasound, IR,
`Hadar, etc., may be used as the imaging device.
`Imaging
`fligital output, or an analog output that is converted by an
`device 13 may have a direct
`A/D converterinto digital signal S.
`Imaging device 13 mayalso be integral with generic
`
`
`image processing system 2
`
`5
`
`j@
`
`if desired.
`signal S may be
`
`is preferably composed of a
`it
`progressive signal,
`
`cession of pairs of interlacedframes, TR; and TR’, and TR2 and TR’, each consisting
`of a succession of horizontal Acannedlines, e.g., fia, lia,...l17 in TRi, and 21 in TRo.
`
`Eachline consists of a succ
`sion of pixels or image-points PI, e.g., a11, a12 and a3 for
`line 41.3; aliz.1 and a1i7.2) f
`line liz; ali and aj2 for line l.;. Signal S(PI) represents
`
`signal S composed ofpix Is PI.
`
`S(PI) includes a frame synchfonization signal (ST) at the beginning of each
`synchronization signalASL) at the beginning ofeach line, and a blanking
`
`gnal (BL). Thus, S(PI) includes/a succession frames, which are representative of the
`
`time domain, and within each frayhe, a series of lines and pixels, which are representative
`
`
`
`of the spatial domain.
`
`
`indicating the existence and localization/of an area or object in motion, and the speed V
`and the oriented direction of displacement DI of each pixel of the image. Also preferably
`output from the system is inputdigit | video signal S, which is delayed (SR) to makeit
`
`30
`
`[|
`SUBSTITUTE SHEET (RULE 28),MSUNG EXHIBIT 1004
`
`SAMSUNG EXHIBIT 1004
`Page 17 of 404
`
`

`

`WO 99/36893
`
`PCT/EP99/00300
`
`10
`
`5
`
`o
`
`15
`
`20
`
`atheWa!Sad
`
`mit
`
`FReyATGbie
`.
`athhegtTEayfeWeWORLAG$l
`
`
`synchronous with the outputZHfor the frame, taking into accountthe calculation time
`
`for the data in composite sighal ZH (one frame). The delayed signal SR is used to
`display the image received byfamera 13 on a monitor or television screen 10, which may
`also be used to display the ihformation containedin composite signal ZH. Composite |
`signal ZH mayalso betransnjitted to a separate processing assembly 10a in which further
`processing of the signal mayfbe accomplished.
`Referring to Fig. 2, spatial and temporal processing unit 11 includes a first
`assembly 1 1a, which consists of a temporal processing unit 15 having an associated
`memory 16, a spatial processing unit 17 having a delay unit 18 and sequencing unit 19,
`and a pixel clock 20, which generates a clock signal HP, and which servesas a clock for
`temporal processing unit 15 and Sequencingunit 19. Clock pulses HP are generated by
`clock 20 at the pixel rate ofthe image, whichis preferably 13.5 MHZ.
`Fig. 3 shows the operation of temporal processing unit 15, the function of which
`is to smooththe video signal and generate a numberof outputs that are utilized by spatial
`processing unit 17. During processing,
`temporal processing unit 15 retrieves from
`memory 16 the smoothed pixel values LI ofthe digital video signal from the immediately
`prior frame, and the values of a smoothing time constant CI for each pixel. As used
`herein, LO and COshall be used to denote the pixel values (L) and time constants (C)
`stored in memory 16 from temporal processing unit 15, and LI and Cl shall denote the
`pixel values (L) and time constants (C) respectively for such values retrieved from
`memory 16for use by temporal processing unit 15. Temporal processing unit 15
`generates a binary output signal DP for each pixel, which identifies whether the pixel has
`undergone significant variation, and a digital signal CO, which represents the updated
`
`Fy
`
`HoeH
`
`iy
`
`eR
`
`PF
`type
`
`calculated value oftime constant C.
`erring to Fig. 3, temporal processing unit 15 includesa first block 15a which
`oy
`
`S the pixels PI of input vided
`signal S. For each pixel PI, the temporal processing
`unit retrieves from memory 16 a gmoothed value LI ofthis pixel from the immediately
`preceding corresponding frame,
`hich was calculated by temporal processing unit 15
`during processing of the immediately prior frame and stored in memory 16 as LO.
`Temporal processing unit 15 calculates the absolute value AB of the difference between
`
` 5
`
`30
`
`
`
`SUNSTITUTE SHEET (RULE 20au4suNGEXHIBIT1004
`ee eT Teo
`
`SAMSUNG EXHIBIT 1004
`Page 18 of 404
`
`

`

`WO 99/36893
`
`.
`
`PCT/EP99/00300
`
`11
`
`each pixel value PI and LI for the same pixel position (for example a,1, of11.1 in TRy and
`of ]1.) in TRa:
`
`= |PI-LI
`Temporal processing unit 15 is controlled by clock signal HP from clock 20 in
`
`
`
`
`
`
`
`
`
`
`20
`
`25
`
`order to maintain synchronization with the incoming pixel stream. Test block 15b of
`
`temporal processing unit 15 receives signal AB and a threshold value SE. Threshold SE
`may be constant, but preferably varies based uponthepixel value PI, and morepreferably
`varies with the pixel value so as to form a gamma correction. Known meansof varying
`SE to form a gammacorrectionis represented by the optional block 15e shown in dashed
`lines. Test block 15b compares, on a pixel-by-pixel basis, digital signals AB and SE in
`order to determine a binary signal DP. If AB exceeds threshold SE, whichindicates that
`
`pixel value PI has undergonesignificant variation as compared to the smoothed value LI
`of the same pixel in the prior frame, DPis set to "1" for the pixel under consideration.
`Otherwise, DP is set to "0" for such pixel.
`When DP= 1, the difference between the pixel value PI and smoothed value LI
`ofthe same pixel in the prior frameis considered too great, and temporalprocessing unit
`15 attempts to reduce this difference in subsequent frames by reducing the smoothing
`time constant C for that pixel. Conversely, ifDP = 0, temporal processing unit 15
`attempts to increase this difference in subsequent frames by increasing the smoothing
`‘time constant C for that pixel. These adjustments to time constant C asa function of the
`value ofDP are made by block 1