`
`PATENT COOPERATION TREATY
`
`
`
`
`
`x
`
`
`
`A
`
`
`&
`PCT
`
`
`$go
`
`
`oNTERNATIONAL PRELIMINARY EXAMINATION
`
`%
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`(PCT Article 36 and Rule 70)
`
`
`
`
`
`See Notification of Transmittal of International
`
`
`
`
`
`
`Preliminary Examination Report (Form PCT/PEA/416)
`
`
`
`
`Priority date (day/month/year)
`
`
`
`15/01/1998
`
`Form PCT/APEA/409 (cover sheet) (January 1994)
`
`Applicant's oragent's file reference
`
`
`
`
`048J PCT 361
`
`
`
`intemationai application No.
`
`
`PCT/EP99/00300
`
`
`
`
`
`FOR FURTHER ACTION
`
`
`
`Internationalfiling date (day/month/year}
`
`
`
`
`15/01/1999
`
`
`
`International Patent Classification (IPC) or national classification and IPC
`
`
`
`
`
`
`
`
`G08B21/00
`
`
`
`
`
`
`
`
`
`
`
`Applicant
`
`HOLDING B.E.V.SA et al.
`
`
`
`
`
`
`1. This internationai preliminary examination report has been prepared by this international Preliminary Examining Authority
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`and is transmitted to the applicant according to Article 36.
`
`
`
`
`
`
`
`
`
`
`
`
`
`This REPORT consists of a total of 7 sheets, including this cover sheet.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`& This report is also accompanied by ANNEXES,i.e. sheets of the description, claims and/or drawings which have
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`been amended and are the basisfor this report and/or sheets containing rectifications made before this Authority
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`(see Rule 70.16 and Section 607 of the Administrative Instructions under the PCT).
`
`
`
`
`
`
`
`
`
`
`
`
`These annexesconsist of a total of 11 sheets.
`
`
`
`
`
`
`
`
`
`
`
`
`
`This report contains indications relating to the following items:
`
`
`
`
`
`
`
`
`
`
`
`
`
`|
`i
`
`
`
`
`
`
`
`
`
`
`Basis of the report
`
`
`
`
`Priority
`
`Non-establishmentof opinion with regard to novelty, inventive step and industrial applicability
`
`
`
`
`
`
`
`
`
`
`
`
`Lack of unity of invention
`
`
`
`
`
`Reasoned statement under Article 35(2) with regard to novelty, inventive step or industrial applicability;
`
`
`
`
`
`
`
`
`
`
`
`
`
`citations and explanations suporting such statement
`
`
`
`
`
`
`Certain documents cited
`
`
`
`Certain defects in the international application
`
`
`
`
`
`
`Certain observations on the international application
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Date of submission of the demand
`
`
`
`
`
`
`
`
`Date of completion of this report
`
`
`
`
`
`
`
`
`09/08/1999
`
`
`
`Name and mailing address ofthe international
`
`
`
`
`
`
`preliminary examining authority:
`
`
`
`European Patent Office
`
`d D-80298 Munich
`
`
`
`
`
`
`Teal. +49 89 2399 -0 Tx: 523656 epmu d
`
`
`
`
`
`
`
`
`Fax: +49 89 2399 - 4465
`
`
`
`
`
`
`
`
`
`
`17.02.2000
`
`
`
`Authorized officer
`
`
`
`
`
`
`Wright, J
`
`Telephone No. +49 89 2399 2705
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Petitioner LG Ex-1023, 0001
`
`Petitioner LG Ex-1023, 0001
`
`

`

`
`
`~
`
`INTERNATIONAL PRELIMINARY
`
`EXAMINATION REPORT
`
`
`
`
`
`
`Intemational application No. PCT/EP99/00300
`
`
`
`
`
`
`1. Basis of the report
`
`
`
`
`
`1. This report has been drawn on the basis of (substitute sheets which have been furnished to the receiving Office in
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`response to an invitation under Article 14 are referred to in this report as “originally filed” and are not annexed to
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`the report since they do not contain amendments.):
`
`
`
`
`
`
`
`Description, pages:
`
`
`
`
`
`2-57
`
`1la
`
`
`
`
`
`Claims, No.:
`
`
`
`1-39
`
`
`
`as originally filed
`
`
`
`
`
`as received on
`
`
`
`
`15/01/2000 with letter of
`
`
`
`
`
`
`11/01/2000
`
`as received on
`
`
`
`
`15/01/2000 with letter of
`
`
`
`
`
`
`11/01/2000
`
`
`
`
`
`Drawings, sheets:
`
`
`
`
`1/20-20/20
`
`
`
`as originally filed
`
`
`
`
`
`2. The amendments haveresulted in the cancellation of:
`
`
`
`
`
`
`
`
`
`
`
`1 ‘the description,
`
`
`OO
`the claims,
`
`
`
`[]
`the drawings,
`
`
`
`
`
`
`pages:
`
`Nos.:
`
`
`sheets:
`
`
`
`
`
`3. C1 This report has been established as if (some of) the amendments had not been made, since they have been
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`considered to go beyond the disclosure asfiled (Rule 70.2(c)):
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Lh . Additional observations,if necessary:
`
`
`
`
`
`
`li. Priority
`
`
`
`
`1.
`
`
`
`[1 This report has been established asif no priority had been claimed due to the failure to fumish within the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`prescribedtimelimit the requested:
`
`
`
`
`
`
`
`
`copy of the earlier application whose priority has been claimed.
`
`
`
`
`
`
`
`
`
`
`
`
`[]
`
`translation of the earlier application whosepriority has been claimed.
`
`
`
`
`
`
`
`
`
`
`
`
`Form PCTAPEA/409 (BoxesI-VIII, Sheet 1) (January 1994)
`
`
`
`
`
`
`
`
`
`
`
`
`Petitioner LG Ex-1023, 0002
`
`

`

`INTERNATIONAL PRELIMINARY
`
`EXAMINATION REPORT
`
`
`
`
`
`
`Intemational application No. PCT/EP99/00300
`
`
`
`
`
`
`2. 0 This report has been established as if no priority had been claimed due to the fact that the priority claim has
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`been found invalid.
`
`
`
`Thus for the purposesofthis report. the internationalfiling date indicated above is considered to be the relevant date.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`3. Additional observations, if necessary:
`
`
`
`
`
`
`see separate sheet
`
`
`
`
`
`V. Reasoned statement under Article 35(2) with regard to novelty, inventive step or industrial
`
`
`
`
`
`
`
`
`
`
`
`
`
`applicability; citations and explanations supporting such statement
`
`
`
`
`
`
`
`
`
`
`
`Yes:
`No:
`
`
`Yes:
`No:
`
`
`Yes:
`No:
`
`
`
`
`
`
`
`
`
`Claims
`Claims
`
`Claims
`Claims
`
`Claims
`Claims
`
`
`
`
`
`
`
`
`
`
`1-39
`
`1-39
`
`1-39
`
`
`
`
`
`
`
`1. Statement
`
`
`
`
`Novelty (N)
`
`
`
`
`Inventive step (IS)
`
`
`
`
`
`Industrial applicability 1A}
`
`
`
`
`2. Citations and explanations
`
`
`
`
`
`
`see separate sheet
`
`
`
`
`
`Vi. Certain documentscited
`
`
`
`
`
`
`1. Certain published documents (Rule 70.10)
`
`
`
`
`
`
`
`
`and / or
`
`
`
`
`2. Non-written disclosures (Rule 70.9)
`
`
`
`
`
`
`
`see separate sheet
`
`
`
`
`
`VII. Certain defects in the international application
`
`
`
`
`
`
`
`
`
`The following defects in the form or contents of the international application have been noted:
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`see separate sheet
`
`
`
`
`
`
`
`Form PCT/IPEA/409 (BoxesI-Vill, Sheet 2) (January 1994)
`
`
`
`
`
`
`
`
`
`
`
`
`Petitioner LG Ex-1023, 0003
`
`

`

`
`
`INTERNATIONAL PRELIMINARY
`
`EXAMINATION REPORT
`
`
`
`
`
`
`Intemational application No. PCT/EP99/00300
`
`
`
`
`
`
`
`VU. Certain observations on the international application
`
`
`
`
`
`
`
`
`
`The following observations on the clarity of the claims, description. and drawings or on the question whether the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`claims are fully supported by the description, are made:
`
`
`
`
`
`
`
`
`
`
`
`
`
`see separate sheet
`
`
`
`
`
`Form PCTAPEA/409 (BoxesI-VIIl, Sheet 3) (January 1994)
`
`
`
`
`
`
`
`
`
`
`
`
`Petitioner LG Ex-1023, 0004
`
`

`

`
`
`Intemational application No. PCT/EP99/00300
`INTERNATIONAL PRELIMINARY
`
`
`
`
`
`
`EXAMINATION REPORT- SEPARATE SHEET
`
`
`
`
`
`
`
`1.
`
`
`
`The following documents will be referred to in this report;
`
`
`
`
`
`
`
`
`
`
`
`D1 WO 98/05002 [Carlus Magnus Ltd.] Published 05.02.1998; filed
`
`
`
`
`
`
`
`
`
`22.07.1997.
`
`[MITSUBISHI]
`[STEED]
`
`
`
`
`
`D2 DE-A-19715519
`
`
`
`D3 WO-A-97/01246
`
`
`
`
`
`
`The examiner is of the opinion thatthe priorities claimed are validly claimed.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Document D1 is publishedafterthe first claimed priority, P1 (FR 60 048) of the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`application but before the second claimedpriority, P2 (PCT EP 98/05383).
`
`
`
`
`
`
`
`
`
`
`
`
`
`As such D1 constitutes prior art within the sense of Rule 64.1 PCT with regard to
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`subject matter which is not covered by the priority document P1. The document
`
`
`
`
`
`
`
`
`
`
`
`
`D1 could constitute a national/regionalprior right as indicated in Rule 64.3 PCT,
`
`
`
`
`
`
`
`
`
`
`
`
`howeverthis is for information of the applicant only and is beyond the scopeof
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`the International Examination.
`
`
`
`
`
`
`
`
`
`
`
`2.
`
`
`In the following discussion, the patentability of the claims will be examined with
`
`
`
`
`
`
`
`
`
`
`
`
`
`regard to the requirements of the PCT. !n particular, the claims will be examined
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`for novelty, as defined in Art. 33(2) PCT, and for inventive step, as definedin Art.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`33(3) PCT. In addition other aspects, such asclarity requirements of Art. 6 PCT,
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`may be discussed as appropriate.
`
`
`
`
`
`
`
`2.1 Conciseness of the claims.
`
`
`
`
`
`
`Art. 6 PCT requires that the claims of the application are concise. This applies not
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`only to individual claims but the overall set of claims, see Guidelines PCT Section
`
`
`
`
`
`
`
`
`
`
`
`
`
`IV-HI-5.
`
`
`
`
`
`in the present case the examiner is of the opinion that the independent claims 1,
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`3,7,9,14,39 constitute an unreasonably large numberof independent claims
`
`
`
`
`
`
`
`
`
`
`
`which could better be claimed in the form of one or two claims per category with
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`the remaining features claimed as independentclaims.
`
`
`
`
`
`
`
`
`
`
`Form PCT/Separate Sheet/409 (Sheet 1) (EPO-April 1997)
`
`
`
`
`
`
`
`
`
`
`
`Petitioner LG Ex-1023, 0005
`
`

`

`Intemational application No. PCT/EP99/00300
`INTERNATIONAL PRELIMINARY
`
`
`
`
`
`
`EXAMINATION REPORT- SEPARATE SHEET
`
`
`
`
`
`
`
`
`2.2
`
`
`
`Independent Claims, Claims 1,3,7,9,14,39
`
`
`
`
`
`
`
`Claims 1,3,7,9,14,39 all differ from the prior art in that they include at least one
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`identification of a sub area of an image using an anthropomorphic model of one
`
`
`
`
`
`
`
`
`
`
`
`
`
`kind or another.
`
`
`
`
`
`
`The closestprior art to the application is considered to be that disclosed in D2.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Whilst D2 discloses for example atcol. 3, lines 19-33 to monitor an area of the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`face including the eyes, there is not disclosed in D2 to “identify” a sub area of an
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`image including the eyes or the corresponding methodof "selecting pixels" of the
`
`
`
`
`
`
`
`
`
`
`
`
`image having characteristics corresponding to a facial characteristic using an
`
`
`
`
`
`
`
`
`
`
`anthropomorphic model (claims 1,3) or one of the kinds of anthropomorphic
`
`
`
`
`
`
`
`
`
`
`models given in the other independent claims.
`
`
`
`
`
`
`
`
`
`
`
`
`The examiner is of the opinion that it the person skilled in the art, presented with
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`the teaching of D2 alone orin combination with other available prior art would not
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`arrive at the subject matter of "identifying" a sub area or "selecting pixels" based
`
`
`
`
`
`
`
`
`
`
`
`
`
`on the models given, rather it would appear from the teaching of D2 that the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`apparatus only looks at the eyes and then monitors movement. Whilst the blinking
`
`
`
`
`
`
`
`
`
`
`
`
`rate is compared to an anthropomorphic model in D2 this is for the purpose of
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`recognising drowsinessof the driver and notto identify a particular area of the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`face.
`
`
`
`
`
`
`
`
`2.3 The remaining dependent claims would appear to be allowable under Art. 33(3)
`
`
`
`
`
`
`
`
`
`
`
`
`
`PCTsince they refer to allowable claims.
`
`
`
`
`
`
`
`
`
`2.4 Clarity of the claims, Art. 6 PCT.
`
`
`
`
`
`
`
`
`
`
`In order to satisfy the requirements of Rule 6.2 B reference signs in the claims
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`should be placed in parenthesis. This applies to all references including "V", page
`
`
`
`
`
`
`
`
`
`
`
`52, first line.
`
`
`
`
`
`
`
`
`
`Conversely, the examiner notes that other expressions placed in brackets are
`
`
`
`
`
`
`
`
`
`
`
`likely to be interpreted as being referencesfalling under Rule 6.2B. In the case of
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`the expression "(328x) or (828y) etc.", appearing in claims 1,2,3,7 and 9 the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Form PCT/Separate Sheet/409 (Sheet2) (EPO-April 1997)
`
`
`
`
`
`
`
`
`
`
`
`
`Petitioner LG Ex-1023, 0006
`
`

`

`Intemational application No. PCT/EP99/00300
`INTERNATIONAL PRELIMINARY
`
`
`
`
`
`EXAMINATION REPORT- SEPARATE SHEET
`
`
`
`
`
`
`claims are rendered unclear. The claims should have been clarified to remove
`
`
`
`
`
`
`
`
`
`
`
`
`these terms from brackets.
`
`
`
`
`
`
`The terms "DP" and "CO"in claims 10 and 23 are not defined in the claims and
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`therefore render the scope of these claims unclear, Art. 6 PCT.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`3.
`
`
`In the following section, certain defects in the International Application will be
`
`
`
`
`
`
`
`
`
`
`
`
`noted.
`
`
`
`The description should have been brought into conformity with the claims placed
`
`
`
`
`
`
`
`
`
`
`
`on file in accordance with the Rule 51 a iii PCT.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Form PCT/Separate Sheet/409 (Sheet 3) (EPO-April 1997)
`
`
`
`
`
`
`
`
`
`
`
`Petitioner LG Ex-1023, 0007
`
`

`

`
`
`- 15-01-2000
`
`
`
`EP 009900300
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`: ese see”—*we” PETEP-OFQQ300..°2
`
`
`
`
`
`
`
`
`
`
`048 J PCT 361
`
`January 11, 2000
`
`
`
`
`
`
`METHOD AND APPARATUS FOR DETECTION OF DROWSINESS
`
`
`
`
`
`
`Inventors:
`
`
`
`
`Dr. Patrick Pirim
`
`
`Dr. Thomas Binford
`
`
`
`
`
`
`
`5
`
`
`
`1.
`
`
`
`
`
`
`BACKGROUND OF THE INVENTION
`
`
`
`Field of the Invention.
`
`
`
`
`The present invention relates generally to an image processing system, and
`
`
`
`
`
`
`
`
`
`
`
`more particularly to the use of a generic image processing system to detect
`
`
`
`
`
`
`
`
`
`
`
`drowsiness.
`
`
`1.Description ofthe Related Art.
`
`
`
`
`
`
`It is well known thata significant number of highway accidents result from
`
`
`
`
`
`
`
`
`
`
`
`drivers becoming drowsy or falling asleep, which results in many deaths and
`
`
`
`
`
`
`
`
`
`
`
`injuries. Drowsiness is also a problem in other fields, such as for airline pilots and
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`powerplant operators, in which great damage mayresult from failure to stay alert.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`A number of different physical criteria may be used to establish when a
`
`
`
`
`
`
`
`
`
`
`
`
`
`person is drowsy, including a change in the duration and interval of eye blinking.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Normally, the duration of blinking is about 100 to 200 ms when awake and about
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`500 to 800 ms when drowsy. The time interval between successive blinks is
`
`
`
`
`
`
`
`
`
`
`
`
`generally constant while awake, but varies within a relatively broad range when
`
`
`
`
`
`
`
`
`
`
`
`
`20=drowsy.
`
`
`Numerous devices have been proposed to detect drowsiness of drivers.
`
`
`
`
`
`
`
`
`
`Such devices are shown, for example, in U.S. Patent Nos. 5,841,354; 5,813,99;
`
`
`
`
`
`
`
`
`
`
`
`5,689,241; 5,684,461; 5,682,144: 5,469,143: 5,402,109: 5,353,013; 5,195,606;
`
`
`
`
`
`
`4,928,090; 4,555,697; 4,485,375; and 4,259,665. In general, these devices fall into
`
`
`
`
`
`
`
`
`
`
`three categories: i) devices that detect movement of the head of the driver, e.g.,
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`tilting; ii) devices that detect a physiological change in the driver, e.g., altered
`
`
`
`
`
`
`
`
`
`
`
`
`heartbeat or breathing, and iit) devices that detect a physical result of the driver
`
`
`
`
`
`
`
`
`
`
`
`
`
`falling asleep, e.g., a reduced grip on the steering wheel. None ofthese devicesis
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`believed to have met with commercial success.
`
`
`
`
`
`
`
`
`10
`
`
`
`15
`
`
`
`25
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`AMENDED SHEET
`
`
`
`
`
`
`Petitioner LG Ex-1023, 0008
`
`

`

`e
`
`- 15-01-2000
`
`
`
`oT)
`
`
`
`10
`
`
`
`15
`
`
`
`20
`
`
`
`EP 009900300
`
`een
`ee
`«
`oa
`
`
`
`
`cs
`ee
`co ee
`8
`
`
`
`
`
`
`° ee oe
`«
`@®
`e
`
`
`
`
`
`
`
`
`
`
`
`
`
`“es PCTLEP-9F/Q0300 oe”
`
`
`048 J PCT 361
`
`
`January 11, 2000
`
`
`
`The German patent application DE-19715515 and the corresponding
`
`
`
`
`
`
`
`
`French patent application FR-2.747.346 disclose an apparatus and a process of
`
`
`
`
`
`
`
`
`
`
`
`evaluation of drowsinesslevel of a driver using a video camera placed near the feet
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`of the driver and a processing unit for the camera image with a software detecting
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`the blinks of the eyes determining the time gap between the beginning and the end
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`ofthe blink. Moreparticularly, a unit 10 ofthe processor realizes:
`
`
`
`
`
`
`
`
`
`
`
`
`® a memorization of the video image and its treatment, so as to determine
`
`
`
`
`
`
`
`
`
`
`
`
`
`an area comprising the driver’s eyes,
`
`
`
`
`
`
`* the detection ofthe time gap between the closing of the driver eyelids and
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`‘their full opening and
`
`
`
`
`© a treatment in a memory 11 and a processor 22 in combination with unit
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`10 to calculate a ratio of slow blink apparition.
`
`
`
`
`
`
`
`
`The object of the international patent application published WO-97/01246
`
`
`
`
`
`
`
`
`
`is a security system comprising a video camera placed within the rear-view mirror
`
`
`
`
`
`
`
`
`
`
`
`of a car and a video screen remotely disposed for the analysis of what is happening
`
`
`
`
`
`
`
`
`
`
`
`
`
`in the car and aroundit, as well as of what happened dueto the recording of the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`output video signal of the camera. This is in fact a concealed camera (within the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`reaf-view mirror), so that it is imperceptible to vandals and thieves and which
`
`
`
`
`
`
`
`
`
`
`
`
`observes a large scope including the inside of the car and its surroundings, the
`
`
`
`
`
`
`
`
`
`
`
`
`
`record allowing one to know later what has happened in this scope (page 6, lines
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`13 to 19), this is not a detector whose effective angle is strictly limited to the car
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`driver face in order to detect its eventual drowsiness and to make him awake.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Commonly-owned PCT Application Serial Nos. PCT/FR97/01354 and
`
`
`
`
`
`
`PCT/EP98/05383 disclose a generic image processing system that operates to
`
`
`
`
`
`
`
`
`
`
`
`25—localize
`
`
`
`e
`
`
`
`ce 8
`e
`e
`
`
`
`
`
`°
`ta
`
`
`.
`eeee
`
`
`
`“3
`tee
`
`
`°
`2
`
`
`
`
`eee son
`
`
`
`&
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`AMENDED SHEET
`
`
`
`
`
`
`Petitioner LG Ex-1023, 0009
`
`

`

`
`
`- 15-01-2000
`
`
`
`EP 009900300
`
`
`
`
`we
`3*¢*
`a ees
`ee es
`Se ee
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`27s gee bs feb ts
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`eo 49
`eee #00
`ee
`"PCTEP-98/00300"
`e
`
`
`
`
`
`
`048 JPCT 361
`
`
`January 11, 2000
`
`
`
`
`
`
`CLAIMS
`
`
`
`1. A process of detecting a person falling asleep, the process comprising
`
`
`
`
`
`
`
`
`
`
`the steps of:
`
`
`
`
`
`
`
`
`5
`
`
`
`10
`
`
`15
`
`
`
`acquiring an image ofthe face (V) ofthe person;
`
`
`
`
`
`
`
`
`
`
`selecting pixels ofthe image having characteristics corresponding to
`
`
`
`
`
`
`
`
`characteristics of at least one eye ofthe person;
`
`
`
`
`
`
`
`
`forming at least one histogram (328x) ofthe selected pixels;
`
`
`
`
`
`
`
`
`
`
`analyzing the at least one histogram (328x) over time to identify
`
`
`
`
`
`
`
`
`
`
`each opening and closing ofthe eye; and
`
`
`
`
`
`
`
`
`determining from the opening and closing information on the eye,
`
`
`
`
`
`
`
`
`
`characteristics indicative of a person falling asleep;
`
`
`
`
`
`
`identifying a sub-area of the image comprising the at least one eye
`
`
`
`
`
`
`
`
`
`
`
`Prior
`to the step of selecting pixels of the image having characteristics
`
`
`
`
`
`
`
`
`
`
`
`corresponding to characteristics of at least one eye, this step of selecting pixels
`
`
`
`
`
`
`
`
`
`
`
`
`comprising selecting pixels within the sub-area of the image, comprising the steps
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`of:
`
`
`
`identifying the head ofthe person in the image; and
`
`
`
`
`
`
`
`
`
`
`identifying the sub-area of the image using an anthropomorphic
`
`
`
`
`
`
`
`
`
`20=model.
`
`
`
`25
`
`
`
`2. The process according to claim 1 wherein the step ofidentifying head
`
`
`
`
`
`
`
`
`
`
`
`
`ofthe person in the image comprisesthe steps of:
`
`
`
`
`
`
`
`
`
`
`selecting pixels of the image having characteristics corresponding to
`
`
`
`
`
`
`
`
`edges of the head ofthe person;
`
`
`
`
`
`
`
`forming histograms (328x, 328y) of the selected pixels projected
`
`
`
`
`
`
`
`
`onto orthogonal axes; and
`
`
`
`
`analyzing the histograms of the selected pixels to identify the edges
`
`
`
`
`
`
`
`
`
`
`of the head of the person.
`
`
`
`
`
`
`3. The process of detecting a person falling asleep,
`
`
`
`
`
`
`
`
`
`the process
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`AMENDED SHEET
`
`
`
`
`
`
`Petitioner LG Ex-1023, 0010
`
`

`

`15-01-2000
`
`
`
`
`
`EP 009900300
`
`
`
`
`.
`e
`s
`e
`ee
`ee
`ae
`eevee
`a
`e
`ee
`ee
`es
`ea
`eo
`ees ee
`e
`
`
`
`
`
`
`
`
`
`
`
`
`e
`e
`e
`e
`e
`ee
`e
`ees
`e ee
`
`
`
`
`
`
`
`
`
`e
`e
`®
`e
`e
`es
`e
`e
`e
`e
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`e
`a
`e
`e
`e
`ee
`e
`e
`s
`e
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`. **—PUT/EP-99/00300 *“50 oF eee
`
`
`
`
`
`
`048 JPCT 361
`
`
`
`January 11, 2000
`
`
`
`
`e
`
`s
`
`5
`
`
`
`10
`
`
`
`15
`
`
`
`20
`
`
`
`25
`
`
`comprising the steps of:
`
`
`
`
`acquiring an image ofthe face V ofthe person;
`
`
`
`
`
`
`
`
`
`
`selecting pixels ofthe image having characteristics corresponding to
`
`
`
`
`
`
`
`
`characteristics of at least one eye ofthe person;
`
`
`
`
`
`
`
`
`formingat least one histogram (328x) of the selected pixels;
`
`
`
`
`
`
`
`
`
`
`analyzing the at least one histogram over time to identify each
`
`
`
`
`
`
`
`
`
`
`opening and closing of the eye; and
`
`
`
`
`
`
`
`determining from the opening and closing information on the eye,
`
`
`
`
`
`
`
`
`
`characteristics indicative of a person falling asleep;
`
`
`
`
`
`
`identifying a sub-area of the image comprising the at least one eye
`
`
`
`
`
`
`
`
`
`
`
`
`prior
`to the step of selecting pixels of the image having characteristics
`
`
`
`
`
`
`
`
`
`
`
`
`corresponding to characteristics of at least one eye, and wherein the step of
`
`
`
`
`
`
`
`
`
`
`
`
`selecting pixels ofthe image having characteristics corresponding to characteristics
`
`
`
`
`
`
`
`
`
`ofat least one eye comprises selecting pixels within the sub-areaofthe image;
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`identifying the location of a facial characteristic of the person in the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`image; and
`
`
`
`identifying the sub-area of the image using an anthropomorphic
`
`
`
`
`
`
`
`
`
`model andthe location ofthe facial characteristic.
`
`
`
`
`
`
`
`
`4. The process according to claim 3 wherein the step of identifying the
`
`
`
`
`
`
`
`
`
`
`
`
`location of a facial characteristic ofthe person in the image comprisesthe steps of:
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`selecting pixels ofthe image having characteristics corresponding to
`
`
`
`
`
`
`
`
`
`
`
`
`
`the facial characteristic;
`
`
`
`
`forming histogramsofthe selected pixels projected onto orthogonal
`
`
`
`
`
`
`
`
`
`
`
`axes; and
`
`
`
`analyzing the histograms of the selected pixels to identify the
`
`
`
`
`
`
`
`
`
`position of the facial characteristic in the image.
`
`
`
`
`
`
`
`
`5. The process according to claim 4 wherein the facial characteristic is
`
`
`
`
`
`
`
`
`
`
`
`
`the nostrils of the person, and wherein the step of selecting pixels of the image
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`having characteristics corresponding to the facial characteristic comprises selecting
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`AMENDED SHEET
`
`
`
`
`
`
`Petitioner LG Ex-1023, 0011
`
`

`

`
`
`- 15-01-2000
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`esse
`
`
`
`e
`
`
`
`
`
`
`.
`
`
`
`
`
`
`
`
`
`e
`
`
`
`
`
`
`
`
`es
`s
`
`
`
`
`
`e
`ee
`
`
`
`
`
`
`
`
`
`
`* PUT/EP-99/00300 *
`
`
`048 J PCT 361
`
`
`January 11, 2000
`
`
`
`
`EP 009900300
`
`
`
`
`
`eoee
`
`
`
`
`
`
`5
`
`
`
`pixels having low luminancelevels.
`
`
`
`
`
`6. The process according to claim 5 further comprising the step of
`
`
`
`
`
`
`
`
`
`
`
`analyzing the histograms of the nostril pixels to determine whether the spacing
`
`
`
`
`
`
`
`
`
`
`
`between the nostrils is within a desired range and whether the dimensions of the
`
`
`
`
`
`
`
`
`
`
`
`
`
`nostrils fall within a desired range.
`
`
`
`
`
`7. The process of detecting a person falling asleep,
`
`
`
`
`
`
`
`
`
`comprising the stepsof:
`
`
`
`
`acquiring an imageofthe face (V) ofthe person;
`
`
`
`
`
`
`
`
`
`
`selecting pixels ofthe image having characteristics corresponding to
`
`
`
`
`
`
`
`
`
`10_characteristics of at least one eye ofthe person;
`
`
`
`
`
`
`
`
`
`
`forming at least one histogram (328x) of the selected pixels;
`
`
`
`
`
`
`
`
`
`
`analyzing the at least one histogram over time to identify each
`
`
`
`
`
`
`
`
`
`
`opening and closing of the eye; and
`
`
`
`
`
`
`
`determining from the opening and closing information on the eye,
`
`
`
`
`
`
`
`
`
`
`15_characteristics indicative of a person falling asleep;
`
`
`
`
`
`
`
`the step of selecting pixels of the image having characteristics
`
`
`
`
`
`
`
`
`
`corresponding to characteristics of at
`least one eye of the person comprises
`
`
`
`
`
`
`
`
`
`
`
`selecting pixels having low luminance levels corresponding to shadowing of the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`the process
`
`
`
`
`
`
`
`
`
`
`
`
`eye; and
`
`
`
`20
`
`
`
`25
`
`
`wherein the step analyzing the at least one histogram over time to
`
`
`
`
`
`
`
`
`
`
`
`identify each opening and closing of the eye comprises analyzing the shape of the
`
`
`
`
`
`
`
`
`
`
`
`
`
`eye shadowing to determine openingsandclosings ofthe eye.
`
`
`
`
`
`
`
`
`
`
`8. The process according to claim 7 wherein the step of forming at least
`
`
`
`
`
`
`
`
`
`
`
`
`
`one histogram of the selected pixels comprises forming histograms of shadowed
`
`
`
`
`
`
`
`
`
`
`pixels ofthe eye projected onto orthogonal axes, and wherein the step of analyzing
`
`
`
`
`
`
`
`
`
`
`
`
`
`the shape of the eye shadowing comprises analyzing the width and height of the
`
`
`
`
`
`
`
`
`
`
`
`
`
`shadowing.
`
`9. The process of detecting a person falling asleep,
`
`
`
`
`
`
`
`
`
`comprising the steps of:
`
`
`
`
`
`the process
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`AMENDED SHEET
`
`
`
`
`
`
`Petitioner LG Ex-1023, 0012
`
`

`

`. 15-01-2000
`
`
`
`
`s
`es
`
`
`tas
`
`
`o
`e
`
`
`
`“
`s
`
`
`
`.
`252
`
`
`
`
`
`
`
`
`
`
`
`©
`
`
`
`e
`eoae
`®
`ae
`e's ee
`
`
`
`“3
`:
`° ooo
`°
`ee
`°
`°
`
`
`
`
`
`
`es
`e
`oe eo
`e
`88
`©
`8s
`
`
`
`
`
`
`
`:
`ee e
`:
`eo 8 °
`
`
`
`
`
`
`
`
`
`
`
`
`
`200 cee
`oe
`POT/EP-99/90306 ©
`
`
`048 JPCT 361
`
`
`January 11, 2000
`
`
`
`
`EP 009900300
`
`
`
`
`
`
`
`
`
`
`
`5
`
`
`
`10
`
`
`
`15
`
`
`20
`
`
`
`25
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`acquiring an image ofthe face V of the person;
`
`
`
`
`
`
`
`
`
`
`selecting pixels ofthe image having characteristics corresponding to
`
`
`
`
`
`
`
`
`characteristics of at least one eye of the person:
`.
`
`
`
`
`
`
`
`
`formingat least one histogram (328x) ofthe selected pixels;
`
`
`
`
`
`
`
`
`
`
`analyzing the at least one histogram over time to identify each
`
`
`
`
`
`
`
`
`
`
`opening and closing of the eye; and
`
`
`
`
`
`
`
`determining from the opening and closing information on the eye,
`
`
`
`
`
`
`
`
`
`characteristics indicative of a person falling asleep;
`
`
`
`
`
`
`the step of selecting pixels of the image having characteristics
`
`
`
`
`
`
`
`
`
`corresponding to characteristics of at
`least one eye of the person comprises
`
`
`
`
`
`
`
`
`
`
`
`selecting pixels in movement correspondingto blinking; and
`
`
`
`
`
`
`
`
`wherein the step analyzing the at least one histogram over time to
`
`
`
`
`
`
`
`
`
`
`
`
`identify each opening and closing of the eye comprises analyzing the number of
`
`
`
`
`
`
`
`
`
`
`
`
`pixels in movement over time to determine openings and closingsofthe eye.
`
`
`
`
`
`
`
`
`
`
`
`
`
`10. The process according to claim 9 wherein the step ofselecting pixels
`
`
`
`
`
`
`
`
`
`
`
`
`of the image having characteristics corresponding to characteristics of at least one
`
`
`
`
`
`
`
`
`
`
`eye of the person comprises selecting having characteristics selected from the
`
`
`
`
`
`
`
`
`
`
`group consisting of i) DP=1, ii) CO indicative of a blinking eyelid,
`iii) velocity
`
`
`
`
`
`
`
`
`
`
`
`indicative of a blinking eyelid, and iv) up and down movementindicative of a
`
`
`
`
`
`
`
`
`
`
`
`
`blinking eyelid.
`
`
`11. The process according to claim 3 wherein the step of identifying a
`
`
`
`
`
`
`
`
`
`
`
`
`facial characteristic of the person in the image comprises the step of searching sub-
`
`
`
`
`
`
`
`
`
`
`
`
`
`images of the image to identify the facial characteristic.
`
`
`
`
`
`
`
`
`
`12. The process according to claim 5 wherein the step of identifying a
`
`
`
`
`
`
`
`
`
`
`
`
`facial characteristic of the person in the image comprises the step of searching sub-
`
`
`
`
`
`
`
`
`
`
`
`
`
`images of the imageto identify the nostrils.
`
`
`
`
`
`
`
`
`13. The process according to claim 11 wherein the facial characteristic is
`
`
`
`
`
`
`
`
`
`
`
`a first facial characteristic and further comprising the stepsof:
`
`
`
`
`
`
`
`
`
`
`using an anthropomorphic model and the location of the first facial
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`AMENDED SHEET
`
`
`
`
`
`
`Petitioner LG Ex-1023, 0013
`
`

`

`
`
`.
`ee
`ee
`ens
`e @e
`
`.
`ee
`°*
`
`
`
`
`es
`@e0
`@©
`
`
`
`e
`ee
`es
`
`
`
`
`
`
`
`SCT/EP-99/80308 «
`
`048 JPCT 361
`
`January 11, 2000
`
`
`
`characteristic to select a sub-area of the image containing a second facial
`
`
`
`
`
`
`
`
`
`
`
`characteristic;
`
`e
`a
`ae ae
`
`
`ees
`
`
`®
`s
`
`
`s
`e
`
`
`
`
`° 53
`e
`
`
`
`
`© ees
`we
`e.
`
`e
`see
`
`
`e
`
`.
`
`
`eee eee
`
`
`as
`s
`e
`
`
`e
`he
`
`
`oes 8
`
`
`
`°
`-#
`*
`
`
`
`
`
`
`
`os
`
`
`
`8
`
`
`
`EP 009900300
`
`e
`
`e
`@
`8
`
`
`
`
`
`
`
`selecting pixels of the image having characteristics corresponding to _
`
`
`
`
`
`
`
`
`
`the second facial characteristic; and
`
`
`
`
`
`analyzing the histograms of the selected pixels of the second facial
`
`
`
`
`
`
`
`
`
`
`
`characteristic to confirm the identification ofthe first facial characteristic.
`
`
`
`
`
`
`
`
`
`14. An apparatus for detecting a person falling asleep,
`
`
`
`
`
`
`
`
`comprising:
`
`
`the apparatus
`
`
`
`
`
`
`
`
`a sensor for acquiring an image of the face of the person, the image
`
`
`
`
`
`
`
`
`
`
`
`
`
`comprising pixels correspondingto the eye of the person;
`
`
`
`
`
`
`
`
`a controller; and
`
`
`
`a histogram formation unit for forming a histogram on pixels having
`
`
`
`
`
`
`
`
`
`selected characteristics,
`
`
`
`
`
`
`
`
`- 15-01-2000
`
`
`
`5
`
`
`
`10
`
`
`
`20
`
`
`25
`
`
`
`the controller controlling the histogram formation unit to select
`
`
`
`
`
`
`
`
`
`15_pixels of the image having characteristics corresponding to characteristics of at
`
`
`
`
`
`
`
`
`
`
`
`
`least one eye of the person and to form a histogram of the selected pixels, the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`controller analyzing the histogram overtime to identify each opening and closing
`
`
`
`
`
`
`
`
`
`
`
`of the eye, and determining from the opening and closing information on the eye,
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`characteristics indicative of a person falling asleep;
`
`
`
`
`
`
`the controller interacting with the histogram formation unit to
`
`
`
`
`
`
`
`
`identify a sub-area of the image comprising the at least one eye, and the controller
`
`
`
`
`
`
`
`
`
`
`
`
`
`controls the histogram formation unit
`to select pixels of the image having
`
`
`
`
`
`
`
`
`
`
`
`characteristics corresponding to characteristics of at least one eye only within the
`
`
`
`
`
`
`
`
`
`
`
`sub-area of the image;
`
`
`
`
`the controller interacting with the histogram formation unit to
`
`
`
`
`
`
`
`
`identify the head of the person in the image; and
`
`
`
`
`
`
`
`
`
`
`the controller
`identifies the sub-area of the image using an
`
`
`
`
`
`
`
`
`anthropomorphic model.
`
`
`15. The apparatus according to claim 14 wherein:
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`AMENDED SHEET
`
`
`
`
`
`
`Petitioner LG Ex-1023, 0014
`
`

`

`a —
`
`- 15-01-2000
`
`
`
`e
`
`
`
`
`
`254
`
`
`
`
`
`e
`
`:
`
`
`
`
`
`
`
`
`EP 009900300
`
`
`
`
`e
`eese
`
`
`
`
`
`
`
`
`wes vee.
`
`
`
`
`
`a
`
`oe
`ee
`os
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`* PTIEFSI9/00309 , ° “ae
`
`
`
`
`048 J PCT 361
`
`
`January 11, 2000
`
`
`
`
`
`
`5
`
`
`
`10
`
`
`
`15
`
`
`20
`
`
`
`25
`
`
`
`the histogram formation unit selects pixels of the image having
`
`
`
`
`
`
`
`
`
`characteristics corresponding to edges of the head of the person and forms
`
`
`
`
`
`
`
`
`
`
`
`histogramsofthe selected pixels projected onto orthogonal axes; and
`
`
`
`
`
`
`
`
`
`
`the controller analyzes the histograms of the selected pixels to
`
`
`
`
`
`
`
`
`
`identify the edges of the head of the person.
`
`
`
`
`
`
`
`
`
`16. The apparatus accordingto claim 14 wherein:
`
`
`
`
`
`
`
`
`the controller interacts with the histogram formation unit to identify
`
`
`
`
`
`
`
`
`
`the location of a facial characteristic of the person in the image; and
`
`
`
`
`
`
`
`
`
`
`
`
`the controller
`identifies the sub-area of the image using an
`
`
`
`
`
`
`
`
`
`anthropomorphic model and thelocation of the facial characte