throbber
PCT
`INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT)
`WO 99/36893
`
`WORLD INTELLECTUAL PROPERTY ORGANIZATION
`International Bureau
`
`(51) International Patent Classification 6 :
`G0SB 21/00
`
`Al
`
`(11) International Publication Number:
`
`(43) International Publication Date:
`
`22 July 1999 (22.07.99)
`
`(21) International Application Number:
`
`PCT/EP99/00300
`
`(22) International Filing Date:
`
`15 January 1999 (15.01.99)
`
`(30) Priority Data:
`98/00378
`PCT/EP98/05383
`
`15 January 1998 (15.01.98)
`25 August 1998 (25.08.98)
`
`FR
`EP
`
`(63) Related by Continuation (CON) or Continuation-in-Part
`(CIP) to Earlier Application
`us
`Filed on
`
`PCT/EP98/05383 (CIP)
`25 August 1998 (25.08.98)
`
`(71) Applicant (for all designated States except US): HOLDING
`B.E.V. S.A. [LU/LU]; 69, route d'Esch, L-Luxembourg
`(LU).
`
`(71)(72) Applicants and Inventors:
`[FR/FR];
`PIRIM, Patrick
`56, rue Patay, F-75013 Paris (FR). BINFORD, Thomas
`[US/US]; 16012 Flintlock Road, Cupertino, CA 95014
`(US).
`
`(74) Agent: PHELIP, Bruno; Cabinet Harle & Phelip, 7, rue de
`Madrid, F-75008 Paris (FR).
`
`(81) Designated States: AL, AM, AT, AU, AZ, BA, BB, BG, BR,
`BY, CA, CH, CN, CU, CZ, DE, DK, EE, ES, Fl, GB, GD,
`GE, GH, GM, HR, HU, ID, IL, IN, IS, JP, KE, KG, KP,
`KR, KZ, LC, LK, LR, LS, LT, LU, LV, MD, MG, MK,
`MN, MW, MX, NO, NZ, PL, PT, RO, RU, SD, SE, SG,
`SI, SK, SL, TJ, TM, TR, TT, UA, UG, US, UZ, VN, YU,
`ZW, ARIPO patent (GH, GM, KE, LS, MW, SD, SZ, UG,
`ZW), Eurasian patent (AM, AZ, BY, KG, KZ, MD, RU, TJ,
`TM), European patent (AT, BE, CH, CY, DE, DK, ES, FI,
`FR, GB, GR, IE, IT, LU, MC, NL, PT, SE), OAPI patent
`(BF, BJ, CF, CG, CI, CM, GA, GN, GW, ML, MR, NE,
`SN, TD, TG).
`
`Published
`With international search report.
`Before the expiration of the time limit for amending the
`claims and to be republished in the event of the receipt of
`amendments.
`
`(54) Title: METHOD AND APPARATUS FOR DETECTION OF DROWSINESS
`
`(57) Abstract
`
`In a process of detecting a person falling asleep, an image of the face of the person is acquired. Pixels of the image having
`characteristics corresponding to an eye of the person are selected and a histogram is formed of the selected pixels. The histogram is
`analyzed over time to identify each opening and closing of the eye, and characteristics indicative of the person falling asleep are determined.
`A sub-area of the image including the eye may be determined by identifying the head or a facial characteristic of the person, and then
`identifying the sub-area using an anthropomorphic model. To determine openings and closings of the eyes, histograms of shadowed pixels
`of the eye are analyzed to determine the width and height of the shadowing, or histograms of movement corresponding to blinking are
`analyzed. An apparatus for detecting a person falling asleep includes a sensor for acquiring an image of the face of the person, a controller,
`and a histogram formation unit for forming a histogram on pixels having selected characteristics. Also disclosed is a rear-view mirror
`assembly incorporating the apparatus.
`
`Petitioner LG Ex-1018, 0001
`
`

`

`FOR THE PURPOSES OF INFORMATION ONLY
`
`Codes used to identify States party to the PCT on the front pages of pamphlets publishing international applications under the PCT.
`
`AL
`AM
`AT
`AU
`AZ
`BA
`BB
`BE
`BF
`BG
`BJ
`BR
`BY
`CA
`CF
`CG
`CH
`CI
`CM
`CN
`cu
`CZ
`DE
`DK
`EE
`
`Albania
`Annenia
`Austria
`Australia
`Azerbaijan
`Bosnia and Herzegovina
`Barbados
`Belgium
`Burkina Faso
`Bulgaria
`Benin
`Brazil
`Belarus
`Canada
`Central African Republic
`Congo
`Switzerland
`C1\te d'Ivoire
`Cameroon
`China
`Cuba
`Czech Republic
`Gennany
`Denmark
`Estonia
`
`ES
`FI
`FR
`GA
`GB
`GE
`GH
`GN
`GR
`HU
`IE
`IL
`IS
`IT
`JP
`KE
`KG
`KP
`
`KR
`KZ
`LC
`LI
`LK
`LR
`
`Spain
`Finland
`France
`Gabon
`United Kingdom
`Georgia
`Ghana
`Guinea
`Greece
`Hungary
`Ireland
`Israel
`Iceland
`Italy
`Japan
`Kenya
`Kyrgyzstan
`Democratic People's
`Republic of Korea
`Republic of Korea
`Kazakstan
`Saint Lucia
`Liechtenstein
`Sri Lanka
`Liberia
`
`LS
`LT
`LU
`LV
`MC
`MD
`MG
`MK
`
`ML
`MN
`MR
`MW
`MX
`NE
`NL
`NO
`NZ
`PL
`PT
`RO
`RU
`SD
`SE
`SG
`
`Lesotho
`Lithuania
`Luxembourg
`Latvia
`Monaco
`Republic of Moldova
`Madagascar
`The fonner Yugoslav
`Republic of Macedonia
`Mali
`Mongolia
`Mauritania
`Malawi
`Mexico
`Niger
`Netherlands
`Norway
`New Zealand
`Poland
`Portugal
`Romania
`Russian Federation
`Sudan
`Sweden
`Singapore
`
`SI
`SK
`SN
`sz
`TD
`TG
`TJ
`TM
`TR
`TT
`UA
`UG
`us
`uz
`VN
`YU
`zw
`
`Slovenia
`Slovakia
`Senegal
`Swaziland
`Chad
`Togo
`Tajikistan
`Turkmenistan
`Turkey
`Trinidad and Tobago
`Ukraine
`Uganda
`United States of America
`Uzbekistan
`Viet Nam
`Yugoslavia
`Zimbabwe
`
`Petitioner LG Ex-1018, 0002
`
`

`

`WO99/36893
`
`PCT /EP99/00300
`
`METHOD AND APPARATUS FOR DETECTION OF DROWSINESS
`
`Binford
`
`BACKGROUND OF THE INVENTION
`
`1.
`
`Field of the Invention.
`
`The present invention relates generally to an image processing system, and
`
`more particularly to the use of a generic image processing system to detect drowsiness.
`
`2.
`
`Description of the Related Art.
`
`It is well known that a significant number of highway accidents result from
`
`drivers becoming drowsy or falling asleep, which results in many deaths and injuries.
`
`Drowsiness is also a problem in other fields, such as for airline pilots and power plant
`
`operators, in which great damage may result from failure to stay alert.
`
`A number of different physical criteria may be used to establish when a person
`
`is drowsy, including a change in the duration and interval of eye blinking. Normally, the
`
`duration of blinking is about 100 to 200 ms when awake and about 500 to 800 ms when
`
`drowsy. The time interval between successive blinks is generally constant while awake, but
`
`varies within a relatively broad range when drowsy.
`
`Numerous devices have been proposed to detect drowsiness of drivers. Such
`
`devices are shown, for example, in U.S. Patent Nos. 5,841,354; 5,813,99;
`
`5,689,241;5,684,461; 5,682,144; 5,469,143; 5,402,109; 5,353,013; 5,195,606; 4,928,090;
`
`4,555,697; 4,485,375; and 4,259,665. In general, these devices fall into three categories: i)
`
`devices that detect movement of the head of the driver, e.g., tilting; ii) devices that detect a
`
`physiological change in the driver, e.g., altered heartbeat or breathing, and iii) devices that
`
`CONFIRMATION COPY
`
`Petitioner LG Ex-1018, 0003
`
`

`

`WO 99/36893
`
`PCT /EP99/00300
`
`detect a physical result of the driver falling asleep, e.g., a reduced grip on the steering wheel.
`
`None of these devices is believed to have met with commercial success.
`
`Commonly-owned PCT Application Serial Nos. PCT/FR97/01354 and
`
`PCT/EP98/05383 disclose a generic image processing system that operates to localize objects
`
`in relative movement in an image and to determine the speed and direction of the objects in
`
`real-time. Each pixel of an image is smoothed using its own time constant. A binary value
`
`corresponding to the existence of a significant variation in the amplitude of the smoothed
`
`pixel from the prior frame, and the amplitude of the variation, are determined, and the time
`
`constant for the pixel is updated. For each particular pixel, two matrices are formed that
`
`include a subset of the pixels spatially related to the particular pixel. The first matrix contains
`
`the binary values of the subset of pixels. The second matrix contains the amplitude of the
`
`variation of the subset of pixels. In the first matrix, it is determined whether the pixels along
`
`an oriented direction relative to the particular pixel have binary values representative of
`
`significant variation, and, for such pixels, it is determined in the second matrix whether the
`
`amplitude of these pixels varies in a known manner indicating movement in the oriented
`
`directior.. In domains that include luminance, hue, saturation, speed, oriented direction, time
`
`constant, and x and y position, a histogram is formed of the values in the first and second
`
`matrices falling in user selected combinations of such domains. Using the histograms, it is
`
`determined whether there is an area having the characteristics of the selected combinations of
`
`domains.
`
`It would be desirable to apply such a generic image processing system to
`
`detect the
`
`drowsiness of a person.
`
`2
`
`Petitioner LG Ex-1018, 0004
`
`

`

`WO99/36893
`
`PCT /EP99/00300
`
`SUMMARY OF THE INVENTION
`
`The present invention is a process of detecting a driver falling asleep in which
`
`an image of the face of the driver is acquired. Pixels of the image having characteristics
`
`corresponding to characteristics.or'at least one eye of the driver are selected and a histogram
`
`is formed of the selected pixels. The histogram is analyzed over time to identify each
`
`opening and closing of the eye, and from the eye opening and closing information,
`
`characteristics indicative of a driver falling asleep are determined.
`
`In one embodiment, a sub-area of the image comprising the eye is determined
`
`prior to the step of selecting pixels of the image having characteristics corresponding to
`
`characteristics of an eye. In this embodiment, the step of selecting pixels of the image having
`
`characteristics of an eye involves selecting pixels within the sub-area of the image. The step
`
`of identifying a sub-area of the image preferably involves identifying the head of the driver,
`
`or a facial characteristic of the driver, such as the driver's nostrils, and then identifying the
`
`sub-area of the image using an anthropomorphic model. The head of the driver may be
`
`identified by selecting pixels of the image having characteristics corresponding to edges of
`
`the head of the driver. Histograms of the selected pixels of the edges of the driver's head are
`
`projected onto orthogonal axes. These histograms are then analyzed to identify the edges of
`
`the driver's head.
`
`The facial characteristic of the driver may be identified by selecting pixels of
`
`the image having characteristics corresponding to the facial characteristic. Histograms of the
`
`selected pixels of the facial characteristic are projected onto orthogonal axes. These
`
`histograms are then analyzed to identify the facial characteristic. If desired, the step of
`
`identifying the facial characteristic in the image involves searching sub-images of the image
`
`until the facial characteristic is found. In the case in which the facial characteristic is the
`
`3
`
`Petitioner LG Ex-1018, 0005
`
`

`

`WO 99/36893
`
`PCT /EP99/00300
`
`nostrils of the driver, a histogram is formed of pixels having low luminance levels to detect
`
`the nostrils. To confirm detection of the nostrils, the histograms of the nostril pixels may be
`
`analyzed to determine whether the spacing between the nostrils is within a desired range and
`
`whether the dimensions of the nostrils fall within a desired range. In order to confirm the
`
`identification of the facial characteristic, an anthropomorphic model and the location of the
`
`facial characteristic are used to select a sub-area of the image containing a second facial
`
`characteristic. Pixels of the image having characteristics corresponding to the second facial
`
`characteristic are selected and a histograms of the selected pixels of the second facial
`
`characteristic are analyzed to confirm the identification of the first facial characteristic.
`
`In order to determine openings and closings of the eyes of the driver, the step
`
`of selecting pixels of the image having characteristics corresponding to characteristics of an
`
`eye of the driver involves selecting pixels having low luminance levels corresponding to
`
`shadowing of the eye. In this embodiment, the step analyzing the histogram over time to
`
`identify each opening and closing of the eye involves analyzing the shape of the eye
`
`shadowing to determine openings and closings of the eye. The histograms of shadowed
`
`pixels are preferably projected onto orthogonal axes, and the step of analyzing the shape of
`
`the eye shado\ving involves analyzing the width and height of the shadowing.
`
`An alternative method of determining openings and closings of the eyes of the
`
`driver involves selecting pixels of the image having characteristics of movement
`
`corresponding to blinking. In this embodiment, the step analyzing the histogram over time to
`
`identify each opening and closing of the eye involves analyzing the number of pixels in
`
`movement corresponding to blinking over time. The characteristics of a blinking eye are
`
`preferably selected from the group consisting of i) DP= 1, ii) CO indicative of a blinking
`
`4
`
`Petitioner LG Ex-1018, 0006
`
`

`

`WO99/36893
`
`PCT /EP99/00300
`
`eyelid, iii) velocity indicative of a blinking eyelid, and iv) up and down movement indicative
`
`of a blinking eyelid.
`
`An apparatus for detecting a driver falling asleep includes a sensor for
`
`acquiring an image of the face af the driver, a controller, and a histogram formation unit for
`
`forming a histogram on pixels having selected characteristics. The controller controls the
`
`histogram formation unit to select pixels of the image having characteristics corresponding to
`
`characteristics of at least one eye of the driver and to form a histogram of the selected pixels.
`
`The controller analyzes the histogram over time to identify each opening and closing of the
`
`eye, and determines from the opening and closing information on the eye, characteristics
`
`indicative of the driver falling asleep.
`
`In one embodiment, the controller interacts with the histogram formation unit
`
`to identify a sub-area of the image comprising the eye, and the controller controls the
`
`histogram formation unit to select pixels of the image having characteristics corresponding to
`
`characteristics of the eye only within the sub-area of the image. In order to select the sub-area
`
`of the image, the controller interacts with the histogram formation unit to identify the head of
`
`the driver in the image, or a facial characteristic of the driver, such as the driver's nostrils.
`
`The controller then identifies the sub-area of the image using an anthropomorphic model. To
`
`identify the head of the driver, the histogram formation unit selects pixels of the image having
`
`characteristics corresponding to edges of the head of the driver and forms histograms of the
`
`selected pixels projected onto orthogonal axes. To identify a facial characteristic of the
`
`driver, the histogram formation unit selects pixels of the image having characteristics
`
`corresponding to the facial characteristic and forms histograms of the selected pixels
`
`projected onto orthogonal axes. The controller then analyzes the histograms of the selected
`
`pixels to identify the edges of the head of the driver or the facial characteristic, as the case
`
`5
`
`Petitioner LG Ex-1018, 0007
`
`

`

`WO 99/36893
`
`PCT /EP99/00300
`
`may be. If the facial characteristic is the nostrils of the driver, the histogram formation unit
`
`selects pixels of the image having low luminance levels corresponding to the luminance level
`
`of the nostrils. The controller may also analyze the histograms of the nostril pixels to
`
`determine whether the spacing between the nostrils is within a desired range and whether
`
`dimensions of the nostrils fall within a desired range. If desired, the controller may interact
`
`with the histogram formation unit to search sub-images of the image to identify the facial
`
`characteristic.
`
`In order to verify identification of the facial characteristic, the controller uses
`
`an anthropomorphic model and the location of the facial characteristic to cause the histogram
`
`formation unit to select a sub-area of the image containing a second facial characteristic. The
`
`histogram formation unit selects pixels of the image in the sub-area having characteristics
`
`corresponding to the second facial characteristic and forms a histogram of such pixels. The
`
`controller then analyzes the histogram of the selected pixels corresponding to the second
`
`facial characteristic to identify the second facial characteristic and to thereby confirm the
`
`identification of the first facial characteristic.
`
`In one embodiment, the histogram formation unit selects pixels of the image
`
`having low luminance levels corresponding to shadowing of the eyes, and the controller then
`
`analyzes the shape of the eye shadowing to identify shapes corresponding to openings and
`
`closings of the eye. The histogram formation unit preferably forms histograms of the
`
`shadowed pixels of the eye projected onto orthogonal axes, and the controller analyzes the
`
`width and height of the shadowing to determine openings and closings of the eye.
`
`In an alternative embodiment, the histogram formation unit selects pixels of
`
`the image in movement corresponding to blinking and the controller analyzes the number of
`
`pixels in movement over time to determine openings and closings of the eye. The
`
`6
`
`Petitioner LG Ex-1018, 0008
`
`

`

`WO 99/36893
`
`PCT /EP99/00300
`
`characteristics of movement corresponding to blinking are preferably selected from the group
`
`consisting of i) DP=l, ii) CO indicative of a blinking eyelid, iii) velocity indicative of a
`
`blinking eyelid, and iv) up and down movement indicative of a blinking eyelid.
`
`If desired, the sensor may be integrally constructed with the controller and the
`
`histogram formation unit. The apparatus may comprise an alarm, which the controller
`
`operates upon detection of the driver falling asleep, and may comprise an illumination source,
`
`such as a source of IR radiation, with the sensor being adapted to view the driver when
`
`illuminated by the illumination source.
`
`A rear-view minor assembly comprises a rear-view mirror and the described
`
`apparatus for detecting driver drowsiness mounted to the rear-view mirror. In one
`
`embodiment, a bracket attaches the apparatus to the rear-view mirror. In an alternative
`
`embodiment, the rear-view mirror comprises a housing having an open side and an interior.
`
`The rear-view mirror is mounted to the open side of the housing, and is see-through from the
`
`interior of the housing to the exterior of the housing. The drowsiness detection apparatus is
`
`mounted interior to the housing with the sensor directed toward the rear-view mirror. If
`
`desired, a joint attaches the apparatus to the rear-view mirror assembly, with the joint being
`
`adapted to maintain the apparatus in a position facing the driver during adjustment of the
`
`mirror assembly by the driver. The rear-view mirror assembly may include a source of
`
`illumination directed toward the driver, with the sensor adapted to view the driver when
`
`illuminated by the source of illumination. The rear-view mirror assembly may also include
`
`an alarm, with the controller operating the alarm upon detection of the driver falling asleep.
`
`Also disclosed is a vehicle comprising the drowsiness detection device.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`Fig. 1 is a diagrammatic illustration of the system according to the invention.
`
`7
`
`Petitioner LG Ex-1018, 0009
`
`

`

`WO 99/36893
`
`PCT /EP99/00300
`
`Fig. 2 is a block diagram of the temporal and spatial processing units of the
`
`invention.
`
`invention.
`
`Fig. 3 is a block diagram of the temporal processing unit of the invention.
`
`Fig. 4 is a block diagram of the spatial processing unit of the invention.
`
`Fig. 5 is a diagram showing the processing of pixels in accordance with the
`
`Fig. 6 illustrates the numerical values of the Freeman code used to determine
`
`movement direction in accordance with the invention.
`
`Fig. 7 illustrates nested matrices as processed by the temporal processing unit.
`
`Fig. 8 illustrates hexagonal matrices as processed by the temporal processing
`
`unit.
`
`unit.
`
`Fig. 9 illustrates reverse-L matrices as processed by the temporal processing
`
`Fig. 10 illustrates angular sector shaped matrices as processed by the temporal
`
`processing unit.
`
`Fig. 11 is a block diagram showing the relationship between the temporal and
`
`spatial processing units, and the histogram formation units.
`
`Fig. 12 is a block diagram showing the interrelationship between the various
`
`histogram formation units.
`
`Fig. 13 shows the formation of a two-dimensional histogram of a moving area
`
`from two one-dimensional histograms.
`
`Fig. 14 is a block diagram of an individual histogram formation unit.
`
`Figs. 15A and 15B illustrate the use of a histogram formation unit to find the
`
`orientation of a line relative to an analysis axis.
`
`8
`
`Petitioner LG Ex-1018, 0010
`
`

`

`WO 99/36893
`
`PCT /EP99/00300
`
`Fig. 16 illustrates a one-dimensional histogram.
`
`Fig. 17 illustrates the use of semi-graphic sub-matrices to selected desired
`
`areas of an image.
`
`Fig. 18 is a side vie'w illustrating a rear view mirror in combination with the
`
`drowsiness detection system of the invention.
`
`Fig. 19 is a top view illustrating operation of a rear view mirror.
`
`Fig. 20 is a schematic illustrating operation of a rear view mirror.
`
`Fig. 21 is a cross-sectional top view illustrating a rear view mirror assembly
`
`incorporating the drowsiness detection system of the invention.
`
`Fig. 22 is a partial cross-sectional top view illustrating a joint supporting the
`
`drowsiness detection system of the invention in the mirror assembly of Fig. 21.
`
`Fig. 23 is a top view illustrating the relationship between the rear view mirror
`
`assembly of Fig. 21 and a driver.
`
`Fig. 24 illustrates detection of the edges of the head of a person using the
`
`system of the invention.
`
`Fig. 25 illustrates masking outside of the edges of the head of a person.
`
`Fig. 26 illustrates masking outside of the eyes of a person.
`
`Fig. 27 illustrates detection of the eyes of a person using the system of the
`
`invention.
`
`Fig. 28 illustrates successive blinks in a three-dimensional orthogonal
`
`coordinate system.
`
`Figs. 29A and 29B illustrate conversion of peaks and valleys of eye movement
`
`histograms to information indicative of blinking.
`
`9
`
`Petitioner LG Ex-1018, 0011
`
`

`

`WO 99/36893
`
`PCT /EP99/00300
`
`Fig. 30 is a flow diagram illustrating the use of the system of the invention to
`
`detect drowsiness.
`
`Fig. 31 illustrates the use of sub-images to search a complete image.
`
`Fig. 32 illustrates tlie use of the system of the invention to detect nostrils and
`
`to track eye movement.
`
`Fig. 33 illustrates the use of the system of the invention to detect an open eye.
`
`Fig. 34 illustrates the use of the system of the invention to detect a closed eye.
`
`Fig. 35 is a flow diagram of an alternative method of detecting drowsiness.
`
`Fig. 36 illustrates use of the system to detect a pupil.
`
`DETAILED DESCRIPTION OF THE INVENTION
`
`The present invention discloses an application of the generic image processing
`
`system disclosed in commonly-owned PCT Application Serial Nos. PCT/FR97/01354 and
`
`PCT/EP98/05383, the contents of which are incorporated herein by reference for detection of
`
`various criteria associated with the human eye, and especially to detection that a driver is
`
`falling asleep while driveing a vehicle.
`
`The apparatus of the invention is similar to that described in the
`
`aforementioned PCT Application Serial Nos. PCT/FR97/01354 and PCT/EP98/05383, which
`
`will be described herein for purposes of clarity. Referring to Figs. 1 and 10, the generic
`
`image processing system 22 includes a spatial and temporal processing unit 11 in
`
`combination with a histogram formation unit 22a. Spatial and temporal processing unit 11
`
`includes an input 12 that receives a digital video signal S originating from a video camera or
`
`other imaging device 13 which monitors a scene 13 a. Imaging device 13 is preferably a
`
`conventional CMOS-type CCD camera, which for purposes of the presently-described
`
`invention is mounted on a vehicle facing the driver. It will be appreciated that when used in
`
`10
`
`Petitioner LG Ex-1018, 0012
`
`

`

`WO 99/36893
`
`PCT /EP99/00300
`
`non-vehicluar applications, the camera may be mounted in any desired fashion to detect the
`
`specific criteria of interest. It is also foreseen that any other appropriate sensor, e.g.,
`
`ultrasound, IR, Radar, etc., may be used as the imaging device. Imaging device 13 may have
`
`a direct digital output, or an analog output that is converted by an AID convertor into digital
`
`signal S. Imaging device 13 may also be integral with generic image processing system 22, if
`
`desired.
`
`While signal S may be a progressive signal, it is preferably composed of a
`
`succession of pairs of interlaced frames, TR1 and TR' 1 and TR2 and TR'2, each consisting of a
`
`succession of horizontal scanned lines, e.g., 111 , 11.2, ••• ,11.1 7 in TR1, and 2.1 in TR2• Each line
`
`consists of a succession of pixels or image-points Pl, e.g., au, au and au for line 111 ; a1 17.,
`
`and al 17.22 for line 1117 ; alu and a1. 2 for line 12.1• Signal S(PI) represents signal S composed of
`
`pixels Pl.
`
`S(PI) includes a frame synchronization signal (ST) at the beginning of each
`
`frame, a line synchronization signal (SL) at the beginning of each line, and a blanking signal
`
`(BL). Thus, S(PI) includes a succession frames, which are representative of the time domain,
`
`and within each frame, a series of lines and pixels, which are representative of the spatial
`
`domain.
`
`In the time domain, "successive frames" shall refer to successive frames of the
`
`same type (i.e., odd frames such as TR1 or even frames such as TR' 1), and "successive pixels
`
`in the same position" shall denote successive values of the pixels (PI) in the same location in
`
`successive frames of the same type, e.g., a11 oflu in frame TR 1 and au of 11.1 in the next
`
`corresponding frame TR2
`
`Spatial and temporal processing unit 11 generates outputs ZH and SR 14 to a
`
`data bus 23 (Fig. 11 ), which are preferably digital signals. Complex signal ZH comprises a
`
`11
`
`Petitioner LG Ex-1018, 0013
`
`

`

`WO 99/36893
`
`PCT /EP99/00300
`
`number of output signals generated by the system, preferably including signals indicating the
`
`existence and localization of an area or object in motion, and the speed V and the oriented
`
`direction of displacement DI of each pixel of the image. Also preferably output from the
`
`system is input digital video signal S, which is delayed (SR) to make it synchronous with the
`
`output ZH for the frame, taking into account the calculation time for the data in composite
`
`signal ZH (one frame). The delayed signal SR is used to display the image received by
`
`camera 13 on a monitor or television screen I 0, which may also be used to display the
`
`information contained in composite signal ZH. Composite signal ZH may also be transmitted
`
`to a separate processing assembly I Oa in which further processing of the signal may be
`
`accomplished.
`
`Referring to Fig. 2, spatial and temporal processing unit 11 includes a first
`
`assembly 11 a, which consists of a temporal processing unit 15 having an associated memory
`
`16, a spatial processing unit 17 having a delay unit 18 and sequencing unit 19, and a pixel
`
`clock 20, which generates a clock signal HP, and which serves as a clock for temporal
`
`processing unit 15 and sequencing unit 19. Clock pulses HP are generated by clock 20 at the
`
`pixel rate of the image, which is preferably 13.5 MHZ.
`
`Fig. 3 shows the operation of temporal processing unit 15, the function of
`
`which is to smooth the video signal and generate a number of outputs that are utilized by
`
`spatial processing unit 17. During processing, temporal processing unit 15 retrieves from
`
`memory 16 the smoothed pixel values LI of the digital video signal from the immediately
`
`prior frame, and the values of a smoothing time constant CI for each pixel. As used herein,
`
`LO and CO shall be used to denote the pixel values (L) and time constants (C) stored in
`
`memory 16 from temporal processing unit 15, and LI and CI shall denote the pixel values (L)
`
`and time constants (C) respectively for such values retrieved from memory 16 for use by
`
`12
`
`Petitioner LG Ex-1018, 0014
`
`

`

`WO99/36893
`
`PCT /EP99/00300
`
`temporal processing unit 15. Temporal processing unit 15 generates a binary output signal
`
`DP for each pixel, which identifies whether the pixel has undergone significant variation, and
`
`a digital signal CO, which represents the updated calculated value of time constant C.
`
`Referring to Fig.·3,"temporal processing unit 15 includes a first block 15a
`
`which receives the pixels PI of input video signal S. For each pixel PI, the temporal
`
`processing unit retrieves from memory 16 a smoothed value LI of this pixel from the
`
`immediately preceding corresponding frame, which was calculated by temporal processing
`
`unit 15 during processing of the immediately prior frame and stored in memory 16 as LO.
`
`Temporal processing unit 15 calculates the absolute value AB of the difference between each
`
`pixel value PI and LI for the same pixel position (for example a1.1, of 11.1 in TR, and of 11.1 in
`
`TR2 :
`
`AB= IPI-LI I
`
`Temporal processing unit 15 is controlled by clock signal HP from clock 20 in
`
`order to maintain synchronization with the incoming pixel stream. Test block 15b of
`
`temporal processing unit 15 receives signal AB and a threshold value SE. Threshold SE may
`
`be constant, but preferably varies based upon the pixel value PI, and more preferably varies
`
`with the pixel value so as to form a gamma correction. Known means of varying SE to form
`
`a gamma correction is represented by the optional block 15e shown in dashed lines. Test
`
`block 15b compares, on a pixel-by-pixel basis, digital signals AB and SE in order to
`
`determine a binary signal DP. If AB exceeds threshold SE, which indicates that pixel value
`
`PI has undergone significant variation as compared to the smoothed value LI of the same
`
`pixel in the prior frame, DP is set to "1" for the pixel under consideration. Otherwise, DP is
`
`set to "O" for such pixel.
`
`13
`
`Petitioner LG Ex-1018, 0015
`
`

`

`WO99/36893
`
`PCT /EP99/00300
`
`When DP = 1, the difference between the pixel value PI and smoothed value
`
`LI of the same pixel in the prior frame is considered too great, and temporal processing unit
`
`15 attempts to reduce this difference in subsequent frames by reducing the smoothing time
`
`constant C for that pixel. Conv~rsely, if DP= 0, temporal processing unit 15 attempts to
`
`increase this difference in subsequent frames by increasing the smoothing time constant C for
`
`that pixel. These adjustments to time constant C as a function of the value of DP are made by
`
`block 15c. If DP = 1, block 15c reduces the time constant by a unit value U so that the new
`
`value of the time 7constant CO equals the old value of the constant CI minus unit value U.
`
`CO=CI-U
`
`If DP = 0, block 15c increases the time constant by a unit value U so that the
`
`new value of the time constant CO equals the old value of the constant Cl plus unit value U.
`
`CO=CI+U
`
`Thus, for each pixel, block 15c receives the binary signal DP from test unit
`
`15b and time constant CI from memory 16, adjusts CI up or down by unit value U, and
`
`generates a new time constant CO which is stored in memory 16 to replace time constant Cl.
`
`In a preferred embodiment, time constant C, is in the form 2P, where pis
`
`incremented or decremented by unit value U, which preferably equals 1, in block 15c. Thus,
`
`if DP= 1, block 15c subtracts one (for the case where U=l) from pin the time constant 2P
`
`which becomes 2P· 1
`
`• lfDP = 0, block 15c adds one top in time constant 2P, which becomes
`
`
`
`2P· 1• The choice of a time constant of the form 2P facilitates calculations and thus simplifies
`
`the structure of block 15c.
`
`Block 15c includes several tests to ensure proper operation of the system.
`
`First, CO must remain within defined limits. In a preferred embodiment, CO must not
`
`become negative (CO2: 0) and it must not exceed a limit N (CO SN), which is preferably
`
`14
`
`Petitioner LG Ex-1018, 0016
`
`

`

`WO 99/36893
`
`PCT /EP99/00300
`
`seven. In the instance in which CI and CO are in the form 2P, the upper limit N is the
`
`maximum value for p.
`
`The upper limit N may be constant, but is preferably variable. An optional
`
`input unit 15f includes a register or memory that enables the user, or controller 42 to vary N.
`
`The consequence of increasing N is to increase the sensitivity of the system to detecting
`
`displacement of pixels, whereas reducing N improves detection of high speeds. N may be
`
`made to depend on PI (N may vary on a pixel-by-pixel basis, if desired) in order to regulate
`
`the variation of LO as a function of the lever of PI, i.e., Nijt = f(Piiji), the calculation of which
`
`is done in block 15f, which in this case would receive the va

This document is available on Docket Alarm but you must sign up to view it.


Or .

Accessing this document will incur an additional charge of $.

After purchase, you can access this document again without charge.

Accept $ Charge
throbber

Still Working On It

This document is taking longer than usual to download. This can happen if we need to contact the court directly to obtain the document and their servers are running slowly.

Give it another minute or two to complete, and then try the refresh button.

throbber

A few More Minutes ... Still Working

It can take up to 5 minutes for us to download a document if the court servers are running slowly.

Thank you for your continued patience.

This document could not be displayed.

We could not find this document within its docket. Please go back to the docket page and check the link. If that does not work, go back to the docket and refresh it to pull the newest information.

Your account does not support viewing this document.

You need a Paid Account to view this document. Click here to change your account type.

Your account does not support viewing this document.

Set your membership status to view this document.

With a Docket Alarm membership, you'll get a whole lot more, including:

  • Up-to-date information for this case.
  • Email alerts whenever there is an update.
  • Full text search for other cases.
  • Get email alerts whenever a new case matches your search.

Become a Member

One Moment Please

The filing “” is large (MB) and is being downloaded.

Please refresh this page in a few minutes to see if the filing has been downloaded. The filing will also be emailed to you when the download completes.

Your document is on its way!

If you do not receive the document in five minutes, contact support at support@docketalarm.com.

Sealed Document

We are unable to display this document, it may be under a court ordered seal.

If you have proper credentials to access the file, you may proceed directly to the court's system using your government issued username and password.


Access Government Site

We are redirecting you
to a mobile optimized page.





Document Unreadable or Corrupt

Refresh this Document
Go to the Docket

We are unable to display this document.

Refresh this Document
Go to the Docket