PCT
`
`WORLD INTELLECTUAL PROPERTY ORGANIZATION
`International Bureau
`
`
`
`INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY(PCT)
`(51) International Patent Classification 6 :
`(11) International Publication Number:
`WO 00/11610
`GO06T 7/20
`.
`_
`(43) International Publication Date:
`
`2 March 2000 (02.03.00)
`
`
`
`(21) International Application Number: PCT/EP98/05383|(81) Designated States: AL, AM, AT, AU, AZ, BA, BB, BG, BR,
`BY, CA, CH, CN, CU, CZ, DE, DK, EE, ES, FI, GB, GE,
`GH, GM, HR, HU,ID,IL, IS, JP, KE, KG, KP, KR, KZ,
`LC, LK, LR, LS, LT, LU, LV, MD, MG, MK, MN, MW,
`MX, NO, NZ, PL, PT, RO, RU, SD, SE, SG, SI, SK, SL,
`TJ, TM, TR, TT, UA, UG, US, UZ, VN, YU, ZW, ARIPO
`patent (GH, GM, KE, LS, MW,SD, SZ, UG, ZW), Eurasian
`patent (AM, AZ, BY, KG, KZ, MD, RU, TJ, TM), European
`patent (AT, BE, CH, CY, DE, DK, ES, FI, FR, GB, GR,
`IE, IT, LU, MC, NL, PT, SE), OAPI patent (BF, BJ, CF,
`CG, CI, CM, GA, GN, GW, ML, MR, NE, SN, TD, TG).
`
`luminance, hue,saturation, speed (V), oriented direction (D1), time constant (CO), first axis (x(m)), and second axis (y(m)).
`
`A method and apparatus
`for localizing an area in rel-
`ative movement and for de-
`termining the speed and di-
`rection thereof in real time is
`disclosed. Each pixel of an
`image is smoothed using its
`own time constant. A binary
`value corresponding to the ex-
`istence of a significant varia-
`tion in the amplitude of the
`smoothed pixel from the prior
`frame, and the amplitudeof the
`variation, are determined, and
`the time constant for the pixel
`is updated. For each particular
`pixel, two matrices are formed
`that include a subsetof the pix-
`els spatially related to the par-
`ticular pixel. The first matrix
`contains the binary values of
`the subset of pixels. The sec-
`ond matrix contains the ampli-
`tude ofthe variation of the sub-
`set of pixels.
`In the first ma-
`trix,
`it is determined whether
`the pixels along an oriented direction relative to the particular pixel have binary values representative of significant variation, and, for such
`pixels, it is determined in the second matrix whether the amplitude of these pixels varies in a known mannerindicating movementin the
`oriented direction. In each of several domains, histogram of the values in the first and second matrices falling in such domain is formed.
`Using the histograms, it is determined whether there is an area having the characteristics of the particular domain. The domains include
`
`(22) International Filing Date:
`
`25 August 1998 (25.08.98)
`
`(63) Related by Continuation (CON) or Continuation-in-Part
`(CIP) to Earlier Application
`US
`Filed on
`
`PCT/FR97/01354 (CIP)
`22 July 1997 (22.07.97)
`
`(71) Applicant (for all designated States except US): HOLDING
`BEV S.A. [LU/LU]; 69, route de d’Esch, L~2953 Luxem-
`burg (LU).
`
`(72) Inventor; and
`(73) Inventor/Applicant(for US only): PIRIM,Patrick [FR/FR]; 56,
`rue Patay, F-75013 Paris (FR).
`
`(74) Agent: PHELIP, Bruno; Cabinet Harlé & Phélip, 7, rue de
`Madrid, F-75008 Paris (FR).
`
`(34) Title:
`
`IMAGE PROCESSING APPARATUS AND METHOD
`
`(57) Abstract
`
`Published
`With international search report.
`
`IPT Ex-2002, p. 0001
`LG v IPT
`IPR2023-00104
`
`IPT Ex-2002, p. 0001
`LG v IPT
`IPR2023-00104
`
`

`

`
`
`FOR THE PURPOSES OF INFORMATION ONLY
`
`
`Codes used to identify States party to the PCT on the front pages of pamphlets publishing international applications under the PCT.
` Albania
`
`Spain
`LS
`Lesotho
`SI
`Slovenia
`ES
`Finland
`LT
`Lithuania
`SK
`Slovakia
`AM
`Armenia
`FI
`
`
`
`France
`LU
`Luxembourg
`SN
`Senegal
`AT
`Austria
`FR
`
`AU
`Australia
`GA
`Gabon
`LV
`Latvia
`SZ
`Swaziland
`
`AZ
`Azerbaijan
`GB
`United Kingdom
`MC
`Monaco
`TD
`Chad
`
`BA
`Bosnia and Herzegovina
`GE
`Georgia
`MD
`Republic of Moldova
`TG
`Togo
`
`BB
`Barbados
`GH
`Ghana
`MG
`Madagascar
`TJ
`Tajikistan
`
`BE
`Belgium
`GN
`Guinea
`MK
`The former Yugoslav
`™T™
`Turkmenistan
`
`BF
`Burkina Faso
`GR
`Greece
`Republic of Macedonia
`TR
`Turkey
`
`ML
`BG
`Bulgaria
`HU
`Hungary
`Mali
`TT
`Trinidad and Tobago
`
`MN
`BJ
`Benin
`IE
`Treland
`Mongolia
`UA
`Ukraine
`
`MR
`BR
`Brazil
`IL
`Israel
`Mauritania
`UG
`Uganda
`MW
`BY
`Belarus
`IS
`Iceland.
`Malawi
`US
`United States of America
`
`MX
`CA
`Canada
`IT
`Italy
`Mexico
`UZ
`Uzbekistan
`
`NE
`CF
`Central African Republic
`JP
`Japan
`Niger
`VN
`Viet Nam
`
`NL
`CG
`Congo
`KE
`Kenya
`Netherlands
`YU
`Yugoslavia
`
`NO
`CH
`Switzerland
`KG
`Kyrgyzstan
`Norway
`ZW
`Zimbabwe
`
`NZ
`cl
`Cote d’Ivoire
`KP
`Democratic People’s
`New Zealand
`
`PL
`CM
`Cameroon
`Republic of Korea
`Poland
`
`PT
`KR
`CN
`China
`Republic of Korea
`Portugal
`
`RO
`KZ
`cu
`Cuba
`Kazakstan
`Romania
`
`RU
`Le
`CZ
`Czech Republic
`Saint Lucia
`Russian Federation
`
`SD
`Li
`DE
`Germany
`Liechtenstein
`Sudan
`DK
`Denmark
`LK
`Sri Lanka
`SE
`Sweden
`
`EE
`Estonia
`LR
`Liberia
`SG
`Singapore
`
`
`
`
`
`
`IPT Ex-2002, p. 0002
`LG v IPT
`IPR2023-00104
`
`IPT Ex-2002, p. 0002
`LG v IPT
`IPR2023-00104
`
`

`

`WO 00/11610
`
`PCT/EP98/05383
`
`IMAGE PROCESSING APPARATUS AND METHOD
`
`BACKGROUND OF THE INVENTION
`
`1. Field of the Invention
`
`10
`
`The presentinvention relates generally to an image processing apparatus, and
`more particularly to a method and apparatus for identifying and localizing an area in
`relative movementin a scene and determining the speed and oriented direction of the area
`in real time.
`
`15
`
`2. Description of the Related Art
`
`The human or animal eye is the best known system for identifying and
`localizing an object in relative movement, and for determining its speed and direction of
`movement. Various efforts have been made to mimic the function of the eye. One type of
`device for this purposeis referred to as anartificial retina, which is shown, for example,
`in GiocomoIndiveri et. al, Proceedings of MicroNeuro, 1996, pp. 15-22 (analog artificial
`retina), and Pierre-Frangois Riiedii, Proceedings of MicroNeuro, 1996, pp. 23-29, (digital
`artificial retina which identifies the edges of an object). However, very fast and high
`capacity memories are required for these devices to operate in real time, and only limited
`information is obtained about the moving areas or objects observed Other examples of
`artificial retinas and similar devices are shown in U S. Patent Nos. 5,694,495 and
`5,712,729.
`
`Another proposed method for detecting objects in an imageis to store a frame
`from a video cameraorother observation sensor in a first two-dimensional memory. The
`frame is composed of a sequence of pixels representative of the scene observed by the
`camera at time t). The video signal for the next frame, which represents the scene at time
`t,, is stored in a second two-dimensional memory.If an object has moved between times ty
`and t,, the distance d by which the object, as represented by its pixels, has moved in the
`scene between t, and t, is determined. The displacement speed is then equal to d/T, where
`
`20
`
`25
`
`30
`
`CONFIRMATION COPY
`
`IPT Ex-2002, p. 0003
`
`LG v IPT
`IPR2023-00104
`
`IPT Ex-2002, p. 0003
`LG v IPT
`IPR2023-00104
`
`

`

`WO 00/11610
`
`PCT/EP98/05383
`
`2
`
`T=t, - tp. This type of system requires a very large memory capacityif it is used to obtain
`precise speedandoriented direction. Information for the movementofthe object. Thereis
`also a delay in obtaining the speed and displacementdirection information corresponding
`to t, + R, where R is the time necessary for the calculations for the period t, - t, system.
`These two disadvantageslimit applicationsofthis type of system.
`Anothertype ofprior image processing system is shown in FrenchPatent No.
`2,611,063, of which the inventor hereof is also an inventor. This patent relates to a
`method and apparatus for real time processing ofa sequenced data flow from the output
`of a camerain order to perform data compression. A histogram of signal levels from the
`camera is formed using a first sequenceclassification law. A representative Gaussian
`function associated with the histogram is stored, and the maximum and minimum levels
`are extracted. The signallevels of the next sequenceare compared with the signal levels
`for the first sequence using a fixed time constant
`identical for each pixel. A binary
`classification signal is generated that characterizes the next sequencewith reference to the
`classification law An auxiliary signal
`is generated from the binary signal
`that
`is
`representative of the duration and position of a range of significant values. Finally, the
`auxiliary signal is used to generate a signal localizing the range with the longest duration,
`called the dominant range. These operations are repeated for subsequent sequences ofthe
`sequencedsignal.
`This prior process enables data compression, keeping only interesting
`parameters in the processed flow of sequenced data. In particular, the process is capable
`of processing a digital video signal
`in order to extract and localize at
`least one
`characteristic of at least one area in the image.It is thus possible to classify, for example,
`brightness and/or chrominance levels of the signal and to characterize and localize an
`object in the image.
`U.S. Patent No. 5,488,430 detects and estimates a displacement by separately
`determining horizontal and vertical changes of the observed area. Difference signals are
`used to detect movementsfrom right to left or from left to right, or from top to bottom or
`bottom to top, in the horizontal andvertical directions respectively. This is accomplished
`by carrying out an EXCLUSIVE ORfunction on horizontal/vertical difference signals and
`on frame difference signals, and by using a ratio of the sums of the horizontal/vertical
`signals and the sums of frame difference signals with respect to a K x 3 window.
`Calculated values of the image along orthogonal horizontal and vertical directions are
`
`10
`
`20
`
`25
`
`30
`
`IPT Ex-2002, p. 0004
`LG v IPT
`IPR2023-00104
`
`IPT Ex-2002, p. 0004
`LG v IPT
`IPR2023-00104
`
`

`

`WO 00/11610
`
`PCT/EP98/05383
`
`3
`
`used with an identical repetitive difference K in the orthogonal directions, this difference
`K being defined as a function ofthe displacement speeds that are to be determined. The
`device determinesthe direction of movement along each of the two orthogonaldirections
`by applyingasetofcalculation operations to the difference signals, which requires very
`complex computations. Additional complex computationsare also necessary to obtain the
`speed and oriented direction of displacement (extraction of a square root to obtain the
`amplitude of the speed, and calculation of the arctan function to obtain the oriented
`direction), starting from projections on the horizontal and vertical axes. This device also
`does not smooth the pixel values using a time constant, especially a time constant that is
`variable for each pixel, in order to compensate for excessively fast variations in the pixel
`values.
`
`10
`
`Finally, Alberto Tomita Sales Representative. and Rokuva Ishii, "Hand Shape
`Extraction from a Sequence ofDigitized Gray-Scale Images," Institute of Electrical and
`Electronics Engineers, Vol. 3, 1994, pp. 1925-1930, detects movement by subtracting
`between successive images, and forming histograms based upon the shape of a human
`hand in order to extract the shape of a human hand in a digitized scene. The histogram
`analysis is based upon a gray scale inherent to the human hand.It does not include any
`means of forming histogramsin the plane coordinates . The sole purpose of the methodis
`to detect the displacement of a human hand, for example, in order to replace the normal
`computer mouse by a hand, the movements of whichareidentified to control a computer.
`It would be desirable to have an image processing system which has a
`relatively simple structure and requiresa relatively small memory capacity, and by which
`information on the movementofobjects within an image can be obtained inreal-time.It
`would also be desirable to have a method and apparatusfor detecting movements that are
`not limited to the hand, but to any object (in the widest sense ofthe term) in a scene, and
`which does not use histograms based on the gray values of a hand, but rather the
`histograms of different variables representative of the displacement and histograms of
`plane coordinates. Such a system would be applicable to many types of applications
`requiring the detection of moving and non-moving objects.
`
`15
`
`20
`
`25
`
`30
`
`IPT Ex-2002, p. 0005
`LG v IPT
`IPR2023-00104
`
`IPT Ex-2002, p. 0005
`LG v IPT
`IPR2023-00104
`
`

`

`WO 00/11610
`
`PCT/EP98/05383
`
`4
`
`SUMMARYOF THE INVENTION
`
`The present invention is a process for identifying relative movement of an
`object in an input signal,
`the input signal having a succession of frames, each frame
`having a succession of pixels. For each pixel of the input signal,
`the input signalis
`smoothedusing a time constantfor the pixel in order to generate a smoothed input signal.
`For each pixel in the smoothed input signal, a binary value correspondingto the existence
`ofa significant variation in the amplitude of the pixel between the current frame and the
`immediately previous smoothed input ‘frame, and the amplitude of the variation, are
`determined.
`
`the time
`Using the existence of a significant variation for a given pixel,
`constant for the pixel, which is to be used in smoothing subsequent frames of the input
`signal, is modified. The time constant is preferably in the form 2°, and is increased or
`decreased by incrementing or decrementing p. For each particular pixel of the input
`signal, two matrices are then formed: a first matrix comprising the binary values of a
`subset of the pixels of the frame spatially related to the particular pixel; and a second
`matrix comprising the amplitude ofthe variation of the subset of the pixels of the frame
`spatially related to the particular pixel. In the first matrix,
`it is determined whether the
`particular pixel and the pixels along an oriented direction relative to the particular pixel
`have binary valuesof a particular value representing significant variation, and, for such
`pixels, it is determined in the second matrix whether the amplitude ofthe pixels along the
`oriented direction relative to the particular pixel varies in a known manner indicating
`movement
`in the oriented direction of the particular pixel and the pixels along the
`oriented direction relative to the particular pixel. The amplitude of the variation of the
`pixels along the oriented direction determines the velocity of movementofthe particular
`pixel and the pixels alongthe orienteddirection relative to the particular pixel.
`In each of one or more domains, a histogram of the valuesdistributed in the
`first and second matrices falling in each such domain is formed. For a particular domain,
`an area of significant variation is determined from the histogram for that domain.
`Histograms of the area ofsignificant variation along coordinate axes are then formed.
`From these histograms, it is determined whether there is an area in movement for the
`particular domain. The domainsare preferably selected from the group consisting ofi)
`
`10
`
`15
`
`20
`
`25
`
`30
`
`IPT Ex-2002, p. 0006
`LG v IPT
`IPR2023-00104
`
`IPT Ex-2002, p. 0006
`LG v IPT
`IPR2023-00104
`
`

`

`WO 00/11610
`
`PCT/EP98/05383
`
`5
`
`luminance, 1i) speed (V),iii) oriented direction (D1), iv) time constant (CO), v) hue, vi)
`saturation, and vii) first axis (x(m)), and viii) secondaxis (y(m)).
`In one embodiment, the first and second matrices are square matrices, with the
`same odd number of rows and columns, centered on the particular pixel.
`In this
`embodiment, the steps of determiningin the first matrix whether the particular pixel and
`the pixels along an oriented direction relative to the particular pixel have binary values of
`a particular value representing significant variation, and the step of determining in the
`second matrix whether the amplitude signal varies in a predetermined criteria along an
`oriented direction relative to the particular pixel, comprise applying nested n x n matrices,
`wherenis odd, centered on the particular pixel to the pixels within each ofthe first and
`10
`second matrices. The process then includes the further step of determining the smallest
`nested matrix in which the amplitude signal varies along an oriented direction around the
`particular pixel.
`
`15
`
`20
`
`In an alternative embodiment, the first and second matrices are hexagonal
`matrices centered on the particular pixel. In this embodiment, the steps of determining in
`the first matrix whether the particular pixel and the pixels along an oriented direction
`relative to the particular pixel have binary values of a particular value representing
`significant variation, and the step of determining in the second matrix whether the
`amplitude signal varies in a predeterminedcriteria along an oriented direction relative to
`the particular pixel, comprise applying nested hexagonal matrices of varying size centered
`on the particular pixel to the pixels within each of the first and second matrices. The
`
`25
`
`process then further includes determining the smallest nested matrix in which the
`amplitude signalvaries along an oriented direction aroundthe particularpixel.
`In a still further embodiment of the invention, the first and second matrices
`are inverted L-shaped matrices with a single row and a single column.
`In this
`embodiment, the steps of determiningin the first matrix whether the particular pixel and
`the pixels along an oriented direction relative to the particular pixel have binary values of
`a particular value representing significant variation, and the step of determining in the
`second matrix whether the amplitude signal varies in a predeterminedcriteria along an
`oriented direction relative to the particular pixel, comprise applying nested n x n matrices,
`wherenis odd, to the single line and the single column to determine the smallest matrix
`in which the amplitude varies on a line with the steepest slope and constant quantification.
`
`30
`
`IPT Ex-2002, p. 0007
`LG v IPT
`IPR2023-00104
`
`IPT Ex-2002, p. 0007
`LG v IPT
`IPR2023-00104
`
`

`

`WO 00/11610
`
`PCT/EP98/05383
`
`6
`
`If desired, successive decreasing portionsofframesof the input signal may be
`considered using a Mallat time-scale algorithm, and the largest of these portions, which
`provides displacement, speed and orientation indications compatible with the value ofp,
`is selected.
`
`In a process of smoothing an inputsignal, for each pixel of the input signal, i)
`the pixel is smoothed using a time constant (CO) for that pixel, thereby generating a
`smoothed pixel value (LO),ii) it is determined whetherthere exists a significant variation
`between such pixel and the samepixel in a previous frame, and lit) the time constant (CO)
`for such pixel to be used in smoothingthe pixel in subsequent framesofthe input signal is
`modified based uponthe existence or non-existence of a significant variation.
`The step of determining the existence of a significant variation for a given
`pixel preferably comprises determining whetherthe absolute value ofthe difference (AB)
`between the given pixel value (PI) and the value of such pixel in a smoothed prior frame
`(LI) exceeds a threshold (SE). The step of smoothing the input signal preferably
`comprises, for each pixel, i) modifying the time constant (CO)for pixel such based upon
`the existence of a significant variation as determinedin the prior step, and ii) determining
`a smoothedvalue forthe pixel (LO) as follows:
`
`PI -LI
`LO=LI "CO
`
`Time constant (CO) is preferably in the form 2°, and p is incremented in the
`event that AB<SE and decremented in the event AB2>SE.
`
`In this process, the system generates an output signal comprising, for each
`pixel, a binary value (DP) indicating the existence or non-existence of a significant
`variation, and the value of the time constant (CO). The binary values (DP) and the time
`constants (CO)are preferably stored in a memory sized to correspond to the framesize.
`A process for identifying an area in relative movement in an input signal
`includes the steps of:
`
`generating a first array indicative of the existence of significant variation in
`the magnitude ofeach pixel between a current frame and a prior frame;
`generating a secondarray indicative of the magnitude ofsignificant variation
`of each pixel between the current frame and a priorframe;
`
`IPT Ex-2002, p. 0008
`LG v IPT
`IPR2023-00104
`
`15
`
`20
`
`25
`
`30
`
`IPT Ex-2002, p. 0008
`LG v IPT
`IPR2023-00104
`
`

`

`WO 00/11610
`
`PCT/EP98/05383
`
`7
`
`establishingafirst moving matrix centered on a pixel under consideration and
`comprising pixels spatially related to the pixel under consideration,
`the first moving
`matrix traversingthefirst array for consideration of each pixel of the current frame; and
`determining whether the pixel under consideration and each pixel of the pixels
`spatially related to the pixel under consideration along an oriented direction relative
`thereto within the first matrix are a particular value representing the presence of
`significant variation, and ifso, establishing in a second matrix within the first matrix,
`centered on the pixel under consideration, and determining whether the amplitude ofthe
`pixels in the second matrix spatially related to the pixel under consideration along an
`oriented direction relative thereto are indicative of movement along such oriented
`direction, the amplitude of the variation along the oriented direction being indicative of
`the velocity of movement, the size of the second matrix being varied to identify the matrix
`size mostindicative of movement.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`The process further comprises, in at least one domain selected from the group
`consisting of i) luminance, ii) speed (V), iii) oriented direction (D1),
`iv) time constant
`(CO), v) hue, vi) saturation, and vii) first axis (x(m)), and viii) second axis (y(m)), and ix)
`data characterized by external inputs, forming a first histogram of the values in such
`domainfor pixels indicative of movement alonganoriented direction relative to the pixel
`under consideration. If desired, for the pixels in the first histogram, histograms of the
`position of such pixels along coordinate axes may be formed, and from such histograms,
`an area of the image meetingcriteria of the at least one domain may be determined.
`A process for identifying pixels in an input signal in one of a plurality of
`classes in one ofa plurality of domains comprises, on a frame-by-framebasis:
`for each pixel of the input signal, analyzing the pixel and providing an output
`signal for each domain containing information to identify each domain in which the pixel
`is classified;
`
`providing a classifier for each domain,theclassifier enabling classification of
`pixels within each domain to selected classes within the domain;
`providing a validation signal for the domains, the validation signal selecting
`one or moreofthe plurality of domains for processing; and
`forming a histogram forpixels of the output signal within the classes selected
`by the classifier within each domain selected by the validation signal.
`
`IPT Ex-2002, p. 0009
`LG v IPT
`IPR2023-00104
`
`IPT Ex-2002, p. 0009
`LG v IPT
`IPR2023-00104
`
`

`

`WO 00/11610
`
`PCT/EP98/05383
`
`8
`
`The process further includes the steps of forming histograms along coordinate
`axes for the pixels within the classes selected by the classifier within each domain
`selected by the validation signal, and forming a composite signal corresponding to the
`spatial position of such pixels within the frame. Pixels falling within limits Las bys Egy Uy in
`the histogramsalong the coordinate axes are then identified, and a composite signal from
`the pixels falling within these limits is formed.
`
`A process for identifying the velocity of movement of an area of an input
`signal comprises:
`
`for each particular pixel of the input signal, forminga first matrix comprising
`binary values indicating the existence or non-existence ofa significant variation in the
`amplitude of the pixel signal between the current frame andaprior frame for a subset of
`the pixels of the frame spatially related to such particular pixel, and a second matrix
`comprising the amplitude of such variation;
`determining in the first matrix whether the particular pixel and the pixels
`along an oriented direction relative to the particular pixel have binary values of a
`particular value representing significant variation, and, for such pixels, determining in the
`second matrix whether the amplitudesofthe pixels along an oriented direction relative to
`the particular pixel vary in a known mannerindicating movementofthe pixel and the
`pixels along an oriented direction relative to the particular pixel, the amplitude of the
`variation along the oriented direction determining the velocity of movement of the
`particular pixel.
`
`20
`
`15
`
`A processfor identifying a non-movingarea in an input signal comprises:
`forming histograms along coordinate axes for pixels of the input signal
`without significant variation betweenthe current frameandaprior frame; and
`25
`forming a composite signal corresponding to the spatial position of such
`pixels within the frame.
`
`30
`
`An apparatus for identifying relative movement in an input signal comprises:
`means for smoothing the input signal using a time constant for each pixel,
`thereby
`generating a smoothedinputsignal;
`means for determining for each pixel in the smoothed inputsignal a binary
`value corresponding to the existence ofa significant variation in the amplitude of the
`pixel signal between the current frame and the immediately previous smoothed input
`frame, and for determining the amplitude ofthe variation;
`
`IPT Ex-2002, p. 0010
`LG v IPT
`IPR2023-00104
`
`IPT Ex-2002, p. 0010
`LG v IPT
`IPR2023-00104
`
`

`

`WO 00/11610
`
`PCT/EP98/05383
`
`9
`
`means for using the existence ofa significant variation for a given pixel to
`modify the time constant for the pixel to be used in smoothing subsequent framesofthe
`input signal;
`
`means for forming a first matrix comprising the binary values of a subset of
`the pixels of the framespatially related to each particular pixel, and for forming a second
`matrix comprising the amplitude of the variation of the subsetof the pixels of the frame
`spatially related to such particularpixel;
`means for determining in the first matrix a particular area in which the binary
`value for each pixel is a particular value representing significant variation, and, for such
`particular area, for determining in the second matrix whether the amplitude varies along
`an oriented direction relative to the particular pixel
`in a known manner indicating
`movementof the pixel in the oriented direction, the amplitude ofthe variation along the
`oriented direction determining the velocity of movementofthepixel.
`An apparatus for smoothing aninput signal comprises:
`means for smoothing each pixel ofthe input signal using a time constant (CO)
`for such pixel, thereby generating a smoothed pixel value (LO) ;
`means for determining the existence of a significant variation for a given
`pixel, and modifying the time constant (CO) for the pixel to be used in smoothing the
`pixel in subsequentframes of the input signal based upon the existence of such significant
`variation.
`
`An apparatus for identifying an area in relative movementin an inputsignal
`
`comprises:
`
`means for generating a first array indicative of the existence of significant
`variation in the magnitude of each pixel between a current frame and a prior frame;
`means for generating a second array indicative of the magnitude ofsignificant
`variation of each pixel between the current frame and a priorframe;
`means for establishing a first moving matrix centered on a pixel under
`consideration and comprising pixels spatially related to the pixel under consideration, the
`first moving matrix traversing the first array for consideration of each pixelof the current
`frame;
`
`means for determining whether the pixel under consideration and eachpixel
`along an oriented direction relative to the pixel under consideration within the first matrix
`is a particular value representing the presence of significant variation, and if so, for
`
`15
`
`20
`
`25
`
`30
`
`IPT Ex-2002, p. 0011
`LG v IPT
`IPR2023-00104
`
`IPT Ex-2002, p. 0011
`LG v IPT
`IPR2023-00104
`
`

`

`WO 00/11610
`
`PCT/EP98/05383
`
`10
`
`establishing a second matrix within the first matrix, centered on the pixel under
`consideration, and for determining whether the amplitude of the pixels in the second
`matrix are indicative of movement along an oriented direction relative to the pixel under
`consideration, the amplitude of the variation along the oriented direction being indicative
`of the velocity of movement, the size of the second matrix being varied to identify the
`matrix size most indicative of movement.
`
`An apparatus for identifying pixels in an input signal in one ofa plurality of
`classes in one ofaplurality of domains comprises:
`means for analyzing each pixel of the input signal and for providing an output
`signal for each domain containing information to identify each domain in which the pixel
`is classified;
`
`10
`
`15
`
`20
`
`25
`
`30
`
`the classifier classifying pixels within each
`a classifier for each domain,
`domainin selected classes within the domain;
`
`a linear combination unit
`for each domain,
`the linear combination unit
`generating a validation signal for the domain,the validation signal selecting one or more
`of the plurality of domains for processing; and
`
`means for forming a histogram for pixels of the output signal within the
`classes selected bythe classifier within each domain selected bythe validation signal.
`An apparatus for identifying the velocity of movement ofan area of an input
`signal comprises:
`
`in the input signal a binary value
`means for determining for each pixel
`corresponding to the existence of a significant variation in the amplitude of the pixel
`signal between the current frame and the immediately previous smoothed input frame, and
`for determining the amplitudeofthe variation,
`
`means for forming,for each particular pixel of the input signal, a first matrix
`comprising the binary valuesof a subsetof the pixels spatially related to such particular
`pixel, and a second matrix comprising the amplitude ofthe variation ofthe subset ofthe
`pixels spatially related to such particular pixel; and
`
`means for determining in the first matrix whether for a particular pixel, and
`other pixels along an oriented direction relative to the particular pixel, the binary value for
`each pixel is a particular value representing significant variation, and, for such particular
`pixel and other pixels, determining in the second matrix whether the amplitude varies
`along an oriented direction relative to the particular pixel in a known manner indicating
`
`IPT Ex-2002, p. 0012
`LG v IPT
`IPR2023-00104
`
`IPT Ex-2002, p. 0012
`LG v IPT
`IPR2023-00104
`
`

`

`WO 00/11610
`
`PCT/EP98/05383
`
`11
`
`movement of the pixel and the other pixels, the amplitude of the variation along the
`oriented direction determining the velocity of movementofthe pixel andthe otherpixels.
`An apparatusfor identifying a non-moving areain an inputsignal comprises:
`means for forming histograms along coordinate axes for pixels of a current
`frame withouta significantvariation from such pixels in a prior frame; and
`means for forming a composite signal correspondingto the spatial position of
`such pixels within the frame.
`
`10
`
`15
`
`invention.
`
`invention.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`Fig. I is a diagrammaticillustration of the system accordingto the invention.
`Fig. 2 is a block diagram of the temporal and spatial processing units of the
`
`Fig. 3 is a block diagram of the temporal processing unit of the invention.
`Fig. 4 is a block diagram ofthe spatial processing unit of the invention.
`Fig. 5 is a diagram showing the processing of pixels in accordance with the
`
`Fig. 6 illustrates the numerical values of the Freeman code used to determine
`
`movementdirection in accordance with the invention.
`
`20
`
`Fig. 7 illustrates two nested matrices as processed by the temporal processing
`
`unit.
`
`unit.
`
`25
`
`unit.
`
`Fig.8 illustrates hexagonal matrices as processed by the temporal processing
`
`Fig.9 illustrates reverse-L matrices as processed by the temporal processing
`
`Fig.9a illustrates angular sector shaped matrices as processed by the temporal
`processing unit.
`
`Fig. 10 is a block diagram showingtherelationship between the temporal and
`spatial processing units, and the histogram formation units.
`Fig. 11 is a block diagram showingtheinterrelationship between the various
`histogram formation units.
`
`30
`
`Fig. 12 showsthe formation of a two-dimensional histogram of a movingarea
`from two one-dimensionalhistograms.
`
`IPT Ex-2002, p. 0013
`LG v IPT
`IPR2023-00104
`
`IPT Ex-2002, p. 0013
`LG v IPT
`IPR2023-00104
`
`

`

`WO 00/11610
`
`PCT/EP98/05383
`
`12
`
`Fig. 13 is a block diagram of an individual histogram formation unit.
`Fig.14 illustrates the use ofthe classifier for finding an alignment of points
`relative to the direction of an analysis axis.
`Fig. 14aillustrates a one-dimensional histogram.
`Fig.
`15
`illustrates
`the
`use of
`the
`system of
`video-conferencing.
`Fig.16 is a top view ofthe system of the invention for video-conferencing.
`Fig.17 is a diagram illustra