RCV. VOREEPA MUENCHEN 0G
`
`12B- 8-98 >
`4,
`
`
`16:47 ;
`
`+49 89 29994465:# 5 ”
`33.01 48745456—-
`
`
`
`
`
`
`
`ev sEpG ar He iB3
`
`
`
`IMAG)! PROCESSING APPARATUS AND METHOD
`
`
`
`
`
`Juventor: Patrick }irim
`
`
`
`
`BACKGROUND OFTH: INVIINTION
`
`
`
`
`
`
`1. l‘icld of the Invention
`
`
`
`
`
`
`The present invention relates generally (o an image processing apparatus, and
`
`
`
`
`
`
`
`
`
`
`
`more particularly to a method and apparatus for identifying and localizing an area in
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`relative movement in a scene and determining the speed and oriented direction of the urea
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`in real time.
`
`
`
`
`2. Description of the Kclated Ari
`
`
`
`
`
`
`The human or animal eye is the best known system for identifying and
`
`
`
`
`
`
`
`
`
`
`
`
`
`localizing an objectin relativemovement, and for determining its speed and dircction of
`
`
`
`
`
`
`
`
`
`
`
`
`
`movement. Variousefforts have been made to mimic the function of the cye. Onc type of
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`device for this purposeis referred to as an artificinl retina, which is shown, for cxample,
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`in GiocomoJndiveri et. al, Procecdingsof MicroNeuro, 1996,pp. 15-22 (analogartificial
`
`
`
`
`
`
`
`
`
`
`
`
`
`retina), and Pierre-Frangois Rucdii, Proceedings ofMicroNcuro, 1996, pp. 23-29, (digital
`
`
`
`
`
`
`
`
`
`
`
`artificial retina which identifies the edges of an object), However, very fast and high
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`capacity memorics are sequired for these devices (0 opcrate in real time, and only limited
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`information is obtained about the moving areas or objects observed Other cxamples of
`
`
`
`
`
`
`
`
`
`
`
`
`artificial retinas and similar devices are shown in U S. Patent Nos. 5,094,495 “and
`
`
`
`
`
`
`
`
`
`
`
`
`
`5,712,729,
`
`—_——
`
`
`
`
`
`
`®@
`
`@
`
`5
`
`
`
`10
`
`
`
`15
`
`
`20
`
`
`
`25
`
`
`30
`
`
`
`
`
`.
`Another proposed method for detecting, objects in an image is (o store a frame
`
`
`
`
`
`
`
`
`
`
`
`
`
`from a video camera or other observation scnsorin a first (wo-dimensional memory. The
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`frune is composed of a scquence of pixels representative of the scene observed by the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`camera at Lime t,. The video signal for the next frame, which represcnts the scene al time
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`{,, is stored in a second two-dimensional memory. If an objoct has moved betweenlimes t,
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`and t,the distance d by which the object, as represented by ils pixcls, has moved in the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`scene between {, and {,is determined. The displacement speed is then equal to d/I, where
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`eee
`
`Petitioner LG Ex-1021, 0001
`
`Petitioner LG Ex-1021, 0001
`
`

`

`“RCV. VONTRPA MPENCHEN OG
`
`
`
`
`
`
`
`
`:25- 8-08 +
`
`'
`
`16:48 :
`
`2
`
`
`
`TBP LANGE 8
`33 3} 48745450-
`
`
`
`
`
`
`
`
`
`~ —
`neypryrhags FF. GR
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`S$
`
`
`
`10
`
`
`
`15
`
`
`20
`
`
`
`
`@
`
`@
`a
`
`
`
`“he
`
`
`
`
`
`
`Te-ty+ ty. This type of system requires a very Jarge memory capacily if il is used io obtain
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`precise spocd and oriented direction. Information for lhc movement Gf the obicet. There is
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`also a delay in obtaining, the speed and displacement direction information corresponding
`
`
`
`
`
`
`
`
`
`
`
`
`to i, ‘1 R, where R is the time necessary for the calculations for the period 1, - (, system.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`These two disadvantages limit applications of this type of systcm.
`
`
`
`
`
`
`
`
`
`
`Another type of prior image processing systemis shown in !rench Patent No.
`
`
`
`
`
`
`
`
`
`
`
`
`
`2,613,063, of which the inventoz hercof is also an inventor. This patent icles 10 4
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`‘method and apparatus for rcal time processing of a sequenced data flow from the output
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`of a cainera in order to perform data compression. A hisiogtamof signal levels fon: the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`camera is fonned using a first sequence classification law. A repicsentative Gaussian
`
`
`
`
`
`
`
`
`
`
`
`
`function associated with the histogram is stored, and the maximum and minimum ievels
`
`
`
`
`
`
`
`
`
`
`
`
`
`wre extracted. The signal Jevels of the next sequenee are compared with the signal ic .cls
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`for the first sequence using a fixed time constant identicad for cach pixel... sinary
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Classification signal is gencrated that characterizes the nex! sequence with refercncs «.. the
`
`
`
`
`
`
`
`
`
`
`
`
`
`classification Jaw An auxiliary signal
`is generated from the bimary signe! wir
`is
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`representative of the duration and position of a range of significant vulues Finais,
`the
`
`
`
`
`
`
`
`
`
`
`
`
`
`auxiliary signal is used to pencrate a signal localizing the range with the longest cist on,
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`called the dominant range. These operations are repeated for subseguent sequence> -
`
`
`
`
`
`
`
`
`
`
`
`sequenced signa).
`
`
`This prior process cnables data compression, keeping onjy inig:.:ing
`
`
`
`
`
`
`
`
`
`
`parameters in the processed flow of sequenced data. In particular, the process i. vapcle
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`_ Of processing @ digital video signal
`in order to extract and loculize ni
`Ives one
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`characteristic of af Ieast one arca in the image.It is thus possible 10 classify, for cxaninic,
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`brightness and/or chrominance levels of the sipnal and to characterize and Jecakz< an
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`U.S. Patent No. 5,488,430 detects and cstimates a displacement by scnaraicly
`
`
`
`
`
`
`
`
`
`
`
`determining horizontal aud vertical changes of the observed area. Difference signah are
`
`
`
`
`
`
`
`
`
`
`
`
`used to detect movenicnte from right to Iefl or fromIch to right, or fron top te batten or
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`bottomto top, in the horizontal andvertical directions respectively. This is accong ished
`
`
`
`
`
`
`
`
`
`
`
`
`
`bycarrying out an EXCL.USIVJ' OR function on horizontal/veriical difference signals and
`
`
`
`
`
`
`
`
`
`
`
`
`on frame difference signals, end by using a ratio of the sums of the horizontal’ veritcal
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`signals and the sume of frame difference signals with respect
`to n K x 3 window.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Calculated values of the image slong orthogonal horizontal and vertical directions are
`
`
`
`
`
`
`
`
`
`
`
`
`
`25~—object in the image. — —
`
`30
`
`
`Petitioner LG Ex-1021, 0002
`
`Petitioner LG Ex-1021, 0002
`
`€
`

`

`RCV. VONSEPA MUENCHEN 0G
`
`
`
`
`
`
`
`725- 3-98 ;
`
`
`16:48 :
`
`
`
`33°
`1 48745456-
`+40 80 2U80740554 7
`
`
`
`
`
`
`aR
`SOT/EP SBME SBS
`
`
`
`
`
`«
`
`
`
`3
`
`®@
`
`used with an identical repetitive difference K in the orthogonal dircetions, this difference
`
`
`
`
`
`
`
`
`
`
`
`
`
`K being defined as a fanction of the displacement speeds that arc (o be determined. ‘The
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`device determines the direction of movement along cach of the iwo orthagonal directions
`
`
`
`
`
`
`
`
`
`
`
`
`by applying # sct of calculation operations to the difference signals, which requires very
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`complex computations. Additional complex computations arc also necessary to obtain the
`
`
`
`
`
`
`
`
`
`
`
`speed and oricnicd direction of displacement (extraction of a square root to obtain the
`
`
`
`
`
`
`
`
`
`
`
`
`
`amplitude of the specd, and calculation of the arctan function to obtain the oriented
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`direction), slarting from projcctions on the horizontal and vertical axes. This device also
`
`
`
`
`
`
`
`
`
`
`
`
`
`docs noi smooth the pixc] valucs using a lime constant, especially a timc constant that is
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`variabic for cach pixel, in order to compensate for excessively fast variations in the pixel
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`valucs,
`
`
`
`
`Hinally, Albcrio Tomita Sates Representative. and RokuveIshii, "Wand Shape
`
`
`
`
`
`
`
`
`
`
`Uxtraction from a Sequence of Vipitized Gray-Scale Images," Institute of Mlcctrical and
`
`
`
`
`
`
`
`
`
`
`lilcctronics Engineers, Vol. 3, 1994, pp. 1925-1930, detects movement by subtracting
`
`
`
`
`
`
`
`
`
`
`
`between successive images, and forming histograms based upon the shape of a human
`
`
`
`
`
`
`
`
`
`
`
`
`
`hand in order to extract the shape of a human hand in a digitized scene. The histogram
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`analysis is based upon a grayscaic inherent to the human hand. }( docs not include any
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`means of forming histograms in the plane coordinates . ‘The sole purpose of the method is
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`1o delect the displacement of a humanhand, for example, in order to replace the normal
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`computer mousc by a hand, the movements of which arc identified 1o control a computer.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Tt would be desirable to have im image processing system which has a
`
`
`
`
`
`
`
`
`
`
`
`
`
`relatively simple stracture and requires a relatively small memory cnpavity, and by which
`
`
`
`
`
`
`
`
`
`
`
`
`information on the movement of objects within an image can be obtained in real-time. It
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`would also be desirable to have a method and apparatus for detccling movements that are
`
`
`
`
`
`
`
`
`
`
`
`
`
`nat limited to the hand, bul to any object (in the widest scnse of the term) in a scene, and
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`which docs not usc histograms based on the gray values of a hand, bul rather the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`histograms of different variables representative of the displacement and histograms of
`
`
`
`
`
`
`
`
`
`
`
`plane coordinates. Such a sysicm would be applicable to many types of applications
`
`
`
`
`
`
`
`
`
`
`
`
`
`requiring the detection of moving, and non-movingobjects.
`
`
`
`
`
`
`
`
`5
`
`
`
`10
`
`
`15
`
`
`20
`
`
`tya
`
`
`30
`
`
`eeneeeee
`
`Petitioner LG Ex-1021, 0003
`
`Petitioner LG Ex-1021, 0003
`
`

`

`= RCV. VON EPA MUENCHEN 0G
`
`
`
`
`
`
`
`
`125- 8-96 +
`
`16640:
`
`33°
`
`
`rn
`
`[
`
`
`
`+49) BO) VUNGWB
`1 48074545600
`
`
`i/t gas ore a4
`
`.)
`
`
`to
`cm
`
`
`
`
`
`
`
`
`‘
`
`
`
`
`
`SUMMARY O}SHL INVENTION
`
`
`
`
`
`sr
`
`
`
`The present invention is a process for identifying relative movement of an
`
`
`
`
`
`
`
`
`
`
`
`
`object in an input signal, the input signal having a succession of frames, cach frame
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`having a succession of pixcls. Kor cach pixel of the input signal,
`the input signal
`is
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`smoothed using a time constant for the pixcl in order to generate a smoothed inpul signal.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Kor cach pixel in the smoothedinput signal, a binary value corresponding to the existence
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`of a significant variation in the amplitude of the pixc) between the current frame and the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`immediately previous smoothed inpul frame, and the amplitude of the vatiation. are
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`determined.
`
`Using the cxistence of & significant variation for a piven pixcl, ‘hc ime
`
`
`
`
`
`
`
`
`
`
`
`
`constant for the pixel, which is to be uscd in smoothing subsequent frames of thinput
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`signal, is modified. ‘The time constant is preferably in the form 2", and is inereased or
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`decreased by incrementing, or dcecrementing p. Yor cach particular pixe) of sh: suput
`
`
`
`
`
`
`
`
`
`
`
`
`
`signal, twa matrices are then formed: a first mairix comprising the binary vac: of a
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`subset of the pixels of the frame spatially related to the particular pixel; and a 3. cond
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`matrix comprising the amplitude of the variation of the subset of the pixcls of th. fame
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`spatially related to the particular pixel, Jn the first matrix, it is determingd woeds
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`particular pixel and the pixels along an oricnted dircetion relative to the particu a7 vixel
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`havo binary values of « particular valuc representing significant variation, and fr. such
`
`
`
`
`
`
`
`
`
`
`
`
`
`pixels, it is determined in the second matrix whether the amplitude of the pixels aicny the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`oriented direction relative to the particular pixe] varics in a known manne: dinicaiing
`
`
`
`
`
`
`
`
`
`
`
`
`
`movement
`in the osiented dircelion of the particular pixel and the pixels along the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`oriented direction relative to the particular pixel. The amplitude of the variation of the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`25~ pixels along the oricnted direction determines thevelocity of movement of the particular
`
`
`
`
`
`
`
`
`
`
`
`pixel andthe pixcls along the oricnted direction relative to the particular pixel.
`
`
`
`
`
`
`
`
`
`
`
`
`
`In cach of onc or moe domains, a histogram of the valucs distributed in the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`first and second matrices falling in cach such domain is formed, For a particular domain,
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`an area of significant variation is determined from the histogram for that domain,
`
`
`
`
`
`
`
`
`
`
`
`
`
`30=Mistogyams of the atca of significant varialion ulong coordinate axes are then formed,
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`l'rom these histograms,
`it is determined whether (here is an area in movement fo the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`particular domain. The domains are preferably sclected from the group consisting of i)
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`e
`
`@
`
`y =
`
`10
`
`
`15
`
`
`
`20
`
`
`ths
`
`aana
`
`Petitioner LG Ex-1021, 0004
`
`Petitioner LG Ex-1021, 0004
`
`

`

`RCV. VONT EPA MUENCHEN OG
`
`
`
`
`
`
`
`:25- 8-98 :
`
`
`LG" HO f
`
`
`
`+40 89 25984465:4 9 7
`33 1] 4874545t-
`
`
`
`
`
`
`
`
`
`
`
`PoTeEP GRIME IBS
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`$
`
`
`
`5
`
`
`
`10
`
`
`20
`
`
`25
`
`
`@
`
`
`
`@
`~
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`luminance,ij) speed (V),iii) oriented direction {121), iv) time constant (CO), v) hue, vi)
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`saturation, und vii) first axis (x(m)), and viii) second axis (y(m)),
`
`
`
`
`
`
`
`
`
`
`
`In onc embodiment,the first and sccond matrices are square matrices, with the
`
`
`
`
`
`
`
`
`
`
`
`
`
`same odd number of rows and columns, centered on the particular pixel.
`In this
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`embodiment, the steps of determining in the first matrix whetherthe particular pixel and
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`the pixcls along un oricnicd directionrelative to the particular pixel have binary valucs of
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`a particular value representing significant variation, and the step of determining in the
`
`
`
`
`
`
`
`
`
`
`
`
`
`sccond inatrix whether the amplitude signal varics in s predetermined criteria along an
`
`
`
`
`
`
`
`
`
`
`
`
`
`oriented direction relative to the particular pixel, comprise applying nested n x n matrices,
`
`
`
`
`
`
`
`
`
`
`
`
`where nis odd, centered on the particolar pixel to the pixels within cach ofthe first and
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`sccond matrices, The process then includes the further slop of determining the smallest
`
`
`
`
`
`
`
`
`
`
`
`
`
`nested matrix in which the amplitude signal varics along an oriented direction around the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`particular pixel.
`
`
`In an alternative embodiment, the first and second matrices are hexagonal
`
`
`
`
`
`
`
`
`
`
`
`15 matrices centered on the particular pixel. In this embodiment, the steps of determining in
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`the first matrix whether the particular pixcl and the pixcls along an oriented dircetion
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`relative to the particular pixel have binary valucs of a particular valuc representing
`
`
`
`
`
`
`
`
`
`
`
`
`
`significant variation, and the sicp of determining in the second matrix whether the
`
`
`
`
`
`
`
`
`
`
`
`
`
`amplitude signa) varies in a predetermined criteria along an oriented dircction relative to
`
`
`
`
`
`
`
`
`
`
`
`
`
`the particular pixel, comprisc applying nested hexagonal matrices of varying size centered
`
`
`
`
`
`
`
`
`
`
`
`
`on the particular pixcl to the pixels within cach of the first and sccond matrices. The
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`process then further includes determining the smallest nested matrix in which the
`
`
`
`
`
`
`
`
`
`
`
`
`amplitudesignal varics along an oricnteddirection around the particular pixcl.
`
`
`
`
`
`
`
`
`
`
`In_a stil} further embodiment of the invention, the first_and_secondmateri
`
`
`
`
`
`
`
`
`
`Jn. this
`are inverted J-shaped matrices with a single row and a single column.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`embodiment, the steps of determining in the first matrix whether the particular pixel and
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`the pixcls along anoriented direction relative to the parlicular pixel have binary values of
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`a particular value representing significant variation, and the step of determining in the
`
`
`
`
`
`
`
`
`
`
`
`
`
`second matrix whether the amplitude signal varics in & predetermined critcria along, an
`
`
`
`
`
`
`
`
`
`
`
`
`
`oriented direction relative (0 the parlicular pixcl, comprise applyingnested n x n matrices,
`
`
`
`
`
`
`
`
`
`
`
`
`
`where 1) is odd, to the single linc and the single column to defermine the smallest matrix
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`in whichthe amplitude varies on a Jine with the steepest slope and constant quantification.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`30
`
`
`
`
`Petitioner LG Ex-1021, 0005
`
`Petitioner LG Ex-1021, 0005
`
`

`

`* RCV. VON! EPA MUEACHEN OG
`
`
`
`
`
`
`
`
`:25- B-98B :
`
`‘
`
`‘
`
`16:50 ;
`
`
`
`
`+49 89 23994465: #10
`33 1 48745456-
`
`
`
`
`
`
`
`AAT/SFOAQiaFe
`
`
`rui/EPSR SEE TRY
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`G
`
`
`
`If desired, successive decreasing portions of frames of the input signat may be
`
`
`
`
`
`
`
`
`
`
`
`
`
`considered using a Mallat time-scale algorithm, and the largest of these portions, which
`
`
`
`
`
`
`
`
`
`
`
`
`
`provides displacement, speed and orientation indications compatible with the valuc of p,
`
`
`
`
`
`
`
`
`
`
`
`
`is sclected,
`
`
`
`
`
`
`
`In a process of smoothing an inputsignal, for cach pixel of the input signal, i)
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`is sinoothed using a time constant (CQ) for that pixcl, thereby penerauing a
`the pixel
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`smoothed pixe) valuc (I.0), ii) it is delermined whether {here exists a significant variation
`
`
`
`
`
`
`
`
`
`
`
`
`between suchpixed and the sane pixe) in a previous frame, and ili) the time constant (CO)
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`for suchpixel to be used in smoothing the pixel in subsequent flames of the input signalis
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`modified based uponthe existence or non-existence of w significant variation.
`
`
`
`
`
`
`
`
`
`
`The step of determining the existence of a significant variation fo 4 given
`
`
`
`
`
`
`
`
`
`
`
`
`pixel preferably comprises determining whether the absolute value of the differences (AB)
`
`
`
`
`
`
`
`
`
`
`
`
`between the piven pixc} valuc (1) and the value of such pixcl in a smoothed poor tame
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`(14) exceeds a threshold (SK). The step of smoothing the input signal
`rreierably
`
`
`
`
`
`
`
`
`
`
`
`
`
`comprises, for cach pixel, i) modifying the me constant (CQ)for pixct such basse .20n
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`the existence of a significant variation as determined in the priorstep, and ii) deter uiv ing
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`a smoothed value for the pixel (1.0) as follows:
`
`
`
`
`
`
`
`
`
`
`P)- hd
`LO =: Li 4 Co
`
`
`
`
`
`
`
`
`Yime constant (CO)is preferably in the form 2", and pis incrementedin the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`event that AB<SEand decremented in (he event AB?SE.
`
`
`
`
`
`
`
`
`
`In-this-proccss,the-systems—penerates-_an_outpul_signalcomprising, for cach
`
`
`
`
`
`
`
`pixel, a binary value (DP) indicating the existence or non-existence of a significant
`
`
`
`
`
`
`
`
`
`
`
`
`
`variation, and the valuc of the time constant (CO). ‘The binary valucs (OP) and the time
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`constants (CQ) are preferably stored in a memory sized {o correspondto the franic size.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`A process for identifying an arca in relative movement in ay input signal
`
`
`
`
`
`
`
`
`
`
`
`
`
`includesthe steps of
`
`
`
`
`generating a first arrny indicative of the existence of sipnificant variation in
`
`
`
`
`
`
`
`
`
`
`
`
`the magnitude of cach pixcl between a current frame and a prior frame,
`
`
`
`
`
`
`
`
`
`
`
`pencrating a sccond array indicative of the magnitude of significant variation
`
`
`
`
`
`
`
`
`
`
`
`of cach pixel between the current frame and a prior frame;
`
`
`
`
`
`
`
`
`
`
`
`
`
`@
`
`
`>
`
`
`
`5
`
`
`
`10
`
`
`
`15
`
`
`20
`
`
`25
`
`
`30
`
`
`
`enE
`
`Petitioner LG Ex-1021, 0006
`
`Petitioner LG Ex-1021, 0006
`
`

`

`zcv. VON: EPA MUENCHEN 0G
`
`
`
`
`
`
`
`195- 8-98 :
`
`
`I6:H1 :
`
`
`
`
`+19) BO BUGUE465. #11
`33 1 4874515t-
`
`POT/EP S&F HE SR"
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`7
`
`
`
`@
`
`
`
`5
`
`
`
`10
`
`
`
`15
`
`
`20
`
`
`establishing a first moving matrix comered on a pixel under consideration and
`
`
`
`
`
`
`
`
`
`
`
`coinprising pixels spatially related 10 the pixel under consideration, the first moving
`
`
`
`
`
`
`
`
`
`
`
`
`matrix (raversing {hefirst array for considcration of cach pixel ofthe curremt frame; and
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`determining whetherthe pixel under consideration and cach pixel of the pixels
`
`
`
`
`
`
`
`
`
`
`
`
`spatially related to the pixel under considcration along an oriented direction relative
`
`
`
`
`
`
`
`
`
`
`
`
`thereto within the first matrix arc a particular value representing the presence of
`
`
`
`
`
`
`
`
`
`
`
`
`
`significant variation, and if so, cstablishing in a sccond matrix within the first matrix,
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`centered on the pixel under consideration, and determining whetherthe amplitude of the
`
`
`
`
`
`
`
`
`
`
`
`
`
`pixels in the second matrix spatially related to the pixel under consideration along an
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`oriented direction relative thereto are indicative of movement along such oriented
`
`
`
`
`
`
`
`
`
`
`
`dircetion, the amplitude ofthe variation ‘along the oricnted dircetion being indicative of
`
`
`
`
`
`
`
`
`
`
`
`
`
`the velocily ofmovement, the size ofthe second matrix being variedto identify the matrix
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`size most indicative of movement.
`
`
`
`
`
`‘The process Jurther comprises, in #t least onc domain selected from the group
`
`
`
`
`
`
`
`
`
`
`
`
`
`consisting of i) luminance, ii) speed (V), ili) oricnied direction (D1), iv) ime constant
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`(CO), ¥) hus,vi) saturation, and vii) first axis (x(n), and viii) sccond axis {y(m)), and ix)
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`data characlerized by external inputs, forming a first histogram of the values in such
`
`
`
`
`
`
`
`
`
`
`
`
`
`domain for pixcls indicative of movement along anoriented direction relative to the pixe)
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`under consideration. If desired, for the pixcls in the first histogram, histograms of the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`position of such pixels slong coordinate axes may be formed, and from such histograms,
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`an arca of the image mecling criteria ofthe at Jeast onc domain may bedetermined.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`A nrocess for identifying pixels in an input signal in one of a piurality of
`classesin onc ofaplurality ofdomainscomprises, ona frame-by-framebasis:
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`——___—___for-cachpixel_of-theinputsignal,analyzingthepixcLandproviding an outpul
`
`
`
`signa) for cach domain containing information to identify cach domain in which the pixel
`25
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`is classified ;
`
`
`providing a classificr for cach domain, the classifier enabling classification of
`
`
`
`
`
`
`
`
`
`
`pixcls within each domain to sclccied classes within the domain;
`
`
`
`
`
`
`
`
`
`
`providing « validation signal for the domains, the validation signal sclecting
`
`
`
`
`
`
`
`
`
`
`30—onc or moreofthe plurality of domains for processing; and
`
`
`
`
`
`
`
`
`
`
`
`
`forming a histogramfor pixels of the output signal within the classes sclected
`
`
`
`
`
`
`
`
`
`
`
`by the classifier within cach domain selected by the validation signal.
`
`
`
`
`
`
`
`
`
`
`
`
`e
`
`
`
`_|
`
`
`
`
`
`
`
`
`
`Petitioner LG Ex-1021, 0007
`
`Petitioner LG Ex-1021, 0007
`
`

`

`"RCV. VON: EPA MUENCHEN 06
`
`
`
`
`
`1BB- BBG 5
`
`1GIBL
`
`33 | 48745456-
`+49 B9 29991465: 412
`
`
`
`eTsEP Ge {HE 983
`
`
`
`§
`
`
`
`
`
`The process further includes the steps of forming histograms along coordinate
`
`
`
`
`
`
`
`
`
`
`
`axes for the pixels within the classcs selected by the classifier within cach domuin
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`sclected by the validation signal, and forming a composite signal corresponding to the
`
`
`
`
`
`
`
`
`
`
`
`
`
`spatial position of suchpixels within the frame. Pixels falling within limits /,, J, 4. J in
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`the histograms along the coordinate axes arc then identified, and a composite signal from
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`the pixels falling within these limits is formed,
`
`
`
`
`
`
`
`
`A process foridentifying the velocity of movement of an areca of an input
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`signal comprises:
`
`
`for cach particular pixel of the input signal, forming a first matrix comprising
`
`
`
`
`
`
`
`
`
`
`
`
`binary values indicating the existence ar non-existence of a significant variation in the
`
`
`
`
`
`
`
`
`
`
`
`
`
`amplitude ofthe pixel signal belweon the current frame and a priorframe for a subsel of
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`the pixels of the frame spatially related {o such particular pixel, and a second matrix
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`comprising the ampliude of such variation;
`
`
`
`
`
`
`determining in the first matrix whether the particular pixel and the pixels
`
`
`
`
`
`
`
`
`
`
`
`
`along an oriented dircction relative to the particular pixe) have hinary valucs af a
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`particular valuc represcnting significant variation, and, for such pixels, determining in the
`
`
`
`
`
`
`
`
`
`
`
`
`second matrix whether the amplitudes of the pixcls along an oriented direction relutive to
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`the particular pixel vary in a known manner indicating movement of the pixcl and the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`pixels along #n oriented direction relative to the particular pixct, the amplitude of the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`variation along the oriented direction determining (he velocity of movement of the
`
`
`
`
`
`
`
`
`
`
`
`
`particularpixel.
`
`
`A process for identifying a non-moving, area in un input signal compriscs:
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`without significant variation betweenthe current frame and a prior frame; and
`
`
`
`
`
`
`
`
`
`
`
`
`forming @ composite signal corresponding16the Spatia) positionofsuch
`
`
`
`
`
`
`
`
`
`
`
`pixcls within the frame.
`.
`
`
`
`
`An apparatus for idenfying rclative movement in an input signal comprises:
`
`
`
`
`
`
`
`
`
`
`means for smoothing the input signal using a time constant for cach pixel,
`thereby
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`generating a smoothed inpul signal;
`
`
`
`
`means for deicrmining for cach pixel in the smoothed input signal a binary
`
`
`
`
`
`
`
`
`
`
`
`
`
`value corresponding to the existence of a significant variation in the amplitude of the
`
`
`
`
`
`
`
`
`
`
`
`
`
`pixel signal between the current frame and the immediately previous smoothed input
`
`
`
`
`
`
`
`
`
`
`
`
`frame, and for determining the amplitudeofthe variation;
`
`
`
`
`
`
`
`
`
`
`@
`:
`
`
`
`®
`
`5
`
`
`
`10
`
`
`
`
`15
`
`
`
`20
`
`
`25
`
`
`30
`
`
`forming histograms along coordinatc axes for pixcls of the input signal
`
`
`
`
`
`cannerrc
`
`Petitioner LG Ex-1021, 0008
`
`Petitioner LG Ex-1021, 0008
`
`

`

`RCV. SON:EPA MUENCHEN 06
`
`
`
`
`
`
`
`7;26- 8-98 ;
`
`
`16:52 :
`
`
`
`
`33 ! 48745456—
`+49 8Y 29994465: #123
`
`
`
`
`
`
`_.
`AGT
`FARARS
`act erp ga sORAB
`
`
`
`
`
`
`
`
`
`
`
`-
`
`9
`
`
`
`means for using the existence of a significant variation for a given pixcl to
`
`
`
`
`
`
`
`
`
`
`
`
`
`modify the time constant for the pixcl io be used in smoothing subsequent frames of the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`input signal;
`
`
`means for forming a first matrix comprising the binary valucs of a subset of
`
`
`
`
`
`
`
`
`
`
`
`
`
`the pixels of the frame spatiallyrelatedto cach particular pixel, and for forming a second
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`matrix comprising the amplitude of the variation of the subset of the pixels of the frame
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`spatially related to such particular pixel;
`
`
`
`
`
`
`means for determining in the first matrix a particular area in whichthe binary
`
`
`
`
`
`
`
`
`
`
`
`
`
`value for each pixc) is a particular vahic representing’ significant vayiation, and,for such
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`particular aroa, for determining in the second matrix whether the amplitude varics along
`
`
`
`
`
`
`
`
`
`
`
`
`
`an oricnted dircetion relative to the particular pixe) in a known manner indicating
`
`
`
`
`
`
`
`
`
`
`
`
`
`movementof the pixel in the oriented direction, the amplitude of the variation along the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`oriented direction determining the velocity of movement of the pixel.
`
`
`
`
`
`
`
`
`
`Anapparatus for smoothing an input signa) comprises:
`
`
`
`
`
`
`
`
`means for smoothing cach pixcl of the input signal using a time constant (CO)
`
`
`
`
`
`
`
`
`
`
`
`
`for such pixel, thereby generating a smoothed pixel value (1.0) ;
`
`
`
`
`
`
`
`
`
`means for determining the existence of a significant variation for a given
`
`
`
`
`
`
`
`
`
`
`
`pixel, and modifying the time constant (CO) for the ‘pixel 1o be used in smoothing the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`pixel in subscguent frames ofthe input signal based upon the existence of suchsignificant
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`~—-varintion.
`
`
`An apparatus for identifying an area ju relative movement in an input signal
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`comprises:
`
`
`means for pencrating # first.atray indicative of the existence of significant
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`variation in the magnitude of cach pixel between a current frame and a prior frame;
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`nicans for pencrating a second array indicative of the magnitude of significant
`
`
`
`
`
`
`
`
`
`
`variation of cachpixel hetween the current frame and & prior frame;
`
`
`
`
`
`
`
`
`
`
`
`means for cstablishing a: first moving matrix centered on a pixel under
`
`
`
`
`
`
`
`
`
`
`
`
`consideration and comprising pixels spatially related to the pixel under consideration, the
`
`
`
`
`
`
`
`
`
`
`
`
`first moving matrix traversingthe first array for consideration ofeach pixel of the current
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`frame;
`
`
`@:
`
`e
`
`5
`
`
`
`10
`
`
`
`15
`
`
`
`20
`
`
`25
`
`30
`
`
`
`means for determining whether the pixcl under consideration and cach pixel
`
`
`
`
`
`
`
`
`
`
`
`along an oriented dircction relative to the pixel under consideration within the first matrix
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`is a particular value represcnting the proscnce of significant variation, and if so, for
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Petitioner LG Ex-1021, 0009
`
`Petitioner LG Ex-1021, 0009
`
`

`

`
`
`*
`
`
`
`RCV. VON EPA MURNCHEN O60
`
`
`
`
`
`
`2325- 8-98 -
`‘
`
`'
`
`16:52 :
`
`B33
`
`1 48715156
`
`
`
`eye
`—ry wy
`cu
`
`
`as
`i
`
`449 BO LH 4
`
`
`
`
`“38
`eit JF
`
`10
`
`
`
`
`
`
`®
`”
`
`@,
`
`5
`
`
`10
`
`
`15
`
`
`20
`
`
`25
`
`
`30
`
`
`cstablishmp a socond matria within the first mattix, eontcred on the pixel under
`
`
`
`
`
`
`
`
`
`
`
`
`
`consideration, and for determining whether the amplitude of the pixcts in the second
`
`
`
`
`
`
`
`
`
`
`
`
`matrix arc indicative of movement slong an oricnied direction rolative to the pixel under
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`considcration, the amplitude of the variation along theoricated direction beingindicative
`
`
`
`
`
`
`
`
`
`
`
`
`of the velocity of movement, thesize of the second matrix being vinied to idemify the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`matrix size most indicative of movement.
`
`
`
`
`
`
`An apparatus for identifying pixels in an input signal m one of a pimalily of
`
`
`
`
`
`
`
`
`
`
`
`
`
`classes in one of a plurality of domains comprises:
`
`
`
`
`
`
`
`
`means for enulyzing cach piacl of the input signal and for providing an output
`
`
`
`
`
`
`
`
`
`
`
`
`
`signal for cach domain containing informationto identify cach damain in whichth: pixe)
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`|
`is Classified;
`
`
`a classifier for cach domuin, the classif