`Kuwashima et a1.
`
`[54] MOVING OBJECT MEASUREMENT DEVICE
`EMPLOYING A THREE-DIMENSIONAL
`ANALYSIS TO OBTAIN CHARACTERISTICS
`OF THE MOVING OBJECT
`
`[76]
`
`[21]
`[22]
`[36]
`
`[37]
`
`[30]
`
`Inventors: Shigesumi Kuwashima. 2-26-2-406
`Chidori. Ohta-Ku. Tokyo 152; Masao
`Shimizu. 3-29-503 Kosugi-cho.
`Nakabara-ku. Kawasaki. Kanagawa 211;
`Torn Nakamura. 838-39 Ishibayashi.
`Nishinasuno-cho. Nasu-gun; Dairoku
`Sekiguchi. 2-23-3 Kalrinokizaka.
`Meguro-ku. Tokyo 152. all of Japan
`Appl. N0.:
`381,864
`PCT‘ Filed:
`Jun. 10, 1994
`
`PCI‘ N0.:
`
`PCT/JP94/00934
`
`§ 371 Date:
`
`Jun. 27, 1995
`
`§ 102(e) Date: Jun. 27, 1995
`PCI‘ Pub. N0.: WO94/29670
`
`PCT Pub. Date: Dec. 22, 1994
`Foreign Application Priority Data
`
`Jun. 10, 1993
`Jun. 10. 1993
`Oct. 19. 1993
`
`[JP]
`[JP]
`[JP]
`
`5-138805
`Japan ..
`5-139629
`Japan .
`Japan .................................. .. 5-261223
`
`[51] Int. Cl.6 ............................ .. G06K 9/00; G06K 9/36;
`G06K 9/32; G03B 13/18
`[52] U.S. C1. ........................ .. 382/107; 382/285; 382/291;
`354/402
`[58] Field of Search ................................... .. 382/107. 285.
`382/291. 302; 354/402
`
`[56]
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`USOO5764786A
`[11] Patent Number:
`[45] Date of Patent:
`
`5,764,786
`Jun. 9, 1998
`
`5,243,418
`5,422,700
`5.515.448
`
`9/1993 Kuno et a1. ........................... .. 382/107
`6/1995 Sada et a1‘
`~
`5/1996 Nishitani ............................... .. 382/107
`
`FOREIGN PATENT DOCUMENTS
`
`50-148024 1l/1975 Japan ............................. .. H04N 5/24
`
`57-201914 12/1982 Japan . . . . . .
`61-45910
`3/1986 Japan .... ..
`63-261102 lO/l988 Japan .... ..
`
`. . . .. GOSD 3/00
`G01B 11/00
`.. G01B 11/00
`
`. . . .. GOSD 3/12
`l-263703 10/1989 Japan . . . . . .
`H04N 5/232
`2-140073 5/1990 Japan .... ..
`GOGF 15/70
`4-1473 H1992 Japan .... ..
`G01B 11/00
`4-95806 3/1992 Japan .... ..
`H04N 5/232
`4-117077 4/1992 Japan .... ..
`G06F 15/70
`4-184577
`7/1992 Japan .... ..
`4-329790 11/1992 Japan ............................ .. GOGF 15/70
`
`Primary Examiner—Andrew W. Johns
`Assistant Examiner-Monica S. Davis
`Attorney Agent, or F irm—Cushrnan Darby & Cushman I?
`Group of Pillsbury Madison & Sutro LLP
`
`[57]
`
`ABSTRACT
`
`The present invention de?nes the area where the targeted
`moving object can move in the three-dimensional real
`spatial coordinate system. then maps it to the two
`dimensional area corresponding to the capturing method and
`the capturing range of the capturing device. and by limiting
`the image processing for obtaining the coordinate to that
`area. obtains the coordinate of the moving object in the
`three-dimensional real space rapidly by using the signal
`which is capturing the moving object. Furthermore. the
`present invention is able to control the capturing direction
`and the capturing range which correspond to the movement
`of the capturing target. to overlap the image signal with the
`data which is always changing by the capturing condition
`and the movement of the measurement data. to improve the
`detection precision. and to track automatically and smoothly.
`
`5,055,926 10/1991 Christensen et al. ................. .. 382/107
`
`38 Claims, 34 Drawing Sheets
`
`Camera Head/Calculation Device
`
`11
`
`Upper
`Calculation
`Device
`
`12
`i
`
`(27
`image
`Ccmpositio
`
`Dem/rice
`
`28
`
`~13
`
`l/O
`
`22
`
`Zoom
`Lens
`213
`i
`I
`Universal
`Head
`
`'
`
`2f’
`Image
`Processing
`Device
`
`Location
`Calculation
`Drive
`Device
`Control F:—->
`Device
`
`\
`
`25
`
`x
`
`26
`
`\‘Camera Head /Ca1culation Device
`
`12
`‘1
`
`_—
`
`Page 1 of 60
`
`SAMSUNG EXHIBIT 1015
`Samsung v. Image Processing Techs.
`
`
`
`US. Patent
`
`Jun. 9, 1998
`
`Sheet 1 0f 34
`
`5,764,786
`
`11
`
`Upper
`Calculation
`Device
`
`238
`
`N13
`
`1 2
`i
`
`$27
`Image
`Compositione
`Device
`'l
`
`I/O
`
`Location
`Calculation 6
`Device
`
`i
`26
`
`1 2
`3
`
`Camera Head/Calculation Device
`
`22
`
`Zoom
`Lens
`
`21
`3
`TV
`camera
`‘
`
`2§3
`
`Universal
`Head
`
`24
`5
`
`Image
`P
`-
`rocessmg
`Device
`
`Drive
`Control
`Device
`S
`25
`
`Camera Head/Calculation Device
`
`Fig. 1
`
`SAMSUNG EXHIBIT 1015
`Page 2 of 60
`
`
`
`US. Patent
`
`Jun. 9, 1998
`
`Sheet 2 of 34
`
`5,764,786
`
`Upper Calculation Device
`
`Location Calculation Device
`
`Define the area in the
`three—dimensional real
`spatial coordinate system
`
`Define the
`three — dimensional
`velocity vector
`
`Notify each location
`calculation device
`
`l
`
`:
`
`Map the three-dimensional
`area and the velocity vector
`to two — dimension
`
`Define it in the image
`processing device
`
`an input signal of
`location calculation
`
`Calculate the three
`dimensional coordinate of
`the moving object
`
`I
`
`Fig. 2
`
`SAMSUNG EXHIBIT 1015
`Page 3 of 60
`
`
`
`US. Patent
`
`Jun. 9, 1998
`
`Sheet 3 0f 34
`
`5,764,786
`
`E III!
`
`ww‘llll
`
`mm A
`
`DmO
`
`Nm
`
`ow (1
`
`a (I
`
`SEE
`
`vm
`
`E9
`mm f 5 l
`an
`at
`
`v50
`
`h
`
`mm
`M l
`
`MAO .11.
`2% .l
`
`Eu<i>
`
`v50
`
`mm
`
`wmm
`
`mm
`
`_m 1
`
`MAO k
`
`mm?»
`
`E500
`
`mm
`
`m .mwm
`
`cm
`
`SAMSUNG EXHIBIT 1015
`Page 4 of 60
`
`
`
`US. Patent
`
`Jun. 9, 1998
`
`Sheet 4 of 34
`
`5,764,786
`
`~21
`
`61
`
`] 1
`I
`
`22
`
`Fig. 4a
`
`61
`
`22
`
`Own
`
`61
`
`21
`
`~62
`
`Fig. 4b
`
`Fig. 4c
`
`SAMSUNG EXHIBIT 1015
`Page 5 of 60
`
`
`
`US. Patent
`
`9
`
`5,764,786
`
`mp
`
`35:083:00mag:85?»
`
`
`Commu®>COU
`
`
`
`Gofimoofimwx<Eofium>
`
`580cmBmc<
`
`m35:065:0065:00582mE>CQ
`65:8mun/w283:5
`
`85$thmE>CQ“802
`
`mmmmm#6035ch<1,35:0#obcoomugEEONCOE
`
`
`mmx<EEONCOIHm
`
`
`
`mugEEONCOE
`
`
`
`us.$386MobcooMEEOONSOONmaS.552mE>CD
`MSImmMMagnum
`
`
`
`
`mafiaSOON
`
`m.mE
`
`882
`
`SAMSUNG EXHIBIT 1015
`
`Page 6 of 60
`
`SAMSUNG EXHIBIT 1015
`Page 6 of 60
`
`
`
`
`
`
`US. Patent
`
`Jun. 9, 1998
`
`Sheet 6 0f 34
`
`5,764,786
`
`horizontal direction address
`
`vertical direction address
`
`\ 101/ /, /
`
`102 103
`
`Fig. 6
`
`SAMSUNG EXHIBIT 1015
`Page 7 of 60
`
`
`
`US. Patent
`
`Jun. 9, 1998
`
`Sheet 7 of 34
`
`5,764,786
`
`
`
`Ux
`
`Fig. 8
`
`SAMSUNG EXHIBIT 1015
`
`Page 8 of 60
`
`SAMSUNG EXHIBIT 1015
`Page 8 of 60
`
`
`
`US. Patent
`
`Jun. 9, 1998
`
`Sheet 8 0f 34
`
`5,764,786
`
`Upper Calculation Device
`
`Location Calculation Device
`
`Define the
`predicted motion
`da :a
`
`Notify the location
`calculation device
`
`2
`
`Calculate the capturing
`direction and the range
`
`l
`
`Define the calculation result as
`the objective value to the drive
`control device
`
`Fig. 9
`
`Upper Calculation Device
`
`Location Calculation Device
`
`Define the three—dirnensional
`movement velocity vector of
`which the moving object can have
`
`Notify the location
`calculation device
`
`l
`
`Map the three — dimensional
`movement velocity vector to
`two — dimensional
`
`Limit the change of capturing
`direction which is faster than the
`two — dimensional movement
`velocity vector
`l
`Define the objective value to
`the drive control device
`
`Fi
`g. 10
`
`SAMSUNG EXHIBIT 1015
`Page 9 of 60
`
`
`
`US. Patent
`
`Jun. 9, 1998
`
`Sheet 9 of 34
`
`5,764,786
`
`Image Processing Device
`
`Location Calculation Device
`
`Obtain the geometric
`center of gravity of the
`moving object
`
`Notify the location
`calculation device of the :>
`obtained geometric center
`of gravity
`1/
`
`Calculate the capturing
`direction and the
`capturing range
`J,
`Define the calculation result
`as the objective value to the
`drive control device
`
`i
`
`Fig. 11
`
`Image Processing Device
`
`Location Calculation Device
`
`Obtain the geometric
`center of gravity of the
`moving object
`
`Notify the location
`calculation device of the 13:)
`obtained geometric center
`of gravity
`1/
`
`Obtain the real
`Spatial Coordinate of
`the moving Object
`
`Define the objective value to the
`drive control device based on the
`obtained coordinate
`
`i
`
`Fig. 12
`
`SAMSUNG EXHIBIT 1015
`Page 10 of 60
`
`
`
`US. Patent
`
`Jun. 9, 1998
`
`Sheet 10 of 34
`
`5,764,786
`
`Upper Calculation
`Device
`
`Location Calculation
`Device
`
`Image Processing
`Device
`
`Define the three—
`dimensional movement
`velocity vector of
`which the moving
`object can have
`
`Notify the
`location
`calculation device
`
`Map the three
`dimensional
`movement velocity
`vector to
`two — dimension
`
`Obtain the real
`spatial coordinate of
`the moving object
`
`Define the objective
`value based on the
`obtained coordinate
`\l/
`Limit the change of
`capturing direction
`which is faster than
`the two—dirnensional
`movement velocity
`vector
`\l/
`Define the objective
`value to the drive
`control device
`
`Fig. 13
`
`Obtain the
`geometric center of
`gravity of the
`moving object
`i
`Notify the location
`calculation device of
`the obtained
`geometric center of
`gravity
`i
`
`SAMSUNG EXHIBIT 1015
`Page 11 of 60
`
`
`
`US. Patent
`
`Jun. 9, 1998
`
`Sheet 11 of 34
`
`5,764,786
`
`Image Processing Device Location Calculation Device
`
`Obtain the geometric
`center of gravity of
`the moving object
`4/
`Notify the location
`calculation device of
`the obtained geometric
`center of gravity
`
`A Calculate the motion
`I’ data of
`the moving Object
`
`s it Within a
`capturing range‘?
`
`Record the motion
`data
`
`Calculate the
`capturing
`direction and the
`range based on
`the motion data
`
`Calculate the
`capturing
`direction and the
`range based on
`the past motion
`data
`
`Define the
`calculation result
`as the objective
`value to the drive
`control device
`
`i
`
`Fig. 14
`
`SAMSUNG EXHIBIT 1015
`Page 12 of 60
`
`
`
`US. Patent
`
`,
`
`Jun. 9, 1998
`
`Sheet 12 of 34
`
`5,764,786
`
`24 f- 1
`Image Processing
`Device
`Image Processing
`Device
`24—2
`24 7 '
`Image Processing
`Device
`Image Processing
`Device
`
`24
`Fig. 15
`
`Data
`7 Selector
`
`fin
`
`Fig. 16
`
`SAMSUNG EXHIBIT 1015
`Page 13 of 60
`
`
`
`US. Patent
`
`Jun. 9, 1998
`
`Sheet 13 of 34
`
`5,764,786
`
`Z
`
`coordinate of the measurement target
`
`W t
`
`>X
`
`Fig. 17a
`
`Z 1;
`
`direction control of the camera
`
`‘ t
`
`> X
`
`Fig. 17b
`
`f
`132 \\
`
`X/IBI
`M/136
`
`133\
`. a‘
`134 \N\
`
`’/,137
`L_//138
`
`135\\\1;/~V139
`
`k
`
`J
`
`SAMSUNG EXHIBIT 1015
`Page 14 of 60
`
`
`
`US. Patent
`
`Jun. 9, 1998
`
`Sheet 14 0f 34
`
`5,764,786
`
`location of the measurement target
`
`v’
`i=1 £2
`timgt
`
`Fig. 19a
`
`area of the measurement target
`
`SAMSUNG EXHIBIT 1015
`Page 15 of 60
`
`
`
`US. Patent
`
`Jun. 9, 1998
`
`Sheet 15 0f 34
`
`5,764,786
`
`Upper Calculation Device
`
`Location Calculation Device
`
`l
`
`Define the shape and the
`motion data to compare in
`the three-dimensional real
`spatial coordinate sysytem
`Map the shape and the motion
`J,
`Notify the location I; data to campare on the two—
`Calculation device
`dimensional area,corresponding
`‘L
`to the capturing area
`
`Detect the capturing
`area of TV camera
`
`Output the mapped data to
`image composition device
`
`Fig. 20
`
`Upper Calculation Device
`
`Location Calculation Device
`l
`Detect the capturing
`area of TV camera
`
`Estimate the measurement
`precision of the motion data
`in the three — dimensional
`real spatial coordinate system
`based on the location of TV
`camrea
`
`‘1/
`
`Notify the location
`calculation device
`‘L
`
`<1:
`
`Notify the upper
`calculation device
`
`calcumte motion data
`
`___-_->
`
`Map the measurernrnt
`precision to the two
`dimensional area
`corresponding to the
`capturing area
`
`Output the motion data and
`the mapped data to image
`composition device
`Fig. 21
`i
`
`SAMSUNG EXHIBIT 1015
`Page 16 of 60
`
`
`
`US. Patent
`
`Jun. 9, 1998
`
`Sheet 16 of 34
`
`5,764,786
`
`Upper Calculation Device
`
`Location Calculation Device
`
`Detect the capturing
`range of TV camera
`
`Notify the location
`calculatlon device
`
`Calculate the mutual
`relation of capturing
`area of each TV camrea
`
`Calculate the capturing
`Notify each location
`calculation device of the ::_> area of other TV camfara
`mutual relation
`to the TV Camera Whwh
`‘1’
`it manages
`
`Output the calculation
`result to the image
`composition device
`
`i
`
`Fig. 22
`
`SAMSUNG EXHIBIT 1015
`Page 17 of 60
`
`
`
`US. Patent
`
`Jun. 9, 1998
`
`Sheet 17 of 34
`
`5,764,786
`
`1l~ Upper
`Calculation
`Device
`
`12
`927
`
`28
`i
`
`N13
`
`Camera Head/ Calculation Device
`4
`Image
`Composition 9
`Device
`
`21
`6
`
`22
`g
`" TV
`Zoom
`Lens _ Camera
`
`Electric
`Universal
`Head
`
`5
`23
`
`Drive
`Control
`Device
`
`a
`:
`'
`
`254
`
`Image
`Processinga
`Device
`
`1/0
`
`0
`Location
`.2
`_ 6
`0 Calculation
`‘5
`DEVICE
`Image ‘
`Processinga *3
`Device
`g
`1
`
`s
`26
`
`s
`24
`
`;
`'
`
`141
`
`'P
`
`Camera Head/ Calculation Device
`
`12
`
`Fig. 23
`
`SAMSUNG EXHIBIT 1015
`Page 18 of 60
`
`
`
`US. Patent
`
`Jun. 9, 1998
`
`Sheet 13 of 34
`
`5,764,786
`
`EN
`
`3% E828
`
`Mrw
`
`Mm
`
`mm
`
`Lwm mm
`mmm Hm
`
`ill
`
`v30
`H H H
`
`l'
`mg _m
`mm
`H
`
`m2<
`v30
`
`E300
`
`mm
`
`wm
`
`v30
`
`Ea;
`
`v50
`
`SAMSUNG EXHIBIT 1015
`Page 19 of 60
`
`
`
`US. Patent
`
`‘ Jun. 9, 1998
`
`Sheet 19 of 34
`
`5,764,786
`
`SAMSUNG EXHIBIT 1015
`Page 20 of 60
`
`
`
`US. Patent
`
`Jun. 9, 1998
`
`Sheet 20 of 34
`
`5,764,786
`
`171
`
`172 173
`
`E33]
`
`Fig. 263
`
`
`
`SAMSUNG EXHIBIT 1015
`
`Page 21 of 60
`
`SAMSUNG EXHIBIT 1015
`Page 21 of 60
`
`
`
`US. Patent
`
`Jun. 9, 1998
`
`Sheet 21 of 34
`
`5,764,786
`
`
`
`Capture the location data
`and the area data of
`itself
`
`
`
`Capture the location data and
`the area data of other devices
`
`
`
`Dalete the location data of
`
`Which area data is increasing
`
`Average the rest
`
`location data
`
`Is the
`
`
`
`
`area data of
`
`increasing?
`
`
`the narrowest valid range
`
`
`Add a flag
`
`
`
`to the centroid
`
`
`
`data which indicate
`
`
`
`
`low reliability
`
`Next process
`
`Fig. 27
`
`SAMSUNG EXHIBIT 1015
`
`Page 22 of 60
`
`SAMSUNG EXHIBIT 1015
`Page 22 of 60
`
`
`
`US. Patent
`
`Jun. 9, 1998
`
`Sheet 22 of 34
`
`5,764,786
`
`Location Calculation
`
`Location Calculation
`
`Device
`
`DeVice ............ . .
`
`
`
`
`Capture the location
`
`data and the area
`
`data from the image
`
`processing device
`
`
`
`
`
`
`data and the area
`
`
`Capture the location
`
`
`data from the image
`
`rocessin device
`
`
`
`
`
`oes
`
`the area
`
`change
`
`suddenly ?
`
` the area
`change
`suddenly ‘7
`
`
`
`
`
`Send data to a
`
`stated location
`
`
`
`calculation device
`
`
`
`
`other location
`
`
`calculation device
`
`
`Calculate the real
`spatial coordinate of
`
`the moving object
`
`Notify other location
`
`calculation devices
`
`
`
`
`
`
`coordinate and its
`
`reliability
`
`
`of
`
`the real spatial
`
`Fig. 28
`
`SAMSUNG EXHIBIT 1015
`
`Page 23 of 60
`
`
`
`Receive data from
`
`SAMSUNG EXHIBIT 1015
`Page 23 of 60
`
`
`
`US. Patent
`
`Jun. 9, 1998
`
`Sheet 23 of 34
`
`5,764,786
`
`®/\/ 181
`i
`18;
`é» «\a
`
`185
`
`Fig. 29
`
`184M ii
`
`Fig. 303
`
`Fig. 30b
`
`Fig. 30c
`
`area
`
`
`
`SAMSUNG EXHIBIT 1015
`
`Page 24 of 60
`
`SAMSUNG EXHIBIT 1015
`Page 24 of 60
`
`
`
`US. Patent
`
`Jun. 9, 1998
`
`Sheet 24 of 34
`
`5,764,786
`
`12
`
`moving
`image
`
`191
`
`moving
`image
`
`192 moving
`image
`
`
`
`
`three - dimensional
`
`data
`
`Fig. 32
`
`SAMSUNG EXHIBIT 1015
`
`Page 25 of 60
`
`SAMSUNG EXHIBIT 1015
`Page 25 of 60
`
`
`
`US. Patent
`
`Jun. 9, 1998
`
`Sheet 25 of 34
`
`5,764,786
`
`201 camera head
`
`
`
`Record the image with the three—dimensional
`
`location information
`
`Fig. 33
`
`SAMSUNG EXHIBIT 1015
`
`Page 26 of 60
`
`SAMSUNG EXHIBIT 1015
`Page 26 of 60
`
`
`
`US. Patent
`
`Jun. 9, 1998
`
`Sheet 26 of 34
`
`5,764,786
`
`
`
`3950onEmmE>oE
`
`F-—...........___.......__..__
`
`E:QO
`
`.$52000
`
`33:08%mm.1!_
`
`IIIIJ.
`
`.Il.ul.lt.ll.ll.l...l..|.|!|.l_
`
`.
`
`common
`
`Etcoo
`
`uofioo.>_«mm-_
`_IIII
`
`.HUmHOQ
`
`SAMSUNG EXHIBIT 1015
`
`Page 27 of 60
`
`SAMSUNG EXHIBIT 1015
`Page 27 of 60
`
`
`
`
`
`US. Patent
`
`Jun. 9, 1998
`
`Sheet 27 of 34
`
`5,764,786
`
`
`
`
`Offset Z
`
`CCD plane
`
`vertical axis
`
`Gjhorizontal axis
`
`Fig. 35
`
`Offset X
`
`negativef
`
`positive
`
`
`
`mam pomt
`
`capturing
`
`center 0
`(center of
`lens)
`
`Optical
`. aXiS
`SUbJeCt
`
`Fig. 36
`
`negative lens positive
`
`
`
`Optical
`axis
`
`subject
`
`mam pomt
`
`Fig. 37
`
`SAMSUNG EXHIBIT 1015
`
`Page 28 of 60
`
`SAMSUNG EXHIBIT 1015
`Page 28 of 60
`
`
`
`US. Patent
`
`Jun. 9, 1998
`
`Sheet 28 of 34
`
`5,764,786
`
`The measurement
`coordinate of
`the Z
`
`image location
`
`measurement device
`
`.
`_
`maln pomt
`
`constraint plane
`
`CCD
`
`TV plane
`
`
`
`targeted object
`
`Fig. 38
`
`X
`
`
`
`constraint plane
`
`TV plane
`
`
` projection
`
`subject
`
`center
`
`X
`
`attention vector
`
`Fig. 39
`
`2‘
`
`vertical
`
`-
`axis
`attention
`
`
`‘ constraint plane
`
`
`
`rotation of
`uniyersal
`head
`
`SAMSUNG EXHIBIT 1015
`
`Page 29 of 60
`
`SAMSUNG EXHIBIT 1015
`Page 29 of 60
`
`
`
`US. Patent
`
`Jun. 9, 1998
`
`Sheet 29 of 34
`
`5,764,786
`
`constraint plane
`
`constraint plane
`
`_X
`I
`A and B can be obtained
`by using arc tan,
`arc cos respectively
`
`
`
` constraint plane
`‘
`‘ él-ocationI where subject
`
`yew
`should be mapfed
`
`CCD plane
`
`
`TV plane
`
`
`
`
`
`X
`
`subject
`
`TV plane
`
`Fig.42
`
`SAMSUNG EXHIBIT 1015
`
`Page 30 of 60
`
`SAMSUNG EXHIBIT 1015
`Page 30 of 60
`
`
`
`US. Patent
`
`Jun. 9, 1998
`
`Sheet 30 of 34
`
`5,764,786
`
`IOCfition 0f
`IJFOJectlonmcenter
`
`Value of D/A vs. location
`of projection center
`
`
`
`—C2)000-1500 —1000 —500
`
`500 1000 1500 2000
`
`0
`D/ A
`Flg. 43
`
`the location Value of D/ A vs. inverse of
`inverse of
`of projection center
`location of projection center
`0.6
`
`0.4
`
`0.3
`
`0.5
`
`
`—2000-1500 —1000 —500
`
`500 1000 1500 2000
`
`0
`D/ A
`
`Fig. 44
`
`SAMSUNG EXHIBIT 1015
`
`Page 31 of 60
`
`SAMSUNG EXHIBIT 1015
`Page 31 of 60
`
`
`
`US. Patent
`
`.
`
`Jun. 9, 1998
`
`Sheet 31 of 34
`
`5,764,786
`
`value of D/A Inverse of distance vs. value of D/A
`1000
`
`500
`
`-500
`
`—1000
`
`-1500
`
`—2000
`0.001
`
`'
`0.002
`
`‘
`0.003
`
`'
`0.004
`
`0.005
`
`0.006
`
`0. 007
`
`Fig. 45
`
`SAMSUNG EXHIBIT 1015
`
`Page 32 of 60
`
`SAMSUNG EXHIBIT 1015
`Page 32 of 60
`
`
`
`US. Patent
`
`Jun. 9, 1998
`
`Sheet 32 of 34
`
`5,764,786
`
`
`
`time(sec)
`
`2048)
`
`Fig.46
`
`0~3(1538-5
`
`1'20
`
`100
`
`080
`
`0.60
`
`0Q0
`
`019
`
`0£00.00
`
`SAMSUNG EXHIBIT 1015
`
`Page 33 of 60
`
`SAMSUNG EXHIBIT 1015
`Page 33 of 60
`
`
`
`US. Patent
`
`Jun. 9, 1998
`
`Sheet 33 of 34
`
`5,764,786
`
`Start
`
`Define the location
`measurement device
`
`Oms
`
`Obtain the information of
`target object from location
`measurement device
`
`
`
`
`
`Read potentiometer
`
`the location value
`Output
`decided by PD) control
`method to servo circuit
`
` No.4
`
`Calculate the Spatial
`coordinate of targeted object
`
`
`
`targeted object
`in the image for
`not
`targeted object on the
`
`
`
`
`
`Predict the location of targeted
`object where it will be when
`the next spatial location is
`caluculated (four points)
`
`
`
`
`
`Define the rotation angle
`of two axes of the
`universal head so that
`the targeted object can
`be searched for
`
`Predict the movement of
`the targeted object based
`on the movement of the
`targeted object so far
`
`
`
`
`
`
`
`Set
`
`the appropriate
`mom value
`
`Decide the direction of
`
`camera based on the
`
`
`predicted location of
`
`
`targeted object and
`calculate the rotation
`
`angle of two axes of
`universal head
`calculated rotation
`
`
`angle of the universal head
`
`larger than the dynamic
`
`characteristic
`
`
`Define the direction of camera
`based on the predicted location
`of targeted object and calculate
`the rotation angle of two axes
`of the universal head
`
`
`
`
`
`
`
`
`
`Set
`
`the zoom wider
`
`
`
`
`Set the zoom value wider
`so that
`the targeted
`object can be caputured
`
`
`
`
`the zoom value by using the
`Set
`location of the targeted object
`in the
`image and the distance to the
`targeted object so that
`the targeted
`object is captured with a defined size
`
`
`
`0
`
`Fig. 47
`
`SAMSUNG EXHIBIT 1015
`
`Page 34 of 60
`
`SAMSUNG EXHIBIT 1015
`Page 34 of 60
`
`
`
`US. Patent
`
`Jun. 9, 1998
`
`Sheet 34 of 34
`
`5,764,786
`
`Decide the focus based on the
`distance of
`targeted object
`
`Read potentiometer
`
`4ms
`
`
`
`Output the position value
`decided by MD control
`method, to servo circuit
`
` NO.1
`
`Communicate with terminal
`
`Read potentiometer
`
`8ms
`
`
`
`Output the position value
`decided by PID control
`method, to servo circuit
`
` No.2
`
`Read potentiometer
`
`12ms
`
`
`
`Output the position value
`decided by PID control
`method, to servo circuit
`
` NO.3
`
`Fig. 48
`
`SAMSUNG EXHIBIT 1015
`
`Page 35 of 60
`
`SAMSUNG EXHIBIT 1015
`Page 35 of 60
`
`
`
`5.764.786
`
`1
`MOVING OBJECT MEASUREMENT DEVICE
`EMPLOYING A THREE-DIMENSIONAL
`ANALYSIS TO OBTAIN CHARACTERISTICS
`OF THE MOVING OBJECT
`
`RELATED APPLICATIONS
`
`This application is a National Phase application of PCT
`Application No. JP 94/00934 filed Jun. 10. 1994.
`
`BACKGROUND OF THE INVENTION
`
`1. Field of the Invention
`
`The present invention is used for measuring moving
`objects. Especially. it is related to the component for quickly
`calculating the coordinate of the captured moving object.
`2. Description of the Related Art
`As a means to measure characteristics of moving objects.
`images of the moving objects have been captured and
`processed as an image signal. In this way. moving objects
`have to be always captured within a screen. Therefore. the
`following methods have been employed.
`(1) When the moving object moves in a wide range. the
`object is measured by placing the moving object in the angle
`of View of the capturing device using a wide angle lens.
`(2) The moving object is measured while placing the
`moving object in the angle of view of the capturing device
`by restricting the movement of the moving object.
`(3) Putting the capturing device on a universal head
`equipped with an angle-encoder. tracking the moving object
`while moving the universal head manually. and calculating
`the coordinate of the moving object for every frame by
`digitizing. etc. using the recorded images and the angles of
`the universal head.
`
`Also. the characteristics of the moving object have been
`obtained by measuring the two-dimensional coordinate of
`the moving object simultaneously using two cameras. and
`calculating the three-dimensional coordinate of the moving
`object by using the two two—dimensional data.
`However. in method (1). the photographed image of the
`moving object itself is too small when a wide angle lens is
`used. Therefore. it is diflicult to separate the moving object
`and the background. Also. because the image processing
`device. which calculates the geometric center of gravity
`(centroid) by inputting images. processes images in the unit
`of pixels; it had not been possible to measure the small
`movement of the moving objects.
`In method (2). the measurable targets are limited because
`the moving range of the moving objects is limited.
`These problems also arise when two cameras are used
`simultaneously. Moreover. when a wide angle lens is used.
`the precision of the three-dimensional coordinate measure-
`ment of the moving objects is lowered. Therefore. these
`measurement methods are used only in the laboratory.
`Moreover. the image processing depends on the measure-
`ment of the two-dimensional coordinates and does not
`
`utilize the characteristics of the movement of the moving
`objects in the three—dimensional measurement space.
`Therefore. it sometimes outputs invalid readings as a result
`of noise included in the image or the noise produced when
`separating the moving objects from the background.
`When measuring the motion or actions of athletes
`outdoors. or in a stadium. method (3) can be used. However.
`a normal
`image signal. one frame corresponds to 1/60
`second. therefore. for example. it is necessary to digitize the
`image of 432000 frames in order to investigate the record for
`
`10
`
`15
`
`25
`
`30
`
`35
`
`45
`
`50
`
`55
`
`65
`
`2
`
`two hours. and 600 processing hours are required. assuming
`that it takes 5 seconds to digitize one frame.
`The present invention solves these problems and provides
`a measurement device which can capture the moving objects
`without using wide angle lenses even if the moving objects
`move in a wide range. and rapidly calculate the coordinate
`of the captured moving objects.
`
`SUMMARY OF THE INVENTION
`
`According to the first point of view of the present
`invention. the moving object measurement device includes
`one or more combinations of capturing an image of means
`to capture the moving objects. a driving means to set the
`capturing direction and the capturing range for the capturing
`means. an image processing means to calculate the location
`in the screen of the moving objects included in the image
`signal obtained by the capturing means. and a location
`calculation means to calculate the coordinate in the real
`space of the moving objects based on the information of the
`location obtained by the image processing means and the
`capturing direction and the capturing range of the capturing
`means when obtaining the image signal. The moving object
`measurement device also includes a three—dimensional area
`setting means to set the area where the targeting moving
`objects can move. an area mapping means to map the area
`in the three—dimensional real space coordinate system on the
`two—dimensional area corresponding to the capturing direc-
`tion and the capturing range of the capturing means. and a
`two-dimensional area setting means to set
`the mapped
`two-dimensional area to the corresponding image process-
`ing means. wherein the image processing means includes an
`area limiting means to limit the targeting area of the calcu-
`lation to the defined nave—dimensional area by the two-
`dimensional area setting means.
`A three-dimensional vector setting means sets the moving
`velocity vector of the targeting moving objects in the three—
`dirnensional real space coordinate system. A vector mapping
`means maps the moving velocity vector in the three-
`dimensional real space coordinate system on the two—
`dimensional moving velocity vector corresponding to the
`capturing direction and the capturing area of the capturing
`means. A vector setting means sets the captured two-
`dimensional moving velocity vector for each corresponding
`capturing means. wherein the image processing means
`includes a means to detect the change of the location of the
`moving objects of the screen of some frames defined before—
`hand and of the current screen in the two-dimensional area
`
`limited by the area limiting means. and can provide a means
`to start up the location calculation means on the condition
`that the detected change of the location is in the area defined
`beforehand for the two-dimensional moving velocity vector
`defined by the vector setting means.
`The capturing means includes the universal head which is
`rotatable around at least one axis and the capturing device
`fixed on the universal head. wherein the driving means
`includes the angle setting means to set the rotation angle
`around each axis of the universal head and a zoom lens
`attached on the capturing device.
`When the precision of the zoom lens and the angle setting
`means are high enough so that there is almost no difference
`between the defined objective value and the actual value. it
`is possible to use the objective value itself for the data of the
`capturing direction and the capturing area of the capturing
`means. However. in practice. it is better to use the actual
`value for the data of the capturing direction and the captur-
`ing area of the capturing means by feedback control which
`
`SAMSUNG EXHIBIT 1015
`
`Page 36 of 60
`
`SAMSUNG EXHIBIT 1015
`Page 36 of 60
`
`
`
`5,764,786
`
`3
`reduces the difference between the actual value and the
`objective value. That
`is.
`the angle setting means must
`include a motor which rotates the universal head on each at
`least one axis. an angle encoder which detects the actual
`rotation angle. and an angle controlling means to control the
`motor for reducing the difference between the detected angle
`and the objective bearing angle defined by the upper-device.
`and the zoom lens must include a zoom driving motor to
`define the zooming position. a zoom encoder to detect the
`actual zooming position. and a zoom controlling means to
`control the zoom driving motor which reduces the difference
`between the detected zooming position and the targeting
`position defined by the upper-device. and the output of the
`angle encoder and the output of the zoom encoder must be
`supplied to the position calculation means as the data on the
`capturing direction and the capturing area of the capturing
`means.
`
`In the present invention. the coordinate of the moving
`object
`is obtained by mapping the area of the three-
`dimensional real space coordinate system where the target-
`ing moving objects can move on the two-dimensional area
`and by limiting analysis to that area. For example. when the
`movement of the moving objects is partially known
`beforehand. such as skiing. skating. athletics. car racing.
`horse racing. bicycle racing. or speedboat racing. it is not
`necessary to measure outside of that area. Therefore. pro-
`cessing on such area should be omitted.
`By this. it is possible to reduce the image signal which is
`the target for processing. and it is possible to calculate the
`coordinate of the moving object in real-time.
`In some cases. it is possible to predict the velocity of the
`moving object beforehand. In such cases. defining the mov-
`ing velocity vector of the moving objects in the three-
`dimensional real space coordinate system. mapping them on
`the two-dimensional moving velocity vector. and begin
`measuring after detecting the moving object in the defined
`two-dimensional area and measuring the velocity vector
`which satisfies a certain condition for the specified velocity
`vector. By this. the possibility of measuring the object other
`than the targeting moving objects when the object moves in
`the capturing area by mistake becomes small.
`The present invention is suitable for the situation where
`multiple capturing means (e.g.. TV cameras) are positioned
`on the movement lane of the targeting moving object. and a
`driving means. an image processing means. and a location
`calculation means are provided. and all of them are system-
`atically managed by a common calculation device. In such
`a case. the calculation device systematically manages the
`coordinate of the moving object captured by each capturing
`means and the status of each capturing means. and it is
`possible to measure the location of the moving objects
`which is moving fast in a wide area effectively and precisely
`by the measurement data from the multiple angles and the
`characteristics of the movement of the moving objects which
`is registered beforehand. Also. it is possible to control the
`direction and the capturing area of the capturing means
`corresponding to the movement of the moving objects. to
`display the comparison data for the image obtained by each
`capturing means. to display the measurement precision. and
`to display other data on the capturing means.
`When capturing the moving objects. in some cases. it is
`possible to partially predict the movement of the moving
`object. For example. in skiing. skating. athletics. car racing.
`horse racing. bicycle racing. and speedboat racing.
`the
`course on which the moving objects move is partially
`known. and their velocity is distributed within a certain
`
`4
`range. In such cases. it is better to control the capturing
`direction and capturing area of the capturing means based on
`them.
`One such method is to define the movement data which is
`predicted for the moving objects on the three-dimensional
`real space coordinate system. and to control the capturing
`direction and area based on them. In this case. it is possible
`to prevent from changing the capturing direction to the
`velocity more than the two—dimensional vector by setting the
`movement vector of the moving objects in the three-
`dirnensional real space coordinate system. and mapping it on
`the two-dimensional vector.
`
`As another method. it is possible to detect the location of
`the moving object in the image from the captured image
`signal. and to control the capturing direction and area based
`on them. When there are multiple moving objects. it is better
`to use the weighted averages of each location. especially
`geometric center of gravity.
`It is also possible to control the capturing direction and the
`area by calculating the coordinate of the moving objects in
`the three—dimensional real space. For this. it is necessary to
`calculate the coordinate of the moving objects rapidly. In
`order to do such calculations. it is necessary to define an area
`in the three—dimensional real space coordinate system. and
`map it on the two-dimensional area based on the capturing
`direction and the area. and process the location detection in
`that area only.
`Even if the direction and the area of the capturing means
`are controlled. sometimes the moving objects get out of the
`capturing area of the capturing means. To cope with such
`cases. it is necessary to measure the motion data of the
`captured moving object beforehand and to keep controlling
`based on the past motion data if the moving object get out
`of the capturing area when it is captured. In this case. it is
`desirable to predict the direction of movement of the moving
`object based on the past motion data. however. it is also
`possible to control the capturing direction to the direction
`defined beforehand.
`
`It is also possible to define the shape and its motion data
`to compare in the three-dimensional real space coordinate
`system. and to map it on the two-dimensional area data
`based on the capturing area of the capturing means. and to
`overlap it on the image signal output by the capturing means.
`That is. not simply overlapping the past images on the
`current image. but overlapping the image on the actual
`image by obtaining the image that is as if captured with the
`capturing means at the present time based on the shape and
`the motion data to compare.
`For example. when capturing the athletics. ski contest. or
`swimming conte