United States Patent [191
`Kuwashima et a1.
`
`[54] MOVING OBJECT MEASUREMENT DEVICE
`EMPLOYING A THREE-DIMENSIONAL
`ANALYSIS TO OBTAIN CHARACTERISTICS
`OF THE MOVING OBJECT
`
`[76]
`
`[21]
`[22]
`[36]
`
`[37]
`
`[30]
`
`Inventors: Shigesumi Kuwashima. 2-26-2-406
`Chidori. Ohta-Ku. Tokyo 152; Masao
`Shimizu. 3-29-503 Kosugi-cho.
`Nakabara-ku. Kawasaki. Kanagawa 211;
`Torn Nakamura. 838-39 Ishibayashi.
`Nishinasuno-cho. Nasu-gun; Dairoku
`Sekiguchi. 2-23-3 Kalrinokizaka.
`Meguro-ku. Tokyo 152. all of Japan
`Appl. N0.:
`381,864
`PCT‘ Filed:
`Jun. 10, 1994
`
`PCI‘ N0.:
`
`PCT/JP94/00934
`
`§ 371 Date:
`
`Jun. 27, 1995
`
`§ 102(e) Date: Jun. 27, 1995
`PCI‘ Pub. N0.: WO94/29670
`
`PCT Pub. Date: Dec. 22, 1994
`Foreign Application Priority Data
`
`Jun. 10, 1993
`Jun. 10. 1993
`Oct. 19. 1993
`
`[JP]
`[JP]
`[JP]
`
`5-138805
`Japan ..
`5-139629
`Japan .
`Japan .................................. .. 5-261223
`
`[51] Int. Cl.6 ............................ .. G06K 9/00; G06K 9/36;
`G06K 9/32; G03B 13/18
`[52] U.S. C1. ........................ .. 382/107; 382/285; 382/291;
`354/402
`[58] Field of Search ................................... .. 382/107. 285.
`382/291. 302; 354/402
`
`[56]
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`USOO5764786A
`[11] Patent Number:
`[45] Date of Patent:
`
`5,764,786
`Jun. 9, 1998
`
`5,243,418
`5,422,700
`5.515.448
`
`9/1993 Kuno et a1. ........................... .. 382/107
`6/1995 Sada et a1‘
`~
`5/1996 Nishitani ............................... .. 382/107
`
`FOREIGN PATENT DOCUMENTS
`
`50-148024 1l/1975 Japan ............................. .. H04N 5/24
`
`57-201914 12/1982 Japan . . . . . .
`61-45910
`3/1986 Japan .... ..
`63-261102 lO/l988 Japan .... ..
`
`. . . .. GOSD 3/00
`G01B 11/00
`.. G01B 11/00
`
`. . . .. GOSD 3/12
`l-263703 10/1989 Japan . . . . . .
`H04N 5/232
`2-140073 5/1990 Japan .... ..
`GOGF 15/70
`4-1473 H1992 Japan .... ..
`G01B 11/00
`4-95806 3/1992 Japan .... ..
`H04N 5/232
`4-117077 4/1992 Japan .... ..
`G06F 15/70
`4-184577
`7/1992 Japan .... ..
`4-329790 11/1992 Japan ............................ .. GOGF 15/70
`
`Primary Examiner—Andrew W. Johns
`Assistant Examiner-Monica S. Davis
`Attorney Agent, or F irm—Cushrnan Darby & Cushman I?
`Group of Pillsbury Madison & Sutro LLP
`
`[57]
`
`ABSTRACT
`
`The present invention de?nes the area where the targeted
`moving object can move in the three-dimensional real
`spatial coordinate system. then maps it to the two
`dimensional area corresponding to the capturing method and
`the capturing range of the capturing device. and by limiting
`the image processing for obtaining the coordinate to that
`area. obtains the coordinate of the moving object in the
`three-dimensional real space rapidly by using the signal
`which is capturing the moving object. Furthermore. the
`present invention is able to control the capturing direction
`and the capturing range which correspond to the movement
`of the capturing target. to overlap the image signal with the
`data which is always changing by the capturing condition
`and the movement of the measurement data. to improve the
`detection precision. and to track automatically and smoothly.
`
`5,055,926 10/1991 Christensen et al. ................. .. 382/107
`
`38 Claims, 34 Drawing Sheets
`
`Camera Head/Calculation Device
`
`11
`
`Upper
`Calculation
`Device
`
`12
`i
`
`(27
`image
`Ccmpositio
`
`Dem/rice
`
`28
`
`~13
`
`l/O
`
`22
`
`Zoom
`Lens
`213
`i
`I
`Universal
`Head
`
`'
`
`2f’
`Image
`Processing
`Device
`
`Location
`Calculation
`Drive
`Device
`Control F:—->
`Device
`
`\
`
`25
`
`x
`
`26
`
`\‘Camera Head /Ca1culation Device
`
`12
`‘1
`
`_—
`
`Page 1 of 60
`
`SAMSUNG EXHIBIT 1015
`Samsung v. Image Processing Techs.
`
`

`

`US. Patent
`
`Jun. 9, 1998
`
`Sheet 1 0f 34
`
`5,764,786
`
`11
`
`Upper
`Calculation
`Device
`
`238
`
`N13
`
`1 2
`i
`
`$27
`Image
`Compositione
`Device
`'l
`
`I/O
`
`Location
`Calculation 6
`Device
`
`i
`26
`
`1 2
`3
`
`Camera Head/Calculation Device
`
`22
`
`Zoom
`Lens
`
`21
`3
`TV
`camera
`‘
`
`2§3
`
`Universal
`Head
`
`24
`5
`
`Image
`P
`-
`rocessmg
`Device
`
`Drive
`Control
`Device
`S
`25
`
`Camera Head/Calculation Device
`
`Fig. 1
`
`SAMSUNG EXHIBIT 1015
`Page 2 of 60
`
`

`

`US. Patent
`
`Jun. 9, 1998
`
`Sheet 2 of 34
`
`5,764,786
`
`Upper Calculation Device
`
`Location Calculation Device
`
`Define the area in the
`three—dimensional real
`spatial coordinate system
`
`Define the
`three — dimensional
`velocity vector
`
`Notify each location
`calculation device
`
`l
`
`:
`
`Map the three-dimensional
`area and the velocity vector
`to two — dimension
`
`Define it in the image
`processing device
`
`an input signal of
`location calculation
`
`Calculate the three
`dimensional coordinate of
`the moving object
`
`I
`
`Fig. 2
`
`SAMSUNG EXHIBIT 1015
`Page 3 of 60
`
`

`

`US. Patent
`
`Jun. 9, 1998
`
`Sheet 3 0f 34
`
`5,764,786
`
`E III!
`
`ww‘llll
`
`mm A
`
`DmO
`
`Nm
`
`ow (1
`
`a (I
`
`SEE
`
`vm
`
`E9
`mm f 5 l
`an
`at
`
`v50
`
`h
`
`mm
`M l
`
`MAO .11.
`2% .l
`
`Eu<i>
`
`v50
`
`mm
`
`wmm
`
`mm
`
`_m 1
`
`MAO k
`
`mm?»
`
`E500
`
`mm
`
`m .mwm
`
`cm
`
`SAMSUNG EXHIBIT 1015
`Page 4 of 60
`
`

`

`US. Patent
`
`Jun. 9, 1998
`
`Sheet 4 of 34
`
`5,764,786
`
`~21
`
`61
`
`] 1
`I
`
`22
`
`Fig. 4a
`
`61
`
`22
`
`Own
`
`61
`
`21
`
`~62
`
`Fig. 4b
`
`Fig. 4c
`
`SAMSUNG EXHIBIT 1015
`Page 5 of 60
`
`

`

`U.S. Patent
`
`Jun. 9, 1998
`
`Sheet 5 of 34
`
`5,764,786
`
`JoUdIAjJIG
`
`JNOIIO
`
`jo1}U09SUIALICG]10}OJ]
`
`
`
`
`
`[01]U0D[O1}U07Z
`
`
`JoJOY,BSUIALIG
`SIXY[BOOA
`
`EL
`
`
`
`JNOfOIJUOs)SIXY[e0IqI9A
`
`UO1}B90]
`
`ajBuy0}as[nd
`SIXY]B0I}J0A
`
`UOISIBAUOD
`
`Japoouqa[suy
`
`
`
`JINJOIyUODSIXY[B]UOZIIOY
`
`€8 Japoouq
`
`SIXY[e}UOZLIOY
`
`a[suy
`
`
`
`sIXy[@]UOZIIO}{
`
`JOJOBSUIALIG
`
`£6
`
`
`
`
`
`}INdIIJOIJUODBUIWIOOZ
`
`G“Sly
`
`W007
`
`Japoouy
`
`||
`
`
`
`SUIALIG]WOOZ
`
`JOJO
`
`SAMSUNG EXHIBIT 1015
`Page 6 of 60
`
`SAMSUNG EXHIBIT 1015
`Page 6 of 60
`
`
`
`
`
`
`
`
`
`

`

`US. Patent
`
`Jun. 9, 1998
`
`Sheet 6 0f 34
`
`5,764,786
`
`horizontal direction address
`
`vertical direction address
`
`\ 101/ /, /
`
`102 103
`
`Fig. 6
`
`SAMSUNG EXHIBIT 1015
`Page 7 of 60
`
`

`

`US. Patent
`
`Jun.9, 1998
`
`Sheet 7 of 34
`
`5,764,786
`
` Me
`
`M, (X,Y: Zi)
`
`(Xe,Ye VAD)
`
`x
`
`Fig. 7
`
`Z
`
`Y
`
`Us
`
`Ve
`
`xX
`
`ui
`
`Vi
`
`Fig. 8
`
`SAMSUNG EXHIBIT 1015
`Page 8 of 60
`
`SAMSUNG EXHIBIT 1015
`Page 8 of 60
`
`

`

`US. Patent
`
`Jun. 9, 1998
`
`Sheet 8 0f 34
`
`5,764,786
`
`Upper Calculation Device
`
`Location Calculation Device
`
`Define the
`predicted motion
`da :a
`
`Notify the location
`calculation device
`
`2
`
`Calculate the capturing
`direction and the range
`
`l
`
`Define the calculation result as
`the objective value to the drive
`control device
`
`Fig. 9
`
`Upper Calculation Device
`
`Location Calculation Device
`
`Define the three—dirnensional
`movement velocity vector of
`which the moving object can have
`
`Notify the location
`calculation device
`
`l
`
`Map the three — dimensional
`movement velocity vector to
`two — dimensional
`
`Limit the change of capturing
`direction which is faster than the
`two — dimensional movement
`velocity vector
`l
`Define the objective value to
`the drive control device
`
`Fi
`g. 10
`
`SAMSUNG EXHIBIT 1015
`Page 9 of 60
`
`

`

`US. Patent
`
`Jun. 9, 1998
`
`Sheet 9 of 34
`
`5,764,786
`
`Image Processing Device
`
`Location Calculation Device
`
`Obtain the geometric
`center of gravity of the
`moving object
`
`Notify the location
`calculation device of the :>
`obtained geometric center
`of gravity
`1/
`
`Calculate the capturing
`direction and the
`capturing range
`J,
`Define the calculation result
`as the objective value to the
`drive control device
`
`i
`
`Fig. 11
`
`Image Processing Device
`
`Location Calculation Device
`
`Obtain the geometric
`center of gravity of the
`moving object
`
`Notify the location
`calculation device of the 13:)
`obtained geometric center
`of gravity
`1/
`
`Obtain the real
`Spatial Coordinate of
`the moving Object
`
`Define the objective value to the
`drive control device based on the
`obtained coordinate
`
`i
`
`Fig. 12
`
`SAMSUNG EXHIBIT 1015
`Page 10 of 60
`
`

`

`US. Patent
`
`Jun. 9, 1998
`
`Sheet 10 of 34
`
`5,764,786
`
`Upper Calculation
`Device
`
`Location Calculation
`Device
`
`Image Processing
`Device
`
`Define the three—
`dimensional movement
`velocity vector of
`which the moving
`object can have
`
`Notify the
`location
`calculation device
`
`Map the three
`dimensional
`movement velocity
`vector to
`two — dimension
`
`Obtain the real
`spatial coordinate of
`the moving object
`
`Define the objective
`value based on the
`obtained coordinate
`\l/
`Limit the change of
`capturing direction
`which is faster than
`the two—dirnensional
`movement velocity
`vector
`\l/
`Define the objective
`value to the drive
`control device
`
`Fig. 13
`
`Obtain the
`geometric center of
`gravity of the
`moving object
`i
`Notify the location
`calculation device of
`the obtained
`geometric center of
`gravity
`i
`
`SAMSUNG EXHIBIT 1015
`Page 11 of 60
`
`

`

`US. Patent
`
`Jun. 9, 1998
`
`Sheet 11 of 34
`
`5,764,786
`
`Image Processing Device Location Calculation Device
`
`Obtain the geometric
`center of gravity of
`the moving object
`4/
`Notify the location
`calculation device of
`the obtained geometric
`center of gravity
`
`A Calculate the motion
`I’ data of
`the moving Object
`
`s it Within a
`capturing range‘?
`
`Record the motion
`data
`
`Calculate the
`capturing
`direction and the
`range based on
`the motion data
`
`Calculate the
`capturing
`direction and the
`range based on
`the past motion
`data
`
`Define the
`calculation result
`as the objective
`value to the drive
`control device
`
`i
`
`Fig. 14
`
`SAMSUNG EXHIBIT 1015
`Page 12 of 60
`
`

`

`US. Patent
`
`,
`
`Jun. 9, 1998
`
`Sheet 12 of 34
`
`5,764,786
`
`24 f- 1
`Image Processing
`Device
`Image Processing
`Device
`24—2
`24 7 '
`Image Processing
`Device
`Image Processing
`Device
`
`24
`Fig. 15
`
`Data
`7 Selector
`
`fin
`
`Fig. 16
`
`SAMSUNG EXHIBIT 1015
`Page 13 of 60
`
`

`

`US. Patent
`
`Jun. 9, 1998
`
`Sheet 13 of 34
`
`5,764,786
`
`Z
`
`coordinate of the measurement target
`
`W t
`
`>X
`
`Fig. 17a
`
`Z 1;
`
`direction control of the camera
`
`‘ t
`
`> X
`
`Fig. 17b
`
`f
`132 \\
`
`X/IBI
`M/136
`
`133\
`. a‘
`134 \N\
`
`’/,137
`L_//138
`
`135\\\1;/~V139
`
`k
`
`J
`
`SAMSUNG EXHIBIT 1015
`Page 14 of 60
`
`

`

`US. Patent
`
`Jun. 9, 1998
`
`Sheet 14 0f 34
`
`5,764,786
`
`location of the measurement target
`
`v’
`i=1 £2
`timgt
`
`Fig. 19a
`
`area of the measurement target
`
`SAMSUNG EXHIBIT 1015
`Page 15 of 60
`
`

`

`US. Patent
`
`Jun. 9, 1998
`
`Sheet 15 0f 34
`
`5,764,786
`
`Upper Calculation Device
`
`Location Calculation Device
`
`l
`
`Define the shape and the
`motion data to compare in
`the three-dimensional real
`spatial coordinate sysytem
`Map the shape and the motion
`J,
`Notify the location I; data to campare on the two—
`Calculation device
`dimensional area,corresponding
`‘L
`to the capturing area
`
`Detect the capturing
`area of TV camera
`
`Output the mapped data to
`image composition device
`
`Fig. 20
`
`Upper Calculation Device
`
`Location Calculation Device
`l
`Detect the capturing
`area of TV camera
`
`Estimate the measurement
`precision of the motion data
`in the three — dimensional
`real spatial coordinate system
`based on the location of TV
`camrea
`
`‘1/
`
`Notify the location
`calculation device
`‘L
`
`<1:
`
`Notify the upper
`calculation device
`
`calcumte motion data
`
`___-_->
`
`Map the measurernrnt
`precision to the two
`dimensional area
`corresponding to the
`capturing area
`
`Output the motion data and
`the mapped data to image
`composition device
`Fig. 21
`i
`
`SAMSUNG EXHIBIT 1015
`Page 16 of 60
`
`

`

`US. Patent
`
`Jun. 9, 1998
`
`Sheet 16 of 34
`
`5,764,786
`
`Upper Calculation Device
`
`Location Calculation Device
`
`Detect the capturing
`range of TV camera
`
`Notify the location
`calculatlon device
`
`Calculate the mutual
`relation of capturing
`area of each TV camrea
`
`Calculate the capturing
`Notify each location
`calculation device of the ::_> area of other TV camfara
`mutual relation
`to the TV Camera Whwh
`‘1’
`it manages
`
`Output the calculation
`result to the image
`composition device
`
`i
`
`Fig. 22
`
`SAMSUNG EXHIBIT 1015
`Page 17 of 60
`
`

`

`US. Patent
`
`Jun. 9, 1998
`
`Sheet 17 of 34
`
`5,764,786
`
`1l~ Upper
`Calculation
`Device
`
`12
`927
`
`28
`i
`
`N13
`
`Camera Head/ Calculation Device
`4
`Image
`Composition 9
`Device
`
`21
`6
`
`22
`g
`" TV
`Zoom
`Lens _ Camera
`
`Electric
`Universal
`Head
`
`5
`23
`
`Drive
`Control
`Device
`
`a
`:
`'
`
`254
`
`Image
`Processinga
`Device
`
`1/0
`
`0
`Location
`.2
`_ 6
`0 Calculation
`‘5
`DEVICE
`Image ‘
`Processinga *3
`Device
`g
`1
`
`s
`26
`
`s
`24
`
`;
`'
`
`141
`
`'P
`
`Camera Head/ Calculation Device
`
`12
`
`Fig. 23
`
`SAMSUNG EXHIBIT 1015
`Page 18 of 60
`
`

`

`US. Patent
`
`Jun. 9, 1998
`
`Sheet 13 of 34
`
`5,764,786
`
`EN
`
`3% E828
`
`Mrw
`
`Mm
`
`mm
`
`Lwm mm
`mmm Hm
`
`ill
`
`v30
`H H H
`
`l'
`mg _m
`mm
`H
`
`m2<
`v30
`
`E300
`
`mm
`
`wm
`
`v30
`
`Ea;
`
`v50
`
`SAMSUNG EXHIBIT 1015
`Page 19 of 60
`
`

`

`US. Patent
`
`‘ Jun. 9, 1998
`
`Sheet 19 of 34
`
`5,764,786
`
`SAMSUNG EXHIBIT 1015
`Page 20 of 60
`
`

`

`US. Patent
`
`Jun.9, 1998
`
`Sheet 20 of 34
`
`5,764,786
`
`171
`
`172 178
`
`Le
`
`Fig. 26a
`
`
`
`SAMSUNG EXHIBIT 1015
`Page 21 of 60
`
`SAMSUNG EXHIBIT 1015
`Page 21 of 60
`
`

`

`U.S. Patent
`
`Jun. 9, 1998
`
`Sheet 21 of 34
`
`5,764,786
`
`
`
`Capture the location data
`
`and the area data of
`itself
`
`
`Capture the location data and
`the area data of other devices
`
`
`
`Dalete the location data of
`which area data is increasing
`
`Average the rest
`
`location data
`
`
`
`
`
`Is the
`
`area data of
`
`
`the narrowest valid range
`
`increasing ?
`Add a flag
`
`to the centroid
`
`
`data which indicate
`
`low reliability
`
`Next process
`
`Fig. 27
`
`SAMSUNG EXHIBIT 1015
`Page 22 of 60
`
`SAMSUNG EXHIBIT 1015
`Page 22 of 60
`
`

`

`U.S. Patent
`
`Jun. 9, 1998
`
`Sheet 22 of 34
`
`5,764,786
`
`Location Calculation
`Device
`
`Location Calculation
`Device ..............
`
`Capture the location
`data and the area
`data from the image
`processing device
`
`
`
`
`
`Capture the location
`data and the area
`
`
`
`
`
`data from the image
`processing device
`
`
`Does
`
`the area the area
`
`
`
`change
`
`suddenly ?
`
`
`
`
`
`
`Send data to a
`Receive data from
`
`stated location
`
`other location
`calculation device
`
`
`
`
`calculation device
`
`
`
`Calculate the real
`
`
`spatial coordinate of
`the moving object
`
`
`Notify other location
`
`
`calculation devices
`of
`the real spatial
`coordinate and its
`
`reliability
`
`Fig. 28
`
`SAMSUNG EXHIBIT 1015
`Page 23 of 60
`
`SAMSUNG EXHIBIT 1015
`Page 23 of 60
`
`

`

`U.S. Patent
`
`Jun. 9, 1998
`
`Sheet 23 of 34
`
`5,764,786
`
`()y~ 181
`\
`enes
`182
`)
`a Ff eS
`
`183
`
`Fig. 29
`
`184[4p
`
`Fig. 30a
`
`Fig. 30b
`
`Fig. 30c
`
`
`
`Fig. 31
`
`SAMSUNG EXHIBIT 1015
`Page 24 of 60
`
`SAMSUNG EXHIBIT 1015
`Page 24 of 60
`
`

`

`US. Patent
`
`Jun. 9, 1998
`
`Sheet 24 of 34
`
`5,764,786
`
`
`
`Del
`
`192 moving
`moving
`191
`moving
`12
`
`
`imageimage image
`
`
`
`> Data Recording||
`Unit Ft
`
`
` three — dimensional
`
`
`
`three — dimensional]
`data
`
`
` 193
`
`Time Base
`Unit
`
`data
`
`Fig. 32
`
`SAMSUNG EXHIBIT 1015
`Page 25 of 60
`
`SAMSUNG EXHIBIT 1015
`Page 25 of 60
`
`

`

`US. Patent
`
`Jun. 9, 1998
`
`Sheet 25 of 34
`
`5,764,786
`
`201 camera head
`
`Record the image with the three — dimensional
`Fig. 33
`
`location information
`
`SAMSUNG EXHIBIT 1015
`Page 26 of 60
`
`SAMSUNG EXHIBIT 1015
`Page 26 of 60
`
`

`

`U.S. Patent
`
`Jun. 9, 1998
`
`Sheet 26 of 34
`
`5,764,786
`
`
`
`
`
`yndynoasewlBZulaow
`
`
`
`yndjnoejepqe{
`
`neee me ent eee eeee ee
`
`jeeds
`
`9}8UIP1I0074
`
` |
`
`peedL922
` wi0o7||adATIG|Zo|—{919d(Fez
`
`IPearid|eze|snoog|
`
`
`
`wu007_J
`
`|
`
`sn30of
`
`|
`
`|
`
`SAMSUNG EXHIBIT 1015
`Page 27 of 60
`
`SAMSUNG EXHIBIT 1015
`Page 27 of 60
`
`

`

`U.S. Patent
`
`Jun.9, 1998
`
`Sheet 27 of 34
`
`5,764,786
`
`
`
`plane
`
`Offset Z
`
`
`
`vertical axis
`
`Cf horizontal axis
`Fig. 35
`
`' positive
`negative,
`
`
`main point
`capturing
`
`center O
`(center of
`lens)
`
`optical
`. axis
`subject
`
`
`
`
`Fig. 36
`
`main point
`
`negative lens positive
`
`
`
`subject
`
`Fig. 37
`
`optical
`axis
`
`SAMSUNG EXHIBIT 1015
`Page 28 of 60
`
`SAMSUNG EXHIBIT 1015
`Page 28 of 60
`
`

`

`U.S. Patent
`
`Jun. 9, 1998
`
`Sheet 28 of 34
`
`5,764,786
`
`The measurement
`
`coordinate of
`the Z
`_
`{constraint plane
`
`image location
`main point
`
`
`measurement device
`CCD TV_plane
`
`
`
`
`targeted object
`
`X
`
`Fig. 38
`
`constraint plane
`
`TV plane
`
`
` projection
`
`
`
`
`
`
`subject
`
`center
`
`attention vector
`
`Fig. 39
`
`Z
`
`| constraint plane
`vertical
`AXIS attention
`vector
`rotation of
`universal
`
`head
`
`horizontal
`
`
`SAMSUNG EXHIBIT 1015
`Page 29 of 60
`
`SAMSUNG EXHIBIT 1015
`Page 29 of 60
`
`

`

`US. Patent
`
`Jun. 9, 1998
`
`Sheet 29 of 34
`
`5,764,786
`
`constraint plane
`—-xX
`constraint plane
`l
`A and B can be obtained
`by using are tan,
`
`CCD plane
`
`arc cos respectively
`
`jSe
`
`
`rotation center
`
`Fig. 41
`
`constraint plane
`
`
`
`f
`constraint plane
`
`vesseines be as
`location | where subject
`
`
`
`vectora
`
`TV plane
`CCD plane
`
`
`
`subject
`
`xX
`
`Fig.42
`
`SAMSUNG EXHIBIT 1015
`Page 30 of 60
`
`SAMSUNG EXHIBIT 1015
`Page 30 of 60
`
`

`

`U.S. Patent
`
`Jun. 9, 1998
`
`Sheet 30 of 34
`
`5,764,786
`
`location of
`projection,Center
`
`Value of D/A vs. location
`
`¥o00-1500 1000-500
`
`500 1000 1500 2000
`
`0
`D/A
`Fig. 43
`
`the location Value of D/A vs. inverse of
`inverse of
`of projection center
`
`of projection center
`location of projection center 0
`
`:
`:
`:
`~—2000-1500 -1000-500
`
`:
`‘
`:
`500 1000 1500 2000
`
`t
`0
`D/A
`Fig. 44
`
`SAMSUNG EXHIBIT 1015
`Page 31 of 60
`
`SAMSUNG EXHIBIT 1015
`Page 31 of 60
`
`

`

`US. Patent
`
`-
`
`Jun. 9, 1998
`
`Sheet 31 of 34
`
`5,764,786
`
`value of D/A Inverse of distance vs. value of D/A
`
`1000
`
`-2000
`0.001
`
`0.002
`
`0.003
`
`
`
`0.004.
`
`0.005
`
`0.006
`
`
`
`0.007
`
`Fig. 45
`
`SAMSUNG EXHIBIT 1015
`Page 32 of 60
`
`SAMSUNG EXHIBIT 1015
`Page 32 of 60
`
`

`

`U.S. Patent
`
`Jun. 9, 1998
`
`Sheet 32 of 34
`
`5,764,786
`
`
`
`time(sec)
`
`2048)
`
`Fig.46
`
`0-3(1538->
`
`SAMSUNG EXHIBIT 1015
`Page 33 of 60
`
`SAMSUNG EXHIBIT 1015
`Page 33 of 60
`
`

`

`U.S. Patent
`
`Jun. 9, 1998
`
`Sheet 33 of 34
`
`5,764,786
`
`Start
`
`Define the location
`measurement device
`
`
`
`Obtain the information of
`target object from location
`measurement device
`
`
`
`
`
`Oms
`
`Read potentiometer
`
`
`
` NO.4
`
`the location value
`Output
`decided by PID control
`method to servo circuit
`
`Calculate the spatial
`coordinate of targeted object
`
`
`
`
`
`Set
`
`the appropriate
`zoom value
`
`
`
`targeted object
`
`not
`in the image for
`
`
`targeted object on the
`ore than a constan
`
`
`period ?
`
`Predict the location of targeted
`
`object where it will be when
`
`Define the rotation angle
`
`
`Predict the movement of
`the next spatial location is
`of two axes of the
`
`
`
`
`the targeted object based
`caluculated (four points)
`
`universal head so that
`
`
`
`on the movement of the
`the targeted object can
`
`targeted object so far
`
`
`be searched for
`
`
`Define the direction of camera
`
`
`based on the predicted location
`
`
`
`Decide the direction of
`of targeted object, and calculate
`
`
`
`
`camera based on the
`
`
`
`the rotation angle of two axes
`of the universal head
`predicted location of
`targeted object, and
`calculate the rotation
`
`
`angle of two axes of
`
`universal head
`
`calculated rotation
`
`angle of the universal head
`
`
`larger than the dynamic
`
`characteristic
`
`
`
`
`
`the zoom value by using the
`Set
`
`
`location of the targeted object
`in the
`Set the zoom value wider
`
`image and the distance to the
`so that
`the targeted
`
`
`
`targeted object so that
`the targeted
`object can be caputured
`
`
`
`object is captured with a defined size
`
`
`
`
`
`
`Set
`
`the zoom wider
`
`(A)
`
`Fig. 47
`
`SAMSUNG EXHIBIT 1015
`Page 34 of 60
`
`SAMSUNG EXHIBIT 1015
`Page 34 of 60
`
`

`

`U.S. Patent
`
`Jun. 9, 1998
`
`Sheet 34 of 34
`
`5,764,786
`
`Decide the focus based on the
`distance of
`targeted object
`
`Read potentiometer
`
`4ms
`
`
`
`Output the position value
`decided by PID control
`method, to servo circuit
`
` NO.1
`
`Communicate with terminal
`
`Read potentiometer
`
`
`
`Output the position value
`decided by PID control
`method, to servo circuit
`
`8ms
`
` NO.2
`
`Read potentiometer
`
`
`
`Output the position value
`decided by PID control
`method, to servo circuit
`
`12ms
`
` NO.3
`
`Fig. 48
`
`SAMSUNG EXHIBIT 1015
`Page 35 of 60
`
`SAMSUNG EXHIBIT 1015
`Page 35 of 60
`
`

`

`5,764,786
`
`1
`MOVING OBJECT MEASUREMENT DEVICE
`EMPLOYING A THREE-DIMENSIONAL
`ANALYSIS TO OBTAIN CHARACTERISTICS
`OF THE MOVING OBJECT
`
`RELATED APPLICATIONS
`
`This application is a National Phase application of PCT
`Application No. JP 94/00934 filed Jun. 10, 1994.
`
`BACKGROUND OF THE INVENTION
`
`1. Field of the Invention
`
`The present invention is used for measuring moving
`objects. Especially,it is related to the componentfor quickly
`calculating the coordinate of the captured moving object.
`2. Description of the Related Art
`As a means to measure characteristics of moving objects,
`images of the moving objects have been captured and
`processed as an image signal. In this way, moving objects
`have to be always captured within a screen. Therefore, the
`following methods have been employed.
`(1) When the moving object moves in a wide range, the
`object is measured by placing the moving object in the angle
`of view of the capturing device using a wide angle lens.
`(2) The moving object is measured while placing the
`moving object in the angle of view of the capturing device
`by restricting the movement of the moving object.
`(3) Putting the capturing device on a universal head
`equipped with an angle-encoder, tracking the moving object
`while moving the universal head manually, and calculating
`the coordinate of the moving object for every frame by
`digitizing, etc. using the recorded images and the angles of
`the universal head.
`
`Also, the characteristics of the moving object have been
`obtained by measuring the two-dimensional coordinate of
`the moving object simultaneously using two cameras, and
`calculating the three-dimensional coordinate of the moving
`object by using the two two-dimensional data.
`However, in method (1), the photographed image of the
`moving object itself is too small when a wide angle lens is
`used. Therefore, it is difficult to separate the moving object
`and the background. Also, because the image processing
`device, which calculates the geometric center of gravity
`(centroid) by inputting images, processes imagesin the unit
`of pixels; it had not been possible to measure the small
`movementof the moving objects.
`In method (2). the measurable targets are limited because
`the moving range of the moving objects is limited.
`These problems also arise when two cameras are used
`simultaneously. Moreover, when a wide angle lens is used,
`the precision of the three-dimensional coordinate measure-
`ment of the moving objects is lowered. Therefore, these
`measurement methods are used only in the laboratory.
`Moreover, the image processing depends on the measure-
`ment of the two-dimensional coordinates and does not
`utilize the characteristics of the movement of the moving
`objects in the three-dimensional measurement space.
`Therefore. it sometimes outputs invalid readings as a result
`of noise included in the image or the noise produced when
`separating the moving objects from the background.
`When measuring the motion or actions of athletes
`outdoors, or in a stadium, method (3) can be used. However.
`a normal
`image signal. one frame corresponds to 1/60
`second, therefore, for example,it is necessary to digitize the
`image of 432000 framesin order to investigate the record for
`
`10
`
`15
`
`25
`
`30
`
`35
`
`45
`
`3a
`
`35
`
`65
`
`2
`two hours, and 600 processing hours are required, assuming
`that it takes 5 seconds to digitize one frame.
`The present invention solves these problems and provides
`a measurementdevice which can capture the moving objects
`without using wide angle lenses even if the moving objects
`move in a wide range, and rapidly calculate the coordinate
`of the captured moving objects.
`
`SUMMARYOF THE INVENTION
`
`According to the first point of view of the present
`invention, the moving object measurement device includes
`one or more combinations of capturing an image of means
`to capture the moving objects. a driving means to set the
`capturing direction and the capturing range for the capturing
`means, an image processing meansto calculate the location
`in the screen of the moving objects included in the image
`signal obtained by the capturing means. and a location
`calculation means to calculate the coordinate in the real
`space of the moving objects based on the information of the
`location obtained by the image processing means and the
`capturing direction and the capturing range of the capturing
`means when obtaining the image signal. The moving object
`measurement device also includes a three-dimensional area
`setting means to set the area where the targeting moving
`objects can move, an area mapping means to map the area
`in the three-dimensional real space coordinate system on the
`two-dimensional area corresponding to the capturing direc-
`tion and the capturing range of the capturing means. and a
`two-dimensional area setting means to set
`the mapped
`two-dimensional area to the corresponding image process-
`ing means, wherein the image processing means includes an
`area limiting meansto limit the targeting area of the calcu-
`lation to the defined two-dimensional area by the two-
`dimensional area setting means.
`A three-dimensional vector setting meanssets the moving
`velocity vector of the targeting moving objects in the three-
`dimensional real space coordinate system. A vector mapping
`Means maps the moving velocity vector in the three-
`dimensional real space coordinate system on the two-
`dimensional moving velocity vector corresponding to the
`capturing direction and the capturing area of the capturing
`means. A vector setting means sets the captured two-
`dimensional moving velocity vector for each corresponding
`capturing means, wherein the image processing means
`includes a meansto detect the change of the location of the
`moving objects of the screen of some frames defined before-
`hand and of the current screen in the two-dimensional area
`limited by the area limiting means, and can provide a means
`to start up the location calculation means on the condition
`that the detected changeof the location is in the area defined
`beforehand for the two-dimensional moving velocity vector
`defined by the vector setting means.
`The capturing means includes the universal head whichis
`rotatable around at least one axis and the capturing device
`fixed on the universal head, wherein the driving means
`includes the angle setting means to set the rotation angle
`around each axis of the universal head and a zoom lens
`attached on the capturing device.
`Whenthe precision of the zoom lens and the angle setting
`means are high enough so that there is almost no difference
`between the defined objective value and the actual value.it
`is possible to use the objective valueitself for the data of the
`capturing direction and the capturing area of the capturing
`means. However, in practice, it is better to use the actual
`value for the data of the capturing direction and the captur-
`ing area of the capturing means by feedback control which
`
`SAMSUNG EXHIBIT 1015
`Page 36 of 60
`
`SAMSUNG EXHIBIT 1015
`Page 36 of 60
`
`

`

`5.764,786
`
`3
`reduces the difference between the actual value and the
`objective value. That
`is,
`the angle setting means must
`include a motor whichrotates the universal head on each at
`least one axis, an angle encoder which detects the actual
`rotation angle, and an angle controlling meansto control the
`motorfor reducingthe difference between the detected angle
`and the objective bearing angle defined by the upper-device.
`and the zoom lens must include a zoom driving motor to
`define the zooming position, a zoom encoder to detect the
`actual zooming position, and a zoom controlling means to
`control the zoom driving motor which reduces the difference
`between the detected zooming position and the targeting
`position defined by the upper-device, and the output of the
`angle encoder and the output of the zoom encoder must be
`supplied to the position calculation meansas the data on the
`capturing direction and the capturing area of the capturing
`means.
`
`In the present invention, the coordinate of the moving
`object
`is obtained by mapping the area of the three-
`dimensional real space coordinate system where the target-
`ing moving objects can move on the two-dimensional area
`and by limiting analysis to that area. For example, when the
`movement of the moving objects is partially known
`beforehand, such as skiing. skating. athletics, car racing,
`horse racing, bicycle racing. or speedboat racing.it is not
`necessary to measure outside of that area. Therefore. pro-
`cessing on such area should be omitted.
`Bythis, it is possible to reduce the image signal which is
`the target for processing. and it is possible to calculate the
`coordinate of the moving object in real-time.
`In somecases,it is possible to predict the velocity of the
`moving object beforehand. In such cases, defining the mov-
`ing velocity vector of the moving objects in the three-
`dimensional real space coordinate system, mapping them on
`the two-dimensional moving velocity vector, and begin
`measuring after detecting the moving object in the defined
`two-dimensional area and measuring the velocity vector
`which satisfies a certain condition for the specified velocity
`vector. By this, the possibility of measuring the object other
`than the targeting moving objects when the object moves in
`the capturing area by mistake becomes small.
`The present invention is suitable for the situation where
`multiple capturing means (e.g.. TV cameras) are positioned
`on the movementlane of the targeting moving object, and a
`driving means, an image processing means, and a location
`calculation means are provided, and all of them are system-
`atically managed by a common calculation device. In such
`a case, the calculation device systematically manages the
`coordinate of the moving object captured by each capturing
`means and the status of each capturing means, and it is
`possible to measure the location of the moving objects
`which is moving fast in a wide areaeffectively and precisely
`by the measurement data from the multiple angles and the
`characteristics of the movementof the moving objects which
`is registered beforehand. Also, it is possible to control the
`direction and the capturing area of the capturing means
`corresponding to the movement of the moving objects, to
`display the comparison data for the image obtained by each
`capturing means, to display the measurementprecision, and
`to display other data on the capturing means.
`When capturing the moving objects, in some cases,it is
`possible to partially predict the movement of the moving
`object. For example, in skiing, skating, athletics, car racing,
`horse racing, bicycle racing, and speedboat racing.
`the
`course on which the moving objects move is partially
`known, and their velocity is distributed within a certain
`
`4
`range. In such cases, it is better to control the capturing
`direction and capturing area of the capturing means based on
`them.
`One such method is to define the movement data which is
`predicted for the moving objects on the three-dimensional
`real space coordinate system. and to control the capturing
`direction and area based on them.In this case,it is possible
`to prevent from changing the capturing direction to the
`velocity more than the two-dimensional vector by setting the
`movement vector of the moving objects in the three-
`dimensional real space coordinate system, and mappingit on
`the two-dimensional vector.
`
`As another method,it is possible to detect the location of
`the moving object in the image from the captured image
`signal, and to control the capturing direction and area based
`on them. When there are multiple moving objects, it is better
`to use the weighted averages of each location, especially
`geometric center of gravity.
`It is also possible to control the capturing direction and the
`area by calculating the coordinate of the moving objects in
`the three-dimensional real space. Forthis, it is necessary to
`calculate the coordinate of the moving objects rapidly. In
`order to do such calculations,it is necessary to define an area
`in the three-dimensional real space coordinate system, and
`map it on the two-dimensional area based on the capturing
`direction and the area, and process the location detection in
`that area only.
`Evenif the direction and the area of the capturing means
`are controlled, sometimes the moving objects get out of the
`capturing area of the capturing means. To cope with such
`cases, it is necessary to measure the motion data of the
`captured moving object beforehand and to keep controlling
`based on the past motion data if the moving object get out
`of the capturing area whenit is captured. In this case.it is
`desirable to predict the direction of movement of the moving
`object based on the past motion data, however, it is also
`possible to control the capturing direction to the direction
`defined beforehand.
`
`It is also possible to define the shape and its motion data
`to compare in the three-dimensional real space coordinate
`system, and to map it on the two-dimensional area data
`based on the capturing area of the capturing means, and to
`overlapit on the image signal output by the capturing means.
`That is, not simply overlapping the past images on the
`current image, but overlapping the image on the actual
`image by obtaining t