I 1111111111111111 11111 lllll lllll lllll lllll 111111111111111 lll111111111111111
`US006144366A
`6,144,366
`[11] Patent Number:
`United States Patent
`[19J
`Numazaki et al.
`Nov. 7, 2000
`[45] Date of Patent:
`
`[54] METHOD AND APPARATUS FOR
`
`GENERATING INFORMATION INPUT
`
`USING REFLECTED LIGHT IMAGE OF
`TARGET OBJECT
`
`5,297,061 3/1994 Dementhon et al. ................... 364/559
`
`
`
`
`
`
`
`5,323,174 6/1994 Klapman et al. ....................... 345/156
`
`
`
`5,353,042 10/1994 Klapman et al. ....................... 345/156
`
`
`5,423,554 6/1995 Davis ...................................... 273/437
`
`
`
`
`5,481,622 1/1996 Gerhardt et al. ........................ 382/103
`
`
`
`5,627,565 5/1997 Morishita ................................ 345/158
`Shunichi Numazaki; Miwako Doi,
`
`
`[75]Inventors:
`
`
`5,686,942 11/1997 Ball ......................................... 345/158
`
`both of Kanagawa; Akira Morishita,
`
`
`
`5,703,356 12/1997 Bidiville et al. .. .... ... ... ... ... ... ... 250/221
`Tokyo; Naoko Umeki; Hiroki Miura,
`
`
`
`5,729,475 3/1998 Romanik, Jr ............................ 364/559
`
`
`both of Kanagawa, all of Japan
`
`
`5,801,704 9/1998 Oohara et al. .......................... 345/358
`
`
`
`5,819,206 10/1998 Horton et al. .......................... 702/150
`Kawasaki,[73] Assignee: Kabushiki Kaisha Toshiba,
`
`
`
`
`
`5,900,863 5/1999 Numazaki ............................... 345/158
`Japan
`
`08/953,667
`[21] Appl. No.:
`
`[22] Filed:Oct. 17, 1997
`
`Primary Examiner-William A. Cuchlinski, Jr.
`
`
`
`
`Assistant Examiner-Marthe Y. Marc-Coleman
`
`
`
`Attorney, Agent, or Firm----Oblon, Spivak, McClelland,
`
`Maier & Neustadt, P.C .
`
`[30]
`
`[57]
`
`
`
`
`Foreign Application Priority Data
`
`ABSTRACT
`
`
`
`A method and an apparatus for generating information input
`
`
`8-275949 Oct. 18, 1996 [JP] Japan ....................................
`
`
`9-019397 Jan. 31, 1997 [JP] Japan ....................................
`
`
`
`
`which are capable of realizing a direct command type
`
`
`
`9-027752 .................................... Feb. 12, 1997 [JP] Japan
`
`
`
`information input scheme by which the gesture or the
`
`
`
`motion can be inputted easily. T he apparatus has a timing
`
`[51] Int. Cl.7 ....................................................... G09G 5/08
`
`
`
`
`
`
`signal generation unit for generating a timing signal; a
`345/157; 345/158 [52] U.S. Cl. ........................... 345/156;
`
`
`
`
`
`
`
`
`lighting unit for emitting a light whose intensity vary in
`[58] Field of Search .....................................
`345/158, 157,
`
`
`
`
`time, according to the timing signal generated by the timing
`
`
`345/156, 419, 425; 463/31, 32, 33, 36,
`
`
`
`
`signal generation unit; and a reflected light extraction unit
`
`
`37; 250/200, 206.1, 208.1, 214 R
`
`
`
`having a sensor array for detecting a reflected light from a
`
`
`
`
`
`target object resulting from the light emitted by the lighting
`
`
`
`
`
`unit, in synchronization with the timing signal generated by
`
`
`
`
`the timing signal generation unit, so as to obtain a spatial
`
`
`
`
`intensity distribution of the reflected light in a form of a
`
`
`
`4,320,292 3/1982 Oikawa et al. ......................... 250/227
`
`
`
`reflected light image indicative of an information input
`
`
`
`
`
`4,516,020 5/1985 Simpson et al. .................... 250/214 L
`
`
`
`
`related to the target object, in separation from an external
`
`
`354/403 4,839,685 6/1989 Ishigu ro et al. ........................
`
`
`
`light that is illuminating the target object.
`
`
`4,841,349 6/1989 Nakano ..................................... 357/30
`
`
`4,956,794 9/1990 Zeevi et al. ............................. 364/559
`
`96 Claims, 101 Drawing Sheets
`
`
`
`5,243,182 9/1993 Murata et al. ....................... 250/222.1
`
`[56]
`
`
`
`References Cited
`
`U.S. PATENT DOCUMENT S
`
`104
`\
`TIMING SIGNAL
`GENERATION -
`UNIT
`
`101
`\
`
`LIGHTING UNIT
`
`102
`\
`REFLECTED LIGHT
`
`EXTRACTION UNIT
`
`103
`\
`FEATURE DATA
`
`GENERATION UNIT
`
`

`

`Sheet 1 of 101 6,144,366
`U.S. Patent
`Nov. 7, 2000
`
`FIG.I
`
`101
`\
`
`
`
`- LIGHTING UNIT
`
`104
`s
`TIMING SIGNAL
`
`GENERATION -
`UNIT
`
`102
`\
`REFLECTED LIGHT
`
`-�
`
`EXTRACTION UNIT
`
`103
`
`FEATURE DATA
`
`
`
`GENERATION UNIT
`
`

`

`112
`
`CONTROL UNIT
`
`TIMING
`
`101
`
`UNIT a------�
`LIGHTING
`
`UNIT
`GENERATION
`
`0
`0
`0
`
`�-..J
`
`DATA
`
`MEMORY
`
`z
`
`FEATURE
`
`110
`
`UNIT
`
`DETECTION
`PHOTO­
`
`UNIT
`
`2ND
`
`AID
`
`CALCULATION
`
`DIFFERENCE
`
`UNIT
`
`DETECTION
`
`115 103
`
`114
`
`109 111
`
`PHOTO­
`1ST
`
`DETECTION
`
`OPTICS
`
`......
`
`......
`
`�
`
`•
`
`•
`
`FIG.2
`
`102 REFLECTED LIGHT
`
`EXTRACTION UNIT
`
`PHOTO­
`
`

`

`EXAMPLE)
`
`(2 X 2 AS
`REGION
`IMAGE
`
`FIG.3
`
`I ....
`
`I
`
`I
`
`17
`
`:· ::
`
`�-
`
`--
`
`L�
`-------
`-· �-
`
`·r�
`
`I
`I
`I
`I
`
`Yccs
`
`Ycc2Ycc3 ---- _,_
`Ycc4--
`
`----
`
`Yss Ycc1
`
`134
`
`I TIMING GENERA TOR
`
`PD
`
`135
`
`I I PD
`
`"7"
`
`I
`
`;J;
`
`I�
`
`-
`
`,o:r ,o:r
`
`I
`
`�136
`
`REGISTERS
`HORIZONTAL
`SHIFf
`
`s.
`
`�
`
`r--, -c::::>------, C:
`
`CIRCUIT r---133
`
`DIFFERENCE
`
`1= 'J'?T
`� �
`I ·-------- 1--Li�
`--
`I L_pD
`i�
`!
`--��--117
`
`� 1:n-I
`�
`
`r.:i r�r�
`,r ,..,
`l;J�� r�
`I \ L
`
`;J;
`
`�
`
`J;. J;-
`..L
`I I I
`
`..L
`
`;J;
`
`u
`
`o>;,::, r--
`en < tI1 tI1r--;,::, tI1 i-3
`
`en
`
`n- i-3
`
`n
`
`I
`
`I
`
`L..
`
`\
`
`I 'PD
`
`"7
`Ir J;.•·J;-)
`
`..I..
`I I I
`
`.L
`
`

`

`SAMPLE 2
`
`SAMPLE I
`
`TRANSFER
`
`0
`0
`0
`
`f'I
`
`z
`
`�-..J
`LIGHT
`
`127
`
`ADDRESS LINE
`
`123 122
`
`126
`
`SIGNAL LINE
`
`VERTICAL
`
`......
`
`......
`
`•
`
`•
`
`RESET
`
`F
`
`124
`
`FIG.4
`
`

`

`Sheet 5 of 101 6,144,366
`U.S. Patent
`Nov. 7, 2000
`
`FIG.5
`
`I
`
`--
`
`I
`
`I
`
`128
`
`LIGHTING -1
`CONTROL PULSE I I
`LIGHTING
`POWER (--� 119
`
`------
`
`-f
`
`l
`
`1
`I
`I
`
`�---------
`
`RESET
`I
`STORING
`PERIOD-I� _
`SAMPLE 1
`
`SAMPLE 2
`
`130
`)
`
`131
`)
`
`'
`
`II(
`
`> I
`
`STORING :
`PERIOD-2
`132
`)
`
`

`

`6,144,366
`Sheet 6 of 101 6,144,366
`U.S. Patent
`Sheet 6 of 101
`Nov.7, 2000
`Nov. 7, 2000
`
`\0
`
`•
`
`U.S. Patent
`
`t)
`
`....-4
`0
`N
`
`Page 7 of 147
`
`

`

`Sheet 7 of 101
`6,144,366
`U.S. Patent
`Nov.7, 2000
`Sheet 7 of 101 6,144,366
`U.S. Patent
`Nov. 7, 2000
`
`FIG.7
`FIG.7
`
`||fe 209
`|BS
`-FAxL_|_
`
`209
`
`203
`
`ow
`
`Page8 of 147
`
`

`

`6,144,366
`U.S. Patent
`6,144,366
`U.S. Patent
`Sheet 8 of 101
`Nov.7, 2000
`Sheet 8 of 101
`Nov. 7, 2000
`
`FIG.8
`FIG.8
`
`
`
`----210
`210
`
`211
`211
`212
`212
`
`,,,.-..
`I �
`�
`
`.,..
`
`-
`
`PPCee171bei|||eenoe
`PES
`CCCPTes
`pititytTiiie
`
`/ r-v ............ ..
`,-,-
`(( � -\ '-
`—fCSE
`' \_�
`�, -
`I
`aaaaa~a~os)LCa
`\ ... '
`|NoAcoMeAT7|Et
`\� D:X:
`[>WW,
`I
`�
`\Q
`- ......... I
`I
`
`,-
`
`I�
`
`-I'
`
`Page 9 of 147
`
`
`

`

`Sheet 9 of 101 6,144,366
`U.S. Patent
`Nov. 7, 2000
`
`FIG.9
`
`STICK SHAPED OBJECT
`v-- 213
`
`
`
`DETECTION UNIT
`
`,,
`
`STICK SHAPED OBJECT TIP
`UNIT v-- 214
`END EXTRACTION
`
`,,
`
`CENTER OF GRAVITY
`
`
`
`DETECTION UNIT
`
`v-- 215
`
`

`

`D
`
`U.S. Patent OlOld
`
`,_ -
`
`-_,,
`�
`
`0
`'l"""'""f
`
`•
`
`\0
`
`.....
`N
`
`Page 11 of 147
`
`6,144,366
`Sheet 10 of 101 6,144,366
`U.S. Patent
`Nov.7, 2000
`Sheet 10 of 101
`Nov. 7, 2000
`
`,_ ,_
`� c
`.... ....
`c
`_,, _,,
`� �
`
`00
`
`.....
`N
`
`
`
`r-- .....
`
`N
`
`

`

`6,144,366
`U.S. Patent
`Nov. 7, 2000 Sheet 11 of 101
`
`FIG.1 lA
`
`FINGER TIP
`MOVEMENT
`����- - - - -_ l � - - _-__
`
`i.---.----.---&.-...--.----�>---
`
`-+--+-......._,__,_____,,____,�I--'>------'"---
`
`
`
`- -----] -- -------
`
`CENTER OF- 1.-----'&-.......__,����"---
`
`GRAVITY
`M OVEMENT
`
`(a)
`
`(b)
`
`FIG.llB
`
`-
`
`FINGER TIP
`- ����-----.-�-MOVEMENT-'---.___.___-L--
`-----_]�------
`- -----_j�------
`-1<--¼-........._. __ __,__,l.-l->.-t�ol---CENTER OF_ �____,�___,�___,"---
`GRAVITY
`MOVEMENT _ +--+-_,__,_____,,____,���
`
`-�---__,._�-�-
`
`(c)
`
`(d)
`
`

`

`6,144,366
`U.S. Patent
`Sheet 12 of 101
`Nov. 7, 2000
`
`FIG.12
`
`I
`I
`I •
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`CALCULATED
`
`I
`I
`
`I
`I
`FINGER TIP
`I
`I
`
`�
`MOVEMENT
`I
`CENTER OF GRAVITY
`I
`I
`DOES NOT CHANGE
`I
`I
`I
`
`X-COORDINATE OF
`
`I
`I
`I
`
`I
`
`•
`
`

`

`status-EXTENDED
`
`[n].
`
`n ].len gth-+ 1
`
`infoStick
`infoStick[
`SEQUENCE
`END OF PIXEL
`OF
`-X-COORDINATE
`infoStick [ n]. xend[ i]
`SEQUENCE
`ST ART OF PIXEL
`-X-COORDINATE
`infoStick[n].xstart[i
`CANDIDATE
`UPDATE DATA OF THAT
`
`OF
`
`OF
`
`]
`
`226
`
`FIG.13
`
`YES
`
`PREVIOU
`CANDIDAT
`Y OF STIC
`ONNECT
`
`220
`
`ELEMENTS
`
`infoStick[n]
`
`CANDIDATE S
`SHAPED OBJECT
`FOR STORING STICK
`STRUCTURE
`PREP ARE SEVERAL
`
`SEQUENCE
`END OF PIXEL
`OF
`-X-COORDINATE
`n] .xend[0]
`infoStick[
`SEQUENCE
`START OF PIXEL
`-X-COORDINATE
`infoStick[n].xstart[0]
`CANDIDA TE
`SHAPED OBJECT
`REGISTER NEW STICK
`
`infoStick[n].status-EXTENDED
`infoStick[n].length-1
`
`tick [ n] .Ien g
`SCRIBED LEN
`APED OBJECT
`GTH OF UPD
`
`GO TO NEXT LINE223
`NOSTICK
`infoStick[n].status­
`
`FINISHED
`
`NO(ONE LINE)
`
`____
`
`__
`
`....:..,__
`
`_.
`
`THAN c CONSECUTIVE
`PIXEL
`
`ANY
`
`ABOVE pFOUND BY
`WITH PIXEL VALUES
`HA YING NO MORE
`PIXEL SEQUENCE
`
`LINE SCA
`HORIZONTAL
`
`ONE LINE
`NOT UPDATED BY THIS222
`OBJECT CANDIDATE
`DELETE STICK SHAPED
`
`

`

`U.S. Patent
`
`Nov. 7, 2000
`Sheet 14 of 101 6,144,366
`
`FIG.14
`
`�229
`
`t
`
`k-1
`
`i=O
`
`t
`
`t
`
`infoStick[n],xend[i]
`
`j=infoStick[n],xstart[i]
`
`CALCULATE TH
`
`TH=(sum/max)2 Xmax
`sum:MEAN OF PIXEL VALUE SUMS PER LINE FOR----230
`1ST & 2ND LINES FROM TOP OF STICK SHAPED
`OBJECT (TOPMOST IS 0-TH LINE)
`
`max:MAXIMUM PIXEL VALUE IN 1ST & 2ND LINES
`
`OBTAIN k & a (0<a:S: 1) THAT SATISFY:
`
`TH= � xsum[i]+a · xsum[k]
`
`WHERE xsum[i] IS PIXEL VALUE SUM OF
`PARTIAL PIXEL SEQUENCE FOR i-TH LINE
`OF STICK SHAPED OBJECT
`MULTIPLY a WITH EACH PIXEL VALUE OF PARTIAL 232
`PIXEL SEQUENCE FOR k-TH LINE
`OBTAIN CENTER OF GRAVITY (xx, yy)
`yy= [ii xsum[i](infoStick[
`n ]. ystart+i)] / TH
`xx{i xpsum[i]
`]/ TH
`xpsum[i]= � j · P(j,infoStick[n].ystart+i)
`
`
`r---..- 231
`
`r---..-
`
`r---..- 233
`
`

`

`238 '---....,.
`
`CONNECTED TO STICK SHAPED OBJECT CANDIDATE
`AFfER PARTIAL PIXEL IN MIDDLE OF y=5 LINE IS
`
`235
`
`infoStick[0]
`.status=EXTENDED
`inf oStick[0]
`.length=2
`infoStick[0]
`.xend[l
`]=7
`infoStick[0]
`.xend[0]=6
`infoStick[0]
`.xstart[
`1 ]=5
`.xstart[0]=6
`.ystart=3
`
`UP TO y=4 LINE
`
`AFfER SCAN IS FINISHED
`
`23 7 '---....,. infoStick[0]
`infoStick[0]
`
`.status=EXTENDED
`.length=
`.xend[0]=6
`.xstart[0]=6
`.ystart=3
`
`infoStick[0]
`infoStick[0]
`infoStick[0]
`infoStick[0]
`infoStick[0]
`
`1
`
`236 '---....,.
`
`IT AS NEW STICK SHAPED OBJECT
`
`CANDIDATE
`REGISTERING
`AFfER FINDING PARTIAL PIXEL IN y=3 LINE &
`
`FIQ--=..: 15=-------------=--=--=-=-=---
`
`0 1 2 3 4 5 6 7 8
`x---
`
`91011
`-------
`
`-------------
`
`.length
`
`infoStick[n]
`
`�
`
`infoStick[n]
`FROM TOP OF STICK SHAPED OBJECT
`IN i-TH LINE
`START OF X-COORDINATE
`
`.xend[i]
`
`.xstart[i]
`
`infoStick[n]
`
`(LINE) AT WHICH
`infoStick[n].ystart
`
`STICK SHAPED OBJECT START
`Y-COORDINATE
`
`l__L___J_...L.....J=
`
`11
`9 10 l-+-+-+-liii4
`234--8 1----+-+-+-+---H
`7
`6H---+-+-+----lf
`5
`
`y O H---+-+-+-+-+----ji-++-t-t--, Hl-+---+---l-+-+--+=l--t-+----t-t---1
`
`(LENGTH IN VERTICAL DIRECTION)
`CURRENT NUMBER OF LINES
`END OF X-COORDINATE
`
`infoStick[0]
`.status=EXTENDED
`infoStick[0]
`.length=3
`infoStick[0]
`.xend[2]=7
`infoStick[0]
`.xend[
`1 ]=7
`.xend[0]=6
`infoStick[0]
`infoStick[0] .xstart[2]
`=5
`infoStick[0]
`]=5
`infoStick[0]
`infoStick[0]
`
`.xstart[l
`.xstart[0]=6
`.ystart=3
`
`IS NOT STICK SHAPED OBJECT CANDIDATE
`
`BY NEW LINE
`
`IS UPDATED (EXTENDED)
`
`EXTENDED:IT
`NOSTICK:IT
`STICK:IT
`
`IS STICK SHAPED OBJECT CANDIDATE
`.status
`
`infoStick[n]
`
`

`

`infoStick[l].status=EXTENDED
`infoStick[
`1
`infoStick[l].xend[0]=6
`
`1] .length=
`
`infoStick[0].status=NOSTICK
`infoStick[0]
`
`.length=
`
`I
`
`242-------.._ infoStick[0].xend[0]=3
`
`1] .xstart[0]=6
`
`infoStick[l].ystart=3
`
`/
`
`infoStick[0]
`infoStick[0].ystart=2
`AFTER SCAN OF y=3 LINE IS FINISHED 243
`
`.xstart[0]=3 infoStick[
`
`infoStick[0].status=EXTENDED
`infoStick[0]
`241 -------.._ infoStick[0]
`infoStick[0]
`infoStick
`AFTER SCAN OF y=2 LINE IS FINISHED
`
`.length=
`.xend[0]=3
`.xstart[0]=3
`
`I
`
`[0]. ystart=2
`
`0 I 2 3 4 5 6 7 8
`
`91011
`
`y O 1-+--+-+--+'"-+-+-+-+-+-+-+-I !! 1-+--+-F"'"t--i---i,,,,,+--+--+-+---+--i
`
`239
`
`x---
`
`FIG.16
`
`11 L......J........L......1.----'"""
`IO 1-+-+-+--t
`
`7
`6 ----
`
`240----8 1-+-+-+--+---tl
`
`

`

`6,144,366
`U.S. Patent
`Sheet 17 of 101
`Nov. 7, 2000
`
`FIG.17
`
`244
`
`X ►
`01234567891011
`yO
`1
`2
`3
`4
`5
`6
`7
`245-- 8
`9
`10
`11 t----+- --+- � i.tffl
`
`

`

`U.S. Patent
`6,144,366
`U.S. Patent
`Sheet 18 of 101 6,144,366
`Nov.7, 2000
`Sheet 18 of 101
`Nov. 7, 2000
`
`FIG
`18
`FIG.18
`
`
`
`Page 19 of 147
`
`

`

`U.S. Patent
`Nov.7, 2000
`Sheet 19 of 101
`6,144,366
`Nov. 7, 2000 Sheet 19 of 101 6,144,366
`U.S. Patent
`
`246247
`
`oO
`<t
`—N
`
`(b)
`
`FIG.19
`
`
`246247
`
` (a)
`
`oO
`
`=N
`
`Page 20 of 147
`
`

`

`Sheet 20 of 101
`6,144,366
`U.S. Patent
`Nov.7, 2000
`6,144,366
`U.S. Patent
`Nov. 7, 2000 Sheet 20 of 101
`
`FIG.20
`
`•
`
`0
`N
`c., �
`�
`
`249250
`
`(c)
`
`249250
`
`(b)
`
`249250
`
`0
`\r)
`('I
`
`(a)
`
`Page 21 of 147
`
`

`

`CURSOR DISPLAY
`
`255
`)
`
`254
`
`)
`
`UNIT
`
`DETECTION
`
`STATE -CLICK SIGNAL
`
`-
`
`STATIONARY
`
`6,144,366
`
`CURSOR
`
`I
`
`UNIT
`
`DETECTION UNIT
`
`VSC
`
`DETERMINATION
`
`
`
`LINONOLLOALYd
`
`POSITION
`
`UNIT
`
`-
`
`CSC
`
`\
`
`253
`
`Sheet 21 of 101
`
`U.S. Patent
`
`Nov.7, 2000
`
`CURSOR POSITION
`
`\
`
`TIVNDISMOTTO
`
`
`HLVLISAUVNOILVLS
`
`252
`
`FIG.21
`
`TINN
`
`AVIdSIGYOSAND
`
`NOILISOdYOSANO
`NOILVNINYALAG
`
`dILYHONIA
`
`NOILISOd
`
`ESCCSC[Sz
`
`IcDi
`
`FINGER TIP
`
`dosano
`
`251
`
`LINN
`
`TINONOLLOYeLAG
`
`
`
`
`

`

`257
`
`)
`
`SIGNAL
`
`-COMMAND OR CLICK
`
`6,144,366
`
`256
`
`)
`
`UNIT
`
`DETECTION
`
`MOVEMENT
`
`TIP POSITION
`
`CURSOR OR FINGER
`
`ii
`
`9S¢
`
`UNIT UNIT
`
`DETECTION
`
`UNIT
`
`-
`
`DETERMINATION
`
`LST\LINQNOLLOALEG
`
`CURSOR DISPLAY
`
`-
`
`POSITION
`
`LNAWNAAOW
`
`FINGER TIP
`
`253
`
`�
`
`Sheet 22 of 101
`
`
`
`TIVNOISMOITIOYOANVNWOD
`
`CURSOR POSITION
`
`)
`252
`
`
`
`NOILISOddLL
`
`)
`
`qdHONITYOAOSANO
`
`251
`
`U.S. Patent
`
`Nov.7, 2000
`
`FIG.22
`
`LIND
`
`AVIdSIGAOSANO
`
`NOILISOdYOSANO
`NOILVNINASALAC
`
`LINN,
`
`LINDNOLLOYLAd
`
`dILYONI
`
`NOILISOd
`
`tSccSé1S¢
`
`c¢DIA
`
`
`
`
`
`

`

`RESULT
`
`INTERPRETATION
`
`�
`
`'"""'
`
`'"""'
`
`331
`
`(
`
`333 ....
`
`INTERPRETATION
`
`UNIT
`
`�
`�
`
`, RULES
`
`MATRIX SHAPE
`DISTANCE
`
`-MEMORY UNIT
`INTENSITY RANGE IMAGE
`LIGHT
`
`TIMING
`
`ATION
`
`INTERPRET
`
`SHAPE
`
`332
`
`�
`
`MEMORY UNIT
`
`SHAPE
`
`FIG.23
`
`

`

`6,144,366
`U.S. Patent
`Nov. 7, 2000
`Sheet 24 of 101
`6,144,366
`U.S. Patent
`Nov. 7, 2000 Sheet 24 of 101
`
`
`
`
`
`-
`
`-­u
`
` FIG,.24
`
`I
`
`--
`
`'"O
`
`
`
`
`
`. . ' • I '. •
`. .. I · .
`. ·
`
`"'F-+--+-1 --
`
`� l--l---l--i---4--!-4-+-i---i--1--1--1--1---i--l--l-+--ol
`
`tjFP_Image
`
`
`
`
`
`
`
`
`Page 25 of 147
`
`
`
`
`

`

`U.S. Patent Nov. 7, 2000 Sheet 25 of 101
`6,144,366
`
`FIG.25
`
`START
`
`STORE RANGE IMAGE
`INTO RANGE IMAGE 401
`MEMORY UNIT
`
`READ DISTANCE MA TRIX 402
`
`DISTANCE MATRIX 403
`PROCESSING
`
`CHING S
`RPRETA
`LE EXI
`
`NO
`�--�
`
`YES
`
`406
`
`CHANGE THRESHOLD
`& GO TO 403
`
`OUTPUT COMMAND
`405
`
`CORRESPONDING TO
`MATCHED SHAPE
`
`END
`
`

`

`6,144,366
`U.S. Patent Nov. 7, 2000 Sheet 26 of 101
`
`FIG.26
`
`
`
`SHAPE INTERPRETATION RULES FOR POINTING
`
`RULE -1 :ONE RECTANGLE-POINTING
`
`RULE -2:TWO RECTANGLES-TWO FINGER OPERATION
`
`
`
`RULE -3:OTHERS--MOTION INPUT
`
`POINTING RULE
`
`
`VERTICAL & HORIZONTAL CHECK
`
`
`
`VERTICAL/HORIZONTAL::::: I-OBJECT SELECT/MOVE FORWARD
`
`
`
`VERTICAL/HORIZONTAL<l--SLOPE CHECK-1
`
`VERTICAL/HORIZONTAL>l--SLOPE CHECK-2
`
`
`SLOPE CHECK-1
`
`
`
`SLOPE:::::0 & CENTER OF GRAVITY RIGHT-ROTATION TO RIGHT
`
`
`
`SLOPE:::::0 & CENTER OF GRAVITY LEFT-ROTATION TO LEFT
`
`SLOPE CHECK-2
`
`SLOPE::::::o & CENTER OF GRAVITY UPPER-TURNING UPWARD
`
`
`SLOPE::::: 0 & CENTER OF GRAVITY LOWER
`--TURNING DOWNWARD
`
`

`

`6,144,366
`U.S. Patent
`6,144,366
`U.S. Patent
`Sheet 27 of 101
`Nov.7, 2000
`Nov. 7, 2000
`Sheet 27 of 101
`
`
`
`C
`
`..
`
`-
`
`PittTeyTee
`ptittttteyLitTTtTrrr
`
`
`
`
`
`
`pttPytTTTaSEReeePtPPPPeepereeeLtTteTTTSERSSSE.|SEeeeenneLtTTteTTTT[|TTTeeTTLttTTPrrryyPittTTePrET
`
`
`Li]tptTeaTyLTTeTTTeeBEER
`(J(2PttTTPTTTTrtLTTTTETTTTTTTTryTy1]tttteey
`
`SERRESERREEREPyTeTeLEPtTpierTTTPereyyrrrSERRE
`PEPTTTee
`PteTTTeLTTTTTTPTPrTrLtteTtaTTerLPreTTrrPre
`PityeetrrrPETTTTPTrrrey
`1}tyty||LtTYPtTt|{|LttTY||tty||PTiTTLttt||e808||ft||LyyyyLtTTY||ttydLTTT|ityPtfta]TTTEETT|ft|egtHLttTT|tttPtTTTTeeTrittTPLittyttyepPePtftPteyTtPTTTLt|PittTTETT|Ppt
`treee
`
`
`t-+----+-1--t-+
`
`1 � 1.
`\ ···•·····
`
`1--+-+-+-+-+--l-l----1'"'4�"-4---1-l---1-+-1 .---..
`1--+-+-+-+-+--+-+-+-+�-+--l-l---1-+-1 ..0
`
`PLTTTTPTETerrtrrtT
`PittTTTetPtittTTTTTyeeer
`
`1-+-+--+-+--+-+--i- �--4--l--4--l--4---lf--l .---..
`t-+-+-+--+-4---1--1----1---4----+-4-+-il--l--� ,:,:s
`
`Page 28 of 147
`
`■ -
`■ -
`
`
`
`
`
`
`
`
`

`

`|||a PTTTTTTTTrtTTT
`
`LETTywTPPTTPTTTTTee|miTPTr||IWTE|||eS||TTAi||ThWaLttTAitt|TThe
`PETTPtTer
`||a|a||aerLtetTP[|MTPTritittyetryLtTTETTeTr
`PLETTTTyTey
`
`
`
`
`
`Page 29 of 147
`
`U.S. Patent
`6,144,366
`U.S. Patent
`Sheet 28 of 101 6,144,366
`Nov.7, 2000
`Sheet 28 of 101
`Nov. 7, 2000
`
`RoaeTE
`
`PPPP
`PP
`
`PTTTTTTSTTT
`LTTTTTy
`LttTerrs
`wt
`LttTTT
`LITttTrrin
`PETTT
`cc
`|tT
`TTRe
`PTeeTTTTY
`mtN
`}TT
`PitTTTyr
`“eS!
`STtTTT
`ye
`PEtTy
`
`a|
`
`
`
`
`
`
`

`

`Nov. 7, 2000 Sheet 29 of 101 6,144,366
`U.S. Patent
`
`FIG.29
`
`
`
`SHAPE INTERPRETATION RULES FOR POWER ON/OFF
`
`RULE -1:ONE RECTANGLE-POINTING
`
`RULE-2:TWO RECTANGLES-TWO FINGERS
`
`
`RULE -3:FIVE RECTANGLES-FIVE FINGERS
`
`POINTING RULE
`AREA CHECK
`
`AREA>> -FIST
`a
`AREAS a-NULL
`
`COMMAND GENERATION
`FIVE FINGERS, FIST, TWO FINGERS-POWER OFF
`
`
`
`TWO FINGERS, FIST, FIVE FINGERS-POWER ON
`
`

`

`U.S. Patent Nov. 7, 2000 Sheet 30 of 101 6,144,366
`
`FIG.30
`
`-----
`
`/
`
`,
`
`/
`
`'
`
`' '
`
`'
`
`\
`
` \
`
`\
`
`'
`\
`
`<
`
`�
`
`\
`
`'
`\
`
`' '
`
`'
`
`�
`
`�
`
`\
`�\
`PYRAMID-, \
`\ I
`
`SPHERE
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`
`
`VIEWPOINT-2 VIEWPOINT-I
`VIEWPOINT-3
`
`

`

`6,144,366
`U.S. Patent
`U.S. Patent
`Sheet 31 of 101 6,144,366
`Sheet 31 of 101
`Nov.7, 2000
`Nov. 7, 2000
`
`7LNIOdMAIA(4
`
`€-LNIOdMAIA(9 [-LNIOdMAIAC
`
`
`
`Page 32 of 147
`
`

`

`-- '
`-t-
`---
`---
`-t-
`-t-
`--
`-- 0:::
`---
`--t-
`--
`□
`-t-
`-t- .-� ---
`
`
`
`--
`t- -
`--
`t- -
`
`--
`--
`
`--
`
`t- -
`--
`t- - D
`
`....:l
`
`---
`0::: ---
`->-- -
`---
`---
`--t-
`---
`---
`
`6,144,366
`U.S. Patent
`Sheet 32 of 101 6,144,366
`U.S. Patent
`Sheet 32 of 101
`Nov.7, 2000
`Nov. 7, 2000
`
`-t-
`
`-- L �
`--
`-
`-
`--
`-
`-
`
`-
`
`-
`
`>--
`
`-
`
`-
`
`--
`
`□
`
`□
`
`f---t-
`
`t-
`--
`
`t-
`--
`
`--t-
`
`--t-
`
`-
`
`--
`--t-
`-
`--t-
`--
`---
`--t-
`---
`---
`--t-
`
`->--
`
`-
`
`->--
`
`-
`
`N ('<")
`
`Page 33 of 147
`
`

`

`6,144,366
`U.S. Patent
`Nov. 7, 2000
`Sheet 33 of 101
`
`FIG.33
`
`SHAPE INTERPRETATION RULES FOR
`
`MOVING OBJECT PARALLAX
`
`RULE-1:TWO RECTANGLES-FACE CHECK
`
`
`RULE -2:OTHERS--NULL
`
`DISTANCE TO LARGER RECTANGLE>DISTANCE TO SMALLER
`
`FACE CHECK
`
`
`RECTANGLE-VIEWPOINT CHECK
`OTHERS---+NULL
`
`VIEWPOINT CHECK
`L =/:c 0---+ VIEWPOINT DIRECTION=arcsin( (L-R)/L)
`
`
`L= 0-- VIEWPOINT DIRECTION=-90
`
`°
`
`

`

`6,144,366
`
`OINT-2
`
`Sheet 34 of 101
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`'
`
`'
`
`I
`
`'
`
`I
`
`I
`
`I
`
`I
`
`I
`
`$!-
`PYRAMID I SPHERE
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`-------�--------------
`
`!
`
`\ VIEWPOINT-I
`,-----------------
`
`/ --
`
`U.S. Patent
`
`Nov.7, 2000
`
`'
`
`I
`
`'
`
`FIG.34
`
`s\€\LNIOdMHIA
`
`T-LNIOdMYIA\JoLeA
`
`\
`
`\
`
`\
`
`\
`
`\
`
`\
`
`\
`
`\
`
`\
`
`\
`
`\
`
`\
`
`\
`
`VIEWPOINT-3
`
`\
`
`\
`
`\
`
`\
`
`,' \
`
`\/
`
`\
`
`/
`
`\
`
`\
`
`\
`
`\
`
`<
`
`
`

`

`RANGE)
`
`(ANGLE)
`(SPEED)
`(SWINGING
`MOTION VECTOR
`
`'
`
`'"""'
`
`'"""'
`
`331
`
`(
`
`343 �
`
`�
`
`UNIT
`EXTRACTION ,......._____.
`IMAGE CHANGE
`
`DISTANCE
`
`342
`
`MEMORY UNIT
`POSITION i,--..__,.,
`COMPARISON
`
`FIG.35
`
`-
`
`MEMORY UNIT
`
`TIMING
`
`INTENSITY RANGE IMAGE MATRIX
`LIGHT
`
`

`

`U.S. Patent
`6,144,366
`000 Sheet 36 of 101
`Nov. 7, 2
`
`FIG.36
`
`START
`
`INITIALIZE COMPARISON
`901
`
`POSITION MEMORY UNIT
`
`STORE RANGE IMAGE
`INTO RANGE IMAGE
`MEMORY UNIT
`
`902
`
`READ DISTANCE MA TRIX
`
`903
`
`EXTRACT CENTER OF
`GRAVITY POSITION
`
`904
`
`OF GRAVITY
`SITION STORED
`ARISON POSI
`MORY UN
`
`YES
`
`NO
`
`TO 907
`
`CALCULATE MOTION VECTOR
`
`
`
`906
`(CALCULATE SWINGING RANGE)
`(908)
`(CALCULATE SPEED)
`(909)
`(910)
`
`(CALCULATE ANGLE)
`
`----------------
`
`STORE NEW CENTER OF GRAVITY
`907
`
`POSITION INTO COMPARISON
`
`POSITION MEMORY UNIT
`
`TO 902
`
`

`

`Sheet 37 of 101 6,144,366
`U.S. Patent
`Nov. 7, 2000
`
`FIG.37
`
`v=[ (Xn, Y n)-(Xn-1, Y n-1) ]/M · 30
`
`FIG.38
`
`
`
`SWINGING RANGE(Xn-Xn-1)/M · 30
`
`�✓ I
`
`I
`
`SPEED(Y
`n-Y n-1 )/M · 30
`
`

`

`Sheet 38 of 101 6,144,366
`U.S. Patent
`Nov. 7, 2000
`
`FIG.39
`
`FINGER AT TIME t FINGER AT TIME t+ I
`
`a)
`
`Xbt,Ybt
`
`
`
`Xst,Yst Xst+ l,Yst+ I
`
`b)
`
`
`
`
`
`Xbt,Ybt X'bt+ 1,Y'bt+ 1
`
`
`
`Xst,Yst X'st+ 1,Y'st+ 1
`
`
`
`
`
`

`

`U.S. Patent Nov. 7, 2000 Sheet 39 of 101
`
`6,144,366
`
`FIG.40
`
`SWINGING
`
`ANGLE
`a
`
`SWINGING
`
`ANGLE a
`
`

`

`6,144,366
`Nov. 7, 2000 Sheet 40 of 101
`U.S. Patent
`
`FIG.41
`
`SWINGING
`
`ANGLE a
`
`

`

`Sheet 41 of 101
`U.S. Patent
`Nov.7, 2000
`U.S. Patent
`Sheet 41 of 101 6,144,366
`Nov. 7, 2000
`
` 6,144,366
`
`FIG.42
`
`Page 42 of 147
`
`

`

`'"""'
`
`'"""'
`
`�
`�
`
`DRAWING UNIT i--------.___,, 346
`
`� 345
`
`CONTROL UNIT
`CHARACTER
`
`333
`
`CONTROL UNIT
`MAPPING i--------.___,, 344
`PARAMETER
`
`RESULT
`INTERPRETATION
`
`(
`
`UNIT
`
`SHAPE
`
`i-----. INTERPRETATION
`
`332
`
`(
`
`UNIT
`SHAPE MEMORY
`
`RANGE (ANGLE)
`TION VECTOR
`
`�SWINGING
`SPEED, MO
`
`OF GRAVITY
`AREA,CENTER
`
`UNIT
`EXTRACTION i--------.___,, 343
`IMAGE CHANGE
`
`:::
`
`TIMING :;. MEMORY UNIT
`
`( DISTANCE
`
`INTENSITY RANGE IMAGE MATRIX
`LIGHT
`
`MEMORY UNIT
`POSITION i--------.___,, 342
`COMPARISON
`
`FIG.43
`
`331
`
`

`

`Sheet 43 of 101
`6,144,366
`U.S. Patent
`Nov.7, 2000
`Sheet 43 of 101 6,144,366
`U.S. Patent
`Nov. 7, 2000
`
`FIG.44
`
`Page 44 of 147
`
`

`

`Sheet 44 of 101
`6,144,366
`U.S. Patent
`Nov.7, 2000
`U.S. Patent
`Sheet 44 of 101 6,144,366
`Nov. 7, 2000
`
`FIG.45
`
`Page 45 of 147
`
`

`

`103
`
`}
`
`UNIT
`
`FEATURE DATA GENERATION
`
`354
`
`(
`
`353
`
`(
`
`IMAGE DATAr
`
`MEMORY UNIT
`
`� IMAGE
`
`EXTRACTED
`
`UNIT
`EXTRACTION
`
`352
`
`(
`
`351
`
`(
`
`.____.___--'-_,; IMAGE MEMORY
`
`.....____,J
`
`UNIT
`
`TIMING -
`
`-------
`
`INTENSITY
`LIGHT
`VISIBLE
`
`ARRAY
`DETECTION
`PHOTO­
`VISIBLE LIGHT
`
`331
`
`(
`
`102
`
`(
`
`MATRIX
`REFLECTION
`
`.____.___--'-_,_ MEMORY UNIT
`RANGE IMAGE
`
`UNIT
`EXTRACTION
`LIGHT
`I
`REFLECTED
`
`TIMIN G -
`
`LIGHT
`REFLECTED
`
`FIG.46
`
`

`

`"'-'
`
`"'-'....
`
`--�-
`
`qJjjJJI --.-.-:":/:
`
`""--·':::°:'''''
`
`�
`
` � 0--, 0--,
`0--, .... �
`LIGHT PHOTO-DETECTOR
`
`(if/I .. ti/I _(if/I VISIBLE
`
`... ;:;::,:,�-...,. ....
`
`�:❖:-,· (if/I (/f/14'Pl NEAR INFRARED PHOTO-DETECTOR
`
`�--
`
`�ts:7'
`
`�
`
`= ......
`
`---------
`
`----------
`
`FIG.47
`
`PHOTO-DETECTION
`,,,,--------------
`�
`
`LENS
`
`'"""'
`0
`'"""'
`
`rF.J.=­�
`
`0
`0
`0
`N
`�-..J
`
`z 0 �
`
`0 ....,
` �
`
`O'I
`
`.... ,i;;..
`
`
`
`...... �
` � �
`
`
`
`d•r:JJ.•
`
`

`

`U.S. Patent Nov. 7, 2000 Sheet 47 of 101 6,144,366
`
`FIG.48
`_ ____ _
`__ ___ _ __ __.A.-_ __
`
`�
`
`ORIGINAL
`
`IMAGE
`
`MASK
`
`---....
`
`\ I
`
`EXTRACTED
`
`IMAGE
`
`

`

`UNIT 103;
`
`FEATURE DATA GENERATION
`
`355
`
`(
`
`RECORDING UNIT
`(COMPRESSION)
`IMAGE
`EXTRACTED
`
`354
`
`(
`
`EXTRACTED
`
`MEMORY UNIT
`
`i-IMAGE
`
`UNIT
`EXTRACTION
`
`331
`
`(
`
`( 102
`
`MATRIX
`REFLECTION
`
`MEMORY UNIT
`RANGE IMAGE
`
` -
`,-i
`
`UNIT
`EXTRACTION
`LIGHT
`REFLECTED
`
`1---'-----+
`
`TIMING -
`
`LIGHT
`REFLECTED
`
`FIG.49
`
`� (
`
`353
`
`IMAGE DAT A
`
`( 351
`
`UNIT
`IMAGE MEMORY 1----'
`
`.---------,
`
`I
`
`ARRAY
`DETECTION
`1--....,__---+-i
`PHOTO­
`VISIBLE LIGHT
`
`TIMING -
`
`INTENSITY
`LIGHT
`VISIBLE
`
`

`

`356
`
`(
`
`UNIT
`TRANSMISSION
`(COMPRESSION)
`
`-�
`IMAGE MEMORY
`
`---+-..i UNIT
`
`UNIT 103;
`
`FEATURE DATA GENERATION
`
`352
`
`(
`
`351
`
`(
`
`ARRAY
`
`TIMING -DETECTION1---
`
`354
`
`(
`
`353
`
`(
`
`IMAGE DATA
`
`MEMORY UNIT
`
`_. IMAGE
`
`EXTRACTED
`
`UNIT
`EXTRACTION
`
`INTENSITY
`LIGHT
`VISIBLE
`
`
`331
`
`(
`
`102
`
`(
`
`MATRIX
`REFLECTION
`
`---+..i MEMORY UNIT
`RANGE IMAGE
`
`I
`
`UNIT
`EXTRACTION
`LIGHT
`REFLECTED
`
`--'--
`
`1---
`
`TIMING -
`
`LIGHT
`REFLECTED
`
`FIG.50
`
`...-------, I
`
`PHOTO-
`VISIBLE LIGHT
`
`--'--
`
`

`

`Sheet 50 of 101 6,144,366
`U.S. Patent
`Nov. 7, 2000
`
`FIG.51
`
`BACKGROUND IMAGE EXTRACTED IMAGE
`
`/
`
`COMPOSED IMAGE
`
`

`

`IMAGE
`COMPOSED
`357
`
`(
`
`UNIT
`COMPOSITION
`IMAGE
`
`6,144,366
`
`354
`
`(
`
`MEMORY UNIT
`
`358--IMAGE
`
`BACKGROUND
`
`352
`
`(
`
`
`351
`
`(
`
`-�
`
`UNIT
`
`ARRAY
`DETECTION
`
`353
`
`
`
`AOVIIAOVWI
`
`
`
`dySOdNWODGNNOYOAVE
`
`IMAGE DATA (
`
`-------
`
`LIGHT
`VISIBLE
`
`VISIBLE LIGHT
`
`LINOAYOWAWN
`
`IN
`
`_ PHOTO-
`
`TENSITY IMAGE MEMORY
`
`EXTRACTED
`
`LINO.
`
`NOILISOdWOD
`
`331
`
`(
`
`MEMORY UNIT
`
`LSet
`
`1--. IMAGE
`
`UNIT
`EXTRACTION
`
`Sheet 51 of 101
`
`Nov. 7, 2000
`
`MATRIX
`REFLECTION
`
`LINNAYOWHWN
`
`FIG.52
`
`GHLOVaLxd
`AHOVWI
`NOILOVALXA
`
`MEMORY UNIT
`LIGHT RANGE IMAGE
`REFLECTED
`
`LIND
`
`UNIT
`EXTRACTION
`LIGHT
`REFLECTED
`
`
`
`LIN?NOILOALAd
`
`102
`
`AVUaAV
`
`(
`
`-
`
`ATdISIA
`
`U.S. Patent
`
`NOLLOYTa
`
`XIULVN
`
`ThonLHOIT
`
`LINA
`
`OVINEFON|gpaaayCsLoddad cSDJA
`
`
`
`HOVNIAUYOWANAOVII|ALISNALNI-OLOHd LHor]|~LHOIT
`
`LINNAUYOWAWNNOLLOVULX4
`
`

`

`354
`
`(
`
`MEMORY UNIT
`IMAGE
`EXTRACTED
`
`353
`
`(
`
`'
`
`MATRIX
`REFLECTION
`
`_ __,
`IMAGE DATA
`UNIT
`EXTRACTION
`
`INTENSITY
`LIGHT
`I VISIBLE
`I UNIT
`-��� IMAGE MEMORY
`
`352
`
`(
`
`351
`
`ARRAY
`DETECTION
`PHOTO­
`VISIB LE LIGHT
`
`TIMING -
`
`SWITCH UNIT 359
`
`MEMORY UNIT
`RANGE IMAGE
`
`UNIT
`EXTRACTI ON
`TIMING _ LIGHT
`r--R...l.E-FL_E_C_T_E_D__I
`t LIGHT
`102 REFLEC TED
`
`331
`
`t 7
`
`FIG.53
`
`

`

`362
`
`(
`
`361
`
`DRAWING UNIT
`
`MEMORY UNIT
`
`IMAGE
`
`Z-VALUE
`
`6,144,366
`
`U.S. Patent
`
`Nov.7, 2000
`
`Sheet 53 of 101
`
`352
`
`351
`
`(
`
`UNIT
`
`(
`
`ARRAY
`
`DETECTION
`
`353
`
`IMAGE DATA (
`
`c9E
`
`UNIT
`
`EXTRACTION
`
`VISIBLE
`
`INTENSITY IMAGE MEMORY
`
`PHOTO-
`VISIBLE LIGHT LIGHT
`
`Z-VALUES
`
`DATA+
`
`IMAGE
`
`331
`
`(
`
`102
`
`(
`
`UNIT
`
`EXTRACTION
`
` vsOld
`LINNONIMVUGLINNOILOdLad
`weestVIVdHOVWIATEISIA
`
`
`HOVNITIND
`
`XTaLVNADVIADNVYLHOIT
`
`
`LINNAYOWSWNOILOVULXA
`GaALOAWAYda.LOd44a
`
`
`LHor1|LHOM!ATaISIA
`
`MEMORY UNIT
`
`LIGHT RANGE IMAGE MATRIX
`
`REFLECTION
`
`LINDAYOWNYWN
`
`ANYIVA-Z
`
`HOVWI
`
`FIG.54
`
`+VLVC
`
`SHNTVA-Z
`
`NOILOVaLxa
`
`LIN‘
`
`NOLLOAWaa
`
`LIGHT
`REFLECTED REFLECTED
`
`LHOIT
`
`AVUAV
`
`
`
`AYOWHWAOVWI|ALISNALNI~OLOHd
`
`
`

`

`| ane8L9=S=v۩ZinIWw
`
`12 t-t----t-+-t-t--t-t-t----t-+-t-t--;--t-t----t
`11
`10 1-+-+-+-+--+--+-1-+-+-+-+-+--+-1-+--l
`9
`8
`7
`6
`5
`4
`3
`2
`1
`
`3141516
`
`l2 34567891011121
`
`-+-+-+-+-+-+-+-+-+-+-+-+--+---+-1
`-+-+-+-+-+-+-+-+-+-+-+-+--+---+-1
`
`16 t-+--+-+-+-+-+-+----+-+-t-t--t-i-t---t--1
`15
`
`
` t-+- t-+-
`13
`14
`
`(
`
`a)
`
`aaa tnNoRnaonlraesy
`
`FNM TMNOM OD
`
`oe=N
`
`—oe
`
`6,144,366
`
`-
`
`10111213141516
`
`1 2 3 4 5 6 7 8 9
`
`16 '---'----'---'--'---'--'-'---'----'---'--'---'--'-'---'--'
`15
`14
`
`1-+--+-+-+---+--+-+-+--+----+----+----+----,1--+--+--1
`
`-+-+-+--+--+-1-+-+-+-+-+--+-1-+--l
`
` 1-+
` t-t-
`13
`12
`
`---t-+-t-t--t-t-t----t-+-t-t--t-t-+----t
`
`c)
`
`1 1
`10 t-+--+-+--+--+-
`9
`8
`7
`6
`5
`4
`3
`2
`1
`
`91011121314
`
`16
`1 2 3 4 5 6 7 8
`
`eSeSeSNX~>oZz.
`15
`
`FIG.55
`
`U.S. Patent
`
`16
`15
`14
`13
`12
`11
`JO
`9
`8
`7
`6
`5
`4
`3
`2
`1
`
`b)
`
`
`

`

`U.S. Patent
`Nov.7, 2000
`Sheet 55 of 101
`6,144,366
`U.S. Patent Nov. 7, 2000 Sheet 55 of 101 6,144,366
`
`FIG.56
`
`
`
`a)
`
`b)
`
`c)
`
`Page 56 of 147
`
`

`

`352
`
`(
`
`INTENSITY
`LIGHT
`VISIBLE
`
`1
`
`35
`
`361
`
`(
`
`353
`
`(
`
`IMAGE DATA
`
`.--.,.........i IMAGE MEMORY
`
`,________.
`
`UNIT
`
`ARRAY
`DETECTION
`PHOTO­
`VISIBLE LIGHT
`
`TIMING -
`
`MEMORY UNIT
`IMAGE
`Z-VALUE
`
`r--------
`
`Z-V ALUES
`DATA+
`IMAGE
`
`MATRIX
`REFLECTION
`
`MEMORY UNIT
`RANGE IMAGE
`
`LIGHT
`REFLECTED
`
`FIG.57
`
`UNIT
`EXTRACTION
`I
`LIGHT
`REFLECTED
`
`TIMING -
`
`UNIT
`
`t---------'--' EXTRACTION
`
`�363
`UNIT
`DETERMINATION
`RANGE
`EXTRACTION
`
` r----.....1-___
`
`7
`
`331
`
`

`

`Sheet 57 of 101
`6,144,366
`U.S. Patent
`Nov.7, 2000
`U.S. Patent
`6,144,366
`Nov. 7, 2000
`Sheet 57 of 101
`
`b) NX
`
`PETETTTey
`PTTTereTT BetI
`rt
`
`"'
`
`___ ,_,_ ,-�
`:,:,;
`''"'''
`-:, \
`
`1
`
`'"''"''''''''
`
`l'r--.
`"
`
`i'--
`i'
`
`\n 11;>9
`
`/.
`
`YO
`C"l
`u
`N
`
`ON
`
`.....
`u
`
`'
`
`>
`
`,--..
`(.)
`
`il
`|_|
`|_|
`tI
`|]
`Tt|
`td
`||
`an
`
`an
`
`CO
`”)
`
`Sa
`
`e
`
`Page 58 of 147
`
`HH
`
`an PPPeryPyTy
`an
`rte
`||
`ritytl
`{|
`|
`Tl
`
`

`

`Sheet 58 of 101 6,144,366
`U.S. Patent
`Nov. 7, 2000
`
`FIG.59
`
`REFLECTED
`.----------,--LIGHT IMAGE � 721
`CONTROL GENERATION
`SIGNAL
`UNIT
`720
`\
`
`TIMING
`CONTROL
`UNIT
`
`SIGNALS
`CORRESPONDING
`TO REFLECTED
`LIGHT IMAGE
`
`'
`
`NONLINEAR
`CONVERSION � 722
`UNIT
`
`i
`SIGNALS CORRESPONDING
`TO RANGE IMAGE
`
`

`

`Sheet 59 of 101 6,144,366
`U.S. Patent
`Nov. 7, 2000
`
`FIG.60
`
`REFLECTED
`�--- LIGHT IMAGE f--,--721
`CONTROL GENERATION
`SIGNAL
`UNIT
`
`723
`\
`TIMING
`CONTROL
`UNIT
`
`SIGNALS
`CORRESPONDING
`TO REFLECTED
`LIGHT IMAGE
`
`COORDINATE
`SIGNAL
`
`i-----724
`A/D
`CONVERTER
`
`DIGITAL
`REFLECTED
`LIGHT IMAGE
`
`CORRECTION � 725
`UNIT
`
`i CORRECTED
`
`RANGE
`IMAGE
`
`

`

`U.S. Patent
`Sheet 60 of 101 6,144,366
`Nov. 7, 2000
`
`FIG.61
`
`.__
`
`__.
`
`REFLECTED
`.---� LIGHT IMAGE -------721
`CONTROL
`GENERATION
`SIGNAL
`UNIT
`723
`\
`
`____
`
`SIGNALS
`CORRESPONDING
`TO REFLECTED
`l LIGHT IMAGE
`
`TIMING
`CONTROL
`UNIT
`
`COORDINATE
`SIGNAL
`
`NONLINEAR
`CONVERSION � 726
`UNIT
`
`SIGNALS
`CORRESPONDING
`TO RANGE
`IMAGE
`
`AID
`-----724
`CONVERTER
`
`DIGITAL
`RANGE
`IMAGE
`
`CORRECTION _______ 727
`UNIT
`
`t CORRECTED
`
`RANGE
`IMAGE
`
`

`

`6,144,366
`U.S. Patent
`Nov. 7, 2000 Sheet 61 of 101
`
`FIG.62
`
`800
`
`700
`
`600
`
`E-< 500
`:::i
`� 400
`:::i
`0 300
`200
`
`100
`
`0
`0
`
`-
`
`50 · 100 150
`250 300
`200
`INPUT
`
`

`

`Sheet 62 of 101 6,144,366
`U.S. Patent
`Nov. 7, 2000
`
`FIG.63
`
`REFLECTED
`.------- LIGHT IMAGE i---- 721
`CONTROL GENERATION
`SIGNAL
`UNIT
`723
`\
`TIMING
`CONTROL
`UNIT
`
`SIGNALS
`CORRESPONDING
`TO REFLECTED
`LIGHT IMAGE
`
`COORDINATE
`SIGNAL
`
`A/D
`i-----724
`CONVERTER
`
`DIGITAL
`REFLECTED
`w LIGHT IMAGE
`
`CORRECTION � 728
`TABLE
`
`i CORRECTED
`RANGE IMAGE
`
`

`

`2000 Sheet 63 of 101 6,144,366
`U.S. Patent
`Nov. 7,
`
`�RDI-
`ATE
`IN
`
`FIG.64
`
`I I
`
`0 1 2
`
`4094 4095
`
`0
`
`1
`
`2
`
`3
`
`4
`
`255 255 255
`255 255 255
`255 255 255
`255 255 255
`255 255 255
`
`5
`
`255 255 255
`
`6
`
`7
`
`8
`
`255 255 255
`255 255 255
`255 255 255
`
`255 255
`255 255
`255 255
`
`255 255
`255 255
`255 255
`255 255
`
`255 255
`255 255
`
`-�
`_,..---
`
`99
`100
`
`101
`
`39 37 40
`38 36 39
`38 36 39
`
`102
`
`103
`
`104
`
`37 35 38
`37 35 38
`36 34 37
`
`105
`
`36 34 37
`
`106
`
`35 33 36
`
`107
`
`108
`
`109
`
`35 33 36
`34 32 35
`34 32 35
`
`=--
`
`---247
`248
`
`249
`250
`
`251
`
`252
`
`253
`254
`255
`
`1 0 2
`1 0 2
`1 0 2
`1 0 2
`1 0 2
`0 1
`0
`0 0 1
`0 0 1
`0 1
`0
`
`38 39
`37 38
`
`37 38
`
`36 37
`
`36 37
`35 36
`
`35 36
`34 35
`34 35
`
`34 34
`33 34
`
`1 1
`
`1
`
`1
`
`1
`
`1
`
`1
`1
`0 1
`0
`0
`
`0
`
`0
`
`0 0
`
`0 0
`
`

`

`6,144,366
`U.S. Patent
`Nov. 7, 2000 Sheet 64 of 101
`
`FIG.65
`
`CONTROL .---
`SIGNAL
`
`-
`
`...i -
`
`REFLECTED
`LIGHT IMAGE
`i------- 721
`GENERATION
`UNIT
`
`--------l
`
`723�
`
`TIMING
`CONTROL
`UNIT
`
`COORDINATE
`SIGNAL
`
`SIGNALS
`CORRESPONDING
`TO REFLECTED
`LIGHT IMAGE
`
`___,----.724
`AID
`CONVERTER
`
`DIGITAL
`REFLECTED
`LIGHT IMAGE
`
`'
`
`730
`\
`
`CORRECTION
`CORRECTION
`t--------' CALCULATION �729
`TABLE
`CORRECTION UNIT
`DATA
`
`�
`
`CORRECTED
`RANGE IMAGE
`
`

`

`6,144,366
`U.S. Patent
`Sheet 65 of 101
`Nov. 7, 2000
`
`�ROI-
`
`IN
`
`FIG.66
`
`I I
`
`ATE
`
`0 1 2
`
`4094 4095
`
`0
`
`255 255 255
`
`255 255
`
`8
`
`296 290 293
`
`298 296
`
`16
`
`191 187 188
`
`24
`
`144 141 141
`
`193 191
`146 144
`
`32
`
`116 114 113
`
`118 116
`
`40
`
`48
`
`56
`
`64
`
`72
`
`80
`
`88
`
`97 95 94
`83 82 80
`72 71 69
`64 62 61
`56 55 53
`50 49 47
`45 44 42
`
`96
`
`40 39 37
`36 35 33
`104
`32 32 29
`112
`120
`29 29 26
`
`128
`
`136
`
`144
`
`26 26 23
`24 23 21
`21 21 18
`
`152
`
`19 18 16
`
`160
`
`168
`
`176
`
`184
`
`192
`
`17 16 14
`15 15 12
`10
`13 13
`11 11 8
`10 10 7
`
`200
`
`8 8 5
`
`208
`
`7
`
`7 4
`
`216
`
`6 5 3
`
`224
`
`4 4 1
`
`232
`
`240
`
`3 3 0
`2 2 0
`
`248
`
`1 1 0
`
`255
`
`0 0
`
`0
`
`99 97
`
`85 83
`74 72
`66 64
`
`58 56
`
`52 50
`
`47 45
`
`42 40
`
`38 36
`
`34 32
`
`31 29
`
`28 26
`
`26 24
`23 21
`
`21 19
`19 17
`
`17 15
`
`15 13
`
`13 11
`10
`
`12
`10
`
`8
`
`9 7
`
`8 6
`6 4
`
`5 3
`
`4
`
`2
`
`3 1
`
`2 0
`
`

`

`U.S. Patent
`Sheet 66 of 101 6,144,366
`Nov. 7, 2000
`
`FIG.67
`
`REFLECTED
`_____ LIGHT IMAGE �721
`CONTROL GENERATION
`SIGNAL
`UNIT
`723
`\
`TIMING
`CONTROL
`UNIT
`
`OUTPUT
`DATA
`
`COORDINATE
`SIGNAL
`
`IMAGE DATA
`STORAGE ------731
`UNIT
`
`IMAGE DATA
`TRANSFER �732
`UNIT
`
`730
`\
`
`CORRECTION
`CORRECTION
`1-------.....,. CALCULATION �729
`TABLE
`CORRECTION UNIT
`DATA
`
`L------
`
`CORRECTED
`RANGE
`IMAGE
`
`

`

`Nov. 7, 2000 Sheet 67 of 101 6,144,366
`U.S. Patent
`
`FIG.68
`
`733
`)
`REFERENCE
`
`OBJECT
`
`ACTIVATION
`UNIT
`
`734
`)
`
`REFERENCE
`
`OBJECT
`
`,
`
`OBJ ECT
`POS ITION
`SPE CIFYING
`
`DAT A
`
`735
`)
`
`721
`)
`CONTROL REFLECTED
`CONTROL SIGNAL LIGHT IMAGE
`UNIT
`GENERATION
`
`UNIT
`
`COOR DINATE
`DATA
`
`OUTPUT
`DISTANCE DATA DATA
`
`736
`)
`CORRECTION
`
`737
`' ' )
`CORRECTION
`
`DATA
`DATA
`STORAGE CORRECTION CALCULATION
`UNIT
`UNIT
`DATA
`
`

`

`Sheet 68 of 101
`6,144,366
`U.S. Patent
`Nov.7, 2000
`U.S. Patent
`Sheet 68 of 101 6,144,366
`Nov. 7, 2000
`
`300
`
`250
`
`200
`
`(a)150
`
`100
`
`50
`
`0
`128
`160
`192
`224
`0 32 64 96 128 160 192 224 256
`
`
`
`FIG.69
`FIG.69
`
`256
`256
`
`300
`
`250
`
`200
`
`(b)
`(b)150
`
`100
`
`50
`
`0
`64
`96
`128
`160
`192
`224
`0 32 64 96 128 160 192 224 256
`
`Page 69 of 147
`
`

`

`U.S. Patent
`Sheet 69 of 101 6,144,366
`Nov. 7, 2000
`
`FIG.70
`
`738
`\
`
`USER
`
`COMMANDING
`
`UNIT
`
`739
`'
`�
`
`CONTROL
`
`UNIT
`
`721
`)
`CONTROL REFLECTED
`SIGNAL LIGHT IMAGE
`GENERATION
`
`UNIT
`
`COOR
`DINATE
`DATA
`
`OUTPUT
`DISTANCE DATA DATA
`
`740
`\
`CORRECTION
`
`741
`' ' \
`CORRECTION
`
`DATA
`
`DATA
`
`STORAGE
`
`UNIT
`
`CORRECTION CALCULATION
`UNIT
`DATA
`
`

`

`U.S. Patent Nov. 7, 2000
`Sheet 70 of 101 6,144,366
`
`FIG.71
`
`PLEASE PLACE THE REFERENCE
`
`PLATE AT DISTANCE