United States Patent [19J
`Numazaki et al.
`
`I 1111111111111111 11111 lllll lllll lllll lllll 111111111111111 lll111111111111111
`US006144366A
`[11] Patent Number:
`[45] Date of Patent:
`
`6,144,366
`Nov. 7, 2000
`
`[54] METHOD AND APPARATUS FOR
`GENERATING INFORMATION INPUT
`USING REFLECTED LIGHT IMAGE OF
`TARGET OBJECT
`
`[75]
`
`Inventors: Shunichi Numazaki; Miwako Doi,
`both of Kanagawa; Akira Morishita,
`Tokyo; Naoko Umeki; Hiroki Miura,
`both of Kanagawa, all of Japan
`
`[73] Assignee: Kabushiki Kaisha Toshiba, Kawasaki,
`Japan
`
`[21] Appl. No.: 08/953,667
`
`[22] Filed:
`
`Oct. 17, 1997
`
`5,297,061
`5,323,174
`5,353,042
`5,423,554
`5,481,622
`5,627,565
`5,686,942
`5,703,356
`5,729,475
`5,801,704
`5,819,206
`5,900,863
`
`3/1994 Dementhon et al. ................... 364/559
`6/1994 Klapman et al. ....................... 345/156
`10/1994 Klapman et al. ....................... 345/156
`6/1995 Davis ...................................... 273/437
`1/1996 Gerhardt et al. ........................ 382/103
`5/1997 Morishita ................................ 345/158
`11/1997 Ball ......................................... 345/158
`12/1997 Bidiville et al. .. .... ... ... ... ... ... ... 250/221
`3/1998 Romanik, Jr ............................ 364/559
`9/1998 Oohara et al. .......................... 345/358
`10/1998 Horton et al.
`.......................... 702/150
`5/1999 Numazaki ............................... 345/158
`
`Primary Examiner-William A. Cuchlinski, Jr.
`Assistant Examiner-Marthe Y. Marc-Coleman
`Attorney, Agent, or Firm----Oblon, Spivak, McClelland,
`Maier & Neustadt, P.C.
`
`[30]
`
`Foreign Application Priority Data
`
`[57]
`
`ABSTRACT
`
`Oct. 18, 1996
`Jan. 31, 1997
`Feb. 12, 1997
`
`[JP]
`[JP]
`[JP]
`
`Japan .................................... 8-275949
`Japan .................................... 9-019397
`Japan .................................... 9-027752
`
`Int. Cl.7 ....................................................... G09G 5/08
`[51]
`[52] U.S. Cl. ........................... 345/156; 345/157; 345/158
`[58] Field of Search ..................................... 345/158, 157,
`345/156, 419, 425; 463/31, 32, 33, 36,
`37; 250/200, 206.1, 208.1, 214 R
`
`[56]
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`4,320,292
`4,516,020
`4,839,685
`4,841,349
`4,956,794
`5,243,182
`
`......................... 250/227
`3/1982 Oikawa et al.
`5/1985 Simpson et al. .................... 250/214 L
`6/1989 Ishiguro et al.
`........................ 354/403
`6/1989 Nakano ..................................... 357/30
`9/1990 Zeevi et al. ............................. 364/559
`9/1993 Murata et al. ....................... 250/222.1
`
`A method and an apparatus for generating information input
`which are capable of realizing a direct command type
`information input scheme by which the gesture or the
`motion can be inputted easily. The apparatus has a timing
`signal generation unit for generating a timing signal; a
`lighting unit for emitting a light whose intensity vary in
`time, according to the timing signal generated by the timing
`signal generation unit; and a reflected light extraction unit
`having a sensor array for detecting a reflected light from a
`target object resulting from the light emitted by the lighting
`unit, in synchronization with the timing signal generated by
`the timing signal generation unit, so as to obtain a spatial
`intensity distribution of the reflected light in a form of a
`reflected light image indicative of an information input
`related to the target object, in separation from an external
`light that is illuminating the target object.
`
`96 Claims, 101 Drawing Sheets
`
`104
`\
`TIMING SIGNAL
`GENERATION
`UNIT
`
`101
`\
`
`LIGHTING UNIT
`
`102
`\
`REFLECTED LIGHT
`EXTRACTION UNIT
`
`103
`\
`FEATURE DATA
`GENERATION UNIT
`
`Page 1 of 147
`
`GOOGLE EXHIBIT 1004
`
`

`

`U.S. Patent
`
`Nov. 7, 2000
`
`Sheet 1 of 101
`
`6,144,366
`
`104
`)
`TIMING SIGNAL
`GENERATION
`UNIT
`
`-
`-
`
`FIG.I
`
`101
`\
`LIGHTING UNIT
`
`102
`\
`REFLECTED LIGHT
`EXTRACTION UNIT
`
`-
`
`-
`
`103
`\
`'
`:- FEATURE DATA
`GENERATION UNIT
`
`Page 2 of 147
`
`

`

`0--,
`0--,
`~
`....
`"'-'
`"'-'
`~
`....
`0--,
`
`'"""'
`'"""' 0
`0 ....,
`N
`~ ....
`'JJ. =(cid:173)~
`
`z 0
`
`0
`0
`0
`N
`~-..J
`~
`
`~ = ......
`~ ......
`~
`•
`r:JJ.
`d •
`
`112
`
`CONTROL UNIT
`TIMING
`
`101
`
`UNIT
`
`HMEMORYH GENERATION
`
`DATA
`FEATURE
`
`AID
`
`)
`103
`
`115
`
`114
`
`FIG.2
`
`110
`
`UNIT
`CALCULATION
`DIFFERENCE
`
`109 111
`
`UNIT
`DETECTION
`PHOTO(cid:173)
`2ND
`
`UNIT
`DETECTION
`PHOTO(cid:173)
`1ST
`
`EXTRACTION UNIT
`REFLECTED LIGHT
`
`102
`
`OPTICS
`DETECTION
`PHOTO(cid:173)
`
`Page 3 of 147
`
`

`

`0--,
`0--,
`~
`....
`"'-'
`"'-'
`~
`....
`0--,
`
`~
`
`'"""'
`'"""' 0
`0 ....,
`~ .....
`'JJ. =(cid:173)~
`
`z 0
`
`0
`0
`0
`N
`~-..J
`~
`
`~ = .....
`~ .....
`~
`•
`r:JJ.
`d •
`
`(2 X 2 AS EXAMPLE)
`REGION
`IMAGE
`
`I
`
`I ....
`
`c
`L ~ i~ ~~--117
`
`1-,
`
`I-
`
`=
`l:l;.~
`·-----------,-....
`I . .,
`·-
`I
`I
`I
`L_pD
`'J IC
`I
`I
`
`PD
`I
`
`I
`
`I
`
`~
`
`--;;:
`~
`
`!
`
`'1
`
`77T
`
`77T'
`
`?.'?'
`
`:::
`
`?.'?'
`
`:::
`
`REGISTERS
`HORIZONTAL SHIFf ~136
`
`DIFFERENCE CIRCUIT ---133
`
`s .
`
`•
`
`c::::>------, c
`
`I
`I
`
`~
`
`...,
`
`FIG.3
`
`Yccs
`
`---~
`--
`Ycc2Ycc3
`Ycc4--
`
`----
`Yss Ycc1
`
`134
`
`I TIMING GENERA TOR
`
`--------._ ---
`
`I
`I
`I
`I
`I
`I
`
`L ~ i~
`
`L
`
`J;.
`..L
`I
`
`I
`
`J;.
`.I.
`I
`
`~~~ r~
`
`i~
`
`-;
`
`;:
`~
`
`.L
`I I
`rC::
`
`'PD
`I
`J;.•·J;. \
`.J..
`I
`::ir~
`
`'7
`Ir
`
`. ..,
`
`or
`
`en
`;,::, r--
`o>
`i-3 n
`n-
`tI1 i-3
`r--;,::,
`tI1 tI1
`en <
`
`I
`
`I
`
`PD
`
`J
`
`\
`135
`
`Page 4 of 147
`
`

`

`0--,
`0--,
`~
`....
`"'-'
`"'-'
`~
`....
`0--,
`
`'"""'
`'"""' 0
`0 ....,
`~ ....
`'JJ. =(cid:173)~
`
`,i;;..
`
`z 0
`
`0
`0
`0
`N
`~-..J
`~
`
`~ = ......
`~ ......
`~
`•
`r:JJ.
`d •
`
`SAMPLE 2
`SAMPLE I
`
`TRANSFER
`
`ADDRESS LINE 127
`
`1q 1~
`
`LIGHT
`
`RESET
`
`~ *;J
`
`--r
`
`121
`~
`124
`
`122
`•
`
`I
`
`123
`
`I
`
`r1
`
`126
`SIGNAL LINE
`VERTICAL
`
`FIG.4
`
`Page 5 of 147
`
`

`

`U.S. Patent
`
`Nov. 7, 2000
`
`Sheet 5 of 101
`
`6,144,366
`
`FIG.5
`
`I
`I
`
`I
`
`I
`(
`
`LIGHTING
`CONTROL PULSE
`
`LIGHTING
`POWER
`
`I
`I
`I
`I
`
`RESET
`
`I
`
`STORING
`1
`PERIOD-I~
`
`SAMPLE 1
`
`SAMPLE 2
`
`I
`
`~
`
`>
`
`128
`)
`129
`~
`
`130
`)
`
`131
`)
`
`II(
`
`'
`>I
`STORING :
`PERIOD-2 1
`
`I 132
`
`~
`
`Page 6 of 147
`
`

`

`U.S. Patent
`
`Nov. 7, 2000
`Nov.7, 2000
`
`Sheet 6 of 101
`Sheet 6 of 101
`
`6,144,366
`6,144,366
`
`U.S. Patent
`
`\0
`• t)
`~
`~
`
`....-4
`0
`N
`
`Page 7 of 147
`
`Page 7 of 147
`
`

`

`U.S. Patent
`U.S. Patent
`
`Nov.7, 2000
`Nov. 7, 2000
`
`Sheet 7 of 101
`Sheet 7 of 101
`
`6,144,366
`6,144,366
`
`FIG.7
`FIG.7
`
`TEE 20
`209
`Ee
`208 Hf
`TSC
`
`\
`
`Page 8 of 147
`
`Page 8 of 147
`
`

`

`U.S. Patent
`U.S. Patent
`
`Nov. 7, 2000
`Nov.7, 2000
`
`Sheet 8 of 101
`Sheet 8 of 101
`
`6,144,366
`6,144,366
`
`FIG.8
`FIG.8
`
`r--- 210
`210
`
`211
`211
`212
`212
`
`aaP
`
`||
`||
`
`oa|ai
`||||a
`
`pitttt|pittETTt
`
`e|||4
`
`v--.,
`I ®00
`~
`e
`pOxaed
`~ ~ I"-
`-
`,...,,
`mE
`NA
`LN
`,--. ,--1 -Ir ~ -
`/ iv i,.,--,,,.
`|FAA
`\
`(|
`'
`I
`--
`,a \t DX i--- ~
`I
`--
`~
`I
`
`_,_
`~
`
`'-
`
`I~
`
`~
`
`i.,
`
`j
`
`'I
`
`\
`
`Page 9 of 147
`
`Page 9 of 147
`
`
`
`
`
`

`

`U.S. Patent
`
`Nov. 7, 2000
`
`Sheet 9 of 101
`
`6,144,366
`
`FIG.9
`
`STICK SHAPED OBJECT
`DETECTION UNIT
`
`__,.... 213
`
`STICK SHAPED OBJECT TIP
`END EXTRACTION UNIT
`
`~ 214
`
`I '
`
`CENTER OF GRAVITY
`DETECTION UNIT
`
`L.r-- 215
`
`Page 10 of 147
`
`

`

`U.S. Patent
`
`Nov. 7, 2000
`Nov.7, 2000
`
`Sheet 10 of 101
`Sheet 10 of 101
`
`6,144,366
`6,144,366
`
`,_ ,_
`
`....
`....
`c c
`_,, _,,
`~ ~
`
`D ~ c
`
`U.S. Patent OlOld
`
`.....
`00
`N
`
`r--.....
`N
`
`,_ --_,,
`
`~
`
`0
`'l"""'""f
`• C,
`Ii--!
`~
`
`Page 11 of 147
`
`Page 11 of 147
`
`

`

`U.S. Patent
`
`Nov. 7, 2000
`
`Sheet 11 of 101
`
`6,144,366
`
`FIG.1 lA
`
`-+ - - -> - . -~~~ --
`
`FINGER TIP
`MOVEMENT
`- _-_ -~~~-
`-
`-
`- _} ~--
`
`- -----] -
`
`- -------
`-+---+--+----''l--"----"------'H-'>--4~>1--- CENTER Op _ _,._....,.__,.l,-->,-~t.+->-,,.1~1---
`GRAVITY
`MOVEMENT
`
`(a)
`
`(b)
`
`FIG.llB
`
`FINGER TIP
`-~~~~~~-1--MOVEMENT--1.-.-~-1.-.-~-1.-.-~4---
`- -----_]~------
`
`- - - - -_j ~-- - - - -
`-
`-~---"~....>..-'~~'1--CENTER OF-~~~~~~>1---
`GRAVITY
`-t--t---P--'..........,.._~~~'I--MOVEMENT_l--+-+-'i-------->---~!+-"--1~>1---
`
`(c)
`
`(d)
`
`Page 12 of 147
`
`

`

`U.S. Patent
`
`Nov. 7, 2000
`
`Sheet 12 of 101
`
`6,144,366
`
`FIG.12
`
`•
`
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`I
`
`I
`I
`I
`I
`I
`: FINGER TIP
`:
`~ MOVEMENT
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`•
`
`CALCULATED
`X-COORDINATE OF
`CENTER OF GRAVITY
`DOES NOT CHANGE
`
`Page 13 of 147
`
`

`

`0--,
`0--,
`~
`....
`"'-'
`"'-'
`~
`....
`0--,
`
`'"""'
`'"""' 0
`0 ....,
`'"""' ~
`~ ....
`'JJ. =(cid:173)~
`
`z 0
`
`0
`0
`0
`N
`~-..J
`~
`
`~ = ......
`~ ......
`~
`•
`r:JJ.
`d •
`
`tick[ n] Jeng
`
`SCRIBED LENG
`APED OBJECT
`GTH OF UPD
`
`infoStick[n].status-EXTENDED
`infoStick[ n ].length-+ 1
`
`SEQUENCE
`END OF PIXEL
`
`-X-COORDINATE OF
`infoStick[ n]. xend[ i]
`
`SEQUENCE
`ST ART OF PIXEL
`
`-X-COORDINATE OF
`infoStick[n].xstart[i]
`CANDIDATE
`UPDATE DATA OF THAT
`
`226
`FIG.13
`
`YES
`
`infoStick[n] .status-EXTENDED
`infoStick[ n] .length-1
`
`SEQUENCE
`END OF PIXEL
`
`-X-COORDINATE OF
`infoStick[ n] .xend[0]
`
`SEQUENCE
`START OF PIXEL
`
`-X-COORDINATE OF
`infoStick[n].xstart[0]
`CANDIDATE
`SHAPED OBJECT
`REGISTER NEW STICK
`
`223
`
`GO TO NEXT LINE
`
`NOSTICK
`infoStick[n].status(cid:173)
`
`ONE LINE
`NOT UPDATED BY THIS~ 222
`OBJECT CANDIDATE
`DELETE STICK SHAPED
`
`NO(ONE LINE)
`/
`LINE SCA
`HORIZONTAL
`
`')
`
`FINISHED
`
`ABOVE pFOUND BY
`WITH PIXEL VALUES
`
`THAN c CONSECUTIVE PIXEL
`
`HA YING NO MORE
`PIXEL SEQUENCE
`
`ANY
`
`PREVIOU
`CANDIDAT
`Y OF STIC
`ONNECT
`
`-
`
`220
`
`infoStick[ n]
`
`CANDIDATES
`SHAPED OBJECT
`FOR STORING STICK
`STRUCTURE ELEMENTS
`PREP ARE SEVERAL
`
`Page 14 of 147
`
`

`

`U.S. Patent
`
`Nov. 7, 2000
`
`Sheet 14 of 101
`
`6,144,366
`
`FIG.14
`
`~229
`
`,.._______ 231
`
`CALCULATE TH
`TH=(sum/max) 2 Xmax
`sum:MEAN OF PIXEL VALUE SUMS PER LINE FOR r- 230
`1ST & 2ND LINES FROM TOP OF STICK SHAPED
`OBJECT (TOPMOST IS 0-TH LINE)
`max:MAXIMUM PIXEL VALUE IN 1ST & 2ND LINES
`t
`OBTAIN k & a (0<a:S: 1) THAT SATISFY:
`k-1
`TH= ~ xsum[i]+a · xsum[k]
`i=O
`WHERE xsum[i] IS PIXEL VALUE SUM OF
`PARTIAL PIXEL SEQUENCE FOR i-TH LINE
`OF STICK SHAPED OBJECT
`t
`MULTIPLY a WITH EACH PIXEL VALUE OF PARTIAL
`PIXEL SEQUENCE FOR k-TH LINE
`t
`OBTAIN CENTER OF GRAVITY (xx, yy)
`
`,.._______
`
`232
`
`yy= [ii xsum[i](infoStick[ n ]. ystart+i)] / TH
`
`xx{i xpsum[i] ] / TH
`
`xpsum[i]=
`
`infoStick[n],xend[i]
`~ j · P(j,infoStick[n].ystart+i)
`j=infoStick[n],xstart[i]
`
`,.._______ 233
`
`Page 15 of 147
`
`

`

`0--,
`0--,
`~
`....
`"'-'
`"'-'
`~
`....
`0--,
`
`'"""
`'""" 0
`0 ....,
`'""" Ul
`~ ....
`'JJ. =(cid:173)~
`
`z 0
`
`0
`0
`0
`N
`~-..J
`~
`
`~ = ......
`~ ......
`~
`•
`r:JJ.
`d •
`
`--------------------------------------
`
`-
`
`CANDIDATE
`REGISTERING IT AS NEW STICK SHAPED OBJECT
`AFfER FINDING PARTIAL PIXEL IN y=3 LINE &
`
`FIQ15
`
`infoStick[n] .status
`
`(LENGTH IN VERTICAL DIRECTION)
`CURRENT NUMBER OF LINES
`
`END OF X-COORDINATE
`
`infoStick[n] .length
`
`infoStick[n] .xend[i]
`
`FROM TOP OF STICK SHAPED OBJECT
`START OF X-COORDINATE IN i-TH LINE
`
`infoStick[n] .xstart[i]
`
`STICK SHAPED OBJECT START
`Y-COORDINATE (LINE) AT WHICH
`
`infoStick[n].ystart
`
`11 ~~~
`9 10 11---+--I -+-I -+1--1
`
`~ I I I I I Li
`5
`
`234--8
`
`rtlllllllllllll
`
`0 1 2 3 4 5 6 7 8 91011
`x----
`
`-
`
`CONNECTED TO STICK SHAPED OBJECT CANDIDATE
`AFfER PARTIAL PIXEL IN MIDDLE OF y=5 LINE IS
`
`238 '---....,.
`
`~
`
`235
`
`infoStick[0] .status=EXTENDED
`infoStick[0] .length=3
`infoStick[0] .xend[2]=7
`infoStick[0] .xend[ 1 ]=7
`infoStick[0] .xend[0]=6
`infoStick[0] .xstart[2] =5
`infoStick[0] .xstart[l ]=5
`infoStick[0] .xstart[0]=6
`infoStick[0] .ystart=3
`
`infoStick[0] .status=EXTENDED
`inf oStick[0] .length=2
`infoStick[0].xend[l ]=7
`infoStick[0] .xend[0]=6
`infoStick[0] .xstart[ 1 ]=5
`infoStick[0] .xstart[0]=6
`infoStick[0] .ystart=3
`
`23 7 '---....,.
`
`AFfER SCAN IS FINISHED UP TO y=4 LINE
`
`infoStick[0] .status=EXTENDED
`infoStick[0] .length= 1
`infoStick[0] .xend[0]=6
`infoStick[0] .xstart[0]=6
`infoStick[0] .ystart=3
`
`236 '---....,.
`
`EXTENDED:IT IS UPDATED (EXTENDED) BY NEW LINE
`NOSTICK:IT IS NOT STICK SHAPED OBJECT CANDIDATE
`STICK:IT IS STICK SHAPED OBJECT CANDIDATE
`
`Page 16 of 147
`
`

`

`0--,
`0--,
`~
`....
`"'-'
`"'-'
`~
`....
`0--,
`
`'"""'
`'"""' 0
`0 ....,
`'"""' O'I
`~ ....
`rF.J. =(cid:173)~
`
`z 0
`
`0
`0
`0
`N
`~-..J
`~
`
`~ = ......
`~ ......
`~
`•
`r:JJ.
`d •
`
`infoStick[l].status=EXTENDED
`infoStick[ 1] .length= 1
`infoStick[l].xend[0]=6
`infoStick[ 1] .xstart[0]=6
`infoStick[l].ystart=3
`
`/
`
`infoStick[0].status=NOSTICK
`infoStick[0] .length= 1
`242-------.._ infoStick[0].xend[0]=3
`infoStick[0] .xstart[0]=3
`infoStick[0].ystart=2
`AFTER SCAN OF y=3 LINE IS FINISHED
`
`243
`
`, ....
`I L::
`
`I
`
`I
`
`.lt
`I I'
`
`I
`
`I
`
`I
`
`infoStick[0].status=EXTENDED
`infoStick[0] .length= 1
`infoStick[0] .xend[0]=3
`infoStick[0] .xstart[0]=3
`infoStick[0]. ystart=2
`AFTER SCAN OF y=2 LINE IS FINISHED
`
`FIG.16
`
`241 -------.._
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I VI I
`
`I
`
`I
`
`I
`
`I
`
`0 1 2 3 4 5/ 6 7 8 9 10 11
`x----
`
`239
`
`11 ,
`fQ I
`240----8
`7
`6 I
`5
`4
`3
`2 I
`y!? I
`
`Page 17 of 147
`
`

`

`U.S. Patent
`
`Nov. 7, 2000
`
`Sheet 17 of 101
`
`6,144,366
`
`FIG.17
`
`244
`X
`►
`01234567891011
`
`yO
`1
`2
`3
`4
`5
`6
`7
`245- 8
`9
`10 11 J . - - -4 -~~
`
`Page 18 of 147
`
`

`

`U.S. Patent
`U.S. Patent
`
`Nov. 7, 2000
`Nov.7, 2000
`
`Sheet 18 of 101
`Sheet 18 of 101
`
`6,144,366
`6,144,366
`
`FIG.18
`18
`FIG
`
`
`
`Page 19 of 147
`
`Page 19 of 147
`
`
`

`

`U.S. Patent
`U.S. Patent
`
`Nov.7, 2000
`Nov. 7, 2000
`
`Sheet 19 of 101
`Sheet 19 of 101
`
`6,144,366
`6,144,366
`
`
`246247
`
` (b)
`
`oO
`<t
`—N
`
`FIG.19
`
`
`
`246247
`
` (a)
`
`oO
`
`=N
`
`Page 20 of 147
`
`Page 20 of 147
`
`

`

`6,144,366
`6,144,366
`
`©W
`
`YNO
`
`o
`On”
`
`=N
`
`N
`
`2
`€a
`<
`
`
`
`U.S. Patent
`U.S. Patent
`
`Nov.7, 2000
`Nov. 7, 2000
`
`Sheet 20 of 101
`Sheet 20 of 101
`
`0
`a
`N
`• c.,
`e
`O
`fo
`~
`~
`
`©W
`
`YN3
`
`On
`
`aN
`
`
`
`Page 21 of 147
`
`Page 21 of 147
`
`

`

`0--,
`0--,
`~
`....
`"'-'
`"'-'
`~
`....
`0--,
`
`'"""'
`'"""' 0
`'"""' 0 ....,
`~ ....
`'JJ. =(cid:173)~
`
`N
`
`z 0
`
`0
`0
`0
`N
`~-..J
`~
`
`~ = ......
`~ ......
`~
`•
`r:JJ.
`d •
`
`55
`\
`
`::-CLICK SIGNAL
`
`254
`
`DETECTION UNIT
`STATIONARY STATE
`CURSOR
`
`'
`
`UNIT
`CURSOR DISPLAY
`
`-
`
`UNIT
`DETERMINATION
`CURSOR POSITION
`
`-
`-
`
`DETECTION UNIT
`POSITION
`FINGER TIP
`
`253
`
`252
`
`251
`
`FIG.21
`
`Page 22 of 147
`
`

`

`0--,
`0--,
`~
`....
`"'-'
`"'-'
`~
`....
`0--,
`
`~ ....
`'JJ. =(cid:173)~
`
`'"""'
`'"""' 0
`0 ....,
`N
`N
`
`z 0
`
`0
`0
`0
`N
`~-..J
`~
`
`~ = ......
`~ ......
`~
`•
`r:JJ.
`d •
`
`,... ,..,,..,
`)
`
`-COMMAND OR CLICK SIGNAL
`
`256
`
`DETECTION UNIT
`MOVEMENT
`
`-TIP POSITION
`
`CURSOR OR FINGER
`
`UNIT
`CURSOR DISPLAY
`
`UNIT
`DETERMINATION
`CURSOR POSITION
`
`DETECTION UNIT
`POSITION
`FINGER TIP
`
`253
`
`252
`
`251
`
`FIG.22
`
`Page 23 of 147
`
`

`

`0--,
`0--,
`~
`....
`"'-'
`"'-'
`~
`....
`0--,
`
`'"""'
`'"""' 0
`0 ....,
`~
`N
`
`~ ....
`'JJ. =-~
`
`z 0
`
`0
`0
`0
`N
`~-..J
`~
`
`~ = ......
`~ ......
`~
`•
`r:JJ.
`d •
`
`INTERPRETATION RESULT
`
`,,
`
`333
`
`UNIT
`INTERPRETATION
`SHAPE
`
`-
`
`, RULES
`
`INTERPR ATION
`SHAPE
`
`332
`
`MEMORY UNIT
`SHAPE
`
`331
`(
`
`MATRIX
`DISTANCE
`
`MEMORY UNIT
`INTENSITY RANGE IMAGE
`LIGHT
`
`-
`
`TIMING
`
`FIG.23
`
`Page 24 of 147
`
`

`

`U.S. Patent
`
`Nov. 7, 2000
`
`Sheet 24 of 101
`
`6,144,366
`
`-(cid:173)u
`
`-
`
`--+-+-+-l--l-+-+---+---4-1--1 - (cid:173)
`+-+-+-lf---+--+-+-+-IH
`(I)
`
`·- :\\
`. ' . I '. .
`.·
`. :
`
`---+--1--1--+-+-I----I - -
`--.+---+-I - -
`~ 1--+--+-+--+--1--1-+-+---1--1--1--1--1---1--1--1-+-l '"O
`·.
`
`<
`
`Page 25 of 147
`
`

`

`U.S. Patent
`
`Nov. 7, 2000
`
`Sheet 25 of 101
`
`6,144,366
`
`FIG.25
`
`START
`
`STORE RANGE IMAGE
`INTO RANGE IMAGE
`MEMORY UNIT
`
`READ DISTANCE MATRIX
`
`DISTANCE MATRIX
`PROCESSING
`
`401
`
`402
`
`403
`
`CHING S
`RPRETAT
`LE EXI
`
`YES
`
`NO
`> - - -~
`
`406
`
`CHANGE THRESHOLD
`& GO TO 403
`
`OUTPUT COMMAND
`CORRESPONDING TO
`MATCHED SHAPE
`
`405
`
`END
`
`Page 26 of 147
`
`

`

`U.S. Patent
`
`Nov. 7, 2000
`
`Sheet 26 of 101
`
`6,144,366
`
`FIG.26
`
`SHAPE INTERPRETATION RULES FOR POINTING
`
`RULE -1:ONE RECTANGLE-POINTING
`RULE -2:TWO RECTANGLES-TWO FINGER OPERATION
`RULE -3:OTHERS--MOTION INPUT
`
`POINTING RULE
`VERTICAL & HORIZONTAL CHECK
`VERTICAL/HORIZONTAL::::: I-OBJECT SELECT/MOVE FORWARD
`VERTICAL/HORIZONTAL<l--SLOPE CHECK-1
`VERTICAL/HORIZONTAL>l--SLOPE CHECK-2
`SLOPE CHECK-1
`SLOPE:::::0 & CENTER OF GRAVITY RIGHT-ROTATION TO RIGHT
`SLOPE:::::0 & CENTER OF GRAVITY LEFT-ROTATION TO LEFT
`
`SLOPE CHECK-2
`SLOPE::::::o & CENTER OF GRAVITY UPPER-TURNING UPWARD
`SLOPE::::: 0 & CENTER OF GRAVITY LOWER
`--TURNING DOWNWARD
`
`Page 27 of 147
`
`

`

`U.S. Patent
`U.S. Patent
`
`Nov. 7, 2000
`Nov.7, 2000
`
`Sheet 27 of 101
`Sheet 27 of 101
`
`6,144,366
`6,144,366
`
`X
`
`-
`
`-
`
`f--t-+--1-+-+-l-+--+4~'-4--1f-l-l----l-l .---..
`f--t-+--l-+-+--+-+-+-+~--+--lf-l-l----4-1 ..0
`
`rEtTTTTTpeeTTTTTTPPeeeTPETPTTTyeptPteTeeeettpitttTeeeTtrtPPTeTeteyett
`
` Pidtt|aPitriyyyetyeeyLEEttitttrtitritteetLEPETTtTtTtttietttttCoLETTTTtTTertrritttt|(q(e TTTTiteeeeeeeTTTyLTTTTPETretTrTtPtyetteryeytaPteTtTeeeeteteTPtTTTPetetTTttSRRPETPTTeTTeTPTTTETEereeeey
`
`
`PTTTTTTtTTTt||PtTTIPtTytTTTETLtmtitTPi{TyTet|iPtTytTPTTe
`LTTtttttteTt|_t|Peteey
`LTTTTtTTeTeLt||
` rorETriTePeeerereTL|pittiyttyteLI}|SERREIRLLT
`
`J(9(p
`
`
`~1~} {
`
`C
`
`[|
`
`.---..
`"'O
`
`1---+-l-l-l-l-.l-,,- "4--'-!--4--1--4--l--+--4-1 .---..
`,:,:s
`t-+-+-+--+-+-4--+--+-+---+--4~--1----1--1-----'
`
`■ ■
`■ ■
`
`Page 28 of 147
`
`Page 28 of 147
`
`
`

`

`U.S. Patent
`U.S. Patent
`
`Sheet 28 of 101
`Sheet 28 of 101
`
`6,144,366
`6,144,366
`
`Nov. 7, 2000
`Nov.7, 2000
`
`Losew
`
`pittTttmt|tt
`
`Paar
`
`Page 29 of 147
`
`Page 29 of 147
`
`
`

`

`U.S. Patent
`
`Nov. 7, 2000
`
`Sheet 29 of 101
`
`6,144,366
`
`FIG.29
`
`SHAPE INTERPRETATION RULES FOR POWER ON/OFF
`
`RULE -1:ONE RECTANGLE-POINTING
`RULE-2:TWO RECTANGLES-TWO FINGERS
`RULE -3:FIVE RECTANGLES-FIVE FINGERS
`
`POINTING RULE
`AREA CHECK
`AREA>> a -FIST
`AREAS a-NULL
`
`COMMAND GENERATION
`FIVE FINGERS, FIST, TWO FINGERS-POWER OFF
`TWO FINGERS, FIST, FIVE FINGERS-POWER ON
`
`Page 30 of 147
`
`

`

`U.S. Patent
`
`Nov. 7, 2000
`
`Sheet 30 of 101
`
`6,144,366
`
`FIG.30
`
`/
`
`- - - - -
`
`,
`
`/
`
`\
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`SPHERE
`
`'
`'
`\
`
`<
`
`' ' ' , /
`'
`
`~ \
`PYRAMID-,
`\
`
`\
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`' ' ' ' ' '
`
`~
`
`~
`
`' '
`
`~
`
`VIEWPOINT-2 VIEWPOINT-I
`
`VIEWPOINT-3
`
`Page 31 of 147
`
`

`

`U.S. Patent
`U.S. Patent
`
`Nov.7, 2000
`Nov. 7, 2000
`
`Sheet 31 of 101
`Sheet 31 of 101
`
`6,144,366
`6,144,366
`
`b)VIEWPOINT-2
`
`
`c)VIEWPOINT-3
`
`a)VIEWPOINT-1
`
`FIG.31
`
`Page 32 of 147
`
`Page 32 of 147
`
`

`

`U.S. Patent
`U.S. Patent
`
`Nov. 7, 2000
`Nov.7, 2000
`
`Sheet 32 of 101
`Sheet 32 of 101
`
`6,144,366
`6,144,366
`
`I
`I
`
`t-- ,-
`
`,- ,-
`
`,- -
`,- -
`
`--
`,--
`--
`,-- □
`- -
`'
`--
`r--; ....l ~
`
`
`
`
`
`C ::.
`,- -
`,- -
`- -
`,- -
`- - □
`,- -
`,- -
`,- -
`,- -
`
`,
`
`0:::
`
`- t - -,-
`
`- t - -,-
`
`---
`---
`- t - --
`---
`---
`- f --
`---
`
`- f - ,-
`
`,-.._
`,.0
`
`,,......,
`(1,)
`
`>--f- -
`>--f- -
`-- -
`>--f- -
`-- -
`>--f- -
`>--f- -
`>--f- -
`.-,- -
`
`....l
`
`,-.._
`,:,::$
`
`,- -
`- -
`,- -
`- -
`,- -
`,- -
`,- -
`--
`--
`--
`
`0:::
`□
`
`D
`
`....l
`
`>--f- -
`
`-- -
`>--f- -
`.-,- -
`_,- -
`.-,- -
`>--f- -
`-- -
`>--f- -
`.-,- -
`
`
`
`
`
`
`
`
`
`
`
`N
`('<")
`•
`C)
`~
`~
`
`Page 33 of 147
`
`Page 33 of 147
`
`
`

`

`U.S. Patent
`
`Nov. 7, 2000
`
`Sheet 33 of 101
`
`6,144,366
`
`FIG.33
`
`SHAPE INTERPRETATION RULES FOR
`MOVING OBJECT PARALLAX
`
`RULE-1:TWO RECTANGLES-FACE CHECK
`RULE -2:OTHERS--NULL
`
`FACE CHECK
`DISTANCE TO LARGER RECTANGLE>DISTANCE TO SMALLER
`RECTANGLE-VIEWPOINT CHECK
`OTHERS---+NULL
`VIEWPOINT CHECK
`L =f:c 0---+ VIEWPOINT DIRECTION=arcsin( (L-R)/L)
`L=0--VIEWPOINT DIRECTION=-90°
`
`Page 34 of 147
`
`

`

`0--,
`0--,
`~
`....
`"'-'
`"'-'
`~
`....
`0--,
`
`,i;;..
`~
`
`'"""
`'""" 0
`0 ....,
`
`~ ....
`'JJ. =(cid:173)~
`
`z 0
`
`0
`0
`0
`N
`~-..J
`~
`
`~ = ......
`~ ......
`~
`•
`r:JJ.
`d •
`
`I
`I
`--
`I
`
`I
`I
`
`I
`
`I
`I
`
`I
`
`I
`I ;
`
`I
`I
`I
`I
`
`I
`I
`
`I
`I
`
`I
`I
`I
`
`I
`
`I
`
`I
`
`' ' I
`' I
`I
`
`PYRAMID ~ _/
`
`SPHERE ~.i.
`
`,------,--------
`
`I
`,,,,----7
`
`/
`
`/
`
`FIG.34
`
`' ' I
`
`"-, \
`
`"'
`
`\
`
`\
`
`\
`
`\
`
`\
`
`\
`
`"'
`
`\
`
`\
`
`\
`
`\
`
`"'
`
`"'
`
`"'
`
`/
`
`/
`
`--\
`\/
`
`\
`
`\
`
`\
`
`\
`
`\
`
`\
`
`\
`
`VIEWPOINT-3
`
`'\
`
`/
`
`<
`
`\
`
`\
`
`\
`
`\
`
`\
`
`\
`
`', ---VIEWPOINT-I \ ,,,,----
`
`Page 35 of 147
`
`

`

`0--,
`0--,
`~
`....
`"'-'
`"'-'
`~
`....
`0--,
`
`'"""'
`'"""' 0
`0 ....,
`Ul
`~
`
`~ ....
`'JJ. =-~
`
`z 0
`
`0
`0
`0
`N
`~-..J
`~
`
`~ = ......
`~ ......
`~
`•
`r:JJ.
`d •
`
`(ANGLE)
`(SPEED)
`(SWINGING RANGE)
`MOTION VECTOR
`
`'
`
`343
`
`UNIT
`EXTRACTION
`IMAGE CHANGE
`
`'
`
`-
`
`342
`
`MEMORY UNIT
`POSITION
`COMPARISON
`
`331
`(
`
`MATRIX
`DISTANCE
`
`-MEMORY UNIT
`INTENSITY RANGE IMAGE
`LIGHT
`
`TIMING
`
`FIG.35
`
`Page 36 of 147
`
`

`

`U.S. Patent
`
`Nov. 7, 2000
`
`Sheet 36 of 101
`
`6,144,366
`
`FIG.36
`
`START
`
`INITIALIZE COMPARISON
`POSITION MEMORY UNIT
`
`STORE RANGE IMAGE
`INTO RANGE IMAGE
`MEMORY UNIT
`
`READ DISTANCE MA TRIX
`
`EXTRACT CENTER OF
`GRAVITY POSITION
`
`901
`
`902
`
`903
`
`904
`
`OF GRAVITY
`SITION STORED
`ARISON POSI
`MORY UN
`
`NO
`
`TO 907
`
`YES
`CALCULATE MOTION VECTOR
`(CALCULATE SWINGING RANGE)
`(CALCULATE SPEED)
`(CALCULATE ANGLE)
`
`_____________ ___.
`
`906
`(908)
`(909)
`(910)
`
`STORE NEW CENTER OF GRAVITY
`POSITION INTO COMPARISON
`POSITION MEMORY UNIT
`
`907
`
`TO 902
`
`Page 37 of 147
`
`

`

`U.S. Patent
`
`Nov. 7, 2000
`
`Sheet 37 of 101
`
`6,144,366
`
`FIG.37
`
`v=[ (Xn, Y n)-(Xn-1, Y n-1) ]/M · 30
`
`FIG.38
`
`SWINGING RANGE(Xn-Xn-1)/M · 30
`
`~✓ I
`I I\
`
`SPEED(Y n-Y n-1 )/M · 30
`
`Page 38 of 147
`
`

`

`U.S. Patent
`
`Nov. 7, 2000
`
`Sheet 38 of 101
`
`6,144,366
`
`FIG.39
`
`FINGER AT TIME t
`
`FINGER AT TIME t+ I
`
`a)
`
`b)
`
`Xbt,Ybt
`
`Xst,Yst Xst+ l,Yst+ I
`
`Xbt,Ybt
`
`X'bt+ 1,Y'bt+ 1
`
`Xst,Yst X'st+ 1,Y'st+ 1
`
`Page 39 of 147
`
`

`

`U.S. Patent
`
`Nov. 7, 2000
`
`Sheet 39 of 101
`
`6,144,366
`
`FIG.40
`
`{
`
`~¼,i
`
`SWINGING
`ANGLE a
`
`SWINGING
`ANGLE a
`
`Page 40 of 147
`
`

`

`U.S. Patent
`
`Nov. 7, 2000
`
`Sheet 40 of 101
`
`6,144,366
`
`FIG.41
`
`I
`
`SWINGING
`ANGLE a
`
`Page 41 of 147
`
`

`

`U.S. Patent
`U.S. Patent
`
`Nov.7, 2000
`Nov. 7, 2000
`
`Sheet 41 of 101
`Sheet 41 of 101
`
`6,144,366
`6,144,366
`
`,....._
`u
`C)
`
`,....._
`.0
`b)
`
`FIG.42
`
`PTTTtTTPTT
`
`aE
`
`aI
`
`f
`
`PteeTTTT |||I
`BRR
`
`ERR
`
`a)
`
`|t|eg
`PiTttTeeTT
`||
`PTTeTTTT
`PreeTeeeeT
`
`Page 42 of 147
`
`Page 42 of 147
`
`

`

`0--,
`0--,
`~
`....
`"'-'
`"'-'
`~
`....
`0--,
`
`~ ....
`'JJ. =-~
`
`'"""'
`'"""' 0
`0 ....,
`N
`,i;;..
`
`z 0
`
`0
`0
`0
`N
`~-..J
`~
`
`~ = ......
`~ ......
`~
`•
`r:JJ.
`d •
`
`346
`
`DRAWING UNIT
`
`345
`
`CONTROL UNIT
`CHARACTER
`
`INTERPRETATION CONTROL UNIT
`
`-
`
`-
`
`-
`
`--
`
`344
`
`MAPPING
`PARAMETER
`
`RANGE (ANGLE)
`TION VECTOR
`
`,SWINGIN(
`SPEED, :M
`
`343
`
`342
`
`UNIT
`EXTRACTION
`IMAGE CHANGE
`
`MEMORY UNIT
`POSITION
`COMPARISON
`
`UNIT
`
`~ INTERPRETATION
`
`UNIT
`SHAPE MEMORY
`
`SHAPE
`OF GRAVITY
`AREA,CENTER
`
`MATRIX
`DISTANCE
`
`MEMORY UNIT
`INTENSITY RANGE IMAGE
`LIGHT
`
`(
`331
`
`TIMING
`
`FIG.43
`
`Page 43 of 147
`
`

`

`U.S. Patent
`U.S. Patent
`
`Nov.7, 2000
`Nov. 7, 2000
`
`Sheet 43 of 101
`Sheet 43 of 101
`
`6,144,366
`6,144,366
`
`FIG.44
`
`'tj-
`'tj-
`•
`t)
`~
`~
`
`Page 44 of 147
`
`1-,-,.,.,.,.,.,.,.,.,.
`
`~
`
`-~
`Jj,c
`
`I
`~ -
`
`Page 44 of 147
`
`

`

`U.S. Patent
`U.S. Patent
`
`Nov.7, 2000
`Nov. 7, 2000
`
`Sheet 44 of 101
`Sheet 44 of 101
`
`6,144,366
`6,144,366
`
`FIG.45
`
`Page 45 of 147
`
`Page 45 of 147
`
`

`

`0--,
`0--,
`~
`....
`"'-'
`"'-'
`~
`....
`0--,
`
`~ ....
`'JJ. =(cid:173)~
`
`'"""'
`'"""' 0
`0 ....,
`Ul
`,i;;..
`
`z 0
`
`0
`0
`0
`N
`~-..J
`~
`
`~ = ......
`~ ......
`~
`•
`r:JJ.
`d •
`
`FEATURE DATA GENERATION UNIT 1037
`
`352
`(
`
`l•I IMAGE MEMORY 1---___.
`
`UNIT
`
`354
`
`(
`
`353
`(
`
`IMAGE DATA
`
`MEMORY UNIT
`IMAGE
`EXTRACTED
`
`1----i
`
`UNIT
`EXTRACTION
`
`,
`
`I
`
`1
`
`351
`(
`ARRAY
`DETECTION
`PHOTO(cid:173)
`VISIBLE LIGHT
`
`INTENSITY
`LIGHT
`VISIBLE
`
`TIMING I-
`
`331
`(
`
`MATRIX
`REFLECTION
`
`MEMORY UNIT
`RANGE IMAGE
`
`102
`
`(
`
`UNIT
`EXTRACTION
`LIGHT
`REFLECTED
`
`TIMING I-
`
`LIGHT
`REFLECTED
`
`FIG.46
`
`Page 46 of 147
`
`

`

`0--,
`0--,
`~
`....
`"'-'
`"'-'
`~
`....
`0--,
`
`~ ....
`'JJ. =(cid:173)~
`
`O'I
`,i;;..
`
`'"""'
`'"""' 0
`0 ....,
`
`z 0
`
`0
`0
`0
`N
`~-..J
`~
`
`~ = ......
`~ ......
`~
`•
`r:JJ.
`d •
`
`-4'P) NEAR INFRARED PHOTO-DETECTOR
`
`~ ~ts:7'
`~O❖O•'" -
`
`~--
`
`--------------------
`
`~
`FIG.47
`
`PHOTO-DETECTION LENS
`,,,,--------------
`
`Page 47 of 147
`
`

`

`U.S. Patent
`
`Nov. 7, 2000
`
`Sheet 47 of 101
`
`6,144,366
`
`FIG.48
`
`________ __.A.-_ _______ _
`---....
`,,--
`
`ORIGINAL
`IMAGE
`
`MASK
`
`\
`
`I
`
`EXTRACTED
`IMAGE
`
`Page 48 of 147
`
`

`

`0--,
`0--,
`~
`....
`"'-'
`"'-'
`~
`....
`0--,
`
`~ ....
`'JJ. =(cid:173)~
`
`'"""'
`'"""' 0
`0 ....,
`00
`,i;;..
`
`z 0
`
`0
`0
`0
`N
`~-..J
`~
`
`~ = ......
`~ ......
`~
`•
`r:JJ.
`d •
`
`FEATURE DATA GENERATION UNIT 103;
`
`355
`
`RECORDING UNIT
`(COMPRESSION)
`IMAGE
`EXTRACTED
`
`354
`
`353
`(
`
`__.
`
`IMAGE DATA
`
`Ll_JJ IMAGE MEMORY _
`
`MEMORY UNIT
`IMAGE
`EXTRACTED
`
`i--1
`
`UNIT
`EXTRACTION
`
`w
`
`331
`(
`
`MATRIX
`REFLECTION
`
`MEMORY UNIT
`RANGE IMAGE
`
`102
`(
`
`UNIT
`EXTRACTION
`LIGHT
`REFLECTED
`
`TIMING I-
`
`LIGHT
`REFLECTED
`
`FIG.49
`
`352
`(
`
`UNIT
`
`351
`(
`ARRAY
`DETECTION
`PHOTO(cid:173)
`VISIBLE LIGHT
`
`TIMING I-
`
`INTENSITY
`LIGHT
`VISIBLE
`
`Page 49 of 147
`
`

`

`0--,
`0--,
`~
`....
`"'-'
`"'-'
`~
`....
`0--,
`
`~ ....
`'JJ. =(cid:173)~
`
`'"""'
`'"""' 0
`0 ....,
`\0
`,i;;..
`
`z 0
`
`0
`0
`0
`N
`~-..J
`~
`
`~ = ......
`~ ......
`~
`•
`r:JJ.
`d •
`
`FEATURE DATA GENERATION UNIT 103
`
`356
`(
`
`UNIT
`TRANSMISSION
`(COMPRESSION)
`
`352
`(
`
`UNIT
`IMAGE MEMORY
`
`-
`
`354
`
`(
`
`353
`(
`
`IMAGE DATA
`'~
`
`MEMORY UNIT
`IMAGE
`EXTRACTED
`
`~
`
`UNIT
`EXTRACTION
`
`I
`
`INTENSITY
`LIGHT
`VISIBLE
`
`351
`(
`ARRAY
`DETECTION
`PHOTO-
`VISIBLE LIGHT
`
`TIMING
`
`331
`(
`
`MATRIX
`REFLECTION
`
`MEMORY UNIT
`RANGE IMAGE
`
`102
`(
`
`UNIT
`EXTRACTION
`LIGHT
`REFLECTED
`
`TIMING
`
`LIGHT
`REFLECTED
`
`FIG.50
`
`Page 50 of 147
`
`

`

`U.S. Patent
`
`Nov. 7, 2000
`
`Sheet 50 of 101
`
`6,144,366
`
`FIG.51
`
`BACKGROUND IMAGE
`
`EXTRACTED IMAGE
`
`/
`
`COMPOSED IMAGE
`
`Page 51 of 147
`
`

`

`0--,
`0--,
`~
`....
`"'-'
`"'-'
`~
`....
`0--,
`
`'"""'
`'"""' 0
`'"""' 0 ....,
`~ ....
`'JJ. =(cid:173)~
`
`Ul
`
`z 0
`
`0
`0
`0
`N
`~-..J
`~
`
`~ = ......
`~ ......
`~
`•
`r:JJ.
`d •
`
`IMAGE
`COMPOSED
`
`357
`(
`
`UNIT
`COMPOSITION
`IMAGE
`
`MEMORY UNIT
`
`BACKGROUND
`
`358~ IMAGE
`
`352
`(
`
`UNIT
`
`351
`(
`ARRAY
`DETECTION
`
`INTENSITY I IMAGE MEMORY•-~
`
`--i PHOTO(cid:173)
`
`354
`(
`
`353
`(
`
`IMAGE DATA
`
`MEMORY UNIT
`
`i-... IMAGE
`
`EXTRACTED
`
`UNIT
`EXTRACTION
`
`~
`
`331
`(
`
`MATRIX
`REFLECTION
`
`MEMORY UNIT
`I RANGE IMAGE
`
`LIGHT
`REFLECTED
`
`FIG.52
`
`VISIBLE
`
`---------, LIGHT
`
`VISIBLE LIGHT
`
`102
`(
`
`UNIT
`EXTRACTION
`LIGHT
`
`-REFLECTED
`
`Page 52 of 147
`
`

`

`0--,
`0--,
`~
`....
`"'-'
`"'-'
`~
`....
`0--,
`
`~ ....
`'JJ. =(cid:173)~
`
`'"""'
`'"""' 0
`0 ....,
`N
`Ul
`
`z 0
`
`0
`0
`0
`N
`~-..J
`~
`
`~ = ......
`~ ......
`~
`•
`r:JJ.
`d •
`
`354
`7
`MEMORY UNIT
`IMAGE
`EXTRACTED
`
`I
`
`I
`
`UNIT
`
`353
`(
`
`~
`
`IMAGE DATA
`
`m IMAGE MEMORY
`
`ARRAY
`DETECTION
`PHOTO(cid:173)
`VISIBLE LIGHT
`
`1
`
`351
`7
`
`TIMING I-
`
`352
`(
`
`INTENSITY
`LIGHT
`VISIBLE
`
`UNIT
`EXTRACTION
`
`J---359
`
`SWITCH UNIT
`
`331
`(
`
`MATRIX
`REFLECTION
`
`MEMORY UNIT
`
`; I RANGE IMAGE
`
`1
`
`UNIT
`
`TIMING 1--1 EXTRACTION
`
`LIGHT
`REFLECTED
`
`LIGHT
`REFLECTED
`
`FIG.53
`
`(
`102
`
`Page 53 of 147
`
`

`

`0--,
`0--,
`~
`....
`"'-'
`"'-'
`~
`....
`0--,
`
`'"""'
`'"""' 0
`0 ....,
`~
`Ul
`
`~ ....
`'JJ. =(cid:173)~
`
`z 0
`
`0
`0
`0
`N
`~-..J
`~
`
`~ = ......
`~ ......
`~
`•
`r:JJ.
`d •
`
`362
`
`DRAWING UNIT
`
`361
`(
`
`"
`
`MEMORY UNIT
`IMAGE
`Z-VALUE
`
`352
`I
`
`UNIT
`IMAGE MEMORY
`
`-
`
`353
`l
`
`IMAGE DATA
`
`UNIT
`EXTRACTION
`
`INTENSITY
`LIGHT
`VISIBLE
`
`351
`I
`ARRAY
`DETECTION
`PHOTO-
`VISIBLE LIGHT
`
`Z-VALUES
`DATA+
`IMAGE
`
`331
`(
`
`MATRIX
`REFLECTION
`
`MEMORY UNIT
`RANGE IMAGE
`
`LIGHT
`REFLECTED
`
`102
`(
`
`UNIT
`EXTRACTION
`LIGHT
`REFLECTED
`
`FIG.54
`
`Page 54 of 147
`
`

`

`0--,
`0--,
`~
`....
`"'-'
`"'-'
`~
`....
`0--,
`
`~ ....
`'JJ. =(cid:173)~
`
`'"""'
`'"""' 0
`0 ....,
`,i;;..
`Ul
`
`z 0
`
`0
`0
`0
`N
`~-..J
`~
`
`~ = ......
`~ ......
`~
`•
`r:JJ.
`d •
`
`~~ I I I I I I I I I I I I I I I I I
`ii-11111111111111111
`
`9
`8
`7
`6
`5
`4
`3
`2
`1
`
`c)
`
`1 2 3 4 5 6 7 8 9 10111213141516
`
`■
`
`16
`15
`14
`13
`12
`11
`JO
`9
`8
`7
`6
`5
`4
`3
`2
`1
`
`b)
`
`~ /
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`I
`
`16 ~~~~~~~~~~~~~~
`~~ I
`~~ I
`9 ~? I
`
`8
`7
`6
`5
`4
`3
`2
`1
`
`a)
`
`1 2 3 4 5 6 7 8 910111213141516
`
`l 2 345678910111213141516
`
`FIG.55
`
`Page 55 of 147
`
`

`

`U.S. Patent
`U.S. Patent
`
`Nov.7, 2000
`Nov. 7, 2000
`
`Sheet 55 of 101
`Sheet 55 of 101
`
`6,144,366
`6,144,366
`
`FIG.56
`
`
`
`a)
`
`b)
`
`c)
`
`c)
`
`Page 56 of 147
`
`Page 56 of 147
`
`

`

`0--,
`0--,
`~
`....
`"'-'
`"'-'
`~
`....
`0--,
`
`~ ....
`'JJ. =(cid:173)~
`
`'"""'
`'"""' 0
`0 ....,
`O'I
`Ul
`
`z 0
`
`0
`0
`0
`N
`~-..J
`~
`
`~ = ......
`~ ......
`~
`•
`r:JJ.
`d •
`
`352
`
`INTENSITY
`LIGHT
`VISIBLE
`
`351
`
`3~1
`
`MEMORY UNIT
`IMAGE
`Z-VALUE
`
`Z-VALUES
`DATA+
`IMAGE
`
`UNIT
`IMAGE MEMORY
`
`353
`(
`
`IMAGE DATA
`
`UNIT
`EXTRACTION
`
`~
`
`r--363
`
`331
`(
`
`MATRIX
`REFLECTION
`
`MEMORY UNIT
`RANGE IMAGE
`
`VISIBLE LIGHT
`
`ARRAY
`DETECTION
`
`·-PHOTO-
`
`UNIT
`DETERMINATION
`RANGE
`EXTRACTION
`
`/
`
`102
`(
`
`-EXTRACTION
`
`LIGHT
`REFLECTED
`
`UNIT
`
`TIMING
`
`TIMING
`
`LIGHT
`REFLECTED
`
`FIG.57
`
`Page 57 of 147
`
`

`

`U.S. Patent
`U.S. Patent
`
`Nov. 7, 2000
`Nov.7, 2000
`
`Sheet 57 of 101
`Sheet 57 of 101
`
`6,144,366
`6,144,366
`
`COZ19Z
`
`(q
`
`,--..
`ro
`
`(p
`
`......
`
`~
`
`"' \
`
`l'r--..
`
`:,::::
`
`I"--
`
`I'-
`
`\
`
`:,:,;;,
`
`"
`
`c.::::::
`
`::,:·
`
`"
`
`,--..
`(.)
`
`8SOIA
`
`Page 58 of 147
`
`Page 58 of 147
`
`
`
`

`

`U.S. Patent
`
`Nov. 7, 2000
`
`Sheet 58 of 101
`
`6,144,366
`
`FIG.59
`
`CONTROL
`SIGNAL
`
`-
`
`REFLECTED
`LIGHT IMAGE _,,-. 721
`GENERATION
`UNIT
`
`720
`~
`
`TIMING
`CONTROL
`UNIT
`
`SIGNALS
`CORRESP ONDING
`ECTED
`TO REFL
`
`MAGE LIGHT I
`
`NONLINEAR
`CONVERSION
`UNIT
`
`_,,-. 722
`
`i
`
`SIGNALS CORRESPONDING
`TO RANGE IMAGE
`
`Page 59 of 147
`
`

`

`U.S. Patent
`
`Nov. 7, 2000
`
`Sheet 59 of 101
`
`6,144,366
`
`723
`\
`
`TIMING
`CONTROL
`UNIT
`
`FIG.60
`
`REFLECTED
`~---.._ LIGHT IMAGE ---- 721
`CONTROL
`GENERATION
`SIGNAL
`UNIT
`
`COORDINATE
`SIGNAL
`
`SIGNALS
`CORRESPONDING
`TO REFLECTED
`LIGHT IMAGE
`
`A/D
`CONVERTER
`
`------724
`
`DIGITAL
`REFLECTED
`LIGHT IMAGE
`
`CORRECTION v-- 725
`UNIT
`
`i
`
`CORRECTED
`RANGE
`IMAGE
`
`Page 60 of 147
`
`

`

`U.S. Patent
`
`Nov. 7, 2000
`
`Sheet 60 of 101
`
`6,144,366
`
`723
`\
`
`TIMING
`CONTROL
`UNIT
`
`FIG.61
`
`REFLECTED
`~ - - LIGHT IMAGE -------721
`CONTROL
`GENERATION
`SIGNAL
`UNIT
`
`COORDINATE
`SIGNAL
`
`SIGNALS
`CORRESPONDING
`TO REFLECTED
`
`7 LIGHT IMAGE
`
`NONLINEAR
`CONVERSION v--- 726
`UNIT
`
`SIGNALS
`CORRESPONDING
`TO RANGE
`IMAGE
`
`AID
`CONVERTER
`
`v---724
`
`DIGITAL
`RANGE
`IMAGE
`
`CORRECTION
`UNIT
`
`-------727
`
`i
`
`CORRECTED
`RANGE
`IMAGE
`
`Page 61 of 147
`
`

`

`U.S. Patent
`
`Nov. 7, 2000
`
`Sheet 61 of 101
`
`6,144,366
`
`FIG.62
`
`\
`
`\
`\
`"' '--
`
`------
`
`50 ·
`
`100
`
`150
`INPUT
`
`200
`
`250
`
`300
`
`800
`
`700
`
`600
`
`~ 500
`~ 400
`~
`0 300
`
`200
`
`100
`
`0
`
`0
`
`Page 62 of 147
`
`

`

`U.S. Patent
`
`Nov. 7, 2000
`
`Sheet 62 of 101
`
`6,144,366
`
`FIG.63
`
`REFLECTED
`~ - - - LIGHT IMAGE 1----721
`CONTROL
`GENERATION
`SIGNAL
`UNIT
`
`L - - - - - - - , . . . . . .1
`
`SIGNALS
`CORRESPONDING
`TO REFLECTED
`LIGHT IMAGE
`
`COORDINATE
`SIGNAL
`
`723
`~
`
`TIMING
`CONTROL
`UNIT
`
`AID
`CONVERTER
`
`1---- 7 24
`
`DIGITAL
`REFLECTED
`, LIGHT IMAGE
`
`CORRECTION ~ 728
`TABLE
`
`i
`
`CORRECTED
`RANGE IMAGE
`
`Page 63 of 147
`
`

`

`U.S. Patent
`
`Nov. 7, 2000
`
`Sheet 63 of 101
`
`6,144,366
`
`FIG.64
`
`I \
`
`~RD I -
`IN
`ATE
`
`0
`
`1
`
`2
`
`4094
`
`4095
`
`--
`
`-~
`-
`
`0
`1
`2
`3
`4
`5
`6
`7
`8
`
`99
`100
`101
`102
`103
`104
`105
`106
`107
`108
`109
`
`247
`248
`249
`250
`251
`252
`253
`254
`255
`
`255
`255
`255
`255
`255
`255
`255
`255
`255
`
`39
`38
`38
`37
`37
`36
`36
`35
`35
`34
`34
`
`1
`1
`1
`1
`1
`0
`0
`0
`0
`
`255
`255
`255
`255
`255
`255
`255
`255
`255
`
`37
`36
`36
`35
`35
`34
`34
`33
`33
`32
`32
`
`0
`0
`0
`0
`0
`0
`0
`0
`0
`
`255
`255
`255
`255
`255
`255
`255
`255
`255
`
`40
`39
`39
`38
`38
`37
`37
`36
`36
`35
`35
`
`2
`2
`2
`2
`2
`1
`1
`1
`1
`
`255
`255
`255
`255
`255
`255
`255
`255
`255
`
`38
`37
`37
`36
`36
`35
`35
`34
`34
`34
`33
`
`1
`1
`1
`1
`0
`0
`0
`0
`0
`
`255
`255
`255
`255
`255
`255
`255
`255
`255
`
`39
`38
`38
`37
`37
`36
`36
`35
`35
`34
`34
`
`1
`1
`1
`1
`1
`0
`0
`0
`0
`
`Page 64 of 147
`
`

`

`U.S. Patent
`
`Nov. 7, 2000
`
`Sheet 64 of 101
`
`6,144,366
`
`723
`\
`
`TIMING
`CONTROL
`UNIT
`
`730
`\
`
`CORRECTION
`TABLE
`
`FIG.65
`
`REFLECTED
`~ - . . - i LIGHT IMAGE ___,,.._721
`CONTROL
`GENERATION
`SIGNAL
`UNIT
`
`COORDINATE
`SIGNAL
`
`SIGNALS
`CORRESPONDING
`TO REFLECTED
`LIGHT IMAGE
`
`·Ii
`
`AID
`CONVERTER
`
`~724
`
`DIGITAL
`REFLECTED
`11 LIGHT IMAGE
`
`CORRECTION
`r - - - - - - - . i - CALCULATION 1-,,-- 729
`CORRECTION UNIT
`DATA
`
`1
`
`CORRECTED
`RANGE IMAGE
`
`Page 65 of 147
`
`

`

`U.S. Patent
`
`Nov. 7, 2000
`
`Sheet 65 of 101
`
`6,144,366
`
`~RO I -
`IN
`ATE
`
`0
`8
`16
`24
`32
`40
`48
`56
`64
`72
`80
`88
`96
`104
`112
`120
`128
`136
`144
`152
`160
`168
`176
`184
`192
`200
`208
`216
`224
`232
`240
`248
`255