I 1111111111111111 11111 111111111111111 11111 1111111111 lllll 111111111111111111
`
`US005900863A
`
`
`United States Patent
`[19J
`Numazaki
`
`5,900,863
`[11]Patent Number:
`May 4, 1999
`[45]Date of Patent:
`
`[54]METHOD AND APPARATUS FOR
`CONTROLLING COMPUTER WITHOUT
`TOUCHING INPUT DEVICE
`
`5,686,940 11/1997 Kuga ....................................... 345/159
`
`
`
`
`
`
`5,686,942 11/1997 Ball ......................................... 345/158
`
`FOREIGN PATENT DOCUMENTS
`
`7-057103 3/1995 Japan .
`Shunichi Numazaki, Yokohama, Japan[75]Inventor:
`
`
`
`
`
`
`
`Kabushiki Kaisha Toshiba, Kawasaki,[73]Assignee:
`
`Japan
`30th General Conference (Zenkoku Taikai) of Information
`
`
`
`
`
`
`
`P rocessing, Society of Japan, pp. 1249-1250 (1985 Spring).
`
`OTHER PUBLICATIONS
`
`[21]Appl. No.: 08/614,502
`
`[22] Filed:Mar. 1 3, 1996
`
`
`
`
`
`Attorney, Agent, or Firm----Oblon, Spivak, McClelland,
`
`Primary Examiner-Jeffery A. Hofsass
`
`
`
`
`Assistant Examiner-John Tweel, Jr.
`
`
`
`
`
`Maier & Neustadt, P.C.
`
`Mar. 16, 1995 [JP] Japan .................................... 7-083460
`[57]
`ABSTRACT
`Sep. 28, 1995 [JP] Japan .................................... 7-250418
`
`
`
`
`A distance image representing a distribution of distances,
`[51]Int. Cl.6
`G09G 5/08
`
`
`
`each measured between the apparatus and each conceivable
`
`
`
`.......................... 345/158; 345/156; 345/157;
`[52]U.S. Cl.
`
`
`
`point existing on an object is input. The minimum points in
`345/159; 345/175
`
`
`the distance image are detected. The minimum points are
`
`
`..................................... 345/158, 157,
`[58]Field of Search
`
`
`some of the conceivable points on the object which have
`
`345/156, 159, 160, 12, 7, 175; 348/140,
`
`
`
`
`smaller distance values than the other conceivable points.
`141
`
`
`The time differences, each representing a change which each
`
`
`point in the distance image assumes as time passes, are
`
`
`
`
`detected. A designating point for designating a command to
`
`
`
`the system is determined. The designating point is one of the
`
`
`minimum points which has changed in the time difference
`11/1988 Denlinger ................................ 345/175
`
`
`4,782,328
`
`
`
`detected. A system is controlled in accordance with motion
`
`
`
`1/1991 Zimmerman et al. .................. 345/156
`4,988,981
`
`
`of the designating point determined.
`
`
`12/1992 Sigel ....................................... 345/157
`5,168,531
`
`
`345/158 11/1994 Pan ..........................................
`5,367,315
`
`10/1995 Geiser ..................................... 345/156
`5,459,488
`
`33 Claims, 46 Drawing Sheets
`
`
`
`[30] Foreign Application P riority Data
`
`. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
`
`[56]
`
`
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`\0
`
`D
`
`ii
`
`-
`
`f
`
`IPR2021-00922
`Apple EX1005 Page 1
`
`

`

`
`U.S. Patent May 4, 1999 Sheet 1 of 46
`5,900,863
`
`DISTANCE
`IMAGE
`INPUTTING
`SECTION
`
`r-'
`
`DISTANCE
`IMAGE
`ANALYZING
`SECTION
`
`,-J
`2
`
`PROCES SING
`SECT\ ON
`
`r1
`3
`
`F I G.
`
`DISTANCE
`IMAGE
`Vi
`INPUTTING
`SECTION
`
`'
`
`1,
`
`4
`\..
`
`MINIMUM
`POINT
`DETECTING
`SECTION
`
`TIME
`5
`DIFFERENCE
`DETECTING
`SECTION
`
`'
`
`v-G
`
`POINT
`DETERMINING
`SECTION
`
`SYSTEM v1
`CONTROL
`SECTION
`
`F 1 G. 2
`
`IPR2021-00922
`Apple EX1005 Page 2
`
`

`

`U.S. Patent
`5,900,863
`
`May 4, 1999 Sheet 2 of 46
`
`w
`
`D
`
`ii
`
`I t X
`;2j I
`
`PSD
`
`F I G. 3
`
`LED ARRAY
`LENS
`
`F I G. 4
`
`------
`
`IPR2021-00922
`Apple EX1005 Page 3
`
`

`

`U.S. Patent May 4, 1999
`
`Sheet 3 of 46
`
`5,900,863
`
`A
`
`�d�
`LEO ARRAY
`...___.,--30
`
`F l G. 5
`
`LED ARRAY
`40
`
`�
`
`POSITION
`42
`SENSING
`DETECTOR
`
`LEO ARRAY
`4i
`CONTROL
`SECTION
`
`DISTANCE 43
`CALCULATING
`SECTION
`
`47
`
`TIMING
`CONTROL
`SECTION
`
`44
`
`MINIMUM
`TIME
`DIFFERENCE
`POINT
`DETECTING
`DETECTING
`SECTION
`SECTION
`
`FIG. 6
`
`------
`
`POINT DETERMINING 46.
`SECTI ON
`
`IPR2021-00922
`Apple EX1005 Page 4
`
`

`

`5,900,863
`U.S. Patent May 4, 1999 Sheet 4 of
`46
`
`.---- - ------- -- - - - - ---------u
`50
`
`50
`
`,---- --
`
`-....- - - - -
`
`---
`
`-- - - - -------.-------0
`
`V
`
`V
`
`SHIFT
`SHIFT
`i----- - - - - -
`REGISTER
`REGISTER
`,
`',
`''
`'
`
`SHIFT
`_ __, REGISTER 1
`I
`I
`I
`
`-----�
`--------------------------
`
`,------
`
`---
`
`---+--
`
`---
`
`---
`
`- - - - - --�r-----.i
`
`V
`
`SHIFT
`- - - - -_____, REGISTER
`
`53
`
`PULSE
`FREQUENCY
`EXTRACTOR
`DIVIDER
`CLOCK
`GENERATING
`OF
`GENERATOR
`� -- - -----.TIMING PULSES
`52
`
`51
`
`F I G. 7
`
`DISTANCE
`CALCULATING
`SECTION
`
`IPR2021-00922
`Apple EX1005 Page 5
`
`

`

`U.S. Patent May 4, 1999
`
`5,900,863
`
`Sheet 5 of 46
`
`�
`-- -----, 0
`a::
`
`RECIPROCAL
`GENERATOR
`
`:i:
`0
`a::
`
`CONSTANT
`MULTIPLIER
`
`CD
`
`- U-
`
`RECIPROCAL
`GENERATOR
`
`in
`<.D
`
`<t
`
`<.!)
`
`en
`
`a::
`<.!)
`
`a::
`I-
`
`::i:::
`
`I-
`::i:::-
`
`u :::>1-<->
`
`c::t!=
`ff") _J(..)
`<.D
`
`+
`
`+
`
`I-
`
`::i:::-u:::> 1-u
`c::t!=
`_JU
`
`Cl
`
`u.
`0
`
`a::
`0
`0
`u
`
`N
`<.D Cl
`
`"'-<t
`
`
`
`N
`
`0
`
`"" <t
`
`<.D
`
`0
`<.D
`
`<.D
`
`0.. (I') Cl
`
`N
`
`IPR2021-00922
`Apple EX1005 Page 6
`
`

`

`LED i x _________ x LED 2 x�_x
`i BACKGROUND
`
`�
`
`��
`
`2 BACKGROUND 3
`
`�----'x
`� ---� ---\ ,---
`BACKGROUND
`
`3
`
`0 ....,
`
`O'I
`,i;;..
`
`.... O'I
`
`
`
`=-
`
`'JJ.
`
`�
`�
`
`= ......
`...... �
`�
`�
`r:JJ.
`d
`
`•
`
`•
`
`2 X BACKGROUND
`73 LATCH TRIGGER f I f I f I
`LED 3 'C \0
`72 OUTPUT OF
`�
` '-<
`
`0--, �
`
`\0 ==....
`
`00
`
`....
`
`Ul
`
`(LE03 }-
`
`( LEO 2 )
`
`( LED i )
`
`FIG. 9 F
`
`i X BACKGROUND
`
`XBACKGROUND
`
`BACKGROU ND VALUE
`75 OUTPUT EXCLUOI
`NG
`FIG. 9 E 74 OUTPUT OF LATCH
`
`SIGNA L
`
`FIG. 9 D
`
`A/0
`
`FIG. 9 C
`
`7i A/0 TRIGGER
`FI G. 9 A 70 OUTPUT OF PSO
`
`SIGNAL
`
`F I G 9 8
`
`·
`
`IPR2021-00922
`Apple EX1005 Page 7
`
`

`

`0 ....,
`
`O'I
`,J;..
`
`-..J
`
`=-�
`
` �
`
`rF.J.
`
`BACKGROUND X
`I f
`
`X
`1
`
`X LED i X LED 2 X LED 3 X LED 4 X. '[_
`BACKGROUND
`
`'"""'
`�,J;..
`'--<
`�
`�
`
`\0
`\0
`\0
`
`\
`
`BACKGROUND
`
`·
`
`�
`
`= ......
`
`......
`�
`�
`
`•
`
`r:JJ.
`d
`
`•
`
`Fl G. i O A 80 OUTPUT OF PSD
`
`BACKGROUND VALUE --J.
`
`FIG. i OF
`
`FIG. iOE
`
`Bi A/D TRIGGER
`
`Fl G. i OB
`
`OF�
`82 OUTPUT
`SIGNAL
`
`83 LATCH TRIGGE R
`
`SIGNAL
`
`FIG. i OD
`
`A/D
`
`FIG. i OC
`
`85 OUTPUT EXCLUDING )>--------i(
`84 OUTPUT OF LATCH
`
`�
`0--,
`00
`= ....
`=
`
`\0
`
`....
`
`Ul
`
`.
`
`_.
`
`LED 1 X LED 2 XL ED 3 X LED 4 )------,[
`
`. .
`
`_ _
`
`_
`
`IPR2021-00922
`Apple EX1005 Page 8
`
`

`

`U.S. Patent
`5,900,863
`
`
`May 4, 1999 Sheet 8 of 46
`
`i-----COMPARATOR Do< D3
`2
`bit
`DISTANCE
`
`iO ___,�COMPARATOR Do< D4
`DATA
`bit
`
`i6
`
`t t
`bit
`05 COMPARATOR Do<D5
`t2 bit
`20bit DG COMPARATOR Do< DG
`2i bit 07 COMPARATOR Do< D7
`22 bitDa COMPARATOR
`00 < De
`
`i03
`
`,--- -- ---, �D > TH
`COMPARATQRr---�
`
`TH
`
`X-COORDINATE 4
`
`Y-COORDINATE 4
`
`F I G.
`i i
`
`IPR2021-00922
`Apple EX1005 Page 9
`
`

`

`U.S. Patent
`May 4, 1999
`
`5,900,863
`
`
`Sheet 9 of 46
`
`:I:
`
`I- A
`
`
`
`v Cl
`<I
`
`a::
`0
`�
`
`if
`0 u
`
`:I:
`
`I- A
`
`
`
`rt"')
`
`0
`
`<I
`
`a::
`0
`�
`a::
`if :ii:
`8
`
`:c
`
`>- A
`
`
`
`N
`C
`<I
`
`a::
`0
`�
`a::
`�
`:E
`8
`
`:c
`
`>- A
`
`
`
`0
`
`<I
`
`a::
`0
`�
`a::
`�
`:E
`0 u
`
`:I:
`>-
`
`Cl
`<I
`
`+
`
`N
`C
`<I
`
`rt"')
`
`Cl
`<I
`
`w
`
`I- en
`
`
`
`N
`
`--
`
`. (..!)
`
`LL
`
`+- .0
`
` .....
`
`
`
`v
`
`+- .0.
`
`
`
`rt"')
`
`+- .0
`
`
`
`N
`
`-..a
`
`+-
`
`(!)
`
`w ..0 a::
`
`I-LL
`
`
`:c
`
`en
`
`0
`
`w
`
`Cl
`
`IPR2021-00922
`Apple EX1005 Page 10
`
`

`

`U.S. Patent May 4, 1999 Sheet 10 of
`5,900,863
`46
`
`Cl
`uJ
`
`�
`z
`<!>
`en
`Ll.l
`Cl
`
`I- z
`
`
`0
`a.. rt')
`
`0
`
`N
`�
`u
`
`Cl
`
`N
`
`0
`
`�
`Cl u
`
`� ro
`
`<C
`Cl
`
`......
`
`Ll.l
`�
`z (.!)
`a::
`
`0
`
`0
`0
`u
`
`-
`
`LL.
`
`0
`
`- 0
`
`�
`Cl u
`
`�
`<C
`Cl
`
`Ll.l
`
`z
`<C
`I-
`en
`Cl
`
`>-
`
`0
`
`>-
`a:: V
`� X
`a::
`a: �
`
`0
`u
`
`X
`
`<!>
`z
`I-
`�
`u
`a::
`I-
`en
`
`z
`
`I
`LL.J
`<!>
`<C a:: c.!)
`O <C
`I- __JenLJ...
`
`IPR2021-00922
`Apple EX1005 Page 11
`
`

`

`
`U.S. Patent
`
`May 4, 1999 Sheet 11 of 46
`
`5,900,863
`
`�
`
`�
`
`v 4o
`LED ARRAY
`
`V42
`POSITION
`SENSING
`DETECTOR
`
`u-4i
`LED ARRAY
`CONTROL
`SECTION
`
`DISTANCE
`�---� CALCULATING
`SECTION
`
`v41
`TIMING
`CONTROL
`SECTION
`
`44
`!
`
`TIME
`DIFFERENCE
`DETECTING
`SECTION
`
`MINIMUM
`POINT
`DETECTING
`SECTION
`
`C ANDIDATE POINT
`STORING SECTION
`
`L,ri20
`
`v i2
`POINT DETERMINING
`SECTION
`
`F I G. i 4
`
`IPR2021-00922
`Apple EX1005 Page 12
`
`

`

`U.S. Patent May 4, 1999
`
`5,900,863
`
`Sheet 12 of 46
`
`rt')
`
`.r
`u
`�
`
`_J
`
`N
`rt')
`�"\. 0
`C> :::.:::: u
`
`lO
`rt')
`-
`\
`a::w
`_J :co
`+
`uo::
`) ,,----
`-z3:o
`I- I-
`_,,
`
`_J
`
`U>U
`
`oo/ o-
`-.:t rt')
`-X
`
`N
`-.:t
`
`Lr
`
`N
`::c
`- �
`
`<(
`_J
`
`LC)
`
`--
`
`. (!)
`
`LL.
`
`.J
`
`-::cu
`
`I- _J
`
`
`
`Cl �-..
`JI
`�----
`
`I -...,...0 0 I
`I
`I v- I rt">
`N
`I...,...
`�\ 0
`I I....
`:::.:::: I a::
`C> u I w
`_1-_ :co
`I _J
`4
`I) I uo::
`3:0
`I-I-
`,, -z
`CDo/ C ) - U>U
`
`_J
`
`I
`
`--0
`
`___
`
`� N
`X
`
`,.--...
`
`I \I
`
` 0 I
`I
`I rt">
`,�
`I �
`I a::
`:::.::::
`Cl u
`w
`,....l-:co
`' I _J
`uo::
`3:o
`I- I-
`,,
`__
`-z
`
`0
`�"\.
`
`...,...
`0
`
`_J
`
`U> U
`�
`
`)
`
`I
`
`tD°J ) -
`
`rt')
`
`X
`
`rt')_,--
`-
`
`-)
`
`IPR2021-00922
`Apple EX1005 Page 13
`
`

`

`5,900,863
`U.S. Patent May 4, 1999 Sheet 13 of
`46
`
`8k��NCE - - - --
`COORDINATE -----
`DATA
`
`- - --t
`
`------
`
`TO MEMORY
`--- CIRCUIT
`
`F I G. i 6
`
`INPUT TRIGGER
`SIGNAL CK\
`TO FU P-FLOP
`
`X i
`FIG. i 7 A
`
`X
`
`X2
`Fl G. i 7 8
`
`X
`
`X3
`FIG. i 7C
`
`X
`
`I
`
`IX I
`!X
`
`03
`Fl G. i 7 D
`
`X
`
`ix
`
`IPR2021-00922
`Apple EX1005 Page 14
`
`

`

`U.S. Patent May 4, 1999
`
`5,900,863
`
`Sheet 14 of 46
`
`l­o0
`
`
`
`I- L
`
`
`
`0
`
`�Cl.
`ug
`_J I.L I-
`<( I :::,
`;z: Cl. U
`c.!:>-O:: _ _J_
`cnLLU
`
`N
`X ><
`
`rt)
`
`X
`
`rt)
`
`0
`
`�N
`��
`a::u
`
`I- _J
`:I: <t
`u:z
`�� _Jen
`
`<:{ CD u
`co CD en
`....... ....... .......
`
`Cl w
`CX)
`(X)
`.......
`
`......
`
`LL
`(X)
`.......
`
`. <.!)
`<.!)
`
`<..!)
`
`<.!) (.!)
`
`LL LL LL
`
`-
`
`-LL-
`
`LL
`
`<.!)
`
`lJ_
`
`IPR2021-00922
`Apple EX1005 Page 15
`
`

`

`5,900,863
`U.S. Patent
`May 4, 1999 Sheet 15 of
`46
`
`40
`
`LED ARRAY
`
`42
`
`POSITION
`SENSING
`DETECTOR
`
`4i
`LED ARRAY
`CONTROL
`SECTION
`
`DISTANCE
`CALCULATING
`SECTION
`
`43
`
`47
`
`TIMING
`CONTROL
`SECTION
`
`45
`
`44
`
`TIME
`MINIMUM
`POINT
`DIFFERENCE
`DETECTING
`DETECTING
`SECTION
`SECTION
`
`CANDIDATE POINT
`STORING SECTION
`
`t20
`
`t53
`
`t 5 t
`
`POINT
`i50
`DETECTING -­
`SECTION
`
`POINT
`POINT
`TRACKING
`STORING
`,-.--.---; SE CT I ON
`SECTION
`
`i52
`SWITCHING
`CONTROL
`SECTION
`
`F I G. i 9
`
`IPR2021-00922
`Apple EX1005 Page 16
`
`

`

`U.S. Patent
`May 4, 1999 Sheet 16 of 46 5,900,863
`
`160
`t
`\
`LATCH
`CIRCUIT
`
`i63
`\
`LATCH
`CIRCUIT
`
`16i
`i62
`i
`\ \
`LATCH
`LATCH
`CIRCUIT
`CIRCUIT
`
`1
`
`t65
`�
`
`DISTANCE
`DISTANCE
`164'-
`CALCULATING CALCULATING
`CIRCUIT
`CIRCUIT
`
`DISTANCE
`CALCULAT ING
`CIRCUIT
`
`I
`
`i66
`
`COMPARATOR
`i67---------
`SECTION
`
`I/
`
`POI NT RENEW! NG f i68
`SECTION
`
`2 0
`F I G.
`
`IPR2021-00922
`Apple EX1005 Page 17
`
`

`

`U.S. Patent May 4, 1999
`
`Sheet 17 of 46
`
`5,900,863
`
`7 l.;40
`
`LED ARRAY
`
`LED ARRAY f 4 i
`CONTROL
`SECTION
`
`�
`POSITION V 42
`SENSING
`DETECTOR
`
`DISTANCE l143
`CALCULATING
`SECTION
`
`v41
`
`TIMING
`CONTROL
`SECTION
`
`45
`
`\ TIME !
`DIFFERENCE
`DETECTING
`SECTION
`l
`
`44
`)
`POINT
`MINIMUM
`DETECTING
`SECTION
`
`1,
`
`POINT
`i50 "-DETECTING ---­
`SECTION
`
`POINT
`POINT
`TRACKING
`STORING
`SECTION
`SECTION
`iti I
`
`�
`i53
`
`SWITCHING Lr i52
`CONTROL
`SECTION
`
`F I G. 2 i
`
`IPR2021-00922
`Apple EX1005 Page 18
`
`

`

`U.S. Patent
`May 4, 1999
`
`5,900,863
`
`Sheet 18 of 46
`
`JzOoa::_
`
`1-t­
`zu
`ow
`ucn
`
`.
`(!)
`
`lJ_
`
`a::
`w
`
`W-..--,
`o>
`'-.,.__ :z
`<t 8
`
`t­a::
`
`�--- ffi
`
`-
`
`N
`
`_. Q_
`:ii:
`<t
`
`w
`b1 a:: z
`I 0
`t-­
`::c t­
`<.!) U
`-W
`_J(f)
`
`<.!)
`:z
`
`t­t- �
`
`
`w:z
`I 0
`t-­::c t­
`
`<.!)U
`-LU
`_J(f)
`
`w
`>
`�
`<tx
`a::w
`C:: C
`LU:z c::
`lJ...o c:: -----N �
`0::-
`
`0,._
`
`�
`
`0,._
`
`N
`
`N
`
`(!)
`
`LL.
`
`IPR2021-00922
`Apple EX1005 Page 19
`
`

`

`
`
`U.S. Patent May 4, 1999 Sheet 19 of 46 5,900,863
`
`CURSOR
`CONTROL
`SECTION
`
`CONTROL
`EXECUTING
`SECTION
`
`COMMAND
`CHANGING
`SECTION
`
`DISTANCE
`IMAGE
`INPUT
`SECTION
`
`MINIMUM
`POINT
`DETECTING
`SECTION
`
`TIME
`DIFFERENCE
`DETECTING
`SECTION
`
`POINT
`DETERMINING
`SECTION
`
`F l G. 2 3
`
`IPR2021-00922
`Apple EX1005 Page 20
`
`

`

`U.S. Patent
`May 4, 1999
`
`5,900,863
`Sheet 20 of
`46
`
`CURRENT FOR
`DRIVING LED
`F\G.25A
`
`CURRENT OUTPUT
`FROM PSD
`FIG.258
`TR I GGER SIGNAL FOR
`AID CONVERSION ___ __.
`FIG.25C
`Fl RST SAMPLE
`-ANO-HOLD
`CIRCUIT 221
`SECOND SAMPLE
`-ANO-HOLD
`CIRCUIT
`AMPLIFIER
`
`220
`
`222
`
`223
`
`A/0
`CONVERTER
`
`F \ G. 26
`
`---+----'
`
`FIG. 27 A CURRENT FOR DRIVING LED
`F \ G. 27 8 CURRENT OUTPUT FROM PSD
`SIGNAL FOR TRIGGERING n1
`FIRST SAMPLE-ANO-HOLD I :
`CIRCUIT
`...... --+-1--1 --
`--�
`SIGNAL FOR TRIGGERING n 1
`SECOND SAMPLE-AND-HOLD
`,___-+-I --
`____ _____,
`CIRCUIT
`TRIGGER SIGNAL FOR
`A/0 CONVERSION __ fl_
`
`I
`I
`I
`I
`
`I
`
`I
`
`I I :
`I I I
`I I I
`I I I
`1
`
`1
`
`I
`
`FIG. 27C
`
`FIG. 270
`
`FIG. 27 E
`
`IPR2021-00922
`Apple EX1005 Page 21
`
`

`

`U.S. Patent
`May 4, 1999
`
`5,900,863
`
`Sheet 21 of 46
`
`230
`<_
`
`23i
`\
`
`232
`\
`
`LIGHT
`-EMITTING
`SECTION
`
`LIGHT
`AMPLIFIER
`-RECEIVING
`SECTION
`
`234
`
`2�3
`
`MAGNITUDE
`DETERMINING
`SECTION
`
`A/D
`CONVERTER
`
`235
`;
`
`236
`
`� ,)
`
`OPERATION
`SECTION
`
`OUTPUT i-------
`SECTION
`
`F \ G.2 8
`
`24i
`240
`)
`!
`DISTANCE
`MINIMUM
`MINIMUM
`POINT
`POINT
`IMA GE
`PROCESSING
`SELECTING
`DETECTING
`INPUT
`SECTION
`SECTION
`SECTION SECTION
`
`242
`,I
`
`243
`;
`
`F I G. 2 9
`
`240
`;
`DISTANCE
`IMAGE
`INPUT
`SECTION
`
`f---
`
`24i
`,J
`
`242
`,J
`
`243
`;
`
`MINIMUM
`MINIMUM
`PROCESSING
`POINT
`POINT
`SELECTING
`SECTION
`DETECTING
`SECTION
`SECTION ----
`
`F I G. 3 0
`
`IPR2021-00922
`Apple EX1005 Page 22
`
`

`

`5,900,863
`U.S. Patent
`
`May 4, 1999 Sheet 22 of 46
`
`POINT
`25i\____ MOTION
`�
`DETECTING
`SECTION
`
`250
`
`252
`\.,.
`
`254
`;
`
`255
`,.J
`
`DISTANCE
`IMAGE
`INPUT
`SECTION
`
`MINIMUM � MINIMUM
`� PROCESSING
`POINT
`POINT
`DETECTING SELECTING
`SECTION
`� SECTION
`SECTION
`
`'
`
`OBJECT
`ANALYZING -
`253"""'\... SECTION
`
`F I G. 3 i
`
`IPR2021-00922
`Apple EX1005 Page 23
`
`

`

`5,900,863
`U.S. Patent May 4, 1999 Sheet 23 of
`46
`
`START
`
`DISTANCE VALUE OF
`
`MINIMUM POI NT
`- dm COUNTER= 0
`
`SELECT ONE FROM
`CIRCUMFERENCE
`OF MINIMUM
`POINTS SATISFYING
`dm<d<dm+th
`DISTANCE VALUE.._d
`
`r---------.-i
`
`NO
`
`NO
`
`YES
`
`COUNTER++
`
`YES
`
`E ND
`
`F I G. 32
`
`IPR2021-00922
`Apple EX1005 Page 24
`
`

`

`�
`0--,
`00
`= ....
`=
`
`\0
`
`....
`
`Ul
`
`SECTION
`PROCES SING
`
`F I G. 3 4
`
`FILTE R
`
`SECTION
`SELECTING
`POINT
`
`MINIMUM
`
`SECTION
`DET CTING
`POl�T
`
`MINIMUM
`
`264
`
`263
`
`262
`
`26i
`
`SECTION
`INPUT
`IMAGE
`DISTANCE
`260
`
`POSIT ION
`
`F I G. 3 3
`
`
`----'
`
`
`
`Si
`
`---thf
`
`= ......
`
`...... �
`�
`�
`
`•
`
`r:JJ.
`d
`
`•
`
`DISTANCE
`
`===� �===------::� tf _____
`
`O'I
`,J;..
`....,
`
`0
`
`,J;..
`N
`....
`
`=­�
`
` �
`'JJ.
`
`\0
`\0
`\0
`'"""'
`�,J;..
`'-<
`�
`�
`
`�
`
`_
`
`______
`
`__
`__
`
`___
`
`...._.._
`
`IPR2021-00922
`Apple EX1005 Page 25
`
`

`

`46 5,900,863
`U.S. Patent May 4, 1999 Sheet 25 of
`
`DISTANCE
`IMAGE
`INPUT
`SECTION
`�
`270
`
`MINIMUM
`POINT
`DETECTING
`SECTION
`
`MINIMUM
`POINT
`SELECTING v212
`SECTION
`
`I
`
`271
`
`275
`
`,)
`
`COMMAND
`EXECUTING
`SECTION
`
`276
`l, "
`
`-
`
`DISPLAY
`CONTROL
`SECTION
`
`POINT-HALTING
`TIME
`DETECTING
`SECTION
`!
`273
`
`274
`(
`POINT-MENU .___
`CORRESPONDENCE
`DETECTING
`SECTION
`F
`I G.
`35
`
`DISTANCE
`MINIMUM MINIMUM
`POINT --� POINT
`IMAGE
`SELECTING Lr 272
`� DETECTING
`INPUT
`SECTION
`SECTION SECTION
`;
`\
`270
`271
`
`275
`
`PO l'NT -HALT I NG i-------+---'
`_----i TI ME
`DETECTING
`SECTION
`
`COMMAND
`EXECUTING
`SECTION
`
`273 274
`(
`POINT-MENU --
`...___ CORRESPONDENCE
`DETECTING
`SECTION
`
`F I G. 3 6
`
`DISPLAY
`CONTROL
`SECTION
`
`!
`276
`
`277�
`SPEECH
`OUTPUT
`�
`SECTION
`
`IPR2021-00922
`Apple EX1005 Page 26
`
`

`

`U.S. Patent May 4, 1999 Sheet 26 of 46
`
`5,900,863
`
`DISTANCE
`MINIMUM
`POINT
`IMAGE
`� DETECTING
`INPUT
`SECTION l\.28i
`SECTION
`!
`280
`
`283
`\
`FIRST
`SECOND
`COMMAND
`POINT-MOTION ��P OINT-MOT ION 1------i EXECUTING
`282 """'\._____
`DETECTING DETECTING SECTION
`SECTION
`SECTION
`
`284
`\
`
`'
`
`MENU
`285 '--- CONTROL
`SECTION
`
`F I G. 3 7
`
`PLAY
`
`MOVE DELETE
`
`F l G. 3 8
`
`RESTAURANT BAR S N ACK BAR CAFE
`
`FRENCE ITALIAN
`
`OTHERS
`
`-3000
`
`~ 6000 ~ iOOOO iOOOO~
`
`F 1 G. 3 9
`
`290
`
`IPR2021-00922
`Apple EX1005 Page 27
`
`

`

`U.S. Patent May 4, 1999 Sheet 27 of 46 5,900,863
`
`PLAY
`
`MOVE
`
`COPY
`
`F I G. 4 0
`
`DELETE
`
`PLAY
`
`F I G. 4 i
`
`COPY
`
`DELETE
`
`IPR2021-00922
`Apple EX1005 Page 28
`
`

`

`.. �
`
`'"""'
`�,J;..
` '-<
`
`��
`
`\0
`\0
`\0
`
`193 ,..__,
`
`j
`
`____ J
`
`= .....
`
`
`�•
`
`
`rJJ.•
`
`..... �
`�--'---,I
`------7
`�
`�
`
`HS
`
`BAND-PASS ABSOLU TE
`7
`
`----
`
`{7
`
`-------
`
`t75 176
`
`------
`
`I�--'--,-�-�-�--_ __,_ __
`
`_______
`
`\._ __
`
`\H
`
`HI 172 1
`
`\A
`
`ECEIVI NG
`
`\90LIGHT-R
`
`LENS"\
`
`PRE-
`
`I AMPL\FI
`
`Ir-------.
`,--------------�-----------�
`
`I
`
`r----�----1
`
`--;-J . �
`DETECTOR I DETECTOR EXTRACTOR DETECTOR I CONVERTER
`
`0 ....,
`
`O'I
`,J;..
`
`00
`N
`
`=­�
`
` � ....
`'JJ.
`
`�
`�
`00
`=
`
`=....
`
`
`
`\0
`
`
`0-.....
`
`F I G. 4 2
`
`t8
`
`f i
`5 ..--r1 SECT I ON t86__rj SECT I ON
`OUTPUT I I OUTPUT
`
`I
`
`mi '--100 179
`-------
`r-1CENTER HALT I GRAVITY CENTER ROD TIP §��E( THAPED I A/ D
`
`\847-I L ____ ---------
`
`102
`
`rn3
`
`93,....___,
`
`CONTROL SECTI ON I.
`
`I rn1� , ,
`
`�-----1
`
`; t9i LIGHT-APP
`
`LENS
`
`LYING 189
`SECTION CIRCUIT
`EMITTING DRIVE
`LIGHT-�--t88
`
`t
`
`1 --------------------------------
`1�---
`
`I
`
`GENERATOR L....-..,--,----'
`
`ER
`ARRAY
`DETECTO R SWl�CH CONVERTER-AMPLIFIER FILTER VALUE INTE GRATOR 1
`�tHOTO-ANA OG l I /
`V
`3 174
`
`IPR2021-00922
`Apple EX1005 Page 29
`
`

`

`U.S. Patent May 4, 1999 Sheet 29 of 46
`
`5,900,863
`
`F I G. 4 3
`
`40t
`
`�
`
`403
`
`�
`404
`
`�
`� ('j;
`
`-
`
`,- -
`---
`r.:-t\/ ---------
`;[' 7 ( \ '
`V -\ ' 1-/ \
`\, n�
`rY/
`I
`' '::::::f) I\S v,
`II
`
`"'-
`
`_,.
`
`402
`
`F 1 G. 4 5
`
`IPR2021-00922
`Apple EX1005 Page 30
`
`

`

`�
`0--,
`00
`=
`=
`
`....
`
`\0
`
`LRESET
`
`Ul ....
`
`f
`305
`
`LRESET
`
`0 l
`
`F I G. 44
`
`O'I
`,J;..
`....,
`
`0
`
`=­�
`
` �
`'JJ.
`
`0
`
`
`....�
`
`\0
`\0
`\0
`'"""'
`�,J;..
`'-<
`�
`�
`
`= ......
`
`---t
`
`)
`304
`-t
`
`f
`303
`)
`302
`
`-t
`
`......
`�
`�
`
`•
`
`r:JJ.
`d
`
`•
`
`30i
`
`_____ MMn r
`
`BAND-PASS
`
`ANALOG SWITCH
`OUTPUT CURRENT OF
`SECTION O
`INPUT CURRENT OF f LIGHT-EMITTING
`
`i89
`
`2 /
`
`i9
`
`A/ D CONVERTER r-i 79
`--------
`VOLTAGE
`J---i78
`INTEGRATOR
`SIGNAL
`VOLTA:t
`GENERATOR --
`ABSOLUTE VALUE i77 SIGNAL
`VOL TAG�r
`F I LTER r i 76
`SIGNAL t
`AMPLIFIER
`I /V CONVERTER-i74
`ANALOG SWITCH
`ARRAY
`PHOTO DETECTOR
`f? "'--""i7 i
`SECTION
`LIGHT -EMITTING
`
`IPR2021-00922
`Apple EX1005 Page 31
`
`

`

`U.S. Patent May 4, 1999 Sheet 31 of
`46 5,900,863
`
`F l G. 4 6
`
`F I G. 4 8
`
`IPR2021-00922
`Apple EX1005 Page 32
`
`

`

`U.S. Patent
`May 4, 1999
`46 5,900,863
`Sheet 32 of
`
`r<> ..r-
`0 0
`w w
`
`LO
`0
`w
`
`0
`w
`
`\
`
`. t!)
`
`LL
`
`IPR2021-00922
`Apple EX1005 Page 33
`
`

`

`U.S. Patent
`May 4, 1999
`
`5,900,863
`
`Sheet 33 of 46
`
`-.....
`
`
`
`C
`
`C
`
`a..
`
`0)
`
`�
`
`-.x.
`
`C
`
`-
`
`- LL-
`
`(\J
`0
`ex:>
`
`.....
`
`-a..
`
`
`
`IPR2021-00922
`Apple EX1005 Page 34
`
`

`

`5,900,863
`U.S. Patent May 4, 1999 Sheet 34 of
`46
`
`-�.-------d-�...------.1--�---
`
`DISTANCE
`THE FINGER
`TIP HAS MOVED
`_.......--+.-------...------.--+-
`
`-.--.--
`
`���>+-------��1==�--
`
`F I G. 5 0 A � F I G. 5 0 8
`
`DISTANCE
`THE
`FINGER TIP HAS
`MOVED
`- ----]--
`
`- -----
`
`- ----
`
`----I-
`DISTANCE THE
`GRAV I TY CENTER
`HAS MOVED
`
`F I G. 5 i A � F I G. 5 t B
`
`IPR2021-00922
`Apple EX1005 Page 35
`
`

`

`46 5,900,863
`U.S. Patent
`May 4, 1999 Sheet 35 of
`
`F l G. 52A
`
`I
`I
`I •
`I
`
`I
`
`I
`
`I
`I
`I
`I
`I
`I
`
`I I
`I
`DISTANCE THE
`
`X-COOROINATE OF I
`GRAVITY CENTER H FINGER TIP HAS
`DOES NOT CHANGE I I MOVED
`I I
`I I
`I
`I
`
`•
`
`F I G. 528
`
`IPR2021-00922
`Apple EX1005 Page 36
`
`

`

`U.S. Patent May 4, 1999 Sheet 36 of 46
`
`5,900,863
`
`<C i!..O
`....J -7
`t5<.!:I I �
`_z , .......
`cncnz i,--
`� 8��1
`<(Q U I
`zo:::w
`N
`o::: <( [L (/) I
`�
`N
`I
`c:i_
`I
`>
`I
`wO <(
`1-t:;:
`o:::
`3� IB
`cnz z
`I-
`0 UJ
`row
`
`...JUI-I
`
`0
`1-
`
`<::t
`(\J
`
`r<>
`LO
`
`--�N
`
`LL.
`
`....JZ
`
`00 o:::-
`I-I-zu
`owucn
`
`f"I')
`N
`
`0
`' <(
`
`O
`
`� N
`
`I
`
`<C<.!:>
`
`,-------------
`I
`� --,
`I 0)
`I N u.
`I O
`1
`I
`I �
`I N
`I
`I I
`I ffi
`I 1-
`I Ww
`0::: 0:::
`I> -
`I t5u.
`U ....J
`I
`>CL
`I'- �
`-<C
`L ___ ---------------�
`
`...... I u0:::
`
`f"I')
`0
`N
`
`(
`
`u
`
`w
`>
`0::
`0
`(/)
`
`-
`
`�
`I ><
`
`/ /
`
`/
`/
`/
`/
`/
`/
`/
`
`---------
`
`0
`N......
`
`Lt")
`0
`N
`
`Y-AXIS DRIVE CIRCUIT
`
`IPR2021-00922
`Apple EX1005 Page 37
`
`

`

`May 4, 1999 Sheet 37 of 46
`U.S. Patent
`
`5,900,863
`
`�
`r--------------
`---7
`I J �
`L!")
`I
`--r I <! o 1-
`I
`� -+- ----; cncnz
`b I
`�I 5� � ::::>
`I
`-z
`--
`�
`� I
`l
`JUI-
` 8��
`:::c
`a:: � I
`I <!OU
`en
`W
`I
`I
`I �8:�
`I
`J
`I
`�
`rj
`r<')I
`o I J
`�
`LC")
`�I <!
`::::>
`� I-
`Z ,,...
`--=:
`..::;
`(/') c:;; z a::
`�z a
`�cno u
`:::c
`ow­JUI­
`<! OU
`I za:::w
`I ._ _ _,
`I
`L __________ -______ __J
`
`(.!)
`
`JZoo a:::-
`LJ...
`1-1- --�
`;zu
`ow (._)Cf)
`
`r<')
`
`u
`
`II
`
`,.,
`
`<(�Cf)
`
`0 r<')
`
`-- I­:::>u
`
`0:: �+---+-��
`u
`�
`0:: 0Cf)
`
`X
`<(
`I
`X
`
`0 -----­r<')
`
`--
`
`Y-AXIS DRIVE CIRCUIT
`
`IPR2021-00922
`Apple EX1005 Page 38
`
`

`

`5,900,863
`U.S. Patent May 4, 1999 Sheet 38 of
`46
`
`t403
`
`i404
`
`F I G. 5 5 A
`
`F I G. 5 5 8
`9 i506
`i5o i
`
` 1
`r_,I
`
`ROD-SHAPED
`OBJECT DETECTOR
`i507
`/
`�----' '�-�----,
`i502
`
`1504
`
`ONE-ROW
`PIXEL VALUE
`UNIT
`SUMMING
`
`ROD TIP
`EXTRACTOR
`
`t508
`\
`
`CENTER
`GRAVITY
`DETECTOR
`
`TIP-EXTRACT
`I NG
`PARAMETER
`CALCULATOR
`)
`t503
`
`-----t 509
`
`)
`
`F I G. 5 6
`
`0
`
`IPR2021-00922
`Apple EX1005 Page 39
`
`

`

`�
`0--,
`00
`= ....
`=
`
`\0
`
`....
`
`Ul
`
`DETECTOR
`f
`t707
`
`-0
`
`-MOT I ON
`
`ENTER
`
`? C
`i70
`
`B
`
`J i705
`
`t704
`
`I
`
`0 ....,
`\0
`�
`
`O'I
`,J;..
`
` � ....
`
`=­�
`
`....,
`
`PROCE SSING SECTION )
`DIGITAL SIGNAL
`
`A/D
`
`SECTION
`
`PROCESSING
`
`f-----9-
`
`SIGNAL
`
`i--
`
`�
`
`SECTION
`SENSOR ANALOG
`
`=......
`
`� ...... �
`
`�
`
`•
`
`r:JJ.
`d
`
`•
`
`-
`
`)
`
`'JJ.
`
`\0
`\0
`\0
`'"""'
`... o
`�,J;..
`'-<
`�
`�
`
`F I G. 57
`
`{
`t608
`
`) I BUTTON-CO NDITION I a
`
`BUTTON
`
`.. DETECTOR
`
`\
`i607
`
`l606
`
`l604
`
`t603
`
`rno2
`
`i60i
`
`SECTION
`
`PROC ESSING
`
`A/D DIGITAL SIGNAL
`
`SECTION
`
`SECTION PROCESSING
`SENSOR ANALOG
`
`SIGNAL
`
`IPR2021-00922
`Apple EX1005 Page 40
`
`

`

`U.S. Patent May 4, 1999
`
`5,900,863
`Sheet 40 of
`46
`
`i802
`�804
`
`□□
`
`i803
`
`\
`\
`I
`
`\
`
`\
`I
`
`\
`
`F I G.
`59
`
`IPR2021-00922
`Apple EX1005 Page 41
`
`

`

`0--, �
`
`00
`= ....
`=
`
`\0
`
`....
`
`Ul
`
`O'I
`,J;..
`....,
`
`0
`
`....
`
`=­�
`
`,J;.. '"""'
` �
`'JJ.
`
`'"""'
`�,J;..
` '-<
`
`��
`
`\0
`\0
`\0
`
`= ......
`
`...... �
`�
`�
`
`•
`
`r:JJ.
`d
`
`•
`
`t9
`
`6 0 A
`
`F I G.
`
`NOT REACHE D
`
`-ED
`infoStick[nJ
`status-EXTEND
`infoS tick[nJ
`length-i
`PIXELS
`-x COORD INATE OF END OF
`infoStick[nJ
`xend [OJ
`PIXELS
`--X COO RD I NATE OF START OF
`infoStick[nJ
`xstor [OJ
`OBJECT
`NEW ROD-SHAPED
`REGISTER
`CONTINUOUS
`NOT
`DETECTED?
`CANDIDATE ALREADY
`CONTINUOUS TO THE
`ARE PIXEL ROWS
`
`09
`i908�CHED
`infoStick[nJ
`length �d
`MINED VALUE?
`OBJECT�
`REACHED A PREDETE R-
`ROD-SHAPED
`HAS LENGTH OF UPDATED
`infoS tick[n] stotus--EXTENOED
`infoSti ck[nJ length-+ i
`PIXEL S
`-X COORDINATE
`OF END OF
`infoStick [nJ · xend [i]
`PIXELS
`-X COORD INATE OF START OF
`infoStick
`[n] . xstort[J]
`.
` CANDI DATE
`UPDATE INFORMAT ION OF THE
`CONTINUOUS
`1907
`t905
`
`06
`t9
`
`�
`
`�90 i
`
`i904
`n status -NOSTICK
`
`--------; MOVE TO NEXT LI NE
`
`infoStick
`ONE LINE
`DATE NOT UPDATED IN THIS
`DELETE ROD-SH APED CANDI­
`/i903
`-TAL� COMPLE ffl
`FOUND IN EACH HOR I ZON
`VALUE OF P OR MORE
`PIXELS,EACH
`HAVING
`ESS THAN C ROWS OF
`SCAN HORIZONTAL
`tlN
`infoStick [nJ
`OBJECT CANDIDATES
`FOR STORING ROD-SHAPED
`PREPARE STORAGE UNITS
`
`OF ONE
`
`<L
`
`IPR2021-00922
`Apple EX1005 Page 42
`
`

`

`46 5,900,863
`U.S. Patent May 4, 1999 Sheet 42 of
`
`200{
`r;y-
`y
`CALCULATE TH
`2002
`TH= ( sum/max )2 x max
`sum: AVERAGE OF TOTAL PIXEL VALUES PER
`I/
`LI NE WI TH RESPECT TO FIRST AND
`
`SECOND LI NES ( UPPERMOST I S O th
`LI NE) FROM ROD-SHAPED OBJECT
`max: LARGEST VALUE AMONG Fl RST AND
`SECOND LINES
`ASSUME PIXEL VALUE SUM OF SOME OF Li 2003
`
`PIXELS IN iTH LINE(UPPERMOST IS
`0TH LINE)OF ROD-SHAPED OBJECT IS
`xsum [i], OBTAIN kand a ( O< a< i)
`
`
`SATISFING TH =.[xsum[i]+a. xsum [k]
`t=O
`
`k-i
`
`=
`
`2004
`MULTIPLY EACH LARGE -VALUE PIXEL V
`
`IN K-TH ROW BY VARIABLE a
`
`k
`
`OBTAIN GRAVITY CENTER ( XX y y)
`[nJ · ystart + i } / TH V 2005
`
`yy = I: xsum[i] (infoStick
`t=O
`xx=-� xpsum [i]/TH
`1= 0
`xpsum [ i] = [ j · P ( j, infostick [n] ystart + i }
`
`
`j = i n f o stick [ uJ · x start [ i J
`
`
`
`infostick [uJ xend [i J
`
`F I G.
`6 0 8
`
`IPR2021-00922
`Apple EX1005 Page 43
`
`

`

`U.S. Patent
`5,900,863
`
`May 4, 1999 Sheet 43 of 46
`
`X
`
`Yo 01234567a9mt1
`l � t--+-+--+---+--+-----+.--.-+-+----<f----+-t-----1
`2i01
`5 t---t-----t---t-----t----1'<-',
`7 t---t-----t---t-----t---t'<-",
`6��1------M
`� t---t-----t---t-----h-T
`
`F I G.
`
`2102�
`
`
`infostick [nJ. Ystart
`LnJ . xstart [iJ
`infostick
`
`infostick(nJ .xend [i]
`
`infostick [nJ. length
`
`infostick [nJ. status
`
`STICK:
`NOST I CK:
`EXTENDED:
`
`F I G.
`
`2103
`
`'------
`
`AFTER LARGE -VALUE PIXELS FOUND IN LI NE
`
`( y = 3) HAVE BEEN REG I STEREO AS A NEW
`
`CANDIDATE FOR A RODSHAPED OBJECT
`
`infostick [OJ. ystart = 3
`
`
`infostick [OJ. xstart[OJ =B
`
`infoSti ck[O]. xend [OJ = B
`length = 1
`infostick[OJ.
`
`inf ostick!OI. status = EXTENDED
`
`F I G.
`6iC
`
`IPR2021-00922
`Apple EX1005 Page 44
`
`

`

`46 5,900,863
`U.S. Patent
`May 4, 1999 Sheet 44 of
`
`
`
`AFTER SCANNING HAS PROCEEDED TO LINE
`
`
`{y=4)
`infostick [OJ. ystart = 3
`ck [OJ. xstart[OJ = a
`infoSti
`= 5
`inf ost i ck lOJ. xstarHn
`inf ostic k [OJ. xend [OJ= a
`inf ostick [OJ . xend [iJ = 1
`
`infostick [OJ. length =2
`[O]. status= EXTENDED
`inf ostick
`
`F I G.
`
`2l05�
`
`AFTER CONNECTING LARGE-VALUE PIXELS
`AT THE MIDDLE PART OF LINE (y =5) TO
`A CANDIDATE FOR A RODSHAPED OBJECT
`
`infoStickLOL ystart = 3
`
`
`i nfost i ck [OJ .xstart[OJ= a
`
`inf ost ick [OJ .xstart [1 J= 5
`[21 = 5
`info stick [OJ. xstart
`infostickLOJ .xend [OJ= a
`infostick[OJ .xend [1 l = 7
`infostick[Ol .xend [2J = 7
`
`
`infostick[OJ .length =3
`
`infost i ck[O] .status = EXTENDED
`
`F I G.
`6iE
`x- -0123456789t0ii
`y 0
`230 i
`
`I i :::=======:=::-=:-��--l-
`
`5 1---1---��
`6 1--+--+-+----+--,!Y-.'
`7 �-+-+-+--�
`2302 � � �-+-+--h-<-
`
`6 3 H L--1...._
`F I G.
`
`m -----+.->-.>,
`�
`L......J.__
`
`IPR2021-00922
`Apple EX1005 Page 45
`
`

`

`U.S. Patent
`May 4, 1999
`
`of 46 5,900,863 Sheet 45
`
`
`
`220i
`
`X---
`
`0123456789mi1
`
`y O 1----1--i--+---�l--------l---l---+---,f---+---l
`d f----+--+----+""+-l---4-..+-t-+-+-t--i
`� l----+--+---+---1--------W 7 l-+--1--+--�
`H �'---"--L>...:
`
`8 t--+-t--t-k-<-
`2202
`9
`iO t--t-t--+-�
`
`F I G. 6 2 A
`
`AFTER LINE ( y=2l HAS BEEN
`SCANNED
`
`infostick [OJ. ystart =2
`
`infost ick [OJ. xstart [OJ = 3
`
`ick[0J. xend [OJ = 3
`2203�
`infost
`= i
`
`infostick[0J. length
`[0J .status= EXTENDED
`infostick
`G.6 2 8
`Fl
`AFTER LI NE ( y = 3 ) HAS BEEN
`SCANNED
`
`infostick [OJ. ystart = 2
`
`
`infostick [OJ. xstart [0]=3
`2204�
`
`infoStick[0J .xend [OJ =3
`. length = i
`infostick[OJ
`
`= NOST I CK
`
`infostick[0] .status
`
`G.62C
`Fl
`
`infoStick [iJ. ystort = 3
`
`infostick [0. xstort [OJ.= 6
`2205�
`
`infos ti ck [iJ. xend [OJ= 6
`
`infostickrn .length = i
`
`i nfostick[O. status = EXTENDED
`
`G.62 D
`Fl
`
`IPR2021-00922
`Apple EX1005 Page 46
`
`

`

`U.S. Patent May 4, 1999
`
`Sheet 46 of 46 5,900,863
`
`2404 fr
`2403
`
`240�
`
`I
`
`F I G.
`64
`
`IPR2021-00922
`Apple EX1005 Page 47
`
`

`

`
`
`METHOD AND APPARATUS FOR
`
`
`
`CONTROLLING COMPUTER WITHOUT
`
`
`
`TOUCHING INPUT DEVICE
`
`1
`
`5,900,863
`
`2
`Pointing devices, such as mouses, have hitherto been
`
`
`
`
`
`
`
`
`
`commonly used to control personal computers and the like.
`
`
`
`
`A pointing device is a data input device which is operated to
`
`
`input position data to a computer. Various types of pointing
`BACKGROUND OF THE INV ENTION
`
`
`
`5 devices are available. The most popular of the pointing
`1. Field of the Invention
`
`
`
`
`device is the mouse. The mouse comprises a housing and a
`
`
`
`
`ball contained in the housing. The ball can rotate in any
`
`
`
`
`The present invention relates to a device designed to input
`
`
`
`direction with respect to the housing.
`
`
`
`data for controlling or remote-controlling, for example, an
`electronic apparatus.
`
`
`
`In use, the mouse is electrically connected to the computer
`
`2. Description of the Related Art 10 and placed on a desk. Once the mouse is put on the desk, the
`
`
`
`
`ball contacts the desk top at one point. As the mouse is
`
`
`
`Most of the data input devices (hereinafter called "control
`
`
`
`moved back, forth, leftward or rightward on the desk, the
`devices") hitherto developed to input control data are hand­
`
`
`
`
`
`
`
`ball rotates with respect to the mouse housing. The angles
`
`held ones. A user operates a control device of this type, while
`
`
`
`
`through which the ball rotate in two directions are detected,
`
`holding it in hand. Control devices most commonly used in
`
`
`
`
`households are remote controllers. A remote controller has 15
`
`
`
`
`either mechanically or optically. The data representing the
`
`
`
`
`
`
`
`angles detected is supplied to the computer In the computer,
`
`
`several buttons on it. The user holds it in a hand, aims it at
`
`
`
`this data is processed into a direction and a distance, for
`
`
`
`an electronic apparatus and pushes one of the buttons. When
`
`
`which the cursor is moved on the display screen.
`
`the button is pushed, the remote controller emits an infrared
`
`
`
`beam to the apparatus, thus giving instructions to the appa­
`
`
`
`
`Pointing devices other than the mouse have been put to
`
`
`
`
`20 use for inputting data to computers. Among them are a touch
`ratus.
`
`
`
`
`
`
`
`pad, a track ball, a small Among the control devices hitherto developed, other than joystick, a touch panel and a pen
`
`
`the hand-held panel. device. ones, is a so-called touch The touch
`
`
`panel comprises a transparent or opaque panel. When a user
`
`
`
`The mouse is easy to operate to input data to the computer.
`
`
`pushes any part of the panel with a finger or an elongated
`
`
`
`However, the mouse can hardly be used unless there is a flat
`
`
`object, the touch panel inputs a data item representing the 25
`
`
`
`wide surface such as the top of a desk. To operate his
`
`
`
`
`pushing of that part or the position of that part. A transparent
`
`
`
`computer outdoors, by using the mouse, the user needs to
`
`panel may be mounted on a display screen displaying
`
`
`
`
`find something having a flat wide surface. In a train or a car,
`
`
`
`images of buttons. In this case, the user may push that part
`
`
`it would be impossible for him to use the mouse to input data
`
`
`
`of the transparent panel located above one of the buttons
`
`
`
`to the computer placed on his laps. Even if a desk is
`30
`
`
`displayed, thereby to input the data item indicating that the
`
`
`
`provided, the mouse cannot be used if the desktop is covered
`
`button has been selected.
`
`
`with books, folders or documents. To use the mouse, the
`
`
`In the field of image recognition, research has been done
`
`desktop has to be cleared.
`
`
`
`
`for a method of recognizing hand signaling and gestures. No
`Portable computers, each having a pointing device other
`
`
`
`
`
`
`practically usable methods have been developed, however.
`
`
`
`than the mouse, are now used in increasing numbers. The
`35
`
`
`
`This is because various requirements are imposed, which are
`
`
`
`
`pointing device on the portable computer is a track ball, a
`
`
`
`
`hard to fulfill in practice. More specifically, it is required that
`
`
`
`touch pad, a small joystick or the like. The track ball is
`
`
`the background on which to display hands or a person be
`
`
`
`
`rotated with a finger. The direction and distance the track
`
`
`monotonous so that image data may be processed more
`
`
`
`
`ball is rotated are detected, and the data representing them
`
`
`
`
`easily. It is also required that the hands or the person be
`
`
`
`is input to the computer. The touch pad, available in various
`40
`
`
`
`displayed in a relatively large size at the central part of a
`
`
`
`
`types, can detect the position of any part touched. When
`
`display screen.
`
`
`
`
`
`stroked with the user's finger, the touch pad generates data
`
`
`
`Also, research has been conducted to facilitate image
`
`
`
`
`representing the direction and distance it is thus stroked.
`
`
`recognition. More precisely, an attempt has been made to
`
`
`
`
`
`A small joystick is shaped like a lever. It is provided on
`
`
`
`develop a method in which stereoscopic analysis is per-
`
`
`
`
`a keyboard. In most cases, it protrudes upwards, from among
`45
`
`
`formed on an object, based on trigonometrical survey, in
`
`
`"G", "H" and "B" keys. When the joystick is rotated, the
`
`
`
`
`order to recognize the three-dimensional shape of the object.
`
`
`
`
`direction and angle by which it is rotated are detected. The
`
`
`Nonetheless, only a few cases are known in which image
`
`
`
`direction and the angles are converted into two data items
`
`
`
`recognition techniques are applied to remote controllers.
`
`
`
`
`which represent the direction and distance for which the
`
`
`
`
`A user cannot operate a remote controller unless he or she
`
`
`
`50 cursor is to move on the display screen. The data items are
`
`
`holds it in a hand. It is undesirable that the remote controller
`input to the computer.
`
`is held in a hand which has gotten dirty.
`
`
`Pointing accomplished by using a mouse, a track ball, a
`
`
`
`
`To operate a touch panel, the user need not hold anything
`
`
`
`touch pa