I 1111111111111111 11111 lllll lllll lllll lllll 111111111111111 1111111111 11111111
`
`US006191773Bl
`
`
`(12)United States Patent
`Maruno et al.
`
`(10)Patent No.:US 6,191,773 Bl
`
`*Feb.20,2001
`(45)Date of Patent:
`
`(54)INTERFACE APPARATUS
`
`(56)
`
`
`
`References Cited
`
`Susumu Maruno, Osaka; Taro
`(75)Inventors:
`
`
`
`Imagawa, Harakata; Michiyo Moriya,
`5,594,469 * 1/1997 Freeman et al. ..................... 345/158
`
`
`
`
`Moriguchi; Kenji Kondo, Neyagawa,
`
`
`5,767,842 * 6/1998 Korth ................................... 345/173
`all of (JP)
`
`
`
`5,864,334 * 1/1999 Sellers .................................. 345/158
`
`U.S. PATENT DOCUMENTS
`
`
`
`
`Matsushita Electric Industrial Co.,
`(73)Assignee:
`Ltd., Osaka (JP)
`
`FOREIGN PATENT DOCUMENTS
`
`
`
`8-16781 * 1/1996 (JP) .
`
`pros­( *) Notice: This patent issued on a continued
`
`
`OTHER PUBLICATIONS
`
`
`
`ecution application filed under 37 CFR
`
`
`
`1.53( d), and is subject to the twenty year
`"Wideo-Based Handshape Recognition Using a Handshape
`
`
`
`
`patent term provisions of 35 U.S.C.
`
`
`Structure Model in Real Time" Grabel et al., 1996 IEEE, pp.
`154(a)(2).
`
`446-450, 1996.*
`
`Under 35 U.S.C. 154(b), the term of this
`
`
`
`* cited by examiner
`
`
`patent shall be extended for O days.
`
`(21)Appl. No.:
`
`08/945,628
`
`Apr. 25, 1996
`(22) PCT Filed:
`
`(86)PCT No.:
`
`
`
`PCT/JP96/01124
`
`Primary Examiner-Dennis-Doon Chow
`
`
`
`
`
`(74)Attorney, Agent, or Firm-Ratner & Prestia
`
`(57)
`
`ABSTRACT
`
`
`
`
`
`Jan. 2, §371 Date: 1998
`
`
`
`§ 102(e) Date: Jan. 2, 1998
`
`(87)PCT Pub. No.: WO96/34332
`
`PCT Pub. Date: Oct. 31, 1996
`
`The invention relates to an interface apparatus for making
`
`
`
`
`
`
`
`input and output of appliances having display such as
`
`
`
`
`computer, word processor, information appliance and
`
`
`
`
`television, comprising recognizing means for recognizing
`
`the shape or move of the hand of an operator, display means
`
`
`
`for displaying the features of the shape or move of the hand
`
`
`
`recognized by the recognizing means as special shape in the
`
`
`
`screen, and control means for controlling the information
`
`
`displayed in the screen by the special shape displayed in the
`
`
`Apr. 28, 1995 (JP) ................................................... 7-105551
`
`
`
`screen by the display means, wherein the two-dimensional
`
`
`Oct. 19, 1995 (JP) ................................................... 7-270828
`
`
`or three-dimensional information displayed in the screen can
`
`
`Nov. 1, 1995 (JP) ................................................... 7-284768
`
`
`
`be selected, indicated or moved only by changing the shape
`
`(51)Int. Cl.7 ....................................................... G09G 5/08
`
`
`or moving the hand, so that the interface apparatus of very
`
`
`
`excellent controllability and high diversity may be pre­
`
`............................................. 345/158; 345/156 (52) U.S. Cl.
`
`
`sented.
`(58)Field of Search
`
`
`.................................. 66/104, 55, 13,
`
`
`
`66/15, 120; 358/403; 379/93.14, 100.08,
`
`
`
`88.13, 88.24; 345/427, 156, 157, 158, 173
`
`
`
`
`
`(30) Foreign Application Priority Data
`
`
`
`
`
`6 Claims, 26 Drawing Sheets
`
`200
`
`3
`
`2
`
`IPR2021-00922
`Apple EX1008 Page 1
`
`

`

`
`U.S. Patent Feb.20,2001
`
`Sheet 1 of 26
`
`US 6,191,773 Bl
`
`200
`
`3
`
`1/ Television
`
`2 Network
`
`a.---2
`
`1
`
`FIG. 1
`
`IPR2021-00922
`Apple EX1008 Page 2
`
`

`

`US 6,191,773
`Bl
`U.S. Patent
`Feb.20,2001
`Sheet 2 of
`26
`
`21
`
`25
`
`2
`
`Frame memory Reference
`image
`memory
`3 '------.-
`22
`Shape identifying
`means
`
`L---r-----'-----'
`
`--'
`
`Display
`control
`unit 23
`
`Icon generating
`unit
`
`24
`
`FIG. 2
`
`FIG. 3(A) FIG. 3(8)
`
`FIG. 3(C)
`
`IPR2021-00922
`Apple EX1008 Page 3
`
`

`

`U.S. Patent Feb.20,2001
`
`
`Sheet 3 of 26
`
`US 6,191,773 Bl
`
`Image difference 221
`
`calculation unit
`
`Contour depicting 222
`
`unit
`
`223
`
`Shape identifying
`unit
`
`22
`
`FIG. 4
`
`
`
`Reference image
`
`Result of operation
`
`Taken image FIG. 5
`
`IPR2021-00922
`Apple EX1008 Page 4
`
`

`

`U.S. Patent Feb. 20, 2001
`Sheet 4 of 26
`
`
`
`US 6,191,773 Bl
`
`FIG. 6(A) FIG. 6(8)
`
`. .
`
`. . .
`
`. .
`
`.
`.
`.
`
`
`
`.. .. . . -.. . .
`
`• •. • •
`• I • • ..
`
`.
`.
`
`.
`
`FIG. 6(D)
`FIG. 6(C)
`
`IPR2021-00922
`Apple EX1008 Page 5
`
`

`

`
`U.S. Patent Feb.20,2001
`
`Sheet 5 of 26
`
`US 6,191,773 Bl
`
`200
`
`2 Network
`
`FIG. 7(A)
`
`200
`
`1/ Television
`
`2 Network
`
`
`
`____________ 1
`
`1
`
`FIG. 7(8)
`
`IPR2021-00922
`Apple EX1008 Page 6
`
`

`

`U.S. Patent Feb. 20, 2001
`Sheet 6 of 26
`
`
`
`US 6,191,773 Bl
`
`3
`
`.._ __ 2
`
`FIG. 7(C)
`
`IPR2021-00922
`Apple EX1008 Page 7
`
`

`

`U.S. Patent Feb. 20, 2001
`Sheet 7 of 26
`
`
`
`US 6,191,773 Bl
`
`3
`
`1.---2
`
`
`
`I1
`
`_k;. / 203
`���
`204 �
`
`«:::
`
`205 206
`
`FIG. 8
`
`IPR2021-00922
`Apple EX1008 Page 8
`
`

`

`I■-
`�
`
`-...,l
`�
`'° I■-
`
`I■-
`_,.a-...
`e rJ'J.
`
`O'I
`N
`0 ....,
`
`00
`
`
`
`=­�
`
`'JJ.
`
`0 '"""'
`
`0
`
`�= N
`
`N
`
`FIG. 9
`
`232
`
`inverting
`Coordinate
`
`unit
`
`unit Icon generating
`
`unit
`
`transforming
`
`24
`
`Coordinate
`
`231
`
`unit
`
`23
`
`control
`Display
`
`Motion recognizing
`
`22
`
`means
`
`ce image
`
`Frame memory Referen
`
`memory
`
`25
`
` ......
`
`...... =
`
`�
`�
`r:JJ. •
`d •
`
`___
`
`262
`
`L..: _____ _j
`I unit 1
`I Image updating
`
`I
`
`_
`
`__._
`
`,---
`
`I ',, 26
`l,
`
`I
`
`Timer
`,---
`--7
`
`261
`
`3
`
`21
`
`2
`
`IPR2021-00922
`Apple EX1008 Page 9
`
`

`

`U.S. Patent Feb.20,2001
`Sheet 9 of 26
`
`
`
`US 6,191,773 Bl
`
`FIG. 10(A) FIG. 10(8)
`
`22
`
`Image difference 221
`
`calculation unit
`
`222
`Contour depicting
`
`unit
`
`Shape change 225
`
`identifying unit
`
`224
`Position detector
`
`
`
`FIG. 11
`
`IPR2021-00922
`Apple EX1008 Page 10
`
`

`

`U.S. Patent Feb.20,2001
`Sheet 10 of 26
`
`
`
`US 6,191,773 Bl
`
`
`
`Reference image
`
`Result of operation
`
`Taken image FIG. 12
`
`IPR2021-00922
`Apple EX1008 Page 11
`
`

`

`U.S. Patent Feb. 20, 2001
`Sheet 11 of 26
`
`
`
`US 6,191,773 Bl
`
`FIG. 13(A) FIG. 13(8)
`
`FIG. 13(C) FIG. 13(D)
`
`IPR2021-00922
`Apple EX1008 Page 12
`
`

`

`
`U.S. Patent Feb.20,2001
`
`
`US 6,191,773 Bl
`
`Sheet 12 of 26
`
`2
`
`3
`
`31
`Frame memory
`
`32
`Motion recognizing
`
`means
`
`Display 33
`control
`unit 331
`Coordinate
`
`
`
`unit transforming unit Icon generating
`
`34
`
`Coordinate
`
`inverting unit
`332
`
`FIG. 14
`
`
`
`-
`
`-- -- - -- --- -- --
`-
`- - -
`
`- - - -- - - -
`
`--,
`
`
`
`
`
`Contour depicting unit 321
`
`r -
`
`--
`
`I
`
`324
`322 Contour waveform
`
`Coordinate table
`
`operation unit
`
`3232
`
`3231
`1 Shape filter 2 Shape filter
`
`
`(hand undulations)
`
`(finger undulations)
`
`3252
`
`3251
`Differentiating Differentiating
`device device
`3261
`�2@2
`Coordinate
`
`Shape judging unit
`
`operation unit
`
`- - J
`
`--
`
`L -- --- ----
`
`----
`
`....... --
`
`Hand shape output Hand coordinate output
`
`FIG. 15
`
`IPR2021-00922
`Apple EX1008 Page 13
`
`

`

`Sheet 13 of 26 US 6,191,773
`Bl
`U.S. Patent Feb. 20, 2001
`
`FIG. 16(A) FIG. 16(8)
`
`. ,
`
`.
`. ..
`• I •
`•
`-
`
`. . . . .
`....
`
`. .
`. ,, .
`-
`. . .. ..
`
`.
`
`.
`
`. .
`..
`. . .
`
`' ,.
`
`-.. '
`
`FIG. 16(C)
`
`FIG. 16(D)
`
`IPR2021-00922
`Apple EX1008 Page 14
`
`

`

`U.S. Patent Feb.20,2001
`Sheet 14 of 26
`
`
`
`US 6,191,773 Bl
`
`(A1)
`
`Start point s
`
`(81)
`
`e
`
`(812)
`
`(811)
`
`(8122)
`
`(8112)
`
`1c1 1c2
`
`FIG. 17
`
`IPR2021-00922
`Apple EX1008 Page 15
`
`

`

`
`
`U.S. Patent Feb.20,2001
`
`US 6,191,773 Bl
`
`Sheet 15 of 26
`
`(A2)
`
`(82)
`
`(822)
`
`(821)
`
`(B222)
`
`(8212)
`
`FIG. 18
`
`IPR2021-00922
`Apple EX1008 Page 16
`
`

`

`U.S. Patent Feb.20,2001
`Sheet 16 of 26
`
`US 6,191,773
`Bl
`
`Start point s
`
`FIG. 19
`
`C2
`
`Start point s
`
`FIG. 20
`
`IPR2021-00922
`Apple EX1008 Page 17
`
`

`

`U.S. Patent Feb.20,2001
`Sheet 17 of 26
`
`US 6,191,773
`Bl
`
`A4
`
`A3
`
`Display device 14----
`
`-
`
`--1 Object coordinate
`memory unit
`
`AS
`
`A2
`
`Contact
`judging
`unit
`
`Cursor
`coordinate
`memory unit
`
`A1
`
`Input device
`
`FIG. 21
`
`IPR2021-00922
`Apple EX1008 Page 18
`
`

`

`
`U.S. Patent Feb. 20, 2001
`
`Sheet 18 of 26 US 6,191,773 Bl
`
`FIG. 22(A) FIG. 22(8)
`
`FIG. 22(C) FIG. 22(0)
`
`FIG. 22(E) FIG. 22(F)
`
`IPR2021-00922
`Apple EX1008 Page 19
`
`

`

`U.S. Patent Feb. 20, 2001 Sheet 19 of 26
`
`Cube
`
`VS 6,191,773 Bl
`
`FIG. 23(A)
`
`Plane
`
`FIG. 23(B)
`
`IPR2021-00922
`Apple EX1008 Page 20
`
`

`

`U.S. Patent Feb.20,2001
`Sheet 20 of 26
`
`
`
`US 6,191,773 Bl
`
`
`
`Example of configuration on virtual space (front view)
`
`
`
`Top
`
`Cursor
`
`Left
`
`Right
`
`
`
`°"' Virtual object
`
`Floor
`
`Bottom
`
`FIG. 24(A)
`
`
`
`Example of configuration on virtual space (side view)
`
`
`
`Top
`
`Rear �Virtual object
`
`Cursor
`
`Front
`
`Bottom
`
`FIG. 24(8)
`
`IPR2021-00922
`Apple EX1008 Page 21
`
`

`

`U.S. Patent Feb.20,2001
`Sheet 21 of 26
`
`
`
`US 6,191,773 Bl
`
`
`
`Display example
`
`
`
`Virtual object
`
`Cursor
`
`FIG. 25
`
`IPR2021-00922
`Apple EX1008 Page 22
`
`

`

`I■-
`�
`
`�
`'° I■-
`I■-
`_,.a-...
`Learn ing means
`
`rJ'J.
`e
`
`I
`45f
`
`0 ....,
`judging
`� ....
`
`means
`Overall
`
`I
`
`45c
`
`O'I
`N
`
`N
`N
`
`
`
`'JJ.=­�
`
`0 '"""'
`0
`�= N
`
`N
`
`�
`
` ......
`
` =
`
`�
`�
`
`......�
`
`r:JJ. •
`d •
`
`� calculating
`
`-
`
`transforming
`Coordinate Distance
`
`/45a
`
`45g,l
`
`means means
`
`Move vector
`
`-
`
`means
`
`-recog nizing
`
`-
`
`-
`
`FIG. 26
`
`i---450
`
`means
`
`Shape judging
`
`-
`
`'
`
`means 45d
`
`calculating
`
`Motion ,-.-45b
`
`\ memory
`Object
`
`means
`
`/
`
`43
`
`Displ ay means
`
`means
`memory
`Cursor
`
`/
`42
`
`I
`
`\.
`
`input means
`
`Sight line
`
`46
`
`/
`
`judging
`means
`
`45 Interaction
`
`Input means
`
`41
`
`IPR2021-00922
`Apple EX1008 Page 23
`
`

`

`
`U.S. Patent Feb. 20, 2001
`Sheet 23 of 26
`
`US 6,191,773 Bl
`
`Buttons
`
`FIG. 27(A)
`
`FIG. 27(8)
`
`Operation buttons
`
`FIG. 27(C)
`
`IPR2021-00922
`Apple EX1008 Page 24
`
`

`

`
`U.S. Patent Feb.20,2001
`
`
`US 6,191,773 Bl
`
`Sheet 24 of 26
`
`. . ..
`.
`. ..
`.
`
`• :-:-·.:·.::: !ti
`
`·. ·.·i•:-:-:-:::�
`--• •,::�v, ..,-,.,..,_M
`• • •• •• •: •::
`If')(:::-.·:::.-.:-.- ....
`
`FIG. 28(A)
`
`
`
`Circumscribing rectangle
`
`/
`/
`/
`
`/
`
`FIG. 28(8)
`
`IPR2021-00922
`Apple EX1008 Page 25
`
`

`

`U.S. Patent Feb. 20, 2001
`
`Sheet 25 of 26
`
`US 6,191,773 Bl
`
`El
`
`FIG. 29(A)
`
`��
`cr�-□
`I , \
`�
`[ r } \
`I I \ \ FIG. 29(8)
`I l \ \
`
`Notice point .,........ El El
`
` \
`l
`I r \ \
`r l
`\ "\
`I l � \ FIG. 29(C)
`
`Ii
`
`Cube
`
`Plane
`
`FIG. 29(D)
`
`IPR2021-00922
`Apple EX1008 Page 26
`
`

`

`U.S. Patent Feb.20,2001
`Sheet 26 of 26
`
`
`
`US 6,191,773 Bl
`
`506
`
`502
`
`501
`
`508
`507
`
`503 504 505
`
`FIG. 30
`
`PRIOR ART
`
`IPR2021-00922
`Apple EX1008 Page 27
`
`

`

`US 6,191,773
`Bl
`
`1
`
`APPARATUS
`INTERFACE
`
`2
`line and the length of
`line,
`contour
`contour
`the angle of the
`and filtered,
`and
`waveform is calculated
`that is, the
`contour
`shape is gen­
`the specified
`the shape waveform expressing
`Phase Application
`of
`is a U.S. National
`This Application
`erated.
`PCT/JP96/01124.
`Application
`PCT International
`5
`means for display­
`display
`cursor
`comprising
`Moreover,
`BACKGROUND OF THE INVENTION
`as a special
`screen
`shape of a hand on the
`ing a feature of the
`means for storing
`the
`shape and manipulating as
`cursor,
`The present invention relates
`apparatus
`for
`to an interface
`displays the
`other than cursor
`object
`with display
`relation
`such as computer
`apparatus
`of information
`input and output
`point represent-
`representative
`and shape of the
`coordinates
`such as
`display
`having a
`and appliance
`and word processor
`10 ing the position
`other than cursor
`of the display
`object
`television.
`and judging the interac­
`and means for calculating
`display,
`it is
`apparatus,
`interface
`In a kind of conventional
`and the display object, manipula­
`display
`tion of the
`cursor
`to display a cursor at a coordinate position
`detected
`designed
`along the intent
`by the interactions
`realized smoothly
`tion is
`for adding some other
`screen,
`by the mouse on a display
`when gripping the displayed virtual
`object in
`operator
`of the
`device,
`or
`in the display
`to the information
`information
`15 the case of display
`manipulator.
`virtual
`display as
`of cursor
`or selecting the displayed
`information.
`changing
`user
`thus constructed,
`as the
`In the interface
`apparatus
`interface
`of this conventional
`FIG. 30 shows an outline
`a hand,
`for example,
`means and shows,
`faces the recognizing
`a host
`501 denotes
`numeral
`30, reference
`In FIG.
`apparatus.
`hand is
`shape of the
`shape corresponding
`to the
`the special
`and virtual operation buttons
`and 502 is a display,
`computer,
`manipulation,
`for screen
`screen
`displayed as an
`icon in the
`are displayed in the display
`host
`502 by the
`503, 504, 505
`20
`is enabled.
`display
`to the icon
`so that control
`according
`a mouse
`506 represents
`numeral
`501. Reference
`computer
`the given hand
`Or when instructed by
`hand gesture,
`the display so as
`501 controls
`host computer
`and the
`cursor,
`corresponding
`to
`shape set
`as a special
`is displayed
`gesture
`with the move of the
`in synchronism
`to move in the
`screen
`and its move is
`the shape of the hand on the display
`screen,
`of the mouse
`basis of the moving distance
`mouse 507, on the
`a virtual switch
`or the like
`and, for example,
`also displayed,
`by the mouse 507. As the user moves the mouse 25
`507 detected
`by the hand
`can be'selected
`screen
`on the display
`displayed
`506 is moved to the position of a
`507, the mouse cursor
`can be
`screen
`on the
`displayed
`or the display
`object
`gesture,
`screen,
`and by
`button in the
`display
`operation
`virtual
`desired
`and therefore
`on the purpose,
`depending
`or carried
`grabbed
`button
`mouse 507, an operation
`508 on the
`a switch
`pressing
`simple
`a very
`mouse or other input
`device,
`without requiring
`so as to instruct action to the
`host computer
`is selected
`501.
`is realized.
`of appliance
`30 manipulation
`the mouse or
`however,
`construction,
`In this conventional
`the interface
`much
`to realize
`possible
`It is further
`to the main body of
`in addition
`is necessary
`the input device
`by sequentially
`and
`ease of manipulation
`in the
`enhanced
`and a table or area for manipulating the
`mouse
`the appliance,
`the interaction with the
`display object
`automatically
`judging
`information
`to portable
`which is not
`suited
`is also needed,
`by the virtual manipulator according
`to be operated
`desired
`by manipulation through the
`Besides,
`or the like.
`appliance
`shape
`as the special
`of operation of the
`operator,
`35 to the intent
`interface.
`and intuitive
`mouse, it is not a
`direct
`virtual
`hand works as
`to the shape of the
`set corresponding
`cursor.
`aside from the mere
`manipulator
`SUMMARY OF THE INVENTION
`
`OF THE DRAWINGS
`BRIEF DESCRIPTION
`an interface
`to present
`invention
`of the
`It is an object
`easily with-
`an appliance
`manipulating
`capable of
`apparatus
`40
`FIG. 1
`apparatus
`of an interface
`is an appearance drawing
`and mouse. It is
`such as keyboard
`out requiring input
`device
`of the invention;
`in a first embodiment
`further
`apparatus
`an interface
`present
`thereof to
`other object
`the interface
`block diagram of
`FIG. 2 is a detailed
`ease of manipulation of indicating or
`in the
`advanced
`of the invention;
`same embodiment
`in the
`apparatus
`by judging interactions
`along the
`object
`the display
`catching
`shape of hand
`an example of
`showing
`45 FIG. 3 is a diagram
`and automatically.
`sequentially
`of the operator
`intent
`same embodiment
`of
`in the
`apparatus
`by the interface
`judged
`apparatus
`In structure, the invention
`an interface
`provides
`the invention;
`means for recognizing the
`shape of
`comprising recognizing
`FIG. 4
`of shape iden­
`an example
`is a diagram showing
`means for displaying the
`display
`a hand of an operator,
`in the same
`embodi-
`means of the interface
`apparatus
`tifying
`by the recog­
`shape of the hand recognized
`features of the
`invention;
`50 ment of the
`and control
`shape,
`as a special
`screen
`means on the
`nizing
`displayed
`in the
`the information
`means for controlling
`of operation by
`an example
`FIG. 5 is a diagram
`showing
`by the
`in the screen
`shape displayed
`by the special
`screen
`unit in the same embodiment;
`difference operation
`an image
`displayed in the
`information
`means, whereby the
`display
`generated
`of icon
`an example
`is a diagram showing
`FIG. 6
`the shape of the
`only by varying
`can be controlled
`screen
`by an icon generating unit in the same
`embodiment;
`hand.
`55
`an operation
`showing
`drawing
`FIG. 7 is an appearance
`apparatus
`an interface
`to present
`It is a further
`object
`same embodiment;
`of the
`of the interface
`apparatus
`example
`by recognizing also
`in ease of manipulation
`much superior
`apparatus
`of an interface
`drawing
`FIG. 8 is an appearance
`move, a frame
`the hand. To recognize the
`the move of
`invention;
`of the
`in a second embodiment
`up the shape or move
`the image picking
`memory for saving
`block diagram of the interface
`apparatus
`detailed
`FIG. 9 is
`image memory for storing the
`hand, and a reference
`of the
`60
`of the invention;
`second embodiment
`in the
`saved in the frame memory as
`image taken before the image
`of shape of
`an example
`showing
`FIG. 10 is a diagram
`by depicting
`and it is achieved
`reference image are
`provided,
`same embodi-
`of the
`apparatus
`by the interface
`hand judged
`the image in the frame
`memory and
`between
`the difference
`ment;
`the reference image stored in the
`image memory.
`reference
`FIG. 11
`is a diagram showing
`of motion
`an example
`shape or move of the
`In other method of recognition, the
`65
`of the same
`the interface
`apparatus
`unit of
`recognizing
`contour
`as the
`taken image is depicted
`user in the
`hand of the
`embodiment;
`and the relation
`between
`is traced,
`its contour
`user, and
`of the
`
`IPR2021-00922
`Apple EX1008 Page 28
`
`

`

`
`
`(cl)is a diagram showing a fourth example of the display
`
`DESCRIPTION OF THE PREFERRED
`
`EMBODIMENTS
`
`
`
`US 6,191,773 Bl
`
`(c) is a diagram showing a third example of the display
`
`4
`3
`FIG. 12 is a diagram showing an example of operation by
`
`
`
`FIG. 28(a) is a diagram showing an example of image of
`
`
`
`
`
`
`an image difference operation unit in the same embodiment;
`
`a hand taken by a camera in the same embodiment;
`
`
`
`
`FIG. 13 is a diagram showing an operation example of the
`
`
`
`
`
`(b)is a diagram showing a binary example of image of a
`same embodiment;
`hand taken by a camera in the same embodiment;
`
`
`
`FIG. 29(a) is a diagram showing an example of image
`5
`
`
`FIG. 14 is a detailed block diagram of an interface
`
`
`
`displayed by display means used in the interface apparatus
`
`
`apparatus in a third embodiment of the invention;
`
`
`in the same embodiment of the invention;
`
`
`
`FIG. 15 is a diagram showing an example of motion
`
`
`
`
`(b) is a diagram showing a second example of the display
`
`
`recognizing unit of the interface apparatus in the third
`screen;
`
`embodiment of the invention;
`10
`
`
`
`
`
`
`
`
`FIG. 16(A) to (D) are diagrams showing examples of icon
`screen;
`
`
`
`
`
`displayed on a display screen by the interface apparatus of
`the same embodiment;
`
`
`
`
`screen; and
`
`
`
`FIG. 17 is a diagram showing operation of motion rec­
`FIG. 30 is an explanatory diagram for explaining a
`
`
`
`
`
`ognizing unit of the interface apparatus in the same embodi-15
`
`
`conventional interface apparatus.
`
`ment of the invention;
`
`
`
`FIG. 18 is a diagram showing operation of motion rec­
`
`
`
`
`
`ognizing unit of the interface apparatus in the same embodi­
`
`ment of the invention;
`FIG. 19 is a diagram showing operation of motion rec-20
`
`
`(First embodiment)
`
`
`
`
`
`ognizing unit of the interface apparatus in the same embodi­
`
`
`
`A first embodiment of the invention relates to an interface
`
`ment of the invention;
`
`
`
`comprising recognizing means such as image pickup device
`
`
`for recognizing the shape of a hand of the operator, display
`
`
`
`FIG. 20 is a diagram showing operation of motion rec­
`
`
`means for displaying the feature of the shape of the hand
`
`
`
`
`
`ognizing unit of the interface apparatus in the same embodi-
`
`
`
`recognized by the recognizing means on a screen as a special
`25
`
`ment of the invention;
`
`
`shape by an icon or the like, and control means for control­
`
`FIG. 21 is a diagram showing an interface apparatus
`
`
`
`
`
`ling the information displayed on the screen by varying the
`
`explaining a fourth embodiment;
`
`
`
`
`shape of the hand by operating the special shape such as icon
`
`
`
`FIG. 22(a) is a diagram showing an open state of cursor
`
`
`
`displayed on the screen by the display means as the so-called
`
`
`
`
`in an example of a cursor used in the interface apparatus of
`30 cursor.
`the same embodiment;
`FIG. 1 shows the appearance of the first embodiment of
`
`
`
`(b)is a diagram showing a closed state of the same
`
`the interface apparatus of the invention. Reference numeral
`embodiment;
`
`
`
`1 denotes a host computer, 2 is a display unit, and 3 is a CCD
`(c) is a diagram showing an open state of cursor in an
`
`
`
`
`
`
`camera for picking up an image. The CCD camera 3 has the
`
`
`
`
`example of a cursor used in the interface apparatus of 35
`
`
`
`
`pickup surface located in the same direction as the display
`the same embodiment;
`
`
`direction, so that the shape of the hand of the user can be
`
`
`(cl)is a diagram showing a closed state of the same
`
`
`
`
`picked up when the user confronts the display screen. On the
`embodiment;
`
`
`
`display, menu 201, 202, and icon 200 reflecting the shape of
`(e) is a diagram showing an open state of cursor in an
`
`
`
`
`
`the hand are displayed.
`
`
`example of a cursor used in the interface apparatus of 40
`
`
`FIG. 2 is a detailed block diagram of the invention. The
`
`
`
`the same embodiment;
`
`image fed in from the CCD camera is stored in a frame
`
`
`(j)is a diagram showing a closed state of the same
`
`memory 21. In a reference image memory 25, a background
`embodiment;
`
`
`image not including person taken previously is stored as
`
`
`FIG. 23(a) is a diagram showing the shape of an example
`
`
`
`reference image. The reference image may be updated
`
`
`of a virtual object used in the interface apparatus of the same 45
`
`
`
`whenever as required.
`embodiment;
`Shape identifying means 22 depicts the difference of the
`
`
`
`(b)is a diagram showing the shape of other example of a
`
`
`
`image saved in the frame memory and the image stored in
`
`
`
`virtual object used in the interface apparatus of the
`
`
`the reference image memory, and removes the background
`same embodiment;
`
`
`
`image from the image, depicts, for example, the portion
`50
`
`
`FIG. 24(a) is a front view showing configuration of cursor
`
`
`
`
`corresponding to the hand of the user, and judges if the shape
`
`
`is, for example, one finger as shown in FIG. 3(A), two
`
`and virtual object in a virtual space;
`
`fingers as shown in FIG. 3(B), or three fingers as shown in
`
`
`
`
`(b)is a side view showing configuration of cursor and
`FIG. 3(C).
`
`virtual object in a virtual space;
`
`
`
`FIG. 25 is a diagram showing a display example of virtual
`FIG. 4 shows a detailed example of the shape identifying
`
`
`
`55
`space for explaining the embodiment;
`
`
`
`means 22, which comprises an image difference operation
`
`
`FIG. 26 is a block diagram showing an example of the
`
`unit 221, a contour depicting unit 222, and a shape identi­
`
`
`interface apparatus of the same embodiment;
`fying unit 223.
`
`
`
`FIG. 27(a) is a diagram showing an example of input
`The image difference operation unit 221 calculates the
`
`
`
`
`
`
`device in input means used in the interface apparatus of the
`
`
`difference of the image saved in the frame memory and the
`60
`same embodiment;
`
`
`
`image stored in the reference image memory as mentioned
`(b) is a diagram showing an example of input device in
`
`
`
`
`
`
`
`
`above. As a result, the object to be detected, for example, the
`
`input means used in the interface apparatus of the same
`
`
`
`user, can be separated from the background portion. For
`embodiment;
`
`
`
`example, when the image difference operation unit 221 is
`
`
`
`
`composed of a simple subtraction circuit, as shown in FIG.
`
`
`
`
`(c)is a diagram showing an example of input device in 65
`
`
`
`5, only the portion of the hand of the user in the image in the
`
`input means used in the interface apparatus of the same
`
`frame memory can be depicted. The contour depicting unit
`embodiment;
`
`IPR2021-00922
`Apple EX1008 Page 29
`
`

`

`
`
`US 6,191,773 Bl
`
`6
`5
`at least of an image pickup unit, a motion recognizing unit
`
`
`
`
`
`
`
`222 depicts the contour shape of the object existing in the
`
`
`for recognizing the shape or move of an object in a taken
`
`
`
`
`image as a result of operation by the image difference
`
`
`
`
`picture, and a display unit for displaying the shape or move
`
`operation unit 221. As a practical method, for example, by
`
`
`
`
`
`
`of the object recognized by the motion recognizing unit, for
`depicting the edge of the image, the contour shape may be
`
`
`
`
`5 storing the image taken by the image pickup unit, and a
`
`easily depicted.
`
`reference image memory for storing the image taken before
`
`
`
`The shape identifying unit 223 identifies specifically the
`
`
`
`
`the image saved in the frame memory as reference image,
`
`
`
`
`contour shape of the hand depicted by the contour depicting
`
`wherein the motion recognizing unit comprises an image
`
`
`
`
`unit 222,and judges if the shape is, for example, one finger
`
`
`
`change depicting unit for depicting the difference between
`
`as shown in FIG. 3(A) or two fingers as shown in FIG. 3(B).
`
`
`the image in the frame memory and the reference image
`10
`
`
`
`As the shape identifying method, for example, template
`
`
`stored in the reference image memory.
`
`
`
`matching, matching technique with shape model, and neural
`
`
`network may be employed, among others.
`
`
`
`FIG. 8 shows the appearance of the second embodiment
`
`
`of the interface apparatus of the invention. In FIG. 8, same
`
`
`
`An icon generating unit 24 generates an icon image as a
`
`
`
`
`constituent elements as in the first embodiment are identified
`
`
`special shape to be shown in the display, on the basis of the
`15
`
`
`with same reference numerals. That is, reference 1 is a host
`
`
`
`result of identifying the hand shape by the shape identifying
`
`
`computer, 2 is a display unit, and 3 is a CCD camera for
`
`
`unit 223. For example, when the result of identifying the
`
`picking up an image. The CCD camera 3 has an image
`
`
`
`shape of the hand was one finger, an icon of numeral "1" is
`
`
`
`
`pickup surface located in the same direction as the display
`
`
`
`generated as shown in FIG. 6(A), or in the case of two
`
`
`
`
`direction, so that the hand gesture of the user can be picked
`
`
`
`
`fingers, an icon of numeral "2" is created as in FIG. 6(B). As
`
`the shape of the icon, alternatively, when the result of 20
`
`
`
`
`up as the user confronts the display surface. On the display
`
`
`
`
`
`
`surface of the display unit 2, virtual switches 204, 205,206,
`
`
`identifying the shape of the hand was one finger, an icon of
`
`
`
`and an icon of an arrow cursor 203 for selecting the virtual
`one finger may be shown as shown in FIG. 6(C), or in the
`
`switches are displayed.
`
`
`
`
`
`case of two fingers, an icon of two fingers may be created as
`
`
`
`shown in FIG. 6(D). A display controller 23 controls the
`
`
`
`FIG. 9 is a block diagram showing a specific constitution
`
`display on the basis of the result of identifying the shape of 25
`
`
`
`
`of the embodiment. The image fed through the CCD camera
`
`the hand by the shape identifying unit 223. For example,
`
`
`3 is saved in a frame memory 21. A preliminarily taken
`
`
`
`while displaying the icon according to the result of
`
`
`
`image is stored in a reference image memory 25 as a
`
`
`
`
`identifying, the menu previously corresponding to the result
`
`
`
`reference image. A reference image updating unit 26 is
`
`
`
`
`of identifying is displayed by emphasis on the basis of the
`
`
`composed of a timer 261 and an image updating unit 262,
`
`
`
`30 and is designed to update the reference image by transferring
`
`hand shape identifying result.
`
`
`
`the latest image stored in the frame memory 21 to the
`
`
`In this embodiment of the invention, an example of
`
`
`
`reference image memory 25 at a specific time interval
`
`
`
`operations described below. As shown in FIG. 7(A), when
`
`indicated by the timer 261.
`
`
`
`the user confronts the appliance having the interface appa­
`
`
`
`ratus of the invention and points out one finger, an icon of
`An motion recognizing unit 22 depicts the difference
`
`
`
`numeral "1" is shown on the display, and the display of 35
`
`
`
`
`between the image saved in the frame memory and the
`
`
`
`television on the first menu is shown by emphasis. At this
`
`
`
`image stored in the reference image memory, and eliminates
`
`
`
`time, by using sound or voice from the display device in tune
`
`the background image from the image, and depicts the
`
`
`with the emphasis display, the attention of the operator may
`
`
`
`portion corresponding, for example, to the hand of the user,
`
`
`
`
`be attracted. Herein, by pointing out two fingers as in FIG.
`
`
`
`and also judges if the shape is one finger as shown in FIG.
`40
`lO(A) or a fist as shown in FIG. lO(B).
`
`
`
`7(B), an icon of numeral "2" is shown on the display and the
`
`
`
`
`display of network on the second menu is shown by empha­
`FIG. 11 shows a detailed example of the motion recog­
`
`
`
`
`sis. In this state, by maintaining the same hand shape for a
`
`
`nizing unit 22, being composed of an image difference
`
`
`
`specific time, the second menu is selected, and an instruction
`
`
`
`operation unit 221, a contour depicting unit 222, a shape
`
`
`
`is given to the host computer so as to display the network
`
`
`
`change identifying unit 225, and a position detector 224.
`terminal. For selection of menu, sound or the like may be 45
`
`
`
`
`
`
`
`The image difference operation unit 221 calculates the
`used at the same time. In the case of hand shape different
`
`
`
`
`difference between the image saved in the frame memory 21
`
`
`from those determined preliminarily as in FIG. 7(C), icon
`
`
`
`and the image stored in the reference image memory 25 as
`
`
`and menu are not shown on the display, and no instruction
`
`
`
`
`mentioned above. Consequently, the object desired to be
`
`is given to the host computer.
`
`
`
`example, the hand portion of the 50 depicted as motion, for
`
`
`
`
`Thus, according to the invention, by identifying the shape
`
`
`
`
`user, can be separated from the background portion, and
`
`
`
`of the hand in the taken image, it is possible to control the
`
`
`only the moving object image can be depicted at the same
`
`
`
`
`computer or appliance on the basis of the result of
`
`
`
`time. For example, when the image difference operation unit
`
`
`
`
`identifying, and it is possible to manipulate without making
`
`
`
`221 is composed of a mere subtraction circuit, as shown in
`
`
`
`contact from a remote distance without using keyboard,
`
`
`
`and only the 55 FIG. 12, the hand portion in the reference image
`
`
`
`
`mouse or other device. Besides, as the result of identifying
`
`
`
`
`hand portion of the latest image in the frame memory can be
`
`
`
`the shape of the hand is reflected in the screen, the user can
`
`
`
`
`depicted, so that only the moving hand portion can be easily
`
`
`
`
`manipulate while confirming the result of identifying, and
`
`
`
`
`identified. The contour depicting portion 222 depicts the
`
`
`ease and secure manipulation is possible.
`
`
`
`
`object existing in the image as the result of operation by the
`
`
`
`
`In this embodiment, this is an example of applying in
`
`
`
`contour 60 image difference operation unit 221, that is, the
`
`
`selection of menu, but by pressing so that the icon display
`
`
`
`shape of the hand portion before moving and after moving.
`
`
`
`according to a specific shape of hand may be replaced by
`
`
`
`
`As an example of practical method, by depicting the edge of
`
`
`
`
`
`
`picture or message, it is also possible to control display and
`
`
`
`the image, the contour shape can be easily depicted.
`
`
`writing of picture or message.
`
`The shape change identifying unit 225 identifies the detail
`
`(Second embodiment)
`
`
`
`65 of the contour shape of the hand portion after moving being
`
`
`
`
`
`
`
`
`A second embodiment of the invention relates to an depicted by the contour depicting unit 222, and judges if the
`lO(A),
`
`
`
`
`
`
`
`interface apparatus comprising a frame memory composed shape is, for example, a finger as shown in FIG. or a
`
`IPR2021-00922
`Apple EX1008 Page 30
`
`

`

`
`
`US 6,191,773 Bl
`
`7
`8
`lO(B).
`
`fist as shown in FIG. At the same time, the position
`posed of at least an image pickup unit, a motion recognizing
`
`
`
`
`
`
`
`detector 224 calculates the coordinates of the center of
`
`
`
`
`unit for recognizing the shape and/or move of the hand of the
`
`
`
`
`gravity of the contour shape of the hand portion of the user
`
`
`
`
`user in the taken image, and a display unit for displaying the
`after moving.
`
`
`
`shape and/or move of the hand of the user recognized by the
`
`
`
`
`5 motion recognizing unit, thereby depicting the contour of
`
`
`
`An icon generating unit 24 generates an icon image to be
`
`
`
`the taken user image, a contour waveform operation unit for
`
`
`shown on the display on the basis of the result of identifying
`
`
`
`tracing the depicted contour, and calculating the relation
`
`the hand shape by the shape change iden