(12) United States Patent
`Maruno et al.
`
`USOO6191773B1
`(10) Patent No.:
`US 6, 191,773 B1
`(45) Date of Patent:
`*Feb. 20, 2001
`
`(*) Notice:
`
`(54) INTERFACE APPARATUS
`(75) Inventors: Susumu Maruno, Osaka; Taro
`Imagawa, Harakata; Michiyo Moriya,
`Moriguchi; Kenji Kondo, Neyagawa,
`all of (JP)
`(73) Assignee: Matsushita Electric Industrial Co.,
`Ltd., Osaka (JP)
`This patent issued on a continued pros
`ecution application filed under 37 CFR
`1.53(d), and is subject to the twenty year
`patent term provisions of 35 U.S.C.
`154(a)(2).
`Under 35 U.S.C. 154(b), the term of this
`patent shall be extended for 0 days.
`(21) Appl. No.:
`08/945,628
`(22) PCT Filed:
`Apr. 25, 1996
`(86) PCT No.:
`PCT/JP96/O1124
`S371 Date:
`Jan. 2, 1998
`S 102(e) Date: Jan. 2, 1998
`(87) PCT Pub. No.: WO96/34332
`PCT Pub. Date: Oct. 31, 1996
`Foreign Application Priority Data
`(30)
`Apr. 28, 1995
`(JP) ................................................... 7-105551
`Oct. 19, 1995 (JP).
`... 7-270828
`Nov. 1, 1995
`(JP) ................................................... 7-284768
`(51) Int. Cl." ....................................................... G09G 5/08
`(52) U.S. Cl. ............................................. 345/158; 34.5/156
`(58) Field of Search .................................. 66/104, 55, 13,
`66/15, 120; 358/403; 379/93.14, 100.08,
`88.13, 88.24; 345/427, 156, 157, 158, 173
`
`
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`1/1997 Freeman et al. ..................... 34.5/158
`5,594,469
`5,767,842 * 6/1998 Korth ..........
`... 34.5/173
`5,864,334
`1/1999 Sellers .................................. 34.5/158
`
`FOREIGN PATENT DOCUMENTS
`8-16781 * 1/1996 (JP).
`OTHER PUBLICATIONS
`“Wideo-Based Handshape Recognition Using a Handshape
`Structure Model in RealTime” Grobel et al., 1996 IEEE, pp.
`446-450, 1996.*
`* cited by examiner
`
`Primary Examiner Dennis-Doon Chow
`(74) Attorney, Agent, or Firm-Ratner & Prestia
`(57)
`ABSTRACT
`The invention relates to an interface apparatus for making
`input and output of appliances having display Such as
`computer, word processor, information appliance and
`television, comprising recognizing means for recognizing
`the shape or move of the hand of an operator, display means
`for displaying the features of the shape or move of the hand
`recognized by the recognizing means as special shape in the
`Screen, and control means for controlling the information
`displayed in the Screen by the Special shape displayed in the
`Screen by the display means, wherein the two-dimensional
`or three-dimensional information displayed in the Screen can
`be selected, indicated or moved only by changing the shape
`or moving the hand, So that the interface apparatus of very
`excellent controllability and high diversity may be pre
`Sented.
`
`6 Claims, 26 Drawing Sheets
`
`IPR2022-00090 - LGE
`Ex. 1008 - Page 1
`
`

`

`U.S. Patent
`
`Feb. 20, 2001
`
`Sheet 1 of 26
`
`US 6,191,773 B1
`
`2OO
`
`
`
`
`
`
`
`
`
`
`
`FIG. 1
`
`IPR2022-00090 - LGE
`Ex. 1008 - Page 2
`
`

`

`U.S. Patent
`
`Feb. 20, 2001
`
`Sheet 2 of 26
`
`US 6,191,773 B1
`
`25
`
`
`
`
`
`Reference image
`memory
`22
`
`Shape identifying
`eaS
`
`
`
`
`
`Display control
`Unit
`23
`
`COn generating
`Unit
`
`24
`
`
`
`
`
`FIG. 2
`
`h FIG. 3(A)
`()
`
`FIG. 3(C)
`
`FIG. 3(B)
`
`IPR2022-00090 - LGE
`Ex. 1008 - Page 3
`
`

`

`U.S. Patent
`
`Feb. 20, 2001
`
`Sheet 3 of 26
`
`US 6,191,773 B1
`
`Image difference - 221
`Calculation Unit
`
`Contour depicting
`Unit
`
`Shape identifying
`unit
`
`222
`
`223
`
`
`
`22
`
`FIG. 4
`
`Reference image
`
`Result of operation
`
`Taken image
`
`FIG. 5
`
`IPR2022-00090 - LGE
`Ex. 1008 - Page 4
`
`

`

`U.S. Patent
`
`Feb. 20, 2001
`
`Sheet 4 of 26
`
`US 6,191,773 B1
`
`2
`
`FIG. 6(A)
`
`FIG. 6(B)
`
`
`
`FIG. 6(C)
`
`FIG. 6(D)
`
`IPR2022-00090 - LGE
`Ex. 1008 - Page 5
`
`

`

`U.S. Patent
`
`Feb. 20, 2001
`
`Sheet S of 26
`
`US 6,191,773 B1
`
`
`
`
`
`
`
`2OO
`
`
`
`
`
`-
`
`FIG. 7(A)
`
`2OO
`
`1
`
`FIG. 7(B)
`
`IPR2022-00090 - LGE
`Ex. 1008 - Page 6
`
`

`

`U.S. Patent
`
`Feb. 20, 2001
`
`Sheet 6 of 26
`
`US 6,191,773 B1
`
`3
`
`O
`
`1
`
`FIG. 7 (C)
`
`IPR2022-00090 - LGE
`Ex. 1008 - Page 7
`
`

`

`U.S. Patent
`
`Feb. 20, 2001
`
`Sheet 7 of 26
`
`US 6,191,773 B1
`
`
`
`FIG. 8
`
`IPR2022-00090 - LGE
`Ex. 1008 - Page 8
`
`

`

`U.S. Patent
`U.S. Patent
`
`Feb. 20, 2001
`Feb. 20, 2001
`
`Sheet 8 of 26
`Sheet 8 of 26
`
`US 6,191,773 B1
`US 6,191,773 B1
`
`
`
`QSRIDT9OUIIAJOY
`
`Asowsw
`
`
`
`sunepdnsew]
`
`yun
`
`
`
`
`
`
`
`
`
`
`
`ve
`
`
`
`
`
`WunsunejauedsUos]
`
`
`
`SUIZIUZ0N0JUONOW
`
`sueow
`
`Ke|dsig
`
`Jos}UO9
`
`yun
`
`dJBUIPIOOZ
`
`
`
`wunsulwmossueyy
`
`
`
`[WunsunJOAUI
`
`aPUIPI007)
`
`6Sls
`
`IPR2022-00090 - LGE
`
`Ex. 1008 - Page 9
`
`IPR2022-00090 - LGE
`Ex. 1008 - Page 9
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Feb. 20, 2001
`
`Sheet 9 of 26
`
`US 6,191,773 B1
`
`FIG. 1 O(A)
`
`FIG. 1 O(B)
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`221
`
`222
`
`225
`
`224
`
`Image difference
`calculation unit
`
`Contour depicting
`unit
`
`Shape change
`identifying unit
`
`Position detector
`
`FIG 11
`
`IPR2022-00090 - LGE
`Ex. 1008 - Page 10
`
`

`

`U.S. Patent
`
`Feb. 20, 2001
`
`Sheet 10 of 26
`
`US 6,191,773 B1
`
`
`
`Reference image
`
`Result of operation
`
`Taken image
`
`FIG. 12
`
`IPR2022-00090 - LGE
`Ex. 1008 - Page 11
`
`

`

`U.S. Patent
`
`Feb. 20, 2001
`
`Sheet 11 of 26
`
`US 6,191,773 B1
`
`X.
`
`FIG. 13(B)
`
`V.
`
`FIG. 13(A)
`
`
`
`FIG. 13(C)
`
`FIG. 13(D)
`
`IPR2022-00090 - LGE
`Ex. 1008 - Page 12
`
`

`

`U.S. Patent
`
`Feb. 20, 2001
`
`Sheet 12 of 26
`
`US 6,191,773 B1
`
`
`
`Frame memory
`
`Motion recognizing
`e2S
`
`31
`
`32
`
`34
`
`
`
`Icon generating unit
`
`
`
`unit
`Coordinate
`transforming unit
`
`Coordinate
`inverting unit
`
`FIG. 14
`
`
`
`
`
`
`
`
`
`
`
`Hand shape output
`
`Hand coordinate output
`FIG. 15
`
`32
`
`IPR2022-00090 - LGE
`Ex. 1008 - Page 13
`
`

`

`U.S. Patent
`
`Feb. 20, 2001
`
`Sheet 13 of 26
`
`US 6,191,773 B1
`
`FIG. 16(A)
`
`FIG. 16(B)
`
`
`
`FIG. 16(C)
`
`FIG. 16(D)
`
`IPR2022-00090 - LGE
`Ex. 1008 - Page 14
`
`

`

`U.S. Patent
`
`Feb. 20, 2001
`
`Sheet 14 of 26
`
`US 6,191,773 B1
`
`(A1)
`
`-
`
`Start points
`
`y
`
`
`
`IPR2022-00090 - LGE
`Ex. 1008 - Page 15
`
`

`

`U.S. Patent
`
`Feb. 20, 2001
`
`Sheet 15 of 26
`
`US 6,191,773 B1
`
`(A2)
`
`
`
`N
`
`(B212)
`
`FIG. 18
`
`IPR2022-00090 - LGE
`Ex. 1008 - Page 16
`
`

`

`U.S. Patent
`
`Feb. 20, 2001
`
`Sheet 16 of 26
`
`US 6,191,773 B1
`
`Start points
`
`FIG. 19
`
`
`
`Start points
`
`FIG. 20
`
`IPR2022-00090 - LGE
`Ex. 1008 - Page 17
`
`

`

`U.S. Patent
`
`Feb. 20, 2001
`
`Sheet 17 of 26
`
`US 6,191,773 B1
`
`
`
`A4
`
`A3
`
`Display device
`
`Object coordinate
`memory unit
`
`Contact
`judging unit
`
`
`
`Cursor coordinate
`memory unit
`
`FIG. 21
`
`IPR2022-00090 - LGE
`Ex. 1008 - Page 18
`
`

`

`U.S. Patent
`
`Feb. 20, 2001
`
`Sheet 18 of 26
`
`S 6,191,773 B1
`U
`
`FIG. 22(A)
`
`FIG. 22(B)
`
`
`
`FIG. 22(E)
`
`FIG. 22(F)
`
`IPR2022-00090 - LGE
`Ex. 1008 - Page 19
`
`

`

`U.S. Patent
`
`Feb. 20, 2001
`
`Sheet 19 of 26
`
`US 6,191,773 B1
`
`Cube
`
`S).
`
`FIG. 23(A)
`
`Plane
`
`FIG. 23(B)
`
`IPR2022-00090 - LGE
`Ex. 1008 - Page 20
`
`

`

`U.S. Patent
`
`Feb. 20, 2001
`
`Sheet 20 of 26
`
`US 6,191,773 B1
`
`Example of configuration on virtual space (front view)
`
`Left
`
`Top
`
`l
`
`Cursor
`
`Right
`
`Virtual object
`
`Floor
`
`BOttom
`FIG. 24(A)
`
`Example of configuration on virtual space (side view)
`
`Top
`
`Rear
`
`l
`
`CurSOr
`Front
`
`Virtual object
`
`FOOr
`
`BOttom
`FIG. 24(B)
`
`IPR2022-00090 - LGE
`Ex. 1008 - Page 21
`
`

`

`U.S. Patent
`
`Feb. 20, 2001
`
`Sheet 21 of 26
`
`US 6,191,773 B1
`
`Display example
`
`Virtual object
`
`
`
`IPR2022-00090 - LGE
`Ex. 1008 - Page 22
`
`

`

`U.S. Patent
`U.S. Patent
`
`Feb. 20, 2001
`
`
`
`97
`OV
`
`aul]ysis
`
`sueowyndul
`
`20Ut1SIG
`
`sunjejnoyes
`
`
`
`
`
`suvousuIspnfUONdeJOIU]~Sr
`
`
`
`2}BUIPJOODJosING
`
`
`
`
`
`SUIWIOSULT}AJOWIDU
`
`suvoUl
`
`
`
`sueollsuvoul
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`uonoW
`
`SUIZIUZ0I01
`
`suvowAejdsiq
`
`Sheet 22 of 26
`
`US 6,191,773 B1
`US 6,191,773 B1
`
`
`
`
`
`suispnlpjesaaO
`
`sueoul
`
`
`
`SUBOWSUIUICI]
`
`JOJD0ADAO
`
`
`
`SUvOWSUIIEIND[VO
`
`suvow
`
`
`
`suispn{adeys
`
`suvoul
`
`9¢Dis
`
`IPR2022-00090 - LGE
`
`Ex. 1008 - Page 23
`
`IPR2022-00090 - LGE
`Ex. 1008 - Page 23
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Feb. 20, 2001
`
`Sheet 23 of 26
`
`US 6,191,773 B1
`
`Buttons
`
`FIG. 27(A)
`
`
`
`Operation buttons
`
`FIG. 27(C)
`
`IPR2022-00090 - LGE
`Ex. 1008 - Page 24
`
`

`

`U.S. Patent
`
`Feb. 20, 2001
`
`Sheet 24 of 26
`
`US 6,191,773 B1
`
`
`
`Circumscribing rectangle
`
`FIG. 28(B)
`
`IPR2022-00090 - LGE
`Ex. 1008 - Page 25
`
`

`

`U.S. Patent
`
`Feb. 20, 2001
`
`Sheet 25 of 26
`
`US 6,191,773 B1
`
`\
`( LIN \
`\
`/
`I
`/
`\
`\
`/ |
`|
`\ \ FIG. 29(A)
`b.
`AHA Fig. 29(e)
`
`Notice point 1
`
`
`
`FIG. 29(D)
`
`IPR2022-00090 - LGE
`Ex. 1008 - Page 26
`
`

`

`U.S. Patent
`
`Feb. 20, 2001
`
`Sheet 26 of 26
`
`US 6,191,773 B1
`
`
`
`FIG. 3O
`PRIOR ART
`
`IPR2022-00090 - LGE
`Ex. 1008 - Page 27
`
`

`

`1
`INTERFACE APPARATUS
`
`US 6,191,773 B1
`
`This Application is a U.S. National Phase Application of
`PCT International Application PCT/JP96/01124.
`
`BACKGROUND OF THE INVENTION
`The present invention relates to an interface apparatus for
`input and output of information apparatus Such as computer
`and word processor and appliance having a display Such as
`television.
`In a kind of conventional interface apparatus, it is
`designed to display a cursor at a coordinate position detected
`by the mouse on a display Screen, for adding Some other
`information to the information in the display device, or
`changing or Selecting the displayed information.
`FIG. 30 shows an outline of this conventional interface
`apparatus. In FIG. 30, reference numeral 501 denotes a host
`computer, and 502 is a display, and virtual operation buttons
`503, 504, 505 are displayed in the display 502 by the host
`computer 501. Reference numeral 506 represents a mouse
`cursor, and the host computer 501 controls the display so as
`to move in the screen in synchronism with the move of the
`mouse 507, on the basis of the moving distance of the mouse
`507 detected by the mouse 507. As the user moves the mouse
`507, the mouse cursor 506 is moved to the position of a
`desired virtual operation button in the display Screen, and by
`pressing a switch 508 on the mouse 507, an operation button
`is selected so as to instruct action to the host computer 501.
`In this conventional construction, however, the mouse or
`the input device is necessary in addition to the main body of
`the appliance, and a table or area for manipulating the mouse
`is also needed, which is not Suited to portable information
`appliance or the like. Besides, by manipulation through the
`mouse, it is not a direct and intuitive interface.
`
`SUMMARY OF THE INVENTION
`It is an object of the invention to present an interface
`apparatus capable of manipulating an appliance easily with
`out requiring input device Such as keyboard and mouse. It is
`other object thereof to present an interface apparatus further
`advanced in the ease of manipulation of indicating or
`catching the display object by judging interactions along the
`intent of the operator Sequentially and automatically.
`In Structure, the invention provides an interface apparatus
`comprising recognizing means for recognizing the shape of
`a hand of an operator, display means for displaying the
`features of the shape of the hand recognized by the recog
`nizing means on the Screen as a special shape, and control
`means for controlling the information displayed in the
`Screen by the Special shape displayed in the Screen by the
`display means, whereby the information displayed in the
`Screen can be controlled only by varying the shape of the
`hand.
`It is a further object to present an interface apparatus
`much Superior in ease of manipulation by recognizing also
`the move of the hand. To recognize the move, a frame
`memory for Saving the image picking up the shape or move
`of the hand, and a reference image memory for Storing the
`image taken before the image Saved in the frame memory as
`reference image are provided, and it is achieved by depicting
`the difference between the image in the frame memory and
`the reference image Stored in the reference image memory.
`In other method of recognition, the shape or move of the
`hand of the user in the taken image is depicted as the contour
`of the user, and its contour is traced, and the relation between
`
`15
`
`25
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`the angle of the contour line and the length of contour line,
`that is, the contour waveform is calculated and filtered, and
`the shape waveform expressing the Specified shape is gen
`erated.
`Moreover, comprising cursor display means for display
`ing a feature of the shape of a hand on the Screen as a special
`shape and manipulating as cursor, means for Storing the
`relation with display object other than cursor displays the
`coordinates and shape of the representative point represent
`ing the position of the display object other than cursor
`display, and means for calculating and judging the interac
`tion of the cursor display and the display object, manipula
`tion is realized Smoothly by the interactions along the intent
`of the operator when gripping the displayed virtual object in
`the case of display of cursor display as virtual manipulator.
`In the interface apparatus thus constructed, as the user
`faces the recognizing means and shows, for example, a hand,
`the Special shape corresponding to the shape of the hand is
`displayed as an icon in the Screen for Screen manipulation,
`So that control according to the icon display is enabled.
`Or when instructed by hand gesture, the given hand
`gesture is displayed as a special shape Set corresponding to
`the shape of the hand on the display Screen, and its move is
`also displayed, and, for example, a virtual Switch or the like
`displayed on the display Screen can be selected by the hand
`gesture, or the display object displayed on the Screen can be
`grabbed or carried depending on the purpose, and therefore
`without requiring mouse or other input device, a very simple
`manipulation of appliance is realized.
`It is further possible to realize the interface much
`enhanced in the ease of manipulation by Sequentially and
`automatically judging the interaction with the display object
`desired to be operated by the Virtual manipulator according
`to the intent of operation of the operator, as the Special shape
`Set corresponding to the shape of the hand works as Virtual
`manipulator aside from the mere cursor.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`FIG. 1 is an appearance drawing of an interface apparatus
`in a first embodiment of the invention;
`FIG. 2 is a detailed block diagram of the interface
`apparatus in the same embodiment of the invention;
`FIG. 3 is a diagram showing an example of Shape of hand
`judged by the interface apparatus in the same embodiment of
`the invention;
`FIG. 4 is a diagram showing an example of shape iden
`tifying means of the interface apparatus in the same embodi
`ment of the invention;
`FIG. 5 is a diagram showing an example of operation by
`an image difference operation unit in the Same embodiment;
`FIG. 6 is a diagram showing an example of icon generated
`by an icon generating unit in the same embodiment;
`FIG. 7 is an appearance drawing showing an operation
`example of the interface apparatus of the same embodiment;
`FIG. 8 is an appearance drawing of an interface apparatus
`in a Second embodiment of the invention;
`FIG. 9 is detailed block diagram of the interface apparatus
`in the second embodiment of the invention;
`FIG. 10 is a diagram showing an example of shape of
`hand judged by the interface apparatus of the same embodi
`ment,
`FIG. 11 is a diagram showing an example of motion
`recognizing unit of the interface apparatus of the same
`embodiment;
`
`IPR2022-00090 - LGE
`Ex. 1008 - Page 28
`
`

`

`3
`FIG. 12 is a diagram showing an example of operation by
`an image difference operation unit in the Same embodiment;
`FIG. 13 is a diagram showing an operation example of the
`Same embodiment;
`FIG. 14 is a detailed block diagram of an interface
`apparatus in a third embodiment of the invention;
`FIG. 15 is a diagram showing an example of motion
`recognizing unit of the interface apparatus in the third
`embodiment of the invention;
`FIG.16(A) to (D) are diagrams showing examples of icon
`displayed on a display Screen by the interface apparatus of
`the same embodiment;
`FIG. 17 is a diagram showing operation of motion rec
`ognizing unit of the interface apparatus in the same embodi
`ment of the invention;
`FIG. 18 is a diagram showing operation of motion rec
`ognizing unit of the interface apparatus in the same embodi
`ment of the invention;
`FIG. 19 is a diagram showing operation of motion rec
`ognizing unit of the interface apparatus in the same embodi
`ment of the invention;
`FIG. 20 is a diagram showing operation of motion rec
`ognizing unit of the interface apparatus in the same embodi
`ment of the invention;
`FIG. 21 is a diagram showing an interface apparatus
`explaining a fourth embodiment;
`FIG. 22(a) is a diagram showing an open State of cursor
`in an example of a cursor used in the interface apparatus of
`the same embodiment;
`(b) is a diagram showing a closed State of the same
`embodiment;
`(c) is a diagram showing an open State of cursor in an
`example of a cursor used in the interface apparatus of
`the same embodiment;
`(d) is a diagram showing a closed State of the same
`embodiment;
`(e) is a diagram showing an open State of cursor in an
`example of a cursor used in the interface apparatus of
`the same embodiment;
`(f) is a diagram showing a closed State of the same
`embodiment;
`FIG. 23(a) is a diagram showing the shape of an example
`of a virtual object used in the interface apparatus of the same
`embodiment;
`(b) is a diagram showing the shape of other example of a
`Virtual object used in the interface apparatus of the
`Same embodiment;
`FIG.24(a) is a front view showing configuration of cursor
`and virtual object in a virtual space;
`(b) is a side view showing configuration of cursor and
`Virtual object in a virtual space;
`FIG.25 is a diagram showing a display example of virtual
`Space for explaining the embodiment;
`FIG. 26 is a block diagram showing an example of the
`interface apparatus of the same embodiment;
`FIG. 27(a) is a diagram showing an example of input
`device in input means used in the interface apparatus of the
`Same embodiment;
`(b) is a diagram showing an example of input device in
`input means used in the interface apparatus of the same
`embodiment;
`(c) is a diagram showing an example of input device in
`input means used in the interface apparatus of the same
`embodiment;
`
`1O
`
`15
`
`25
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`US 6,191,773 B1
`
`4
`FIG. 28(a) is a diagram showing an example of image of
`a hand taken by a camera in the Same embodiment;
`(b) is a diagram showing a binary example of image of a
`hand taken by a camera in the same embodiment;
`FIG. 29(a) is a diagram showing an example of image
`displayed by display means used in the interface apparatus
`in the same embodiment of the invention;
`(b) is a diagram showing a second example of the display
`Screen,
`(c) is a diagram showing a third example of the display
`Screen,
`(d) is a diagram showing a fourth example of the display
`Screen; and
`FIG. 30 is an explanatory diagram for explaining a
`conventional interface apparatus.
`DESCRIPTION OF THE PREFERRED
`EMBODIMENTS
`(First embodiment)
`A first embodiment of the invention relates to an interface
`comprising recognizing means Such as image pickup device
`for recognizing the shape of a hand of the operator, display
`means for displaying the feature of the shape of the hand
`recognized by the recognizing means on a Screen as a special
`shape by an icon or the like, and control means for control
`ling the information displayed on the Screen by varying the
`shape of the handby operating the Special shape Such as icon
`displayed on the Screen by the display means as the So-called
`CUSO.
`FIG. 1 shows the appearance of the first embodiment of
`the interface apparatus of the invention. Reference numeral
`1 denotes a host computer, 2 is a display unit, and 3 is a CCD
`camera for picking up an image. The CCD camera 3 has the
`pickup Surface located in the same direction as the display
`direction, So that the shape of the hand of the user can be
`picked up when the user confronts the display Screen. On the
`display, menu 201, 202, and icon 200 reflecting the shape of
`the hand are displayed.
`FIG. 2 is a detailed block diagram of the invention. The
`image fed in from the CCD camera is Stored in a frame
`memory 21. In a reference image memory 25, a background
`image not including person taken previously is Stored as
`reference image. The reference image may be updated
`whenever as required.
`Shape identifying means 22 depicts the difference of the
`image Saved in the frame memory and the image Stored in
`the reference image memory, and removes the background
`image from the image, depicts, for example, the portion
`corresponding to the hand of the user, and judges if the shape
`is, for example, one finger as shown in FIG. 3(A), two
`fingers as shown in FIG. 3(B), or three fingers as shown in
`FIG. 3(C).
`FIG. 4 shows a detailed example of the shape identifying
`means 22, which comprises an image difference operation
`unit 221, a contour depicting unit 222, and a shape identi
`fying unit 223.
`The image difference operation unit 221 calculates the
`difference of the image Saved in the frame memory and the
`image Stored in the reference image memory as mentioned
`above. As a result, the object to be detected, for example, the
`user, can be separated from the background portion. For
`example, when the image difference operation unit 221 is
`composed of a simple Subtraction circuit, as shown in FIG.
`5, only the portion of the hand of the user in the image in the
`frame memory can be depicted. The contour depicting unit
`
`IPR2022-00090 - LGE
`Ex. 1008 - Page 29
`
`

`

`S
`222 depicts the contour shape of the object existing in the
`image as a result of operation by the image difference
`operation unit 221. As a practical method, for example, by
`depicting the edge of the image, the contour shape may be
`easily depicted.
`The shape identifying unit 223 identifies specifically the
`contour shape of the hand depicted by the contour depicting
`unit 222,and judges if the shape is, for example, one finger
`as shown in FIG.3(A) or two fingers as shown in FIG.3(B).
`AS the shape identifying method, for example, template
`matching, matching technique with shape model, and neural
`network may be employed, among others.
`An icon generating unit 24 generates an icon image as a
`Special shape to be shown in the display, on the basis of the
`result of identifying the hand shape by the shape identifying
`unit 223. For example, when the result of identifying the
`shape of the hand was one finger, an icon of numeral “1” is
`generated as shown in FIG. 6(A), or in the case of two
`fingers, an icon of numeral “2” is created as in FIG. 6(B). As
`the shape of the icon, alternatively, when the result of
`identifying the shape of the hand was one finger, an icon of
`one finger may be shown as shown in FIG. 6(C), or in the
`case of two fingers, an icon of two fingers may be created as
`shown in FIG. 6(D). A display controller 23 controls the
`display on the basis of the result of identifying the shape of
`the hand by the shape identifying unit 223. For example,
`while displaying the icon according to the result of
`identifying, the menu previously corresponding to the result
`of identifying is displayed by emphasis on the basis of the
`hand shape identifying result.
`In this embodiment of the invention, an example of
`operations described below. As shown in FIG. 7(A), when
`the user confronts the appliance having the interface appa
`ratus of the invention and points out one finger, an icon of
`numeral “1” is shown on the display, and the display of
`television on the first menu is shown by emphasis. At this
`time, by using Sound or voice from the display device in tune
`with the emphasis display, the attention of the operator may
`be attracted. Herein, by pointing out two fingers as in FIG.
`7(B), an icon of numeral “2” is shown on the display and the
`display of network on the Second menu is shown by empha
`Sis. In this State, by maintaining the same hand shape for a
`Specific time, the Second menu is Selected, and an instruction
`is given to the host computer So as to display the network
`terminal. For Selection of menu, Sound or the like may be
`used at the Same time. In the case of hand shape different
`from those determined preliminarily as in FIG. 7(C), icon
`and menu are not shown on the display, and no instruction
`is given to the host computer.
`Thus, according to the invention, by identifying the shape
`of the hand in the taken image, it is possible to control the
`computer or appliance on the basis of the result of
`identifying, and it is possible to manipulate without making
`contact from a remote distance without using keyboard,
`mouse or other device. Besides, as the result of identifying
`the shape of the hand is reflected in the Screen, the user can
`manipulate while confirming the result of identifying, and
`ease and Secure manipulation is possible.
`In this embodiment, this is an example of applying in
`Selection of menu, but by pressing So that the icon display
`according to a specific shape of hand may be replaced by
`picture or message, it is also possible to control display and
`Writing of picture or message.
`(Second embodiment)
`A Second embodiment of the invention relates to an
`interface apparatus comprising a frame memory composed
`
`15
`
`25
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`US 6,191,773 B1
`
`6
`at least of an image pickup unit, a motion recognizing unit
`for recognizing the shape or move of an object in a taken
`picture, and a display unit for displaying the shape or move
`of the object recognized by the motion recognizing unit, for
`Storing the image taken by the image pickup unit, and a
`reference image memory for Storing the image taken before
`the image Saved in the frame memory as reference image,
`wherein the motion recognizing unit comprises an image
`change depicting unit for depicting the difference between
`the image in the frame memory and the reference image
`Stored in the reference image memory.
`FIG. 8 shows the appearance of the second embodiment
`of the interface apparatus of the invention. In FIG. 8, same
`constituent elements as in the first embodiment are identified
`with Same reference numerals. That is, reference 1 is a host
`computer, 2 is a display unit, and 3 is a CCD camera for
`picking up an image. The CCD camera 3 has an image
`pickup Surface located in the same direction as the display
`direction, So that the hand gesture of the user can be picked
`up as the user confronts the display Surface. On the display
`surface of the display unit 2, virtual Switches 204, 205.206,
`and an icon of an arrow cursor 203 for selecting the virtual
`Switches are displayed.
`FIG. 9 is a block diagram showing a Specific constitution
`of the embodiment. The image fed through the CCD camera
`3 is Saved in a frame memory 21. A preliminarily taken
`image is Stored in a reference image memory 25 as a
`reference image. A reference image updating unit 26 is
`composed of a timer 261 and an image updating unit 262,
`and is designed to update the reference image by transferring
`the latest image Stored in the frame memory 21 to the
`reference image memory 25 at a Specific time interval
`indicated by the timer 261.
`An motion recognizing unit 22 depicts the difference
`between the image Saved in the frame memory and the
`image Stored in the reference image memory, and eliminates
`the background image from the image, and depicts the
`portion corresponding, for example, to the hand of the user,
`and also judges if the shape is one finger as shown in FIG.
`10(A) or a fist as shown in FIG. 10(B).
`FIG. 11 shows a detailed example of the motion recog
`nizing unit 22, being composed of an image difference
`operation unit 221, a contour depicting unit 222, a shape
`change identifying unit 225, and a position detector 224.
`The image difference operation unit 221 calculates the
`difference between the image Saved in the frame memory 21
`and the image Stored in the reference image memory 25 as
`mentioned above. Consequently, the object desired to be
`depicted as motion, for example, the hand portion of the
`user, can be separated from the background portion, and
`only the moving object image can be depicted at the same
`time. For example, when the image difference operation unit
`221 is composed of a mere Subtraction circuit, as shown in
`FIG. 12, the hand portion in the reference image and only the
`hand portion of the latest image in the frame memory can be
`depicted, So that only the moving hand portion can be easily
`identified. The contour depicting portion 222 depicts the
`object existing in the image as the result of operation by the
`image difference operation unit 221, that is, the contour
`shape of the hand portion before moving and after moving.
`AS an example of practical method, by depicting the edge of
`the image, the contour shape can be easily depicted.
`The shape change identifying unit 225 identifies the detail
`of the contour shape of the hand portion after moving being
`depicted by the contour depicting unit 222, and judges if the
`shape is, for example, a finger as shown in FIG. 10(A), or a
`
`IPR2022-00090 - LGE
`Ex. 1008 - Page 30
`
`

`

`7
`fist as shown in FIG. 10(B). At the same time, the position
`detector 224 calculates the coordinates of the center of
`gravity of the contour shape of the hand portion of the user
`after moving.
`An icon generating unit 24 generates an icon image to be
`shown on the display on the basis of the result of identifying
`the hand shape by the shape change identifying unit 225. AS
`examples of icon image, for example, when the result of
`identifying the hand shape is one finger, for example, the
`arrow marked icon as shown in FIG. 13(A) may be
`generated, or in the case of a first shape, the X-marked icon
`as shown in FIG. 13(B) may be generated. Or, if the
`identifying result of hand shape is two fingers, an icon
`mimicking two fingers as shown, for example, in FIG. 13(C)
`may be generated, or in the case of a first, an icon mimicking
`the fist may be generated as shown in FIG. 13(D).
`A display controller 23 controls the display position of the
`icon generated by the icon generating unit 24 on the display
`2, and is composed of a coordinate transforming unit 231
`and a coordinate inverting unit 232. The coordinate trans
`forming unit 231 transforms from the coordinates of the
`taken image into display coordinates on the display 2, and
`the coordinate inverting unit 232 inverts the lateral positions
`of the transformed display coordinates. That is, the coordi
`nates of the center of gravity in the image of the portion
`corresponding to the user's hand detected by the position
`detector 224 are transformed into display coordinates on the
`display 2, and the lateral coordinates are inverted to display
`an icon in the display 2. By this manipulation, when the user
`moves the hand to the right, the icon moves to the right on
`the display Screen, like a mirror action.
`In thus constituted embodiment, an example of operation
`is described below. As shown in FIG. 8, when the user
`confronts the appliance incorporating the interface apparatus
`of the embodiment and points out one finger of the hand, the
`arrow cursor appearing on the display moves to an arbitrary
`position corresponding to the move of the hand. Then, by
`moving the hand to an arbitrary one of the Virtual Switches
`204, 205, 206 shown on the display 2, the arrow cursor is
`moved, and when the hand is gripped to form a fist, the one
`of the virtual Switches 204, 205, 206 is selected, and an
`instruction is given to the host computer 1.
`In this embodiment, it is designed to recognize the shape
`and move of the object in the taken image, but it may be also
`designed to recognize either the Shape or the move of the
`object in the taken image.
`Thus, according to the invention, which comprises the
`motion recognizing unit for recognizing the shape and/or
`move of the objectin the taken image, display unit for
`displaying the shape and/or move of the object recognized
`by the motion recognizing unit, frame memory for Saving
`the image taken by the image pickup means, and a reference
`image memory for Storing the image taken before the image
`Saved in the frame memory as reference image, by depicting
`the difference between the image in the frame memory and
`the reference image Stored in the reference image memory in
`the motion recognizing unit, when the user confronts the
`image pickup unit and gives instruction by, for example, a
`hand gesture, the given han