(19) United States
`(12) Patent Application Publication (10) Pub. No.: US 2003/0048260A1
`Matusis
`(43) Pub. Date:
`Mar. 13, 2003
`
`US 200300.48260A1
`
`(54) SYSTEM AND METHOD FOR SELECTING
`ACTIONS BASED ON THE DENTIFICATION
`OF USERS FINGERS
`(76) Inventor: Alec Matusis, Stanford, CA (US)
`Correspondence Address:
`LUMEN INTELLECTUAL PROPERTY
`SERVICES
`45 CABOTAVENUE, SUITE 110
`SANTACLARA, CA 95051 (US)
`
`(21) Appl. No.:
`(22) Filed:
`
`10/222,195
`Aug. 16, 2002
`9
`Related U.S. Application Data
`
`(60) Provisional application No. 60/313,083, filed on Aug.
`17, 2001.
`
`Publication Classification
`
`(51) Int. Cl." ....................................................... G09G 5/00
`
`
`
`(52) U.S. Cl. .............................................................. 345/173
`
`(57)
`
`ABSTRACT
`
`Provided is a System and method that increases the func
`tionality of input devices and control panels. A dependent
`relationship between n functions and n fingertips is associ
`ated with an input Sensor. Including different motions for
`each fingertip could extend this dependent relationship and
`further increase functionality. A user Selects only one of
`his/her fingertips, which then activates the input Sensor
`(through on/off activation and/or motion). The Selected
`fingertip is the only fingertip that is required to activate the
`input Sensor, thereby allowing the input Sensor to be arbi
`trary Small. An imaging means is included to identify which
`fingertip activates the input Sensor. Imaging means requires
`the acquisition of at least one image of a part of the user's
`hand large enough to identify the Selected fingertip activat
`ing the input Sensor. A processing means is included to
`determine from data of the input Sensor and acquired images
`which function is Selected.
`
`910
`
`922
`
`IPR2022-00602
`Apple EX1032 Page 1
`
`

`

`Patent Application Publication Mar. 13, 2003 Sheet 1 of 14
`
`US 2003/0048260A1
`
`0 I UOQ0Unq | quum?L
`
`
`
`I 9 Inã1-I
`
`IPR2022-00602
`Apple EX1032 Page 2
`
`

`

`Patent Application Publication Mar. 13, 2003 Sheet 2 of 14
`
`US 2003/0048260A1
`
`
`
`Z 9.Inã1-I
`
`IPR2022-00602
`Apple EX1032 Page 3
`
`

`

`Patent Application Publication Mar. 13, 2003 Sheet 3 of 14
`
`US 2003/0048260A1
`
`
`
`IPR2022-00602
`Apple EX1032 Page 4
`
`

`

`Patent Application Publication
`
`Mar. 13, 2003 Sheet 4 of 14
`
`US 2003/0048260A1
`
`
`
`IPR2022-00602
`Apple EX1032 Page 5
`
`

`

`Patent Application Publication Mar. 13, 2003 Sheet 5 of 14
`
`US 2003/0048260A1
`
`
`
`Cr
`y
`f
`
`t
`
`IPR2022-00602
`Apple EX1032 Page 6
`
`

`

`Patent Application Publication Mar. 13, 2003 Sheet 6 of 14
`
`US 2003/0048260A1
`
`
`
`-- y
`
`t
`
`IPR2022-00602
`Apple EX1032 Page 7
`
`

`

`Patent Application Publication Mar. 13, 2003 Sheet 7 of 14
`
`US 2003/0048260A1
`
`
`
`f
`
`O
`was
`lf
`
`t s
`
`IPR2022-00602
`Apple EX1032 Page 8
`
`

`

`Patent Application Publication Mar. 13, 2003 Sheet 8 of 14
`
`US 2003/0048260A1
`
`
`
`S
`
`IPR2022-00602
`Apple EX1032 Page 9
`
`

`

`Patent Application Publication Mar. 13, 2003. Sheet 9 of 14
`
`US 2003/0048260A1
`
`
`
`S
`
`s
`
`i.
`
`IPR2022-00602
`Apple EX1032 Page 10
`
`

`

`Patent Application Publication Mar. 13, 2003 Sheet 10 of 14 US 2003/0048260A1
`
`or
`y
`lf
`
`S co
`
`2,
`
`s
`
`
`
`s
`
`IPR2022-00602
`Apple EX1032 Page 11
`
`

`

`Patent Application Publication Mar. 13, 2003 Sheet 11 of 14
`
`US 2003/0048260A1
`
`
`
`IPR2022-00602
`Apple EX1032 Page 12
`
`

`

`Patent Application Publication Mar. 13, 2003 Sheet 12 of 14 US 2003/0048260A1
`
`
`
`IPR2022-00602
`Apple EX1032 Page 13
`
`

`

`Patent Application Publication Mar. 13, 2003 Sheet 13 of 14
`
`US 2003/0048260A1
`
`0I9 I
`
`
`
`
`
`JºÁBIA GIO
`
`
`
`ÇI 9.InãIH
`
`00£ I
`
`IPR2022-00602
`Apple EX1032 Page 14
`
`

`

`Patent Application Publication Mar. 13, 2003 Sheet 14 of 14
`
`US 2003/0048260 Al
`
`suonoun,]
`sdyajssuly
`
`non Ow wm
`
`
`
`JasulyxopuyJodiyrasuLjy
`
`
`
`Jasury[PPIJodnsesury
`
`
`
`IgsuLysunryjodnrosulj
`
`
`
`IaBuLyapyjodnszesury
`
`
`
`Aade,,oyuoGNWHLHOM
`
`
`
`auoud119)
`
`OOrT
`
`IPR2022-00602
`Apple EX1032 Page 15
`
`WittWH
`
`plons
`
`IPR2022-00602
`Apple EX1032 Page 15
`
`
`
`
`
`
`

`

`US 2003/0048260A1
`
`Mar. 13, 2003
`
`SYSTEMAND METHOD FOR SELECTING
`ACTIONS BASED ON THE DENTIFICATION OF
`USERS FINGERS
`
`CROSS-REFERENCE TO RELATED
`APPLICATIONS
`0001. This application is cross-referenced to and claims
`priority from U.S Provisional application No. 60/313,083
`filed Aug. 17, 2001, which is hereby incorporated by refer
`CCC.
`
`FIELD OF THE INVENTION
`0002 This invention relates generally to input devices.
`More particularly, the present invention relates to Systems
`for Selecting actions or communicating intents based on the
`identification of user's fingers through imaging.
`
`BACKGROUND
`Input devices that allow a user to select an action
`0.003
`are well known in the art and can take different forms.
`Examples of input devices are, for instance, a keyboard, a
`mouse, a touchSensor pad or panel, a Switch, a button, or the
`like. A user pressing a key on the keyboard, clicking a
`clicker on a mouse, touching a Sensor pad, flipping a Switch
`or pushing a button, could for instance establish activation
`of the input device and trigger an action. The various kinds
`of input devices are used for different types of applications
`Such as entering data in a computer-related System, operat
`ing a remote control, handling a personal data assistant,
`operating an audio-visual device, operating an instrument
`panel, which are merely examples of the different types of
`applications where input devices or Sensors are used.
`0004 One of the main problems in the art of input
`devices or Sensors is the issue of increasing functionality and
`improving user-friendlineSS while minimizing the size of the
`input device. In general, the current input devices could be
`distinguished into two categories. The first category relates
`to input devices whereby the action is independent from
`what actually caused the activation of the input device. The
`Second category relates to input devices whereby the action
`is dependent from what actually caused the activation of the
`input device.
`0005. An example of the first category of input devices
`could be illustrated through the use of a keyboard. If a user
`wants to select the letter “d” on a keyboard, then the user
`could activate the letter “d” with any finger of his/her left or
`right hand, or with any other object or device that can isolate
`the “d' key from the other keys and activate or press the “d”
`key. In other words, it does not matter what actually acti
`vates the “d' key. Therefore the action of any key on a
`keyboard is categorized as being independent from what
`actually caused the action of that particular key. Further
`more, each key on a keyboard is related to one action or
`function. As a perSon of average skill in the art would readily
`appreciate, this example merely illustrates the concept of the
`first category of input devices and this concept also applies
`to other input devices, Such as a virtual keyboard, a mouse,
`Switch, button, touchpad, touchscreen or the like.
`0006 Korth in U.S. Pat. No. 5,767,842 teaches the use of
`a virtual keyboard instead of a physical keyboard. In Korth,
`the movements of a user's fingers are interpreted as opera
`
`tions on a non-existent virtual keyboard. An image data
`acquisition System is used for monitoring positions of the
`user's fingers with respect to the virtual keys on the Virtual
`keyboard. The monitored positions of the fingers of the
`user's hand operating the Virtual keyboard are then corre
`lated to the corresponding key locations on the Virtual
`keyboard. In case of a virtual keyboard, the “d' key is only
`existent in the virtual sense as a virtual “d' key. Therefore,
`also for Korth's virtual keyboard, it does not matter what
`actually activates the virtual “d' key and the action of a key
`on a virtual keyboard is also categorized as being indepen
`dent from what caused the action of that particular virtual
`key.
`0007 One way of increasing the functionality of a key on
`any type of keyboard is to use an alternative key in combi
`nation with the “d' key. For instance, one could use the
`“shift” key in addition to the “d' key to produce capital letter
`“D’. For a keyboard or similar input device to increase the
`number of actions or functions, the number of combinations
`of keys needs to increase or the size of a keyboard needs to
`increase which both would result in an input device that is
`impractical. On the other hand it would be possible to
`decrease the size of the keypads, however, this would also be
`impractical Since the user's fingers might be getting too big
`in order to discriminate one particular key. However, in all
`Such Solutions, the action of a key, whether there are a lot of
`combinations, a lot of keys or there are a lot of keys in a
`Small Space, would still be categorized as being independent
`from what caused the action of that particular key.
`0008 Another method to increase the functionality of an
`input device is taught in cell phones. Cell phones teach one
`Solution to maximize the number actions using a key that is
`capable of generating different actions. A Single key on a cell
`phone would normally be associated with four different
`actions. For instance, Such a key could have one number,
`Such as “3' and three different letters, such as "D”, “E”, and
`“F”. The activation of "D' is based on one touch on the key,
`“E” is based on two touches on the key, “F” is based on three
`touches on the key and “3” is based on four touches on the
`key. However, as a perSon of average skill would readily
`acknowledge, Such input devices are user-unfriendly Since it
`requires a lot of effort to generate a word like for instance
`“Cell Phone'.
`0009 Bisset et al. in U.S. Pat. No. 5,825,352 teaches the
`use of multiple fingers for emulating mouse button and
`mouse operations on a touch Sensor pad. The Sensor pad
`Senses the proximity of multiple Simultaneous fingers or
`other appropriate objects to the touch Sensors. Bisset et al.
`teaches that their invention can be described in most of its
`applications by establishing one finger as controlling move
`ment of the cursor, and the Second finger as controlling
`functions equivalent to a mouse button or Switch. In this
`context according to Bisset et al., one finger may be con
`sidered the point finger, while the other finger is the click
`finger.
`0010 Although, the method taught by Bisset et al.
`teaches the possibility of using one Sensor pad to generate
`multiple actions using a combination of fingers or objects,
`there is absolutely no correlation between the combination
`of fingers or objects and the following action. For instance,
`the two fingers in Bisset et al. could be an index finger and
`thumb. However, the two fingers could also be an index
`
`IPR2022-00602
`Apple EX1032 Page 16
`
`

`

`US 2003/0048260A1
`
`Mar. 13, 2003
`
`finger and middle finger. For the method of Bisset et al. it
`does not matter which combination of fingers or even
`objects is used. Therefore, the action that results from a
`combination of fingers or objects on a Sensor pad as taught
`in Bisset et al. is also categorized as being independent from
`what actually caused the action. Furthermore, the method by
`Bisset et al. might work well for a Sensing pad on a Standard
`size notebook, it would be difficult to use the method taught
`by Bisset et al. for Small input device, e.g. where the Sensor
`or input device is Smaller than the Size of two fingers or tips
`of fingers. Consequently, the functionality would decrease
`Significantly.
`0.011) An example of the second category of input
`devices, whereby the action is dependent from what actually
`caused the activation of the input device, is taught through
`the use of a large touchscreen in U.S. Pat. No. 6,067,079 to
`Shieh who teaches a virtual pointing device for touch
`Screens. Shieh teaches that in response to the user placing
`his/her hand on a touchscreen, the touchscreeen detects the
`Sound pattern of the user's palm Site of the hand.
`0012. The areas of the touchscreen under user's hand
`then becomes activated Such that certain predefined move
`ments of the user's fingers, thumb and/or palm on those
`activated areas cause certain functions to be invoked. Shieh
`further teaches that a Single click on, for instance, a finger
`print area invokes a single function, Such as the “open’
`function.
`0013 In Shieh, the action is correlated with a part of the
`hand. However, placement of the hand can be anywhere and
`in any orientation on the touchscreen as long as touchscreen
`is able to detect the Sound pattern of the palm Site of the
`hand. The placement of the hand on the touchscreen is
`irrelevant as long as a Sound image of the palm Site of the
`hand can be obtained and the relative position e.g. a thumb
`can be distinguished using the Sounds handprint to produce
`the single action predefined for the thumb. In other words,
`the absolute position of the thumb with respect to the sensor
`or input device is irrelevant to the Selection process of an
`action, Since the relative position of the thumb to hand is
`what matters.
`0.014
`Furthermore, Shieh's method relies heavily on a
`large touch Screen to obtain the Sound hand image. It would
`therefore be difficult to apply Shieh's method in an appli
`cation with a touchscreen that is Smaller than the size of a
`hand whereby it would be impossible to obtain the sound
`handprint. If Shieh's method would be applied on a smaller
`touchscreen, the functionality of Shieh's method would
`decrease significantly, Since for example to differentiate
`between three fingers, all three fingers would have to be
`contacting the touchscreen at the same time.
`0.015
`Accordingly, with the increasing demand of
`Smaller input devices and enhancement of functionality,
`there is still a strong need to develop new Systems and
`methods that would be able to maximize the number of
`actions while minimizing the Size of the input device.
`Additionally, in many cases there is a need for a user to
`Select one out of Several actions or functions with his/her
`hands when it is impossible or unsafe to look at the input
`device. This situation arises when a user controls a car, a
`plane, or Some other machinery, and therefore (s)he has to
`look in a specific direction, which may prevent the user from
`looking at the controls. A similar need arises when the user's
`
`field of view is limited, for example while looking through
`a viewfinder, or when the input device is not visible at all,
`e.g. in the dark. In all these situations there is a need to Select
`one out Several functions with user's hands based on tactile
`feedback only, without looking at the controls.
`
`SUMMARY OF THE INVENTION
`0016. The present invention provides a system and
`method that increases the functionality of input devices and
`control panels. The System and method include a dependent
`relationship between n functions and n fingertips. The Sys
`tem and method further include an input Sensor, which is
`asSociated with the n functions. A user Selects only one of
`his/her fingertips. The Selected fingertip then touches and
`activates the input Sensor. The Selected fingertip is the only
`fingertip that is required to touch and activate the input
`Sensor, thereby allowing the input Sensor to be arbitrary
`small. Up to 8 different functions can be defined for a single
`input Sensor in which each function is correlated and depen
`dent on a fingertip of left or right hand. If multiple input
`Sensors were used in a System, the functionality of that
`System would then increase Significantly. Furthermore, the
`total number of functions for one input Sensor could be
`further increased to 10 when all the fingertips and thumbs
`are defined in the dependent relationship between functions
`and fingertips (and thumbs).
`0017. It would even be possible to further increase the
`number of possible functions for a single input Sensor. This
`could be established by having an input Sensor that is not
`only capable of detecting on/off activation as a result of a
`fingertip touching or activating the input Sensor, but also
`capable of detecting a motion that is performed by the user
`at the same time when the user activated the input Sensor. In
`general, m, . . . , m, motions could be defined respectively
`corresponding to n fingertips whereby the total number of
`Selectable functions for that Single input Sensor increases to
`
`0018 (whereby m are integers; note that n fingertips is
`also corresponding to n functions).
`0019. Once the user selects a fingertip, he/she is aware of
`the Selected function, however, the System or device on
`which the user wants to select the function is not. In order
`for the System and method of the present invention to
`determine and identify which fingertip touches and activates
`the input Sensor an imaging means is included. The imaging
`means requires the acquisition of at least one image (or
`images) of a part of the user's hand large enough to identify
`the Selected fingertip that activates the input Sensor. After the
`image is obtained, the image is processed by a processing
`means to determine which fingertip touched and activated
`the input Sensor. The present invention could further include
`a feedback means (e.g. through executing the Selected
`function, providing Sound, providing a display or the like) to
`provide the user feedback over the selected function.
`0020. In view of that which is stated above, it is the
`objective of the present invention to provide a System and
`method to Select a function from n functions on an input
`
`IPR2022-00602
`Apple EX1032 Page 17
`
`

`

`US 2003/0048260A1
`
`Mar. 13, 2003
`
`Sensor, whereby the input Sensor is associated with the n
`functions and whereby the n functions corresponds to n
`fingertips.
`0021. It is another objective of the present invention to
`provide an input Sensor that is capable of detecting m, . . .
`, m, motions respectively corresponding to n fingertips
`whereby the total number of selectable functions for the
`input Sensor increases to
`
`0022. It is yet another objective of the present invention
`to Select a function by Selecting only one fingertip at a time
`and only the Selected fingertip touches and activates the
`input Sensor.
`0023. It is still another objective of the present invention
`to provide input Sensors that are arbitrary Small or input
`Sensors that are Substantially as Small as the Selected fin
`gertip.
`It is still another objective of the present invention
`0024.
`to provide input Sensors that are Substantially larger than the
`Selected fingertip, which touches and activates the input
`SCSO.
`0.025. It is still another objective of the present invention
`to provide input sensors with tactile stimuli.
`0026. It is still another objective of the present invention
`to provide a system and method in which it would be
`possible to Successfully Select a function in case the user is
`prevented from looking at the input Sensor or the Selected
`fingertip while the user Selects and activates the input Sensor.
`0027. It is still another objective of the present invention
`to provide an imaging means to image a part of Said user's
`hand large enough to identify the Selected fingertip that
`activates the input Sensor.
`0028. It is still another objective of the present invention
`to provide a processing means to determine the Selected
`function from the identified fingertip by the imaging means
`and the dependent relationship between the n functions and
`the n fingertips.
`0029. It is still another objective of the present invention
`to provide a processing means to determine the Selected
`function from the identified fingertip by the imaging means
`and the dependent relationship between the n fingertips and
`m, . . . , m, motions corresponding to the n fingertips.
`0030 The advantage of the present invention over the
`prior art is that the present invention enables one to increase
`the functionality of Systems without necessarily increasing
`the number of input devices or input Sensors. Another
`advantage of the present invention is that it allows a manu
`facturer to develop Systems that maximizes the number of
`possible functions or actions of the System while minimizing
`the size of the System. Still another advantage of the present
`invention is that it would allow a user to use tactile infor
`mation from touching the Sensor with the Selected fingertip,
`to Select a function from a plurality of functions without
`looking at the controls.
`
`BRIEF DESCRIPTION OF THE FIGURES
`0031. The objectives and advantages of the present
`invention will be understood by reading the following
`detailed description in conjunction with the drawings, in
`which:
`0032 FIG. 1 shows an example of a dependent relation
`ship between fingertips and functions according to the
`present invention;
`0033 FIG. 2 shows an example of the method steps for
`Selecting a function based on the Selection of the corre
`sponding fingertip according to the present invention;
`0034 FIG. 3 shows an example of a dependent relation
`ship between fingertips, motions and functions according to
`the present invention;
`0035 FIG. 4 shows an example of the method steps for
`Selecting a function based on the Selection of the corre
`sponding fingertip and motion according to the present
`invention;
`0036 FIGS. 5-10 show examples of different types of
`possible input Sensors according to the present invention.
`FIGS. 5-10 also show exemplary selections of a fingertip to
`touch and activate the input Sensors according to the present
`invention;
`0037 FIG. 11 shows an example of the system according
`to the present invention;
`0038 FIG. 12 shows an example of an image acquired
`through the imaging means according to the present inven
`tion; and
`0039 FIGS. 13-14 show examples of how the system and
`method of the present invention could be applied.
`
`DETAILED DESCRIPTION OF THE
`INVENTION
`0040 Although the following detailed description con
`tains many Specifics for the purposes of illustration, anyone
`of ordinary skill in the art will readily appreciate that many
`variations and alterations to the following exemplary details
`are within the Scope of the invention. Accordingly, the
`following preferred embodiment of the invention is set forth
`without any loSS of generality to, and without imposing
`limitations upon, the claimed invention.
`0041. The present invention provides a system and
`method 100 for selecting a function from a plurality of
`functions with his/her fingertip. In general, there could be n
`functions whereby each of the n functions corresponds with
`in fingertips. For the purpose of the present invention,
`function has the same meaning as action or intent. AS it is
`shown in FIG. 1, there is a dependent relationship between
`each fingertip and the corresponding function. The least
`number of dependent relationships is 2, i.e. when n is 2. The
`example shown in FIG. 1 shows the fingertips of the left and
`right hand. Including all the fingertips it would be possible
`to define a maximum of 8 different functions, i.e. when n is
`8. The determination of which fingertip should correspond to
`which function is completely arbitrary and Simply a matter
`of choice or preference. The correspondence, i.e. the depen
`dent relationship, between fingertip and function is usually
`preset in a System by the manufacturer. However, it is also
`possible for the manufacturer to allow the user of the system
`
`IPR2022-00602
`Apple EX1032 Page 18
`
`

`

`US 2003/0048260A1
`
`Mar. 13, 2003
`
`to define this corresponding relationship, as he/she prefers.
`Furthermore, the total number of functions could be
`increased to 10 if one also includes the thumb of the left and
`right hand as shown in FIG. 1.
`0042. As it is shown in FIG. 2, the key idea of the system
`and method 200 of the present invention is that a user selects
`210 only one fingertip at a time. The user is aware of the
`particular function that corresponds to the Selected fingertip.
`With the Selected fingertip, i.e. only the Selected fingertip,
`the user touches and activates 220 an input Sensor. It is
`important to realize that the user is not using his/her other
`fingertips when touching the input Sensor. This offers great
`advantages to Systems and methods in which it would now
`be possible to maximize the number of functions while
`minimizing the size of the input Sensor. With a Single input
`Sensor, a manufacturer of the device or System has the
`opportunity to define up to 10 different functions, i.e. when
`n is 10, which correspond to different fingertips for a single
`input Sensor. This would not only increase the functionality
`of the System, it would also make the Selection proceSS
`easier as well as it would decrease potential injuries Such as
`repetitive Strain injuries associated with repetitive typing or
`pressing.
`0043. Once the user selects a fingertip, he/she is aware of
`the Selected function, however, the System or device on
`which the user wants to Select the function is not. Imaging
`230 is used in order for the system and method of the present
`invention to determine and identify which fingertip touches
`and activates the input Sensor. Imaging 230 requires at least
`one image of a part of the user's hand large enough to
`identify the Selected fingertip that activates the input Sensor.
`After the image is obtained, the image is processed 240 to
`determine which fingertip touched and activated the input
`Sensor (more details about imaging and processing are
`provided infra). Processing includes that the identified fin
`gertip based on imaging is compared in a look-up table. The
`look-up table contains the dependent relationship between
`the fingertips and functions in order to determine the cor
`responding function for the identified fingertip.
`0044) Understanding the concept of the present invention
`described so far, it would be possible to further increase the
`number of possible functions for a single input Sensor. This
`is established by having an input Sensor that is not only
`capable of detecting on/off activation, but also capable of
`detecting a motion that is performed by the user at the same
`time when the user activated the input Sensor. For only one
`fingertip one could then define p motions for a Single input
`Sensor (whereby p is an integer). In general, m, . . . , m,
`motions could be defined respectively corresponding to n
`fingertips whereby the total number of selectable functions
`for that Single input Sensor increases to
`
`0045 (whereby m are integers; note that n fingertips is
`also corresponding to n functions as discussed Supra with
`respect to FIGS. 1-2). FIG. 3 shows an example of two
`different fingertips for the right hand whereby each fingertip
`corresponds to an upward motion and a downward motion.
`
`By having two fingertips (i.e. when n is 2) and two different
`motions for each fingertip (i.e. when m is 2 and m is 2) the
`total number of different functions is then 4, i.e. m+m=4.
`FIG. 4 shows a system and method 400 that is similar to
`system and method 200 as it is discussed Supra and with
`respect to FIG. 2. The difference between FIG. 2 and FIG.
`4 is the addition of providing motion 410 by the selected
`fingertip. Since a function is now dependent on the Selected
`fingertip and the provided motion by the Selected fingertip,
`processing 420 now further includes determining the func
`tion that corresponds to the identified fingertip based on
`imaging 230. A look-up table that contains the dependent
`relationship between the fingertips, motions and functions is
`used to determine the functions given the identified finger
`tip.
`0046) The input sensor could be an arbitrary small input
`Sensor. The input Sensor could also be Substantially as Small
`as or Smaller than the Selected fingertip. Input Sensor could
`include any kind of electrical elements or heat-conducting
`elements to either Sense binary on/off activation and/or
`resistive membrane position elements or position Sensor
`elements to Sense motion. Input Sensors could therefore take
`different forms Such as, for instance, but not limited to, a
`keypad, button, a contact point, a Switch, a touchscreen, a
`trackpad, or a heat-conducting pad. Although for Some
`applications it would be preferred and advantageous to
`utilize a Small input Sensor, Such as a Small keypad, the
`present invention is not limited to the use of a Small input
`Sensor. The concept of the present invention would also
`work for large input sensors. It would for instance be easier
`for a user to locate a large input Sensor, large input Sensors
`would be advantageous for the applications when the user
`has to Select one out of a plurality of functions without
`looking at the input Sensor, based on the tactile feedback
`only. These large input sensors (e.g. Substantially larger than
`the area of a fingertip) would be equipped with a coordinate
`location mechanism (such as in laptop trackpads) for iden
`tifying the coordinate of the contact point of the Selected
`fingertip with the input sensor, which would then be used by
`the image recognition algorithm.
`0047 FIGS. 5-10 show different examples of input sen
`sors or devices. FIG. 5 shows the dorsal site of a user's right
`hand 510. User's right hand 510 shows the dorsal part 511
`of the hand which is opposite from the palm of the hand,
`thumb 512, index finger 513, middle finger 514, ring finger
`515, and little finger 516. Thumb 512, index finger 513, ring
`finger 515, and little finger 516 are shown in a flexed
`position (i.e. bringing the fingertips in a direction toward the
`palm site of the hand), whereas index finger 513 is in an
`extended position, Substantially extended position or par
`tially flexed position. It would only be necessary for the
`non-Selected fingers to not obscure the View of the Selected
`finger by the imaging device; thus the non-Selected fingers
`can also be in Substantially extended or partially flexed
`position. In the example of FIG. 5, the user has selected
`fingertip. 513-FT of index finger 513 to touch and activate
`input sensor 520. Input sensor 520 could be a keypad, a
`Switch or a button. It should be noted that the size of input
`sensor 520 (530 shows a top view of input sensor 520) in this
`example is substantially as Small as fingertip. 513-FT.
`0048 FIG. 6 shows a similar example as in FIG. 5 with
`the difference that the user has selected fingertip. 514-FT of
`middle finger 514 to touch and activate input sensor 520. In
`
`IPR2022-00602
`Apple EX1032 Page 19
`
`

`

`US 2003/0048260A1
`
`Mar. 13, 2003
`
`the example of FIG. 6, the user has selected fingertip
`514-FT of middle finger 514 to touch and activate input
`sensor 710. Input sensor 710 could be an arbitrary small
`input device or sensor. It should be noted that the size of
`input sensor 710 (720 shows a top view of input sensor 710)
`in this example is substantially smaller than fingertip. 514
`FT.
`0049 FIG. 8 shows an example of multiple input sensors
`820 that are distributed on top of a support surface 810. In
`the example of FIG. 8, the user has selected (1) fingertip
`513-FT of index finger 513 and (2) input sensor 822 out of
`all 12 input sensors 820 to touch and activate input sensor
`822. In this example, input sensors 820 are shown are
`keypads or buttons. It should be noted that the size of input
`sensors 820 (830 shows a top view of input sensors 820) in
`this example are each Substantially as Small as fingertip
`51.3-FT.
`0050 FIG. 9 shows input sensors 920 distributed in a
`similar fashion as in FIG. 8 with the difference that input
`sensors 920 are now underneath a surface 910. An example
`of Support surface 910 is a touchscreen, whereby input
`sensors 920 are distributed underneath the touchscreen. In
`the example of FIG. 9, the user has selected (1) fingertip
`513-FT of index finger 513 and (2) input sensor 922 out of
`all 12 input sensors 920 to touch and activate input sensor
`922. Surface 910 could be transparent so that the user has the
`opportunity to recognize the location of each of the input
`sensors 920, or surface 910 could has markings or illustra
`tions to help visualize and/or localize where the user should
`touch Surface 910 in order to select the intended input
`sensor. It should be noted that the size of input sensors 920
`(930 shows a top view of input sensors 920) in this example
`are each substantially as small as fingertip. 513-FT.
`0051 FIGS. 5-9 show examples in which the user could
`activate the input Sensor with a fingertip either by pressing
`the input Sensor, touching the input Sensor, flipping the input
`Sensor, bending the input Sensor, or the l