111111
`
`1111111111111111111111111111111111111111111111111111111111111111111111111111
`US 20090326406Al
`
`(19) United States
`c12) Patent Application Publication
`Tan et al.
`
`(10) Pub. No.: US 2009/0326406 Al
`Dec. 31, 2009
`(43) Pub. Date:
`
`(54) WEARABLE ELECTROMYOGRAPHY-BASED
`CONTROLLERS FOR HUMAN-COMPUTER
`INTERFACE
`
`(75)
`
`Inventors:
`
`Desney Tan, Kirkland, WA (US); T.
`Scott Saponas, Seattle, WA (US);
`Dan Morris, Bellevue, WA (US);
`Jim Turner, Monroe, WA (US)
`
`Correspondence Address:
`MICROSOFT CORPORATION
`ONE MICROSOFT WAY
`REDMOND, WA 98052 (US)
`
`(73) Assignee:
`
`MICROSOFT CORPORATION,
`Redmond, WA (US)
`
`(21) Appl. No.:
`
`12/404,223
`
`(22) Filed:
`
`Mar. 13, 2009
`
`Related U.S. Application Data
`
`(63) Continuation-in-part of application No. 12/146,471,
`filed on Jun. 26, 2008.
`
`Publication Classification
`
`(51)
`
`Int. Cl.
`(2006.01)
`A61B 510488
`(2006.01)
`H03K 17194
`(52) U.S. Cl. ........................................... 600/546; 341/20
`(57)
`ABSTRACT
`
`A "Wearable Electromyography-Based Controller" includes
`a plurality of Electromyography (EMG) sensors and provides
`a wired or wireless human-computer interface (HCl) for inter(cid:173)
`acting with computing systems and attached devices via elec(cid:173)
`trical signals generated by specific movement of the user's
`muscles. Following initial automated self-calibration and
`positional localization processes, measurement and interpre(cid:173)
`tation of muscle generated electrical signals is accomplished
`by sampling signals from the EMG sensors of the Wearable
`Electromyography-Based Controller. In operation, the Wear(cid:173)
`able Electromyography-Based Controller is donned by the
`user and placed into a coarsely approximate position on the
`surface of the user's skin. Automated cues or instructions are
`then provided to the user for fine-tuning placement of the
`Wearable Electromyography-Based Controller. Examples of
`Wearable Electromyography-Based Controllers
`include
`articles of manufacture, such as an armband, wristwatch, or
`article of clothing having a plurality ofintegratedEMG-based
`sensor nodes and associated electronics.
`
`APPLY SENSOR NODES OR
`ARTICLE(S) HAVING INTEGRAL
`SENSOR NODES TO APPROXIMATE
`POSITIONS ON BODY
`100 _;
`:
`
`1 o5
`
`.-------------------------·
`
`:
`: USER GESTURE MODULE:
`:(DIRECT USER TO PERFORM:
`: PARTICULAR GESTURES
`:
`:
`AND/OR MOTIONS)
`:
`------------t-----c~~;-
`
`·-----------_'f------------.
`.---------------.X-------c.-----.
`:
`:
`SELF-CALIBRATION MODULE:
`:
`:
`'
`:
`(ASSOCIATE MUSCLE GENERATED
`:ELECTRICAL SIGNALS WITH SPECIFIC~-~
`:
`GESTURES AND/OR MOTIONS)
`:
`:
`~~~~~~~~~~~~~~~]~~~~~~~~~~~~~~~~ 1 :~~----_-_-_-_-_]_-_-_-_-_-_-_-_-_-_-_-_~
`
`TRAINING MODULE
`
`:
`:
`:
`:
`
`: USER FEEDBACK MODULE: :
`: POSITIONAL LOCALIZATION MODULE: :
`,
`(DETERMINE BODY POSITIONS
`:. • .; (DIRECT REPOSITIONING OF,
`:
`AND RELATIVE POSITIONS
`•
`:
`SENSOR NODES)
`:
`OF SENSOR NODES)
`:
`:
`:
`:
`
`:
`
`:
`
`- ;;;::;- ------- ·:----- -----------· 130 ------------r---....::.::;;;--
`.-------------___ '( _____________ c.
`:
`: ONBOARD MULTIPLEXING MODULE:
`r----------------'
`(DETERMINE WHICH
`:
`:
`SENSOR NODES TO USE)
`:
`----------------1----------------·
`----------------i---------..c..~~:
`:
`POWER CONTROL MODULE
`:
`!. -------------- ·:----------------~ 40
`f
`T
`SIGNAL CAPTURE MODULE:
`(MEASURE MUSCLE GENERATED
`ELECTRICAL SIGNALS USING
`SELECTED SENSOR NODES)
`
`150
`
`!
`
`r-145
`
`CONTROL MODULE:
`HCI CONTROL OF COMPUTING
`DEVICES AND/OR OTHER DEVICES
`BASED ON MEASURED MUSCLE
`GENERATED ELECTRICAL SIGNALS
`'
`. ---------------i------------ -C.160
`! ~~~i~~ri~~~~:~i~~~~ !
`--------------------------------_.
`
`INITIAL SETUP AND USE OF THE WEARABLE
`ELECTROMYOGRAPHY-BASED CONTROLLER
`
`1 of 26
`
`FITBIT EXHIBIT 1007
`
`

`
`Patent Application Publication
`
`Dec. 31, 2009 Sheet 1 of 9
`
`US 2009/0326406 A1
`
`~------------------------1
`: USER GESTURE MODULE:
`:
`:(DIRECT USER TO PERFORM:
`: PARTICULAR GESTURES
`:
`:
`AND/OR MOTIONS)
`:
`------------~-----\::~~;-
`·------------~------------.
`:
`1
`TRAINING MODULE
`
`APPLY SENSOR NODES OR
`ARTICLE(S) HAVING INTEGRAL
`SENSOR NODES TO APPROXIMATE
`POSITIONS ON BODY
`1oo--'
`:
`105
`·---------------~-------1::-----·
`:
`SELF-CALIBRATION MODULE:
`:
`:
`(ASSOCIATE MUSCLE GENERATED
`:
`:ELECTRICAL SIGNALS WITH SPECIFIC~-~
`:
`GESTURES AND/OR MOTIONS)
`:
`:
`1
`1;0::;-------~------------
`l ____________ f ____________ l
`l·--------------~----------------
`: POSITIONAL LOCALIZATION MODULE: :
`: USER FEEDBACK MODULE: :
`:.-~(DIRECT REPOSITIONING OF 1
`(DETERMINE BODY POSITIONS
`1
`AND RELATIVE POSITIONS
`SENSOR NODES)
`:
`:
`1
`:
`:
`OF SENSOR NODES)
`:
`:
`:
`·;;;::;---------:----------------~ 130 ------------~---~·;;;--
`.----------------~-------------!::
`:
`:
`:
`I ONBOARD MULTIPLEXING MODULE:
`r----------------1
`1
`I
`:
`(DETERMINE WHICH
`:
`:
`SENSOR NODES TO USE)
`----------------~----------------·
`135
`----------------Y----------~--.
`:
`POWER CONTROL MODULE
`:
`~---------------~----------------·
`•
`140
`I
`
`· - - - - - - - - - - - - - - - , - - - - - - - - - - - - - - - -
`
`I
`I
`
`SIGNAL CAPTURE MODULE:
`(MEASURE MUSCLE GENERATED
`ELECTRICAL SIGNALS USING
`SELECTED SENSOR NODES)
`
`150
`
`155
`
`~
`
`,145
`
`CONTROL MODULE:
`HCI CONTROL OF COMPUTING
`DEVICES AND/OR OTHER DEVICES
`BASED ON MEASURED MUSCLE
`GENERATED ELECTRICAL SIGNALS
`
`I
`
`. I
`160
`·---------------lr.-------------~
`:
`CONTROL FEEDBACK MODULE:
`:
`:
`(PROVIDE HAPTIC, VISUAL OR
`:
`:
`AUDIBLE FEEDBACK USER)
`--------------------------------J
`1
`INITIAL SETUP AND USE OF THE WEARABLE
`ELECTROMYOGRAPHY-BASED CONTROLLER
`FIG. 1
`
`2 of 26
`
`

`
`Patent Application Publication Dec. 31, 2009 Sheet 2 of 9
`
`US 2009/0326406 Al
`
`EXAMPLE OF ARMBAND WITH INTEGRAL EMG
`SENSOR NODES FOR IMPLEMENTING THE
`WEARABLE ELECTROMYOGRAPHY -BASED
`CONTROLLER
`
`235
`230
`
`FIG. 2
`
`3 of 26
`
`

`
`Patent Application Publication Dec. 31, 2009 Sheet 3 of 9
`
`US 2009/0326406 Al
`
`SENSOR NODES (360)
`ON HEAD (370)
`
`SENSOR NODES (300)
`ON ARM (310)
`
`SENSOR NODES (340)
`ON CHEST (350)
`
`SENSOR NODES (320)
`ON LEG (330)
`
`)::t j::t
`_____ )::t )::t
`~
`)::t
`
`FIG. 3
`
`4 of 26
`
`

`
`Patent Application Publication Dec. 31, 2009 Sheet 4 of 9
`
`US 2009/0326406 Al
`
`.-----------------------------------------------
`------~
`~-----···
`POWER MANAGEMENT
`i
`:
`:
`MODULE
`!
`'
`'
`'
`L •• •• •• , •• •• •• ••••• •• •• ••••• •• •• ••••• •• •• ••••• ·'
`490 _I
`....-----------1 + POWER SOURCE (BATTERY,
`FUEL CELL, WIRED, ETC.)
`
`490
`
`EXEMPLARY
`EMG SENSOR NODE
`BLOCK DIAGRAM
`
`_______ c.~~~-----.
`'--··--·---~-~-f~~-----··.)
`
`/ BIO-STIMULUS •• ••
`
`i- ----------- __ .. __ ------------.
`'
`'
`'
`'
`:
`:
`CONTROL
`i
`-------------------------\. _;-·-----------~
`PROCESSOR
`:
`'
`~ ------------------------,---:
`'-485
`
`, .... __ _c435
`LOW-PASS i ·::::c .. • i
`FILTER!, -..:.· i
`----l·--·
`'
`'
`'
`'
`
`-----.. -----
`~
`
`, •••• ___ r.-440
`i •::::c_ .• i LOW-PASS
`i' -..:.· i FILTER
`----l·--·
`'
`'
`'
`'
`r4 45
`
`ANALOG TO DIGITAL CONVERTER
`
`+
`
`r 450
`
`DIGITAL SIGNAL PROCESSING MODULE
`
`.&
`
`·------------------------t _______________________ _
`i WIRELESS COMMUNICATIONS i
`i
`i
`MODULE (RF, IR, ETC.)
`4 5J:T· -------~~ ~~ ~~ ~~~ J~ ~~ ~~~ ~~ ~~ ~-------------:
`
`~460
`:
`!
`! ANTENNA
`'
`'
`'
`'
`~- ------------------.. !
`
`. ______________________ .. t. __________________ .. C 4 6 5
`.&
`i
`i
`DIRECT WIRED
`i
`i COMMUNICATIONS MODULE
`t _______________________ • ________________________ :
`
`·----------t ----------.
`! DATA CABLE r470
`
`'
`'
`'
`'
`'
`'
`~- -------------------- J
`
`FIG. 4
`
`5 of 26
`
`

`
`Patent Application Publication Dec. 31, 2009 Sheet 5 of 9
`
`US 2009/0326406 Al
`
`EXEMPLARY ELECTRICAL
`STIMULUS CIRCUIT
`
`Inductor
`
`-10uH
`
`520\
`
`Resistor r 530
`
`1
`
`Transistor
`
`__ .....____
`
`Capacitor
`- 6pF
`
`- 4.7k n
`
`500 _! N
`
`r550
`
`-25V
`
`r540
`
`Electrode
`
`-=- Electrode
`
`FIG. 5
`
`EXEMPLARY ELECTRICAL
`STIMULUS WAVEFORM
`
`•••
`
`\ \ 600 ~----------------------------------------~-----------
`
`ToFF
`
`FIG. 6
`
`6 of 26
`
`

`
`Patent Application Publication Dec. 31, 2009 Sheet 6 of 9
`
`US 2009/0326406 Al
`
`EFFECTS OF NODE POSITION ON BODY
`
`RELATIVELY FLAT SURFACE
`OF BODY CAUSING NODES
`TO COUPLE WELL
`
`CURVATURE OF BODY
`CAUSING NODES TO
`NOT COUPLE WELL.
`
`NODES ON ABOUT THE
`SAME PLANE COUPLING
`WELL, NOT SENSITIVE
`TO ROTATION
`
`NODES AROUND BODY
`TISSUE HAVE WEAK RF
`LINK DUE TO ABSORBTION
`OF RADIATED ENERGY
`
`BODY
`
`FIG. 7
`
`7 of 26
`
`

`
`Patent Application Publication Dec. 31, 2009 Sheet 7 of 9
`
`US 2009/0326406 Al
`
`EXEMPLARY USE OF SOUND
`SOURCE FOR POSITIONAL
`CALl BRA TION OF EMG SENSORS
`
`FIG. 8
`
`8 of 26
`
`

`
`Patent Application Publication Dec. 31, 2009 Sheet 8 of 9
`
`US 2009/0326406 Al
`
`905
`
`910
`
`~~930
`
`915
`
`;I
`
`935 ~D
`
`945 _)
`
`WIRELESS EMG SENSORS IN
`COMMUNICATION WITH EXEMPLARY
`HUBS AND/OR COMPUTING DEVICES
`
`FIG. 9
`
`9 of 26
`
`

`
`Patent Application Publication Dec. 31, 2009 Sheet 9 of 9
`
`US 2009/0326406 Al
`
`1000
`
`GENERAL PURPOSE COMPUTER
`
`1010
`
`PROCESSING
`UNIT(S)
`
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`I
`I
`
`:~~~::\..------·
`: REMOVABLE :
`,_-------------·
`:
`: STORAGE
`,--------------
`NON-
`:
`1
`: REMOVABLE :
`:
`: STORAGE
`
`-----'C 1080--
`
`1020
`
`I
`
`--......1'-----
`
`SYSTEM
`MEMORY
`
`i-------------------------------~
`1015
`:
`,••••••••••.£_
`I
`I
`I
`I
`I
`I
`I
`----~
`GPU(S)
`:
`:
`I
`I
`, _____________ )
`I
`I
`~------------------
`
`·-----------------1
`·---i
`:
`,------1'------·
`,------1'------·
`I
`I
`I
`I
`:
`INPUT
`:
`:
`OUTPUT
`:
`: DEVICE(S)
`:
`: DEVICE(S)
`:
`'------f-\.:. ~ ~40 ,_ -----f-- -c~ 050
`I
`I
`I
`I
`
`COMMUNICATIONS
`INTERFACE
`
`1030
`
`I
`I
`I
`I
`
`'f
`
`I
`I
`I
`I
`
`'f
`
`FIG. 10
`
`10 of 26
`
`

`
`US 2009/0326406 AI
`
`Dec. 31, 2009
`
`1
`
`WEARABLE ELECTROMYOGRAPHY-BASED
`CONTROLLERS FOR HUMAN-COMPUTER
`INTERFACE
`
`CROSS-REFERENCE TO RELATED
`APPLICATIONS
`
`[0001] This application is a Continuation-in-Part (CIP) of
`U.S. patent application Ser. No. 12/146,471, filed on Jun. 26,
`2008, and entitled "RECOGNIZING GESTURES FROM
`FOREARM EMG SIGNALS."
`
`BACKGROUND
`
`[0002]
`1. Technical Field
`[0003] A "Wearable Electromyography-Based Controller"
`provides a physical device, worn by or otherwise attached to
`a user, that directly senses and decodes electrical signals
`produced by human muscular activity using surface Elec(cid:173)
`tromyography (sEMG) sensors. The resulting electrical sig(cid:173)
`nals provide a muscle-computer interface for use in control(cid:173)
`ling or interacting with one or more computing devices or
`other devices coupled to a computing device.
`[0004] 2. Related Art
`[0005]
`In general, as is well known to those skilled in the
`art, Electromyography (EMG) measures the muscle electrical
`activity during muscle contractions as an electrical potential
`between a ground electrode and a sensor electrode. EMG can
`measure signals either directly within the muscle (invasive
`EMG) or on the skin above a muscle (surface EMG).
`[0006]
`Invasive EMG is very accurate in sensing muscle
`activation, but is generally considered to be impractical for
`human-computer interaction applications as it requires
`needle electrodes to be inserted through the skin and directly
`into the muscle fibers. In contrast, surface EMG, while less
`accurate, only requires that conductive sensors be placed on
`the surface of the skin. Surface EMG is fundamentally noisier
`than invasive EMG since motor unit action potentials
`(MUAPs) must pass though body tissues such as fat and skin
`before they can be captured by a sensor on the surface. Due to
`the high sensitivity of EMG sensors required to detect these
`signals, they also typically detect other electrical phenomena
`such as activity from other muscles, skin movement over
`muscles, and environmental noise, etc.
`[0007] The EMG signal is an electrical potential, or volt(cid:173)
`age, changing over time. The raw signal is an oscillating wave
`with an amplitude increase during muscle activation. Most of
`the power of this signal is contained in the frequency range of
`5 to 250 Hz. A typical statistic computed over the raw EMG
`signal for diagnosis of muscle activity is the windowed root
`mean squared (RMS) amplitude of the measured potential.
`This RMS measure of EMG signals has typically been
`employed for diagnostic purposes such as evaluating muscle
`function during rehabilitation after a surgery or for measuring
`muscle activation to assess gait. RMS amplitude is a rough
`metric for how active a muscle is at a given point in time.
`Consequently, since most EMG-based applications have
`originated and are used in medical and/or clinical settings,
`certain assumptions are generally made about preparation
`and setup ofEMG measurement devices, and about the mea(cid:173)
`surement and processing of EMG signals.
`[0008] For example, since the medical utility of attaining
`the best possible signal is high, there is typically no perceived
`need to reduce the cost of preparation and setup at the cost of
`signal accuracy. Specifically, in setting up EMG devices in a
`
`clinical setting, the skin is typically first cleaned with an
`abrasive so that dead skin cells are removed. EMG sensors are
`then typically carefully placed by an expert, who can locate
`the exact locations of muscle bellies and find the optimal
`placement. Further, in some cases, a current is then applied
`through the sensors to test sensor placement accuracy. For
`example, if the electrode is placed on a muscle that is
`expected to control a particular finger and a current is applied
`to the muscle via the electrode in the EMG sensor, the
`expected finger should twitch, if not, then sensor would be
`relocated to the correct position.
`[0009] Further, since EMG sensors are generally carefully
`placed in clinical settings, they are usually treated as being
`static (i.e., mapped directly to specific muscles of interest).
`Consequently, clinicians tend to place many constraints on
`users/patients in these scenarios. For example, the users/pa(cid:173)
`tients may not be allowed to move their bodies in certain ways
`(e.g., rotating the arm, since this would move the surface
`sensors away from the muscles of interest).
`[0010] Human-computer interfaces (HCI) have been pri(cid:173)
`marily implemented by monitoring direct manipulation of
`devices such as mice, keyboards, pens, dials, touch sensitive
`surfaces, etc. However, as computing and digital information
`becomes integrated into everyday environments, situations
`arise where it may be inconvenient or difficult to use hands to
`directly manipulate an input device. For example, a driver
`attempting to query a vehicle navigation system might find it
`helpful to be able to do so without removing his or her hands
`from the steering wheel. Further, a person in a meeting may
`wish to unobtrusively and perhaps invisibly interact with a
`computing device. Unfortunately, the general assumptions
`described above with respect to the setup and use of conven(cid:173)
`tional EMG sensors and signal measurement tend to make the
`use and setup of conventional EMG systems impractical for
`typical HCI purposes which allow a user to control and inter(cid:173)
`act with computing systems, applications, and attached
`devices.
`
`SUMMARY
`
`[0011] This Summary is provided to introduce a selection
`of concepts in a simplified form that are further described
`below in the Detailed Description. This Summary is not
`intended to identifY key features or essential features of the
`claimed subject matter, nor is it intended to be used as an aid
`in determining the scope of the claimed subject matter.
`[0012] A "Wearable Electromyography-Based Controller"
`as described herein, provides a hardware device or devices,
`implemented in various form factors, which is either worn by
`the user or temporarily attached to the user's body. In com(cid:173)
`bination with associated initialization and configuration soft(cid:173)
`ware and user interface techniques, the Wearable Elec(cid:173)
`tromyography-Based Controller provides a human computer
`interface (HCI) device that allows the user to control and
`interact with computing systems and attached devices via
`electrical signals generated by the movement of the user's
`muscles following initial automated self-calibration and posi(cid:173)
`tiona! localization processes. In other words, the Wearable
`Electromyography-Based Controller provides a user wear(cid:173)
`able muscle-computer interface (muCI).
`[0013]
`In general, the Wearable Electromyography-Based
`Controller includes one or more integrated Electromyogra(cid:173)
`phy (EMG) sensor nodes. The EMG sensor nodes within the
`Wearable Electromyography-Based Controller measure
`muscle electrical activity for use in muscle-computer inter-
`
`11 of 26
`
`

`
`US 2009/0326406 AI
`
`Dec. 31, 2009
`
`2
`
`action applications. However, unlike conventional Elec(cid:173)
`tromyography (EMG) measurement systems, the Wearable
`Electromyography-Based Controller described herein
`requires only general positional placement on the user's body.
`In fact, this general placement of the Wearable Electromyo(cid:173)
`graphy-Based Controller is enabled by including more EMG
`sensors than are expected to be necessary to measure muscle
`electrical activity. An automated positional localization pro(cid:173)
`cess is then used to automatically identify and select a subset
`of some or all of the sensor nodes that are in an appropriate
`position to collect muscle electrical signals corresponding to
`particular user gestures or movements.
`[0014] More specifically, because various embodiments of
`the Wearable Electromyography-Based Controller includes
`an excess ofEMG sensors, the initial positional localization
`process allows the overall system to self-select a set of one or
`more appropriate sensor nodes within the Wearable Elec(cid:173)
`tromyography-Based Controller in order to capture appropri(cid:173)
`ate muscle electrical signals for controlling and interacting
`with computing systems and attached devices.
`[0015] As noted above, the Wearable Electromyography(cid:173)
`Based Controller can be implemented in various forms,
`including wearable devices or articles of clothing. For
`example, the Wearable Electromyography-Based Controller
`can be implemented as an armband, a wristwatch, eyeglasses
`(with sensors integrated into the frame), a shirt, gloves, or
`other article of clothing worn by the user, or any other physi(cid:173)
`cal device or collection of devices worn by the user. Further,
`it should also be understood that a user can wear multiple
`Wearable Electromyography-Based Controllers, with each
`such Wearable Electromyography-Based Controller being
`used to interact with either the same or a different computing
`device or application.
`[0016]
`In view of the above summary, it is clear that the
`Wearable Electromyography-Based Controller described
`herein provides a unique device for measuring user muscle
`electrical activity for interacting with and controlling one or
`more computing devices following an initial positional local(cid:173)
`ization process for self-selecting a set of appropriate EMG
`sensor nodes to capture the electrical activity associated with
`particular user gestures or motions. In addition to the just
`described benefits, other advantages of the Wearable Elec(cid:173)
`tromyography-Based Controller will become apparent from
`the detailed description that follows hereinafter when taken in
`conjunction with the accompanying drawing figures.
`
`DESCRIPTION OF THE DRAWINGS
`
`[0017] The specific features, aspects, and advantages of the
`claimed subject matter will become better understood with
`regard to the following description, appended claims, and
`accompanying drawings where:
`[0018] FIG. 1 provides an exemplary architectural flow
`diagram that illustrates program modules for implementing
`various embodiments of the Wearable Electromyography(cid:173)
`Based Controller, as described herein.
`[0019] FIG. 2 provides an example of a wireless embodi(cid:173)
`ment of the Wearable Electromyography-Based Controller
`implemented in the form of an armband, as described herein.
`[0020] FIG. 3 illustrates individual EMG-based sensor
`nodes coupled to various parts of the user's body for imple(cid:173)
`menting various embodiments of the Wearable Electromyo(cid:173)
`graphy-Based Controller, as described herein.
`
`[0021] FIG. 4 illustrates an exemplary block diagram
`showing general functional units of individual EMG-based
`sensor nodes for implementing various embodiments of the
`Wearable Electromyography-Based Controller, as described
`herein.
`[0022] FIG. 5 illustrates an exemplary electrical stimulus
`circuit as a component of the EMG-based sensor node, for
`implementing various embodiments of the Wearable Elec(cid:173)
`tromyography-Based Controller, as described herein.
`[0023] FIG. 6 illustrates an exemplary electrical stimulus
`waveform generated by the electrical stimulus circuit of FIG.
`5, for implementing various embodiments of the Wearable
`Electromyography-Based Controller, as described herein.
`[0024] FIG. 7 illustrates the effect of placing sensor nodes
`in different relative positions on the user's body with respect
`to the use of radio-frequency (RF) based positional localiza(cid:173)
`tion of sensor nodes, as described herein.
`[0025] FIG. 8 illustrates the use of a sound source for pro(cid:173)
`viding positional localization of sensor nodes, as described
`herein.
`[0026] FIG. 9 illustrates the use of wireless EMG-based
`sensor nodes with various types of hubs and/or computing
`devices, for implementing various embodiments of the Wear(cid:173)
`able Electromyography-Based Controller, as described
`herein.
`[0027] FIG. 10 is a general system diagram depicting a
`simplified general-purpose computing device having simpli(cid:173)
`fied computing and I/0 capabilities for interacting with the
`Wearable Electromyography-Based Controller, as described
`herein.
`
`DETAILED DESCRIPTION OF THE
`EMBODIMENTS
`
`[0028]
`In the following description of the embodiments of
`the claimed subject matter, reference is made to the accom(cid:173)
`panying drawings, which form a part hereof, and in which is
`shown by way of illustration specific embodiments in which
`the claimed subject matter may be practiced. It should be
`understood that other embodiments may be utilized and struc(cid:173)
`tural changes may be made without departing from the scope
`of the presently claimed subject matter.
`[0029] 1.0 Introduction:
`[0030]
`In general, a "Wearable Electromyography-Based
`Controller," as described herein, provides various techniques
`for measuring user muscle electrical activity to interact with
`and control one or more computing devices. More specifi(cid:173)
`cally, the Wearable Electromyography-Based Controller pro(cid:173)
`vides a wearable device having a set of electromyography
`(EMG) sensor nodes for detecting a user's muscle-generated
`electrical signals for interacting with and/or controlling gen(cid:173)
`eral purpose computing devices, applications running on such
`computing devices, personal music players, physical devices
`coupled to a computing device (such as, for example, a pan(cid:173)
`tilt-zoom camera, home automation system, musical instru(cid:173)
`ment, etc.), game consoles, televisions or other multimedia
`devices, virtual devices such as a virtual piano or virtual
`guitar implemented within a computing environment, etc.
`Similarly, the Wearable Electromyography-Based Controller
`can be used to provide control of electromechanical pros(cid:173)
`thetic devices such as prosthetic hands, arms, legs, etc., by
`performing particular motions or gestures which in turn cause
`specific muscles of the user to generate electrical signals that
`are then used to activate one or more pre-determined motions
`in the prosthetic device.
`
`12 of 26
`
`

`
`US 2009/0326406 AI
`
`Dec. 31, 2009
`
`3
`
`[0031] The Wearable Electromyography-Based Controller
`is implemented in various form factors, including sets of
`individual sensor nodes (wired or wireless), wearable devices
`including a plurality of sensor nodes, or articles of clothing
`including a plurality of sensor nodes. For example, in various
`embodiments, the Wearable Electromyography-Based Con(cid:173)
`troller is implemented as an armband, a wristwatch, eye(cid:173)
`glasses, an article of clothing worn by the user (such as a shirt,
`gloves, shoes, pants, headband, etc.), or any other physical
`device or collection of devices worn by the user that has
`sufficient contact with the surface of the user's skin to the user
`to measure the electrical activity one or more of the user's
`muscles. Further, it should also be understood that a user can
`wear multiple Wearable Electromyography-Based Control(cid:173)
`lers, with each such Wearable Electromyography-Based Con(cid:173)
`troller being used to interact with either the same or a different
`computing device, application, or other attached device.
`[0032] For example, in one embodiment, the EMG sensor
`nodes of the Wearable Electromyography-Based Controller
`are placed in a simple band which is worn around the users
`forearm in order to sense muscle activity associated with
`specific finger and hand gestures (see discussion of FIG. 2 for
`an example of this embodiment). One example of this
`embodiment is described in a co-pending U.S. patent appli(cid:173)
`cation entitled "RECOGNIZING GESTURES FROM
`FOREARM EMG SIGNALS," filed Jun. 26, 2008, and
`assigned Ser. No. 12/146,471, the subject matter of which is
`incorporated herein by this reference. This co-pending patent
`application generally describes an initial process for learning
`muscle electrical signals corresponding to particular user ges(cid:173)
`tures or motions for use in allowing the Wearable Elec(cid:173)
`tromyography-Based Controller to provide the desired
`human computer interface (HCl), also referred to herein as a
`muscle-computer interface (muCI).
`In contrast to conventional Electromyography
`[0033]
`(EMG) measurement systems, the Wearable Electromyogra(cid:173)
`phy-Based Controller described herein requires only general
`positional placement on the user's body. In fact, this general
`placement of the Wearable Electromyography-Based Con(cid:173)
`troller is enabled by including more EMG sensors than are
`expected to be necessary to measure muscle electrical activ(cid:173)
`ity, with automated sensor selection being performed during
`an initial positional localization process. Consequently, the
`initial positional localization process allows the overall sys(cid:173)
`tem to self-select a set of one or more appropriate sensors
`within the Wearable Electromyography-Based Controller in
`order to capture appropriate muscle electrical signals for con(cid:173)
`trolling and interacting with computing systems and attached
`devices. Note that in various embodiments, the automated
`positional localization is repeated either periodically, or on as
`as-needed basis, in case the Wearable Electromyography(cid:173)
`Based Controller moves relative to one or more particular
`muscles during use.
`[0034] Further, in various embodiments, non-selected
`EMG sensors within the Wearable Electromyography-Based
`Controller are automatically turned off in order to save power.
`This embodiment is particularly useful in wireless implemen(cid:173)
`tations of the Wearable Electromyography-Based Controller
`where an onboard battery (replaceable or chargeable), fuel
`cell, photovoltaic power cell, etc., is used to energize selected
`EMG sensor nodes and associated circuitry. It should also be
`
`noted that in wireless implementations of the Wearable Elec(cid:173)
`tromyography-Based Controller, communication between
`the Wearable Electromyography-Based Controller and one or
`more computing systems is accomplished via conventional
`wireless communications protocols such as, for example,
`radio frequency (RF) communications, infrared (IR) based
`communications, Bluetooth, Zigbee, etc. In this case, the
`Wearable Electromyography-Based Controller includes one
`or more wireless transmitters, and optionally one or more
`receivers, for directly interfacing with one or more computing
`devices, or interfacing with one or more "hubs" that serve as
`intermediaries for interfacing the Wearable Electromyogra(cid:173)
`phy-Based Controller with one or more computing devices.
`It should be understood that various embodiments
`[0035]
`of the Wearable Electromyography-Based Controller are
`implemented in wired embodiments, such as, for example, by
`including an integrated USB cable or the like that both pro(cid:173)
`vides the necessary power for the EMG sensor nodes and
`provides a communications pathway between the Wearable
`Electromyography-Based Controller and one or more com(cid:173)
`puting devices. As in the wireless case, in wired embodi(cid:173)
`ments, the Wearable Electromyography-Based Controller
`communicates either directly with computing devices, or
`with those computing devices via an intermediary hub.
`In addition, given the various wired and wireless
`[0036]
`embodiments of the Wearable Electromyography-Based
`Controller described above, it should be understood that
`hybrid embodiments using various elements ofboth the wired
`and wireless configurations are enabled. For example, in one
`embodiment, a power cable provides operational power,
`while wireless communications are then enabled by one or
`more transmitters/receivers integrated into, or coupled to, the
`Wearable Electromyography-Based Controller. For example,
`in these types of hybrid embodiments, the power cable (e.g.,
`a power cable connected to a transformer or other power
`source, or a USB power cable connected to a computing
`device or transformer, etc.) provides operational power to the
`Wearable Electromyography-Based Controller, while the
`wireless
`transmitters/receivers provide communications
`between the Wearable Electromyography-Based Controller
`and one or more computing devices or intermediary hubs
`within wireless range of the Wearable Electromyography(cid:173)
`Based Controller.
`[0037] Some of the advantages offered by the Wearable
`Electromyography-Based Controller are that the Wearable
`Electromyography-Based Controller enables muscle electri(cid:173)
`cal signal based HCl or muCl type control of computing
`devices, applications, and attached devices with little or no
`preparation and setup on the part of the user. In fact, in the
`simplest embodiment, the user simply generally places the
`Wearable Electromyography-Based Controller in an approxi(cid:173)
`mate position (e.g., forearm, wrist, legs, chest, shoulders,
`back, etc.) that requires little or no expertise or attention to
`specific sensor node placement. Further, the automated posi(cid:173)
`tionallocalization of the Wearable Electromyography-Based
`Controller allows users to move freely as they would if they
`were not wearing the device.
`In addition, the use of multiple EMG sensor nodes in
`[0038]
`combination with one or more Wearable Electromyography(cid:173)
`Based Controllers also allows various applications associated
`with the Wearable Electromyography-Based Controller to
`detect muscle strain and provide ergonomic feedback to the
`user. Consequently, in various embodiments, the Wearable
`Electromyography-Based Controller can be used for training
`
`13 of 26
`
`

`
`US 2009/0326406 AI
`
`Dec. 31, 2009
`
`4
`
`a user to execute complex sets of specific muscular activities,
`such as required for playing musical instruments or sports, or
`for operating particular devices by providing immediate hap(cid:173)
`tic, audio, or visual feedback to the user in response to specific
`motions or gestures by the user.
`[0039]
`In view of the above summarized capabilities, and in
`further view of the following detailed description, it should be
`understood that the Wearable Electromyography-Based Con(cid:173)
`trollerprovides users with a "universal" input mechanism that
`can be used to control any computing device, applications
`running of computing devices, electronic or mechanical
`devices coupled to a computing device, or any other elec(cid:173)
`tronic device (television, radio, appliance, light switch, etc.)
`having an appropriate infrastructure or interface for receiving
`input from a wired or wireless controller. Note also that the
`use of a small wearable device such as the Wearable Elec(cid:173)
`tromyography-Based Controller, which may be under the
`user's clothes, if desired, provides a mechanism that is unob(cid:173)
`trusive (i.e., the user can be using her hands to perform other
`tasks while using Wearable Electromyography-Based Con(cid:173)
`troller to provide active control of one or more devices).
`Further, it should also be appre