Session 4 @pr1/rich5/CLS_sj-ibm/GRP_sj-ibm/JOB_sj0300/DIV_sj3940-00a 06/08/00
`
`by J. A. Paradiso
`K.-Y. Hsiao
`A. Y. Benbasat
`Z. Teegarden
`
`As an outgrowth of our interest in dense wireless
`sensing and expressive applications of wearable
`computing, the Responsive Environments Group
`at the MIT Media Laboratory has developed a
`very versatile human-computer interface for the
`foot. By dense wireless sensing, we mean the
`remote acquisition of many different parameters
`with a compact, autonomous sensor cluster. We
`have developed such a low-power sensor card to
`measure over 16 continuous quantities and
`transmit them wirelessly to a remote base
`station, updating all variables at 50 Hz. We have
`integrated a pair of these devices onto the feet
`of dancers and athletes, measuring continuous
`pressure at three points near the toe, dynamic
`pressure at the heel, bidirectional bend of the
`sole, height of each foot off conducting strips in
`the stage, angular rate of each foot about the
`vertical, angular position of each foot about the
`Earth’s local magnetic field, as well as foot tilt
`and acceleration, 3-axis shock acceleration (from
`kicks and jumps), and position (via an integrated
`sonar). This paper describes the sensor and
`electronics systems, then outlines several
`projects in which we have applied these shoes
`for interactive dance and the capture of high-
`level podiatric gesture. We conclude by outlining
`several applications of our sensor system, which
`are unrelated to footwear.
`
`Wearable technology has long had application
`
`in musical expression. A historical example
`can be seen in the “one-man-band,” 1 a concept that
`dates back well over a century, long before the dawn
`of electronics. Figure 1 shows a modern incarnation
`in such a rig, with each “instrument” mounted for
`convenient access, responding to the action of a par-
`ticular limb or a specific, controllable motion of the
`wearer. Since the instruments were traditionally
`
`acoustic, each made a particular kind of sound, and
`the “action-to-audio” mapping was essentially static.
`In order to attain a timbral richness approaching that
`of a “band,” many such instruments were scattered
`about the body. Despite the apparent clutter, per-
`formers could use these adornments to charm and
`amuse audiences with occasionally virtuosic (al-
`though often acrobatic) musical expression as they
`appropriately flailed away.
`
`With the dawn of electronics, the situation evolved.
`Now the instruments themselves did not have to be
`mounted on the performer’s body, since they could
`be replaced by a set of electronic sensors that picked
`up the motion cues and controlled a remote music
`synthesizer. In the 1980s, the MIDI (Musical Instru-
`ment Digital Interface) standard and digital synthe-
`sis brought these systems even further, since now a
`computer could be easily placed in the loop, recog-
`nizing particular motions from real-time analysis of
`the sensor signals and producing a more complex,
`dynamic, and captivating software mapping of sound
`onto action. This was a very liberating process, be-
`cause the sensor systems freed the body from bear-
`ing the burden of the instruments, and advances in
`synthesis and data interpretation freed the sounds
`from being tied to simple causal definitions.
`
`Most projects in such electronic musical “wear-
`ables” 2,3 come under the rubric of “interactive
`rCopyright 2000 by International Business Machines Corpora-
`tion. Copying in printed form for private use is permitted with-
`out payment of royalty provided that (1) each reproduction is done
`without alteration and (2) the Journal reference and IBM copy-
`right notice are included on the first page. The title and abstract,
`but no other portions, of this paper may be copied or distributed
`royalty free without further permission by computer-based and
`other information-service systems. Permission to republish any
`other portion of this paper must be obtained from the Editor.
`
`IBM SYSTEMS JOURNAL, VOL 39, NOS 3&4, 2000
`
`0018-8670/00/$5.00 © 2000 IBM
`
`PARADISO ET AL. 1
`
`IPR2020-00910
`Garmin, et al. EX1010 Page 1
`
`

`

`Session 4 @pr1/rich5/CLS_sj-ibm/GRP_sj-ibm/JOB_sj0300/DIV_sj3940-00a 06/08/00
`
`Figure 1 A simple one-man marching band setup (left); an example of pre-electronics wearable technology for musical
`expression (right)
`
`— For position only —
`
`dance.” 4 An early example 5 is found in the work of
`composer Gordon Mumma, who adorned dancers
`with accelerometers to control analog synthesizers
`in performances of the 1960s. The well-known per-
`formance artist Laurie Anderson publicized these
`concepts in her shows of the 1980s, 6 using active ap-
`parel such as body suits adorned with percussive
`pickup transducers and neckties with embedded mu-
`sic keyboards. In the 1990s, several systems of this
`sort appeared. Many, such as Mark Coniglio’s MI-
`DIdancer, 7 The Danish Institute of Electronic Mu-
`sic 8 (DIEM) digital dance interface, and the Yamaha
`Miburi,** 3 were based around placing a set of re-
`sistive bend sensors across the dancer’s joints to ob-
`tain dynamic articulation. Because the Miburi was
`a commercial product, it was packaged as a complete
`system, including finger controllers for each hand,
`a wireless interface, an embedded synthesizer, and
`a set of shoes with piezoelectric taps at the toe and
`the heel, with each shoe wired to the central belt-
`pack transmitter.
`
`The foot of a trained dancer is a very expressive, mul-
`timodal appendage, capable of articulating much
`more than simple taps. Shoe interfaces for musical
`performances, however, were dominated by such tap
`implementations 9 and, until now, have not appre-
`ciably diversified from the toe-heel piezoelectrics.
`
`Different applications have resulted in the adoption
`of other technologies for foot sensing, although es-
`sentially all of these instances concentrate on sens-
`ing only a small set of particular parameters. For ex-
`ample, podiatric treatment centers and product
`development groups at sports shoe companies use
`densely pixilated pressure sensors 10 to observe the
`dynamic pressure distribution on the shoe soles dur-
`ing walking and running. In these applications, the
`shoe is often tethered to a data acquisition system
`through a multiconductor cable. Much coarser pres-
`sure sensor arrays (e.g., sensing at only a few places)
`have been used in portable commercial products,
`such as devices to warn patients with podiatrial neu-
`
`2
`
`PARADISO ET AL.
`
`IBM SYSTEMS JOURNAL, VOL 39, NOS 3&4, 2000
`
`IPR2020-00910
`Garmin, et al. EX1010 Page 2
`
`

`

`Session 4 @pr1/rich5/CLS_sj-ibm/GRP_sj-ibm/JOB_sj0300/DIV_sj3940-00a 06/08/00
`
`ropathy about potentially damaging footfalls 11 and
`shoes to interactively coach a golfer on his or her
`dynamic balance. 12 A pressure-sensing overshoe has
`also been incorporated in “Cyberboot,” 13 developed
`at the National Center for Supercomputing Appli-
`cations (NCSA) to incorporate foot gesture into vir-
`tual reality installations. The “Fantastic Phantom
`Slipper” 14 was an installation that used a pressure-
`sensing shoe with an active IR (infrared) optical sys-
`tem that tracked translational position across a small
`area, enabling users to step on animated insects that
`were projected onto the floor. Retrofits to jogging
`sneakers are now being brought to market that use
`inertial sensors for quantifying footfalls 15 and esti-
`mating elapsed distance (e.g., pedometry). 16
`
`The “expressive footwear” device developed in the
`MIT Media Lab Responsive Environments Group
`breaks these niches by using a diverse sensor suite
`to measure many (16) different parameters at the
`foot, detecting essentially everything that the foot is
`able to do, and telemetering the data back to a re-
`mote host computer in real time, leaving each shoe
`entirely untethered. Most human-computer inter-
`faces concentrate on precisely measuring gesture ex-
`pressed by the hands and fingers, devoting little, if
`any, attention to the feet. We have developed an in-
`terface that breaks this tradition, by measuring many
`parameters articulated at the foot.
`
`The sensor system and shoe hardware
`
`Our instrumented shoe was initially proposed 17 in
`1997, then refined 18,19 in 1998, and perfected 20 in
`1999. Figure 2 shows a diagram of the sensor system
`for our current shoe. Figure 3 shows a photograph
`of our original shoe system from 1997, grafted onto
`a Capezio Dansneaker**, and Figure 4 shows our
`final design affixed to a Nike Air Terra Kimbia (the
`electronics are normally obscured by a protective Lu-
`cite** cover, which was removed for this photo-
`graph). Figure 5 shows a close-up of the final ver-
`sion of the shoe electronics card, which can be seen
`to have advanced considerably beyond the initial
`working prototype of Figure 3.
`
`Shoe design and fabrication. A standard foam in-
`sole (represented by a dotted line in Figure 2) is em-
`bedded with an array of tactile sensors. Two stan-
`dard force-sensitive resistors (FSRs) 21 are placed at
`the left and right in the forward region of the shoe,
`yielding continuous pressure there and responding
`to the dancer’s rocking of the foot side-to-side. An-
`other FSR is placed forward of the toes, at right an-
`
`gles to the sole so it responds to downward pressure
`during pointing, when the shoe is vertical. Originally,
`this sensor was also inside the shoe compartment,
`but was moved outside for more reliable operation,
`since its performance varied considerably across dif-
`ferent dancers’ feet. For easier integration, a more
`malleable “FlexiForce**” 22 FSR was used here (its
`foil cable is seen running across the side of the sole
`
`in Figure 4). At the heel, where dynamic pressure
`is more relevant, we placed a strip of PVDF (poly-
`vinylidene fluoride), 23 a piezoelectric foil that re-
`sponds to changes in force. 24 Two back-to-back re-
`sistive bend sensors, 25 which were placed across the
`middle of the insole behind the toes, measured the
`sole’s bidirectional bend.
`
`A strip of copper mesh adhering to the bottom of
`the insole acted as a pickup electrode, capacitively
`coupling to transmitting electrodes, placed on the
`stage, that broadcast a constant sinusoidal signal at
`’55 kHz. When the dancer is above one of these
`plates, the signal received at the shoe decreases with
`the distance of the shoe from the plate, 26 giving an
`indication of the height of the shoe above the stage.
`Another electrode (not shown in Figure 2) is placed
`above the insole, just below the dancer’s foot, and
`is connected to the local electronics ground. This
`breaks the symmetry 27 between the pickup electrode,
`isolated below the insole, and the local shoe elec-
`tronics ground, which is now effectively coupled to
`the dancer’s body. The dancer, in turn, is ambiently
`coupled to the house ground, enabling current to
`flow from the transmitter plates into the shoe, hence
`allowing the shoe system to capacitively receive the
`transmitted 55 kHz signal. The height of the foot is
`inferred from the detected signal strength.
`
`A small (21⁄4" 3 31⁄4") circuit board is affixed to the
`outside edge of the shoe on a metal mount, contain-
`ing additional sensors and electronics. In our orig-
`inal design, the orientation of the foot at an angle,
`
`IBM SYSTEMS JOURNAL, VOL 39, NOS 3&4, 2000
`
`PARADISO ET AL. 3
`
`IPR2020-00910
`Garmin, et al. EX1010 Page 3
`
`

`

`Session 4 @pr1/rich5/CLS_sj-ibm/GRP_sj-ibm/JOB_sj0300/DIV_sj3940-00a 06/08/00
`
`Figure 2 Functional diagram of the Expressive Footwear electronics and sensor suite
`— For position only —
`
`20 KBITS/SEC RF LINK
`
`BASE STATION
`RECEIVER
`
`3-AXIS COMPASS
`3-AXIS SHOCK
`ACCELEROMETER
`
`f
`
`(YAW)
`
`TWIST (GYRO)
`
`PIC 16C711
`
`TILT
`ACCELEROMETER
`
`TRANSMIT
`ANTENNA
`
`9 VOLTS
`
`9 V
`BATTERY
`
`DYNAMIC
`PRESSURE
`(PVDF)
`
`SENSOR INSOLE
`
`CONTINUOUS
`PRESSURE (FSRs)
`
`(ROLL)
`C
`
`CAPACITIVE
`COUPLING
`
`POWER
`SWITCH
`
`RF
`TRANSMITTER
`
`SONAR
`RECEIVER
`
`RESISTIVE
`BEND
`SENSOR
`
`u
`(PITCH)
`
`PICKUP
`ELECTRODE
`
`40 KHZ SONAR
`PROJECTOR
`(4 PROJECTOR HEADS AROUND STAGE)
`
`55 KHZ
`OSCILLATOR
`
`FIELD-SENSING
`TRANSMITTER ELECTRODE
`(UNDER STAGE)
`
`f, about the vertical when the foot was nearly level
`was obtained from an 1525 analog electromechan-
`ical compass, 28 a small gimbaled magnet with quadra-
`ture position measured by a pair of Hall sensors,
`manufactured by the Dinsmore Instrument Corpo-
`ration in Flint Michigan. This monitored the orien-
`tation of the foot relative to the ambient (Earth’s)
`magnetic field. While the Dinsmore device was ad-
`equate for capturing slower motion during initial op-
`eration, after several hours of use the mechanics
`would start to fail and the gimbal would stick. The
`large forces and shock impulses encountered at a
`
`dancer’s foot are quite hostile to any fragile devices.
`In subsequent versions, the electromechanical com-
`pass was replaced with an all-solid-state device us-
`ing permalloy bridge sensors,
`the Honeywell
`HMC2003 3-axis magnetic sensor, 29 which we mod-
`ified 30 for 5-volt operation and higher gain. Although
`this sensor was quite reliable and gave wonderful,
`prompt 3-axis rotational response (another degree-
`of-freedom above the Dinsmore), permalloy bridges
`can drift over time as the sensing elements lose their
`magnetization. Therefore, a set of “strapping” pins
`was provided on the shoe card. By momentarily con-
`
`4
`
`PARADISO ET AL.
`
`IBM SYSTEMS JOURNAL, VOL 39, NOS 3&4, 2000
`
`IPR2020-00910
`Garmin, et al. EX1010 Page 4
`
`

`

`Session 4 @pr1/rich5/CLS_sj-ibm/GRP_sj-ibm/JOB_sj0300/DIV_sj3940-00a 06/08/00
`
`Figure 3 The original working prototype shoe
`
`Figure 5 A close-up of the most recent sensor circuit card
`
`— For position only —
`
`— For position only —
`
`Figure 4 The modern, perfected shoe with protective
`electronics cover removed
`
`— For position only —
`
`necting an 18-volt source across these pins, all mag-
`netic bridges would be subject to a brief current pulse
`that would magnetically saturate the permalloy,
`strapping it to maximum sensitivity. Over normal us-
`age, this strapping procedure would be adequate for
`at least several days, if not weeks, of operation.
`
`Because spins are important gestures to detect, we
`mounted another rotational sensor, a compact gy-
`roscope (a Murata GyroStar** vibrating-reed de-
`vice 31), on the sensor board, aligned with the axis of
`the ankle. This provided a direct measurement of
`angular rate about the vertical, giving clear response
`to spins and twists.
`
`A 2-axis, 62 G (where G is the acceleration of grav-
`ity) MEMs (microelectromechanical systems) accel-
`erometer from Analog Devices (the ADXL202) 32
`measured the tilt of the shoe with respect to the grav-
`ity vector and responded to the moderate acceler-
`ations of foot swings. Impact shocks and kicks, at
`higher G levels, were measured in 3-axes by a triple
`piezoelectric accelerometer (the ACH-04-08-05 from
`Measurement Specialties). 33
`
`A small (1 centimeter diameter) piezoceramic so-
`nar receiver (e.g., the Polaroid 40KR08 34) detects
`40 kHz pings sent from as many as four locations
`around the stage. By timing the reception of their
`first arrival, the translational position of the shoe can
`be tracked. The current shoe system is able to re-
`ceive pings across a distance of roughly 20 feet us-
`ing our current projectors, which are standard 1.5-cm
`diameter 40 kHz piezoceramic sources ganged in
`pairs. Additional range can be attained with more
`powerful emitters. With four independent projectors,
`at least one shoe is generally able to detect the sig-
`nals from at least two projectors in our present per-
`formance configuration (see the section on dance ap-
`plications later in this paper), fixing the dancer’s
`position on the plane of the stage.
`
`A “Peripheral Interface Controller” PIC 16C711 mi-
`crocomputer from Microchip Systems, clocked at 16
`MHz, is embedded onto the shoe card to digitize all
`signals and produce a serial data stream, which is
`broadcast to a base station through a small radio fre-
`quency (RF) transmitter, currently the “TX” series
`from Radiometrix. 35 Each shoe streams data at a sep-
`arate frequency (418 and 433 MHz). The 20 Kb/s
`
`IBM SYSTEMS JOURNAL, VOL 39, NOS 3&4, 2000
`
`PARADISO ET AL. 5
`
`IPR2020-00910
`Garmin, et al. EX1010 Page 5
`
`

`

`Session 4 @pr1/rich5/CLS_sj-ibm/GRP_sj-ibm/JOB_sj0300/DIV_sj3940-00a 06/08/00
`
`peak transmitter data rate enables a full state up-
`date rate from each shoe that approaches 50 Hz. Our
`shoes use a helical stub antenna that protrudes be-
`hind the heel, as seen in Figures 2–4. This enables
`the shoe’s transmissions to easily be received across
`
`a normal stage; we have used them successfully be-
`yond 100 feet from their base stations, but this per-
`formance depends, of course, on the local RF envi-
`ronment. Although the output of these transmitters
`is just under a milliwatt, they are still too strong for
`FCC (Federal Communications Commission) regu-
`lations, which allow 2–3 mW (microwatt) in these
`bands. In addition, emission at 418 MHz is limited
`to brief duration. The corresponding European lim-
`its, for which these transmitters were designed, are
`much more liberal. Such rules certainly restrict the
`carefree operation of our present system. As out-
`lined in the last section of this paper, we are cur-
`rently developing higher-bandwidth, channel-shared
`communications hardware that will allow for the le-
`gal operation of multiple embedded transmitters that
`meet our requirements.
`
`All onboard shoe electronics draw a current of about
`50 mA (milliampere) at 5 volts. The original shoe
`system used an onboard 1⁄2 AA-size 6.2-volt lithium
`camera battery, which provided a useful life of a few
`hours. After the first model, however, we moved to
`an off-card 9-volt alkaline battery, which provides for
`at least a half-day of very stable continuous perfor-
`mance. Although the operation could be extended
`significantly by substituting a switching regulator for
`the on-card series regulator or only powering the
`compass module (which consumes nearly half of the
`board’s current) during its readout, 36 this battery life
`span was already sufficient for our performance ap-
`plications, so the additional design complication was
`not warranted.
`
`Using the shoe. It is much easier to work with this
`shoe system than most other types of wearable in-
`
`terfaces. One only needs to put on the shoes and flip
`their power switches; there are no connectors, teth-
`ers, cables, harnesses, etc., to worry about. Although
`some of the sensor systems (e.g., the sonar) could
`be well implemented at other locations on the body,
`having all devices concentrated at the shoes greatly
`simplified the setup. Many dancers have worked with
`this system and have encountered few, if any, prob-
`lems with the mechanics and location of the elec-
`tronics module or antenna. Of the two, the antenna
`proved the most restrictive, since it could limit an-
`kle motion. It should be noted, however, that all of
`our dancers worked in a freeform, interpretive, and
`improvisational modern genre, as opposed to tra-
`ditional styles such as tap and ballet that may involve
`more constraints. With more engineering (e.g., go-
`ing to an embedded loop antenna and distributing
`the electronics throughout the shoe), the system can
`be made much more innocuous. In addition, the cur-
`rent device is largely hardwired into a particular shoe.
`Additional design work can make such a system mod-
`ular, perhaps clipping onto a shoe with an adjust-
`able insole that is adaptable across a wide range of
`foot sizes.
`
`Electronics, base stations, and system
`integration
`
`This section describes the electronics design and in-
`tegration of the shoe system components. More de-
`tail can be found in Reference 30. Figure 6 shows
`a block diagram of the electronics for the embed-
`ded shoe system. All sensors, except for the sonar
`and the two low-G accelerometer channels, produce
`analog voltages, which are conditioned, routed to
`CMOS (complementary metal oxide semiconductor)
`multiplexers, and then digitized by the 8-bit converter
`onboard the PIC.
`
`Electronics. Signal conditioning for the FSR sensors
`is simply an emitter follower; because no voltage gain
`is required, this allows the series gain-setting resis-
`tor in the FSR to become as large as needed to pro-
`vide adequate response to toe pressure, while pre-
`senting a low impedance to the analog to digital
`inputs of the PIC. Likewise, the PVDF signals are buff-
`ered by a junction field-effect transistor (JFET) source
`follower, enabling the PVDF shunt resistance (which
`limits the low-frequency bandwidth) to be set at 40
`MV (megaohm). The back-to-back bend sensors are
`fed through a differential amplifier to give bidirec-
`tional response. The capacitive pickup signal is first
`conditioned by a passive LC bandpass filter tuned to
`the 55 kHz transmitter (rejecting ambient back-
`
`6
`
`PARADISO ET AL.
`
`IBM SYSTEMS JOURNAL, VOL 39, NOS 3&4, 2000
`
`IPR2020-00910
`Garmin, et al. EX1010 Page 6
`
`

`

`Session 4 @pr1/rich5/CLS_sj-ibm/GRP_sj-ibm/JOB_sj0300/DIV_sj3940-00a 06/08/00
`
`Figure 6 Block diagram for the shoe-mounted circuitry
`
`— For position only —
`
`INPUTS
`
`CONTROL
`
`OUTPUT
`
`8:1
`ANALOG
`MUX
`
`INPUTS
`
` GAIN
`BUFFER
`
`DISCRIMINATOR
`
`PZT
`SONAR
`RECEIVER
`
`COMPASS
`AXIS 3
`
`SHOCK
`ACCEL
`AXIS 1
`
`TIPTOE
`FSR
`
`PING
`THRESHOLD
`ADJ.
`
`SONAR
`LED
`
`GAIN
`BUFFER
`
`ENVELOPE
`DETECTOR
`
`EMITTER
`FOLLOWER
`
`RF
`TRANSMITTER
`
`3V POWER
`(FOR TXM
`SERIES)
`
`MUX
`CONTROL
`OUT
`
`DIGITAL IN
`
`INTERRUPT
`IN
`
`SERIAL
`DATA
`OUT
`
`PIC16C711
`
`DIGITAL IN
`
`ANALOG IN
`
`CONTROL
`
`OUTPUT
`
`8:1
`ANALOG
`MUX
`
`LOWG
`ACCEL 1
`
`LOWG
`ACCEL 2
`
`ENVELOPE
`DETECTOR
`
`EMITTER
`FOLLOWER
`
`EMITTER
`FOLLOWER
`
`DIFFERENTIAL
`AMP
`
`+-
`
`SOURCE
`FOLLOWER
`
`GAIN
`BUFFER
`
`55 KHZ
`LC FILTER
`
`LEFT FSR
`(UNDER BALL
`OF FOOT)
`
`RIGHT FSR
`(UNDER BALL
`OF FOOT)
`
`BEND
`SENSOR
`
`HEEL
`PVDF
`
`GYRO
`
`CAPACITIVE
`PICKUP
`ELECTRODE
`
`COMPASS
`AXIS 1
`
`COMPASS
`AXIS 2
`
`9V
`BATTERY
`
`5V
`REGULATOR
`
`3V
`REGULATOR
`
`BATTERY-LOW
`GATE OUT
`
`+5V
`
`INPUTS
`
`SPARE ANALOG
`INPUT HEADER
`
`GAIN
`BUFFER
`
`GAIN
`BUFFER
`
`ENVELOPE
`DETECTOR
`
`ENVELOPE
`DETECTOR
`
`SHOCK
`ACCEL
`AXIS 2
`
`SHOCK
`ACCEL
`AXIS 3
`
`ground at other frequencies), then fed through a gain
`block and half-wave envelope detector that extracts
`the positive amplitude of the received signal. The
`three shock accelerometer signals are amplified, then
`time-stretched with a similar half-wave envelope de-
`tector, allowing them to be reliably digitized by the
`PIC across its data acquisition cycle. Although this
`loses polarity information, the raw accelerometer sig-
`nals are too narrow to be detected by the PIC at its
`50 Hz sampling updates. The signals coming directly
`from the Murata gyroscope are perfectly within the
`0–5-volt digitization range without further condition-
`
`ing, as are the 3 output signals from the Honeywell
`compass after it was modified, as mentioned earlier.
`In addition, the regulated 3-volt supply used by the
`RF transmitter module is digitized by the PIC and
`transmitted with each data set, since it is used to con-
`tinuously monitor the 5-volt supply, which is used as
`the A/D reference. The 3-volt input will appear to
`grow as the 5-volt supply droops.
`
`The latest version of the shoe electronics card has
`two 8-channel analog multiplexers. With the 4 an-
`alog inputs already available on the PIC, there are
`
`IBM SYSTEMS JOURNAL, VOL 39, NOS 3&4, 2000
`
`PARADISO ET AL. 7
`
`IPR2020-00910
`Garmin, et al. EX1010 Page 7
`
`

`

`Session 4 @pr1/rich5/CLS_sj-ibm/GRP_sj-ibm/JOB_sj0300/DIV_sj3940-00a 06/08/00
`
`18 available analog channels. Since the shoe system
`uses only 14 of these, the extra 4 inputs are brought
`to a header, where they are available for other de-
`vices (e.g., these are useful when the card is embed-
`ded in systems other than the shoe, as mentioned in
`the section on other applications later in this paper.
`
`The two low-G accelerometer outputs are digital 1
`kHz pulse trains, with the duty cycle of each pulse
`corresponding to the detected acceleration along the
`respective axis. They are thus connected directly to
`a pair of PIC digital inputs. After the PIC digitizes
`the analog data, it uses software to measure the ac-
`celerometer pulse-widths, retaining 8 bits of reso-
`lution.
`
`The signal from the sonar receiver is likewise first
`amplified (since the piezoceramic head is already
`highly resonant, there is no bandpass filtering). The
`signal is then routed through a half-wave envelope
`detector and sent to a discriminator with adjustable
`threshold (setting the sonar sensitivity). The discrim-
`inator output is applied to a PIC digital input that
`can generate an interrupt when the discriminator
`goes high, executing a tight segment of code that
`starts the timer of the PIC and sets a “sonar received”
`flag. When the PIC is about to transmit the byte in
`the serial data record dedicated to the sonar, it checks
`this flag to see if a ping was received, and if so, it
`sends the timer value (otherwise it sends zero). This
`parameter is thus the latency between the time when
`the ping was received and the time when the sonar
`byte was transmitted. Making the sonar threshold
`manually adjustable allows the user to set the trade-
`off between sonar sensitivity (e.g., range of opera-
`tion) and any 40 kHz background noise. Most of this
`noise is caused when the dancer lands hard from a
`jump or stomps a foot; because the accelerometers
`also detect this state nicely, any such spurious sonar
`spikes that coincidentally occur can be removed in
`the base station or subsequent PC software.
`
`The primary 5-volt supply for the shoe hardware is
`conditioned by a low-dropout series regulator that
`produces a battery-low gate, which is tripped when
`the battery drops below 5.3 volts. This gate is also
`read by the PIC and encoded into its data transmis-
`sion.
`
`Base stations. Figure 7 is a block diagram for the
`base station. It is much simpler, mainly consisting of
`a PIC 16C73 microcomputer (during the hardware
`design cycle, it was the smallest PIC with hardware
`serial ports) that receives serial input from a Radiom-
`
`etrix RX-series RF receiver, which picks up transmis-
`sions from the shoe and sends serial output to a
`RS-232 driver (for communicating with a personal
`computer serial port).
`
`In order to provide an appropriately zero-balanced
`RF serial stream, the shoe’s PIC uses a very simple,
`brute-force variation of Manchester encoding, in
`which it first sends all data bytes for a full record of
`sensor values and then sends their binary compli-
`ments. Additionally, by comparing each data byte
`in a record with its transmitted compliment, RF re-
`ception errors are detected, and individual bad bytes
`are “failed” and ignored, thus keeping the rest of
`the record intact. In order to enable the base station
`to quickly synchronize to the shoe’s data cycle, the
`first byte in a record is marked with a unique code
`(either 254 or 255, depending on the battery-low
`gate). This code is not permitted to appear in sub-
`sequent values.
`
`System integration. Figure 8 shows a high-level block
`diagram of the entire expressive footwear system. In
`the current rendition, two base stations are needed,
`one for each shoe. Each base station listens for its
`shoe at a different RF frequency (as mentioned, 418
`or 433 MHz). One of the base stations, deemed the
`“master,” also has an onboard 55 kHz sine wave gen-
`erator and driver for the electric field transmitter
`plates, which are detectable by both shoes. The mas-
`ter’s PIC additionally generates four gates for the so-
`nar pingers, each of which produces a few-millisec-
`ond burst of 40 kHz ultrasound when triggered. The
`master pulses a sonar gate every tenth of a second,
`going round-robin through all connected pingers.
`The master’s PIC uses its timer to measure the in-
`terval between sending the ping and receiving a valid
`byte detected by sonar from the shoe. The value of
`the sonar byte sent from the shoe (containing the
`latency in the shoe) is then subtracted from the value
`of the master’s timer (containing the acoustic tran-
`sit time plus shoe latency), resulting in the amount
`of time it took the ping to reach the shoe, hence the
`distance of the shoe from the pinger. This sonar sys-
`tem works satisfactorily, providing 8 bits of position
`resolution across a 30-foot range. As seen in Figure
`8, a pulse from the master synchronizes the slave base
`station when the master sends each ping. This pulse
`interrupts the slave’s PIC to start its timer, enabling
`the same sonar algorithm to work there. The master
`and slave base stations keep track of which sonar
`head was the last to ping, sending that address along
`to the host personal computer (PC) with every data
`record.
`
`8
`
`PARADISO ET AL.
`
`IBM SYSTEMS JOURNAL, VOL 39, NOS 3&4, 2000
`
`IPR2020-00910
`Garmin, et al. EX1010 Page 8
`
`

`

`Session 4 @pr1/rich5/CLS_sj-ibm/GRP_sj-ibm/JOB_sj0300/DIV_sj3940-00a 06/08/00
`
`Figure 7 Block diagram for a base station
`
`— For position only —
`
`TO HOST PC
`
`19.2 KB RS-232
`SERIAL SIGNAL
`
`MAX233 LINE
`CONVERTER
`
`SONAR PINGER X4
`
`(MASTER ONLY)
`
`EMITTERS
`
`PZT
`
`EMITTERS
`
`PZT
`
`EMITTERS
`
`PZT
`
`EMITTERS
`
`PZT
`
`EDGE-TRIGGERED
`40KHZ PULSE
`BURST
`
`EDGE-TRIGGERED
`40KHZ PULSE
`BURST
`
`EDGE-TRIGGERED
`40KHZ PULSE
`BURST
`
`EDGE-TRIGGERED
`40KHZ PULSE
`BURST
`
`PIC16C73
`
`PACKET FRAME
`RECOGNITION
`ERROR DETECTION
`
`SONAR
`TIMING
`
`5V SERIAL
`PACKETS
`
`RF
`RECEIVER
`
`SONAR MODE
`SELECT
`DIP SWITCHES
`(MASTER ONLY)
`
`PROJECTOR 1
`
`SYNCH
`
`PING GATE
`
`PROJECTOR 1
`
`SYNCH
`
`PING GATE
`
`TO
`SLAVE
`(MASTER ONLY)
`
`FROM
`MASTER
`(SLAVE ONLY)
`
`CAPACITIVE ELECTRODE TRANSMITTER
`
`BUFFER/DRIVER
`
`FREQUENCY
`TUNE
`
`CW 55 KHZ VCO
`(8038)
`
`AMPLITUDE
`ADJUST
`
`TO CAPACITIVE
`TRANSMIT ELECTRODE
`
`The current system produces a pair of 19.2 KBaud
`serial streams from master and slave base stations
`that are combined in the analysis and content soft-
`ware running on the host PC. The 50 Hz state-up-
`date cycle is primarily limited by the 20 Kb/s RF data
`rate, which is at the edge of capability for the Ra-
`diometrix transmitter and receiver modules that we
`are currently using. A more efficient zero-balancing
`scheme would likewise speed up the effective data
`rate to within a factor of two. The data interpreta-
`tion algorithm running on the host PC provides an-
`other layer of error protection by ignoring any spu-
`rious “spikes” on most sensor signals (e.g., data that
`abruptly jump from the baseline to a significant value
`then return directly to the baseline on the subsequent
`sample). This introduces an intrinsic delay of one
`20-millisecond data cycle.
`
`System performance
`
`The data stream produced by the shoe system is very
`rich, providing much detail on the gait and foot
`dynamics. This can be seen in the sample data plot-
`ted in Figure 9, which shows a 12-second “stripchart”
`excerpt of the raw outputs as wirelessly received at
`the host PC, from all 16 sensor subsystems on a sin-
`gle instrumented Nike sneaker. At the beginning of
`the data sample, the user walked toward the sonar
`head, starting roughly 15 feet away and ending up
`a foot or two from the head after 6 seconds. This is
`clearly seen in the sonar range data, plotted at top
`left, where individual footfalls create a stairstep struc-
`ture. In this example, only one sonar projector was
`used, pinging at 10 Hz. The regular signature of the
`gait can be seen in the pressure and bend signals,
`plotted below the sonar. The difference between the
`
`IBM SYSTEMS JOURNAL, VOL 39, NOS 3&4, 2000
`
`PARADISO ET AL. 9
`
`IPR2020-00910
`Garmin, et al. EX1010 Page 9
`
`

`

`Session 4 @pr1/rich5/CLS_sj-ibm/GRP_sj-ibm/JOB_sj0300/DIV_sj3940-00a 06/08/00
`
`Figure 8 Configuration of the fully deployed Expressive Footwear System
`— For position only —
`
`55 KHZ
`CAPACITIVE
`TRANSMIT ELECTRODE
`
`HOST PC
`
`AUDIO RACK
`
`MIDI
`STREAM
`
`PACKET FILTERING
`GESTURE RECOGNTION
`DATA -> MUSIC
`MAPPING
`
`MIDI INTERFACE
`SYNTHESIZERS
`MIXER
`AUDIO OUT/SPEAKERS
`
`RS-232
`
`COM1
`
`COM2
`
`SONAR PINGERS
`
`RS-232
`
`LEFT SHOE
`
`SENSORS
`DATA CONVERSION
`DATA COLLECTION
`SERIALIZATION
`
`433 MHZ RF
`
`MASTER BASE
`STATION
`
`PACKET RECOGNITION
`LINE CONVERSION
`SONAR CONTROL/
`TIMING
`PAN OSCILLATOR
`
`SYNCHRONIZATION
`
`SONAR
`
`418 MHZ RF
`
`SENSORS
`DATA CONVERSION
`DATA COLLECTION
`SERIALIZATION
`
`RIGHT SHOE
`
`PACKET RECOGNITION
`LINE CONVERSION
`SONAR TIMING
`
`SLAVE BASE
`STATION
`
`FSR and PVDF response is obvious, the former pro-
`viding steady-state pressures as the toes bear down,
`and the latter giving a differential signal that responds
`to the attack and release of the heel. The FSR signal
`decreases with increasing pressure. As noted in Fig-
`ure 9, the FSRs are biased to be slightly insensitive
`for a conventional person’s walk, yielding more range
`for a dancer up on his or her toes, where the pres-
`sures are higher.
`
`After about 7 seconds, the walking stops, and the
`foot is moved about more wildly, as can be noted in
`the drop in the consistency of the pressure signals.
`At roughly 11 seconds, the foot is rotated perpen-
`dicular to the floor, and the wearer presses the front
`of the shoe against the ground, as seen in the tip pres-
`sure plotted in the second graph at left (because this
`action is very deliberate and the data very clean, it
`is a good candidate to