Expressive Footwear for Computer-Augmented Dance Performance
`
`Joseph A. Paradiso and Eric Hu
`MIT Media Laboratory
`20 Ames St.
`Cambridge, MA 02139
`(+1)-617-253-8988
`joep,human@media.mit.edu
`
`Abstract
`A sensor system is described for instrumenting a pair of
`dancing shoes in order to capture many expressive degrees of
`freedom and use them to drive music synthesizers and
`computer graphics in a real-time performance. Dynamic
`pressure is measured at three points in the shoe sole, as are
`the bend of the sole, pitch and yaw shoe angles, and
`translational shoe positions. Data will be transmit across a
`19.2 kbaud wireless link, enabling updates at 10 msec
`intervals.
`
`Keywords: Musical gesture sensing, body suit, shoe
`sensors, foot sensing, computer dance performance.
`
`1 .
`
`Introduction
`Technologies under development for wearable computer
`systems can
`revolutionize several
`fields of artistic
`performance, such as theater or dance. Already, several
`artists have used body suits
`in computer-augmented
`performance [1,2]. Most of these devices merely employ
`embedded mechanical switches or piezoelectric sensors to
`detect simple strikes at several body locations. Others, such
`as in the Yamaha Miburi system [3], also use bend sensors
`to give continuous estimates of limb positions. None of
`these systems have attempted to instrument shoes with any
`degree of versatility, although the Miburi shoes provide
`piezoelectric triggers. Since the feet of a well-trained dancer
`are highly expressive appendages, an intimate computer-
`augmented performance needs realtime measurements of the
`many continuous parameters that can be acquired in the
`footwear. This paper describes a suite of sensors that we are
`now implementing into a set of dancing sneakers. As the
`signal conditioning needs are minimal and the processors
`and transmitters required are very small, we are building all
`electronics directly onto the shoe, avoiding difficulties with
`tethers running to a central communications unit.
`Figure 1 shows a diagram of the shoe instrumented with
`a sensor array; we are now using a Capezio Dance Sneaker
`for the actual implementation. All sensors and subsystems
`are described below:
`
`2 . Sensors and Electronics
`Two piezoelectric pads (“1” & “2” in Fig. 1) made from
`flat, laminated sheets of PVDF (polyvinylidene fluoride)
`polymer [4] are placed at the front of the shoe sole, below
`the regions covered by the big and small toes. One PVDF
`pad (“3” in Fig. 1) is placed at the heel. PVDF has already
`been inserted into a set of shoes to measure footfalls for
`wearable computing applications [5]; the signals are several
`volts into a high-impedance (‡
` 10 MegOhm) load, thus
`require only simple buffering before digitization. The two
`front pads will measure differential toe pressure, while the
`back pad will measure dynamic pressure at the heel.
`Although PVDF does not provide a steady-state force
`measurement, a dancer is usually in motion, and the most
`important performance features arise from dynamics.
`We measure the sole deflection with a bidirectional FSR
`strip (“4” in Fig. 1) optimized for bending response [6],
`placed across the center of the sole where most bend occurs.
`A dancer will often twist the foot in pitch (q ),
`independently of bending the sole (as measured above). The
`pitch coordinate
`is measured by a micromechanical
`accelerometer used as a tilt sensor (“6” in Fig. 1), where it
`senses the pitch component of the gravitational acceleration
`vector (g). Both single and dual-axis devices are commonly
`available in small packages, with resolutions at the milli-g
`
`Figure 1. Shoe-borne instrumentation
`
`__________________________________________________________________________________________________
`
`Presented at the First International Symposium on Wearable Computers, Cambridge MA. October 1997
`
`FITBIT, INC. v. LOGANTREE LP
`Ex. 1021 / Page 1 of 2
`
`

`
`in Fig. 2). When transmitting its raster of acquired data, the
`microcontroller will add a set of bytes that specify the
`relative time at which the sonar ping were received. Each
`transmitter pings at a different frequency, thus all can fire
`simultaneously. Since PVDF is inherently broadband, its
`frequency response and selectivity can be set through a
`simple electronic filter.
`Electric field sensing [11] will directly monitor the
`elevation of each foot; a copper strip above the sole
`transmits at 50-100 kHz, and the strength of the signal
`capacitively induced onto electrodes mounted atop the stage
`indicates the vertical shoe displacement.
`3 . Conclusions
`A simple suite of compact, commercially-available
`sensors can easily be built into a shoe to measure many
`degrees of expression, together with a small microprocessor
`and wireless transmitter. This system will be used in a
`series of computer-augmented dance performances.
`4 . Acknowledgments
`We are grateful to our colleagues at the MIT Media
`Laboratory for many discussions, especially Tod Machover,
`who has long been interested in expressive digital footwear.
`We appreciate the dance advice and prototype trials by
`Yuying Chen, and thank Jack Memishian from Analog
`Devices for donating the ADXL202’s and Vic Chatigny of
`AMP Sensors for donating the PVDF. We acknowledge the
`support of the Things That Think Consortium and our other
`sponsors at the MIT Media Laboratory.
`5 . References
`[1] L. Anderson, Stories from the Nerve Bible, HarperCollins,
`NY, 1994.
`[2] M. Coniglio, Troika Ranch, The MidiDancer system, See:
`“http://www.art.net/~troika/mididancer.html”.
`[3] Yamaha Corporation, The Miburi performance
`See: “http://www.yamaha.co.jp/news/96041001.html”.
`[4]
`J. Paradiso, “The Interactive Balloon: Sensing, Actuation,
`and Behavior in a Common Object,” IBM Systems Journal,
`35(3&4), 1996, pp. 473-487.
`[5] S. Mann, “Smart Clothing: The Wearable Computer and
`WearCam,” Personal Technologies, V. 1(1), 1997, pp. 21-27.
`[6] FLX01 sensor from The Images Co., Staten Island, NY.
`[7] C. Verplaetse, “Inertial Proprioceptive Devices: Self-
`motion-sensing Toys and Tools,”
`IBM Systems Journal,
`35(3&4), 1996, pp. 639-650.
`[8] Honeywell, Inc., Solid State Electronics Center, Plymouth,
`MN, Magnetic Sensor Products.
`[9] See “http://www.microchip.com/”.
`[10] Abacom Technologies, Etobicoke, Ontario, Canada.
`[11] J. Paradiso, N. Gershenfeld, “Musical Applications of
`Electric Field Sensing,” Computer Music Journ. 21(2), 1997.
`
`system,
`
`Figure 2. Sensing of shoe position
`
`level [7]. Although the kinematics of the foot preclude the
`roll angle from being fully articulated, a dual-axis
`accelerometer (e.g., the ADXL202) will also measure this
`quantity. Besides responding to tilt, these accelerometers
`will also detect changes in foot velocity, plus give very
`clear indications and profiles of jumps and impacts.
`The yaw coordinate (f ) will change as the dancer turns
`throughout the performance space. This can be measured
`directly (at least when the foot is oriented flat down) by
`embedding a small electronic compass into the sole (“5” in
`Fig. 1). We are now using a compact, magnetic vector
`sensor based on a permalloy bridge [8] to estimate bearing
`from the Earth’s magnetic field. To obtain better response,
`we are also exploring the installation of a compact,
`micromechanical gyroscope [7]; any gyro drift can be
`compensated by the compass measurements.
`Since signal conditioning requirements are minimal for
`this system, all electronics can be housed in a small
`compartment outside the shoe (“8” in Fig. 1). This will
`include a minimal embedded controller with 8-bit A/D
`converter to log and serialize the sensor data (e.g., a PIC
`16C71
`from Microchip Systems
`[9]), an analog
`multiplexer, battery adequate for at least 30 minutes of
`continuous operation, and a low-power RF transmitter, such
`as the TXM series of circa 400 MHz FM devices from
`Abacom Technologies [10], which occupy a 1 x 3 cm.
`surface-mount PC card. Since these transmitters can pass
`20K bits/second, the sensor data can be updated at roughly
`10 msec intervals. Because the range is so limited, a small
`antenna (“9” in Fig. 1) can either extend from the controller
`compartment, as in the figures, or be realized as a simple
`wire laminated onto the shoe itself. To enable continuous
`data transmission without the need of synchronization, both
`left and right feet will transmit on different RF carriers.
`These signals will be received and processed at a base
`station (“12” in Fig. 2), which analyzes the data and
`presents relevant features to the computers controlling the
`music and other aspects of the performance.
`Translational position of the shoes can be measured by a
`scanning laser rangefinder (11 in Fig. 2) or directly via
`active sonar. We are mounting a strip of PVDF on the
`outside of the shoe (“7” in Fig. 1) to detect ultrasound
`pulses from sonar transmitters at the stage perimeter (“10”
`
`2
`
`Ex. 1021 / Page 2 of 2