NEWS
`
`, COLUMNS
`
`14 Industry Trends
`New Technologies Place Video
`in Your Hand
`George Lawton
`18 Technology News
`Aspect-Oriented Programming
`Takes Aim at Software
`Complexity
`Sandra Kay Miller
`23 News Briefs
`Hacker Launches Cyberattack
`from Security Site • New
`Bus Technology Boosts
`Performance • Peer-to-Peer
`Goes Hand to Hand
`
`MEMBERSHIP NEWS
`
`84 Computer Society
`Connection
`Strategic Plan Maps Society's
`Goals • Student Winners
`Recognized for Technical
`Writing, Academics, and
`Involvement
`88 Call and Calendar
`
`TH E TR U T H A B O U T B I S T
`
`Looking for innovative ways to test
`
`embedded core-based chips? See the
`
`online tutorial "Built-In Self-Test
`
`for Systems-on-Chip" at http://
`
`aomputer.org/DT-tutorials/BIST.
`
`Membership Magazine
`of the
`
`IEEE~
`COMPUTER
`SOCIETY
`
`100 Entertainment
`Computing
`The Curse of Fast Iron
`James F. Dunnigan
`104 Communications
`Packet Scheduling in Next-
`Generation Multiterabit
`Networks
`Itamar Elhanany, Michael
`Kahane, and Dan Sadat
`107 Standards
`Software Architecture:
`Introducing IEEE Standard 1471
`Mark W. Maier, David Emery,
`and Rich Hilliard
`112 The Profession
`Virtual Organization:
`The New Feudalism
`Abbe Mowshoiuitz
`
`DEPARTMENTS
`
`4 Letters
`26 Article Summaries
`96 New Books
`97 Products
`
`10 IEEE Computer Society
`Membership Application
`91 Advertiser/Product Index
`92 Career Opportunities
`
`GOOGLE EXHIBIT 1040
`GOOGLE v. NEONODE
`IPR2021-01041
`
`Page 1 of 11
`
`

`

`ABOUT THIS ISSUE
`
`W hile transaction
`
`processing and Web
`servers dominate the
`high-end computing
`market, multimedia
`and DSP applications
`are beginning to
`prevail in the
`desktop and mobile(cid:173)
`computing markets.
`These applications
`exhibit behavioral
`characteristics that are quite
`different from traditional general(cid:173)
`purpose applications and require a
`new set of design constraints, such
`as very specialized performance at
`very low power and cost. This
`special issue captures some of
`these exciting trends in processor
`architecture.
`
`IEE Computer Society: http:/ /computer.ore
`computer: http:/ /computer.ore/computer
`computer@computer.org
`IEE Computer Society Publications Office: + 1 714 821 8380
`
`COMPUTING
`PRACTICES
`
`28 ltsy: Stretching
`the Bounds of Mobile
`Computing
`William R. Hamburgen,
`Deborah A . Wallach,
`Marc A. Viredaz,
`Lawrence S. Brakmo,
`Carl A . Waldspurger,
`Joel F. Bartlett,
`Timothy Mann,
`and Keith I. Farkas
`
`PERSPECTIVES
`
`38 The Cooler the
`Better: New
`Directions in the
`Nomadic Age
`Tsugio Mak imoto,
`Kazuh iko Eguchi,
`and Mitsugu Yoneyama
`
`43 Foundries and the
`Dawn of an Open
`IP Era
`Shang-yi Chiang
`
`COVER
`
`47 Early 21st Century
`Processors
`Sriram Vajapeyam
`and Mateo Va lero
`
`52
`
`59
`
`Power: A First-Class
`Architectural Design
`Constraint
`Trevor Mudge
`
`Instruction-Level
`Distributed
`Processing
`James E. Smith
`
`66 Speculative
`Mu ltithreaded
`Processors
`Gurindar S. Sohi
`and Amir Roth
`
`75 Embedded Computer
`Architecture and
`Automation
`B. Ramakrishna Rau
`and Michael S. Schlansker
`
`w RIGHTS RESERVED. ABSTRACTING IS PERMITTED
`
`APA COPYRIGHT© 2001 BY THE INSTITUTE OF ELEC(cid:173)
`~ " ' TR ICAL AND ELECTRONICS ENGINEERS INC. ALL
`
`WITH RF.DITTO THE SOURCE. LIBRARIES ARE PER(cid:173)
`MITTED TO PHOTOCOPY BEYOND THE LIMITS OFUS
`COPYRIGHT LAW FOR PRIVATE USE OF PATRONS: ( I ) THOSE POST-
`1977 ARTICLES THAT CARRY A CODE AT THE BOTTOM OF THE
`FIRST PAGE, PROVIDED THE PER-COPY FEE INDICATED IN THE
`CODE IS PAID THROUGH THE COPYRJGHT CLEARANCE CENTER,
`222 ROSEWOOD DR., DANVERS, MA 01923; (2) PRE-1978 ARTI(cid:173)
`CLES \Vlll-lOUT FEE. FOR on IER COPYING, REPRJNT, OR REPUB-
`1 ICATION PERMISSION, WRITE TO COPYR IGHTS AND PERMIS(cid:173)
`SIONS DEPARTMENT, IEEE PUBLI ATIONS ADMIN!SlllATION, 445
`IIOF.s LANE, P.O. Box 1331, PISCATAWAY, NJ 08855-1331.
`
`Cover design and artwork
`by Dirk Hagner
`
`NEXT MONTH
`Software
`Engineering on
`Internet Time
`
`GOOGLE EXHIBIT 1040
`GOOGLE v. NEONODE
`IPR2021-01041
`
`Page 2 of 11
`
`

`

`I This ma terial may bi! protected by Copyright law lTitle 17 U.S. Code)
`
`I
`
`COMPUTING PRACTICES
`
`Itsy: Stretching the Bounds
`of Mobile Computirig
`
`A prototype pocket computer that has
`enough processing power and memory
`capacity to run cycle-hungry applications
`such as continuous-speech recognition
`and real-time MPEG-1 movie decoding
`has proved to be a useful experimental
`tool for interesting app_lications, systems
`work, and power studies.
`
`William R.
`
`Deborah A.
`Wallach
`Marc A.
`Viredaz
`Lawrence S.
`Brakmo
`Carl A.
`Waldspurger
`Joel F.
`Bartlett
`Timothy
`Mann
`Keith I.
`Farkas
`Compaq
`Computer
`Corporation ,
`Corporate
`Research
`
`Hamburg en T he advent of fast, power-thrifty micro(cid:173)
`
`processors has made possible pocket-size
`computers with performance approach(cid:173)
`ing that of desktop PCs. This new class
`of mobile computers enables applica(cid:173)
`tions and user-interface modalities not feasible with
`traditional personal digital assistants and cell phones,
`while placing new demands on batteries and power
`management. We built Compaq's Itsy pocket com(cid:173)
`puter research prototype to explore the possibilities,
`demands, and limitations of mobile computing.
`Our primary hardware goals were to attain high per(cid:173)
`formance with minimal power consumption, size, and
`weight. At the same time, we needed a rich feature set
`to support user-interface and applications research and
`the flexibility to easily add new capabilities. To meet
`these goals, we used daughtcrcards to provide ltsy with
`comprehensive expansion capability. Fine-grain hard(cid:173)
`ware control supports flexible power management and
`monitoring. Developers can use the Linux operating
`system with extensions for a flash file system, resource
`sharing, and power management to rapidly prototype
`operating system extensions and new applications. Itsy
`has sufficient processing power and memory capacity
`to run cycle-hungry applications such as continuous
`speech recognition, a full-fledged Java virtual machine,
`and real -time MPEG-1 movie decoding.
`
`HARDWARE
`We began our hardware effort by constructing 75
`systems that we used to start software development. 1
`The experience we gained in building and using these
`sys tems influenced our subsequent des ign, ltsy v2.
`
`111D
`
`Computer
`
`Figure 1 shows this design's general architecture, and
`Table 1 lists its primary specifications.
`Our design focused on two goals: packing m ' xi(cid:173)
`mum performance into a unit that people can com(cid:173)
`fortably carry all day in a pocket or purse and en(cid:173)
`abling easy customization and extension of the ~ys(cid:173)
`tem hardware and software. Itsy is only slightly larger
`than a credit card, but it incorporates these other
`desirable features. Criteria such as cost or suitabil ity
`for volume manufacturing, which are critical for com(cid:173)
`mercial products, played no significant role.
`A small system's battery and display are generall) its
`largest and heaviest components, so they establish a l0•ver
`bound on the system's size and weight. For Itsy, we
`selected a lithium-ion cell just large enough to provide a
`full day of intermittent use, and the smallest available LCD
`with sufficient resolution for a rich graphical inrerfac '.
`We ruthlessly excluded any features or components
`that would bloat the system. For example, version 1
`users wanted a thumbwheel encoder, a cursor button,
`a good speaker, and a stylus slot, but because all fo ur
`of these features would not fit within our space bJd(cid:173)
`get, we excluded the stylus slot. Similar considerations
`led us to omit a bulky stereo headphone jack a nd codec
`in favor of a smaller monaural headset jack and a
`monaural codec that includes a touch-screen con(cid:173)
`troller. Finally, a radio transceiver was clearl y de ir(cid:173)
`able, but we found no obvious best choice to include
`in the base system. Therefore, we relegated experi(cid:173)
`mental radios to the daughtercard or serial interfaces.
`As a result of these choices, the complete Itsy is unly
`70 percent larger in volume than it would be if it c0n(cid:173)
`tained only the battery and display.
`
`d018·9162/01/$10.00 © 20011EEE
`
`GOOGLE EXHIBIT 1040
`GOOGLE v. NEONODE
`IPR2021-01041
`
`Page 3 of 11
`
`

`

`LCD,
`back light
`
`Touch
`screen
`
`LED
`
`Available
`daughtercard
`functionality
`Software
`~---;:=:=::=:::;-+I modem
`A/Dlnput
`
`GPIOs
`SSP
`+ - - - - - - -+ISDLC
`2 UARTs
`
`Memory bus
`(static memory,
`DRAM,
`twoPCMCIA
`sockets)
`
`Codec
`GPIO
`lrDA
`LCD
`LED
`Li -ion
`
`Coder/decoder
`General-purpose 1/0 pin
`Infrared Data Associ ation standard port
`Liquid crystal display
`Light-emitting diode
`Lithium-ion
`
`PCMCIA
`RS-232
`SDLC
`SSP
`UART
`USB
`
`Personal Computer Memory Card lnt'I Assoc.
`Serial interface
`Synchronous data link controller
`Synchronous serial port
`Universal asynchronous receiver-transmitter
`Universal serial bus
`
`Figure 1-ltsy v2
`architecturp. Most of
`the hardware is
`implemented on the
`motherboard
`(depicted in blue).
`The LCD and the
`touch screen attach
`to the motherboard
`via dedicated
`connectors. Other
`parts of the hardware
`(shown in yellow)
`are implemented on
`a separate flexible
`circuit board.
`A daughtercard
`connector is avail(cid:173)
`able to interface with
`additional hardware.
`Daughtercards can
`use any of the
`features listed in
`the red box.
`
`Processor
`The Stro ngARM SA-11 00 is a low-power 32-bit
`microprocessor that implements the ARM instruction
`set. 2
`3 This processor was a clea r choice beca use its
`·
`integer perfo rm a nce approaches th at o f des ktop
`processors, but it uses an o rder of magnitude less
`power. It also provides a useful collecti on of periph(cid:173)
`eral devi ces, as well as power-savin g fea tures th at
`researchers ca n exploit. To minimi ze energy use, the
`StrongARM supports software-controllable cl ock fre(cid:173)
`,quency and two low-power modes: idle and sleep. In
`Jdle mode, the clock to the processor core is gated off,
`aving power du e to the CMOS ci rcuit techno logy,
`While the rest of th e chip remain s powered and all
`peripherals rema in ena bled. [n sleep mode, most of
`the processor is unpowered, and onl y the rea l-time
`clock and the wa ke- up circuit remain ena bled.
`Optionall y, the system clock ca n remain ena bled for
`aster wake-up.
`
`In contrast to the usual power-saving passive-matrix
`isplays developers commonl y choo e fo r small sys(cid:173)
`ms, ltsy's LCD has some particularly usefu l charac(cid:173)
`teristics. It 0. 18-mm pixe l pitch is 25 to 30 percent
`mailer than the typica l pitch of small matrix displays,
`permitting dense text and crisp graphics. It multiline
`address ing techno logy provides higher contrast th an
`is typica l of a passive display. Fin ally, the LCD's built(cid:173)
`in I -bit-per-pixel memory and a progra mma ble-logic(cid:173)
`device (PLD ) a ux ili ary controller make it possible to
`display a static monochrome image while the proces(cid:173)
`sor is in sleep mode.
`
`Memory
`Th e most frequent complaint from ltsy vl proto(cid:173)
`type user was the limitati on of having onl y 4 Mbytes
`o f fl ash memory, so we started the v2 motherboa rd
`des ign with th e memory system. As Figure 2 shows,
`we chose a micro-ball -grid array package for the fl ash
`instea d of a peripheral lead package. Alth ough th e
`mi cro-BGA call s for a more compli ca ted assembl y
`process, it offers three tim es the mounting density. We
`chose the moth erboard width to allow dense tiling of
`the fl as h across the board's full width . We arranged
`the DRAMs on the opposite side to match the fl ash
`tiling. ltsy v2 has tw ice as much DRAM and eight
`times as much fl ash as Itsy vl , with only a 3 percent
`increase in boa rd area.
`
`Table 1. ltsy v2 specifications (without daughtercard).
`
`Component or characteristic
`Processor
`Dynamic RAM (DRAM)
`Flash memory
`Processor-core voltage
`Main voltage
`LCD
`Battery
`Size
`Weight
`
`Specification
`StrongARM SA-1100 (59 MHz-191 MHz) *
`32 Mbytes (50 ns, extended data out)
`32 Mbytes (90 ns)
`1.5 V or 1.23 V (selectable)
`3.05 V
`320 x 200 pixels, 15 gray levels
`Rechargeable Li-ion (2.2 W x h)
`118 mm x 65 mm x 16 mm
`130 gm
`
`' The processor's manual guarantees operation up to a 191 -MHz core frequency at
`1.5 V, but all ltsy systems built to date function correctly at 206 MHz and even higher.
`
`April 2001
`
`ml
`
`GOOGLE EXHIBIT 1040
`GOOGLE v. NEONODE
`IPR2021-01041
`
`Page 4 of 11
`
`

`

`After choosing the number of DRAM chips (4) and
`their individual capacity (64 Mbits), we selected the
`device width. Our choices were to implement one bank
`using four 8-bit parts (32 Mbytcs per bank) or two
`banks of two 16-bit parts each (16 Mbytes per bank).
`Because the StrongARM only supports up to fo ur
`banks, the one-ba nk option offers better expansion
`capabilities. However, we chose the two-bank design
`for two reasons. First, refreshing the 16-bit parts con(cid:173)
`sumes less power-about 5.2 mW versus 9.5 mW for
`32 Mbytes-because 16-bit parts have a different inter(cid:173)
`nal topology. Second, reading data out of th e 16-bit
`parts consumes less power because on ly two parts are
`active instead of four. For example, copying memory
`using eight-word bursts at 206 MHz requires about
`490 mW rather than 630 mW.
`
`Power system
`Our DRAM selection demonstrates how we used
`architectura l decisions and component selection to
`stretch battery life. However, to make the best use of
`limited battery energy, we had to consider both power
`supply and consumption.
`Because battery voltage varies widely during use,
`the design needs voltage regulators to provide the sys-
`
`tern with constant supply voltages. Linear regular lrs
`are sma ll, cheap, and widely used in sma ll devices, hut
`they have poor efficiency, particularly when the input
`voltage is much larger than the output. Therefore, we
`used switching regulators for our two main supphes.
`However, because clean audio was a key require~ ..:nt
`for speech recognition, we chose a sma ll, low-noise,
`linear regulator for the ana log circuitry.
`ltsy uses logic specified to operate at 3.3 ± 0. 3 V,
`but the power supp ly is set to 3.05 V, on ly sliglitly
`above the minimum. Because power consumpt'on
`increases with the sq uare of the voltage, this reduc(cid:173)
`tion from the usual 3.3 V, combined with the use of a
`switching regulator, results in an alm ost 15 percent
`power savings. Although a lower system vol t.,ge
`reduces noise margins, test systems have operated -,Ii(cid:173)
`ab ly at as low as 2. 7 V, indicating that this design point
`was a reasonable trade-off.
`
`Power-saving mechanisms
`These power strategies were necessary but not 0 11f(cid:173)
`ficient, so we developed additional hardware features
`that a llow the software to make the best use of avai l(cid:173)
`able energy. The hardware does not automatically dis(cid:173)
`able external peripherals when the processor ent..: rs
`
`(a)
`
`(b)
`
`Figure 2. The (a) top and (b) bottom sides of the ltsy v2 motherboard. Using micro-ball-grid array packages for the processor ancf
`flash memory offers a very high mounting density.
`
`lllll!J
`
`Computer
`
`GOOGLE EXHIBIT 1040
`GOOGLE v. NEONODE
`IPR2021-01041
`
`Page 5 of 11
`
`

`

`leep mode. Instead, the software can turn each unit
`n and off individually. This strategy lets the operat(cid:173)
`g system disable any of these units while the proces(cid:173)
`or is running, or conversely, any of the units can
`remain active while the processor is in sleep mode. For
`example, the operating system usually puts the DRAM
`· self-refresh mode during sleep, but it can also
`hoose to completely unpower the DRAM. We can
`use these mechanisms to implement a wide variety of
`leep modes, ranging from deep sleep, which main(cid:173)
`ins only the real-time clock, to light sleep, which
`eeps the LCD enabled, the DRAM contents pre(cid:173)
`rved, the clock on, and most interrupts configured
`o wake up the processor.
`Itsy's design lets the software choose to power the
`rocessor core at either 1.5 V or 1.23 V. Although
`1.23 V is below the manufacturer's specification, it is
`feat moderate clock rates and yields about a 30 per(cid:173)
`m power savings for the core, translating to 10 to 20
`rcent overall.
`ltsy can monitor its own power and energy use.
`oltage measurements and precision current-sense
`resistors with differential amplifiers allow sampling of
`he power on four separate paths: from the USB or
`ther external source, to and from the battery, into the
`rocessor core supply, and combined into the main and
`alog supplies, External laboratory instruments can
`onitor four additional current-sense resistors (for the
`DRAM and the three supply outputs). A battery mon(cid:173)
`. or circuit integrates power into and out of the bat(cid:173)
`tery to continuously monitor the state of charge.
`Accurately measuring the state of charge has proven
`hallenging because the battery monitor chip has two
`· itations: It cannot handle Itsy's large dynamic-cur(cid:173)
`em range, and it provides no method of correcting
`or current-measurement input offset errors, which
`re significant in a system that spends much of its time
`m sleep mode.
`
`ughtercard interface
`To facilitate development of high-performance
`ardware extensions, the daughtercard interface
`ports the full memory bus-all 32 data bits and all
`6 address signals-and most other useful signals.
`These other signals implement serial ports, 14 gen(cid:173)
`eral-purpose 1/0 pins (GPIOs), a software modem,
`d a single general-purpose analog input. Because
`the serial ports and the GPIOs share many processor
`ins, these features are not all available simultane(cid:173)
`ously.
`
`To exploit Itsy's flexible hardware, we needed flex(cid:173)
`ible software. We also wanted a comprehensive,
`robust environment for large applications. Open(cid:173)
`Source software was attractive because it eliminated
`
`barriers to sharing code. To meet these goals,
`we selected Linux, initially porting Russell
`King's ARM distribution (version 2.0.30) to an
`SA-1100 evaluation system and then to the Itsy
`platform. We then made significant additional
`changes to better support handheld computing,
`including improvements for power manage(cid:173)
`ment and memory-based file systems. In addi(cid:173)
`tion, we designed and implemented sessions, a
`new device-sharing model that lets Itsy run dif(cid:173)
`ferent application environments concurrently.
`
`Sessions, a new
`device-sharing
`model, lets ltsy run
`different application
`environments
`concurrently.
`
`File systems
`Itsy uses the Linux Ramdisk driver to provide
`dynamic memory partitioning between process
`address spaces and memory-resident file systems. This
`implementation is ideal for a handheld device because
`it does not waste space on redundant copies of data in
`the buffer cache and virtual memory system.
`For additional savings, we modified the Ramdisk
`driver to discard blocks containing only zeros. This
`simple but surprisingly effective form of compression
`allowed us to create a large file system that does not
`consume physical memory until it stores actual data.
`Changes to a Ramdisk are lost when power fails,
`so Itsy also bas a flash-based file system for stable,
`writeable storage. This consists of an ordinary Linux
`ext2 file system that expects to run on top of a disk,
`plus a block device driver that emulates a disk in a
`portion of the flash. Because flash has very different
`properties from a disk, this emulation is not trivial:
`The system must erase flash before writing it, the min(cid:173)
`imum erase unit is large-----128 or 256 Kbytes on Itsy(cid:173)
`and erasing is slow-typically 700 ms per erase unit.
`We used the industry-standard flash translation
`layer (FTL) data structure for disk emulation. We
`based our driver on code from the Linux PCMCIA
`subsystem, modified to work with Itsy's onboard flash.
`An FTL driver keeps a map from virtual disk block
`addresses to physical flash addresses. It handles reads
`simply by looking in the map to find the correct data.
`Writes are more complicated.
`When the driver must write to a virtual block,
`it finds a free-previously erased-physical block,
`writes the data into it, and updates the map. The phys(cid:173)
`ical block previously mapped to this virtual address,
`if any, is now unused, but the driver cannot erase it if
`other blocks in the same erase unit are still in use.
`When no free blocks remain, the driver reclaims space
`by choosing an erase unit that contains some unused
`blocks, moving any in-use blocks still in the unit to a
`spare unit, and erasing the old unit, which becomes
`the new spare.
`We observed a serious performance problem in this
`scheme. When the FTL driver reclaims space, it can
`free up only blocks that it knows are unu ed. FTL is
`
`April 2001
`
`GOOGLE EXHIBIT 1040
`GOOGLE v. NEONODE
`IPR2021-01041
`
`Page 6 of 11
`
`

`

`Figure 3. An example
`of sessions. The
`session labeled
`X Windows is in the
`foreground, and the
`others are in the
`background. The
`audio driver mixes its
`inputs, attenuating
`the inputs from the
`background sessions.
`The display driver
`shows one of the
`foreground session's
`frame buffers. When
`the user presses a
`button, only the
`foreground session
`is notified.
`
`Background
`session.:...-----
`
`fl MPEG player
`
`Foreground
`session:---(cid:173)
`X Windows
`IC!l!llll
`
`, .
`
`ltsy
`
`Background
`session:-~----
`
`mmi
`
`II
`ltsy
`
`□
`
`a disk emulator, not a file system, so it receives on ly
`virtual block read and write requests. Therefore, it
`does not find out that the driver has freed a physical
`block until the file system overwrites the correspond(cid:173)
`ing virtual block with new data. In the worst case,
`when the FTL driver must write to a virtual block, it
`knows about only one unused physical block-the one
`containing the virtua l block's o ld va lue. So it must
`erase and recopy a whole erase unit for every write,
`slowing FTL performance to a crawl. The Linux FTL
`driver avoids the worst case by defining the flash's vir(cid:173)
`tual size as on ly 95 percent of its physical size, so that
`5 percent of the physical blocks are always known to
`be unused. As a result, a single erase can sometimes
`reclaim more than one block. However, FTL still per(cid:173)
`forms as poorly as if the flash were 95 percent full, no
`matter how empty it really is.
`To correct this problem, we made a small modifica(cid:173)
`tion to the Linux ext2 file system. The file system now
`informs the block driver when it frees a virtual block,
`enabling the driver to reclaim the corresponding phys(cid:173)
`ical block ahead of time. This change greatly reduces
`the number of erases per write when the flash is not full.
`
`Power management
`To let Itsy operate at reduced power, we modified
`Linux to take advantage of the StrongARM proces(cid:173)
`sor's low-powered operating modes-idle and sleep(cid:173)
`and variable clock speeds.
`Exploiting idle mode is straightforward. The
`processor can enter and exit the mode in just a few
`clock cycles, so we modified the kernel idle loop to
`
`enter idle mode. When an interrupt occurs, the prou:s(cid:173)
`sor exits idle mode and returns to normal operati ,n.
`The user cannot detect idle mode.
`Sleep mode saves more power than idle mode. At
`the same time, it makes a bigger impact on the system,
`both in terms of software and user detectabi hLy,
`because sleep mode unpowers most of the proces<"r.
`In particular, the on-chip peripheral controllers,
`including the LCD controller and the UARTs, don't
`work. Sleep mode can be ini ti ated by user request
`(such as a button press), by the kernel, or by a system
`power fau lt. Entering sleep mode takes about 1 ,O
`µs; exiting takes about 10 µs if the clock was left
`enabled, 157 µs if it was disabled.
`We developed a power management module tha t
`coordinates the suspension and resumption of devi ... ..;s
`when sleep mode starts and ends. Each device registers
`callbacks that the power manager executes to deter(cid:173)
`mine whether or when the devices are ready to suspend.
`Another callback demands that the devices save their
`state immediately, pending suspension. When the
`processor exits sleep mode, the operating systr rn
`reestablishes its previous state (stack and registers), and
`the power manager calls the devices for reinitialization.
`The power manager module also suspends and
`resumes devices when changing the processor clc ·k
`speed. Suspending and resuming devices, except rhe
`LCD, is necessary only to prevent temporary glitches,
`and it adds about 400 µs to the base clock switching
`time of 125 µs. Resetting the LCD takes much longer,
`but the processor can perform useful computati nn
`during this time.
`
`Computer
`
`GOOGLE EXHIBIT 1040
`GOOGLE v. NEONODE
`IPR2021-01041
`
`Page 7 of 11
`
`

`

`Vlrtualizing ltsy devices
`Unlike a unified windowing system that demands
`compliance from all applications, the sessions device(cid:173)
`sharing model supports the peaceful coexistence of
`distinct yet concurrently executing "worlds," each of
`which sees its own virtual Itsy. Thus, Itsy can concur(cid:173)
`rently run systems as disparate as X Windows, Java,
`and Squeak, as well as full-screen stand-alone appli(cid:173)
`cations and virtual consoles.
`A session consists of one or more processes. At any
`given time, one session is distinguished as the fore(cid:173)
`ground session; the others are background sessions.
`As Figure 3 shows, foreground-session processes
`receive physical input events, such as button presses or
`touch-screen samples. They also have exclusive-or
`preferred-access to physical output devices. For
`example, a session's active frame buffer, if any, is vis(cid:173)
`ibly mapped onto the LCD. Processes in background
`sessions, on the other hand, do not receive physical
`input events, and their output is typically hidden.
`When a process opens a physical device, it accesses
`only a session-filtered instance of that device. The
`input that each physical device generates is replicated
`to all instances in the foreground session. Output is
`merged from all instances in the foreground session.
`A device can drop, attenuate (for example, for audio),
`store, or specially handle background-session output.
`A user-level process serves as a session manager to
`determine which virtual Itsy instance is currently
`mapped to the physical hardware. Figure 4 shows a
`session manager running on Itsy.
`Each physical device also exports a raw interface
`that applications requiring continuous access can use.
`For example, a speech recognition application running
`in the background can use the raw interface to snoop
`on all audio input, translate it, and offer the transcribed
`text to any application in the foreground session.
`
`EXPERIENCES WITH USER INTERFACES
`ltsy's system software makes porting existing desk(cid:173)
`top applications easy, but unfortunately, some
`assumptions of the desktop interaction style are inap(cid:173)
`propriate for small portable devices. Our initial
`attempts at using Squeak's graphical user interface on
`ltsy demonstrated the difficulty of mapping a device
`with a large display and three-button mouse to a
`device with a small display and a stylus. We developed
`conventions that made the Squeak GUI usable, but it
`was by no means easy to use.
`Users cannot move a stylus on a handheld device as
`easily and accurately as they move a mouse on a desk(cid:173)
`top. As a result, mastering handheld interfaces that
`require precision pointing and tapping is difficult.
`Additionally, a desktop GUI has the user's full attention
`for extended periods, but the only portable application
`domain that can demand such attention is gaming.
`
`Figure 4. A session
`manager running on
`ltsy v2. Several
`applications are
`simultaneously
`running in the back(cid:173)
`ground, including
`an MPEG player, a
`battery monitor, and
`a voice organizer.
`
`Clearly, as the Palm operating system demonstrates,
`GUis must be specially designed for small systems.
`As portable devices shrink further, stylus-based
`input becomes even more difficult. This problem led
`us to experiment with two nontraditional user inter(cid:173)
`faces: speech-based input and output and gesture(cid:173)
`based input.
`
`Speech
`One promising interface for a tiny device is speech.
`For output, ltsy uses DECtalk, a commercial text-to(cid:173)
`speech engine. For speech input, a more challenging
`problem, we had two speech recognition systems
`ported to ltsy. The first, TalkSoft, provides speaker(cid:173)
`independent, small-vocabulary, command-and-con(cid:173)
`trol speech recognition in a small memory footprint.
`We have successfully used DECtalk and TalkSoft to
`build a speech-based multimedia e-mail program.
`Although not a complete implementation, this pro(cid:173)
`gram has successfully demonstrated the feasibility of
`speech-based interaction in realistic environments,
`even using ltsy's bui lt-in microphone in a crowded
`room.
`Dragon Systems ported its speech recognition
`system to ltsy. This system includes both a speaker(cid:173)
`independent, grammar-based, continuous-speech,
`command-and-control engine and Dragon's Naturally(cid:173)
`Speaking, a speaker-dependent, continuous-speech
`dictation engine with a 30,000-word vocabulary.
`Together, the two engines offer the potential for a rich
`speech-only user environment.
`The promise of speech recognition has been par(cid:173)
`tially realized in the sense that adequate performance
`is no longer an obstacle to using speech as an inter(cid:173)
`face for pocket-sized devices. However, we must still
`meet the challenge of building effective speech(cid:173)
`centered or multimodal user interfaces.
`
`Gesture
`Most desktop-computer input methods rely on
`physical manipulation of an object such as a keyboard
`or mouse. A sy terns shrink, we can use the motion
`of the system itself for user input. Tilting a handheld
`
`April 2001 ~
`
`GOOGLE EXHIBIT 1040
`GOOGLE v. NEONODE
`IPR2021-01041
`
`Page 8 of 11
`
`

`

`Figure 5. A photo
`album application
`on ltsy. When a user
`makes a gentle
`fanning gesture with
`the ltsy, a two-axis
`micromachined
`accelerometer senses
`the fore and aff and
`left and right tilting,
`and the photo album
`application displays
`the next picture.
`
`computer to navigate through a document has long
`been anticipated, but sensors have only recently
`become small enough and cheap enough for develop(cid:173)
`ers to embed them in handheld devices that implement
`the tilt-to-scroll method.
`We extended the tilt-to-scroll method to include the
`use of gestures to issue commands. Our user interface,
`which we call Rock 'n' Scroll, lets users gesture to
`scroll, make selections, and issue commands, without
`resorting to any other input method. 4
`A photo album application demonstrates this inter(cid:173)
`face. The user tilts the album on either axis to scroll
`through miniature photographs until finding a picture
`of interest. When the user makes a gentle fanning ges(cid:173)
`ture, the album zooms in on that picture. The user can
`make additional fanning gestures to step through the
`rest of the album, return to the miniatures, or disable
`and enable scrolling. Pictures are availab le in land(cid:173)
`scape and portrait mode; holding the unit in the new
`orientation for a few moments reorients the picture.
`Early experiments with a handheld mockup demon(cid:173)
`strated that users quickly learn to operate Rock 'n'
`Scroll and gave us some insight into user expectations.
`These positive results, as well as improvements in sen(cid:173)
`sor technology, encouraged us to incorporate Rock 'n'
`Scroll as a standard input method on Itsy. Figure 5
`shows an implemention of the photo album applica(cid:173)
`tion on Itsy. A two-axis micromachined accelerome(cid:173)
`ter senses fore and aft and left and right tilting, and
`software converts the accelerometer's outputs to ges(cid:173)
`ture commands. In our user study, we observed that
`tilting the mockup to play a simple game fascinated
`nearly all participants. This observation was confirmed
`by the enthu siastic reception of our port of id
`Software's Doom game in which users tilt the Itsy to
`navigate thro ugh a three-dimensional environment.
`
`El
`
`Computer
`
`HEAVY LIFTING WITH A TINY BATTERY
`To meet project goals, we needed sufficient proce~s(cid:173)
`ing power and memory capacity to run next-generJ(cid:173)
`tion applications and user interfaces, as well as sufficient
`battery life to run realistic user interface experiments.
`We ran performance and energy consumption tests to
`assess how well we fulfilled these needs.
`
`Performance
`By running the Dhrystone