COMBINING AUXILIARY FINGER INPUT WITH THUMB TOUCH
`FOR SINGLE-HANDED MOBILE DEVICE INTERFACES
`
`
`by
`
`
`
`ANDREAS HOLLATZ
`
`
`
`
`
`
`
`
`
`
`
`A thesis submitted to the School of Computing
`
`in conformity with the requirements for
`
`the degree of Master of Science
`
`
`
`
`
`
`
`
`
`Queen’s University
`
`Kingston, Ontario, Canada
`
`(October 2015)
`
`
`
`Copyright ©Andreas Hollatz, 2015
`
`
`
`
`
`
`
`Neonode Smartphone LLC, Exhibit 2040
`Page 2040 - 1
`IPR2021-01041, Google LLC v. Neonode Smartphone LLC
`
`

`

`ProQuest Number:
`
`10663674
`
`All rights reserved
`
`INFORMATION TO ALL USERS
`The quality of this reproduction is dependent upon the quality of the copy submitted.
`
`In the unlikely event that the author did not send a complete manuscript
`and there are missing pages, these will be noted. Also, if material had to be removed,
`a note will indicate the deletion.
`
`Published by ProQuest LLC (
`2017
`
`ProQuest
`
`10663674
`). Copyright of the Dissertation is held by the Author.
`
`All rights reserved.
`This work is protected against unauthorized copying under Title 17, United States Code
`Microform Edition © ProQuest LLC.
`
`ProQuest LLC.
`789 East Eisenhower Parkway
`P.O. Box 1346
`Ann Arbor, MI 48106 - 1346
`
`(cid:3)(cid:3)(cid:3)(cid:3)
`
`(cid:3)(cid:3)(cid:3)(cid:3)
`
`(cid:3)
`
`(cid:3)
`
`(cid:3)(cid:3)
`
`(cid:3)(cid:3)(cid:3)
`
`(cid:3)
`
`(cid:3)(cid:3)
`
`(cid:3)(cid:3)
`
`Neonode Smartphone LLC, Exhibit 2040
`Page 2040 - 2
`IPR2021-01041, Google LLC v. Neonode Smartphone LLC
`
`

`

`
`
`Abstract
`
`This thesis reports on the use of auxiliary finger input to complement touch-only interactions on
`
`mobile devices. While a majority of touchscreen based mobile devices support multi-touch input,
`
`mobile device interactions in one-handed usage scenarios are usually limited to a single point of
`
`contact with the screen. In most cases, the thumb is the preferred source of touch input. Selecting
`
`user interface elements, such as buttons and sliders, requires frequent movement of the thumb,
`
`occludes a display, and, to reach targets, demands frequent adjustments of grip.
`
`To tackle these usability problems of single-handed usage scenarios, we explored the use of the
`
`auxiliary fingers — that is, the fingers that grip, support, and make contact with a mobile device
`
`— as additional input channels. Sensing input from the auxiliary fingers might lead to
`
`significantly less thumb movement, with target selection and other interactions distributed across
`
`all five digits. We built a series of mobile device prototypes that sense isometric pressure at
`
`different areas on their surfaces. To evaluate the performance of this interaction paradigm, we
`
`measured task completion times and error rates for common mobile tasks, including document
`
`formatting, application switching, and map navigation, and validated that the use of additional
`
`fingers for input led to performance gains. We follow-up with a study to measure each finger’s
`
`ability to apply pressure on the side of the device and measured the effect of this pressure on the
`
`thumb's range of motion around the screen. Finally, we provide software and hardware design
`
`recommendations based on these studies.
`
`
`
`
`
`ii
`
`Neonode Smartphone LLC, Exhibit 2040
`Page 2040 - 3
`IPR2021-01041, Google LLC v. Neonode Smartphone LLC
`
`

`

`Co-Authorship
`
`The prototype design and experimental evaluation on pages 24-43 were conducted collaboratively
`
`with David Holman and Amartya Banerjee.
`
`
`
`iii
`
`Neonode Smartphone LLC, Exhibit 2040
`Page 2040 - 4
`IPR2021-01041, Google LLC v. Neonode Smartphone LLC
`
`

`

`
`
`Table of contents
`
`HAPTER!1!!!!INTRODUCTION!...................................................................................................!1!
`
`1.1! BACKGROUND!AND!MOTIVATION!.............................................................................................!1!
`1.2! CONTRIBUTIONS!AND!THESIS!OUTLINE!.......................................................................................!2!
`
`! C
`
`CHAPTER!2!!!!RELEATED!WORK!.................................................................................................!5!
`2.1! SOFT!MACHINES!....................................................................................................................!5!
`2.2! ONE9HANDED!MOBILE!INTERACTION!........................................................................................!9!
`2.3! GRASP9BASED!INTERACTION!..................................................................................................!12!
`2.4! AUGMENTED!MOBILE!INTERFACES!(SENSOR!FUSION)!.................................................................!13!
`2.5! DEFORMABLE!MOBILE!INTERACTIONS!.....................................................................................!15!
`2.6!
`ISOMETRIC!FORCE!BASED!INTERACTION!...................................................................................!16!
`2.7! HAND!ANATOMY!AND!FUNCTION!...........................................................................................!17!
`2.7.1! Anatomy,...................................................................................................................,17!
`2.7.1.1!
`Forearm!and!Extrinsic!Muscles!.........................................................................................................!17!
`2.7.1.2!
`The!Wrist!...........................................................................................................................................!18!
`2.7.1.3!
`The!Hand!...........................................................................................................................................!19!
`2.7.1.4!
`The!Digits!..........................................................................................................................................!19!
`2.7.2! Hand,Function,...........................................................................................................,20!
`2.7.2.1!
`Internal!and!Manipulation!Forces!.....................................................................................................!20!
`2.7.2.2!
`Finger!independence!of!movement!..................................................................................................!21!
`2.7.2.3! Movement!Speed!..............................................................................................................................!22!
`
`CHAPTER!3!!!!THE!UNIFONE!PROTOTYPE!................................................................................!24!
`3.1! UNIFONE!DESIGN!PROCESS!....................................................................................................!24!
`3.1.1! Unifone,Auxiliary,Touch,Gestures,.............................................................................,30!
`3.2! DISTINGUISHING!HOLDING!GRASP!AND!INPUT!SQUEEZE!...............................................................!31!
`3.3! FORCE!SENSORS!..................................................................................................................!31!
`3.4! PHYSICAL!CONNECTION!TO!DEVICE!.........................................................................................!33!
`3.5! SENSOR!PROCESSING!AND!DEVICE!COMMUNICATION!................................................................!34!
`iv
`
`
`
`Neonode Smartphone LLC, Exhibit 2040
`Page 2040 - 5
`IPR2021-01041, Google LLC v. Neonode Smartphone LLC
`
`

`

`IOS!EXPERIMENT!SOFTWARE!ARCHITECTURE!............................................................................!34!
`3.6!
`3.7! UNIFONE!INTERACTIONS!.......................................................................................................!36!
`3.8! UNIFONE!EVALUATION!.........................................................................................................!37!
`3.9! UNIFONE!EXPERIMENT!DESIGN!..............................................................................................!39!
`3.9.1! Participants,...............................................................................................................,40!
`3.10! UNIFONE!RESULTS!.............................................................................................................!40!
`3.10.1! Scrolling,task,..........................................................................................................,40!
`3.10.2! Formatting,task,......................................................................................................,41!
`3.10.3! Application,switching,.............................................................................................,41!
`3.10.4! Map,navigation,......................................................................................................,41!
`3.11! DISCUSSION!......................................................................................................................!42!
`
`4.5!
`
`CHAPTER!4!!!!THE!ISOPHONE!PROTOTYPE!..............................................................................!44!
`4.1!
`ISOPHONE!CONCEPT!.............................................................................................................!44!
`4.2! HARDWARE!........................................................................................................................!45!
`4.3! SOFTWARE!.........................................................................................................................!47!
`4.3.1! Device,Communication,.............................................................................................,47!
`4.4! SIGNAL!PROCESSING!AND!ANALYSIS!........................................................................................!48!
`4.4.1.1!
`Thresholding!.....................................................................................................................................!48!
`4.4.1.2!
`Blob!Finding!and!Finger!Positions!.....................................................................................................!48!
`4.4.1.3! Gesture!Recognition!and!Event!handling!..........................................................................................!49!
`INTERACTIONS!.....................................................................................................................!50!
`4.5.1.1! Home!Gesture!...................................................................................................................................!50!
`4.5.1.2!
`Zoom!Control!....................................................................................................................................!51!
`ISOPHONE!EXPERIMENT!........................................................................................................!51!
`4.6!
`4.6.1! Experiment,Description,.............................................................................................,51!
`4.6.2! Participants,...............................................................................................................,53!
`4.6.3! Results,......................................................................................................................,54!
`4.6.3.1!
`Range!of!motion!results!....................................................................................................................!54!
`4.6.3.3!
`TLX!Results!........................................................................................................................................!54!
`4.6.3.4!
`Inter9response!interval!results!..........................................................................................................!55!
`4.6.4! Discussion,.................................................................................................................,56!
`4.6.4.1!
`Screen!Area!.......................................................................................................................................!56!
`4.6.4.2!
`Inter9response!Intervals!....................................................................................................................!59!
`v
`
`
`
`Neonode Smartphone LLC, Exhibit 2040
`Page 2040 - 6
`IPR2021-01041, Google LLC v. Neonode Smartphone LLC
`
`

`

`CHAPTER!5!!!!DESIGN!GUIDELINES!..........................................................................................!61!
`5.1! GENERAL!DESIGN!GUIDELINES!...............................................................................................!61!
`5.2! HARDWARE!DESIGN!GUIDELINES!............................................................................................!62!
`5.2.1! Dimensions,...............................................................................................................,62!
`5.2.2! Pressure,ranges,........................................................................................................,63!
`5.2.3! Sensor,Position,.........................................................................................................,64!
`! SOFTWARE!DESIGN!GUIDELINES:!.................................................................................................!66!
`5.3! FORMULATING!AN!EFFECTIVE!GESTURAL!LANGUAGE!...................................................................!66!
`
`CHAPTER!6!!!!CONCLUSION!AND!FUTURE!WORK!....................................................................!69!
`6.1! CONCLUSION!......................................................................................................................!69!
`6.2! FUTURE!WORK!....................................................................................................................!70!
`
`REFERENCES!..........................................................................................................................!73!
`
`APPENDIX!A:!!AVERAGE!PRESSURE!AT!SCREEN!POSITION!.......................................................!78!
`
`
`
`
`
`
`
`
`vi
`
`Neonode Smartphone LLC, Exhibit 2040
`Page 2040 - 7
`IPR2021-01041, Google LLC v. Neonode Smartphone LLC
`
`

`

`List of Figures
`
`
`FIGURE!1.!THE!XEROX!5700!PRINTING!SYSTEM!WITH!A!TOUCHSCREEN!INTERFACE![50]!....................................................!7!
`FIGURE!2.!THE!IBM!SIMON!WAS!THE!WORLD’S!FIRST!SMARTPHONE![7]!........................................................................!8!
`FIGURE!3.!THE!NEONODE!N1!WAS!THE!FIRST!MOBILE!TO!USE!SWIPE!GESTURES![46]!........................................................!8!
`FIGURE!4.!THE!ORIGINAL!APPLE!IPHONE!MARKED!THE!BEGINNING!OF!THE!CURRENT!MOBILE!PARADIGM![28]!........................!8!
`FIGURE!5.!THE!XEROX!PARCTAB![49]!..................................................................................................................!11!
`FIGURE!6.!SINGLE!HANDED!ZOOM!FROM!SONY!.......................................................................................................!12!
`FIGURE!7.!COMPARISON!OF!INTER9RESPONSE!INTERVAL!FOR!FINGER!MOTION!FOR!A!RANGE!OF!ACTIVITIES![26]!...................!23!
`FIGURE!9.!SENSOR!PLACEMENT!FOR!THE!FIRST!ITERATION!OF!THE!UNIFONE!PROTOTYPE!..................................................!26!
`FIGURE!10.!AN!EARLY!ITERATION!OF!THE!UNIFONE!PROTOTYPE:!A!SILICON!SKIN!WRAPS!AROUND!PRESSURES!SENSOR!ATTACHED!
`TO!AN!IPOD!TOUCH.!A!PHIDGETS!INTERFACE!KIT,!ATTACHED!TO!A!DESKTOP!COMPUTER,!ACQUIRES!THE!PRESSURE!DATA.
`!.............................................................................................................................................................!27!
`FIGURE!11!(A)!TOP!SQUEEZE!GESTURE:!THE!USER!POSITIONS!THE!INDEX!AND!MIDDLE!FINGERS!NEAR!THE!TOP!CORNER!OF!THE!
`MOBILE.!TYPICALLY,!THE!OTHER!FINGERS!ARE!NOT!RAISED!(THE!LITTLE!FINGER!IS!SHOWN!EXTENDED!FOR!CLARITY)!(B)!
`MIDDLE!SQUEEZE!GESTURE.!THE!MIDDLE!AND!RING!FINGER!ARE!POSITIONED!AROUND!THE!MIDDLE!OF!THE!DEVICE!AND!
`PUSH!INWARDS,!AND!(C)!THE!RING!AND!LITTLE!FINGERS!SQUEEZE!THE!BOTTOM!CORNER!INWARDS.!...........................!31!
`FIGURE!13.!THE!UNIFONE!PRESSURE!SENSOR!MODULE:!TWO!ALUMINUM!PLANKS!SANDWICH!PRESSURE!SENSORS!ON!OPPOSITE!
`ENDS.!.....................................................................................................................................................!36!
`FIGURE!14!RESULTS!FOR!UNIFONE!EXPERIMENT.!STANDARD!ERROR!ABOVE!AND!BELOW!THE!MEAN!IS!SHOWN.!...................!42!
`FIGURE!15.!THE!ISOPHONE!PROTOTYPE!WITH!LIVE!SENSOR!FEEDBACK!SHOWN!ON!SCREEN.!..............................................!45!
`FIGURE!16.!THE!ISOPHONE!SENSOR!DESIGN!SCHEMATIC.!...........................................................................................!46!
`FIGURE!17.!THE!ISOPHONE!SENSOR!TO!DEVICE!CONNECTION!SCHEMATIC.!....................................................................!47!
`FIGURE!18.!STATE!DIAGRAM!FOR!ISOPHONE!INPUT!AND!GESTURE!CLASSIFICATION.!........................................................!50!
`FIGURE!19.!FORCE!VS.!TIME!DIAGRAM!USED!TO!ILLUSTRATE!ISOPHONE’S!INTER9RESPONSE!INTERVAL!................................!53!
`FIGURE!20.!THUMB!CONTACT!FREQUENCY!DIFFERENCE!FROM!AVG.!FOR!THE!(A)!INDEX,!(B)!MIDDLE!(C)!RING!AND,!(D)!LITTLE!!
`FINGERS.!RED!AREAS!REPRESENT!SCREEN!LOCATIONS!THAT!WERE!CONTACTED!MORE!FREQUENTLY!THAN!AVG.!.............!58!
`FIGURE!21.!NORMALIZED!PRESSURE!FROM!THE!MIDDLE!AND!RING!(A),!AND!INDEX!AND!RING!(B)!......................................!59!
`FIGURE!22.!GRAPHICAL!REPRESENTATIONS!OF!DATA!COLLECTED!DURING!THE!ISOPHONE!STUDY.!AVERAGES!ACROSS!
`PARTICIPANTS!ARE!SHOWN!FOR!A)!SCREEN!AREA!REACHED!BY!THE!THUMB!B)!TLX!SCORE!(REMINDER:!A!LOWER!TLX!SCORE!
`INDICATES!A!MORE!FAVORABLE!RESULT)!C)!INTER9RESPONSE!INTERVAL!D)!FORCE!PER!FINGER.!..................................!66!
`
`
`
`
`
`
`vii
`
`Neonode Smartphone LLC, Exhibit 2040
`Page 2040 - 8
`IPR2021-01041, Google LLC v. Neonode Smartphone LLC
`
`

`

`List of Tables
`
`
`TABLE!1.!SCREEN!AREA!REACHED!BY!THUMB!IN!CM2!.................................................................................................!54!
`TABLE!2.!TLX!SCORES!........................................................................................................................................!55!
`TABLE!3!INTER9RESPONSE!INTERVALS!IN!MILLISECONDS!.............................................................................................!56!
`
`
`
`
`
`viii
`
`Neonode Smartphone LLC, Exhibit 2040
`Page 2040 - 9
`IPR2021-01041, Google LLC v. Neonode Smartphone LLC
`
`

`

`
`
`Chapter 1
`
`
`
`Introduction
`
`1.1 Background and Motivation
`
`Advancements in mobile technology have enabled highly portable devices that, compared to
`
`earlier feature phones, contain a significant amount of computational power. Before the iPhone’s
`
`introduction in 2007, many mobile devices had their interactive surface area divided between a
`
`small screen and a number of physical controls. Since then, there has been a trend of increased
`
`display size and a corresponding decrease in the physical space allotted for hardware controls.
`
`This effect is explained by the touch-sensitive display (touchscreen), and its dual purpose for
`
`input and output; it affords the replacement of physical controls with virtual buttons and gestures.
`
`Although devices with touchscreen make many tasks easier, such as web browsing or composing
`
`an email, they are still limited for more complex tasks, such as document editing and three-
`
`dimensional spatial navigation.
`
`In particular, one limitation is that the viewable area of a touchscreen is partially occluded by a
`
`user's finger when in use. Although a minor obstruction creates a negligible impact on tasks that
`
`do not require precise targeting, such as scrolling through an email, it is prohibitive when more
`
`precise targeting or manipulation is required (e.g., selecting a sentence while editing a
`
`document). Additionally, complex tasks such as text entry, drawing, and 3D spatial interactions
`
`often require numerous on-screen buttons and additional Graphical User Interface (GUI) controls
`
`that occupy valuable display area.
`
`Often, touch gestures are used to increase the range and number of interaction available without
`
`occupying additional display space. These gestures can be highly usable when they embody a
`
`
`
`1
`
`Neonode Smartphone LLC, Exhibit 2040
`Page 2040 - 10
`IPR2021-01041, Google LLC v. Neonode Smartphone LLC
`
`

`

`strong physical metaphor such as swipe-to-scroll and pinch-to-zoom navigation gestures found
`
`on most smartphones today. As the number of available gestures increases, so does complexity
`
`and, typically, user error. Even relatively simple gesture combinations, such as slide-to-scroll
`
`and tap-to-select, often result in false positives. This is further complicated when a user has only
`
`one hand available, leaving the thumb of their grasping hand to act as a single touch point. When
`
`designing interactions for one-handed use, the contact area between device and user becomes an
`
`even more valuable and premium resource. We must maximize its use as an interactive surface if
`
`we are to create better mobile user experiences.
`
`1.2 Contributions and thesis outline
`
`Although gestures that use tilt, acceleration, spatial location, and even deformation have been
`
`actively researched in one-handed scenarios [6,13,16,18] the user’s additional fingers that grasp,
`
`or rest along the device are often overlooked as input channels. They can be used to support or
`
`extend the thumb’s primary pointing behavior. In this thesis, we explore how input from these
`
`auxiliary fingers can impact the usability of mobile interactions.
`
`As an initial step in this space, we focus on one-handed interactions that rely on a user’s
`
`nondominant grasp. Users often adopt one-handed strategies when interacting with mobile
`
`devices; this leaves a hand free to manage the demands of the real world. The thumb, stabilized
`
`by the hand’s supporting fingers, acts as the primary pointing digit in these scenarios. The
`
`supporting fingers are sometimes delegated to controlling hardware buttons for settings like
`
`volume or power. Most often, they are unused. Though poorly suited for precision, their
`
`movement is not completely inhibited. Even when the thumb is actively targeting, the individual
`
`fingers of a user’s nondominant grasp are capable of coarse isometric manipulations; the
`
`fingertips protrude and curl around the edge of the device. When grasping a mobile phone this
`
`way, gestures like squeezing the edges of a device are feasible as an input modality.
`
`
`
`2
`
`Neonode Smartphone LLC, Exhibit 2040
`Page 2040 - 11
`IPR2021-01041, Google LLC v. Neonode Smartphone LLC
`
`

`

`Leveraging this auxiliary finger input also influences the design tradeoff between screen size and
`
`interface complexity. Modern mobile devices, such as the Apple iPhone and Samsung Galaxy,
`
`have adopted high-resolution multi-touch displays that make it easier to interact with complex
`
`content. With greater demands on display area, some applications, like Google Earth, limit the
`
`functionality of their mobile version. Others leave their functionality intact and distribute
`
`workflows across numerous screens and repetitive interface button presses. Instead, using
`
`auxiliary input supports transferring interface control elements off the display; for example,
`
`auxiliary input gestures can be mapped to general navigational controls, leaving more area free to
`
`render content. Although dedicated hardware buttons, such as the trumpet-like buttons in
`
`Weiser’s PARCTab [51], can theoretically achieve similar results, an industrial design that is
`
`ergonomic for a range of hand and finger sizes is challenging (especially if buttons are placed
`
`along the top edge of a device larger than the ParcTab).
`
`To address these problems and to explore the use of auxiliary finger input, we present two
`
`prototypes that improve one-handed mobile interaction: Unifone and Isophone. The first
`
`prototype, Unifone, senses isometric manipulations using a pressure-distributing accessory placed
`
`along its outer edge. Using this hardware sensor, Unifone affords coarse targeting [17]: instead of
`
`a precisely located hardware button, users squeeze the phone near its middle or corners, an
`
`ergonomic improvement that can adjust sensor location for each user and enhance the pointing
`
`behavior of the thumb (see section 4.3). We report on the evaluation of Unifone’s three squeeze-
`
`based gestures—a middle squeeze, top corner squeeze, and bottom corner squeeze—and compare
`
`them against thumb-only interaction in a set of common mobile tasks. When tightly coupled with
`
`the movement of the thumb, Unifone’s squeeze gesture results in superior performance for a set
`
`of common mobile tasks.
`
`Unifone’s force-distributed sensor layout reduced the need for precise targeting, but it still placed
`
`significant constraints on how the device needed to be held, with some fingers on its upper half
`
`
`
`3
`
`Neonode Smartphone LLC, Exhibit 2040
`Page 2040 - 12
`IPR2021-01041, Google LLC v. Neonode Smartphone LLC
`
`

`

`and others on the lower half. Isophone, the second prototype, uses a more sophisticated hardware
`
`design with a high spatial resolution sensor array, capable of differentiating individual fingers
`
`placed anywhere along its length. This further eliminates the need for precise finger placement
`
`during interaction and allowed us to perform a study informing decisions about the placement of
`
`hardware controls on the side of the device as well as the placement of onscreen software based
`
`controls.
`
`This thesis is presented in six chapters. This first chapter has introduced the limitations of a
`
`conventional mobile touchscreen device and revealed the motivation behind including input from
`
`auxiliary fingers into their design. The second chapter provides an overview of past work in
`
`mobile device interaction design, sensor augmentation and the related mechanics of the human
`
`hand. Our first prototype, Unifone, is discussed in chapter three along with a detailed discussion
`
`of finger pressure sensing requirements and their enabling technologies. Our discussion moves
`
`from discovery — in a series of iterative prototypes we tried a number of different sensor
`
`placements and interaction types — to refinement in the Unifone prototype where we improve on
`
`the interaction schemes that showed the most promise. We then discuss a comprehensive user
`
`study. The second major prototype, Isophone, is presented in chapter four. Using this prototype
`
`we evaluated the impact of using different fingers on the thumb’s ability to interact with the
`
`screen. We provide a summary of our hardware and software recommendations gathered from
`
`our user studies in chapter five. We conclude with a discussion of how we foresee the research
`
`continuing and how new technologies will enable this work to be applied to non-planar
`
`deformable devices in chapter six.
`
`
`
`
`
`4
`
`Neonode Smartphone LLC, Exhibit 2040
`Page 2040 - 13
`IPR2021-01041, Google LLC v. Neonode Smartphone LLC
`
`

`

`
`
`Chapter 2
`
`
`
`Related Work
`
`Our research builds upon and draws cues from the following areas of previous research: (1) the
`
`genesis of ‘soft machine’ mobile interfaces, one-handed mobile usage, and interaction techniques
`
`related to grasp; (2) extending one-handed interactions through spatial sensing, surface
`
`deformation, and examples that leverage pressure-based input for one-handed and bimanual
`
`interaction; (3) an overview of the anatomy and function of the human hand. This final section
`
`draws many references relating to human hand performance from the fields of anatomy,
`
`ergonomics and neuroscience, and is particularly relevant to the study reported in chapter four.
`
`2.1 Soft Machines
`
`Soft machines were proposed by Nakatini and Rohrlich in [31] as a digital alternative to “hard
`
`machines” which they describe as "machines such as stoves, radios and copiers operated with
`
`knobs, switches, keys, pushbuttons and other familiar controls. Hard machines have many
`
`characteristics that make for ease of learning, efficiency of operation and ease of transfer, but
`
`they are ultimately limited by their ‘hardness.’” The authors point to several aspects of hard
`
`machines that offer usability advantages over the general-purpose computers of their time. The
`
`modularity of a special purpose machine keeps the complexity of interaction within reasonable
`
`limits. Scrutiny of the machine’s form leads a user to form conjectures about its function and
`
`operation. The one-to-one mapping between controls and their associated operations limits these
`
`conjectures to a reasonable number and the immediate feedback of physical controls allows a user
`
`to test their conjectures and stimulates the formation of new conjectures in a short amount of
`
`time. In contrast to the symbolic operations that require the learning of a human invented
`
`5
`
`Neonode Smartphone LLC, Exhibit 2040
`Page 2040 - 14
`IPR2021-01041, Google LLC v. Neonode Smartphone LLC
`
`

`

`language, the manual controls of a machine, “conform to a universal language based on the
`
`physical laws that govern the interactions between physical objects” and can often be learned
`
`without instruction or training. This ability to casually learn a machine, the ability to transfer
`
`knowledge between machines, and the efficiency of specialized controls are identified by the
`
`authors as the primary advantages of a hard machine over a general-purpose computer.
`
`While physical controls on hard machines clearly offer usability advantages, their material
`
`inflexibility is limiting. As stated by Nakatini and Rohrlich, “we are now in an awkward situation
`
`where the functionality of machines is easily changed by software, but the inflexibility of hard
`
`controls severely limits the changes that can be accommodated without changing the hardware or
`
`compromising the operability of the machine.” They suggest that a way out of this “awkward
`
`situation” is through the use of what they refer to as a “soft machine,” or a virtual representation
`
`of a physical machine composed of computer images displayed on a touch-screen display. Such a
`
`machine offers the “universal language” of physical controls as well as the flexibility and support
`
`of sequential disclosure of controls associated a general-purpose computer.
`
`The earliest commercial implementation of a soft machine is found on the control console of the
`
`1980 XEROX 5700 photocopier. Xerox introduced the system out of necessity, a growing
`
`number of photocopier features such as copying, duplexing, reducing, collating, stapling,
`
`typesetting and printing, would have required more than 130 buttons to control [8]. Instead of
`
`creating an extremely large, physically cumbersome, and overwhelming console, Xerox engineers
`
`opted for a black and white screen with an infrared touch sensor. A home screen presented soft
`
`button representations of the features available machine, and once touched the screen was
`
`replaced by the specific controls available to that feature.
`
`
`
`6
`
`Neonode Smartphone LLC, Exhibit 2040
`Page 2040 - 15
`IPR2021-01041, Google LLC v. Neonode Smartphone LLC
`
`

`

`
`
`Figure 1. The Xerox 5700 printing system with a touchscreen interface [50]
`
`
`
`In 1992, IBM and Bell South brought the soft machine philosophy to a mobile device with the
`
`release of the Simon smartphone. With limited network coverage, and with miniaturization of
`
`technology still in nascent stages, the Simon was could not gain widespread adoption. However,
`
`the interface style became quite popular on other mobile devices such as the Apple Newton and
`
`the Palm Pilot. The large number of features and constrained dimensions of these handheld
`
`devices made soft controls even more attractive than they had been on a desktop counterpart.
`
`Largely relying on discrete soft buttons combined with pen input, these early mobile soft
`
`machines lacked the continuous direct input gestures and more sophisticated physical metaphors
`
`that researchers had been developing on larger mobile, and stationary systems [23].
`
`
`
`7
`
`Neonode Smartphone LLC, Exhibit 2040
`Page 2040 - 16
`IPR2021-01041, Google LLC v. Neonode Smartphone LLC
`
`

`

`The Neonode N1 was the first commercially available mobile device to make extensive use of
`
`swipe gestures appropriate for one-handed use, including a browser that scrolled content
`
`vertically with swipes. The gestures were more limited than the continuous direct input based
`
`gestures we are all familiar with today. Swiping up scrolled content up and swiping down scrolled
`
`content down similar to the PC touchpads of the time. However, its menu lists did have a
`
`highlighted element that moved down with a downward swipe and up with an upward swipe in
`
`what was very close to a direct physical mapping. It was not until the release of the first iPhone in
`
`200