Technische Universität München
`
`Fakultät für Maschinenwesen
`
`Lehrstuhl für Ergonomie
`
`Suitability of Touch Gestures and Virtual Physics in Touchscreen User
`
`Interfaces for Critical Tasks
`
`Dipl.-Ing. Jurek Breuninger
`
`Vollständiger Abdruck der von der
`
`Fakultät Maschinenwesen der Technischen Universität München
`
`zur Erlangung des akademischen Grades eines
`
`Doktor-Ingenieurs (Dr.-Ing.)
`
`genehmigten Dissertation.
`
`Vorsitzender:
`
`Prof. Dr.-Ing. Gunther Reinhart
`
`Prüfer der Dissertation: 1. Prof. Dr. phil. Klaus Bengler
`
`2. Prof. Dr. Heinrich Hußmann
`
`Die Dissertation wurde am 13.11.2019 bei der Technischen Universität Mün-
`
`chen eingereicht und durch die Fakultät Maschinenwesen am 28.05.2020 an-
`
`genommen.
`
`Neonode Smartphone LLC, Exhibit 2018
`Page 2018 - 1
`IPR2021-00144, Samsung Elecs. Co. Ltd. et al. v. Neonode Smartphone LLC
`
`

`

`Neonode Smartphone LLC, Exhibit 2018
`Page 2018 - 2
`IPR2021-00144, Samsung Elecs. Co. Ltd. et al. v. Neonode Smartphone LLC
`
`

`

`Neonode Smartphone LLC, Exhibit 2018
`Page 2018 - 3
`IPR2021-00144, Samsung Elecs. Co. Ltd. et al. v. Neonode Smartphone LLC
`
`

`

`ZUSAMMENFASSUNG
`
`Zusammenfassung
`
`Das Ziel dieser Forschungsarbeit war es zu untersuchen, ob moderne Touchscreen-
`
`Interaktionskonzepte, die auf Consumer-Electronic-Geräten wie Smartphones etab-
`
`liert sind, für zeit- und sicherheitskritische Anwendungsfälle wie Maschinensteuerung
`
`und Medizingeräte geeignet sind. Mehrere gebräuchliche Interaktionskonzepte mit
`
`und ohne Touch-Gesten und virtueller Physik wurden experimentell auf ihre Effizienz,
`
`Fehlerrate und Nutzerzufriedenheit bei der Aufgabenlösung untersucht. Basierend
`
`auf den Resultaten werden Empfehlungen für das Scrollen in Listen und das horizon-
`
`talen Navigieren in mehrseitigen Software-Dialogen ausgesprochen.
`
`Der Text gibt eine Übersicht der speziellen Eigenschaften von Touchscreen-Mensch-
`
`Maschine-Schnittstellen und der Unterschiede zu zeigerbasierten Eingabegeräten. Er
`
`beschreibt den aktuellen Stand des Touchscreen-Interaktionsdesigns, v.a. die Be-
`
`sonderheiten moderner Touch-Interaktion, nämlich Touch-Gesten und virtuelle Phy-
`
`sik. Die größten Herausforderungen für Touchscreen-Interaktionsdesign sind Feed-
`
`forward, Feedback, Größe der interaktiven Elemente, Kompatibilität, Effekte virtueller
`
`Physik und Interferenz. Basierend auf einem einfachen qualitativen Modell der Ein-
`
`flussfaktoren beim Touchscreen-Interaktionsdesign sollten die folgenden Hypothesen
`
`zu Effizienz und Sicherheit moderner Touchscreen-Interaktion überprüft werden:
`
`Touch-Gesten führen zu langsamerer Aufgabenerfüllung, höherer Fehlerrate, aber
`
`besserer Nutzerbewertung. Beim Scrollen führt virtuelle Trägheit zu schnellerer Auf-
`
`gabenerfüllung, aber auch zu mehr Über-das-Ziel-Hinausschießen und höherer Feh-
`
`lerrate. Seitenweises Blättern führt zu schnellerer Aufgabenerfüllung und geringerer
`
`Fehlerrate als kontinuierliche Inhalte. Um dies zu überprüfen, wurden mehrere Expe-
`
`rimente durchgeführt, die Interaktionskonzepte häufiger Aufgaben vergleichen: Me-
`
`nüs, Funktionswähler, Zahleneingabe, Listen-Scrollen und horizontaler Ansichts-
`
`wechsel. Der Einfluss des Interaktionsdesigns auf Eingabegeschwindigkeit, Fehlerra-
`
`te und Nutzerbewertung wird für Listen-Scrollen und horizontalen Ansichtswechsel
`
`deutlich gezeigt. Eine mit Wischgesten gesteuerte Liste mit virtueller Trägheit und
`
`Alphabetleiste ist die beste Wahl für das Scrollen von Listen aller Längen. Um hori-
`
`zontal durch Ansichten zu navigieren, sind Tabs die geeignetste Wahl für kritische
`
`Aufgaben. Touch-Gesten können zu höherer Fehlerrate führen, aber vernünftig ge-
`
`staltete Konzepte mit Touch-Gesten können dennoch für kritische Aufgaben geeignet
`
`sein. Die Nutzerbewertung von Touch-Interaktionskonzepten korreliert stark mit der
`
`Eingabegeschwindigkeit. Fehler scheinen keinen Einfluss darauf zu haben.
`
`I
`
`Neonode Smartphone LLC, Exhibit 2018
`Page 2018 - 4
`IPR2021-00144, Samsung Elecs. Co. Ltd. et al. v. Neonode Smartphone LLC
`
`

`

`ABSTRACT
`
`Abstract
`
`The goal of this research was to examine if modern touchscreen interaction concepts
`
`that are established on consumer electronic devices like smartphones can be used in
`
`time-critical and safety-critical use cases like for machine control or healthcare appli-
`
`ances. Several prevalent interaction concepts with and without touch gestures and
`
`virtual physics were tested experimentally in common use cases to assess their effi-
`
`ciency, error rate and user satisfaction during task completion. Based on the results,
`
`design recommendations for list scrolling and horizontal dialog navigation are given.
`
`The text gives an overview of the special characteristics of touchscreen human–
`
`machine interfaces and their differences to pointer-based input devices. It describes
`
`the state of the art of user interface design for touchscreens, particularly the interac-
`
`tion concepts that distinguish modern touchscreen interaction with tablets and
`
`smartphones from older interaction concepts, namely touch gestures and virtual
`
`physics. Due to the use of these interaction concepts and the special characteristics
`
`of touchscreens, the main challenges of user interface design for touchscreen are
`
`feedforward, feedback, size of interactive elements, compatibility, effects of virtual
`
`physics, and interference. Based on a simple qualitative model of influence factors in
`
`touchscreen interaction design, the following hypotheses concerning the efficiency
`
`and safety of modern touchscreen interaction are to be tested: Touch gestures lead
`
`to slower task completion, higher error rate, but better user rating. For scrolling tasks,
`
`virtual inertia leads to faster task completion, but more overshooting and higher error
`
`rate. Paged content leads to faster task completion and lower error rate than contin-
`
`uous content. To test the hypotheses, several experiments were conducted that
`
`compare interaction concepts in common tasks: Menus, function selectors, numerical
`
`input, list scrolling, and horizontal content change. For list scrolling and horizontal
`
`content change, the influence of interaction design on input speed, error rate, and
`
`user rating is clearly shown. A list that can be moved with a swiping gesture and that
`
`has virtual inertia and an alphabetic index bar is the best choice for scrolling lists of
`
`all lengths. To navigate through horizontal content, tabs are the most suitable choice
`
`for critical tasks. The use of touch gestures can lead to higher error rates, but rea-
`
`sonably designed concepts with touch gestures can still be suitable for critical tasks.
`
`The user ratings of touch interaction concepts correlate strongly with the input speed.
`
`Errors and overshoots seem to have no impact.
`
`II
`
`Neonode Smartphone LLC, Exhibit 2018
`Page 2018 - 5
`IPR2021-00144, Samsung Elecs. Co. Ltd. et al. v. Neonode Smartphone LLC
`
`

`

`ACKNOWLEDGEMENTS
`
`Acknowledgements
`
`I would like to thank the following people for their help and support during the work
`
`on this research project and my time at the Chair of Ergonomics:
`
`− Professor Klaus Bengler for giving me the opportunity to work as a researcher
`
`and teaching assistant at the Chair of Ergonomics, for encouraging young re-
`
`searchers to question existing assumptions while scrutinizing und challenging
`
`their approaches carefully.
`
`− Professor Heiner Bubb for giving me the opportunity to conduct my first re-
`
`search in software ergonomics as a student, which led me to making my pas-
`
`sion into my profession, and for being a kind, attentive and always available
`
`partner for discussions concerning research methodology and ergonomics in
`
`general.
`
`− Severina Popova-Dlugosch, Uwe Herbst, and Carsten Dlugosch for their help
`
`and support concerning project work, publications, lectures, student supervi-
`
`sion, and discussing research approaches; for being my dearest colleagues
`
`and friends during this great time and afterwards.
`
`− Professor Armin Eichinger, Michael Stecher, Andreas Haslbeck and my other
`
`colleagues at the Chair of Ergonomics for giving insights and opinions on re-
`
`search methodologies, relevant literature and being helpful and encouraging
`
`discussion partners.
`
`− Professor Heinrich Hußmann for reviewing this thesis and his input on litera-
`
`ture.
`
`− Professor Erich Hollnagel and Professor Don Norman for being heartening,
`
`but challenging in discussing the relevance of this research, methodology, and
`
`publication strategy.
`
`− Steffen Bauereiß, Emmanuel el-Khoury, Michael Enslin, Jakob Haug, Benedikt
`
`Hirmer, Lisa Hüfner, Clara Lange, Amel Mahmuzic, Felix Menzel, Nađa
`
`Šahinagić, and Tom Schelo for carrying out experiments that were part of this
`
`research project.
`
`− My wife and family for everything they do.
`
`III
`
`Neonode Smartphone LLC, Exhibit 2018
`Page 2018 - 6
`IPR2021-00144, Samsung Elecs. Co. Ltd. et al. v. Neonode Smartphone LLC
`
`

`

`TABLE OF CONTENTS
`
`Table of Contents
`
`1
`
`2
`
`3
`
`3.1
`
`3.1.1
`
`3.1.2
`
`3.1.3
`
`3.2
`
`3.2.1
`
`3.2.2
`
`3.2.3
`
`3.2.4
`
`3.2.5
`
`3.2.6
`
`3.2.7
`
`4
`
`4.1
`
`4.1.1
`
`4.1.2
`
`4.1.3
`
`4.1.4
`
`4.2
`
`4.2.1
`
`4.2.2
`
`4.2.3
`
`4.2.4
`
`4.3
`
`4.4
`
`INTRODUCTION
`
`A SHORT HISTORY OF TOUCHSCREEN INTERACTION
`
`HUMAN FACTORS IN TOUCHSCREEN INTERACTION
`
`Fundamentals of Human–Computer Interaction
`
`Criticality of Tasks
`
`Usability
`
`Established Usability Requirements
`
`1
`
`4
`
`12
`
`12
`
`12
`
`14
`
`17
`
`Special Characteristics of Touchscreen Interaction and Differences
`
`to Pointer-Based Input Devices
`
`Occlusion
`
`Feedback
`
`Precision
`
`Variability
`
`Posture
`
`Complexity
`
`Summary
`
`TOUCHSCREEN INTERACTION DESIGN
`
`Basics
`
`Differences in Interaction Design compared to Pointer-Based Input
`Devices
`
`Models of Touchscreen Input Speed
`
`Standards
`
`Guidelines
`
`Gesture-Based Interaction
`
`Direct Manipulation
`
`Virtual Physics
`
`Semantic and Symbolic Gestures
`
`18
`
`18
`
`19
`
`20
`
`23
`
`25
`
`28
`
`29
`
`30
`
`30
`
`30
`
`32
`
`32
`
`33
`
`33
`
`34
`
`40
`
`43
`
`Difference in Gesture-Based Interaction compared to Pointer-Based Input
`Devices
`44
`
`Novel Interaction Concepts
`
`Common Usability Problems in Touchscreen Interaction Design
`
`44
`
`45
`
`IV
`
`Neonode Smartphone LLC, Exhibit 2018
`Page 2018 - 7
`IPR2021-00144, Samsung Elecs. Co. Ltd. et al. v. Neonode Smartphone LLC
`
`

`

`TABLE OF CONTENTS
`
`Feedforward
`
`Feedback
`
`Size of Interactive Elements
`
`Compatibility
`
`Effects of Virtual Physics
`
`Interference
`
`A MODEL FOR ERGONOMIC TOUCHSCREEN INTERACTION
`
`The Human–Machine Control Loop
`
`Quality of the Result
`
`Factors that Influence the Quality of the Result
`
`Known Dependencies in the Model
`
`RESEARCH QUESTIONS
`
`Applicability of the Model to Critical Tasks
`
`Primary Research Interest
`
`Further Pending Research Questions
`
`Feedforward of Direct Manipulation
`
`Interference
`
`Strain during Repeated or Continuous Use
`
`Input Gain
`
`Hypotheses
`
`EXPERIMENTS
`
`Considerations on Experiment Design
`
`Overview
`
`Conscious Activation
`
`Menus
`
`Function Selectors
`
`Numerical Input
`
`Smart Home Control Demonstrator
`
`List Scrolling
`
`4.4.1
`
`4.4.2
`
`4.4.3
`
`4.4.4
`
`4.4.5
`
`4.4.6
`
`5
`
`5.1
`
`5.1.1
`
`5.1.2
`
`5.2
`
`6
`
`6.1
`
`6.2
`
`6.3
`
`6.3.1
`
`6.3.2
`
`6.3.3
`
`6.3.4
`
`6.4
`
`7
`
`7.1
`
`7.2
`
`7.3
`
`7.4
`
`7.5
`
`7.6
`
`7.7
`
`7.8
`
`7.8.1
`
`7.8.2
`
`7.8.3
`
`Variants
`
`Procedure
`
`Results
`
`V
`
`45
`
`46
`
`47
`
`48
`
`50
`
`50
`
`52
`
`52
`
`53
`
`53
`
`54
`
`56
`
`56
`
`56
`
`57
`
`57
`
`57
`
`58
`
`58
`
`58
`
`61
`
`61
`
`62
`
`62
`
`64
`
`66
`
`68
`
`70
`
`70
`
`73
`
`78
`
`82
`
`Neonode Smartphone LLC, Exhibit 2018
`Page 2018 - 8
`IPR2021-00144, Samsung Elecs. Co. Ltd. et al. v. Neonode Smartphone LLC
`
`

`

`TABLE OF CONTENTS
`
`7.8.4
`
`Discussion
`
`7.9
`
`7.9.1
`
`7.9.2
`
`7.9.3
`
`7.9.4
`
`8
`
`8.1
`
`8.2
`
`8.3
`
`9
`
`Horizontal Content Change
`
`Variants
`
`Procedure
`
`Results
`
`Discussion
`
`PRACTICAL IMPLICATIONS
`
`Recommendations based on the Results
`
`Assessment of Validity and Practicality
`
`Publication of the Results
`
`SUMMARY AND OUTLOOK
`
`GLOSSARY
`
`REFERENCES
`
`APPENDIX A: STATISTICAL ANALYSIS
`
`APPENDIX B: QUESTIONNAIRES
`
`97
`
`100
`
`102
`
`107
`
`114
`
`123
`
`127
`
`127
`
`129
`
`131
`
`132
`
`134
`
`137
`
`159
`
`204
`
`VI
`
`Neonode Smartphone LLC, Exhibit 2018
`Page 2018 - 9
`IPR2021-00144, Samsung Elecs. Co. Ltd. et al. v. Neonode Smartphone LLC
`
`

`

`
`
`
`
`
`
`VII
`
`Neonode Smartphone LLC, Exhibit 2018
`Page 2018 - 10
`IPR2021-00144, Samsung Elecs. Co. Ltd. et al. v. Neonode Smartphone LLC
`
`

`

`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Everything is best for something and worst for something else.
`
`— Bill Buxton
`
`VIII
`
`Neonode Smartphone LLC, Exhibit 2018
`Page 2018 - 11
`IPR2021-00144, Samsung Elecs. Co. Ltd. et al. v. Neonode Smartphone LLC
`
`

`

`INTRODUCTION
`
`1 Introduction
`
`Displays that can not only output information, but also detect and localize touches on
`
`their surface to serve as an input device, have been available for several decades
`
`now and are a proven and common input technology in many modern electronic de-
`
`vices. These touchscreens offer a number of advantages over other input devices
`
`like keyboards, mice, touch pads, voice or gesture recognition. They need almost no
`
`additional space in all devices that already have displays. By closely integrating input
`
`and output, they allow for a form of human–computer interaction that can be consid-
`
`ered especially direct because information presentation, user input and visual feed-
`
`back all take place at the same location.
`
`Although touchscreens are a long-standing technology (Shneiderman, 1991) as far
`
`as computer technology is concerned, they have only become as important and
`
`ubiquitous as they are today in the last few years. Because of vast technical im-
`
`provements and an ongoing process of miniaturization, touchscreens are today
`
`available and suitable for a wide variety of device classes, foremost for numerous
`
`forms of mobile devices. Especially the establishment of smartphones, which began
`
`with the iPhone in 2007, has put a mobile touchscreen device in almost everybody’s
`
`pocket. Tablets and convertible laptop computers continue to add to the success of
`
`the touchscreen. With these new device classes, new interaction paradigms were
`
`introduced and established, mainly by the most successful vendors Apple, Google,
`
`and Microsoft. These new paradigms make use of the improved abilities of modern
`
`capacitive touchscreens to detect sliding finger motions on the screen continuously
`
`and without delay (Figure 1). Faster microprocessors allow instantaneous and realis-
`
`tic dynamic visual feedback based on physical metaphors. Touch gestures and virtu-
`
`al physics have become state of the art in touchscreen devices and they are used in
`
`almost all modern consumer electronics.
`
`Yet the high rate of innovation of user interfaces that is driven by the high-volume
`
`market and short development cycles of consumer electronics, which allow for fast
`
`return of investment and quick changes in strategy, has not arrived in other fields.
`
`Where investment cycles are longer lasting, introduction of new technologies will oc-
`
`cur with a delay. More importantly, in fields where the human–computer interaction is
`
`part of a task that might have severe consequences for economic profitability or hu-
`
`man safety, decision makers are more likely to trust in proven concepts than to adapt
`
`1
`
`Neonode Smartphone LLC, Exhibit 2018
`Page 2018 - 12
`IPR2021-00144, Samsung Elecs. Co. Ltd. et al. v. Neonode Smartphone LLC
`
`

`

`INTRODUCTION
`
`young technologies (Hartmann, 2012; Wiedenberg, 2012). Therefore, although ges-
`
`ture-based touchscreen interaction has been the state of the art for some time, it is
`
`only adapted slowly in factories, power plants, process engineering and medical de-
`
`vices.
`
`Figure 1: An overview of possible touchscreen gestures. [Source: gestureworks.com]
`
`
`
`While they have been using touchscreen technology for many years, they tend to
`
`offer conservative virtual-button-based user interfaces (Figure 2). Others copy ele-
`
`2
`
`Neonode Smartphone LLC, Exhibit 2018
`Page 2018 - 13
`IPR2021-00144, Samsung Elecs. Co. Ltd. et al. v. Neonode Smartphone LLC
`
`

`

`INTRODUCTION
`
`ments of consumer electronic user interfaces without any adaption to the circum-
`
`stances of their field of application. Moreover, some develop new interaction con-
`
`cepts without any experimental validation, which is concerning from an ergonomic
`
`point of view.
`
`Figure 2: An engineer using a touchscreen in an industrial environment. [Source:
`www.heidenhain.de]
`
`
`
`For an adaption of modern interaction paradigms and a suitable and correct imple-
`
`mentation in the devices, easily applicable and scientifically verified guidelines are
`
`needed that explicitly address touchscreen interaction for critical tasks without arbi-
`
`trary focus on vendor-specific hardware, software frameworks or visual design strat-
`
`egies. This thesis documents research that aims to find and validate touchscreen
`
`interaction paradigms that are suitable for critical tasks. Certain use cases were stud-
`
`ied to give recommendations of ergonomic design while tapping the full potential of
`
`modern touchscreen technology.
`
`3
`
`Neonode Smartphone LLC, Exhibit 2018
`Page 2018 - 14
`IPR2021-00144, Samsung Elecs. Co. Ltd. et al. v. Neonode Smartphone LLC
`
`

`

`A SHORT HISTORY OF TOUCHSCREEN INTERACTION
`
`2 A Short History of Touchscreen Interaction
`
`Interacting with objects directly on a computer screen is a technology almost as old
`
`as electronic computers themselves. At first, it was only possible with stylus-like de-
`
`vices, called light guns or light pens (Figure 3), which were used as early as 1952
`
`with the MIT’s Whirlwind computer (Carlson, 2009; Freedman, 2015).
`
`Figure 3: MIT's Whirlwind computer was the first to allow for direct interaction on the
`screen using a light pen. [Source: https://history-computer.com/ModernComputer/
`Electronic/Whirlwind.html]
`
`
`
`The first descriptions of the mode of operation of capacitive touchscreen and working
`
`prototypes of ‘touch displays’ that could be operated with the fingers were published
`
`by Johnson in the 1960s (Johnson, 1965, 1967). They were intended for radar opera-
`
`tors as described by Orr and Hopkins (1968), who were the first to analyze the poten-
`
`tial of this new input technology to improve the workplace and performance of air traf-
`
`fic controllers. These early touchscreens used thin copper wires stretched over the
`
`display, which obstructed the view of the operator somewhat depending on the densi-
`
`ty of the wire matrix (Figure 4).
`
`The first transparent touchscreen was developed and put to daily use at CERN in the
`
`early 1970s, but it was originally only able to detect nine different touch areas on the
`
`screen, later sixteen (CERN, 2010). Touchscreens spread more with the invention of
`
`optical touchscreens in 1972 (US3775560, 1973) and were integrated into computers
`
`like the University of Illinois’ PLATO IV system (Figure 6) and in 1983 into the com-
`
`mercially available HP-150 (Figure 7; YouTube, 2008). The first computer input sys-
`
`tem that allowed multi-touch was a camera-based touch pad rather than a
`
`touchscreen (Mehta, 1982).
`
`4
`
`Neonode Smartphone LLC, Exhibit 2018
`Page 2018 - 15
`IPR2021-00144, Samsung Elecs. Co. Ltd. et al. v. Neonode Smartphone LLC
`
`

`

`A SHORT HISTORY OF TOUCHSCREEN INTERACTION
`
`Figure 4: The first touchscreen had visible wires running across the screen. [Source:
`mraths.org.uk]
`
`
`
`Figure 5: The first transparent, capacitive touchscreens (bottom) were developed and
`used at CERN. [Source: https://cerncourier.com/the-first-capacitative-touch-screens-
`at-cern/]
`
`
`
`5
`
`Neonode Smartphone LLC, Exhibit 2018
`Page 2018 - 16
`IPR2021-00144, Samsung Elecs. Co. Ltd. et al. v. Neonode Smartphone LLC
`
`

`

`A SHORT HISTORY OF TOUCHSCREEN INTERACTION
`
`Figure 6: The University of Illinois' PLATO IV computer system was the first widely
`deployed computer with optical
`touchscreen.
`[Source: https://archives.library.
`illinois.edu/erec/University%20Archives/1505050/BrownBag/BBPlatoIV.htm]
`
`
`
`Figure 7: The Hewlett-Packard 150 computer system was the first commercially
`available
`touchscreen computer.
`[Source: http://www.vintagecomputing.com/
`index.php/archives/356/retro-scan-of-the-week-the-hp-150-touchscreen-computer]
`
`
`
`6
`
`Neonode Smartphone LLC, Exhibit 2018
`Page 2018 - 17
`IPR2021-00144, Samsung Elecs. Co. Ltd. et al. v. Neonode Smartphone LLC
`
`

`

`A SHORT HISTORY OF TOUCHSCREEN INTERACTION
`
`Krueger (1983, 1991) was the first to describe in depth the possibilities of gestural
`
`human–computer interaction without additional technical devices (e.g. mouse, stylus,
`
`glove) on the basis of Video Place and later Video Desk (Krueger, Gionfriddo, & Hin-
`
`richsen, 1985). Although those systems were not touchscreens in the narrow sense,
`
`in one of the described configurations they worked like modern touch tables. Trans-
`
`parent capacitive touchscreens with multi-touch capabilities were developed at Bell
`
`Labs in 1984 (US4484179, 1984). Although it had to be controlled with a stylus, the
`
`GRiDPad (Figure 8) was the first self-contained mobile touchscreen device in 1989
`
`(Atkinson, 2008).
`
`Figure 8: The first self-contained mobile touchscreen device, the GRiDPad. [Source:
`https://oldcomputers.net/gridpad.html]
`
`
`
`In 1993, Wellner (1993) showed with the DigitalDesk how touchscreen interaction
`
`can be used to augment a work environment like a classic desktop. The first com-
`
`mercially available portable device with a finger-operated touchscreen was the IBM
`
`Simon (Figure 9), considered the first smartphone by some (Buxton, 2007). It was
`
`sold between 1994 and 1995. To address the limitations of touchscreens concerning
`
`haptic feedback, tangible interfaces were introduced in 1995 (Fitzmaurice, Ishii, &
`
`Buxton, 1995). The Portfolio Wall by Alias|Wavefront was a commercially available
`
`wall display that recognized many of the now common touch gestures for direct ma-
`
`nipulation and menu control in 1999 (Buxton, 2007). By 2001, the Diamond Touch
`
`table by Mitsubishi Research Labs (Figure 10) was able to distinguish applied pres-
`
`sure and hands and fingers of different users (Dietz & Leigh, 2001). In 2002, Rekimo-
`
`to (2002) introduced a sensor technology that is able to recognize hand positions,
`
`shapes and their distance from the surface. This capacitive system does not suffer
`
`from light occlusion problems like camera-based ones and can be fully integrated into
`
`the surface.
`
`7
`
`Neonode Smartphone LLC, Exhibit 2018
`Page 2018 - 18
`IPR2021-00144, Samsung Elecs. Co. Ltd. et al. v. Neonode Smartphone LLC
`
`

`

`A SHORT HISTORY OF TOUCHSCREEN INTERACTION
`
`Figure 9: The IBM Simon from 1994 is considered the first smartphone by some.
`[Source:
`https://commons.wikimedia.org/wiki/File:IBM_SImon_in_charging_station.
`png]
`
`
`
`Figure 10: Diamond Touch is a touchscreen table that can be interacted with by sev-
`eral users.
`[Source: MERL-LOBBY by Mergatroid212; https://en.wikipedia.org
`/wiki/File:MERL-LOBBY.JPG ; license: CC BY 3.0]
`
`
`
`8
`
`Neonode Smartphone LLC, Exhibit 2018
`Page 2018 - 19
`IPR2021-00144, Samsung Elecs. Co. Ltd. et al. v. Neonode Smartphone LLC
`
`

`

`A SHORT HISTORY OF TOUCHSCREEN INTERACTION
`
`The Neonode N1 (Figure 11), available in 2004, was the first smartphone to use a
`
`touchscreen as primary input and to support touch gestures for several functions
`
`(Blickenstorfer, 2006; Joire, 2007). Its vibration motor offered some sort of haptic
`
`feedback. The Lemur music controller (Figure 12) was the first commercially availa-
`
`ble touchscreen device with unlimited touch points in 2005 (Stantum Technologies,
`
`2015).
`
`Figure 11: The first smartphone to support touch gestures: The Neonode N1 [Source:
`http://www.gsmhistory.com/vintage-mobiles/fig-36-neonode-n1/]
`
`
`
`Figure 12: The Lemur music controller was the first commercially available
`touchscreen
`device
`that
`supported
`unlimited multi-touch.
`[Source:
`http://www.jazzmutant.com/]
`
`
`
`9
`
`Neonode Smartphone LLC, Exhibit 2018
`Page 2018 - 20
`IPR2021-00144, Samsung Elecs. Co. Ltd. et al. v. Neonode Smartphone LLC
`
`

`

`A SHORT HISTORY OF TOUCHSCREEN INTERACTION
`
`PlayAnywhere was the first touch table that was able to identify and interact with ob-
`
`jects. It displayed corresponding visual output to enhance the possibilities of tangible
`
`interfaces (Wilson, 2005). It led to a commercial product in 2007, the Microsoft Sur-
`
`face, later renamed PixelSense (Robertson, 2012). In its latest iteration, Samsung
`
`SUR40 (Figure 13), it is also an image processor, like a camera (Microsoft, 2015a)
`
`and can detect objects even at some distance.
`
`Figure 13: Touchscreen tables like the Samsung SUR40 allow multi-touch gesture
`interaction and can
`recognize objects
`that
`lie on
`the surface.
`[Source:
`http://nsquaredblog.blogspot.com/2012/07/australian-launch-event-for-samsung.html]
`
`
`
`However, the main cause for today’s massive ubiquity and popularity of touchscreen
`
`devices are modern smartphones and tablets, which were made popular by Apple
`
`beginning in 2007 (Figure 14) with the iPhone and in 2010 with the iPad (Figure 15).
`
`The ongoing commercial success of these device classes leads to rapidly rising sales
`
`of touchscreens (Figure 16) and to a continuing integration of touchscreens into a
`
`wide variety of electronic devices like home appliances, industrial machines, and
`
`medical equipment.
`
`10
`
`Neonode Smartphone LLC, Exhibit 2018
`Page 2018 - 21
`IPR2021-00144, Samsung Elecs. Co. Ltd. et al. v. Neonode Smartphone LLC
`
`

`

`A SHORT HISTORY OF TOUCHSCREEN INTERACTION
`
`Figure 14: The devices that defined modern touchscreen interaction: The original
`Apple iPhone (2007) [Source: https://www.macworld.com/article/3204152/original-
`2007-iphone-photo-album.html]
`
`
`
`
`
`
`
`Figure 15: In 2010, Apple increased the ubiquity of touchscreen devices by introduc-
`ing the iPad. With tablet computers, mobile touchscreen interaction is not limited to
`small screens anymore. [Source: https://www.macwelt.de/a/ipad-1-das-kann-das-
`erste-apple-tablet-heute-noch,3060023]
`
`Figure 16: Annual touchscreen revenues and forecast based on 2012 data. [Source:
`https://www.prweb.com/releases/npd-displaysearch/analysis/prweb9705889.htm]
`
`
`
`11
`
`Neonode Smartphone LLC, Exhibit 2018
`Page 2018 - 22
`IPR2021-00144, Samsung Elecs. Co. Ltd. et al. v. Neonode Smartphone LLC
`
`

`

`HUMAN FACTORS IN TOUCHSCREEN INTERACTION
`
`3 Human Factors in Touchscreen Interaction
`
`3.1 Fundamentals of Human–Computer Interaction
`
`3.1.1 Criticality of Tasks
`
`This thesis focuses on the evaluation of modern software user interfaces on
`
`touchscreen devices intended for use cases where operators have to fulfill critical
`
`tasks. The notion of critical tasks is known in a variety of fields, notably in project
`
`management, where it describes a task in a project that lies on the critical path and
`
`thus influences the time plan of the project. It was originally called critical jobs by the
`
`inventors of Critical Path Planning, Kelley and Walker (1959). Another common use
`
`of the concept of critical tasks deals with their effect on human safety. The definition
`
`used in this thesis is mainly based on the latter, safety-critical tasks, which is the pre-
`
`dominant meaning in the field of human factors. Yet it is extended to include econom-
`
`ic requirements, which are essential to most industrial use cases that are part of the
`
`motivation for this research. This secondary focus on efficiency shows ties to the role
`
`of criticality in Critical Path Planning. All mentions of critical tasks in this text refer to
`
`tasks that can have significant influence on the safety of humans or the economic
`
`viability of a process, as defined by the Department of Defense (2013): “A critical task
`
`is one requiring human performance which, if not accomplished in accordance with
`
`system requirements, will likely have adverse effects on cost, system reliability, effi-
`
`ciency, effectiveness, or safety.” The criticality of the common intended tasks distin-
`
`guishes touchscreen interaction with most consumer electronics from interaction with
`
`devices for healthcare, facility management, and plant control. Those environments,
`
`where tasks as defined by the Department of Defense can occur or are part of the
`
`regular line of action, will be called “critical task environments” in this text. The follow-
`
`ing factors mainly influence the criticality of tasks.
`
`3.1.1.1 Risk
`
`The main difference between a critical task and a non-critical task is that there is a
`
`significant risk of unsuccessful completion of the task. Farmer (1977) defined risk as
`
`the product of the probability of an event and the adversity of its results. This means
`
`the task is either hard to complete successfully or the consequences of an unsuc-
`
`cessful completion are severe (or both). As mentioned above, this can concern either
`
`12
`
`Neonode Smartphone LLC, Exhibit 2018
`Page 2018 - 23
`IPR2021-00144, Samsung Elecs. Co. Ltd. et al. v. Neonode Smartphone LLC
`
`

`

`HUMAN FACTORS IN TOUCHSCREEN INTERACTION
`
`the safety of the people and the material involved or the economic viability of the pro-
`
`cess that contains the task. Tasks with high risk usually result in high costs to
`
`achieve acceptable system reliability, effectiveness, efficiency, and safety. An unsuc-
`
`cessful completion of the task is the result of some kind of error during the procedure.
`
`The error can occur on the part of the machine or on the part of the user. The focus
`
`of this research lies in the human–machine interface, so it is mainly concerned with
`
`understanding and minimizing the risk that is a result of the design of this interface or
`
`can be influenced by the design of the whole man–machine system. Machine failures
`
`may be unavoidable and not be caused by user actions, but may require the possibil-
`
`ity of restarting gracefully and lessen consequences. This adds additional require-
`
`ments to the human–machine interface, where this restarting or additional adjustment
`
`processes have to be triggered. Human errors may be results of individual capabili-
`
`ties or circumstances, but are often also strongly influenced by the design of the hu-
`
`man–machine interface (Reason, 1990).
`
`3.1.1.2 Time Budget
`
`If one follows the criticality definition by the Department of Defense (2013), critical
`
`tasks can be found in any corporate environment because here most tasks have to
`
`be effective and efficient to assure the economic viability of a company. If tasks can
`
`be completed faster, more can be accomplished in the same time frame. This in-
`
`crease in efficiency is desirable from an economic point of view. Therefore, while
`
`there is no immediate necessity for the users to operate faster than they would nor-
`
`mally, the organizational process might include incentives to do so (e.g. wages or
`
`career advancement dependent on throughput).
`
`The time budget can also be clearly defined by process design. In any non-trivial
`
`process, tasks are usually dependent on certain circumstances, usually induced by
`
`other tasks. To assure the effectiveness and efficiency of those tasks, they often
`
`have to be completed within a loosely or very concisely defined time frame. If there is
`
`a clear dependency on another task, the time frame begins with the completion of
`
`this preceding task. If the result of the preceding task is not permanent, the following
`
`task cannot be carried out successfully anymore at some point. While this dependen-
`
`cy on other tasks is often a result of economic considerations in industrial applica-
`
`tions, it can also be a result of technical restrictions, medical requirements, or other
`
`13
`
`Neonode Smartphone LLC, Exhibit 2018
`Page 2018 - 24
`IPR2021-00144, Samsung Elecs. Co. Ltd. et al. v. Neonode Smartphone LLC
`
`

`

`HUMAN FACTORS IN TOUCHSCREEN INTERACTION
`
`uncontrollable circumstances. Examples would be working on a product while it is hot
`
`enough to be formed or examining a patient while a medication is in effect.
`
`Repeatedly working on tasks under high time pressure is known to worsen perfor-
`
`mance and increase human error (Reason, 1990; Schmidtke, 1993), thus influencing
`
`economic viability and possibly safety. If a time budget is inherent in a task, the de-
`
`sign of the man–machine interface has to ensure the best possible usage of this time
`
`frame. This means that the number of necessary steps, required precision, and the
`
`cognitive and physical workload should be as low as possible. Since the performance
`
`of users with technical systems is influenced by their familiarity with these systems
`
`and their un