Multi-Modal Text Entry and Selection on a Mobile Device
`
`
`
`
`
`
`Amy Karlson
`Brian Meyers
`Ben Bederson
`
`
`
`David Dearman
`
`University of Toronto
`
`
`
`
`ABSTRACT
`
`Microsoft Research
`
`Microsoft Research
`
`University of Maryland
`
`Rich text tasks are increasingly common on mobile devices,
`requiring the user to interleave typing and selection to produce the
`text and formatting she desires. However, mobile devices are a
`rich input space where input does not need to be limited to a
`keyboard and touch. In this paper, we present two complementary
`studies evaluating four different input modalities to perform
`selection in support of text entry on a mobile device. The
`modalities are: screen touch (Touch), device tilt (Tilt), voice
`recognition (Speech), and foot tap (Foot). The results show that
`Tilt is the fastest method for making a selection, but that Touch
`allows for the highest overall text throughput. The Tilt and Foot
`in users performing and
`methods—although fast—resulted
`subsequently correcting a high number of text entry errors,
`whereas the number of errors for Touch is significantly lower.
`Users experienced significant difficulty when using Tilt and Foot
`in coordinating the format selections in parallel with the text
`entry. This difficulty resulted in more errors and therefore lower
`text throughput. Touching the screen to perform a selection is
`slower than tilting the device or tapping the foot, but the action of
`moving the fingers off the keyboard to make a selection ensured
`high precision when interleaving selection and text entry.
`Additionally, mobile devices offer a breadth of promising rich
`input methods that need to be careful studied in situ when
`deciding if each is appropriate to support a given task; it is not
`sufficient to study the modalities independent of a natural task.
`
`KEYWORDS: Mobile device, multi-modal,
`text entry,
`formatting, target selection, foot, tilt, touch, speech
`
`INDEX TERMS: H5.m. Information interfaces and presentation
`(e.g., HCI): Miscellaneous.
`
`text
`
`1
`
`INTRODUCTION
`
`Text entry is a fundamental and common activity that users
`perform on their mobile devices. Although the mobile phone is
`most commonly used to send and receive simple unadorned text
`messages [7], users are looking for new ways to improve the
`expressivity their devices can afford. Rich text entry tasks such as
`writing a detailed and structured email, posting an update to a
`Blog, and editing a Word document are becoming increasingly
`more common. These rich text task are typically supported by
`selectable interface features that allow for faster (e.g., word
`completion), accurate (e.g., word correction), descriptive (e.g.,
`font, format, colour) and structured (e.g., bullets, indentation) text
`
`dearman@dgp.toronto.edu
`{karlson, brianme}@microsoft.com
`bederson@cs.umd.edu
`
`Graphics Interface Conference 2010
`31 May - 2 June, Ottawa, Ontario, Canada
`
`Copyright held by authors. Permission granted to
`CHCCS/SCDHM to publish in print form, and ACM
`to publish electronically.
`
`
`
`Figure 1. A user entering text on a mobile device and selecting the
`appropriate character level formatting.
`
`entry. Selecting these features is currently accomplished by
`touching an on-screen widget, or navigating among a list of
`options using the directional pad.
`Touching an on-screen widget or manipulating the directional
`pad are natural methods of selection, but they require users to
`interleave selection and typing, slowing the user‘s rate of text
`input. Given that text entry is already considered a bottleneck for
`expression on mobile devices, we wondered how well alternative
`input channels (e.g., accelerometers and speech recognition)
`might be used to efficiently complement text entry. These
`alternatives have the potential to be used in parallel with typing,
`allowing the user to dedicate her fingers to the primary typing task
`and reduce the impact that editing functions have on throughput.
`The focus of this research is to better understand the efficiency
`potential that alternate input channels hold for increasing the
`expressivity and throughput of mobile text entry. Specifically, we
`are interested in comparing screen touching to alternate input
`channels as a means to support selection during text entry. The
`input channels we have chosen to evaluate are device tilt (Tilt),
`voice recognition (Speech), and foot tap (Foot). While there are a
`large number of alternate input channels that we could explore,
`we compared Touch with Tilt, Speech, and Foot because they
`cover a range of input technologies supported by devices today
`(noting that foot sensing is becoming more common place with
`the Nike + iPod Sports Kit [9]).
`To better understand the relative tradeoffs that screen touch,
`device tilt, voice input and foot tapping offer users for performing
`multimodal edit-based selections during text entry, we conducted
`a controlled laboratory evaluation. Our goal was not to evaluate
`the technologies themselves, but rather their unique impacts on
`the flow of text input and formatting. Overall, we found:
`
`
`
`Touch resulted in the highest text throughput. Thus, our core
`hypothesis that parallel input channels would be faster was
`false. Coordinating selection and typing was difficult using
`tilt, foot and voice.
`
` A significant trade-off exists between selection speed and
`accuracy. Selection was quickest with device tilt and foot
`tapping, but screen touch and voice resulted in fewer errors.
`
`19
`
`2010
`
`Petitioner Samsung Ex-1035, 0001
`
`

`

`
`
`
`
`There are interesting human performance issues with respect
`to the orientation of a target within an input type. For
`example, tapping the ball of the foot is more accurate than
`using the heel.
`The time to select a target is slower than the time to resume
`typing the text.
`
`2
`
`RELATED WORK
`
`The dominant modes for interacting with a mobile device
`currently require a user to touch the screen or use the directional
`pad. However, a mobile phone can support alternate input
`modalities such as accelerometers [3, 4, 10, 11, 15, 19, 20],
`speech recognition [14, 16, 17, 22], cameras [23] and chording
`[24] have been explored, some of which are already common in
`today‘s smart mobile devices.
`Touching the screen with a finger or stylus, manipulating the
`directional pad and typing on a QWERTY or 12-button keypad
`are typical modes for interacting with mobile devices. The
`majority of phones provide one or more of these modes to interact
`with the information space. ChordTap by Wigdor et al., based on
`the principles of a chorded keyboard, extends the default keypad
`by adding three buttons on the back of a mobile phone [24]. The
`ChordTap keys allow users to differentiate between the multiple
`characters for each button on a 12 button keypad.
`Speech has been used as a means to provide direct input [22]
`and facilitate text correction [1, 5]. Jiang et al. fused word
`candidates generated by typing Chinese characters while speaking
`in parallel to generate a single reduced word set [5]. Similarly, Ao
`et al. corrected recognition errors in Chinese handwriting by
`having people verbally repeat the sentence [1].
`Tilt and orientation of a mobile device have been used
`extensively to navigate through lists and menus [3, 4, 10, 11] and
`disambiguate between characters when typing [11, 15, 19, 20, 24].
`Oakley et al. used tilt to navigate through one dimensional lists
`and menus [10, 11], invoking commands by rotating the device
`into one of three target regions along a 90 degree rotational space
`(vertical to horizontal). Unigesture, a tilt-to-write system by
`Sazawal et al., enabled single handed text entry [20]. Rather than
`typing on a keypad, three to four characters are organized into
`seven regions that are selected using the orientation of the device.
`Partridge et al. refined the unigesture technique in TiltType [15],
`allowing users to disambiguate between the characters within a
`region by pressing one of four buttons. TiltText, a technique
`designed specifically for a mobile phone, utilized 30 and 60
`degree rotations along the vertical and horizontal plane to
`disambiguate between the available characters on a standard 12-
`button mobile phone keypad [24]. Vision TiltText [23] mimicked
`the functionality of Wigdor et al.‘s TiltText, but used the mobile
`phones built-in camera to differentiate between characters by
`detecting the user‘s hand movement, rather than the devices tilt.
`Foot based input for a mobile device is generally a discounted
`mode of interaction. However, many professional examples
`confirm that feet can be elegantly engaged in tasks (e.g.,
`musicians, audio scribes). Pearson and Weiser explored alternate
`topologies of foot movement and the design space to support
`interacting with desktop computing [16]. They later implemented
`one such design called the planar slide mole, assessing its
`performance against a mouse for target selection [17]. Although
`the mouse was generally faster and less prone to errors, the foot
`was extremely quick in gesturing. Pakkanen et al. conducted a
`similar study utilizing a trackball to perform target selection,
`comparing the foot to the hand [14]. Their results highlight that
`although the foot is not as dexterous as the hand, it is appropriate
`for tasks that do not require precision or quick homing and
`
`selection. The design of our study is in keeping with these
`recommendations. The foot is not required to perform homing,
`only selection.
`Many alternate input modalities exist for mobile devices to
`support direct text input. These methods have never been directly
`compared in order to assess their effectiveness as a parallel input
`channel to enrich text entry. Understanding the performance of
`these alternate input modes will provide a better understanding of
`how multimodal selection and text entry can be integrated [12]
`and how to best support user‘s integration patterns [13].
`
`3
`
`STUDY
`
`We performed two studies in order to explore human capabilities
`while performing selections during text entry. Four input modes
`were compared: standard screen touching (Touch), device tilting
`(Tilt), voice recognition (Speech), and foot tapping (Foot). Tilt,
`Speech and Foot all allow the user to keep her fingers on the
`keyboard, and so have the potential to be used independently from
`the act of letter selection. However, these channels also have
`characteristics that may impair the text entry task. For example,
`Speech is considered a ―natural‖ form of input that does not
`require much in the way of physical effort, but verbalizing
`commands while typing words may impose additional cognitive
`demands. Tilt may be familiar to many modern mobile phone or
`game users, but coordinating Tilt with text entry may be difficult
`or uncomfortable in practice. Finally, while many professional
`examples confirm that feet can be elegantly engaged in tasks, Foot
`selection for the uninitiated user may simply be too awkward and
`too ―distant‖ from the mental/manual focus of the task to be used
`effectively.
`To evaluate user performance with our chosen input methods,
`we devised two experiments. Experiment one involved a stimulus-
`response Target Selection task to establish the speed and accuracy
`with which users can make target selections using each of our four
`chosen input modes. In experiment two, we used a Text
`Formatting task to evaluate how quickly and accurately users can
`apply text formatting using each input method while completing a
`text entry
`task. The purpose of studying
`the
`two
`tasks
`independently was to isolate systematic differences in users‘
`abilities to perform selections using the different input methods
`(Target Selection) from other influences affecting the flow and
`throughput of text entry (Text Formatting). Initially we had
`envisioned using a more common word correction or prediction
`feature for this task. However, those tasks would not have allowed
`us to control for when the user applied the correction or prediction
`feature. The usefulness of correction and prediction features is
`dependent on the input text and the user‘s perception that
`selecting a word is faster than correcting or typing the word. Text
`Formatting was a realistic text entry task that allowed us to
`maintain control over when and where a selection is made.
`The Touch, Tilt, Speech and Foot input methods vary greatly in
`the granularity of expression they support. For example, voice
`supports a large unconstrained input space, limited only by the
`human capacity to label and verbalize a selection. In contrast,
`researchers have formally characterized the limits of hand tilt to a
`much smaller input space [18]. Because our focus was on
`understanding the relative tradeoffs between the input methods
`during text entry and not comparing the expressive limits of each
`method, we chose a selection space of four options to achieve
`parity across the input methods. Limiting the selection space also
`allowed for straightforward visual mappings between the input
`gestures and on-screen selection targets.
`
`20
`
`2010
`
`Petitioner Samsung Ex-1035, 0002
`
`

`

`3.1
`
`Target Selection and Text Formatting Tasks
`
`The Target Selection experiment involved a stimulus-response
`task designed to evaluate the speed and accuracy with which
`participants could identify and select on-screen targets in four
`different positions using the four input methods (Figure 2: top
`row). The goal of this task was to understand users‘ motor
`abilities across
`input methods. The
`target placement and
`alignment differed for each input method, but for each method the
`placement corresponded to the physical action necessary to
`perform a selection (Figure 2). Each trial began with a blank
`screen, requiring the participant to press the ‗F‘ and ‗J‘ keys
`simultaneously to display the target object, shown in red. The start
`posture was designed to ensure that the device was held in a
`consistent manner across trials and participants. Selection time
`was calculated from the time the ‗F‘ and ‗J‘ keys were pressed
`until a selection was made.
`The Text Formatting experiment involved a modified text entry
`task that required participants to reproduce short text phrases that
`included visual formatting characteristics (Figure 2: bottom row).
`The goal of this task was to evaluate the speed and accuracy with
`which users could interleave the selection and de-selection of
`formatting states while concurrently entering text, and how the
`four input methods impact the primary text entry task. This is in
`contrast to the Target Selection task, which simply evaluated the
`user‘s ability to execute four distinct commands using each of the
`four input types.
`Participants were required to enter the characters of a text
`phrase and apply formatting to the text at various positions in the
`phrase. The tasks allowed for partially formatted words, meaning
`that format mode activations were required both at the beginning
`and middle of a word, and format mode deactivations were
`required at both the middle and end of a word. In practice, the
`format commands were modal—only one format could be active
`at a time. For example, selecting ‗Orange‘ would activate the
`orange text mode. All subsequent characters entered would be
`shown in orange until the user selected ‗Orange‘ again; returning
`the text mode to ―regular‖ entry mode.
`The placement and alignment of the selectable format buttons
`was identical to that of the Target Selection task. The interface
`supported formatting sequences of three or more characters. Words
`of three to five characters could be formatted in whole, while words
`
`
`
`
`
`Figure 3. Experimental setup for the Foot input condition.
`
`To conduct this study we developed an application test-bed that
`provides input to a HTC Touch Pro 2 (shown in
`of six or more characters could be formatted in whole or in part. A
`partially formatted word would always start and end with an
`unformatted character. Formatting characters according to this
`schema ensured that a minimum of three characters would be
`entered between toggling formats. The phrases were chosen
`randomly from MacKenzie‘s English phrase dictionary [7]. No
`phrases were repeated and the order was consistent between
`participants. After each phrase was entered and properly formatted,
`the participant pressed the ‗Enter‘ key to advance to the next phrase.
`
`3.2
`
`Apparatus
`
`Figure 1) using screen Touch, Tilt of the device, Speech, and Foot
`tapping. The test-bed consisted of two components: a desktop
`computer running Windows Vista and the HTC Touch Pro 2
`running Windows Mobile 6.1. The desktop and mobile device was
`connected wirelessly by a dedicated Linksys Wireless router using
`802.11g.
`Foot and Speech input was accomplished through the desktop
`computer by wirelessly communicating commands to the mobile
`device. Input for Foot was performed using two X-keys 3 switch
`foot pedals connected to the desktop computer via USB. One foot
`pedal was placed sideways under each foot such that the ball and
`heel of the foot depressed the respective left and right switch
`(Figure 3). In the default state, the switches were depressed by the
`pressure of the user‘s foot resting on them. A selection was
`registered when a switch was released, not pressed. This
`implementation allows for four possible inputs by lifting the ball
`or heel of the left and right foot. For example, lifting the heel of
`
`
`
`
`
`
`
`
`
`
`
`
`
`Figure 2. The software interface for the Target Selection (top) and Text Formatting (bottom) tasks. Presented are the Foot (left), Tilt (middle)
`and Touch/Voice (right) interfaces. The target to select for the Target Selection task is red. The phrase used in the Text Formatting task
`includes all four formats: ‘eason’ is orange; ‘lfe’ is green; ‘love’ is bold; and ‘the’ is underlined.
`
`21
`
`2010
`
`Petitioner Samsung Ex-1035, 0003
`
`

`

`the right foot would select the red target (Figure 2: top-left) or
`‗Bold‘ format mode (Figure 2: bottom-left).
`Input with the Speech condition was performed by saying the
`target‘s label. The speech recognition component of our test bed
`was implemented using a Wizard of Oz simulation. We chose not
`use computer-based speech recognition because the systems we
`tested for the desktop and mobile device incurred a noticeable lag
`between the time when a label was spoken and interpreted. We
`wanted the latency between saying a label and it being selected to
`be as small as possible to allow for a fair comparison. To
`accomplish this, we relied on a human wizard to listen to the
`participants‘ verbal selection and press one of four corresponding
`keys on a keyboard connected to the desktop computer. The
`selection was then wirelessly communicated to the mobile device.
`For example, saying ―four‖ would result in the selection of the red
`target (Figure 2: top-right). Similarly, saying ―Bold‖ would select
`the ‗Bold‘ target and enter ‗Bold‘ input mode (Figure 2: bottom-
`right).
`Tilt and Touch inputs were implemented directly on the mobile
`device and did not require the desktop computer. Input using the
`Tilt of the mobile device was implemented by sampling the
`integrated six degree of freedom accelerometer at 25 Hz. We
`interpreted four Tilt gestures: tilting the device forward, backward,
`left and right. Gestures exceeding 30 degree changes from a
`continually updated ―neutral‖ position were recognized as a
`command along the direction of movement. We chose to implement
`a relative rather than absolute origin because it allowed users to
`choose a comfortable position and angle at which to hold the
`device. For example, tilting the device backward would select the
`red target (Figure 2: top-middle), or the ‗Underline‘ formatting state
`(Figure 2: bottom-middle).
`Input using Touch was performed by physically pressing the
`appropriate target displayed on the device‘s resistive screen. For
`example, pressing the red target would select it (Figure 2: top-
`right). Similarly, pressing the ‗Bold‘ target would activate the
`‗Bold‘ formatting state (Figure 2: bottom-right).
`
`3.3
`
`Participants
`
`Twenty-four participants recruited from the general population took
`part in the study – 11 females and 13 males. The age of participants
`varied between 18 and 38, with a median age of 26. We recruited
`participants that currently owned a mobile device (e.g., Blackberry,
`HTC Touch, iPhone or iPod Touch) that they currently use to enter
`text on a daily basis and have done so consistently for at least the
`past four months. All participants owned a device with either a
`physical or touch screen based QWERTY keyboard. All but one
`participant were right handed. Participants were compensated
`(removed for review) for their time.
`
`3.4
`
`Procedure
`
`The Target Selection experiment was administered first. The
`participant was first introduced to all four input methods and
`trained how to use each. The number of training trials varied
`across participants, but always continued until both the participant
`and experimenter
`felt comfortable with
`the participant‘s
`performance. The participant then completed the selection task for
`each of the four input methods, completing all trials for a given
`input method before continuing to the next method. Participants
`were instructed to make the selections as quickly and accurately
`as possible.
`For the Text Formatting experiment, the participant was first
`asked to read a document describing the formatting task and then
`enter 10 to 12 training phrases for each input method. If the
`participant felt uncomfortable after the training tasks for any input
`
`mode, she was allowed to continue training until both the
`participant and experimenter
`felt comfortable with her
`performance. After training on all input methods, the testing phase
`began, during which the participant completed four sets of test
`phrases grouped by input type. Participants were instructed to
`enter and format the text as quickly and accurately as possible,
`and to correct all mistakes. However, perfect input was not
`enforced. After completing trials for each input type, a modified
`NASA TLX survey was administered. After the final input
`method was completed, a concluding survey was administered
`asking participants to rank the inputs in order of preference and to
`provide subjective feedback with respect to the least and most
`preferred input.
`
`3.5
`
`Design
`
`The order Tilt, Touch, Speech and Foot were presented was fully
`counterbalanced across the twenty-four participants for the Target
`Selection and Text Formatting experiments. The Target Selection
`experiment was a 4×4 design. It comprised the following factors
`and levels:
`
`
`
`
`
`Input Type {Touch, Tilt, Speech, Foot}
`
`Target Position {1, 2, 3, 4}
`
`Each Input Type was evaluated over six blocks of trials (1
`training; 5 testing) with 20 test trials per block - five trials for
`each of the four Target Positions. Each participant performed
`4×5×4×5 = 400 test trials or 9,600 among all 24 participants. The
`presentation order of the target positions was randomly assigned
`within each block, but consistent across participants. The first
`block for each Input Type was training and excluded from the
`analysis.
`The Text Formatting experiment was a 4×3×4 design. It
`comprised the following factors and levels:
`
`
`
`
`
`
`
`Input Type {Touch, Tilt, Speech, Foot}
`
`Format Position {Start, Middle, End}
`
`Target Position {1, 2, 3, 4}
`
`Each Input Type was evaluated over five blocks of trials (one
`training and four testing) with between 8 and 12 phrases per
`block. Each block required 48 format selections – four trials for
`each Format Position × Target Position. Participants performed
`4×4×3×4×4 = 768 format selection and entered 3,111 characters
`or 18,432 format selections and 74,664 characters among all 24
`participants. The length of the words and the number of words per
`phrase dictated the overall number of phrases required to meet the
`48 format selections per block. The presentation order of the
`format position and the type of format to be applied was randomly
`assigned within blocks, but presented consistently across
`participants. The first block for each Input Type was training and
`excluded from the analysis.
`
`4
`
`RESULTS
`
`The Target Selection and Text Formatting experiments were
`conducted independently, and thus analyzed separately. Selection
`time trials that exceeded three standard deviations from the mean
`were removed as outliers. To account for the variability in human
`selection, the median selection time for each participant was used in
`the analysis. Timing data was analyzed with repeated measures
`ANOVAs using Wilk‘s Lambda. Event-count measures such as
`error were analyzed with nonparametric Friedman tests and post-
`hoc pairwise comparisons were conducted with the Wilcoxon test.
`All post-hoc comparisons were conducted using Holm‘s sequential
`Bonferroni correction.
`
`22
`
`2010
`
`Petitioner Samsung Ex-1035, 0004
`
`

`

`Time
`
`(ms)
`
`Tilt
`
`Touch
`
`Speech
`
`Foot
`
`ErrorRate(%)
`
`ONF&FDD@®
`
`0.17
`
`Touch
`
`0.13
`
`Speech
`
`Tilt
`
`
`
`Foot
`
`Figure 5. The selection error rate for the Target Selection
`experiment grouped by the Input Types.
`
`that overall differences in
`wizard). Together with the fact
`selection times across positions was very small (< 41 ms) it is
`unlikely that the position effect has practical meaning with respect
`to user performance for Speech.
`
`Figure 4. The average selection time for the Target Selection
`experiment grouped by the Input Types. Theerror bars indicate
`the standard deviation.
`
`4.1.2|Target Selection Errors
`In addition to evaluating the participants’ performance for each
`The overall selection error rate was 2.47%. A Friedman test
`Input Type, we evaluated how each input impacted the task of text
`showed a significant main effect ofInput Type, 6. nem = 55.29,
`entry by analyzing the participant’s character stream with
`Wobbrock and Myers TextTest StreamAnalyzer [25].
`p<0.001. The error rates for Touch (0.17%) and Speech (0.13%)
`Wedo not directly compare the Target Positions across the four
`are negligible (<1%) and significantly lower than Tilt (3.21%) and
`Input Types because the positions vary and are therefore not
`Foot (6.38%), all with p<0.001. In addition, the error rate for Foot
`equivalent. Rather, we focus our analysis on the selection time
`is greater than Tilt, p<0.005.
`and error between Target Positions within Input Types.
`Pairwise analysis of the Target Positions within the Input Types
`yielded significant differences in the error rate for Tilt
`(3, x=24) =
`7.88, p<0.05) and Foot
`(77g, 24) = 8-52, p<0.05). Post-hoc
`pairwise comparisons were conducted with the Wilcoxon test.
`Tilt. Forward tilt (n=29; 1.21%) resulted in a higher error rate
`than backwards (n=10:; 0.04%), p<0.005. No differences were
`foundfor left (n=18: 0.75%) andright (n=20; 0.83%).
`Foot. Theerror rate for the left-heel (n=52; 2.17%) was greater
`than the right-ball (n=27: 1.13%), p<0.005. No differences were
`found forthe left-ball (m=32; 1.33%) or right-heel (n=42: 1.75%).
`Wealso compared the combinederror rate ofthe ball and heel of
`the left and right foot. Overall, the heel (n=94; 3.92%) has an
`error rate greater than the ball of the foot (n=59: 2.46%), p<0.05.
`Since most of our participants were right-footed, it makes sense
`that participants would be most agile with the ball of their
`dominant foot (right) and least agile with the heel of their non-
`dominant foot(left).
`
`Target Selection Results
`4.1
`Seventy-six outliers (0.8%)—three standard deviations from the
`mean—were excluded from the analysis. Analysis of the blocks
`yielded no evidence of a learning effect: therefore all test trials are
`includedin the time anderror analyses.
`
`Target Selection Time
`4.1.1.
`Repeated measures ANOVA on the median selection times
`yielded a significant effect of Input Type. F3, 11=879.98. p<0.001.
`Post-hoc pairwise comparisons show that the overall selection
`time for Speech (1172 ms) was slower than the other Input Types
`(Figure 4), all at p<0.001. Although Touch (656 ms) and Foot
`(635 ms) have similar mean selection times, Tilt (588 ms) was
`foundto be faster than the other Input Types,all at p<0.001.
`Repeated measures ANOVAs for Target Positions within Input
`Types found a significanteffect on selection time for Tilt (F3, 2=8.64,
`p<0.001), Touch (F3, 2=13.80, p<0.001) and Speech (Fs, 2=11.10,
`p<0.001), but not for Foot.
`Tilt. Tilting the device forward (561 ms) wasfaster than tilting
`in the other directions: forward was 8.2% faster than left (594 ms:
`p<0.05), 5.9% faster than right (591 ms; p<0.001), and 5.3%
`faster than backward (607 ms: p<0.01).
`Touch. The second target (680 ms; ordered left to right) was
`4.0% slower than the first (654 ms: p<0.001), and 7.1% slower
`than the fourth (635ms:; p<0.001). While somewhatsurprising, the
`slower selection time for the second target is likely attributed to
`the majority of our participants being right-hand dominant. This
`agrees with our observation that many participants opted to use
`their right thumb to reach across the screen to the second target
`rather than using their proximally closer left thumb.
`Speech. The fourth target (1199ms; ordered left to right) was
`3.5% slower than the first (1158ms; p<0.001), and 3.9% slower
`than the second (1154ms; p<0.001). While the slower selection
`time of the fourth target might be attributed to a tendency for
`participants to scan left to right, this explanation seems somewhat
`suspect since we would have expected target recognition to occur
`preattentively and be immuneto position bias. Furthermore, it is
`useful
`to remember that
`the Speech condition involved two
`independent human response
`components
`(participant
`and
`
`Text Formatting Results
`4.2.
`Two-hundred and ten (1.1%) selections were identified as outliers
`(greater than three standard deviations from the mean) and
`removed from the analysis. The timing data for P1 using Touch
`(192 entries) was not included in the analysis because of a
`software loggingerror. In the analysis we differentiate:
`e
`Selection Time — the time difference between typing a
`character and selecting a subsequent formatting.
`Resumption Time — the time difference between selecting a
`format and typing a subsequentcharacter.
`Pairwise comparison shows that selection time is slower,
`p<0.001, than resumption time (Figures 6 and 7).
`
`e
`
`4.2.1
`
`Format Selection Time
`
`Repeated measures ANOVAofthe median selection time yielded
`a significant effect of Input Type. F3, 9 = 95.23, p<0.001, and
`FormatPosition, Fy, 9, = 15.0, p<0.001. Similar to the Target
`Selection results, post-hoc pairwise comparisons show that the
`selection time for Speech (1146 ms) was slower than the other
`Input Types (Figure 6), all at p<0.001. Although Touch (855 ms),
`Tilt (797 ms) and Foot (834 ms) have similar mean selection
`times, Tilt was found to be faster than Touch, p<0.001. Analysis
`
`Graphics Interface 2010 fa
`
`23
`
`Petitioner Samsung Ex-1035, 0005
`
`Petitioner Samsung Ex-1035, 0005
`
`

`

`1600
`1400
`1200
`1000
`800
`600
`400
`200
`0 4
`
`
`
`Time(ms)
`
`Ss
`
`R
`
`Ss
`
`R
`
`Ss
`
`R
`
`Ss
`
`R
`
`Tilt
`
`Touch
`
`Speech
`
`Foot
`
`Figure 6. The average selection time (S) and resumption time (R)
`for the Text Formatting experiment grouped by the Input Types. The
`error bars indicate the standard deviation.
`
`1600
`1400
`1200
`@ 1000
`E 800
`£
`600
`- 400
`200
`0 4
`
`Ss
`
`R
`
`iS
`
`R
`
`986
`
`Ss
`
`R
`
`Start
`
`Middle
`
`End
`
`Figure 7. The average selection time (S) and resumption time (R)
`for the Text Formatting experiment grouped by the Format
`Positions. The error bars indicate the standard deviation.
`
`of the FormatPosition revealed that toggling a format selection at
`the End of a word (839 ms) is faster than the Start (905ms:
`p<0.01) and Middle of a word (985 ms: p<0.001).
`Repeated measures ANOVAs for Target Position within Input
`Types found a significant main effect on selection time for Touch
`(F3, 19=11.04, p<0.001), Speech (F3, 19=11.62, p<0.001), and Foot
`(F3, 13 = 7.30. p<0.005). but not Tilt.
`Touch. The second target (920 ms) is 13.6% slower than the
`third (810ms) and 12.1% slower than the fourth (821 ms), all at
`p<0.001, which matches our findings from the Target Selection
`study.
`Speech. The second target (1192 ms) is slower than all the
`other target positions: 5.7% slower than the first (1128 ms;
`p<0.005): 7.6% slower than the third (1108 ms; p<0.001); and
`3.3% slower than the fourth (1154 ms; p<0.05).
`Foot. Theleft-heel (903 ms) is 14.0% slower than the right-ball
`(792 ms; p<0.005) and 11.3% slower than the right-heel (811 ms;
`p<0.001), but not the left-ball (834 ms).
`
`Format Resumption Time
`4.2.2
`Repeated