UNITED STATES PATENT AND TRADEMARK OFFICE
`BEFORE THE PATENT TRIAL AND APPEAL BOARD
`__________________
`
`GOOGLE INC.
`Petitioner
`
`v.
`
`KONINKLIJKE PHILIPS ELECTRONICS N.V.
`Patent Owner
`__________________
`
`U.S. PATENT NO. RE44,913
`Case Nos. IPR2017-00386, IPR2017-00387
`__________________
`
`DECLARATION OF DR. ANDREW COCKBURN
`
`
`
`
`
`
`
`Google Ex. 1002
`Google v. Philips
`
`

`
`
`
`
`
`TABLE OF CONTENTS
`
`I. 
`INTRODUCTION ........................................................................................... 1 
`BACKGROUND AND QUALIFICATIONS ................................................. 2 
`II. 
`III.  COMPENSATION AND RELATIONSHIP TO THE PARTIES .................. 7 
`IV.  LEGAL STANDARDS ................................................................................... 7 
`A. 
`Priority Date .......................................................................................... 7 
`B. 
`Claim Construction ............................................................................... 7 
`C. 
`Anticipation ........................................................................................... 8 
`D.  Obviousness ........................................................................................... 8 
`E. 
`Person of Ordinary Skill in the Art ....................................................... 9 
`SUMMARY OF OPINION ........................................................................... 10 
`V. 
`VI.  BACKGROUND ........................................................................................... 11 
`A. 
`The Increasing Capabilities of Computing Devices ........................... 11 
`B. 
`Touch Input ......................................................................................... 14 
`C.  Methods for Expressing Intentions Through Touch ........................... 15 
`D. 
`Interfaces for Accessing Functionality on Small Devices .................. 18 
`E. 
`Text Entry on Mobile Touchscreen Devices ....................................... 23 
`1.  Mobile Keyboards and Key Overloading ................................. 24 
`2. 
`Key Overloading on Mobile Devices ....................................... 32 
`VII.  THE ’913 PATENT ....................................................................................... 36 
`A. 
`The Specification of the ’913 Patent ................................................... 36 
`B. 
`The Claims of the ’913 Patent ............................................................. 41 
`1. 
`Independent Claims 1, 3 and 4 .................................................. 41 
`2. 
`Dependent Claim 2 ................................................................... 43 
`3. 
`Dependent Claims 5, 9 and 13 .................................................. 45 
`4. 
`Dependent Claims 6, 10 and 14 ................................................ 45 
`5. 
`Dependent Claims 7, 11 and 15 ................................................ 46 
`6. 
`Dependent Claims 8, 12 and 16 ................................................ 46 
`Prosecution History of the ’913 Patent ............................................... 47 
`1. 
`Prosecution of the ’318 Patent .................................................. 47 
`2. 
`Prosecution of the ’913 Reissue Patent ..................................... 48 
`CONSTRUCTION OF THE CHALLENGED CLAIMS ............................. 56 
`A. 
`“character” ........................................................................................... 56 
`B. 
`“keypad” .............................................................................................. 56 
`C. 
`“touchscreen” ...................................................................................... 57 
`D.  Means-Plus-Function Limitations ....................................................... 57 
`1. 
`Claim 4 ...................................................................................... 58 
`2. 
`Claim 6 ...................................................................................... 63 
`
`B. 
`
`II. 
`
`
`
`Google Ex. 1002
`Google v. Philips
`
`

`
`
`
`
`
`
`
`3. 
`Claim 7 ...................................................................................... 63 
`Claim 8 ...................................................................................... 64 
`4. 
`III.  DETAILED EXPLANATION OF GROUNDS FOR INVALIDITY .......... 65 
`A. 
`Claims 1 and 3-16 Are Rendered Obvious by Sakata II ..................... 65 
`1. 
`Overview and Explanation of Sakata II .................................... 65 
`2. 
`Independent Claim 1 ................................................................. 82 
`3. 
`Claim 3 .................................................................................... 106 
`4. 
`Claim 4 .................................................................................... 107 
`5. 
`Dependent Claims 5, 9, and 13 ............................................... 112 
`6. 
`Dependent Claims 6, 10, and 14 ............................................. 112 
`7. 
`Dependent Claims 7, 11, and 15 ............................................. 114 
`8. 
`Dependent Claims 8, 12, and 16 ............................................. 116 
`Claims 1 and 3-16 Are Rendered Obvious Over Sakata II in
`View of Buxton ................................................................................. 118 
`Claims 1-5, 9, and 13 Are Obvious Over Hoeksma in view of
`Sakata II ............................................................................................. 125 
`1. 
`Overview of Hoeksma ............................................................ 126 
`2. 
`Claims 1 and 2 ......................................................................... 129 
`3. 
`Dependent Claim 9 ................................................................. 143 
`4. 
`Claims 3 and 13 ....................................................................... 144 
`5. 
`Claims 4 and 5 ......................................................................... 145 
`IV.  CONCLUSION ............................................................................................ 149 
`
`
`B. 
`
`C. 
`
`
`
`ii
`
`Google Ex. 1002
`Google v. Philips
`
`

`
`
`
`I, Dr. Andrew Cockburn, hereby state the following:
`
`I.
`
`INTRODUCTION
`
`1.
`
`I have been retained by Google Inc. (“Google”) in connection with the
`
`filing of a Petition for inter partes review of U.S. Patent No. RE44,913 (“the ’913
`
`patent”) (Ex. 1001). The opinions presented here are my own.
`
`2.
`
`The claims of the ’913 patent generally relate to a method of
`
`improved character input that employs a keypad that is capable of selecting both
`
`“primary” and “secondary” characters associated with particular keys. The method
`
`involves the use of a “short press” to select “primary” characters and the use of a
`
`“long press” to facilitate selections of secondary characters.
`
`3.
`
`I have been asked to explain the bases for my opinions that claims 1-
`
`16 of the ’913 patent are unpatentable because they are anticipated and/or would
`
`have been obvious to the person of ordinary skill in view of the prior art.
`
`4.
`
`In reaching my opinions regarding the invalidity of these patent
`
`claims, I have relied on the documents cited herein, and I have relied as well on my
`
`decades of knowledge and experience in the field of Human-Computer Interaction
`
`(“HCI”).
`
`5.
`
`This declaration is based on information currently available to me. I
`
`reserve the right to supplement my opinions in response to arguments raised by
`
`Philips or in response to any additional information that becomes available to me.
`
`
`
`Google Ex. 1002
`Google v. Philips
`
`

`
`
`
`II. BACKGROUND AND QUALIFICATIONS
`6.
`I am a Professor at the Department of Computer Science and Software
`
`Engineering at the University of Canterbury, New Zealand. I also currently head
`
`the HCI (which stands for “Human-Computer Interaction”) and Multi-Media
`
`research group at the University of Canterbury.
`
`7.
`
`In 1988, I was awarded a Bachelor of Science with Honors in
`
`Computer Science from the University of York, England.
`
`8.
`
`In 1993, I was awarded a Ph.D. from the University of Stirling,
`
`Scotland. My thesis was on “Computer Supported Cooperative Work” which
`
`relates to forms of group interaction supported on computer.
`
`9.
`
`In 1993, I joined the University of Canterbury as a Lecturer in the
`
`Department of Computer Science (now the Department of Computer Science and
`
`Software Engineering). I was subsequently promoted to a Senior Lecturer, and
`
`then an Associate Professor, before my appointment as a Professor in 2010. I
`
`currently hold this title of Professor.
`
`10.
`
`I have over 23 years’ experience in the area of HCI. The field of HCI
`
`generally is concerned with ways of understanding and improving the interaction
`
`between humans and computers, with a view to understanding, evaluating,
`
`designing and building new styles of interactions that improve on one or more of
`
`
`
`2
`
`Google Ex. 1002
`Google v. Philips
`
`

`
`
`
`the end goals of making computers faster to learn, more intuitive, more efficient to
`
`use and more subjectively satisfying.
`
`11. Throughout my career, I have published the results of many research
`
`projects that have involved building new user interfaces or reviewing existing user
`
`interfaces for performing a particular task, and evaluating their effectiveness. This
`
`includes publications relating to:
`
`(a)
`
`investigating user experiences with web navigation interfaces, for
`
`example when using the “back” button on web browsers;
`
`(b)
`
`reviewing and improving various interface schemes for traversing
`
`through documents in computer applications, including zooming, scrolling and
`
`other techniques;
`
`(c)
`
`analyzing new interfaces for text entry on mobile and touch-sensitive
`
`devices;
`
`(d)
`
`addressing the problems arising from the small form factor of mobile
`
`devices with touch-sensitive displays;
`
`(e)
`
`examining the influence of haptic feedback on user performance with
`
`mouse and touchscreen input devices;
`
`(f)
`
`evaluating the importance of spatially stable displays in user
`
`interfaces; and
`
`
`
`3
`
`Google Ex. 1002
`Google v. Philips
`
`

`
`
`
`(g)
`
`designing and evaluating a pen-based user interface for musical input
`
`on a touchscreen.
`
`12.
`
`I also have extensive experience in designing and building new user
`
`interfaces and reviewing existing user interfaces. This includes a number of
`
`projects regarding the design, development and evaluation of user interfaces that I
`
`have undertaken with companies in the computing and HCI industry, such as:
`
`(a) working with Airbus SAS in 2016 on methods to assist pilot
`
`interaction with touchscreens in turbulent environments;
`
`(b) working with Hewlett-Packard Research Labs from 2010-2012 on the
`
`design and evaluation of pointing techniques for remote displays, such as
`
`interactive TVs;
`
`(c)
`
`a number of projects from 2006-2010 working with Logitech on the
`
`design, development, evaluation and improvement of user interfaces for next
`
`generation mice, including in relation to scrolling and window management tasks;
`
`(d) working with IBM Almaden Research in 2006 on the design,
`
`development and evaluation of user interfaces for touch-sensitive text entry on
`
`mobile devices;
`
`(e) working with Digit Wireless in 2002 on the evaluation of user
`
`performance for user interfaces for digital text entry on mobile devices; and
`
`
`
`4
`
`Google Ex. 1002
`Google v. Philips
`
`

`
`
`
`(f) working with Microsoft Research in 1999 on the development,
`
`evaluation and improvement of user interfaces for web browsing, in particular the
`
`mechanisms for revisiting pages (such as through the “back” button or
`
`bookmarks).
`
`13. At the University of Canterbury, I currently teach the following
`
`courses:
`
`(a)
`
`a course on introductory computer programming designed for first
`
`year students across disciplines; and
`
`(b)
`
`courses on HCI for computer science students at all university levels
`
`(including honors and masters level students).
`
`14.
`
`I also manage an active research lab with a number of graduate level
`
`students. I have supervised twelve students to successful completion of their
`
`Ph.D.’s in the field of HCI. In addition, I have supervised graduate students who
`
`have worked on mobile text entry techniques as part of their work in my lab.
`
`15.
`
`In 2015, I was elected to the Association of Computing Machinery
`
`(ACM) Computer Human Interaction (CHI) Academy, which honors the principal
`
`leaders in the field of HCI. I have been a continuous member of the ACM for over
`
`two decades.
`
`
`
`5
`
`Google Ex. 1002
`Google v. Philips
`
`

`
`
`
`16. Since the early 1990s, I have routinely attended annual conferences
`
`relating to the field of HCI and have regularly read journals that cover research in
`
`the field of HCI.
`
`17.
`
`I have served on the editorial boards of leading journals in the HCI
`
`field (including ACM Transactions on Computer-Human Interaction, Human-
`
`Computer Interaction Journal, and International Journal of Human-Computer
`
`Studies, Interacting with Computers, and Foundations and Trends In Human
`
`Computer Interaction), and have participated in the review process for numerous
`
`articles, including articles related to text entry on mobile devices.
`
`18.
`
`I have served in senior leadership roles for the leading conference in
`
`the field of HCI—the ACM CHI Annual Conference on Human Factors in
`
`Computing Systems. I have served as paper and notes chair for CHI 2014 and
`
`2015, and as subcommittee chair for CHI 2011.
`
`19.
`
`I have authored over one hundred and fifty publications related to HCI
`
`in international journals and conferences. These publications include, among
`
`others, several publications related to text entry in graphical user interfaces.
`
`20. Details of my professional qualifications and background are more
`
`fully set forth in my curriculum vitae, a copy of which is attached as Exhibit 1.
`
`6
`
`
`
`
`
`
`
`Google Ex. 1002
`Google v. Philips
`
`

`
`
`
`III. COMPENSATION AND RELATIONSHIP TO THE PARTIES
`
`21.
`
`I am being compensated at an hourly rate of $475 USD, plus
`
`expenses, for the time I spend working on this matter. This is my standard
`
`consulting rate. I am an independent party and my compensation is not contingent
`
`upon the outcome of this matter.
`
`22.
`
`It is my understanding that Koninklijke Philips Electronics N.V.
`
`(“Philips”), is listed as the assignee of the ’913 patent. Prior to this matter, I have
`
`not been employed or retained by Google or Philips. I own no stock in Google or
`
`Philips, and am aware of no other financial interest I have with those companies.
`
`IV. LEGAL STANDARDS
`
`23. Although I am not an attorney and do not expect to offer any opinions
`
`regarding the law, I have been informed of certain legal principles that I relied on
`
`in forming the opinions set forth in this report.
`
`A.
`
`24.
`
`Priority Date
`
`I have been asked to assume that the priority date of the ’913 patent is
`
`June 30, 2001.
`
`B. Claim Construction
`
`25.
`
`I understand that the broadest reasonable construction standard applies
`
`in inter partes review proceedings. I understand that, under this standard, claim
`
`terms are presumed to be given their ordinary and customary meaning as would be
`
`
`
`7
`
`Google Ex. 1002
`Google v. Philips
`
`

`
`
`
`understood by one of ordinary skill in the art at the time of the invention. The
`
`presumed meaning of a claim term may be rebutted based on the disclosures of the
`
`patent’s specification. I also understand that, under the broadest reasonable
`
`construction standard, it is appropriate to consider the file history of the patent in
`
`determining how a claim term would be understood by the person of ordinary skill
`
`in the art.
`
`26.
`
`I have used the broadest reasonable construction standard to construe
`
`each of the claim terms discussed herein.
`
`C. Anticipation
`
`27.
`
`I understand that for a claim to be anticipated, a single prior art
`
`reference must disclose to the person of ordinary skill in the art, either expressly or
`
`inherently, each and every limitation set forth in the claim as arranged in the claim.
`
`I understand that claims are unpatentable if they are anticipated by the prior art.
`
`D. Obviousness
`
`28.
`
`I understand that even if a claim is not anticipated, an invention that
`
`would have been obvious to the person of ordinary skill at the time of the invention
`
`is not patentable. I understand that an invention is obvious if the person of
`
`ordinary skill in the art would have had reason to make the invention, and a
`
`reasonable expectation of success in so doing. I understand that obviousness is
`
`determined by considering several factors, including: the state of the art at the time
`
`
`
`8
`
`Google Ex. 1002
`Google v. Philips
`
`

`
`
`
`the invention was made; the level of ordinary skill in the art; differences between
`
`what is described in the art and the claims at issue; and objective evidence of
`
`nonobviousness (such as commercial success, long-felt but unsolved needs, failure
`
`of others, and unexpected results) or obviousness (such as simultaneous invention).
`
`E.
`
`29.
`
`Person of Ordinary Skill in the Art
`
`I understand that the person of ordinary skill in the art (“POSA”) is a
`
`hypothetical person who is presumed to have known all of the relevant art at the
`
`time of the invention. I understand that the person of ordinary skill in the art may
`
`possess the education, skills, and experience of multiple actual people who would
`
`work together as a team to solve a problem in the field. I understand that factors
`
`that may be considered in determining the level of ordinary skill in the art may
`
`include: (1) the educational level of the inventor; (2) type of problems encountered
`
`in the art; (3) prior art solutions to those problems; (4) rapidity with which
`
`innovations are made; (5) sophistication of the technology; and (6) educational
`
`level of active workers in the field.
`
`30. On the basis of my consideration of these factors and my experience
`
`in the human computer interaction field, including my familiarity with text entry
`
`graphical user interface design, I have been asked to opine as to the level of skill of
`
`the hypothetical person of ordinary skill in the art to which the ’913 patent is
`
`directed. In my opinion, the hypothetical person of ordinary skill in the art would
`
`
`
`9
`
`Google Ex. 1002
`Google v. Philips
`
`

`
`
`
`have at least an undergraduate degree in computer science or computer
`
`engineering, or the equivalent. In addition, the POSA would have at least two
`
`years of experience in designing and/or implementing user interfaces, or equivalent
`
`academic experience.
`
`V.
`
`SUMMARY OF OPINION
`
`31.
`
`I understand that Google requests inter partes review of Claims 1-16
`
`of the ’913 patent, titled “Text Entry Method and Device Therefor.” The ’913
`
`patent issued on May 27, 2014 and has been assigned to Philips.
`
`32.
`
`In my opinion, Claims 1-16 would have been anticipated and/or
`
`obvious to the person of ordinary skill as of June 30, 2001 based on the following
`
`references:
`
`(1) Japanese Unexamined Patent Application Publication No. 2000-
`
`148366 (“Sakata II”) (Ex. 1004), which published on May 26, 2000. I
`
`have been asked to assume that is prior art to the ’913 patent. I have
`
`reviewed both the original Japanese application and a certified
`
`translation thereof, which are included in Ex. 1004.
`
`(2) U.S. Patent No. 6,094,197 to Buxton (“Buxton”) (Ex. 1006),
`
`which was issued from an application filed in the United States May
`
`17, 1995. I have been asked to assume that Buxton is prior art to the
`
`’913 patent.
`
`
`
`10
`
`Google Ex. 1002
`Google v. Philips
`
`

`
`
`
`(3) U.S. Patent No. 6,271,835 to Hoeksma (“Hoeksma”) (Ex. 1005),
`
`which issued from an application filed in the United States on
`
`September 3, 1998. I have been asked to assume that Hoeksma is
`
`prior art to the ’913 patent.
`
`33.
`
`It is my opinion that claims 1 and 3-16 of the ’913 patent would have
`
`been obvious over Sakata II, that claims 1 and 3-16 would have been obvious over
`
`Sakata II in view of Buxton, and that claims 1-5, 9, and 13 would have been
`
`obvious over Hoeksma and Sakata II.
`
`VI. BACKGROUND
`
`A. The Increasing Capabilities of Computing Devices
`34.
`In the late 1970s and early 1980s, personal computers with graphical
`
`user interfaces became widely available. Their relatively low cost, convenient size,
`
`and graphical user interfaces made computing accessible to home users and in
`
`schools.
`
`35. The use of graphical user interfaces was particularly important in
`
`enabling non-experts to successfully interact with computers. Rather than having to
`
`learn, remember, and type commands with a specific syntax, users could instead
`
`interact with familiar objects, such as files, folders, printers, and windows on a
`
`desktop.
`
`
`
`11
`
`Google Ex. 1002
`Google v. Philips
`
`

`
`
`
`36.
`
`In examining the “glowing enthusiasm” for new forms of graphical
`
`user interfaces (in contrast to “grudging acceptance or outright hostility” for
`
`command languages), Shneiderman (1983) introduced the term “direct
`
`manipulation” to characterize the user’s ability to interact with displayed objects
`
`through “physically obvious and intuitively natural means” (such as a mouse). Ex.
`
`1011 (Shneiderman, B., “Direct Manipulation: A Step Beyond Programming
`
`Languages”, Computer, IEEE., 16(8), pp. 57-69 (Aug. 1983).).
`
`37. Since the introduction of personal computers, many of their hardware
`
`capabilities have improved dramatically. These include processing speed, memory
`
`size, display resolution, and network communication capabilities. These
`
`improvements have allowed computers to support richer media (e.g., audio and
`
`video processing), more computationally demanding tasks (e.g., searching large
`
`information spaces), and collaborative activities (e.g., media sharing over
`
`distance).
`
`38. Gradual hardware improvements also opened opportunities to
`
`miniaturize computing devices. “Luggable” computers first appeared in early
`
`1970s, with laptop and handheld form-factors becoming available in the early
`
`1980s. As processing and other capabilities improved over time, functions and
`
`applications that initially required larger computing devices became available on
`
`smaller devices.
`
`
`
`12
`
`Google Ex. 1002
`Google v. Philips
`
`

`
`
`
`39. The advent of the World Wide Web and software web-browsers to
`
`access it in the early 1990s had a dramatic effect on the ease with which networked
`
`information could be accessed through computing devices. As the range of
`
`resources on the web rapidly increased, it became increasingly desirable to make
`
`these resources available to users in smaller, mobile devices.
`
`40. Prior to the mid-1990s mobile computing devices could be broadly
`
`categorized as developing along two parallel tracks: increasingly capable
`
`cellphones (also called “feature phones”), and increasingly capable personal digital
`
`assistants (PDAs). The primary focus of cellphones was mobile telephony,
`
`including short-message service capabilities. The primary focus of PDAs was to
`
`provide computing and information management utilities, including calendars,
`
`reminders, and to-do lists.
`
`41.
`
`In the mid-1990s the feature phone and PDA tracks of mobile
`
`development converged, with the IBM Simon device representing the first
`
`wireless, personal communicator. The IBM Simon included a variety of PDA
`
`utilities (a calendar, address book, note-taking utility, and so on), as well as various
`
`functions based on cellular telephony (including the ability to send and receive
`
`phone calls, faxes, and emails). Nokia’s 9000 Communicator, released in 1996,
`
`additionally incorporated a mobile web browser.
`
`
`
`13
`
`Google Ex. 1002
`Google v. Philips
`
`

`
`
`
`42. With the advent of PDAs and wireless, personal communicators in the
`
`mid-1990s (and their miniaturization), there was a need to optimize the user
`
`experience. The user interface needed to be readable but easy to use at the same
`
`time. Touchscreens, which I discuss below, were implemented on such devices to
`
`facilitate user inputs.
`
`B.
`Touch Input
`43. As of the mid-1990s (and much earlier), touchscreens were
`
`implemented on mobile devices, including PDAs and wireless, personal
`
`communicators. Touchscreens employ touch input, which allows the user to input
`
`information to a computing device by placing an object onto, or near to, a surface
`
`that registers the location of the object. Typically, the object is a stylus or finger.
`
`44. Various hardware technologies had been developed to support touch
`
`input since the 1960s, with related “lightpen” technologies first appearing in the
`
`1950s. Three important forms of hardware technologies for sensing touch inputs
`
`are resistive, capacitive, and inductive. The benefits of resistive technologies
`
`include their ability to sense any form of contact, including a gloved hand or
`
`pencil. Capacitive technologies facilitate concurrent sensing of multiple contacts.
`
`Inductive technologies facilitate touch inputs without requiring actual contact with
`
`the surface.
`
`
`
`14
`
`Google Ex. 1002
`Google v. Philips
`
`

`
`
`
`45. A touch sensitive surface may be overlaid on top of a computer
`
`display, in which case the display becomes a touchscreen, which affords a variety
`
`of interaction benefits to users. In particular, users can interact directly with items
`
`on the display by moving their finger or stylus onto an object or icon displayed on
`
`the screen. This ability to directly interact with objects on the screen using touch
`
`input eases problems that can arise from the physical separation between the input
`
`device and display (as occurs with a mouse, for example) – these problems include
`
`the cognitive and physical effort of mapping movement at one place to output at
`
`another, as well as the need for additional physical space for the separately located
`
`input device.
`
`46. Numerous researchers from the mid-1980s onwards have referred to
`
`touchscreen input as being more “natural”, “intuitive”, and “easy to learn” than
`
`other input methods.
`
`C. Methods for Expressing Intentions Through Touch
`47. Prior to the priority date of the ’913 patent, touch input created
`
`slightly different opportunities for expressing user intentions than those provided
`
`by an indirect pointing device, such as a mouse. A mouse would typically support
`
`several buttons and a scrollwheel, creating opportunities for different intentions to
`
`be expressed on objects by pressing different buttons or rotating the wheel.
`
`Furthermore, the location of a visible cursor was continually updated when the
`
`
`
`15
`
`Google Ex. 1002
`Google v. Philips
`
`

`
`
`
`mouse moved, allowing users to view the correspondence between cursor location
`
`and underlying objects on the display. These properties of mouse interaction would
`
`allow a user to, for example, move the cursor over an object, then carry out
`
`different functions on it with different mouse buttons, such as selecting it by
`
`clicking the left mouse button or viewing a menu of object commands by clicking
`
`the right mouse button.
`
`48. Touch input, in contrast, did not normally support physical buttons
`
`(unless a specialized stylus was used), and depending on the sensing hardware
`
`used, it would not normally support cursor tracking either. Consequently, different
`
`methods of expressing intentions were developed for touch input. Different
`
`functions might be executed on an object by tapping or double-tapping on an
`
`object. Long-duration stationary “dwell” or “hold” actions were also commonly
`
`used to activate specific object functions. For example, the Apple Newton
`
`MessagePad used stationary “hold” actions to select objects for movement, as the
`
`manual excerpt below shows.
`
`
`
`16
`
`Google Ex. 1002
`Google v. Philips
`
`

`
`
`
`
`Ex. 1012 (Apple Newton MessagePad Handbook (1995)) (“Newton Manual”) at
`
`64.
`
`49. The notion of using a predetermined time to determine a second
`
`interpretation for a user’s action was well known prior to the priority date of the
`
`’913 patent. The long-duration stationary “hold” action for object selection on the
`
`Apple Newton MessagePad was one such example. Another example that was
`
`widely used involved using a long-press to initiate an auto-repeat mode of
`
`character entry on virtual and physical keyboards and keypads. For example, the
`
`Psion 3a physical keyboard would enter a single character if a key was pressed for
`
`less than a predetermined time, but enter an “auto-repeat” mode if pressed and held
`
`for longer than a predetermined time. Ex. 1016 (Psion Series 3a User Guide
`
`(1993)) at 4. The Simon mobile device used a similar method of time-based
`
`determination for auto-repeat with its “Clear” virtual button for deleting characters:
`
`“The Clear button erases previously entered characters, one at a time, like the
`17
`
`
`
`Google Ex. 1002
`Google v. Philips
`
`

`
`
`
`backspace key on the standard keyboard. If you touch and hold the Clear button, it
`
`will repeat.” Ex. 1017 (IBM Simon Users Manual (1994)) at 19.
`
`50.
`
`“Marking menus” provide another well-known example of such user
`
`interfaces. Ex. 1019 (Kurtenbach, G. and Buxton, W., The Limits of Expert
`
`Performance Using Hierarchic Marking Menus. Proceedings of the INTERCHI
`
`‘93 Conference on Human Factors in Computing Systems, 482-87, New York:
`
`ACM (1993)). Marking menus were designed to help users attain expert levels of
`
`command selection performance. An expert user could select an item by making a
`
`rapid directional gesture; but if the user paused for more than a predetermined time
`
`during execution of their gesture, then a menu of alternative options was displayed
`
`to assist them by visually guiding their selection.
`
`51. A predetermined time was also used to change between primary and
`
`secondary character interpretations of key-presses, using methods such as “Half-
`
`QWERTY”, which I describe later.
`
`D.
`Interfaces for Accessing Functionality on Small Devices
`52. Regardless of the size of the computer display, interface designers
`
`faced challenges in organizing interface components to best meet the user’s needs.
`
`53.
`
`In general, interface designers would seek to maximize three aspects
`
`of interaction—learnability (or the intuitiveness of the interface), efficiency (the
`
`
`
`18
`
`Google Ex. 1002
`Google v. Philips
`
`

`
`
`
`ability to quickly complete tasks), and subjective satisfaction (the degree to which
`
`users liked using the system).
`
`54. By the late 1980s, various user interface guidelines for assisting the
`
`attainment of these goals were well known and were included in standard text
`
`books for computer science undergraduates (see, for example, Ex. 1020
`
`Shneiderman, B., Designing the User Interface: Strategies for Effective Human-
`
`Computer Interaction, Ch. 2, 60-62, Addison-Wesley, Reading, Mass. (1987)
`
`(providing a set of “golden rules” of interface design)). Among these guidelines,
`
`user interface “consistency” was known to be a “golden rule” for successful
`
`interface design. Id. at 61. The benefits of consistency included the fact that users
`
`could successfully transfer their understanding from one interface to another,
`
`thereby easing the learning burden of a new interface.
`
`55.
`
`In Chapter 5 of his influential text on “Usability Engineering,”
`
`Nielsen described the critical role of consistency in interface design as follows:
`
`“Consistency is one of the most basic usability principles. If users know that the
`
`same command or the same action will always have the same effect, they will feel
`
`more confident in using the system . . . because they will already have part of the
`
`knowledge needed to operate new parts of the system,” and “The same information
`
`should be presented in the same location on all screens.” Ex. 1021 (Nielson, J.,
`
`
`
`19
`
`Google Ex. 1002
`Google v. Philips
`
`

`
`
`
`Usability Engineering, Ch. 5, Morgan Kaufmann Pub., San Francisco (1993) at
`
`132.
`
`56.
`
`It was well understood that consistency should be applied to elemental
`
`interface controls, such as the layout of a keyboard or keypad on a mobile device,
`
`as doing so facilitated rapid understanding of the method for text entry on a device.
`
`It also made users less prone to make errors. Indeed, consistency was so elemental
`
`for keypad and keyboard layouts that International Standards were defined to guide
`
`and advise designers (such as ISO/IEC 9995, first defined in 1984).
`
`57. Conforming to keyboard/keypad standards, and permitting a default
`
`state, was well-known in the art