`
`
`
`
`
`UNITED STATES PATENT AND TRADEMARK OFFICE
`____________________
`
`
`BEFORE THE PATENT TRIAL AND APPEAL BOARD
`
`
`____________________
`
`GOOGLE LLC,
`
`Petitioner,
`
`v.
`
`NEONODE SMARTPHONE LLC,
`
`Patent Owner.
`
`____________________
`
`Case No. IPR2021-01041
`U.S. Patent No. 8,095,879
`
`____________________
`
`
`PETITION FOR INTER PARTES REVIEW
`
`
`

`

`TABLE OF CONTENTS
`
`I.
`Relief Requested .............................................................................................. 1
`The ’879 Patent ................................................................................................ 2
`II.
`III. Claim Construction .......................................................................................... 4
`IV. Level of Ordinary Skill .................................................................................... 4
`V.
`The Challenged Claims Are Unpatentable and Should Be Cancelled ............ 5
`A.
`[Ground 1] Claims 1-5, 13, and 15-17 are rendered obvious by
`Robertson and Maddalozzo ................................................................... 5
`1.
`Robertson .................................................................................... 5
`2. Maddalozzo ............................................................................... 10
`3.
`Claim 1 ...................................................................................... 11
`a.
`[1Preamble] “A non-transitory computer readable
`medium storing a computer program with
`computer program code, which, when read by a
`mobile handheld computer unit, allows the
`computer to present a user interface for the mobile
`handheld computer unit, the user interface
`comprising:” ................................................................... 12
`i.
`Motivation ............................................................ 15
`ii.
`Expected success .................................................. 18
`[1a] “a touch sensitive area in which a
`representation of a function is provided,” ...................... 19
`[1b] “wherein the representation consists of only
`one option for activating the function and” .................... 23
`[1c] “wherein the function is activated by a multi-
`step operation comprising (i) an object touching
`the touch sensitive area at a location where the
`representation is provided and then (ii) the object
`
`b.
`
`c.
`
`d.
`
`IPR2021-01041
`U.S. Patent 8,095,879
`
`
`ii
`
`

`

`IPR2021-01041
`U.S. Patent 8,095,879
`
`
`e.
`
`gliding along the touch sensitive area away from
`the touched location,” ..................................................... 25
`[1d] “wherein the representation of the function is
`not relocated or duplicated during the gliding.” ............. 29
`[Claim 2] “wherein the function, when activated, causes
`the user interface to display icons representing different
`services or settings for a currently active application.” ............ 30
`i.
`Motivation ............................................................ 32
`ii.
`Expected success .................................................. 33
`[Claim 3] “wherein the user interface is characterised in,
`that a selection of a preferred service or setting is done by
`tapping on a display icon corresponding to the preferred
`service or setting.”..................................................................... 34
`i.
`Motivation ............................................................ 35
`ii.
`Expected success .................................................. 36
`[Claim 4] “wherein the function, when activated, causes
`the user interface to display a keyboard and a text field.” ........ 37
`i.
`Motivation ............................................................ 40
`ii.
`Expected success .................................................. 40
`[Claim 5] “wherein said text field is used for inputting
`and editing of text through said keyboard.” .............................. 41
`[Claim 13] “wherein the user interface is characterised in,
`that said representation of said function is located at the
`bottom of said touch sensitive area.” ........................................ 41
`[Claim 15] “characterised in, that said computer program
`code is adapted to function as a shell upon an operating
`system.” ..................................................................................... 43
`[Claim 16] “wherein the representation is finger-sized.” ......... 44
`
`4.
`
`5.
`
`6.
`
`7.
`
`8.
`
`9.
`
`10.
`
`iii
`
`

`

`B.
`
`11.
`
`IPR2021-01041
`U.S. Patent 8,095,879
`
`Motivation ............................................................ 44
`i.
`Expected success .................................................. 45
`ii.
`[Claim 17] “wherein the location where the
`representation is provided does not provide touch
`functionality for a different function.” ...................................... 46
`[Ground 2] Claims 6-7, and 9 are rendered obvious by
`Robertson, Maddalozzo, and Vayda ................................................... 47
`1.
`Overview of Vayda ................................................................... 47
`2.
`[Claim 6] “wherein the function, when activated, causes
`the user interface to display a list with a library of
`available applications and files on the mobile handheld
`computer unit.” .......................................................................... 49
`i.
`Motivation ............................................................ 51
`ii.
`Expected success .................................................. 52
`[Claim 7] “wherein the user interface is characterised in,
`that a selection of an application or file is done by gliding
`the object along said touch sensitive area so that a
`representation of a desired one of said application or file
`is highlighted, raising said object from said touch
`sensitive area, and then tapping on said touch sensitive
`area.” ......................................................................................... 53
`i.
`Motivation ............................................................ 57
`ii.
`Expected success .................................................. 58
`[Claim 9] “wherein the user interface is characterised in,
`that, one item in said list is highlighted by a moveable
`marking, and the user interface enables list navigation
`whereby gliding the object along the touch sensitive area
`in a direction towards te top of said list or towards the
`bottom of said list causes said marking to move in the
`same direction without scrolling the list.” ................................ 59
`
`3.
`
`4.
`
`iv
`
`

`

`IPR2021-01041
`U.S. Patent 8,095,879
`
`
`C.
`
`D.
`
`[Ground 3] Claim 12 rendered obvious by Robertson,
`Maddalozzo, and Bedford-Roberts ..................................................... 61
`1.
`[Claim 12] “wherein the user interface is characterised in,
`that an active application, function, service or setting is
`advanced one step by gliding the object along the touch
`sensitive area from left to right, and that the active
`application, function, service or setting is closed or
`backed one step by gliding the object along the touch
`sensitive area from right to left.” .............................................. 61
`i.
`Motivation ............................................................ 63
`ii.
`Expected success .................................................. 64
`[Ground 4] Claims 1, 4-6, 13, and 15-17 are rendered obvious
`by Tarpenning ...................................................................................... 65
`1.
`Tarpenning ................................................................................ 65
`2.
`Claim 1 ...................................................................................... 67
`a.
`Element [1Preamble] ...................................................... 68
`b.
`Element [1a] .................................................................... 71
`c.
`Element [1b] ................................................................... 75
`d.
`Element [1c] .................................................................... 76
`i.
`Motivation ............................................................ 82
`ii.
`Expected success .................................................. 83
`Element [1d] ................................................................... 84
`e.
`Claim 4 ...................................................................................... 84
`i.
`Motivation ............................................................ 89
`ii.
`Expected success .................................................. 90
`Claim 5 ...................................................................................... 90
`
`3.
`
`4.
`
`v
`
`

`

`5.
`
`2.
`
`E.
`
`IPR2021-01041
`U.S. Patent 8,095,879
`
`Claim 6 ...................................................................................... 90
`i.
`Motivation ............................................................ 93
`ii.
`Expected success .................................................. 94
`Claim 13 .................................................................................... 95
`6.
`Claim 15 .................................................................................... 96
`7.
`Claim 16 .................................................................................... 96
`8.
`Claim 17 .................................................................................... 97
`9.
`[Ground 5] Claims 2-3, 7, and 9 are rendered obvious by
`Tarpenning and Vayda ........................................................................ 98
`1.
`Claim 2 ...................................................................................... 98
`i.
`Motivation ............................................................ 99
`ii.
`Expected success .................................................. 99
`Claim 3 ....................................................................................100
`i.
`Motivation ..........................................................101
`ii.
`Expected success ................................................101
`Claim 7 ....................................................................................101
`3.
`Claim 9 ....................................................................................102
`4.
`[Ground 6] Claim 12 is rendered obvious by Tarpenning and
`Bedford-Roberts ................................................................................102
`1.
`Claim 12 ..................................................................................102
`VI. Non-Institution Under 35 U.S.C. § 325(d) Would Be Improper .................103
`VII. Non-Institution Under 35 U.S.C. § 314(a) Would Be Improper .................104
`A. Denial of institution under Apple v. Fintiv would be improper ........104
`
`F.
`
`vi
`
`

`

`IPR2021-01041
`U.S. Patent 8,095,879
`
`
`B. Denial of institution under General Plastic and Valve Corp.
`would be improper .............................................................................105
`VIII. Mandatory Notices .......................................................................................106
`A.
`Real Parties-in-Interest Under 37 C.F.R. § 42.8(b)(1) ......................106
`B.
`Related Matters Under 37 C.F.R. § 42.8(b)(2) .................................106
`C.
`Lead and Back-Up Counsel Under 37 C.F.R. § 42.8(b)(3) ..............107
`D.
`Service Information Under 37 C.F.R. § 42.8(b)(4) ...........................108
`IX. Standing .......................................................................................................108
`X.
`Conclusion ...................................................................................................108
`
`
`vii
`
`

`

`
`
`LIST OF EXHIBITS
`
`IPR2021-01041
`U.S. Patent 8,095,879
`
`
`Description
`Exhibit
`Ex-1001 U.S. Patent No. 8,095,879 (“the ’879 patent”)
`Ex-1002
`File History of U.S. Patent No. 8,095,879
`Ex-1003 Declaration of Jacob O. Wobbrock, Ph.D.
`Ex-1004 CV of Jacob O. Wobbrock, Ph.D.
`Ex-1005 George G. Robertson, Buttons as First Class Objects on an X
`Desktop, UIST: Proceedings of the ACM Symposium on User
`Interface Software and Technology: Hilton Head, South Carolina,
`USA, pp. 35-44 (Nov. 11-13, 1991) (“Robertson”)
`Ex-1006 U.S. Patent No. 7,768,501 to Maddalozzo Jr. et al. (“Maddalozzo”)
`Ex-1007 U.S. Patent No. 5,745,717 to Vayda et al. (“Vayda”)
`Ex-1008 U.S. Patent No. 5,870,092 to Bedford-Roberts (“Bedford-Roberts”)
`Ex-1009 U.S. Patent No. 6,181,344 to Tarpenning et al. (“Tarpenning”)
`Ex-1010 U.S. Patent Application Publication No. 2001/0035880 to Musatov
`et al. (“Musatov”)
`Ex-1011 U.S. Patent Application Publication No. 2002/0149569 to Dutta
`et al. (“Dutta”)
`Ex-1012 U.S. Patent Application Publication No. 2003/0013483 to Ausems
`et al. (“Ausems”)
`Ex-1013 U.S. Patent No. 6,249,277 to Varveris (“Varveris”)
`Ex-1014 U.S. Patent No. 6,344,848 to Rowe et al. (“Rowe”)
`Ex-1015 U.S. Patent No. 6,388,870 to Canova Jr. et al. (“Canova”)
`Ex-1016 U.S. Patent No. 7,081,882 to Sowden et al. (“Sowden”)
`
`viii
`
`

`

`IPR2021-01041
`U.S. Patent 8,095,879
`
`
`Description
`Exhibit
`Ex-1017 U.S. Patent No. 5,347,295 to Agulnick et al. (“Agulnick”)
`Ex-1018 Declaration of Rachel J. Watters
`Ex-1019 Declaration of Kelley M. Hayes Greenhill
`Ex-1020 Wisconsin MARC Record for Robertson, available at https://wrlc-
`gu.primo.exlibrisgroup.com/discovery/sourceRecord?vid=01WRLC
`_GUNIV:01WRLC_GUNIV&docId=alma9911002247004101&rec
`ordOwner=01WRLC_NETWORK (last visited May 26, 2021)
`
`Ex-1021
`
`Library of Congress MARC Record for Robertson, available at
`https://catalog.loc.gov/vwebv/staffView?searchId=16367&recPo
`inter=0&recCount=25&bibId=11489112 (last visited May 27,
`2021)
`Ex-1022 U.S. Copyright Office Record, available at
`https://cocatalog.loc.gov/cgi-
`bin/Pwebrecon.cgi?v1=8&ti=1,8&Search%5FArg=Proceedings
`%20of%20the%20ACM%20Symposium%20on%20User%20Int
`erface%20Software%20and%20Technology&Search%5FCode=
`TALL&CNT=25&PID=S5YOB6rlECB6M_LEJD300IQpy&SE
`Q=20210527133326&SID=3 (last visited May 27, 2021)
`Ex-1023 WorldCat Record for OCLC Control Number 28864712, available
`at https://www.worldcat.org/title/proceedings-of-the-acm-
`symposium-on-user-interface-software-and-
`technology/oclc/28864712 (last visited June 2, 2021)
`Ex-1024 WorldCat Record for OCLC Control Number 270712133, available
`at https://www.worldcat.org/title/proceedings-of-the-acm-
`symposium-on-user-interface-software-and-technology-hilton-head-
`south-carolina-usa-november-11-13-1991/oclc/270712133 (last
`visited June 2, 2021)
`Ex-1025 Neonode Smartphone LLC v. Apple Inc., No. 6:20-cv-00505-ADA,
`Docket No. 40, Order Granting Stay (W.D. Tex. Dec. 8, 2020)
`
`ix
`
`

`

`IPR2021-01041
`U.S. Patent 8,095,879
`
`
`Description
`Exhibit
`Ex-1026 Neonode Smartphone LLC v. Samsung Electronics Co. Ltd.,
`No. 6:20-cv-00507-ADA, Text Order Granting Stay (W.D. Tex.
`Dec. 11, 2020)
`
`Ex-1027
`
`Excerpts from Robert W. Scheifler & James Gettys, X Window
`System, Version 11, Release 5 (Digital Press 3d ed. 1992)
`Ex-1028 U.S. Patent No. 5,745,116 to Pisutha-Arnond (“Pisutha-Arnond”)
`Ex-1029 U.S. Patent No. 5,644,628 to Schwarzer et al. (“Schwarzer”)
`
`
`
`x
`
`

`

`Relief Requested
`Petitioner Google LLC requests review under 35 U.S.C. § 311 of claims 1-7,
`
`IPR2021-01041
`U.S. Patent No. 8,095,879
`
`
`I.
`
`9, 12-13, and 15-17 of U.S. Patent 8,095,879 and cancellation under pre-AIA
`
`35 U.S.C. § 103 based on:
`
`Ex-1005 Robertson, Buttons as First Class Objects on an X Desktop,
`
`(November 11-13, 1991); prior art under § 102(b)
`
`Ex-1006 U.S. Patent 7,768,501 (“Maddalozzo”); filed May 1, 1998;
`
`issued August 3, 2010; prior art under § 102(e)
`
`Ex-1007 U.S. Patent 5,745,717 (“Vayda”); issued April 28, 1998; prior art
`
`under § 102(b)
`
`Ex-1008 U.S. Patent 5,870,092 (“Bedford-Roberts”); issued February 9,
`
`1999; prior art under § 102(b)
`
`Ex-1009 U.S. Patent 6,181,344 (“Tarpenning”); issued January 30, 2001;
`
`prior art under § 102(b)
`
`
`
`Ground
`
`Claims
`
`§ 103 Basis
`
`1
`
`1-5, 13, 15-17 Rendered obvious by Robertson and Maddalozzo
`
`1
`
`

`

`IPR2021-01041
`U.S. Patent No. 8,095,879
`
`
`Ground
`
`Claims
`
`§ 103 Basis
`
`2
`
`3
`
`4
`
`5
`
`6
`
`6-7, 9
`
`Rendered obvious by Robertson, Maddalozzo, and
`
`Vayda
`
`12
`
`Rendered obvious by Robertson, Maddalozzo, and
`
`Bedford-Roberts
`
`1, 4-6, 13, 15-17 Rendered obvious by Tarpenning
`
`2-3, 7, 9
`
`Rendered obvious by Tarpenning and Vayda
`
`12
`
`Rendered obvious by Tarpenning and
`
`Bedford-Roberts
`
`II. The ’879 Patent
`The ’879 patent describes a touch-based user interface for a mobile handheld
`
`computer unit (e.g., “mobile phone,” “PDA,” “laptop computer”). Ex-1001, 1:6-9,
`
`1:24-33. The user interface includes a “touch sensitive area” 1 (red) divided into
`
`menu area 2 and display area 3. Menu area 2 includes “representation[s]” (blue) of
`
`predefined functions 21, 22, 23 activated when “touch sensitive area 1 detects a
`
`movement [(orange)] of an object 4 [finger, pen] with its starting point A within
`
`the representation [(blue)] of a function on the menu area 2 and with a direction B
`
`2
`
`

`

`from the menu area 2 to the display area 3 [(yellow)],” as shown. Ex-1001, 3:50-
`
`IPR2021-01041
`U.S. Patent No. 8,095,879
`
`
`54, 4:1-11, 2:10-14, 6:11-15.
`
`
`
`
`
`
`
`Ex-1001, FIGS. 1, 2 (annotated).
`
`When the functions for representations 21 and 23 are activated by a flick
`
`gesture, display area 3 displays icons representing services or functions (green,
`
`Figure 3) or a menu/library of applications and files (bright green, Figure 6), as
`
`shown. Ex-1001, 4:7-23.
`
`3
`
`

`

`IPR2021-01041
`U.S. Patent No. 8,095,879
`
`
`
`
`Ex-1001, FIGS. 3, 6 (annotated).
`
`III. Claim Construction
`No terms need to be construed to resolve unpatentability. Realtime Data,
`
`LLC v. Iancu, 912 F.3d 1368, 1375 (Fed. Cir. 2019).
`
`IV. Level of Ordinary Skill
`A person of ordinary skill in the art (“POSITA”) of the ’879 patent would
`
`have at least a bachelor’s degree in Computer Science, Human-Computer
`
`Interaction, Symbolic Systems, or related engineering disciplines, and at least two
`
`years of experience designing and programming graphical user interfaces. Relevant
`
`work experience can substitute for formal education and advanced degree studies
`
`could substitute for work experience. Ex-1003, ¶49.
`
`4
`
`

`

`IPR2021-01041
`U.S. Patent No. 8,095,879
`
`
`V. The Challenged Claims Are Unpatentable and Should Be Cancelled
`A.
`[Ground 1] Claims 1-5, 13, and 15-17 are rendered obvious by
`Robertson and Maddalozzo
`1.
`Robertson
`Robertson discloses a gesture-based “high-level user interface toolkit”
`
`having configurable buttons (“XButtons”) that can be used with “touch screens for
`
`[user] input.” Ex-1005, Abstract, §§ 1, 1.2, 2. Robertson activates a button’s
`
`corresponding functions using gestures, e.g., “pen-based gestural input[s],” such as
`
`the user touching the screen and then sliding or gliding the pen across the screen.
`
`Ex-1005, §§ 3-3.1. When “a user gestures at an XButton, a gesture parser interprets
`
`… pen movement and classifies it as one of a small set of easily differentiated
`
`gestures (flick left, flick right, flick up, flick down, click, rubout, check, or insert),”
`
`and then “executes the appropriate action (i.e., there is an action for each of the
`
`gestures)” corresponding to the gesture. Ex-1005, § 3.1. The screen also shows a
`
`“track” of the gesture being drawn. Id.
`
`Annotated Figure 1 shows a touch screen (red) having a “Phone” button
`
`(blue) with a corresponding function to dial a phone number using a multi-step
`
`“flick right” gesture, activated by the user touching the pen on the “Phone” button
`
`and drawing a line to the right on the screen (orange). Ex-1005, §§ 3-3.1, 4.2.
`
`5
`
`

`

`IPR2021-01041
`U.S. Patent No. 8,095,879
`
`
`555-123-4567
`
`Ex-1005, FIG. 1 (annotated).
`
`The user can also activate the “button editor” menu function of Figure 3
`
`using an “Insert” gesture—drawing a “caret” starting on the button to be edited—
`
`as shown. Ex-1005, §§ 3.2-3.3.
`
`
`
`6
`
`

`

`IPR2021-01041
`U.S. Patent No. 8,095,879
`
`
`555-123-4567
`
`
`
`Ex-1005, FIG. 1 (annotated).
`
`Robertson uses toolkits to create buttons and assign functions to gestures
`
`using “the operating system command language.” Ex-1005, Abstract, §§ 3.2-3.3.
`
`Robertson discloses command languages, like Unix and Lisp, but does not limit
`
`itself to these languages. Ex-1005, §§ 1.2, 3.2; Ex-1003, ¶67.
`
`7
`
`

`

`IPR2021-01041
`U.S. Patent No. 8,095,879
`
`Annotated Figure 3 shows a button editor to configure button attributes (e.g.,
`
`title, bitmap, foreground, background, text) and assign functions (green) activated
`
`by gestures (orange). Ex-1005, §§ 3.2-3.3.
`
`Ex-1005, FIG. 3 (annotated).
`
`8
`
`
`
`

`

`IPR2021-01041
`U.S. Patent No. 8,095,879
`
`Robertson was publicly available to POSITAs more than one year before the
`
`’879 patent filing as part of published materials from the ACM’s 4th Annual
`
`Symposium on User Interface Software and Technology (UIST), held November
`
`11-13, 1991. Ex-1005; Ex-1003, ¶70. These papers were distributed to conference
`
`attendees, including POSITAs, at the conference. Ex-1003, ¶71. This is confirmed
`
`by the U.S. Copyright Office’s publication date of October 28, 1991. Ex-1019,
`
`¶21; Ex-1022. Robertson bears the ACM’s 1991 copyright date and numerical
`
`publication code and was available for purchase for $1.50. Ex-1005, 11 fn.;
`
`Ex-1003, ¶¶73-74. ACM publications were widely distributed and reliably
`
`published on or about the ACM copyright date, qualifying Robertson as publicly
`
`available on or about 1991. Microsoft Corp. v. IPA Techs. Inc., IPR2019-00811,
`
`Paper 44, at 142 (Oct. 15, 2020); Ericsson Inc. v. Intellectual Ventures I LLC,
`
`IPR2014-00527, Paper 41, at 10-12 (May 18, 2015); Ex-1003, ¶75.
`
`Robertson was catalogued and shelved at the Library of Congress and would
`
`have been publicly available to POSITAs through searching as of about September
`
`or October 1993. Ex-1019, ¶¶16-18. Robertson was also catalogued and shelved at
`
`the University of Wisconsin library and made publicly available to POSITAs
`
`through searching about August 20, 1998. Ex-1018, 1-2.
`
`9
`
`

`

`2. Maddalozzo
`Maddalozzo discloses mobile handheld computers, e.g., “laptop,”
`
`IPR2021-01041
`U.S. Patent No. 8,095,879
`
`
`“notebook,” and “calculator-size mini-portable computers,” having touch-screen
`
`user interfaces. Ex-1006, 1:12-35, 2:1-22. Like Robertson, these devices may run
`
`Unix or similar operating systems. Ex-1006, 4:36-40. These handheld computers
`
`have touch-sensitive on-screen keyboards (purple) allowing users to input text into
`
`a text field (orange). Ex-1006, Abstract, FIG. 6.
`
`10
`
`

`

`IPR2021-01041
`U.S. Patent No. 8,095,879
`
`
`
`
`
`Ex-1006, FIG. 6 (annotated).
`
`
`
`3.
`Claim 1
`Robertson and Maddalozzo render obvious claim 1.
`
`Robertson teaches or suggests each element of claim 1. Maddalozzo
`
`discloses, confirms, and renders obvious Robertson’s computing device being a
`
`mobile handheld computer unit having a “non-transitory computer readable
`
`medium” in [1Preamble]. Ex-1003, ¶79.
`
`11
`
`

`

`IPR2021-01041
`U.S. Patent No. 8,095,879
`
`
`a.
`
`[1Preamble] “A non-transitory computer readable
`medium storing a computer program with computer
`program code, which, when read by a mobile
`handheld computer unit, allows the computer to
`present a user interface for the mobile handheld
`computer unit, the user interface comprising:”1
`Robertson discloses or suggests [1Preamble]. Ex-1003, ¶¶80-89.
`
`Maddalozzo discloses and renders obvious the computer unit being a “mobile
`
`handheld computer unit” and the computer program being stored in a “non-
`
`transitory computer readable medium.” Ex-1003, ¶80.
`
`Robertson discloses a computer program with code for implementing
`
`gesture-activated buttons (e.g., XButtons), which, when read by a computing
`
`device, presents a user interface for receiving gestural inputs (e.g., touch-screen
`
`inputs) of the device. Ex-1005, Abstract, §§ 3.1-3.2, 4.1; Ex-1003, ¶81.
`
`Robertson discloses a “user interface toolkit, called XButtons” (computer
`
`program code) that presents user-activatable “on-screen buttons” to activate
`
`functions in a computing system, like an X Windows system. Ex-1005, Abstract.
`
`Robertson’s XButtons are implemented by computer programs having program
`
`
`1 Petitioner does not concede the preamble is limiting. If Patent Owner argues it is,
`
`Robertson and Maddalozzo render obvious [1Preamble].
`
`12
`
`

`

`code, like the “operating system command language.” Ex-1005, §§ 1.2, 3.2, 4.1;
`
`IPR2021-01041
`U.S. Patent No. 8,095,879
`
`
`Ex-1003, ¶82.
`
`Robertson suggests that the computing device can be a mobile handheld
`
`computer unit. Robertson describes “X”-based commands, Unix-based commands,
`
`and Lisp-based commands. Ex-1005, Abstract, §§ 1.2, 2.1, 3-3.2. A POSITA
`
`would understand that Robertson’s teachings can apply to many types of gesture-
`
`based devices, including handheld computing devices, like laptop computers and
`
`PDAs. Ex-1003, ¶83. Although one goal of Robertson is to make buttons user-
`
`editable, a POSITA would understand that Robertson’s gesture activations applied
`
`to any device capable of receiving and processing gesture-based inputs, e.g., laptop
`
`computers, PDAs, and other small form-factor devices. Id.
`
`Robertson teaches activating a button’s functions through “pen-based
`
`gestural input[s]” in addition to “pressing” (e.g., tapping on a button) to execute
`
`the button’s function(s). Ex-1005, § 3.1. A POSITA would have known that pen-
`
`based input devices were a method of inputting commands to mobile handheld
`
`computer units, including touch-screen laptops and PDAs (e.g., Palm OS
`
`devices/Palm Pilot, IBM Simon, Apple Newton and MessagePad, Windows Pocket
`
`PC devices, and Nokia Symbian-based devices). Ex-1003, ¶84.
`
`A POSITA would understand that Robertson’s computer unit stores
`
`computer program code on a non-transitory computer readable medium for
`
`13
`
`

`

`implementing the user interface because without computer program code or a
`
`IPR2021-01041
`U.S. Patent No. 8,095,879
`
`
`storage medium, the computer would not function. Ex-1003, ¶85. Maddalozzo
`
`confirms and renders obvious the non-transitory computer readable medium. Id.
`
`Maddalozzo discloses a computer system including “a graphical user interface that
`
`resides within a machine-readable medi[um] to direct the operation of [a] computer
`
`system.” Ex-1006, 3:32-34. Maddalozzo’s mobile computer includes various types
`
`of non-transitory machine-readable media, e.g., “RAM 34, ROM 36, … or optical
`
`disk,” for storing code. Ex-1006, 4:28-34. Dutta also confirms POSITAs knew to
`
`implement graphical user interfaces using “software residing in computer readable
`
`media” within a computer unit, e.g., PDA. Ex-1011, [0021].
`
`A POSITA would have found it obvious to store Robertson’s user interface
`
`computer program code in a non-transitory computer readable medium of the
`
`laptop, PDA, or other handheld computing device to provide a unitary system.
`
`Ex-1003, ¶86.
`
`Robertson discloses a computer unit having a user interface for “pen-based
`
`gestural input[s],” but does not specify the type of computer unit. See generally
`
`Ex-1005; Ex-1003, ¶87. Robertson’s user interface using “pen-based gestural
`
`input[s]” suggests that the device could be a mobile handheld computing unit
`
`(Ex-1005, § 3.1), and Maddalozzo renders obvious implementing Robertson’s
`
`teachings in a mobile handheld computer unit, e.g., a laptop or PDA. Ex-1003, ¶87.
`
`14
`
`

`

`Maddalozzo discloses different mobile handheld computer devices and
`
`IPR2021-01041
`U.S. Patent No. 8,095,879
`
`
`explains that touch-screen technology was commonly used in mobile handheld
`
`computer units, like laptops, notebook computers, and calculator-size mini-
`
`portable computers. Ex-1006, 1:12-35, 2:1-22; Ex-1003, ¶88.
`
`A POSITA would have found it obvious to implement Robertson’s
`
`XButtons (or other gesture-activated buttons) in mobile handheld computer units to
`
`enhance the user experience through more capable buttons supporting gesture
`
`activations. Ex-1003, ¶89. POSITAs knew handheld computers could run X
`
`protocol and/or Unix using touch screens. Ex-1010 (Musatov), [0034], [0038],
`
`[0155]; Ex-1006, 4:32-40; Ex-1003, ¶89. A POSITA would also have been
`
`motivated to use Robertson’s gesture-based user interface on mobile computing
`
`devices with small touch screens and without external keyboards to enhance the
`
`functionality of the devices and improve user experience. Ex-1003, ¶89.
`
`i. Motivation
`Robertson and Maddalozzo are directed to touch-based user interfaces.
`
`Ex-1005, Abstract, §§ 3-3.1; Ex-1006, 1:13-16, 6:13-19; Ex-1003, ¶90. A POSITA
`
`would understand that Robertson’s teachings apply to many types of gesture-based
`
`devices. Ex-1003, ¶90. Maddalozzo’s mobile handheld devices, e.g., laptop or
`
`PDA, work with “[a]ny suitable operating system and associated graphical user
`
`interface,” including Robertson’s “Unix” operating system or Lisp-based
`
`15
`
`

`

`commands.2 Ex-1006, 4:32-38; Ex-1005, Abstract, §§ 1.2, 2.1, 3-3.2; Ex-1003,
`
`IPR2021-01041
`U.S. Patent No. 8,095,879
`
`
`¶90.
`
`A POSITA would have been motivated to implement Robertson’s gesture-
`
`based operations in mobile handheld computer units (e.g., laptop, PDA-type
`
`device) having non-transitory computer readable media to provide a fully
`
`integrated handheld device that stores and executes user interface program code to
`
`eliminate any need for external computing components and to provide a simple,
`
`configurable multi-function user interface. Ex-1003, ¶92.
`
`“Touch screen technology [was] increasingly being implemented … with
`
`portable computers,” and the “increasingly portable” nature of such devices would
`
`have motivated a POSITA to implement Robertson’s gesture-based function
`
`activation to keep the form factor small, while still allowing for convenient user
`
`interfaces. Ex-1006, 1:28-33; Ex-1003, ¶¶91-93.
`
`
`2 Robertson’s teachings apply to other command languages. Ex-1005, Abstract,
`
`§§ 1.2, 2.1, 3-3.2; Ex-1003, ¶90.
`
`16
`
`

`

`A POSITA would have been motivated to implement Robertson’s gesture-
`
`IPR2021-01041
`U.S. Patent No. 8,095,879
`
`
`based operations on mobile computing devices, e.g., mobile laptop computers3 or
`
`PDA-style devices, to implement gesture-based action buttons using “pen-based
`
`gestural input[s]” common for such handheld devices and more convenient to
`
`operate on smaller screens. Ex-1003, ¶93; Ex-1005, § 3.1. A POSITA would have
`
`been motivated to implement gesture-based interfaces on mobile computing
`
`devices because consumers desired portability and gestures were more convenient
`
`for such devices, often lacking external mice and keyboards. Ex-1006, 1:28-60;
`
`Ex-1003, ¶91.
`
`Robertson’s teachings would have been implemented on X-based handheld
`
`devices (e.g., laptop computers, PDAs) to take advantage of operations the “Unix
`
`Shell” command language provides to design gesture-based buttons that are more
`
`useful and convenient for users. Ex-1005, Abstract, §§ 1.2, 2.1, 3.2, 5; Ex-1003,
`
`¶93. A POSITA would also have been motivated to implement Robertson’s
`
`teachings on other mobile devices (non-X-based, non-Unix) that use well-known
`
`
`3 The ’879 patent explains that “[m]obile handheld computers” have “various
`
`embodiments,” including the “personal digital assistant (PDA)” and “laptop
`
`computer, which is getting smaller and … competing in size with the PDA’s.”
`
`Ex-1001, 1:24-33.
`
`17
`
`

`

`gesture-based inputs to provide convenient execution of commands and functions
`
`IPR2021-01041
`U.S. Patent No. 8,095,879
`
`
`using the limited screen space available. Ex-1003, ¶93; see Ex-1009 (Tarpenning);
`
`Ex-1028 (Pisutha-Arnond); Ex-1029 (Schwarzer); Ex-1012 (Ausems); Ex-1008
`
`(Bedford-Roberts); Ex-1007 (Vayda (stylus)).
`
`ii.
`Expected success
`Implementing pen-based gesture inputs on touch-screen mobile devices was
`
`well-known to POSITAs. Ex-1003, ¶95; see Ex-1009 (finger/stylus inputs to
`
`perform touch-screen gestures); Ex-1010 (mobile handheld computer with touch
`
`screen implementing X-protocol); Ex-1011 (PDA with touch-screen user
`
`interface); Ex-1012; Ex-1013; Ex-1014; Ex-1015. The ’879 patent acknowledges
`
`laptop computers, PDAs, and mobile phones having touch-screen gestural inputs
`
`were previously known. Ex-1001, 1:24-33.
`
`A POSITA would have known how to implement Robertson’s touch-based
`
`gestures on mobile handheld devices because Robertson teaches principles for
`
`designing touch-based gestures applicable to various systems using touch-screen
`
`inputs and gestures. Ex-1003, ¶96. Robertson describes using multiple command
`
`languag