Case: IPR2015-00476
`Patent: 7,218,313
`
`
`UNITED STATES PATENT AND TRADEMARK OFFICE
`_________________________
`
`BEFORE THE PATENT TRIAL AND APPEAL BOARD
`________________________
`
`
`
`SONY COMPUTER ENTERTAINMENT AMERICA LLC
`Petitioner
`
`v.
`
`APLIX IP HOLDINGS CORPORATION
`Patent Owner
`
`________________________
`
`
`
`Case No. IPR2015-00476
`
`Patent No. 7,218,313
`
`
`
`DECLARATION OF DR. KARON MACLEAN
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Mail stop PATENT BOARD
`Patent Trial and Appeal Board
`U.S. Patent & Trademark Office
`P.O. Box 1450
`Alexandra, VA 22313-145
`
`
`

`
`
`I.
`
`
`
`II.
`
`TABLE OF CONTENTS
`
`Page
`
`Background & Qualifications .................................................................................... 1
`
`A.
`
`B.
`
`C.
`
`D.
`
`E.
`
`Education background and career history .................................................... 1
`
`Research expertise ........................................................................................... 2
`
`Collaborations with Industry ......................................................................... 6
`
`Professional service and recognition ............................................................. 7
`
`Exhibits analyzed ............................................................................................. 8
`
`Legal Framework ...................................................................................................... 10
`
`A.
`
`B. Differences Between the Art and the Invention ....................................... 12
`
`Scope and Content of the Prior Art ............................................................ 11
`
`C.
`
`The Level of Skill in the Art ......................................................................... 13
`
`D. Objective Indicia............................................................................................ 13
`
`
`III. Opinion ...................................................................................................................... 14
`
`Background of the Technology ................................................................... 14
`
`
`A.
`
`
`
`1.
`
`2.
`
`Short history of handheld computing devices up to 2003 ............ 14
`
`Short History of touchpads and use on computers generally and
`also on handheld devices up to 2003 ............................................... 15
`
`3. Designing Handheld Devices ........................................................... 18
`
`Summary of the ’313 Patent ......................................................................... 20
`
`Person of ordinary skill in the art ................................................................ 23
`
`
`ii
`
`B.
`
`C.
`
`
`
`

`
`D.
`
`It is not obvious to combine Pallakoff and Liebenow with respect to
`claims 21-24, 26, 52-56 and 58..................................................................... 24
`
`
`
`
`
`
`
`
`
`
`
`
`
`1.
`
`2.
`
`3.
`
`4.
`
`5.
`
`6.
`
`7.
`
`Pallakoff and Liebenow represent fundamentally different design
`approaches, which do not work in combination ........................... 26
`
`Pallakoff teaches modifier buttons on the side, and teaches away
`from modifier buttons on the back, making implementation with
`Liebenow’s rear touchpad incompatible ......................................... 28
`
`Liebenow does not teach use of a rear surface touch panel for
`modification of front-surface input elements or their functions,
`nor does Liebenow’s touch panel implementation support such a
`use. Instead, Liebenow explicitly places modifier buttons on the
`front (or side) .................................................................................... 32
`
`Liebenow’s invention of rear-surface touch panel input, intended
`to facilitate typing on a rear surface while finger locations are
`displayed on the front display, is not functionally compatible
`(using methods taught by Liebenow) with front-surface key
`function modification. It would need substantial modification to
`be combined ....................................................................................... 33
`
`Pallakoff requires simultaneous activation of a combination of
`side-buttons, but multiple-touch sensing is not taught or available
`in Liebenow’s invention .................................................................... 34
`
`It is impractical to replace Pallakoff’s side-located modifier
`buttons with Liebenow’s back-surface touch pad due to user
`feedback needs ................................................................................... 38
`
`The proposed benefit of combination proposed by the 476
` Petition is of limited viability and is incompatible with
`Pallakoff .............................................................................................. 41
`
`
`E. Hedberg is not analogous to the ’313 patent ............................................. 45
`
`IV. Conclusion ........................................................................................................... 47
`
`
`
`iii
`
`

`
`I. Background & Qualifications
`
`
`1.
`
`I have summarized in this section my educational background, career
`
`history, and other relevant qualifications. I have also attached a current version of my
`
`Curriculum Vitae as Ex. 2008.
`
`A. Educational background and career history
`
`2.
`
`I am presently a Full Professor at the University of British Columbia,
`
`with a regular appointment in Computer Science in the Faculty of Science, and a
`
`courtesy appointment in Mechanical Engineering in the Faculty of Applied Science. I
`
`have recently been a Visiting Professor at the University of Colorado (Boulder,
`
`Colorado, USA) and at the University of Canterbury (Christchurch, NZ).
`
`3.
`
`In 1986 I received a B.S. degree in Mechanical Engineering and
`
`Biological Sciences from Stanford University. In 1988 I received a M.S. in Mechanical
`
`Engineering from Massachusetts of Technology, and 1996 a Ph.D. in Mechanical
`
`Engineering from Massachusetts of Technology.
`
`4.
`
`From 1989 to 1991 I worked as a project engineer at the University of
`
`Utah’s Center for Engineering Design in Salt Lake City, UT. From 1996 to 2000 I was
`
`a Member of Research Staff and Project Lead at Interval Research Corporation in
`
`Palo Alto, CA.
`
`
`
`1
`
`

`
`B. Research expertise
`
`5.
`
`The majority of my research relates to different aspects of the design of
`
`physical interfaces for human use. These draw on disciplines of Human Computer
`
`Interaction (HCI), robotics and mechatronics, human biomechanics, psychophysics
`
`and cognition, and design practices more broadly. As a sample, my research includes
`
`design of interaction techniques for handheld devices, invention of flexible sensors
`
`for touch sensing and machine-learning recognition of human gestural touch,
`
`prototyping tools for haptic designers, psychophysically-based tools for creating sets
`
`of “haptic icons” (meaningful signals rendered by tactile and force-feedback
`
`actuators), and design and deployment of emotional touch in therapeutic applications
`
`of touch-based affective robots. Other research interests include human-robot
`
`interaction (HRI), accelerometer based motion recognition and interactive motion
`
`guidance for users, and mobile tools for systematizing health issue investigation.
`
`6. With my students and postdocs, I have co-authored over 100 peer-
`
`reviewed publications in these areas. In the fields of HCI and HRI, top-tier (peer-
`
`reviewed, 20-25% acceptance) conferences are a primary mode of publication. My
`
`team has received seven “Best Paper” awards and two additional runners-up in the
`
`last 10 years at such conferences, including the ACM Conference on Human Factors
`
`in Computing Systems (CHI), ACM/IEEE Human Robot Interaction (HRI), IEEE
`
`HAPTICS and ACM ICMI (Int’l Conference on Multimodal Interaction).
`
`
`
`2
`
`

`
`7.
`
`I am a co-inventor on four patents, filed between 1999 and 2001 and
`
`listed in my CV, which address novel physical interaction design concepts arising
`
`from research of the team I led at Interval Research in Palo Alto While they rely on
`
`creative and sometimes sophisticated disposition of technology elements that were
`
`newly available at the time, their primary contributions were in how to make
`
`technology work in a physical way for real problems that people had or foreseeably
`
`would have – for example, in controlling new forms of digital media, and the
`
`emerging challenges of mobile interactions.
`
`8.
`
`I have graduated 33 graduate students and supervised 6 postdocs and 52
`
`directed undergraduate research projects since 2000. Many of these individuals are
`
`presently employed in influential human-computer interaction roles in prominent
`
`companies including Apple, RIM, Microsoft, Google, Phillips, Canada’s National
`
`Research Council, Immersion Corp., Dolby, and the National Basketball Association
`
`as well as numerous smaller startups and other ventures. Two initiated a startup based
`
`on the research they did with me (Haptok Inc.).
`
`9.
`
`I currently am solely or jointly responsible for CAN$0.7M in research
`
`funding (FY 2014 funding at UBC and elsewhere) and for CAN$7.8M in grants
`
`overall since 2000 from (for example) Canada’s NSERC, CIHR and CFI funding
`
`agencies, the USA’s NSF and numerous private companies including Nokia, Nissan,
`
`Immersion, and General Motors. The majority of this funding involves multi-
`
`disciplinary and multi-institutional projects.
`
`3
`
`
`
`

`
`10. My work in human interface systems has its roots in an undergraduate
`
`research job at Stanford/VA Rehabilitation Research & Development Center, where
`
`in the early 1980’s I worked with Felix Zajac and his group on modeling the human
`
`musculoskeletal system. I continued in bioengineering with my Master’s research in
`
`MIT’s Mechanical Engineering Design Division, and a rehabilitation group creating
`
`computer-controlled prosthetics and robotics. My own research in functional
`
`neuromuscular stimulation was ultimately directed at restoring function to spinal-cord
`
`injured humans; with my supervisor William Durfee I characterized motor responses
`
`to neural stimulation, and devised a system by which to control motor activation. I
`
`then worked for two years at the University of Utah’s Center for Engineering Design,
`
`a world leader in complex human-controlled robotics, on the design and digital
`
`controls for high-degree of freedom, teleoperated, force-feedback robots for undersea
`
`and animatronic applications.
`
`11.
`
`For my MIT PhD (1990-1996) I developed and psychophysically
`
`characterized one of the very first of what later came to be known as haptic interfaces.
`
`At a time we knew absolutely nothing of how humans would perceive virtual force
`
`displays or how to create high fidelity haptic representations of real environments. I
`
`studied this by simulating and haptically rendering linear pushbutton switch profiles
`
`with high performance motors and examining human responses to these displays
`
`relative to traditional switches. MIT at this time was a haptics breeding-ground,
`
`spawning the majority of early haptics researchers and entrepreneurs internationally.
`
`
`
`4
`
`

`
`For example, my fellow students and friends included Thomas Massie, inventor of the
`
`Phantom haptic device; Margaret Minsky, credited with publication of a first haptic
`
`application; Ed Colgate, who later became the first Editor-in-Chief of the new IEEE
`
`Transactions on Haptics, and also chaired the first haptics conference in 1992; and
`
`Elaine Chen, a haptic inventor.
`
`12.
`
`I took up a postdoc and then researcher / team leader roles at Interval
`
`Research Corporation (1996-2000), a Palo Alto technology think-tank funded by Paul
`
`Allen, where I led a diverse team of designers and engineers in developing novel
`
`haptic interaction techniques and technology, predominantly for handheld interaction
`
`with physical devices. This was a period of frenetic development in user interface
`
`technology, much of it centered in Silicon Valley, and Interval had a front-row seat to
`
`observe and participate in it.
`
`13. At UBC, I am now primarily engaged on three threads of work: physical
`
`and generally handheld interface design including using these devices for motion
`
`sensing and guidance feedback, with a special focus on attentional processes; affective
`
`human-robot interaction; and mobile health applications. My background, spanning
`
`robot controls, bioengineering /psychophysics and human computer interaction
`
`(HCI) design, is somewhat unusual and I have exploited this to bridge my various
`
`communities. At present, I am leading a group of about ten students and postdocs at
`
`UBC from a mix of computational, engineering and arts backgrounds, working on
`
`interaction techniques and sensing technology for mobile and HRI contexts.
`
`
`
`5
`
`

`
`14.
`
`In summary, my research has been largely focused on physical human
`
`interfaces and human neuro, musculoskeletal and cognitive systems since the early
`
`1980’s, and I have been actively engaged in communities at the forefront of
`
`developments in these areas throughout my career.
`
`C.
`
`Collaborations with Industry
`
`
`
`15. As an HCI designer, it has always been useful to align with end-users
`
`and industries who inform me of human problems and industry needs. I will mention
`
`a few. Immersion Corp has been a primary holder of haptic intellectual property since
`
`the late 1990’s, whom I consulted for in 2000, and have received funding from and
`
`sent students to for employment ever since. I have worked closely with two
`
`automotive companies, Nissan on cockpit controls and General Motors on human-
`
`robot interaction for collaborative automation. I have given invited talks at Apple
`
`regarding my interaction design techniques, and three of my graduates work there
`
`now on the Apple Watch project. At RIM, one of my graduates spearheaded the push
`
`to put haptics onto the Blackberry in the late 2000’s. I am on the advisory board of
`
`Tangible Haptics, a Northwestern startup by Edward Colgate commercializing
`
`variable-friction surface haptic feedback, a concept which he invented; my students
`
`and I developed interaction techniques for it which then won a Best of CHI award. I
`
`am on the advisory board of the EU research consortium WEARHAP. I have also
`
`
`
`6
`
`

`
`worked with numerous small companies over the years in collaborative and advisory
`
`capacities.
`
`D.
`
`16.
`
`Professional service and recognition
`
`I have co-chaired major academic conferences, most recently the 2010
`
`and 2012 IEEE Haptics Symposium, and served in 2014 as Area Chair (Interaction
`
`Techniques and Devices) for the largest HCI conference, ACM CHI. I currently serve
`
`on the editorial board of the Int’l Journal of Human-Computer Studies (IJHCS), and
`
`was a founding associate editor of the IEEE Transactions on Haptics, whose 2008
`
`creation I orchestrated. I have served on numerous program committees in HCI,
`
`robotics, haptics and mobile interaction. In 2014-2015, I was on the Awards
`
`committees for all three major haptics conferences (Eurohaptics 2014, Haptics
`
`Symposium 2014 and Worldhaptics 2015).
`
`17.
`
`I am a Senior Member of the Association for Computing Machinery
`
`(ACM), and a Senior Member of the Institute for Electrical and Electronics
`
`Engineering (IEEE), active in the Computer Society and the Robotics and
`
`Automation Society.
`
`18.
`
`I have been the prime architect of UBC’s HCI curriculum, creating it in
`
`2000 and twice re-vamping a program that has reappeared in many other universities;
`
`it is now among the largest in North America. I have also taught numerous summer
`
`
`
`7
`
`

`
`schools and workshops with the aim of bringing HCI practices to engineering haptics
`
`audiences, and vice versa.
`
`19.
`
`I am currently leading a major pan-university initiative to create an HCI
`
`Institute at UBC, including proposals to secure major funding for it.
`
`20.
`
`I was the recipient of the 2001 Peter Wall Early Career Scholar, the 2006
`
`Izzak Walton Killam Memorial Faculty Research Fellowship, the 2008 Charles A
`
`McDowell Award (the top research prize at UBC), and received a 2013 NSERC
`
`Accelerator Grant, awarded to top science researchers nationally in Canada.
`
`E. Exhibits analyzed
`
`
`
`
`
`
`Exhibit 1005
`
`Petition for Inter Partes Review of U.S. Patent No. 7,218,313
`Decision Instituting Inter Partes Review of U.S. Patent No. 7,218,313
`(Paper 11)
`Exhibit 1001 U.S. Patent No. 7,218,313 to Beth Marcus et al.
`Exhibit 1004 U.S. Patent Publication No. 2002/0163504 to Pallakoff
`U.S. Patent Application Publication No. 2002/0118175 to Liebenow
`et al.
`Exhibit 1007 U.S. Patent No. 6,469,691 to Armstrong
`Exhibit 1008
`International Publication No. WO1999/18495 to Hedberg
`Exhibit 1009 Expert Declaration of Dr. Gregory F. Welch
`Amended Complaint in Aplix IP Holdings Corporation v. Sony
`Computer Entertainment Inc. and Sony Computer Entertainment
`America LLC, Case No. 1:14-cv-12745
`Allen, J. P., Handheld Computing Predictions: What Went Wrong?,
`Proceedings of the 1st International Symposium on Handheld and
`Ubiquitous Computing, Karlsruhe, Germany: Springer-Verlag, 1999,
`pp. 117-123
`Wikipedia entry on “List of Blackberry products” at
`https://en.wikipedia.org/wiki/List_of_BlackBerry_products,
`accessed 8/3/2015
`Keyboard image at http://www.computerhistory.org/
`collections/catalog/102642008, accessed 8/2/2015
`
`Exhibit 2005
`
`Exhibit 2011
`
`Exhibit 2012
`
`Exhibit 2013
`
`
`
`8
`
`

`
`Exhibit 2015
`
`Exhibit 2016
`
`Exhibit 2017
`
`Exhibit 2018
`
`Exhibit 2019
`
`Exhibit 2020
`
`Exhibit 2021
`
`Exhibit 2022
`
`Exhibit 2023
`
`Exhibit 2014 Patent US 5,305,017
`Wikipedia entry on “Touchpad” at
`https://en.wikipedia.org/wiki/Touchpad, accessed 8/1/2015
`Buxton, W., Multi-Touch Systems that I Have Known and Loved, at
`www.billbuxton.com/multitouchOverview.html, accessed 8/2/2015
`Walker, G., A Review of Technologies for Sensing Contact Location on the
`Surface of a Display, Journal of the Society for Information Display,
`vol. 20:8, pp. 413-440, 2012
`Wikipedia entry on “IBM Simon” at https://en.wikipedia.org/wiki/
`IBM_Simon, accessed 8/2/2015
`Wikipedia entry on “Casio PB 1000” at
`https://en.wikipedia.org/wiki/Casio_PB-1000, accessed 8/5/2015
`Blickenstorfer, C., NeoNode N1, Can a unique interface put this compelling
`smart phone on the map? At
`http://pencomputing.com/WinCE/neonode-n1-review.html,
`accessed 8/2/2015
`Wikipedia entry on “List of iPod models” at
`https://en.wikipedia.org/wiki/List_of_iPod_models, accessed
`8/2/2015
`Barker, M., Microsoft Teams with Interlink Electronics for Xbox
`Controllers, at www.Gamasutra.com, accessed 8/2/2015
`Hinckley, K., Pierce, J., Sinclair, M., and Horvitz, E., Sensing
`Techniques for Mobile Interaction, Proceedings of the 13th Annual ACM
`Symposium on User Interface Software and Technology, San Diego,
`California, USA: ACM, 2000, pp. 91-100
`Microchip AR1000 Series Resistive Touch Screen Controller Data
`Sheet (2009-2012) at
`http://ww1.microchip.com/downloads/en/DeviceDoc/41393B.pdf
`Elo Touch Solutions: Tyco Electronics Introduces the Industry's First Multi-
`Touch Gestures Technology for Analog Resistive Touchscreens, December 4,
`2008
`True Multi Touch on Analog Resistive at www.haptyc.com, accessed
`8/26/15
`Wikipedia entry on "Cirque Corporation" at
`https://en.wikipedia.org/wiki/Cirque_Corporation, accessed
`8/26/15
`Wikipedia entry on "iPod Classic Second Generation" at
`https://en.wikipedia.org/wiki/IPod_Classic#2nd_generation,
`accessed 8/26/15
`
`Exhibit 2024
`
`Exhibit 2025
`
`Exhibit 2026
`
`Exhibit 2027
`
`Exhibit 2028
`
`
`
`9
`
`

`
`Exhibit 2029
`
`Exhibit 2031
`
`Exhibit 2032
`
`Wikipedia entry on "iPod click wheel" at
`https://en.wikipedia.org/wiki/IPod_click_wheel, accessed 8/26/15
`Exhibit 2030 PCMag.com review: Fingerworks iGesture Pad, February 3, 2004
`Wikipedia entry on “Camera phone” at
`https://en.wikipedia.org/wiki/Camera_phone, accessed 8/3/2015
`Partridge, K., Chatterjee, S., Sazawal, V., Borriello, G., and Want, R.,
`Tilttype: Accelerometer-Supported Text Entry for Very Small
`Devices,” in Proceedings of the 15th annual ACM symposium on
`User interface software and technology. Paris, France: ACM, 2002,
`pp. 201-204
`Wigdor, D. and Balakrishnan, R., Tilttext: Using Tilt for Text Input
`to Mobile Phones” in Proceedings of the 16th annual ACM
`symposium on User interface software and technology. Vancouver,
`Canada: ACM, 2003, pp. 81-90
`Buxton, W., Hill, R., and Rowley, P., Issues and techniques in touch-
`sensitive tablet input, SIGGRAPH Computer Graphics, vol. 19:3, pp.
`215-224, July 1985.
`Wikipedia entry on “Touchscreen” at
`https://en.wikipedia.org/wiki/Touchscreen, accessed 8/1/2015
`
`Exhibit 2033
`
`Exhibit 2034
`
`Exhibit 2035
`
`II. Legal Framework
`
`
`21. Counsel has informed me that a patent claim is obvious if the
`
`differences between it and the prior art are such that it would have been obvious at
`
`the time the invention was made to a person having ordinary skill in the relevant art.
`
`To determine obviousness I must consider the scope and content of the prior art, the
`
`differences between the prior art and the claimed invention, the level of ordinary skill
`
`in the field of the invention, and any relevant objective considerations of
`
`nonobviousness. I have taken each of these factors into account in my analysis
`
`below.
`
`
`
`10
`
`

`
`A.
`
`22.
`
`Scope and Content of the Prior Art
`
`I am informed that prior art for obviousness purposes is limited to art
`
`that is analogous, meaning that it is from the same “field of endeavor” as the patented
`
`invention, or is “reasonably pertinent” to the particular problem with which the
`
`inventor was involved.
`
`23.
`
`I understand that two pieces of art that relate to the same general subject
`
`matter are not necessarily in the same field of endeavor. Similarity in the structure
`
`and function of the invention and the prior art is indicative that the prior art is within
`
`the same field of endeavor, and differences in structure and function suggest the
`
`contrary.
`
`24.
`
`I am further informed that another way that art can be considered
`
`analogous is if it is “reasonably pertinent” to the problem the inventor was trying to
`
`solve. A reasonably pertinent reference is one that logically would have commended
`
`itself to an inventor's attention in considering the problem the inventor was trying to
`
`solve.
`
`25. Even when art is analogous, however, I understand that it is also
`
`important to consider the differences between the art and the invention as claimed
`
`and whether a person of ordinary skill would have had a motivation to combine the
`
`references in the manner of the claim.
`
`
`
`11
`
`

`
`B. Differences Between the Art and the Invention
`
`26. Most inventions are an assembly of pre-existing technology. It is
`
`therefore not proper to say that an invention was obvious merely because it is a
`
`recombination of the prior art. Such an invention might be perfectly obvious in
`
`hindsight, but that is not the test for validity purposes. Rather, the test is it would
`
`have been obvious at the time of the invention to a person of ordinary skill who did
`
`not have the benefit of the inventor’s teachings. Moreover, sometimes it is the
`
`recognition of the problem to be solved, rather than the claimed solution, that was
`
`non-obvious.
`
`27. A reason to combine references might come from one of various
`
`sources. For instance, there might be a specific marketplace pressure or existing
`
`design demand that would motivate a person of ordinary skill in the art to make the
`
`combination. It does not matter what the source of the reason to combine is. What
`
`is important is clearly to identify a reason that a person of ordinary skill at the relevant
`
`time would have appreciated. Otherwise, the combination is obvious only in
`
`hindsight.
`
`28.
`
`I understand rather that there has to be some set of facts apart from the
`
`teachings of the patent itself to provide the glue to combine the art in the manner of
`
`the claims. One cannot simply put together pieces of art like a jigsaw puzzle using the
`
`patent as the picture on the puzzle box lid. That is impermissible hindsight.
`
`
`
`12
`
`

`
`29.
`
`I understand that it can also be pertinent whether there was a reason in
`
`the art not to make the claimed combination. It could be that a person of ordinary
`
`skill would have considered the combination difficult to make or that the prior art
`
`“teaches away” from the claimed invention.
`
`C. The Level of Skill in the Art
`
`30.
`
`I am informed that obviousness is to be judged from the perspective of
`
`a person of ordinary skill, not the perspective of an expert. Even if the invention
`
`would have been obvious to the brightest minds in the field, that does not render it
`
`unpatentable. I understand that highly educated and trained individuals know more,
`
`and thus more inventions are obvious to them. On the other hand, individuals with
`
`little education or training would have more difficulty in making connections and
`
`therefore fewer inventions would be obvious to them.
`
`D. Objective Indicia
`
`31.
`
`Finally, I am informed that one should take into account any objective
`
`evidence that suggest that the invention may have been obvious or nonobvious. For
`
`instance, if many different inventors came up with the same combination around the
`
`same time, that may suggest that it was an obvious combination to make. On the
`
`other hand, if the elements of the invention were around for a long time without
`
`anyone making the combination, that is an objective fact suggesting that the
`
`
`
`13
`
`

`
`combination may not have been so obvious. I understand that there is no exhaustive
`
`list of objective factors.
`
`III. Opinion
`
`
`A.
`
`Background of the Technology
`
`1.
`
`Short history of handheld computing devices up to 2003
`
`32. A number of technological and commercial developments were
`
`influential or significant with respect to handheld computing device interfaces and
`
`usage. While hand-sized, portable computers have existed in some form since at least
`
`the late 1980s, Palm Computing’s 1996 release of the Palm Pilot inaugurated the
`
`“PDA” (personal digital assistant) era and the idea of a handheld device that could
`
`work as a complement to a personal computer (Ex. 2011, Allen, at p. 5). Pen-based
`
`user input dominated handheld devices around this time.
`
`33. The decade from 1995 through the mid-2000’s saw enormous
`
`exploration and innovation in handheld input modalities, in part because the purpose
`
`and potential of handheld computing was far from resolved: what would they be good
`
`for, who would use them, and how would these devices fit into the growing
`
`ecosystem of other computational instruments at users’ disposal?
`
`34. Devices during this period tended to be specialized. Cell phones, PDAs,
`
`music players, cameras, game consoles and remote controls had and still have the
`
`liberty to focus on working well for one thing. Devices were marketed as
`
`
`
`14
`
`

`
`“smartphones” as early as 2000 (e.g. the Ericsson R380, Palm Kyocera, and Treo 90),
`
`but combined a small number of relatively compatible functions, e.g., a PDA plus a
`
`mobile phone. Blackberry released the first “convergent” 6000 series smartphone in
`
`2003 (Ex. 2012 at p. 1), with push email, mobile telephony, text, internet fax and web
`
`browsing.
`
`35. However, it was not until the 2007 iPhone and the 2008 Android HTC
`
`Dream that the “Swiss Army knife” smartphone truly emerged. From that point,
`
`smartphone uses spanned business, music and gaming as well as telephony and
`
`calendaring, and third party “apps” that it seemed could do just about everything else.
`
`After this point, smartphones lost most differentiation in size, shape and input
`
`modalities, with general convergence onto a thin slab with touchscreen controls.
`
`36.
`
`Innovation driven by use cases and enabling of helpful interaction
`
`techniques: Academic literature around this time illustrates considerable attention to
`
`how users might provide input to handheld devices, and also how users might use
`
`these devices to interact with other aspects of their environment. Their reports
`
`address both purpose (potential tasks, often newly envisioned ones) and interaction
`
`techniques (often enabled by new sensing modalities).
`
`2.
`
`Short history of touchpads, and their use on computers
`generally and on handheld devices up to 2003
`
`37. Touch sensors have proved attractive computer input elements in
`
`contexts where space is limited. The first appearance of touchpads as a commercially
`
`
`
`15
`
`

`
`available computer input modality is commonly cited as an inclusion on the Apollo
`
`computer’s keyboard (Exs. 2013, 2015 at p. 2) in 1982, although there were earlier
`
`commercial uses. It spread more widely as GlidePoint technology, invented in the
`
`1980’s (U.S. Patent No. 5,305,017, Ex. 2014), developed by Cirque Corporation
`
`(1991) and incorporated into early Apple products (1994 PowerBook series). Several
`
`US companies, including Logitech, Synaptics and MicroTouch Systems, released their
`
`own versions in the mid-1990’s; descendants of these early technologies have
`
`continued to dominate consumer electronics ever since (many sources, including a
`
`reasonable summary on Wikipedia, Ex. 2015). Multitouch variants and interaction
`
`techniques that exploited them attained substantial visibility by the mid 1980’s (e.g.,
`
`Ex. 2012, Buxton, at p. 1), and appeared in some commercial settings, e.g. air traffic
`
`control terminals even earlier, in the 1960’s (Ex. 2017, Walker, p. 413).
`
`38. Leading technologies: The earliest technology families utilized as
`
`computer input elements are capacitive (allows sensing of presence and absolute
`
`position of a bare finger based on its electrical properties) and resistive (measures the
`
`resistance drop when two electrically striped sheets are pressed together, thus
`
`requiring some touch pressure). These have different strengths and weaknesses; for
`
`example, resistive touchpads (can have higher resolution, require nonzero pressure)
`
`work well with stylus input, whereas capacitive touchpads and touchscreens require a
`
`bare finger but accommodate light touch. The capacitive touch screen we see in
`
`smartphones today is comprised of multiple layers of glass with fused conductive
`
`
`
`16
`
`

`
`coating, but other construction methods and materials support sensing on curved and
`
`even flexible surfaces. Many variants have been derived from both.
`
`39. Touch sensing on handhelds: IBM’s Simon Personal Communicator
`
`(1994), was touted as the first device to include telephone and PDA features, and also
`
`had a stylus touchscreen with handwriting input (Ex. 2018 at p. 1); but it was predated
`
`by non-phone devices such as the 1987 Casio pocket computer PB-1000 (Ex. 2019),
`
`which had 16 touch-sensitive keys (“LCD sensitive areas”) built into the screen in a
`
`4x4 grid. The 2004 Neonode N1 Mobile phone (announced in 2002) had an optical
`
`touch-screen, and was considered the first touchscreen to go beyond finger-tapping
`
`on buttons to make use of finger swipe-type gestures (Ex. 2020 at pp. 1-3). The
`
`iPhone, responsible for massively popularizing multitouch input in handhelds, was
`
`released in 2007.
`
`40. Of non-screen touch sensors, the best known in this period may be
`
`Apple’s iPod, first released in 2001 with a mechanical scroll wheel, and with a one-
`
`dimensional touch-sensitive wheel in 2002 (see e.g. Ex. 2021). The “click wheel”
`
`innovation where mechanical tactile switches were incorporated beneath the touch
`
`wheel did not appear until 2004.
`
`41. Another relevant player is Interlink Electronics (1984), a company which
`
`has produced wide varieties of flexible force-sensing resistors packaged in ways
`
`convenient for prototyping and manufacture, and initially most popular with
`
`electronic music instrument controllers. In the early 2000’s, these enabled inexpensive
`
`
`
`17
`
`

`
`analog sensing of point and 2-D position and pressure. Interlink participated in the
`
`Xbox console design (2001), providing six buttons with pressure sensitivity, e.g. for
`
`sensitive control of character movements (Ex. 2022).
`
`42. The incorporation of touch sensing on hand-sized surfaces to enable
`
`interactions in ways other than buttons or stylus movements on a flat touch-sensitive
`
`screen developed more slowly than gestural control. Ex. 2023, a paper related to
`
`sensing techniques for mobile interaction, describes a simple prototype capacitive
`
`sensor applied to a curved surface to indicate touch contact or non-contact over a
`
`larger surface (e.g., a mouse) without specific location information. This led others to
`
`envi