5/10/2018
`
`From touch displays to the Surface: A brief history of touchscreen technology I Ars Technica
`~ .
`·-:-:-:-:-:-:-:-:-:-:-:-:-·
`
`Historians generally consider the n,-~,t finger-driven touchscreen to have been invented by E.A.
`Johnson in 1965 at the Royal Radar Establishment in Malvern, United Kingdom. Johnson originally
`described his work in an article entitled '·Touch di~T!·J>'······J nove: ,npuvoutput de\1·:Cc· re;,-- c,:,:-npute,·~,"
`published in Electronics Letters. The piece featured a diagram describing a type of touchscreen
`mechanism that many smartphones use today-what we now know as capacitive touch. Two years
`later, Johnson further expounded on the technology with photographs and diagrams in "Touch
`Displays: A Programmed Man-Machine Interface," published in Ergonomics in 1967.
`
`A capacitive touchscreen panel uses an insulator, like glass,
`that is coated with a transparent conductor such as indium tin
`oxide (ITO). The "conductive" part is usually a human finger,
`which makes for a fine electrical conductor. Johnson's initial
`technology could only process one touch at a time, and what
`we'd describe today as "multitouch" was still somewhat a ways ...,
`<::
`away. The invention was also binary in its interpretation of
`""'
`......
`"-·
`i~
`touch-the interface registered contact or it didn't register
`contact. Pressure sensitivity would arrive much later.
`
`[k~~~~
`~~'d t,1,&1lr~,g:
`Ct>t,oo~~
`~~1:-~,~~~
`-&.~~""&i..-:.):: .. d,:.~
`
`~t"":-~~~h~
`~,S,(<>g
`
`t1~~88
`~,"ill
`
`How capacitive touch screens work.
`
`Even without the extra features, the early touch interface idea
`had some takers. Johnson's discovery was eventually adopted by air traffic controllers in the UK and
`remained in use until the late 1990s.
`
`1970s: Resistive touchscreens are invented
`
`Although capacitive touchscreens were designed first, they were eclipsed in the early years of touch
`by resistive touchscreens. American inventor Dr. G. Samuel Hurst developed resistive touchscreens
`almost accidentally. The Berea College Magazine fnr- alumni de~:,Jii:,ed it like this:
`
`https://arstechnica.com/gadgets/2013/04/from-touch-displays-to-the-surface-a-brief-history-of-touchscreen-technology/
`
`0004
`
`Verizon 1024 P4
`U.S. Patent No. 8,213,970
`
`Page 1432
`
`

`

`5/10/2018
`
`From touch displays to the Surface: A brief history of touchscreen technology I Ars Technica
`
`Hurst and the research team had been working at the University of Kentucky. The university tried to
`file a patent on his behalf to protect this accidental invention from duplication, but its scientific
`origins made it seem like it wasn't that applicable outside the laboratory.
`
`Hurst, however, had other ideas. "I thought it might be useful for other things," he said in the article.
`In 1970, after he returned to work at the Oak Ridge National Laboratory (ORNL), Hurst began an
`after-hours experiment. In his basement, Hurst and nine friends from various other areas of
`expertise set out to refine what had been accidentally invented. The group called its fledgling venture
`'T
`aphic.~:," and the team discovered that a touchscreen on a computer monitor made for an
`excellent method of interaction. All the screen needed was a conductive cover sheet to make contact
`with the sheet that contained the X- and Y-axis. Pressure on the cover sheet allowed voltage to flow
`between the X wires and the Y wires, which could be measured to indicate coordinates. This discovery
`helped found what we today refer to as resistive touch technology (because it responds purely to
`pressure rather than electrical conductivity, working with both a stylus and a finger).
`
`As a class of technology, resistive touchscreens tend to be very affordable to produce. Most devices
`and machines using this touch technology can be found in restaurants, factories, and hospitals
`because they are durable enough for these environments. Smartphone manufacturers have also
`used resistive touchscreens in the past, though their presence in the mobile space today tends to be
`confined to lower-end phones.
`
`,<:; ~,
`""®
`
`~~1111111111■
`
`A second-gen AccuTouch curved touchscreen from EloTouch.
`
`https://arstechnica.com/gadgets/2013/04/from-touch-displays-to-the-surface-a-brief-history-of-touchscreen-technology/
`
`0005
`
`Page 1433
`
`

`

`5/10/2018
`
`From touch displays to the Surface: A brief history of touchscreen technology I Ars Technica
`Elographics didn't confine itself just to resistive touch, though. The group eventually paten,.,,,d the first
`curved glass touch interface. The patent was titled "electrical sensor of plane coordinates" and it
`provided details on "an inexpensive electrical sensor of plane coordinates" that employed
`"juxtaposed sheets of conducting material having electrical equipotential lines." After this invention,
`Elographics was sold to "good folks in California" and became EloTouch Systems.
`
`By 1971, a number of different touch-capable machines had been introduced, though none were
`pressure sensitive. One of the most widely used touch-capable devices at the time was the University
`of Illinois's PLATO IV terminal-one of the tit~:t generalized computer assisted instruction systems.
`The PLATO IV eschewed capacitive or resistive touch in favor of an infrared system (we'll explain
`shortly). PLATO IV was the first touchscreen computer to be used in a classroom that allowed
`students to touch the screen to answer questions.
`
`The PLATO IV touchscreen terminal.
`
`Florence is a former reviews editor at Ars.
`
`.-,
`Page: 1 •",
`x~~: ~j Next······,'
`
`https://arstechnica.com/gadgets/2013/04/from-touch-displays-to-the-surface-a-brief-history-of-touchscreen-technology/
`
`0006
`
`Page 1434
`
`

`

`From touch displays In lne Surf am: A brief history of torn:hscreen !rn::hnoklgy I Afs Te,Jmirn
`
`1980s: The decade of touch
`
`In 1982, the first human-controlled multitour.h device was developed at the UmversIty of
`Toronto by Nimish Mehta. It wasn't so much a tour.hsueen as it was a tour.h-tablet. The Input
`Research Group at the university figured out that a frosted-glass panel with a camera behind it
`could detec! action as it rewgni;-ed the different "black spots" showing up on-:;creen. Bill
`Bu:<ton has pbyed a ht1ge role in the development !)f multitouch ter.hn!)logy {most notably with
`the Portfol10Wall, to be dismssed a bit later}, and he deemed Mehl:a's invention important
`enough w indude in his mformai t,,,,d!:',i! of computer input devices:
`
`Or1!!l m trne fir$t dieg111ms depicting
`mult!toud1 input.
`
`Shmtly thereafter, gestural inter<1cti<m was introduced by Myron Krueger, <1n American computer artist who developed an opti<<1I
`system that could track hand mO\'ements. Krueger introduced Video Place (later called Video Desk) in '!983, though he'd been
`working on the system since the late 19l0s. It used proJectors and video cameras to track hands, fmgers, and the people they
`belonged to. Unlike multitouch, it wasn't entirely aware of who or what was touching, though the software could react to different
`poses. TI1e display depic!ed what looked like ,;hadows in a simulated space.
`
`---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------.
`Myron Kreuger-Video Place-1989
`
`I
`
`rillps.//arster.lmir.a.rnm/gadgets/2013/04!from-tooch--disp!ays-b-lhe-surface-a-brief-histmy--0f-touchscreen--ter.hnology/2/
`
`I
`
`0007
`
`Page 1435
`
`

`

`51'!012018
`
`From tooth displays lo lne Surf aw: A brief history of torn:hscreen !rn::hnokigy I Afs Ter.hnirn
`
`:
`
`--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------·
`Bill Bunon !11tr1:1dw::es the l"ortfo!i1:1Wall ami detllli!s. some of lti. ab!l!ties.
`
`Though it wasn't technically touch-based-lt relied on "dwell tlme" before it would execute an action-Buxton :·eg,":~:,: it as one of
`lhe technologies that ''wmle the book' in terms of unencumbered .. , rich gestural interanion, The work was more than a decade
`ahead of il:s time and was hugely influential. yet not as acknowledged as it should be." Krueger also pioneered viri:uai reality arid
`interactive art later on in his career.
`
`Touchscreens began being heavily commerclalized at the beginning of the '1980s. HP {then still formally known as Hewlett-Packard)
`wssed ilS hat in wilh the ;-;;:,_ ·i SU in September of 1983. The rnmputer used MS-DOS and featured a 9-inch Sony CRT surrounded by
`infrared {IR) ernitt.m; and detectors that wuid sense where the user's finger came down on the stf'een. Th~, ,system ;:ost aboua
`$2,795, bul: it was 1101 immediatdy embraced because it had sonw usabilily issues. For iris!arn:e, poklng al: lhe sneeri would in !urn
`block other IR rays that rnuld tell the rnmputer where the finger was pointing. This ,,,,;;.:,te,: in what some called "Gorma Arm,"
`referring to muscle fatigue that came from a user sticking his or her hand out for so long.
`
`rillps://arsteclmica.cmn/gadgets/2013/04!from-tooch--disp!ays.b-tl!e.surface.a.brief-histoey . .af.touchscreen--technology/2/
`
`0008
`
`Page 1436
`
`

`

`5/10/2018
`
`From touch displays to the Surface: A brief history of touchscreen technology I Ars Technica
`
`::,·;urg,·, I The HP-150 featured MS-DOS and a 9-inch touchscreen Sony CRT.
`
`A year i-it;,,·. multitouch technology took a step forward when Bob Boie of Bell Labs developed the first transparent multitouch
`screen overlay. As Ars '. .. ,.,-,."'' last year:
`
`The discovery helped create the multitouch technology that we use today in tablets and smartphones.
`
`1990s: Touchscreens for everyone!
`
`IBM's Simon Personal Communicator: big handset, big screen, and a stylus for touch input.
`
`https://arstechnica.com/gadgets/2013/04/from-touch-displays-to-the-surface-a-brief-history-of-touchscreen-technology/2/
`
`0009
`
`Page 1437
`
`

`

`5/10/2018
`
`From touch displays to the Surface: A brief history of touchscreen technology I Ars Technica
`
`In 1993, IBM and BellSouth teamed up to launch the Simon Personal Communicator, one of the first cellphones with touchscreen
`technology. It f 0;;t,.:: ,,c paging capabilities, an e-mail and calendar application, an appointment schedule, an address book, a
`calculator, and a pen-based sketchpad. It also had a resistive touchscreen that required the use of a stylus to navigate through
`menusandtoinputda~.
`
`The original MessagePad 100.
`
`Apple also launched a touchscreen PDA device that year: the Newton PDA. Though the Newton platform had begun in 1987, the
`Message Pad was the first in the series of devices from Apple to use the platform. As Time ,~c,1 ;,;:, Apple's CEO at the time, John
`Sculley, actually coined the term "PDA'' (or "personal digital assistant"). Like IBM's Simon Personal Communicator, the MessagePad
`100 featured handwriting recognition software and was controlled with a stylus.
`
`Early reviews of the Message Pad focused on its useful features. Once it got into the hands of consumers, however, its
`shortcomings became more apparent. The handwriting recognition software didn't work too well, and the Newton didn't sell that
`many units. That didn't stop Apple, though; the company made the Newton for six more years, ending with the MP2000.
`
`https://arstechnica.com/gadgets/2013/04/from-touch-displays-to-the-surface-a-brief-history-of-touchscreen-technology/2/
`
`0010
`
`Page 1438
`
`

`

`5/10/2018
`
`From touch displays to the Surface: A brief history of touchscreen technology I Ars Technica
`
`The first Palm Pilot.
`
`Three years later, Palm Computing followed suit with its own PDA, dubbed the Pilot. It was the first of the company's many
`generations of personal digital assistants. Like the other touchscreen gadgets that preceded it, the Palm 1000 and Pilot 5000
`required the use of a stylus.
`
`Palm's PDA gadget had a bit more success than IBM and Apple's offerings. Its name soon became synonymous with the word
`"business," helped in part by the fact that its handwriting recognition software worked very well. Users used what Palm called
`"Graffiti" to input text, numbers, and other characters. It was simple to learn and mimicked how a person writes on a piece of
`paper. It was eventually implemented over to the Apple Newton platform.
`
`PDA-type devices didn't necessarily feature the finger-to-screen type of touch screens that we're used to today, but consumer
`adoption convinced the companies that there was enough interest in owning this type of device.
`
`Near the end of the decade, University of Delaware graduate student Wayne Westerman published a doctoral dissertation entitled
`"Hand Tracking, Finger Identification, and Chordic Manipulation on a Multi-Touch Surface." The paper iJ,,t-i::t,d the mechanisms
`behind what we know today as multitouch capacitive technology, which has gone on to become a staple feature in modern
`touchscreen-equipped devices.
`
`https://arstechnica.com/gadgets/2013/04/from-touch-displays-to-the-surface-a-brief-history-of-touchscreen-technology/2/
`
`The iGesture pad manufactured by FingerWorks.
`
`0011
`
`Page 1439
`
`

`

`5/10/2018
`
`From touch displays to the Surface: A brief history of touchscreen technology I Ars Technica
`
`Westerman and his faculty advisor, John Elias, eventually formed a company called FingerWorks. The group began producing a line
`of multitouch gesture-based products, including a gesture-based keyboard called the TouchStream. This helped those who were
`suffering from disabilities like repetitive strain injuries and other medical conditions. The iGesture Pad was also released that year,
`which allowed one-hand gesturing and maneuvering to control the screen. FingerWorks was eventually acquired by Apple in 2005,
`and many attribute technologies like the multitouch Trackpad or the iPhone's touchscreen to this acquisition.
`
`2000s and beyond
`
`With so many different technologies accumulating in the previous decades, the 2000s were the time for touchscreen technologies
`to really flourish. We won't cover too many specific devices here (more on those as this touchscreen series continues), but there
`were advancements during this decade that helped bring multitouch and gesture-based technology to the masses. The 2000s were
`also the era when touchscreens became the favorite tool for design collaboration.
`
`2001: AliaslWavefront's gesture-based Portfolio Wall
`
`As the new millennium approached, companies were pouring more resources into integrating touchscreen technology into their
`daily processes. 3D animators and
`were especially targeted with the advent of the PortfolioWall. This was a large-format
`touchscreen meant to be a dynamic version of the boards that design studios use to track projects. Though development started in
`1999, the PortfolioWall was unveiled at SIGGRAPH in 2001 and was produced in part by a joint collaboration between General
`Motors and the team at Alias I Wavefront. Buxton, who now serves as principal research at Microsoft Research, was the chief
`scientist on the project. "We're tearing down the wall and changing the way people effectively communicate in the workplace and
`do business," he ~akl back then. "PortfolioWall's gestural interface allows users to completely interact with a digital asset. Looking
`at images now easily become part of an everyday workflow."
`
`Portfolio Wall
`
`https://arstechnica.com/gadgets/2013/04/from-touch-displays-to-the-surface-a-brief-history-of-touchscreen-technology/2/
`
`0012
`
`Page 1440
`
`

`

`5/HJ/2018
`
`From touch disp!11ys In lne Surf am: A brief hi:otnry of torn:hscreen !rn::hnoklgy I Afs Te,Jmirn
`
`The PortfolioWall used a simple, 0asy-to-us0, gesture-based irnertac0- !tallowed u;;0rs to inspect and maneuver lmag0s,
`animations, and 3D files with just their fingers. It was also easy to scale images, fetch 3D models, and play back video. A later
`version added sketch and text annotation, the ability to launch third-party app!Ecations, and a Maya-based 3D viewmg tool to use
`panning, mtilting, zooming, and viewing for 30 models. Fur the most pilrt, the product was considered a digital rnrkboard for
`design--wntrk professions. it also (,Y.t a whopping $38,000 to get the whole 5et up in:s.tillled----$3,000 for the presenter ltsel! ilnd
`$35,000 for the ;;erver.
`
`The PmtfolioWall also addressed the fact that while traditional mediums like
`day models and full-size drawings were still important to the design process,
`they were :s.lowly being augmented by digitill tuol5. The device included add(cid:173)
`(ms that virtually emulated those tangible mediums and ,;erved as a
`presentation tool for designers to show off their work in pmgress.
`
`Another main draw of the PortfolioWall was its •awareness server," which
`helped fodlitate wllilb01ilteon ilCro:s.s a ne!work so that teams didn't have to
`be in th0 same mom to review a project. Teams could have multiple wall,; in
`different spaces and still rnllaborate remotely.
`
`The PortfolioWall was eventually laid to rest in 2008, but it was a prime
`exilmple of how gestures interncteng with the touchscreen rnuld help control
`an entire operating system,
`
`The PtirtfolioW~II ~!lowed dlffiign1era tti .li~p!~y fu!l-s;cele
`30modds..
`
`Page: ·1 2 3 Next-----,
`
`READER WMMENTS
`
`SHARE THIS STORY
`
`rillps://arsteclmica.r.rnn/gadgetsi2013/04!from-tooch--disp!ays-b-lhe-5l.lrface-a-brief-histmy--0f-touchscreero--techriology/2/
`
`0013
`
`Page 1441
`
`

`

`5/10/2018
`
`From touch displays to the Surface: A brief history of touchscreen technology I Ars Technica
`
`Using the Sony SmartSkin.
`
`In 2002, Sony introduced a flat input surface that could recognize multiple hand positions and touch
`points at the same time. The company called it S,'n-,FTS.k.in. The technology worked by calculating the
`distance between the hand and the surface with capacitive sensing and a mesh-shaped antenna.
`Unlike the camera-based gesture recognition system in other technologies, the sensing elements
`were all integrated into the touch surface. This also meant that it wouldn't malfunction in poor
`lighting conditions. The ultimate goal of the project was to transform surfaces that are used every
`day, like your average table or a wall, into an interactive one with the use of a PC nearby. However,
`the technology did more for capacitive touch technology than may have been intended, including
`introducing multiple contact points.
`
`How the SmartSkin sensed gestures.
`
`Jun Rekimoto at the Interaction Laboratory in Sony's Computer Science Laboratories noted the
`advantages of this technology in a \Nhiterup-:T. He said technologies like SmartSkin offer "natural
`support for multiple-hand, multiple-user operations." More than two users can simultaneously touch
`the surface at a time without any interference. Two prototypes were developed to show the
`SmartSkin used as an interactive table and a gesture-recognition pad. The second prototype used
`finer mesh compared to the former so that it can map out more precise coordinates of the fingers.
`Overall, the technology was meant to offer a real-world feel of virtual objects, essentially recreating
`how humans use their fingers to pick up objects and manipulate them.
`
`https://arstechnica.com/gadgets/2013/04/from-touch-displays-to-the-surface-a-brief-history-of-touchscreen-technology/3/
`
`0014
`
`Page 1442
`
`

`

`Fmm touch displays In lne Surf am: A brief hi:otnry of torn:hscreen !ochnoklgy I Afs Ter.hnirn
`2002-2004: Failed tablets and Microsoft Research's TouchLight
`
`Multitouch technology struggled In the mainstream, appearing In
`specialty devices but never quite catching a big break. One almost
`came in 2002, when Canada-based DSI Datotech developed the
`HandGear + GRT device (the acronym "GRT" referred to the device's
`Gesture Recognition Technology}. The device's multipoint touch pad
`worked a bit like the aforementioned iGesture pad in that it could
`recognize various gestures and allow users to use it as an input
`device to control their computers. "We wanted to make quite sure
`that HandGear would be easy to use," VP of Marketing Tim
`Heaney stdd in a press release. "So the technology was designed to
`recognize hand and finger movements which are completely
`natural, or Intuitive, to the user, whether they're left- or right(cid:173)
`handed. After a short !earning-period, they're literally able to concentrate on the work at hand, rather
`than on what the fingers are doing."
`
`A muh:itouch tablet input device
`named HandGear.
`
`HandGear also enabled users to "grab" three-dimensional objects in real-time, further extending that
`idea of freedom and productivity in the design process. The company even made the AP! available for
`developers via AutoDesk. Unfortunately, as Buxton mentions in his G'/erviev,, of mu!titouch, the
`company ran out of money before their product shipped and DSI dosed its doors.
`
`Touchllght - Microsoft Research
`
`rillps://arstechrsia:1.rnrn/gadgets/2013/04!from-tooch--disp!ays-b-lhe--surface-a-bl"ief.-histrny--0f-touchscreen--technology/3/
`
`0015
`
`Page 1443
`
`

`

`5/10/2018
`
`From touch displays to the Surface: A brief history of touchscreen technology I Ars Technica
`
`Andy Wilson explains the technology behind the Touch light.
`
`Two years later, Andrew D. Wilson, an employee at Microsoft Research, jevc-in~x:d a gesture-based
`imaging touchscreen and 3D display. The Touch light used a rear projection display to transform a
`sheet of acrylic plastic into a surface that was interactive. The display could sense multiple fingers
`and hands of more than one user, and because of its 3D capabilities, it could also be used as a
`makeshift mirror.
`
`The Touchlight was a neat technology demonstration, and it was eventually licensed out for
`production to F:on Pf:ai!<.'/ before the technology proved too expensive to be packaged into a
`consumer device. However, this wouldn't be Microsoft's only foray into fancy multitouch display
`technology.
`
`2006: Multitouch sensing through "frustrated total internal reflection"
`
`In 2006, Jeff Han gave the first public demonstration of his intuitive, interface-free, touch-driven
`computer screen at a TED Conference in Monterey, CA. In his p!<:'sent,:,ticn, Han moved and
`manipulated photos on a giant light box using only his fingertips. He flicked photos, stretched them
`out, and pinched them away, all with a captivating natural ease. "This is something Google should
`have in their lobby," he joked. The demo showed that a high-resolution, scalable touchscreen was
`possible to build without spending too much money.
`
`https://arstechnica.com/gadgets/2013/04/from-touch-displays-to-the-surface-a-brief-history-of-touchscreen-technology/3/
`
`0016
`
`Page 1444
`
`

`

`Fmm touch displays In lne Surf am: A brief hi:otnry of torn:hscreen !ochnoklgy I Afs Ter.hnirn
`
`A diagram of Jeff Han's mu!tltm.1ch sensing used FT!ft
`
`Han had discovered that the "robust" multitouch sensing was possible using "frustrated total internal
`reflection" (FT!R). a technique from the biometrics community used for fingerprint imaging. FTIR
`\.>vork:s by shining light through a plece of acryllc or p!exiglass. The light (infrared is commonly used}
`bounces back and forth between the top and bottom of the acrylic as it travels. When a finger
`touches down on the surface, the beams scatter around the edge where the finger is placed, hence
`the term "frustrated." The images that are generated look like white blobs and are plcked up by an
`infrared camera. The computer analyzes where the finger ls touching to mark its placement and
`asslgn a coordinate. The software can then analyze the coordinates to perform a certain task, like
`resize or rotate objects.
`
`Jeff Han on TED Talks
`
`rillps://arstechrsia:1.rnrn/gadgets/2013/04!from-tooch--disp!ays-b-lhe--surface-a-bl"ief.-histrny--0f-touchscreen--technology/3/
`
`0017
`
`Page 1445
`
`

`

`5/10/2018
`
`From touch displays to the Surface: A brief history of touchscreen technology I Ars Technica
`
`Jeff Han demonstrates his new "interface-free" touch-driven screen.
`
`After the TED talk became a YouTube hit, Han went on to launch a startup called Perceptive Pixel. A
`year following the talk, he told VV!red that his multitouch product did not have a name yet. And
`although he had some interested clients, Han said they were all "really high-end clients. Mostly
`defense."
`
`Last year, Hann <:;,-_iid his company to Microsoft in an effort to make the technology more mainstream
`and affordable for consumers. "Our company has always been about productivity use cases," Han
`told AIIThingsD. "That's why we have always focused on these larger displays. Office is what people
`think of when they think of productivity.
`
`2008: Microsoft Surface
`
`Before there was a 10-inch tablet, the name "Surface" referred to Microsoft's high-end tabletop
`graphical touchscreen, originally built inside of an actual IKEA table with a hole cut into the top.
`Although it was demoed to the public in 2007, the idea originated back in 2001. Researchers at
`Redmond envisioned an interactive work surface that colleagues could use to manipulate objects
`back and forth. For many years, the work was hidden behind a non-disclosure agreement. It took 85
`prototypes before Surface 1.0 was ready to go.
`
`As Ars vvrc.:t~,- in 2007, the Microsoft Surface was essentially a computer embedded into a medium(cid:173)
`sized table, with a large, flat display on top. The screen's image was rear-projected onto the display
`surface from within the table, and the system sensed where the user touched the screen through
`cameras mounted inside the table looking upward toward the user. As fingers and hands interacted
`with what's on screen, the Surface's software tracked the touch points and triggered the correct
`actions. The Surface could recognize ~,e\,c-r_:ii much points i.~t _:1 UnK', as well as objects with small
`"domino" stickers tacked on to them. Later in its development cycle, Surface also gained the ability to
`identify devices via RFID.
`
`I Microsoft unveils the SURFACE
`
`https://arstechnica.com/gadgets/2013/04/from-touch-displays-to-the-surface-a-brief-history-of-touchscreen-technology/3/
`
`0018
`
`Page 1446
`
`

`

`Fmm touch displays In lne Surf am: A brief hi:otnry of torn:hscreen !ochnoklgy I Afs Ter.hnirn
`
`Bill Gates demonstrates the Microsoft Surface.
`
`The original Surface was unveiled at the A!! Things D conference in 2007. Although many of its design
`concepts weren't new, it very effectively illustrated the real-world use case for touchscreens
`integrated into something the size of a coffee table. Microsoft then brought the 30-inch Surface to
`demo it at CES 2008, but the company exp!icit!y said that it was targeting the "entertainment retail
`space." Surface was designed primarily for use by Microsoft's commercial customers to give
`consumers a taste of the hardware. The company partnered up with several big name hotel resorts,
`like Starwood and Harrah's Casino, to showcase the technology in their lobbies. Companies like AT&T
`used the Surface to showcase the latest handsets to consumers entering their brick and mortar retail
`locations.
`
`Microsoft Surface Demo @ CES 2008
`
`rillps://arstechrsia:1.rnrn/gadgets/2013/04!from-tooch--disp!ays-b-lhe--surface-a-bl"ief.-histrny--0f-touchscreen--technology/3/
`
`0019
`
`Page 1447
`
`

`

`Fmm touch displays In lne Surf am: A brief hi:otnry of torn:hscreen !ochnoklgy I Afs Ter.hnirn
`
`Surface at CES 2008.
`
`Rather than refer to it as a graphic user interface (GU!), Microsoft denoted the Surface's interface as a
`natural user interface, or "NUL" The phrase suggested that the technology would feel almost
`instinctive to the human end user, as natural as interacting with any sort of tangible object in the reai
`world. The phrase also referred to the fact that the interface was driven primarily by the touch of the
`user rather than input devices. (Plus, NUl-"new-ey"-made for a snappy, marketing-friendly
`acronym.)
`
`Samsung SUR40 with Microsoft® Plxe!Sense™
`
`Microsoft introduces the Samsung SUR40.
`
`In 2011, Microsoft p<:1rtn~~r~id up with manufacturers like Samsung to produce sleeker, newer tabletop
`Surface hardware. For example, the Satnsung SU~M-0 has a 4(Hnch 1080p LED, and It drastically
`
`rillps://arstm:hrsirn.rnr11/gadgets/2013/04!from-tooch--disp!ays-(0--lhe--surface-a-bl"ief.-histrny--0f-touchscmen--technology/3/
`
`0020
`
`Page 1448
`
`

`

`5/10/2018
`
`From touch displays to the Surface: A brief history of touchscreen technology I Ars Technica
`
`reduced the amount of internal space required for the touch sensing mechanisms. At 22-inches thick,
`it was thinner than its predecessors, and the size reduction made it possible to mount the display on
`a wall rather than requiring a table to house the camera and sensors. It cost around $8,400 at the
`time of its launch and ran Windows 7 and Surface 2.0 software.
`
`Last year, the company rebranded the technology as PixelSense once Microsoft i:",t(l'.:r.l:.1U:'c.l its
`unrelated Surface tablet to consumers. The name "PixelSense" refers to the way the technology
`actually works: a touch-sensitive protection glass is placed on top of an infrared backlight. As it hits
`the glass, the light is reflected back to integrated sensors, which convert that light into an electrical
`signal. That signal is referred to as a "value," and those values create a picture of what's on the
`display. The picture is then analyzed using image processing techniques, and that output is sent to
`the computer it's connected to.
`
`PixelSense features four main components that make up its technology: it doesn't require a mouse
`and keyboard to work, more than one user can interact with it at one time, it can recognize certain
`objects placed on the glass, and it features multiple contact points. The name PixelSense could also
`be attributed to that last bit especially-each pixel can actually sense whether or not there was touch
`contact.
`
`Although it would make an awesome living room addition, Microsoft continues to market the Surface
`hardware as a business tool rather than a consumer product.
`
`Touch today-and tomorrow?
`
`It can't be understated-each of these technologies had a monumental impact on the gadgets we
`use today. Everything from our smartphones to laptop trackpads and WACOM tablets can be
`somehow connected to the many inventions, discoveries, and patents in the history of touchscreen
`technology. Android and iOS users should thank to E.A. Johnson for capacitive touch-capable
`
`https://arstechnica.com/gadgets/2013/04/from-touch-displays-to-the-surface-a-brief-history-of-touchscreen-technology/3/
`
`0021
`
`Page 1449
`
`

`

`5/10/2018
`
`tOl
`
`From touch displays to the Surface: A brief history of touchscreen technology I Ars Technica
`SUBSCRIPTIONS
`
`e (POS) system.
`
`In the next part of our series, we'll dive deeper on the devices of today. (Just how has the work
`of FingerWorks impacted those iDevices anyway?) But history did not end with 2011, either. We'll also
`discuss how some of the current major players-like Apple and Samsung-continue contributing to
`the evolution of touchscreen gadgets. Don't scroll that finger, stay tuned!
`
`Page: ··; 2 3
`
`Florence is a former reviews editor at Ars.
`
`https://arstechnica.com/gadgets/2013/04/from-touch-displays-to-the-surface-a-brief-history-of-touchscreen-technology/3/
`
`0022
`
`Page 1450
`
`

`

`Exhibit 1017
`Exhibit 1017
`
`Page 1451
`
`Page 1451
`
`

`

`UNITED STA 1ES p A 1ENT AND TRADEMARK OFFICE
`
`UNITED STA TES DEPARTMENT OF COMMERCE
`United States Patent and Trademark Office
`Address: COMMISSIONER FOR PATENTS
`P.O. Box 1450
`Alexandria, Virginia 22313-1450
`www.uspto.gov
`
`APPLICATION NO.
`
`FILING DATE
`
`FIRST NAMED INVENTOR
`
`ATTORNEY DOCKET NO.
`
`CONFIRMATION NO.
`
`90/006,572
`
`03/25/2003
`
`6251537
`
`1568.1061RE
`
`3636
`
`06/30/2010
`
`49455
`7590
`SIBIN MCEWEN, LLP
`1400 EYE STREET, NW
`SUIIB 300
`WASHINGTON, DC 20005
`
`EXAMINER
`
`JOHNSON, JERRYD
`
`ART UNIT
`
`PAPER NUMBER
`
`3991
`
`MAIL DATE
`
`DELIVERY MODE
`
`06/30/2010
`
`PAPER
`
`Please find below and/or attached an Office communication concerning this application or proceeding.
`
`The time period for reply, if any, is set in the attached communication.
`
`PTOL-90A (Rev. 04/07)
`
`Page 1452
`
`

`

`UNITED STATES PATENT AND TRADEMARK OFFICE
`
`BEFORE THE BOARD OF PATENT APPEALS
`AND INTERFERENCES
`
`Ex parte SAMSUNG DISPLAY DEVICES CO., LTD.,
`Patent Owner and Appellant
`
`Appeal 2008-005992
`Reexamination Control 90/006,572
`Patent 6,251,537 Bl
`Technology Center 3900
`
`Deci