US 8,686,952 B2
`(10) Patent No.:
`a2) United States Patent
`Burroughetal.
`(45) Date of Patent:
`Apr.1, 2014
`
`
`US008686952B2
`
`(54) MULTI TOUCH WITH MULTI HAPTICS
`.
`:
`
`Inventors: BenjeminIPepe NowtomMA(US).
`
`.
`
`>
`
`>
`
`.
`
`(75)
`
`.
`.
`(73) Assignee: Apple Inc., Cupertino, CA (US)
`
`(*) Notice:
`
`Subject to any disclaimer, the term ofthis
`‘
`*
`ec isexrended66.{diusted under 35
`S.C,
`154(b)
`by
`ays.
`
`(21) Appl. No.: 12/419,174
`
`(22)
`
`(65)
`
`Filed:
`
`Apr. 6, 2009
`as
`.
`Prior Publication Data
`US 2010/0156818 Al
`Tun. 24. 2010
`
`Related U.S. Application Data
`(60) Provisional application No. 61/140,519, filed on Dec.
`23, 2008.
`
`(51)
`Int. Cl.
`(2006.01)
`GO6F 3/041
`(2006.01)
`(52) Cec
`GO6F 3/016 (2013.01)
`CPC .
`USPC. esc. 345/173;345/87; 345/156; 345/168,
`;
`;
`;
`;
`345/169
`
`(58) Field of Classification Search
`CPC vieeeccccceseesetensesceecsessenecaeeseeescnecaeeeens GO6F 3/016
`See application file for complete search history.
`
`(56)
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`6,337,678 Bl
`6,819,312 B2
`7,148,789 B2
`7,336,260 B2
`7,369,115 B2
`
`1/2002 Fish
`11/2004 Fish
`12/2006 Sadleretal.
`2/2008 Martinetal.
`§/2008 Cruz-Hernandezet al.
`
`..........
`
`2003/0122779 Al
`2003/0179190 Al
`2005/0017947 A1*
`
`2006/0109256 Al
`
`7/2003 Martin etal.
`9/2003 Franzen
`1/2005 Shahoian etal. w..0.... 345/156
`
`5/2006 Grant et al.
`
`2006/0119586 Al
`6/2006 Grantet al.
`2007/0146317 Al
`6/2007 Schena
`2007/0229455 Al
`10/2007 Martinetal.
`2007/0229478 Al
`10/2007 Rosenberg etal.
`345/173
`seaboeeod “I . hooey qamstein
`eal
`Z'
`esterman ct al.
`2008/0024459 AL*
`1/2008 Poupyrevet al. wesc. 345/173
`2008/0062122 Al
`3/2008 Rosenbergetal.
`2008/0062143 Al
`3/2008 Shahoian etal.
`2008/0062144 Al
`3/2008 Shahoian etal.
`2008/0068348 Al
`3/2008 Rosenbergetal.
`2008/0084384 Al
`4/2008 Gregorioetal.
`2008/0111788 Al
`5/2008 Rosenbergetal.
`2008/0170037 Al
`7/2008 Cruz-Hernandez etal.
`
`.ccscceue 345/173
`2009/0002328 AL*
`1/2009 Ullrichet al.
`2009/0085878 Al*
`4/2009 Heubel et al.
`.....ccsesee 345/173
`OTHER PUBLICATIONS
`How Haptics Will Change the Way We Interact With Machines,
`http://www.popularmechanics.com/technology/industry/4253368.
`html.
`iPhone Haptics, http://code.google.com/p/iphone-haptics/.
`es
`:
`cited by examiner
`Primary Examiner — Joseph Haley
`Assistant Examiner — Ifedayo Iluyomade
`(74) Attorney, Agent, or Firm — Brownstein Hyatt Farber
`Schreck, LLP
`°
`
`ABSTRACT
`
`57
`(67)
`Methods and systems for processing touch inputs are dis-
`closed. The invention in one respect includes reading data
`froma multi-touch sensing device such as a multi-touch touch
`screen wherethe data pertains to touch input with respect to
`the multi-touch sensing device, and identifying at least one
`multi-touch gesture based on the data from the mulli-touch
`sensing device and providing an appropriate multi-haptic
`response.
`
`16 Claims, 21 Drawing Sheets
`
`Touch event
`
`
`
`Acoustic
`Isolator
`121
`
`ae
`ant
`
`Hz
`
`Touch event
`
`
`
`
`
`
`
`
`
`
`
`| Micro-
`HOOeeeet tad
`1 Haptic profiles ha
`Housing 102
`
`‘
`
`
`
`~___Operationat components
`104
`
`~__Operational components
`
`APPLE 1014
`
`
` ‘— Processor 106
`
`
`
`
`
`Acoustic
`Isolator
`421
`
`410
`
`Housing 162
`
`APPLE 1014
`
`1
`
`

`

`U.S. Patent
`
`Apr. 1, 2014
`
`Sheet 1 of 21
`
`US 8,686,952 B2
`
`O04
`
`WLbig
`
`SoepalutJesh
`
`Aeidsiq
`
`CLLB0lAag
`
`jeuoReioda
`
`aiuasje
`
`Sjuauoduics
`
`POL
`
`
`
`2
`
`
`

`

`U.S. Patent
`
`Apr.1, 2014
`
`Sheet2 of 21
`
`US 8,686,952 B2
`
`Acoustic
`
`svar
`
`Haptic feedback
`H(d)
`
`
`Uppersurface
`Touch event
`
`126
`
`
`
`
`
`HO0) Peene 3g|ny
`
`
`
`: Touch sense layer 122
`
`
`Display device 112
`
`apticactuator,==Hapticactuator|
`
`| eeewoe
`
`ae
`
`haptic
`
`
`
`
`Housing 102
`
`
`Operational components
`104
`
`Fig. 1B
`
`3
`
`

`

`U.S. Patent
`
`Apr. 1, 2014
`
`Sheet 3 of 21
`
`US8,686,952 B2
`
`Acoustic
`lsolator
`121
`
`Touch event
`
`.
`
`| Haptic actuator |
`
`x Pr Micro-controller |
`
`: Haptic profiles «-~
`
`Processor 106
`
`
`
`T
`
`
`|
`
`Housing 102
`
`Operational components
`104
`
`Fig. 1C
`
`4
`
`

`

`U.S. Patent
`
`Apr. 1, 2014
`
`Sheet 4 of 21
`
`US 8,686,952 B2
`
`Acoustic
`isolator
`421
`
`110
`/
`
`.-
`roN
`wk
`
`Heompound <
`t
`t
`t
`t
`t
`t
`{
`t
`
`\
`
`H
`
`Touch event
`
`
`
`
`}
`
`‘
`
` 24
`
`_ ge
`ow
`ZN
`tf
`to
`to
`to
`to
`to
`to
`1
`4
`\
`to
`{4
`to
`‘
`i
`Protectivelayer 120
`
`Touch senselayer 122
`
`
`
`
`
` Display device 112
`
`“Haptic actuator |
`
`_ Haptic actuator
`
`Housing 102
`
`
`
`Operational components
`104
`
`Fig. 1D
`
`5
`
`

`

`U.S. Patent
`
`Apr. 1, 2014
`
`Sheet 5 of 21
`
`US8,686,952 B2
`
`
`
`Hhousing
`
`
`
`
`
`
`Housing 102
`
`
`Acoustic
`isolator
`121
`
`Touch event
`T
`
`_
`
` : 124
`Touch sense layer 122
`
`Display device 112
`
`Haptic actuator
`
`Operational components
`104
`
`Fig. 1E
`
`6
`
`

`

`U.S. Patent
`
`Apr. 1, 2014
`
`Sheet 6 of 21
`
`US8,686,952 B2
`
`Silent area
`
`arere
`
`128
`
`H(X) = H(ni, Ma, ny) ~ Cano
`
`Ee
`
`EE
`
`EEE
`
`Fig. 2A
`
`7
`
`

`

`U.S. Patent
`
`Apr. 1, 2014
`
`Sheet 7 of 21
`
`US8,686,952 B2
`
`
`
`Fig. 2B
`
`8
`
`

`

`U.S. Patent
`
`Apr. 1, 2014
`
`Sheet 8 of 21
`
`US 8,686,952 B2
`
`00€ €“bis
`
`7]yjbue|
`
`uonoesU0
`
`uoisuedxa
`
`<p
`
`A+il-
`
`9
`
`
`
`

`

`U.S. Patent
`
`Apr. 1, 2014
`
`Sheet 9 of 21
`
`US 8,686,952 B2
`
`00&
`
`pbis
`
`7]uybus|
`
`><
`
`UONoesU0D
`
`
`
`+49010)paddy
`
`
`
`(W/4ainsseid)
`
`||I
`
`
`
`|uoisuedxa At/-
`
`eeeydA
`
`10
`
`10
`
`
`

`

`U.S. Patent
`
`Apr.1, 2014
`
`Sheet 10 of 21
`
`US 8,686,952 B2
`
`Sense output node
`
`502
`
`
`
`Voltage sensed?
`
`ctuator in passive
`
`506
`
`
`
`modeit
`
` y A
`
`Sensed voltage greater
`
`than threshold?
`
`
`Actuator in active
`mode
`
`y
`
`
`
`510
`
`512
`
`Retrieve haptic
`profile
`
`Fig. 3
`
`x 5
`
`00
`
`514
`
`n
`
`518
`
`
`Generate haptic response
`by actuator based on
`
`retrieved haptic profile
`
`Haptic respone
`
`
`complete?
`
`516
`
`y
`
`Set to passive
`
`11
`
`

`

`U.S. Patent
`
`Apr. 1, 2014
`
`Sheet 11 of 21
`
`US8,686,952 B2
`
`image feature
`612
`
`Device
`100
`
`ee
`
`
`
`12
`
`

`

`US 8,686,952 B2
`
`~<a— 702
`
`Region 1
`
`Region 2
`
`Target
`706
`
`U.S. Patent
`
`Apr. 1, 2014
`
`Sheet 12 of 21
`
`
`
`a
`
`a
`
`“
`
`.
`
`.
`tee on
`ee .
`Region 2
`
`~
`
`N
`
`N.
`
`So
`
`NN
`
`ee
`
`Target
`708
`
`i
`
`:
`
`i\
`
`4
`
`/
`
`/
`
`i
`i
`
`:
`i
`po
`{
`\
`i
`
`\
`\
`
`\\
`
`\
`
`\
`
`‘
`
`\
`
`\
`“
`
`\
`
`M
`
`— ~
`
`\
`
`\
`
`\
`
`\
`
`\
`
`\
`
`\\
`
`|
`
`i
`
`/
`
`4
`
`i
`
`/
`
`4
`
`La
`
`a
`
`13
`
`13
`
`

`

`US 8,686,952 B2
`
`Sheet 13 of 21
`
`
`
`
`
`
`
`
`
`gIIA=Jabuyjouonisod5|wom.6Zs.blezuoiBes|uolBadJODIE}zuoibesJODIE}|uoiBeuye6se}
`
`U.S. Patent
`
`jebuey
`
`Aysuaqul
`
`Apr. 1, 2014
`
`Jueosainb
`
`.ondegg‘614“4ayoyndeyxxXyo6e}
`
`
`yebseyyobieyHHajejsjusosainbHajeys
`
`'Hbeybyesuodse!
`
`
`
`
`
`(x)eyeulpio-05YyOnoL
`
`zuoibesZ08Z08c08
`
`14
`
`14
`
`
`
`

`

`U.S. Patent
`
`Apr. 1, 2014
`
`Sheet 14 of 21
`
`US 8,686,952 B2
`
`
`
`
`
`V6‘bl
`
`006
`
`496uyOouonisod
`
`°H
`
`
`g6‘bi
`
`(x)a}euIpsO-09YONO|
`
`Aysusqul
`
`ondey
`
`esuodsel
`
`15
`
`15
`
`

`

`U.S. Patent
`
`Apr.1, 2014
`
`Sheet 15 of 21
`
`US 8,686,952 B2
`
`~aJoyeoipui
`
`aBpesoyeoipuiapa
`
`
`
` 0)ayeulpio-09yonoyJabuyjouonisod gor‘bis
`
`Aysuayu}
`
`ondey
`
`esuodsal
`
`VoLbig
`
`
`
`ZLOaunjeayabeuu|
`
`ainjyeajJoQabpe
`
`16
`
`16
`
`
`
`

`

`U.S. Patent
`
`Apr.1, 2014
`
`Sheet 16 of 21
`
`US 8,686,952 B2
`
`1100
`
`the presence ofat
`
`least a first finger
`and a second finger
`are detected
`
`4102
`
`
`
`the presence of the
`
`
`two fingers represents
`
`a gesture
`
`1104
`
`haptic devices nearest the
`touch poit are sct to active
`mode‘in order to provide a
`vibrotactile
`
`distance between at least the
`
`1106
`
`1108
`
`1110
`
`Distance increasing?
`
`two fingers is compared.
`
`
`Fig. 11
`
`Generate zoom in signal
`
`Generate zoom out signal
`
`Generate zoomin haptic
`signal
`
`Generate zoom out haptic
`signal
`
`
`
`1114
`
`1118
`
`17
`
`17
`
`

`

`U.S. Patent
`
`Apr.1, 2014
`
`Sheet 17 of 21
`
`US 8,686,952 B2
`
`1262
`
`1210
`
`4204
`
`1208
`
`FIG. 12A
`
`1202
`
`1206
`
`FIG. 12B
`
`18
`
`18
`
`

`

`U.S. Patent
`
`Apr.1, 2014
`
`Sheet 18 of 21
`
`US 8,686,952 B2
`
`
`
`19
`
`19
`
`

`

`U.S. Patent
`
`Apr.1, 2014
`
`Sheet 19 of 21
`
`US 8,686,952 B2
`
`N. California
`
`
`
`‘ane
`
`"i
`
`FIG. 12F
`
`20
`
`20
`
`

`

`U.S. Patent
`
`Apr.1, 2014
`
`Sheet 20 of 21
`
`US 8,686,952 B2
`
`1236
`
`7
`
`f
`
`San\Franctsep
`eos
`
`1204
`
`‘.
`
`CK
`
`O) S
`1218—“NON,
`ae
`’, SanCards~..¢"
`\
`Teme
`-
`
`.
`
`4208
`
`"
`
`.
`
`.
`ae
`onan
`:
`
`21
`
`we
`
`i
`
`.
`
`~
`
`Se "~\vot
`
`“
`“s
`+
`.,
`v4 SanrQaros
`:
`Ye
`eo
`
`21
`
`

`

`U.S. Patent
`
`Apr. 1, 2014
`
`Sheet 21 of 21
`
`US8,686,952 B2
`
`1300
`
`virtual scroll wheel
`is presented on the
`display
`
`1302
`
`
`SL epresenceofthe
` 1304
`wheel is set.
`
`two fingers represents
`a gesture
`
`the initial position of the
`fingers on the virtual scroll
`
`1306
`
`
`
`rotate signal is generated
`
`
`when the angle of the fingers
`change relative to the
`reference point
`
`
`
`rotate signal can be used to
`rotate the virtual scroll
`wheel inthe direction of
`finger rotation
`
`1308
`
`1310
`
`Generate haptic click signal
`
`4312
`
`Fig. 13
`
`22
`
`22
`
`

`

`US 8,686,952 B2
`
`1
`MULTI TOUCH WITH MULTI HAPTICS
`
`CROSS REFERENCE TO RELATED
`APPLICATIONS
`
`This patent application takes priority under 35 U.S.C. 119
`(e) to U.S. Provisional Application Ser. No. 61/140,519
`entitled MULTI TOUCH WITH MULTI HAPTICSby Bur-
`rough et al., filed Dec. 23, 2008 which is incorporated by
`reference in its entirety for all purposes.
`
`BACKGROUNDOF THE INVENTION
`
`1. Field of the Invention
`
`The present invention relates generally to providing multi-
`touch/multi-haplic systems and methods.
`2. Description of the Related Art
`Multi-touch devices have advantages over conventional
`single point sensing touch devicesin that they can distinguish
`more than one object (finger) in contrast to single point
`devices that are simply incapable of distinguishing multiple
`objects. In most cases, multi-touch devices monitor a sensing
`surface for a touch or near touch, and when a touch occurs
`determines the distinct areas of contact and identifies the
`contacts via their geometric features and geometric arrange-
`ment. Onceidentified or classified, the contacts are monitored
`for various motions, actions or events. The contacts and
`motions thereof are then converted into inputs for controlling
`someaspectof an electronic device.
`Multi-touch devices can be embodied in various forms
`
`including butnotlimit to standard touch pads, large extended
`palm pads, touch screens, touch sensitive housings, etc. Fur-
`thermore, multi-touch devices can be placed in variouselec-
`tronic devices including but not limited to computers such as
`tablet computers, laptop computers, desktop computers as
`well as handheld computing devices such as media players
`(e.g., music, video, games), PDAs, cell phones, cameras,
`remote controls, and/or the like. The multi-touch devices can
`also be placed on dedicated input devices such as touch screen
`monitors, keyboards, navigation pads, tablets, mice, and the
`like. Essentially, multi-touch devices can be applied to any
`surface, and can be found in any consumerelectronic product
`that requires inputs.
`Since multi-touch devices provide a numberof inputting
`operationsat a single location (input surface), inputting with
`multi-touch devices can be very efficient. The user can main-
`tain their hand(s) at the multi-touch surface without having to
`move their hand(s) to address other input devices. For
`example, conventional systems typically include a keyboard
`and a separate mouse.In order to use the mouse,the user must
`movetheir hand from the keyboard and onto the mouse. In
`order to keyboard efficiently (both hands), the user must
`movetheir hand from the mouseto the keyboard. This input-
`ting sequenceis very inefficient. For one, only one device can
`be used effectively at a given time. For another, there is
`wasted time between each inputting step. In contrast, with
`multi-touch surfaces the user can generate both static com-
`mands(e.g., keyboarding) and manipulative commands(e.g.,
`tracking) from the same location andat the same time. The
`user therefore does not have to movetheir hands to perform
`different inputting tasks. The user simply provides different
`chordsorfinger motions to generate a numberofinputs either
`sequentially or simultaneously. In one example, the user can
`provide key commands with taps at specific locations of the
`multi-touch surface while allowing tracking from all loca-
`tions of the multi-touch surface.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`However, research has shown that providing the multi-
`touch surface with the ability to provide physical (haptic)
`feedback makes the multi-touch experience even moreeffi-
`cient and realistic to the user. For example, physical key-
`boards provide a physical indication (a bump, for example)
`indicative of the homekey. This physical sensation can not be
`provided by a conventional multi-touch system thereby fore-
`ing the user to visually locate the home key thereby making
`keyboard use less efficient and fatiguing. However, by pro-
`viding a physical facsimile of the home key bump using an
`actuatorthat provides a physical sensation to the user provid-
`ing an approximate representation of the bump, the user’s
`experience of the MT keyboard (and any multi-touch system
`for that matter) can be more realistic and therefore more
`enjoyable.
`Therefore, a system that enhances the multi-touch experi-
`ence byincorporating a corresponding physical response(s) is
`described.
`
`SUMMARYOF THE INVENTION
`
`The invention relates, in one embodiment, to an apparatus
`and method for providing multi-touch haptic feedback. The
`apparatus includes, at least, a touch pad having a touch sen-
`sitive surface arrangedto receive a user provided multi-touch
`event associated with at least two different locations on the
`touch sensitive surface, a multi-touch detection mechanism
`operatively coupled to the touch sensitive surfacethat detects
`the multi-touch event and generates a corresponding a multi-
`touchsignal, and a plurality ofhaptic feedback devices opera-
`tively coupled to the multi-touch detection mechanism and
`the touch sensitive surface cooperatively arranged to concur-
`rently provide tactile feedback at each of the at least two
`different locations on the touch sensitive surface in response
`to the multi-touch signal wherein the tactile feedback at each
`of the at least two different locations are discreet from one
`
`another. When the multi-touch signal indicates that the multi-
`touch event
`is a dynamic multi-touch event indicating a
`change in the multi-touch event, then thetactile feedback at
`each ofthe at least two different locations is updatedto reflect
`the changein the multi-touch event.
`It should be noted that in some casesthe tactile feedback
`event can be different for each of the at least two different
`locations.
`The invention relates, in another embodiment, to a multi-
`touch haptic mechanism. The multi-touch haptic mechanism
`includes, at least, a touch pad having a touchsensitive surface
`arranged to detect a user touch event at substantially any
`location on the touch sensitive surface and a plurality of
`independent haptic devices operatively coupled to the touch
`sensitive surface each providing a corresponding type of tac-
`tile feedback thereby providing a tactile feedback at substan-
`tially any location on the touch sensitive surface at which the
`user touch eventhas occurred, wherein each ofthe plurality of
`independent haptic devices only responds to the user touch
`event in one or moreassociated regionsof the touch sensitive
`surface. When at least two of the plurality of independent
`haptic devices cooperate to provide a type of haptic response
`that is different than that type provided byeitheroftheat least
`two independenthaptic devices separately.
`Theinvention relates, in another embodiment, to an inte-
`grated device arranged to act as both a force sensing device
`and a haptic feedback device. The device includes, at least, a
`touch sensitive surface, a controller unit, and a mechanical
`actuator coupled with the controller unit and the touch sensi-
`tive surface. The integrated device acts as the force sensing
`device by generating an output voltage in direct proportion to
`
`23
`
`23
`
`

`

`US 8,686,952 B2
`
`3
`a force applied to the mechanical actuatorby a user touching
`the touch sensitive surface, sensing the output voltage by the
`controller unit and converting the sensed output voltage to an
`indication of the applied force. Only when the sensed output
`voltage exceeds a voltage threshold level does the integrated
`device act as the haptic feedback device by halting the sensing
`of the output voltage by the controller unit activating the
`mechanical actuator by the controller unit, wherein theacti-
`vated mechanical actuator imparts a physical force to the
`touch sensitive surface that results in a vibro-tactile response
`(subcutaneoustissue activated) felt by the user commensurate
`with the force applied by the user.
`The invention relates, in another embodiment, to an elec-
`tronic device. The electronic device includes, at least, a touch
`pad having a touch sensitive surface arranged to process a
`user touch event and a plurality of haptic feedback devices
`each of which is operatively coupled to the touch sensitive
`surface and each respondingto the user touch event only in a
`specific region of the touch sensitive surface and arranged to
`providetactile feedback singly or in combination with others
`of the plurality of haptic feedback devices in response to the
`user touch event. When the touch sensitive regionsofat least
`two ofthe plurality ofhaptic devices overlap,ifthe user touch
`event occurs in the overlapping region, then the at least two
`haptic devices cooperate to provide a combinedhaptic feed-
`back response based upon the location in the overlapping
`region of the user touch event.
`‘The invention relates, in another embodiment, to an elec-
`tronic device. The electronic device includes, at least, a touch
`pad having a touchsensitive surface arranged to receive a user
`touch event provided by a user, a controller coupled and in
`communication with the touch pad arrangedto at least ana-
`lyze the user touch event and/ora state of the touch pad and
`based upon the analysis provide a user touch event signal in
`responseto the user touch event, and at least one haptic device
`operatively coupled to the controller arranged to receive the
`user touch event signal, wherein the at least one haptic device
`respondsto the user touch event signal by providing an appro-
`priate haptic feedback response to the user based upon the
`analysis provided bythe controller.
`In one embodiment,the touch sensitive surface is arranged
`to receive different types of user touch events each being
`characterized by an amountof pressure applied on the touch
`sensitive surface by a user and at least one haptic device
`operatively coupled to the touch sensitive surface arranged to
`provide a specific type of tactile feedback corresponding to
`the amountof pressure applied to the touch sensitive surface
`by the user.
`It should be noted that
`
`in each of the embodiments
`
`described above, the methods can be implemented using a
`touch based input device such as a touch screen or touch pad,
`moreparticularly a multi-touch touch based input device, and
`even more particularly a multi-touch touch screen.It should
`also be noted thatthe gestures, gesture modes, gestural inputs,
`etc. can correspond to any of those described below in the
`detailed description. For example, the gestures can be asso-
`ciated with zooming, panning, scrolling, rotating, enlarging,
`floating controls, zooming targets, paging, inertia, keyboard-
`ing, wheeling, and/orthe like.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`The invention will be readily understood by the following
`detailed description in conjunction with the accompanying
`drawings, wherein like reference numerals designate like
`structural elements, and in which:
`
`4
`FIGS. 1A-1E are a series of block diagrams of a system,in
`accordance with one embodimentof the present invention.
`FIGS. 2A-2B shows a multi-point multi-haptic system
`having a multi-touch surface that incorporates a plurality of
`haptic devices in accordance with an embodiment of the
`invention.
`FIG. 3 shows a schematic diagram of a representative
`piezo-electric haptic assembly.
`FIG. 4 shows a schematic diagram of the haptic assembly
`shownin FIG. 3 configured to act as a pressure sensor.
`FIG.5 showsa flowchart detailing a process in accordance
`with an embodimentofthe invention.
`
`5
`
`10
`
`FIG. 6 showsdisplay device displaying representative hap-
`tic active GUI elements in accordance with an embodiment of
`the invention.
`
`15
`
`FIG. 7 showsrepresentative GUI button elements in accor-
`dance with an embodimentofthe invention.
`
`FIGS. 8A-8B showsrepresentative GUI button element
`and associated haptic profile in accordance with an embodi-
`mentof the invention.
`FIGS. 9A-9B showsa representative slider element and
`associated haptic profile in accordance with an embodiment
`of the invention.
`FIGS. 10A-10B shows a feature edge detection system in
`accordance with an embodimentof the invention.
`FIG. 11 is a diagram of a zoom gesture method 1100 in
`accordance with an embodimentof the invention.
`FIGS. 12A-12Hillustrates a display presenting a GUI
`object in the form of a map of North America with embedded
`levels which can be zoomed.
`FIG. 13 is a diagram of a GUI operational method in
`accordance with one embodimentof the present invention.
`
`DETAILED DESCRIPTION OF SELECTED
`EMBODIMENTS
`
`Reference will now be madein detail to selected embodi-
`ments an example ofwhichis illustrated in the accompanying
`drawings. While the invention will be described in conjunc-
`tion with a preferred embodiment,it will be understoodthatit
`is not intendedto limit the invention to one preferred embodi-
`ment. To the contrary, it is intended to cover alternatives,
`modifications, and equivalents as can be included within the
`spirit and scope ofthe invention as defined by the appended
`claims.
`
`The invention relates to multi-touch haptic feedback.
`Multi-touch haptic feedback refers to haptic techniques
`capable of providing multiple and discretely located haptic
`sensations across a surface. The haptic system can for
`example includea plurality of haptic nodes, each of which is
`capable of issuing vibro-tactile sensations (at the same time
`or different times and with the same intensity or different
`intensity). The haptic nodes can for example be configured in
`a matrix or array. In one embodiment, the haptic nodes are
`mapped to touch sensing nodes. Each touch sensing node can
`be assigned one or more haptic nodes. The haptic nodes are
`typically proximate the touch sensing nodes to which it has
`been assigned.
`In one embodiment, the touch sensing surface is a multi
`touch surface thus making a multi touch multi-touch haptic
`device. In so doing haptic feedback can be provided that
`indicates information about a multi touch event. For example,
`the surface under a moving finger can be actuated while the
`surface under the non-moving finger remains static.
`In
`another example, the surface under the movingfingeris actu-
`ated concurrently with a signal being passed to the other
`finger indicating that a multi touch action is occurring. In this
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`24
`
`24
`
`

`

`US 8,686,952 B2
`
`6
`cavity configured to at least partially enclose any suitable
`numberof operational electronic components 104 used by
`portable electronic device 100 to carry out its intended func-
`tions. Operational electronic components 104 can include
`processor 106 that can operate (in conjunction with an oper-
`ating system) to execute computer code and produce and use
`data. Processor 106 can be implemented on a single-chip,
`multiple chips or multiple electrical components. For
`example, various architectures can be used for the processor
`106, including dedicated or embedded processor, single pur-
`pose processor, controller, ASIC, and so forth. The operating
`system, other computer code and data can reside within a
`memory 108 that can be operatively coupled to processor 106.
`By way of example, memory 108 can include Read-Only
`Memory (ROM), Random-Access Memory (RAM), flash
`memory, hard disk drive and/or the like. Operational compo-
`nents 104 can also include a numberof input/output (1/O)
`devices 109. Such devices can include audio output devices
`such as headphonejacks, data ports (such as I.E.E.E. 1392
`compliant, USB, etc.), and so on.
`Portable electronic device 100 can also include a user
`
`10
`
`15
`
`20
`
`5
`way, the signals taken together can indicate the nature of the
`underlying action being taken by the user. For example, if an
`object (such as an image) is being expandedor reducedin size
`a larger/more intense signal could be generated (either by
`increasing frequency or amplitude). It is also contemplated
`that isolated feedback can be used to provide an on-screen
`click-wheelor other such user input where the touch screen is
`used to simulate the “clicks” of the click wheel both audibly
`and viatactile feedback.
`
`The described embodiments generally pertain to gestures
`and methods ofimplementing gestures with associated physi-
`cal feedback with touch sensitive devices. Examples oftouch
`sensitive devices include touch screens and touch pads. One
`aspect of the invention describes a touch sensitive input
`device able to recognize at least two substantially simulta-
`neously occurring gestures using at least two different fingers
`or other objects (hereinafter referred to as a multi-touch
`event). The touch sensitive input device communicates with
`an array of haptic feedback devices(also referred to as haptic
`actuators) each arranged to provide haptic feedback accord-
`ing to a haptic profile in response to a multi-touch event. In
`another aspect of the invention, each finger receives different
`haptic feedback (multi-haptic) depending upon the location
`on the touch sensitive input device each finger is placed. In
`another aspect of the invention, a compoundhaptic feedback
`can be provided that combines the output from at least two
`different haptic actuators to form the compoundresponsethat
`is different from that provided by the two originating haptic
`actuators. In another embodiment, an integrated device is
`described that can act as both a force sensing device and a
`haptic feedback device. In still another embodiment, a hand-
`held portable device is described having a housing and a user
`interface are acoustically isolated from each other. In this
`way, the housing and userinterface and having non-interfer-
`ing and independenthaptic responses.
`These and other aspects of the invention are discussed
`below with referenceto FIGS. 1-13. However, those skilled in
`the art will readily appreciate that the detailed description
`given herein with respect to these figures is for explanatory
`purposes as the invention extends beyond these limited
`embodiments.
`
`interface 110 that can operate to both receive user inputs and
`provide information to a user. In the described embodiment,
`user interface 110 can include display device 112 that can be
`operatively coupled to processor 106 by way of bus 114.
`Display device 112 can correspond to any knowndisplay
`technology such as a plasma, LCD,or an organic light emit-
`ting diode (OLED). It should be noted that in the embodiment
`shown in FIGS. 1A-1E, display device 112 is integrated with
`the electronic device 100. However, display device 112 can
`also be configured as a component separate from portable
`electronic device 100 in which case display device 112 would
`be considered a peripheral device that can be coupled to
`portable electronic device 100 by way of a wired connection
`(such as a peripheral bus or cable) or a wireless connection
`such as IR, RF, Bluetooth or the like (among others).
`In somecases, display device 112 presents graphical user
`interface (GUI) 116 on display device 112. GUI 116 can
`provide an easy to use interface between a user of portable
`electronic device 100 and the operating system or application
`running thereon. Generally speaking, GUI 116 iconically
`represents programs,
`files and operational options with
`FIG. 1A-1E are block diagramsof a representative elec-
`graphical images. The graphical images can include win-
`tronic device or system 100, in accordance with one embodi-
`dows, fields, dialog boxes, menus, icons, buttons, cursors,
`ment of the present invention. Electronic device 100 can
`scroll bars, etc. Such images can be arranged in predefined
`correspond to a computer (such as a desktops or laptops) as
`well as small form factor electronic devices that can include
`layouts, or can be created dynamically to serve the specific
`actions being taken by a user. During operation, the user can
`portable consumerelectronic products such as cell phones,
`select and activate various graphical images in orderto ini-
`PDA, media players and/or the like. As such, portable elec-
`tiate functions and tasks associated therewith. By way of
`tronic device 100 can be sized for one-handed operation and
`example, a user can select a button that opens, closes, mini-
`placement into small areas such as a pocket. Portable elec-
`mizes, or maximizes a window, or an icon that launches a
`tronic device 100 can process data and more particularly
`particular program. GUI 116 can additionally or alternatively
`media such as audio, video, images, etc. As such, the portable
`display information, such as noninteractive text and graphics,
`electronic device 100 can correspondto a music player, game
`for the user on the display device 112.
`player,video player, personaldigital assistant (PDA), such as,
`55
`
`for example, an iPod™, an iPod Nano™,an iPod Shuffle™, As shown moreclearly in FIG. 1B inaside view perspec-
`an iPod™ Touch or an iPhone™available by Apple Inc. of
`tive of device 100, user interface 110 can include protective
`Cupertino, Calif. In some cases, portable electronic device
`layer 120 disposed on top of display device 112. In this way,
`100 can communicate wirelessly (with or without the aid of a
`protective layer 120 can be used as protective top layer of
`wireless enabling accessory system) and/or via wired path-
`transparent or semitransparent material
`(clear)
`thereby
`ways(e.g., using traditional electrical wires).
`affording display device 112 protection from potentially
`Portable electronic device 100 includes a housing 102.
`damaging external
`insults caused by, for example, sharp
`Housing 102 can be formed of any number of materials
`objects, dropping, and so on andyetstill allow any image
`including for example plastics, metals, ceramics andthelike.
`presented by display device 112 to be clearly viewed by a
`In one embodiment, housing 102 can be formedofstainless
`user. Protective layer 120 can be formed of many well known
`steel in order to provide an aesthetic and appealing look and
`transparent materials such as glass (e.g., referred to as cover
`feel as well as provide structural integrity and support forall
`glass), and more particularly highly polished glass. It should
`sub-assemblies installed therein. Housing 102 can define a
`be appreciated, however, that other transparent materials (or
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`60
`
`65
`
`25
`
`25
`
`

`

`US 8,686,952 B2
`
`7
`at least translucent materials) such asclearplastic may also be
`used. In some embodiments, protective top layer 120 can be
`acoustically isolated from housing 102 using, for example,
`acoustic isolation buffers 121. By acoustically isolating hous-
`ing 102 and protective top layer 120 from each other, it is
`possible to provide separate and independent haptic
`responses, one directed at housing 102 and anotherdirectedat
`protective top layer 120 without interfering with each other.
`For example, it may be desirable to provide one type ofhaptic
`response at protective layer 120 and another type haptic
`response at housing 102 at the sametime orat a different time
`independent of each other or in some cases one being the
`result of or related to the other.
`User interface 110 can be touch sensitive suitable for
`
`receiving one or more user touch events by which information
`can be passed between the user and the portable electronic
`device 100. In somecases, the one or more inputs in the form
`of user touch events can be substantially simultaneously
`received (e.g., multi-touch). In these embodiments, user
`interface 110 is rendered touch sensitive by meansof a touch
`sensing layer 122 that can be disposed below protective layer
`120 such that touch sensing layer 122 is between protective
`layer 120 and the display device 112. This arrangementcan be
`accomplished by, for example, applying touch sensing layer
`122 to display device 112 or by applying touch sensing layer
`122 to protective layer 120 using any numberof attachment
`processes, such as printing, depositing,
`laminating, etc.
`Touch sensing layer 122 generally includes at least one touch
`sensing device 124 configured to detect an object in close
`proximity to or exerting pressure on an upper surface 126 of
`protective layer 120. In keeping with the wide applicability of
`the invention, sensing device 124 can be widely varied and
`can be configured to activate as the finger touches the upper
`surface 126. In the simplest case, an electrical signal is pro-
`duced eachtimea finger (or other appropriate object) passes
`a sensor. The numberofsignals in a given time frame may
`indicate location, direction, speed and acceleration of the
`finger on the touch sensitive portion,i.e., the more signals, the
`more the user movedhisor herfinger.
`Touch sensing layer 122 can be configured to act as a
`multi-touch input device. Mul