(19) United States
`(12) Patent Application Publication (10) Pub. No.: US 2010/0156818 A1
`Burrough et al.
`(43) Pub. Date:
`Jun. 24, 2010
`
`US 20100156818A1
`
`(54) MULTI TOUCH WITH MULTI HAPTICS
`
`Publication Classi?cation
`
`(75) Inventors:
`
`Bobby Burrough, Cupertino, CA
`(US); Benjamin J. Pope, Newton,
`MA (Us)
`
`(51) Int‘ Cl‘
`G06F 3/041
`G08B 6/00
`
`(200601)
`(2006.01)
`
`Correspondence Address:
`
`PO. BOX 1687
`CUPERTINO, CA 95015-1687 (US)
`_
`_
`(73) Ass1gnee:
`APPLE INC., Cupert1no, CA (U S)
`
`(21) APP1- NOJ
`_
`(22) Flled:
`
`12/419,174
`
`Apr‘ 6’ 2009
`Related U 5 Application Data
`'
`'
`(60) Provisional application No. 61/140,519, ?led on Dec.
`23, 2008.
`
`Ulsl
`
`~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ -
`
`(57)
`
`ABSTRACT
`
`Methods and systems for processing touch inputs are dis
`closed. The invention in one respect includes reading data
`from a multi-touch sensing device such as a multi-touch touch
`screen Where the data pertains to touch input With respect to
`the multi-touch sensing device, and identifying at least one
`multi-touch gesture based on the data from the multi-touch
`sensing device and providing an appropriate multi-haptic
`response.
`
`Haptic feedback
`H(d)
`
`Upper Surface
`126
`
`Touch event
`
`Acoustic
`lsoléaior
`
`110
`
`124
`
`Protective layer 120
`Touch sense layer 122
`Display device 112
`Haptic actuator
`\140
`
`Haptic actuator l
`136 ;>
`
`121
`
`,,,,,,,,,,,,,,,,,,,,,,,,,
`
`,
`
`|
`
`a
`haptic
`
`s1
`
`121
`
`,,,,,,,,,,,,,,,,
`Micro-controller
`
`T, f
`’“ °
`
`
`
` 134 Howl: ---------------------------- - 1 -------------------------- - Housing 102 Haptic profiles 4/
`
`
`
`
`
`
`
`llllllllllllllllllllllllllllllllllllll Processor 106
`
`Operational components
`104
`
`APPLE INC.
`EXHIBIT 1005 - PAGE 0001
`
`

`
`Patent Application Publication
`
`Jun. 24, 2010 Sheet 1 0f 21
`
`US 2010/0156818 A1
`
`
`
`womtmg 6mm
`
`Q? P
`
`wEwgQEQu
`
`wow
`
`APPLE INC.
`EXHIBIT 1005 - PAGE 0002
`
`

`
`Patent Application Publication
`
`Jun. 24, 2010 Sheet 2 0f 21
`
`US 2010/0156818 A1
`
`Haptic feedback
`H(d)
`
`Upper Surface
`126
`
`Touch event
`
`Acoustic
`isolzazor
`
`110
`
`Protective layer 120
`Touch sense layer 122
`
`x
`
`,
`
`Display device 112
`
`\140
`
`,,,,,,,,,,,,,,,,,,,,,,,,
`
`l
`
`T,
`
`121
`
`134
`Hml: Micro-controller
`Housing 102
`
`Haptic profiles 4/ \
`
`‘"ff’ 777777777777777777777777777777777777 7‘
`l—‘ Processor106
`
`Operationai components
`104
`
`Fig. 1B
`
`APPLE INC.
`EXHIBIT 1005 - PAGE 0003
`
`

`
`Patent Application Publication
`
`Jun. 24, 2010 Sheet 3 0f 21
`
`US 2010/0156818 A1
`
`Acoustic
`lsolator
`121
`
`110
`
`121
`
`Touch event
`
`Protective layer 120
`I Touch sense layer 122
`Dispiay device 112
`
`Haptic actuator
`136) A
`
`121
`
`r i , 7 7 7 , , , 7 i , , 7 7 o o , i , 7 7 i , , , ,
`
`, , 7 i 7 o , 7 , 7 7 i , 7 , , 7 , , 7 i i , , 7 , , 7 i , 7 7 , , , , 7 i 7 , 7 7 o H
`
`Micro-controller
`
`TWO
`
`Haptic profiles 4/134 1%"'ekéééééé?oek
`
`Housing 102
`
`\Operational components
`
`104
`
`Fig. 1c
`
`APPLE INC.
`EXHIBIT 1005 - PAGE 0004
`
`

`
`Patent Application Publication
`
`Jun. 24, 2010 Sheet 4 0f 21
`
`US 2010/0156818 A1
`
`Acoustic
`
`lsolator 121
`
`110
`
`Hcompound
`
`1
`
`1
`
`|
`
`H2
`
`H1
`
`Protective‘ layer 120
`Touch sense layer 122
`
`124 1
`
`a
`
`I
`
`Display device 112
`
`k140
`
`iiiiiiiiiiiiiiiiiiiiiii 2
`
`'2
`
`h
`
`\, 121
`
`f 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
`
`2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
`
`2 2 2 2 2 2 2 2 2 2 2 2 2,
`
`1 32
`\ l
`
`' Miékdéb??bhé?
`
`
`
`
`HM: . Housing 102 Haptlc pro?les 4/ . 134 ‘\
`
`
`
`
`
`Tinfo
`
`
`
`~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Processor 106
`
`\Operational components
`
`104
`
`Fig. 1D
`
`APPLE INC.
`EXHIBIT 1005 - PAGE 0005
`
`

`
`Patent Application Publication
`
`Jun. 24, 2010 Sheet 5 0f 21
`
`US 2010/0156818 A1
`
`Acoustic
`isolator
`121
`
`Touch event
`
`Protective layer 120
`Touch sense layer 122
`
`121
`
`J42
`
`Ihousing
`
`\
`140
`
`136 j
`
`1
`
`121
`
`f , , 7 , , , 7 , , 7 7 , , i , , 7 7 , 7 i V 7 i 7 , i 7 V 7 7 , 7 i , , 7 7 , 7 7 , , 7 7 , 7 7 , , , 7 , , 7 , , , 7 , , 7 7 , , i , , i 7 , 7 i V 7 i , , i 7 V 7 7 , , H
`
`Hhousing
`
`1
`i
`
`132 '\
`Micro-controiler
`_ . Hap’uc pro?les 4/ 134
`
`
`
`H9O‘
`
`
`
`Tinfo
`
`~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Processor106
`
`Housing 102
`
`\
`\Operational components
`
`104
`
`Fig. 1E
`
`APPLE INC.
`EXHIBIT 1005 - PAGE 0006
`
`

`
`Patent Application Publication
`
`Jun. 24, 2010 Sheet 6 0f 21
`
`US 2010/0156818 A1
`
`H0O : H(n1, n1, n1)
`
`i
`
`Silent area
`
`Fig; 2A ~
`
`APPLE INC.
`EXHIBIT 1005 - PAGE 0007
`
`

`
`Patent Application Publication
`
`Jun. 24, 2010 Sheet 7 0f 21
`
`US 2010/0156818 A1
`
`//
`
`s
`
`//
`
`Fig. 2B
`
`APPLE INC.
`EXHIBIT 1005 - PAGE 0008
`
`

`
`Patent Application Publication
`
`Jun. 24, 2010 Sheet 8 0f 21
`
`US 2010/0156818 A1
`
`n_ Q28
`
`4 £82
`
`corombcoo
`
`com
`
`m @E
`
`wow
`
`31%
`
`mom
`
`\l
`
`\\
`n‘‘.
`
`ll‘lllllll-lll'll
`
`APPLE INC.
`EXHIBIT 1005 - PAGE 0009
`
`

`
`Patent Application Publication
`
`Jun. 24, 2010 Sheet 9 0f 21
`
`US 2010/0156818 A1
`
`APPLE INC.
`EXHIBIT 1005 - PAGE 0010
`
`

`
`Patent Application Publication
`
`Jun. 24, 2010 Sheet 10 0f 21
`
`US 2010/0156818 A1
`
`> Sense output node
`
`504
`
`502
`
`Voltage sensed?
`
`Actuator in passive
`mode
`
`Sensed voltage greater
`than threshold?
`
`508
`
`Actuator in active
`mode
`
`510
`
`i
`
`Retrieve haptic
`profile
`
`i
`
`Generate haptic response
`by actuator based on
`retrieved haptio profile
`
`512
`
`N914
`
`518
`?
`Set to passive
`mode
`
`Haptic respone
`complete?
`
`Fig. 5
`
`APPLE INC.
`EXHIBIT 1005 - PAGE 0011
`
`

`
`Patent Application Publication
`
`Jun. 24, 2010 Sheet 11 0f 21
`
`US 2010/0156818 A1
`
`f
`
`GUI 600 \1
`
`\
`
`Device
`100
`K’
`
`Image feature
`612
`
`1 12
`
`610
`
`610
`
`610
`
`610
`
`610
`
`610
`
`Key
`Pad
`608
`
`610
`
`610
`
`610
`
`\ Slider
`
`604
`
`
`
`Button 602
`
`/ k
`
`/
`
`)
`
`\\\\ \
`
`/
`
`77/
`
`\
`
`/
`
`Fig. 6
`
`APPLE INC.
`EXHIBIT 1005 - PAGE 0012
`
`

`
`Patent Application Publication
`
`Jun. 24, 2010 Sheet 12 0f 21
`
`US 2010/0156818 A1
`
`Region 1
`
`Region 2
`
`Target
`706
`
`<— 702
`
`///Regio?{1 \\
`
`704
`
`/
`
`g’
`
`1
`
`Target
`
`70s
`
`*
`
`APPLE INC.
`EXHIBIT 1005 - PAGE 0013
`
`

`
`Patent Application Publication
`
`Jun. 24, 2010 Sheet 13 0f 21
`
`US 2010/0156818 A1
`
`60:: B 5:58 \
`
`l
`
`I
`
`
`
`29w Emowwiw
`
`mm I N: K NI K NI K .
`
`
`
`00 2256-00 £032.
`
`
`
`E99 69E 698
`
`
`
`I I 23m Ewowomsw I
`
`K K K
`
`.0 I I I @MMMMNE
`
`N 862 N8 N8 N8
`
`A / r c062 4 ,
`
`\~
`
`r :QQE 6 :2 N :062 6 K_ H _\ c089 592
`
`m m m N :052
`
`wwwwwww
`
`BECEE
`
`APPLE INC.
`EXHIBIT 1005 - PAGE 0014
`
`

`
`Patent Application Publication
`
`Jun. 24, 2010 Sheet 14 0f 21
`
`US 2010/0156818 A1
`
`ma .5
`
`
`
`3 $5520.00 5030.?
`
`f @825
`
`2%:
`
`wwcoawwh
`
`
`
`
`
`O O a I‘ .W mLiikriinz iiiiiiii E F N O a
`
`APPLE INC.
`EXHIBIT 1005 - PAGE 0015
`
`

`
`Patent Application Publication
`
`Jun. 24, 2010 Sheet 15 0f 21
`
`US 2010/0156818 A1
`
`23mm”. 06 m Q26
`
`m2 .3
`
`
`
`2 29:90.00 £030.?
`
`<2 .mm
`
`
`
`NE @552 mmmg
`
`
`
`E6065 wmum 6:865 mono
`
`
`
`
`
`56:25
`
`oznmI
`
`wwconmwh
`
`595 D6 coEwoQ
`
`APPLE INC.
`EXHIBIT 1005 - PAGE 0016
`
`

`
`Patent Application Publication
`
`Jun. 24, 2010 Sheet 16 0f 21
`
`US 2010/0156818 A1
`
`Fig. 11
`
`1100
`
`the presence of at
`least a ?rst ?nger
`and a second ?nger
`are detected
`
`1 102
`
`e presence of the
`two ‘ringers represents ~
`
`1104
`
`a gesture ‘
`
`1106
`haptic devices nearest the p
`touch point are set to active N
`mode in order to provlde a
`vibrotaetile
`
`distance between at least the A1) 08
`a two ?ngers is compared
`
`1 1 10
`
`‘ Distance increasing? 1
`
`1112
`Generate zoom in signal -/\/
`
`Generate Zoom out signal
`
`L
`
`Generate Zoom in haptic
`signal
`
`1116
`
`‘
`
`l
`
`Generate zoom out haptie p’
`signal
`
`1114
`
`1118
`
`APPLE INC.
`EXHIBIT 1005 - PAGE 0017
`
`

`
`Patent Application Publication
`
`Jun. 24, 2010 Sheet 17 0f 21
`
`US 2010/0156818 A1
`
`1202
`
`i _
`
`“1204,
`
`FIG. 12A
`
`1204 ‘
`
`~ FIG. 12B
`
`1206 ii Hm)
`
`APPLE INC.
`EXHIBIT 1005 - PAGE 0018
`
`

`
`Patent Application Publication
`
`Jun. 24, 2010 Sheet 18 0f 21
`
`US 2010/0156818 A1
`
`1 204
`
`FIG.
`
`120
`
`~ d
`
`1204
`
`1204‘ .
`
`i
`
`*\ ~
`
`‘
`
`FIG
`
`APPLE INC.
`EXHIBIT 1005 - PAGE 0019
`
`

`
`Patent Application Publication
`
`Jun. 24, 2010 Sheet 19 0f 21
`
`US 2010/0156818 A1
`
`1204
`
`FIG.
`ii 125
`
`~ 1204‘
`
`APPLE INC.
`EXHIBIT 1005 - PAGE 0020
`
`

`
`Patent Application Publication
`
`Jun. 24, 2010 Sheet 20 of 21
`
`US 2010/0156818 A1
`
`APPLE INC.
`EXHIBIT 1005 - PAGE 0021
`
`

`
`Patent Application Publication
`
`Jun. 24, 2010 Sheet 21 of 21
`
`US 2010/0156818 A1
`
`Virtlial scroll Wheel ‘
`is presented on the
`display *
`
`i
`
`' t epresence ofthe
`
`a two fingers represents
`‘
`* a gesture
`
`the initial position of the
`fingers on the virtual scroll
`wheel is set.
`
`rotate signal is generated
`when the angle of the fingers
`change relative to the
`reference point
`
`rotate signal can be used to
`rotate the Virtual scroll
`‘ Wheel in the direction of
`
`finger rotation i
`
`Generate haptic click signal
`
`APPLE INC.
`EXHIBIT 1005 - PAGE 0022
`
`

`
`US 2010/0156818 Al
`
`Jun. 24, 2010
`
`MULTI TOUCH VVITH MULTI HAPTICS
`
`CROSS REFERENCE TO RELATED
`APPI JCATIONS
`
`[0001] This patent application takes priority under 35 U.S.
`C. ll9(e) to U.S. Provisional Application Ser. No. 61/140,
`519 entitled MULTI TOUCH WITH MULTI HAPTICS by
`Burrough et al., filed Dec. 23, 2008 which is incorporated by
`reference in its entirety for all purposes.
`
`BACKGROUND OF THE IN VEN TIDN
`
`. Field of the Invention
`I
`[0002]
`[0003] The present invention relates generally to providing
`multi-touch/multi-haptic systems and methods.
`[0004]
`2. Description of the Related Art
`[0005] Multi-toucl1 devices l1ave advantages over co11ven-
`tional single point sensing touch devices in that they can
`distinguish more than one object (finger) in contrast to single
`point devices that are simply incapable ofdistinguishing mul-
`tiple objects. In most cases, multi-toucl1 devices monitor a
`sensing surface for a touch or near touch, and when a touch
`occurs determines the distinct areas of contact and identifies
`the contacts via their geometric features and geometric
`arrangement. Once identified or classified, the contacts are
`monitored for various motions, actions or events. The con-
`tacts and motions thereof are then converted into inputs for
`controlling some aspect of an electronic device.
`[0006] Multi-touch devices can be embodied in various
`forms including but not limit to standard touch pads, large
`extended palm pads, touch screens, touch sensitive housings,
`etc. Furthermore, multi—touch devices can be placed in vari-
`ous electronic devices including but not lir11ited to computers
`such as tablet computers, laptop computers, desktop comput-
`ers as well as handheld computing devices such as media
`players (e.g., music, video, games), PDAs, cell phones, cam-
`eras, remote controls, and/or the like. The multi—touch devices
`can also be placed on dedicated input devices such as touch
`screen monitors, keyboards, navigation pads, tablets, mice,
`and the like. Essentially, multi—touch devices can be applied
`to any surface, and can be found in any consumer electronic
`product that requires inputs.
`[0007]
`Since multi—touch devices provide a number of
`inputting operations at a single location (input surface), i11p ut-
`ting with multi—touch devices can be very efficient. The user
`can maintain their hand(s) at the multi—touch surface without
`having to move their hand(s) to address other input devices.
`For example, conventional systems typically include a key-
`board and a separate mouse. In order to use the mouse, the
`user must move their hand from the keyboard and onto the
`mouse. In order to keyboard efficiently (both hands), the user
`must move their hand from the mouse to the keyboard. This
`inputting sequence is very inefficient. For one, only one
`device can be used effectively at a given time. For another,
`there is wasted time between each inputting step. In contrast,
`with multi—touch surfaces the user can generate both static
`commands (e.g., keyboarding) and manipulative commands
`(e.g., tracking) from the same location and at tl1e same time.
`The user therefore does not have to move their hands to
`perform different inputting tasks. The user simply provides
`different chords or finger motions to generate a number of
`inputs either sequentially or simultaneously. In one example,
`the user car1 provide key co111n1ands witl1 taps at specific
`
`locations of the multi—touch surface while allowing tracking
`from all locations of the multi—touch surface.
`[0008] However, research has shown that providing the
`multi—touch surface with the ability to provide physical (hap-
`tic) feedback makes the multi—touch experience even more
`efficient and realistic to the user. For example, physical key-
`boards provide a physical indication (a bump, for example)
`indicative of tl1e home key. This physical sensation can not be
`provided by a conventional multi—touch system thereby forc-
`ing the user to visually locate the home key thereby making
`keyboard use less efficient and fatiguing. However, by pro-
`viding a physical facsimile of the home key bump using an
`actuator that provides a physical sensation to the user provid-
`ing an approximate representation of the bump, the user’s
`experience ofthe MT keyboard (and any multi—touch system
`for that matter) can be more realistic and therefore more
`enjoyable.
`[0009] Therefore, a system that enhances the multi—touch
`experience by incorporating a corresponding physical
`response(s) is described.
`
`SUMMARY OF THE INVENTION
`
`in one embodiment, to an
`[0010] The invention relates,
`apparatus and method for providing 1nulti—touch haptic feed-
`back. The apparatus includes, at least, a touch pad having a
`touch sensitive surface arranged to receive a user provided
`multi—touch event associated with at least two different loca-
`tions on the touch sensitive surface, a multi—touch detection
`mechanism operatively coupled to the touch sensitive surface
`that detects tl1e multi-toucl1 event and generates a correspond-
`ing a multi—touch signal, and a plurality of haptic feedback
`devices operatively coupled to the multi—touch detection
`mechanism and the touch sensitive surface cooperatively
`arranged to concurrently provide tactile feedback at each of
`tlie at least two different locations on tl1e touch sensitive
`surface in response to the multi—touch signal wherein the
`tactile feedback at each of the at least two different locations
`are discreet from one another. When the multi—touch signal
`indicates that tl1e multi—touch event is a dynamic multi—touch
`event indicating a change in the multi—touch event, then the
`tactile feedback at each of the at least two different locations
`is updated to reflect the change in the multi—touch event.
`[0011]
`It should be noted that in some cases the tactile
`feedback event ca11 be different for each of tlie at least two
`different locations.
`[0012] The invention relates, in another embodiment, to a
`1nulti—touch haptic mechanism. The multi—touch haptic
`mechanism includes, at least, a touch pad having a touch
`sensitive surface arranged to detect a user touch event at
`substantially any location on the touch sensitive surface and a
`plurality ofindependent haptic devices operatively coupled to
`the touch sensitive surface each providing a corresponding
`type of tactile feedback thereby providing a tactile feedback
`at substantially any location on tl1e touch sensitive surface at
`which the user touch event has occurred, wherein each of the
`plurality of independent haptic devices only responds to the
`user touch event in one or more associated regions of the
`touch sensitive surface. When at least two of the plurality of
`independent haptic devices cooperate to provide a type of
`haptic response that is different than that type provided by
`either of the at least two independent haptic devices sepa-
`rately.
`[0013] The invention relates, in another embodiment, to an
`integrated device arranged to act as both a force sensing
`
`APPLE INC.
`EXHIBIT 1005 - PAGE 0023
`
`

`
`US 2010/0156818 Al
`
`Jun. 24, 2010
`
`device and a haptic feedback device. The device includes, at
`least, a touch sensitive surface, a controller unit, and a
`mechanical actuator coupled with the controller unit and the
`touch sensitive surface. The integrated device acts as the force
`sensing device by generating an output voltage in direct pro-
`portion to a force applied to the mechanical actuator by a user
`touching the touch sensitive surface, sensing the output volt-
`age by the controller unit and converting the sensed output
`voltage to an indication of the applied force. Only when the
`sensed output voltage exceeds a voltage threshold level does
`the integrated device act as the haptic feedback device by
`halting the sensing ofthe output voltage by the controller unit
`activating the mechanical actuator by the controller unit,
`wherein the activated mechanical actuator imparts a physical
`force to the touch sensitive surface that results in a vibro-
`tactile response (subcutaneous tissue activated) felt by the
`user con1rnensurate with the force applied by the user.
`[0014] The invention relates, in another embodiment, to an
`electronic device. The electronic device includes, at least, a
`touch pad having a touch sensitive surface arranged to process
`a user touch event and a plurality of haptic feedback devices
`each of which is operatively coupled to the touch sensitive
`surface and each responding to the user touch event only in a
`specific region of the touch sensitive surface and arranged to
`provide tactile feedback singly or in combination with others
`of the plurality of haptic feedback devices in response to the
`user touch event. When the touch sensitive regions of at least
`two ofthe plurality ofhaptic devices overlap, ifthe user touch
`event occurs in the overlapping region, then the at least two
`haptic devices cooperate to provide a combined haptic feed-
`back response based upon the location in the overlapping
`region of the user touch event.
`[0015] The invention relates, ir1 another embodiment, to ar1
`electronic device. The electronic device includes, at least, a
`touch pad having a touch sensitive surface arranged to receive
`a user touch event provided by a user, a controller coupled and
`in commtmication with the touch pad arranged to at least
`analyze the user touch event and/or a state of the touch pad
`and based upon the analysis provide a user touch event signal
`in response to the user touch event, and at least one haptic
`device operatively coupled to the controller arranged to
`receive the user touch event signal, wherein the at lea st one
`haptic device responds to the user touch event signal by
`providing an appropriate haptic feedback response to the user
`based upon the analysis provided by the controller.
`[0016]
`In one embodiment, the touch sensitive surface is
`arranged to receive different types of user touch events each
`being characterized by an amount of pressure applied on the
`touch sensitive surface by a user and at least one haptic device
`operatively coupled to the touch sensitive surface arranged to
`provide a specific type of tactile feedback corresponding to
`the amount of pressure applied to the touch sensitive surface
`by the user.
`[0017]
`It should be noted that in each of the embodiments
`described above, the methods can be implemented using a
`touch based input device such as a touch screen or touch pad,
`more particularly a multi-touch touch based input device, and
`even more particularly a multi-touch touch screen. It should
`also be noted that the gestures, gesture modes, gestural inputs,
`etc. can correspond to any of those described below in the
`detailed description. For example, the gestures can be asso-
`
`ciated with zooming, parming, scrolling, rotating, enlarging,
`floating controls, zooming targets, paging, inertia, keyboard-
`ing, wheeling, and/or the like.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`[0018] The invention will be readily understood by the
`following detailed description in conjunction with the accom-
`panying drawings, wherein like reference numerals designate
`like structural elements, and in which:
`[0019]
`FIGS. 1A—1E are a series of block diagrams of a
`system, in accordance with one embodiment of the present
`invention.
`[0020]
`FIGS. 2A-2B shows a multi-point multi-haptic sys-
`tem having a multi-touch surface that incorporates a plurality
`of haptic devices in accordance with an embodiment of the
`invention.
`[0021]
`FIG. 3 shows a schematic diagram ofa representa-
`tive piezo-electric haptic assembly.
`[0022]
`FIG. 4 shows a schematic diagram of the haptic
`assembly shown in FIG. 3 configured to act as a pressure
`sensor.
`
`FIG. 5 shows a flowchart detailing a process in
`[0023]
`accordance with an embodiment of the invention.
`[0024]
`FIG. 6 shows display device displaying representa-
`tive haptic active GUI elements in accordance with an
`embodiment of the invention.
`[0025]
`FIG. 7 shows representative GUI button elements in
`accordance with an embodiment of the invention.
`
`FIGS. 8A-8B shows representative GUI button ele-
`[0026]
`ment and associated haptic profile in accordance with an
`embodiment of the invention.
`[0027]
`FIGS. 9A—9B shows a representative slider element
`and associated haptic profile in accordance with an embodi-
`ment of the invention.
`[0028]
`FIGS. 10A-10B shows a feature edge detection sys-
`tem in accordance with an embodiment of the invention.
`
`FIG. 11 is a diagram ofa zoom gesture method 1 1 00
`[0029]
`in accordance with an embodiment of the invention.
`[0030]
`FIGS. 12A—12H illustrates a display presenting a
`GUI object in the form of a map of North America with
`embedded levels which can be zoomed.
`[0031]
`FIG. 13 is a diagram ofa GUI operational method i11
`accordance with one embodiment of the present invention.
`
`DETAILED DESCRIPTION OF SELECTED
`EMBODIMENTS
`
`[0032] Reference will now be r11ade in de ail to selected
`embodiments an example ofwhich is illustrated in the accom-
`panying drawings. While the invention will be described in
`conjunction with a preferred embodiment, it will be under-
`stood that it is not intended to limit the invention to one
`preferred embodiment. To the contrary, it is intended to cover
`alternatives, modifications, and equivalents as can be
`included within the spirit and scope of the invention as
`defined by the appended claims.
`[0033] The invention relates to multi-touch haptic feed-
`back. Multi-touch haptic feedback refers to haptic techniques
`capable of providing r11ultiple and discretely located haptic
`sensations across a surface. The haptic system can for
`example include a plurality of haptic nodes, each of which is
`capable of issuing vibro—tactile sensations (at the same time
`or different times and with the same inte11sity or different
`intensity). The haptic nodes can for example be configured i11
`
`APPLE INC.
`EXHIBIT 1005 - PAGE 0024
`
`

`
`US 2010/0156818 A1
`
`Jun. 24, 2010
`
`a matrix or array. In one embodiment, the haptic nodes are
`mapped to touch sensing nodes. Each touch sensing node can
`be assigned o11e or more haptic nodes. The haptic nodes are
`typically proximate the touch sensing nodes to which it has
`been assigned.
`[0034]
`In one embodiment, the touch sensing surface is a
`multi touch surface thus making a multi touch multi-touch
`haptic device. In so doing haptic feedback can be provided
`that indicates information about a multi touch event. For
`example, the surface under a moving fmger can be actuated
`while the surface under the non-moving finger remains static.
`In another example, the surface under the moving finger is
`actuated concurrently with a signal being passed to the other
`finger indicating that a multi touch action is occurring. In this
`way, the signals taken together can indicate the nature of the
`underlying action being taken by the user. For example, if an
`object (such as an image) is being expanded or reduced in size
`a larger/more intense signal could be generated (either by
`increasing frequency or amplitude). It is also contemplated
`that isolated feedback ca11 be used to provide an 011-screen
`click-wheel or other such user input where the touch screen is
`used to simulate the “clicks” of the click wheel both audibly
`and via tactile feedback.
`
`[0035] The described embodiments generally pertain to
`gestures and methods of implementing gestures with associ-
`ated physical
`feedback with touch sensitive devices.
`Examples of touch sensitive devices include touch screens
`and touch pads. One aspect ofthe invention describes a touch
`sensitive input device able to recognize at least two substan-
`tially simultaneously occurring gestures using at least two
`different fingers or other objects (hereinafter referred to as a
`multi-touch event). The touch sensitive input device commu-
`nicates with an array ofhaptic feedback devices (also referred
`to as haptic actuators) each arranged to provide haptic feed-
`back according to a haptic profile in response to a m11lti-touch
`event. In another aspect of the invention, each finger receives
`different haptic feedback (multi-haptic) depending upon the
`location on the touch sensitive input device each finger is
`placed. In another aspect ofthe invention. a compound haptic
`feedback can be provided that combines the output from at
`least two different haptic actuators to form the compound
`response that is different from that provided by the two origi-
`nating haptic actuators. In another embodiment, an integrated
`device is described that can act as both a force sensing device
`and a haptic feedback device. In still another embodiment, a
`handheld portable device is described having a housing and a
`user interface are acoustically isolated from each other. In this
`way, the housing and user interface and having non—interfer—
`ing and independent haptic responses.
`[0036] These and other aspects of the invention are dis-
`cussed below with reference to FIGS. 1-13. However, those
`skilled in the art will readily appreciate that the detailed
`description given herein with respect to these figures is for
`explanatory purposes as the invention extends beyond these
`limited embodiments.
`
`FIG. 1A-1 E are block diagrams ofa representative
`[0037]
`electronic device or system 100,
`in accordance with one
`embodiment of the present invention. Electronic device 100
`can correspond to a computer (such as a desktops or laptops)
`as well as small form factor electronic devices that can
`include portable consumer electronic products such as cell
`phones, PDA, media players and/orthe like.As such, portable
`electronic device 100 can be sized for one-handed operation
`and placement i11to small areas such as a pocket. Portable
`
`electronic device 100 can process data and more particularly
`media such as audio, video, images, etc. As such. the portable
`electronic device 100 can correspond to a music player, game
`player, video player, personal digital assistant (PDA), such as,
`for example, an iPodTM, an iPod NanoTM, an iPod ShuffieTM,
`an iPodTM Touch or an iPhoneTM available by Apple Inc. of
`Cupertino, Calif. In some cases, portable electronic device
`100 can communicate wirelessly (with or without the aid of a
`wireless enabling accessory system) and/or via wired path-
`ways (e.g., using traditional electrical wires).
`[0038]
`Portable electronic device 100 includes a housing
`102. Housing 102 can be formed of any munber of materials
`including for example plastics, metals, ceramics and the like.
`In one embodiment, housing 102 can be formed of stainless
`steel in order to provide an aesthetic and appealing look and
`feel as well as provide structural integrity and support for all
`sub-assemblies installed therein. Ilousing 102 can define a
`cavity configured to at least partially enclose any suitable
`number of operational electronic components 104 used by
`portable electronic device 100 to carry out its intended fur1c-
`tions. Operational electronic components 104 can include
`processor 106 that can operate (in conjunction with an oper-
`ating system) to execute computer code and produce and use
`data. Processor 106 can be implemented on a single-chip,
`multiple chips or multiple electrical components. For
`example, various architectures can be used for the processor
`106, including dedicated or embedded processor, single pur-
`pose processor, controller, ASIC, and so forth. The operating
`system, other computer code and data can reside within a
`memory 108 that can be operatively coupled to processor 106.
`By way of example, memory 108 can include Read-Only
`Memory (ROM), Random—Access Memory (RAM), flash
`memory. hard disk drive and’or the like. Operational compo-
`nents 104 can also include a number of input/output (I/O)
`devices 109. Such devices can include audio output devices
`such as headphone jacks, data ports (such as I.E.E.E. 1392
`compliant, USB, etc.), and so on.
`[0039]
`Portable electronic device 100 can also include a
`user interface ll0 that can operate to both receive user inputs
`and provide information to a user. In the described embodi-
`ment, user interface 110 can include display device 112 that
`can be operatively coupled to processor 106 by way of bus
`114. Display device 112 cancorrespond to any known display
`teclmology such as a plasma, LCD, or an organic light emit-
`ting diode (OLED). It should be noted that in the embodiment
`shown in FIGS. 1A-1E, display device 112 is integrated with
`the electronic device 100. However, display device 112 ca11
`also be configured as a component separate from portable
`electronic device 100 in which case display device 112 would
`be considered a peripheral device that can be coupled to
`portable electronic device 100 byway of a wired comiection
`(such as a peripheral bus or cable) or a wireless connection
`such as IR, RF, Bluetooth or the like (among others).
`[0040]
`In some cases, display device 112 presents graphical
`user interface (GUI) 116 on display device 112. GUI 116 ca11
`provide an easy to use interface between a user of portable
`electronic device 100 and the operating system or application
`running thereon. Generally speaking, GUI 116 iconically
`represents programs,
`files and operational options with
`graphical images. The graphical images can include win-
`dows, fields, dialog boxes, menus, icons, buttons, cursors,
`scroll bars, etc. Such images can be arranged in predefined
`layouts, or can be created dynamically to serve the specific
`actions being taken by a user. During operation, the user ca11
`
`APPLE INC.
`EXHIBIT 1005 - PAGE 0025
`
`

`
`US 2010/0156818 Al
`
`Jun. 24, 2010
`
`select and activate various graphical images in order to ini-
`tiate functions and tasks associated therewith. By way of
`example, a user can select a button that opens, closes, mini-
`mizes, or maximizes a window, or an icon that latmches a
`particularprogram. GUI 116 can additionally or alternatively
`display information, such as non interactive text and graphics,
`for the user on the display device 112.
`[0041] As shown more clearly in FIG. 1B in a side view
`perspective of device 100, user interface 110 can include
`protective layer 120 disposed on top of display device 112. In
`this way, protective layer 120 can be used as protective top
`layer of transparent or semitransparent material
`(clear)
`thereby affording display device 112 protection from poten-
`tially damaging external insults caused by, for example, sharp
`objects, dropping, and so on and yet still allow any image
`presented by display device 112 to be clearly viewed by a
`user. Protective layer 120 can be formed of many well known
`transparent materials such as glass (e.g., referred to as cover
`glass), and more particularly highly polished glass. It should
`be appreciated, however, that other transparent materials (or
`at least translucent materials) such as clear plastic may also be
`used. In some embodiments, protective top layer 120 can be
`acoustically isolated from housing 102 using, for example,
`acoustic isolation buffers 121. By acoustically isolating hous-
`irrg 102 and protective top layer 120 from each other, it is
`possible to provide separate and independent haptic
`rcsponscs, onc directed at housing 102 and another directed at
`protective top layer 120 without interfering with each other.
`For example, it may be desirable to provide one type ofhaptic
`response at protective layer 120 and another type haptic
`response at housing 102 at the same time or at a different time
`independent of each other or in some cases one being the
`result of or related to the other.
`
`[0042] User interface 110 can be touch sensitive suitable
`for receiving one or more user touch events by which infor-
`mation can be passed between the user and the portable elec-
`tronic device 100. In some cases, the one or more inputs in the
`form ofuser touch events can be substantially simultaneously
`received (e.g., multi-touch).
`In these embodiments, user
`interface 110 is rendered touch sensitive by means of a touch
`sensing layer 122 that can be disposed below protective layer
`120 such that touch sensing layer 122 is between protective
`layer 120 and the display device 1 12. This arrangement c