aaRiitanifibutoinititiiiiimmanommimmirmumuitunilBIRIMINUIIII11111111111111111111IIIIIIIEDMMEMEDIMMIMEMIIII1
`
`1"- (cid:9)
`
`..inmmummumunumunninnmumnimmumucanumuumumimiumatiiiiimuirinmuutunaultounlIMIMMIIMEIHOMITIIIIII
`
`1W. (cid:9)
`-4t)
`
`110),VILL 11011MONVITIVERTt FaCESEIVIESt
`
`uometp,
`
`UNITED STATES DEPARTMENT OF COMMERCE
`United States Patent and Trademark Office
`
`December 08, 2015
`
`THIS IS TO CERTIFY THAT ANNEXED HERETO IS A TRUE COPY FROM
`THE RECORDS OF THIS OFFICE OF:
`
`U.S. PATENT: 8,659,571
`ISSUE DATE: February 25, 2014
`
`By Authority of the
`Under Secretary of Commerce for Intellectual Property
`and Director of the United States Patent and Trademark Office
`
`T. LAWRENCE
`Certifying Officer
`
`APPLE INC.
`EXHIBIT 1001 - PAGE 0001
`
`(cid:9)
`

`

`1111111111111111111111111491111111111111111111111111111111111111
`
`(12) United States Patent
`Birnbaum et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 8,659,571 B2
`*Feb. 25, 2014
`
`(54)
`
`MODEL FOR SHARED
`INTERACTIVITY
`ON MOBILE DEVICES
`FEEDBACK
`
`(71)
`
`Applicant.
`
`Immersion Corporation, San Jose, CA
`(US)
`
`(72)
`
`Inventors:
`
`David Birnbaum, San Jose, CA (US);
`Chris Ullrich, San Jose, CA (US); Jason
`Short, San Francisco, CA (US); Ryan
`Devenish, San Francisco, CA (US)
`
`(73)
`
`Assignee.
`
`Immersion Corporation, San Jose, CA
`(US)
`
`( * )
`
`Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`
`This patent is subject to a terminal dis-
`claimer.
`
`(21) Appl. No.: 13/773,191
`
`(22) Filed: (cid:9)
`
`Feb. 21, 2013
`
`(65)
`
`Prior Publication Data
`
`US 2013/0300683 Al (cid:9)
`
`Nov. 14, 2013
`
`Related U.S. Application Data
`
`(63) Continuation of application No. 13/592,685, filed on
`Aug. 23, 2012, now Pat. No. 8,493,354.
`
`(2006.01)
`
`(51) Int. Cl.
`G06F 3/041 (cid:9)
`(52) U.S. Cl.
`USPC (cid:9)
`(58) Field of Classification Search
`USPC (cid:9)
` 345/156-184; 178/18.01-18.09, 18.11;
`340/4.12, 407.1, 407.2; 463/30
`See application file for complete search history.
`
` 345/173; 340/407.2
`
`(56) (cid:9)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`9/1997 (cid:9) Baudel et al. (cid:9)
`5,666,499 A * (cid:9)
`5,825,308 A * (cid:9) 10/1998 (cid:9) Rosenberg (cid:9)
`6,061,004 A * (cid:9)
`5/2000 (cid:9) Rosenberg (cid:9)
`6,088,019 A * (cid:9)
`7/2000 (cid:9) Rosenberg (cid:9)
`6,100,874 A * (cid:9)
`8/2000 (cid:9) Schena et al. (cid:9)
`6,166,723 A * (cid:9) 12/2000 (cid:9) Schena et al. (cid:9)
`6,211,861 B1* (cid:9)
`4/2001 (cid:9) Rosenberg et al. (cid:9)
`6,252,579 B1 * (cid:9)
`6/2001 (cid:9) Rosenberg et al. (cid:9)
`6,300,936 B1 * (cid:9) 10/2001 (cid:9) Braun et al. (cid:9)
`6,337,678 B1 * (cid:9)
`1/2002 (cid:9) Fish (cid:9)
`6,429,846 B2 * (cid:9)
`8/2002 (cid:9) Rosenberg et al. (cid:9)
`6,448,977 B1 * (cid:9)
`9/2002 (cid:9) Braun et al. (cid:9)
`6,717,573 B1 * (cid:9)
`4/2004 (cid:9) Shahoian et al. (cid:9)
`6,819,312 B2* (cid:9) 11/2004 (cid:9) Fish (cid:9)
`7,024,625 B2 * (cid:9)
`4/2006 (cid:9) Shalit (cid:9)
`7,084,854 B1 * (cid:9)
`8/2006 (cid:9) Moore et al. (cid:9)
`7,088,342 B2 * (cid:9)
`8/2006 (cid:9) Rekimoto et al. (cid:9)
`7,113,177 B2* (cid:9)
`9/2006 (cid:9) Franzen (cid:9)
`(Continued)
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`715/808
`
`341/20
`
`341/20
`345/156
`345/157
`345/184
`345/163
`715/856
`345/156
`345/156
`345/156
`715/701
`345/161
`345/156
`715/702
`345/157
`345/169
`345/173
`
`Primary Examiner — Stephen Sherman
`(74) Attorney, Agent, or Firm — Thomas A. Hassing
`
`ABSTRACT
`(57) (cid:9)
`A system that produces a dynamic haptic effect and generates
`a drive signal that includes a gesture signal and a real or
`virtual device sensor signal. The haptic effect is modified
`dynamically based on both the gesture signal and the real or
`virtual device sensor signal such as from an accelerometer or
`gyroscope, or by a signal created from processing data such as
`still images, video or sound. The haptic effect may optionally
`be modified dynamically by using the gesture signal and the
`real or virtual device sensor signal and a physical model, or
`may optionally be applied concurrently to multiple devices
`which are connected via a communication link. The haptic
`effect may optionally be encoded into a data file on a first
`device. The data file is then communicated to a second device
`and the haptic effect is read from the data file and applied to
`the second device.
`
`30 Claims, 15 Drawing Sheets
`
`15
`
`10 /
`
`13
`
`O
`
`12
`
`PROCESSOR
`
`20 (cid:9)
`
`22
`
`OUTPUT DEVICE
`DRIVE
`MODULE
`
`MEMORY
`
`16 (cid:9)
`
`18
`
`OUTPUT DEVICE
`DRIVE CIRCUIT
`
`OUTPUT
`DEVICE
`
`APPLE INC.
`EXHIBIT 1001 - PAGE 0002
`
`(cid:9)
`(cid:9)
`(cid:9)
`(cid:9)
`(cid:9)
`(cid:9)
`(cid:9)
`(cid:9)
`(cid:9)
`(cid:9)
`(cid:9)
`(cid:9)
`(cid:9)
`(cid:9)
`(cid:9)
`(cid:9)
`(cid:9)
`(cid:9)
`(cid:9)
`

`

`US 8,659,571 B2
`Page 2
`
`(56) (cid:9)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`7,205,978 B2 * 4/2007 Poupyrev et al. (cid:9)
`7,336,260 B2 * 2/2008 Martin et al. (cid:9)
`7,446,456 B2* 11/2008 Maruyama et al. (cid:9)
`7,456,823 B2 * 11/2008 Poupyrev et al. (cid:9)
`7,468,573 B2 * 12/2008 Dai et al. (cid:9)
`7,528,508 B2*
`5/2009 Bruwer (cid:9)
`7,554,246 B2* 6/2009 Maruyama et al. (cid:9)
`7,592,999 B2 *
`9/2009 Rosenberg et al. (cid:9)
`7,663,604 B2 *
`2/2010 Maruyama et al. (cid:9)
`7,755,607 B2 *
`7/2010 Poupyrev et al. (cid:9)
`7,808,488 B2 * 10/2010 Martin et al. (cid:9)
`7,821,498 B2 * 10/2010 Kramer et al. (cid:9)
`7,825,903 B2 * 11/2010 Anastas et al. (cid:9)
`RE42,064 E * 1/2011 Fish (cid:9)
`7,890,863 B2 *
`2/2011 Grant et al. (cid:9)
`7,920,131 B2* 4/2011 Westerman (cid:9)
`7,924,144 B2* 4/2011 Makinen et al. (cid:9)
`7,969,288 B2 * 6/2011 Braun et al. (cid:9)
`7,973,769 B2 *
`7/2011 Olien (cid:9)
`7,978,181 B2 *
`7/2011 Westerman (cid:9)
`7,982,588 B2 *
`7/2011 Makinen et al. (cid:9)
`7,982,720 B2 *
`7/2011 Rosenberg et al. (cid:9)
`8,004,492 B2* 8/2011 Kramer et al. (cid:9)
`8,031,181 B2* 10/2011 Rosenberg et al. (cid:9)
`8,035,623 B2 * 10/2011 Bruwer (cid:9)
`8,059,105 B2 * 11/2011 Rosenberg et al. (cid:9)
`8,098,235 B2 *
`1/2012 Heubel et al. (cid:9)
`2001/0035854 Al * 11/2001 Rosenberg et al. (cid:9)
`2002/0044132 Al * 4/2002 Fish (cid:9)
`2002/0177471 Al * 11/2002 Kaaresoja et al. (cid:9)
`2003/0063128 Al * 4/2003 Salmimaa et al. (cid:9)
`2003/0100969 Al *
`5/2003 Jones (cid:9)
`2003/0162595 Al *
`8/2003 Serbanescu (cid:9)
`2003/0206202 Al * 11/2003 Moriya (cid:9)
`2004/0002902 Al *
`1/2004 Muehlhaeuser (cid:9)
`2005/0057528 Al * 3/2005 Kleen (cid:9)
`2005/0179617 Al *
`8/2005 Matsui et al. (cid:9)
`2005/0212760 Al * 9/2005 Marvit et al. (cid:9)
`2005/0245302 Al * 11/2005 Bathiche et al. (cid:9)
`2006/0022952 Al * 2/2006 Ryynanen (cid:9)
`2006/0026536 Al * 2/2006 Hotelling et al. (cid:9)
`2006/0061545 Al * 3/2006 Hughes et al. (cid:9)
`2006/0097991 Al *
`5/2006 Hotelling et al. (cid:9)
`2006/0119586 Al * 6/2006 Grant et al. (cid:9)
`2006/0181510 Al *
`8/2006 Faith (cid:9)
`2006/0192760 Al *
`8/2006 Moore et al. (cid:9)
`2006/0197752 Al * 9/2006 Hurst et al. (cid:9)
`2006/0255683 Al * 11/2006 Suzuki et al. (cid:9)
`2006/0256074 Al * 11/2006 Krum et al. (cid:9)
`2006/0279476 Al * 12/2006 Obata (cid:9)
`2006/0279542 Al * 12/2006 Flack et al. (cid:9)
`2006/0284849 Al * 12/2006 Grant et al. (cid:9)
`2007/0066283 Al * 3/2007 Haar et al. (cid:9)
`2007/0139366 Al * 6/2007 Dunko et al. (cid:9)
`2007/0150826 Al * 6/2007 Anzures et al. (cid:9)
`2007/0152984 Al *
`7/2007 Ording et al. (cid:9)
`2007/0236450 Al * 10/2007 Colgate et al. (cid:9)
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`345/157
`345/169
`310/328
`345/173
`310/317
`307/140
`310/338
`345/156
`345/173
`345/156
`345/169
`345/163
`345/173
`345/156
`715/702
`345/173
`340/407.2
`340/407.1
`345/168
`345/173
`340/407.2
`345/173
`345/156
`345/173
`345/173
`345/173
`345/173
`345/156
`345/156
`455/567
`345/810
`700/97
`472/1
`345/846
`705/26
`345/173
`345/7
`345/156
`463/1
`345/173
`715/863
`345/156
`345/173
`345/173
`345/158
`345/163
`345/173
`310/317
`345/156
`345/2.3
`345/158
`345/173
`455/412.2
`345/156
`715/772
`345/173
`345/156
`
`345/168
`
`2007/0247429 Al * 10/2007 Westerman (cid:9)
`345/173
`
`2007/0247442 Al * 10/2007 Andre et al. (cid:9)
`463/42
`
`2007/0265096 Al * 11/2007 Kouno et al. (cid:9)
`345/173
`
`2007/0279392 Al * 12/2007 Rosenberg et al. (cid:9)
`345/173
`
`2008/0024459 Al *
`1/2008 Poupyrev et al. (cid:9)
`345/177
`
`2008/0055277 Al * 3/2008 Takenaka et al. (cid:9)
` 178/18.03
`2008/0060856 Al * 3/2008 Shahoian et al. (cid:9)
`
`345/156
`2008/0068334 Al * 3/2008 Olien et al. (cid:9)
`
`345/156
`2008/0088580 Al * 4/2008 Poupyrev et al. (cid:9)
`
`345/156
`2008/0111788 Al *
`5/2008 Rosenberg et al. (cid:9)
`
`345/173
`2008/0180406 Al *
`7/2008 Han et al. (cid:9)
`
`715/763
`2008/0216001 Al * 9/2008 Ording et al. (cid:9)
`
`455/466
`2008/0287147 Al * 11/2008 Grant et al. (cid:9)
`
`463/39
`2008/0300055 Al * 12/2008 Lutnick et al. (cid:9)
`
`345/156
`2008/0303782 Al * 12/2008 Grant et al. (cid:9)
`
`345/173
`2009/0002328 Al *
`1/2009 Ullrich et al. (cid:9)
` 340/407.2
`2009/0079550 Al * 3/2009 Makinen et al. (cid:9)
`
`345/173
`2009/0085878 Al * 4/2009 Heubel et al. (cid:9)
` 340/407.2
`2009/0109007 Al * 4/2009 Makinen et al. (cid:9)
`
`345/173
`2009/0128503 Al *
`5/2009 Grant et al. (cid:9)
` 455/556.1
`2009/0137269 Al *
`5/2009 Chung (cid:9)
` 178/18.04
`2009/0166098 Al *
`7/2009 Sunder (cid:9)
` 340/407.2
`2009/0167508 Al *
`7/2009 Fadell et al. (cid:9)
` 340/407.2
`2009/0167509 Al *
`7/2009 Fadell et al. (cid:9)
`
`345/173
`2009/0167704 Al *
`7/2009 Terlizzi et al. (cid:9)
` 178/18.03
`2009/0250267 Al * 10/2009 Heubel et al. (cid:9)
`
`345/174
`2009/0256817 Al * 10/2009 Perlin et al. (cid:9)
`
`455/73
`2009/0270046 Al * 10/2009 Lai (cid:9)
`
`345/173
`2009/0284485 Al * 11/2009 Colgate et al. (cid:9)
`
`345/168
`2009/0315830 Al * 12/2009 Westerman (cid:9)
`
`340/669
`2010/0013653 Al *
`1/2010 Birnbaum et al. (cid:9)
`
`345/156
`2010/0013761 Al *
`1/2010 Birnbaum et al. (cid:9)
`
`709/206
`2010/0017489 Al *
`1/2010 Birnbaum et al. (cid:9)
`
`715/863
`2010/0017759 Al *
`1/2010 Birnbaum et al. (cid:9)
`
`345/173
`2010/0045619 Al *
`2/2010 Birnbaum et al. (cid:9)
` 340/407.2
`2010/0085169 Al * 4/2010 Poupyrev et al. (cid:9)
` 178/18.03
`2010/0108408 Al *
`5/2010 Colgate et al. (cid:9)
`
`340/3.1
`2010/0127819 Al *
`5/2010 Radivojevic et al. (cid:9)
`
`345/179
`2010/0149134 Al *
`6/2010 Westerman et al. (cid:9)
`
`345/173
`2010/0156818 Al *
`6/2010 Burrough et al. (cid:9)
`
`345/173
`2010/0214243 Al *
`8/2010 Birnbaum et al. (cid:9)
`2010/0231539 Al *
`9/2010 Cruz-Hernandez et al. .. 345/173
`2010/0245254 Al *
`9/2010 Olien et al. (cid:9)
`
`345/173
`2010/0265208 Al * 10/2010 Kim et al
`
`345/174
`2010/0313124 Al * 12/2010 Privault et al. (cid:9)
`
`715/702
`2010/0328053 Al * 12/2010 Yeh et al. (cid:9)
` 340/407.2
`2011/0021272 Al *
`1/2011 Grant et al. (cid:9)
`
`463/30
`2011/0025609 Al *
`2/2011 Modarres et al. (cid:9)
`
`345/173
`2011/0043454 Al *
`2/2011 Modarres et al. (cid:9)
`
`345/173
`2011/0043527 Al *
`2/2011 Ording et al. (cid:9)
`
`345/428
`2011/0102340 Al *
`5/2011 Martin et al. (cid:9)
`
`345/173
`2011/0105103 Al *
`5/2011 Ullrich (cid:9)
`
`455/420
`2011/0109588 Al *
`5/2011 Makinen et al. (cid:9)
`
`345/174
`2011/0138277 Al * 6/2011 Grant et al. (cid:9)
`
`715/702
`2011/0260988 Al * 10/2011 Colgate et al. (cid:9)
`
`345/173
`2011/0264491 Al * 10/2011 Birnbaum et al. (cid:9)
` 705/14.4
`2012/0068957 Al * 3/2012 Puskarich et al. (cid:9)
`
`345/174
`2012/0081276 Al * 4/2012 Ullrich et al. (cid:9)
`
`345/156
`2012/0105333 Al *
`5/2012 Maschmeyer et al. (cid:9)
`
`345/173
`
`* cited by examiner
`
`APPLE INC.
`EXHIBIT 1001 - PAGE 0003
`
`(cid:9)
`

`

`U.S. Patent (cid:9)
`
`Feb. 25, 2014 (cid:9)
`
`Sheet 1 of 15 (cid:9)
`
`US 8,659,571 B2
`
`V ""
`
`6
`
`LL
`
`/AAA
`AAAA
`
`VVVV
`
`w
`O t
`5 D W °
`O C:
`H 0
`D Ili
`Cl. >
`I— —
`D Cr
`O a
`
`PROCESSOR
`
`0
`(N
`
`0
`
`6 43
`
`APPLE INC.
`EXHIBIT 1001 - PAGE 0004
`
`

`

`U.S. Patent (cid:9)
`
`Feb. 25, 2014 (cid:9)
`
`Sheet 2 of 15 (cid:9)
`
`US 8,659,571 B2
`
`CNI
`
`6
`
`LL
`
`APPLE INC.
`EXHIBIT 1001 - PAGE 0005
`
`

`

`U.S. Patent
`
`Feb. 25, 2014
`
`Sheet 3 of 15
`
`US 8,659,571 B2
`
`303
`
`411
`
`4091
`
`407i
`
`FIG. 4B
`
`APPLE INC.
`EXHIBIT 1001 - PAGE 0006
`
`(cid:9)
`(cid:9)
`(cid:9)
`

`

`U.S. Patent
`
`Feb. 25, 2014 (cid:9)
`
`Sheet 4 of 15
`
`US 8,659,571 B2
`
`505
`
`FIELD FORCE
`
`FIG. 5
`
`FIG. 6
`
`APPLE INC.
`EXHIBIT 1001 - PAGE 0007
`
`(cid:9)
`(cid:9)
`

`

`U.S. Patent (cid:9)
`
`Feb. 25, 2014 (cid:9)
`
`Sheet 5 of 15 (cid:9)
`
`US 8,659,571 B2
`
`701
`
`703
`
`
`
`%***44-gallillil
`
`*
`
`FIG. 7
`
`APPLE INC.
`EXHIBIT 1001 - PAGE 0008
`
`

`

`U.S. Patent (cid:9)
`
`Feb. 25, 2014 (cid:9)
`
`Sheet 6 of 15 (cid:9)
`
`US 8,659,571 B2
`
`809
`
`811
`
`FINGER
`SPEED (vf)
`
`805 -\n TANGENTIAL
` SPEED (vt)
`
`807
`
`Ft
`
`803
`
`
`
`815
`
`FIG. 8A
`
`801 *
`
`809
`
`811
`
`LATERAL
`FORCES
`803
`
`813
`
`TANGENTIAL
`AIR GAP 805 SPEED (vt)
`SQUEEZE
`FORCE (Ft)
`
`FIG. 8B
`
`APPLE INC.
`EXHIBIT 1001 - PAGE 0009
`
`(cid:9)
`

`

`U.S. Patent (cid:9)
`
`Feb. 25, 2014 (cid:9)
`
`Sheet 7 of 15
`
`US 8,659,571 B2
`
`V
`cy)
`C
`LL.
`
`0
`
`APPLE INC.
`EXHIBIT 1001 - PAGE 0010
`
`(cid:9)
`

`

`U.S. Patent (cid:9)
`
`Feb. 25, 2014 (cid:9)
`
`Sheet 8 of 15 (cid:9)
`
`US 8,659,571 B2
`
`O v-
`
`LL
`
`03
`O
`
`LL
`
`•
`
`•
`
`APPLE INC.
`EXHIBIT 1001 - PAGE 0011
`
`

`

`U.S. Patent (cid:9)
`
`Feb. 25, 2014 (cid:9)
`
`Sheet 9 of 15
`
`US 8,659,571 B2
`
`APPLE INC.
`EXHIBIT 1001 - PAGE 0012
`
`(cid:9)
`

`

`U.S. Patent (cid:9)
`
`Feb. 25, 2014 (cid:9)
`
`Sheet 10 of 15 (cid:9)
`
`US 8,659,571 B2
`
`g########################11111111
`
`1 101
`
`1103
`
`FIG. 11
`
`APPLE INC.
`EXHIBIT 1001 - PAGE 0013
`
`

`

`U.S. Patent (cid:9)
`
`Feb. 25, 2014 (cid:9)
`
`Sheet 11 of 15 (cid:9)
`
`US 8,659,571 B2
`
`APPLE INC.
`EXHIBIT 1001 - PAGE 0014
`
`

`

`U.S. Patent (cid:9)
`
`Feb. 25, 2014 (cid:9)
`
`Sheet 12 of 15 (cid:9)
`
`US 8,659,571 B2
`
`APPLE INC.
`EXHIBIT 1001 - PAGE 0015
`
`

`

`U.S. Patent (cid:9)
`
`Feb. 25, 2014 (cid:9)
`
`Sheet 13 of 15 (cid:9)
`
`US 8,659,571 B2
`
`C START )
`
`1301
`
`RECEIVE A DEVICE SENSOR SIGNAL AT
`TIME T1
`
`V
`
`(- 1303
`
`RECEIVE A GESTURE SIGNAL AT TIME T2
`
`1305
`
`COMPARE THE DEVICE SENSOR SIGNAL
`WITH A HAPTIC EFFECT SIGNAL TO
`GENERATE A DEVICE SENSOR SIGNAL
`DIFFERENCE VECTOR
`
`1307
`
`COMPARE THE GESTURE SIGNAL WITH A
`HAPTIC EFFECT SIGNAL TO GENERATE A
`GESTURE DIFFERENCE VECTOR
`
`(— 1309
`
`OPTIONALLY RECEIVE AN ANIMATION OR
`PHYSICAL MODEL DESCRIPTION
`
`1311
`
`V
`GENERATE AN INTERACTION
`PARAMETER USING THE GESTURE
`DIFFERENCE VECTOR, THE SIGNAL
`DIFFERENCE VECTOR AND ANIMATION
`OR PHYSICAL MODEL DESCRIPTION
`
`1313
`APPLY A DRIVE SIGNAL TO A HAPTIC
`OUTPUT DEVICE ACCORDING TO THE
`INTERACTION PARAMETER
`
`C END )
`
`FIG. 13
`
`APPLE INC.
`EXHIBIT 1001 - PAGE 0016
`
`

`

`U.S. Patent (cid:9)
`
`Feb. 25, 2014 (cid:9)
`
`Sheet 14 of 15 (cid:9)
`
`US 8,659,571 B2
`
`START )
`
`-1401
`
`ENABLE A COMMUNICATION LINK
`BETWEEN FIRST AND SECOND DEVICES
`HAVING FIRST AND SECOND HAPTIC
`OUTPUT DEVICES
`
`-1403
`RECEIVE A FIRST GESTURE SIGNAL OR
`DEVICE SENSOR SIGNAL FROM THE
`FIRST DEVICE AND COMMUNICATE IT TO
`THE SECOND DEVICE
`
`r1405
`OPTIONALLY RECEIVE A SECOND
`GESTURE SIGNAL OR DEVICE SENSOR
`SIGNAL FROM THE SECOND DEVICE AND
`COMMUNICATE IT TO THE FIRST DEVICE
`
`-1407
`
`GENERATE AN INTERACTION
`PARAMETER USING THE FIRST GESTURE
`OR SENSOR SIGNAL AND THE OPTIONAL
`SECOND GESTURE OR SENSOR SIGNAL
`
`(-1409
`
`CONCURRENTLY APPLY A DRIVE SIGNAL
`TO THE FIRST AND SECOND HAPTIC
`OUTPUT DEVICES ACCORDING TO THE
`INTERACTION PARAMETER
`
`( END )
`
`FIG. 14
`
`APPLE INC.
`EXHIBIT 1001 - PAGE 0017
`
`

`

`U.S. Patent (cid:9)
`
`Feb. 25, 2014 (cid:9)
`
`Sheet 15 of 15 (cid:9)
`
`US 8,659,571 B2
`
`( START )
`
`V
`
`i--1501
`
`RECEIVE A GESTURE SIGNAL OR DEVICE
`SENSOR SIGNAL FROM A FIRST DEVICE
`
`- 1503
`STORE OR ENCODE THE GESTURE OR
`SENSOR SIGNAL IN A DATA FILE ON THE
`FIRST DEVICE
`
`(-1505
`COMMUNICATE THE DATA FILE TO A
`SECOND DEVICE HAVING A HAPTIC
`OUTPUT DEVICE
`
`-1507
`READ THE GESTURE OR SENSOR SIGNAL
`FROM THE DATA FILE ON THE SECOND
`DEVICE
`
`(-1509
`APPLY A DRIVE SIGNAL TO THE HAPTIC
`OUTPUT DEVICE ACCORDING TO THE
`SIGNAL
`
`c (cid:9)
`
`END
`
`FIG. 15
`
`APPLE INC.
`EXHIBIT 1001 - PAGE 0018
`
`

`

`US 8,659,571 B2
`
`1
`INTERACTIVITY MODEL FOR SHARED
`FEEDBACK ON MOBILE DEVICES
`
`CROSS-REFERENCE TO RELATED
`APPLICATIONS
`
`This application claims the benefit of priority under 35
`USC §120 to copending application Ser. No. 13/592,685,
`filed Aug. 23, 2012, which claims the benefit of priority to
`application Ser. No. 13/472,709, filed May 16, 2012, which
`claims the benefit of priority to application Ser. No. 13/397,
`142, filed Feb. 15, 2012.
`
`FIELD OF THE INVENTION
`
`One embodiment is directed generally to a user interface
`for a device, and in particular to producing a dynamic haptic
`effect using multiple gesture signals and real or virtual device
`sensor signals.
`
`BACKGROUND INFORMATION
`
`Electronic device manufacturers strive to produce a rich
`interface for users. Conventional devices use visual and audi-
`tory cues to provide feedback to a user. In some interface
`devices, kinesthetic feedback (such as active and resistive
`force feedback) and/or tactile feedback (such as vibration,
`texture, and heat) is also provided to the user, more generally
`known collectively as "haptic feedback" or "haptic effects".
`Haptic feedback can provide cues that enhance and simplify
`the user interface. Specifically, vibration effects, or vibrotac-
`tile haptic effects, may be useful in providing cues to users of
`electronic devices to alert the user to specific events, or pro-
`vide realistic feedback to create greater sensory immersion
`within a simulated or virtual environment.
`In order to generate vibration effects, many devices utilize
`some type of actuator or haptic output device. Known haptic
`output devices used for this purpose include an electromag-
`netic actuator such as an Eccentric Rotating Mass ("ERM") in
`which an eccentric mass is moved by a motor, a Linear Reso-
`nant Actuator ("LRA") in which a mass attached to a spring is
`driven back and forth, or a "smart material" such as piezo-
`electric, electro-active polymers or shape memory alloys.
`Haptic output devices also broadly include non-mechanical
`or non-vibratory devices such as those that use electrostatic
`friction (ESF), ultrasonic surface friction (USF), or those that
`induce acoustic radiation pressure with an ultrasonic haptic
`transducer, or those that use a haptic substrate and a flexible or
`deformable surface, or those that provide projected haptic
`output such as a puff of air using an air jet, and so on.
`Traditional architectures that provide haptic feedback only
`with triggered effects are available, and must be carefully
`designed to make sure the timing of the haptic feedback is
`correlated to user initiated gestures or system animations.
`However, because these user gestures and system animations
`have variable timing, the correlation to haptic feedback may
`be static and inconsistent and therefore less compelling to the
`user. Further, device sensor information is typically not used
`in combination with gestures to produce haptic feedback.
`Therefore, there is a need for an improved system of pro-
`viding a dynamic haptic effect that includes multiple gesture
`signals and device sensor signals. There is a further need for
`providing concurrent haptic feedback to multiple devices
`which are connected via a communication link.
`
`SUMMARY OF THE INVENTION
`
`One embodiment is a system that produces a dynamic
`haptic effect and generates a drive signal that includes a
`
`5
`
`2
`gesture signal and a real or virtual device sensor signal. The
`haptic effect is modified dynamically based on both the ges-
`ture signal and the real or virtual device sensor signal such as
`from an accelerometer or gyroscope, or by a signal created
`from processing data such as still images, video or sound. The
`haptic effect may optionally be modified dynamically by
`using the gesture signal and the real or virtual device sensor
`signal and a physical model. The haptic effect may optionally
`be applied concurrently to multiple devices which are con-
`10 nected via a communication link The haptic effect may
`optionally be encoded into a data file on a first device. The
`data file is then communicated to a second device and the
`haptic effect is read from the data file and applied to the
`second device.
`
`15
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`20 (cid:9)
`
`35 (cid:9)
`
`FIG. 1 is a block diagram of a haptically-enabled system
`according to one embodiment of the present invention.
`FIG. 2 is a cut-away perspective view of an LRA imple-
`mentation of a haptic actuator according to one embodiment
`of the present invention.
`FIG. 3 is a cut-away perspective view of an ERM imple-
`mentation of a haptic actuator according to one embodiment
`25 of the present invention.
`FIGS. 4A, 4B and 4C are views of a piezoelectric imple-
`mentation of a haptic actuator according to one embodiment
`of the present invention.
`FIG. 5 is a view of a haptic device using electrostatic
`30 friction (ESF) according to one embodiment of the present
`invention.
`FIG. 6 is a view of a haptic device for inducing acoustic
`radiation pressure with an ultrasonic haptic transducer
`according to one embodiment of the present invention.
`FIG. 7 is a view of a haptic device using a haptic substrate
`and flexible or deformable surface according to one embodi-
`ment of the present invention.
`FIGS. 8A and 8B are views of a haptic device using ultra-
`sonic surface friction (USF) according to one embodiment of
`40 the present invention.
`FIGS. 9A, 9B and 9C are screen views of a user initiated
`dynamic haptic effect according to one embodiment of the
`present invention.
`FIGS. 10A, 10B, 10C, 10D, 10E and 1OF are screen views
`45 of encoding a haptic effect into a data file according to one
`embodiment of the present invention.
`FIG. 11 is a screen view of a user initiated dynamic haptic
`effect according to one embodiment of the present invention.
`FIGS. 12A, 12B, 12C, 12D and 12E are screen views of
`50 applying a haptic effect concurrently to multiple devices
`according to one embodiment of the present invention.
`FIG. 13 is a flow diagram for producing a dynamic haptic
`effect with a gesture signal and a device sensor signal accord-
`ing to one embodiment of the present invention.
`FIG. 14 is a flow diagram for concurrently applying a
`haptic effect to multiple devices according to one embodi-
`ment of the present invention.
`FIG. 15 is a flow diagram for encoding and applying a
`haptic effect using a data file according to one embodiment of
`60 the present invention.
`
`55 (cid:9)
`
`DETAILED DESCRIPTION
`
`As described below, a dynamic haptic effect refers to a
`65 haptic effect that evolves over time as it responds to one or
`more input parameters. Dynamic haptic effects are haptic or
`vibrotactile effects displayed on haptic devices to represent a
`
`APPLE INC.
`EXHIBIT 1001 - PAGE 0019
`
`

`

`US 8,659,571 B2
`
`3
`change in state of a given input signal. The input signal can be
`a signal captured by sensors on the device with haptic feed-
`back, such as position, acceleration, pressure, orientation, or
`proximity, or signals captured by other devices and sent to the
`haptic device to influence the generation of the haptic effect.
`A dynamic effect signal can be any type of signal, but does
`not necessarily have to be complex. For example, a dynamic
`effect signal may be a simple sine wave that has some prop-
`erty such as phase, frequency, or amplitude that is changing
`over time or reacting in real time according to a mapping
`schema which maps an input parameter onto a changing
`property of the effect signal. An input parameter may be any
`type of input capable of being provided by a device, and
`typically may be any type of signal such as a device sensor
`signal. A device sensor signal may be generated by any
`means, and typically may be generated by capturing a user
`gesture with a device. Dynamic effects may be very useful for
`gesture interfaces, but the use of gestures or sensors are not
`necessarily required to create a dynamic signal.
`One common scenario that does not involve gestures
`directly is defining the dynamic haptic behavior of an ani-
`mated widget. For example, when a user scrolls a list, it is not
`typically the haptification of the gesture that will feel most
`intuitive, but instead the motion of the widget in response to
`the gesture. In the scroll list example, gently sliding the list
`may generate a dynamic haptic feedback that changes accord-
`ing to the speed of the scrolling, but flinging the scroll bar may
`produce dynamic haptics even after the gesture has ended.
`This creates the illusion that the widget has some physical
`properties and it provides the user with information about the
`state of the widget such as its velocity or whether it is in
`motion.
`A gesture is any movement of the body that conveys mean-
`ing or user intent. It will be recognized that simple gestures
`may be combined to form more complex gestures. For
`example, bringing a finger into contact with a touch sensitive
`surface may be referred to as a "finger on" gesture, while
`removing a finger from a touch sensitive surface may be
`referred to as a separate "finger off' gesture. If the time
`between the "finger on" and "finger off' gestures is relatively
`short, the combined gesture may be referred to as "tapping";
`if the time between the "finger on" and "finger off' gestures is
`relatively long, the combined gesture may be referred to as
`"long tapping"; if the distance between the two dimensional
`(x,y) positions of the "finger on" and "finger off' gestures is
`relatively large, the combined gesture may be referred to as
`"swiping"; if the distance between the two dimensional (x,y)
`positions of the "finger on" and "finger off' gestures is rela-
`tively small, the combined gesture may be referred to as
`"smearing", "smudging" or "flicking". Any number of two
`dimensional or three dimensional simple or complex gestures
`may be combined in any manner to form any number of other
`gestures, including, but not limited to, multiple finger con-
`tacts, palm or first contact, or proximity to the device. A
`gesture can also be any form of hand movement recognized
`by a device having an accelerometer, gyroscope, or other
`motion sensor, and converted to electronic signals. Such elec-
`tronic signals can activate a dynamic effect, such as shaking
`virtual dice, where the sensor captures the user intent that
`generates a dynamic effect.
`FIG. 1 is a block diagram of a haptically-enabled system 10
`according to one embodiment of the present invention. Sys-
`tem 10 includes a touch sensitive surface 11 or other type of
`user interface mounted within a housing 15, and may include
`mechanical keys/buttons 13. Internal to system 10 is a haptic
`
`4
`feedback system that generates vibrations on system 10. In
`one embodiment, the vibrations are generated on touch sur-
`face 11.
`The haptic feedback system includes a processor 12.
`5 Coupled to processor 12 is a memory 20 and an actuator drive
`circuit 16, which is coupled to a haptic actuator 18. Processor
`12 may be any type of general purpose processor, or could be
`a processor specifically designed to provide haptic effects,
`such as an application-specific integrated circuit ("ASIC").
`Processor 12 may be the same processor that operates the
`entire system 10, or may be a separate processor. Processor 12
`can decide what haptic effects are to be played and the order
`in which the effects are played based on high level param-
`15 eters. In general, the high level parameters that define a par-
`ticular haptic effect include magnitude, frequency and dura-
`tion. Low level parameters such as streaming motor
`commands could also be used to determine a particular haptic
`effect. A haptic effect may be considered dynamic if it
`20 includes some variation of these parameters when the haptic
`effect is generated or a variation of these parameters based on
`a user's interaction.
`Processor 12 outputs the control signals to drive circuit 16
`which includes electronic components and circuitry used to
`25 supply actuator 18 with the required electrical current and
`voltage to cause the desired haptic effects. System 10 may
`include more than one actuator 18, and each actuator may
`include a separate drive circuit 16, all coupled to a common
`processor 12. Memory device 20 can be any type of storage
`30 device or computer-readable medium, such as random access
`memory (RAM) or read-only memory (ROM). Memory 20
`stores instructions executed by processor 12. Among the
`instructions, memory 20 includes an actuator drive module 22
`which are instructions that, when executed by processor 12,
`35 generate drive signals for actuator 18 while also determining
`feedback from actuator 18 and adjusting the drive signals
`accordingly. The functionality of module 22 is discussed in
`more detail below. Memory 20 may also be located internal to
`processor 12, or any combination of internal and external
`40 memory.
`Touch surface 11 recognizes touches, and may also recog-
`nize the position and magnitude or pressure of touches on the
`surface. The data corresponding to the touches is sent to
`processor 12, or another processor within system 10, and
`45 processor 12 interprets the touches and in response generates
`haptic effect signals. Touch surface 11 may sense touches
`using any sensing technology, including capacitive sensing,
`resistive sensing, surface acoustic wave sensing, pressure
`sensing, optical sensing, etc. Touch surface 11 may sense
`so multi-touch contacts and may be capable of distinguishing
`multiple touches that occur at the same time. Touch surface 11
`may be a touchscreen that generates and displays images for
`the user to interact with, such as keys, dials, etc., or may be a
`touchpad with minimal or no images.
`System 10 may be a handheld device, such as a cellular
`telephone, PDA, computer tablet, gaming console, etc. or
`may be any other type of device that provides a user interface
`and includes a haptic effect system that includes one or more
`ERMs, LRAs, electrostatic or other types of actuators. The
`60 user interface may be a touch sensitive surface, or can be any
`other type of user interface such as a mouse, touchpad, mini-
`joystick, scroll wheel, trackball, game pads or game control-
`lers, etc. In embodiments with more than one actuator, each
`actuator may have a different output capability in order to
`65 create a wide range of haptic effects on the device. Each
`actuator may be any type of haptic actuator or a single or
`multidimensional array of actuators.
`
`55 (cid:9)
`
`APPLE INC.
`EXHIBIT 1001 - PAGE 0020
`
`

`

`US 8,659,571 B2
`
`6
`5
`("EMF") of the rotating mass 301 is received and used to
`FIG. 2 is a cut-away side view of an LRA implementation
`determine the angular speed of ERM 18. In another embodi-
`of actuator 18 in accordance to one embodiment. LRA 18
`includes a casing 25, a magnet/mass 27, a linear spring 26, (cid:9)
`ment, the drive period and the monitoring period are concur-
`and an electric coil 28. Magnet 27 is mounted to casing 25 by (cid:9)
`rent and the present invention dynamically determines the
`spring 26. Coil 28 is mounted directly on the bottom of casing 5 angular speed of ERM 18 during both the drive and monitor-
`25 underneath magnet 27. LRA 18 is typical of any known
`ing periods.
`LRA. In operation, when current flows through coil 28 a
`FIGS. 4A-4C are views of a piezoelectric implementation
`magnetic field forms around coil 28 which in interaction with
`of a haptic actuator 18 according to one embodiment of the
`the magnetic field of magnet 27 pushes or pulls on magnet 27. (cid:9)
`present invention. FIG. 4A shows a disk piezoelectric actua-
`One current flow direction/polarity causes a push action and l() for that includes an electrode 401, a piezo ceramics disk 403
`the other a pull action. Spring 26 controls the up and down (cid:9)
`and a metal disk 405. As shown in FIG. 4B, when a voltage is
`movement of magnet 27 and has a deflected up position where (cid:9)
`applied to electrode 401, the piezoelectric actuator bends in
`it is compressed, a deflected down position where it is (cid:9)
`response, going from a relaxed state 407 to a transformed
`expanded, and a neutral or zero-crossing position where it is (cid:9)
`state 409. When a voltage is applied, it is that bending of the
`neither compressed or deflected and which is equal to its 15 actuator that creates the foundation of vibration. Alterna-
`resting state when no current is being applied to coil 28 and
`tively, FIG. 4C shows a beam piezoelectric actuator that oper-
`there is no movement/oscillation of magnet 27. (cid:9)
`ates similarly to a disk piezoelectric actuator by going from a
`For LRA 18, a mechanical quality factor or "Q factor" can (cid:9)
`relaxed state