US007667692B2
`
`(12) Ulllted States Patent
`Marcus et al.
`
`(10) Patent N0.:
`(45) Date of Patent:
`
`US 7,667,692 B2
`*Feb. 23, 2010
`
`(54) HUMAN INTERFACE SYSTEM
`
`5,189,416 A
`
`2/1993 Estes
`
`(75) Inventors: Beth Marcus, Bedford, MA (US); W.
`David Lee, Newton, MA (US)
`
`(Continued)
`
`(73) Assignee: Zeemote, Inc., Chelmsford, MA (US)
`
`FOREIGN PATENT DOCUMENTS
`
`( * ) Notice:
`
`Subject to any disclaimer, the term ofthis
`patent is extended or adjusted under 35
`USC 154(b) by 0 days.
`
`This patent is subject IO a terminal (115-
`claimer.
`
`DE
`
`298 23 417 U1
`
`5/1999
`
`(Continued)
`
`OTHER PUBLICATIONS
`
`(21) Appl, NO; 12/329,411
`
`(22) F1led:
`
`Dec. 5, 2008
`
`State Intellectual Property Of?ce of PRC Noti?cation of First Of?ce
`Action dated Nov. 21, 2008, Chinese Patent Appln No.
`2007101533713.
`
`(Continued)
`
`(65)
`
`Prior Publication Data
`Us 2009/0143142 A1
`Jun 4’ 2009
`
`Primary ExamineriRicardo L Osorio
`(74) Attorney, Agent, or FirmiFish & Richardson P.C.
`
`Related US. Application Data
`
`(57)
`
`ABSTRACT
`
`(63) Continuation of application No. 11/747,863, ?led on
`May 11, 2007, noW Pat. No. 7,463,245, Which is a
`continuation of application No. 10/699,555, ?led on
`Oct. 31, 2003, noW Pat. No. 7,218,313.
`
`(51) Int. Cl.
`(2006.01)
`G09G 5/00
`(52) US. Cl. ..................................... .. 345/169; 345/ 168
`(58) Field of Classi?cation Search ............... .. 345/156,
`345/168, 169, 173; 400/472; 341/22
`See application ?le for Complete Search history
`_
`References Clted
`U.S. PATENT DOCUMENTS
`
`(56)
`
`4,793,312 A 12/1988 Doinagaetal.
`4,867,028 A
`9/1989 Jones
`4,891,777 A
`1/1990 Lapeyre
`4,896,554 A
`1/1990 Culver
`4,912,462 A
`3/1990 WashiZuka et al.
`
`A human interface con?gured to optimiZe a biomechanical
`effect of a human user’s opposing thumb and ?ngers by
`including, on one surface, one or more softWare con?gurable
`input elements manipulatable by a user’s thumb(s) or a stylus,
`and, on another surface, one or more software con?gurable
`selection elements manipulatable by a user’s ?nger(s). A
`selection element may be a pressure sensor pad con?gurable
`to represent delineated active areas that are mapped to one Or
`more input functions. Shape changing media may be pro
`vided to permit a user to tactilely discriminate betWeen delin
`eated active areas. Tactile feedback may be provided to a user
`through palpable detents, vibratory or force producing units.
`lnputting data may include mapping each selection element
`to a shift function, mapping each input element to text func
`tions, and using the selection elements to shift betWeen text
`?lnctions associated With an input element to input a desired
`text ?inction.
`
`20 Claims, 9 Drawing Sheets
`
`SCEA Ex. 1001 Page 1
`
`

`

`US 7,667,692 B2
`Page 2
`
`US. PATENT DOCUMENTS
`
`5,365,589 A 11/1994 GutowitZ
`5,432,510 A
`7/1995 Matthews
`5,473,325 A 12/1995 McAlindon
`5,512,919 A
`4/1996 Amki
`5,515,305 A
`5/1996 Register et al.
`5,612,690 A
`3/1997 Levy
`5,782,642 A
`7/1998 Goren
`5,824,931 A 10/1998 Papadopfmlos
`5 359,629 A
`V1999 Tognazzlm
`5,900,864 A
`5/1999 Macdonald
`5 973521 A 10/1999 Levy
`6’005’496 A 12/1999 Hargreaves et a1‘
`6’084’576 A
`7/2000 Le? St 31'
`6:107:988 A
`8/2000 PhllhpPS
`6,115,028 A
`9/2000 Balakrishnan et al.
`6,184,804 B1
`2/2001 Harrison
`6,219,731 B1
`4/2001 GutowitZ
`6,228,709 B1
`5/2001 Hsieh
`6,232,956 B1
`5/2001 Mailman
`6,297,752 B1
`10/2001 Ni
`6,377,685 B1
`4/2002 Krishnan
`RE37,723 E
`6/2002 Goren
`6,512,511 B2
`1/2003 Willner et al.
`6,520,699 B2
`2/2003 Abe
`6,541,715 B2
`4/2003 Swanson
`6,542,091 B1
`4/2003 Rasanen
`6,546,239 B1
`4/2003 PaZdersky et al.
`6,573,844 B1
`6/2003 Venolia et al.
`6,606,486 B1
`8/2003 Cubbage et al.
`6,654,733 B1
`11/2003 Goodman et al.
`6,703,963 B2
`3/2004 Higginson
`6,738,045 B2
`5/2004 Hinkley et al.
`6,741,235 B1
`5/2004 Goren
`6,760,013 B2
`7/2004 Willner et al.
`6,865,718 B2
`3/2005 Montalcini
`6,885,317 B1
`4/2005 GutowitZ
`6,885,318 B2
`4/2005 Bickerton
`6,909,424 B2
`6/2005 Liebenow et al.
`6,911,608 B2
`6/2005 Levy
`6,947,028 B2
`9/2005 Shkolnikov
`6,980,200 B2 12/2005 Goren
`7,072,975 B2
`7/2006 Kato
`7,092,734 B2
`8/2006 Herle et al.
`2002/0019259 A1
`2/2002 Armstrong
`2002/0023265 A1
`2/2002 Mercalf
`2002/0163504 A1 11/2002 Pallakoff
`2003/0020692 A1
`1/2003 Grif?n et al.
`2003/0048205 A1
`3/2003 He
`2003/0061103 A1
`3/2003 Kanai
`2003/0083114 A1
`5/2003 Lavin et al.
`2003/0095156 A1
`5/2003 Klein et al.
`2003/0122784 A1
`7/2003 Shkolnikov
`2003/0169188 A1
`9/2003 Chang et al.
`2003/0193418 A1 10/2003 Shi
`2004/0107303 A1
`6/2004 Mulligan
`2004/0208681 A1 10/2004 Dechene
`2005/0093846 A1
`5/2005 Marcus et al.
`
`FOREIGN PATENT DOCUMENTS
`
`DE
`EP
`EP
`EP
`EP
`EP
`JP
`KR
`W0
`WO
`WO
`
`2002 27645 A1
`0251477
`0585730
`1103883
`1293882
`1 376 319 A1
`2000-267787 A
`19990072889
`91/ 05303
`WO 03/0071 17 A2
`03/042805
`
`4/2002
`1/1988
`3/1994
`5/2001
`3/2003
`1/2004
`9/2000
`9/ 1999
`4/1991
`1/2003
`5/2003
`
`W0 03/052948 A1
`W0
`W0 WO 2004/019315 A1
`
`6/2003
`3/2004
`
`http://www.alphagrips.com/
`
`OTHER PUBLICATIONS
`“Introducing the Dvorak Keyboard”, http://www.mwbrooks.com/
`dvomld‘
`Alphagrip,
`AlphagripAG5UsersManual.pdf.
`Amy K. Karlson, Benjamin B. Bederson, John SanGiovanni, 2004.
`AppLens and LaunchTile: Two Designs for One-Handed Thumb Use
`on Small Devices, http://hcil.cs.umd.edu/trs/2004-37/2004-37.html.
`Andriy Pavlovych, Wolfgang SturZlinger: Less-Tap: A Fast and
`Easy-to-learn Text Input Technique for Phones. Graphics Interface
`2003,
`97-104,
`http://www.graphicsinterface.org/cgi-bin/
`DownloadPaper?name:2003/170/paper170.pdf.
`Atrua: sensor company, http://www.atrua.com/s-mobilephones.
`html.
`Baillie, L., KuncZier, H., and Anegg, H. 2005. Rolling, rotating and
`imagining in a virtual mobile world. In Proceedings of the 7th inter
`national Conference on Human Computer interaction with Mobile
`Devices &Amp; Services (SalZburg, Austria, Sep. 19-22, 2005).
`MobileHCI ’05, vol. 111. ACM Press, New York, NY, 283-286.
`http://doi.acm.org/10.1145/1085777.1085833.
`Bartlett, J. F. 2000. Rock ‘n’ Scroll Is Here to Stay. IEEE Comput.
`Graph. Appl. 20, 3 (May 2000), 40-45. http://portal.acm.org/citation.
`cfm?id:618728&coll:Portal&d1:GUIDE&CFID:665 88306
`&CFTOKEN:73460863#.
`http://mobilitytoday.com/news/005986/
`Bluetooth
`GPS,
`mobilityibuyGPSnowii -Blueiblueto othiGPS.
`BuXton, “A Directory of Sources for Input Technologies”, Oct. 1,
`2003, http://www.billbuXton.com/InputSources.html.
`BuXton, “An Introduction to Human Input to Computers”, Apr. 6,
`1999, http://www.billbuXton.com/input01.Introduction.pdf.
`BuXton, “Human Input to Computer Systems: Theories, Techniques
`and Technology”, http://billbuXton.com/inputManuscript.htlm.
`C. MetZger, M. Anderson, T. Starner, 2004. FreeDigiter: A Contact
`Free Device for Gesture Control. Eighth IEEE International Sympo
`sium on Wearable Computers (ISWC’04) pp. 18-21. http://www.
`wirelessrerc.gatech.edu/projects/development/D1?les/iswc04 -
`freedigiterpdf.
`Chipman, L. E., Bederson, B. B., and Golbeck, J. A. 2004. SlideBar:
`analysis of a linear input device. Behav. Inf. Tech. 23, 1 (Jan. 2004),
`1-9. http://portal.acm.org/citation.cfm?id:993182.993184#, http://
`www.cs.umd.edu/Library/TRs/CS-TR-4471/CS-TR-4471.pdf.
`Chording and TiltingiDaniel Wigdor (thesis)i2004- describes
`chordtap and tilttap (also covered in depth in the paper referenced
`below), http ://www. dgp.toronto .edu/~dwigdor/research/thesi s/sub
`mitted.html.
`Daniel Fallmana , Andreas Lund , Mikael Wiberg, ScrollPad: Tan
`gible Scrolling with Mobile Devices, Proceedings of the Proceedings
`of the 37th Annual Hawaii International Conference on System Sci
`ences (HICSS’04)iTrack 9, p. 90286.3, Jan. 5-8, 2004. http://portal.
`acm.org/citation.cfm?id:963347&coll:GUIDE&dl:GUIDE
`&CFID:66483658&CFTOKEN:36023921, http://danielfallman.
`org/resources/papers/fallman-hicss37.pdf.
`Daniel Wigdor , Ravin Balakrishnan, TiltTeXt: using tilt for text input
`to mobile phones, Proceedings of the 16th annual ACM symposium
`on User interface software and technology, p. 81-90, Nov. 2-5, 2003,
`Vancouver, Canada, http://portal.acm.org/citation.cfm?id:964705,
`http://www.dgp.toronto.edu/~ravin/papers/uist2003itilttext.pdf.
`DigitWireless: FastTap, http://www.digitwireless.com/?ash/down
`1O ad/fastap‘pdf'
`Donner, J. (2005). Research Approaches to Mobile Use in Develop
`ing World: A Review of the Literature. International Conference on
`Mobile Communication and Asian Modernities City University of
`Hong Kong, Jun. 7-8, 2005.
`Eleksen, www.eleksen.com.
`Eslambolchilar, P., Murray-Smith, R. (2004). Tilt-Based Automatic
`Zooming and Scaling in Mobile DevicesiA state-space implemen
`tation. In Proc. of Mobile Human-Computer Interaction (MobileHCI
`2004), Glasgow, UK, Sep. 2004: In S. Brewster and M. Dunlop
`
`SCEA Ex. 1001 Page 2
`
`

`

`US 7,667,692 B2
`Page 3
`
`http://www.
`
`(Eds.). Mobile Human-Computer-InteractioniMobileHCI 2004,
`Lecture Notes in Computer Science, vol. 3160, Berlin: Springer,
`120-131.
`JPO Noti?cation of Reason(s) for Refusal; Dispatch Date: Jul. 8,
`2008; Dispatch No. 396667.
`Exideas,
`http://www.exideas.com/ME/index.html,
`exideas.com/ME/HardKeyhtml.
`GamePad, http://www.mobilemag.com/ content/ 100/345/C5578/ .
`Goldstein, M., et al., “The Finger-Joint-Gesture Wearable Keypad,”
`Ericsson Radio Systems AB., pp. 9-18.
`Green, N., Kruger, J., Faldu, C., and St. Amant, R. 2004. A reduced
`QWERTY keyboard for mobile text entry. In CHI ’04 Extended
`Abstracts on Human Factors in Computing Systems (Vienna, Aus
`tria, Apr. 24-29, 2004). CHI ’04. ACM Press, NewYork, NY, 1429
`1432. http://portal.acm.org/citation.cfm?id:986082&coll:GUIDE
`&dl:GUIDE&CFID:66591340&CFTOKEN:6294934.
`H. Kober, E. Skepner, T. Jones, H. GutowitZ, S. MacKenZie, 2001.
`Linguistically Optimized Text Entry on a Cell Phone. In Proceedings
`of the CHI 2001. http://www.eatoni.com/research/chipdf.
`Harrison, B. L., Fishkin, K. P., Gujar, A., Mochon, C., and Want, R.
`1998. Squeeze me, hold me, tilt me! An exploration of manipulative
`user interfaces. In Proceedings of the SIGCHI Conference on Human
`Factors in Computing Systems (Los Angeles, California, United
`States, Apr. 18-23, 1998). C. Karat, A. Lund, J. CoutaZ, and J. Karat,
`Eds. Conference on Human Factors in Computing Systems. ACM
`Press/Addison-Wesley Publishing Co., NewYork, NY, 17-24. http://
`portal.acm.org/citation.cfm?id:274647&coll:Portal&dl:GUIDE
`&CFID:66588306&CFTOKEN:73460863&CFID:66588306
`&CFTOKEN:73460863#.
`Hinckley, K., Cutrell, E., Bathiche, S., and Muss, T. 2002. Quantita
`tive analysis of scrolling techniques. In Proceedings of the SIGCHI
`Conference on Human Factors in Computing Systems: Changing
`Our World, Changing Ourselves (Minneapolis, Minnesota, USA,
`Apr. 20-25, 2002). CHI ’02. ACM Press, New York, NY, 65-72.
`http://doi.acm.org/10.1145/503376.503389.
`Hinckley, K., Pierce, J., HorvitZ, E., Sinclair, M. Foreground and
`Background Interaction with Sensor-enhanced Mobile Devices,
`ACM TOCHI (Transactions on Computer-Human Interaction) Spe
`cial Issue on Sensor-Based Interaction, 12(1), Mar. 2005, pp. 31-52.
`http://portal.acm.org/citation.cfm?id:1057240&coll:GUIDE
`&dl:GUIDE&CFID:66591340&CFTOKEN:6294934.
`Hinkley, K., Pierce, J ., Sinclair, M., and HorvitZ, E. Sensing Tech
`niques for Mobile Interaction. UIST 2000: ACM Symposium on User
`Interface Software and Technology, pp. 91-100. http://portal.acm.
`org/citation.cfm?id:3 54417&coll:GUIDE&dl:GUIDE
`&CFID:6648365 8&CFTOKENI3 6023921.
`Howard.co.kriThe mouse phone, http://www.howard.co.kr/com
`puter/mouse/mousephone.htrn.
`Innovative Ergonomic Solutions, Cirque Pocket Keyboard, http://
`www.iesproducts.com/key-misc-pockethtml.
`Introducing the Dvorak Keyboard, http://www.mwbrooks.com/
`dvorald.
`Jeong-Hoon Shin and Kwang-Seok Hong. An improved alphanu
`meric input algorithm using gloves. http://www.complexity.org.au/
`conference/upload/shin01/shin01 .pdf.
`K. Lyons, T. Starner, D. Plaisted, J. Fusia, A. Lyons, A. Drew, E. W.
`Looney, 2004. “Twiddler Typing: One-Handed Chording Text Entry
`for Mobile Phones,” Proc. Conf. Human Factors in Computing Sys
`tems (SIGCHI 01), ACM Press, 2004, pp. 671-678. http://www.cc.
`gatechedu/fac/Thad.Starner/p/030i10iMTE/twiddler-chi04.pdf.
`K. Lyons. Everyday wearable computer use: A case study of an expert
`user. In Proceedings of Mobile HCI 2003, pp. 61-75, 2003. http://
`www.cc.gatech.edu/ccg/publications/everydayicase.pdf.
`Kiyokuni Kawachiya , Hiroshi Ishikawa, NaviPoint: an input device
`for mobile information browsing, Proceedings of the SIGCHI con
`ference on Human factors in computing systems, p. 1-8, Apr. 18-23,
`1998, Los Angeles, California, United States http://portal.acm.org/
`citation.cfm?id:274645&coll:Portal&dl:GUIDE
`&CFID:66588306&CFTOKEN:73460863.
`Kjeldskov, J. and Graham, C. (2003). A Review of Mobile HCI
`Research Methods. In Proc. of Mobile Human-Computer Interaction
`(MobileHCI 2003), Udine Italy, Sep. 2003: In L. Chittaro (Ed.).
`
`Mobile Human-Computer-InteractioniMobileHCI 2003, Lecture
`Notes in Computer Science, vol. 2795, Berlin: Springer, 317-335.
`Kjeldskov, J. Stage, J. (2004). New Techniques for Usability Evalu
`ation of Mobile Systems. International Journal of Human-Computer
`Studies, May 2004, 60 (5-6): 599-620.
`KranZ, M., Holleis, P., Schmidt, A. “DistScrollia new one-handed
`interaction device”. In Proceedings of the 5th International Workshop
`on Smart Appliances and Wearable Computing, Jun. 10, 2005. http://
`www.hcilab.org/documents/DistScrol
`lAnewOneHandedInteractionDevice-KranZHolleisSchmidt
`IWSAWC2005.pdf.
`Kyocera Candid KX16, http://www.mobilemag.com/ content/ 100/
`340/C4392/.
`Lee Butts , Andy Cockburn, An evaluation of mobile phone text input
`methods, Third Australasian conference on User interfaces, p. 55-59,
`Jan. 1, 2002, Melbourne, Victoria, Australia, http://www.crpit.com/
`confpapers/CRPITV7Butts.pdf.
`Lee, S. and Hong S.H.. Chording as a Text Entry Method in Mobile
`Phones. In Proceedings of the MobileHCI 2004: 6th International
`Symposium, Glasgow, UK, Sep. 13-16, 2004.
`Lee, S., Hong, S. H., and Jeon, J. W. 2002. Designing a universal
`keyboard using chording gloves. SIGCAPH Comput. Phys. Handi
`cap. , 73-74 (Jun. 2002), 142-147. http://doi.acm.org/10.1145/
`960201 .957230.
`Lumsden, J ., Gammell, A. (2004). Mobile Note Taking: Investigating
`the Efficacy of Mobile Text Entry. In Proc. of Mobile Human-Com
`puter Interaction (MobileHCI 2004), Glasgow, UK, Sep. 2004: In S.
`Brewster and M. Dunlop (Eds.). Mobile Human-Computer
`InteractioniMobileHCI 2004, Lecture Notes in Computer Science,
`vol. 3160, Berlin: Springer, 156-168.
`M. D. Dunlop and A. Crossan, “Dictionary based text entry method
`for mobile phones”, published in Brewster, SA, and Dunlop, M.D.,
`(editors). Proceedings of Second Workshop on Human Computer
`Interaction with Mobile Devices, Aug. 1999. http://www.cis.strath.
`ac.uld~mddfresearch/publications/99dunloperossan.pdf.
`M. Kolsch, M. Turk, 2002. Keyboards without Keyboards: A Survey
`of Virtual Keyboards. UCSB Technical Report 2002-21, Jul. 12,
`2002. http://www.cs.ucsb.edu/research/techireports/reports/2002
`21.pdf.
`MacKay, B., Dearman, D., Inkpen, K., andWatters, C. 2005. Walk ’n
`scroll: a comparison of software-based navigation techniques for
`different levels of mobility. In Proceedings of the 7th international
`Conference on Human Computer interaction with Mobile Devices
`&Amp; Services (SalZburg, Austria, Sep. 19-22, 2005). MobileHCI
`’05, vol. 111.ACM Press, NewYork, NY, 183-190. http://portal.acm.
`org/citation.cfm?id:1085808&coll:GUIDE&dl:GUIDE
`&CFID:66591340&CFTOKEN:6294934.
`MacKenZie, I. S. (2002). KSPC (keystrokes per character) as a char
`acteristic of text entry techniques. Proceedings of the Fourth Inter
`national Symposium on Human-Computer Interaction with Mobile
`Devices, pp. 195-210. Heidelberg, Germany: Springer-Verlag.
`MacKenZie, I. S., & Soukoreff, R. W. Phrase sets for evaluating text
`entry techniques. Extended Abstracts of the ACM Conference on
`Human Factors in Computing Systems4CHI 2003, pp. 754-755
`New York: ACM.
`MacKenZie, S., & Soukoreff, W. (2002). Text entry for mobile com
`puting: Models and methods, theory and practice. Human-Computer
`Interaction. 17. p. 147-198. http://www.yorku.ca/macldhci3-2002.
`pdf.
`Microth KeyWheel, http://www.microth.com/circumscript/over
`view.asp.
`Mikael Goldstein, Didier Chincholle, Morten Back (2000). Assess
`ing Two New Wearable Input Paradigms: The Finger-Joint-Gesture
`Palm-Keypad Glove and the Invisible Phone Clock. Personal and
`Ubiquitous Computing, vol. 4, Issue 2/ 3.
`Mikael Goldstein and Didier Chincholle The Finger-Joint-Gesture
`Wearable Keypad. Ericsson Radio Systems.
`Min, Lin and Sears, Andrew (2005). Graphics Matter: A Case Study
`of Mobile Phone Keypad Design for Chinese Input. CHI 2005, Late
`Breaking Results: Posters, Portland, Oregon. Apr. 2-7, 2005.
`MotorolaiiTAP,
`http://news.Zdnet.co.uldhardware/mobile/
`0,39020360,39118435.00.htm.
`
`SCEA Ex. 1001 Page 3
`
`

`

`US 7,667,692 B2
`Page 4
`
`http://wireless.ign.com/articles/657/
`
`http://www.n-e-ware.com/Downloads/KeySticld330/
`NE-Ware,
`KSUserManual330i01.pdf.
`Nokia 6620 with a control stick, http://nds2.nokia.com/?les/support/
`nam/phones/guides/6620iUSien.PDF.
`OnisZcZak, A., & MacKenZie, I. S. (2004). A comparison of two input
`methods for keypads on mobile devices. Proceedings of NordiCHI
`2004, pp. 101-104. New York: ACM. http://www.yorku.ca/macld
`nordichi2004.pdf.
`Orientation-based interaction for Mobile Devices. J. Darnauer, S.
`Garrity and T. Jim, Stanford University, pp. 1-4, found on the internet
`at http://hci .stanford.edu/srk/c s3 77a-mobile/proj ect/?nal/darnauer
`garrity-kim.pdf.
`Partridge, K., Chatterjee, S., SaZawal,V., Borriello, G., and Want, R.
`TiltType: accelerometer- supported text entry for very small devices,
`Proceedings of the 15th annual ACM symposium on User interface
`software and technology, Oct. 27-30, 2002, Paris, France.
`Pirhonen, A., Brewster, S., and Holguin, C. 2002. Gestural and audio
`metaphors as a means of control for mobile devices. In Proceedings
`of the SIGCHI Conference on Human Factors in Computing Sys
`tems: Changing Our World, Changing Ourselves (Minneapolis, Min
`nesota, USA, Apr. 20-25, 2002). CHI ’02. ACM Press, NewYork, NY,
`291-298. http://doi.acm.org/10.1145/503376.503428.
`Prevalent Devices LLC, http://www.prevalentdevices.com/manual3
`5-06 .pdf.
`Slingshot,
`Qualcomm
`657041p1.html.
`Rakkolainen, I. (2003). MobiVRia novel user interface concept for
`mobile computing. In: Bieber, K. & Kirste, T. (eds.), Proceedings of
`the 4th International Workshop on Mobile Computing, IMC 2003,
`Jun. 17-18, 2003, Rostock, Germany, pp. 107-1 12. http://www.cs.tut.
`?/~ira/IMC2003.pdf.
`Rekimoto, J. Tilting operations for small screen interfaces. Proceed
`ings of the 9th annual ACM symposium on User Interface software
`and technology, pp. 167-168, Nov. 6-8, 1996, Seattle. http://portal.
`acm.org/citation.cfm?id:2371 15&coll:GUIDE&dl:GUIDE
`&CFID:6648365 8&CFTOKENI3 6023921.
`Rosenberg, R. (1998). Computing without Mice and Keyboards: Text
`and Graphic Input Devices for Mobile Computing. Ph.D. Thesis,
`Dept. of Computer Science, University College, London, 1998.
`http://www.obscure.org/rosenberg/.
`Samsung Game Pad, http://www.cellphonemall.net/wireless/store/
`accessorydetail.asp?id:23198&phoneid:334.
`Scott MacKenZie , Hedy Kober , Derek Smith , Terry Jones , Eugene
`Skepner, LetterWise: pre?x-based disambiguation for mobile text
`input, Proceedings of the 14th annual ACM symposium on User
`interface software and technology, Nov. 11-14, 2001, Orlando,
`Florida.
`Sega into cell phones, http://www.phoneyworld.com/newspage.
`aspx?n:1745 .
`Sengital Ltd. Tilt sensor replacement for PDA, http://sengital.manu
`facturer.globalsources.com/si/6008823523892/ProductDetail/PDA
`keyboard/productiid-1001050135/action-GetProduct.htm.
`Sharp Vodafone ZTCJ01, http://www.slashphone.com/93/3123.
`htrnl.
`
`http://www.
`
`Sony Clie game controller PEGA-GC10,
`palminfocenter.com/viewistory.asp?ID:4295.
`Soukoreff, R. W. and MacKenZie, I. S. (2004). Recent Developments
`in Text-Entry Error Rate Measurement. CHI 2004, Late Breaking
`Results Paper, Vienna Austria, Apr. 24-29, 2004.
`Starner, T. “Keyboards Redux: Fast Mobile Text Entry”. Pervasive
`Computing, Jul.-Sep. 2004, pp. 97-101. http://www.cc.gatech.edu/
`fac/Thad.Starner/p/magaZine/2004-3-keyboard-redux.pdf.
`Synaptics,
`http://www.synaptics.com/products/pdf/
`mobiletouchipbpdf.
`http://www.tegic.com/pdfs/salessheets/
`TegiciT9,
`T9%20Adaptive%20Text%20Input%20Sales%20Sheet%201.pdf,
`http://www.tegic.com/pdfs/salessheets/
`T9%20Adaptive%20Text%20Input%20Sales%20Sheet%202.pdf,
`http://www.tegic.com/pdfs/salessheets/
`T9%20Adaptive%20Text%20Input%20Sales%20Sheet%203.pdf,
`http://www.tegic.com/pdfs/salessheets/
`Sloppy%20Type%20Sales%20Sheet.pdf.
`The GamePad, http://www.kotaku.com/gaming/cell-phones/killer
`cell-phone-game-controller-130968.php.
`Thumbscript, http://www.thumbscript.com/index.htrnl, http://www.
`thumbscript.com/howitworks.htrnl, http://www.thumbscript.com/
`technotes.html.
`Twiddler, http://www.handykey.com/, http://www.handykey.com/
`Keymap.pdf.
`Unidenti?ed and Undated Document discussing alternative designs
`to QWERTY Keyboard, pp. 2-10.
`Varatouch: sensor company, http://www.esato.com/news/ article.
`php/id:388.
`Virpi Roto, Nokia Research. Browsing on Mobile Phones. http://
`www.research.att.com/~rjana/WF12iPaper1.pdf.
`Wigdor, D. and Balakrishnan, R. “A Comparison of Consecutive and
`Concurrent Input Text Entry Techniques for Mobile Phones”, Con
`ference on Human Factors, Apr. 24-29, 2004, vol. 6, No. 1, pp. 81-88,
`http://portal.acm.org/citation.cfm?id:985703,
`http://www.dgp.
`toronto .edu/~ravin/papers/chi2004ic oncurrenttextinputpdf.
`Wobbrock, J. O., ForliZZi, J., Hudson, S. E. and Myers, B. A.
`WebThumb: interaction techniques for small-screen browsers. Proc.
`UIST, ACM Press (2002), 205-208.
`XEG Mobile Phone Pad, http:// us. giZmodo .com/ gadgets/ cellphones/
`gaming-on-the-go-with-xeg-136414.php,
`http://www.
`akihabaranews.com/en/news-10615
`XEG%2C+the+mobile+phone+pad.html.
`Yuvee: special keypad layout, www.yuvee.com, http://www.yuvee.
`com/builtin1.shtml,
`http: //www.yuvee .com/builtiinib. shtml,
`http://www.yuvee.com/testdrive2.shtrnl.
`Zhai, S., Smith, BA, and Selker, T. Improving Browsing Perfor
`mance: A study of four input devices for scrolling and pointing tasks,
`Proceedings of the IFIP TC13 Interantional Conference on Human
`Computer Interaction, p. 286-293, Jul. 14-18, 1997.
`Zicorp4eZiTap, http://www.Zicorp.com/eZiTap.htm.
`Communication from the European Patent Of?ce dated Feb. 11,
`2009.
`
`SCEA Ex. 1001 Page 4
`
`

`

`U.S. Patent
`
`3,mF
`
`Mm
`
`7,SU
`
`7
`
`2B2
`
`
`
` zmm2m5gm
`
`aa
`
`Hmm
`
`
`
`3:05:003%:598005
`
`a
`
`EIH
`
`
`
`
`
`9mow/omomeon20%
`
`9ll6,EM
`
`
`
`%883mm39:canmowfiBE
`
`aa.
`
`SCEA EX. 1001 Page 5
`
`SCEA Ex. 1001 Page 5
`
`
`
`
`

`

`US. Patent
`
`Feb. 23, 2010
`
`Sheet 2 of9
`
`US 7,667,692 B2
`
`HOwwUOOHAH
`
`25
`
`SIH A
`
`‘520.5200 5&5
`3
`
`
`
`39; $0M
`
`SCEA Ex. 1001 Page 6
`
`

`

`U S Patent
`
`Feb. 23, 2010
`
`Sheet 3 0f9
`
`US 7,667,692 B2
`
`Fig. 3b
`
`302
`
`SCEA Ex. 1001 Page 7
`
`

`

`US. Patent
`
`Feb. 23, 2010
`
`Sheet 4 0f9
`
`US 7,667,692 B2
`
`Fig. 3c
`
`SCEA Ex. 1001 Page 8
`
`

`

`U S. Patent
`
`Feb. 23, 2010
`
`Sheet 5 0f9
`
`US 7,667,692 B2
`
`SCEA Ex. 1001 Page 9
`
`

`

`U S. Patent
`
`Feb. 23, 2010
`
`Sheet 6 0f9
`
`US 7,667,692 B2
`
`,6,
`
`kw
`
`m .5
`
`
`mam/Man 6,60%
`7.”, 6 iii:
`
`SCEA Ex. 1001 Page 10
`
`

`

`U S. Patent
`
`Feb. 23, 2010
`
`Sheet 7 0f9
`
`US 7,667,692 B2
`
`@ .5 8 .3
`
`6, ,5,
`
`SCEA Ex. 1001 Page 11
`
`

`

`U S Patent
`
`Feb. 23, 2010
`
`Sheet 8 0f9
`
`US 7,667,692 B2
`
`3
`
`SCEA Ex. 1001 Page 12
`
`

`

`US. Patent
`
`Feb. 23, 2010
`
`Sheet 9 0f9
`
`US 7,667,692 B2
`
`502
`
`§
`
`
`
`g 625 66562
`
`V1036 Z Eooq
`
`%
`
`REES
`
`aw
`
`w .wE
`
`mg
`
`3585
`
`mmH
`
`BPEm
`
`Q
`
`SCEA Ex. 1001 Page 13
`
`

`

`US 7,667,692 B2
`
`1
`HUMAN INTERFACE SYSTEM
`
`PRIORITY INFORMATION
`
`This application is a continuation of US. application Ser.
`No. 11/747,863, ?led on May 11, 2007, and claims priority
`under 35 U.S.C. 119(e) to US. application Ser. No. 11/747,
`863, ?led on May 11, 2007; and to US. application Ser. No.
`10/699,555, ?led on Oct. 31, 2003; and the disclosure of the
`prior applications is considered part of (and is incorporated
`by reference in) the disclosure of this application.
`
`BACKGROUND
`
`The following description relates to human interface and
`input systems for electronic devices, particularly hand-held
`electronic devices, such as cell phones, personal digital assis
`tants (“PDAs”), pocket personal computers, smart phones,
`hand-held game devices, bar-code readers, remote controls,
`and other similar input devices having a keypad or one or
`more input elements.
`Electronic devices have become increasingly sophisticated
`and physically smaller due in part to a decrease in the price of
`processing poWer and a concurrent increase in demand by
`consumers for smaller devices. Such devices, hoWever, tend
`to be limited in function and utility by the user’s ability to
`interface With the device for data input (e.g., text, numeric,
`and functional input) and/or device control, Which becomes
`increasingly more di?icult to do as the available space on the
`device’s surface for positioning the input elements, Which are
`used for data input and/or device control, continues to
`decrease.
`Various human interface and input systems and techniques
`for hand-held electronic devices have been developed for data
`input and device control. These include miniature keyboards
`and keypads used in combination With chordal input tech
`niques, modal input techniques and/or smart keys; and touch
`screens used in combination With on-screen keyboard or key
`pad softWare or hand-Writing recognition softWare.
`Keyboard or Key Pad Used With Chordal, Modal and Smart
`Key Techniques
`Miniature keyboards and keypads are similar to their stan
`dard full-siZe versionsiie, a keyboard generally has a full
`set or substantially full set of numeric, character, and func
`tional input elements, While key pads typically have a reduced
`set of numeric, character and/or functional input elements
`compared to keyboards. These miniature input devices typi
`cally are designed to ?t the available space on one surface of
`a hand-held electronic device or are designed as small, easily
`transportable, external plug-in devices. Thus, as hand-held
`electronic devices become smaller, the siZe of the input ele
`ments typically has been reduced in order for the desired
`number of input elements to ?t on one surface of the elec
`tronic device.
`For data input and device control, miniature keyboards and
`keypads typically either require one of tWo input tech
`niquesiuse of one or more thumbs or ?ngers to press the
`desired input elements or use of a stylus to “peck” the desired
`input elements (Which is usually done Where the input ele
`ment is of smaller siZe). Various techniques, such as chordal
`input techniques, modal input techniques and smart keys,
`have been developed and implemented to improve the e?i
`ciency and effectiveness of using miniature keyboards and
`keypads.
`Chordal Input Techniques
`Chordal input techniques generally are based upon the
`principle that characters, symbols, Words, phrases or concepts
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`can be represented by a reduced set of input elements. Thus,
`by only having to press a reduced combination of input ele
`ments, functionality can be increased and quicker and more
`accurate data input can be realiZed. Chordal input techniques
`can be used on any keyboard or keypad con?guration or any
`device having more than one input element, and typically
`results in feWer input elements or more functions compared to
`conventional keyboards or keypads. An example of an elec
`tronic device using tWo-handed chordal input techniques is a
`court reporter or stenographer’ s typeWriter. One chordal input
`technique using a keypad to decrease the number of actua
`tions to achieve a large number of functions is described in
`US. Pat. No. 5,973,621 to Levy, entitled “Compact Keyed
`Input Device,” Which is incorporated herein by reference.
`Modal Input Techniques
`Modal input techniques are based on the concept that func
`tions of the electronic device, e.g., text messaging in a cell
`phone or PDA, can be accessed by pressing a particular input
`element (or combination of elements) on a keyboard or key
`pad. Once that particular input element is pressed, the func
`tionality of all or a portion of the input elements on the
`keyboard or keypad may change. Modal techniques typically
`are used in calculators, cell-phones, and PDAs. For example,
`in cell phones, a modal technique called multi-tap is common,
`in Which individual input elements on the keypad are associ
`ated With multiple symbols, such as characters, letters, num
`bers, icons or other types of symbols, Which tends to reduce
`the number of input elements required to achieve the desired
`functions, e. g., a tWelve-input-element keypad can be used to
`represent all letters of the English alphabet and the decimal
`digits. A user can input a desired symbol Within a set of
`symbols associated With a certain input element by tapping on
`that particular input element With a thumb, ?nger, or stylus,
`one or more times to input the desired character. Thus, if a
`user desires to send a text message, the user may press a
`functional input element, e.g., a mode key, to access the text
`messaging functionality of the cell phone and then tap an
`individual input element one or more times to select the
`associated symbol for input. The number of taps needed to
`input a particular symbol may differ depending on the lan
`guage character set chosen. For example, Japanese keypad or
`keyboards typically require a minimum set of 46 characters
`for text input, While English or American keyboards and
`keypads usually require a minimum set of 26 characters for
`text input. These modal input techniques have gained some
`popularity as users perform more text functions.
`Smart Keys
`Smart keys are typically used on keypads and refer to a
`single key or combination of keys that, When pressed, predict
`the users next logical action. Some implementations Work
`better than others and some applications reduce the number of
`keystrokes required to complete a function better than others.
`Word-predictor softWare, for example, attempts to predict the
`Word or character the user intends to input based upon one or
`more letters inputted by the user and the likely probabilities
`Within a given language. The probability of the softWare
`guessing correctly increases With the length of the Word or
`number of letters or characters inputted. In a device using
`smart keys on the keypad, a user may tap the keys 2, 2 and 8
`in sequence to generate the Word “cat” and the device Would
`display that Word ?rst because it is usually the most common
`combination, Whereas the Word “bat,” Which can be generated
`by pressing the same keys, Would not be displayed ?rst
`because it is not as common. Also, the Word “cat” may be
`displayed after pressing the 2 key the second time based on a
`guess by the Word-predictor softWare.
`
`SCEA Ex. 1001 Page 14
`
`

`

`US 7,667,692 B2
`
`3
`Smart keys also are typically used for Japanese data input
`where a user phonetically inputs letters representing the
`sound of the Japanese character (e.g., a Kanji character).
`Based on the inputted letters, the predictor software guesses
`the Japanese character. To select the character, a user would
`press the accept button or use the scrolling function to go to
`the next character with a similar set of phonetic inputs.
`Touch Screen Using On-Screen Keyboard or Handwriting
`Recognition Software
`Using on-screen keyboard or keypad software with a touch
`screen offers users the ability to enter data with ?ngers or
`thumbs on a screen-siZed keyboard or buttons, allowing faster
`data input without a stylus or physical keyboard or keypad
`accessory; while using handwriting recognition software
`with a touch screen, such as Graf?tiO on the Palm operating
`system, offers users the ability to enter text with a stylus by
`writing the text directly on the touch screen. Touch screens
`usually consume more power and are more expensive than
`non touch-sensitive screens. This higher power consumption
`can be a problem for hand-held electronic devices, which
`typically have limited power resources. Moreover, touch
`screens usually require the user to use both hands (e.g., one
`hand is used to hold and steady the device while the other
`hand is used to grasp the stylus), which is generally undesir
`able fo