US007218313B2
`
`(12)
`
`United States Patent
`Marcus et al.
`
`(10) Patent N0.:
`(45) Date of Patent:
`
`US 7,218,313 B2
`May 15, 2007
`
`(54) HUMAN INTERFACE SYSTEM
`
`6,084,576 A
`
`7/2000 Leu et al.
`
`(75) Inventors: Beth Marcus, Bedford, MA (US); W.
`David Lee, Newton, MA (U S)
`
`(73) Assignee: Zeetoo, Inc., Bedford, MA (US)
`
`( * ) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U_S~C_ 154(1)) by 502 days'
`
`.
`(Commued)
`FOREIGN PATENT DOCUMENTS
`
`EP
`EP
`EP
`WO
`WO
`
`0585730
`1103883
`1 293 882 A2
`91/05303 A1
`03/042805
`
`3/1994
`5/2001
`3/2003
`4/1991
`5/2003
`
`(21) Appl. No.: 10/699,555
`(22) Filed:
`Oct. 31, 2003
`
`(65)
`
`Pl‘iOl‘ PllbliC?tiOIl Data
`
`Us Zoos/0093846 A1
`
`May 5’ 2005
`
`OTHER PUBLICATIONS
`Andriy Pavlovych, Wolfgang StiirZlinger: Less-Tap: A Fast and
`Easy-to-learn Text Input Technique for Phones. Graphics Interface
`2003,
`97-104
`http://www.graphicsinterface.org/cgi-bin/
`DownloadPaper?name:2003/170/paper170.pdf.
`
`(51) Int. Cl.
`(2006.01)
`G09G 5/00
`(52) us. Cl. ..................................... .. 345/169; 345/168
`(58) Field of Classi?cation Search .............. .. 345/156
`
`(Continued)
`R_ d O _
`E _
`P _
`_
`("my mmmerf 1“? 0 509°
`(74) Ammey’ Age”: "r F’rmiplsh & Rlchardson P'C'
`
`345/169, 173, 168; 400/472; 341/22
`'
`'
`See appl1cat1on ?le for complete search h1story.
`
`(57)
`
`ABSTRACT
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`4,793,312 A 12/1988 Doinaga et al.
`4,867,028 A
`9/1989 Jones
`4,891,777 A
`1/1990 Lapeyre
`4,896,554 A
`1/1990 Culver
`4,912,462 A
`3/1990 WashiZuka et al.
`5,189,416 A
`2/1993 Estes
`5,365,589 A 11/1994 Gutowitz
`5,432,510 A
`7/1995 Matthews
`5,473,325 A 12/1995 McAlindon
`5,512,919 A
`4/1996 Araki
`5,515,305 A *
`5/1996 Register et a1. ........... .. 708/145
`5,612,690 A
`3/1997 Levy
`5,782,642 A
`7/1998 Goren
`5,824,931 A 10/1998 Papadopoulos
`5,859,629 A
`1/1999 TognaZZini
`5,900,864 A
`5/1999 Macdonald
`5,973,621 A 10/1999 Levy
`6,005,496 A 12/1999 Hargreaves et al.
`
`A human interface con?gured to optimize a biomechanical
`e?fect of a human user’s opposing thumb and ?ngers by
`including, on one surface, one or more software con?gurable
`input elements manipulatable by a user’s thumb(s) or a
`stylus, and, on another surface, one or more software con
`?gurable selection elements manipulatable by a user’s
`?nger(s). A selection element may be a pressure sensor pad
`con?gurable to represent delineated active areas that are
`mapped to one or more input functions. Shape changing
`media may be provided to permit a user to tactilely dis
`criminate between delineated active areas. Tactile feedback
`may be provided to a user through palpable detents, vibra
`tory or force producing units. Inputting data may include
`mapping each selection element to a shift function, mapping
`each input element to text functions, and using the selection
`elements to shift between text functions associated with an
`input element to input a desired text function.
`
`60 Claims, 9 Drawing Sheets
`
`SCEA Ex. 1001 Page 1
`
`

`
`US 7,218,313 B2
`Page 2
`
`US. PATENT DOCUMENTS
`
`8/2000 Phillipps ................... .. 345/156
`6,107,988 A *
`9/2000 Balakrishnan et al.
`6,115,028 A
`2/2001 Harrison
`6,184,804 B1
`4/2001 GutowitZ
`6,219,731 B1
`5/2001 Hsieh
`6,228,709 B1
`5/2001 Mailman
`6,232,956 B1
`6,297,752 B1* 10/2001 Ni ............................. .. 341/22
`6,377,685 B1
`4/2002 Krishnan
`RE37,723 E
`6/2002 Goren
`6,512,511 B2
`1/2003 Willner et a1.
`6,520,699 B2
`2/2003 Abe
`6,541,715 B2
`4/2003 Swanson
`6,542,091 B1
`4/2003 Rasanen
`6,546,239 B1
`4/2003 PaZdersky et al.
`6,573,844 B1
`6/2003 Venolia et al.
`6,606,486 B1
`8/2003 Cubbage et al.
`6,654,733 B1
`11/2003 Goodman et al.
`6,703,963 B2
`3/2004 Higginson
`6,738,045 B2
`5/2004 Hinckley et al.
`6,741,235 B1
`5/2004 Goren
`6,760,013 B2
`7/2004 Willner et a1.
`6,865,718 B2
`3/2005 Montalcini
`6,885,317 B1
`4/2005 GutowitZ
`6,885,318 B2
`4/2005 Bickerton
`6,909,424 B2 *
`6/2005 Liebenow et al. ........ .. 345/169
`6,911,608 B2
`6/2005 Levy
`6,947,028 B2 *
`9/2005 Shkolnikov ............... .. 345/156
`6,980,200 B2 12/2005 Goren
`11/2002 Pallakolf
`2002/0163504 A1
`2003/0020692 A1
`1/2003 Grif?n et al.
`2003/0048205 A1
`3/2003 He
`2003/0061103 A1
`3/2003 Kanai
`2003/0169188 A1
`9/2003 Chang et al.
`2003/0193418 A1 10/2003 Shi
`2004/0208681 A1* 10/2004 Dechene ................... .. 400/472
`
`OTHER PUBLICATIONS
`
`Scott MacKenZie, Hedy Kober, Derek Smith, Terry Jones, Eugene
`Skepner, LetterWise: pre?x-based disambiguation for mobile text
`input, Proceedings of the 14th annual ACM symposium on User
`interface software and technology, Nov. 11-14, 2001, Orlando,
`Florida.
`H. Kober, E. Skepner, T. Jones, H. GutowitZ, S. MacKenZie, 2001.
`Linguistically Optimized Text Entry on a Cell Phone. In Proceed
`ings of the CHI 2001. http://www.eatoni.com/research/chi.pdf.
`Mikael Goldstein and Didier Chincholle The Finger-Joint-Gesture
`Wearable Keypad. Ericsson Radio Systems.
`Rosenberg, R. (1998). Computing without Mice and Keyboards:
`Text and Graphic Input Devices for Mobile Computing. Ph.D.
`Thesis, Dept. of Computer Science, University College, London,
`1998. http://www.obscure.org/rosenburg/.
`MacKay, B., Dearman, D., Inkpen, K., and Watters, C. 2005. Walk
`’n scroll: a comparison of software-based navigation techniques for
`different levels of mobility. In Proceedings of the 7th international
`Conference on Human Computer interaction with Mobile Devices
`& Amp; Services (SalZburg, Austria, Sep. 19-22, 2005). MobileHCI
`’05, vol. 111. ACM Press, New York, NY, 183-190. http://portal.
`acm.org/citation.cfm?id:1085808&coll:GUIDE&dl:GUIDE
`&CFID:66591340&CFTOKEN:6294934.
`KranZ, M., Holleis, P., Schmidt, A. “DistScrollia new one-handed
`interaction device”. In Proceedings of the 5th International Work
`shop on Smart Appliances and Wearable Computing, Jun. 10, 2005.
`http://www.heilab.org/documents/DistScrol
`lAnewOneHandedInteractionDevice-KranZHolleisSchmidt
`IWSAWC2005 .pdf.
`Daniel Fallmana, Andreas Lund, Mikael Wiberg, ScrollPad: Tan
`gible Scrolling with Mobile Devices, Proceedings of the Proceed
`ings of the 37th Annual Hawaii International Conference on System
`Sciences (HICSS’04)iTrack 9, p. 90286.3, Jan. 5-8, 2004. http://
`portal.acm/org/citation.cfm?id:963347&coll:GUIDE&dl:GUIDE
`
`http://daniel.fallman.
`
`&CFID:66483658&CFTOKEN:36023921
`org/resources/papers/fallman-hicss37.pdf.
`Chipman, L. E., Bederson, B. B., and Golbeck, J. A. 2004. SlideBar:
`analysis ofa linear input device. Behav. Inf. Tech. 23, 1 (Jan. 2004),
`1-9. http://portal.acm.org/citation.cfm?id:993182.993184# http://
`www.cs.umd.edu/Library/TRs/CS-TR-4471/CS-TR-4471.pdf.
`Orientation-based interaction for Mobile Devices. J. Darnauer, S.
`Garrity and T. Jim, Stanford University. http://hci.stanford.edu/srld
`cs377a-mobile/project/?nal/darnauer-garrity-kim.pdf.
`Rekimoto, J. Tilting operations for small screen interfaces. Proceed
`ings of the 9th annual ACM symposium on User Interface software
`and technology, pp. 167-168, Nov. 6-8, 1996, Seattle. http://portal.
`acm.org/citation.cfm?id:237115&coll:GUIDE&dl:GUIDE
`&CFID:6648365 8&CFTOKENI36023921.
`Hinckley, K., Pierce, J. Horvits, E., Sinclair, M. Foreground and
`Background Interaction with Sensor-enhanced Mobile Devices,
`ACM TOCHI (Transactions on Computer-Human Interaction) Spe
`cial Issue on Sensor-Based Interaction, 12 (1), Mar. 2005, pp. 31-52.
`http://portal.acm.org/citation.cfm?id:1057240&coll:GUIDE
`&dl:GUIDE&CFID:66591340&CFTOKEN:6294934.
`Hinckley, K., Cutrell, E., Bathiche, S., and Muss, T. 2002. Quan
`titative analysis of scrolling techniques. In Proceedings of the
`SIGCHI Conference on Human Factors in Computing Systems:
`Changing Our World, Changing Ourselves (Minneapolis, Minne
`sota, USA, Apr. 20-25, 2002). CHI ’02. ACM Press, New York, NY,
`65-72. http://doi.acm.org/10.1145/503376.503389.
`Harrison, B. L., Fishkin, K. P., Gujar, A., Mochon, C., and Want, R.
`1998. Squeeze me, hold me, tilt me! An exploration of manipulative
`user interfaces. In Proceedings of the SIGGCHI Conference on
`Human Factors in Computing Systems (Los Angeles, California,
`United States, Apr. 18-23, 1998). C. Karat, A. Lund, J. CoutaZ, and
`J. Karat, Eds. Conference on Human Factors in Computing Sys
`tems. ACM Press/Addison-Wesley Publishing Co., New York, NY,
`17-24. http://portal.acmorg/citation.cfm?id=274647&coll=Portal
`&dl:GUIDE&CFID:66588306&CFTOKEN:73460863
`&CFID:66588306&CFTOKEN:73460863#.
`Kiyokuni Kawachiya, Hiroshi Ishikawa, NaviPoint: an input device
`for mobile information browsing, Proceedings of the SIGCHI
`conference on Human factors in computing systems, p. 1-8, Apr.
`18-23, 1998, Los Angeles, California, United States http://portal.
`acm.org/citation.cfm?id:274645&coll:Portal&dl:GUIDE
`&CFID:66588306&CFTOKEN:73460863.
`Hinkley, K., Pierce, J., Sinclair, M., and HorvitZ, E. Sensing
`Techniques for Mobile Interaction. UIST 2000: ACM Symposium
`on User Interface Software and Technology, pp. 91-100. http://
`portal.acm.org/citation.cfm?id:354417&coll:GUIDE&dl:GUIDE
`&CFID:6648365 8&CFTOKENI36023921.
`Baillie, L., KuncZier, H., and Anegg, H. 2005. Rolling, rotating and
`imagining in a virtual mobile world. In Proceedings of the 7th
`international Conference on Human Computer interaction with
`Mobile Devices & Amp; Services (SalZburg, Austria, Sep. 19-22,
`2005). MobileHCI ’05, vol. 111. ACM Press, New York, NY,
`283-286. http://doi.acm.org/10.1145/1085777.1085833.
`Amy K. Karlson, Benjamin B. Bederson, John SanGiovanni, 2004.
`AppLens and LaunchTile: Two Designs for One-Handed Thumb
`Use on Small Devices http://hcil.cs.umd.edu/trs/2004-37/2004-37.
`htrnl.
`Virpi Roto, Nokia Research. Browsing on Mobile Phones. http://
`www.research.att.com/~rjana/WF12iPaper1.pdf.
`Qualcomm.Slingshot
`http://wireless.ign.com/articles/657/
`657041p1.html.
`GamePad http://www.mobilemag.com/content/100/345/C5578/.
`Kyocera Candid KX16 http://www.mobilemag.com/content/ 100/
`340/C4392/.
`XEG Mobile Phone Pad http://us.giZmodo.com/gadgets/cellphones/
`gaming-on-the-go-with-xeg-136414.php
`http://www.
`akihabaranews.com/en/news-10615
`EXG%2C+the+mobile+phone+pad.html.
`Samsung Game Pad http://www.cellphonemall.net/wireless/store/
`accessorydetail.asp?id:23 1 98&phoneid:3 34.
`Thumbscript http://www.thumbscript.com/index.html http://www.
`thumbscript.com/howitworks.htrnl
`http://www.thumbscript.com/
`technotes.html.
`
`SCEA Ex. 1001 Page 2
`
`

`
`US 7,218,313 B2
`Page 3
`
`http://www.
`
`http://www.
`
`http://www.exideas.com/ME/indexhtml
`Exideas
`exideas.com/ME/HardKey.html.
`NE-Ware
`http://www.n-e-ware.com/Downloads/KeySticld330/
`KSUserManual330i01.pdf.
`Synaptics
`http://www.synaptics.com/products/pdf/
`mobiletouchipbpdf.
`Atrua: sensor company http://www.atrua.com/s-mobilephones.
`htrnl.
`Varatouch: sensor company http://www.esato.com/news/article.
`php/id:388.
`Eleksen http://www.eleksen.com.
`SHARP Vodafone ZTCJ01 http://www.slashpbone.com/93/3123.
`htrnl.
`The GamePad http://www.kotaku.com/gaming/cell-phones/killer
`cell-phone-game-controller-130968.php.
`Sony Clie
`game controller PEGA-GC10
`palminfocenter.com/viewistory.asp?ID:4295.
`Bluetooth
`GPS
`http://mobilitytoday.com/news/005986/
`mobilityibuyGPSnowii-BlueibluetoothiGPS.
`Alphagrip http://www.alphagrips.com/AlphagripAG5UsersManual.
`pdf.
`http://www.tegic.com/pdfs/salessheets/T9
`TegiciT9
`&20Adaptive%20Text%20Input%20Sales%20Sheet%201 .pdf
`http://www.tegic.com/pdfs/salessheets/
`T9%20Adaptive%20Text%20Input%20Sales%20Sheet%202.pdf
`http://www.tegic.com/pdfs/salessheets/
`T9%20Adaptive%20Text%20Input%20Sales%20Sheet%203.pdf
`http://www.tegic.com/pdfs/salessheets/
`Sloppy%20Type%20Sales%20Sheet.pdf.
`Zicorp4eZiTap http://www.Zicorp.com/eZiTap.htm.
`MotorolaiiTAP
`http://news.Zdnet.co.uk/hardware/mobile/
`0,39020360,39118435,00.htm.
`DigitWireless: FastTap http://www.digitwireless.com/?ash/down
`load/fastappdf.
`Microth KeyWheel http://www.microth.com/circumscript/over
`view.asp.
`Yuvee; special keypad layout www.yuvee.com http://www.yuvee.
`com/builtin1.shtml http://www.yuvee.combuiltiinib.shtml http://
`www.yuvee.com/testdrive2.shtrnl.
`Twiddler http://www.handykey.com/ http://www.handykey.com/
`Keymap.pdf.
`Sengital Ltd. Tilt sensor replacement for PDA http://sengital.manu
`facturer. globalsourcescom/ si/ 6008823 523892/ProductDetail/PDA
`keyboard/productiid-1001050135/action-GetProduct.htm.
`Howard.co.kriThe mouse phone http:www.howard.co.kr/com
`puter/mouse/mousephone.htrn.
`Nokia 6620 with a control stick http://nds2.nokia:com/?les/support/
`nam/phones/guides/6620iUSien.PDF.
`Sega into cell phones http://www.phoneyworld.com/newspage.
`aspx?n:1745 .
`Prevalent Devices LLC http://www.prevalentdevices.com/manual3
`5-06.pdf.
`M. Kolsch, M. Turk, 2002. Keyboards without Keyboards: A Survey
`of Virtual Keyboards. UCSB Technical Report 2002-21, Jul. 12,
`2002. http://www.cs.ucsb.edu/research/techireports/reports/2002
`21.pdf.
`Jeong-Hoon Shin and Kwang-Seok Hong. An improved alphanu
`meric input algorithm using gloves. http://www.complexity.org.au/
`conference/upload/shin01/shin01 .pdf.
`C. MetZger, M. Anderson, T. Starner, 2004. FreeDigiter: A Contact
`Free Device for Gesture Control. Eighth IEEE International Syrn
`posium on Wearable Computers (ISWC’04) pp. 81-21. http://www.
`wirelessrerc.gatech.edu/projects/development/D1?les/iswc04
`freedigiterpdf.
`Rakkolainen, I. (2003). MobiVRia novel user interface concept
`for mobile computing. In: Bieber, K. & Kirste, T. (eds.), Proceed
`ings of the 4th International Workshop on Mobile Computing, IMC
`2003, Jun. 17-18, 2003, Rostock, Germany, pp. 107-112. http://
`www.cs.tut.?/~ira/IMC2003.pdf.
`Lumsden, J ., Gammell, A. (2004). Mobile Note Taking: Investigat
`ing the E?icacy of a Mobile Text Entry. In Proc. of Mobile
`Human-Computer Interaction (MobileHCI2004), Glasgow, UK,
`Sep. 2004: In S. Brewster and M. Dunlop (Eds.). Mobile Human
`
`Computer-InteractioniMobileHCI 2004, Lecture Notes in Com
`puter Science, vol. 3160, Berlin: Springer, 156-168.
`MacKenZie, I. S., & Soukorelf, R. W. Phrase sets for evaluating text
`entry techniques. Extended Abstracts of the ACM Conference on
`Human Factors in Computing Systems4CHI 2003, pp. 754-755
`New York: ACM.
`MacKenZie, IS. (2002). KSPC (keystrokes per character) as a
`characteristic text entry techniques. Proceedings of the Fourth
`International Symposium on Human-Computer Interaction with
`Mobile Devices, pp. 195-210. Heidelberg, Germany: Springer
`Verlag.
`Soukoreff, R. W. and MacKenZie, I. S. (2004). Recent Develop
`ments in Text-Entry Error Rate Measurement. CHI 2004, Late
`Breaking Results Paper, Vienna Austria, Apr. 24-29, 2004.
`Lee, S. and Hong S.H.. Chording as a Text Entry Method in Mobile
`Phones. In Proceedings of the MobileHCI 2004: 6th International
`Symposium, Glasgow, UK, Sep. 13-16, 2004.
`Green, N., Kruger, J., Faldu, C., and St. Amant, R. 2004. Areduced
`QWERTY keyboard for mobile text entry. In CHI ’04 Extended
`Abstracts on Human Factors in Computing Systems (Vienna, Aus
`tria, Apr. 24-29, 2004). CHI’04. ACM Press, New York, NY,
`1429-1432.
`http://portal.acm.org/citation.cfm?id:986082
`&coll:GUIDE&d1:GUIDE&CFID:66591340
`&CFTOKEN:6294934.
`Partridge, K., Catterjee, S., SaZawal, V., Borriello, G., and Want, R.
`TiltType: accelerometer-supported text entry for very small devices,
`Proceedings of the 15th annual ACM symposium on User interface
`software and technology, Oct. 27-30, 2002, Paris, France.
`Mikael Goldstein, Didier Chincholle, Morten Back (2000). Assess
`ing Two New Wearable Input Paradigms: The Finger-Joint-Gesture
`Palm-Keypad Glove-and-the-Invisible Phone Clock. Personal and
`Ubiquitous Computing, vol. 4, Issue 2/3.
`Bartlett, J. F. 2000. Rock ’n Scroll Is Here to Stay. IEEE Comput.
`Graph. Appl. 20, 3 (May 2000), 40-45. http://portal.acm.org/cita
`tion.cfm?id:618728&coll:Portal&d1:GUIDE&CFID:66588306
`&CFTOKEN:73460863#.
`Eslambolchilar, P., Murray-Smith, R. (2004). Tilt-Based Automatic
`Zooming and Scaling in Mobile DevicesiA state-space implemen
`tation.
`In Proc. of Mobile Human-Computer Interaction
`(MobileHCI 2004), Glasgow, UK, Sep. 2004: In S. Brewster and M.
`Dunlop (Eds.). Mobile Human-Computer-InteractioniMobileHCI
`2004, Lecture Notes in Computer Science, vol. 3160, Berlin:
`Springer, 120-131.
`Zhai, S., Smith, BA, and Selker, T. Improving Browsing Perfor
`mance: A study of four input devices for scrolling and pointing
`tasks, Proceedings of the IFIP TC13 Interantional Conference on
`Human-Computer Interaction, p. 286-293, Jul. 14-18, 1997.
`Wobbrock, J. O., ForliZZi, J. Hudson, S. E. and Myers, B. A.
`WebThumb: interaction techniques for small-screen browsers. Proc.
`UIST, ACM Press (2002), 205-208.
`Lee, S., Hong, SH, and Jeon, J. W. 2002. Designing a universal
`keyboard using chording gloves. SIGCAPH Comput. Phys. Handi
`cap., 73-74 (Jun. 2002), 142-147. http://doi.acm.org/10.1145/
`960201.957230.
`Pirhonen, A., Brewster, S., and Holguin, C. 2002. Gestural and
`audio metaphors as a means of control for mobile devices. In
`Proceedings of the SIGCHI Conference on Human Factors in
`Computing Systems: Changing Our World, Changing Ourselves
`(Minneapolis, Minnesota, USA, Apr. 20-25, 2002). CHI ’02. ACM
`Press, New York, NY, 291-298. http://doi.acm.org/10.1145/503376.
`503428.
`Kjeldskov, J. and Graham, C. (2003). A Review of Mobile HCI
`Research Methods. In Proc. of Mobile Human-Computer Interac
`tion (MobileHCI 2003), Udine Italy, Sep. 2003: In L. Chittaro (Ed.).
`Mobile Human-Computer-InteractioniMobileHCI 2003, Lecture
`Notes in Computer Science, vol. 2795, Berlin: Springer, 317-335.
`Donner, J. (2005). Research Approaches to Mobile Use in Devel
`oping World: A Review of the Literature. International Conference
`on Mobile Communication and Asian Modernities City University
`of Hong Kong, Jun. 7-8, 2005.
`Min, Lin and Sears, Andrew (2005). Graphics Matter: A Case Study
`of Mobile Phone Keypad Design for Chinese Input. CHI 2005, Late
`Breaking Results: Posters, Portland, Oregon. Apr. 2-7, 2005.
`
`SCEA Ex. 1001 Page 3
`
`

`
`US 7,218,313 B2
`Page 4
`
`Mackenzie, S., & Soukoreff, W. (2002). Text entry for mobile
`computing: Models and methods, theory and practice. Human
`Computer Interaction. 17. p. 147-198. http://www.yorku.ca/macld
`hci3-2002.pdf.
`Starner, T. “Keyboards Redux: Fast Mobile Text Entry”. Pervasive
`Computing, Jul.-Sep. 2004, pp. 97-101. http://www.cc.gatech.edu/
`fac/ Thad. Starner/ p/ magaZine/ 2004 -3 -keybo ard-redux.pdf.
`Lee Butts, Andy Cockburn, An evaluation of mobile phone text
`input methods, Third Australasian conference on User interfaces, p.
`55-59, Jan. 01, 2002, Melbourne, Victoria, Australia http://www.
`crpit.com/confpapers/CRPITV7Butts.pdf.
`Chording and Tilting-Daniel Wigdor (thesis)i2004-describes
`chordtap and tilttap (also covered in depth in the paper referenced
`below) http ://www. dgp .toronto .edu/~dwigdor/research/thesis/sub
`mitted.html.
`Wigdor, D. and Balakrishnan, R. “A Comparison of Consecutive
`and Concurrent Input Text Entry Techniques for Mobile Phones”,
`Conference on Human Factors, Apr. 24-29, 2004, vol. 6, No. 1, pp.
`81-88 http://portal.acm.org/citation.cfm?id:985703 http://www.
`dgp.toronto.edu/~ravin/papers/chi2004iconcurrenttextinput.pdf
`OnisZcZak, A., & MacKenZie, I. S. (2004). A comparison of two
`input methods for keypads on mobile devices. Proceedings of
`NordiCHI 2004, pp. 101-104. New York: ACM. http://www.yorku.
`ca/macldnordichi2004.pdf.
`K. Lyons. Everyday wearable computer use: A case study of an
`expert user. In Proceedings of Mobile HCI 2003, pp. 61-75, 2003.
`http://www.cc.gatech.edu/ccg/publications/everydayicasepdf.
`K. Lyons, T. Starner, D. Plaisted, J. Fusia, A. Lyons, A. Drew, E. W.
`Looney, 2004. “Twiddler Typing: One-Handed Chording Text Entry
`for Mobile Phones,” Proc. Conf. Human Factors in Computing
`Systems (SIGCHI 01), ACM Press, 2004, pp. 671-678. http://www.
`cc.gatech.edu/fac/Thad.Starner/p/030i10iMTE/twiddler-chi04.
`pdf.
`
`Daniel Wigdor, Ravin Balakrishnan, TiltText: using tilt for text
`input to mobile phones, Proceedings of the 16th annual ACM
`symposium on User interface software and technology, p. 81-90,
`Nov. 2-5, 2003, Vancouver, Canada http://portal.acm.org/citation.
`cfm?id:964705
`http://www.dgp.toronto.edu/~ravin/papers/
`uist2003itilttext.pdf.
`M. D. Dunlop and A. Crossan, “Dictionary based text entry method
`for mobile phones”, published in Brewster, S.A., and Dunlop, M.D.,
`(editors). Proceedings of Second Workshop on Human Computer
`Interaction with Mobile Devices, Aug. 1999. http://www.cis.strath.
`ac.uld~mddfresearch/publications/99dunlopcrossan.pdf.
`Kjeldskov, J. Stage, J. (2004). New Techniques for Usability Evalu
`ation of Mobile Systems. International Journal of Human-Computer
`Studies, May 2004, 60 (5-6): 599-620.
`Buxton, “A Directory of Sources for Input Technologies”, Oct. 1,
`2003, http://www.billbuxton.com/InputSources.html.
`“Introducing the Dvorak Keyboard”, http://www.mwbrooks.com/
`dvorak/.
`Innovative Ergonomic Solutions, Cirque Pocket Keyboard, http://
`www.iesproducts.com/key-misc-pockethtml.
`Buxton, “Human Input to Computer Systems: Theories, Techniques
`and Technology”, http://www.billbuxton.com/inputManuscript.
`htlm.
`Buxton, “An Introduction to Human Input to Computers”, Apr. 6,
`1999, http://www.billbuxton.com/input01.Introduction.pdf.
`Unidenti?ed and Undated Document discussing alternative designs
`to QWERTY Keyboard, pp. 2-10.
`
`* cited by examiner
`
`SCEA Ex. 1001 Page 4
`
`

`
`U.S. Patent
`
`May 15, 2007
`
`Sheet 1 0f 9
`
`US 7,218,313 B2
`
`EIH
`
`6:938 :55
`
`$32G
`
`N_ M
`
`
`
`085G ow?ogm
`
`a
`
`M .3
`
`gm
`
`me ~
`
`H
`
`3
`
`homwooo?
`
`%
`
`822
`
`@0502
`
`$4
`
`g
`
`20%
`
`SCEA Ex. 1001 Page 5
`
`

`
`U.S. Patent
`
`May 15, 2007
`
`Sheet 2 0f 9
`
`US 7,218,313 B2
`
`m Emmi
`
`8330i
`
`%
`
`6295200 3&5
`3
`
`5&5 EoE
`
`bnEomm<
`
`
`g
`
`
`
`3&5 58%
`
`SCEA Ex. 1001 Page 6
`
`

`
`U.S. Patent
`
`May 15, 2007
`
`Sheet 3 0f 9
`
`US 7,218,313 B2
`
`@
`
`m E
`
`SCEA Ex. 1001 Page 7
`
`

`
`U.S. Patent
`
`May 15, 2007
`
`Sheet 4 0f 9
`
`US 7,218,313 B2
`
`Fig 3c
`
`SCEA Ex. 1001 Page 8
`
`

`
`U S. Patent
`
`M y15,2007
`
`Sheet 5 0m
`
`US 7,218,313 B2
`
`402
`
`454 450
`
`SCEA Ex. 1001 Page 9
`
`

`
`U.S. Patent
`
`M y 15, 2007
`
`Sheet 6 0f 9
`
`US 7,218,313 B2
`
`SCEA Ex. 1001 Page 10
`
`

`
`U S. Patent
`
`M y 15, 2007
`
`Sheet 7 0f 9
`
`US 7,218,313 B2
`
`Fig. 6b
`
`Fig 6a
`
`SCEA Ex. 1001 Page 11
`
`

`
`U.S. Patent
`
`M y 15, 2007
`
`Sheet 8 0f 9
`
`US 7,218,313 B2
`
`Fig. 7b
`
`SCEA Ex. 1001 Page 12
`
`

`
`U.S. Patent
`
`May 15, 2007
`
`Sheet 9 0f 9
`
`US 7,218,313 B2
`
`#62
`
`§
`
`
`
`
`
`.olwlw. v35 M62502
`
`mlmw
`
`
`
`x5502 130A
`
`g
`
`316G
`
`w PEwE
`
`BEBE
`
`mwh
`
`“I N
`
`00
`
`520m
`
`a
`
`SCEA Ex. 1001 Page 13
`
`

`
`US 7,218,313 B2
`
`1
`HUMAN INTERFACE SYSTEM
`
`BACKGROUND
`
`2
`input techniques is a court reporter or stenographer’s type
`Writer. One chordal input technique using a keypad to
`decrease the number of actuations to achieve a large number
`of functions is described in US. Pat. No. 5,973,621 to Levy,
`entitled “Compact Keyed Input Device,” Which is incorpo
`rated herein by reference.
`Modal Input Techniques
`Modal input techniques are based on the concept that
`functions of the electronic device, e.g., text messaging in a
`cell-phone or PDA, can be accessed by pressing a particular
`input element (or combination of elements) on a keyboard or
`keypad. Once that particular input element is pressed, the
`functionality of all or a portion of the input elements on the
`keyboard or keypad may change. Modal techniques typi
`cally are used in calculators, cell-phones, and PDAs. For
`example, in cell phones, a modal technique called multi-tap
`is common, in Which individual input elements on the
`keypad are associated With multiple symbols, such as char
`acters, letters, numbers, icons or other types of symbols,
`Which tends to reduce the number of input elements required
`to achieve the desired functions, e.g., a tWelve-input-element
`keypad can be used to represent all letters of the English
`alphabet and the decimal digits. A user can input a desired
`symbol Within a set of symbols associated With a certain
`input element by tapping on that particular input element
`With a thumb, ?nger, or stylus, one or more times to input the
`desired character. Thus, if a user desires to send a text
`message, the user may press a functional input element, e. g.,
`a mode key, to access the text messaging functionality of the
`cell phone and then tap an individual input element one or
`more times to select the associated symbol for input. The
`number of taps needed to input a particular symbol may
`differ depending on the language character set chosen. For
`example, Japanese keypad or keyboards typically require a
`minimum set of 46 characters for text input, While English
`or American keyboards and keypads usually require a mini
`mum set of 26 characters for text input. These modal input
`techniques have gained some popularity as users perform
`more text functions.
`Smart Keys
`Smart keys are typically used on keypads and refer to a
`single key or combination of keys that, When pressed,
`predict the users next logical action. Some implementations
`Work better than others and some applications reduce the
`number of keystrokes required to complete a function better
`than others. Word-predictor softWare, for example, attempts
`to predict the Word or character the user intends to input
`based upon one or more letters inputted by the user and the
`likely probabilities Within a given language. The probability
`of the softWare guessing correctly increases With the length
`of the Word or number of letters or characters inputted. In a
`device using smart keys on the keypad, a user may tap the
`keys 2, 2 and 8 in sequence to generate the Word “cat” and
`the device Would display that Word ?rst because it is usually
`the most common combination, Whereas the Word “bat,”
`Which can be generated by pressing the same keys, Would
`not be displayed ?rst because it is not as common. Also, the
`Word “cat” may be displayed after pressing the 2 key the
`second time based on a guess by the Word-predictor soft
`Ware.
`Smart keys also are typically used for Japanese data input
`Where a user phonetically inputs letters representing the
`sound of the Japanese character (e.g., a Kanji character).
`Based on the inputted letters, the predictor softWare guesses
`the Japanese character. To select the character, a user Would
`press the accept button or use the scrolling function to go to
`the next character With a similar set of phonetic inputs.
`
`The following description relates to human interface and
`input systems for electronic devices, particularly hand-held
`electronic devices, such as cell phones, personal digital
`assistants (“PDAs”), pocket personal computers, smart
`phones, hand-held game devices, bar-code readers, remote
`controls, and other similar input devices having a keypad or
`one or more input elements.
`Electronic devices have become increasingly sophisti
`cated and physically smaller due in part to a decrease in the
`price of processing poWer and a concurrent increase in
`demand by consumers for smaller devices. Such devices,
`hoWever, tend to be limited in function and utility by the
`user’s ability to interface With the device for data input (e.g.,
`text, numeric, and functional input) and/or device control,
`Which becomes increasingly more dif?cult to do as the
`available space on the device’s surface for positioning the
`input elements, Which are used for data input and/or device
`control, continues to decrease.
`Various human interface and input systems and tech
`niques for hand-held electronic devices have been developed
`for data input and device control. These include miniature
`keyboards and keypads used in combination With chordal
`input techniques, modal input techniques and/ or smart keys;
`and touch screens used in combination With on-screen
`keyboard or keypad softWare or hand-Writing recognition
`softWare.
`Keyboard or Key pad Used With Chordal, Modal and
`Smart Key Techniques
`Miniature keyboards and keypads are similar to their
`standard full-siZe versionsiie, a keyboard generally has a
`full set or substantially full set of numeric, character, and
`functional input elements, While key pads typically have a
`reduced set of numeric, character and/or functional input
`elements compared to keyboards. These miniature input
`devices typically are designed to ?t the available space on
`one surface of a hand-held electronic device or are designed
`as small, easily transportable, external plug-in devices.
`Thus, as hand-held electronic devices become smaller, the
`siZe of the input elements typically has been reduced in order
`for the desired number of input elements to ?t on one surface
`of the electronic device.
`For data input and device control, miniature keyboards
`and keypads typically either require one of tWo input tech
`niquesiuse of one or more thumbs or ?ngers to press the
`desired input elements or use of a stylus to “peck” the
`desired input elements (Which is usually done Where the
`input element is of smaller siZe). Various techniques, such as
`chordal input techniques, modal input techniques and smart
`keys, have been developed and implemented to improve the
`ef?ciency and effectiveness of using miniature keyboards
`and keypads.
`Chordal Input Techniques
`Chordal input techniques generally are based upon the
`principle that characters, symbols, Words, phrases or con
`cepts can be represented by a reduced set of input elements.
`Thus, by only having to press a reduced combination of
`input elements, functionality can be increased and quicker
`and more accurate data input can be realiZed. Chordal input
`techniques can be used on any keyboard or keypad con?gu
`ration or any device having more than one input element,
`and typically results in feWer input elements or more func
`tions compared to conventional keyboards or keypads. An
`example of an electronic device using tWo-handed chordal
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`SCEA Ex. 1001 Page 14
`
`

`
`US 7,218,313 B2
`
`3
`Touch Screen Using On-Screen Keyboard or Handwriting
`Recognition Software
`Using on-screen keyboard or keypad software with a
`touch screen offers users the ability to enter data with ?ngers
`or thumbs on a screen-siZed keyboard or buttons, allowing
`faster data input without a stylus or physical keyboard or
`keypad accessory; while using handwriting recognition soft
`ware with a touch screen, such as Graf?tiO on the Palm
`operating system, offers users the ability to enter text with a
`stylus by writing the text directly on the touch screen. Touch
`screens usually consume more power and are more expen
`sive than non touch-sensitive screens. This higher power
`consumption can be a problem for hand-held electronic
`devices, which typically have limited power resources.
`Moreover, touch screens usually require the user to use both
`hands (e.g., one hand is used to hold and steady the device
`while the other hand is used to grasp the stylus), which is
`generally undesirable for interfacing with and controlling
`one handed hand-held electronic device, such as cell phones.
`Handwriting recognition software has improved the slow
`ness and awkwardness inherent in stylus, ?nger or thumb
`input but other drawbacks still remain, such as high power
`consumption, the necessity to use both hands, and lack of
`tactile feedback to inform a user when an input element has
`been. Moreover, recognition software requires training to
`use properly, and, even then, still results in a high error rate.
`Game Control
`For game control, many of the above approaches have
`been used, but in most hand-held electronic devices, a user
`typically controls game play through the use of some form
`of input element, such as on a miniature keypad and/or
`directional pad (“D-pad”), which typically is located on the
`front surface of the device. Game control on some hand-held
`electronic devices, such as cell phones, is inherently one
`handed or at most two thumbed because of the siZe of the
`device, while game control on other hand-held electronic
`devices, such as PDAs and conventional game console
`controllers, is typically two-handed. The input elements
`associated with game control on these devices are typically
`digital even though analog input elements have been used on
`game controllers for PC and console game systems, such as
`Microsoft’s Xbox or Sony’s Play Station 2.
`
`SUMMARY
`
`The present inventors recogniZed that conventional
`human interface and input systems for hand-held electronic
`devices tended to be relatively in?exible, cumbersome, and
`inef?cient to use, among other reasons, because they were
`not designed to take advantag