1111111111111111 IIIIII IIIII 1111111111 11111 11111 111111111111111 11111 111111111111111 11111111
`US 20100017872Al
`
`c19) United States
`c12) Patent Application Publication
`Goertz et al.
`
`c10) Pub. No.: US 2010/0017872 Al
`Jan. 21, 2010
`(43) Pub. Date:
`
`(54) USER INTERFACE FOR MOBILE
`COMPUTER UNIT
`
`(75)
`
`Inventors:
`
`Magnus Goertz, Stockholm (SE);
`Joseph Shain, Rehovot (IL)
`
`Correspondence Address:
`Joseph Shain
`Neonode Technologies, Linnegatan 89
`Stockholm 11523 (SE)
`
`(73) Assignee:
`
`NEONODE TECHNOLOGIES,
`Stockholm (SE)
`
`(21) Appl. No.:
`
`12/486,033
`
`(22) Filed:
`
`Jun.17,2009
`
`Related U.S. Application Data
`
`(63) Continuation-in-part of application No. 10/315,250,
`filed on Dec. 10, 2002.
`
`(60) Provisional application No. 61/132,469, filed on Jun.
`19, 2008.
`
`Publication Classification
`
`(51)
`
`Int. Cl.
`G06F 3/048
`(2006.01)
`G06F 3101
`(2006.01)
`G06F 21100
`(2006.01)
`(2006.01)
`G06F 3/041
`(52) U.S. Cl. ........... 726/16; 715/784; 715/702; 345/173;
`715/773
`
`(57)
`
`ABSTRACT
`
`A computer readable medium storing a computer program
`with computer program code, which, when read by a mobile
`computer unit, allows the computer to present a user interface
`for the mobile computer unit. The user interface includes a
`touch sensitive area in which a portion of a first window and
`a portion of a second window are displayed. The user scrolls
`the first window when an object touches a corresponding
`location in the touch sensitive area at which the first window
`is displayed, and scrolls the second window when an object
`touches a corresponding location in the touch sensitive area at
`which the second window is displayed. The user interface
`displays a plurality of keys which are selected when an object
`touches a corresponding key location and are activated when
`the object applies additional pressure to the corresponding
`key location. Alternatively, the key is activated when an
`object touches its corresponding location. The user interface
`presents keys in a manner facilitating entry of Chinese char(cid:173)
`acters using stroke and Pinyin input.
`
`200
`
`200 ..
`
`200 ..
`
`._-_lrWII
`.:lit.'/!' ,1
`
`200
`
`. ...... 200
`
`PINYIN
`
`200
`
`Apple and Samsung Ex. 1113
`Apple Inc., Samsung Electronics Co., Ltd., and
`Samsung Electronics America, Inc. v. Firstface Co., Ltd.
`IPR2019-00612
`Page 00001
`
`

`

`Patent Application Publication
`
`Jan. 21, 2010 Sheet 1 of 12
`
`US 2010/0017872 Al
`
`-~-101 / 121
`
`...,
`, ... , .. F>z£0~'.t:
`(1:-)~.Z.l .,..( s_f."
`
`"
`""
`., v.
`1/"
`.,._"4 """.._ ".~~;"·~ _;\."~ ;J-..._,_
`-~_"-, ~ .... ,~:"
`..
`
`•
`
`¥ faib·+t-¥£4
`-t ,;j
`
`i21
`
`>t{:*;r$:m}4 "Ft.:z}
`::...~:-:~: :> =.? t)(.~).:,~
`100-- z
`"
`<..._"4 '< '-A 1/ ~.o;••~:,z .. " & ?"• ,'
`.. _'--(._~ ....
`..
`
`"
`
`._""'
`
`v',
`
`"
`
`/E(
`
`103
`
`100
`
`v 4 '>,
`
`121
`l
`,····,···,,
`,t~>%>~S~<~
`1 u- 'GU...,x&)
`/.< !,~ >;"' l} ~( )
`
`~----.-
`t::.:
`
`102
`
`121
`
`"'
`
`-:i,,·p:l,•~•@
`
`_1~:x :.? t)(.~t:.
`""-i '":<.._,,;•·;A"& J.._,_~'
`
`-'•
`
`"."'(~'.v,'"
`
`104
`
`121
`l .... , ... , ... T .. ,
`,j'l;pjfy;;,_&•i'l',•~"'
`- Y~J.1. .. x,:J.L)
`1 08
`4 ;,; _.-:( !r~ ;.; ,.. G ? )
`/\,":;;;;·/., <. V'-'
`
`..
`
`'IJII"
`
`111
`... ~~:::*~~---4:7:t_f;
`(1:-:) ~.Z. l .. /,..3.! "
`~~4 '"":: .. ~); ... ;:_,,(":& J:.._,_'
`...
`-~-~'-. ~ .... ,~:"
`
`112
`f,i:'\i,;\~1,' ·i:t:iz
`1..:.:-.."x"'21 ,., ""- I
`\..' ••• ~ .,-. ••• • •• ✓'\...~· .....
`
`('--..-i "",\",;•·~)-."fr')_"_".,,
`
`.. "~ '.
`""'!<~""
`""
`
`"
`
`-'•
`
`--100
`
`,;-···---
`
`'
`
`l
`
`I
`
`:
`
`<
`
`~fil::~?::l••~:i:~
`,1:.:::::21~ JL
`
`> 4-; '
`/'.
`
`1};":."7
`
`61 "
`> .. •
`..
`
`"'
`,~>;!:'.>:[• !-t''il'
`1 H"' Y~tL .. >"3 J '\
`4 ~ ;z !r~ , ,.. ~ f )
`
`100
`
`;".":-:'::;'~/.,'/""
`
`".
`
`'
`
`y
`/\,":!::':•/.,, ; V'-'
`
`•
`
`STROKE
`
`c,p-
`
`113
`~~t.l:;<W~~~:.:~
`1•u'\ lt__. "3j ",
`
`?"'4-; )"- :}~v' .. ~l)
`
`_,,"::::;-/. > V'-'
`
`..
`
`):._":::;;<~/.,'1/""
`
`".
`
`100--
`
`114
`aHi0 ,},•li>''1il'
`..,
`':.'~:,"'::~.t::X:!U.:.:
`4 ~, . i ~ 5:--:_)~ fU ,,
`
`~ •• , ' " , . .~ . ) ' ,
`
`V
`
`FIG. 1
`
`IPR2019-00612 Page 00002
`
`

`

`Patent Application Publication
`
`Jan. 21, 2010 Sheet 2 of 12
`
`US 2010/0017872 Al
`
`200 ..
`
`200 ..
`
`PINYIN
`
`?.no
`
`?.00
`
`FIG. 2
`
`IPR2019-00612 Page 00003
`
`

`

`Patent Application Publication
`
`Jan. 21, 2010 Sheet 3 of 12
`
`US 2010/0017872 Al
`
`Fig. 3
`
`,~From: Karl Asp, Please call me
`From: Summer, CALL ME!! . . . . .
`
`•
`
`:
`
`•~RSS
`
`llEWS
`
`SPORT
`
`MORE
`
`Kid shot down scool class in
`~
`-
`-
`---
`-- -
`- --- -
`--
`I
`>
`\
`l \ ! I
`' I l l
`(cid:141)
`•
`dee 7
`12:00 Lunch Mika
`14:00 Dev
`.:.~_"'
`You have
`,
`.all · -, \ _
`new Gmo1ls
`- - - - -
`
`Mikael Hagma
`Steve Jobs
`Rod Johas~n
`
`A gadget is an application running on
`the phone. In their icon like state
`Gadgets are also running showing
`information like latest incoming
`message, current song played,
`currency, wheather etc. A gadget can
`be any type of application.
`
`-Streaming Radio
`-Note book
`-Web mail
`-Phone application
`-Clock
`etc.
`
`A Gadget can be updated with
`information both from the phone
`(new SMS) or from a third party
`over the internet (RSS Feed)
`
`A Gadget in icon state at the
`home screen can only take a
`defined space
`
`In most cases a gadget can beexpandedeither displayed like an overlay
`directly on the home screen like the volume control or displayedas a full
`screen application like the camera .
`
`IPR2019-00612 Page 00004
`
`

`

`Patent Application Publication
`
`Jan. 21, 2010 Sheet 4 of 12
`
`US 2010/0017872 Al
`
`Problem: It is hard to hit the right item on a touch display because when you move the finger to and from the
`surface it is also easy move it in X-Y direction by mistake especialy when used with one hand.
`
`Solution: Use a conventional touchtechnology to find the object in X-Y position. Add tactile or graphical ilustration
`were your finger is and use a force sensor to the display to read the activation.
`
`Fig. 4
`
`Fig. 5
`
`Fig. 6
`
`Find object by sweeping the screen.
`E.g. Links in a web site
`
`+
`
`Find object by sweeping the screen.
`E.g. Links in a web site
`
`Fig. 7
`
`Fig. 8
`
`IPR2019-00612 Page 00005
`
`

`

`Patent Application Publication
`
`Jan. 21, 2010 Sheet 5 of 12
`
`US 2010/0017872 Al
`
`Power ON
`Fig. 10
`Fig. 9
`
`Power OFF
`Fig. 11
`
`-0701lJ,.'d>
`
`12·070IICrMool -
`
`,.......o,oonm
`12:010llOFn.lott
`
`.,,....,
`
`-Press Home key
`
`• 12't.
`
`Qlnof1;
`
`1/ount
`
`(]I It NEON
`~3 _\'ideo ii
`® .
`
`Key lock
`Fig. 12
`Fig. 13
`
`(]I It NEON
`
`c .........
`
`Vo1ur.>to
`
`+12%
`
`Press oft Gadget or
`
`hold Home key for 5 second)( e y IO Ck
`high security
`Fig. 15
`
`Fig. 14
`
`i-Ol'OIIJ<>lil>
`
`12,070:IIOF,--.,_
`~ii~/
`
`('i
`
`NEON
`
`:at
`CB
`ean-lVw-n,,
`.,,...,
`~31Vidoo "iil
`@
`
`$MS
`
`Tap key lock to lock
`
`i--o7'<.IQ-
`
`12;07ot0F~ -
`
`c-n, \l$N
`
`(]I It
`~3 Vtdeo ,Q,
`
`NEON
`•12%
`
`SMS
`
`-®
`
`-
`
`Tap home button
`to start
`
`Type unlock
`code
`
`...
`
`lllf!ID ------ma• -Tap home button
`
`to start. Tap code
`one. two, three or four digits.
`
`Home/ return from application
`Fig. 16
`Fig. 17
`
`'P:'.:::::::-
`
`~
`
`~:£,~i
`
`NEON
`
`(]I It
`'-• ~ ., ....
`5:3 V~oo ii
`@
`
`Tap home
`
`IPR2019-00612 Page 00006
`
`

`

`Patent Application Publication
`
`Jan. 21, 2010 Sheet 6 of 12
`
`US 2010/0017872 Al
`
`Scroll alternatives
`Fig. 18
`
`r
`
`~0100.Jr,t,b
`12·01 O!lQ~,ut.Q&I
`
`dZlllJiiiiJ -.. oo ..... ~ ::£. t
`
`T~ ~~
`.,.,.,( ......
`it
`c.,,,.,, ""'-
`3 Video
`
`NEON
`•12~.
`
`@ -
`
`~
`
`Alt4
`Scroll directly on screen.
`Then we miss possibility to
`touch and drag on other
`gadgets
`
`Fig. 19
`
`Fig. 20
`
`Fig. 21
`
`e (cid:173)
`,.(11..,-.l.lG
`
`e :£0 i
`
`T~ ~ , - -
`,.,_,.-
`- - - . - - - - - -
`NEON
`-+12%
`
`c.,nm Vcune
`
`~ If:
`~ 3 Video
`
`SMS
`
`!...a.iOT'OCJCUI
`
`12:07090F~ --
`
`Camera
`
`VolUOne
`
`it
`Ill
`, .. ,
`~3 Video
`@
`
`Bl
`
`NEON
`+12%
`
`~
`
`~"";:~ .}
`-~·· ~:; \.'
`~O7-00JaOO VI
`
`1Ullltejsu.AIIO
`
`@
`
`'\ . •:;:•
`12:07 IJGl1Hn.&Qd -1 3 -
`,_
`-· ~~~-
`--Ill ~ NEON
`~3 Vodeo -®
`
`camera
`
`•12"
`
`"'"
`
`,oQ
`SMS
`
`Alt 1
`Scrollbar put finger
`on the side and scroll
`down to get lower down
`on the page
`
`Alt2
`Drag button
`Put finger on button and
`drag screen up and down
`
`Alt3
`Two drag buttons. Same
`function as Alt 2.
`Put finger on the lower scroll
`button and drag screen up
`or on the upper icon and
`drag down.
`
`IPR2019-00612 Page 00007
`
`

`

`Patent Application Publication
`
`Jan. 21, 2010 Sheet 7 of 12
`
`US 2010/0017872 Al
`
`. 22
`Fig
`
`Notes:
`OiMer,Milk.
`Vlsir Art muse.um
`
`•
`
`Home screen Based on scroll Alt 1
`
`I'
`
`SR-j ~!wG~,il T~affic
`
`34·
`
`Jam
`
`"I
`
`Optional feature: Put your finger on the scroll bar
`~~
`andle and drag up to display upper Gadgets that are
`h
`ess frequently used.The screen is automatically
`I
`crolled back 3 seconds after the finger is removed
`s
`
`V
`
`lii'ill Message -
`,-
`
`~ 07:00Jobb
`09:0 Frukosl
`
`12:07 ~ -
`
`14.00 Hej skaJMG
`
`I'\ @ .:;:~
`"""'
`
`e,r,~
`MediaPlayer
`
`Li.-.:.f:l•fn ..... "
`
`@
`
`Telephone
`O.~ ~l@B
`li,Lisa
`!.\,1;~ ~,1 Dan,el
`
`Camera
`
`Volume
`
`NEON
`+12%
`
`.,,
`
`H,
`
`,)
`
`Put your finger on the screen
`and scroll bar handle and drag down to display display
`more Gadgets down the page.
`Use Home button to get back to the home area.
`
`More Gadgets
`
`Read about new Gadgets
`Updated by
`
`Home
`area
`
`I
`
`'""
`
`rn Jt
`Sign 3 Video
`ffl1 WEB
`
`200
`SMS
`
`- - - - -·~
`
`link
`
`... ..,;IJ; You(B U4Jtcom
`link
`Fickr
`
`'J1!E8
`!:nt,
`
`'
`'WEB
`I link
`(cid:127):
`
`Face
`bOOK
`
`. ... ,;.,. ....
`~ Google search
`*****
`~ See VldlltO dips
`***** B WebnomlromCNN
`*~
`
`finding content
`on the .veb
`
`on the web get
`most seen film on the
`
`Including an RSS feed
`
`IPR2019-00612 Page 00008
`
`

`

`Patent Application Publication
`
`Jan. 21, 2010 Sheet 8 of 12
`
`US 2010/0017872 Al
`
`Fav 5
`Fig. 23 Fig. 24
`
`History
`Fig. 25
`
`r{~o
`GoC"
`
`Fig.26
`
`~01-00.-
`12.070911foi<<>SI
`lWIDiili1
`
`~,,er
`
`TilNPhO"t
`
`~.:...:..
`
`"'-IO'Oll..,.._'IIIO @j1· i
`~·~--.
`c..,-.,.. - +12%
`[]I • NEON
`S:3 II Fav
`® '
`
`5
`
`@
`
`Tap Call to make a call
`Tap SMS to SMS that person
`More for rearranging/edit. ..
`
`Tap history
`
`Tap to call or SMS
`
`Optional: Included SMS history.
`
`IA'\
`\U)
`
`Tap fav 5
`
`Telephone
`Fig. 28
`Fig. 27
`
`[]I
`c..,...,,1
`
`Volume
`
`NEON
`•12~
`Fav
`1deo 5
`
`It
`,.,. 3 v.
`:>s
`. ®
`
`Tap telephone
`
`Active call
`Fig. 29
`.. ·;,
`
`3d,f
`6mno
`
`~tll:fl.
`
`Lyn H
`L-?
`1abc
`4ghi
`5;ll
`Sluv
`-0
`
`)pi:r,
`
`(cid:141)
`More
`
`®
`
`Fig. 30
`
`-
`
`IPR2019-00612 Page 00009
`
`

`

`Patent Application Publication
`
`Jan. 21, 2010 Sheet 9 of 12
`
`US 2010/0017872 Al
`
`Volume
`Fig. 31
`
`Fig. 32
`
`Mp3 player
`Fig. 34
`Fig. 33
`
`+------ti,,j
`
`-,M,0100-
`12:07"'°~-
`
`'"''"''"
`
`1: 31 Video
`'"' ®
`
`Press volume
`button
`
`=3 Video
`'"'
`[i-."\
`~
`
`At this point you can select to
`change another sound source.
`Control disappears after 4
`seconds
`
`NEON
`Canie<B ~ •12%
`
`a, It
`~ 3 History F5av
`
`'"' @
`
`~VrAutri+!•12%
`
`a, JI:
`! NEON
`""'3 H.' Fav
`IS ory 5
`
`;:S
`
`~
`\!,;JI
`
`Tap player
`
`Cantrall disappears after 4
`seconds
`
`Operator add
`_ Fig. 35
`
`1
`
`"'-11m,....-.
`
`Fig. 37
`3 . - - - -
`
`11h11. · ·
`
`t
`
`~3lvideo Fav
`5
`I
`
`Sr.AS
`
`®
`
`2. Fig. 36
`
`Market 100 Free SMS
`User group: A34
`
`1 Go 1
`
`Add:
`
`1m11Mw~1
`
`Operators screen
`
`4. Tap Add -> operator web page
`or java app.
`
`Alt 1 End user can move it around. A click trough creates kickback.
`Alt 2 Fixed position. Reduced monthly fee.
`
`IPR2019-00612 Page 00010
`
`

`

`Patent Application Publication
`
`Jan. 21, 2010 Sheet 10 of 12
`
`US 2010/0017872 Al
`
`Video
`Fig. 38
`
`.... ,.',...
`v~;~
`
`iii,,,illM01«JJabO
`
`121-
`,.1111 ... -....a
`
`-·
`
`T~ ~
`?;,:"•
`..... ·-~
`
`12,0]1111-0ltw.OII - ~
`~
`® "I• -::=·
`-
`...... ,_
`~!.. NEON
`~31 Video
`®
`
`•12%
`
`SMS
`
`Fig. 39
`
`oc
`OCC:
`OD
`
`Tap the screen and use controls
`Controls disappears after 4
`seconds
`
`© /.
`
`©
`
`Fig. 42
`
`08-543 464 23
`Lennart
`__,
`Lisab
`(A
`~
`Lotta
`I
`Ll{'llnH
`2abc
`3def
`1.,-?
`5jkl
`5mno
`4ghi
`7pqrs
`Stuv
`9wxyz
`(cid:141)
`-0
`
`@ /.
`
`Send Picture to
`
`Tap video
`
`Camera
`Fig. 40
`
`,_,... --·-
`
`~'\A•:::•
`"~t' -·,., -
`
`®
`
`Tap Camera
`
`Fig. 41
`
`Use controls
`
`IPR2019-00612 Page 00011
`
`

`

`Patent Application Publication
`
`Jan. 21, 2010 Sheet 11 of 12
`
`US 2010/0017872 Al
`
`Messaging SMS
`Fig. 43
`Fig. 44
`
`Fig. 45
`
`DD
`~')()0
`DD
`
`© /.
`
`Ill It:
`
`Carne,ra VOIJIM
`
`NEON
`+12%
`
`SMS
`
`f:' 3 Video
`@
`
`@ /4
`
`Tap Message gadget
`
`Select Message
`
`Could also be same structure as call history
`See previous description.
`
`IPR2019-00612 Page 00012
`
`

`

`Patent Application Publication
`
`Jan. 21, 2010 Sheet 12 of 12
`
`US 2010/0017872 Al
`
`Pan home screen
`
`Fig. 47
`
`Home screen
`
`Fig. 46
`
`Neonode 2008-04-20
`Magnus Goertz
`
`Problem: Only one browser session can be opend at a time.
`How can we tell the user that in an intuitive way and still stay
`in the same screen and keep important information at the top?
`
`Solution: Always open the browser to the bottom of the screen
`and keep content above the browser Gadget.
`
`Pan up and down.
`Put you finger on the screen
`and drag up or down
`
`Open gadget I browser
`
`Open gadget / browser
`
`Fig. 49
`
`Open gadget I browser
`
`Fig. 50
`
`Push/Tap and release a
`Gadget to open/activate
`
`When Web-Gadget is activated -all
`gadgets beneath Web-Gadget slides
`down and disappears. Instead the
`browser takes this space
`
`Paning Gadget
`
`Fig. 51
`
`Paning Gadget
`
`Fig. 52
`
`Paning Web
`
`Fig. 53
`
`Paning Web
`
`Fig. 54
`
`....... -torchretayln -"'°""""""
`
`""'=(cid:173)
`""'""'
`
`I
`
`Paning abow bowser window will result
`in changing size of browser window.
`
`Paning inside bowser window will result
`in paning browser content
`
`IPR2019-00612 Page 00013
`
`

`

`US 2010/0017872 Al
`
`Jan. 21, 2010
`
`1
`
`USER INTERFACE FOR MOBILE
`COMPUTER UNIT
`
`CROSS REFERENCES TO RELATED
`APPLICATIONS
`
`[0001] This application is a continuation-in-part of pending
`application U.S. Ser. No. 10/315,250, filed on Dec. 10, 2002,
`entitled USER INTERFACE FOR MOBILE HANDHELD
`COMPUTER UNIT. This application claims priority from
`provisional application No. 61/132,469, filed on Jun. 19,
`2008, entitled IMPROVED KEYPAD FOR CHINESE
`CHARACTERS.
`
`FIELD OF THE INVENTION
`
`[0002] The field of the present invention is user interfaces
`for electronic devices and, more particularly, to touch screen
`user interfaces.
`
`BACKGROUND OF THE INVENTION
`
`[0003] Touch screens provide user interfaces through
`which a user enters input to a computing device by touching
`a screen at a selected location, with a stylus or with his finger.
`[0004] Conventional touch screens are limited as to the
`types of user inputs that they can recognize. For example,
`conventional touch screens are unable to distinguish between
`a soft tap and a hard press. In some prior art embodiments
`users initially select an item on the screen, and then subse(cid:173)
`quently activate the selected item. However, because prior art
`touch screens do not distinguish between degrees of pressure,
`the user is required to remove his finger or stylus from the
`screen and activate his selection with a second tap. It would be
`advantageous to produce touch screens that distinguish
`between varying amounts of pressure applied to the screen so
`that a user can select an item by touching its location on the
`screen, and then activate the item by applying additional
`pressure to the touch location without having to first remove
`his finger or stylus from the screen.
`In conventional touch screens the keys are often
`[0005]
`small relative to the touch area, especially in handheld
`devices. The keys are also often situated close together. This
`can make it difficult to determine which key is being pressed
`by the user. It would be advantageous to clearly indicate to the
`user which key the user has selected, and furthermore, to
`allow the user to accept or reject the selected key without first
`removing his finger or stylus from the screen in order to
`perform a second tap.
`[0006] Current user interfaces are basic and often require
`navigation through a series of menus in order to perform a
`desired operation. The present invention presents a user inter(cid:173)
`fere that is versatile in providing the user with many options,
`while requiring only few selections to activate a desired func(cid:173)
`tion. To further enhance user experience, certain functions are
`performed automatically without requiring the user to enter a
`selection.
`
`SUMMARY OF THE DESCRIPTION
`
`[0007] Aspects of the present invention relate to user inter(cid:173)
`faces designed for use with a touch screen. The present inven(cid:173)
`tion relates to computer readable media storing computer
`programs with computer program code, which, when read by
`a computer unit, allows the computer to present a user inter(cid:173)
`face for the computer unit.
`
`In accordance with embodiments of the present
`[0008]
`invention, the computer unit features a touch sensitive display
`area. According to preferred embodiments an initial display
`configuration presents a plurality of gadgets on the display.
`These gadgets are small areas of the screen that indicate
`which functions each gadget will perform when activated by
`the user. When a gadget is activated, typically by the user
`touching the area of the display on which the gadget is dis(cid:173)
`played, the gadget increases in size and provides the user with
`icons and information about the gadget's functions. Signifi(cid:173)
`cantly, the gadget does not cover the entire display area. Thus,
`when a gadget, or even a plurality of gadgets, is active, the
`primary display of all available gadgets is still accessible.
`This primary display can be compared to a desktop in com(cid:173)
`puter operating system user interfaces. However, this primary
`display in the user interface of the present invention is not the
`same as a desktop where active windows can cover icons on
`the desktop. In the present invention, gadgets are arranged in
`a manner that open gadgets do not cover other gadgets.
`Rather, when an open gadget expands in size, other gadgets
`are shifted to make room for the expended gadget. This allows
`the user to scroll the primary display or desktop to view any
`gadget. In the context of the present invention, this primary
`display area that includes both open and closed gadgets is
`called the home window. The user scrolls the home window to
`view gadgets that are shifted beyond the viewable area of the
`display. An expanded, or activated, gadget has an expanded
`window, but often it is not large enough to display everything
`contained in that gadget window. To view contents of the
`gadget not displayed in the visible portion of the gadget
`window, the user scrolls the gadget window. Thus, two dif(cid:173)
`ferent scroll operations are provided: scrolling the home win(cid:173)
`dow and scrolling a gadget window. According to one
`embodiment, scrolling is executed by gliding a finger or sty(cid:173)
`lus along the touch screen to shift the active display area of the
`home window or of the gadget. The scrolling affects the home
`window if the finger glide began in an area of the screen that
`does not belong to an active gadget; the scroll affects an active
`gadget window if the finger glide began inside that active
`gadget window.
`[0009] Various embodiments of the invention support sev(cid:173)
`eral methods of scrolling a window. According to one
`embodiment, scrolling is done when the user touches the
`display inside the window area, for example at an edge of the
`window, or on an icon, such is a arrow or scrollbar, indicating
`a scroll operation. According to another embodiment, scroll(cid:173)
`ing is done by the user touching the window with a finger or
`stylus and then gliding the finger or stylus along the torch
`sensitive screen in a direction indicating the desired direction
`of the scroll. When the content of the home display is larger in
`two dimensions than the actual display screen, this operation
`is like panning an image or map. When it is larger along only
`one axis ( e.g., only vertically), the scrolling only scrolls in the
`one axis even when the glide is not orthogonal along the one
`axis.
`[0010] Another aspect of the present invention relates to
`computer readable media storing a computer program with
`computer program code, which, when read by a mobile hand(cid:173)
`held computer unit, allows the computer to present a user
`interface for the mobile handheld computer unit. The user
`interface features a touch sensitive area in which representa(cid:173)
`tions of a plurality of keys are displayed, and each key is
`mapped to a corresponding location in the touch sensitive
`area at which the representation of the key is displayed. A key
`
`IPR2019-00612 Page 00014
`
`

`

`US 2010/0017872 Al
`
`Jan. 21, 2010
`
`2
`
`in this context includes, inter alia, alphabetic keys such as in
`a QWERTY keypad, numeric keys and also icons represent(cid:173)
`ing programs or functions. Akey is selected, but not activated,
`when an object touches the corresponding location. This
`intermediate status of being selected but not activated facili(cid:173)
`tates the user to subsequently activate a desired key and avoid
`activating a neighboring key that the user selected but does
`not wish to activate. A selected key is activated when the
`object touching it applies additional pressure to the key loca(cid:173)
`tion.
`[0011] According to preferred embodiments of the inven(cid:173)
`tion, when a key is selected, the user interface generates a
`secondary representation of the key, such as a callout balloon
`containing the key representation. The callout balloon is
`placed away from the key location (being touched) so that the
`user can easily view which key is selected without lifting his
`finger. According to another embodiment, an audio represen(cid:173)
`tation of the selected key is generated so the user hears which
`key was selected.
`[0012] According to still further features in preferred
`embodiments of the invention, the user touches the screen
`(with a finger or stylus) at a first location, for example select(cid:173)
`ing a first key. The user then glides his finger or sty !us over the
`screen to additional locations. At each additional location a
`new key is selected and the previously selected key is dese(cid:173)
`lected. The user can activate any selected key by applying
`additional pressure to the screen. The user does not have to
`remove the object from the screen to glide and select addi(cid:173)
`tional keys even after activating a first key.
`[0013] Additional touch pressure is detected in various
`ways according to several embodiments. According to one
`embodiment, the touch sensitive area is a light-based touch
`screen operable to detect different levels of touch pressure.
`For example, light-based touch screens typically include a
`calculating unit operable to identify the size, shape and con(cid:173)
`tours of an area being touched based or a pattern of obstructed
`light. See applicant's co-pending U.S. patent application Ser.
`No. 10/494,055, titled ON A SUBSTRATE FORMED OR
`RESTING DISPLAY ARRANGEMENT, the contents of
`which are incorporated herein by reference. When a finger or
`flexible object is used as a touch object, as additional pressure
`is applied to the touch surface, the contact area of the finger or
`object touching the screen increases. Thus, additional pres(cid:173)
`sure is detected when an increase in the contours of the
`covered touch area is detected.
`[0014] Alternatively, or in combination with the above, the
`touch sensitive area features both a touch screen operable to
`identify a touch location on the screen and a pressure sensor
`operable to detect pressure applied to the screen but not
`sensitive to the location of the object applying the pressure.
`[0015] Other aspects of the present invention relate to con(cid:173)
`venient arrangement and function oficons to perform popular
`functions within a user interface. Thus, a camera gadget fea(cid:173)
`tures a multimedia messaging service (MMS) button facili(cid:173)
`tating sending an active photo in an MMS message; a key lock
`gadget locks the computer and displays an instrumental key(cid:173)
`pad for entering a musical code to unlock the computer; a
`reporting gadget displays information for a first period of
`time and is then automatically deactivated. Several reporting
`gadgets are provided, including a gadget that displays the
`time of day; a gadget displays a weather forecast; a gadget
`that displays stock market information.
`[0016] According to still further features in preferred
`embodiments the reporting gadget continues to display its
`
`information for a second period of time if the gadget is
`touched during the first period of time. I.e., automatic deac(cid:173)
`tivation after the first period of time is canceled.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`[0017] The present invention will be more fully understood
`and appreciated from the following detailed description,
`taken in conjunction with the drawings in which:
`[0018] FIG. 1 is a series of screens illustrating Chinese
`stroke input, in accordance with an embodiment of the
`present invention;
`[0019] FIG. 2 is a series of screens illustrating Chinese
`Pinyin input, in accordance with an embodiment of the
`present invention;
`[0020] FIGS. 3 is a screen of a handheld terminal display(cid:173)
`ing a plurality of gadgets, in accordance with an embodiment
`of the present invention;
`[0021] FIGS. 4-6 are diagrams illustrating selection of ele(cid:173)
`ments displayed on a touch screen. A selected element is
`displayed in a callout as shown in FIGS. 5-6. The user does
`not lift his finger off the screen between selection of different
`elements, as in FIGS. 5 and 6. The use simply glides his finger
`along the screen. This is shown in FIG. 7. To activate a
`selected element the user presses on the screen without first
`lifting his finger off the screen as shown in FIG. 8 and in the
`series of FIGS. 7-8, all in accordance with an embodiment of
`the present invention;
`[0022] FIGS. 7-8 illustrate a side view of a user selecting
`and activating an on-screen element, in accordance with an
`embodiment of the present invention;
`[0023] FIGS. 9-11 illustrate turning a phone on and off.
`FIG. 9 illustrates pressing the home key on a turned-off
`phone. FIG. 10 illustrates the phone when it is turned on. FIG.
`11 illustrates pressing the home key to turn a phone off.
`[0024] FIGS. 12-15 illustrate locking and unlocking a
`phone. FIG. 12 illustrates activating a lock gadget onscreen to
`lock the phone. FIG. 13 illustrates pressing the home button
`on a locked phone to unlock the phone. FIG. 14 illustrates the
`unlocked phone. FIG. 15 shows pressing the home key on a
`locked phone to display a keypad for entering a security code
`to unlock the phone.
`[0025] FIGS. 16-17 illustrate pressing the home key to
`return to the home display screen. FIG. 16 shows an open
`active calling gadget. Pressing the home button presents the
`home screen shown in FIG. 17.
`[0026] FIGS. 18-21 illustrate scrolling options. In the
`phone shown in FIG. 18 scrolling is done by gliding a finger
`or stylus along the touch sensitive screen. FIG. 19 illustrates
`a scrollbar along one edge of the screen for scrolling up and
`down. FIGS. 20-21 illustrate arrow buttons for scrolling up
`and down.
`[0027] FIG. 22 illustrates scrolling the home display
`screen, in accordance with an embodiment of the present
`invention;
`[0028] FIGS. 23-24 illustrate a Fav5 gadget, in accordance
`with an embodiment of the present invention;
`[0029] FIGS. 25-26 illustrate a History gadget, in accor(cid:173)
`dance with an embodiment of the present invention;
`[0030] FIGS. 27-28 illustrate a Calling gadget, in accor(cid:173)
`dance with an embodiment of the present invention;
`[0031] FIGS. 29-30 illustrate placing a call using a Calling
`gadget, in accordance with an embodiment of the present
`
`IPR2019-00612 Page 00015
`
`

`

`US 2010/0017872 Al
`
`Jan. 21, 2010
`
`3
`
`invention; FIG. 30 illustrates the shortcut buttons presented
`when an incoming call is received (i.e., when the phone is
`ringing).
`[0032] FIGS. 31-32 illustrate volume control, in accor(cid:173)
`dance with an embodiment of the present invention;
`[0033] FIGS. 33-34 illustrate an MP3 player gadget, in
`accordance with an embodiment of the present invention;
`[0034] FIGS. 35-37 illustrate an operator promotion gad(cid:173)
`get, in accordance with an embodiment of the present inven(cid:173)
`tion;
`[0035] FIGS. 38-39 illustrate a video player gadget, in
`accordance with an embodiment of the present invention;
`[0036] FIGS. 40-42 illustrate a camera gadget and shortcut
`buttons provided when the camera gadget is activated, in
`accordance with an embodiment of the present invention;
`[0037] FIGS. 43-45 illustrate an SMS gadget and shortcut
`buttons provided when the SMS gadget is activated, in accor(cid:173)
`dance with an embodiment of the present invention; FIG. 45
`illustrates how related SMS messages are displayed as a
`conversation in an active SMS gadget.
`[0038] FIGS. 46-54 illustrate scrolling the home display
`screen and gadget windows, in accordance with an embodi(cid:173)
`ment of the present invention;
`
`DETAILED DESCRIPTION
`
`[0039] Touch screen keypads offer great flexibility in key(cid:173)
`pad interface design-flexibility that cannot be achieved with
`electromechanical keypads. Custom design keypads can be
`generated on a touch screen, where the markings on each key
`and the functions that each key provides are optimized for a
`designated application. Moreover, touch screen keypads can
`change modes, from one pad of keys and associated functions
`to a different pad of keys and associated functions. Custom
`keypads are of particular advantage for multi-lingual appli(cid:173)
`cations.
`[0040] A general description of touch screen keypad inter(cid:173)
`faces, in accordance with embodiments of the present inven(cid:173)
`tion, and several examples thereof, are provided in the Appen(cid:173)
`dix and described in detail herein below.
`[0041] Embodiments of the present invention relate to
`improved keypads for inputting Chinese characters using
`XT9 stroke input, and using Chinese Pinyin. XT9 stroke input
`builds Chinese characters using six basic strokes, and offers
`selection of possible characters and phrases based on a set of
`strokes that have been input. Chinese Pinyin uses Latin char(cid:173)
`acters that transliterate a sound or a syllable, in combination
`with a digit that represents an intonation or inflection. E.g.,
`Ma in a rising tone is m-a-1, and Ma in a descending tone is
`m-a-2.
`[0042] Reference is now made to FIG. 1, which is an illus(cid:173)
`tration of an improved stroke-based keypad for entering Chi(cid:173)
`nese characters by inputting strokes, in accordance with an
`embodiment of the present invention. Shown in FIG. 1 is a
`keypad 100, used for inputting basic strokes. Also shown in
`FIG. 1 is a sequence of strokes 101-106 that are successively
`input by a user, and a corresponding sequence of Chinese
`characters 111-114 that are entered in a message being com(cid:173)
`posed.
`[0043]
`In accordance with an embodiment of the present
`invention, keypad 100 is generated and displayed on a touch
`screen. Keypad 100 has fewer than the standard 12 keys in a
`touch pad, allowing more room on screen for displaying
`characters.
`
`[0044] Further in accordance with an embodiment of the
`present invention, the keys of keypad 100 are customized so
`that they contain only relevant information. For example, a
`prior art keypad displays a digit, 3-4 characters, and a basic
`Chinese stroke, all inside one key, even though in XT9 stroke
`mode the basic Chinese stroke is the only useful one. The
`custom keys of the present invention display only the basic
`Chinese strokes, or the strokes and numbers, but no charac(cid:173)
`ters.
`[0045] There are two types of key presses supported by
`keypad 100-regular and long. A regular key press adds the
`stroke shown on the key to the series of strokes 101-106
`already pressed. As strokes are successively entered, a num(cid:173)
`bered array 121 of Chinese characters or phrases is dynami(cid:173)
`cally displayed along the top of the keypad. These characters
`or phrases are predicted based on the key presses already
`entered. In order to select one of the numbered elements of
`array 121, the user performs a long key press on that number.
`Alternatively, the user may keep entering strokes until only
`one option remains.
`[0046] Often, however, more predicted characters or
`phrases exist than can be displayed along the top of the
`keypad. The N2 phone, manufactured by Neonode of Stock(cid:173)
`holm, Sweden, has a joystick button at the bottom of the
`phone. Twitching the joystick up/down displays different sets
`of predicted characters or phrases. When the intended ch