as United States
`a2) Patent Application Publication (0) Pub. No.: US 2006/0026535 Al
`
` Hotelling et al. (43) Pub. Date: Feb. 2, 2006
`
`
`ZOQIL
`US 20060026535A1
`
`(54) MODE-BASED GRAPHICAL USER
`INTERFACES FOR TOUCH SENSITIVE
`INPUT DEVICES
`
`(22)
`
`Filed:
`
`Jan. 18, 2005
`
`Related U.S. Application Data
`
`(75)
`
`Inventors: Steve Hotelling, San Jose, CA (US);
`Brian Q. Huppi, San Francisco, CA
`(US); Joshua A.Strickon, San Jose,
`CA (US); Duncan Robert Kerr, San
`Francisco, CA (US); Bas Ording, San
`Francisco, CA (US); Imran Chaudhri,
`San Francisco, CA (US); Greg
`Christie, San Jose, CA (US); Jonathan
`P. Ive, San Francisco, CA (US)
`
`Correspondence Address:
`BEYER WEAVER & THOMAS LLP
`P.O. BOX 70250
`OAKLAND,CA 94612-0250 (US)
`
`(73) Assignee: Apple Computer Inc.
`
`(63) Continuation-in-part of application No. 10/903,964,
`filed on Jul. 30, 2004.
`
`Publication Classification
`
`(51)
`
`Int. Cl.
`(2006.01)
`GO6F 3/00
`(52) US. Cle cecccssscesscsssssssssseenssssensserseseeeeseseessee 715/863
`
`(57)
`
`ABSTRACT
`
`A user interface methodis disclosed. The method includes
`detecting a touch and then determining a user interface mode
`when a touch is detected. The method further includes
`activating one or more GUI elements based on the user
`
`(21) Appl. No.:
`
`11/038,590
`
`interface mode and in response to the detected touch.
`
`HP Inc. - Exhibit 1020 - Page 1
`
`HP Inc. - Exhibit 1020 - Page 1
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 1 of 52
`
`US 2006/0026535 Al
`
`OUTPUT
`
`©©
`
`VODEVICE
`
`GUI
`
`GESTURE
`
`HP Inc. - Exhibit 1020 - Page 2
`
`HP Inc. - Exhibit 1020 - Page 2
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 2 of 52
`
`US 2006/0026535 Al
`
`100 _
`
`
`READ IMAGE FROM TOUCHSCREEN
`
`
`
`CONVERT IMAGE TO COLLECTION OR
`LIST OF FEATURES
`
`
`
`CLASSIFY AND GROUP FEATURES
`
`CALCULATE KEY PARAMETERS OF
`
`FEATURE GROUPS
`
`ASSOCIATE GROUP(S) TO ULI.
`ELEMENT(S)
`
`112
`
`FEEDBACK
`
`PERFORM
`ACTION
`
`PROVIDE USER
`
`102
`
`104
`
`106
`
`108
`
`110
`
`114
`
`FIG. 2
`
`HP Inc. - Exhibit 1020 - Page 3
`
`HP Inc. - Exhibit 1020 - Page 3
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 3 of 52
`
`US 2006/0026535 Al
`
`120
`
`TOUCH1
`
`(ID,)
`
`A,
`
`TOUCH2
`(ID,)
`
`6,
`
`y
`
`1
`
`122A
`
`A
`
`2
`
`X
`
`*
`
`Y,
`
`122B
`
`FIG. 3A
`
`X,
`
`
`
`FIG. 3B
`
`HP Inc. - Exhibit 1020 - Page 4
`
`HP Inc. - Exhibit 1020 - Page 4
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 4 of 52
`
`US 2006/0026535 Al
`
`130
`
`ID,
`
`_t
`wv Dy
`
`Dave =
`
`D,*D,
`
`es
`D7 Cc
`uc
`
`ID,
`
`130
`
`FIG. 4
`
`HP Inc. - Exhibit 1020 - Page 5
`
`HP Inc. - Exhibit 1020 - Page 5
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 5 of 52
`
`US 2006/0026535 Al
`
`150
`
`RECEIVE GROUP OF FEATURES
`
`
`
`IS THERE A
`CHANGE IN # OF
`FEATURES?
`
`NO
`
`152
`
`
`158
`
`CALCULATE CURRENT
`
`PARAMETER VALUES
`
`CALCULATEINITIAL
`PARAMETER VALUES
`
`
`
`
`
`REPORTINITIAL AND
`
`CURRENT PARAMETER VALUES
`
`160
`
`FIG. 5
`
`END
`
`HP Inc. - Exhibit 1020 - Page 6
`
`HP Inc. - Exhibit 1020 - Page 6
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 6 of 52
`
`US 2006/0026535 Al
`
`i
`
`i”
`
`HP Inc. - Exhibit 1020 - Page 7
`
`HP Inc. - Exhibit 1020 - Page 7
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 7 of 52
`
`US 2006/0026535 Al
`
`
`
`
`FIG.6B
`
`HP Inc. - Exhibit 1020 - Page 8
`
`HP Inc. - Exhibit 1020 - Page 8
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 8 of 52
`
`US 2006/0026535 Al
`
`
`
`
`FIG.6C
`
`HP Inc. - Exhibit 1020 - Page 9
`
`HP Inc. - Exhibit 1020 - Page 9
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 9 of 52
`
`US 2006/0026535 Al
`
`<wt
`©=
`
`188
`
`
`FIG.6D
`
`HP Inc. - Exhibit 1020 - Page 10
`
`HP Inc. - Exhibit 1020 - Page 10
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 10 of 52
`
`US 2006/0026535 Al
`
`
`
`
`190
`
`FIG.6E
`
`HP Inc. - Exhibit 1020 - Page 11
`
`HP Inc. - Exhibit 1020 - Page 11
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 11 of 52
`
`US 2006/0026535 Al
`
`CD)
`4Cy
`
`KT).
`
`&
`
`Z|
`
`:
`
`SO ~~ Ep)
`or
`DP)
`
`FIG.6F
`
`HP Inc. - Exhibit 1020 - Page 12
`
`HP Inc. - Exhibit 1020 - Page 12
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 12 of 52
`
`US 2006/0026535 Al
`
`200~
`
`202
`
`DETECT USER INPUT
`
`
`
`
`
`
`CONTROL ACTIONS
`
`CLASSIFY USER INPUT AS
`TRACKING OR SELECTION
`INPUT OR GESTURE INPUT
`
`206
`
`PERFORM TRACKING
`
`DURING USER INPUT
`
`PERFORM GESTURE
`
`208
`
`FIG. 7
`
`300 ~
`
`
`OUTPUT GUI OBJECT
`
`302
`
`
`
`
`RECEIVE GESTURE INPUT OVER
`GUI OBJECT
`
`MODIFY GUI OBJECT BASED ON
`AND IN UNISON WITH THE
`GESTURE INPUT
`
`304
`
`306
`
`FIG. 9
`
`HP Inc. - Exhibit 1020 - Page 13
`
`HP Inc. - Exhibit 1020 - Page 13
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 13 of 52
`
`US 2006/0026535 Al
`
`252
`
`CAPTUREINITIAL IMAGE
`
`DETERMINE TOUCH MODE
`BASED ON INITIAL IMAGE
`
`254
`
`256
`
` CAPTURE NEXT IMAGE
`
` \
`
`
`
`
`
`260
`
`
`SET NEXT IMAGE
`ASINITIAL IMAGE
`
` DETERMINE IF
`
`MODE CHANGED?
`
`
`
`COMPAREINITIAL AND NEXT
`IMAGES
`
`
`262
`
`FIG. 8
`
`HP Inc. - Exhibit 1020 - Page 14
`
`HP Inc. - Exhibit 1020 - Page 14
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 14 of 52
`
`US 2006/0026535 Al
`
`“S
` 352
`
`
`
`DETECT PRESENCEOFFIRST FINGER AND SECOND FINGER ON
`A TOUCH SENSITIVE SURFACE AT THE SAME TIME
`
`
`
`COMPARE DISTANCE BETWEEN
`TWO FINGERS
`
`
`
`358
`
`
`
`
`GENERATE ZOOM OUT
`356
`GENERATEZOOMIN
`SIGNAL IF DISTANCE
`
`SIGNALIF DISTANCE
`
`
`BETWEEN TWO FINGERS
`
`BETWEEN TWO FINGERS
`
`DECREASES
`
`INCREASES
`
`
`
`
`FIG. 10
`
`DETECT PRESENCE OF FIRST
`FINGER AND SECOND FINGER
`ON A TOUCH SENSITIVE
`SURFACE AT THE SAME TIME
`
`MONITOR POSITION OF TWO
`OBJECTS WHEN MOVED
`TOGETHER ACROSS TOUCH
`SENSITIVE SURFACE
`
`INITIAL POSITION
`
`GENERATE PAN SIGNAL WHEN
`THE POSITION OF THE FINGERS
`CHANGESRELATIVE TO AN
`
`FIG. 12
`
`HP Inc. - Exhibit 1020 - Page 15
`
`HP Inc. - Exhibit 1020 - Page 15
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 15 of 52
`
`US 2006/0026535 Al
`
`368
`
`364
`
`FIG. 11A
`
`364
`
`366
`
`FIG. 11B
`
`HP Inc. - Exhibit 1020 - Page 16
`
`HP Inc. - Exhibit 1020 - Page 16
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 16 of 52
`
`US 2006/0026535 Al
`
`FIG. 110
`
`366
`
`
`364
`
`372
`
`N. California
`
`FIG. 11D
`
`366
`
`HP Inc. - Exhibit 1020 - Page 17
`
`HP Inc. - Exhibit 1020 - Page 17
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 17 of 52
`
`US 2006/0026535 Al
`
`N. California
`
`FIG. 11E
`San Jose
`
`364
`
`FIG. 11F
`
`HP Inc. - Exhibit 1020 - Page 18
`
`HP Inc. - Exhibit 1020 - Page 18
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 18 of 52
`
`US 2006/0026535 Al
`
`376
`
`7
`
`364
`
`;
`San Francisco
`
`\ .
`
`:
`
`\,
`
`\,
`CK
`
`880
`
`366
`
`378
`
`.
`
`
`
`e SanJose
`PN
`ee”
`
`
`
`\ NO 7 ,
`\. SanCarlgs ~-<
`* TOR
`
`‘ n
`
`FIG. 11G
`
`378
`
`366
`
`San'Qarlos
`‘~
`
`|,
`
`\
`
`v
`
`HP Inc. - Exhibit 1020 - Page 19
`
`HP Inc. - Exhibit 1020 - Page 19
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 19 of 52
`
`US 2006/0026535 Al
`
`368 FIG. 13A
`
`368
`
`368
`
`364
`
`366
`
`FIG. 13B
`
`HP Inc. - Exhibit 1020 - Page 20
`
`HP Inc. - Exhibit 1020 - Page 20
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 20 of 52
`
`US 2006/0026535 Al
`
`368
`
`366
`: \ CS
`/
`
`364
`
`SQ
`
`366
`
`FIG. 13C FIG. 13D
`
`HP Inc. - Exhibit 1020 - Page 21
`
`HP Inc. - Exhibit 1020 - Page 21
`
`

`

`Patent Application Publication Feb. 2,2006 Sheet 21 of 52
`
`US 2006/0026535 Al
`
`450 ~
`
`WHEN THE ANGLE CHANGES
`
`DETECT PRESENCEOF FIRST
`FINGER AND SECOND FINGER
`ON A TOUCH SENSITIVE
`SURFACE AT THE SAME TIME
`
`GENERATE INITIAL ANGLE AT
`SET DOWN
`
`GENERATE ROTATE SIGNAL
`
`FIG. 14
`
`500 >
`502
`
`
`DETECT PRESENCE AN OBJECT
`
`ON TOUCH SENSITIVE SURFACE
`
`
`
`
`
`RECOGNIZE OBJECT
`
`504
`
`DISPLAY IMAGEIN VICINITY OF
`
`596
`
`OBJECT
`
`FIG. 16
`
`HP Inc. - Exhibit 1020 - Page 22
`
`HP Inc. - Exhibit 1020 - Page 22
`
`

`

`Patent Application Publication Feb. 2,2006 Sheet 22 of 52
`
`US 2006/0026535 Al
`
`368
`
`364
`
`366
`
` 366
`
`FIG. 15B
`
`368
`
`364
`
`366
`
`FIG. 15C 8
`
`HP Inc. - Exhibit 1020 - Page 23
`
`HP Inc. - Exhibit 1020 - Page 23
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 23 of 52
`
`US 2006/0026535 Al
`
`17A
`
`
`
`FIG.17C
`
`FIG.
`
`HP Inc. - Exhibit 1020 - Page 24
`
`HP Inc. - Exhibit 1020 - Page 24
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 24 of 52
`
`US 2006/0026535 Al
`
`oO
`io
`
`oO
`
`NL
`
`o
`
`LL
`NR
`vr
`
`O L
`
`L
`
`COON
`
`N 1
`
`0
`
`17D
`
`FIG.
`
`HP Inc. - Exhibit 1020 - Page 25
`
`HP Inc. - Exhibit 1020 - Page 25
`
`

`

`600™
`
`
`DISPLAY PAGE FROM A GROUP
`OF PAGES
`
`Patent Application Publication Feb. 2, 2006 Sheet 25 of 52
`
`US 2006/0026535 Al
`
`DISPLAY CONTROL BOX HAVING
`CONTROL BUTTONS
`
`552
`
`
`
`
`
`
`
`
`ENLARGE CONTROL BOX AND CONTROL BUTTONS WHEN
`THE PRESENCE OF AN OBJECT IS DETECTED OVER THE
`CONTROL BOX
`
`556
`
`554
`
`
`
`GENERATE CONTROLSIGNAL ASSOCIATED WITH A SELECTED BUTTON
`OF THE CONTROL BOX WHEN THE PRESENCE OF THE OBJECTIS
`DETECTED OVER ONE OF THE ENLARGED CONTROL BUTTONS
`
`FIG. 18
`
`602
`
` 604
`
`
`SY
`DETECT PRESENCE OF OBJECT
`OVER PREDETERMINED REGION
`OF DISPLAYED PAGE
`
`
`
`
`606
`GENERATE PAGE TURN SIGNAL WHEN THE
`
`
`DETECTED OBJECT IS SLID HORIZONTALLY ACROSS
`
`
`THE PAGE IN THE PREDETERMINED REGION OF THE
`
`PAGE
`
`
`FIG. 20
`
`HP Inc. - Exhibit 1020 - Page 26
`
`HP Inc. - Exhibit 1020 - Page 26
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 26 of 52
`
`US 2006/0026535 Al
`
`
`580
`
`FIG. 19A
`
`FIG. 19B
`
`HP Inc. - Exhibit 1020 - Page 27
`
`HP Inc. - Exhibit 1020 - Page 27
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 27 of 52
`
`US 2006/0026535 Al
`
`
`
`
`FIG. 19C
`
`516
`
`FIG. 19D
`
`HP Inc. - Exhibit 1020 - Page 28
`
`HP Inc. - Exhibit 1020 - Page 28
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 28 of 52
`
`US 2006/0026535 Al
`
`HP Inc. - Exhibit 1020 - Page 29
`
`HP Inc. - Exhibit 1020 - Page 29
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 29 of 52
`
`US 2006/0026535 Al
`
`HP Inc. - Exhibit 1020 - Page 30
`
`HP Inc. - Exhibit 1020 - Page 30
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 30 of 52
`
`US 2006/0026535 Al
`
`650 |
`
`652
`
`DISPLAY IMAGE
`
`DETECT SCROLLING OR PANNING STROKE
`
`DETERMINE SPEED AND DIRECTION OF SCROLLING OR PANNING
`STROKE
`
`654
`
`656
`
`658
`
`MOVE IMAGE IN ACCORDANCE WITH THE SPEED AND DIRECTION
`OF THE SCROLLING OR PANNING STROKE
`
`660
`
`SLOWING MOTION OF IMAGE IN ACCORDANCE WITH INERTIA
`PRINCIPALS WHEN THE PANNING OR SCROLLING STROKE IS NO
`LONGER DETECTED
`
`
`
`
`FIG. 22
`
`702
`
`DISPLAY KEYBOARD
`
`oN
`
`
`
`
`
`
`
`
`
`
`
`GENERATE SINGLE CONTROL FUNCTION WHEN THE
`FIRST OBJECT IS DETECTED OVERTHE FIRST KEY
`
`
`AND WHEN THE SECOND OBJECT IS DETECTED OVER
`
`THE SECOND KEY
`.
`
`
`DETECT PRESENCE OF A FIRST OBJECT
`OVER A FIRST KEY AND A SECOND
`OBJECT OVER A SECOND KEY AT THE
`SAME TIME
`
`704
`
`706
`
`FIG. 24
`
`HP Inc. - Exhibit 1020 - Page 31
`
`HP Inc. - Exhibit 1020 - Page 31
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 31 of 52
`
`512
`
`
`
`US 2006/0026535 Al
`
`
`681
`
`
`Faeaaa ETSeeee
`
`
`
`FIG. 23B
`
`|
`
`HP Inc. - Exhibit 1020 - Page 32
`
`HP Inc. - Exhibit 1020 - Page 32
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 32 of 52
`
`US 2006/0026535 Al
`
`FIG. 23D
`
`
`
`HP Inc. - Exhibit 1020 - Page 33
`
`HP Inc. - Exhibit 1020 - Page 33
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 33 of 52
`
`US 2006/0026535 Al
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`576A
`
`
`
`
`
`
`
` CL CO
`
`
`
`
`
`730
`
`HP Inc. - Exhibit 1020 - Page 34
`
`HP Inc. - Exhibit 1020 - Page 34
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 34 of 52
`
`US 2006/0026535 Al
`
`512
`
`Saceveuaasaccevssocceusecescrscresccnsisssrscesconesessesesssesessessssaseas
`Y
`i oa
`“=
`qQ
`
`on
`
`576A
`
`
`
`|
`ito | 7
`
`
`
`
`
`
`
`
`
`
`S\ GrTTTyT-y +
`a
`COLO
`Ly.
`\O~ OD) Vo
`\
`4
`a L)
`
`(
`
`W
`IS
`
`576B
`
`FIG. 25C
`
`512
`
`uanunconecvaserevessecsececscssterstorossseccsseecensosseseeseseesecorasaenone
`Y
`00QO._WORD PROCESSOR
`©
`
`qQq
`
`520
`
`576A
` Sad,
`
`
`
`
`
`
`
`
`HP Inc. - Exhibit 1020 - Page 35
`
`HP Inc. - Exhibit 1020 - Page 35
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 35 of 52
`
`US 2006/0026535 Al
`
`750i.
`
`PRESENT VIRTUAL SCROLL
`WHEEL
`
`752
`
`754
`
`
`
`
`
`
`
`DETECT PRESENCE OF
`FINGER(S) ON TOUCHSCREEN
`
`756
`
`
`SET INITIAL POSITION OF
`
`FINGER(S) ON VIRTUAL SCROLL
`WHEEL
`
`758
`
`
`GENERATE ROTATE SIGNAL
`
`WHENFINGER(S) MOVES ABOUT
`
`THE VIRTUAL SCROLL WHEEL
`
`FIG. 26
`
`HP Inc. - Exhibit 1020 - Page 36
`
`HP Inc. - Exhibit 1020 - Page 36
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 36 of 52
`
`US 2006/0026535 Al FIG. 27B
`
`HP Inc. - Exhibit 1020 - Page 37
`
`HP Inc. - Exhibit 1020 - Page 37
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 37 of 52
`
`US 2006/0026535 Al
`
`512
`
`760
`
` FIG. 27D
`
`HP Inc. - Exhibit 1020 - Page 38
`
`HP Inc. - Exhibit 1020 - Page 38
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 38 of 52
`
`US 2006/0026535 Al
`
`[800
`
`TOUCH
`DETECTED?
`
`
`
`804
`
`DETERMINE USER
`
`INTERFACE MODE
`
`DISPLAY GUI ELEMENT BASED ON
`USER INTERFACE MODE AND IN
`CONJUNCTION WITH TOUCH
`
`806
`
`802
`
`
`
`
`
`ENABLE GUI ELEMENT
`FUNCTIONALITY
`
`
`
`
`DISABLE GU! ELEMENT
`
`DE-
`FUNCTIONALITY AND
`
`ACTIVATE GUI
`REMOVE GUI ELEMENT
`ELEMENT?
`
`FROM DISPLAY
`
`
`
`
`
`FIG. 28
`
`HP Inc. - Exhibit 1020 - Page 39
`
`HP Inc. - Exhibit 1020 - Page 39
`
`

`

` O
`iT”
`
`fo)
`
`Qi
`©
`
`S
`
`oO
`
`a
`o
`
`“
`:
`
`Patent Application Publication Feb. 2, 2006 Sheet 39 of 52
`
`US 2006/0026535 Al
`
`a
`CN
`©)
`i
`
`oo
`
`2
`
`©
`
`HP Inc. - Exhibit 1020 - Page 40
`
`HP Inc. - Exhibit 1020 - Page 40
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 40 of 52
`
`US 2006/0026535 Al
`
`30D
`30B
`
`FIG.
`
`FIG.
`
`HP Inc. - Exhibit 1020 - Page 41
`
`QNO
`
`N0
`
`FIG.30C
`
`
`
`FIG.30A
`
`<N
`
`N©
`
`HP Inc. - Exhibit 1020 - Page 41
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 41 of 52
`
`US 2006/0026535 Al
`
`820C
`
`820D
`
`
`
`FIG.31C
`
`826D
`
`
`
`FIG.31D
`
`OoKe
`O
`
`No
`
`31B
`
`820B
`
`FIG.
`
`826B
`
`
`
`FIG.31A
`
`HP Inc. - Exhibit 1020 - Page 42
`
`HP Inc. - Exhibit 1020 - Page 42
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 42 of 52
`
`US 2006/0026535 Al
`
`DETERMINE APPLICATION
`
`852
`
`SELECT APPROPRIATE GUI MODE
`
`DETERMINE CURRENTSTATE OF
`APPLICATION
`
`DETERMINE TOUCH
`
`FIG. 32
`
`HP Inc. - Exhibit 1020 - Page 43
`
`HP Inc. - Exhibit 1020 - Page 43
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 43 of 52
`
`US 2006/0026535 Al
`
`902
`
`DISPLAY LIST OF SONGS
`
`904
`
`
`TOUCH
`DETECTED?
`
`
`
`
`YES
`
`ACTIVATE VIRTUAL SCROLL WHEEL
`IN ADDITION TO LIST OF SONGS
`
`906
`
`916
`
`WHEEL
`
`DEACTIVATE
`VIRTUAL SCROLL
`
`910
`
`
`
`
`
`SCROLLING
`
`TOUCH EVENT
`PERFORMED?
`
`NO
`
`
`
`SCROLL THROUGHLIST OF SONGSIN
`ACCORDANCE WITH TOUCH EVENT
`
`
`FIG. 33
`
`HP Inc. - Exhibit 1020 - Page 44
`
`HP Inc. - Exhibit 1020 - Page 44
`
`

`

`Patent Application Publication Feb. 2,2006 Sheet 44 of 52
`
`US 2006/0026535 Al
`
`
`
`
`
`936
`
`FIG. 34D
`
`
`
`932A
`
`|
`
`FIG. 34C
`
`
` A 936
`
`
`FIG. 34F
`
`HP Inc. - Exhibit 1020 - Page 45
`
`HP Inc. - Exhibit 1020 - Page 45
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 45 of 52
`
`US 2006/0026535 Al
`
`28
`
`930B
`
`music managemen
`
`ariseTSiDum
`
`[a
`
`FIG. 35A
`
`928
`
`c®eowca€|9a)5©
`
`925a
`
`music managemen
`
`WtteaieilAFEL
`
`FIG. 35B
`
`28
`
`oOOoOo
`
`acoNrs)
`
`FIG. 35C
`
`HP Inc. - Exhibit 1020 - Page 46
`
`HP Inc. - Exhibit 1020 - Page 46
`
`
`
`
`
`
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 46 of 52
`
`US 2006/0026535 Al
`
`music managemen
`
`|(
`
`2
`
`3
`
`28
`
`930B
`
`936
`
`925
`
`eaE|ac||IOaS|&
`
`FIG. 35D
`
`bo
`
`2
`
`FIG. 35E
`
`music managemen
`
`[scngname ime|
`pees
`
`aae
`[eng
`
`inrtHn224cE
`
`HP Inc. - Exhibit 1020 - Page 47
`
`HP Inc. - Exhibit 1020 - Page 47
`
`
`
`
`
`
`
`

`

`Patent Application Publication Feb. 2,2006 Sheet 47 of 52
`
`US 2006/0026535 Al
`
`944 942
`
`photo Managemen
`
`photomanagemen
`Photo
`Photo
`
`2
`
`3
`
`;
`
`|
`
`4
`
`[oooNNphoto managemen
`
`Fig. 36B
`
`Fig. 36C
`
`943
`
`936
`
`HP Inc. - Exhibit 1020 - Page 48
`
`HP Inc. - Exhibit 1020 - Page 48
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 48 of 52
`
`US 2006/0026535 Al
`
`952
`
`950
`
`
`
`weeeeeeeeKeo
`
`956
`
`962
`
`GUI ELEMENT
`
`ACTIVATEFIRST GUI
`ELEMENT
`
`ACTIVATE SECOND
`
`960
`
`DEACTIVATE FIRST
`
`GU! ELEMENT
`
`DEACTIVATE
`SECOND GUI
`ELEMENT
`
`958
`
`966
`
`FIG. 37
`
`HP Inc. - Exhibit 1020 - Page 49
`
`HP Inc. - Exhibit 1020 - Page 49
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 49 of 52
`
`US 2006/0026535 Al
`
`music managemen'
`
`oo00
`
`960
`
`playing: song 1 aa Oo
`
`$ESI
`aren)(rongrareTITS aTOUTTOOTS
`ei
`party shuMe
`radio
`a
`90s music
`80s music
`70s music
`top 25 most played
`
`964
`
`
`
`
`
`
`
`
`968
`
`top 25 most payed
`
`FIG. 38A
`
`960
`
`964
`
`900
`
`frou
`
`
`
`
`top 25 most played
`
`EL
`
`i
`
` ‘ul
`
`
`: EIi :1
`
`elesHeeIA
`
`
`
`
` FIG. 38C
`
`HP Inc. - Exhibit 1020 - Page 50
`
`HP Inc. - Exhibit 1020 - Page 50
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 50 of 52
`
`US 2006/0026535 Al
`
`966
`
`=|fs[)oacal—203E
`
`960
`
`playing: song 1
`
`<> O
`
`FIG. 38D
`
`=o—aoo=5E272)S|£
`
`<a> O
`rary
`x0)OT e
`a
`
`playing: song 1
`
`960
`
`FIG. 38E
`
`CcaEoO||aE2a]SI
`
`playing: song 1
`
`ae
`
`HP Inc. - Exhibit 1020 - Page 51
`
`HP Inc. - Exhibit 1020 - Page 51
`
`
`
`€
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 51 of 52
`
`US 2006/0026535 Al
`
`960
`
`
`asic
`top 25 most played
`
`FIG. 38G
`
`960
`
`FIG. 38H
`
`HP Inc. - Exhibit 1020 - Page 52
`
`HP Inc. - Exhibit 1020 - Page 52
`
`

`

`Patent Application Publication Feb. 2, 2006 Sheet 52 of 52
`
`US 2006/0026535 Al
`
`990
`
`playing: song 1)
`
`960
`
`ae|
`
`964
`
`
`FIG. 38l
`
`962
`
`music managemen
`
`960
`
`96
`
`FIG. 38J
`
`HP Inc. - Exhibit 1020 - Page 53
`
`HP Inc. - Exhibit 1020 - Page 53
`
`

`

`US 2006/0026535 Al
`
`Feb. 2, 2006
`
`MODE-BASED GRAPHICAL USER INTERFACES
`FOR TOUCH SENSITIVE INPUT DEVICES
`
`CROSS-REFERENCE TO RELATED
`APPLICATIONS
`
`[0001] This application is a Continuation-in-Part of U.S.
`patent application Ser. No. 10/903,964, filed on Jul. 30,
`2004, entitled “GESTURES FOR TOUCH SENSITIVE
`INPUT DEVICES,” and which is herein incorporated by
`reference.
`
`[0002] This application is also related to: (i) U.S. patent
`application Ser. No. 10/256,716 filed, on Sep. 26, 2002 and
`entitled “METHOD AND APPARATUS FOR ACCELER-
`ATED SCROLLING’;(ii) U.S. patent application Ser. No.
`10/259,159, filed on Sep. 26, 2002 and entitled “METHOD
`AND APPARATUS FOR USE OF ROTATIONAL USER
`INPUT”; (iii) U.S. patent application Ser. No. 10/072,765,
`filed on Feb. 7, 2002 and entitled “MOUSE HAVING A
`ROTARYDIAL”; and (iv) U.S. patent application Ser. No.
`10/840,862, filed on May 6, 2004 and entitled “MULTI-
`POINT TOUCH SCREEN”. Eachof these related applica-
`tion are hereby incorporated herein by reference
`
`BACKGROUNDOF THE INVENTION
`
`[0003]
`
`1. Field of the Invention
`
`[0004] The present invention relates generally to gesturing
`associated with touch sensitive devices.
`
`[0005]
`
`2. Description of the Related Art
`
`[0006] There exist today many styles of input devices for
`performing operations in a computer system. The operations
`generally correspond to moving a cursor and makingselec-
`tions on a display screen. The operations may also include
`paging,
`scrolling, panning, zooming, etc. By way of
`example, the input devices may include buttons, switches,
`keyboards, mice, trackballs, touch pads, joy sticks, touch
`screens and the like. Each of these devices has advantages
`and disadvantages that are taken into account when design-
`ing the computer system.
`
`[0007] Buttons and switches are generally mechanical in
`nature and provide limited control with regards to the
`movement of
`the cursor and making selections. For
`example, they are generally dedicated to moving the cursor
`in a specific direction (e.g., arrow keys) or to making
`specific selections (e.g., enter, delete, number, etc.).
`
`In mice, the movement of the input pointer corre-
`[0008]
`sponds to the relative movements of the mouseas the user
`moves the mouse along a surface. In trackballs, the move-
`ment of the input pointer correspondsto the relative move-
`ments of a ball as the user movesthe ball within a housing.
`Mice and trackballs also include one or more buttons for
`making selections. Mice mayalso include scroll wheels that
`allow a user to move through the GUI by simply rolling the
`wheel forward or backward.
`
`[0009] With touch pads, the movementof the input pointer
`corresponds to the relative movements of the user’s finger
`(or stylus) as the finger is moved along a surface ofthe touch
`pad. Touch screens, on the other hand, are a type of display
`screen that has a touch-sensitive transparent panel covering
`the screen. When using a touch screen, a user makes a
`
`selection on the display screen by pointing directly to GUI
`objects on the screen (usually with a stylus or finger).
`
`In order to provide additionally functionality, ges-
`[0010]
`tures have been implemented with some of these input
`devices. By way of example, in touch pads, selections may
`be made when oneor more taps are detected on the surface
`of the touch pad. In some cases, any portion of the touch pad
`may be tapped, and in other cases a dedicated portion of the
`touch pad maybe tapped. In additionto selections, scrolling
`may be initiated by using finger motion at the edge of the
`touch pad.
`
`([0011] US. Pat. Nos. 5,612,719 and 5,590,219, assigned
`to Apple Computer, Inc. describe some other uses of ges-
`turing. U.S. Pat. No. 5,612,719 discloses an onscreen button
`that is responsive to at least two different button gestures
`made on the screen on or near the button. U.S. Pat. No.
`5,590,219 discloses a method for recognizing an ellipse-type
`gesture input on a display screen of a computer system.
`
`In recent times, more advanced gestures have been
`[0012]
`implemented. For example, scrolling may be initiated by
`placing four fingers on the touch pad so that the scrolling
`gesture is recognized and thereafter moving these fingers on
`the touch pad to perform scrolling events. The methods for
`implementing these advanced gestures, however, has several
`drawbacks. By way of example, once the gesture isset, it
`cannot be changed until the user resets the gesture state. In
`touch pads, for example,if four fingers equals scrolling, and
`the user puts a thumb down after the four fingers are
`recognized, any action associated with the new gesture
`including four fingers and the thumb will not be performed
`until the entire hand is lifted off the touch pad and put back
`downagain (e.g., reset). Simply put, the user cannot change
`gesture states midstream. Along a similar vein, only one
`gesture may be performed at any given time. That
`is,
`multiple gestures cannot be performed simultaneously.
`
`[0013] Based on the above, there is a need for improve-
`ments in the way gestures are performedon touch sensitive
`devices.
`
`SUMMARYOF THE INVENTION
`
`[0014] The invention pertains to gestures and methods of
`implementing gestures with
`touch sensitive devices.
`Examples of touch sensitive devices include touch screens
`and touch pads. The invention also pertains to user interfaces
`and methods of implementing user interfaces with displays.
`
`[0015] The invention can be implemented in numerous
`ways, including as a method, graphical user interface, com-
`puting device, or a computer readable medium. Several
`embodiments of the invention are discussed below.
`
`[0016] The invention relates, in one embodiment, to a user
`interface method. The method includes detecting a touch.
`The method also includes determining a user interface mode
`when a touch is detected. The method further includes
`activating one or more GUI elements based on the user
`interface mode and in response to the detected touch.
`
`{0017] The inventionrelates, in one embodiment, to a user
`interface method. The method includes displaying a group of
`media items. The method also includes detecting a touch
`over the group of media items that are displayed. The
`method further includes activating a virtual scroll wheel
`
`HP Inc. - Exhibit 1020 - Page 54
`
`HP Inc. - Exhibit 1020 - Page 54
`
`

`

`US 2006/0026535 Al
`
`Feb. 2, 2006
`
`when a touch is detected over the displayed media items.
`Theactivation of the virtual scroll wheel includes displaying
`and enabling the functionality ofthe virtual scroll wheel, the
`virtual scroll wheel providing a touch region where a user
`swirls their finger in order to traverse through the group of
`media items. The method additionally includes determining
`if a touch event is performedrelative to the touch region of
`the virtual scroll wheel. Moreover,
`the method includes
`scrolling through the group of media items whena scrolling
`touch event is performed.
`
`to a
`in one embodiment,
`[0018] The invention relates,
`method performed on a user operated electronic device
`having a display and a touch screen. The method includes
`determining if a touch is detected. The methodalso includes
`monitoring and analyzing the current operating conditions
`whena touch is detected. The methodalso includesactivat-
`
`ing a first GUI elementfora first set of operating conditions.
`The method additionally includes activating a second GUI
`element for a second set of operating conditions.
`
`to a
`in one embodiment,
`[0019] The invention relates,
`method performed on a computing device having a display
`and a touch sensing input device. The method includes
`sensing touches. The method also includes displaying and
`enabling a GUI element when a touch is detected. The GUI
`element is based on at least one of the following: (a) the
`application currently running on the computing device; (b)
`the current state of the application; and/or (c) one or more
`characteristics of the touch. The characteristics include, for
`example, touch location, touch ID, numberof touches, and
`touch motion. The method further includes disabling and
`removing the GUI element from display when one of the
`following events occurs: (a) the touch is no longer detected,
`(b) a touch hasnot been detected for a preset amount oftime,
`(c) a certain amount of time has gone by since the step of
`displaying and enabling, (d) a user selection.
`
`[0020] The invention relates, in another embodiment, to a
`computing system. The computing system includesa display
`device configured to display a graphical user interface. The
`system also includes a touch screen positioned over the
`display device. The touch screen is configured to detect
`touches that occur over the display device. The system
`further includes a processor operatively coupled to the
`display device and the touch screen. The processorinstructs
`the display device to display one or more GUI elements in
`responseto a touch, and performsactions associated with the
`GUIelement when touch events are detected relative to the
`displayed GUI elements.
`
`[0021] The invention relates, in another embodiment, to a
`computing device. The computing device includes a proces-
`sor. The computing device also includes a touch screen
`capable of sensing touch events. The computing device
`further includes a display configured to simultaneously
`display a plurality of media items and a virtual scroll wheel.
`The virtual scroll wheel provides a region where touch
`events are performed in order to implement a scrolling
`action. The scrolling action allowsa userto traverse through
`the plurality of media items.
`
`[0022] Other aspects and advantagesof the invention will
`become apparent from the following detailed description,
`taken in conjunction with the accompanying drawings,
`illustrating by way of example the principles of the inven-
`tion.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`[0023] The invention will be readily understood by the
`following detailed description in conjunction with the
`accompanying drawings, wherein like reference numerals
`designate like structural elements, and in which:
`
`FIG.1 is a block diagram of a computer system, in
`[0024]
`accordance with one embodiment of the present invention.
`
`[0025] FIG.2 is a multipoint processing method, in accor-
`dance with one embodiment of the present invention.
`
`[0026] FIGS. 3A andB illustrate an image, in accordance
`with one embodimentof the present invention.
`
`FIG.4 illustrates a group of features, in accordance
`[0027]
`with one embodimentof the present invention.
`
`[0028] FIG.5 is a parameter calculation method,in accor-
`dance with one embodiment of the present invention.
`
`[0029] FIGS. 6A-6Gillustrate a rotate gesture, in accor-
`dance with one embodiment of the present invention.
`
`[0030] FIG.7 is a diagram of a touch-based method,in
`accordance with one embodiment of the present invention.
`
`{0031] FIG. 8 is a diagram of a touch-based method, in
`accordance with one embodiment of the present invention.
`
`[0032] FIG. 9 is a diagram of a touch-based method, in
`accordance with one embodimentof the present invention.
`
`[0033] FIG. 10 is a diagram of a zoom gesture method,in
`accordance with one embodiment of the present invention.
`
`[0034] FIGS. 11A-11H illustrates a zooming sequence,in
`accordance with one embodiment of the present invention.
`
`[0035] FIG. 12 is a diagram of a pan method, in accor-
`dance with one embodiment of the present invention.
`
`[0036] FIGS. 13A-13Dillustrate a panning sequence, in
`accordance with one embodiment of the present invention.
`
`in
`[0037] FIG. 14 is a diagram of a rotate method,
`accordance with one embodiment of the present invention.
`
`[0038] FIGS. 15A-15C illustrate a rotating sequence, in
`accordance with one embodimentof the present invention.
`
`[0039] FIG. 16 is a diagram of a GUI operational method,
`in accordance with one embodiment of the present inven-
`tion.
`
`floating control
`a
`[0040] FIGS. 17A-17E illustrate
`sequence, in accordance with one embodimentofthe present
`invention.
`
`FIG.18 is a diagram of a GUI operational method,
`(0041]
`in accordance with one embodimentof the present inven-
`tion.
`
`zooming target
`a
`[0042] FIGS. 19A-19D illustrate
`sequence, in accordance with one embodimentofthe present
`invention.
`
`[0043] FIG. 20 is a diagram of a GUI operational method,
`in accordance with one embodimentof the present inven-
`tion.
`
`[0044] FIGS. 21A-21Dillustrate a page turning sequence,
`in accordance with one embodimentof the present inven-
`tion.
`
`HP Inc. - Exhibit 1020 - Page 55
`
`HP Inc. - Exhibit 1020 - Page 55
`
`

`

`US 2006/0026535 Al
`
`Feb. 2, 2006
`
`FIG.22 is a diagram of a GUI operational method,
`[0045]
`in accordance with one embodimentof the present inven-
`tion.
`
`elements based on the user interface mode(e.g., displaying
`a GUI element). Another aspect of the inventionrelates to a
`virtual scroll wheel.
`
`[0046] FIGS. 23A-23Dillustrate an inertia sequence, in
`accordance with one embodiment of the present invention.
`
`FIG.24 is a diagram of a GUI operational method,
`[0047]
`in accordance with one embodimentof the present inven-
`tion.
`
`[0048] FIGS. 25A-25D illustrates a keyboard sequence,in
`accordance with one embodiment of the present invention.
`
`FIG.26 is a diagram of a GUI operational method,
`[0049]
`in accordance with one embodimentof the present inven-
`tion.
`
`scroll wheel
`a
`27A-27D illustrates
`[0050] FIGS.
`sequence, in accordance with one embodimentofthe present
`invention.
`
`[0051] FIG. 28 is user interface method, in accordance
`with one embodimentof the present invention.
`
`in
`[0052] FIGS. 29A-29D illustrate a transition effect,
`accordance with one embodiment of the present invention.
`
`in
`[0053] FIGS. 30A-30D illustrate a transition effect,
`accordance with another embodiment of the present inven-
`tion.
`
`in
`[0054] FIGS. 31A-31D illustrate a transition effect,
`accordance with another embodimentof the present inven-
`tion.
`
`[0055] FIG. 32 is a determination method, in accordance
`with one embodiment of the present invention.
`
`FIG.33 is a user interface method, in accordance
`[0056]
`with one embodimentof the present invention.
`
`[0057] FIG. 34A-34F illustrates a sequence associated
`with the method shown in FIG. 33, in accordance with one
`embodiment of the present invention.
`
`[0058] FIG. 35A-35F illustrates a sequence associated
`with the method shownin FIG. 33, in accordance with one
`embodimentof the present invention.
`
`interface
`user
`a
`[0059] FIGS. 36A-36C illustrate
`sequence, in accordance with one embodimentofthe present
`invention.
`
`[0061] The invention pertains to gestures and methods of
`implementing gestures with touch sensitive devices.
`Examples of touch sensitive devices include touch screens
`and touch pads. The invention also pertains to user interfaces
`and methodsof implementing user interfaces with displays.
`One aspect of the invention relates to determining a user
`interface mode based on one or more conditions. Another
`
`aspect of the invention relates to activating one or more GUI
`
`FIG.37is a user interface method, in accordance
`[0060]
`with one embodimentof the present invention FIGS. 38A-
`38J illustrate a user interface sequence, in accordance with
`[0066] The computer system 50 also includes a display
`one embodimentof the present invention.
`device 68 that is operatively coupled to the processor 56.
`The display device 68 maybealiquid crystal display (LCD)
`DETAILED DESCRIPTION OF THE
`(e.g., active matrix, passive matrix and the like). Alterna-
`INVENTION
`tively, the display device 68 may be a monitor such as a
`monochromedisplay, color graphics adapter (CGA)display,
`enhanced graphics adapter (EGA) display, variable-graph-
`ics-array (VGA) display, super VGA display, cathode ray
`tube (CRT), and the like. The display device may also
`correspond to a plasma display or a display implemented
`with electronic inks.
`
`[0062] These and other aspects of the invention are dis-
`cussed below with reference to FIGS. 1-38J. However,
`those skilled in the art will readily appreciate that
`the
`detailed description given herein with respect to these fig-
`ures is for explanatory purposes as the invention