as) United States
`a2) Patent Application Publication (10) Pub. No.: US 2005/0024341 Al
`(43) Pub. Date: Feb. 3, 2005
`
`Gillespie et al.
`
`US 20050024341A1
`
`(54)
`
`(75)
`
`TOUCH SCREEN WITH USER INTERFACE
`ENHANCEMENT
`
`Inventors: David W. Gillespie, Los Gatos, CA
`(US); Ray Trent, San Jose, CA (US);
`Andrew C. Hsu, Belmont, CA (US);
`Leslie R. Grate, Waltham, MA (US)
`
`Publication Classification
`
`Ente C07 caccccsscsssssssssnsssssesnstsnssnsssevee G09G 5/00
`(SL)
`(52) US. Ch.
`cesescsssssssssstsssinessstnstvassetnstnnsvesse 345/173
`
`(57)
`
`ABSTRACT
`
`Correspondence Address:
`SIERRA PATENT GROUP, LTD.
`PO BOX6149
`STATELINE, NV 89449 (US)
`
`(73)
`
`Assignee: Synaptics, Inc.
`
`(21)
`
`Appl. No.:
`
`10/125,067
`
`(22)
`
`Filed:
`
`Apr. 17, 2002
`
`Related U.S. Application Data
`
`(60)
`
`Provisional application No. 60/291,694,filed on May
`16, 2001.
`
`The present invention is a graphical user interface in a
`computing device having a processor running an operating
`system and a display. The graphical user interface comprises
`a touch screen and a driver coupling the touch screen to the
`operating system. The driver can display a plurality of icons
`on the touch screen, or a plurality of screen images having
`at least one icon, with each of the icons associated with
`operations on the display and/or the touch screen. Other
`embodiments include the touch screen having unactivated
`and activated states, as well as the presence of an application
`programming interface that enables an application to display
`at least one image on the touch screen.
`
`428
`
`416
`
`400
`
`412
`
`426 418
`
`APPLE 1030
`
`APPLE 1030
`
`1
`
`

`

`Patent Application Publication Feb. 3, 2005 Sheet 1 of 14
`
`US 2005/0024341 Al
`
`
`
`Figure 2
`
`2
`
`

`

`Patent Application Publication Feb. 3, 2005 Sheet 2 of 14
`
`US 2005/0024341 Al
`
`300
`
`414
`
`3
`
`

`

`Patent Application Publication Feb. 3, 2005 Sheet 3 of 14
`
`US 2005/0024341 Al
`
`<+—WwW
`
`OaQ2-<x
`
`Figure 5
`
`4
`
`

`

`Patent Application Publication Feb. 3, 2005 Sheet 4 of 14
`
`US 2005/0024341 Al
`
`600
`
`604
`
`602
`
`Figure 6A
`
`aa
`
`Figure 6B
`
`5
`
`

`

`Patent Application Publication Feb. 3, 2005 Sheet 5 of 14
`
`US 2005/0024341 Al
`
`703
`
`700
`
`710 \
`
`700
`
`720 ~_
`
`700
`
`722
`
`Figure 7A
`
`Figure 7B
`
`Figure 7C
`
`730
`
`732 LB
`
`700
`
`740 ® 742
`
`746
`
`\
`
`744
`
`|
`
`Figure 7D
`
`Figure 7E
`
`6
`
`

`

`Patent Application Publication Feb. 3, 2005 Sheet 6 of 14
`
`US 2005/0024341 Al
`
`800
`
`Slide 6: 2001 sales data
`
`Used Fred’s estimates for UK rgn
`
`Remember to highlight new prods!
`
`Projections
`for 2002 sales
`a
`
`806 oeaawsnOweey
`826
`
`Dept intro
`
`<1
`
`Figure 8A
`
`Chapter QG
`
`Time 94:58
`
`802
`
`820
`
`Cy
`
`822
`
`$24
`
`Figure 8B
`
`7
`
`

`

`Patent Application Publication Feb. 3, 2005 Sheet 7 of 14
`
`US 2005/0024341 Al
`
`
`
`Paying too much for jewelry?
`
`www.hot-jewelry.com
`
`842
`
`
`
`Figure 8C
`
`Figure 8D
`
`870
`
`8
`
`

`

`Patent Application Publication Feb. 3, 2005 Sheet 8 of 14
`
`US 2005/0024341 Al
`
`Format
`
`Painter.
`
`902
`
`908
`
`Figure 9
`
`Copies
`the format from a selected
`or text and applies it to the
`object
`or text you click.
`To copy the
`object
`to more than one item, double-
`format
`click the icon, click each item you
`want to format,
`then press ESC.
`
`More Help
`
`
`9
`
`

`

`Patent Application Publication Feb. 3, 2005 Sheet 9 of 14
`
`US 2005/0024341 Al
`
`mH
`c. Flaps
`
`Throttle
`
`
`Figure 10B
`
`Figure 10C
`
`10
`
`

`

`Patent Application Publication Feb. 3, 2005 Sheet 10 of 14
`
`US 2005/0024341 Al
`
`1100 1102
`3.14159265359
`
`Figure 11
`
`11
`
`

`

`Patent Application Publication Feb. 3, 2005 Sheet 11 of 14
`
`US 2005/0024341 Al
`
`
`Magnification Level
`
`Figure 13A
`
`1320
`
`Figure 13B
`
`12
`
`

`

`{
`
`1408
`
`printf ("Hello");
`
`getUserName ();
`1402
`
`Patent Application Publication Feb. 3, 2005 Sheet 12 of 14
`
`US 2005/0024341 Al
`
`Figure 13C
`
`Figure 14
`
`13
`
`

`

`Patent Application Publication Feb. 3, 2005 Sheet 13 of 14
`
`US 2005/0024341 Al
`
`1500
`
`a Stop! This computeris registered to
`John Q. Doe
`
`Enter your PIN
`
`and touch Begin to proceed.
`
`Sean] [0|
`
`Figure 15A
`
`1520
`
`\y
`
`1522
`
`1524
`
`1528
`
`1530
`
`Display &
`Central
`Hard Disk
`Processor
`
`
`
`
`
`
`Other
`Devices
`
`
`
`
`
`
`Peripheral
`Touch
`
`Controller
`
`
`Screen
`
`
`1526
`
`Figure 15B
`
`14
`
`

`

`Patent Application Publication Feb. 3, 2005 Sheet 14 of 14
`
`US 2005/0024341 Al
`
`1600
`
`1610
`
`1612
`
`1614
`
`Touch
`Screen
`
`Peripheral
`Controller
`
`Hardware
`
`Driver
`1624
`
`Application pTeeseee
`
`1602
`
`1604
`
`1606
`
`1622
`
`Pointing
`Device
`Driver
`
`1620
`
`Touch
`Screen
`Driver
`
`Software
`
`1640
`
`1642
`
`1644
`
`Figure 16
`
`15
`
`

`

`US 2005/0024341 Al
`
`Feb. 3, 2005
`
`TOUCH SCREEN WITH USER INTERFACE
`ENHANCEMENT
`
`PRIORITY TO RELATED APPLICATIONS
`
`[0001] The present application claims priority to U.S.
`Patent Provisional Application Ser. No. 60/291,694, entitled
`“Touch Screen with User Interface Enhancement”, filed on
`May 16, 2001, which is incorporated herein in its entirety.
`
`BACKGROUND
`
`[0002] The present invention relates to computer interface
`devices, and moreparticularly, to a computer touch pad with
`integrated display device, and enhancementsto the portable
`computer user interface employing same.
`
`[0003] Touch pads are widely used in computer applica-
`tions, particularly as pointing devices in portable computers.
`In typical usage,
`the touch pad is a featureless, finger
`sensitive surface in a rectangular opening of the palm rest of
`the computer. The touch pad servessolely as an input device
`for the computer. The touch pad functions primarily as a
`cursor pointing device, but some touch padsoffer additional
`functions.
`
`[0004] For example, U.S. Pat. No. 5,543,591 to Gillespie
`et al. discloses a typical prior art touch pad sensor in which
`finger tapping gestures in designated regions of the touch
`surface invoke special commands on the computer. U.S. Pat.
`No. 5,943,052 to Allen et al. discloses a touch pad in which
`finger motions in designated regions invoke a scrolling
`command. These tap regions and scrolling regions have
`proven useful to expert users but confusing to novice users
`as the regions are invisible to the eye but different
`in
`behavior. Marking the regions with screen-printed icons on
`the opaque sensor surface can help, but it can also lead to
`greater confusion if the regions are software configurable.
`
`[0007] A transparent touch pad suitable for placement over
`a display such as an LCD screen has been developed andis
`disclosed and claimed in co-pending U.S. patent application
`Ser. No. 09/415,481, filed Oct. 8, 1999, assigned to the same
`assignee as the present invention. This application discloses
`a touch screen having the small size and low cost of a
`conventional touch pad for portable computers and notes
`that
`the touch pad and display could be included in a
`personal computer to enhance the user interface in various
`ways, but it does not disclose details of the software imple-
`mentation, nor how such a device can simultaneously func-
`tion as the pointing device of the computer, nor how this
`arrangement enhancesthe user interface.
`
`SUMMARY
`
`[0008] The drawbacks and disadvantages of the prior art
`are overcome by the touch screen with user interface
`enhancement.
`
`[0009] The present invention is a graphical user interface
`in a computing device having a processor running an oper-
`ating system and a display. The graphical user interface
`comprises a touch screen and a driver coupling the touch
`screen to the operating system. The driver can display a
`plurality of icons on the touch screen,or a plurality of screen
`images having at least one icon, with each of the icons
`associated with operations on the display and/or the touch
`screen. Other embodiments include the touch screen having
`unactivated and activated states, as well as the presence of
`an application programminginterface that enables an appli-
`cation to display at least one image on the touch screen.
`
`BRIEF DESCRIPTION OF THE DRAWING
`FIGURES
`
`[0010] Referring now to the figures, wherein like elements
`are numberedalike:
`
`a diagram showing an illustrative
`([0012] FIG. 2 is
`embodiment of a touch screen in greater detail;
`
`[0005] A further disadvantage of prior art touch pads is
`that they use up a significant fraction of the surface area of
`{0011] FIG.1is a diagram showing a notebook computer
`the computer for a single dedicated input function. Other
`system with main display, keyboard, and touch screen;
`pointing devices such as isometric joysticks (see, e.g., U.S.
`Pat. No. 5,521,596 to Selker et al) and force sensing keys
`(see, e.g., U.S. Pat. No. 4,680,577 to Straayer et al) have
`been proposed as compactalternatives, but these devices are
`[0013] FIG.3isa diagram illustrating an example default
`not as expressive or as easy to use as touch pads.
`image for use when the touch screen is operating as a
`conventional touch pad;
`
`[0006] Touch screens are also well knownin the art. One
`example of a touch screen is disclosed in U.S. Pat. No.
`4,806,709 to Blair. In typical use, the main display screen of
`a computer is overlaid with or implemented as a touch
`sensitive input device. This eliminates the need to dedicate
`separate parts of the surface of the computer for input and
`output. If the touch screen serves as the main pointing device
`of the computer, pointing is accomplished by a direct
`mapping from finger position to selection of a point on the
`screen beneath the finger. This direct mapping makes touch
`screens easy to understand and use. However, touch screens
`are impractical for everyday use as the main display of a
`computer because the user’s arm tires from being continu-
`ously held up to touch the screen. If the touch screen is laid
`flat to avoid arm wear, the arm tends to rest on the touch-
`sensing surface and, with many touch sensing technologies,
`this disrupts the ability to sense the finger. Touch screens the
`size of a main computer display may also be prohibitively
`bulky or expensive for use in applications that do not require
`them.
`
`[0014] FIG. 4 is a diagram illustrating an example of a
`first “iconic” usage mode of the touch screen;
`
`[0015] FIG. 5 is a diagram illustrating the touch screen
`image of FIG. 4 modified to indicate the activated state of
`the touch screen using a dashed line around each icon that
`is touch-sensitive in the activated state;
`
`[0016] FIG. 6A is a diagram illustrating a portion of the
`keyboard featuring several keys; FIG. 6B is a diagram
`illustrating one possible arrangement of a special
`touch
`sensitive region or second touch sensor could be provided
`that activates the touch screen when touched;
`
`[0017] FIG. 7A is a diagram illustrating small icons that
`may be smaller than a finger and may be completely
`obscured by the finger when the finger touches them;
`
`[0018] FIG. 7B through 7E illustrate several mechanisms
`to eliminate the problem of obscuring small icons;
`
`16
`
`16
`
`

`

`US 2005/0024341 Al
`
`Feb. 3, 2005
`
`[0019] FIGS. 8A through 8D are diagrams illustrating
`use of a small control panel on the touch screen associated
`with an application, reserving the entire main display for
`visual data associated with the application;
`
`[0020] FIG. 9 is a diagram showing an example use of the
`touch screen to display subsidiary help text;
`
`[0021] FIG. 10A is a diagram illustrating employment of
`the touch screento display a find/replace dialog on the touch
`screen, leaving the main display free to display a document
`unobstructed;
`
`[0022] FIG. 10B is a diagram illustrating use of the touch
`screen to act as a joystick emulator while displaying the
`control layout established by the game, leaving the main
`display free to display game graphics unobstructed;
`
`[0023] FIG. 10C is a diagram illustrating an example in
`which a touch screen image includes icons drawn from a
`typical toolbar,
`leaving the main display free to display
`document or an image unobstructed;
`
`[0024] FIG. 11 is a diagram illustrating a pop-up image
`including various icons representing commonly used tools
`and software applications on the computer;
`
`[0025] FIG. 12 is a diagram illustrating a pop-up calcu-
`lator application that operates entirely within the touch
`screen;
`
`[0026] FIGS. 13A and 13B are diagrams illustrating
`different features of a magnifier as a pop-up image on a
`touch screen, leaving the main display undisturbed;
`
`[0027] FIG. 13C is a diagram illustrating a debugger
`implemented as a pop-up application on a touch screen,
`providing a secondary debugging display with no extra cost
`or bulk;
`
`[0028] FIG. 14 is a diagram illustrating an example of an
`ideographic handwriting entry system on a touch screen in
`which a handwriting entry area respondsto finger touch to
`enter an ideographic character;
`
`[0029] FIG. 15A is a diagram illustrating use of a touch
`screen as a user interface device for a computer security
`interlock;
`
`[0030] FIG. 15B is a diagram illustrating an exemplary
`hardware architecture for implementing the computer secu-
`rity interlock of FIG. 15A; and
`
`[0031] FIG. 16 is a diagram illustrating an exemplary
`software architecture for a touch screen.
`
`shown in FIG.1, but other locations are equally applicable,
`such as above the keyboard, adjacent to the keyboard or
`main display, or located in a separate enclosure connected by
`cable or wireless link to the computer. Although touch
`screen 106 usually replaces the conventional touch pad of a
`computer, touch screen 106 could be introduced in addition
`to the other user interface devices of the computer.
`
`[0034] FIG. 2 illustrates an illustrative embodiment of
`touch screen 106 in greater detail. Touch screen assembly
`200 consists of touch sensor 202, display 204, and backlight
`206 stacked or laminated together. Touch screens can be
`built in a variety of alternative ways as are well knownin the
`art. For example, touch sensor 202 can be an active sensor
`employing capacitive, resistive, inductive, or other methods,
`or it can be a passive surface on which touch sensing is
`accomplished by optical, acoustic, or other methods.
`Capacitive touch sensors are ideally suited for use in the
`present invention due to their sensitivity, low cost, rugged-
`ness, and suitability to small sensing areas. However, any
`touch screen technology would serve for the present inven-
`tion.
`
`[0035] Similarly, display 204 can be a liquid crystal dis-
`play (LCD), organic light emitting diode (OLED)display,
`electroluminescent display, or any other type of small dis-
`play suitable for mounting in a portable computer. LCD
`displays are ideally suited for use in the present invention
`due to their low cost and availability, but other types of
`displays may be employed. Display 204 may becolor or
`monochrome, and need not have the same resolution, color
`capabilities, or other qualities as the main display of the
`computer.
`
`[0036] The touch screen assembly may include a backlight
`206 to enhance readability in all lighting conditions. In
`alternative embodiments, backlight 206 may be replaced by
`a frontlight, passive reflector, or other light source, or it may
`be omitted altogether.
`
`[0037] Touch screen assembly 200 mayinclude additional
`layers or components to assist the mounting or mechanical
`properties of the touch screenorto integrate the touch screen
`with other components of the computer system. The touch
`screen mayalso include hardened,antireflective, textured, or
`other surface layers. The inclusion, omission, or nature of
`these additional layers and components is immaterial to the
`present invention.
`
`[0038] Touch sensor 202 is connected to touch sensing
`controller 208. The nature of controller 208 depends on the
`design of touch sensor 202 and its details are immaterial to
`the present invention. Likewise, display 204 is connected to
`a suitable display controller 210, and backlight 206,
`if
`present, is connected to backlight controller 212. Each of
`[0032] Those of ordinary skill in the art will realize that
`controllers 208, 210, and 212 communicate with host com-
`the following description of the present invention is illus-
`puter 214. In an illustrative embodiment, controllers 208,
`trative only and not in any way limiting. Other embodiments
`210, and 212 are connected to a central
`touch screen
`of the invention will readily suggest themselves to such
`controller 216 that connects to host computer 214 byasingle
`skilled persons.
`interface 218. Interface 218 may be a mouse interface such
`as PS/2, or a general purpose peripheral interface such as the
`Universal Serial Bus (USB). USB hasthe advantage of high
`bandwidth and wide availability. Any of controllers 208,
`210, 212, and 216 may be implementedas chips or discrete
`components, combined onto fewer chips or one chip, inte-
`grated with assembly 200, or combined with other functions
`of host computer 214. Host computer 214 may be embodied
`
`DETAILED DESCRIPTION
`
`FIG.1 illustrates a notebook computer system 100
`[0033]
`with main display 102 and keyboard 104. Touch screen 106
`is mounted in palm rest 110. The touch screen is typically
`equipped with left and right “mouse” buttons 108. Touch
`screen 106 is integrated into computer system 100 in a
`similar way as a touch pad would be in a prior art computer.
`Touch screen 106 will usually be located in the palm rest as
`
`17
`
`17
`
`

`

`US 2005/0024341 Al
`
`Feb. 3, 2005
`
`in the central processing unit of computer system 100, a
`peripheral processor such as a USB host controller, or a
`combination thereof.
`
`corner tap region that simulates a right mouse button click,
`and an icon 308 which represents a logo for the computer
`vendor.
`
`In an alternative illustrative embodiment, control-
`[0039]
`lers 208, 210, and 212 may connect to host computer 214
`through different
`interfaces. For example,
`touch screen
`controller 208 could connect as a conventional touch pad
`using a PS/2 interface, while display controller 210 and
`backlight controller 212 connect by USB orby a specialized
`display interface.
`
`[0040] Because touch screen 106 of FIG. 1 replaces a
`conventional touch pad, touch screen 106 usually serves as
`a conventional pointing device for the computer. For this
`reason, the touch screen must be able to interface to the
`computer as a conventional mouse. This is a further reason
`for interface 218 to be either a mouseinterface such as PS/2,
`or a general interface such as USBthat includes support for
`conventional mice. Interface 218 may also provide for an
`alternate or extended interface protocol
`that allows for
`additional information about finger activity to be commu-
`nicated to computer 214, and for computer 214 to control
`display 204 and backlight 206. This additional finger activity
`information may include the absolute location of the finger
`on the sensor surface. When appropriate driver software is
`loaded onto computer 214, the driver software can enable
`the alternate or extended interface protocol to support the
`user interface enhancements of the present invention. When
`other driver software, such as a conventional mouse or touch
`pad driver, is loaded instead, interface 218 can revert to
`mouse or touch pad compatibility using touch sensor 202 as
`a conventional touch pad, and controller 210 or 216 can
`operate the display autonomously, such as by furnishing a
`suitable default display image for display 204.
`
`[0041] When the touch screen is used as a conventional
`touch pad, finger motions on the touch sensor (e.g., in a
`cursor positioning region, which could identify a starting
`position) will typically cause corresponding motions of a
`cursor on the main display, and clicks of “mouse” buttons
`(or action control icons) 108 will typically cause special
`actions, such as selections on the main display. Tapping
`gestures may be interpreted as “mouse” clicks or other
`special actions, as disclosed in U.S. Pat. No. 5,543,591.
`Other gestures may also be recognized, such as scrolling
`motions as disclosed in U.S. Pat. No. 5,943,052. The default
`display image may include graphical icons to indicate spe-
`cial tapping or scrolling regions on the touch sensor surface
`or the default screen image may be a blank screen with only
`a manufacturer’s logo.
`
`In one embodiment, the cursor positioning region
`[0042]
`is denoted by the absence of icons for actions, other than
`cursor positioning. However, there are many different ways
`of identifying the cursor positioning region on the touch
`screen, such examples include, but are not limited to, a box
`could enclose the cursor positioning region, a shaded region
`or icon could cover the entire cursor positioning region, or
`an icon could be centered in an otherwise blank area, thus
`labeling the blank area as a cursor positioning region.
`
`[0044] Alternatively, computer system 100 of FIG. 1 can
`include a secondary pointing device, such as an isometric
`joystick located in keyboard 104 or an external mouse,
`which relieves touch screen 106 from the responsibility of
`functioning as primary pointing device in additiontoits role
`as an enhanceduser interface device.
`
`touch pad with default screen
`[0045] A conventional
`imageis just one of several general modesof usage that are
`envisioned for the touch screen of the present invention.
`Subsequent drawing figures illustrate several other usage
`modes that employ the touch screen as a fully interactive
`input/output device to enhance the user interface of the
`computer system. These general usage modes include
`“iconic,“auxiliary,” and “pop-up” touch screen modes,
`each with a variety of possible applications. The same touch
`screen can operate in each of these various modes, or other
`modes, at different
`times. The different modes can also
`appearon the screen at the same time; for example, icons can
`appear in an auxiliary or pop-up image, or an auxiliary or
`pop-up image could be overlaid in a window on the iconic
`mode image instead of fully replacing that image.
`
`[0046] FIG. 4 illustrates an example of a first “iconic”
`usage mode of the touch screen. In the iconic mode, the
`screen displays an image that includes a numberof small
`icons such as pictures or buttons. The touch sensor operates
`as a touch pad pointing device in iconic mode, in which
`finger motions and taps on the sensor are generally inter-
`preted the same as when the touch screen operates as a
`conventional touch pad. The screen image in iconic mode
`may include elements in common with the default image of
`FIG.3, as the two modesoperate similarly. Iconic mode will
`generally display additional iconsrelating to softwarethat is
`running on the computer and other aspects of the operation
`of the computer.
`
`image 400
`In the example image of FIG. 4,
`[0047]
`includesscroll arrow icons 402 and 404 and a touch region,
`such as illustrated by corner tap icon 406 in common with
`FIG.3. Logo 308 has been omitted from image 400 in this
`example to reduce clutter.
`In an alternate embodiment,
`non-critical graphics from the default
`image could be
`retained as a background image on which icons overlap; in
`yet another embodiment, a different image suchas static or
`dynamic “wallpaper” may serve as a background image.
`
`In example image 400, additional icons have been
`[0048]
`added to represent various system status indicators and
`functions. Icon 410 defines a second touch region or a comer
`tapping region to activate the “back” function of web
`browsers and other software. As the user enables and dis-
`
`ables special tap regions and changestheir assigned func-
`tions, such as by using a software control panel, the tap
`region icons such as icons 406 and 410 can appear, disap-
`pear, move, and change in shape to reflect
`the current
`settings.
`
`FIG.3 illustrates an example default image for use
`[0043]
`whenthe touch screen is operating as a conventional touch
`pad. FIG. 3 depicts the image on the touch screen display as
`seen by the user. Image 300 includes arrow icons 302 and
`304 indicating scrolling regions, an icon 306 indicating a
`
`Icon 412 is a continuous display of the time and
`[0049]
`date. This icon would normally have no effect on the
`interpretation of finger taps within its region. Instead, a
`finger tap within its boundaries would be interpreted as a
`simulated mouse button click, just as if the tap occurred
`
`18
`
`18
`
`

`

`US 2005/0024341 Al
`
`Feb. 3, 2005
`
`away from any icon. If every icon responded specially to
`finger taps, the main function of tapping to simulate a mouse
`click would become too inconvenient
`to use. A visual
`convention may be used to indicate which icons represent
`tap-sensitive regions; in the example of FIG.4, dashed lines
`426 and 428are used to indicate these regions.
`
`Icon group 414includesthe traditional set of status
`[0050]
`icons that appear on modern portable computers, such as
`numeric keypad lock, caps lock, scroll
`lock, hard disk
`activity, battery life, and system power. By locating these
`system icons on the touch screen display,
`the system
`designer eliminates the need for the special dedicated LED
`or LCDstatus displays that are typically used in prior art
`computers.
`
`Insomeprior art portable computers, the dedicated
`[0051]
`system status displays are situated so that they are visible
`even whenthe cover of the computer is closed over the main
`display. The touch screen of the present invention could
`similarly be situated so that all or part of the screen image
`is visible when the coveris closed, for example, by causing
`the touch screen to protrude from under the cover or by
`cutting a notch in the cover over the location of the touch
`screen. This arrangement would allow the user to monitor
`battery recharging and other quiescent activities of the
`computer system while the computer is not in use.
`
`Icon 416 is an e-mail notification status icon; icon
`[0052]
`416 may, for example, change to a new shape or animated
`image to indicate that e-mail has arrived. Icon 418 similarly
`notifies the user of imminent appointments. These icons
`suggest a natural action that could be taken when the user
`taps on the icons, such as opening the associated e-mail
`reading or appointment scheduling software. Because these
`icons are located nearer the center of the touch sensing area
`and could easily be tapped by accident, icons 416 and 418
`may be madesensitive to finger taps only when they have
`been activated by some separate means such as pressing a
`special function key on keyboard 104.
`
`Icons 420 and 422 represent commandsto select
`[0053]
`pop-up applications on the touch screen. Icon 420 selects an
`application launcher.
`Icon 422 selects a calculator or
`numeric keypad. Like icons 416 and 418, icons 420 and 422
`may be madesensitive to finger taps only when the touch
`screen is in the activatedstate.
`
`be used to decide on the number, types, and placement of
`icons in the default iconic screen. Also, it may be advanta-
`geous to allow the user to select which icons are present and
`to rearrange the icons, possibly using a software control
`panel. Because the numberof candidate icons likely exceeds
`available space,
`it may be desirable to provide multiple
`iconic screen layouts selectable by some means such as in
`the software control panel or by tapping on an icon on the
`touch screen.
`
`It will be obviousto oneskilled in the art that many
`[0056]
`other images, logos, status indicators, command buttons,
`controls, and other types of icons can share the touch screen
`display in the iconic usage mode. These icons can be purely
`display indicators, or they can indicate control regions that
`respond specially to finger motions and/or fingertaps, either
`at all times or only when the touch screen has been activated
`in a special way. Some icons may be built-in by the system
`designer, such as the system status icons or the logo of the
`computer manufacturer. Other icons may be created and
`maintained by application software running on the com-
`puter, such as an e-mail notification icon.
`
`[0057] The activated state of the touch screen may be
`indicated by a visual convention. FIG.5 illustrates the touch
`screen image of FIG. 4 modified to indicate the activated
`state of the touch screen using a dashed line around each
`icon that is touch-sensitive in the activated state. In image
`500, dashed lines 516, 518, 520, and 522 have surrounded
`certain icons to indicate that finger taps in the regions near
`these icons will be interpreted as special commandsto the
`icons. Similarly, dashed outline 524 indicates that finger
`motions in the volume control region will adjust the setting
`of the control. Outline 512 for the time and date icon has
`
`become dashed to indicate that a tap on this icon will
`activate a special function such as setting the time or
`accessing a world clock. Outline 514 for the system status
`icons remains solid to indicate that, in the example of FIG.
`5,
`these icons have no special
`tapping functions in the
`activated state. Dashed lines 526 and 528 remainto indicate
`that the corner tap regions continue to have their special tap
`interpretations when the touch screen is in the activated
`state. Many other visual conventions would serve equally
`well
`to indicate touch-sensitive icons, such as solid or
`colored lines, colored or inverted backgrounds, changes in
`brightness or coloration of the activated icons, changes in
`shape or animation of the activated icons, or other well-
`known conventions for highlighting a portion of an image.
`
`Icon 424 represents the volume control for the
`[0054]
`sound system and speakers of the computer.
`Icon 424
`[0058] The example of FIGS. 4 and5illustrates the same
`includes a visual slider and “thumb.” The position of the
`set of icons in the activated and unactivated state. However,
`thumb on the slider reflects the current volume setting.
`activation of the touch screen could also create additional
`When the touch screen is in the activated state, finger
`motions within the volume control region can move the
`thumb to a different location on the slider to adjust the
`volume level. When the touch screen is not in the activated
`
`state, icon 424 is a visual display only and has no special
`interpretation when touched. Similar slider controls may be
`provided to adjust other system parameters such as the
`sound balance among several sound sources, the brightness
`and contrast of the main screen or touch screen, or the power
`managementstrategy.
`
`[0055] The icons depicted in FIG.4 are illustrative of the
`types of icons that can be provided on the iconic mode
`screen. FIG. 4 does not necessarily represent
`the ideal
`selection or placement of icons. Human-factors testing may
`
`icons that are not present, for example to reduce clutter,
`whenthe touch screen is not in the activated state. Existing
`icons could also be removed or rearranged, although to
`avoid confusion, this could be done only to replace icons not
`useful in the activated state, such as icon 414 of FIG.4, with
`other icons that are most useful when activated, such as
`icons similar to icons 420 and 422.
`
`[0059] There are many possible alternative mechanisms
`for the user to signal the activation of touch screen icons. In
`the simplest case, the icons are either always inactive or
`always active for tapping or motion commands. The corner
`tapping and scrolling region icons 302, 304, and 306 of
`FIGS. 3 and 402, 404, 406, and 410 of FIG. 4 are examples
`
`19
`
`19
`
`

`

`US 2005/0024341 Al
`
`Feb. 3, 2005
`
`of icons that are active at all times. The logo icon 308 of
`FIG. 3 and system status icons 414 of FIG. 4 are examples
`of iconsthat are inactive at all times. For simple touch screen
`images like that of FIG. 3, all icons may fall into these
`simple categories and no overt activation mechanism is
`needed. For more elaborate touch screen imageslike that of
`FIGS. 4 and 5, an overt activation mechanism is necessary
`for those icons that must respond to touch but cannot
`reasonably be made touch sensitive at all times.
`
`In an illustrative embodiment, a key on the main
`[0060]
`keyboard 104 of computer system 100 of FIG. 1 is desig-
`nated as the touch screen activation key. FIG. 6A illustrates
`a portion 600 of the keyboard featuring several keys. Most
`keys of the keyboard, such as letter “Z” key 602, have
`preassigned functions that do not overlap well with touch
`screen activation. Even the existing shifting keys such as
`shift key 604, control key 606, and Alt key 614 are not
`suitable because they are often pressed in conjunction with
`mouseclicks in application software for features such as
`extending selections; hence, it is desirable for the user to be
`able to tap normally on the touch sensor to simulate a mouse
`click while these shifting keys are pressed.
`
`[0061] Function or “Fn” key 608 is common on the
`keyboardsof portable computers. This key, when held down,
`changesthe interpretations of various other keys to perform
`special control functions. For example,
`in one portable
`computer, the arrow keys change to screen brightness con-
`trols, certain letter keys change to become a numeric key-
`pad, and various other keys change to control the external
`video and various other functions. The alternate “Fn” func-
`tions of the various keys are often indicated by blue writing
`next to the white writing indicating the primary function of
`a key. Because the “Fn” key is often absent on desktop
`computers, software typic