(19) United States
`(12) Patent Application Publication (10) Pub. No.: US 2002/0135615 A1
`Lang
`(43) Pub. Date:
`Sep. 26, 2002
`
`US 2002O135615A1
`
`(54) OVERLAID DISPLAY FOR ELECTRONIC
`DEVICES
`
`(75) Inventor: Eric G. Lang, Redmond, WA (US)
`
`Correspondence Address:
`Homer L. Knearl
`Merchant & Gould P.C.
`P.O. Box 2903
`Minneapolis, MN 55402-0903 (US)
`
`(73) Assignee: Microsoft Corporation
`
`(21) Appl. No.:
`
`09/773,971
`
`(22) Filed:
`
`Jan. 31, 2001
`
`Publication Classification
`(51) Int. Cl." ....................................................... G06F 3/00
`(52) U.S. Cl. ............................................ 345/764; 345/766
`(57)
`ABSTRACT
`An interface System and method for inputting control Signals
`to an electronic device with a display and at least one input
`element. An application module is coupled with the input
`element and performs at least one task in response to the
`activation signal. The application module is additionally
`coupled to an information module and a control module. The
`information module receives at least one information image
`from the application module, and the control module
`receives at least one control image from the application
`module. Furthermore, the control image is associated with
`the input element. A rendering module coupled with the
`information module and the control module is used to create
`a compound image. The compound image created is a
`combination of the content image and control image Such
`that the content image appears in front of the control image.
`
`102
`
`108
`
`104
`
`12
`
`106
`
`
`
`116
`
`110
`
`110
`
`Apple Inc.
`Exhibit 1013
`Page 001
`
`

`

`Patent Application Publication Sep. 26, 2002. Sheet 1 of 13
`
`US 2002/0135615 A1
`
`112
`
`114
`
`--
`
`102
`
`108
`
`104
`
`O6
`
`2
`
`O
`
`N 120
`
`11 O
`
`11 O
`
`Fig. 1
`
`Apple Inc.
`Exhibit 1013
`Page 002
`
`

`

`Patent Application Publication Sep. 26, 2002 Sheet 2 of 13
`
`US 2002/0135615 A1
`
`MEMORY
`
`204
`
`re
`
`DMA
`CONTROLLER
`
`
`
`214
`
`
`
`
`
`INPUT
`ELEMENTS
`
`I/O PORTS
`
`DISPLAY
`
`210
`
`208
`
`Fig. 2
`
`Apple Inc.
`Exhibit 1013
`Page 003
`
`

`

`Patent Application Publication Sep. 26, 2002 Sheet 3 of 13
`
`US 2002/0135615 A1
`
`Stock Quotes
`
`Symbol
`
`SINDU
`
`Last
`
`10,940.530
`
`SCOMPX
`
`3,940.350
`
`336
`
`
`
`
`
`
`
`
`
`
`
`338
`
`
`
`
`
`SINX
`
`CSCO
`
`MSFT
`
`
`
`
`
`
`
`
`
`
`
`1,394.460
`
`109.500
`
`97.875
`
`184.000
`
`Apple Inc.
`Exhibit 1013
`Page 004
`
`

`

`Patent Application Publication
`
`US 2002/0135615 A1
`
`312
`
`338
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`336
`
`Fig. 3B
`
`Apple Inc.
`Exhibit 1013
`Page 005
`
`

`

`Patent Application Publication
`
`Sep. 26, 2002. Sheet 5 of 13
`
`US 2002/0135615 A1
`
`
`
`
`
`
`
`
`
`
`
`
`
`338
`
`x X x X x X x X xx
`316
`
`Fig. 3C
`
`Apple Inc.
`Exhibit 1013
`Page 006
`
`

`

`Patent Application Publication Sep. 26, 2002 Sheet 6 of 13
`
`US 2002/0135615 A1
`
`
`
`402
`
`Generate Control
`SCee
`
`s
`ASSOCiate tasks
`with input
`elements
`
`404
`
`4O6
`
`Generate
`information Screen
`
`
`
`Combine information
`SCreen and Control Screen
`into composite Screen
`
`408
`
`Display
`composite screen
`
`410
`
`Receive activation
`signals from input
`elements
`
`412
`
`
`
`Perform taSKS
`associated with
`input elements
`
`414
`
`Fig. 4
`
`End
`
`Apple Inc.
`Exhibit 1013
`Page 007
`
`

`

`Patent Application Publication Sep. 26, 2002 Sheet 7 of 13
`
`US 2002/0135615 A1
`
`516
`
`Display
`Element
`
`Application
`Module
`508
`
`
`
`Rendering
`Module
`
`504
`
`:
`
`
`
`Information
`Module
`
`
`
`Control
`Module
`
`Apple Inc.
`Exhibit 1013
`Page 008
`
`

`

`Patent Application Publication Sep. 26, 2002 Sheet 8 of 13
`
`US 2002/0135615 A1
`
`Display
`Element
`
`to
`
`516
`
`514
`
`;
`
`Operating
`System
`508
`
`
`
`y
`
`Rendering
`Module
`
`512
`
`510
`
`802 N1
`
`
`
`information
`Module
`
`Control
`Module
`
`Driver
`
`520
`
`input
`Element
`
`518
`
`Application
`Module
`
`:
`
`-------------------------------------------
`
`506 :
`
`504
`
`Fig. 6
`
`Apple Inc.
`Exhibit 1013
`Page 009
`
`

`

`Patent Application Publication Sep. 26, 2002 Sheet 9 of 13
`
`US 2002/0135615 A1
`
`
`
`338
`
`Apple Inc.
`Exhibit 1013
`Page 010
`
`

`

`Patent Application Publication Sep. 26, 2002 Sheet 10 of 13
`
`US 2002/0135615 A1
`
`
`
`312
`
`338
`
`Fig. 8
`
`Apple Inc.
`Exhibit 1013
`Page 011
`
`

`

`Patent Application Publication Sep. 26, 2002 Sheet 11 of 13
`
`US 2002/0135615 A1
`
`
`
`338
`
`708
`
`336
`
`Apple Inc.
`Exhibit 1013
`Page 012
`
`

`

`Patent Application Publication Sep. 26, 2002 Sheet 12 of 13
`
`US 2002/0135615 A1
`
`340
`
`
`
`332
`
`sco
`
`Sar,
`dea buddy - I'll buy
`&
`
`338
`
`336
`
`Apple Inc.
`Exhibit 1013
`Page 013
`
`

`

`Patent Application Publication
`
`Sep. 26, 2002. Sheet 13 of 13
`
`US 2002/0135615 A1
`
`O Bean D
`
`
`
`Display information
`Screen in foreground
`
`Display control
`screen in background
`
`1106
`
`LOad a
`Character Set
`
`Divide character set
`into character subsets
`
`1108
`
`
`
`
`
`Display control
`images representing
`the character subsets
`
`1110
`
`Receive Selection
`Signal for a character
`Subset
`
`1112
`
`
`
`ls an
`individual character
`Selected?
`
`Display selected
`character
`
`
`
`Update character set
`to selected subset
`
`Select another
`character?
`
`Yes
`
`1 120
`
`Fig. 11
`
`Apple Inc.
`Exhibit 1013
`Page 014
`
`

`

`US 2002/0135615 A1
`
`Sep. 26, 2002
`
`OVERLAID DISPLAY FOR ELECTRONIC
`DEVICES
`
`TECHNICAL FIELD
`0001. The present invention relates to an interface for
`electronic devices, and more Specifically to a graphical
`interface Showing both an information Screen and a control
`Screen in an overlapping manner.
`
`BACKGROUND OF THE INVENTION
`0002 Small computing devices, such as personal digital
`assistants (PDAS) and Smart watches, typically have a lim
`ited Surface area on which to provide a display Screen and
`user input elements. Because of this Spatial constraint, input
`hardware, Such as push buttons, knobs, and joysticks, are
`often assigned multiple functions in an effort to decrease the
`number of input elements needed on the device. For
`example, the same push button may be used to Select menu
`options, enter data values, and maneuver a cursor during the
`device's operation. Generally, by assigning multiple func
`tions to the input elements, the number of input elements on
`the device is decreased and more room is made available for
`a display Screen.
`0003. One drawback of assigning multiple functions to
`input elements is that users may be required to remember
`how the functions of each input element change during the
`coarse of device operation. A Solution to this is to devote a
`region of the display to remind the user what functions are
`currently assigned to input elements. Accordingly, the dis
`play may include, for example, a bottom line Stating, “Press
`F2 to save, F3 to exit.” Devoting a region of the display to
`list input element assignments, however, decreases the
`amount of room available on the display for non-control
`information.
`0004 Another conventional method of increasing the
`display size of Small electronic devices is to miniaturize the
`input hardware So that leSS Surface area is taken up by input
`elements. Miniaturizing input hardware, however, reduces
`their handineSS and often makes portable computing devices
`awkward to use.
`0005 Some portable computing devices utilize touch
`Sensitive displays for both outputting information and
`receiving user input. In Such a configuration, the display is
`typically Separated into an output region and a touch
`Sensitive input region. The output region of the display
`provides information to the user while the input region
`typically includes virtual input elements, Such as radio
`buttons and Slide-bars, for receiving user input. A touch
`Sensitive display also allows for Virtual input elements to be
`added and removed according to the requirements of various
`device applications being executed.
`0006 Although conventional touch-sensitive displays
`may offer a more flexible input interface for Smaller elec
`tronic devices, there still exists a tradeoff between the
`amount of display area devoted to outputting information to
`the user and the amount of display area devoted to receiving
`user input. For example, increasing the input area may
`facilitate input entry, but this also leaves less room on the
`display to output information. Thus, conventional user inter
`face techniques may not provide a large enough area for both
`outputting information and receiving user input in Small
`devices.
`
`SUMMARY OF THE INVENTION
`0007. In accordance with the present invention, the above
`and other problems are Solved by combining informational
`images and input control images in a display Such that both
`may occupy Substantially the entire display area Simulta
`neously. Thus, increasing the size of the input control images
`does not diminish the size of the informational images, and
`vise Versa.
`0008 Briefly stated, the present invention involves a user
`interface for inputting control Signals to an electronic device
`having a display and at least one input element. When
`implemented as a method, the method may include the acts
`of displaying an information Screen in the display fore
`ground and displaying at least one control image in the
`display background Such that the control image appears
`behind the information Screen. Moreover, the control image
`is associated with the input element and may indicate a task
`to be performed by the electronic device when the input
`element is activated. When the input element is activated, an
`activation Signal is received and the activation of the input
`element is detected.
`0009. The invention may also be implemented as an
`article of manufacture Such as a computer program product
`or computer readable media. The computer program product
`may be a computer Storage media readable by a computer
`System and encoding a computer program of instructions for
`executing the above computer process. The computer pro
`gram product may also be a propagated Signal on a carrier
`readable by a computing System and encoding a computer
`program of instructions for executing the above computer
`proceSS.
`0010 Another aspect of the present invention is an inter
`face System for inputting control Signals into an electronic
`device. When implemented as an interface System, the
`interface System includes at least one input element adapted
`to provide an activation Signal when the input element is
`activated. An application module is also coupled with the
`input element and performs at least one task in response to
`the activation signal. The application module is additionally
`coupled to an information module and a control module. The
`information module receives at least one information image
`from the application module, and the control module
`receives at least one control image from the application
`module. Furthermore, the control image is associated with
`the input element. A rendering module coupled with the
`information module and the control module is used to create
`a compound image. The compound image created is a
`combination of the content image and control image Such
`that the content image appears in front of the control image.
`The interface System also includes a display element
`coupled with the rendering module for displaying the com
`pound image.
`0011 Yet another aspect of the present invention is a
`method for inputting characters to an electronic device. The
`electronic device includes a graphical user interface with a
`display and a plurality of input elements. The method
`includes a display operation for displaying an information
`Screen in a display foreground. Another display operation
`for displaying a control Screen in a display background, with
`the display background appearing behind the display fore
`ground is also performed. A load operation for loading a
`character Set is performed. The character Set includes a
`
`Apple Inc.
`Exhibit 1013
`Page 015
`
`

`

`US 2002/0135615 A1
`
`Sep. 26, 2002
`
`plurality of individual characters. A divide operation for
`dividing the character Set into character Subsets is per
`formed. The character Subsets are represented in the control
`Screen during a resenting operation. A receiving operation
`receives a Selection signal for one of the character Subsets.
`The range of the Selectable character Set is narrowed to the
`Selected character Subset during a narrowing operation. The
`dividing, representing, receiving, and narrowing operations
`are repeated until a Selection of one of the individual
`characters is made.
`0012. These and various other features as well as advan
`tages, which characterize the present invention, will be
`apparent from a reading of the following detailed description
`and a review of the associated drawings.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`0013 FIG. 1 shows an exemplary electronic device
`embodying the present invention.
`0.014
`FIG. 2 shows a simplified representation of a
`device architecture for implementing the present invention.
`0.015
`FIG. 3A shows a smart watch device embodying
`the present invention with exemplary information Screen
`COntentS.
`0016 FIG. 3B shows a smart watch device embodying
`the present invention with exemplary control Screen con
`tentS.
`0017 FIG. 3C shows a smart watch device embodying
`the present invention with exemplary composite Screen
`COntentS.
`0.018
`FIG. 4 shows an operational flow diagram of the
`Steps taken for inputting control Signals to an electronic
`device as contemplated by the present invention.
`0.019
`FIG. 5 shows a system embodying the present
`invention.
`0020 FIG. 6 shows another system embodying the
`present invention.
`0021
`FIG. 7 shows an exemplary control screen for
`another embodiment of the present invention.
`0022 FIG. 8 shows an exemplary control screen after a
`Selection of a character Subset is made from FIG. 7.
`0023 FIG. 9 shows an exemplary composite screen after
`a selection of a character Subset is made from FIG. 8.
`0024 FIG. 10 shows an exemplary composite screen for
`another embodiment of the present invention.
`0.025
`FIG. 11 shows an operational flow diagram of the
`Steps taken for inputting characters to an electronic device as
`contemplated by the present invention.
`
`DETAILED DESCRIPTION OF THE
`INVENTION
`0026. It is contemplated that the present invention is
`utilized in electronic devices with graphical user interfaces,
`and preferably in portable computer-based devices, Such as
`a personal digital assistants (PDAS), Smart watches, mobile
`telephones, and the like. The invention is described in detail
`below with reference to the figures. When referring to the
`
`figures, like Structures and elements shown throughout are
`indicated with like reference numerals.
`0027. In FIG. 1, an exemplary electronic device 102
`embodying the present invention is shown. The electronic
`device 102 includes a housing 104 containing the various
`components of the device 102. The housing 104 is made
`from a durable material, Such as a metallic alloy or a hard
`plastic, capable of withstanding the rougher treatment asso
`ciated with portable devices. The device 102 may also
`include a protective case or cover (not shown) to further
`prevent damage. Moreover, a strap 120 or belt clip (not
`shown) may be provided to hold the portable device 102
`proximate the user.
`0028. The device 102 may include one or more input
`elements 110 mounted on the housing 104. The input
`elements 110 provide activation signals to the device 102
`which are responsive to user interaction. Thus, the input
`elements allow a user to control the device 102 by selecting
`various tasks during different operating Stages of the device
`102. It is contemplated that several types of input elements
`110 may be used in conjunction with the present invention.
`The input elements 110 used may include, but are not limited
`to, push button Switches, rocker Switches, joysticks, rotary
`dials, Slide bars, and touch-Sensitive displayS.
`0029. The device 102 has a communication port 112 for
`communicating with other electrical devices. The commu
`nication port 112 may carry out wire based communications
`and/or wireleSS communications. Various communication
`protocols may be Supported by the communication port 116,
`including HyperText Transfer Protocol (HTTP), Post Office
`Protocol (POP), Time Division Multiple Access (TDMA),
`Code Division Multiple Access (CDMA), and Wireless
`Application Protocol (WAP). It should be noted that the
`protocols listed above are provided as examples only; it is
`contemplated that many other protocols known by those
`skilled in the art may be supported by the Smart watch 102.
`In one embodiment of the present invention, the Smart watch
`102 is part of a wireless piconet, such as a BLUETOOTH
`(TM) WAP. BLUETOOTH is a Trademark owned by Tele
`fonaktiebolaget LM Ericsson.
`0030. An expansion slot 114 on the device 102 allows for
`other equipment to be coupled with the device 102. For
`example, an external storage media (not shown), Such as a
`memory cartridge, magnetic disk drive, or optical disc drive
`may be coupled to the device 102 through the expansion slot
`114. The expansion slot 114 may also be used to connect the
`device 102 to other peripherals, Such as a printer, a Scanner,
`and a digital camera (not shown).
`0031. The electronic device 102 includes a speaker 116
`and a microphone 118. The speaker 116 can be used to play
`recorded music, provide auditory alarms, and produce other
`sound output. The microphone 118 can be used to detect
`Sound for recording, pick-up voice commands, and carry out
`telephone communications.
`0032. A display 106 on the front face of the electronic
`device 102 is used to display informational images and
`control images in accordance with the present invention. The
`display 106 is preferably a liquid crystal display (LCD),
`however, other types of displayS, Such as a cathode ray tube
`(CRT), may be used. Furthermore, the display 106 may be
`a monochrome, gray Scale, or color display. In one embodi
`
`Apple Inc.
`Exhibit 1013
`Page 016
`
`

`

`US 2002/0135615 A1
`
`Sep. 26, 2002
`
`ment of the invention, the display 106 includes touch
`Sensitive input elements which provide activation signals to
`the device 102 when the display 106 is contacted by the user.
`A stylus 108 or other pointing device can be used in
`conjunction with a touch-Sensitive display 106 to activate a
`Small region of the touch-Sensitive Screen.
`0.033 AS discussed in greater detail below, the present
`invention blends an information Screen and a control Screen
`in an overlapping fashion Such that both Screens are dis
`played in the same display region Simultaneously. The
`information and control Screens are combined using graphi
`cal blending techniques Such as alpha blending, Simulated
`alpha blending, and XORing. By doing So, the amount of
`display Space available to show information images is not
`dependent on the display area occupied by control images.
`Thus, the user interface of the present invention is capable
`of utilizing substantially all of the display area for both
`displaying information and receiving user input.
`0034). With reference now to FIG. 2, a simplified repre
`Sentation of the device architecture for implementing the
`present invention is shown. The electronic device 102
`includes a central processing unit (CPU) 202 which is
`primarily responsible for carrying out arithmetic, logic, and
`control operations. The CPU 202 may include a floating
`point unit (FPU) and/or a co-processor (not shown) for
`accelerated graphics performance. Additionally, the CPU
`202 may be a general purpose processor, a digital Signal
`processor (DSP), or other state machine circuit.
`0.035 A memory unit 204 for storage of data and program
`code is coupled with the CPU 202. The memory unit 204
`may include a memory cache, random access memory
`(RAM), video RAM (VRAM), and read only memory
`(ROM). In addition, the memory unit 204 encompasses mass
`Storage media, Such as magnetic and optical memory media.
`0036) The CPU202 also communicates with input/output
`(I/O) ports 206 which receive and transmit data from and to
`the outside environment. For example, the I/O ports 206
`may connect the CPU 202 with a display 206, input elements
`210, and a network 212. The CPU 202 may access the I/O
`ports 206 as either memory mapped I/O Space or as Sepa
`rately mapped I/O space. In addition, the I/O ports 206 may
`also be configured to Support interrupt driven CPU access.
`0037. The device 102 can include a dynamic memory
`access (DMA) controller 214 which enables the I/O ports
`206 to read and write data from and to the memory unit 204
`without involving the CPU 202. The DMA controller 214 is
`especially useful when bit-mapped images of the display
`208 are stored in the memory unit 204. The DMA controller
`214 allows the display 208 to quickly read the stored
`bit-mapped images without slowing down CPU perfor
`CC.
`0.038 According to one embodiment of the present inven
`tion, the memory unit 204 contains dedicated Space for
`Storing an information Screen, a control Screen, and a
`composite Screen. AS used herein, a “Screen” is a digital
`representation of the display content. The amount of
`memory Space required to Store a Screen is typically depen
`dent on the display resolution and color depth of the Screen.
`For example, a high resolution Screen display generally
`requires more memory Space to Store images than a low
`resolution Screen display. In one embodiment of the present
`
`invention, the control Screen is of lower resolution and color
`depth than the information Screen and the composite Screen.
`0039. A computing device, such as electronic device 102,
`typically includes at least Some form of computer-readable
`media. Computer readable media can be any available media
`that can be accessed by the electronic device 102. By way
`of example, and not limitation, computer-readable media
`might comprise computer Storage media and communication
`media.
`0040 Computer storage media includes volatile and non
`Volatile, removable and non-removable media implemented
`in any method or technology for Storage of information Such
`as computer readable instructions, data Structures, program
`modules or other data. Computer Storage media includes, but
`is not limited to, RAM, ROM, EEPROM, flash memory or
`other memory technology, CD-ROM, digital versatile disks
`(DVD) or other optical storage, magnetic cassettes, mag
`netic tape, magnetic disk Storage or other magnetic Storage
`devices, or any other medium that can be used to Store the
`desired information and that can be accessed by the com
`puting system 200.
`0041 Communication media typically embodies com
`puter-readable instructions, data Structures, program mod
`ules or other data in a modulated data Signal Such as a carrier
`wave or other transport mechanism and includes any infor
`mation delivery media. The term “modulated data signal”
`means a signal that has one or more of its characteristics Set
`or changed in Such a manner as to encode information in the
`Signal. By way of example, and not limitation, communi
`cation media includes wired media Such as a wired network
`or direct-wired connection, and wireleSS media Such as
`acoustic, RF, infrared, and other wireleSS media. Combina
`tions of any of the above should also be included within the
`Scope of computer-readable media. Computer-readable
`media may also be referred to as computer program product.
`0042. In FIG. 3A, exemplary information screen con
`tents in a Smart watch device 304 are shown. AS introduced
`above, the Smart watch 304 may include push buttons 332,
`334,336 and 338, a strap 308, and a display 310. The display
`310 may or may not be a touch-sensitive display. Although
`the information screen 302 is shown providing investment
`data, generally any form of display output may be displayed
`in the information screen 302. Thus, the information Screen
`302 may include text images, graphics images, Video
`images, or a combination thereof. AS used herein, an “infor
`mation image” referS collectively to the various informa
`tional objects contained in the information screen 302.
`0043. In FIG. 3B, the smart watch device 304 is shown
`with exemplary control Screen contents, which are repre
`sented with crosshatched lines. The control Screen 312
`includes one or more control images 314, 316, 318, and 320
`which, in general, convey Symbolic representations of Vari
`ous tasks which the user can Select. For example, control
`image 316 may indicate that the information Screen will
`Scroll down if an input element associated with the control
`image 316 is activated. Similarly, control image 318 may
`indicate execution of a Scroll up task, and control images
`320 and 322 may indicate execution of a play music task and
`a stop music task, respectively. It is contemplated that the
`control images used to indicate tasks are simple, low
`resolution images with only a few colors. The control Screen
`
`Apple Inc.
`Exhibit 1013
`Page 017
`
`

`

`US 2002/0135615 A1
`
`Sep. 26, 2002
`
`312 may also contain dividing lines 322 and other images to
`help the user distinguish various regions 324, 326, 328, and
`330 of the display.
`0044) In accordance with one embodiment of the present
`invention, control images 314, 316, 318, and 320 are asso
`ciated with input elements. A control image is associated
`with an input element by positioning the control image in the
`display 310 proximate the input element. For example,
`control images 314, 316, 318, and 320 are associated with
`input elements 332,334,336 and 338, respectively. Thus, an
`in or down stroke of push button 332 causes the information
`screen 302 to scroll up. In another embodiment button 332
`might have multiple Strokes Such as a Stroke in each of four
`directions. In this embodiment only one button would be
`required to activate each of the four control images 314,316,
`318 and 320 as each control image would be associated with
`a stroke direction. If a touch-Sensitive display is present,
`various display regions may be used as input elements. Thus,
`control images 314, 316, 318, and 320 may alternatively be
`asSociated with touch-Sensitive display regions 324, 326,
`328, and 330, respectively.
`0045. In FIG. 3C, the smart watch device 304 is shown
`with exemplary composite Screen contents in the display
`310. The composite screen 340 is a combined image of both
`the information screen 302 (shown in solid lines) and the
`control screen 312 (shown in cross hatched lines). Since the
`display 310 is utilized to present both the information screen
`302 and the control screen 312 in the same physical location,
`a user interface with a relatively large input area may be
`achieved without compromising the amount of information
`presented to the user. Thus, large control imageS can be
`generated in the display 310 for easy user interaction.
`Additionally, the control imageS can be created, modified, or
`deleted according to the input requirements of the Software
`being executed in the device 304.
`0046 Preferably, the information screen 302 and the
`control Screen 312 are combined Such that the information
`Screen 302 appears to be in the display foreground and the
`control Screen appears to be in the display background.
`Combining the information screen 302 and the control
`Screen 312 in Such an overlapping or watermark fashion may
`be achieved using Software, hardware, or a combination of
`both. For example, the information screen 302 may be
`Superimposed over one or more control imageS 314, 316,
`318, and 320 using alpha blending, simulated alpha blend
`ing, or XORing techniques.
`0047. In FIG. 4, an operational flow diagram of the acts
`taken for inputting control Signals to an electronic device as
`contemplated by one embodiment of the present invention
`are shown. The logical operations of the various embodi
`ments of the present invention are implemented (1) as a
`Sequence of computer implemented Steps or program mod
`ules running on a computing System and/or (2) as intercon
`nected machine logic circuits or circuit modules within the
`computing System. The implementation is a matter of choice
`dependent on the performance requirements of the comput
`ing System implementing the invention. Accordingly, the
`logical operations making up the embodiments of the
`present invention described herein are referred to variously
`as operations, Structural devices, acts or modules. It will be
`recognized by one skilled in the art that these operations,
`Structural devices, acts and modules may be implemented in
`
`Software, in firmware, in Special purpose digital logic, and
`any combination thereof without deviating from the Spirit
`and Scope of the present invention as recited within the
`claims attached hereto.
`0048. A control screen operation 402 generates one or
`more control images. AS previously mentioned, it is con
`templated that the control Screen resides in the device
`memory 204 (see FIG. 2) and is bit-mapped to the display.
`In one embodiment of the invention, the control Screen is
`generated and modified by an application program display
`ing information in the display. If more than one application
`makes use of the display, then each application may access
`and modify the control Screen according to the display area
`taken up by that application. AS described below, the control
`Screen can be generated and modified through an operating
`environment of the electronic device 102 or an application
`program interface (API).
`0049. When the control screen is generated, control
`images are associated with input elements by Virtue of their
`position on the display. Thus, placing a control image next
`to an input element associates the control image with the
`input element. If a touch-Sensitive display is used, associ
`ating the control image is accomplished by checking
`whether a touch coordinate returned by the touch-Sensitive
`display falls within the display area occupied by the control
`Image.
`0050 Associate operation 404 assigns input elements to
`tasks. AS used herein, tasks refer to program code that is
`executed when an input element is activated. A task may
`include a single instruction code, a Series of instruction
`codes, or an entire program. A task may be associated with
`an input element by executing or branching to the task when
`an activation Signal is received from the input element.
`Alternatively, the task may be executed as part of an
`interrupt Service routine initiated when an activation Signal
`from the input element is received.
`0051. An information screen operation 406 generates the
`text images, graphical images and Video imageS. The infor
`mation Screen, like the control Screen, may be Stored in the
`device memory 204 (see FIG. 2) and bit-mapped to the
`display. AS mentioned previously, the information Screen
`can contain Such elements as text images, graphical images,
`and Video images.
`0052 Composite screen operation 408 combines the
`information Screen and the control Screen to build a com
`posite Screen. This operation may utilize known image
`manipulation techniqueS Such as alpha blending, Simulated
`alpha blending, and XORing. For example, a transparency
`mask or “alpha channel” may be specified for both the
`information Screen and the control Screen. Pixel values in
`each Screen are then multiplied by their respective alpha
`channel values. The information Screen and the control
`Screen are then overlaid by adding corresponding pixel
`locations in each Screen, with the result Stored in the
`composite Screen. By adjusting alpha channel values, one
`Screen may be brought forward while the other Screen can
`appear to fall to the background. Preferably, the information
`Screen and the control Screen are blended Such that the
`control Screen appears behind the information Screen in an
`overlapping or watermark fashion. However in an alterna
`tive embodiment, the control Screen might be in the fore
`ground and overlay an information Screen in the back
`
`Apple Inc.
`Exhibit 1013
`Page 018
`
`

`

`US 2002/0135615 A1
`
`Sep. 26, 2002
`
`ground. The composite Screen may be generated or built by
`dedicated hardware in the electronic device or by Software
`executed in the CPU202 (see FIG. 2). The composite screen
`may also be Stored in reserved memory.
`0.053
`Display operation 410 displays the composite
`screen is displayed in the display 106 (FIG. 1). In one
`embodiment of the present invention, a display driver con
`tinuously updates the display Screen by accessing the device
`memory and activating display pixels according to compos
`ite Screen data. Furthermore, access to the composite Screen
`may be carried out through the DMA controller 214 (see
`FIG. 2).
`0.054
`Activation signals from one or more input elements
`are received