United States Patent (19)
`Cullen et al.
`
`III USOO5732230A
`5,732,230
`Mar. 24, 1998
`
`Patent Number:
`11
`45 Date of Patent:
`
`54
`
`75
`
`73
`
`21
`22
`(51)
`(52)
`58
`
`56)
`
`COMPUTER USER INTERFACE FOR
`MANIPULATING MAGE FRAGMENTS
`USING DRAG, DROP AND MERGE
`OPERATIONS
`
`Inventors: John F. Cullen, Redwood City; Mark
`Peairs; Peter E. Hart, both of Menlo
`Park, all of Calif.
`Assignees: Richo Company Ltd., Tokyo, Japan;
`Richo Corporation, Menlo Park, Calif.
`
`Appl. No.: 446,196
`Filed:
`May 19, 1995
`Int. Cl. ... G06F 3/00
`U.S. Cl. .......................... 395/339; 395/135; 358/450
`Field of Search ................................... 395/155, 157,
`395/159, 161, 135,339, 133, 138; 358/450
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`6/1991 Cok............................................. 382/1
`5,022,085
`8/1992 Ise et al. ...
`... 38.2/41
`5,140,647
`5,185,808 2/1993 Cok ............................................ 382/1
`
`5,473,737 12/1995 Harper .................................... 39S/31
`5,611,033
`3/1997 Pilleloud et al. ....................... 395/135
`FOREIGN PATENT DOCUMENTS
`0235902
`1/1987 European Pat. Off. .
`4-235464 8/1992 Japan.
`WO 93/12501
`6/1993 WIPO :
`Primary Examiner Raymond J. Bayerl
`Assistant Examiner-A. Katbab
`Attorney, Agent, or Firm-Charles J. Kulas; Philip H.
`Albert; Townsend and Townsend and Crew LLP
`57
`ABSTRACT
`A system for manipulating image fragments so that image
`processing devices such as copiers, fax machines and scan
`ners may efficiently process oversize images. The system
`provides a user interface so that when an oversize image is
`scanned in multiple parts to produce multiple image frag
`ments the user can manipulate the image fragments by
`performing drag, drop and merge operations on the image
`fragments. Embodiments of the invention include the use of
`a touch screen or mouse to allow the user to perform the
`operations. Other operations are selectable such as rotating
`an image fragment, zooming in or out on portions of the
`displayed fragments, merging the fragments into an inte
`grated image and storing or printing the image fragments.
`17 Claims, 10 Drawing Sheets
`
`SCANOWERSE MAGE
`INTO TWO OR MORE FRAGMENTS
`
`04
`
`M 00
`
`
`
`
`
`
`
`
`
`
`
`INPUT IMAGE FRAGMENTS
`INTO SYSTEM
`
`DISPLAY FASE
`FRAGMENTS ON SCREEN
`
`SELECT A FRAGMENT
`
`OVE SELECTED
`FRAGMENT
`
`DROP SELECTED
`FRAGMEN AT NEW
`POSITION
`
`
`
`DONE MOVING
`FRAGMENTS
`
`8
`
`20
`
`REGISTER FRAGMENS
`NTO CORRECT
`CORRESPONDENCE
`
`ERGE IMAGE
`FRAGMENTS
`
`
`
`OPUT HE INTEGRATED
`IMAGE
`
`22
`
`APPL-1014
`APPLE INC. / Page 1 of 17
`
`

`

`U.S. Patent
`
`Mar 24, 1998
`
`Sheet 1 of 10
`
`99
`
`
`
`
`
`
`
`APPL-1014
`APPLE INC. / Page 2 of 17
`
`

`

`U.S. Patent
`
`Mar. 24, 1998
`
`Sheet 2 of 10
`
`5,732,230
`
`202
`
`200
`M
`
`SCANNER
`
`
`
`
`
`204
`
`MEMORY
`
`206
`
`DISPLAY
`
`
`
`
`
`
`
`
`
`
`
`
`
`27
`Visis
`N.
`N.
`Y LI I IND
`N
`
`
`
`
`
`
`
`PRINTER
`
`A/G 2
`
`APPL-1014
`APPLE INC. / Page 3 of 17
`
`

`

`U.S. Patent
`
`Mar. 24, 1998
`
`Sheet 3 of 10
`
`5,732,230
`
`/100
`
`ENTER
`
`02
`
`04
`SCAN OVERSIZE MAGE
`INTO TWO OR MORE FRAGMENTS
`INPUT IAGE FRAGMENTS
`ri06
`INTO SYSTEM
`
`OSPLAY MAGE
`FRAGMENTS ON SCREEN
`SELECT A FRAGMEN
`
`08
`
`MOVE SELECTED
`FRAGMENT
`
`DROP SELECTED
`FRAGMENT AT NEW
`POSITION
`
`
`
`
`
`
`
`
`
`DONE MOVING
`FRAGMENTS
`
`YES
`REGISTER FRAGMENTS
`NTO CORRECT
`CORRESPONDENCE
`
`MERGE MAGE
`FRAGMENTS
`
`OUTPUT THE INTEGRATED
`IMAGE
`
`22
`
`EXIT
`
`8
`
`I9
`
`20
`
`A76 3.
`
`APPL-1014
`APPLE INC. / Page 4 of 17
`
`

`

`U.S. Patent
`
`Mar. 24, 1998
`
`Sheet 4 of 10
`
`5,732,230
`
`
`
`sa
`9
`e
`se
`C
`s
`
`Z.
`
`e
`
`2
`
`V
`
`S
`
`APPL-1014
`APPLE INC. / Page 5 of 17
`
`

`

`U.S. Patent
`
`Mar. 24, 1998
`
`Sheet 5 of 10
`
`5,732,230
`
`
`
`s
`O
`e
`se
`2
`s
`m
`2.
`
`s
`
`P
`
`cS
`
`S
`
`APPL-1014
`APPLE INC. / Page 6 of 17
`
`

`

`U.S. Patent
`
`Mar. 24, 1998
`
`Sheet 6 of 10
`
`5,732,230
`
`
`
`
`
`3108 [EOS
`
`9 39/3/
`
`APPL-1014
`APPLE INC. / Page 7 of 17
`
`

`

`U.S. Patent
`
`Mar. 24, 1998
`
`Sheet 7 of 10
`
`5,732,230
`
`
`
`
`
`308 flÓS
`
`3 :
`: E
`0 0 O
`KX Cs
`
`?I O W „H y? º J, N I MI?I S []
`
`Z 39/…/
`
`APPL-1014
`APPLE INC. / Page 8 of 17
`
`

`

`U.S. Patent
`
`Mar. 24, 1998
`
`Sheet 8 of 10
`
`5,732,230
`
`376
`
`PROCESS
`
`SOURCE
`
`378
`
`380
`
`390
`
`392
`
`LAYOUT
`
`OUTPUT
`
`ta
`
`FIG.
`
`2d
`
`t
`Ge
`
`&.
`
`4&- z o<
`
`>
`wv
`2
`
`374
`
`APPL-1014
`APPLEINC./ Page 9 of 17
`
`APPL-1014
`APPLE INC. / Page 9 of 17
`
`

`

`U.S. Patent
`
`Mar. 24, 1998
`
`Sheet 9 of 10
`
`5,732,230
`
`
`
`
`
`cidae. [108
`
`APPL-1014
`APPLE INC. / Page 10 of 17
`
`

`

`U.S. Patent
`
`Mar. 24, 1998
`
`Sheet 10 of 10
`
`5,732,230
`
`
`
`
`
`£108 [108
`
`3? OV di Hºl J, N I HA SIT
`
`¿ya
`
`O/ 39/3/
`
`APPL-1014
`APPLE INC. / Page 11 of 17
`
`

`

`5,732,230
`
`O
`
`15
`
`20
`
`25
`
`30
`
`35
`
`1.
`COMPUTER USER INTERFACE FOR
`MANIPULATING MAGE FRAGMENTS
`USING DRAG, DROP AND MERGE
`OPERATIONS
`Notice Regarding Copyrighted Material
`A portion of the disclosure of this patent document
`contains material which is subject to copyright protection.
`The copyright owner has no objection to the facsimile
`reproduction by anyone of the patent document or the patent
`disclosure as it appears in the Patent and Trademark Office
`file or records, but otherwise reserves all copyright rights
`whatsoever.
`BACKGROUND OF THE INVENTION
`This invention relates generally to a user interface for
`manipulating images and specifically to a computer user
`interface for combining image fragments.
`Today, image processing devices allow images to be
`"captured" by computer systems by, e.g., scanning a printed
`version of an image to obtain a digital representation of the
`image. Also, digital representations of images can be printed
`to generate a hard copy of the image. Examples of image
`processing devices are copiers, fax machines and scanners.
`These systems now use advanced technology to allow a
`human operator to manipulate the captured image by
`reducing, enlarging, adjusting the contrast, resolution or
`color of images, etc. While today's basic image processing
`devices are well-suited to handling standard size images.
`such as an image on an 8.5"X11" sheet of paper, problems
`arise in these devices where an oversize image needs to be
`broken into image fragments in order to capture the image
`into a device. Problems also arise where image fragments
`need to be re-assembled for printing as a single integrated
`image on one or more sheets of paper.
`For example, a problem with copy machines arises when
`it is desired to copy an oversize image, such as a map or
`poster. This is because the configuration of the copy machine
`will usually allow only portions, or fragments, of the over
`size image to be scanned in each pass of the copier's
`scanning mechanism. This means that the human user of the
`copier needs to manually position the oversize image and
`make multiple scans of portions of the map or poster.
`Because the user must visually align the oversize image on
`the copier's platen, often without the aid of any registration
`marks, the user ends up with a hodgepodge collection of
`non-uniform fragments of the oversize image spread out
`over the papers. In the worst case, the user must then
`manually assemble the image fragments by cropping and
`taping together the pages.
`Similarly, fax machines are limited to accepting paper of
`fixed and relatively small dimensions. If an oversize docu
`ment is wider than that allowable by the fax machine the
`document must be broken up into smaller images on smaller
`sheets of paper. The oversize image is then transmitted as
`several pieces to a receiving fax machine. A user at the
`receiving fax machine then goes through a similar process to
`piece together the oversize document's image from the
`multiple fragments of the document.
`Another instance of the shortcomings of today's imaging
`devices to handle oversize documents can be seen in a
`hand-held scanning device. With the hand-held scanner the
`image must be scanned multiple times by having the user
`"swipe" the hand-held scanner over the image. Where a
`document is large, the hand-held scanning operation results
`in many image fragments that need to be pieced together.
`
`2
`Approaches to deal with this problem include the matching
`together of 2 images captured sequentially by the hand
`scanner. However, an approach such as this, described in
`European patent application IPN WO 93/12501, fails to
`provide an adequate way for a user to work with large
`images that have been fragmented into several pieces by the
`scanning process.
`Other approaches for combining images in today's image
`processing devices include, U.S. Pat. No. 5,022,085 to
`David R. Cok. This patent discloses an imaging data mecha
`nism for merging, or compositing, a first image onto a
`second image. Once composited, the Cok invention uses
`special techniques for reducing boundary artifacts caused
`when a first image edge lies on top of a second image.
`Another system disclosed in European patent application
`number 87300562.3 shows compositing a first image onto a
`second image. The invention discloses techniques for detect
`ing the first image's edges so that the compositing operation
`can be made more automatic. Japanese patent JP 4235464
`discloses a system for merging two different images input
`from main and sub-image sensor units. U.S. Pat. No. 5,140,
`647 discloses applying markers to document images and
`then using the markers to register the images.
`The above references do not solve the problems of the
`prior art in handling oversize images efficiently in image
`processing devices such as copiers, fax machines and scan
`ners. Some of the references are related to compositing
`operations where a first image is laid on top of a second
`unrelated image and an attempt is made to make the images
`look as though they are one image by removing border
`artifacts. This differs from the situation where image frag
`ments need to be aligned adjacent to one another so that, for
`example, an image detail such an object or letter that is
`broken up over two image fragments is made to appear
`whole again. All of the references fail to disclose a system
`for efficiently manipulating large numbers of image frag
`ments to create a large composited image.
`Accordingly, it is desirable to provide an image process
`ing device that allows efficient manipulation of image frag
`ments so that oversize images can be handled with greater
`ease and accuracy.
`SUMMARY OF THE INVENTION
`The present invention provides a graphical user interface
`(GUI) to image processing devices. The GUI allows a user
`to perform operations such as selecting, dragging and drop
`ping displayed representations of the image fragments to
`indicate a user preferred alignment for the fragments. The
`use of such an approach greatly simplifies the user's task in
`handling oversize documents with standard image process
`ing devices such as copiers, fax machines and scanners.
`One embodiment of the invention includes a method for
`registering image fragments in a computer system. The
`computer system includes a processor coupled to a user
`input device, scanning device and display screen. The
`method comprises the steps of using the scanning device to
`input two or more image fragments of an original image;
`using the processor to display at least a portion of the first
`image fragment and at least a portion of the second image
`fragment simultaneously on the display screen; accepting
`signals from the user input device to select the first image
`fragment displayed on the display screen; moving the first
`image fragment on the display screen so that a portion of the
`first image fragment is adjacent to and aligned with the
`portion of the second image fragment; and registering, by
`using the processor, the first and second image fragments
`
`45
`
`50
`
`55
`
`65
`
`APPL-1014
`APPLE INC. / Page 12 of 17
`
`

`

`5,732,230
`
`O
`
`15
`
`20
`
`25
`
`30
`
`35
`
`3
`into a combined image that closely approximates at least a
`portion of the original image, wherein the registering step
`uses the adjacency and alignment from the previous step.
`An apparatus for registering image fragments in a com
`puter system is also disclosed. The apparatus comprises a
`processor; a memory coupled to the processor; a user input
`device coupled to the processor; a display screen coupled to
`the processor; a data input device restoring representations
`of two or more image fragments of an original image into the
`memory; display means for displaying at least a portion of
`the first image fragment and at least a portion of the second
`image fragment simultaneously on the display screen; selec
`tion means for accepting signals from the user input device
`to select the first image fragment displayed on the display
`screen; redisplay means for moving the first image fragment
`on the display screen in response to signals from the user
`input device; position determining means generating one or
`more parameters describing the locations of the moved first
`image fragment relative to the second image fragment when
`the moved first image fragment is adjacent to, and aligned
`with, a portion of the second image fragment; and registra
`tion means coupled to the display means for receiving the
`parameters and for registering the first and second image
`fragments into a combined image based on the parameters.
`BRIEF DESCRIPTION OF THE DRAWINGS
`FIG. 1 shows basic subsystems of a computer system
`suitable for use with the present invention;
`FIG. 2 shows a configuration of subsystems for a pre
`ferred embodiment;
`FIG. 3 shows flowchart 100 illustrating a method of the
`present invention.
`FIG. 4 shows a first screen display of the user interface of
`the present invention;
`FIG.S shows a second screen display of the user interface
`of the present invention;
`FIG. 6 shows a third screen display of the user interface
`of the present invention with 6 manipulated image frag
`ments:
`FIG. 7 shows a fourth screen display of the user interface
`of the present invention with merged fragments;
`FIG. 8 shows a fifth screen display of the user interface of
`the present invention;
`FIG. 9 shows a sixth screen display of the user interface
`of the present invention; and
`FIG. 10 shows a seventh screen display of the user
`interface of the present invention.
`DESCRIPTION OF THE PREFERRED
`EMBODMENT
`FIG. 1 shows basic subsystems of a computer system
`suitable for use with the present invention. In FIG. 1.
`computer system 10 includes bus 12 which interconnects
`major subsystems such as central processor 14, system
`memory 16, input/output (I/O) controller 18, an external
`device such as a printer 20, parallel port 22, display screen
`24 via display adapter 26, serial port 28, keyboard 30 and
`fixed disk32. Many other devices can be connected such as
`scanning device 34 connected via external interface 36,
`mouse 38 connected via serial port 28 and touch screen 40
`connected directly. Many other devices or subsystems (not
`shown) may be connected in a similar manner. Also, it is not
`necessary for all of the devices shown in FIG. 1 to be present
`to practice the present invention, as discussed below. The
`
`4
`devices and subsystems may be interconnected in different
`ways from that shown in FIG. 1. The operation of a
`computer system such as that shown in FIG. 1 is readily
`known in the art and is not discussed in detail in this
`application.
`FIG. 2 shows configuration 200 for a preferred embodi
`ment including scanner 202, memory 204. CPU206, display
`208, touch screen 210 and printer 212. Configuration 200
`could implement, for example, a copier machine. Configu
`ration 200 could also be a portion of hardware in a fax
`machine or scanner. The present invention is adaptable for
`use in any system where an oversize image must be scanned
`piecemeal so that fragments of the oversize image are input
`into the system.
`Display 208 can be a liquid crystal display (LCD) screen
`or a cathode ray tube (CRT) screen or other type of display
`screen or panel. The display screen is similar to that in use
`on standard computers such as personal computers or work
`stations employing a CRT screen or monitor. Various forms
`of user input devices may be used with the present invention.
`For example, even though a touch screen is shown in FIG.
`2, a mouse input device that allows a user to move a pointer
`displayed on the display screen in accordance with user hand
`movements is a standard user input device. A mouse usually
`includes one or more buttons on its surface so that the user
`may point to an object on the screen by moving the mouse
`and may select the object, or otherwise activate the object,
`by depressing one or more buttons on the mouse. The touch
`screen allows a user to point to objects on the screen to select
`an object and to move the selected object by pointing to a
`second position on the screen. As discussed below, various
`buttons and controls may be displayed on the screen for
`activation by using the mouse or touch screen.
`FIG. 3 shows flowchart 100 illustrating a method of the
`present invention.
`The flowchart is illustrative of merely the broad logical
`flow of steps to achieve a method of the present invention
`and that steps may be added to, or taken away from, the
`flowchart without departing from the scope of the invention.
`Further, the order of execution of steps in the flowchart may
`be changed without departing from the scope of the inven
`tion. Additional considerations in implementing the method
`described by the flowchart may dictate changes in the
`selection and order of steps.
`In general, the flowcharts in this specification include one
`or more steps performed by software routines executing in
`a computer system such as computer system 1 of FIG.1. The
`routines may be implemented by any means as is known in
`the art. For example, any number of computer programming
`languages, such as "C", Pascal, FORTRAN, assembly
`language, etc.. may be used. Further, various programming
`approaches such as procedural object oriented or artificial
`intelligence techniques may be employed.
`The steps of the flowcharts may be implemented by one
`or more software routines, processes, subroutines. modules,
`etc. Some considerations such as interrupt driven, polled, or
`other implementation schemes may affect the order of steps
`performed by software. A multiprocessing or multitasking
`environment could allow steps to be executed "concur
`rently."
`Flowchart 100 is entered at step 102. At step 104, the
`original oversize image is scanned into two or more image
`fragments. An example of an oversize image is a map.
`Typically, a map is much larger than the 8.5"X11" paper size
`that most copiers, fax machines and scanners accept. While
`some copiers do allow for larger image sizes, copiers that
`
`45
`
`50
`
`55
`
`65
`
`APPL-1014
`APPLE INC. / Page 13 of 17
`
`

`

`5
`acceptimages much larger than 11"X17" are more expensive
`and often require specially trained operators. Note that the
`oversize image may be spread across two or more separate
`pieces of paper. The image fragments may be obtained by
`loading electronic representations of the image fragments
`directly into the system of the present invention. In this case,
`the scanning step 104 is bypassed. Other ways to obtain
`multiple image fragments from an original oversize image
`are possible.
`At step 106 the obtained image fragment data is input into
`the system of the present invention. Usually this step results
`in the image fragment data being stored in the memory or
`other storage medium of the system. In the case where the
`original oversize image is scanned the image fragment data
`is obtained from the scanner and is transferred into the
`memory of the computer system which performs the steps
`described below.
`After step 106 is executed step 108 is performed to
`display the image fragments on the display screen. Step 108
`is performed by processor 14 which controls the selection
`and display of image fragments stored in memory 16 onto
`display screen 24.
`In a preferred embodiment, all of the image fragments are
`displayed on the display screen at one time. In order to
`display all of the multiple image fragments on the display
`screen at the same time, the image fragments are reduced.
`For example, an 8.5"X11" format page might be shown as a
`rectangle with the same aspect ratio with a size on the order
`of 1" to 2" on a side. The need for reducing an image on the
`display is necessary when the screen resolution is not fine
`enough to show all of the fragments at a nominal level of
`detail. In a preferred embodiment, when a page is reduced a
`reduced image of the information printed on each page is
`also displayed on the reduced page so that each page appears
`on the screen with a suggestion of its image. This allows the
`pages to be distinguished easily on the screen.
`Note that the shape of each fragment on the screen
`approximates the shape obtained from the scan or other
`source of image fragments. That is, where the original
`oversize image has been scanned in rectangular regions the
`image fragments are shown on the display as rectangular
`areas having the same relative dimensions as the scanned
`region. For a copy machine, this will typically be an 8.5"x
`11" or 11"x7" page that is shown as an accordingly reduced
`rectangle on the display screen. However, where the scanner
`is a hand-held scanner, the scan "footprint” may vary so that
`the image fragment is a long, narrow band or other irregular
`shape. Regardless of the shape of the scan footprint the
`present invention maintains an approximation to the foot
`print in the display of the image fragment on the screen.
`The display screen should have sufficiently high resolu
`tion to allow the user to identify image fragments by larger,
`gross aspects of the portion of the image on the image
`fragment. It is especially desirable for text to be readable,
`where such text is relatively small in the original oversize
`image. As discussed below, the invention provides for
`"zooming" in on an image fragment to view more of the
`detail of the image portion contained in the fragment.
`Steps 110-116 form a loop that describes the steps a user
`performs to manipulate image fragments to create an inte
`grated image. At step 110, the user selects an image fragment
`on the display screen. Where the user input device is a
`mouse, the user may perform step 110 by moving a pointer
`on the display screen over an image to be selected. The user
`moves the pointer by manipulating a mouse as is known in
`the art. Once the pointer is over the desired image fragment,
`
`35
`
`45
`
`50
`
`55
`
`65
`
`5,732,230
`
`10
`
`15
`
`25
`
`30
`
`6
`the user depresses a button on the mouse to select the image
`fragment. Many other forms of user input may be used to
`perform step 110 and other steps such as 112 and 114. For
`example, a trackball, so-called "point pad,” dataglove, digi
`tizing tablet, etc., may be employed. A preferred embodi
`ment uses a touch screen so that the user may point to or
`touch areas of the display screen directly with their fingers
`or some other object such as a stylus. The use of the touch
`screen eliminates the need for a flat surface to operate the
`mouse so that the system of the present invention may be
`made more compact.
`Next, step 112 is performed so that the user can move the
`selected fragment to a new position on the display screen.
`Where a touch screen is used, the user may point to a new
`position for the fragment or may "drag" the fragment about
`the screen by pointing while moving the position of the
`finger on the screen. Where amouse is the user input device,
`the user may perform step 112 by using the mouse to move
`the pointer on the screen so that the selected fragment is
`dragged along with the pointer. When the user has moved the
`selected fragment to a desired position, the user executes
`step 114 so that the selected fragment is released or
`"dropped" to assume its new position on the display screen.
`Step 116 indicates that steps 110-114 may be performed
`repeatedly, as needed, so that the user may arrange image
`fragments freely on the screen. The purpose of the fragment
`manipulation is to allow the user to easily put the image
`fragments together on the screen and build the original
`oversize image out of the image fragments. It is not neces
`sary for all of the image fragments to be displayed on the
`display screen at one time, although this is preferable to aid
`in the image fragment manipulation. Image fragments may
`be overlaid onto other image fragments or positioned so that
`the edges of different image fragments are adjacent.
`Image fragments can also be transparent. This allows a
`user to view image fragments underlying other image frag
`ments. It also allows the user interface to generate a uniform
`background where fluctuations in the scanning of the image
`fragment produce an undesirable varying background. An
`undesirable varying background occurs, for example, where
`engineering schematics, blueprints or drawings originally
`have a uniform background but whose scanned fragments
`have varying backgrounds because of the different levels of
`gray detection due to the imprecise auto gain control in
`digital scanners. By making the background (varying levels
`of gray) transparent while retaining the foreground infor
`mation containing the lines, symbols, text, etc., the user
`interface of the present invention can generate a uniform
`background that provides the user with more discernible and
`pleasing images.
`Once the user is done manipulating image fragments. the
`loop of steps 110-116 is exited and step 118 is performed.
`Step 118 registers the image fragments into an integrated
`image, according to the current orientation of the image
`fragments on the display screen. In a preferred embodiment,
`step 118 is invoked by user selection. The user is able to
`select registration of the image fragments in the current
`orientation of fragments on the display screen by activating
`a predetermined user interface control, for example, by
`activating a "button" on the screen. The activation of the
`control to register the image fragments causes the processor
`to execute instructions to calculate relative positions for
`each image fragment. Another control is provided by to
`allow the user to "merge” the image fragments into a single
`integrated image made up of multiple image fragments. The
`merge operation uses the relative positions calculated by the
`register operation to achieve the integrated image. Thus, in
`
`APPL-1014
`APPLE INC. / Page 14 of 17
`
`

`

`5,732,230
`
`O
`
`15
`
`20
`
`25
`
`30
`
`35
`
`7
`a preferred embodiment the dual steps of registering and
`merging are required. However, for ease of discussion
`"registering" is generally regarded as including the merging
`operation in this specification.
`A simple way to achieve registration is to use the coor
`dinates of each image fragment in their relative positions on
`the display screen. That is, there is no further automated
`registration of the image fragments beyond the user's own
`positioning. In most cases, this may be sufficient, since, as
`discussed below, the user is provided with various tools to
`aid in accurate alignment, positioning and merging of the
`image fragments to produce an integrated image.
`Where automated registration is desired, the present
`invention allows for, e.g., computerized registration, where
`the processor makes decisions based on a predetermined
`algorithm as to how to align shapes, objects, etc., from one
`edge of an image fragment to an adjacent edge of another
`image fragment. The shapes, objects, etc., are discrete
`pictures broken up across two or more image fragments. For
`example where a line segment extends from the first image
`fragment to a second image fragment and the user has placed
`an edge of the first image fragment cutting through the line
`segments adjacent to an edge of the second image fragment
`that also cuts through the second image fragment's portion
`of the line segment, the processor could execute instructions
`to ensure that the image fragments are "fine tuned" so that
`the split ends of the line segment abut as precisely as
`possible. Other forms of automated registration are possible
`by using special "fiducial.” or registration marks, on each
`image fragment and aligning the image fragments by ensur
`ing that the fiducials exactly overlie each other. Extrapola
`tion of image data to extend features beyond current image
`fragments, or interpolation to derive image data between
`two fragments, could also be employed. Also, "smoothing"
`or other forms of image enhancement could be used to
`improve the integrated image after merging. For example,
`image smoothing or averaging would be useful at the joining
`point of two image fragments where there is not an exact
`match and a noticeable edge, or artifacting, occurs.
`Next, after the image fragments have been registered, step
`119 is performed to merge the image fragments into an
`integrated image. In a preferred embodiment, the step of
`merging the image fragments creates a single digital file
`containing the information in each of the image fragments
`included in the registered image. The file may reside in
`memory or in persistent storage, such as on a magnetic disk.
`The file can be of various formats. For example, the file can
`be a simple bitmap representation of the registered image.
`Another possibility is for the file to contain the image
`fragments as discrete objects along with information about
`the alignment of each image fragment in the registered
`image. Other file formats are possible.
`The integrated, or merged, image is output at step 120. An
`example of outputting the integrated image could be printing
`the integrated image in reduced form on a standard sheet of
`paper, printing the integrated image to full scale on a large
`sheet of paper, electronically transferring the integrated
`image to another device, storing the integrated image, etc.
`After outputting the integrated image, the method described
`by flowchart 100 in FIG. 2 is exited at step 122.
`Note that the steps of flowchart 100 may be selectively
`repeated, for example, to add more image fragments to an
`existing integrated image. An additional image fragment can
`be selected at step 110 after a first integrated image has been
`created as described above. The additional image fragment
`is then moved, dropped, registered and merged, as desired,
`to add the additional image fragment to the existing inte
`grated image.
`
`8
`In a fax machine application, the invention is particularly
`useful at either the receiving or sending fax machines to
`allow a human user to drag, drop and merge, or otherwise
`manipulate, image fragments from multiple sheets of paper.
`For example, at the sending fax machine multiple sheets of
`paper, each containing an image fragment of an original
`image, can be scanned by the fax machine. Prior to sending
`the image fragments to the receiving fax machine, the image
`fragments can be processed as described above in flowchart
`100 to obtain an integrated image. The image fragments can
`then be transmitted along with coordinate information for
`allowing a processor in the receiving fax machine to auto
`matically reproduce the original image.
`The reproduced image can be displayed on a display
`screen for further manipulation by a human operator at the
`receiving fax machine. For example, the operator at the
`receiving fax machine may desire to reduce the received
`image to fit on a single sheet of paper before printing. Or the
`receiving operator may wish to re-align the received image
`over multiple sheets of paper before printing. By allowing
`the receiving operator to access the electronic version of the
`original image as an integrated image, the operator is given
`greater flexibility over the receipt of the image. Likewise,
`the sending operator may decide, for example, to reduce the
`integrated image before sending.
`FIG. 4 shows a screen display of the user interface of the
`present invention. In FIG. 4, screen display 300 includes
`work area 302 that occupies essentially all of the screen
`except for title bar 304. Buttons, such as process buttons
`306, layout buttons 308, output buttons 310 and DONE
`button 312 overlay work area 302 toward the right of the
`screen. Indicators 314 also occupy some space on top of
`work area 302.
`FIG. 5 shows screen display 340 after image fragments
`have been loaded into the system. That is, FIG. 5 shows
`screen display 340 as it would appear after steps 104-108 of
`flowchart 100 of FIG. 3 have been executed. In FIG. 5, six
`rectangular page fragments are arbitrarily arranged as two
`rows and