a2, United States Patent
`US 6,710,791 B1
`(10) Patent No.:
`*Mar. 23, 2004
`(45) Date of Patent:
`Kodamaet al.
`
`US006710791B1
`
`(54) PORTABLE INFORMATION GATHERING
`APPARATUS AND INFORMATION
`MANAGEMENT METHOD
`
`(75)
`
`Inventors: Yoshiyuki Kodama, Nagano-ken (JP);
`Yoshiteru Uchiyama, Nagano-ken (JP)
`
`(73) Assignee: Seiko Epson Corporation, Tokyo (JP)
`
`(*) Notice:
`
`This patent issued on a continued pros-
`ecution application filed under 37 CFR
`1.53(d), and is subject to the twenty year
`patent
`term provisions of 35 U.S.C.
`154(a)(2).
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`US.C. 154(b) by 0 days.
`
`(21) Appl. No.: 08/792,026
`
`(22)
`
`Filed:
`
`Jan. 31, 1997
`
`(30)
`
`Foreign Application Priority Data
`
`Feb. 1, 1996
`
`(IP) voces cece eeesccesseeesseecseseneesensaes 8-016903
`
`Tint, C17 ieee ececececeeeseeseeeneeneeeeeeneenneene G09G 5/00
`(SL)
`(52) US. CMe
`ceecssecccccssssssssseee 345/835; 345/839; 707/7;
`715/537
`(58) Field of Search oo... eee 345/173, 169,
`345/350, 340, 156, 179, 1, 901, 903, 77,
`357, 104, 439, 835, 838, 839; 379/93.23,
`93.17; 707/501, 7, 100, 101, 102; 178/18.03,
`18.04, 19.01; 715/515, 522, 525, 526, 530,
`537
`
`(56)
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`......... 361/679
`5,268,817 A * 12/1993 Miyagawaet al.
`5,401,917 A *
`3/1995 Yoshida et al... 345/179
`5,406,307 A *
`4/1995 Hirayamaetal.
`.......... 345/179
`5,453,761 A *
`9/1995 Tanaka occ 345/173
`5,467,102 A * 11/1995 Kuno etal. wv. 345/1
`
`5/1997 Kikinis et al. 0... 345/156
`5,634,080 A *
`
`9/1997 Huffman et al... 345/173
`5,663,748 A *
`
`9/1997 Hirayama .......... 345/350
`5,673,406 A *
`
`1/1998 Sotomayor .............08 707/501
`5,708,825 A *
`
`4/1998 Rowe et al. we. 395/615
`5,737,599 A *
`.....0.0.. 345/340
`5,745,109 A *
`4/1998 Nakano et al.
`8/1998 Cecchini et al.
`............ 345/357
`5,790,122 A *
`OTHER PUBLICATIONS
`
`“Mastering WordPerfect 5.1 & 5.2 for Windows” Alan
`Simpson, 1993, pp. 74-76 and 250256.*
`
`* cited by examiner
`
`Primary Examiner—Dennis-Doon Chow
`(74) Attorney, Agent, or Firm—Hogan & Hartson LLP
`
`(57)
`
`ABSTRACT
`
`A portable information gathering apparatus and information
`managing method in the same make it possible to capture
`information entered through an input/display tablet, image
`information entered through an image input device and
`voice information entered through a voice input device, and
`other type of information for individual pages, each corre-
`sponding to all information displayed at one time on a
`screen. In response to an instruction given by a user of the
`portable information gathering apparatus, an attach-to-page
`icon setting unit attaches icons to desired pages to manage
`the page information in units of pages. Several of these icons
`are: an icon for managing the information captured in
`respective pages by time-related data such as date and time;
`an icon for specifying a particular page; an icon serving as
`a memo guide; and an icon for managing the information
`captured in the respective pages by the type of information.
`
`5,049,862 A *
`
`9/1991 Dao et al. we 345/179
`
`18 Claims, 17 Drawing Sheets
`
`28
`
`70
`
`
`
`NEW YEAR’S PARTY
`
`AT XXX’s HOME
`
`7:00 PM
`
`101
`
`116b
`
`1
`
`APPLE 1016
`
`APPLE 1016
`
`1
`
`

`

`U.S. Patent
`
`Mar. 23, 2004
`
`Sheet 1 of 17
`
`US 6,710,791 B1
`
`ATTACH-TO—PAGE
`ICON SETTING
`PROCESSOR
`
`NEXT
`PAGE
`BUTTON
`
`PREVIOUS
`PAGE
`BUTTON
`
`CAMERA
`BUTTON
`
`VOICE
`INPUT
`BUTTON
`
`41
`
`BUTTON
`SIGNAL
`PROCESSOR
`
`21
`
`2234
`
`33
`
`35
`
`36
`
`
`
`cpy REALTIME||POWERFLASH
`
`MEMORY
`CLOCK
`SWITCH
`
`2|301#|411
`
`SERIAL
`WA
`LCD
`CCD
`
`
`CONTROLLER CONTROLLER||°/ CONTROLLER
`
`|
`|
`|
`t
`|
`j
`i
`(
`{
`}
`
`'
`I
`'
`4
`'
`!
`!
`'
`’
`J
`'
`
`
`
`|
`1424
`{
`|
`I
`
`‘
`|
`j
`|
`'
`|
`|
`I
`|
`!
`l
`|
`28 | |
`
`MICROPHONE
`‘TABLET||1
`
`SPEAKER Po PHOTODIODE
`Me
`
`t |
`
`422
`
`|
`
`'
`
`|
`!
`t
`\
`(
`
`425
`
`uneIb rewisumrer]|“YY| |
`
`
`
`CCD
`
`AMPLIFIE
`
`31
`
`AMPLIFIER]
`
`32
`
`t
`1}
`|}
`|
`
`
`
`423
`
`INFRARED
`RECEIVER/
`
`RS
`
`2
`
`

`

`U.S. Patent
`
`Sheet 2 of 17
`
`Mar. 23, 2004
`
`US 6,710,791 B1
`
`3
`
`

`

`U.S. Patent
`
`Mar. 23, 2004
`
`Sheet 3 of 17
`
`US 6,710,791 B1
`
`101
`
`103
`
`104
`
`ee
`
`(oy BPD
`A 4 Ay
`Z
`7
`rf Cn ies “ay
`,“a
`ev
`ba “yy
`
`XX
`ARRIVAL
`xx HOUR
`
`105d
`
`106b
`
`105b
`
`FIG. 4
`
`1050 102 tetapie DEPARTURE
`
`105
`
`1064
`
`%
`
`4
`
`

`

`U.S. Patent
`
`Mar. 23, 2004
`
`Sheet 4 of 17
`
`US 6,710,791 B1
`
`HARDWARE
`DEVELOPMENT
`
`MR. TARO SUWA
`
`XX ENGINEERING SECTION
`
`1996. 01.01
`
`14:00
`
`"
`
`FIG. OA
`
`
`
`
`FULL—COLOR PRINTER
`
`¥
`
`99.800
`
`EXPECTED TO BE
`RELEASED ON
`
`1996. 01.01
`
`14:30
`
`"
`
`FIG. 5B
`
`5
`
`

`

`U.S. Patent
`
`Mar. 23,2004
`
`Sheet 5 of 17
`
`US 6,710,791 B1
`
`FIG. 6A
`
`
`Slel7[slo]
`
`[-[_]montH (_[_Joay
`[-T Jxour []IMIN.
`
`oy[2]5]41
`
`=]
`
`6
`
`

`

`[| ]montH ("|Joay
`[-TJxour (_[JMIN.
`
`ot25/4)
`
`FIG. 7
`
`U.S. Patent
`
`Mar. 23, 2004
`
`Sheet 6 of 17
`
`US 6,710,791 B1
`
`28
`
`70
`
`BGuoo
`
`
`103
`
`FIG. 8
`
`7
`
`

`

`U.S. Patent
`
`Mar. 23,2004
`
`Sheet 7 of 17
`
`US 6,710,791 B1
`
`115
`
`28
`
`70
`
`73
`ane ae
`
`
`: stelrials) [&]|Gaze 7s
`
`74,
`
`AUTO-ERASE SET [||montH [|Jay
`From
`
`CONFIRMATION
`REQUIRED LH
`
`117116
`FIG. 9
`
`TEL
`TEL MEMO
`
`118
`
`FIG. 10
`
`8
`
`

`

`U.S. Patent
`
`Mar. 23, 2004
`
`Sheet 8 of 17
`
`US 6,710,791 B1
`
`CREATE STAMP
`
`FIG. L1A
`
`FIG. 11C
`
`"TOUCH REGISTERING LOCATION”
`
`FIG. 11B
`
`"TAKE PICTURE”
`
`123
`
`9
`
`

`

`U.S. Patent
`
`Mar.23, 2004
`
`Sheet 9 of 17
`
`US 6,710,791 B1
`
`Ist
`page
`
`page
`
`2nd
`
`Address indicating 1st page
`Address indicating 2nd page
`Address indicating 3rd page
`Address indicating 4th page
`
`Address indicating next page
`Address indicating previous page
`
`Addresses indicating input dato
`
`Address indicating additional
`
`information
`
`Address indicating next page
`Address indicating previous page
`
`Addresses indicating input data
`
`Address indicating additional
`
`information
`
`Address indicating next page
`Address indicating previous page
`
`Addresses indicating input data
`
`Address indicating additional
`
`information
`
`FIG. 12
`
`
`
`
`
`
`
`
`
`
`Image data
`on 1st page
`Voice data
`on ist page
`Ink data
`on Ist page
`Additional
`information
`on ist page
`
`Image data
`on 2nd page
`Voice data
`on 2nd page
`Ink data
`on 2nd page
`Additional
`information -
`
`on 2nd page
`
`
`
`
`
`
`
`
`
`
`Image data
`on 3rd page
`Voice data
`on Srd page
`Ink data
`on 3rd page
`Additional
`information
`on Jrd page
`
`10
`
`

`

`U.S. Patent
`
`Mar. 23, 2004
`
`Sheet 10 of 17
`
`US 6,710,791 B1
`
`NEW YEAR’S PARTY
`
`AT XXX's HOME
`
`7:00 PM
`
`101
`
`116b
`
`FIG. 18
`
` Additional
`information
`on 4th page
`
`
`
` Ink information
`on 4th page
`
`
`
`
`Code of alarm stamp
`Stamp position: x—coordinate
`Stamp position: y—coordinate
`Alarm set: year
`Alarm set: month
`Alarm set: day
`Alarm set: hour
`Alarm set: minute
`Code of party stamp
`Stamp position: x—coordinate
`Stamp position: y—coordinate
`END
`
`11
`
`

`

`U.S. Patent
`
`Mar. 23, 2004
`
`Sheet 11 of 17
`
`US 6,710,791 B1
`
`Ps? (80, 220)
`
`(30, 220)
`
`Pst
`
`Pxy (319, 239)
`
`12
`
`

`

`
`
`$24 10©|sevecteD STAMP
`N
`OTHER
`FIG. 19
`PROCESSING
`S216
`RECORD INFORMATION
`ON SELECTED STAMP
`
`DISPLAY
`
`
`
`
`
`
`
`U.S. Patent
`
`Mar. 23, 2004
`
`Sheet 12 of 17
`
`US 6,710,791 B1
`
`FROM
`NORMAL MODE
`
`
`
`S4
`
`N
`OTHER
`PROCESSING
`
`0®
`FIG. 17
`
`sos.
`
`Y
`
`10®
`FIG. 18
`
`$211
`
`DISPLAY
`LIST OF STAMPS
`
`
` IN LIST
`
`OF STAMP
`
`Y
`ERASE
`LIST OF STAMP
`
`END
`
`CHANGE PAGE No.
`OF QUICK STAMP
`
` $218
`
`DISPLAY
`BACKGROUND
`
`FIG. 16
`
`$219
`
`BACKGROUND
`STAMP
`'
`
`S44
`
`SET
`NORMAL MODE
`
`13
`
`

`

`U.S. Patent
`
`Mar. 23, 2004
`
`Sheet 13 of 17
`
`US 6,710,791 B1
`
`IS
`: S31
`
`
`
`
`TOUCHED
`REGULAR
`STAMP
`
`?
`
`
`TOUCHED
`USER-DEFINED
`
`STAMP
`
`?
`
`
`Y
`
`
`
`
`
`MOVE STAMP
`OR
`
`335
`
`SET REGULAR
`MODE
`
`
`
`
`TOUCHED
`
`ALARM
`STAMP
`
` TOUCHED
`
`
`$42
`
`
`
`DISPLAY ALARM
`349
`
`SETTING SCREEN
`
`DISPLAY AUTO-ERASE
`
`SETTING SCREEN
`
`
` $44
`
`
`
`
`FIG. 17
`
`SET NORMAL
`MODE
`
`END
`
`14
`
`

`

`U.S. Patent
`
`Mar.23, 2004
`
`Sheet 14 of 17
`
`US 6,710,791 B1
`
`$221
`
`DISPLAY LIST OF
`RETRIEVAL MODES
`
`$222
`
`y
`
`TABLET
`TOUCH
`?
`
`Y
`
`$223
`
`
`STAMP
`
`RETRIEVAL
`?
`
`$224
`
`DISPLAY LIST
`OF STAMPS
`
`
`
`TABLET
`TOUCH
`?
`
`
`|x $225
`
`
`
`
`SPECIFIED STAMP
`
`.
`
`$223
`OTHER
`PROCESSING
`
`$227
`
`END
`
`PROVIDE 9-SEGMENT
`DISPLAY SHOWING
`9 PAGES WITH
`
`~
`
`BUTTON ON
`
`NEXT
`9-SEGMENT
`DISPLAY
`
`
` NEXT PAGE
`
`
` PREVIOUS
`
`PAGE ON
`
`PREVIOUS
`
`9—SEGMENT
`
`DISPLAY
`
`
`
`TABLET
`TOUCH
`
`N
`
`Y
`
`S234
`
`DISPLAY TOUCHED PAGE
`FULLY ON SCREEN
`
`ENO
`
`15
`
`

`

`U.S. Patent
`
`Mar. 23, 2004
`
`Sheet 15 of 17
`
`US 6,710,791 B1
`
`FIG. 19
`
`9248
`
`DISPLAY "CREATE STAMP”
`SCREEN
`
`
`
`REGISTER CREATED
`
`
` SELECTED
`"CREATE STAMP”
`
`
`
`
`
`
`OTHER PROCESSING
`
`Y
`
`
`
`
`
`STAMP
`
`16
`
`

`

`U.S. Patent
`
`Mar. 23, 2004
`
`US 6,710,791 B1
`
`FIG. 20A
`
`FIG. 20B
`
`Sheet 16 of 17
`
`FIG. 20C
`
`17
`
`

`

`U.S. Patent
`
`Mar. 23, 2004
`
`Sheet 17 of 17
`
`US 6,710,791 B1
`
`
`
`FIG. 21
`PRIOR ART
`
`18
`
`

`

`US 6,710,791 B1
`
`1
`PORTABLE INFORMATION GATHERING
`APPARATUS AND INFORMATION
`MANAGEMENT METHOD
`
`BACKGROUND OF THE INVENTION
`
`1. Field of the Invention
`
`invention relates to portable information
`The present
`gathering equipment such as a pen computer and, more
`particularly,
`to a portable information gather apparatus
`which can function to manage captured information on a
`page basis, and an information management method by the
`apparatus.
`2. Description of the Related Art
`In recent years, pen computers and other similar devices
`have been coming into widespread use as portable informa-
`tion gathering equipment. There are being developed pen
`computers which can not only record information written on
`a tablet-screen with an input pen, but can also capture
`images and spoken statements. There are increasing tenden-
`cies to enhance the multifunctional features of such pen
`computers for greater convenience.
`FIG. 21 is a schematic block-diagram showing conven-
`tional portable information gathering equipmentconstructed
`to perform the function of recording information written on
`a tablet screen with an input pen and also the function of
`capturing images and voice inputs. This portable informa-
`tion gathering equipmentis constituted primarily by a CPU
`1, a ROM 2 in which processing programs andthe like are
`stored, a data memory 3, a tablet 6 which combines a
`digitizer 4 and an LCD 5 into one piece, an image input
`device 7, a voice input device 8, a voice output device 9, an
`information gathering menu selector 10, and a system bus 11
`for connecting these components to the CPU 1.
`Judging from the illustrated configuration, this type of
`conventional portable information gathering equipmentis
`designed to capture image and voice information in addition
`to the pen input information which is known as “ink infor-
`mation” captured through the tablet 6.
`In the conventional portable information gathering
`equipment, however, whenever entering information, a
`menuselection has to be performed through the information
`gathering menu selector 11 to properly set the mode in
`accordance with the type of information to be captured
`before entering the information. For instance, to enter an
`image and then characters, the mode must beset first for
`image input
`to capture an image,
`then the mode must
`changed to character input to take in characters.
`For the portable information gathering equipment to be
`user-friendly, all the different types of captured information,
`namely, the image information, the voice information, and
`the ink information, should be handled as a collection on a
`page basis (hereinafter referred to as “page data”). For
`example, to record the information about a certain product
`in terms of one-page information, it would be convenient to
`enter the image of the product and add the description of the
`image in terms of voice information or ink information
`entered using a pen, thus handling all the image, voice, and
`ink information as a piece of page data, that is, handling
`them in units of pages.
`is desirable that multimedia
`it
`More specifically,
`information,
`including image information, voice
`information, and ink information, captured by a user arbi-
`trarily be stored automatically in a storing means as page
`data, i.e. a single piece of collective data on a page basis
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`without the need for operation by the user so that the user is
`not even aware of the processing.
`This type of conventional portable information gathering
`equipment, however, has not been designed to automatically
`handle all of the image information, voice information, and
`ink information as page data for each page without the need
`for the performance of operations by the user of the equip-
`ment.
`
`Furthermore, this type of information gathering equip-
`ment handles a wide variety of information and the volume
`may be as large as over 100 pages. Therefore, the user is
`required to manage entered information by alarm setting or
`some other method according to the type of information
`whenreading a particular piece of information from among
`the information which hasbeenstored, or reading all related
`information, or if entered information has any time limit.
`In this type of conventional portable information gather-
`ing equipment such as a pen computer,
`it is possible to
`manage the captured information to a certain extent by
`setting an alarm orthe like by writing date, time, contents of
`information, etc. according to a predetermined format.
`However, if such equipment allows information including
`pen input, image, and voice information to be entered as
`desired while it does not have any specified format, thenit
`would have to have a function for managing such informa-
`tion wheneverit is entered.
`
`SUMMARYOF THE INVENTION
`
`Accordingly, it is an object of the present invention to
`provide a portable information gathering apparatus and an
`information management method in the same which obviate
`the need of setting mode according to information to be
`captured whetherit is image information, voice information,
`or ink information, which enable a user thereof to enter any
`type of information as he or she desires, and which permit
`efficient managementof information byefficiently retrieving
`desired information among entered information, by associ-
`ating the information on different pages, or by providing
`information with alarms.
`
`To this end, according to one aspect of the present
`invention, there is provided a portable multimedia informa-
`tion gathering apparatus for gathering and displaying
`entered information and for also storing the gathered
`information, which is equipped at least with: information
`input means for entering information; page adding icon
`setting means for pasting an icon corresponding to the
`information, which has been entered for each page equiva-
`lent to one screen by the information input means,
`to a
`specified page; and data storing means wherein the infor-
`mation entered through the information input meansand the
`information regarding the icon pasted to the specified page
`are written at a specified address.
`The information to be enteredis at least one of: informa-
`
`tion entered through the input/display tablet for capturing
`information when the screen thereof is touched and also for
`
`image information
`displaying the captured information;
`entered through image input means, and voice information,
`or spoken statements, entered through voice input means.
`The icon pasted to the specified pageis at least one of: an
`icon for managing the information captured in respective
`pages by time-related data such as date and time; an icon for
`specifying a particular page; an icon serving as a memoran-
`dum guide; and an icon for managing the information
`captured in the respective pages by the type of information.
`The information regarding the icon pasted onthe specified
`page written to the data storing means is the information
`19
`
`19
`
`

`

`US 6,710,791 B1
`
`3
`indicating the type of icon and positional information indi-
`cating the position where the icon is pasted. The icon
`information for the icon for managing information captured
`in the respective pages by the time-related data such as date
`and time is the time-related information including date and
`time as well as the information indicating the type of icon
`and the positional information.
`With this arrangement, an icon corresponding to infor-
`mation captured on each page can be pasted to captured
`information, and these different types of icons are pasted on
`a page basis as desired,
`thus enabling the pages to be
`managed by the types of icons pasted on the pages. This
`makes it possible to set such functions as alarm, To Do,
`quick call, and automatic erase in units of pages, to permit
`a background stamp such as a telephone memorandum to be
`pasted on each page, and to classify page information by
`type by using the icons. These features are extremely
`convenient for managing andretrieving data which has been
`entered.
`
`invention,
`According to another aspect of the present
`there is provided an information managing method in a
`portable multimedia information gathering apparatus which
`gathers and displays entered information andalso stores the
`gathered information, whereby an icon, which corresponds
`to the information captured for each page equivalent to one
`screen by information input means, can be pasted to a
`specified page, and when the icon has been pasted in a
`predeterminedposition on the page, the information entered
`through the information input means together with the
`information regarding the pasted icon are written at a
`specified address in data storing means, thus implementing
`information management based on the written information
`regarding the icon.
`least one of the
`The information to be entered is at
`information captured through the input/display tablet which
`captures information whenthe screen thereof is touched,the
`image information captured through the image input means,
`and the voice information captured through the voice input
`means.
`
`The icon pasted on a specified page is at least one of the
`following icons: an icon for managing information captured
`into the pages by time-related data such as date and time; an
`icon for specifying a particular page; an icon serving as a
`memorandum guide; and an icon for managing information
`captured into the pages by the type of information.
`The information regarding the icon pasted on the specified
`page which is written to the data storing means is the
`information indicating the type of icon and the positional
`information indicating the position wherethe icon is pasted.
`The information regarding the icon,if the icon is the one for
`managing the information captured into the pages by time-
`related data such as date and time, is the information related
`to time including date and time.
`Use of the method described above allows icons to be
`pasted according to the information captured into the pages
`and also the pages to be managed by the type of icon by
`pasting different types of icons as desired in units of pages.
`This makesit possible to set such functions as alarm, To Do,
`quick call, and automatic erase in units of pages, to permit
`a background stamp such as a telephone memorandum to be
`pasted on each page, and to classify page information by
`type by using the icons. These features are extremely
`convenient for managing andretrieving data which has been
`entered. Applying the method in accordance with the present
`invention to a pen computer or the like turns it into an
`extremely user-friendly pen computer.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`4
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIG. 1 is a block diagram illustrative of the configuration
`of an embodiment of apparatus according to the present
`invention;
`FIG. 2 is an external view of a pen computer to which the
`present invention is applied;
`FIG. 3 is a diagram showing an example of a function
`menu selector which has been set beforehand;
`FIG. 4 is a diagram showing an example ofa list of stamps
`which is displayed when a stamp is selected through the
`function menu selector;
`FIGS. 5A and 5B are diagrams showing an example of
`displayed information which has been captured into the pen
`computer;
`FIGS. 6A and 6B are diagrams illustrating the display
`operation when an alarm stamp has beenselected;
`FIG. 7 is a diagram illustrating the display operation when
`a ToDo stamp has been selected;
`FIG. 8 is a diagram illustrating the display operation when
`a quick stamp has been selected;
`FIG. 9 is a diagram illustrating the display operation when
`an erase stamp has been selected;
`FIG. 10 is a diagram illustrating the display operation
`when a background stamp has been selected;
`FIGS. 11A, 11B and 11C are diagrams illustrating the
`display operation when a user-defined stamp has been
`selected;
`FIG. 12 is a diagram illustrating the contents of a flash
`memory;
`
`FIG. 13 is a diagram showing an example of a display
`screen wherein a stamp has been pasted;
`FIG. 14 is a diagram illustrating the contents of the
`memory of additional information for the page information
`to which a certain stamp has been attached;
`FIG. 15 is a diagram illustrating the positional informa-
`tion of the attached stamp;
`FIG. 16 is a flowchart illustrating a part of a processing
`procedure in the embodimentin accordance with the present
`invention;
`FIG. 17 is a flowchart illustrating a part of a processing
`procedure in the embodimentin accordance with the present
`invention;
`FIG. 18 is a flowchart illustrating a part of a processing
`procedure in the embodimentin accordance with the present
`invention;
`FIG. 19 is a flowchart illustrating a part of a processing
`procedure in the embodimentin accordance with the present
`invention;
`FIGS. 20A, 20B and 20C are diagrams illustrating a
`displaying operation for retrieving in the embodiment; and
`FIG. 21 is a block diagram showing a schematic configu-
`ration of a conventional multimedia information gathering
`apparatus.
`
`DESCRIPTION OF THE PREFERRED
`EMBODIMENT
`
`An embodiment of the present invention will now be
`described. This embodiment will use a pen computer as the
`portable information gathering equipment.
`FIG. 1 is a block diagram illustrative of the configuration
`of the entire embodiment of a pen computer to which the
`present invention has been applied. The pen computer is
`20
`
`20
`
`

`

`US 6,710,791 B1
`
`5
`constituted primarily by: a CPU 21; a ROM 22 forstoring
`basic processing programs for the pen computer and a
`program and data necessary for gathering information
`according to the present invention (the processing procedure
`based on the program will be described in detail later); a
`CCD camera 23 for capturing image information; a CCD
`controller 24 for driving the CCD camera 23 and for
`converting the signals received from the CCD camera to
`digital signals; a microphone 25 for capturing voice infor-
`mation; an amplifier 26 for amplifying voice signals
`received from the microphone 25; an analog-to-digital con-
`verter 27 for converting the voice signals to digital signals;
`an input/display tablet 28 which combines a digitizer 281
`and a liquid crystal display (LCD) 282 into one unit and
`which captures information including characters and line
`drawings and which also displays the captured information;
`an LCD circuit controller 29; a digital-to-analog converter
`30 for converting the voice signals to analog signals when
`outputting voice signals; an amplifier 31; a speaker 32; a
`flash memory 33 serving as a data storing meansfor storing
`data captured by a user (the data to be stored in the flash
`memory and the storing areas thereof will be described in
`detail later); a RAM 34 used as a work memory when
`implementing various types of processing in the pen com-
`puter; a real time clock 35 which indicates the current time
`and provides time information and date information; a
`powerswitch 36; page feed buttons which are comprised of
`a next page button 37 and a previous page button 38; a
`camera button 39; a button signal processor 41 which detects
`the ON/OFFstate of a button such as a voice input button 40
`and issues signals corresponding to the ON/OFFstate; an
`external equipment connecting device 42 for the transfer of
`information betweenitself and external information process-
`ing equipmentsuch as a personal computer; a system bus 43
`for interconnecting the individual sections and component
`units; and an attach-to-page icon setting processor 44 which
`carries out various types of processing on attach-to-page
`icons which are pasted to respective pages so as to manage
`information captured into the pages by time-related data
`such as date and time or classify the information by type or
`to indicate a background of a memorandum,etc. The attach-
`to-page icons, which are to be attached to pages, are put on
`respective pages as if they were stamped thereon;therefore,
`the icons will be hereinafter referred to as “stamps” which
`will be described in detail later.
`
`The external equipment connecting device 42 is con-
`structed primarily by: a serial controller 421; an RS232C
`port 422 used as an interface for connection to external
`information processing equipment such as a personal com-
`puter; an infrared receiver/transmitter 423 for performing
`optical communication with information processing equip-
`ment such as a personal computer; a light emitting diode
`(LED) 424; and a photo diode 425.
`FIG. 2 shows an external view of the pen computer in
`accordancewith the present invention; componentsidentical
`to those shown in FIG. 1 are assigned the same reference
`numerals as in FIG. 1. Among the constituent elements
`shown in FIG. 1, those that appear in the external view of
`FIG. 2 include the CCD camera 23 (more precisely a lens
`23a of the CCD camera 23), the camera button 39, the tablet
`28,
`the microphone 25,
`the speaker 32,
`the voice input
`button 40, the next page button 37, the previous page button
`38, and the power switch 36. Provided on the screen of the
`tablet 28 is a function menuselector 70. An input pen 51 is
`used to draw characters or lines on the screen of the tablet
`28 and to point at icons.
`As illustrated in FIG. 3, the function menu selector 70
`pre-displays icons indicating menus,including a “find” icon
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`6
`71, an icon 72 for calling up the list of stamps, a quick call
`icon 73 for immediately calling a particular page, a key
`word-setting icon 74 for finding information, a 9-page
`display icon 75 for displaying the information of nine pages
`at the same time on the tablet screen, and an icon 76 for
`specifying other menus. These icons are shown in the form
`of patterns or the like which visually show the functions of
`the icons. When one of the icons is pointed at, the contents
`of, or information identifying, the icon are displayed on the
`screen.
`
`For instance, among the icons mentioned above, if the
`icon 72 for calling up the list of stamps is pointed at, then
`the list of stamps as shown in FIG. 4 will appear on the
`screen of the tablet 28. These stamps are used to manage
`information by each pageor to provide backgroundpictures.
`The meaningsand functionsof the individual, stamps will be
`described later.
`
`When the 9-page display icon 75 is pointed at after the
`power switch is turned ON,a total of nine pages of infor-
`mation are displayed at the same time in reduced sizes,
`including four pages each preceding and following a par-
`ticular page which is the page displayed when the power
`switch is turned ON and which is normally the page previ-
`ously processed. Therefore, this icon serves also as an index;
`however, no further description will be given aboutthis icon
`becauseit is not directly connected to the present invention.
`In such a pen computer when characters or line pictures
`are written on the screen of the tablet 28 with the input pen
`51, the CPU 21 inside reads coordinates from the digitizer
`281 and displays the writing trace in the corresponding
`position of the LCD 282 and stores the coordinate data at a
`predetermined address of the flash memory 33.
`To input an image through the CCD camera 23, the lens
`23a of the CCD camera 23 is pointed at the image to be
`captured and the camera button 39 isfirst depressed halfway.
`This causes the image entered through the lens 23a to be
`projected as an animation on the screen of the tablet 28; and
`when the camera button 39 is further depressed, the image
`is locked and the locked imageis captured. The image signal
`is converted to a digital signal by the CCD cameracontroller
`24 before it is stored at a predetermined addressin the flash
`memory 33.
`To input spoken information, the voice input button 40 is
`depressed to enable audible speech to be captured through
`the microphone 25; the spoken sounds are then amplified
`through the amplifier 26 and converted to a digital signal by
`the analog-to-digital converter 27 before it is stored at a
`predetermined address in the flash memory 33.
`The operation for capturing information is performed in
`units of screens, i.e. on a page basis. The following will
`describe the processing procedure.
`If the pen computer has never been used, when the power
`switch 36 is turned ON, nothing will be displayed on the
`screen of the tablet 28; this screen will be regarded as the
`first page. However, the following will describe an example
`wherein some information has already been captured into
`the first page and additional information will be captured on
`the second and subsequent pages.
`In this example, the user of the pen computer will meet a
`person, whom he has never met before, to have a business
`discussion during which he will capture information regard-
`ing the person and the details of the discussion and also
`information about a product, which is the topic of the
`business discussion.
`
`Whenthe userfirst turns ON the power switch 36, the first
`page into which information has already been captured will
`21
`
`21
`
`

`

`US 6,710,791 B1
`
`7
`be displayed. Pressing the next page button 37 causes the
`screen of the second page, into which no information has
`been captured yet, to be displayed. With the blank second
`page on the screen, in order to capture the image of the face
`of the person with whom the user is meeting, the user points
`the lens 23a of the CCD camera 23 at the person and
`depresses the camera button 39 halfway as previously
`described,
`then further, e.g. fully, depresses the camera
`button 39. This causes the then-existing face image, which
`is a stationary image, to be captured and the face image to
`be displayed on the screen of the tablet 28 at the same time,
`as shown in FIG. 5A; the image data is stored at a prede-
`termined address of the flash memory 33. The position and
`size of the area where the image is displayed may bepreset
`in the display area on the screen ofthe tablet 28 or they may
`be set according to the situations of each case.
`FIGS. 5A and 5B show examples where an area of about
`one-quarter of the display screen at top left thereof is set as
`the image area.
`The user of the pen computer then writes with the input
`pen 51 memo-like information including the name of the
`person, the company nameand the departmentor section to
`which he belongs as the information regarding the person.
`The ink information, such as characters or line drawings,
`entered with the input pen 51 is displayed on the screen of
`the tablet 28 as the user draws them andit is also stored at
`a predetermined address in the flash memory 33.
`Furthermore, to record an importantpart of the discussion
`in terms of spoken statements, the voice input button 40 is
`depressed to capture the such statements. The captured voice
`is subjected to digital conversion and stored as a voice signal
`at a predetermined address in the flash memory 33. Upon
`completion of capturing the voice, an icon 52 indicating the
`completion of recording appears at a predetermined point on
`the screen of the tablet 28. This icon 52 is used also for
`
`function reading. To be more specific, when the data on the
`page is read later, the voice information is read and repro-
`duced from the predetermined address in the flash memory
`33 by pointing at the icon 52 with the input pen 51.
`FIG. 5A shows an example where the information which
`has been captured according to the procedure described
`aboveis displayed on the screen of the tablet 28, that is, the
`information on the second page. As it is seen from. FIG. 5A,
`the visual information of the page information of one page
`unit is given by face image information 61, ink information
`62 such as characters and line drawings, and the icon 52
`which indicates the presence of the voice information in the
`captured information.
`the
`In addition to the information mentioned above,
`information indicating the date at that point is also displayed
`automatically by the real-time clock 35. The date and time
`information is also written at a predetermined address in the
`flash memory 33 as part of the additional information to go
`with the captured information which includes the image,
`voice, and ink information.
`to
`To go on to the next page, namely the third page,
`capture new information after finishing the collection of
`information for the second page, the user presses the next
`page button 37 to use the third page for capturing more
`information.
`
`The write area for the data stored in the flash memory 33
`will be described later.
`
`FIG. 5B showsan example of the information written to
`the third page in which an image 63 of t