(12)
`
`United States Patent
`Dawson
`
`(10) Patent N0.:
`(45) Date of Patent:
`
`US 6,252,588 B1
`*Jun. 26, 2001
`
`US006252588B1
`
`(54) METHOD AND APPARATUS FOR
`PROVIDING AN AUDIO VISUAL EMAIL
`SYSTEM
`
`.
`(75) Inventor‘ John Dawson’ Palo Alto’ CA (Us)
`
`(73) Assignee: ZenEek gfechnology, Inc., Foster City,
`CA US
`
`(*) Notice:
`
`This patent issued on a continued pros-
`ecution application ?led under 37 CFR
`1.53(d), and is subject to the tWenty year
`patent term provisions of 35 U.S.C.
`6409(2)
`_
`_
`_
`_
`SubJeCt_tO any dlsclalmer>_ the term of thls
`Patent 15 extended or adlusted under 35
`U-S-C- 154(k)) by 0 days-
`
`(21) Appl. N0.: 09/098,362
`.
`_
`Jun‘ 16’ 1998
`(22) Flled'
`(51) Int. c1.7 .................................................... .. G06F 13/00
`(52)
`345/329; 709/206; 345/331
`(58) Field of Search ................................... .. 345/339 440
`345/302 331 329 330 332. 707/10. 706/206’
`’
`’
`’
`’
`2,45 204’ 237 206
`’
`’
`’
`
`(56)
`
`References Cited
`
`Us PATENT DOCUMENTS
`709006
`5 564 018 * 10/1996 Flores et al
`5,579,472 * 11/1996 Keyworth, 11 et a1. ............ .. 345/326
`5,742,769 * 4/1998 Lee et al. ............ ..
`709/206
`5,752,059 * 5/1998 Holleran ............................. .. 709/245
`
`5,778,054 * 7/1998 Kimura etal. .................. .. 379/93.23
`5,793,365 * 8/1998 Tang et al. ......................... .. 345/329
`5,818,935 * 10/1998 Maa ........... ..
`380/20
`5,867,281 * 2/1999 NoZoe et al.
`. 358/402
`5,867,654 * 2/1999 Ludwig et al. ..
`709/204
`6,014,689 * 1/2000 Budge et al. ...................... .. 709/206
`
`* Cited by examiner
`Primary Examiner—Raymond J. Bayerl
`ASSiSl/lnl EX?min€r—Tad6SSe Hailu
`(74) Attorney, Agent, or Firm—The Hecker LaW Group
`
`ABSTRACT
`(57)
`An audio visual e-mail system and method are presented.
`The audio visual e-mail system of the invention reduces the
`complexity of sending and receiving audio visual e-mail
`messages to a level that alloWs a user to send and receive
`audio visual e-mail With a minimum of inconvenience. The
`prior art paradigm of selecting e-mail addresses from a list
`of long and complex e-mail addresses is replaced by select
`ing a thumbnail image of the intended recipient. Instead of
`trying to guess the identity of the sender of an e-mail
`message, an e-mail message Created according to the inven
`tion is automatically “signed” by a thumbnail image of the
`Sender‘ Instead of Sending long Written descriptions that are
`prone to misunderstanding and dif?cult to read, the sender of
`an e-mail message according to the invention can send a
`photo of the object to be described together With a verbal,
`audio description. In one embodiment, a user uses a
`television-type remote control as a command interface to
`create audio visual messages containing a picture, a verbal
`description in the user’s oWn Words, and a thumbnail portrait
`of the user
`
`11 Claims, 12 Drawing Sheets
`
`1803 i
`'
`'
`
`PHONE
`JACK
`
`______________________________________________________________ __ £1801
`f 1805
`/1807
`
`COM
`MANAGER
`
`MIME
`UNIT
`
`/ 1817
`
`MICRO
`PHONE
`
`g
`l
`3
`3
`
`i
`:
`5
`
`1811
`
`1809
`GRAPHICS
`RECIPIENT
`MAIL
`HANDLER H DATABASE —’ ENGINE
`___T
`
`1813 _|
`SOUND
`ENGINE
`
`782’ l
`IEEE1394
`MANAGER
`
`7823
`
`LAYOUT
`ENGINE
`
`INPUT
`SYSTEM
`
`/1825
`
`[1819
`
`SPEAKER
`
`/-1827
`
`DISPLAY
`
`;
`1815 i
`
`I
`l
`5
`
`1829
`
`1831
`
`IR
`COMMANDS
`
`EXTERNAL
`DEVICES
`
`Snap's Exhibit No. 1061
`Page 1
`
`

`

`U.S. Patent
`
`Jun. 26, 2001
`
`Sheet 1 0f 12
`
`US 6,252,588 B1
`
`MORE INFO
`
`m
`EEIIEI
`M
`107\
`mi ELM
`\113
`/ CLEAR \
`\111
`/
`109/
`/[I||Z||3l
`115/
`Ill 15 151
`[Z] IE 131
`m1
`ENTER
`
`105
`
`117
`
`119\
`
`127\
`
`/121 C123 125
`
`/ 129
`
`FIG. 1
`
`Snap's Exhibit No. 1061
`Page 2
`
`

`

`US 6,252,588 B1
`
`U.S. Patent
`
`Jun. 26, 2001
`
`Sheet 2 of 12
`
`00¢
`
`aOLs
`
`dG ¢Old
` INUdTidCavMeOdAldidSWMGNDSW179Nz07FNEE”102Qvau{azEANppgNNsehateaBo&WySSIW08MIIALaFTi©
`
`AUNTS
`0%a26lzale912
`YFQNISHLOZJHLNOONLTIN8B
`
`
`
`
`
`
`
`
`
`
`
`INVdFOVSSIWFHLMOHSOLFYFHHOMTI4
`
`Snap's Exhibit No. 1061
`Page 3
`
`Snap's Exhibit No. 1061
`Page 3
`
`

`

`U.S. Patent
`
`Jun. 26, 2001
`
`Sheet 3 of 12
`
`US 6,252,588 B1
`
`
`
`W'V00:11LYHLOZFHINODNUFIWHNOWHIINODOLSISIHL
`
`‘MINIOTYYI
`
`HtYFONIS
`
`
`
`‘SauvoqdISI
`
`lS,QA
`dOlS806ALUNIWSONITHdSHOWLSSIH0CVLONDNaS
`
`
`
`
`BelFe/BoAieog.
`
`
`
`€Old
`
`Snap's Exhibit No. 1061
`Page 4
`
`Snap's Exhibit No. 1061
`Page 4
`
`

`

`U.S. Patent
`
`Jun. 26, 2001
`
`Sheet 4 0f 12
`
`US 6,252,588 B1
`
`[401
`SELECT IMAGE
`RESOURCE
`
`PRESENT
`IN COMPUTER
`.7
`
`/ 419
`
`SCAN IMAGE
`
`[417
`
`v
`RETRIEVE
`IMAGE
`
`CONVERT INTO
`USABLE FORMAT
`
`STORE ON
`HARD DISK
`
`CONVERT TO
`C ORREC T FORMAT
`
`ATTACH IMAGE
`= FILE TO E-MAIL
`MSG.
`
`SEND MESSAGE
`
`FIG. 4
`
`Snap's Exhibit No. 1061
`Page 5
`
`

`

`U.S. Patent
`
`Jun. 26, 2001
`
`Sheet 5 0f 12
`
`US 6,252,588 B1
`
`/
`
`HAPPY H ULIDA YS
`
`[501
`[503 x
`
`5507
`
`/
`
`FIG. 5
`
`f
`
`[605/ 6%
`[RECORDING...43 SEC]
`
`\
`
`/
`
`FIG. 6
`
`Snap's Exhibit No. 1061
`Page 6
`
`

`

`U.S. Patent
`
`Jun. 26, 2001
`
`Sheet 6 0f 12
`
`US 6,252,588 B1
`
`/
`
`/ 701
`/707 \
`N0 PHOTOS AVAILABLE
`703
`TYPEA TEXT MESSAGE FOLLOWED BYENTER
`/
`0H HIT srop TO RETURN 10 FIRST SCREEN
`
`ZQQ
`
`FIG. 7
`
`\
`
`/
`
`Snap's Exhibit No. 1061
`Page 7
`
`

`

`U.S. Patent
`
`Jun. 26, 2001
`
`Sheet 7 0f 12
`
`US 6,252,588 B1
`
`[907
`/903 \
`
`‘x /f\ \/921
`11/ 1
`K ' 1Q '29
`
`15 [917 907
`’
`:1
`
`§
`
`/
`
`f
`
`905
`
`\
`
`SEND MESSAGE
`PICK MORE RECIPIENTS \
`\922
`
`911 [913
`[909
`’
`/i
`m \
`:1,
`
`919
`FIG.9
`
`r
`
`/1001
`\
`HAPPY HOLIDAYS! ” "m
`
`[1909 1011/1013 [1015/1017
`'
`j
`'
`/
`
`1007
`
`Snap's Exhibit No. 1061
`Page 8
`
`

`

`U.S. Patent
`
`Jun. 26, 2001
`
`Sheet 8 0f 12
`
`US 6,252,588 B1
`
`/1101
`f HAPPY HOLIDAYS!
`
`1103
`
`\
`
`f
`
`1205<
`
`FIG. 11
`
`HAPPY HOLIDAYS!
`
`/1201
`
`/ 1203
`
`{R23
`11219
`
`[1209 1211 [1213 /1215/1217
`E 1207
`j
`
`FIG. 12
`
`Snap's Exhibit No. 1061
`Page 9
`
`

`

`U.S. Patent
`
`Jun. 26, 2001
`
`Sheet 9 0f 12
`
`US 6,252,588 B1
`
`/ 1301
`/ /DELETING THIS MESSAGE SENT BYXX 12/123/98 \
`“*2”
`HAPPY HOLIDAYS! “)3
`
`1311 [1313 [1315/1317
`
`1307 5
`
`j
`
`/ 1401
`\
`1403\/
`7407\\PH0NE NUMBER FOHINTEHNETACCESS: ll]
`1411\\PPP LOG/N PBOMPT: E
`1415\\PPP LOG/N NAME:
`1}
`1419\\PPP PASSWORD PROMPI-
`[I]
`1423QPPPPASSWORDI @
`MQEQPAPPRESS: @
`1431\\DNS SERVER ADDRESS:
`l]
`1435\\P0P3 SERVER NAME:
`E]
`1439\\SMTP SERVER NAME:
`|Il
`1443\\EMAIL LOG/N NAME.-
`|I_]
`\EMA/L LOG/N PASSWORD;
`llgfj
`PRESS ENTER TO SAVE CHANGES AND TRY IT. / 1447
`\ PRESS STOP T0 QUIT WITHOUT SAVING CHANGES. /
`
`FIG. 14
`
`Snap's Exhibit No. 1061
`Page 10
`
`

`

`U.S. Patent
`
`Jun. 26, 2001
`
`Sheet 10 0f 12
`
`US 6,252,588 B1
`
`1503\/
`1507\\PARAMETER 1-‘
`1511\\PARAMETER 2'
`\PARAMETER 3;
`
`\
`IE:
`III
`1|
`
`PRESS ENTER TO SAVE CHANGES AND TRY II
`\ PRESS STOP TO QUIT WITHOUT SAVING CHANGES. /
`
`FIG.15
`
`6
`
`CURRENT PICTURE
`
`[1601
`W
`
`1621
`MASAKI@ZENTEK. 0011/1 /
`
`g i?
`g
`
`[1609 1611 [1613 [1615/1617
`'
`'
`'
`f
`
`1605
`
`6 61
`
`1619
`
`@1 6
`
`1607
`
`j
`
`Snap's Exhibit No. 1061
`Page 11
`
`

`

`U.S. Patent
`
`Jun. 26, 2001
`
`Sheet 11 0f 12
`
`US 6,252,588 B1
`
`1732\
`f1731
`/ 1729\A
`MASAKI@ZENTEK. 00M
`/1727
`DAWSON@ZENTEK. 00M
`/1723
`DDAVIS@ZENTEK. COM
`
`Wk
`
`172“
`
`[1701
`\
`
`1702
`
`1703\
`NEW
`
`f 1718
`
`W
`7733
`/1709r1711,1713/1715/1717
`1705
`I ”\
`[1/ i,
`17071704
`
`Q@ @PQQD/
`
`\
`
`\1719
`
`j
`
`Snap's Exhibit No. 1061
`Page 12
`
`

`

`U.S. Patent
`
`Jun. 26, 2001
`
`Sheet 12 0f 12
`
`US 6,252,588 B1
`
`B2 \ ........................................................ -
`
`:2 \ m 82 \ 222 \
`265 m 25 252%
`.822 m “.52 28
`
`m l‘ l ‘ < :
`
`M22 / 1
`
`
`
`\ 22% m \ $93 $222 5%
`
`\ 225% n 1 23% 8225s 1 2225mm 1 1 $2
`
`
`
`
`22 M _| 22\ :2\ %2\ >
`
`
`
`1 " 26E 25E 25.2% 1 1 $33:
`
`
`
`
`
`
`
`" 255 $232 22%
`
`_
`
`2 6E
`
`B2 \ $2 \
`
`253% 822.28
`2255 2
`
`v63.
`
`Snap's Exhibit No. 1061
`Page 13
`
`

`

`US 6,252,588 B1
`
`1
`METHOD AND APPARATUS FOR
`PROVIDING AN AUDIO VISUAL E-MAIL
`SYSTEM
`
`BACKGROUND OF THE INVENTION
`
`1. Field of the Invention
`This invention relates to electronic messaging systems.
`2. Background Art
`With the proliferation of personal computers and com
`munications netWorks such as the internet, electronic mail,
`commonly referred to as “e-mail,” has become a popular
`mechanism for the eXchange or distribution of information
`among individuals, and Within or betWeen enterprises, for
`both private and commercial purposes. There are a number
`of disadvantages of current e-mail systems. One disadvan
`tage is that a computer system is needed. Another disadvan
`tage is that it is dif?cult to use if a sender Wants to attach
`pictures or audio messages to an e-mail message. These
`disadvantages can be understood by revieWing the Way
`e-mail Works.
`An e-mail message may be analogiZed to a posted letter
`or piece of mail. HoWever, instead of a physical object that
`is itself physically transported from a sender to a recipient,
`an e-mail message is an electronic representation that is
`communicated electronically through a communications
`netWork. Examples of communications netWorks used for
`communicating e-mail messages include, but are not limited
`to, telecommunications netWorks, Wide area netWorks
`(WAN s), local area netWorks (LAN s), the Internet, intranets,
`eXtranets, Wireless networks, and other netWorks over Which
`electronic, digital, and/or analog data may be communi
`cated.
`Prior Art E-mail Systems
`E-mail messages are created, sent, received, and read
`using a communications program, often referred to as a
`“mail” or “e-mail” application program. An interface of one
`prior art e-mail program is illustrated in FIG. 2. The eXample
`of FIG. 2 is the interface of the mail module of Netscape
`Communicator, an internet broWser application program.
`The interface is displayed in a WindoW 200. The interface
`includes a roW of control buttons 201—210 arranged along
`the top of WindoW 200, column heading boXes 212—215
`arranged beloW control buttons 201—210, and a display ?eld
`217 that displays a list of messages. In the eXample of FIG.
`2, the list indicates a single message 216. The information
`displayed for message 216 in display area 217 includes an
`icon 218 indicating that message 216 is an e-mail message,
`teXt 219 indicating the subject of message 216, namely
`“Meeting on the 20th”, teXt 220 indicating the source or
`destination of message 216, in this case “Sender” and teXt
`221 indicating the time that message 216 Was sent or
`received, in this case 9:20 PM.
`Control buttons 201—210 are used to create, modify and
`manipulate messages. Button 201, “Get Msg” is used to
`display a message that has been selected from the list of
`messages displayed in display area 217 (messages can also
`be opened by double clicking on them). Button 202, “NeW
`Msg” is used to generate a neW message form that can be
`completed and mailed to a recipient. “Reply” button 203 is
`used to generate a reply form for replying to a message that
`is currently in vieW. When activated, a message form is
`generated that has as its address the address of the sender of
`the message being vieWed. Optionally, the reply message
`may include the entire teXt of the sender’s message. “For
`Ward” button 204 generates a message form that includes the
`sender’s message, but With a blank address, so that the
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`message may be optionally annotated and forWarded to
`another recipient.
`Button 205, “File”, is used to save a message into a ?le in
`a teXt or other format. Button 206, “Print”, is activated to
`print the message on an attached printer. “Security” button
`208 activates security options for a message such as, for
`eXample, encryption, or the use of a digital certi?cate or
`digital signature. Messages can be deleted using “Delete”
`button 209. “Stop” button 210 is used to interrupt or stop
`operations.
`Column heading boXes 212—215 contain column headings
`for the information displayed in display area 217. The
`headings speci?ed in column heading boXes 212—215 are
`“Subject”, “To/From”, “Date”, and “Priority”, respectively.
`Messages in the list can be sorted by subject, by sender or
`recipient, by ascending or descending date, by priority, or by
`any combination thereof.
`An e-mail message form 300 generated using the eXample
`e-mail program of FIG. 2 is illustrated in FIG. 3. E-mail
`message form 300 includes a palette of control buttons
`301—308, an address entry area 309, a subject entry area 310,
`and a message editing area 311. Address entry area 309
`indicates the destination(s) for the message, in this case
`“Receiver@receiver.com”. Subject entry area 308 indicates
`the subject of the message, here “Meeting on the 20th”. The
`body 311 of the message is displayed in editing area 312.
`“Send” button 301 is activated to initiate the transmission
`of the message from the sender to the recipient. “Quote”
`button 302 is used to insert teXt from another message into
`body 311 of message displayed in editing area 312.
`“Address” button 303 prompts the sender to enter an address
`of the recipient or to select an address from a stored address
`book. “Spelling” button 305 performs a spell check on the
`message teXt, and “Save” button 306 is used to save a
`message as a teXt ?le.
`“Attach” button 304 is used to attach one or more elec
`tronic ?les to the e-mail message. The operation of this
`e-mail ?le attachment feature illustrates disadvantages and
`limitations of the prior art. Often a sender Wishes to send one
`or more ?les to a recipient. One method of sending a ?le to
`a recipient is to copy the information from the ?le (eg the
`teXt from a teXt ?le) and paste that information into the body
`309 of an e-mail message. If the ?le is large, this may not be
`possible, since some e-mail programs have limitations on
`the siZe of the body of an e-mail message. In other cases, the
`?les represent non-text data, such as sound, images, or
`movies, for eXample, that cannot be easily pasted into the
`body of an e-mail message. In such circumstances, the ?le
`attachment feature is used.
`When “Attach” button 304 is activated, a dialogue boX
`appears that alloWs the send er to navigate through a ?le
`system and select ?les to be attached to the e-mail message.
`After one or more ?les are selected, the sender causes the
`e-mail program to transmit the e-mail message and attached
`?le(s) to a recipient. When the recipient reads the message,
`the reader’s e-mail program displays an indicator indicating
`that one or more ?les are attached. If the recipient uses the
`e-mail program of FIGS. 2 and 3 the recipient may retrieve
`the attached ?le(s) by activating the “Attach” button. Upon
`activating the “Attach” button, the recipient is presented
`With a dialogue boX that enables the recipient to retrieve the
`attached ?le or ?les and store them in the recipient’s ?le
`system.
`Attaching Image Files to E-mail Messages
`Users often desire to send one or more pictures With an
`e-mail message. These pictures can be photographs, digital
`photographs, computer draWings, graphic images, or any
`
`Snap's Exhibit No. 1061
`Page 14
`
`

`

`US 6,252,588 B1
`
`3
`other type of image. These kinds of image ?les often require
`preparatory Work to make them suitable for attachment to
`e-mail messages. Accordingly, attaching image ?les to an
`e-mail message can be a complex process.
`Before an image ?le can be attached to an e-mail message,
`the ?le must be present on the computer, or must be placed
`on the computer. If the ?le is already located on the
`computer the user must remember What the ?le is named,
`and Where it is located. Often it is necessary not only locate
`the ?le but also to vieW it ?rst to make sure that it is indeed
`the correct ?le that is intended to be sent. The ?le must be
`in a format that the person receiving the ?le has a capability
`to use. If it is not the ?le may have to be converted, using
`a conversion program, to a common format (such as GIF or
`JPEG) that the person receiving the ?le has the capability of
`vieWing.
`If the ?le is not already on the computer it must be taken
`from another source and placed on the computer. There are
`a variety of sources for such ?les. For example, the image
`?le may come from a picture taken by a digital camera, from
`?les on the internet, or from a CD ROM.
`If the image is not already in a computer readable form,
`(for example, if the image is a photographic print or a
`neWspaper clipping) it must ?rst be converted into such a
`form. This can be done, for example, by using a process
`called scanning. When an image from a photograph or
`neWspaper article is scanned it is placed on a device called
`a scanner Which is attached to a computer. A scanner
`illuminates consecutive small sections of the item to be
`scanned and converts the small section into a series of
`numbers that represent the colors and intensity at discrete
`points of the section. The computer stores these numbers in
`a ?le, and then the scanner moves on to the next small
`section of the item to be scanned and the process is repeated
`over and over until the entire item has been scanned and
`converted into a sets of numbers and placed in the computer
`?le. The ?le can then be used to recreate an image of the item
`scanned on another computer. HoWever, because raW
`scanned ?les are often large and therefore dif?cult to
`transmit, they are often converted into more compact ?les
`(such as GIF or J PEG) Which are smaller and therefore more
`easily transmitted. Scanned ?les are typically converted
`using conversion programs, and then stored in their smaller
`converted form on the hard disk of the computer.
`Once a ?le exists in an appropriate format on the com
`puter it can be sent in the form of an e-mail attachment. To
`attach an image ?le to an e-mail message a user utiliZes the
`attachment feature of the user’s e-mail program, for
`example, by clicking the “Attach” button 304 of the e-mail
`program of FIG. 3. Typically, a dialog box appears With a
`representation of the ?le system hierarchy of the computer.
`The user then navigates through the ?le system hierarchy to
`?nd the desired image ?le. (The user may need to ?rst locate
`and vieW the ?le to verify that it is the correct ?le.) The user
`then attaches the ?le, usually by double clicking the mouse
`button on the name of the ?le. At that point the ?le has been
`attached to the e-mail and is ready to be sent. The user then
`activates the “send” command of the user’s e-mail program
`(e. g. by clicking on the “send” button 301 of FIG. 3), and the
`e-mail along With the attachment is transmitted.
`FIG. 4 illustrates a How diagram of the process of
`inputting and attaching an image ?le to an e-mail as an
`e-mail attachment using an e-mail program of the prior art.
`Referring to FIG. 4, at step 401 the image resource ?le to be
`sent as an e-mail attachment is selected. At step 403 a
`determination is made as to Whether the selected image ?le
`is present in the computer. If the image ?le to be sent as an
`
`10
`
`15
`
`25
`
`35
`
`45
`
`55
`
`65
`
`4
`attachment is already present on the computer, the process
`proceeds to step 405 Where the image to be sent is located.
`After the image is located the process proceeds to decision
`block 407 at Which a determination is made as to Whether the
`image is in the correct format. If it is determined at decision
`block 407 that the image ?le is in the correct format, the
`process proceeds to step 411. If the image is not in the
`correct format, the process proceeds to step 409 Where the
`image is converted to the correct format. The process then
`proceeds to step 411.
`At step 411 the image ?le in the correct format is attached
`to the e-mail message. The sequence then proceeds to step
`413, Where the message containing the attached image ?le is
`transmitted to the recipient.
`If it is determined at decision block 403 that the image ?le
`to be sent as an attachment is not already present on the
`computer, the process proceeds to decision block 415. At
`decision block 415 a determination is made as to Whether the
`image is in a computer readable ?le format. If the image is
`in a computer readable ?le format (e.g., on a CD-ROM or
`retrievable from the internet), the process proceeds to step
`417, Where the image is retrieved from the appropriate
`source. The process then proceeds to decision block 407.
`If it is determined at decision block 415 that the image is
`not in a computer readable form, the process proceeds to
`step 419. At step 419, the image is converted to computer
`readable form, for example by scanning in the image With a
`scanner. The process then proceeds to step 421 Where the
`scanned image is converted into a usable format suitable for
`the recipient Who Will receive the ?le as an attachment. The
`process then proceeds to step 423. At step 423 the scanned
`and converted image is stored as a ?le on the hard disk. The
`process then proceeds to step 411.
`Attaching a Sound File to an E-mail Message
`Attaching a sound ?le to an e-mail message is as complex
`as attaching an image ?le. Before a sound ?le can be
`attached to an e-mail message the sound ?le must be present
`on the computer, or must be placed on the computer. If the
`?le is already located on the computer the user must remem
`ber What the ?le is named, and Where it is located. Often it
`is necessary not only locate the ?le but also to listen to it ?rst
`to make sure that it is indeed the correct ?le that is intended
`to be sent. The ?le must be in a format that the person
`receiving the ?le has a capability to use. If it is not the ?le
`may have to be converted, using a conversion program, to a
`common format (such as WAV) that the person receiving the
`?le has the capability of playing.
`If the sound is not already in a computer readable ?le
`format, (for example if it is a spoken message, or is analog
`data contained on an audio tape) it must ?rst be converted to
`a computer readable form. This can be done, for example, by
`using a microphone and a sound card installed in the
`computer for this purpose. To input a sound message into the
`computer the user Will typically activate a sound recording
`program. The user Will then activate a “record” function
`Within the sound recording program. When the recording
`function has been activated the user speaks the message,
`plays the audio tape containing the message or otherWise
`creates the sound that is to be recorded. When the message
`is completed the user activates the “stop recording” function
`of the sound recording program. The microphone and sound
`card combination convert the sound that is entering the
`microphone While the recording function is active into a
`digital format that can be stored on the computer. One
`process for doing so is knoWn as “sampling”.
`The process of sampling can be reversed and the sampled
`digital sound data converted back into sound by recreating
`
`Snap's Exhibit No. 1061
`Page 15
`
`

`

`US 6,252,588 B1
`
`10
`
`15
`
`25
`
`35
`
`5
`the intensity and frequency of the sound represented by the
`stored digital data. To do so a playback program is typically
`used. The playback program takes the digital sound ?le that
`the computer has stored and converts it back to the original
`sound through the use of a sound card, ampli?er and speaker
`Within the computer.
`If the sampling process is not rapid enough (eg if it does
`not meet the Well knoWn “Nyquist” criteria), the quality of
`the sound reproduced from the digital sound ?le Will not be
`very high, and the sound Will be distorted. To keep this
`sampling rate high (typically tens of thousands of samples
`per second) the computer must store tens of thousands of
`numbers per second to accurately reproduce a sampled
`sound. A ?le representing a feW seconds of sound can
`therefore easily groW to be quite large.
`It is because of this large ?le siZe that sound ?les are not
`usually stored as raW ?les, but, like large image ?les, are
`encoded to make their siZe smaller. A sound encoding
`program is used for this purpose. Asound encoding program
`can take advantage of the fact that sound in general and
`speech in particular has patterns of frequency and intensity,
`and use those patterns to reduce the siZe of the ?le. There are
`many different types of encoding used in sound, the WAV
`?le encoding being one of the most popular.
`After the sound has been recorded by the computer it
`usually is listened to by the user to ensure that the correct
`sound clip is contained Within the ?le. Typically though the
`?le Will contain some unintended sounds at the beginning or
`end, or may not contain the entire sound message intended
`to be recorded. If the entire message is not contained Within
`the ?le the normal procedure is to re-record the message. If
`the ?le contains additional sounds the ?le can be edited
`using a sound editing program. Usually a sound editing
`program is included as part of a recording program.
`A sound editing program usually generates a visual rep
`resentation of the sound in the form of a linear Waveform
`graph. Several seconds of sound can typically be displayed
`on the screen at a time. When the sound ?le is played using
`the sound editing program, the Waveform is highlighted to
`shoW the part of the Waveform that is currently being heard.
`The user of the program can then select and delete the
`undesired portions of the sound ?le. In that Way a user can,
`for example, eliminate a preliminary clearing of a throat
`prior to the speech intended to be recorded, or perhaps the
`beginning of a next sentence that Was accidentally recorded.
`After the ?le has been properly edited it can be stored on
`the computer’s hard disk under a suitable name. It can also
`be converted, if necessary to a common format such as WAV
`that the intended recipient of the ?le has the capability of
`playing.
`Once the ?le exists in an appropriate format on the
`computer it is ready to be sent as an e-mail attachment.
`The How process of inputting and attaching a sound ?le to
`an e-mail message is similar to the process of attaching an
`image ?le to an e-mail message, described above With
`reference to FIG. 4. The user must locate the ?le or import
`the ?le to be attached, playback the ?le to ensure it contains
`the intended sound clip, put it in the correct format, and then
`attach it to an e-mail message.
`Receiving and Decoding E-mail Attachments
`When a user receives an e-mail message that includes
`audio and/or image attachments, a complex series of opera
`tions must be performed before the recipient can access the
`attached ?les. First, the recipient must determine Whether
`the recipient has received an e-mail message containing
`image or sound ?le attachments. The recipient looks for neW
`messages using a mail checking function of the user’s e-mail
`
`45
`
`55
`
`65
`
`6
`program, for example by clicking on the “Get Msg” button
`201 of the e-mail program of FIG. 2. Identifying information
`for any neW message that has been received is added to the
`message WindoW on the recipient’s display. For example in
`FIG. 2, the identifying information “Meeting on the 20th”
`for message 216 appears in the message WindoW. (If the
`message is unread it sometimes Will appear in boldface
`type).
`To ?nd out if the e-mail message contains an attachment
`the user must open the e-mail message. For example, the
`e-mail may be opened by clicking tWice on the subject of the
`message. The message Will then open up in a WindoW as
`shoWn, for example, in FIG. 3. If there is an attachment an
`attachment indicator is typically displayed, such as, for
`example, an icon at the end of the message.
`Once the user has determined that there is an attachment
`to the e-mail message, the user must retrieve and save the
`attachment. To save an attachment the attachment must ?rst
`be selected. This may be done, for example, by clicking on
`the attachment icon. The user must then specify Where the
`attachment is to be saved. Typically, after the attachment has
`been selected, a dialog box appears that alloWs the user to
`navigate through the ?le system and select Where the
`attached message Will be saved. The same dialog box Will
`usually also alloW the user to change the name of the
`attached ?le if desired.
`After the attached ?le has been saved, the user may close
`or delete the e-mail message. To vieW or listen to the
`attached ?le, the user must ?rst navigate through the ?le
`hierarchy to ?nd the correct ?le and determine Whether it is
`in a format compatible With the user’s computer system. If
`the ?le is not in a format that is compatible With the
`programs that are present in the user’s computer, the user
`may need to convert the ?le using a conversion program and
`then save the converted ?le. If the ?le is not in a usable
`format, and if the proper conversion softWare to convert it to
`the correct format is not available, the user may either obtain
`the necessary softWare, or simply discard the ?le as useless.
`Once the ?le is in the proper format the user then may
`access (vieW or listen to) the ?le. To access the ?le the user
`must choose an appropriate accessing program and activate
`it. Once the accessing program has been activated the user
`must again navigate through the ?le system hierarchy to
`retrieve the desired ?le.
`Because separate programs are typically required to read
`an e-mail message and to access attached ?les it is difficult
`to read a message and access an attached ?le at the same
`time. To do so both the e-mail program and the access
`program must be displayed on the user’s computer monitor
`at the same time. Even if both programs are displayed at the
`same time usually only one can be active. HoWever, to read
`different parts of the e-mail message the e-mail program
`must be the active program, While to move, adjust, or re-siZe
`an image, or to listen to a sound attachment ?le, the image
`or sound accessing program must be the active program.
`Often the only practical solution is to print out the text of an
`e-mail message on a computer, and then With the print out
`of the message in hand access the attached ?le on the
`computer using the appropriate accessing program.
`One can see that for both sound and image ?le attach
`ments the process of obtaining the ?le, converting the ?le to
`the proper format and using an accessing program to access
`the ?le can take many steps and be quite time consuming. In
`addition to being quite time consuming there is no guarantee
`that a user Will be able to decode an attached ?le, either
`because of lack of proper decoding hardWare or softWare, or
`because the process can often be so complex that the user
`may lose interest and simply decide it is not Worth the
`trouble.
`
`Snap's Exhibit No. 1061
`Page 16
`
`

`

`US 6,252,588 B1
`
`7
`Other Disadvantages of E-Mail
`The use of e-mail requires a certain level of computer
`knowledge and familiarity. An e-mail program can be quite
`overWhelming if the user is a person Who is easily intimi
`dated by computers. It can also be confusing to computer
`literate individuals Who have not had a great deal of e-mail
`experience or are presented With an e-mail program that they
`are not familiar With.
`The addresses of e-mail messages also can be quite
`confusing. Instead of having an easily understood form such
`as “Aunt Matilda from Seattle” e-mail addresses often take
`arbitrary and confusing forms such as “1654256@com
`puserve.com”. Because received e-mail messages are usu
`ally identi?ed by the e-mail address of the sender, an e-mail
`recipient Will often have to actually open a message to
`determine Who it is from. Even if the e-mail user success
`fully opens the mail message, the user may be confronted
`With a message that is contained Wholly or partly in an
`attached ?le. The attached ?le may have a voice message
`Within it, or a picture, or video that requires activation of a
`special accessing program before the entire message can be
`delivered.
`E-mail also sometimes presents a problem in the manner
`that e-mail messages are displayed. For example, one user
`may send another user a message in a font that, While
`readable on the sender’s machine, is dif?cult to read on the
`recipient’s machine. This can be the case even if each user’s
`e-mail program is from the same company, and the version
`is different. For example, an e-mail message sent by a user
`using a Japanese version of Microsoft Exchange may be
`unreadable When sent to a user using an American version of
`Microsoft Exchange.
`SUMMARY OF THE INVENTION
`The audio visual e-mail system presented here brings
`electronic mail Within the reach of everyone, no matter What
`their previous experience With computers and e-mail. It
`alloWs those Who have never used a computer to easily send
`messages to anyone With a similar system, or even to other
`existing systems.
`Th