United States Patent [191
`Wang et al.
`
`I|l|||llllllllIllllllllllll11111141111111"!llllHlllllllllllllllll
`
`[11] Patent Number:
`[45} Date of Patent:
`
`5,461,711
`Oct. 24, 1995
`
`[54] METHOD AND SYSTEM FOR SPATIAL
`ACCESSING OF TIME-BASED
`INFORMATION
`
`5,109,482
`5,191,645
`5,202,961
`5,388,197
`
`4/1992 Bohrman ............................... .. 395/154
`3/1993 Carlucci et al.
`395/159
`4/1993 Mills et a1.
`395/159
`2/1995 Rayner ...... ..
`395/154
`
`
`
`Inventors: weijia Wangj Sunnyvale; Sean M. White, San Francisco, both of Cal1f.
`
`
`
`5,404,444 Primary Examiner_Raymond J_ Baysrl Billings ................................. ..
`
`
`
`[73] Assignee: Interval Research Corporation, Palo
`
`Attorney’ Agent’ or Flrm—Bm0kS & Kushman
`
`A110, Cahf-
`
`[57]
`
`ABSTRACT
`
`_
`
`[21] APPL NOJ 172,637
`
`[22] Filed:
`
`Dec. 22, 1993
`
`6
`[51] Int. Cl. ........................................ ..: ........... .. (3061?‘ 3/02
`[52] US. Cl. ........................ .. 395/161,395/155,-395/154,
`_
`395/159’ 345/156’ 345/173
`[58] Field Of Search ................................... ,. 395/ 155, 154,
`395/ 161, 152, 119, 160, 159; 345/173,
`156’ 179’ 180
`
`[56]
`
`_
`References cued
`US PATENT DOCUMENTS
`
`A method and ‘system for accessing time-based- information
`based on spatial coordinate informatlon obtaIued from a
`user. Time-based information is mapped into a spatial rep
`resentation by mapping and transforming the timing of
`timebased information Segments to Spatial Coordinates_
`Such Spatial coordinates could be either one’ two’ or three
`dimensional. With this mapping, segments of time-based
`information can be associated with different parts of a Spatial
`object, a spatial representation, or different position of a
`spatial movement. These segments can be accessed by the
`user by physical inputs in such a fashion that the kinesthetic
`or touching memory of the user can be relied upon to
`r?-aegless information and to create a sense of the whole in
`e 1 ormation.
`
`6/1988 Blanton et a1. ................... .. 395/152 X
`4,752,836
`4,943,866 7/1990 Barker et a1.
`395/155 X
`4,964,004 10/1990 Barker ............................... .. 345/156 X
`
`44 Claims, 7 Drawing Sheets
`
`TIME-BASED N 20
`INFORMATION
`
`TlM|NG~TO-SPAT|AL _,\_/22
`CONVERTER
`
`26
`
`4'0 ACCESSED
`
`- — - - - - - >
`
`YNFoRMATIoN
`
`24w SPATIAL
`ACCESS DEVICE
`
`36
`
`USER INTERFACE
`DEVICE
`
`ACCESSED
`INFORMATION
`
`INPUT SPATIAL
`COORDINATE
`34
`
`38
`
`23\/\ PLAYBACK
`DEVICE
`
`SENSORY
`OUTPUT
`
`32
`
`I
`~42
`I
`ISENSORY
`I OUTPUT
`i
`USER
`
`30
`
`Samsung USP 7,973,773
` Exhibit 1010 Page 1
`
`

`
`US. Patent
`
`Oct. 24, 1995
`
`Sheet 1 0f 7
`
`5,461,711
`
`MAP THE TIMING OF
`TIME~BASED INFORMATION N10
`INTO SPATIAL COORDINATES
`
`t _ _ _ _ _ _ _ __
`I
`I
`I
`I
`I
`
`N18
`
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`DETERMINE INPUT M12
`SPATIAL COORDINATE
`
`l
`
`ACCESS INFORMATION
`CORRESPONDING TO INPUT N14
`SPATIAL COORDINATE
`
`I
`
`PLAYBACK ACCESSED
`INFORMATION
`
`16
`
`FIG. 1
`
` Exhibit 1010 Page 2
`
`

`
`U.S. Patent
`
`0a. 24, 1995
`
`Sheet 2 of 7
`
`5,461,711
`
`TIME-BASED EV 2o
`INFORMATION
`
`TlMlNG-TO-SPATIAL _,\/22
`CONVERTER
`
`40 ACCESSED
`
`INFORMATION '
`
`— - - - - - - >
`
`2e
`8
`
`36
`24¢ SPATIAL
`ACCESS DEVICE /
`
`usER INTERFACE
`DEVICE
`
`ACCESSED
`
`INFORMATION
`
`NP T PATIAL
`ICOLCJDRSDINATE
`34
`
`38
`
`l
`
`:»\/42
`1
`ISENSORY
`I OUTPUT
`+
`
`28
`\_/~
`
`PLAYBACK
`DEVICE
`
`I
`
`SENSORY
`OUTPUT
`
`32
`
`USER
`
`I
`
`30
`
`FIG. 2
`
` Exhibit 1010 Page 3
`
`

`
`US. Patent
`
`Oct. 24, 1995
`
`Sheet 3 of 7
`
`5,461,711
`
`,\_/50 /
`
`ASSIGN
`TIMING MARKS
`I
`
`MAP TIMING MARKS TO
`SPATIAL COORDINATES
`
`52
`l'\/
`
`FIG. 3
`
`54
`
`/
`
`/14
`
`COMPARE INPUT SPATIAL COORDINATE
`TO MAPPED SPATIAL COORDINATE
`
`FOUND A
`MATCHED SPATIAL
`COORDINATE
`
`NEXT SPATIAL
`COORDINATE
`
`I 56
`
`IDENTIFY SEGMENT CORRESPONDING
`TO MATCHED SPATIAL COORDINATE ~58
`
`FIG. 4
`
` Exhibit 1010 Page 4
`
`

`
`U.S. Patent
`
`Oct. 24, 1995
`
`Sheet 4 of 7
`
`5,461,711
`
`>mos_m_>_
`
`88o_o:<
`
`zo_»<sEou_z_
`
`mo<nEm:z_ow
`
`mo_>mo
`
`
`
`v.o<m><._n_O_QD<
`
`mo_>m_o
`
`8
`
`
`
`”mo_>mozo_»<_2mon.wz<m»
`
`
`
`
`
`mxm<s_oz_s_:.n_<_>_
`
`H<S_mOu_ONO._.Z_
`
`m.O_n_
`
`Exhibit 1010 Page 5
`
` Exhibit 1010 Page 5
`
`
`
`
`
`
`

`
`US. Patent
`
`0a. 24, 1995
`
`Sheet 5 of 7
`
`5,461,711
`
`FIG. 6
`
`i
`
` Exhibit 1010 Page 6
`
`

`
`US. Patent
`
`0a. 24, 1995
`
`Sheet 6 of 7
`
`5,461,711
`
`64\
`812
`(
`
`86
`
`88
`
`7/7
`//
`
`Lil‘? 90
`PAGE [2].
`
`‘ 19mm;
`
`HlGHLlGHT > 92
`
`/////////////,
`
`I
`
`S
`84
`
`E
`
`sTYLus > 94
`W
`
`7
`
`\ in
`
`PAGE w
`
`PAGE V PAGE X PAGE Y
`
`PAGE 2
`
`FIG. 9
`
` Exhibit 1010 Page 7
`
`

`
`US. Patent
`
`Oct. 24, 1995
`
`Sheet 7 of 7
`
`5,461,711
`
`100
`
`__g__+
`
`102
`
`FIG. 8a
`
`10°
`\
`
`100
`\ FIG. 80
`
`106
`
`FIG. 8b
`
`, , ,
`
` Exhibit 1010 Page 8
`
`

`
`5,461,711
`
`1
`METHOD AND SYSTEM FOR SPATIAL
`ACCESSING OF TIME-BASED
`INFORMATION
`
`TECHNICAL FIELD
`
`The present invention relates to a method and system for
`accessing time’based information based on spatial coordi
`nate information obtained from a user.
`
`BACKGROUND ART
`
`2
`e?icient method exists for prioritizing the information.
`2. Utilizing the characteristics of the information, such as
`the amplitude of the waveform of a recorded speech,
`and structural patterns to visually annotate the infor
`mation so that di?erent segments can be easily distin
`guished. Thus this approach relies upon an effective
`automatic feature extraction system. This approach is
`also di?icult to implement and not effective for all
`sources of time-based information.
`Other prior art systems have addressed the presentation of
`recorded speech based upon the prior knowledge of the
`structure of the underlying text.
`An example of one such
`system is the multimedia package “From Alice to Ocean”
`marketed by Apple Computers. This package presents the
`story of a woman’ s journey across the Australian desert. The
`text of the story, as read by the author, has been stored in
`convenient segments. These segments of text are accessed
`by a user along with corresponding visual images based
`upon visual cues displayed on a computer screen.
`A second prior art system of this kind is a CD-ROM
`game/reader, “Just Grandma and Me,” marketed by Broder
`bund. This package allows a user to play back prerecorded
`spoken words corresponding to the words of a story dis
`played on a computer screen.
`These systems and their underlying methods do not
`address the problem of accessing time-based information
`without the additional a priori information provided by the
`presence of the text. If the underlying text is known, the
`solution to the problem is trivialized since the text itself
`could be used to directly access the information, e. g. directly
`reading or text—to-speech translation. This is not an effective
`approach for time-based information which lacks a priori
`access to a textual interpretation or other similar script.
`
`SUMMARY OF THE INVENTION
`
`It is thus an object of the present invention to provide a
`method and system for permitting a user to spatially access
`and play back time-based information.
`It is also an object of the present invention to create a
`sense of the whole for accessing time-based information
`which is analogous to what peripheral vision creates in
`reading printed text.
`It is thus a further object of the present invention to
`provide a method and system for accessing time-based
`information by converting it to spatially distributed infor‘
`mation so that the information can be easily accessed and
`browsed.
`It is thus an additional object of the present invention to
`provide a method and system for converting or mapping
`time-based information to spatially distributed information.
`Moreover, an object of the present invention is to provide
`a method and system for accessing the spatialized informa
`tion in a way that is relevant to the time-based origins of the
`information.
`An additional object of the present invention is to provide
`a method and system which allow a user to utilize kines
`thetic and tactile memories to memorize the locations of
`segments of the spatialized information and obtain a sense of
`the whole such that the user can achieve an apparent parallel
`and arbitrary access of the information which resembles
`visual reading.
`A further object of the present invention is to provide a
`method and system of using physical inputs from a user
`which include any human body movement (e.g., ?ngers,
`tongue, head, shoulder, knees, toe, etc. .
`.
`) to scan a
`
`Absorbing and processing time~based information is more
`difficult than absorbing spatially distributed information.
`This is, in part, due to the difficulty in creating a “sense of
`the whole” from time-based information. For instance, it is
`a common experience that visual reading of printed text is
`much easier and faster than listening to recorded speech. The
`focus of the eye (fovea vision) can move quickly from one
`place to another; the peripheral vision fuzzily views a large
`area around the fovea so a reader knows where the fovea
`vision is with respect to the rest of the printed text (e.g., the
`rest of page, the rest of the sentence, the rest of the word) at
`any given moment. It is the peripheral vision that enables the
`fovea vision to locate quickly the places the reader wants to
`examine. Therefore, visual reading can be thought of as
`“instant arbitrary accessing”.
`In contrast the conventional devices which people use to
`access time-based information, for instance, an audio and/or
`video player, only display such information sequentially
`(e.g., playing back recorded audio or video frames). In other
`words, the conventional devices present time-based infor
`mation only along a time axis. Since time is sequential, at
`any moment, the person who is accessing such information
`is exposed to only a small segment of the information. Given
`that a human’s temporal memory is short and has poor
`resolution of time, it
`is di?icult for users to
`determine the
`relative timing mark of a particular information segment
`with respect to the entire piece of information being dis
`played or accessed.
`Even if the underlying devices allow users to quickly
`move from one part of the speech to another (e.g., a digital
`storing device that allows random access), a listener nor
`mally would not do so, simply because he/she doesn’t know
`where he/she is with respect to the entire information at any
`moment, thus he/she doesn’t know where to move. Thus the
`di?iculty of accessing time~based information stems chie?y
`from the lack of the capability to generate a sense of the
`whole due to the combination of characteristics of human
`sight, hearing, memory, and the way conventional devices
`work. This temporal limitation makes it much more difficult
`to construct a general outlook of the set of information.
`If time-based information, such as recorded speech, could
`be translated precisely into printed text or some other spatial
`visual codes such as pictures, accessing the time-based
`information could
`be turned into reading. However, auto
`matic speech recognition is far from perfect. Thus alterna
`tive ways to absorb and present recorded speech are needed.
`Many prior art systems have attempted to solve this
`problem. Most of the previous research has emphasized one
`of the following approaches:
`extracting important
`1. Condensing the information by
`
`(situation dependent) portions or by throwing out insig
`ni?cant portions. This approach is not effective in all
`circumstances since it assumes that some of the time
`based information, is in fact, insigni?cant and that an
`
`20
`
`25
`
`30
`
`35
`
`45
`
`65
`
` Exhibit 1010 Page 9
`
`

`
`5,461,711
`
`3
`spatialized representation of time-based information
`A more speci?c object of the present invention is to
`provide a method and system for permitting a user to
`spatially access and play back time-based information, by
`physical inputs, in such a manner that the human spatial
`memory of the user can be relied upon to re-access infor
`mation previously accessed. In this manner, the information
`can be skimmed by the user to determine portions of the
`information which warrant a more careful scanning.
`In carrying out the above objects, the present invention
`provides a method for permitting a user to spatially access
`and play back time-based information having a plurality of
`segments. The method includes the step of mapping each of
`the plurality of segments of the time-based information to a
`corresponding one of a plurality of
`mapped spatial coordi
`nates. Next an input spatial coordinate is determined, cor
`responding to a ?rst physical input of the user generated by
`a movement of a portion of the user’ s body to a ?rst position.
`A ?rst segment of the time-based information whose
`mapped spatial coordinate corresponds to the input spatial
`coordinate is then accessed, and the ?rst segment is played
`back to the user. After the portion of the user’s body has
`moved to a second position, a human spatial memory of the
`user can be used to return the portion of the user’s body to
`the ?rst position to re-access and play back the ?rst segment
`of time-based information by repeating the steps above.
`In carrying out the above objects, the present invention
`further provides a method for permitting a user to spatially
`access and play back time-based information having a
`plurality of segments. The method includes assigning a
`sequence of corresponding timing marks to the time-based
`information which denote the passage of time in the time
`
`based information, forming the plurality of
`segments based
`on the sequence of timing marks, and mapping each of the
`sequence of
`timing marks into a corresponding one of a
`plurality of mapped spatial coordinates.
`An input spatial coordinate is determined which corre
`sponds to a ?rst physical input of the user generated by a
`movement of a portion of the user’s body to a ?rst position.
`A ?rst segment of the time-based information whose
`mapped spatial coordinate corresponds to the input spatial
`coordinate is accessed. This accessing is performed by
`comparing the input spatial coordinate to one or more of the
`plurality of mapped spatial coordinates to identify a mapped
`spatial coordinate which matches the input spatial coordi
`nate. The ?rst segment is identi?ed as the segment which
`corresponds to the matched mapped spatial coordinate based
`on the corresponding timing mark. The ?rst segment is then
`played back to the user. After the portion of the user’s body
`has moved to a second position, a human spatial memory of
`the user can be used to return the portion of the user’s body
`to the ?rst position to re-access and play back the ?rst
`segment of time-based information by repeating the steps
`above.
`In carrying out the above objects, the present invention
`further provides a system for implementing each of the
`methods described.
`The objects, features and advantages of the present inven
`tion are readily apparent from the following detailed
`description of the best mode for carrying out the invention
`when taken in connection with the accompanying drawings.
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`25
`
`30
`
`35
`
`50
`
`55
`
`60
`
`FIG. 1 is a ?ow-chart representation of one embodiment
`of the me of the present invention,
`FIG. 2 is a block-diagram representation of one embodi
`ment of the system of the present invention,
`
`65
`
`4
`FIG. 3 is a ?ow-chart representation of the method of
`transforming time-based information into spatial-based
`information of one embodiment of the present invention,
`FIG. 4 is a ?ow-chart representation of a method for
`accessing time-based information corresponding to an input
`spatial coordinate of one embodiment of the present inven
`tion.
`FIG. 5 is a block-diagram representation of an alternative
`embodiment of the system of the present invention.
`FIG. 6 is a pictorial representation of one embodiment of
`the system of the present invention which emphasizes a
`two-dimensional touch sensitive user interface device.
`FIG. 7 is a pictorial representation of an alternative
`two-dimensional user interface device for use with one
`embodiment of the system of the present invention.
`FIGS. 8a-c present scanning patterns to be used with one
`embodiment of the system of the present invention. alter
`native two-dimensional user-interface devices for use with
`FIG. 9 presents a spatial multiple page structure used with
`one embodiment of the present invention.
`While the invention will be described in connection with
`the several embodiments, it will be understood that this
`description is not intended to limit the invention to these
`embodiments. On the contrary, the invention is intended to
`cover all alternatives, modi?cations and equivalents as may
`be included within the spirit and scope of the invention as
`de?ned by the appended claims.
`
`BEST MODE FOR CARRYING OUT THE
`INVENTION
`
`presented by
`The present invention solves the problems
`
`the prior art in accessing time-based information by map
`ping and transforming the timing of time-based information
`segments to spatial coordinates. Such spatial coordinates
`could be any Cartesian or non-Cartesian coordinates in one,
`two, or three dimensional Euclidean or non-Euclidean space.
`With this mapping, segments of time-based information can
`be associated with different parts of a spatial object, a spatial
`representation, or a different position of a spatial movement.
`It should be understood that the term, “time-based infor
`mation," as used herein should be interpreted broadly to
`include, but not be limited to, any general signal or collec
`tion of data which can be expressed in temporal form.
`Examples include audio signals, whether analog or digital,
`such as speech, music, and MIDI signals; visual informa
`tion, such as video, animation and other graphical informa
`tion; and other temporal data such as electromagnetic sig
`nals, stock closing information, daily temperature data, etc.
`Turning now to FIG. 1, a ?ow-chart representation of one
`embodiment of the present invention is presented. The
`
`timing of time-based information, such as audio, video, or
`any other time-based signal, is mapped to a set of spatial
`coordinates as shown in step 10. Various methods are
`possible for mapping the timing of time-based information
`to spatial coordinates. These include, but are not limited to:
`mapping continuous timing marks of time-based informa
`tion to spatial coordinates, mapping sequential numbers of
`words (in a piece of speech) to spatial coordinates, and
`mapping other structural formations found in time-based
`information to spatial representations.
`These spatial coordinates may have any number of
`degrees of freedom. Thus, the transformation of the present
`invention may map the time-based information into three
`coordinates in a three-dimensional space, map the time
`
` Exhibit 1010 Page 10
`
`

`
`5,461,711
`
`10
`
`5
`based information into three coordinates of position and
`three coordinates of orientation of a geometric solid in
`three-dimensional space, or map the time-based information
`into a set of positions and orientations for a series of
`connected joints in
`a three’dimensional space.
`An input spatial coordinate is then determined based upon
`a physical input from the user generated by the movement of
`the user’s body as shown in step 12. This step could be
`performed by sensing the position, velocity, acceleration
`and/or orientation of one or more portions or
`joints of the
`user’ s body. Examples include but are not limited to utilizing
`multiple ?nger positions on a position~digitizer pad that can
`convert ?ngers’ position on the pad to a set of spatial
`coordinates, positions generated by a hand controller or
`tongue-controlled positioning device, or positions generated
`by an interface device which responds to head movement.
`Interface devices of this sort are commonly used in virtual
`reality applications.
`The determined input spatial coordinate must be in the
`same coordinate space as the mapped spatial coordinates of
`the timebased information. Thus, each segment of time
`based information would correspond to a possible input
`spatial coordinate.
`A segment of the time-based information whose mapped ‘
`spatial coordinate corresponds to the input spatial coordinate
`is then accessed as shown in step 14, and the accessed ?rst
`segment of time-based information is then played back to
`the user as shown in step 16. At this point the method may
`end, or optionally, may continue by determining a new input
`spatial coordinate accessing further information correspond
`ing to the new input spatial coordinate and playing back this
`new accessed information as shown by path 18 and steps 12,
`14, and 16. By this means, after the portion of the user’s
`body has been moved to a subsequent position, human
`spatial memory can be used to return the user’s body to the
`?rst position to re-access and play back the segment of
`time-based information. This human spatial memory could
`be kinesthetic memory, the memory that memorizes spatial
`positions of human physical movement through sensing
`muscle tension. The term “Human Spatial Memory” should
`be broadly constituted to also cover other spatial memories
`of the user such as touching memory of the user, the memory
`that memorizes spatial
`locations on a portion of a human
`body (e.g., a palm) where skin is stimulated by touching.
`In this fashion, the sense of the whole capability of
`peripheral vision provided in visual reading by human
`memories can be emulated by spatial movement and posi
`tioning of the user’s body. This emulation may occur by
`utilizing a ?nger’s kinesthetic memory, a palm’s tactile and
`touching memory, a tongue’s kinesthetic memory, or other
`types of human spatial memory based upon the choice of
`interface device and based upon the choice of possible input
`spatial coordinates.
`FIG. 2 presents a block-diagram representation of one
`embodiment of the system of the present invention. The
`timing of time-based information 20, having a plurality of
`segments, is mapped to a set of spatial coordinates by
`timing-to-spatial converter 22. This converter could be
`implemented by a programmable logic array or discrete
`logic device, or by an analog computational device. How
`ever, in the preferred embodiment, this timing-to-spatial
`conversion is performed by software or ?rmware running on
`a central processing unit of a computer, such as a micro
`processor.
`User interface device 26 determines an input spatial
`coordinate corresponding to a physical input generated by
`
`6
`the user 30. Arrow 34 represents the physical input from user
`30 to user interface device 26. User interface device 26 is
`also capable of providing a sensory output or feedback 42 to
`user 30. This sensory output may take the form of some
`vibration, touch, temperature, or other sensory information
`which corresponds to the accessed time-based information
`and which aids the user in obtaining a sense of the whole of
`the time-based information in input coordinate space.
`User interface device 26 in turn provides the determined
`input spatial coordinate 36 to spatial access device 24. This
`spatial access device accesses a segment of time-based
`information whose mapped spatial coordinate corresponds
`to the input spatial coordinate. This accessed information 38
`is transferred from spatial-access device 24 to playback
`device 28 in order to provide a sensory output 32 to user 30.
`Sensory output 32 of playback device 28 is provided in a
`manner which emulates the form of the original form of the
`time-based information. Thus, if the original information is
`audio, the playback device plays back this audio information
`to the user as sound. Further, if the time-based information
`is graphical information such as video or animation, then the
`playback device plays back the time-based information to
`the user in the form of graphical images. These sensory
`outputs 32 of playback device 28 should be contrasted with
`the sensory output 42 of user interface device 26, whose
`purpose is to aid users in accessing the information. The
`sensory output 42 would generally be in a form which differs
`from the original form of the time-based information.
`Examples of such sensory output include vibration and other
`tactile output as well as supplementary audio and visual cues
`which do not emulate the time-based information.
`Turning now to FIG. 3, one method is presented for
`performing the step of mapping the timing of the time-based
`information into a set of spatial coordinates, as shown in step
`10 of FIG. 1. The time~based information is assigned a series
`of sequential timing marks as shown in step 50. These timing
`marks could be equally spaced in time, or unequally spaced.
`If the timing marks were unequally spaced, they could be
`arranged based upon the structural attributes of the infor
`mation. For example, in the case of speech, the timing marks
`could be used to set boundaries between syllables, words,
`sentences, paragraphs, or pages of spoken words thereby
`forming segments of time-based information. These timing
`marks are then each mapped into a corresponding one of a
`plurality of mapped spatial coordinates which in turn cor
`respond to the set of possible input spatial coordinates as
`shown in step 52.
`The segments of the time-based information which are
`accessed may correspond to the time-based information
`between two successive timing marks. In this case, the
`segments are non-overlapping. This need not be the case for
`all applications. For instance, a segment corresponding to a
`given timing mark may be the time based information from
`the preceding timing mark to the subsequent timing mark.
`Further, ?ltering or windowing may be used to allow a
`segment to fade in at the beginning and/or fade out at the
`end.
`Turning now to FIG. 4, a ?ow-chart representation of one
`embodiment of step 14, accessing information correspond
`ing to an input spatial coordinate, as shown in FIG. 1, is
`presented. The input spatial coordinate is compared to a
`mapped spatial coordinate as shown in step 54. If the input
`spatial coordinate matches the mapped spatial coordinate,
`then a segment of the time-based information is identi?ed
`which corresponds to the matched storage spatial coordinate
`as shown in step 58. If the input spatial coordinate does not
`match the mapped spatial coordinate, then a next-mapped
`
`35
`
`40
`
`55
`
`65
`
` Exhibit 1010 Page 11
`
`

`
`5,461,711
`
`7
`spatial coordinate is chosen for the comparison as shown in
`step 56, and step 54 is repeated until a match is found.
`To summarize, the three major principles behind this
`invention are: (i) mapping the timing of time-based infor
`mation to a set of spatial coordinates; (ii) utilizing kines
`thetic and touching memory to help construct a sense of the
`whole for accessing the information; and (iii) using various
`modes of interaction to aid users in moving around in the
`information and controlling the transformation. A general
`procedure for constructing an embodiment of the present
`invention is given as follows:
`(I) Choose the form of physical movement and physical
`input to signify an input spatial coordinate. In particu
`lar, any moveable portion of the human body can be
`used to generate a physical input.
`(2) Decide what patterns of movement to use.
`(3) Choose a mapping from the timing marks of the
`time-based information to the spatial coordinates that
`the moving pattern traverses. At this stage, physical
`(e.g., electronically) devices are involved to sense the
`spatial movement and display the sound according to a
`given time-space mapping.
`Turning now to FIG. 5, a block diagram representation of
`an alternative embodiment of the system of the present
`invention is presented. Given that the ear is the second best
`information-absorbing sensory organ (next to the eyes), in
`this embodiment of the present invention, we consider using
`the ears to listen to sound (as opposed to converting sound
`into the media perceived by other sensory modality, e.g.,
`tactile). To move from part of spatially distributed sound to
`another, one can use any portion of the human body that can
`move spatially. One of the most convenient portions of the
`body is the hand, i.e., palm and ?ngers. Fingers can move
`quickly enough to match the speed with which the cars can
`listen. In this embodiment of the present invention, the
`kinesthetic and touching memories associated with one or
`more ?ngers, coupled with listening, are used to create a
`sense of the whole, similar to an “overview” capability in
`reading.
`This embodiment of the present invention uses a touch
`sensitive pad to provide input spatial coordinates in a
`two-dimensional space. The timing of time-based informa
`tion is converted to two-dimensional spatialized data which
`is then browsed and accessed with ?ngers. Timing marks are
`mapped onto a page format, i.e., sound are distributed on the
`surface in rows like printed text. The ?ngers provide a
`source of kinesthetic and tactile memory which allow the
`user to create a representation of the information.
`More particularly, the timing of audio information 60 is
`transformed into a two-dimensional spatial format by trans
`formation device 62. This transfonnation device creates a
`series of timing marks in the audio information and maps
`these timing marks into a set of two-dimensional coordi
`nates. A user 69 accesses this audio information by means of
`?nger 66. The device that senses ?nger position and move
`ment is two-dimensional interface device 64, a touch-sen
`sitive pad which generates a two~dimensional coordinate of
`the position where the hand is touching the surface. These
`coordinates are transferred from interface device 64 to
`transformation device 62 to determine the audio information
`which corresponds to these two~dimensional coordinates.
`This audio information is in turn transferred back to inter
`face device 64, which provides a vibration feedback to ?nger
`66. This vibration feedback is implemented by providing a
`vibratory output by means of an activator or speaker (not
`shown) attached to the interface device 64. The vibration
`output is in proportion to an audio signal created from the
`
`10
`
`15
`
`20
`
`25
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`8
`accessed audio information. The accessed audio information
`is further provided to audio playback device 68, which plays
`back the accessed audio information to the ears 67 of user
`69. The memory 65 of user 64 guided by touch and memo
`rized position of the user’s ?nger in “the whole” allows the
`user to re-access a previously accessed portion of audio
`signal.
`Note thatthere are many spatial distribution patterns of
`the sound on such a surface. Examples include:
`(i) Playing back only the segment which is corresponding
`to the spatial location the ?nger is pointing to;
`(ii) Playing back a certain length of sound starting from a
`segment which corresponds to the spatial location of
`the ?nger;
`(iii) Playing back a certain length of sound starting from
`the signi?cant place (such as a beginning of a word)
`which is the nearest to the segment which corresponds
`to the ?nger location; or
`(iv) In both (ii) and (iii), when playing back the sound, the
`sound volume fades with time or is ?ltered in
`some
`form.
`FIG. 6 presents a pictorial diagram which represents the
`operation of the system of FIG. 5. Audio information 60 is
`stored in computer 74. Computer 74, in turn, assigns a
`sequence of timing marks which denote the passage of time
`in the audio information 60. The segments of audio infor
`mation between each successive pair of timing marks are in
`turn mapped to a set of two-dimensional coordinates which
`correspond to the two-dimensional coordinates of touch
`sensitive pads 70 of two-dimensional interface device 64. A
`particular two-dimensional coordinate is accessed by ?nger
`66 touching that coordinate. This ?nger position, in the form
`of an input spatial coordinate, is fed to computer 74, which
`accesses the corresponding segment of audio information
`and plays back that information to speaker 78, which gen
`erates an acoustic output. Further, computer 74 sends the
`accessed segment of audio information to interface device
`64, which vibrates in proportion to the accessed audio
`segment in a manner which is felt by ?nger 66. In this
`application, the ?nger provides a source of kinesthetic and
`tactile memory which allows the user to create a sense of the
`whole in the information.
`The operation of the system of FIGS. 5 and 6 is perhaps
`best described in terms of the following hypothetical appli
`ca