11111111111111111111111111111
`1111111111111111111111111111
`i
`M
`5,367,621
`[11] Patent Number:
`[45] Date of Patent: Nov. 22, 1994
`
`1i1
`12
`171
`E1
`10
`101
`
`United States Patent [19]
`Cohen et al.
`
`[54] DATA PROCESSING METHOD TO
`PROVIDE A GENERALIZED LINK FROM A
`REFERENCE POINT IN AN ON-LINE BOOK
`TO AN ARBITRARY MULTIMEDIA OBJECT
`WHICH CAN BE DYNAMICALLY UPDATED
`[75] Inventors: Amy S. C. Cohen, Silver Spring;
`Christopher F. Gleason,
`Gaithersburg; Donald R. Hyatt,
`Silver Spring, all of Md.; Michael E.
`Moran, Woodstock, N.Y.; Jeffrey N.
`Stevens, Gaithersburg, Md.; Alan J.
`Wecker, Haifa, Israel
`International Business Machines
`Corporation, Armonk, N.Y.
`755,709
`Sep. 6, 1991
`
`[73] Assignee:
`
`
`
`
`GO6F 15/62
`395/154; 395/144;
`395/148; 395/275; 395/600
` 395/153, 154, 275, 600,
`[58] Field of Search
`395/148, 144; 340/717, 721, 345; 345/1, 2;
`364/282.1
`
`[21] Appl. No.:
`[22] Filed:
`[51] Int. C1 5
`[52] U.S. Cl.
`
`[56]
`
`References Cited
`U.S. PATENT DOCUMENTS
` 364/900
`4,435,779 3/1984 Mayer et al.
` 340/721
`4,692,757 9/1987 Tsuhara et al.
` 364/900
`4,815,029 3/1989 Barker et al.
` 364/900
`4,829,470 5/1989 Wang
` 364/518
`4,893,256 1/1990 Rutherfoord et al.
` 364/200
`4,914,586 4/1990 Swinehart et al.
` 364/513
`4,931,950 6/1990 Isle et al.
` 364/523
`4,933,880 6/1990 Borgendale et al.
` 364/521
`4,939,672 7/1990 Meadows
` 382/56
`4,941,193 7/1990 Barnsley et al.
` 364/518
`4,967,375 10/1990 Pelham et al.
` 364/900
`4,974,194 11/1990 Barker et al.
` 364/900
`5,008,853 4/1991 Bly et al.
`395/154
`5,109,482 4/1992 Bohrman
`FOREIGN PATENT DOCUMENTS
`4157962 8/1990 European Pat. Off. .
`8811925 5/1989 Japan .
`OTHER PUBLICATIONS
`Gibson, et al., "Set of Algorithms for Defining, Query-
`
`•••••••••At
`
`ing, . . . " vol. 30, No. 8, Jan. 1988, pp. 123-127, IBM
`TDB.
`Kendrick, et al., "Method for Supporting Non-Text
`Objects in ... " IBM TDB, vol. 30, No. 8, Jan. 1988, pp.
`155-162.
`Elliott, et al., "Technique for Retaining, Saving and . .
`. " IBM TDB, vol. 30, No. 8, Jan. 1988, pp. 163-164.
`J. Nielsen, "Hypertext II", SIGCHI Bulletin, vol. 21,
`No. 2, Oct. 1989, pp. 41-47.
`vol. 21, No. 1, pp. 65-67 J. Nielsen, "Hyperhyper: De-
`velopments Across the Field of Hypermedia-A Mini
`Trip Report: BCS Workship, London, UK, 23 Feb.
`1989, " SIGCHI Bulletin.
`(List continued on next page.)
`Primary Examiner—Heather R. Herndon
`Assistant Examiner—Joseph H. Feild
`Attorney, Agent, or Firm—John E. Hoel; Edward H.
`Duffield
`ABSTRACT
`[57]
`A generalized link is provided in a data processing sys-
`tem to enable referencing from a point within an orga-
`nized hierarchy of an on-line softcopy text, to an arbi-
`trary multimedia object. The multimedia object may be
`represented by data which is also contained within the
`softcopy book or alternately which may be separate
`from the book as a separate file or a separate data base.
`This enables an author at the time of writing the soft-
`copy book, to specify specific multimedia hardware and
`software support for the display of multimedia presenta-
`tions accompanying the text in the book. Provision is
`made for the author to specify alternate forms of multi-
`media presentation, where a particular specified multi-
`media apparatus or supporting software is not present at
`a workstation. An improved method for dynamically
`updating a softcopy book to a new edition is also dis-
`closed.
`
`8 Claims, 18 Drawing Sheets
`
`1111•••••••01.11,
`igim•••••18A
`1••••••••••••11.1
`
`*O.
`WWII*
`
`111•Ii0•10um.• Wuce=7:—
`
`Mg-
`
`••••••••••••=••••••••
`
`NFLE Ex. 1020 - Page 1
`
`NFLE 1020 - Page 1
`
`

`

`5,367,621
`Page 2
`
`OTHER PUBLICATIONS
`
`J. Nielsen, "Trip Report: Hypertext '89," SIGCHI Bul-
`letin, vol. 21, No. 4, Apr. 1990, pp. 52-61.
`A. Pearl, "Sun's Link Service: A Protocol for Open
`Linking", Hypertext '89 Proceedings, Nov. 1989, pp.
`137-146.
`"An Overview of Hypertext and Hypermedia," Con-
`cepts & Issues, EP10-010-701, Nov. 1989.
`
`"Multimedia Authoring Systems," PC Magazine, Jul.
`1990, pp. 163-192.
`E. Bender, "Desktop Multimedia: You Ain't Seen
`Nothing Yet," PC World, Mar. 1990, pp. 191-196.
`S. Ditlea, "Hypted," PC/Computing, Oct. 1990,
`201-210.
`Beauliev et al., "Hierarchy in Picture Segmentation: A
`Stepwise Optimization Approach", IEEE Transactions
`on Pattern Analysis and Machine Intelligence, Feb.
`1989, pp. 150-163.
`
`NFLE Ex. 1020 - Page 2
`
`NFLE 1020 - Page 2
`
`

`

`U.S. Patent
`
`Nov. 22, 1994
`
`Sheet 1 of 18
`
`5,367,621
`
`FIG. 1
`
`100
`FILE INDEX 10.5
`LINK DESCRIPTION TAGS 102
`BOOK TEXT WITH TAGS Ea
`INTERNAL ANIMATION OBJECT 106
`INTERNAL AUDIO OBJECT IN
`INTERNAL GRAPHICS OBJECT DI
`
`
`
`(FIG. la)
`(FIG. lb)
`(FIG. 1c
`(FIG. 1d)
`(FIG. le)
`
`FIG. 1 c
`
`106 —\
`OBJECT=lamily_dip.and Motion Picture in Bit Map Animation Sequence
`1111111111111111
`1111111111111111 1111111111111111 1111111111111111
`1100111111111111 1111111111111111 1111111111111111
`1111111111111111
`1111111111111111
`1111001111111111 1111001111111111 1111111111111111
`1111111111001111
`1111110011111111 1111110011111111 1111110011111111
`1111111100111111 1111111100111111 1111111100111111
`1111111100111111
`1111111111001111
`1111111111111111 1111111111001111 1111111111001111
`1111111111111111 1111111111111111 1111111111110011
`1111111111110011
`1111111111111111 1111111111111111 1111111111111111
`1111111111111111
`
`180a
`
`180b
`
`180c --)
`
`180i
`
`• • •
`
`108
`
`OBJECT=lrumpet.aud'
`0001 0010 0011 0100 0101 0110
`0110 0101 0100 0100 0101 0110
`0100 0010 0010 0010 0001 0000
`
`FIG. id
`Audio Data 184
`0111 0111
`0110 0110 0101
`0000 0000 0000
`
`NFLE Ex. 1020 - Page 3
`
`NFLE 1020 - Page 3
`
`

`

`U.S. Patent
`
`Nov. 22, 1994
`
`Sheet 2 of 18
`
`5,367,621
`
`FIG. 1 a
`
`102
`
`:LDESC ID-leptLmovle OBJTYPE=vIdeo
`
`OBJECT=lamlly_cilp.vicis
`STORE=external
`DATA='video.exe CD Video File Format A'
`RUNTDR='African Elephant Family Video'
`
`AUTO1AUNCH=no
`VIEW=center
`STYLE=2
`
`ALT OBJECTassfamily clip.anms
`ALT-STORE=Internal
`ALT DATA=AAnimation; 100 frames; B/W bttmap'
`ALT TEXT='Aflican Elephant Family Animation'
`
`ALT TEXT2=1Video not available'
`
`:LDESC ID=eleph_sotmd OBAYPE=audlo
`
`OBJECT-'trumpet.aud'
`STORE=Intemal
`DATA="auclo.exe DeltaMod File Format B'
`RUNTD(TdUsten to the Elephant lumpet'
`
`AUTOIAUNCH=no
`
`ALT TD(T2=1Audlo not available'
`
`:LDESC ID=pop_graphic
`
`OBJTYPE=graphic
`
`OBJECTsepopulation.gphs
`STORE=Internal
`DATA='graph.exe GOCA Format C'
`
`AUTOIAUNCH=yes
`VIEW=bottom
`SlYLE=2
`
`ALT TEXT2='Graphics not available'
`
`120
`4-1
`
`140
`4-1
`
`150
`4-1
`
`NFLE Ex. 1020 - Page 4
`
`NFLE 1020 - Page 4
`
`

`

`U.S. Patent
`
`Nov. 22, 1994
`
`Sheet 3 of 18
`
`5,367,621
`
`FIG. lb
`
`104
`
`:H1.AFRICAN WILDLIFE.
`
`:P.Studies have been conducted over the last decade on the
`changing populations of wildlife around the world. Inroads
`made by civilization have drastically reduced the habitats
`for Indigenous species. A typical population undergoing such
`changes Is that of the elephant.
`
`:L LlNeleph_movie.Motion Picture of African Elephant family:eL
`
`:P.The elephant Is an endangered species. The members of the
`family Elephantidae are indigenous to both Asia and Africa.
`The African elephant Is tall, large-eared mammal of
`tropical Africa. The Indian elephant Is a relatively
`small-eared mammal of the forests of southeast Ma.
`Listen to the sound of an elephant's trumpet.
`
`:L UD--eleph sound..Audio 1- Elephant's TrumpeteL.
`
`:P.Several International organizations have focused attention on
`the plight of the world's elephant populations.
`
`:L L1D=pop_graphIc.Rgure 1: Worldwide Elephant PopulatIon:eL
`
`:P.The graph in Figure 1, produced by the Serengeti Foundation
`of Tanzania, Is based on their comprehensive surveys of wildlife
`parks and zoos around the world. The graph shows the growth
`of zoo populations for the elephant compared to the growth of
`wild populations in Ma and Africa.
`
`160
`
`162
`4-1
`
`164
`4-1
`166
`4-1
`
`168
`4-J
`170
`4-1
`
`172
`
`174
`4-1
`
`NFLE Ex. 1020 - Page 5
`
`NFLE 1020 - Page 5
`
`

`

`U.S. Patent
`
`Nov. 22, 1994
`
`Sheet 4 of 18
`
`5,367,621
`
`FIG. 2
`195
`
`OBJECT='family_clip.vid'
`
`CD Video File FormatD
`
`• • • I
`
`FIG. 1 e
`
`1107
`
`Graphic Data 190)
`OBJECT='population.gph'
`JD element (Row 1. CoI1) to (Row 2. Col 2)
`E01 Horizontal Size (5700)
`E02 Vertical Size (1300)
`E03 Vector (300,500) to (1100,500)
`E04 Vertical String (300,300) "Elephants"
`E05 Box (800,700) to (1100,900)
`E06 Horizontal String (700,700) "Asia"
`E07 Vector (1100,500) to (1100,3400)
`E08 Box (600,1600) to (1100,1800)
`E09 Horizontal String (500,1400) "Africa"
`E10 Box (300,2700) to (1100,2900)
`Ell Horizontal String (200,2600) "Zoos"
`El 2 Horizontal String (1200,600) "Figure 1: Worldwide Elephant Population"
`E13 Horizontal String (300,3900) "Data:"
`E14 Horizontal String (400,3900) "Serengeti"
`El 5 Horizontal String (500,3900) "Foundation"
`El 6 Horizontal String (600,3900) "Survey"
`
`NFLE Ex. 1020 - Page 6
`
`NFLE 1020 - Page 6
`
`

`

`U.S. Patent
`
`Nov. 22, 1994
`
`Sheet 5 of 18
`
`5,367,621
`
`/ p.m row
`
`rm.
`
`rams
`
`Ina=
`
`8
`
`/=••1
`
`r•loa
`
`row
`
`...... .111=1••
`
`IMMO j11••••r r11.1••
`
`"MN h.
`
`fai
`
`"Me
`
`ran IMMO
`
`IIMM FINSO
`roma ram rim
`
`P"'▪ "
`
`IMMO
`
`a
`
`a
`
`•
`
`•
`
`a
`I
`I
`I
`
`a
`$
`I
`I
`
`N
`
`NFLE Ex. 1020 - Page 7
`
`I..' 8 I
`r_ r- r- Ir.. r....
`E3
`'r— r
`r..... r.... ram IlMalk
`......
`ir.
`11.
`an rum r r IMMO
`,_ ,_ ,_
`p_ 8
`f ... r... r.—
`1,....
`Q
`r_ r_ r
`rm. r ,......
`P... r r ,..... p_ , ,_
`
`IMIIM
`
`r
`
`r
`
`r—
`
`r.•
`
`• r—
`
`r— r—
`r— r. , r— r.... r r— er-
`1...•, r— r....
`r— r— ir—, r—.
`1... r r...... 11. r r— r...
`,_ ,_ r_ ,_
`..... tr,...
`r.,—
`8
`1... r"..... 1.... e—
`P... r•••• #...
`I_ r_
`8
`rm... row , ,_ r_ r ,_
`. ir _
`„, r_
`
`Immo ii. rime roam
`
`......
`
`.0
`0
`c0
`
`...... 1111111,
`
`IR,, r.m,
`',MOM r.... ii--, r—,
`. 1.... I.... r—
`r,,,.. ...•. ram eon
`rINIMF r f.... r I.•
`N... r— r I... ,-- r,... ro
`3 r,....
`r 8 r—, r.... 1..., r,... f•
`,_ 8 r...,
`
`1....
`
`f.
`
`NFLE 1020 - Page 7
`
`

`

`wawa 'STE
`
`•AoN
`
`1'661
`
`81 JO 9 PailS
`
`0001 0000 0000 0000 0000
`0101 0110 0110 0110 0101
`0101 0110 0111 0111 0111
`
`0100 0010 0010 0010
`0110 0101 0100 0100
`0001 0010 0011 0100
`
`AUTO DATA 184
`
`FIG. 3b
`
`SOUND SIGNAL 184'
`SAMPLED ANALOG /
`
`8 off Ed - Not .X1 aldm
`
`NFLE 1020 - Page 8
`
`

`

`U.S. Patent
`
`Nov. 22, 1994
`
`Sheet 7 of 18
`
`5,367,621
`
`FIG.
`3c
`
`E04
`,4)
`
`Africa
`
`Ele-
`phants Asia
`
`Zoos
`
`
`
`E142
`
`Data:
`Serengeti
`Foundation
`Survey
`
`DISPLAYED
`GRAPHICS
`190'
`
`Figure 1: Worldwide Elephant Population
`
`-E12
`
`Graphics Data 190
`in element (Row 11Co11) to (Row 2. Col 21
`E01 Horizontal Size (5700)
`E02 Vertical Size (1300)
`E03 Vector (300,500)to (1100,500)
`E04 Vertical String (300,300) "Elephants"
`E05 Box (800,700) to (1100,900)
`E06 Horizontal String (700,700) "Asia"
`E07 Vector (1100,500) to (1 1 00,3400)
`E08 Box (600,1600) to (11-00,1800)
`E09 Horizontal String (500,1400) "Africa"
`E10 Box (300,2700) to (1100,2900)
`El 1 Horizontal String (200,2600) "Zoos"
`E12 Horizontal String (1200,600) "Figure 1: Worldwide Elephant Population"
`E13 Horizontal String (300,3900) "Data:"
`E14 Horizontal String (400,3900) "Serengeti"
`El 5 Horizontal String (500,3900) "Foundation"
`E16 Horizontal String (600,3900) "Survey"
`
`NFLE Ex. 1020 - Page 9
`
`NFLE 1020 - Page 9
`
`

`

`wawa •s-fl
`
`t661 `ZZ 'sox
`
`81 JO 8 PeqS
`
`•••, =MEM. Ma'
`
`111.110
`
`all0•111•
`
`MINN
`
`•••••••
`
`AWN
`
`LINK BUFFER 234
`
`ALT TEXT2=1Audio not available'
`AUTOLAUNCH=no
`RUNTEXT=listen to the Elephant Trumpet'
`DATA='audio.exe \ DettaMod File Format B'
`STORE=Internal
`OBJECT=strumpet.aud'
`OBJ1YPE=audio
`
`140
`
`:LDESC ID=eleph sound
`
`ot offEd - Not .X1 aldm
`
`(Fig. le)
`(Fig. 1d)
`(Fig. 1 c)
`(Fig. 1 b)
`(Fig. la)
`
`Internal Graphics Object 110
`Internal Audio Object 108
`internal Animation Object 106
`Book Text with Tags 104
`Link Description Tags 102
`
`100
`
`AUDIO_DATA.DVR
`TAPE_ AUDIO.DVR
`CD_ AUDIO.DVR
`
`FORMAT B
`DeltaMod
`20-20000Hz
`
`high fl
`stereo
`
`Software Driver
`
`Characteristics
`
`Hardware
`
`300
`
`Book File Buffer 232
`
`audio
`
`Function
`I/O Profile Buffer 230
`
`WORKSTATION 200
`
`FIG. 4a
`
`MEMORY 220
`
`NFLE 1020 - Page 10
`
`

`

`lualud *S'Il
`
`81 JO 6 40911S
`
`•••••••• •
`
`awes GM=
`
`momme
`
`••••• mmit
`
`MM./. OM worm ••• gamma
`
`011•11MID
`
`I•••
`
`IMMO,
`
`•OMM
`
`iM=
`
`Figure 1: Worldwide Elephant Population
`
`Survey
`Foundation
`Serengeti
`Data:
`
`Zoos
`
`ivo•
`
`Africa
`
`Asia
`
`Elephants
`
`Display Buffer 238
`
`II offEd - Not .xa ald1.1
`
`FIG. 4b
`
`
`
`the plight of the world's elephant populations
`:P.Several international organizations have focused attention on 4,---170
`:L Licl=eleph sound.Audio 1 - Elephant's Trumpet:eL.
`168
`Listen to the sound of an elephant's trumpet.
`166
`small-earned mammal of the forests of southeast Asia.
`tropical Africa. The Indian elephant Is a relatively
`
`Page Buffer
`
`236
`
`
`
`ilmilmi••• •
`
`MEP
`
`••=••••
`
`,iy• mmi
`
`land mimmn.
`
`OM* Olual
`
`AMR 0•••••
`
`=MI
`
`M.•
`
`MO =MEM MO
`
`NFLE 1020 - Page 11
`
`

`

`111aIed 'Ka
`
`t661 6ZZ •sox
`
`81 Jo oT pails
`
`FIG. 4
`
`FIG, 4c
`
`FIG. 4b
`
`FIG. 4a
`
`TO HIFI
`
`& CD
`TAPE
`
`FROM
`—1—TO &
`
`dilIMMININIMIM11101••••••1111011MIMMEr•
`
`ADAPTER
`MOUSE
`
`ADAPTER
`AUDIO
`
`ADAPTER
`
`VIDEO
`
`INTRF
`LAN
`
`
`
`216
`
`N
`
`214
`
`\
`
`212
`
`\
`
`210
`
`DISPLAY &
`
`ADAPTER
`
`KEYBD
`
`DASD
`
`CPU
`
`z1 offEd - Not .X1 aldm
`
`208
`
`206
`
`204\
`
`BUS 202
`
`FIG. 4c
`
`174'
`
`Drivers and Operating System 590
`
`I/O Handler Programs 500, 530 and 560 (Figs. 7a, 7b and 7c)
`
`Softcoppy Book Read Program 400 (Fig. 6)
`
`of Tanzania, Is based on their comprehensive surveys of wildlife..
`The graph in Figure 1, produced by the Serengeti Foundation
`
`Mal MMMOMM MIN
`
`ONO
`
`MINS
`
`i•MMIMPi Em•
`
`•=1.
`
`1/M/0/0
`
`=IV MOM=
`
`411
`
`MOO
`
`NFLE 1020 - Page 12
`
`

`

`U.S. Patent
`
`Nov. 22, 1994
`
`Sheet 11 of 18
`
`5,367,621
`
`FIG. 5
`
`300
`
`I/O
`FUNCTION
`
`HARDWARE
`TYPES
`
`CHARAC-
`TERISTICS
`
`SOFTWARE
`DRIVERS
`
`keyboard
`
`std
`
`pointer
`device
`
`mouse
`
`touch
`overlay
`
`display
`
`high res
`
`disk store
`
`hard drive
`
`printer
`
`audio
`
`video
`
`video
`
`laser
`
`stereo
`high fi
`
`VHS
`Tape
`
`Compact
`Disc
`
`std
`
`.050
`
`stylus .050
`finger .500
`
`1000x1000
`256 colors
`GOCA
`FORMAT C
`
`100 MB
`
`300 dpi
`
`20-20000 Hz
`DeltaMod
`FORMAT B
`
`Read/Write
`
`CD Video
`FORMAT A
`Read/Write
`
`BIOS
`
`MOUSE.DVR
`
`OVRLAY.DVR
`GESTURE.DVR
`
`BIOS
`ANIMATION.DVR
`GRAPHICS.DVR
`
`BIOS
`
`PRINTER.DVR
`
`CD_AUDIO.DVR
`TAPE AUDIO.DVR
`AUDIO_DATA.DVR
`
`VHS TAPE.DVR
`VHS DATA.DVR
`
`CD_VIDEO.DVR
`VIDEO_DATA.DVR
`
`NFLE Ex. 1020 - Page 13
`
`NFLE 1020 - Page 13
`
`

`

`U.S. Patent
`
`Nov. 22, 1994
`
`Sheet 12 of 18
`
`5,367,621
`
`FIG. 6
`
`LOAD USER'S WORKSTATION PROFILE 300 (FIG. 5)
`
`t#
`
`LOAD BOOK FILE 100 (FIG. 1
`
`LOAD UNK DESCRIPTION TAGS 102 INTO UNK BUFFER 234
`
`LOAD NEW PAGE INTO PAGE BUFFER 236
`
`DISPLAY PAGE TEXT
`
`LOCATE UNK TAGS IN PAGE
`
`j
`DO ANY UNK TAGS HAVE A UNK DESCRIPTION
`WITH AUTOLAUNCH='YES'?
`
`408
`
`r P10
`
`X412
`
`414
`
`NO
`
`NO
`
`IF YES, THEN GET DATA STRING FROM UNK DESCRIPTION
`
`X416
`
`418
`
`OUTPUT DATA STRING TO START THE EXECUTION OF THE I/O
`HANDLER PROGRAM SPECIFIED IN STRING AND GO TO STEP 420
`420
`
`WAIT FOR POINTER TO ACTIVATE A UNK TAG
`OR UNTIL NEW PAGE REQUESTED
`
`ARE ANY UNK TAGS ACTIVATED BY THE POINTER?
`
`422
`
`
`IF YES, THEN GET DATA STRING FROM UNK DESCRIPTION
`
`426
`F.
`OUTPUT DATA STRING TO START THE EXECUTION OF THE I/O
`HANDLER PROGRAM SPECIFIED IN STRING AND GO TO STEP 420
`428
`
`v424
`
`IF NO, THEN GO TO STEP 420
`
`NFLE Ex. 1020 - Page 14
`
`NFLE 1020 - Page 14
`
`

`

`U.S. Patent
`
`Nov. 22, 1994
`
`Sheet 13 of 18
`
`5,367,621
`
`FIG. 7a
`
`SOFTCOPY BOOK READ PROGRAM OUTPUTS DATA
`STRING (VIDEO.EXEWD VIDEO FILE FORMAT A')
`
`VIDEO HANDLER PROGRAM 500, VIDEO.EXE BEGINS
`
`PARSE DATA STRING TO IDENTIFY VIDEO
`HARDWARE/SOFTWARE SUPPORT REQUIRED
`
`508
`—I DOES WORKSTATION PROFILE INCLUDE REQUIRED VIDEO SUPPORT(
`510
`0
`IF NO VIDEO SUPPORT, THEN DOES BOOK FILE HAVE ALT OBJECT?
` 512
`
`IF NO, THEN DISPLAY ALT TEXT2 ("VIDEO NOT AVAILABLE')
`AND RETURN
`ELSE IF YES, THEN GET ALT DATA STRING ('\ANIMATION...')
`$
`PARSE ALT DATA STRING TO IDENTIFY
`TYPE OF VIDEO SUPPORT REQUIRED
`I
`
`DOES WORKSTATION PROFILE INCLUDE
`REQUIRED VIDEO SUPPORT?
`$
`IF NO, THEN DISPLAY ALT TEXT2 ("VIDEO NOT AVAILABLE')
`AND RETURN
`ELSE IF YES, OUTPUT ALT DATA STRING TO VIDEO
`SUPPORT AND ACCESS ALT OBJECT AT ALT STORE
`LOCATION AND OUTPUT ALT OBJECT TO VIDEO SUPPORT
`
`14
`
`1,16 1,18
`
`YES
`
`DISPLAY ALT OBJECT (ANIMATION) AND DISPLAY ALT
`
`TEXTV20
`
`RETURN
`
`522
`
`524
`
`IF WORKSTATION PROFILE INCLUDES REQUIRED VIDEO SUPPORT,
`THEN OUTPUT DATA STRING TO VIDEO SUPPORT AND ACCESS OBJECT
`AT STORE LOCATION AND OUTPUT OBJECT TO VIDEO SUPPORT
`
`DISPLAY OBJECT (CD VIDEO) AND DISPLAY RUNTEXT
`
`RETURN
`
`528
`
`NFLE Ex. 1020 - Page 15
`
`NFLE 1020 - Page 15
`
`

`

`U.S. Patent
`
`Nov. 22, 1994
`
`Sheet 14 of 18
`
`5,367,621
`
`FIG. 7b
`
`( 53°
`r32
`SOFTCOPY BOOK READ PROGRAM OUTPUTS DATA
`STRING ('AUDIO.EXUELTAMOD FILE FORMAT B')
`
`AUDIO HANDLER PROGRAM 530, AUDIO.EXE BEGINS
`
`PARSE DATA STRING TO IDENTIFY AUDIO
`HARDWARE/SOFTWARE SUPPORT REQUIRED
`
` 536
`
`538
`
`DOES WORKSTATION PROFILE INCLUDE REQUIRED AUDIO SUPPORT(
`540
`O
`IF NO AUDIO SUPPORT, THEN DOES BOOK FILE HAVE ALT OBJECT?
`
` 2
`IF NO, THEN DISPLAY ALT TEXT2 ("AUDIO NOT AVAILABLE"
`AND RETURN
`ELSE IF YES, THEN GET ALT DATA STRING
`
`PARSE ALT DATA STRING TO IDENTIFY
`TYPE OF AUDIO SUPPORT REQUIRED
`
`DOES WORKSTATION PROFILE INCLUDE
`REQUIRED AUDIO SUPPORT?
`
`546
`
`IF NO, THEN DISPLAY ALT TEXT2 ("AUDIO NOT AVAILABLE")
`AND RETURN
`ELSE IF YES, OUTPUT ALT DATA STRING TO AUDIO
`SUPPORT AND ACCESS ALT OBJECT AT ALT STORE
`LOCATION AND OUTPUT ALT OBJECT TO AUDIO SUPPORT
`
` 548
`
`YES
`
`DISPLAY ALT OBJECT AND DISPLAY ALT TEXT
`
`RETURN
`
`552
`
`135°
`
`554
`
`IF WORKSTATION PROFILE INCLUDES REQUIRED AUDIO SUPPORT,
`THEN OUTPUT DATA STRING TO AUDIO SUPPORT AND ACCESS OBJE
`AT STORE LOCATION AND OUTPUT OBJECT TO AUDIO SUPPORT
`
`PLAY OBJECT (TRUMPET.AUD) AND DISPLAY RUNTEXT
`558
`
`RETURN
`
`NFLE Ex. 1020 - Page 16
`
`NFLE 1020 - Page 16
`
`

`

`U.S. Patent
`
`Nov. 22, 1994
`
`Sheet 15 of 18
`
`5,367,621
`
`FIG. 7c-
`SOFTCOPY BOOK READ PROGRAM OUTPUTS DATA
`STRING eGRAPH.EXEAGOCA FORMAT C')
`
`560
`
`562
`
`
`
`566
`
`GRAPHICS HANDLER PROGRAM 560, GRAPH.EXE BEGINS
`I
`PARSE DATA STRING TO IDENTIFY GRAPHICS
`SOFTWARE SUPPORT REQUIRED
`i
`DOES WORKSTATION PROFILE INCLUDE REQUIRED GRAPHICS SUPPORT?!
`z.574
`it NO
`IF NO GRAPHICS SUPPORT, THEN DOES BOOK FILE HAVE ALT OBJECT?!
` 572
`IF NO, THEN DISPLAY ALT TEXT2 ("GRAPHICS NOT AVAILABLE")
`AND RETURN
`ELSE IF YES, THEN GET ALT DATA STRING
`$
`PARSE ALT DATA STRING TO IDENTIFY
`TYPE OF GRAPHICS SUPPORT REQUIRED
`
` 568
`
`DOES WORKSTATION PROFILE INCLUDE
`REQUIRED GRAPHICS SUPPORT?
`
`r76
`
` 578
`IF NO, THEN DISPLAY ALT TEXT2 ("GRAPHICS NOT AVAILABLEY
`AND RETURN
`ELSE IF YES, OUTPUT ALT DATA STRING TO GRAPHICS
`SUPPORT AND ACCESS ALT OBJECT AT ALT STORE
`LOCATION AND OUTPUT ALT OBJECT TO GRAPHICS SUPPORT
`
`DISPLAY ALT OBJECT AND DISPLAY ALT TEXT
`
`YES
`
`RETURN
`
`584
`
`el IF WORKSTATION PROFILE INCLUDES REQUIRED GRAPHICS SUPPORT,
`THEN OUTPUT DATA STRING TO GRAPHICS SUPPORT AND ACCESS OBJE
`AT STORE LOCATION AND OUTPUT OBJECT TO GRAPHICS SUPPORT
`
`582
`
`DISPLAY OBJECT (POPULATION.GPH) AND DISPLAY RUNTEXT
`588
`
`RETURN
`
` 586
`
`NFLE Ex. 1020 - Page 17
`
`NFLE 1020 - Page 17
`
`

`

`U.S. Patent
`
`Nov. 22, 1994
`
`Sheet 16 of 18
`
`5,367,621
`
`FIG, 8a
`100. Th
`
`File Index j..Q5
`
`offset
`object
`value
`name
`013A
`'Idesc
`table'
`05B3
`'book
`text'
`3000
`dip.anm'
`'family
`4000
`'trumpet.aud'
`5000
`opulation.gph'
`'
`n
`amily_2nd.anm -0.,_§.901___ name
`
`Link Description Tags
`Book Text with Tags
`Int' Animation Object
`Inr Audio Object
`Id Graphics Object
`addeqpInr 2nd Ed. Animation Object
`Link Description Tags 1Q2
`:LDESC ID=eleph_movie OBJTYPE=video
`OBJECT='family dip.vid'
`STORE=extemaT
`DATA='video.exe 1CD VIDEO File Format A'
`RUNTEXT='African Elephant Family Video'
`AUTOLAUNCH=no
`VIEW=center
`new
`STYLE=2
`changed --0- ALT OBJECTAamily 2nd.anms '4---- name
`ALT-STOREAnternai
`optional change --0- ALT—DATAA Animation; 150 frames; B/W bitmap'
`optional change --IP- ALT TEXT-'New African Elephant Family Animation'
`ALT TEXT2-2Video not available'
`Book Text with Tags im
`(Fig. 1b)
`Internal Animation Object Ex
`(Fig. 1c) (100 frames)
`Internal Audio Object na
`(Fig. 1d)
`Internal Graphics Object 11
`(Fig. 1 e)
`added,.3` Internal 2nd Edition Animation Object la
`
`(150 frames)
`
`120
`
`
`NFLE Ex. 1020 - Page 18
`
`NFLE 1020 - Page 18
`
`

`

`U.S. Patent
`
`Nov. 22, 1994
`
`Sheet 17 of 18
`
`5,367,621
`
`FIG. 8b
`
`100"
`
`File Index 1Q5
`
`Link Description Tags
`Book Text with Tags
`Int' 2nd Ed. Animation Object
`Int' Audio Object
`Int' Graphics Object
`
`object
`name
`'Idesc table'
`'book text'
`'bail dip.anm'
`'trumpet.aud'
`'population.gph'
`
`offset
`value
`013A same
`0563 name
`3000 ..J
`4500
`5500
`
`120
`
`Link Description Tags 12
`:LDESC ID=eleph_movie OBJTYPE=video
`OBJECTAamily dip.vid'
`STORE=extemaT
`DATA='video.exe \ CD VIDEO File Format A'
`RUNTEXT='African Elephant Family Video'
`AUTOLAUNCH=no
`VIEW=center
`STYLE=2
`no change ---0- ALT OBJECTAamily clip.anm'
`ALT—STORE=intemar
`optional change ---*- ALT—DATA=1 Animation; 150 frames; B/W bitmap'
`optional change ----•► ALT—TEXT.'New African Elephant Family Animation'
`ALT TEXT2=1Video not available'
`
`(Fig. 1b)
`Book Text with Tags 1M
`sub-
`.11.5
`Internal 2nd Edition Animation Object
`,p
`stituted
`Internal Audio Object 1Q8,
`(Fig. 1d)
`Internal Graphics Object 110
`(Fig. 1e)
`
`(150 frames)
`
`NFLE Ex. 1020 - Page 19
`
`NFLE 1020 - Page 19
`
`

`

`U.S. Patent
`
`Nov. 22, 1994
`
`Sheet 18 of 18
`
`5,367,621
`
`FIG. 8c
`802
`LOAD UPDATE FILE CONTAINING NEW OBJECT,
`
`x 800
`
`804
`CHECK PRESENCE OF EXISTING EDITION OF BOOK FILE 100
`ON READ/WRITE MEDIUM (EG DASD 206)
`
`806
`IDENTIFY NAME OF EXISTING INTERNAL OBJECT TO BE UPDATED IN BOOK FILE
`808
`GET LOCATION OFFSET OF EXISTING OBJECT FROM FILE INDEX 105
`810 -1
`IF UPDATE IS TO ADD NEW OBJECT AND KEEP EXISTING OBJECT (FIG. 8A)
`THEN WRITE NEW OBJECT AT END OF BOOK FILE 100 USING NEW OBJECT NAME
`AND WRITE NEW OBJECT NAME AND ITS OFFSET INTO FILE INDEX 105
`AND RETURN
`
`812
`ELSE UPDATE IS TO REPLACE EXISTING OBJECT WITH NEW OBJECT (FIG. 8B)
`8 i4
`
`IF NEW OBJECT IS NOT LARGER THAN EXISTING OBJECT
`THEN OVERWRITE EXISTING OBJECT WITH NEW OBJECT IN BOOK FILE 100
`(NO CHANGE TO NAME OR OFFSET IN FILE INDEX 105)
`AND RETURN
`
`816
`
`ELSE NEW OBJECT IS LARGER THAN EXISTING OBJECT
`
`818
`OVERWRITE EXISTING OBJECT WITH NEW OBJECT IN BOOK FILE 100
`(NO CHANGE TO NAME OR OFFSET OF NEW OBJECT IN FILE INDEX 105)
`AND REPOSITION REMAINING OBJECTS IN BOOK FILE 100
`AND INCREMENT THEIR OFFSETS IN FILE INDEX 105
`AND RETURN
`
`NFLE Ex. 1020 - Page 20
`
`NFLE 1020 - Page 20
`
`

`

`1
`
`5,367,621
`
`2
`representation and presentation of multimedia objects
`are not constrained by the mechanism that invokes the
`link from the book to the object.
`It is a further object of the invention to provide an
`5 improved softcopy book reading method which enables
`the easy updating of multimedia objects contained in the
`book, to new editions.
`SUMMARY OF THE INVENTION
`10 These and other objects, features and advantages are
`provided by the invention disclosed herein. A method,
`program and data processing system are disclosed, for
`providing a generalized link from a reference point
`15 within an organized hierarchy of a formatted text
`stream in an on-line book, to an arbitrary type multime-
`dia object. The method begins by storing a formatted
`text stream in the data processing system. The format-
`ted text stream includes a link description which con-
`20 tains multimedia type information, object location infor-
`mation and multimedia control information for a target
`multimedia object. The formatted text stream further
`includes a link tag associated with the link description,
`which identifies a source location in the formatted text
`25 stream from which a link is established to the target
`multimedia object.
`The method continues by storing the multimedia
`object in the data processing system at a location identi-
`fied by the location information, the object including
`30 multimedia data representing a multimedia presenta-
`tion. The method then continues by storing a multime-
`dia handler program in the data processing system, the
`handler program controlling operations of a multimedia
`output device characterized by the multimedia type
`35 information.
`The method then continues by displaying the format-
`ted text stream in the data processing system. The
`method then continues by activating the link tag in the
`data processing system and in response thereto, trans-
`40 ferring from the link description to the multimedia han-
`dler program the multimedia control information. The
`method then continues by executing the multimedia
`handler program using the control information and in
`response thereto, transferring from the location identi-
`45 fled by the location information to the multimedia out-
`put device, the multimedia data from the multimedia
`object. The method concludes by producing the multi-
`media presentation with the multimedia output device
`using the multimedia data from the multimedia object.
`50 Multimedia objects may be represented by data sepa-
`rate from the book or they may be encapsulated within
`the book. A description of the link is encoded in the
`softcopy book, which includes information about the
`55 type and location of the multimedia object to be refer-
`enced, and which includes multimedia control informa-
`tion. The softcopy book reading program embodying
`the method, invokes the appropriate multimedia han-
`dler program to support the presentation of the multi-
`60 media object. The nature, representation and presenta-
`tion of the multimedia object are not constrained by the
`mechanism that invokes the link.
`Many different kinds of multimedia objects can be
`linked into a softcopy book. Multimedia objects such as
`65 high resolution, photographic quality graphics, motion
`video, or sound can be supported by the invention. In
`addition, other functions which can be included in an
`expanded definition of multimedia, can also be pres-
`
`DATA PROCESSING METHOD TO PROVIDE A
`GENERALIZED LINK FROM A REFERENCE
`POINT IN AN ON-LINE BOOK TO AN
`ARBITRARY MULTIMEDIA OBJECT WHICH
`CAN BE DYNAMICALLY UPDATED
`
`BACKGROUND OF THE INVENTION
`1. Technical Field
`The invention disclosed broadly relates to data pro-
`cessing and more particularly relates to the display of
`multimedia presentations from an on-line book.
`2. Background Art
`Large, softcopy documents have been stored in data
`processing systems for later retrieval and display. The
`retrieval of documents using a computer has been a
`prominent application in both business and library sci-
`ence. However, the presentation of large, softcopy doc-
`uments for display on a monitor screen, has been in a
`serial, page-by-page format. The page-by-page presen-
`tation has made the on-line review of large, softcopy
`documents a slow and inefficient operation. To solve
`this problem of page-by-page presentation, softcopy
`book reading programs have been devised. For exam-
`ple, the IBM BookManager ( Tm ) READ program
`helps the user manage, search and look at on-line books.
`There are two complementary BookManager products,
`BookManager BUILD creates on-line books from files
`marked-up with Generalized Markup Language. The
`BookManager READ product can then manage, search
`and show the on-line books created by BookManager
`BUILD. The BookManager READ program product is
`described in the IBM publication "BookManager (TM )
`READ, Displaying On-Line Books," publication num-
`ber SC23-0449-0, February 1989. The BookManager
`BUILD program product is described in the IBM publi-
`cation "BookManager ( TM ) BUILD—Preparing On-
`Line Books," publication number SC23-0450-0, Febru-
`ary 1989. These books are available from IBM branch
`offices.
`Conventional softcopy book reading programs typi-
`cally display the pages of the softcopy book on a con-
`ventional monitor display screen. Multimedia presenta-
`tion formats such as audio presentation., video presenta-
`tion, sequential image animation presentation, and the
`like have not been considered a part of the presentation
`format which can be included and displayed in softcopy
`on-line books. Among the problems confronting the
`prior art which have prohibited the provision of multi-
`media presentation from on-line books is the lack of
`standardization in multimedia output devices and soft-
`ware drivers. This is due in part to the rapid rate of
`development of new multimedia output devices and it is
`further due in part to a lack of organized standards in
`the industry.
`OBJECTS OF THE INVENTION
`It is therefore an object of the invention to provide an
`improved softcopy book reading method.
`It is another object of the invention to provide an
`improved softcopy book reading method capable of
`multimedia presentations.
`It is still a further object of the invention to provide
`an improved softcopy book reading method which
`provides generalized links to arbitrary multimedia ob-
`jects.
`It is still a further object to provide an improved
`softcopy book reading method wherein the nature,
`
`NFLE Ex. 1020 - Page 21
`
`NFLE 1020 - Page 21
`
`

`

`5,367,621
`
`10
`
`15
`
`30
`
`55
`
`20
`
`3
`ented, such as a spread sheet, or an engineering diagram
`using a computer aided design data base.
`The author of a softcopy book can specify several
`types of multimedia presentation and the method of the
`invention selects the type which is compatible with the 5
`hardware available in the workstation where the book is
`read.
`An additional feature of the invention is the ability to
`dynamically update the softcopy book to new editions
`with new multimedia presentations.
`The resulting data processing method provides a
`generalized link from a reference point in a softcopy,
`on-line book to an arbitrary multimedia object, which
`accommodates a variety of multimedia hardware and
`updating of books to new editions.
`BRIEF DESCRIPTION OF THE DRAWINGS
`These and other objects, features and advantages will
`be more fully appreciated with reference to the accom-
`panying figures.
`FIG. 1 is an organizational diagram of a softcopy
`book file 100.
`FIG. la illustrates the link description tags 102 in the
`softcopy book file of FIG. 1.
`FIG. lb illustrates the softcopy book text with tags 25
`104 in the softcopy book file of FIG. 1.
`FIG. lc illustrates the internal animation object 106 in
`the softcopy book file of FIG. 1.
`FIG. ld illustrates the internal audio object 108 in the
`softcopy book file of FIG. 1.
`FIG. le illustrates the internal graphics object 110 in
`the softcopy book file of FIG. 1.
`FIG. 2 illustrates the external video object 195.
`FIG. 3a illustrates the relationship between the bit
`mapped frames in an animation sequence format and the 35
`corresponding displayed appearance of the frames.
`FIG. 3b illustrates how a sampled analog sound signal
`is converted into audio data for an audio object.
`FIG. 3c illustrates the relationship between displayed
`graphics and the graphics data which generates the 40
`display.
`FIG. 4 is an architectural diagram of an example
`workstation for displaying a softcopy book using the
`method of invention.
`FIG. 5 illustrates a user's workstation profile which 45
`characterizes the workstation of FIG. 4.
`FIG. 6 illustrates a flow diagram of the softcopy book
`READ program in accordance with the method of the
`invention.
`FIG. 7a illustrates an example video handler program 50
`in accordance with the invention.
`FIG. 7b illustrates a flow diagram of an audio handler
`program in accordance with the invention.
`FIG. 7c illustrates a flow diagram of a graphics han-
`dler program in accordance with the invention.
`FIG. 8a illustrates the softcopy book file 100' with
`the addition of a new internal object.
`FIG. 8b illustrates the softcopy book file 100" with
`the substitution of a new internal object.
`FIG. 8c illustrates a flow diagram of a method for 60
`updating the book file to a new edition.
`DESCRIPTION OF THE PREFERRED
`EMBODIMENT
`The above referenced IBM publications describing 65
`BookManager BUILD and BookManager READ pro-
`gram products, describe the principles of operation of a
`softcopy book reading program whose presentation
`
`4
`format primarily displays on a monitor display screen.
`Additional background description of the Book-
`Manager product can be found in the following IBM
`patent applications:
`Abandoned U.S. patent application Ser. No. 317,248,
`filed Feb. 20, 1989, entitled "A Method for Dis-
`playing On-Line Information," by E. Cassorla, et
`al., assigned to the IBM Corporation and incorpo-
`ra