United States Patent [19]
`Gudmundson et al.
`
`|||||||||||||||||||||||||
`
`US005680619A
`[11] Patent Number:
`[45] Date of Patent:
`
`5,680,619
`Oct. 21, 1997
`
`[54] HIERARCHICAL ENCAPSULATION OF
`INSTANTIATED OBJECTS INA
`MULTIMEDIA AUTHORING SYSTEM
`
`[75] Inventors: Norman K. Gudmundson, San Mateo;
`R. Hamish Forsythe, Palo Alto;
`Wayne A. Lee, San Mateo, all of Calif.
`[73] Assignee: mEactory, Inc., Burlingame, Calif.
`
`ABSTRACT
`[57]
`An application development system, optimized for author
`ing multimedia titles, enables its users to create selectively
`reusable object containers merely by defining links among
`instantiated objects. Employing a technique known as Hier
`archical Encapsulation, the system automatically isolates the
`external dependencies of the object containers created by its
`users, thereby facilitating reusability of object containers
`and the objects they contain in other container environ
`ments. Authors create two basic types of objects: Elements,
`[21] Appl. No.: 415,848
`which are the key actors within an application, and
`[22] Filed:
`Apr. 3, 1995
`Modifiers, which modify an Element’s characteristics. The
`object containers (Elements and Behaviors—i.e., Modifier
`[51] Int. Cl* ~~~~~~~~~~~~~~ G06F 9/40
`containers) created by authors spawn hierarchies of objects,
`[52] U.S. Cl. … 395/701
`including the Structural Hierarchy of Elements within
`[58] Field of Search ..................................... 395/650, 700,
`Elements, and the Behavioral Hierarchy, within an Element,
`395/701
`of Behaviors (and other Modifiers) within Behaviors. The
`system utilizes an Element’s dual hierarchies to make that
`Element an environmental frame of reference to the objects
`it contains. Through techniques known as Hierarchical Mes
`sage Broadcasting, Hierarchical Variable Scoping and Hier
`archical Relative Positioning, objects automatically receive
`messages sent to their object container and access data
`known to their object container. An Element's position is
`even determined relative to the position of its parent Ele
`ment container. The system is highly extensible through a
`Component API in which Modifiers and Services that sup
`port them can be added and integrated seamlessly into the
`system. The system’s architecture is substantially platform
`independent, automatically allowing most author’s titles to
`run on multiple platforms. In addition, the entire authoring
`environment can be ported relatively easily to a variety of
`platforms due to the isolation a platform-dependent layer
`within the system.
`48 Claims, 46 Drawing Sheets
`
`[56]
`
`References Cited
`U.S. PATENT DOCUMENTS
`5,287,447 2/1994 Miller et al. ............................ 395/157
`5,423,041
`6/1995 Burke et al. ............................ 395/700
`5,493,680 2/1996 Danforth ....
`. 395/700
`5,499,333 3/1996 Doudnikoff et al. ..
`. 395/600
`5,499,371
`3/1996 Henninger et al. ..................... 395/700
`OTHER PUBLICATIONS
`Booch, “Elements of the Object Model”, Object Oriented
`Analysis and Design with Applications, 2nd Ed. 1994, pp.
`40–77.
`
`Primary Examiner—Kevin A. Kriess
`Assistant Examiner—John L. Chavis
`Attorney, Agent, or Firm—Daniel R. Siegel; Ariel Reich
`
`PROJECT P
`
`SECTIOt, S1
`
`SECTION S2
`
`WARIABLE
`
`112
`
`105
`
`SECTION SS2
`
`
`
`
`
`
`
`SHARED SCENE
`
`108->
`
`ELEMENT E1
`
`ELEMENT E3
`
`
`
`
`
`l 11
`
`Et_EMENT E4.
`
`114-7 t 15 - 1 16
`
`1 : 8
`
`Page 1 of 88
`
`GOOGLE EXHIBIT 1022
`
`

`

`U.S. Patent
`
`Oct. 21, 1997
`
`Sheet 1 of 46
`
`5,680,619
`
`1 O1
`
`FIG, .
`
`PROJECT P
`
`1 O2
`
`1 O3
`
`SECTION S1
`
`SECTION S2
`
`104
`
`SUBSECTION SS1
`
`VARIABLE
`V1
`
`112
`
`1 05
`
`SECTION SS2
`
`1 O7
`
`
`
`1 O6
`
`VARIABLE
`V2
`
`113
`
`SHARED SCENE
`SCO
`
`SCENE SC 1
`
`108
`
`109
`
`1 1 C
`
`ELEMENT E1
`
`ELEMENT E2
`
`ELEMENT E3
`
`1 11
`
`ELEMENT E4 ||MOD MOD MOD | M4
`M :
`M2 ||
`|| M3
`1 17
`
`1 18
`BEHAVIOR
`B1
`
`1 14- 1 15 - 1 16
`
`1 1 9
`
`MOD ||BEHAVIOR VARIABLE
`M5
`B2
`V3
`
`120
`
`121
`
`122
`
`M6
`
`M7
`
`1 23
`
`Page 2 of 88
`
`

`

`U.S. Patent
`
`WD192m0
`
`Sheet 2 of 46
`
`5,680,619
`5,680,619
`
`
`
`
`
`
`
`
`
`
`
`IgfiofiugaBEE:
`
`
`
` 058332::_wmm
`
`:3.5;
`
`25%BEE:
`
`US. Patent
`
`
`358.3ng
`
`„z ryw º ?o ** ** * * ?
`
`\\\w39..bwmmowmmammmomnn
`
`Page 3 of 88
`
`

`

`U.S. Patent
`US. Patent
`
`Oct. 21, 1997
`Oct. 21, 1997
`
`Sheet 3 of 46
`Sheet 3 of 46
`
`5,680,619
`5,680,619
`
`:
`
`:
`
`:
`
`:
`
`5‘s
`
`g
`
`
`
`5%
`3g
`055
`gm
`
`: i
`
`É
`
`É
`
`#
`
`Page 4 of 88
`
`

`

`U.S. Patent
`
`Oct. 21, 1997
`
`Sheet 4 of 46
`
`r
`
`5,680,619
`
`3020-?e, [[Auntitled—1–303
`J02b
`[[F. untitled section-T 304
`J02c
`305 _-[]untitled subsection
`302d
`305-I?è untitled scene
`308-?ºl Grophic Modifier
`307--Thuntilled SCEIlê
`
`502e
`
`
`
`727
`
`Library Name
`
`Trash Con
`728 7296
`
`
`
`Sqve Pop-Up
`730
`
`729e-Hº Change Scene Modifier
`729f #| Bob's Movie
`
`FIG, 6
`
`520
`
`Page 5 of 88
`
`

`

`U.S. Patent
`
`Oct. 21, 1997
`
`Sheet 5 of 46
`
`5,680,619
`
`7540
`
`751
`
`754b
`
`754c
`
`752
`
`754d
`
`EID - -----------------
`
`==E---
`
`753
`
`70/
`
`851
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Toon recelved Mouse Down
`Messenger sent Author Event 1 to Graphic Element
`Graphic Element received Author Event 1
`Scene Trqnsition Modifier received Author Event
`Image Effect Modifier received Author Event 1
`Movie received Author Event 1
`Behavior received Author Event 1
`Miniscript Modifier received Author Event 1
`
`
`
`FIG, 8
`
`Page 6 of 88
`
`

`

`U.S. Patent
`
`Oct. 21, 1997
`
`Sheet 6 of 46
`
`5,680,619
`
`FIG. 9(a)
`902
`903
`904
`FEEH tº MFIºn=
`Cel: |003 || Selection: TII)|[=|III). Show:
`
`------- TÉ
`
`901
`
`900
`
`
`
`
`
`
`
`
`
`============== Source File Info ==::::::::::=======
`| Path: Kitty:Projects:MF:Interface:
`Elements:Toon
`911
`002 Dog Walking 00007 PICS
`003 Dog,008
`PICT
`004 Dog.009
`PICT
`002 Dog.004
`PICT
`002 Dog.003
`PICT
`002 Dog.004
`PICT
`002 Dog,003
`
`
`
`
`
`
`
`
`
`-Compressor=
`Type:
`| Animation IT
`Depth:
`H255 ºf TIF
`[T]Randomly Accessible
`Quality
`
`Least
`
`Low Medium
`
`High
`
`Most
`
`
`
`Untitled—1
`
`000000|º
`
`º-ÉH;
`FIG, 9( d) |Climbing
`OOATOO9.
`
`Page 7 of 88
`
`

`

`U.S. Patent
`
`Oct. 21, 1997
`
`Sheet 7 of 46
`
`5,680,619
`
`
`
`95]
`
`== Element info E
`
`Source File Path: Macintosh HD:Eqrth Folder:
`-
`Ocean Folder:Fish
`
`964
`
`Initiqi State
`959
`[T] Mute
`º-T Loop
`Poused S-F Back ond forth
`
`-
`
`Volume:
`100%
`
`962
`Rote: AT- Layer:
`[T][3].
`
`955
`
`956
`
`957
`
`958
`
`963
`
`961
`
`950
`
`FIG, [O
`
`Page 8 of 88
`
`

`

`U.S. Patent
`
`Oct. 21, 1997
`Oct. 21, 1997
`
`Sheet 8 of 46
`Sheet 8 of 46
`
`5,680,619
`
`5,680,619
`
`
`
`<>
`t?
`
`
`
`
`
`woQOEoEum323.8E
`
`US. Patent
`
`[-
`
`@@
`
`mum33:5
`
`Sm:535E
`
`530905EE
`
`xcH22820:.
`
`----~~~~----------------------
`========::=??===
`
`
`
`umzocmSEoEoo
`
`
`
`958;8:02
`
`EmEom325:
`
`umtSmmcoum
`
`
`
`
`
`
`
`
`
`Page 9 of 88
`
`

`

`U.S. Patent
`
`Oct. 21, 1997
`
`Sheet 9 of 46
`
`5,680,619
`
`1001
`1000
`=\ Integer Variable =
`
`1002
`
`1003
`
`
`
`
`
`1004
`
`1005
`
`FIG, 12(a)
`
`=\ Integer Variable —
`
`1022
`
`|
`
`Integer Range Variable
`
`Walue
`
`Start:
`
`End:
`
`
`
`
`
`1024
`
`1025
`
`FIG, 12(b)
`
`Page 10 of 88
`
`

`

`U.S. Patent
`
`Oct. 21, 1997
`
`Sheet 10 of 46
`
`5,680,619
`
`1041
`== Vector Variable =
`
`1040
`
`Angie:
`
`Magnitude:
`
`
`
`
`
`
`
`104.4
`
`FIG, 12(c)
`
`1045
`
`! 061
`
`! 060
`
`
`
`= Boolean Variable =
`
`1064
`
`1065
`
`FIG, 12(d)
`
`Page 11 of 88
`
`

`

`U.S. Patent
`
`Oct. 21, 1997
`
`Sheet 11 of 46
`
`5,680,619
`
`1121
`
`4 120
`
`-
`
`= Point Variable E
`
`1122
`
`Point Variable
`
`
`
`Wolue
`
`X:
`
`Y;
`
`-
`1123
`
`
`
`
`
`
`
`1124
`
`F.G. 12(g)
`
`1125
`
`1141
`
`1 140
`
`= Time Variable F
`
`Time Voriqble
`
`Walue
`
`00:00.00
`
`
`
`
`
`
`
`
`
`1144
`
`1145
`
`FIG, 12(h)
`
`Page 12 of 88
`
`

`

`U.S. Patent
`
`Oct. 21, 1997
`
`Sheet 12 of 46
`
`5,680,619
`
`1081
`
`1080
`
`E=Siringvariable =
`|-º
`
`Walues
`
`"MIropolis'
`
`1082
`
`
`
`1083
`
`
`
`1103
`
`1 104
`
`1105
`
`FIG, 12(f)
`
`Page 13 of 88
`
`

`

`U.S. Patent
`
`Oct. 21, 1997
`
`Sheet 13 of 46
`
`5,680,619
`
`120.1
`
`1200
`
`H Messenger E
`
`
`
`1203
`
`
`
`
`
`
`
`
`
`
`
`Message
`Event/Commond:
`
`Destinotion [TE]
`
`Message Options
`DX Immediate
`
`1209
`
`-
`
`1210
`
`1211
`
`FIG. [3(a)
`
`Page 14 of 88
`
`

`

`U.S. Patent
`
`Oct. 21, 1997
`
`Sheet 14 of 46
`
`5,680,619
`
`1221
`
`1220
`
`# If Messenger É
`
`Execute when:
`
`Message
`Event/Commond:
`With:
`TIE,
`Destination TIE,
`
`-i?-i tº
`
`
`
`
`
`1230
`
`Message Options
`Dº Immediate
`
`1231
`
`1232
`
`FIG, 3(b)
`
`Page 15 of 88
`
`

`

`U.S. Patent
`
`Oct. 21, 1997
`
`Sheet 15 of 46
`
`5,680,619
`
`
`
`
`
`Execute when:
`
`
`
`Terminate when:
`
`1244
`
`1241
`1240
`= Timer Messenger E
`tºg_-5 tº
`-[E.T. E. E.
`F. El
`
`1245
`
`
`
`
`
`247
`124
`
`
`
`1248
`
`
`
`1252
`
`Delay For
`00:00.00
`
`DZ Loop timer 4-> -->
`º
`1246
`
`Message
`Event/Commond.
`
`With:
`
`[TE][Til?º
`º Destination
`TTE
`
`D
`º
`
`1249
`
`Message Options
`
`- DX|Immedicte
`
`Ex Coscode
`
`DZ Relay
`
`1251
`
`1253
`
`1254
`
`FIG, [3(c)
`
`Page 16 of 88
`
`

`

`U.S. Patent
`
`Oct. 21, 1997
`
`Sheet 16 of 46
`
`5,680,619
`
`1262
`
`
`
`
`
`1268
`
`
`
`
`
`1270
`
`#261
`1260
`H Boundary Detection Messenger =
`|
`Bonday Detection Mºsell
`Enqble when:
`Disqble when:
`FTT Rºl
`
`El
`
`1264
`
`1266
`Delay Element's Parent's Borders
`AT-->
`(C) On first detection
`O While detected O Exiting
`
`Message Specifications
`Message/Command:
`
`With:
`
`Rºl
`TTE
`
`Destinqtion
`
`1267
`
`
`
`Rºl
`
`1275
`
`Message Options
`DX|Immediate
`
`DX Cascade
`
`DX Relay
`
`1271
`
`FIG, [3(d)
`
`Page 17 of 88
`
`

`

`[:2] Inn-'- @
`Detect Elements
`@ C) On first contact
`0 While in contact
`
`0 Exiting
`
`1293
`1299
`
`1295
`
`Message/Command:
`
`With:
`
`1 289
`
`1290
`
`I To collision elements(s)
`
`© To other destination
`
`1292’
`
`US. Patent
`
`Oct. 21, 1997
`
`Sheet 17 of 46
`
`5,680,619
`
`—-———————.______
`
`Collision Messenger
`‘22 I i:@3112,
`
`Enable When:
`
`Disable When:
`
`1284
`
`I
`
`1285
`
`Collide With:
`An E
`t
`y
`amen
`Detect Layer
`1285 I E Front
`E Behind
`1287|
`Message Specifications
`1288 [glitz]
`
`@
`
`Page 18 of 88
`
`

`

`U.S. Patent
`
`Oct. 21, 1997
`
`Sheet 18 of 46
`
`5,680,619
`
`1301
`1300
`# Hypertext Messenger E
`T [Epeted Messenger
`Execute when:
`
`On;
`
`[…]
`
`A-> -1305
`R7] [T]Remain ºnted
`
`[º]
`
`
`
`
`
`With: : Ha*=
`
`
`
`1307
`
`Apply
`Highlight Effect:
`
`Messoge
`Event/Command:
`
`t
`
`
`
`1302
`
`1306
`
`1308
`
`Destinotion
`
`TTE
`
`SNZ
`
`
`
`1309
`
`Message Options
`DZ Immediate
`
`1310
`
`FIG. I3(f)
`
`Page 19 of 88
`
`

`

`U.S. Patent
`
`Oct. 21, 1997
`
`Sheet 19 of 46
`
`5,680,619
`
`
`
`1321
`1320
`= Keyboard Messenger ---
`|
`Keyboard Messenger
`
`Execute When
`
`Key State
`
`
`
`1322
`
`
`
`
`
`Message
`Event/Command:
`
`With:
`
`*~|-|[TE][T][7]
`T.TIRZ.
`
`
`
`Destination
`
`-
`
`1327
`
`
`
`1328
`
`Message Options
`DZ Immediate
`
`Page 20 of 88
`
`

`

`U.S. Patent
`
`Oct. 21, 1997
`1401
`
`Sheet 20 of 46
`
`5,680,619
`
`1400
`
`1402
`
`1 404
`
`Execute When:
`
`Specifications
`(C) Next scene in subsection
`O Previous scene in subsection
`OSpecify scene:
`
`I? Add to destination background
`|I|Add to return list
`
`
`
`FIG, 14(b)
`
`Page 21 of 88
`
`

`

`U.S. Patent
`
`Oct. 21, 1997
`
`Sheet 21 of 46
`
`5,680,619
`
`1441
`
`1440
`
`Specifications
`Tronsition:
`
`1444
`
`
`
`
`
`Scene Change Modifier
`1505
`Scene Chonge Modifier with "Add to Return List" *::::
`
`1525
`ene Chonge Modifier
`1535
`
`Return Modifier
`1545
`
`Page 22 of 88
`
`

`

`U.S. Patent
`
`
`
`Oct. 21, 1997
`
`Sheet 22 of 46
`
`5,680,619
`
`
`
`
`
`|E| (EN PEIL?E,
`
`auON @|
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Page 23 of 88
`
`

`

`U.S. Patent
`
`Oct. 21, 1997
`
`Sheet 23 of 46
`
`5,680,619
`
`1641
`
`1640
`
`Execute When:
`
`Terminate When:
`
`|R1||Nº||Fl
`1643–4|Mouse Up
`pecifications
`| =Path= Position
`Cel:
`
`1644
`
`
`
`1645 {# Hº?ºf 9 i??ºjo? 12
`
`
`
`T .
`
`FIG, 16(c)
`
`1647
`1648
`
`1654
`
`1661
`= Vector Motion Modifief
`
`1660
`
`Execute When:
`
`Termingte. When:
`
`*—HIZiºn.
`1663–1-?ºup-Fi None-º-TR
`1664 |
`Specifications
`With:
`
`1865–Hºw
`
`---
`
`IE
`
`º 1666
`
`1667
`
`FIG, 16(d)
`
`Page 24 of 88
`
`

`

`U.S. Patent
`
`Oct. 21, 1997
`
`Sheet 24 of 46
`
`5,680,619
`
`1701
`
`1700
`
`ETHGººdwº E
`1702–HF Icºphºmo?e. T
`Apply When:
`Remove When:
`*-i-B?l?r? IF Erºs
`Specifications===
`Ink:
`1705-HETIE TH-- 1708
`|
`F] []-—1709
`T |Rectangº
`Fl
`
`
`
`
`
`
`
`Matte:
`
`| Shape:
`
`º, º wº
`
`Elº,
`
`jºr
`
`
`
`
`
`1711
`
`
`
`Border =
`
`T][3] [I]
`
`
`
`1722
`
`1723
`
`1725
`
`Apply When:
`Remove When:
`|Scene Stºred ºil- None
`Specifications
`Color Table:
`
`Page 25 of 88
`
`

`

`U.S. Patent
`
`Oct. 21, 1997
`
`Sheet 25 of 46
`
`5,680,619
`
`1741
`
`1740
`
`FEGrºom wome E
`*—-ºfºº
`
`
`
`
`
`Remove When:
`
`
`
`TE 1744
`1743
`Cº-º: |
`| Style
`-
`1745 H-Hºr?e # Tiki 1746
`CConce?)(CIOKD
`FIG, [7(e) " "
`
`
`
`E. image Effect Modifier
`
`Apply When:
`
`Remove When:
`
`Specifications
`Image Effect:
`
`Bevel Width:
`
`Tone Amount:
`
`Ex|Extend to margins
`
`E=IÉ) [54][3]
`
`-
`
`1764
`
`1766
`
`Page 26 of 88
`
`

`

`U.S. Patent
`
`Oct. 21, 1997
`
`Sheet 26 of 46
`
`5,680,619
`
`1801
`1800
`E Sound Modifier =
`1802
`T
`[Sound Modifier
`| Execute When:
`Terminate When:
`*—it?.T.T.T.E. Tº TIRZ:
`Specifications
`Sound
`SystemTºp Tºl (Freiew D
`
`
`
`
`
`1804
`
`
`
`1805
`
`
`
`
`
`Concel
`
`1807
`
`1808
`
`1821
`== Sound Fade Modifier =
`1822
`l
`Sound Fode Modifier
`| Execute When:
`Termingte When:
`1823–1t?ºn. Fºll.
`Tº [Nº.
`2 || 7 |
`Specifications
`Fode to:
`Duration:
`100%
`00:03,00
`
`
`
`
`
`1825
`
`
`
`1824
`
`
`
`t
`
`1826
`
`FIG, 18(b)
`
`Page 27 of 88
`
`

`

`U.S. Patent
`
`Oct. 21, 1997
`
`Sheet 27 of 46
`
`5,680,619
`
`Apply When:
`Remove When:
`Farent Fº?? El Tºng NTF
`Specifications
`Font:
`?åenevºls:
`|E|| | Dunderine
`ÉpîF| | I] Shadow
`[T]Condensed
`Extended
`
`Alignment:
`
`Size:
`
`t
`
`Outline
`
`1904
`
`1910
`
`
`
`1921
`
`1920
`
`Apply When:
`
`Remove When:
`
`- - -
`
`? tº
`
`-
`
`1924
`
`== Cº-P
`ºn-Histº
`1923–H-Rºuse over TE Nºne TNTE
`Specifications
`Cursor:
`
`
`
`1925
`
`
`
`FIG. 20 * *
`
`Page 28 of 88
`
`

`

`U.S. Patent
`
`Oct. 21, 1997
`
`Sheet 28 of 46
`
`5,680,619
`
`
`
`1981
`
`1980
`
`1982
`
`set my Fioqt to 4,555
`
`Page 29 of 88
`
`

`

`U.S. Patent
`
`Oct. 21, 1997
`
`Sheet 29 of 46
`
`5,680,619
`
`1941
`1940
`1945
`= Classification Modifie? =
`|Cassification Modifier
`|
`1942
`Apply. When:
`Remove When: W_ --
`Porent Engbled
`|Fi None
`1943
`|E|
`Classificqtion
`1944–-Forêts dissº? F.
`
`
`
`Hºrrºr
`
`
`
`1962
`1963
`
`|| |s*Wous modifierº |
`Execute When:
`F.
`--Specifications-----------------
`Set:
`1964—Hincoming Da??|E||
`| To:
`we-H None
`|E|.
`
`1966
`
`Page 30 of 88
`
`

`

`US. Patent
`
`Oct. 21, 1997
`
`Sheet 30 of 46
`
`5,680,619
`
`FIG. 24
`
`EDIT MDDE FUNCTIONALHY
`
`S RUNTIME FUNCTIONALITY
`5
`
`.3 TITLE CONTENT
`5 A: PERIPHERALS
`
`BUILDER
`
`LIBRARIES
`
`TOON
`EDITOR
`
`SYMBOL MANAGER
`
`19
`
`VIEw
`
`VIEW
`
`VIEW
`S
`STRUCTURE
`VIEw
`
`BEHAVIOR
`VIEW
`
`33
`
`54
`
`.
`
`31
`
`32
`
`AUTHORING GUI
`DRAG D
`DROP
`
`LINKING
`
`MORPHING
`
`43
`
`44
`
`MESSAGING
`
`LOG
`
`41
`
`42
`
`23
`
`I
`
`2
`
`21
`
`MODIFIERS
`
`I
`
`SERVICES
`
`24
`
`50
`
`COMPONENT MANAGER
`SERVICE
`COMPONENT
`MANAGER
`ARI
`
`E
`;
`:
`';
`5
`
`:
`
`E
`403
`
`CROSS—PLATFORM ARCHITECTURE
`
`;
`
`Page 31 of 88
`
`

`

`U.S. Patent
`
`Oct. 21, 1997
`
`Sheet 31 of 46
`
`5,680,619
`5,680,619
`
`
`
`Salads.00:Imung;do:IEng;do:
`
`£56
`
`wememan:a:
`$42,331E2«2
`
`
`
`ozC<mwuoEm:meIEmsmd
`
`Er
`
`32%
`
`028$
`
`om:mmm?
`newmm:B——mmEEo:—Erase:—EEOC:‘_.
`
`US. Patent
`
`mm.oE
`
`(NOLIO3SgnS)
`Azofiwmmamv
`
`Esau
`
`
`
`
`
`Page 32 of 88
`
`

`

`U.S. Patent
`
`Oct. 21, 1997
`
`Sheet 32 of 46
`
`5,680,619
`
`FIG, 26
`
`
`
`
`
`
`
`SERVICE
`WRAPPER
`
`1.65
`
`SERVICE
`WRAPPER
`
`1 66
`
`SERVICE
`WRAPPER
`
`167
`
`164:
`
`SERVICE
`
`SERVICE
`
`SERVICE
`
`168
`
`169
`
`17O
`
`Page 33 of 88
`
`

`

`U.S. Patent
`
`Oct. 21, 1997
`
`Sheet 33 of 46
`FIG, 27
`
`5,680,619
`
`175
`
`SCENE 1
`
`SCENE 2
`
`SCENE 3
`
`
`
`
`
`
`
`APPLICATION
`
`STRUCTURAL ELEMENTS
`
`TABLE OF ASSETS
`SCENE 1 OBJECTS
`BACKGROUND PICTURE
`
`176
`
`177
`178
`179
`18O
`
`
`
`
`
`TABLE OF ASSETS
`SCENE 2 OBJECTS
`
`181
`182
`
`
`
`ASSETS
`
`TABLE OF ASSETS
`SCENE 3 OBJECTS
`
`183
`184
`
`Page 34 of 88
`
`

`

`U.S. Patent
`
`Oct. 21, 1997
`
`Sheet 34 of 46
`
`5,680,619
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`SYSTEM
`PROCESSING
`
`
`
`PENDING
`MESSAGE
`TO SERVICE/
`MODIFER
`7
`
`POST MODIFIER/
`SERVICE
`THREAD
`
`
`
`CAN
`THEY BE
`DEFERRED
`7
`
`PERFORM SERVICE/
`MODIFIER
`PROCESSING
`
`NOTIFY THOSE
`ELEMENTS
`DRAW COMPLETE
`
`
`
`
`
`
`
`PERFORM
`END PLAY
`PROCEDURES
`
`ANIMATE
`SHARED SCENE
`
`257
`
`
`
`ANIMATE
`ACTIVE SCENE
`258
`
`PROCESS
`POSTED MODIFIER/
`SERVICE THREADS
`
`PERFORM ALL
`PENDING DRAWS
`
`ANY
`ELEMENTS
`NEED TO KNOW
`
`
`
`FIG, 28
`
`
`
`RELINQUISH
`CONTROL
`2
`
`Page 35 of 88
`
`

`

`U.S. Patent
`
`Oct. 21, 1997
`
`Sheet 35 of 46
`
`5,680,619
`
`
`
`SNNNNNNNNNNNNNNNNNNNNNNNNNNNNN
`
`62 "
`
`Page 36 of 88
`
`

`

`Oct. 21, 1997
`
`64cm63aehS
`
`5,680,619
`
`mOHUDmHmZOO0:200
`
`
`
`US. Patent
`
`Page 37 of 88
`
`

`

`U.S. Patent
`
`Oct. 21, 1997
`
`Sheet 37 of 46
`
`5,680,619
`
`RESOURCES
`DIRECTORY
`
`
`
`
`
`276
`
`IDENTIFY
`KIT FILES
`277
`
`COUNT
`COMPONENTS
`278
`
`BUILD
`COMPONENT
`TABLES
`
`LOAD KIT,
`INITIALIZE
`SELF RECORDS
`
`INITIALIZE
`METHODS
`
`BUILD TABLES
`AND
`RESOLVE SIZES
`
`
`
`
`
`
`
`
`
`
`
`
`
`FIG, 3.
`
`Page 38 of 88
`
`

`

`U.S. Patent
`
`Oct. 21, 1997
`
`Sheet 38 of 46
`
`5,680,619
`
`ë, Fle
`
`Edit
`
`Fomat
`
`Amange
`
`Object
`
`Wew
`
`Window
`
`J8
`
`320
`
`330
`
`
`
`U
`
`|
`|
`|United Sºtomºl United Sºciºl|United Scene v Kliº
`#LE=NEWSNAKE:Structure
`D;
`v. 450 [] Snakehead
`&
`451
`Graphic Modifier
`452 Al SnokeheadVector
`453 [2] Snakeheadmotion
`454
`Snakehit Sound
`455
`Boundary Detection Messenger
`460 [] SnakeMiddle
`461
`Graphic Modifier
`462 || "... Full Motion Modifier
`463
`Timer Messenger
`470 [] SnakeMiddle
`47 | [F Graphic Modifier
`472
`Full Motion Modifier
`473
`Timer Messenger
`480 [] SnakeMiddle
`481
`Graphic Modifier
`482
`Full Motion Modifier
`483
`Timer Messenger
`490 [] Snake Toil
`491
`Graphic Modifier
`492
`Messenger
`493
`Full Motion Modifier
`494
`Timer Messenger s
`
`450
`
`470
`
`490
`
`Sº, Sº,
`
`Page 39 of 88
`
`

`

`US. Patent
`
`Oct. 21, 1997
`
`Sheet 39 of 46
`
`5,680,619
`
`@EEEE
`351 352 353 354 3 5
`
`56
`
`351 [E SchoolingProbabiiity
`352 [E SwitchProbobility
`353 [E TopBottom
`354 [E] Lefmigm ,
`355 [E] MaxSwitchAngle
`356 [El Graphic Modifier
`360 [:1
`
`Transparent
`36}
`352 @ MyVelocity
`365
`NewVelocity
`364
`Drag Motion Modifier
`355
`Vector Motion Modifier
`.366
`Tick Messenger
`37o
`StuyInTank
`
`395 9.1 Miniscn'pt Modifier
`
`TE Border Detection Messenger
`371
`LR Border Detection Messenger
`372
`373 21!] BounceOffBorder
`374 m BounceSound
`
`380
`
`Trend—Setting Behavior
`
`DecideToRandomlySwitch
`381
`382 3g lnvokeRondomDirec’don
`
`390
`
`Schooling Behavior
`
`391 24 Schooling Decision
`392
`Delayed MyHeoding Broadcast
`
`395
`
`ChangeVector
`
`.396 IE LeftCeiRango
`397
`RIghtCelRange
`
`Page 40 of 88
`
`

`

`U.S. Patent
`
`Oct. 21, 1997
`
`Sheet 40 of 46
`
`5,680,619
`
`2120
`
`2220
`| Window X ||
`
`2240
`||
`
`FIG, 34(a) "
`
`
`
`FIG, 34(b) Tº
`
`FG 34(c)."
`
`
`
`
`
`FG 34(3)." | 2500
`
`0
`FIG. 34(e)
`| Window X |
`
`FIG, 34(f) 2600
`
`| |
`
`FIG, 34(g) Tº
`
`Page 41 of 88
`
`

`

`U.S. Patent
`
`Oct. 21, 1997
`
`Sheet 41 of 46
`
`5,680,619
`
`2000
`2010
`2020
`2030
`2040
`2100
`2101
`2102
`2103
`2104
`2105
`2106
`2120
`212]
`2122
`2123
`2124
`2125
`
`~
`x7.
`~7.
`
`*7
`v
`
`WINDOW Example
`?º Window Selection
`If Window Subsection
`[] Window Shored Scene
`[] Window Scene
`[]
`
`?.
`
`Nome
`BoundingRect TopLeft
`BoundingRect_BottomRight
`Graphic Modifier
`Image Effect Modifier
`CREATE Window
`TitleBar
`EA Text Style Modifier
`Graphic Modifier
`TitleRect TopLeft
`TitleRect Bottom Right
`$º CREATE TitleBor
`
`FIG, 35(a)
`
`Page 42 of 88
`
`

`

`U.S. Patent
`
`Sheet 42 of 46
`
`Oct. 21, 1997
`[I]
`
`.
`
`Window X + Minhºax
`Nome
`BoundingRect_TopLeft
`BoundingRect BottomRight
`Graphic Modifier
`Image Effect Modifier
`3. CREATE Window
`ºf MiniMaxFesponse
`Max Response
`Min Response
`TitleBor
`#A Text Style Modifier
`Graphic Modifier
`TitleRect_TopLeft
`TitleRect Bottom Right
`CREATE TitleBar
`MinºloxResponse
`
`[A]
`
`-
`
`?.
`*) ji
`
`~ * = * * *
`
`Image Effect Modifier
`Graphic Modifier
`MinFlag
`BoxRect_TopLeft
`BoxRect BottomRight
`Minl(XONDimensions
`CREATE Minbox
`MinMoxBehavior
`$.
`DetectMinboxSelect
`3. Minºlox Response
`
`FIG, 35(b)
`
`5,680,619
`2200
`220.1
`2202
`2203
`2204
`2205
`2206
`2207
`2208
`2209
`2220
`2221
`2222
`2225
`2224
`2225
`2226
`2240
`2241
`2242
`2243
`2244
`22:45
`2246
`2247
`2248
`2249
`2250
`
`Page 43 of 88
`
`

`

`US. Patent
`
`Oct. 21, 1997
`
`She_et 43 of 46
`
`5,680,619
`
`v
`
`E]
`
`Window X +MinMux + Drag
`Name
`
`BoundingRecLTopLeft
`
`BoundingRecLBotlomRight
`
`Graphic Modifier
`
`image Effect Modifier
`
`CREATE Window
`
`MinMaxResponse
`
`{a
`
`Window DRAG Behavior
`2]
`Follow Mouse
`iitieBor
`Text Style Modifier
`
`FIG. 35(d)
`
`2300
`2301
`
`2302
`
`2303
`
`2304
`
`2305
`
`2306
`
`2307
`
`2308
`2309
`2320
`2321
`
`2322
`
`2323
`
`2324
`
`2325
`
`2326
`
`2327
`
`2328
`
`2329
`
`2330
`
`233]
`2332
`2340
`
`Graphic Modifier
`
`TitieRecLTopLeft
`
`TitieRecL BottomRight
`
`CREATE TitieBar
`
`MindeResponse
`
`TiiIeBar DRAG Behavior
`’H‘r
`
`MouseOffset
`
`MouseDown
`
`CachouseOffset
`
`TitiaBorMouseTracking
`MouseUp
`
`4 Q
`
`_J
`.
`
`'i
`
`E]
`
`MinBox
`
`FIG. 35(c)
`
`[E]
`
`Window X'
`Nome
`
`BoundingRecLTopLef’L
`
`BoundingRecLBottomRight
`
`Graphic Modifier
`CREATE Window
`
`iitieBor
`
`Page 44 of 88
`
`

`

`U.S. Patent
`
`Oct. 21, 1997
`
`Sheet 44 of 46
`
`5,680,619
`
`<z
`
`|E||
`
`~z
`
`~7.
`
`~z
`
`~z
`
`2500
`Window X + WindowShode
`2501
`Nome
`2502
`BoundingRect TopLeft
`2503
`BoundingRect BottomRight
`2504
`Graphic Modifier
`2505
`CREATE Window
`3.
`2506
`*| WINDOW WinShaderesponse
`2507
`Shodedown Response
`2508
`Shadeup Response
`2520
`[A] TitleBor
`2521
`EA Text Style Modifier
`2522
`Graphic Modifier
`2523
`TitleRect_TopLeft
`2524
`TitleRect BottomRight
`2525
`3. CREATE TitleBor
`2526
`* WinShadeBehavior
`2527
`WinShadeflag
`2528
`WinShade Sound
`<!;
`2529
`INTFlag
`$...
`25.30
`*] Detect?)oubleClick
`2531
`[?º]
`ClickFog
`2532
`3. INIFog
`3. MouseupResponse 2533
`Ö| Clicktimer
`2534
`3. SingleClick
`2535
`
`FIG, 35(e)
`
`Page 45 of 88
`
`

`

`BoundingRecLTopLeft
`
`BoundingRecLBottomRight
`
`Graphic Modifier
`
`Image Effect Modifier
`
`CREATE Window
`
`WINDOW WinShodeRespanse
`
`Window DRAG Behavior
`
`TitleBar
`
`Text Style Modifier
`
`Graphic Modifier
`
`TitleRecLTopLeft
`
`TitleRecLBottomRight
`
`CREATE TitleBar
`
`WmShadeBehavior
`
`TItIeBar DRAG Behavior
`
`US. Patent
`
`Oct. 21, 1997
`
`Sheet 45 of 46
`
`fl
`
`Window X + MndowShade + Drag
`
`Nome
`
`IQQEEWEEE
`
`5,680,619
`
`Page 46 of 88
`
`

`

`U.S. Patent
`
`v
`
`Oct. 21, 1997
`[] ALL
`
`Sheet 46 of 46
`
`:
`
`[[F]
`
`Nome
`BoundingRect Tople?
`BoundingRect_BottomRight
`Grophic Modifier
`Image Effect Modifier
`CREATE Window
`?.
`* Mink?qxResponse
`* WINDOW WinShadeResponse
`& Window DRAG Behavior
`TitleBor
`[A] Text Style Modifier
`Graphic Modifier
`TitleRect TopLeft
`TitleRect BottomRight
`& CREATE TitleBor
`ºf MinwoxResponse
`sº WinShadeBehavior
`& TitleBor DRAG Behavior
`[] Minbox
`Image Effect Modifier
`Graphic Modifier
`MinFlag
`BoxRect TopLeft
`BoxRect_Bottom Right
`MiniCONDimensions
`CREATE Minbox
`MinbloxBehavior
`MINBOX WinShaderesponse
`
`FIG, 35(g)
`
`5,680,619
`2700
`2701
`2702
`2703
`2704
`2705
`2706
`2707
`2708
`2709
`2720
`2721
`2722
`2723
`2734
`2725
`2726
`27.27
`2728
`2740
`2741
`2742
`2743
`2744
`2745
`2746
`2747
`2748
`2749
`
`Page 47 of 88
`
`

`

`1
`HIERARCHICAL ENCAPSULATION 0F
`INSTANTIATED OBJECTS IN A
`MULTIMEDIA AUTHORING SYSTEM
`
`I. BACKGROUND
`A. Field of the Invention
`B. Description of the Related Art
`1. Reusability and the Modular Interface
`2. Encapsulation and the Object Interface
`3. The Need for Selective Reusability of "User Objects”
`a. An OOP Windowing Example
`4. The Lack of Selective Reusability in Multimedia
`Authoring Systems
`'
`II. SUMMARY OF THE INVENTION
`1]]. BRIEF DESCRIPTION OF THE DRAWINGS
`IV. DETAILED DESCRIPTION
`A. External Architecmre—“Author’s-Eye View”
`1. Objects: Elements and Modifiers
`2. Object Containers: Elements and Behaviors
`3. Object Hierarchies: Structural Hierarchy and Behav-
`ioral Hierarchy
`a. Types of Elements in the Structural Hierarchy
`( 1) Projects
`(2) Sections
`(3) Subsections
`(4) Scenes
`(5) Media Elements
`b. Isolation of External Dependencies of Object Con-
`tainers
`
`5,680,619
`
`2
`(2) Modifier Connections
`(3) Player and Asset Connections
`(4) Touch-Up Process
`b. Aliasing
`2. Saving Projects and Titles
`3. Runtime Functionality
`a. Message Targeting and Dispatching
`b. Event Loop Processing
`C. Component API—“Programmer’s-Eye View”
`1. Object Model
`2. Component Methods
`3. Registration Process
`4. Core and Service Utilities
`D. Examples
`1. Snake
`2. Fish
`
`3. Windowing System
`TABLES
`
`Table I. System Messages And Commands
`Table II. Message Destinations
`Table III. Message Data Structure
`Table IV. Base Component Class Functions
`Table V. Sample Instance Data Structure
`FIGURES
`
`FIG. 33 Implementation Of Fish
`
`(1) Hierarchical Message Broadcasting
`(2) Hierarchical Variable Scoping
`(3) Hierarchical Relative Positioning
`c. Selective Reusability through Adoption and Trans-
`plantation
`'
`. Object Authoring Interface
`a. Layout View
`b. Layers View
`c. Structure View
`(1. Asset Palette
`e. Library
`f. Alias Palette
`g. Messaging Log
`h. men Editor
`1. Modifier Palettes
`j. Drag and Drop
`5. Object Messaging
`a. Categories of Messages
`(1) Author Messages
`(2) Environment Messages
`(3) Commands
`b. Parts of a Message
`(1) Message Name
`(2) Message Destination
`(3) Message Data
`(4) Message Path
`6. Object Configuration
`a. Elements
`b. Modifiers
`(1) Behavior
`(2) Variables
`(3) Capabilities
`B. Core System Architecture
`1. Edit Mode Functionality
`a. Connecting Objects
`(1) Element To Element Connection
`
`FIG. 1 Dual Hierarchy
`FIG. 2 Layout View With Tool, Object Information And
`Modifier Palettes
`
`FIG. 3 Layers View
`FIG. 4 Structure View
`FIG. 5 Asset Palette
`
`FIG. 6 Library
`FIG. 7 Alias Palette
`
`FIG. 8 Messaging Log
`FIGS. 9(a)—(d) m'I‘oon Editor
`FIG. 10 Element Configuration Dialog
`FIG. 11 Behavior Interface
`
`FIGS. 12(a)—(h) Variable Interfaces
`FIGS. 13(a)—(g) Messenger Interfaces
`FIGS. 14(a)—(c) Scene—Based Modifier Interfaces
`FIG. 15 Scene Change And Return Modifiers
`FIGS. 16(a)—(d) Motion Modifier Dialogs
`FIGS. 17(a)—(d) Visual Modifier Dialogs
`FIGS. 18(a)—(d) Sound Modifier Dialogs
`FIG. 19 Style Modifier Dialog
`FIG. 20 Cursor Modifier Dialog
`FIG. 21 Miniscript Modifier Dialog
`FIG. 22 Classification Modifier Dialog
`FIG. 23 Set Value Modifier Dialog
`FIG. 24 System Architecture
`FIG. 25 Object Interconnections
`FIG. 26 Service Connections
`
`FIG. 27 Title Storage
`FIG. 28 Event Loop Processing
`FIG. 29 Interfaces Architecture
`
`FIG. 30 Operation Of Object Model
`FIG. 31 Loading Of Components
`FIG. 32 Implementation Of A Snake
`
`Page 48 of 88
`
`

`

`5,680,619
`
`terns.
`
`4
`objects typically communicate by directly calling one anoth-
`er’s capabilities or “methods.”
`OOP application development systems generally require
`their users to define abstract “classes” of objects that define
`the characteristics of the objects in that class. The actual
`objects in the user’ 5 application are instantiated as needed at
`“runtime” from this class “template.” Such systems also
`typically include certain built-in reusable class libraries to
`provide the user with an
`base of functionality.
`The instantiated objects that perform the various tasks
`within the user’ 5 application are reusable in large part due to
`a process known as “encapsulation.” Encapsulation involves
`defining a class of objects with a well-defined external
`interface, and “hiding” within the objects the code and data
`necessary to implement that interface. In other words, the
`implementation of'an object’s interface is “encapsulated”
`entirely within the object.
`Thus, programmers can reuse objects in various different
`contexts within their application. All other objects that are
`aware of this interface can “reuse” this object because its
`external dependencies are isolated within its modular
`“object interface.”
`'
`Programmers also can use a process known as “inherit-
`ance” to “specialize” or modularly extend an object’s func—
`tionality by reusing some or all of that object’s character-
`istics via its interface or class definition—1e. , the public
`methods and data structures that define the template for
`instances of that class. For example, programmers may wish
`to extend the base functionality of an existing class (whether
`created by the programmer or provided in a reusable class
`library) by adding a new method to objects in that class, or
`replacing (i.e., overriding) or supplementing (i.e.,
`overloading) an existing method.
`By defining a sub-class of objects that “inherit” the
`characteristics of the “parent” class, the programmer can
`reuse (as well as add to, replace or supplement) those
`inherited characteristics. For example, a “car” or “bus”
`sub-class might inherit characteristics from a more general
`“vehicle” class. A car is a “kind of” vehicle, and may inherit,
`for example, a vehicle’s color and speed properties, as well
`as steering and braking methods.
`3. The Need for Selective Reusahility of “User Objects”.
`Despite their facilities for reusability of objects, OOP
`application development systems remain diflicult to use. It is
`far simpler for application developers to create and config—
`ure instantiated objects directly, as opposed to defining
`abstract classes each time they need to create a slightly
`difierent object or to model the relationships or interaction
`among existing objects.
`This problem is exacerbated when an application devel-
`oper creates and desires to reuse compound or aggregate
`“user objects"—i.e., a group of objects that are related in the
`context of a particular application, but do not necessarily
`share common characteristics. These “user objects,” in addi—
`tion to requiring extensive inter—object communication,
`often involve “part of” or “container” relationships that
`occur frequently in any modularly designed application.
`Inheritance and encapsulation are well-suited to model
`“kind of” relationships to further specialize the atomic
`objects from which an application will be built. All modular
`systems require certain atomic “building bloc
`” which are
`reusable and provide suflicient performance, even if diflicult
`to modify.
`To create applications of any significant complexity,
`however, one must combine these atomic objects into more
`complex “user objects” that interact with one another. It is
`this common modular process that illustrates the significant
`limitations on reusability imposed by traditional OOP sys-
`
`3
`FIGS. 34(a)—(g) Selectively Reusable Windows Example
`FIGS. 35(a)—(g) Implementation Of Windows Example
`I. BACKGROUND
`
`A. Field of the Invention
`This invention relates to application development systems
`generally, and in particular to systems for authoring inter—
`active multimedia applications.
`B. Description of the Related Art
`Since the advent of computers many decades ago, com-
`puter scientists have labored to build increasingly powerful
`computer hardware that is faster and cheaper to build than its
`predecessors. Today’s computers can perform,
`in one
`second, many millions of “add,” “shift,” “load,” “store” and
`other relatively simple functions.
`To perform tasks of any significant complexity, however,
`one must somehow cause computers to perform vast num—
`bers of these simple functions in some sequence. Computer
`software, in its most basic form, comprises programs or
`sequences of instructions, each of which directs a computer
`to perform the function corresponding to that instruction.
`Yet, even as computers have become more powerful,
`developers of computer software continue to struggle to
`create complex programs without having to “reinvent the
`wheel” for each task they direct computers to perform. This
`need for “reusability” permeates virtually every aspect of
`software development, and is driven ultimately by the end
`user’s desire for “ease of use.”
`1. Reusability and the Modular Interface.
`Although there are many varied approaches to this basic
`problem of reusability, one technique remains constant—
`“modular programming," i.e., the “bootstrapping” of mul-
`tiple simple functions into “modules” of greater and greater
`complexity that can be reused through higher—level “modu—
`lar interfaces.” Virtually all software relies on modular
`programming to some extent.
`For example, assemblers and compilers enable program—
`mers to bootstrap from the machine language defined by a
`computer’s instruction set to a higher-level, more human-
`readable language. Similarly, operating systems perform
`many of the low-level support tasks commonly needed by
`applications software developers (e.g., file system, memory
`management, basic GUI routines, etc), and then provide
`developers wiflr a library of “reusable” modules.
`In essence, all application development systems provide
`to their users a language, an application programming inter—
`face (“API”), or some other form of modular interface
`designed to facilitate development of general purpose or
`highly specialized applications. Such modular interfaces
`provide reusability of the “hidden” functionality which
`implements that interface.
`2. Encapsulation and the Object Interface.
`One of the most popular current trends designed to
`promote the creation of reusable software is in the field of
`object-oriented programming (“OOP”). There are a variety
`of GOP languages and application development tools on the
`market today (e.g.