United States Patent
`Walton et al.
`
`[19]
`
`[11] Patent Number:
`
`5,883,639
`
`[45] Date of Patent:
`
`Mar. 16, 1999
`
`Us005883639A
`
`Inc.,
`
`“VAPS—Conceptual Overview”, Virtual Prototypes,
`May 17, 1991.
`“Creating Graphical Interfaces Through Graphical Interac-
`tion”, Singh et al., Oct. 6, 1990.
`“Suit: Software User Interface Design Tool”, Kishi ct al.,
`Sep. 13, 1991.
`“Linked—List Visualization for Debugging”, Shimomura et
`al., May 1991.
`“Visual Programming in a Computer Animation Environ-
`ment”, Van Reeth et al., Oct. 6, 1990.
`“Visualization of Experimental Systems”, Haarslev et al.,
`Oct. 12, 1988.
`
`(List continued on next page.)
`
`Primary Examiner—.Ioseph H. Field
`
`[57]
`
`ABSTRACT
`
`A system for providing a simple, easy to learn and flexible
`means of creating user interfaces to products under devel-
`opment without the need of a programming language or the
`need to learn a large set of complicated commands. The
`Visual Software Engineering (“VSE”) system of thc inven-
`tion uses a simple concept of defining both input to and
`output from graphical objects in an object-oriented system
`by providing examples of what the user desires the graphical
`object to do. This technique is referred to herein as “anima-
`tion by example”. In accordance with this process, the user
`creates a user interface by drawing the user interface with a
`graphics editor and then defining the output behavior (i.e.,
`graphics manipulation) of the user interface components by
`showing each state or frame as an animation. This is
`accomplished by changing the object using a graphic editor
`function such as move or rotate and storing each of the
`frames with the object as a behavior state. Just as with
`dcfining the output, the input is dcfincd by giving the graphic
`object an example of what type of input to look for, and once
`it finds that input, it tells the object which frame to output or
`change to. Application code can then drive the animation or
`read the input by accessing the frame numbers assigned to
`each of the example frames.
`
`39 Claims, 19 Drawing Sheets
`
`Microfiche Appendix Included
`(3 Microfiche, 229 Pages)
`
`[54] VISUAL SOFTWARE ENGINEERING
`SYSTEM AND METHOD FOR DEVELOPING
`VISUAL PROTOTYPES AND FOR
`CONNECTING USER CODE TO THEM
`
`[75]
`
`Inventors: Thomas J. Walton; Kipper K.
`Fulghum; Thomas R. Batcha, all of
`Colorado Springs; Michael T. Juran,
`Highlands Ranch; Eric J. Kuzara,
`Colorado Springs, all of Colo.; Thomas
`F. Kraemer, Corvalis, Oreg.; Roy L.
`Johnson, Fort Collins, Colo.
`
`[73] Assignee: Hewlett-Packard Company, Palo Alto,
`Calif.
`
`[21] Appl. No.: 887,828
`
`[22]
`
`Filed:
`
`Jul. 3, 1997
`
`Related U.S. Application Data
`
`[63] Continuation of Ser. No. 695,540, Aug. 12, 1996, aban-
`doned, which is a continuation of Ser. No. 383,560, Feb. 2,
`1995, abandoned, which is a continuation of Ser. No.
`847,569, Mar. 6, 1992, abandoned.
`
`Int. Cl.6 ..................................................... .. G06T 1/00
`[51]
`[52] U.S. Cl.
`........................ .. 345/473; 345/433; 345/334;
`345/349
`[58] Field of Search ................................... .. 345/418, 419,
`345/473, 326, 474, 475, 333-335, 348-349,
`351, 433
`
`[56]
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`...................... .. 364/200
`5/1989 Barstow et al.
`364/192
`9/1989 Love et al.
`364/200
`2/1990 Kodosky et al.
`364/200
`4/1990 Kodosky et al.
`.. 364/518
`8/1991 Cunningham et al.
`395/152
`11/1993 Susman .............
`434/43
`11/1993 Slye et al.
`......................... .. 395/500
`1/1996 Joseph et al.
`OTHER PUBLICATIONS
`
`
`
`.
`
`4,827,404
`4,870,561
`4,901,221
`4,914,568
`5,041,992
`5,261,041
`5,261,820
`5,485,600
`
`“VAPS by Example”, Copyright Owned by Virtual Proto-
`types Inc. Publication I.D.: 39, Revision 0, Dated Jul. 5,
`1991.
`
`WORK SPACE
`
`[20
`
`USER CODE
`
`LIBRARIES or STANDARD
`AND USER-CREATED
`comvouans
`
`
`
`H0
`
`DRAWING AND BEHAVIOR EDITOR
`
`
`CONNECT
`OBJECTS
`
`
`
`#INCLUDE CONIPONENTS.H
`
`INHILEIOISPLAYJIALUE - TRUE)
`~ READ_BAR(|NFUT)
`VALUE =(INPlJT-X-XX)/YY
`«D SELCAUCEIVALUE):
`
` # DEIINE READ_BAR
`
`# DEFINE SELCAUCE
`
`
`
`CONNECT OBJECTS
`TO CODE
`
`PICK A PLACE
`OBJECTS
`
`CREATE AND/OR
`_'M0Dm0BJEm __
`
`CONNECT OBJECTS
`TO OBJECTS
`
`1
`
`GOOGLE-1004
`
`1
`
`GOOGLE-1004
`
`

`
`5,883,639
`Page 2
`
`OTHER PUBLICATIONS
`
`“VPL: An Active, Declarative Visual Programming Sys-
`tem”, Lau—Kee et al., Oct. 11, 1991.
`“CIAO: A Powerful Synchronisation & Controlling Mech.
`for Human Computer Interfaces”, Papageorgiou Oct. 11,
`1991.
`
`“Vaps Conceptual Overview,” Virtual Prototypes Inc., May
`17, 1991.
`“Creating Graphical Interfaces Through Graphical Interac-
`tion,” Singh, et al. Oct. 6, 1990.
`“Suit: Software User Interface Design Tool,” Kish, et al.,
`Sep. 13, 1991.
`“VPL: An Active, Declarative Visual Programming Sys-
`tem,” Lau—Kee, et al., Oct. 11, 1991.
`
`“CIAO: A Powerful Synchronisation & Controlling Mecha-
`nism for Human Computer Interfaces,” Papageorgiou Oct.
`11, 1991.
`
`Proceedings of the European Computer Graphics Confer-
`ence and Exhibition (Eurographics ’89), 4 Sep. 1989, Ham-
`burg, FRG pp. 3-14, Oijing Mao: “Micro—UIDT: A User
`Interface Development Tool”.
`
`Research Disclosure No. 317, Sep. 1990, Emsworth, GB, p.
`749 Disclosure No. 31761 “Intelligent Object—Oriented
`Graphics”.
`
`IEEE Software, vol. 6, No. 3, May 1989, New York, US, pp.
`15-23, Brad A. Myers: “User—Interface Tools: Introduction
`and Survey”.
`
`2
`
`

`
`U.S. Patent
`
`Mar. 16, 1999
`
`Sheet 1 of 19
`
`5,883,639
`
`
`
`:2:wa<>-:§_e:_é
`
`
`
`:Ec;:a2_7m=._<>
`
`:_:z_:z:§1!
`
`
`
`"$33:§s-:m4
`
`_+2_:z2..§3302.»
`
`2.22%332%:
`
`._E§-$g22
`
`252228
`
`382.
`
`23:3H8228
`
`All
`
`232..3232°H8228
`
`4|232..:53.A22::
`:55
`
`35..¢5:
`
`£339
`
`3
`
`
`

`
`U.S. Patent
`
`Mar. 16, 1999
`
`Sheet 2 0f 19
`
`5,883,639
`
`
`
`E20Ema...§:_%a
`
` zo:§_z<523A_5.5%32::$1;..<:3.._E20§__:§A_322::5:,
`
`g_><_§2.22SW30E:zo_z:Es§52
`A___s_:z_z<2£EEE_A_
`.23.._
`
`,_o:__,__Ezo:§__,_<
`
`225:::a-:w
`
`§§_z<
`
`_I
`
`
`
`|:||II||.|I.|_
`
`\:,:._maE:E2::
`
`
`
`:32:.25:3:
`
`\*m_3<>E2E3:
`
`33:VE5213
`
`5a<>:<_o-:m
`
`
`
`32:“$<Zo>I2._%_E5:;"$8$3
`
`4
`
`

`
`U.S. Patent
`
`Mar. 16, 1999
`
`Sheet 3 of 19
`
`5,883,639
`
`320
`
`LIBRARY OF
`GRAPHICAL OBJECTS
`
`330
`
`DISPLAY
`
`3|0
`
`USERINPUT
`
`% GRAPHWS
`ED|TOR
`
`0
`
`'00
`
`IIIIII
`
`6
`
`300
`
`fl
`
`ANIMATOR
`
`RECORD/ PLAYBACK
`DEVICE
`
`035
`
`370
`
`,
`
`EXTERNAL
`DATABASE
`
`USEWS SOURCE
`CODE
`
`360
`
`5
`
`

`
`U.S. Patent
`
`m
`
`5,883,639
`
`asWEEE__smgmas52%~§_B
`Q».E:E5:2;
`
`gzawE8_=n_§N2m555H.,§z_;
`
`\§\V.§.~F.H
`
`O;
`
`6
`
`
`
`

`
`U.S. Patent
`
`Mar. 16, 1999
`
`Sheet 5 of 19
`
`5,883,639
`
`§
`
`§:_z=
`
`E225
`
`N?
`
`E82,?
`
`52%
`
`g_>§mE
`
`Sq
`
`
`
`xSV.PKas525m5
`
`«Q
`
`g_:_E5
`
`g_><§E
`
`Q?
`
`$3
`
`#58200E,
`
`isso
`
`7
`
`
`
`
`
`

`
`U.S. Patent
`
`Mar. 16, 1999
`
`Sheet 6 of 19
`
`5,883,639
`
`2.522333:3532>2225
`2.328
`
`28
`
`3333:33:.E3333
`
`.5:22222:29:
`
`52.22528_5.2220
`
`2223332228222_
`
`22:22M522229::
`
`29522283:3
`
`2252
`
`=:2_3328:2;
`
`33;3:33.
`
`3:,3:33;3:3.3,z_333:3$333333333333a3»3,_,,::3::m__:m..:3<>$._33%3333333333333::3
`
`3:33
`
`33:3:_:3333:3:_3333333:3333333»3333:3333333.:3:333333:.
`:::3:333::2333333::33:3$3333:3
`3:,3:33333:,3:3.
`
`33
`
`226222:5228.
`
`2333333232_3.5.222:8
`
`523222292.u_.E._.“5oz2_23822:;
`
`
`3:322.82.22222222:
`2.52292228_2:35
`222333333239
`8323E3333
`
`«.25;3:332_
`
`333:3E33333...
`
`3333
`
`.3;3:33W.3533:,333E3?333233
`.9rK33:3:33:
`
`8
`
`
`
`
`
`
`
`
`
`

`
`U.S. Patent
`
`Mar. 16, 1999
`
`Sheet 7 of 19
`
`5,883,639
`
`as5:3~s_>sE
`
`gm
`
`:52:SE22E3
`
`.”_N_.u,zzgsEu
`
`52%m=.:320;:
`
`5,:So.323
`
`:35552M5253
`
`M213855.3E.35
`
`$2223122.5%
`
`ii:33$82
`
`
`
`$8g_><§mz_..:o8SEEEjfl
`
`
`
`
`
`W25g_><_EEEES
`
`O8meE;
`
`5%:32
`
`New
`
`3»E2EE
`
`%_>s:m
`
`:,_#_E5E:M52S552g_:§.:E
`
`Him52$.
`
`E53523$oz
`
`5&8259
`
`t3_><$m
`
`2m_E§“:
`
`S:35
`
`9
`
`
`
`
`
`
`

`
`U.S. Patent
`
`Mar. 16, 1999
`
`Sheet 8 of 19
`
`5,883,639
`
`TOO
`
`WAIT FOR
`
`STATE NAME
`
`DEFINE INPUT
` CREATE NEW
`
`REGISTER NEW
`STATE WITH
`BEHAVIOR ROUTER
`
`‘
`
`
`STORE INPUT
`WITH OBJECT
`
`CREATE A NEW
`STATE VALUE
`
` STORE INPUT
`
`INFORMATION WITH
`BEHAVIOR STATE
`
`
`
`VSE OBJECT INPUT DEFINITION
`
`Fig. 7
`
`10
`
`10
`
`

`
`U.S. Patent
`
`Mar. 16, 1999
`
`Sheet 9 of 19
`
`5,883,639
`
`804
`
`CREATE NEW
`STATE NAME
`
`800
`
`WAIT FOR
`DEFINE BEHAVIOR
`
`DOES STATE
`VALUE EXIST?
`
`YES
`
`
`
`REGISTER NEW STATE
`WITH BEHAVIOR ROUTER
`
`806
`
`
`
`808
`
`CREATE NEW
`
`_
`
`DOES STATE
`
` YES
`STATE VALUE I VALUE EXIST?
`
` 3:2
`
`
`VSE OBJECT
`BEHAVIOR DEFINITION
`
`Fig. 8
`
`GET omecrs
`GRAPHICS sum:
`
`STORE OBJECT'S
`GRAPHICS STATE
`
`11
`
`

`
`U.S. Patent
`
`Mar. 16, 1999
`
`Sheet 10 of 19
`
`5,883,639
`
`
`
`WAIT FOR A BEHAVIOR
`STATE WHICH CONSISTS
`OF A STATE NAME
`AND STATE VALUE
`
`700
`
`
`
`
`
`902
`
`SAVE NEW STATE VALUE
`FOR THE STATE NAME
`
`IN THE BEHAVIOR EVENT CACHE
`
`CO THROUGH LIST
`OF REGISTERED
`BEHAVIOR RECEIVERS
`
`FOUND ANOTHER
`RECEIVER?
`
`YES
`
`908
`
`
`
`
`CALL RECEIVER AND
`PASS IT THE STATE
`NAME AND VALUE GIVEN
`
`BEHAVIOR ROUTER 412
`
`F "g. .9/a/
`
`12
`
`12
`
`

`
`U.S. Patent
`
`Mar. 16,1999
`
`Sheet 11 of 19
`
`5,883,639
`
`920
`
`
`
` WAIT FOR A
`
`STATE NAME
`
`SEARCH FOR STATE NAME
`IN BEHAVIOR EVENT CACHE
`
` 292
`
`FOUND STATE
`NAME?
`
`YES
`
`STATE NAME RETURN A VALUE OF 0
`
`RETURN STATE VALUE
`ASSOCIATED WITH
`
`
`
`BEHAVIOR ROUTER 412
`
`Fig. 9/0)
`
`13
`
`13
`
`

`
`U.S. Patent
`
`Mar. 16, 1999
`
`Sheet 12 of 19
`
`5,883,639
`
`
`
`
` WAIT FOR NEW
`I000
`
`BEHAVIOR STATE
`NAME AND VALUE
`(FROM BEHAVIOR ROUTER)
`
`
`
`STATE NAME
`DEFINED?
`
`
`LOWER
`HIGHER
`STATE
`VALUE
`STATE VALUE
`STATE VALUE
`DEFINED?
`DEFINED?
`DEFINED?
`
`
`
`
`
`
`
`
`
`
`
`
`
`LOAD OBJECT
`F WITH LOWER
`
`STATE VALUE
`
`
`
`LOAD NEW STATE
`INFORMATION
`INTO OBJECT
`
`
`
`
` LOAD TMPORARY
`
`OBJECT WITH
`HIGHER STATE VALUE
`
`EXECUTION OF
`OUTPUT BEHAVIOR
`
`r'g_ /0
`
`
`
`MERGE TEMPORARY
`0BJECT'S TRANSFORM
`WITH CURRENT OBJECT
`
`
`14
`
`14
`
`

`
`U.S. Patent
`
`Mar. 16, 1999
`
`Sheet 13 of 19
`
`5,883,639
`
`H00
`
`H02 REGISTER WITH BEHAVIOR
`
`
`
`ROUTER FOR BEHAVIOR EVENTS
`
`H04
`
`
`
`WAIT FOR A BEHAVIOR EVENT
`FROM THE BEHAVIOR ROUTER OR A
`REQUEST TO SET BEHAVIOR EVENT
`SELECTION FROM CLIENT SERVER
`
`
`
`
`
` H06
`
` LOOK FOR STATE NAME ASSOCIATED
`
`
`2
`
`WITH BEHAVIOR EVENT IN
`BEHAVIOR EVENT SELECTION LIST
`
`
`
`
`RECEIVED A
`BEHAVIOR EVENT?
`
`NO
`
` III4
`
`RECEIVED A REQUEST
`TO SET BEHAVIOR
`EVENT SELECTION?
`
`
`
`IIIO
`
`STATE NAME FOUND
`IN SELECTION LIST?
`
`
`
`
`SEND STATE NAME AND STATE
`VALUE FOR BEHAVIOR EVENT T0
`CLIENT (USER PROGRAM)
`
`.
`
`
`
`ADD NAME TO BEHAVIOR
`EVENT SELECTION LIST
`
`
`
`BEHAVIOR EVENT RECEIVER
`FOR USER APPLICATION
`
`Fig. II
`
`15
`
`15
`
`

`
`U.S. Patent
`
`Mar. 16, 1999
`
`Sheet 14 of 19
`
`5,883,639
`
`
`
`
`
`2%mm>_oEW522E20::52:zsaog52
`
`52meE:
`
`
`
`~_o_mw,arw_momHé_4~$,_.w_?mE:g_><§m
`
`E>a8zE:g_:_$E335.5%
`
`252%20.55:2%g_>§253
`
`E:g_><§:5mm2E3:25.:22:
`
`525me§_$
`
`:65g.>$_:E2
`
`22558meE581.
`
`22:E3E2:2...$2.3E3
`
`5::2:2.5.:Ns.u.~Ln
`
`
`
`E52%_><§E3E;E:522fi_.flo“_
`
`2,:M3,;55as5?4EW3:02.552&322
`E32:2%£22Es5:E2
`
`g_>§m:55
`
`16
`
`55%20.5228
`
`E2:E3
`
`52GoNos
`
`as
`
`:25
`
`16
`
`
`
`
`
`
`

`
`U.S. Patent
`
`Mar. 16, 1999
`
`Sheet 15 of 19
`
`5,883,639
`
`START
`
`I300
`
`OPEN OUTPUT FILE
`
`I302
`
`REGISTER WITH BEHAVIOR
`ROUTER FOR BEHAVIOR EVENTS
`
`I304
`
`I306
`
`RESET TIMER
`
`I3I4
`
`
`
`WRITE STATE NAME AND
`WAIT FOR BEHAVIOR
`STATE VALUE TO OUTPUT
`EVENTOR STOP REQUEST
`
`
`FILE WITH TIME STAMP
`FROM USER
`
`
`
`
`
`‘
`
`GET TIME SINCE
`LAST TIMER RESET
`
`I3|2
`
`-
`
`
`
`RECEIVED
`BEHAVIOR
`EVENT?
`
`RECEIVED
`STOP
`REOUEST?
`
`YES
`
`UNREGISTER WITH
`BEHAVIOR ROUTER FOR
`BEHAVIOR EVENTS
`
`CLOSE OUTPUT FILE
`
`I322
`
`STOP
`
`I3|8
`
`I32
`
`°
`
`17
`
`BEHAVIOR RECORDER 4l8
`
`Fig. /3
`
`17
`
`

`
`U.S. Patent
`
`Mar. 16, 1999
`
`Sheet 16 of 19
`
`5,883,639
`
`START
`
`'4°°
`
`I402
`
`I404
`
`OPEN INPUT FILE
`
`GET LINE FROM FILE
`
`
`
`BEHAVIOR PLAYER 4|8
`
`.?ET'g7. I54!
`
`I420
`
`READ STATE NAME AND
`VALUEFROMIJNEIN HLE
`
`I422
`
`SEND STATE NAME AND
`VALUE T0 BEHAVIOR
`ROUTER FOR ROUTING
`
`18
`
`NEW LINE
`AVAILABLE?
`
`NO
`
`CLOSE FILE
`
`STOP
`
`MI2
`
`GET TIME STAMP
`ON LINE
`
`I40
`
`8
`
`|4|4
`
`.
`
`SET WAIT TIMER
`FROM TIME STAMP
`
`'4'°
`
`CHECK WAIT TIMER
`
`I4M
`
`|4I8
`
`18
`
`

`
`U.S. Patent
`
`Mar. 16, 1999
`
`Sheet 17 of 19
`
`5,883,639
`
`I500
`
`WAIT FOR
`USER INPUT
`
`
` I502
`
`
`
`GRAB EXAMPLE
`MOUSE AND
`MOVE?
`
` CHANGE EXAMPLE
`
`CURSOR SIZE OR
`SHAPE?
`
`
`
`PRESS EXAMPLE
`MOUSE BUTTONS?
`
` STORE LAST
`EXAMPLE MOUSE
`
`EVENT TYPE AS MOTION
`
`
`
`
`
`STORE LAST
`EXAMPLE MOUSE
`EVENT TYPE
`
`
`
`
`AS BUTTON PRESS
`
`
`CHANGE SIZE
`OR SHAPE OF
`EXAMPLE CURSOR
`
`CHANCE POSITION
`OF EXAMPLE
`CURSOR
`
`I506
`
`EXAMPLE MOUSE
`
`Fig. /5
`
`19
`
`19
`
`

`
`U.S. Patent
`
`Mar. 16, 1999
`
`Sheet 18 of 19
`
`5,883,639
`
`I600
` WAR FORINPUT
`FROM USER
`
`IBB
`
`ARE TWO INPUT
`AREAS DEHNED?
`
`YES
`
`N0
`
`IBM
`
`
`
`INPUT DEFINED
`FOR OBJECT?
`
`YES
`
`INPUT OBJECT
`BASED?
`
`NO
`
`
`
`
`I606
`
`YES
`
`ENABLE
`FUNCHON
`SET?
`
`
`INTERPOLATE
`GET STATE
`INPUTINFO
`BETWEENINPUT
`
`—
`AREAS?
`
`YES
`
`
`
`
`
`
`
`I620
`
`
`
`INPUT EVENT
`TYPE CORRECT?
`
`YES
`
`
`DETERMINE
`INTERPOLATION
`VALUE
`
`
`
`IS INPUT EVENT
`IN THE CORRECT
`AREA?
`
`
`
`TELL BEHAVIOR ROUTER
`TD ROUTE NEW
`BEHAVIOR STATE
`
`EXECUTION OFINPUT
`BEHAVIOR OF OBJECT
`
`Fig. /6
`
`20
`
`20
`
`

`
`U.S. Patent
`
`Mar. 16, 1999
`
`Sheet 19 of 19
`
`5,883,639
`
`H
`
`FILE
`
`VSE
`VSE GRAPHICS EDITOR
`
`fl
`
`DRAWING AIDS
`OPEN VIEW ANIMATE
`LIBRARY REC/PLAY CODE LINK
`
`VIEW
`
`—TUREGRUUNUDUAUNURUUNUD MAIN VSE VIEW
`
`0 EDIT
`
`1
`9 RUN
`
`BEHAVIOR EDITOR
`OUTPUT FUNCTION NAME: FZ1
`
`'7
`OUTPUT STATE:9.
`DEFINE OUTPUT
`CURRENT FUNCTION NAME: £145
`MOVE
`SCALE
`fiLEJ§TEE\LlJTNT)TTl|\OET&ME 3
`W W"
`'2
`“NW
`1 NEXT STATE; __Q__ V
`
`ERASE
`F‘/LL
`= [3
`0
`TEXT
`
`GROUP
`
`CUT
`
`PASTE
`
`UNGROUP
`
`COPY
`
`SELECT ALL
`
`DEFINE INPUT
`ENABLE FUNCTION NAME:
`ENABLE STATE: 9
`
`1
`
`21
`
`

`
`5,883,639
`
`1
`VISUAL SOFTWARE ENGINEERING
`SYSTEM AND METHOD FOR DEVELOPING
`VISUAL PROTOTYPES AND FOR
`CONNECTING USER CODE TO THEM
`
`CROSS-REFERENCES TO RELATED
`APPLICATIONS
`
`This is a continuation of application Ser. No. 08/695,540,
`filed Aug. 12, 1996 (abandoned), which was a continuation
`of application Ser. No. 08/383,560, filed Feb. 2, 1995 (now
`abandoned), which was a continuation of application Ser.
`No. 07/847,569, filed Mar. 6, 1992 (now abandoned), all of
`which are incorporated by reference herein for all that they
`disclose.
`
`COPYRIGHT NOTICE & AUTHORIZATION TO
`COPY MICROFICHE APPENDIX
`
`A portion of the disclosure of this patent document
`(APPENDIX A) is available on microfiche (3 microfiche,
`frames 1-229) and contains material which is subject to
`copyright protection. The copyright owner has no objection
`to facsimile reproduction by anyone of the patent document
`or the patent disclosure, (including microfiche APPENDIX
`A) as it appears in the Patent & Trademark Office patent file
`or records, but otherwise reserves all copyright rights what-
`soever.
`
`BACKGROUND OF THE INVENTION
`
`1. Field of the Invention
`
`The present invention relates to a graphical user interface
`design tool for creating interactive graphical user interfaces
`for hardware instruments and real-time systems such as
`VCRs, digital voltmeters, automobiles, and the like.
`2. Description of the Prior Art
`Most consumer products require some type of user inter-
`face. For example, automobiles have a speedometer and a
`steering wheel, while VCRs have switches and knobs and
`digital voltmeters have pushbuttons and LED read-outs.
`Such user interfaces often determine the market success of
`a consumer product because it
`is the primary point of
`interaction between the consumer product and the end user.
`As a result, without a user interface which enables the end
`user to properly interact with the product, even a techno-
`logically sound product will be rendered virtually useless to
`the end user.
`
`To overcome this problem, product developers have often
`created physical prototypes which they employ to test the
`usability of the product and the product’s interface.
`Typically, this is done before the product goes into final
`production. However, since creating a physical prototype
`requires the manufacture of physical components, building
`a physical prototype is an expensive and slow process. In
`fact, since the prototype often cannot be built until
`the
`underlying product is operational, the interface to the user
`has often delayed the introduction of the product.
`In an effort to save time and money, product developers
`have relatively recently begun to create graphical, software
`versions of their interfaces that may be employed to test the
`usability of their proposed product and interface. While such
`“simulated” user interfaces have sped up the prototyping
`process, such “simulated” user interfaces have also taken a
`great deal of time to create. Moreover, such “simulated” user
`interfaces have required computer graphics experts for their
`creation. In the face of the increasing software cost for the
`development of many consumer products, the use of com-
`
`10
`
`15
`
`30
`
`35
`
`40
`
`45
`
`SO
`
`55
`
`60
`
`65
`
`2
`puter graphics experts to create such “simulated” user inter-
`faces is no longer cost effective.
`A number of software tools, such as prototyping tools,
`system modeling tools, and system simulation tools have
`been used by such computer graphics experts for creating
`such “simulated” user interfaces in order to prove concepts
`and test ideas with end users. Such tools typically provide
`some form of animation of the models created so that the
`system “nins” in a manner similar to the final system. Such
`systems thus permit the product developer to build a “run-
`ning” model of a proposed system to show to an end user in
`order to get feedback on the product requirements. Ideally,
`such a system provides the “running” model without requir-
`ing additional “code” to be written while also providing the
`product developer with some aids for partitioning the system
`into its eventual hardware and software components.
`However, since the animation of a such a “simulated"
`prototype is provided by specifying the system behavior via
`unique, proprietary languages that must be learned by the
`product developer or iniplemented by a computer graphics
`expert, the resulting system is not reusable as the product
`moves into implementation, typically does not allow the
`developer to readily create custom graphics, and unduly
`adds to the expense of product development.
`More recently, graphics systems have been developed that
`allow product developers to graphically create user inter-
`faces by picking and placing pre-programmed graphical user
`interface components such as sliders or knobs on the display
`screen. Ilowever, the number of components available for
`such prior art systems is limited and the behavior of the
`interface components is limited to what the original creator
`of the components required. As a result, if a new or custom
`component needs to be added, then a graphics programming
`expert is still required to create the custom component. Also,
`the existing interface components of such systems are very
`diflicult to change or modify.
`Such prior art graphics tools are typically referred to as
`user interface toolkits, user interface builders, and user
`interface management systems. Such graphics tools aid the
`product developer with the creation of screen-oriented
`human interfaces which the product developer can then use
`to create the actual interface to be included with the product.
`Development of such screen-oriented human interfaces has
`been simplified by providing automatic generation of the
`code required to create a “production” interface; however,
`such graphics tools are specifically designed to meet the
`needs of only those product developers creating applications
`that have screen-oriented interfaces, as is typical of infor-
`mation systems. A system with a more general purpose
`application is desired.
`The latest prior art graphics systems allow product devel-
`opers to create output behavior (animation) for user interface
`objects by drawing the object with a graphical editor and
`then animating the object by textual description. Such sys-
`tems require the input behavior (such as how the graphical
`object receives and processes mouse or keyboard events) to
`be described textually or using some type of computer
`language. These input objects have also been limited to
`several specific types of input objects such as sliders or
`knobs. For example, a system of this type referred to as the
`SL-Graphical Modeling System (SL-GMS®) is available
`from SL Corporation. The SL-GMS® system purportedly
`allows programmers and non-programmers to use a drawing
`tool to design portable application interfaces,
`to connect
`screen objects to external data sources, to animate screen
`objects to reflect real-time changes in application data, and
`to use screen objects to control the application.
`
`22
`
`22
`
`

`
`5,883,639
`
`3
`The SL-GMS® system is a development system for use in
`building and managing graphics screens that can be embed-
`ded in a user application. The SL-GMS® system uses
`object-oriented architecture to provide direct, table-driven
`access to application data and over 40 pre-defined actions
`for screen objects. A drawing tool
`is also provided for
`creation of custom screen objects using a set of “building
`blocks” that the developer can use for modifications and
`extensions by calls to a function library. A mouse-driven
`graphics editor is also provided which allows users to create
`graphical objects and to position them on the screen. For
`example, a meter or gauge can be constructed from objects
`such as rectangles, lines and text. Once created, the appear-
`ance of each object can be modified by using the graphics
`editor to vary such standard graphical attributes as color, line
`width, fill percent and style, size, rotation, position and text
`font. Graphical attributes which can be specified by the
`graphics editor of the SL-GMS® system can also be modi-
`fied dynamically to reflect changes in external data. For
`example, this may be accomplished by causing the needle of
`a meter to rotate within a specified range as an external data
`variable changes. Also, a hierarchical, object-oriented
`approach is used to allow properties to be attached to all
`“instances” of the graphics elements. Functions then may be
`specified by the graphics editor to animate screen objects by
`establishing direct connections between the screen elements
`and application database variables. In this manner, screen
`objects may be animated using real-time data by the
`approximately 40 graphical action attributes allowed by the
`system. However, the SL-GMS® system is limited in that
`the o11tp11t and input behavior must be described textually.
`An improved system is thus still desired which may be easily
`used by direct manipulation.
`Other systems, such as the Dataviews® Graphics Soft-
`ware from V.I. Corporation, the Virtual Prototyping System
`(VAPS®) from Virtual Prototypes, Inc., the AC-100 System
`from Integrated Systems, Inc., and the Action!® System
`from ExperTelligence, Inc., have also been developed to
`allow a user to provide animated graphics in response to
`outputs from an application program. However, none of
`these systems allow definition of graphical input or output to
`the system in a completely graphical way. A system which
`is easy to use by direct manipulation of the objects is still
`desired.
`
`In addition to their difficulty of use, prior art graphical
`prototype development systems also have limitations in that
`they cannot break up hierarchial objects into their smaller
`components and cannot connect application code while the
`graphics editor is running. In addition, libraries of compo-
`nents have been difficult to use and create. A system with
`these and other features which is simple, easy to learn and
`flexible is desired. The present invention has been designed
`to meet these needs.
`
`SUMMARY OF THE INVENTION
`
`The present invention meets the above-mentioned needs
`in the art by providing a simple, easy to learn and flexible
`means of creating graphical user interfaces without knowl-
`edge of programming languages and without having to learn
`a large set of complicated commands. Instead, the Visual
`Software Engineering (“VSE”) system of the invention is
`premised on the simple concept of defining both input to and
`output from the graphical user interface by having the user
`provide examples of what he or she intends the object to do
`in response to certain user application inputs. The technique
`of the invention may be thought of as an extension of the
`aforementioned graphical prototype development systems to
`include “animation by example”.
`
`10
`
`15
`
`30
`
`35
`
`40
`
`45
`
`SO
`
`55
`
`60
`
`65
`
`4
`When designing a user interface in accordance with the
`present invention, the interface developer creates the desired
`user interface by first drawing the user interface with a
`graphics editor. The user then defines output behavior of (or
`animates) the user interface components by showing each
`state or frame of the animation. This is accomplished by
`changing the graphical objects in the user interface using a
`graphics editor function such as move or rotate and then
`storing with the graphical object in an object-oriented data-
`base each of the changed states of the graphical object as
`respective “frames” of graphical information. In the process
`of creating these states, the user may assign a frame name
`and a number to a series of stored frames as a behavior
`
`function. Also, just as with defining output (or animation),
`the input may be defined by giving the graphical object an
`example of what type of input to look for. Then, once the
`graphical object finds that input, the graphical object is told
`which frame to output or change to. Application code can
`then drive the animation or read the input by accessing the
`frame numbers assigned to each of the example frames.
`Such application code may be code running on a host, in an
`instruction set simulator, or in the physical device,
`for
`example. This allows users to migrate their application code
`from host to simulation to emulation.
`
`All graphical objects that are created in accordance with
`the invention can be stored in a library and reused. They can
`also be retrieved from the library, changed or customized
`through normal graphics editing commands, and then stored
`in the library as a new component. The user can also store
`the entire design of his or her user interface on disk and save
`the commands that are entered during the development of
`the interface so that these commands also can be replayed at
`a later time. Also, the system of the invention may also use
`interpolation when defining input and output behavior and
`may also provide a multiple View graphical editor.
`In
`addition, hierarchial objects may be created whereby indi-
`vidual elements of the hierarchy may always be accessed.
`In accordance with a preferred embodiment of the
`invention, a system is provided for creating and animating
`graphical objects by directly manipulating the graphical
`objects on a display screen. Such a system preferably
`comprises a graphics editor for creating and manipulating
`the graphical objects on the display screen under user
`control and means for animating the graphical objects by
`storing information representing manipulations of the
`graphical objects as respective behavior states of the graphi-
`cal objects and recreating the manipulations by specifying
`the corresponding behavior states of the graphical objects.
`The animated graphical objects are then displayed on the
`display screen. In the preferred embodiment, the animating
`means comprises a behavior editor which defines what input
`behavior a graphical object has by selectively mapping a
`user defined input region, behavior state name and value to
`a user input operation and which defines what output behav-
`ior a graphical object has by mapping the current behavior
`state of each graphical object to a user defined output state
`name and value. The behavior editor may also comprise
`means for sending a given behavior state to the graphical
`objects by specifying the output state name and value.
`In a preferred embodiment of the invention, the animating
`means comprises an example mouse input device with an
`associated example mouse pad for defining operations done
`with a hardware mouse input device with an associated
`hardware mouse pad. The example mouse comprises a
`graphical mouse object representing the hardware mouse
`with the associated hardware mouse pad on the display
`screen and means for manipulating the graphical mouse
`
`23
`
`23
`
`

`
`5,883,639
`
`5
`object so as to simulate the operation of the hardware mouse.
`Preferably, the manipulating means comprises means for
`pressing and releasing one or more buttons on the example
`mouse pad under control of the hardware mouse and asso-
`ciated hardware mouse pad, whereby a button to be
`depressed on the example mouse pad is selected by the user
`by using the hardware mouse and the button on the example
`mouse pad is depressed by the user by depressing a corre-
`sponding button on the hardware mouse pad. The manipu-
`lating means preferably also comprises means for moving
`the example mouse on its associated example mouse pad
`under control of the hardware mouse and associated hard-
`ware mouse pad. As a result, the example mouse is selected
`by the user by using the hardware mouse and is moved by
`moving the hardware mouse such that the example mouse is
`moved a corresponding amount on its associated example
`mouse pad on the display screen relative to the movement of
`the hardware mouse.
`
`the present invention further comprises an
`Preferably,
`interface for linking the behavior states of the graphical
`objects created by the graphics editor to user defined source
`code or an external database.
`
`The scope of the invention also includes a method of
`creating and animating graphical objects by directly manipu-
`lating the graphical objects on a display screen. Such a
`method in accordance with the invention preferably com-
`prises the steps of:
`drawing desired graphical objects with a graphics editor;
`manipulating the desired graphical objects to different
`display states;
`defining desired output behavior for the desired graphical
`objects for different input states of the desired graphical
`objects;
`storing the defined output behavior with the correspond-
`ing graphical object for respective input states; and
`animating the graphical objects by designating display
`states to be displayed and then displaying the desig-
`nated display states in a desired order as an animation.
`Preferably, the manipulating step of the method of the
`invention comprises a step of moving, rotating, scaling,
`stretching, cutting, filling, pasting, mapping or unmapping
`the graphical object. Also, the animating step preferably
`comprises the step of using an example mouse input device
`with an associated example mouse pad to define operations
`done with a hardware mouse input device with an associated
`hardware mouse pad. Preferably, the manipulating step of
`the method of the invention comprises the steps of pressing
`and releasing one or more buttons on the example mouse pad
`under control of the hardware mouse and associated hard-
`
`ware mouse pad. This can be accomplished by selecting a
`button to be depressed on the example mouse pad using the
`hardware mouse and depressing the button on the example
`mouse pad by depressing a corresponding button on the
`hardware mouse pad. At this time, the example mouse can
`be moved on its associated example mouse pad under
`control of the hardware mouse and associated hardware
`mouse pad by selecting the example mouse using the
`hardware mouse and moving the hardware mouse such that
`the example mouse is moved a corresponding amount on its
`associated example mouse pad on the display screen relative
`to the movement of the hardware mouse.
`The invention also preferably comprises a method of
`defining user input to graphical objects as respective behav-
`ior statcs of the graphical objccts. In accordance with the
`invention, such a method comprises the steps of:
`defining at least one region on a display screen relative to
`each graphical object which may receive user input;
`
`10
`
`15
`
`30
`
`35
`
`40
`
`45
`
`SO
`
`55
`
`60
`
`65
`
`6
`describing what input event types for the at least one
`region relative to each of the graphical objects can
`simulate output behavior states of the graphical objects
`which control output behavior for the graphical objects;
`defining desired input behavior states for the graphical
`objects which can simulate the output behavior states of
`the graphical objects;
`storing the defined input behavior states with correspond-
`ing graphical objects for respective output behavior
`states; and
`stimulating the graphical objects with the defined input
`behavior states so as to cause a change in at least one
`output behavior state of at least one of the graphical
`objects, thereby changing a display state of at least one
`of the graphical objects.
`Such a method in accordance with the invention may also
`further comprise the step of retrieving graphical objects with
`pre-defined input behavior states from a library and assign-
`ing input behavior of the retrieved graphical objects to the
`graphical objects being described in the describing step.
`The VSE system of the invention so designed saves
`development time and cost in that the software interface
`eliminates the need to develop early hardware prototypes to
`demonstrate the appearance and behavior of the system. The
`software interface is also much easier, far less expensive,
`and less time consuming to modify than a hardware imple-
`mentation. It can also be readily tailored