`
`SA
`US 20020186245A1
`
`as) United States
`a2) Patent Application Publication (10) Pub. No.: US 2002/0186245 Al
`
` Chandhokeetal. (43) Pub. Date: Dec. 12, 2002
`
`
`(54) SYSTEM AND METHOD FOR
`CONFIGURING A HARDWAREDEVICE TO
`EXECUTE A PROTOTYPE
`
`(76)
`
`Inventors: Sundeep Chandhoke, Austin, TX (US);
`Nicolas Vazquez, Austin, TX (US);
`Kevin L. Schultz, Georgetown, TX
`(US)
`
`Correspondence Address:
`Jeffrey C. Hood
`Conley, Rose & Tayon, P.C.
`P.O. Box 398
`Austin, TX 78767 (US)
`:
`10/200,091
`
`(21) Appl. No.:
`
`(22)
`
`Filed:
`
`Jul. 19, 2002
`
`Related U.S. Application Data
`(63) Continuation-in-part of application No. 09/595,003,
`filed on Jun. 13, 2000, Continuation-in-part of appli-
`cation No. 09/745,023, filed on Dec. 20, 2000. Con-
`linuation-in-part of application No. 09/949,783, filed
`on Sep. 10, 2001.
`
`Publication Classification
`
`(51) Wnts.” sacsczcsccceasnscancasnntemnniaaic G09G 5/00
`(52) US. C0.
`cecccsssssssnssessvesenstnsevsnensnesnnsssnesensee 345/764
`
`5
`(57)
`
`rey —
`ABSTRACT
`
`Asystem and methodfor configuring a hardware device to
`execute a prototype. The hardware device may include a
`programmable hardware element and/or a processor and
`memory. The prototype may comprise a software entity
`operable to perform a plurality of executable operations,
`wherein the prototype is created by a user without requiring
`user programming. For example, a prototyping environment
`may provide a library of operations that are specific to a
`particular problem domain and may enable the user to select
`:
`.
`:
`.
`. .
`various operations from the library for
`inclusion in the
`prototype, The prototyping environment may include a
`graphical user interface that is streamlined for interactively
`experimenting with various parameters or properties asso-
`ciated with the selected operations and seeing the effects of
`adjusting the parameters.
`
`
`
`
`
`Display a graphical userinterface of a prototyping environment,
`wherein the graphical user interface provides graphical access to a set
`of operations
`401
`
`
`
`
`
`|Le
`
`
`
`
`Receive userinput to the graphical user interface specifying desired
`operationsto include in a prototype
`403
`
`7
`
`Deploy the prototype on a hardware device
`405
`
`|
`
`
`The hardware device executes to perform the functionality specified by
`the prototype
`407
`
`1
`
`Shenzhen Tuozhu 1011
`
`1
`
`Shenzhen Tuozhu 1011
`
`

`

`Patent Application Publication
`
`Dec. 12, 2002
`
`Sheet 1 of 49
`
`US 2002/0186245 Al
`
`Jajndwo9
`
`waysks
`
`a)
`
`28
`
`2
`
`

`

`Patent Application Publication
`
`Dec. 12, 2002 Sheet 2 of 49
`
`
`
`spueoguolisinboy
`
`eyeq-ul-Bnid
`
`viI
`
`&
`
`wo”
`
`29YIOMJAN
`
`peseg-sejndwog
`
`S]USWNJ}SU|
`
`BEL,
`
`@uonisinboy
`
`sbeu
`
`
`
`}S9}Jepuf)UN)
`
`US 2002/0186245 Al
`
`&ois
`
`El
`
`
`
`JOs}UODUOROY|OS|
`
`3
`
`
`
`

`

`Patent Application Publication Dec. 12,2002 Sheet 3 of 49
`
`US 2002/0186245 Al
`
`oN
`
`“Ss
`
`y 1”
`I
`aaeee
`
`aa
`
`FIG. 3A
`
`
`
`4
`
`

`

`Patent Application Publication Dec. 12,2002 Sheet 4 of 49
`
`US 2002/0186245 Al
`
`
`
`5
`
`

`

`Patent Application Publication
`
`Dec. 12, 2002 Sheet 5 of 49
`
`US 2002/0186245 Al
`
`LI
`
`ejeqebeu
`
`
`
`Jayjouogsng
`
`gol
`
`
`
`OZ}snguolsuedx3
`
`oaplA Z9LSng}SOH
`uoljisinboyuonisinboy(0nlso5,WoopsSAUPJeH
`
`
`
`
`GSOld
`
`vil|vel
`
`pegSdlAeq
`BELZe
`
`08t
`
`AIOws/\
`
`J8}|OUOD
`
`v9L
`
`Xo
`oO©0
`OoOo=
`
`
`
`Aiowseyyuleyy
`
`901XK
`
`c8
`
`6
`
`
`
`
`

`

`Patent Application Publication
`
`Dec. 12, 2002 Sheet 6 of 49
`
`US 2002/0186245 Al
`
`eSS®c<3Oo
`
`nNo“I
`
`V9Sls
`
`a[hejOoAUON
`
`80zsngje007
`
`gLzsngeyeq/|OujUOD
`
`sJEMPIeH
`
`(W9d3)
`
`902
`
`
`
`ajqewwesbold_
`
`
`
`——soepe}u|sng
`
`912
`
`Aiowsy|
`
`88
`
`JOSS990Jd
`
`Che
`
`7
`
`

`

`Patent Application Publication Dec. 12,2002 Sheet 7 of 49
`
`US 2002/0186245 Al
`
`FIG.6B
`
`
`
`©x
`
`Nu
`
`d=
`
`faa)
`ow—_
`a
`QO=
`°o
`=
`oO
`oO
`
`uw
`—_
`a
`
`IO Connector 202
`
`LocalBus208
`
`Processor 212
`
`
`
`
`
`
`
`o
`8
`‘E
`oo
`n
`
`— o
`
`8
`
`

`

`Patent Application Publication Dec. 12,2002 Sheet 8 of 49
`
`US 2002/0186245 Al
`
`IO Connector 202
`
`FIG.6C
`
`Interface 216
`
`&swo
`EZO0OS8
`o PAWN
`mH © kL
`o Lt
`ta
`Oo
`
`LocalBus208
`
`Control/DataBus218
`
`®@
`
`Bus
`
`9
`
`

`

`Patent Application Publication
`
`Dec. 12, 2002 Sheet 9 of 49
`
`US 2002/0186245 Al
`
`V2‘Sls
`
`elowed 8LZSngB}eq/|OjU0D
`
`782
`
`ejqewwesbold
`
`aJeMpleH
`
`(vOd4)
`
`—g90epaju]sn
`902Ha}U]SNg
`
`91Z
`
`
`
`Aiowayeeu}
`
`y9z
`
`9[IEJOA-UOCN
`
`[ows
`
`88z
`
`JOSS99014
`
`Zz
`
`VO6L
`
`
`
`BIOWeDYWEWS
`
`10
`
`10
`
`

`

`Patent Application Publication
`
`Dec. 12, 2002 Sheet 10 of 49
`
`US 2002/0186245 Al
`
`dZ‘Old
`
`elawen Sizsngeyeq/|oujuog
`
`
`
`Kiowayye6euu|
`
`>8Z
`
`G06}
`
`
`
`elawegWewWS
`
`282
`
`ELS
`
`—
`
`JOSSe001q
`
`
`
`eoepaju|sng
`
`912
`
`J9}UNOD/JOW!|
`
`91607
`
`982
`
`11
`
`11
`
`

`

`Patent Application Publication
`
`Dec. 12, 2002 Sheet 11 of 49
`
`US 2002/0186245 Al
`
`Jz
`‘Old
`
`282
`
`BOWED @lzsngeyeq/joUOD
`
`
`
`Aioweyyebew)
`
`v8z
`
`JayUNODOWI]
`
`91607
`
`98z
`
`S061
`
`elawe)WeWS
`
`
`
`ajqewwesboldAuowey
`
`eueMpley99z
`
`__
`
`(¥Od4)
`
`
`
`__naneaoeyej}u|sng
`
`9|I}2|OA-UON
`
`912
`
`12
`
`12
`
`

`

`Patent Application Publication Dec. 12,2002 Sheet 12 of 49
`
`US 2002/0186245 Al
`
`Display a graphical user interface of a prototyping environment,
`wherein the graphical user interface provides graphical access to a set
`of operations
`401
`
`Receive userinput to the graphical user interface specifying desired
`operations to include in a prototype
`403
`
`
`
`407
`
`
`
`Deploy the prototype on a hardware device
`405
`
`The hardware device executes to perform the functionality specified by
`the prototype
`
`FIG. 8
`
`13
`
`13
`
`

`

`Patent Application Publication Dec. 12,2002 Sheet 13 of 49
`
`US 2002/0186245 Al
`
`(Step 403)
`
`
`
`Add an operation to the prototype in responseto userinput
`421
`
`
`|
`
`Update graphical userinterfaceto illustrate effect of including the
`operation in the prototype
`423
`
`429
`
`Display a graphical panel for configuring the operation
`425
`
`Receive userinput to the graphical panel for configuring the operation
`427
`
`
`Update graphical user interfaceto illustrate effect of configuring the
`operation
`
`FIG. 9
`
`14
`
`14
`
`

`

`Patent Application Publication Dec. 12,2002 Sheet 14 of 49
`
`US 2002/0186245 Al
`
`(Steps 405 and 407)
`
`
`
`
`
`Programmatically generate a
`
`graphical program based on the
`prototype
`431
`
`
`
`
`Modify the graphical program in
`responseto userinput
`433
`
`Deploy the graphical program on
`the hardware device
`435
`
`
`
`
`
`
`
`
`
`The hardware device executes to
`
`perform the functionality specified
`by the graphical program
`437
`
`
`FIG. 10
`
`15
`
`15
`
`

`

`Patent Application Publication
`
`Dec. 12, 2002 Sheet 15 of 49
`
`US 2002/0186245 Al
`
`
`
`mopulj\,Buisse001gMOPUI\BOUSIBJOY,
`
`
`
`
`
`
`—_—<—<—<—$———/syusunnsuyrueneNyFl
`
`GOHFNSMengqeosie)270)eu]oeISI,OVW
`
`
`
`ASEee
`
`’-4Ae“.fmr”,ie
`
`
`
`labeipelsae[uresalore
`
`MOPUI\\WOOZ
`
`
`
`azisgabew|MOPUIAAGUIS
`
`16
`
`16
`
`
`
`

`

`Patent Application Publication
`
`US 2002/0186245 Al
`
`Dec. 12, 2002 Sheet 16 of 49
`
`HORNE]
`savanna)Sj(OCSPefindSeespina](e[eel]ifra‘aP|HeeaeeligUCISIAa’VWieee
`
`
`
`£9refvj>iece
`
`
`02°00LOAD]Pg
`
`
`ooze“OrFeA,UTEOOS2«=aTeAuNUXe}y
`-LPPLergotoy]
`SerJOFUN
`
`Com
`foloS10eetahrxwearifey«[eysce
`Cwm:poi}
`
`~~|>|Ha]4by
`[8eels
`apeAS
`
`oor
`
`17
`
`17
`
`

`

`Patent Application Publication
`
`Dec. 12, 2002 Sheet 17 of 49
`
`US 2002/0186245 Al
`
`*.
`
`
`
` eStoreSe|peCONeTYKj
`————————————————————,=~squamNE]
`
`
`
`deyduosAmugaeosteigUOISIAwoVWI
`
`COSE2«=HEAUINURe|0051SEAUTR
`LE99L‘engAURe
`
`.Swe*ae7«Ne
`Cartageleoflt
`..eeeJi2teitetavasbt?ao
`someeeBeeFe
`‘.7
`ZISCISWAGOLRAN
`
`oO,HF]
`
`[riziexzic
`
`18
`
`18
`
`
`

`

`f
`
`3 § :i 5 5 &
`
`oo
`
`19
`
`Patent Application Publication Dec. 12,2002 Sheet 18 of 49
`
`US 2002/0186245 Al
`
`.i
`
`e
`
`aa
`
`ll
`a
`
`él]
`;
`
`19
`
`

`

`Patent Application Publication
`
`Dec. 12, 2002
`
`Sheet 19 of 49
`
`US 2002/0186245 Al
`
`
`
`Baayen):ppoyessy|ony
`
`
`
`Adoyuy‘pjoysas4|ony
`
`
`
`Sua-ploysal4|ONY
`
`
`
`aainosafew|
`
`poyaery|
`
`PEMONeN|
`
`
`sjuawNysuy]
`
`20
`
`20
`
`

`

`i 3‘ 3 : & 2
`
`1
`
`Patent Application Publication Dec. 12,2002 Sheet 20 of 49
`
`US 2002/0186245 Al
`
`21
`
`

`

`Patent Application Publication
`
`Dec. 12, 2002 Sheet 21 of 49
`
`
`
`
`
`[LUOfe
`
`PSEPokedPein[otiesfora
`dayjduosfleuigajeasiouyigo)afew)oj
`pleawe
`
`2QMEUGIH-UOYN|CAUOT)S144
`pyoysasy|PRUEH“pjajsaniy
`
`US 2002/0186245 Al
`
`>behCLGMZIC
`[one
`
`
`
`
`
`
`
`
`
`
`
`3PI07BADUAY‘ADO|OLCIOW“APY
`
`
`
`Sapoy|MBojeycioy“apy
`
`
`
`uitag
`
`USM]
`
`22
`
`22
`
`

`

`Patent Application Publication
`
`Dec. 12, 2002 Sheet 22 of 49
`
`US 2002/0186245 Al
`
`8bSls
`
`
`
`BuisseooigJayeabew|
`
`jeulbuo
`
`(B
`
`df-¢e}9\))
`ebew
`
`23
`
`23
`
`

`

`Patent Application Publication
`
`Dec. 12, 2002 Sheet 23 of 49
`
`US 2002/0186245 Al
`
`6}OldTas
`
`
`
`yoaloideazesauaBb0]ay)NOApinoy,
`
`
`
`yoaloudajoumM,3
`
`3}j8UQ
`
`allajbuis240
`
`suninjos\'*'\:3
`
`;Uo}]e307
`
`;aWeN
`
`332[04q|
`
`
`
`IISEG|ENSIAIJOSOADIW“EE
`
`
`
`
`
`TAD/SMOpUIMge)Gp
`
`
`
`
`
`++)|ENSIAYOSOIIWGO
`
`a44aping§E7
`
`24
`
`24
`
`
`

`

`Patent Application Publication
`
`Dec. 12, 2002 Sheet 24 of 49
`
`US 2002/0186245 Al
`
`02Sls
`
`|jasue>
`
`|Yysiul|<aN|ypeg>
`
`TS
`
`yyedqdi3s
`
`
`
` ysiquopaaesduosyyuoSUAaIND43
`
`25
`
`
`
`
`
`1]0l2SBINSBj]JIBjasaseajd
`
`Tdays-
`
`
`
`pieziasUOIesIUayBpojss]
`
`25
`
`
`
`

`

`Patent Application Publication
`
`Dec. 12, 2002 Sheet 25 of 49
`
`US 2002/0186245 Al
`
`beSls
`
`||a2Ue>ysiul4
`
`<xan|yoeg>
`
`puezimGAL
`(NOAJDJsallyGuimoyjoyau]Sa}e9.9
`
`
`
`
`pdays-paezi~uolyesauayapo)x
`
`__|
`
`VOWELSj
`
`AABWLUNG
`
`26
`
`26
`
`
`
`
`
`

`

`Dec. 12, 2002 Sheet 26 of 49
`
`US 2002/0186245 Al
`
`Patent Application Publication
`
`
`—_]e¢1?e|_ajapiojoeeyOYA«©PICHIKDV)Pl4se4LOYA]
`
`
`PnjonicyOVWG01-}00'0-{00!-}iTop
`foofoofoo||-oe
`
`
`O00OE
`
`atedOYA
`
`27
`
`27
`
`

`

`Patent Application Publication Dec. 12,2002 Sheet 27 of 49
`
`US 2002/0186245 Al
`
`
`
`
`
`Create an image
`processing prototype
`
`861
` Programmatically create
`
`a graphical program
`
`
`based on the prototype
`862
`
`
`
`
`Couple an image
`acquisition device to the
`
`
`computer system
`864
`
`
`
`
`Download a hardware
`
`
`configuration based on
`the graphical program to
`
`
`a programmable
`hardware element in the
`image acquisition device
`
` 866
`
`
`
`
`Optionally disconnect the
`image acquisition device
`from the computer
`system; deploy the
`
`
`image acquisition device
`868
`
`
`
`
`
`Image acquisition device
`acquires an image of an
`object
`
` 870
`
`
`
`Image acquisition device
`executes to perform the
`image processing
`
`
`function on the image
`872
`
`
`
`
`
`Present a user interface
`onadisplay
`aA
`FIG. 23
`
`
`28
`
`28
`
`

`

`Patent Application Publication Dec. 12,2002 Sheet 28 of 49
`
`US 2002/0186245 Al
`
`
`
`Create an image
`
`processing prototype
`
`861
`
`
`
` Programmatically create
`
`a graphical program
`based on the prototype
`862
`
`
`
`
`Couple an image
`
`acquisition device to the
`
`
`computer system
` 864
`
`
`
`
`
`Download the graphical
`
`program to the memory
`in the image acquisition
`
`
`device
`
`867
` Optionally disconnect the
`
`
`
`
`image acquisition device
`
`from the computer
`
`system; deploy the
`image acquisition device
`868
`
`
`
`
`Image acquisition device
`acquires an imageof an
`object
`870
`
`
`
`Image acquisition device
`executes to perform the
`image processing
`function on the image
`
`873 Present a userinterface
`
`on a display
`ae
`
`29
`
`FIG. 24
`
`29
`
`

`

`US 2002/0186245 Al
`
`909
`
`atedPart*
`
`Sorgen
`
`arnee
`
`etyEe
`
`=:
`
`easBeSoe
`
`
`
`“A+t=Aa32=Z809Q=old==3a
`4adopaaague Patent Application Publication
`
`
`
`
`009
`
`30
`
`30
`
`

`

`Patent Application Publication
`
`Dec. 12, 2002
`
`Sheet 30 of 49
`
`US 2002/0186245 Al
`
`aseSls
`
`
`
`
`
`aLea
`
`eeecheeeeicsny
`
`PraRisee
`
`SoateS
`
`a
`
`31
`
`31
`
`

`

`Patent Application Publication
`
`Dec. 12, 2002
`
`Sheet 31 of 49
`
`US 2002/0186245 Al
`
`OSC
`Sls
`
`s1yjtuy
`
`x
`
`crest} eer
`
`32
`
`32
`
`

`

`Patent Application Publication
`
`Dec. 12, 2002
`
`Sheet 32 of 49
`
`US 2002/0186245 Al
`
`as¢‘Old
`
`iPeerrfa
`
`00g
`Ranevans
`
`fayeaaaa
`
`33
`
`33
`
`

`

`Patent Application Publication
`
`Dec. 12, 2002
`
`Sheet 33 of 49
`
`US 2002/0186245 Al
`
`ds¢Sls
`
`
`
`34
`
`34
`
`

`

`Patent Application Publication
`
`Dec. 12, 2002
`
`Sheet 34 of 49
`
`US 2002/0186245 Al
`
`dS¢‘Sls
`
`lieSeeSeebeees
`
`
`
`
`
`PeetpisiMa
`
`
`
`35
`
`35
`
`

`

`Patent Application Publication
`
`Dec. 12, 2002
`
`Sheet 35 of 49
`
`US 2002/0186245 Al
`
`|
`
`Teneota.
`
`
`
`GarSROPOM:(e8HOSOL”
`
`NEeeSeet
`
`
`
`Voc“Old
`
`Saree
`esa
`
`DoodGeGaTET”
`
`36
`
`36
`
`

`

`Patent Application Publication
`
`Dec. 12, 2002
`
`Sheet 36 of 49
`
`US 2002/0186245 Al
`
`a9¢Old ==a=anee5aepronto“
`
`
`
`vesicbhaita'J|PaRAun
`
`
`
`23P3x
`
`oes
`
`
`
`2GHMOpUM:ORSPOT.ART,
`
`37
`
`37
`
`
`

`

`Patent Application Publication
`
`Dec. 12, 2002
`
`Sheet 37 of 49
`
`US 2002/0186245 Al
`
`38
`
`38
`
`
`

`

`Patent Application Publication
`
`Dec. 12, 2002
`
`Sheet 38 of 49
`
`US 2002/0186245 Al
`
`d92‘Sls
`
`panowqusey2070=JT
`
`20)
`
`39
`
`39
`
`

`

`Patent Application Publication
`
`Dec. 12, 2002
`
`Sheet 39 of 49
`
`US 2002/0186245 Al
`
`d9¢Ola
`
`T=Er
`
`Taeae[eezoman
`
`40
`
`40
`
`

`

`Patent Application Publication
`
`Dec. 12, 2002
`
`Sheet 40 of 49
`
`US 2002/0186245 Al
`
`A9¢‘Sls
`
`
`
`41
`
`41
`
`

`

`Patent Application Publication
`
`Dec. 12, 2002
`
`Sheet 41 of 49
`
`US 2002/0186245 Al
`
`pe
`
`992‘Sls
`
`
`
`‘fsgketeeeescatatates
`
`*eedesteoth
`
`illem@opeaeee
`
`Tes
`
`|ii}
`
`Seeecs
`
`eae
`
`eeae ae
`
`—————
`
`a mEETS
`
`42
`
`42
`
`
`
`
`
`
`
`

`

`Patent Application Publication Dec. 12,2002 Sheet 42 of 49
`
`US 2002/0186245 Al
`
`Create a motion control
`prototype
`964
`
`Programmatically create
`a graphical program
`based onthe prototype
`962
`
`Couple a motion control
`device to the computer
`system
`964
`
`Download a hardware
`configuration based on
`the graphical program to
`a programmable
`hardware elementin the
`motion control device
`966
`
`Optionally disconnect the
`motion control device
`from the computer
`system; deploy the
`motion control device
`968
`
`Motion contro! device
`executes to perform the
`motion control function ,
`e.g., to control movement
`of one or more objects
`972
`
`974
`
`Present a user interface
`on a display
`
`FIG. 27
`
`43
`
`43
`
`

`

`Patent Application Publication Dec. 12,2002 Sheet 43 of 49
`
`US 2002/0186245 Al
`
`Create a motion control
`prototype
`961
`
`Programmatically create
`a graphical program
`based on the prototype
`962
`
`Couple a motion control
`device to the computer
`system
`964
`
`Download the graphical
`program to the memory
`in the motion control
`device
`967
`
`Optionally disconnect the
`motion control device
`from the computer
`system; deploy the
`motion control device
`
`974
`
`executes to perform the
`motion control function ,
`@.g., to control movement
`of one or more objects
`
`Present a userinterface
`on a display
`
`FIG. 28
`
`44
`
`44
`
`

`

`US 2002/0186245 Al
`
`62‘Sls
`
`Patent Application Publication
`
`Dec. 12,2002 Sheet 44 of 49
`
`
`
`wieiBold1aAeS
`
`90S
`
`Ajjeonewwesbod0}|d¥
`
`jeoiudesBeyipeyayeaso
`
`wesBoid
`
`vos
`
`jual|D
`
`wesBod
`
`z0S
`
`45
`
`45
`
`

`

`Patent Application Publication
`
`Dec. 12, 2002
`
`Sheet 45 of 49
`
`US 2002/0186245 Al
`
`0€‘Sls
`
`
`
`
`
`jeuoljoun4aJeEMPJEHaJeMpley
`
`
`
`
`
`aJempleyISINONuondioseq
`
`80€90€vOE
`
`
`
`apoysulyoe|\payidwio.)uondiuoseg
`
`
`
`
`
`ve(apogeounog@POD891NOS
`__ZLEOle
`
`
`eel,wesbeigaeoudessadAjojold
`—_ToS__
`
`cOeweJbolg008
`
`
`
`46
`
`46
`
`
`
`
`

`

`VHDLBackEnd330A Dec. 12, 2002 Sheet 46 of 49
`
`GraphicalProgram 301
`
`EDIFBackEnd330B
`
`Tree 302
`
`==D
`
`a
`
`©Q
`
`O>
`
`Patent Application Publication
`
`US 2002/0186245 Al
`
`FIG.31
`
`CBackEnd330C
`
`47
`
`47
`
`

`

`“81d0Aleiqr
`
`uoljoun4payidwog
`
`syo0|g
`
`Zve
`
`aJeEMpJeH
`
`
`
`oyloadsjebiel
`
`UOIJEWIOJU|
`
`bye
`
`Dec. 12, 2002
`
`Sheet 47 of 49
`
`US 2002/0186245 Al
`
`ce’Slq
`
`Patent Application Publication
`
`pewiuesbogV9Odsoyiseds-WOd4SJEMPIE}HwesbeigAjeatydeig
`
`
`
`
`
`
`
`YOdsol!wesBoldISIN®Neyeqwes6oig’||edAjojolguojdiiosaqoe
`
`SbeOpeOvev0eZ0€sainjonys00€
`
`—a=a———-LOE
`
`48
`
`48
`
`
`
`

`

`Patent Application Publication
`
`Dec. 12, 2002
`
`US 2002/0186245 Al
`
`ce“Old
`
`Sheet 48 of 49
`
`
`
`
`
`
`
`
`
`
`
`SPO)SUNOSpoyidwioy)uondusseqgai)weibeig”eyeqwesboigadAjojold
`
`(apoyaainosapogseZLEOle—Loe__pleadinosZOESaIN}ONys00g
`
`
`
`
`PoAUFapoyaulyoeyyaIeMYOSjeaiudeis
`
`
`
`
`
`
`
`
`
`
`
`yalqoUeugSOLJ2Jq!]SSEID
`
`seuelqi]
`
`SoUeIgr]one
`
`__ose
`
`ZSESUONIUYOC
`
`49
`
`49
`
`
`
`

`

`Patent Application Publication
`
`Dec. 12, 2002 Sheet 49 of 49
`
`US 2002/0186245 Al
`
`Programmatically create
`graphical program from
`
`prototype
`510
`
`
`
`
`
`
`Selecta portion of the
`graphical program to
`export to a hardware
`implementation
`512
`
`
`
`
`
`Build a VDiagram tree
`
`representing the selected
`program portion
`
`
`514
`
`
`
`
`
`
`
`
`Compile unselected
`
`portion of the graphical
`program into machine
`
`
`codefor oOexecution
`
`
`5
`
`
`
`
`Library of pre-compiled
`function blocks
`
`542
`
`
`
`Call an appropriate back
`
`end program to generate a
`
`
`hardware description from
`the eee tree
`
`
`
`
`Hardware target specific
`information
`544
`
`Convert the hardware
`description to an FPGA-
`specific netlist
`518
`
`Compile thenetlist into an
`FPGAprogram file
`520
`
`
`
`
`Transfer FPGA program
`file to programmable
`hardware (FPGA)to
`produce programmed
`hardware equivalent to
`graphical program
`522
`
`
`
`
`FIG. 34
`
`50
`
`50
`
`

`

`US 2002/0186245 Al
`
`Dec. 12, 2002
`
`SYSTEM AND METHOD FOR CONFIGURING A
`HARDWAREDEVICE TO EXECUTE A
`PROTOTYPE
`
`CONTINUATION DATA
`
`[0001] This application is a continuation-in-part of U.S.
`patent application Ser. No. 09/595,003 titled “System and
`Method for Automatically Generating a Graphical Program
`to Implement a Prototype,” filed Jun. 13, 2000, whose
`inventors were Nicolas Vazquez, Jeffrey L. Kodosky, Ram
`Kudukoli, Kevin L. Schultz, Dinesh Nair, and Christophe
`Caltagirone.
`
`[0002] This application is also a continuation-in-part of
`U.S. patent application Ser. No. 09/745,023 titled “System
`and Method for Programmatically Generating a Graphical
`Program in Response to Program Information”,filed Dec.
`20, 2000, whose inventors were Ram Kudukoli, Robert Dye,
`Paul F. Austin, Lothar Wenzel, and Jeffrey L. Kodosky.
`[0003]
`‘This application is also a continuation-in-part of
`U.S. patent application Ser. No. 09/949,783 titled “System
`and Method for Deploying a Graphical Program on an
`Image Acquisition Device”, filed Sep. 10, 2001, Kevin L.
`Schultz, Jeffrey L. Kodosky, Hugo Andrade, Brian Keith
`Odom and Cary Paul Butler.
`
`FIELD OF THE INVENTION
`
`invention relates to a system and
`[0004] The present
`method for configuring a hardware device to execute a
`prototype. The hardware device may include a program-
`mable hardware element and/or a processor and memory.
`The prototype may comprise a software entity operable to
`perform a plurality of executable operations, wherein the
`prototype is created by a user without requiring user pro-
`gramming.
`
`DESCRIPTION OF THE RELATED ART
`
`[0005] The continually improving price and performance
`of today’s computers make computer-based measurement
`and automation systems more accessible to users who were
`previously limited by inflexible, costly measurement and
`automation methods. By employing computer-based mea-
`surement and automation systems, users can lower capital
`costs, system development costs, and system maintenance
`costs, while at the same time improving the time to market
`and quality of their products. Virtual instrumentation has
`been an industry paradigm shift. Through the power of
`computer-based tools, scientists and engineers can build
`their own solutions exactly suited to their needs rather than
`being confined to a limited, inflexible, and expensive selec-
`tion of fixed-function, traditional instruments.
`
`[0006] Today, many hardware devices includeat least one
`functional unit, such as a processor and memory, or a
`programmable hardware element, which enables the device
`to be programmed with or execute program instructions,
`e.g., to control the device. For example, a “smart camera”
`may include a functional unit operable to execute program
`instructions to acquire and analyze an image, ¢.g., to check
`for visual defects present in a manufactured object. Inas-
`much as a primary goal of virtual
`instrumentation is to
`provide the user the maximum amount offlexibility to create
`his/her own applications and/or define his/her own instru-
`ment functionality, it is desirable to extendthe level at which
`the user is able to program a device, ¢.g., extend the level at
`
`which a user of instrumentation or industrial automation
`hardwareis able to program an instrument. The evolution of
`the levels at which the user has been able to program an
`instrument is essentially as follows.
`
`1. User level software (LabVIEW, LabWin-
`[0007]
`dows CVI, Visual Basic, etc.)
`2. Kernel level software
`
`[0008]
`
`3. Auxiliary kernel level software (a second
`(0009]
`kernel running along side the main OS, e.g., InTime,
`VentureCom, etc.)
`
`[0010]
`
`4. Embedded kernel level software
`
`[0011]
`
`5. Hardware level software (FPGA)
`
`In general, going down the abovelist, the user is
`[0012]
`able to create software applications which provide a more
`deterministic real-time response. Currently, some program-
`ming development
`tools for instrumentation or industrial
`automation provide an interface at level 1 above. In general,
`most users are unable and/or not allowedto program at the
`kernel level or auxiliary kernel level.
`It would be highly
`desirable to provide the user with the ability to develop user
`level software which operates at the embeddedkernel level
`and/or the hardware level.
`
`[0013] Atype of software application referred to herein as
`a “prototyping environment” is becoming increasingly
`popular. In general, a prototyping environment may refer to
`a specialized application that provides an environment that
`is conducive to rapidly and conveniently prototyping a
`problem solution, i.c., a prototype. As used herein, a “pro-
`totype” may comprise a software entity operable to perform
`a plurality of executable operations, wherein the prototype is
`created by a user without requiring user programming. For
`example, the user may create the prototype by interacting at
`a high level with a graphical user interface ofthe prototyping
`environment. One example ofa prototyping environment is
`an image processing prototyping environmentthat enables a
`user to easily develop a script or sequence, i.e., a prototype,
`including various operations that are applied to images, ¢.g.,
`for use in a machine vision, pattern matching, shape match-
`ing, or other imaging application. The prototyping environ-
`ment may enable the user to create the prototype without
`requiring the user to write code in a programming language
`or minimizing the amount of code the user would otherwise
`have to write. Thus, a prototyping environment may enable
`a user to develop a solution for a problem by working at a
`higher level of abstraction than a programming environment
`provides.
`
`It would be desirable to provide the user with the
`[0014]
`ability to develop a prototype using a prototyping environ-
`ment, which can then be readily deployed onto a hardware
`device. This would provide the user with the dual benefits of
`being able to program device functionality at the highest
`level possible (¢.g., prototypes), while also providing the
`ability to have program instructions execute directly on an
`embedded processor or in hardware for increased speed and
`efficiency.
`
`SUMMARY OF THE INVENTION
`
`[0015] One embodiment of the invention comprises a
`system and method for configuring a hardware device to
`execute a prototype. The hardware device may include a
`
`51
`
`51
`
`

`

`US 2002/0186245 Al
`
`Dec. 12, 2002
`
`programmable hardware element and/or a processor and
`memory. The prototype may comprise a software entity
`operable to perform or specify a plurality of executable
`operations, wherein the prototype is created by a user
`withoul requiring user programming. The prototype may be
`crealed using a prototyping environment application. For
`example, the prototyping environment may provide a library
`of operations that are specific to a particular problem domain
`and may enable the user to select various operations from the
`library for inclusion in the prototype. The prototyping envi-
`ronment may include a graphical user interface that
`is
`streamlined for interactively experimenting with various
`parameters or properties associated with the selected opera-
`tions and seeing the effects of adjusting the parameters.
`
`[0016] The prototyping environment may be used to gen-
`erate a sequence, solution, or script, also called a prototype,
`which represents an algorithm or process designed by the
`user in the prototyping environment. For example,
`the
`prototype may include a plurality of operations for imple-
`menting or specifying a solution for a motion control
`application, a data acquisition andcontrol application, a test
`and measurement application, an image processing or
`machine vision application, a process control application, a
`man-machine interface application, a simulation application,
`a hardware-in-the-loop validation application, or any of
`various other types of applications.
`
`[0017] After the prototype has been created, the prototype
`may be deployed on the hardware device. The hardware
`device may be coupled to the computer system to facilitate
`the deployment of the prototype. In one embodiment,
`the
`hardware device may be comprised in the computer system,
`In other embodiments the hardware device may be coupled
`to the computer system through a network, such as the
`Internet, or may be coupled to the computer system through
`wireless means.
`
`[0018] Deploying the prototype on the hardware device
`may comprise configuring the hardware device to perform at
`least a portion of the operations in the prototype. In one
`embodiment, a program based on the prototype may be
`programmatically generated, and deploying the prototype
`may comprise storing this program or a user-modified ver-
`sion of the program on (at least part of) a “functional unit”
`of the hardware device, or configuring the functional unit
`with all or part of the program, such that after the deploy-
`ment, the functional unit is operable to execute or implement
`the functionality of the program.
`
`the term “functional unit” may
`[0019] As used herein,
`include a processor and memory or a programmable hard-
`ware element. The term “functional unit” may include one or
`more processors and memories and/or one or more program-
`mable hardware elements. As used herein, the term “pro-
`cessor” is intended to include any of various types of
`processors, CPUs, microcontrollers, or other devices
`capable of executing software instructions. As used herein,
`the term “programmable hardware element” is intended to
`include various types of programmable hardware, reconfig-
`urable hardware, programmable logic, or field-program-
`mable devices (FPDs), such as one or more FPGAs (Field
`Programmable Gate Arrays), or one or more PLDs (Pro-
`grammable Logic Devices), such as one or more Simple
`PLDs (SPLDs)or one or more Complex PLDs (CPLDs), or
`other types of programmable hardware.
`
`In various embodiments, any of various methods or
`[0020]
`techniques may be used in deploying the prototype on the
`functional unit of the hardware device. In the preferred
`embodiments, the process of deploying the prototypeis fully
`or mostly automatic so that once the prototype is created,
`little or no work is required on the user’s part to deploy the
`prototype on the hardware device. For example, the proto-
`typing environment may include functionality for program-
`matically converting the prototype to a format usable to
`configure the device and for configuring the device accord-
`ing to this format, or the prototyping environment may
`interface with one or more other applications to perform the
`converting and configuring. The user may not be required to
`provide user input, or may be required to provide a minimal
`amount of user input. In one embodiment, a graphical user
`interface wizard, i.e., one or more graphical panels or dialog
`boxes may be displayed to guide the user through the
`process of deploying the prototype on the device.
`
`[0021] After the hardware device has been configured, the
`hardware device may execule to perform the functionality
`specified by the prototype. The hardware device performing
`the functionality specified by the prototype may comprise
`the functional unit on the hardware device executing at least
`a portion of the operations in the prototype, i.c., executing
`program instructions and data implementing the operations.
`
`[0022] As described above, in one embodiment, deploying
`a prototype on a hardware device may comprise program-
`matically generating a program basedon the prototype, and
`then deploying the generated program on the hardware
`device. In one embodiment, the program that is program-
`matically generated based on the prototype may be a graphi-
`cal program. The generated graphical program may include
`various interconnected nodes or icons which visually rep-
`resent the operations in the prototype and which are execut-
`able to perform the operations as configured by the user. In
`various embodiments, this programmatic generation may be
`performed in any of various ways. In one embodiment, the
`prototyping environment application may include function-
`ality for both creating the prototype and generating the
`graphical program. In another embodiment, the prototyping
`environment application may interface with another pro-
`gram, ¢.g., a graphical programming development environ-
`ment application,
`to request the graphical program to be
`generated, e.g., according to a client/server methodology.
`The graphical program may be programmatically generated
`with little or no user input received during the generation
`process.
`
`[0023] The user may optionally modify the graphical
`program thal was programmatically generated, e.g.,
`to
`develop a customized or optimized graphical program. For
`example,
`the user may desire to add functionality to the
`graphical program which was not included in the prototype
`from which the graphical program was generated. For
`example, by directly modifying the graphical source code of
`the program, the user may be able to customize the graphical
`program to perform operations not available from within the
`prototyping environment. The user may utilize a graphical
`programming development environment separate from the
`prototyping environment to interactively modify the gener-
`ated program.
`
`52
`
`52
`
`

`

`US 2002/0186245 Al
`
`Dec. 12, 2002
`
`[0024] The generated graphical program or the user-modi-
`fied version of the graphical program may then be deployed
`on the hardware device to configure the device to perform
`the function implemented by the graphical program, and
`thus to perform the function specified by the prototype on
`which the graphical program is based.
`
`In various embodiments, the process of converting
`[0025]
`the prototype to a graphical program and then configuring
`the device according to the graphical program may vary in
`terms of the flow experienced by the user. For example, in
`one embodiment, the user may create a prototype in the
`prototyping environment and may then request
`the proto-
`typing environment to deploy the prototype on a hardware
`device. For example, the user may invoke a dialog box or
`wizard to specify information regarding the deployment,
`such as a selected hardware device on which to deploy the
`prototype, selected operations of the prototype to deploy,
`etc.
`
`[0026] After the user has filled in the information in the
`dialog box or wizard, a graphical program based on the
`prototype may be automatically generated, and the hardware
`device may then be configured to execute the graphical
`program. The prototyping environment may interact with
`one or more other applications in this process, such as a
`graphical programming development environment. For
`example, the prototyping environment may interface with
`the graphical programming development environment
`to
`generate the graphical program and may then request the
`graphical programming development environment
`to con-
`figure the selected hardware device according to the graphi-
`cal program. However, this may appear to the user as a
`seamless process, and the user may never know that
`the
`graphical programming development environment was
`involved and may not even know that a graphical program
`was ever generated.
`
`In another embodiment, the entire process of mov-
`[0027]
`ing from the prototype to the configured hardware device
`may be less integrated, i.e., more knowledge of and control
`over the process may be provided to the user. For example,
`the prototyping environment may generate the graphical
`program, and the user may then open the graphical program
`in a graphical programming development environment and
`request the graphical programming development environ-
`ment
`to deploy the generated graphical program on a
`selected hardware device. Also, as described above, the user
`may first modify the generated graphical program before
`requesting the graphical program to be deployed on the
`device.
`
`instead of a graphical
`In another embodiment,
`[0028]
`program, a text-based program based on the prototype may
`be automatically generated. The text-based program may
`include program code in any of various text-based program-
`ming languages, such as C, C++, Visual C++, Visual Basic,
`Java, Pascal, etc. The generated text-based program may
`inchide various functions or methods which are executable
`to perform the operations configured by the user in the
`prototype. The text-based program may then be used in
`configuring the hardware device, similarly as described
`above.
`
`the hardware device may
`In one embodiment,
`[0029]
`include a processor and a memory, and deploying the
`graphical program on the hardware device may include
`
`transferring the graphical program to the memory on the
`hardware device, where the proces