(19) United States
`(12) Patent Application Publication (10) Pub. N0.: US 2003/0187950 A1
`Rising, III
`(43) Pub. Date:
`Oct. 2, 2003
`
`US 20030187950A1
`
`(54) METHOD AND SYSTEM FOR UTILIZING
`EMBEDDED MPEG-7 CONTENT
`DESCRIPTIONS
`
`(75) Inventor: Hawley K. Rising III, San Jose, CA
`(Us)
`
`corresponqence_Address3
`J‘ihn P‘ 0 Ramon
`5I6LP
`1511
`sacragllgna) CZ 553;: (Us)
`’
`(73) Assignee; SONY CORPORATION & SONY
`ELECTRONICS INC,
`
`(21) Appl- N05
`_
`_
`(22) Flled'
`
`10/112,610
`
`Mal" 29’ 2002
`Publication Classi?cation
`
`(51) Int. Cl.7 .......................... .. G06F 17/30; G06F 15/16
`
`(52) US. Cl. ............................. .. 709/218; 707/3; 709/231
`
`57
`(
`)
`
`ABSTRACT
`
`An apparatus for selecting digital material objects contain
`ing MPEG-7 content descriptions in response to MPEG-7
`content description queries. The apparatus comprises an
`MPEG-7 content description query generation tool coupled
`With a search engine con?gured for searching and compar
`ing embedded MPEG-7 META tag information Within ?le
`headers, or database information thereof, to the MPEG-7
`content description query. The query tool and search engine
`may be implemented separately or in combination. By Way
`of example, the MPEG-7 query generation tool is imple
`mented Within an application program such as a broWser, or
`Web site, to alloW the user to readily and accurately de?ne
`the content being sought. The content query being sent to a
`search engine con?gured for performing MPEG-7 content
`searches that return hits based on content related matches.
`
`12
`
`10
`
`______________ I
`
`r"
`
`—
`
`18
`
`M
`
`l “
`
`MA T
`
`"-
`
`_
`
`l
`1
`
`Display
`
`o
`
`-
`Query J L'brary
`
`30
`
`
`
`/; 14
`
`
`Content '
`lnformatlon
`
`
`A _ ____ Jnleriesewn? - l M ore <~
`
`20a
`Non tex't ,
`a
`20b
`
`unc'
`.
`Saved Queries
`
`jl
`
`l
`‘
`
`36
`
`I
`
`32
`
`Billing
`
`Submit
`
`Buffer
`
`~22
`
`'1»
`
`r
`Converter
`
`l
`ll
`Syntax
`
`Checker
`
`26\
`
`24
`/_
`
`Rules for
`Query Tool
`MPEG_7
`
`r___
`
`Hit
`Processing
`l
`:
`l
`ll
`i
`
`l
`
`/ 38
`Queue
`
`:
`
`Parser
`
`S arch
`'
`4- strl'gguti?e
`
`results
`-------- I.
`
`l 1
`1:
`i
`:1
`
`
`
`l L-
`
`mm
`
`, H" ,F “““““““““ "*mzé ---- ‘A: ---------- 1/ 44
`
`l
`/
`=
`
`I
`l
`
`l
`l
`
`Page 1 of 28
`
`FORD 1214
`
`

`
`Patent Application Publication
`
`Oct. 2, 2003 Sheet 1 of 15
`
`US 2003/0187950 A1
`
`vb__.M_EEssay:L
`
`
`.mm8.33Ino4<0mfio*
`
`L_ls!t'i8_E§m.o_oo-
`
`8.$938I“V
`
`mm5.39I_m
`
`mm:-.~.§Imm
`
`go:29>Em2:
`
`
`
`‘1z-,H1E%<TEE_o§a:
`
`
`-o5VNmN“Eo_ooho#EMgmfig20.02:
`.o_oOmxmJ.wm9.coo$n_
`
`
`
`_x“_m>._oE_._o_ooflH:_.
`
`em._o@>m:@m_omOWtmmamfimeINV
`
`
`
`88m_o_ooLomCmfimeI__‘_‘
`
`_..G_n_
`
`:2.o_::
`
`
`E:3&5£30:._vm_EEm9m_I1_mm8m.N_§I_m
`\om:.m.m.m3IK////om:.m.N.m$Ino
`...852aI9 %ms_@262Qmm5.33I5
`
`mm$m.m.o$I;.
`
`omfimdavImm
`
`Page 2 of 28
`
`FORD 1214
`
`Page 2 of 28
`
`FORD 1214
`
`

`
`Patent Application Publication
`
`Oct. 2, 2003 Sheet 2 0f 15
`
`US 2003/0187950 A1
`
`E. \\
`
`“r ......................................................... L ..................................... :
`
`u 8 . , (2
`
`_
`
`m < row 4
`
`m mEmwoooE l .E 3
`
`" womni a 8
`
`
`¢wl\ 7 f
`
`M 62. E35 xMEMwO 6tm>¢o0 m 6.52:1
`
`
`
`:1 mm) swam EM 2:5
`
`_ \ > 8N
`
`
`
`
`
`" mm wwwwwmu um>mw new i 0.622 __ @6202 -\ co=mE._oE_
`
`moaEmxw 89295 i \ >655 \ >630 @620
`
`
`
`.UEE ||||||| L ......... ii I #6260
`
`w_.
`
`mESom
`506mm mcEw
`
`
`
`Ewan i
`
`-----———-----——H————--—---’
`
`Nm
`
`Page 3 of 28
`
`FORD 1214
`
`

`
`Patent Application Publication
`
`Oct. 2, 2003 Sheet 3 0f 15
`
`US 2003/0187950 A1
`
`@EH
`
`om
`
`
`
`mum nm>> <
`
`m .GE
`
`E/\ A
`
`.0.
`
`6m 5262;?
`
`Em wBEw
`
`2262.
`
`Q05
`9. ow
`
`359.0,
`
`EM _ 9E1
`
`Page 4 of 28
`
`FORD 1214
`
`

`
`Patent Application Publication
`
`Oct. 2, 2003 Sheet 4 0f 15
`
`US 2003/0187950 A1
`
`@EE
`
`om
`
`8 /
`
`‘ woEmQoE
`
`é: :30
`
`_ 2%. T825 28: :wmé 26:1; =5 _ 2m
`
`
`
`
`
`U
`
`/
`
`Page 5 of 28
`
`FORD 1214
`
`

`
`Patent Application Publication
`
`Oct. 2, 2003 Sheet 5 0f 15
`
`US 2003/0187950 A1
`
`1 | l | I
`
`:41
`
`96 nm>> <
`
`om
`
`m .GE
`
`33$ Egg _ 5L
`
`EQEEwmQ
`
`WOmEE:
`
`EwEoO
`
`QUE U
`
`E wmEmQoE U
`
`_ 2% _ 2855 28: E21 $5 :3 _ 2E
`
`E650
`
`\i/ N@
`
`Page 6 of 28
`
`FORD 1214
`
`

`
`M
`
`0I%OG_n_
`
`JW5
`
`
`
`ML:E
`
`3will
`
`0223..0moo92w_8ozSuv_9S3.:mmwc:U.m.._.%_oU.5:
`
`
`
`cum0um.fiaumEmaoo».wmn__2_|_Sown;
`
`
`
`
`E_.=:_xNE«wooG_0Ev~_wm53>><Uj_IL§%s_
`HH32mHe»mS_o:.=m0:zlu|J
`
`5.».m:llII'sH_as
`e"H%§;3_
`
`a
`
`t
`
`|_
`
`.m_I\Eoo.wmmq>cm.>>>>>>\\xE;_mwm:ou<L
`._|.5
`
`
`3wmaE_as“.QmE95masxoon_O29%muEn_O9mmcemmm
`mu:_.._04/_.\|mmmfiuummtoemanc:_.u_0Wm co._mmw;
`mmo_>w._aO2m__>I>N2;oELuw.wmm:n_%:m
`
`
`
`o_u:muciO
`
`
`
`
`
`
`mmB:o>mu_cemmw®EOIgmocmmgmmvmEm.>>._on_xomm
`
`
`
`taP
`
`t
`
`._hpp
`
`A\\
`mE
`
`L
`
`.1rBm.EEC
`uEDDDDDP4u_Q|Im:o>ms26.1tumgmmufi_®I_28+?...H_.>..
`
`‘Ixv:
`
`Page 7 of 28
`
`FORD 1214
`
`Page 7 of 28
`
`FORD 1214
`
`
`

`
`Patent Application Publication
`
`Oct. 2, 2003 Sheet 7 0f 15
`
`US 2003/0187950 A1
`
`X
`Search
`72
`Customize
`x 0 New
`Choose a category for your search
`0 Find a person's address
`0 Find a business
`0 Find a Web page
`0 Previous searches
`0 tEind a map d
`O _ook up a wor
`@ Find image~—/ 78a
`0 Find audio\-/ 78b
`0 Find videk, 78C
`
`74
`
`76 {
`
`// 78
`
`/’ 8°
`
`82 m MPEG-7 Content search
`/- 84
`Keywords: r I
`J / 86
`'
`mage type
`88am to [:1-_88b
`Cost
`Maximum
`_
`Structure r
`usf‘ 9O
`Shape
`I
`tax?" 92
`Color
`I
`rtxf‘ 94
`Texture r
`I*\ \96
`‘\
`Ordering:
`98
`Image type
`/T\ 1 oo
`
`102a \
`
`" @ éeywords
`0st
`1 02b 5.)) Texture
`
`\
`\_
`v
`104
`
`Search \ __106
`
`FIG. 7
`
`Page 8 of 28
`
`FORD 1214
`
`

`
`Patent Application Publication
`
`Oct. 2, 2003 Sheet 8 0f 15
`
`US 2003/0187950 A1
`
`@EH
`
`60.? E26 D
`
`
`
`5.95 >>wz -\\ m:
`
`95 nw>> <
`
`w .0E
`
`Page 9 of 28
`
`FORD 1214
`
`

`
`Patent Application Publication
`
`Oct. 2, 2003 Sheet 9 0f 15
`
`US 2003/0187950 A1
`
`QUE
`
`om?
`
`U
`
`688 v6
`
`I
`
`mm?
`
`$T\\\ mik 22K
`
`
`
`new? \1i 2 52 $60 952003;
`
`a: ti “3:: 5
`
`
`
`82 M36 2m @2
`
`Page 10 of 28
`
`FORD 1214
`
`

`
`Patent Application Publication
`
`Oct. 2, 2003 Sheet 10 of 15
`
`US 2003/0187950 A1
`
`onVii!
`
`
`
`
`ommm.fl._._zam<m_zmooomkm2
`
`wmN\l9:80.6mE_._..mm_:mmmu>.._o.oow_.o:m.u_cmmuéo.:o=woo:_:uc:9_mv_omn_emm_.
`mmr_mEn_52n_z<n_z<.A2am_§o_>__Em9
`
`
`
`
`Q,_§.._o;.M%,_%o%mHTaIm_4amaw_;%.m.223.:Vpt.1
`
`
`Z_>>O._m_m_m_>Om_...e...CE:oo9:56:82E._m.rmom\..oE_m._.
`
`
`Hll:|._.wcm_._o..o_>.mmZumEwZE._®._.N2-mE_m.._.
`
`
`
`e.wco=.o<33.om_.
`
`
`
`ot
`
`
`_vA_fl_I._wn__:m_bozoco_a_.ommn_EmEooEL\
`
`
`~N~\LEcom..0_.EE2_n__.._o..2mEmo_.v.96.38550.:._t<e
`vNN\l...Ha..Q_.
`
`
`
`
`m_oohn__:m_>520\_.2<1om:amsdomc_>
`
`
`I.Efiesmn_Ob%£93:mm.mnm...%...;aton:
`
`
`--owo_>--¢mv\,»\\ ”wEm2E5»N2mC\\.<E._m.rL#\mt
`
`
`
`
`RF:6:_302:329¢T_$c__39>_Em_®__n_._/N:
`
`
`
`
`
`
`
`o_.._.o_.._
`
`
`~mm\zj1lI_w:o_§_Emom98new$3Amea3émso
`
`
`
`
`
`vmm\Ib111_.wm_Emxmno_.N1..movm\\@orm
`.vI:.:®._.5( genzmi/mfimmBo_mn_Sm.
`
`
`.o_:o__axm_E.a:mE:_22:0omcoammmH/r_.\
`ommm5.3wt
`
`F1
`
`Page 11 of 28
`
`FORD 1214
`
`Page 11 of 28
`
`FORD 1214
`
`
`
`

`
`eMP
`
`P
`
`A\memu
`
`m.r
`
`
`.m.200DwmmMmomm:=o:.=.wD..PNHuom.—n_:mD[296%New
`
`
`
`
`
`.m__:vA__U_I_.mmmE_c_mu__:m_bozoE%___2.__D
`
` .‘um,,_22..T§.z_>:w_oo»E.omc__%_>_:3_251/3
`
`
`23xo._.WS._o_ooWowmm:Bo::mI.
`
`um.|_|::1
`2.oR_(xwm_/mamaM30mom»-om_n__>_
`nmccmm»
`.m2Bxm._.Dowm
`
`
`
`W_mo:mo1_._En:m4/omm
`
`mam
`
`cam5:1---.....-..............:MI»!3:8.*HwwNN..0mn:>_
`
`Em
`
`0N0M.
`
`1Am.
`
`w_._.G_..._BJ
`
`Page 12 of 28
`
`FORD 1214
`
`Page 12 of 28
`
`FORD 1214
`
`
`

`
`Patent Application Publication
`
`Oct. 2, 2003 Sheet 12 0f 15 US 2003/0187950 A1
`
`304
`‘R302
`\m Search Site
`308/"\ ABC lnc.com \ 312
`
`300 /
`
`f 310
`
`306
`
`/
`
`(
`
`\\\>\ ABC lnc.com
`I’ 314/
`
`I
`
`316\‘
`
`318a
`\“ Return
`
`”
`
`318b/_ Order n of m
`/ Query String
`318C
`
`FIG. 12
`
`y/
`324
`\* Search Site
`328/'\ ABC |nc.com
`
`[330
`
`326 \/
`
`ABC lnc.c0m
`
`_
`
`\ 332
`
`FIG. 13
`
`Page 13 of 28
`
`FORD 1214
`
`

`
`Patent Application Publication
`
`Oct. 2, 2003 Sheet 13 0f 15 US 2003/0187950 A1
`
`o;
`
`N€!\\\
`
`3% m3
`
`MET‘
`
`wwwi
`
`25 ‘
`
`o3 )
`
`3/
`lllllllllllllllll i NLUmEE
`
`
`(my?
`
`3. .GE
`
`@
`
`EmEoO
`
`wQK h
`w?
`
`twmE
`
`25m
`
`www.15
`
`Page 14 of 28
`
`FORD 1214
`
`

`
`Patent Application Publication
`
`Oct. 2, 2003 Sheet 14 0f 15 US 2003/0187950 A1
`
`450
`
`Obtain
`Content
`information
`
`i
`
`*Convert
`information w
`to MPEG-7
`454
`description
`i
`
`Embed N 456
`opening delimiter
`
`i
`
`insert
`MPEG-7
`description
`i
`
`\_/\ 458
`
`Embed N 460
`ciosing delimiter
`
`i
`
`Store
`Augmented w 462
`content
`
`464
`
`FIG. 15
`
`Page 15 of 28
`
`FORD 1214
`
`

`
`Patent Application Publication
`
`Oct. 2, 2003 Sheet 15 0f 15 US 2003/0187950 A1
`
`
`
`
`
`v.5: cozqzowmv wlmvmnzz
`
`Page 16 of 28
`
`FORD 1214
`
`

`
`US 2003/0187950 A1
`
`Oct. 2, 2003
`
`METHOD AND SYSTEM FOR UTILIZING
`EMBEDDED MPEG-7 CONTENT DESCRIPTIONS
`
`CROSS-REFERENCE TO RELATED
`APPLICATIONS
`[0001] Not Applicable
`
`STATEMENT REGARDING FEDERALLY
`SPONSORED RESEARCH OR DEVELOPMENT
`[0002] Not Applicable
`
`REFERENCE TO A COMPUTER PROGRAM
`APPENDIX
`[0003] Not Applicable
`
`BACKGROUND OF THE INVENTION
`
`[0004] 1. Field of the Invention
`
`[0005] The invention pertains to content query mecha
`nisms for use With digital content-laden material, and more
`particularly to methods of constructing and executing que
`ries directed at digital materials containing, or associated
`With, MPEG-7 content descriptions.
`[0006] 2. Description of the Background Art
`[0007] Vast quantities of content are currently available
`Within private and public databases, With internet-based
`content aggregation and distribution being a highly visible
`segment of Internet commerce. Systems and applications
`capable of searching through content databases, hoWever,
`have been largely con?ned to ?nding textually based
`resources. The selection of multimedia content from a
`database generally remains dif?cult, inaccurate, and highly
`laborious. It Will be appreciated that searching for multime
`dia content Within a large repository has largely outpaced the
`ability of current query and searching tools. Presently, a
`party searching for content to meet a set of selection
`criterion is often required to load and manually check
`objects against the selection criterion.
`[0008] Recently, hoWever, standards have begun to
`emerge for characteriZing various multimedia sources. The
`MPEG-7 standard is one such standard that provides a
`substantially uniform set of metrics for identifying charac
`teristics of a digital source, in particular multimedia sources.
`A Working group of ISO/IEC referred to as the Moving
`Pictures Expert Group has created a content description
`system knoWn as MPEG-7. The neW MPEG-7 standard
`provides a content description standard, Which should not be
`confused With multimedia representation standards, such as
`MPEG-1, and MPEG-2. The MPEG-7 standard provides a
`universal and extensible set of content descriptions Which
`utiliZe description schemes and descriptors to describe
`aspects of digital material objects including: format, siZe,
`type, encoding, basic structures, colors, textures, shapes,
`motion, speech annotations, timbre, melody, content related
`terms, syntactic structures, semantic structures, syntactic
`semantic links, relationships, as Well as a large set of
`additional content description metrics. FIG. 1 is an example
`of graphical color characteriZation, generally referred to as
`color histogramming, for an image Whose colors are shoWn
`quantiZed into individual RGB components. The color infor
`mation from the histogram is an example of information
`Which may be included Within an MPEG-7 content descrip
`
`tion. Although MPEG-7 may be utiliZed for characteriZing
`multimedia content, there is an attendant lack of convenient
`methods and systems by Which a user may search content
`resources associated With MPEG-7 descriptions.
`
`[0009] Therefore, a need exists for a method and system
`Which facilitates the selection of content, in particular forms
`of multimedia content, from Within a database or netWork.
`The present system ful?lls that need, as Well as others, While
`being organiZed to ease implementation Within the present
`netWork infrastructure and to overcome de?ciencies in pre
`viously developed mechanisms for selecting content.
`
`BRIEF SUMMARY OF THE INVENTION
`
`[0010] The present invention comprises methods and sys
`tems for providing efficient and accurate content searching
`of content resources and content information databases,
`associated With digital material objects, such as multimedia
`?les Which may include such ?le types as audio, video,
`animations, music, images, sound effects, graphics, CAD
`?les, recordings, icons, and other forms of information
`Which extend beyond a text-only nature. It Will be appreci
`ated that in order to search for non-textual content, a content
`description system must be employed Wherein the desired
`search metrics may be quantiZed and compared With a target
`set of resources in Which matches are being sought. Aviable
`content description system should be capable of describing
`the various forms of multimedia content, While providing a
`standard that facilitates interoperability betWeen business
`parties, for example betWeen parties engaged in quid pro quo
`content transactions. The present invention provides various
`methods and systems for searching digital material objects
`according to MPEG-7 content descriptions. By Way of
`example, the MPEG-7 descriptions upon Which the searches
`are based preferably contain MPEG-7 META data Within the
`header of the digital material. For example a ?le may contain
`a markup language, such as SGML, or subsets such as those
`inclusive of HTML, XML, or WAP. An example of META
`tag embedding is described at a later point Within the
`speci?cation.
`[0011] HoWever, the embedding of content information,
`by itself, does not facilitate content based searches Within a
`set of material resources. The present invention teaches
`methods and systems that utiliZe embedded MPEG-7 con
`tent description information to increase the ef?ciency and
`accuracy of searches and queries based on MPEG-7 content
`descriptions. Furthermore, the present method is capable of
`providing information about hoW a search progressed so that
`the content information and search structures may be
`improved While information is accumulated about user pref
`erences.
`
`[0012] By Way of example, the present invention provides
`a system comprising one or more computers having netWork
`interface connectivity for communicating to and from users
`for executing content related searches. Programming execut
`able on the computers include an interface routine con?g
`ured for receiving MPEG-7 content descriptions, and a
`string search routine con?gured to search through an asso
`ciated database, or ?le collection, and return information
`about ?les Whose associated headers contain MPEG-7 con
`tent description elements that coincide With the MPEG-7
`content descriptions received by the interface routine. The
`programming may further include routines for collecting
`
`Page 17 of 28
`
`FORD 1214
`
`

`
`US 2003/0187950 A1
`
`Oct. 2, 2003
`
`information about the search queries Which may be passed
`to a destination Web site upon selecting a link to it amongst
`the search results. It Will be appreciated that Web servers are
`commonly employed to provide services such as searching,
`Wherein the present invention may be readily implemented
`on these systems, and others, by the inclusion of additional
`programming as described.
`
`[0013] To alloW a user to search a set of data resources
`containing MPEG-7 content descriptions, a mechanism is
`required for describing the content being sought, referred to
`as an MPEG-7 content description query; and an associated
`mechanism for applying the query against the set of
`resources, referred to as a search engine con?gured for
`MPEG-7 content description searching. The present inven
`tion describes query generation tools and search engines
`con?gured for searching content resources Which have an
`associated MPEG-7 content description, such as embedded
`META data containing MPEG-7 content descriptions. It Will
`be appreciated that the systems and methods described may
`be utiliZed separately or in combination.
`
`[0014] The query generation tool comprises an interface
`upon Which an MPEG-7 content description query is con
`structed, and preferably includes a set of query resources
`that facilitate query construction. By Way of example, the
`interface may be hosted on a Web site Which may provide
`query functions, query examples, saved queries, macros, a
`syntax checker, and a query converter Which formats the
`query for a speci?c search resource to Which the query is to
`be submitted. The MPEG-7 query generation tool preferably
`alloWs the user to create queries based on content charac
`teristics and content statistics, such as color histogramming,
`and further provides for the entry of non-text forms of
`characteriZation data, such as content query metrics via
`graphical manipulation and sketches. In addition, the query
`generator preferably provides a query library of query
`elements and query functions such that a user, even one With
`scant knoWledge of MPEG-7 constructs, may readily con
`struct accurate MPEG-7 based queries. Included in the
`functions are macro constructs Which are selected, or
`entered, by the user as a compact form of parameter entry
`that may then be converted to a standardiZed, or customiZed,
`representation as the query is passed to a speci?c search
`engine. A simple example of such a macro could be con
`sidered in the selection of a “time of day” parameter
`associated With a video clip, Wherein a user speci?ed time
`period of “sunset” for the ?lming of a piece of video footage
`can be expanded to include terms, phrases, and functions
`Which are synonymous With “sunset” to assure a broadened
`search breadth Within the collection of content resources. In
`addition, a library of query examples and elements is pro
`vided that alloWs a user to select prede?ned query elements
`for inclusion Within a query under construction. The user
`may elect to modify existing queries, or alter existing
`queries for neW search metrics, or cut and paste only the
`desired query elements. The present system also preferably
`provides for the creation of complex structured queries,
`Wherein term elements of a search are constructed and then
`may be assembled in various Ways to arrive at the resultant
`MPEG-7 content description query. The use of terms facili
`tates the rapid creation of query variations and list ordering
`preferences Without the necessity of reWriting the separate
`terms of the query.
`
`[0015] Response ordering may be user selected as either
`inherently derived or explicitly stated. Inherent result order
`ing can be performed based on the order and relationship of
`elements Within the MPEG-7 search query, as Well as the
`relative paucity, or abundance, of result matching in accord
`With the separate terms of the query. In explicitly stated
`response ordering, the user speci?es response ordering terms
`that de?ne hoW the responses, “hits”, are to be arranged.
`Explicit result ordering preferably still maintains an inherent
`ordering Within its stated subterms, and that of the query
`itself, Which operate to re?ne the explicit ordering. The
`ordering of the results and display methods thereof being
`performed Within the query generation engine, a result
`display routine, or the search engine receiving the query and
`associated list order information.
`
`[0016] The query tool preferably also provides intelligent
`query building Wherein the user speci?es a knoWn element,
`query, or query element, along With an associated relation
`ship in response to Which the system generates queries based
`on the knoWn information and expressed relationships that
`the user can prune to more readily arrive at a desired target
`query. For example, consider a simple case in Which the user
`selects an image, for instance of a ripe banana, and directs
`the system to generate a query for similar images. The
`system characteriZes the image and can collect additional
`feedback from the user (i.e. range of image siZes) to improve
`the accuracy of the search query being generated.
`
`[0017] The syntax of the queries being built Within the
`system are checked for compliance With the syntax de?ni
`tions for the query generation tool and in relation to the
`MPEG-7 de?nitions. Warnings and errors are generated
`according to preset preferences When the text of the query
`does not comply With the query generation tool or the
`MPEG-7 de?nitions. It Will be appreciated that the syntax
`entered into the system may not be directly compliant With
`MPEG-7 content descriptions, because the query generation
`tool is capable of supporting extended query generation
`features, such as macros and other extended functionality.
`
`[0018] The generated MPEG-7 queries are submitted for
`execution to a search engine that is con?gured to search
`content resources, that Wholly or in part, are associated With
`MPEG-7 content descriptions that are preferably embedded
`as META data Within the digital material object headers.
`Responses from the search engine may be displayed Within
`the query generation tool, and the query may be iteratively
`adjusted until a suitably limited set of accurately targeted
`results is returned. In addition, the system alloWs the user to
`select objects Within the returned object list, and character
`istics thereof, as inputs to further re?ne the generated
`MPEG-7 content search query.
`
`[0019] It should be appreciated that search engines are
`typically capable of performing a “direct search” Within a
`collection (database) of digital material objects, or an “indi
`rect search” Within a database of information assembled
`about a collection of digital material objects. By Way of
`example, search engines utiliZed for internet searching
`execute their searches Within a database containing an
`indexed set of keyWords and other information that Was
`previously assembled by analyZing each piece of digital
`material submitted to create database entries. It Will be
`recogniZed that the use of direct searching Within a collec
`tion spanning the siZe and having the distributed nature of
`
`Page 18 of 28
`
`FORD 1214
`
`

`
`US 2003/0187950 Al
`
`Oct. 2, 2003
`
`the Internet Would result in ponderously sloW searches While
`unduly increasing Internet traf?c. The ability of a search
`engine to perform both direct and indirect searches is Well
`known in the art. References, therefore, to one form of
`searching imply that the alternative form of searching can be
`similarly applied; barring any express statements to the
`contrary for speci?c situations. For example, references to a
`“collection of digital material objects containing embedded
`MPEG-7 content descriptions” Within Which a search is to
`be performed are also applicable to a “database of informa
`tion about a collection of digital material objects containing
`embedded MPEG-7 content descriptions” Within Which a
`search is to be performed, since the MPEG-7 content
`descriptions can be made available in either case.
`
`[0020] The MPEG-7 query generation tool may be hosted
`Within the aforementioned Web site, or alternatively hosted
`as an application executing Within other computational
`resources, such as in concert With a broWser, or in associa
`tion With a search routine. Users may gain the bene?ts of the
`aforementioned MPEG-7 query generation tool in a variety
`of Ways, including purchasing query generation softWare,
`paying per use charges that are based on subscriptions, time
`of use, queries generated, and so forth. The query generation
`tool may be utiliZed by third parties Which provide query
`generation services to users. In addition, the query genera
`tion and search capability may be provided free so that traf?c
`may be driven to a site, or sites.
`
`[0021] After creation, an MPEG-7 content description
`query is submitted for execution to a search engine con?g
`ured according to the present invention Which provides for
`MPEG-7 content description searching. The search engine
`receives and parses the incoming content description queries
`to create intermediate search information Which is applied
`Within the search engine for comparisons With prospective
`digital material objects, or information collected about said
`objects. Although search engines traditionally create data
`bases of indexed keyWords, and relationships, against Which
`incoming queries are applied; it should, hoWever, be appre
`ciated that collections (especially limited collections stored
`locally) may be searched directly. The search engine of the
`present invention is con?gured to recogniZe MPEG-7 con
`tent descriptions associated With the digital material objects
`being searched, and to apply MPEG-7 query terms against
`these descriptions in search of a match. The search engine
`may additionally support functional extensions of the
`MPEG-7 content descriptions to correspond With functions
`supported in an associated query generation tool. Query
`matches found by the search engine may be directly returned
`to the query generation tool, or user. Alternatively, the
`returned matches may be ordered Within the search engine
`according to information contained Within the query, or
`explicit ordering statements. For example, the listing order
`may be determined by explicit query statements that de?ne
`the desired ordering of the responses; While inherent order
`ing information can be derived from the relationships of
`query elements themselves, such as by their relative position
`Within the query.
`
`[0022] In addition it should be appreciated that MPEG-7
`content descriptions may be automatically generated, such
`as by the back end of the search engine, or a spider program,
`such as selectively based on a fee, When it analyZes a site for
`inclusion or update. The routine may characteriZe multime
`dia content on the site and generate MPEG-7 content
`
`descriptions accordingly. Descriptions such as these can
`then be utiliZed automatically or perhaps for a fee, as
`folloWs: (1) included in search database but not necessarily
`embedded Within the elements of content, (2) automatically
`appended Within the headers of the site, and (3) sent to the
`Webmaster of the site for their use in changing the site
`content.
`
`[0023] The present invention also teaches query capture
`methods Which are capable of supplying originating query
`information to vendors associated With searches Whereby the
`vendors can collect market information that can be utiliZed
`to improve the quality of the embedded content information
`and determine the direction and extent of ongoing marketing
`efforts.
`
`[0024] An object of the invention is to increase the speed
`and accuracy With Which desired content, in particular
`multimedia content, may be extracted from a content
`resource.
`
`[0025] Another object of the invention is to provide a
`content searching system and method that may be readily
`implemented Within the infrastructures of existing content
`resources.
`
`[0026] Another object of the invention is to provide con
`tent searching mechanisms for use With content descriptions
`de?ned by MPEG-7.
`
`[0027] Another object of the invention is to provide a
`content searching system and method that accepts non
`textual user content information and thereafter generates a
`corresponding MPEG-7 content query.
`
`[0028] Another object of the invention is to provide a
`content searching system and method in Which the user may
`input sketches to de?ne relevant MPEG-7 content descrip
`tion metrics for a query.
`
`[0029] Another object of the invention is to provide a
`content searching system and method in Which the user may
`input content samples and relationships betWeen the samples
`and content being sought as input for query generation.
`
`[0030] Another object of the invention is to provide a
`content searching system and method in Which the query and
`associated matches returned from one MPEG-7 content
`query may be utiliZed Within subsequent queries to increase
`search accuracy.
`
`[0031] Another object of the invention is to provide an
`MPEG-7 query generation tool that may be utiliZed Within
`a variety of applications, such as Web broWsers, Web sites,
`stand-alone applications, or integrated Within existing appli
`cations.
`
`[0032] Another object of the invention is to provide an
`MPEG-7 query generation tool in Which the ordering of the
`results may be explicitly stated by the user.
`
`[0033] Another object of the invention is to provide an
`MPEG-7 query generation tool that provides a library of
`functions from Which the user may construct a query.
`
`[0034] Another object of the invention is to provide an
`MPEG-7 query generation tool that alloWs the user to build
`queries from examples and saved queries.
`
`[0035] Another object of the invention is to provide an
`MPEG-7 query generation tool that alloWs a query to be
`
`Page 19 of 28
`
`FORD 1214
`
`

`
`US 2003/0187950 A1
`
`Oct. 2, 2003
`
`de?ned as a series of related terms so that query variations
`may be readily built and tested.
`
`[0036] Another object of the invention is to provide an
`MPEG-7 query generation tool that is capable of general
`iZing user input, to thereby reduce the amount of user input
`required.
`[0037] Further objects and advantages of the invention
`Will be brought out in the folloWing portions of the speci
`?cation, Wherein the detailed description is for the purpose
`of fully disclosing preferred embodiments of the invention
`Without placing limitations thereon.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`[0038] The invention Will be more fully understood by
`reference to the folloWing draWings Which are for illustrative
`purposes only:
`
`[0039] FIG. 1 is a conventional color histogram generated
`for a multimedia image, Which exempli?es non-textual color
`information as may be characteriZed Within an MPEG-7
`content description.
`
`[0040] FIG. 2 is a block diagram of an MPEG-7 content
`searching system according to an embodiment of the present
`invention, shoWn With a query generation tool coupled With
`a search engine.
`
`[0041] FIG. 3 is a screen for a broWser interface according
`to an aspect of the present invention, shoWn With a pull
`doWn menu for vieWing MPEG-7 content descriptions.
`
`[0042] FIG. 4 is a screen from a broWser interface shoWn
`With an option menu associated With a selected content
`element.
`
`[0043] FIG. 5 is a screen from the broWser of FIG. 4,
`con?gured according to an aspect of the present invention
`for providing MPEG-7 information about the selected con
`tent element.
`
`[0044] FIG. 6 is a screen for a broWser interface according
`to an aspect of the present invention, shoWn having a search
`screen in Which MPEG-7 content descriptions may be
`entered.
`
`[0045] FIG. 7 is a detailed vieW of the search screen of
`FIG. 6, Which exempli?es ?elds for selecting multimedia
`content based on MPEG-7 content.
`
`[0046] FIG. 8 is a screen for an MPEG-7 query generation
`tool interface according to an aspect of the present invention,
`shoWn With a query ?le pull doWn menu upon Which a neW
`query is being selected.
`
`[0047] FIG. 9 is a screen for an MPEG-7 query generation
`tool interface as depicted in FIG. 8, shoWn With high level
`search de?nition ?elds for a neW MPEG-7 content descrip
`tion query.
`
`[0048] FIG. 10 is a screen for an MPEG-7 query genera
`tion tool interface according to an aspect of the present
`invention, shoWn With an MPEG-7 content description
`query building screen.
`
`[0049] FIG. 11 is a screen for an MPEG-7 query genera
`tion tool according to an aspect of the present invention,
`
`shoWn for providing intelligent query generation based on
`samples and relationships that constrain the search upon
`Which the query is built.
`
`[0050] FIG. 12 is a data How diagram of a query capture
`mechanism according to an aspect of the present invention,
`shoWn transitioning from a link in a set of search returns to
`a Web page that is provided With information.
`
`[0051] FIG. 13 is a data How diagram of a query capture
`mechanism according to an aspect of the present invention,
`shoWn transitioning from a link in a set of search returns to
`a Web page Which can access query information from the
`parent Web page.
`
`[0052] FIG. 14 is a schematic of a system Which embeds
`MPEG-7 content descriptions Within a digital material
`object, shoWn for use With content and information reposi
`tories comprising either local databases or netWork derived
`resources.
`
`[0053] FIG. 15 is a ?oWchart of a process for embedding
`the content descriptions Within a digital material object.
`
`[0054] FIG. 16 is a text section Within an HTML header
`segment Which exempli?es META tag use.
`
`DETAILED DESCRIPTION OF THE
`INVENTION
`
`[0055] Referring more speci?cally to the draWings, for
`illustrative purposes the present invention is embodied in the
`systems and methods generally shoWn in FIG. 2 through
`FIG. 16. It Will be appreciated that the systems may vary as