PROCEEDINGS
`
`Fifth International Conference
`
`on Computer Vision
`
`Massachusetts Institute of Technology, Cambridge, Massachusetts
`
`Iune 20 — 23, 1995
`
`
`Sponsored by
`IEEE Computer Society Technical Committee on Pattern Analysis and
`Machine Intelligence
`
`

`
`
`
`IEEE Computer Society Press
`10662 Los Vaqueros Circle
`P.O. Box 3014
`Los Aiamitos. CA 90720~1 264
`
`C0PY1'i3ht © 1995 by The Institute of Electrical and Electronics Engineers, Inc,
`All rights reserved.
`
`Copyright and Reprint Permissions.‘ Abstracting is permitted with credit to the source. Libraries may
`photocopy beyond the limits of US copyright law, for private use of patrons, those articles in this volume um
`carry a code at the bottom of the first page, provided that the per-copy fee indicated in the code is paid
`through the Copyright Clearance Center. 222 Rosewood Drive, Danvers, MA 01923.
`
`other COP.‘/ins. reprint, or republication requests should be addressed to: IEEE Copyrights Manager, [E53
`Service Center, 445 Hoes Lane, P.O. Box 1331, Piscataway, NJ 08855-1331.
`
`The papers in this book comprise the proceedings ofthe meeting mentioned on the cover and titlepage. "They"
`reflect the authors’ opinions and, in the interests of timely dissemination, are published as pre.sented'and.-
`without change. Their inclusion in this publication does not necessariiy constitute endorsement
`tire.
`editors, the IEEE Computer Society Press, or the institute ofEiectricai and Eiectronics Engineers, Inc.
`
`IEEE Computer Society Press Order Number PR07042
`Library of Congress Number 95-7730]
`[BEE Catalog Number 95CB35744
`ISBN 0-8186-7042-8
`
`.
`
`Additional copies may be orderedfnom:
`
`IEEE Service Center
`445
`
`[BEE
`
`"
`
`"
`
`'
`
`
`
`IBEE Computer society Press
`Customer Service‘"Center
`10662LosV.aque_t'OsCi!'.0l¢
`P.O.Box 3014“ _
`_
`__ _
`LosA1amitos;__(.3A‘
`Tet:
`-+1-"714-—_‘8__
`Fax: +1-714932
`
`_
`'
`
`-Z
`
`
`
`

`
`Mosaic Based Representations of Vid
`_
`and Their Applications
`
`3
`e0 equences
`
`.
`
`'
`1
`
`Michal Irani, P. Anandan and Steve H
`.
`a
`c1[\I);ii(iSl"T‘°H Research center
`,_ Princeton NJ 08543-5300
`Email: micha1@sarnofl'.co;m
`
`su
`
`Abstract
`
`'
`
`'_y, there has been a growing interest in the
`. aic images to represent the information con.
`_
`“ii video sequences. This paper systematically
`how to go beyond thinking of the mosaic
`-a visualization device, but rather as a basis
`-representation of ‘video sequences. We do.
`‘:5
`"
`at types of mosaics called the static
`arnic mosaic that are suitable for different
`aarios. We discuss a series of extensions
`c mosaics to proioide representations at
`'al and temporal resolutions and to han-
`..-information. We describe techniques for
`ents of the mosaic construction process,
`"
`inte ration, and residual analysis.
`.1 .t.lpp_ications of mosaic representa-
`cornpression, enhancement, en-
`.,_ and other applications in video
`,. and manipulation.
`
`-
`
`
`
`treated.
`
`The purpose of this paper is to develop a taxonomy
`of mosaics by carefully considering the various issues
`that arise in developing mosaic representations. Once
`this taironomy is available, it can be readily seen how
`the various types of mosaics can be used for different
`applications. The paper includes examples of several
`applications of mosaics, including video compression,
`video visualization, video enhancement, and a other
`applications.
`
`2 The Mosaic Representation
`A mosaic image is constructed from all frames in a
`scene sequence, giving a panoramic view of the scene.
`Although the idea of a mosaic image is simple and
`clear, a closer look at the definition reveals a number
`of subtle variations. In this section we describe differ-
`ent “types” of mosaics that arise out of the types of
`considerations outlined above.
`
`2.1 Static Mosaic
`The static mosaic is the most common mosaic rep-
`resentation L10, 12, 11, 9, 7], although it is usually not
`referred to y this name. It has been previously re-
`ferred to as “mosaic” or as “salient still".
`It will be
`shown (in Section 4) how the static mosaic can also
`be extended to represent temporal subsamples of key
`events in the sequence to produce a static “event” mo-
`saic (or “synopsis” mosaic).
`_
`The input video sequence is usually segmented into
`contiguous scene subsequences (e.g., see [13ll\ and a
`static 'rno<sa.ic image is constructed for each scene, by
`_ -, all frames of that subsequence to a fixed co-
`9
`system. The aligned images are then inte-
`_-grated using-different types of temporal filters into a
`"
`image, and the slgruficant residuals (i.e., data
`us‘ "by the mosaic] are computed for each
`'
`__ to the mosaic Image.
`.'8 tie mosaic images are shown in
`the-constructed mosaic image
`, -with blurry crowd, and
`‘ athe constructed mosaic
`
`~
`
`H .-
`
`'
`
`'=sou.1-co of -iinformatiou. Its two
`are. the ability to ob-
`g siet, ofviews of a scene, and
`
`
`
`

`
`
`
`5
`
`
`Fit
`if
`in
`|fl
`
`No
`II?
`in
`
`ii
`L,
`E
`ii;
`[.3
`ti
`Ia
`
`Q
`"l
`&‘
`3‘
`“il
`1-
`- W
`'_;§
`-\
`
`
`
`Figure 2: Static mosaic image of a baseball ga.mese4
`quence. Top rows: Four out ofa 90 frame sequence
`obtained by a camera panning from right to'lefig-.-
`and zooming in on the runners. Bottom
`statiic mosaic image constructed using a tempzeriil?
`me ran.
`
`.
`the amount of “residual” information in the %dyn%'e'
`mosaic will be smaller than that in the static
`-
`- 8
`dynamic mosaic is therefore a more cfi5cient.scel:i¢'1¢P'-
`resentation than the static mosaic. Howev_el:-
`its incremental frame reconstruction, it
`£a.lE‘;J;:l:I)-!lt(fflTf£ifldO_f3 access touilnilividuall
`aorvieom
`.r‘..=
`dynamic mosaic, howev:i?,T3 aanliiiliaalntool for--lbw
`i-ate transmission (see Section 4).
`2.3 Temporal pyramid
`A natural extension of the staic and dyll
`saics is the temporal pyramid which" -it.I
`'.
`of static mosaics whose levels chrrespondfl "
`‘'‘‘"‘°“!1tS _of temporal integration.
`:..
`-1.
`organization is similar to s'pa.tia,]__ .
`3:sleritationi_ The finest level con‘-es."
`a Sam
`-
`_
`'
`.
`"
`coarser 1e\iial1:§.r:fb1hhs:d1hput seq‘? "i
`poral integration and doliri1";cc'
`way to represent such a.
`" '
`L3-Dlacian pyra,1-njd_
`single static mosaic"
`‘-
`sent residuals estimated
`
`.
`T
`
`l“ig1ire 1: Static mosaic image of a table-tfenni:
`ganic seqiieiice. Top row: Three out of a 300 F311;
`sequence obtained by a camefi-‘1 l3'anm_n§§_ across t 8
`scene. Bottom row: The static mosaic Image Con"
`structed using a temporal median.
`
`representation are the changes in the scene with re-
`spect to the background [e.g., moving playersl‘; S?‘-‘
`tion 3.3 presents a method for detectin‘g such resid-
`uals”.
`The mosaic image, along with the frame
`alignment transformations, and with the “reSldll3«1S
`together constitute a complete and cflicienil. represen-
`tation, from which the video sequence can be_frilly
`reconstructed. Applications of the static mosaic are
`described in Section 4.
`
`2.2 Dynamic Mosaic
`Since the static mosaic contains the common infor-
`mation of a set of frames, it cannot completely depict
`the dynamic aspects of the video sequence. This re-
`quires a dynamic mosaic, which is a sequence of evolv-
`ing mosaic images, where the content of each new mo-
`saic image is updated with the most current informa~
`tioii from the most recent frame.
`
`The sequence of dynamic mosaics can be visualized
`either with a stationary background (e.g., by com-
`pletely removing any camera induced motion), or in
`a manner such that each new mosaic image frame is
`aligned to the corresponding input video image frame.
`In the former case, the coordinate system of the mosaic
`is fixed, whereas in the latter case the mosaic is viewed
`within a moving coordinate system.
`In some cases a
`third alternative may be more appropriate, wherein
`a portion of the camera motion [e.g., high frequency
`_]l[-tel‘)
`is removed or a preferred camera trajectory is
`synthesized.
`Note that in the dynamic mosaic the
`moving objects do not blur out or disappear (as 0p_
`posed to the static mosaics in Figs. 1 and 2) but are
`constantly being updated.
`’
`The complete dynamic mosaic r
`'
`video sequence consists of the firepresentatlon of the
`51 d
`‘
`-
`and the incremental alignment paraIi(iIiiiidi‘]sicaliIi1<(i)si’;‘i1Te
`incremental “residuals” that represent the chan
`«
`.
`.
`ges.
`
`

`
`.
`»
`into
`ll’l0."id.lII‘ llllagtf, and H“.
`3.
`-'
`i
`cant
`f
`‘residuals licl.wcci1 the Int
`rames.
`
`Q-9-‘iiiputritron of s;gm'f,‘_
`Jsaic and the individual
`
`Image Alignment
`3‘1
`the chosen world
`molélgfgidallgnmeflt depends on_
`“ed to 2D motion model.
`'llie alignment can be lim-
`more C
`lparainetric motion models, or can utilize
`t t_
`Omp EX 3D motion models and layered represen-
`a 1053- The examples in this paper utilize 2D motion
`nfiodelsi In particular a 6~parameter afline transforma-
`l310T1 311531 an 8-parameter quadratic transformation, to
`‘dpproximate the motions between two images. Work
`011 3D Image alignment is currently in progress and
`described in [?, 6, 8],
`The alignment of all image frames in the video se-
`quence to form the mosaic image can be performed in
`one of the following ways:
`(i) Successive images are
`first aligned, then the computed motion parameters
`are cascaded to determine the alignment parameters
`between any_ frame to a chosen reference frame.
`(ii)
`Each image is aligned directly to the current compos-
`ite mosaic image usin the constructed mosaic image
`as the reference -
`-
`(i.e.,
`ed coordinate system), or (111)
`The current mosaic image is aligned to the new image,
`using the new image as the reference (i.e., dynamic co-
`ordinate system)
`To align two images we use the hierarchical di-
`rect registration technique described in [1, 5 This
`technique first constructs a Laplacian pyrarni
`from
`each of the two input images, and then estimates the
`motion parameters in a coarse-fine manner. Within
`each level
`the Sum of squared difference
`SSD)
`measure is used as a. match measure: E(-[u ) :
`
`
`
`
`' Evolution of the dynamic mosaic images
`table-tennis game sequence. Left column.‘
`as from the original sequence. Right col»
`.
`.
`.
`.
`.
`tetfrresgfindlng iynalmlc m°9aa“i11ma5e5'
`_ e 1°51 1°“ O t’ e P 3'33’ an i‘ 3 crowd
`tly eing updated to match the current
`
`
`
`
`
`aim,
`
`
`
`_
`_
`residual maps of static vs. dynalnlfi
`in'le frame from the table-tennis
`he residual map computed for
`‘frame "in the static representar
`. was signify more significant
`-
`'
`com I ted for
`E
`I-§;3aIniclli‘i:-ipresen—
`
`'
`
`EL: (I(x,t) — I(:r — _u(x),t — 1)]? where x_ : Er, y)
`enotes_ the spatial image position of a point,
`the
`Laplacian pyramid) me e intensity and u(x) :
`u(;g,y),:.- :1:,y)) denotes t e image velocity at that
`point, an the sum is computed over all the points
`within the region and {I1} is used to denote the entire
`t’ n field within that region.
`mo'Ii1(iis measure is ininimized with respect
`to the
`quadratic image motion parameters:
`u(x) = 1913 +1223: +ps + per” +ps:cg
`—.:
`:c+ y+ps+p7=y+Psy
`_.
`imized via the
`'
`{fife} obfesctivepéunction E({u}) is
`gew _fra.me at.
`Gauss-Newton optimization technique. After iterat-
`i0.8QI'.f-"-
`el
`'
`ing a certain number of times within a pyramid level,
`the process continues at the next finer level. More
`details can be found in [1, .5].
`3.2 Image Integration
`"
`-' -are. aligned),
`
`'
`'
`"
`-the‘ mosaic im-
`One
`in ,t__.-lie:-‘d _-
`. "r "integrating the
`
`

`
`
`
`IP''='=* —
`e :
`{M')ENl:
`I= i: i?:m‘r,i“;i::.a:‘a. is, ., e,:r,3i
`ltlrilgtigalfiilfiintensity gradient at pixel (:c,y) in Em“:
`mall neighborhood of pixel (3,3)
`t
`_
`_I:, N(31. .1!) is a fghborhood), C is used to_avoid
`ical instabilities and to suppress noise. Fig. 4 showsan
`f significant frame residuals detected for the
`3 dynamic representations in the table.
`'
`e uence.
`_
`_
`_
`enxfisofllgh the same significance measure is um;
`ith the static and the d,ym.l'm]c mosaic’ the lfimtions
`ldhd magnitudes of the significant residuals differ l;___.,.._
`
`tween the two schemes even whei_1 3-PPl1e_d to the
`'_,.
`sequence. In the case_ of the static mosaic, the g'__
`:9.
`canoe measures in regions ofobjects that move wit
`spect to the background are usually: larger than in the
`case of the dynamic mosaic, as moving objects tend.-$9
`blur out or even disappear_iri_the static mosaic, and
`hence the changes will be significant. In the dyri_ml;'~
`case, the mosaic is constantly being updated with
`most recent information, and therefore, the charig@_§_in_._
`image regions that correspond to independently inn -
`
`ing objects will be smaller between the predicts
`actual frame. In the dynamic mosaic, howev_cr.,.
`residuals will be obtained at image boundafies‘
`the static case.
`
`Z |I¢(1=a'.‘y:’) - I:P”d(3i‘.y.')|
`)eN(=.irl
`
`lllll-l
`
`_
`
`t
`
`4 Mosaic Applications
`The most obvious applications of
`tation are video compression (since mos "
`cient scene representations) and as a means
`afization (since mosaics provide ‘a. wide
`field of view). These will be discussed .
`and _4.2. However, mosaics are else,
`applications, such as scene change‘
`video search and video indexing,
`and man1pu1ation,a.nd others. '
`
`w -
`
`"
`
`"
`
`.
`
`.
`poral average wlivlft?
`one of sevc-ral choices.
`of mosaics. Fag-.
`l-11¢‘
`"'_
`.
`'
`-
`«,9 gen-
`rrease with the. ilist.3',lC'e Ol ll P”"~’l f"°m.'l'S mm -
`ter [to accoiirit for aligniiient
`t
`hoiiiidaries); the weights can be the 01!
`mcess 0
`maps computed in the motion estimatltlllil P4) yield_
`tlie dorniiiaiit “bat-kE§1"0U“d_
`(See 315% 11%‘ Ck’ round
`ing a iriore complete inosaic image of} e
`a ufdn ob_
`scene (i.e., less “ghost-like’ traces of
`f01'eS1'fi
`fiverse
`jects): the weights can also correspond to t 6 ‘.6 im_
`of these outlier rejection maps, yielding a ITIOIS31 f the
`age which contains a panoramic image not 011 Yko 1 Ce
`scene, but also of the foreground event that too [3 8.
`in that scene sequence (see Fig. 5).
`_
`1.
`(iv) Integration in which the most recent rnforrnci :01},
`"re, that which is found in the most recent frame, 15
`used for updating the mosaic (see Fig. 3l-
`,
`(v) Alternative integration schemes for
`image 9"‘
`hancement, such as Super-resolution [3].
`_
`mosaic image whose resolution and image quality sur-
`passes those of airy of the original image frames. See
`more details in Section 4.
`
`3.3 Significant Residual Estimation
`
`The complete sequence representation includes the
`mosaic image, the transformation parameters that re-
`late the mosaic to each individual frame, and the resid-
`ual differences between the mosaic image and the in-
`dividual frames. To reconstruct any given frame in its
`own coordinate system, the mosaic image is warped
`using the corresponding mosaic-to—irnage transforma-
`tion and composed with the residuals for that frame.
`In the case of the static mosaic, the differences are
`directly estimated between a single reference (static)
`mosaic and each frames, and the reconstruction is
`straightforward.
`In the case of the dynamic mosaic,
`however, the residuals are incremental, being with re-
`spect to the previous mosaic image frame, and the
`reconstruction proceeds sequentially from frame to
`frame.
`
`frame and the
`Residuals between the current
`mosaic—based predicted frame occur for several rea-
`sons: object or illumination change, residual misalign-
`merits, interpolation errors durin warping and noise
`Of these the object changes are t e most semanticallir
`significant, and in some cases the illumination changes
`are as Well.
`
`The efiiciency of the re
`t,
`if
`-
`raised by assigning a signifigifiizinmaedgilirdlatg lllie mafia‘
`uals, and using those to weight the residuals A:re3ld-
`tive way of determining semantically signifiéant r°e1:§§-
`1 _
`uals is to consider not only the residual intensit b . t
`.
`y .u .
`also the the magnitude of local residual
`the local misalignments) between the fraggt-$1;(i.,e_.&
`the magnitudle: iii theeraeldiilliailalrillinofii
`To app: ' ;
`Si(::,y) of the normal flow 1:(1agn,it_-,u3_:_Bfat__
`(any) at timet is con-iputgd;
`
`from the mos '
`
`d (;h
`
`._ _
`
`'
`
`4.1 Mosaic Based Video"
`Since mosaics provide an
`sentinga video sequence‘
`to consider is video
`e.“°e3 bet’-Ween -static and
`l"°n§ that were -outliflied.
`to differences m.{-,]15-- __
`-.
`time transmission 9
`the other fan,
`'
`'
`, be static. a.
`C338‘ ma .
`_
`.
`
`
`"
`
`

`
`and prioritize the residuals in other re
`ins them-
`
`g-Ions bef°“3 C0d~
`
`vs-:::...r:tme
`Transmission
`‘
`-
`1-eqllll‘
`Prgcessing
`ence the d
`is the natural choice for this app‘1icatiohim$i,cer$s?"C
`components in the transmission codec are lncpelfiigr
`taidgmimic mosaic construction, incremental residual
`estimation by comparison to the reconstructed mosaic
`from} the previous time instance, the computation f
`fljgnificance measures of the residuals and spatial cod
`grid decoding. As is typical of any predictive cod:
`.'--".--Zgystem, the coder maintains a d
`d
`-
`—
`.
`"order to be in synchrony with Eli: l‘((33i’.Ze‘lv\::i11nTllir
`= codlllg of the images and the residuals can ‘D:
`
`a on an available technique, e.g., Discrete Cosine
`'-..'_:‘f‘a'.ii;sform DOT) or wavelets.
`
`
`
`
`
`..
`
`ression for Storage: For storage a,pplica__
`is important to provide random access to in-
`However, the coding process need
`no in real time and can be done in an of.
`e-.- Therefore, the static mosaic is a natural
`application.
`'to:rage_ codec, the sequence is processed in
`I e- with these major steps being: static mo-
`on, residual estimation for each frame,
`'
`-, and spatial coding and decod-
`e and the individual residuals.
`fended individual residuals are
`oded mosaic and after perform-
`‘
`-transformation and im-
`iiiv-iidual frames can be dis-
`
`_‘_ras;se_nt-ative frames of a
`viewed from a fly-
`'u'e*‘n'<.:e.I was sampled at
`"u
`were spa.-
`The sequence was
`
`at of‘ Kbits/sec
`df 7
`with
`
`
`
`'
`
`
`
`"
`
`.
`-
`-
`.
`'
`useful w
`ays of visualizing l.lie same video seqiience us-
`_
`‘US; mosaic re resentat'
`9
`'
`-
`-
`-
`Own use:
`P
`Ions. Lach visualization has its
`
`K93’ frame mosair" Civen a video se-
`.
`_
`»-
`I
`_
`.
`.quence seg-
`;Tg:£lil;edrnBI;l;i)cCpntlgu0US clips of scene seqncnces,_a
`the Scene ca image of the most salient features in
`The Static I1
`e_coristructed for each scene sequence.
`resent th _ mosaics images (e,g,, Figs, 1 and 2) rep-
`C
`th
`tel: scenes better than any single frame, and
`_aI1
`ere ore be used as key frames” for rapid brows-
`lng through the entire digitally-stored video sequence
`l12]- ‘Other applications of the key frame mosaic are
`describe 11'} detail in Section 4.4.
`
`Synopsis mosaic: While the key frame mosaic is
`useful for capturing the background,
`in some cases,
`It may be desirable to get a synopsis of the event
`that takes place within the video sequence. This can
`be achieved through a mosaic that captures the fore-
`ground event. The synopsis (or event) mosaic is con-
`structed by_usin
`the residual maps (e.g., Fig. 4) as
`weights during t e integration process, allowing for
`foreground moving objects to be retained within the
`mosaic. Fig. 6.b shows an example of a synopsis mo-
`saic for the baseball sequences. Note that regular av-
`eraging of the aligned frames will not maintain the
`foreground moving objects, but will rather make them
`either completely disappear or significantly fade out.
`
`Mosaic video: The panoramic visualization pro-
`vided by the mosaics is useful not only as a static
`image, but for dynamic video visualization as well. In
`this case, a new video sequence is generated (called
`the "mosaic video”) which is a. sequence of mosaic im-
`ages. This type of visualization simulates the output
`of a virtual camera with desired features. The sim-
`plest example of this is stabilized video mosaic dis-
`play, in which case the camera motion is completely
`removed. The previously shown Fig. 3 shows an ex-
`ample of video mosaics. Such a display has uses in
`various applications such as remote navigation, and re-
`mote surveillance. Similar mosaic Visualizations have
`also been suggested by [10].
`4.3 Mosaic Based Video Enhancement
`Mosaic representations can serve as a useful and ef-
`fieient tool for producing high quality stills from video
`as enhancing an entire video sequence.
`‘resolution of an image is determined by the
`characteristics of the camera: the optics, the
`oi-“the detector elements, and their spatial re-
`d "
`
`ilicrease in the sampling rate could, how-
`achieved by obtaining more samples of the im-
`'
`'
`from aseqnence of images in which
`pears moving at subpixel displace-
`I
`-'
`'
`-g the sequence frames over
`_
`'6, can provide higher sampling rate
`'
`scene", and hence integrating over
`' her spatial resolution. When
`'é_ -camera is also known or can
`rfor deblurring, the increase in
`
`

`
`
`
`standard
`ic-based—compression vs.
`-
`.
`.
`,_ H
`v
`_
`fd namic 111033
`.
`_
`hgure :3:
`1l'aI1SIT1lS5lOI1 COI1'lpI'€SS]0l1.
`results 0
`Y
`Left column. Some representatwe
`'
`ence.
`.
`?1PEG Cfonllglfsfiion Sn a storage—i1o$i;f:d?:rC\;(EflI£:.I?:oergfigureconstructed frames after “Slug dynamic
`'
`'
`sequence.
`secon
`rames o a ‘-
`Iumn: For comparison: Th
`'
`-
`’
`.
`_
`.
`{'32 Ixblts/sec. R39“ '5'“
`3
`mosaic-based—comp1-essiori at. a constant bit rate 0
`t the same bit rat.
`-
`'
`PEG om ression of the Sequence 3 .
`.
`.
`*9
`;-q~co1:13séLrE%t;:d/frame§;f§e:h:s:(%%f:::£1cc::r?nT‘Ehe recfignstii-ucted quality of the running soldiers 1n the
`
`1.e.,
`
`1 5 se ..
`
`-
`
`,
`
`images of the bottom row.
`
`mosaic of th b
`
`b ll
`
`'
`
`'
`
`nopsas
`
`Figure 6: Synopsis mosaic image of the baseball
`game sequen .
`Th
`'
`of the scene Ctfritfijtit th: Zifetxii; b£E%§grrF1?:‘:ym°33r1_Q
`that occurrc-3 inafiieZceiieeqgnintioepiniiiiilienléaitllie evenci
`Hers)‘
`( e
`s owlngthe trajectories ofthe t.wo1-un_
`
`mosaic i.
`
`., h
`
`‘
`
`-
`
`-
`
`groun
`
`d c0n.e.8po.,.
`
`1 90 seqne
`
`'
`
`)
`
`310
`
`

`
`
`
`.
`
`.
`
`is even more. pronounced. This method is
`as super-resolution [3, 4,_9].
`.. cgggzcy of usrn mosaics for-_ video enhance-
`fie to the fact 1'. at the mosaic is an efiicgem
`man of the video sequence. Rather than en-
`., {tunes one-by-one (as IS suggested in [4])
`exit of the entire sequence (or layer) ié
`step within the mosaic coordinate Sy3_
`then are the enhanced frames retrieved
`ced rnosa1c.d fr
`ame from a sequence of
`‘eves an enhance
`a'...'sr1eserted.-truck imaged from a remote
`, _ ance video. In this example, all the
`‘very poor quality and very noisy.
`_; ained a_. single static scene that could
`_'
`using 2D ahgnment. The entire
`' enhanced by constructing a. sin 13
`(3 mosaic, and then retrieving t e
`sic back into their original coordi-
`to the inverse 2D parametric
`
`,
`
`1,
`
`5: Based Applications
`'ma..ge.s for various other ap-
`ed E10, 12]. Some of these
`digital ibraries [12], and with
`‘ti
`video in video
`'
`"ts.
`_-mg and manipulation ap-
`e--key-frarne or the syn-
`to significantly reduce
`
`representation may be deve10ped_ We have also de-
`scr'b
`-
`.
`.
`1 ed ii number of different applications of the mo-
`saic
`'
`-
`.
`ampfeefiresentatuons and illustrated them with real ex-
`
`References
`[1]
`‘]‘R- Bergen. P. Anandan, K..l. Hanna, and R. Hin-
`801'aI1l. Hierarchical model-based motion estimation.
`In Proc. European Conference on Computer Vision.
`Pages 237-252, Santa Margarita. Ligure, May 1992.
`
`[2]
`
`M- Ifafli. S. Hsu, and P. Anandan. Mosaic based video
`compression.
`In Proceedings of SPIE Conference on
`Electronic Imaging, February 1995.
`
`M. Irani and S. Peleg. Improving resolution by image
`registration. CVGIP: Graphical Models and Image
`Processing, 53:231—239, May 1991.
`
`M. Irani and S. Peleg. Using motion analysis for image
`enhancement. Journal of Visual Communication and
`Image Representation, 4(4):32-iv-335, December 1993.
`
`[51
`
`M. Irani, B. Ronsso. and S. Peleg. Computing occlud-
`ing and transparent motions. International Journal of
`Computer Vision, l2(l):5—16. January 1994.
`
`[5]
`
`[7]
`
`[81
`
`[9]
`
`R. Kurnar, P. Anandan, and K. Hanna. Direct recov-
`ery of shape from multiple views: a parallax based
`approach. In Proc 12th ICPR, 1994.
`
`Rakesh Kumar, P. Anandan, and K. Hanna. Shape
`recovery from multiple views: a parallax based ap-
`proach.
`In DARPA IU Workshop, Monterey, CA,
`November 1994.
`
`Rakesh Kumar, P. Anandan, M. Irani, J. R. Bergen,
`and K. J. Hanna. Representation of scenes from col-
`lections of images. In submitted to Workshop on Rep-
`resentations of Visual Scenes ‘.95.
`
`3. Mann and ILW. Picard. Virtual bellows: Con-
`an-acting high quality stills from video. In Proc. IEEE
`Int. Gonf. on Image Proc., November 1994.
`
`[11]'
`
`[iifl] P.C.. McLean. Structured video coding. Master’s the-
`-sis, MIT, June 1991.
`Szelislri.
`Image rnosaicing for tele~rea.hty
`'a‘.fiilieations'. Technical Report CRL 94/2, Digital
`Equij5_rnent- Corporation, 1994.
`I,
`'
`I -and W. Bender. Salient video stills: Con-
`
`:°
`
`
`
`'
`
`(e.g. ,_ signs
`at whole In the