US008924506B2
`
`(121 United States Patent
`Levanon et al.
`
`(10) Patent No.:
`
`(45) Date of Patent:
`
`US 8,924,506 B2
`*Dec. 30, 2014
`
`(54) OPTIMIZED IMAGE DELIVERY OVER
`LIMITED BANDWIDTH COMMUNICATION
`CHANNELS
`
`(75)
`
`Inventors:
`
`Isaac Levanon. Raanana. IL (US):
`Yonatan Lavi. Roanana. IL (US)
`
`(73) Assignee: Bradium Technologies LLC. Suffem.
`NY (US)
`
`( * ) Notice:
`
`Subject to any disclaimer. the temt of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 865 days.
`
`This patent is subject to a tenninal dis-
`claitner.
`
`(21) Appl.No.: 13/027,929
`
`(22)
`
`Filed:
`
`Feb. 15, 20H
`
`(65)
`
`Prior Publication Data
`US 2011/0175914Al
`Jul.2l.2011
`
`Related US. Application Data
`
`(63) Continuation-in-part of application No. 12/619.643.
`iiled on Nov. 16. 2009. now Pat. No. 7.908.343. which
`is a continuation of application No. 10/035987. filed
`on Dec. 24. 2001. now Pat. No. 7.644.131.
`
`(60) Provisional application No. 60/258.488. filed on Dec.
`27. 2000. provisional application No. 60/258,489.
`filed on Dec. 27. 2000. provisional application No.
`60/258,465.
`filed o11 Dec. 27. 2000. provisional
`application No. 60/258,468. filed on Dec. 27. 2000.
`provisional application No. 60/258.466. filed on Dec.
`27. 2000. provisional application No. 60/258,467.
`filed on Dec. 27. 2000.
`
`(51)
`
`Int. Cl.
`G06F 15/16
`C0617 3/14
`G06T 3/40
`
`(2006.01)
`(2006.01)
`(2006.01)
`
`(52) U.S. Cl.
`CPC .......... .. G061’ 3/I454 (2013.01): G06T3/4092
`(2013.01): G090 2340/02 (2013.01): G090
`2350/00 (2013.01)
`USPC ......... .. 709/217: 709/203; 345/625: 382/232:
`382/305
`
`(58) Field ofClassiflcation Search
`CPC
`G06T 3/4092: G09G 2340/02: G06F 3/14
`USP(‘
`............... .. 709/202. 203. 2l7. 230. 246. 247:
`345/625: 382/232. 305
`See application file for complete search history.
`References Cited
`
`(56)
`
`U.S. PATENT DOCUMIENTS
`
`5.929.860 A
`6.l82.ll4 B1
`
`7.51999 Hoppe
`[(2001 Yapetal.
`
`(Continued)
`OTHER PUBLICATIONS
`
`l-loppe. H. “Progressive Meshes”. SIGGRAPH ’96: Proceedings of
`the 23rd annual conference on Computer graphics and interactive
`techniques. 1996. pp. 99-108.
`
`Primary Examiner / David Lazaro
`(74) Attorney, Agent. or Firm — Anatoly S. Weiser. Esq.;
`Acuity Law Group. PC.
`(57)
`ABSTRACT
`
`Large-scale images are retrieved over network communica-
`tions channels for display on a client device by selecting an
`update image parcel relative to an operator controlled image
`viewpoint to display via the client device. A request is pre-
`pared for tl1e update image parcel and associated with a
`request queue for subsequent issuance over a communica-
`tions channel. 'l'he update image parcel is received from the
`communications chaimel and displayed as a discrete portion
`of the predetermined image. The update image parcel opti-
`mally has a fixed pixel array size. is received in a single and or
`plurality of network data packets. and were the fixed pixel
`array may be constrained to a resolution less than or equal to
`the resolution of the client device display.
`
`21 Claims. 5 Drawing Sheets
`
`
`
`MN?-§.i(
`
`Microsoft Corp- Exhibit 1002
`
`

`
`
`
`
`US 8,924,506 B2
`Page 2
`
`
`
`(56)
`
`
`
`References Cited
`
`
`US, PATENT DOCUMENTS
`
`
`
`
`
`6,212,301 B1
`
`
`6,314,452 B1
`
`
`6,326,965 B1
`
`
`
`4/2001 Warner et al.
`
`
`
`
`11/2001 Dekel et al.
`
`
`
`
`12/2001 Castelli et al.
`
`
`
`
`
`6,345,279 B1
`
`
`6,346,938 B1
`
`
`6,397,259 B1
`
`
`6,608,933 B1
`
`
`6,704,024 B2
`
`
`6,711,297 B1
`
`
`6,801,665 B1
`
`
`
`2/2002 Li et al.
`
`
`
`2/2002 Chan et al.
`
`
`
`5/2002 Lincke et al.
`
`
`
`
`8/2003 Dowell et al.
`
`
`
`
`3/2004 Robotham et al.
`
`
`3/2004 Chang et al.
`
`
`
`10/2004 Atsumi et 211.
`
`
`
`
`
`
`
`Microsoft Corp. Exhibit 1002
`
`Microsoft Corp. Exhibit 1002
`
`

`
`
`U.S. Patent
`
`
`
`Dec. 30, 2014
`
`
`
`
`
`Sheet 1 of5
`
`
`
`
`
`
`US 8,924,506 B2
`
`
`
`\‘
`-....mzo-\\.E
`
`,§,..\..........w,}
`~‘
`{\\\.“
`
`I-.’A<'§
`
`.
`\~m.\
`:4
`‘
`:
`s‘
`\
`5' :.‘333.‘.I.\.~\\‘§
`;
`\l
`ii
`§\\\\\\\ss«\«\\“
`§€\\
`\\ \\\\VV
`$:\§'{§"{§§§§§‘C§§\
`R ~‘\ \\« \\‘
`§
`\\ Av ms-vw I
`\\
`R
`.
`R xxevns-\~\
`§
`\K\\~.\s-.\-.\-e.-.9
`\
`-6
`\...\\?\.....
`-“
`Tx.
`~‘};‘\
`
`§
`
`mwn \1\
`\
`\
`§
`§
`\
`§
`§
`3
`*
`§
`\
`g
`§
`
`. \‘\
`
`\
`\
`3
`§
`\
`§
`§
`§
`~
`3
`g
`;
`§
`
`4 .-V
`§ c‘
`.»~~
`
`~§
`
`_,..»~§» .,..-~
`
`.5
`S
`§
`$\
`N
`is
`§
`gg
`§
`
`.
`
`V’
`
`s
`.
`
`\
`
`. N _Y
`
`§:§m§§.
`
`§
`
`\\-.\\w.w.m-:-#*‘
`
`(\7_q_
`\\
`~§:“§ ‘EM 4%
`
`.
`
`.‘
`..»“‘
`M-»‘
`
`
`
`
`.
`
`‘N
`N...
`
`_..-“‘
`
`-
`
`§
`*
`
`-Ht“
`9*‘
`
`.
`
`-
`
`‘if
`*9;
`
`\
`
`M......,_‘
`\\v.\\
`€*x
`3
`\\‘'V\
`§§
`u
`:
`“\~x»«““.§~Q§
`i
`z:
`‘
`‘_ :
`\
`'
`;s§§§:%
`‘R’ $3‘
`
`.mmm- ‘-“‘::.‘\\\
`.m..»- :_M,&..~.
`- \x w\\.
`\\ _“_,‘.NgQ‘\“‘"“
`“\'\
`’.'»«-““"
`.\M“_‘_\.
`-_
`
`\»?i.=.~*:i.§
`-SK
`z T“:
`:\
`?3§%3x\s‘.~‘. i.-‘§§§
`
`
`\
`- _.§
`
`
`
`
`§\\\\\\\\\\\\\\\\\\\\\\\\~\
`~\
`.
`.-
`-.
`
`..-*‘\ """~‘x
`‘
`~>‘«-‘
`
`.,...w.\\w.\w.\-~‘
`‘_.N...\..»..
`
`V
`§
`L{§§§\‘c{.‘§“¥§§3‘.}=5$
`§\\§\\\~\\\\§\\\§§§\\\\\§A\;
`...“.\..~.\
`,»-* N.»-*-““ZI‘...‘.:>»~»:.~;;\“-\\K
`.
`N.»-7~“W{‘_
`3
`-
`‘
`'-\_;..
`-\ M,_.§§N:;‘\
`“__..»-~“
`I.--'~“"
`\..~»-~*
`§
`‘
`~
`
`\
`\
`\
`:
`\
`.
`3
`§
`2
`
`
`
`\'w~,.».-'~"‘
`Q
`
`_
`
`~‘
`
`€,.-
`
`_
`‘
`‘3 '3"
`..»..
`..
`
`‘
`
`\\
`_
`.
`§§§§*§§
`‘
`i.
`
`§
`\
`‘ ‘\\s\\ “\.‘\\.
`5*
`s
`‘
`K
`\
`»
`=
`arm;-§:\
`‘
`k
`3
`\-\- ‘-n\-(-
`§
`>.-‘$323: §R.*3i-3‘.
`“““\“u““““w.“‘o
`
`S
`g
`§
`s
`§
`
`
`
`i
`_
`3*
`.3
`_.
`
`3’
`
`,4‘
`.-"
`
`
`
`~\
`..
`\
`3
`\
`..
`S
`3
`‘
`
`ix-"
`
`{,.¢
`
`//
`
`
`
`,.z»mMw..mmm.«-
`
`\w»-
`g
`
`..m\.m“-\\\
`\
`..~ \\
`\\
`w...~><\~§\ \
`
`_“.-~
`
`\
`,,,\..-¥"
`
`j'1:45:by7/'3Mrr/1'
`
`
`
`
`
`
`
`PI
`
`-4(
`
`
`
`Z14955
`
`;39
`
`/
`
`-
`
`\\"__‘_...»-*"“
`
`»mmmmmyznn.a
`
`own
`ac‘ . § we
`
`
`
`
`
`
`
`
`
`Microsoft Corp. Exhibit 1002
`
`Microsoft Corp. Exhibit 1002
`
`

`
`
`U.S. Patent
`
`
`
`Dec. 30, 2014
`
`
`
`
`
`Sheet 2 of5
`
`
`
`
`
`US 8,924,506 B2
`
`
`
`‘xx\xxxxxx-s\sssss\s'~\\'\V(\\\\\\\\\
`%
`.
`.
`S
`‘N =3-Kan;-1
`* .. .. .
`\
`\
`;‘mm-
`ix
`......w.\.....§m~..~...~mo
`‘
`Q
`
`.-i “
`q.
`“
`
`3 -'1‘
`{
`S.-v Q. S»;
`
`
`
`§;-‘:
`~'-“
`
`\
`
`.
`
`‘
`
`..
`
`.
`
`“
`
`\
`3
`§
`-.~»~
`«\«\‘\.v:_\-
`‘
`§“mmm§..§
`§ “
`V‘
`\
`§ 3\:?i§§:“\‘§ .33 :
`i‘
`g
`3
`~\ r ~o -
`§§ .e*§s:.=.m
`_
`,.~
`.‘
`.
`‘.\._,§
`3:. Q .~
`" :
`.9{x‘~}L\§.§$
`‘
`§
`§§
`§-§.$$.{§s.
`:\§ww“‘:w:;“_“§
`
`3
`
`‘
`
`. .
`
`\\\\\\v\
`
`\
`\
`
`‘
`§
`
`gs}
`
`._.
`-\.
`.>~:‘:u:z,
`. ---- ---.\
`‘R:
`§
`$ ¥.‘3€.§<$3%%%is
`i
`ii \.
`§\\\\\\\~\\\s~
`§
`£3
`§
`k
`...»..w.»......m£...m\mm.‘.
`“
`V
`§*:§.m»s§.»¥§:ss*§:3a§~ .
`mm “M
`"““‘““““
`R» .3‘: .
`§
`..
`.
`..\{
`-.\»m~\.“w.m...~...mm..:«
`
`ix‘ \‘§
`_¢o .x}g'}
`
`s\._ ‘\
`
`__ ,.
`x}
`3 ma:
`‘\\‘p\
`~.
`.
`E\§“5{\§§.§$§1‘
`
`§
`s
`\
`3..........§\\
`\
`
`N
`3‘
`*" ‘
`.{3§\‘¢§ §»_5
`
`‘
`,
`.
`.
`_
`.
`\,
`‘N .. V,‘ 5:. ~.\
`3\e>s‘§:s‘:m§§ \\:a‘§ie‘§§< §3§
`
`
`
`A r\
`§%{‘s
`
`"A"
`
`'
`§§;§ M
`\\
`
`¥~
`
`...~m\...\\\\\
`
`
`
`.
`..,.....~..\\.~“m“\\m..........~..» w.“.mm\....\mm.‘..‘..
`k
`...‘........E
`§
`5
`3
`3
`.,
`m“....\..........3
`'
`
`
`i‘$t‘.‘:s_§'i\\““§ §‘§i§.i“-‘§‘2'
`.1\a.~“»~k§ Mm ».
`_\
`.~
`0 ,.
`‘,..-"
`9*
`
`\
`
`’
`
`
` §§§§& fii
`
`*
`'
`7
`"
`'
`i
`\\\\\\\\\\\\\\\\\V -w......\..
`“M
`"\\w~:\
`V M...
`---.
`
`.
`
`>
`
`‘
`
`‘
`
`\
`§
`
`«
`§§;{**;}
`_,§«..._,.,..\.»\.
`-
`\
`§2~ui..i>;~i»‘a
`
`i
`z
`.
`$\\\\\\\\\\\\\\\§\\\\\‘\
`is-'~'-'~'-xv-'-r\':\~\»\\~\\\\\\\.\\\\\.~§
`\
`.
`3
`S
`‘
`\
`a
`3
`-x
`§
`5
`3
`: §§~§;:\
`3
`S
`3‘
`\
`§.\ ~.“ \
`‘
`§\\..M..w...~.§.§
`§
`\
`x
`§
`§
`§
`S
`$
`3
`*
`S
`E
`E
`s
`z
`»‘
`. -.-\\““-.-.~“ ».».».\-.\»“\.~\u\§
`\.\u\«u-;«s«ss»»~s~§
`§
`X
`*
`..,
`§
`‘$6
`5\\\~a\\\\\\\\\\v.~\\§-~\~\\~\~\\-‘~19.-.-.
`~\
`\ '
`‘ \'
`Q‘
`‘E
`-
`;\_\ ,\
`i
`is §3.§‘§;‘.«$Ai}§i.
`é\§\\NV\\\V\\\\\\\\\h\\\\\\\§\h\hV»\'u'§\°
`23$:
`
`'
`
`Microsoft Corp. Exhibit 1002
`
`Microsoft Corp. Exhibit 1002
`
`

`
`U.S. Patent
`
`Dec. 30, 2014
`
`Sheet 3 of5
`
`US 8,924,506 B2
`
`NAVIGATION
`
`Dmmus
`VIEWABLE PARCELS
`
`Fon UPDATE
`
`80
`
`FIG. 5
`
`82
`
`
`
`
`
`Tam
`
`Ounome Nouss
`
`lssus Pmoszm
`REOUESIS
`
`REQUEST PARCEL
`
`DOWNLOADS
`
`Nnwoxx
`
`33
`
`
`
`SYORE mm In -
`Ouzxoms Non:
`
`94
`
`
`
`Wm Fox
`REQUEST Inmo
`Pmonm REGUES1
`FREE
`Wamuc
`
`
`Sum Hneum
`
`100 J
`
`105
`
`Microsoft Corp. Exhibit 1002
`
`

`
`U.S. Patent
`
`Dec. 30, 2014
`
`Sheet 4 of 5
`
`US 8,924,506 B2
`
`mass: ¥:§a*a:é:§
`fiaszzris
`§.<:z<m:s:4 ass
`
`€?§€€¥és'&'§¥t‘;$i
`
`9x§a§m=
`Ssm‘ mm as
`
`Sax: Féfifit
`
`§€.§%:‘*.‘.~€:z ms:
`?.ss:2:‘£:. {£3
`8:s§u¥ xi-‘.é’£%t}§¥
`
`is {M-mt 33m
`Liiikfiiii ¥s§1:¥{§i ?
`
`fimza imtmr
`mas $3
`ilmxsm §4§§fi§§R¥
`
`'
`
`{ma ?§mss‘:
`
`T
`
`iksszgi
`
`E
`
`Ekmzmxs
`{}§’§§&U\i {mm
`{Km i.
`
`. xmuwwn» «wean-A
`
`E ism ¥:ia‘es.ssé;
`{Riki}? REE
`
`3
`
`A 52
`w...,£f.“......_..... _
`9 Ass§ss§:;:stsas ;
`i
`:°::§::s1sm§;sx
`
`§§§$f§§ £‘£i$§‘i§§$
`§’.§R{§§$ $3‘
`ifiifi. E.
`
`Microsoft Corp. Exhibit 1002
`
`

`
`
`U.S. Patent
`
`
`
`
`
`
`
`
`
`
`
`Sheet 5 of 5
`
`
`
`
`
`
`
`%§.»§§.§§§swRx..x..\$5Aw.»..SN“.....\.Wu..Awe...“\.......3...xwNW5WX.m.D...x....3:...a:2...\“n.\.,§{SWm........%..man..xNm«xV...N»\\.“1..weWe«S..C0.»mmM>““\\\R»atNax:m.
`
`.\...~\\...n....\s\\..x.>...
`
`
`
`
`
`.N«33m““\\\\i\\\>\5\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\5\\5)\5\\\\\5\\$\\.5\\\\\\\\\\\\\\f\\IIII\\\\\\\\\\\\\\.\\\\\\\\\\\5\\\\\\\\\\\\
`.“xm\......\..:...\..xmHw\_..xx..““..\\..§»..“........{m.w..R“R.w\s.«._.\\\\\u“v\\\.\m\\s._.3‘““M...»x...333....\\.\§\&\..\.§.....
`..........._...
`...e..\\\..muS\~wnu.“mN..\...mNw\\\.tax......ms...»\xbxw5....«§-\\x-x..\.““aw...n“M?«..12NN...\\\.x.\\.\..3.....\...............\\
`.§\..\\\\\\\\\\\\\.\\.\\.sssssxsxx§\\\..\\\\§\\\\u...x\.\.m\s.¢..
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`.<n.\“.2»N...4u.~.\\..4}.“mu.\\N...““..as1W...yH«X.33\\..\w..\\0.XWNWmy2“x.N“3“.3...“3.Q§%\\»“«..\..0“..\..“»N““.\..N9.»NMNm“N3....3.......\asNNNw..§...§......§.&
`
`?$$$».......3................\$..........u...x..it.sx\.2.
`
`
`\&\\\\\).i\\\)\\\\\\\\\\\\\\\.....\..v.mm....ush..».“Mmm“....\
`
`............w...........~.............-.V
`
`.\$33s....s.....K......\.\......§.......2...§.§.3$:3....S..s§§S$\.....$....3§.§§§a:§.3.\§§ S§§§§§§§§§§.::§....s...........3§ss2s....ss.»
`
`..Nu.“..
`
`..
`
`.V:
`
`
`..,\\.¢\£38..K.....
`xx..S.w...\.WWNSn.“M.UVANW«..3.?N.M:1.....w.....\.§3.Mxmm“.9.MSNE..M:2.x%\\
`.5..x.....9.......3x.\xxx....\\.\9\V...4.....$\&...«as.\\.VAla.ix:
`
`
`flmmu.\.$.mw..\9....“».\..§§§W&«H.xii...“§3.WWanW...\Qm%...2..xa.»\...EWW.”$3..“F:...S:$..,W».»..\3No33..§..\.v».x.<\..«..“..Q..«xx...snxs.\\\.\.aux~*\....~nu;.9«.<as......N.era~..:.N““.Mm‘RmR:M“V(V.V33..
`
`
`
`
`...\x-NNWW.\MJ\¢ssu.....§33.»§.§
`
`
`..usv..u.4\.\.nq.ux\-\.s\K...3.33k“w§...z$_.man“»\.\\\\\\\\“H9“.Ln.kw«“.5.”{J68Wt»«admum“Y:x....“..a..§HMV«“iv”{L\s..uNx.\.x.x\«Hum3k.“N22.
`
`
`
`
`..\x“Q\\\x\\\\3$\\\§\\\.\hSS\\.S9RS8...\\\\\\\.\\s\\.&&$.&\\\§ ..&R§\&a¢\\\
`4...
`
`
`.3.....§...§§..§..\§§§§§§\N.Mn..u.5.Ms...Wm...
`...~N...\\m..VN\N\m«
`
`0NA5P._am«uhn.....N9..\Wn.\..\\\M}.4N\n\.
`
`5\...{.\.....ex»
`
`
`.
`
`ssssssssssxxxxx.<\\\\\\\\\\\\\\\\..c.\\\\\n\\\\\\\\\\\\\\\\\\\\\\\\>33.s>:5.s>.>is>.<>.\>2§\§5:33S:is>25i5§s\tS§s§$>%\\§s\3x§S:x\§>3!<s5i533§\\\\§s:sx
`
`\
`
`Microsoft Corp. Exhi
`
`it 1002
`
`Microsoft Corp. Exhibit 1002
`
`
`
`
`

`
`2
`
`
`
`US 8,924,506 B2
`
`
`
`
`1
`OPTIMIZED IMAGE DELIVERY OVER
`
`
`
`
`LIMITED BANDWIDTH COMMUNICATION
`
`
`CHANNELS
`
`
`
`
`
`PRIORITY CLAIMS/RELATED APPLICATIONS
`
`
`
`
`
`This application is a continuation in part of and claims
`
`
`
`
`
`
`
`
`
`
`priority under 35 USC 120 to U.S. patent application Ser. No.
`
`
`
`
`
`
`
`
`
`
`
`12/619,643 filed on Nov. 16, 2009 now U.S. Pat. No. 7,908,
`
`
`
`
`
`
`
`
`
`
`
`343 which in turn in a continuation of and claims priority
`
`
`
`
`
`
`
`
`
`
`
`under 35 USC 120 to U.S. patent application Ser. No. 10/035,
`
`
`
`
`
`
`
`
`
`
`
`987 filed on Dec. 24, 2001 and entitled “Optimized image
`
`
`
`
`
`
`
`
`
`
`delivery over limited bandwidth communication channels”
`
`
`
`
`
`
`(that now issued on Jan. 5, 2010 as U.S. Pat. No. 7,644,131)
`
`
`
`
`
`
`
`
`
`
`
`
`which in turn claims the benefit under 35 USC 1 19(e) of U.S.
`
`
`
`
`
`
`
`
`
`Provisional Application Nos. 60/258,488, 60/258,489,
`
`
`
`
`
`60/258,465, 60/258,468, 60/258,466, and 60/258,467, all
`
`
`
`
`
`
`filed Dec. 27, 2000, all of which are incorporated herein by
`
`
`
`
`
`
`
`
`
`
`
`reference. The present application is also related to the co-
`
`
`
`
`
`
`
`
`
`pending application Ser. No. 10/035,981 entitled “System
`
`
`
`
`
`
`and Methods for Network Image Delivery with Dynamic
`
`
`
`
`
`
`
`Viewing Frustum Optimized for Limited Bandwidth Com-
`
`
`
`
`
`
`munication Channels, Levanon et al., filed on Dec. 24, 2001
`
`
`
`
`
`
`
`
`
`(now U.S. Pat. No. 7,139,794 issued on Nov. 21, 2006 and
`
`
`
`
`
`
`
`
`
`
`
`which is assigned to the Assignee of the present Application.
`
`
`
`
`
`
`
`
`
`
`FIELD
`
`
`
`
`
`
`The disclosure is related to network based, image distribu-
`
`
`
`
`
`
`
`
`tion systems and, in particular, to a system and methods for
`
`
`
`
`
`
`
`
`
`
`
`efiiciently selecting and distributing image parcels through a
`
`
`
`
`
`
`
`narrowband or otherwise limited bandwidth communications
`
`
`
`
`
`
`channel to support presentation of high-resolution images
`
`
`
`
`
`
`
`subject to dynamic viewing frustums.
`
`
`
`
`
`BACKGROUND
`
`
`
`The Internet and or other network systems may provide a
`
`
`
`
`
`
`
`
`
`unique opportunity to transmit for example complex images,
`
`
`
`
`
`
`
`
`typically large scale bit-maps, particularly those approaching
`
`
`
`
`
`
`
`photo-realistic levels, over large area and or distances. In
`
`
`
`
`
`
`
`
`
`common application, the images may be geographic, topo-
`
`
`
`
`
`
`
`graphic, and or other highly detailed maps. The data storage
`
`
`
`
`
`
`
`
`
`
`requirements and often proprietary nature of such images
`
`
`
`
`
`
`
`
`could be such that conventional interests may be to transfer
`
`
`
`
`
`
`
`
`
`
`the images on an as-needed basis.
`
`
`
`
`
`
`In conventional fixed-site applications, the image data may
`
`
`
`
`
`
`
`
`be transferred over a relatively high-bandwidth network to
`
`
`
`
`
`
`
`
`client computer systems that in turn, may render the image.
`
`
`
`
`
`
`
`
`
`
`Client systems may typically implement a local image navi-
`
`
`
`
`
`
`
`gation system to provide zoom and or pan functions based on
`
`
`
`
`
`
`
`
`
`
`user interaction. As well recognized problem with such con-
`
`
`
`
`
`
`
`
`ventional systems could be that full resolution image presen-
`
`
`
`
`
`
`
`
`tation may be subject to the inherent transfer latency of the
`
`
`
`
`
`
`
`
`
`
`network. Different conventional systems have been proposed
`
`
`
`
`
`
`to reduce the latency affect by transmitting the image in
`
`
`
`
`
`
`
`
`
`highly compressed formats that support progressive resolu-
`
`
`
`
`
`
`tion build-up of the image within the current client field of
`
`
`
`
`
`
`
`
`
`
`
`view. Using a transform compressed image transfer function
`
`
`
`
`
`
`
`increases the field of the image that can be transferred over a
`
`
`
`
`
`
`
`
`
`
`
`fixed bandwidth network in unit time. Progressive image
`
`
`
`
`
`
`
`
`resolution transmission, typically using a differential resolu-
`
`
`
`
`
`tion method, permits an approximate image to be quickly
`
`
`
`
`
`
`
`
`presented with image details being continuously added over
`
`
`
`
`
`
`
`time.
`
`Tzou, in U.S. Pat. No. 4,698,689, describes a two-dimen-
`
`
`
`
`
`
`
`
`sional data transform system that supports transmission of
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`5
`
`10
`
`
`
`15
`
`
`
`20
`
`
`
`25
`
`
`30
`
`
`
`35
`
`
`
`40
`
`
`
`45
`
`
`
`50
`
`
`
`55
`
`
`
`60
`
`
`
`65
`
`
`
`differential coefiicients to represent an image. Subsequent
`
`
`
`
`
`
`
`transmitted coefiicient sets are progressively accumulated
`
`
`
`
`
`
`with prior transmitted sets to provide a succeedingly refined
`
`
`
`
`
`
`
`
`image. The inverse-transform function performed by the cli-
`
`
`
`
`
`
`
`ent computer is, however, highly compute intensive. In order
`
`
`
`
`
`
`
`
`
`to simplify the transform implementation and further reduce
`
`
`
`
`
`
`
`
`the latency of presenting any portion of an approximate
`
`
`
`
`
`
`
`
`
`image,
`images are sub-divided into a regular array. This
`
`
`
`
`
`
`
`
`
`enables the inverse-transform function on the client, which is
`
`
`
`
`
`
`
`
`
`time-critical, to deal with substantially smaller coefficient
`
`
`
`
`
`
`
`data sets. The array size in Tzou is fixed, which leads to
`
`
`
`
`
`
`
`
`
`
`
`
`progressively larger coefiicient data sets as the detail level of
`
`
`
`
`
`
`
`
`
`
`the image increases. Consequently, there is an inherently
`
`
`
`
`
`
`
`
`increasing latency in resolving finer levels of detail.
`
`
`
`
`
`
`
`
`An image visualization system proposed byYap et al., U.S.
`
`
`
`
`
`
`
`
`
`Pat. No. 6,182,114, overcomes some of the foregoing prob-
`
`
`
`
`
`
`
`
`lems. TheYap et al. system also employs a progressive encod-
`
`
`
`
`
`
`
`
`ing transform to compress the image transfer stream. The
`
`
`
`
`
`
`
`
`transform also operates on a subdivided image, but the divi-
`
`
`
`
`
`
`
`
`sion is indexed to the encoding level of the transform. The
`
`
`
`
`
`
`
`
`
`
`encoded transform coefiicient data sets are, therefore, of con-
`
`
`
`
`
`
`
`
`stant size, which supports a modest improvement in the algo-
`
`
`
`
`
`
`
`
`rithmic performance of the inverse transform operation
`
`
`
`
`
`
`required on the client.
`
`
`
`
`Yap et al. adds utilization of client image panning or other
`
`
`
`
`
`
`
`
`
`image pointing input information to support a foveation-
`
`
`
`
`
`
`
`based operator to influence the retrieval order of the subdi-
`
`
`
`
`
`
`
`
`
`vided image blocks. This two-dimensional navigation infor-
`
`
`
`
`
`
`mation is used to identify a foveal region that is presumed to
`
`
`
`
`
`
`
`
`
`
`
`be the gaze point of a client system user. The foveation opera-
`
`
`
`
`
`
`
`
`
`
`tor defines the corresponding image block as the center point
`
`
`
`
`
`
`
`
`
`of an ordered retrieval of coefiicient sets representing a vari-
`
`
`
`
`
`
`
`
`able resolution image. The gaze point image block represents
`
`
`
`
`
`
`
`
`the area of highest image resolution, with resolution reduc-
`
`
`
`
`
`
`
`
`tion as a function of distance from the gaze point determined
`
`
`
`
`
`
`
`
`
`
`by the foveation operator. This technique thus progressively
`
`
`
`
`
`
`
`
`builds image resolution at the gaze point and succeedingly
`
`
`
`
`
`
`
`
`
`outward based on a relatively compute intensive function.
`
`
`
`
`
`
`
`
`Shifts in the gaze point can be responded to with relative
`
`
`
`
`
`
`
`
`
`
`
`speed by preferentially retrieving coefficient sets at and near
`
`
`
`
`
`
`
`
`
`the new foveal region.
`
`
`
`
`Significant problems remain in permitting the convenient
`
`
`
`
`
`
`
`and effective use of complex images by many different types
`
`
`
`
`
`
`
`
`
`
`of client systems, even with the improvements provided by
`
`
`
`
`
`
`
`
`
`the various conventional systems. In particular, the imple-
`
`
`
`
`
`
`
`mentation of conventional image visualization systems is
`
`
`
`
`
`
`generally unworkable for smaller, often dedicated or embed-
`
`
`
`
`
`
`
`ded, clients where use ofimage visualization would clearly be
`
`
`
`
`
`
`
`
`
`
`beneficial. Conventional approaches effectively presume that
`
`
`
`
`
`
`client systems have an excess of computing performance,
`
`
`
`
`
`
`
`
`memory and storage. Small clients, however, typically have
`
`
`
`
`
`
`
`
`restricted performance processors with possibly no dedicated
`
`
`
`
`
`
`
`floating-point support, little general purpose memory, and
`
`
`
`
`
`
`
`extremely limited persistent storage capabilities, particularly
`
`
`
`
`
`
`relative to common image sizes. A mobile computing device
`
`
`
`
`
`
`
`
`
`such as mobile phone, smart phone, tablet and or personal
`
`
`
`
`
`
`
`
`
`
`digital assistant
`(PDA)
`is a characteristic small client.
`
`
`
`
`
`
`
`
`Embedded, low-cost kiosk, automobile navigation systems
`
`
`
`
`
`
`and or Internet enabled/connected TV are other typical
`
`
`
`
`
`
`
`
`examples. Such systems are not readily capable, if at all, of
`
`
`
`
`
`
`
`
`
`
`
`performing complex, compute-intensive Fourier or wavelet
`
`
`
`
`
`
`transforms, particularly within a highly restricted memory
`
`
`
`
`
`
`
`address space.
`
`
`As a consequence of the presumption that the client is a
`
`
`
`
`
`
`
`
`
`
`substantial computing system, conventional image visualiza-
`
`
`
`
`
`tion systems also presume that the client is supported by a
`
`
`
`
`
`
`
`
`
`
`
`complete operating system. Indeed, many expect and require
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Microsoft Corp. Exhibit 1002
`
`Microsoft Corp. Exhibit 1002
`
`

`
`
`
`US 8,924,506 B2
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Thus, a general purpose of the present invention is to pro-
`
`
`
`
`
`
`
`
`
`
`vide an efiicient system and methods of optimally presenting
`
`
`
`
`
`
`
`
`image data on client systems with potentially limited process-
`
`
`
`
`
`
`
`
`ing performance, resources, and communications bandwidth.
`
`
`
`
`
`
`This is achieved in the present invention by providing for
`
`
`
`
`
`
`
`
`
`
`the retrieval of large-scale images over network communica-
`
`
`
`
`
`
`tions charmels for display on a client device by selecting an
`
`
`
`
`
`
`
`
`
`update image parcel relative to an operator controlled image
`
`
`
`
`
`
`
`
`viewpoint to display via the client device. A request is pre-
`
`
`
`
`
`
`
`
`
`
`pared for the update image parcel and associated with a
`
`
`
`
`
`
`
`
`
`request queue for subsequent issuance over a communica-
`
`
`
`
`
`
`
`tions channel. The update image parcel is received from the
`
`
`
`
`
`
`
`
`
`communications channel and displayed as a discrete portion
`
`
`
`
`
`
`
`of the predetermined image. The update image parcel opti-
`
`
`
`
`
`
`
`
`mally has a fixed pixel array size, is received in a single and or
`
`
`
`
`
`
`
`
`
`
`
`
`plurality of network data packets, and were the fixed pixel
`
`
`
`
`
`
`
`
`
`
`array may be constrained to a resolution less than or equal to
`
`
`
`
`
`
`
`
`
`
`
`
`the resolution of the client device display.
`
`
`
`
`
`
`
`An advantage of the present invention is that both image
`
`
`
`
`
`
`
`
`
`parcel data requests and the rendering of image data are
`
`
`
`
`
`
`
`
`
`optimized to address the display based on the display resolu-
`
`
`
`
`
`
`
`
`
`tion of the client system.
`
`
`
`
`
`
`
`
`
`
`3
`an extensive set of graphics abstraction layers to be provided
`
`
`
`
`
`
`
`
`
`by the client system to support the presentation of the deliv-
`
`
`
`
`
`
`
`
`
`
`ered image data. In general, these abstraction layers are con-
`
`
`
`
`
`
`
`
`
`ventionally considered required to handle the mapping of the
`
`
`
`
`
`
`
`
`
`image data resolution to the display resolution capabilities of 5
`
`
`
`
`
`
`
`
`
`the client system. That is, resolution resolved image data
`
`
`
`
`
`
`
`
`
`provided to the client is unconstrained by any limitation in the
`
`
`
`
`
`
`
`
`
`
`
`client system to actually display the corresponding image.
`
`
`
`
`
`
`
`
`Consequently,
`substantial processor performance
`and
`
`
`
`
`
`memory can be conventionally devoted to handling image
`
`
`
`
`
`
`
`
`data that is not or cannot be displayed.
`
`
`
`
`
`
`
`
`Another problem is that small clients are generally con-
`
`
`
`
`
`
`
`
`strained to generally to very limited network bandwidths,
`
`
`
`
`
`
`
`
`particularly when operating under wireless conditions. Such
`
`
`
`
`
`
`
`limited bandwidth conditions may exist due to either the
`
`
`
`
`
`
`
`
`
`direct technological constraints dictated by the use of a low
`
`
`
`
`
`
`
`
`
`bandwidth data channel or indirect constraints imposed on
`
`
`
`
`
`
`
`
`relatively high-bandwidth charmels by high concurrent user
`
`
`
`
`
`
`
`loads. Cellular connected PDAs and webphones are examples
`
`
`
`
`
`
`
`
`of small clients that are frequently constrained by limited
`
`
`
`
`
`
`
`
`
`bandwidth conditions. The conventionally realizable maxi-
`
`
`
`
`
`mum network transmission bandwidth for such small devices
`
`
`
`
`
`
`
`
`may range from below one kilobit per second to several tens
`
`
`
`
`
`
`
`
`
`
`
`of kilobits per second. While Yap et al. states that
`the
`
`
`
`
`
`
`
`
`
`
`
`described system can work over low bandwidth lines, little
`
`
`
`
`
`
`
`
`
`more than utilizing wavelet-based data compression is
`
`
`
`
`
`
`
`advanced as permitting effective operation at low communi-
`
`
`
`
`
`
`
`cations bandwidths. While reducing the amount of data that
`
`
`
`
`
`
`
`
`
`must be carried from the server to the client is significant, Yap
`
`
`
`
`
`
`
`
`
`
`
`
`et al. simply relies on the data packet transfer protocols to
`
`
`
`
`
`
`
`
`
`
`
`provide for an efficient transfer ofthe compressed image data.
`
`
`
`
`
`
`
`
`
`
`Reliable transport protocols, however, merely mask packet
`
`
`
`
`
`
`
`losses and the resultant, sometimes extended, recovery laten-
`
`
`
`
`
`
`
`cies. When such covered errors occur, however, the aggregate
`
`
`
`
`
`
`
`
`bandwidth of the connection is reduced and the client system
`
`
`
`
`
`
`
`
`
`can stall waiting for further image data to process.
`
`
`
`
`
`
`
`
`
`Consequently, there remains a need for an image visual-
`
`
`
`
`
`
`
`
`ization system that can support small client systems, place
`
`
`
`
`
`
`
`
`few requirements on the supporting client hardware and soft-
`
`
`
`
`
`
`
`
`ware resources, and efiiciently utilize low to very low band-
`
`
`
`
`
`
`
`
`
`width network connections.
`
`
`
`SUMMARY
`
`4
`
`
`Another advantage of the present invention is that the pri-
`
`
`
`
`
`
`
`
`
`oritization of image parcel requests is based on an adaptable
`
`
`
`
`
`
`
`
`
`
`parameter that minimizes the computational complexity of
`
`
`
`
`
`
`
`determining request prioritization and, in turn, the progres-
`
`
`
`
`
`
`
`sive improvement in display resolution within the field of
`
`
`
`
`
`
`
`
`view presented on a client display.
`
`
`
`
`
`A further advantage of the present invention is that the
`
`
`
`
`
`
`
`
`
`client software system requires relatively minimal client pro-
`
`
`
`
`
`
`
`cessing power and storage capacity. Compute intensive
`
`
`
`
`
`
`
`numerical calculations are minimally required and image
`
`
`
`
`
`
`
`parcel data is compactly stored in efiicient data structures.
`
`
`
`
`
`
`
`
`
`The client software system is very small and easily down-
`
`
`
`
`
`
`
`
`
`loaded to conventional computer systems or embedded in
`
`
`
`
`
`
`
`conventional dedicated function devices, including portable
`
`
`
`
`
`devices, such as PDAs, tablets and webphones.
`
`
`
`
`
`
`
`Still another advantage of the present invention is that
`
`
`
`
`
`
`
`
`
`image parcel data requests and presentation can be readily
`
`
`
`
`
`
`
`
`
`optimized to use low to very low bandwidth network connec-
`
`
`
`
`
`
`
`
`
`tions. The software system of the present invention provides
`
`
`
`
`
`
`
`
`for re-prioritization of image parcel data requests and presen-
`
`
`
`
`
`
`
`
`tation in circumstances where the rate of point-of-view navi-
`
`
`
`
`
`
`
`
`gation exceeds the data request rate.
`
`
`
`
`
`
`Yet another advantage ofthe present invention is that image
`
`
`
`
`
`
`
`
`
`
`parcel data rendering is performed without requiring any
`
`
`
`
`
`
`
`
`complex underlying hardware or software display subsystem.
`
`
`
`
`
`
`
`The client software system ofthe present invention includes a
`
`
`
`
`
`
`
`
`
`bit-map rendering engine that draws directly to the video
`
`
`
`
`
`
`
`
`
`memory ofthe display, thus placing minimal requirements on
`
`
`
`
`
`
`
`
`
`any underlying embedded or disk operating system and dis-
`
`
`
`
`
`
`
`
`play drivers. Complex graphics and animation abstraction
`
`
`
`
`
`
`layers are not required.
`
`
`
`
`Still another advantage of the present invention is that
`
`
`
`
`
`
`
`
`image parcel block compression is used to obtain fixed size
`
`
`
`
`
`
`
`
`
`transmission data blocks. Image parcel data is recoverable
`
`
`
`
`
`
`
`from transmission data using a relatively simple client
`
`
`
`
`
`
`
`decompression algorithm. Using fixed size transmission data
`
`
`
`
`
`
`blocks enables image data parcels to be delivered to the client
`
`
`
`
`
`
`
`
`
`
`in bounded time frames.
`
`
`
`
`A yet further advantage of the present invention is that
`
`
`
`
`
`
`
`
`
`multiple data forms can be transferred to the client software
`
`
`
`
`
`
`
`
`
`system for concurrent display. Array overlay data, correlated
`
`
`
`
`
`
`
`positionally to the image parcel data and generally insensitive
`
`
`
`
`
`
`
`
`to image parcel resolution, can be initially or progressively
`
`
`
`
`
`
`
`
`provided to the client for parsing and parallel presentation on
`
`
`
`
`
`
`
`
`
`a client display image view.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`10
`
`
`
`15
`
`
`
`20
`
`
`
`25
`
`
`30
`
`35
`
`40
`
`45
`
`
`
`
`
`
`
`
`
`50
`
`
`
`55
`
`
`
`60
`
`
`
`65
`
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`
`
`
`
`
`
`These and other advantages and features of the present
`
`
`
`
`
`
`
`
`
`invention will become better understood upon consideration
`
`
`
`
`
`
`
`of the following detailed description of the invention when
`
`
`
`
`
`
`
`
`
`considered in connection with the accompanying drawings,
`
`
`
`
`
`
`
`in which like reference numerals designate like parts through-
`
`
`
`
`
`
`
`
`out the figures thereof, and wherein:
`
`
`
`
`
`
`FIG. 1 depicts a preferred system environment within
`
`
`
`
`
`
`
`which various embodiments of the present invention can be
`
`
`
`
`
`
`
`
`utilized;
`
`FIG. 2 is a block diagram illustrating the preparation of
`
`
`
`
`
`
`
`
`
`
`image parcel and overlay data set that are to be stored by and
`
`
`
`
`
`
`
`
`
`
`
`
`
`served from a network server system in accordance with a
`
`
`
`
`
`
`
`
`
`
`preferred embodiment of the present invention;
`
`
`
`
`
`
`FIG. 3 is a block diagram of a client system image presen-
`
`
`
`
`
`
`
`
`
`
`tation system constructed in accordance with a preferred
`
`
`
`
`
`
`
`embodiment of the present invention;
`
`
`
`
`
`
`
`
`
`
`
`Microsoft Corp. Exhibit 1002
`
`Microsoft Corp. Exhibit 1002
`
`

`
`
`
`US 8,924,506 B2
`
`
`
`
`5
`FIG. 4 provides a data block diagram illustrating an opti-
`
`
`
`
`
`
`
`
`
`mized client image block processing path constructed in
`
`
`
`
`
`
`
`accordance with a preferred embodiment of the present
`
`
`
`
`
`
`
`invention;
`
`FIG. 5 is a process flow diagram showing a main process-
`
`
`
`
`
`
`
`
`
`ing thread implemented in a preferred embodiment of the
`
`
`
`
`
`
`
`
`present invention;
`
`
`FIG. 6 provides a process flow diagram showing a network
`
`
`
`
`
`
`
`request thread implemented in a preferred embodiment of the
`
`
`
`
`
`
`
`present invention;
`
`
`FIG. 7 provides a process flow diagram showing a display
`
`
`
`
`
`
`
`image rendering thread implemented in a preferred embodi-
`
`
`
`
`
`
`ment of the present invention;
`
`
`
`
`
`FIG. 8 provides a process flow diagram showing the parcel
`
`
`
`
`
`
`
`
`map processing performed preliminary to the rendering of
`
`
`
`
`
`
`
`
`image data parcels in accordance with a preferred embodi-
`
`
`
`
`
`
`
`
`ment of the present invention;
`
`
`
`
`
`FIG. 9 provides a process flow diagram detailing the ren-
`
`
`
`
`
`
`
`
`dering and progressive prioritization of image parcel data
`
`
`
`
`
`
`
`download requests in accordance with a preferred embodi-
`
`
`
`
`
`
`
`ment of the present invention; and
`
`
`
`
`
`
`FIG. 10 provides a process flow diagram detailing the
`
`
`
`
`
`
`
`
`
`determination of an optimal detail level for image parcel
`
`
`
`
`
`
`
`
`
`presentation for a current viewing frustum in accordance with
`
`
`
`
`
`
`
`
`a preferred embodiment of the present invention.
`
`
`
`
`
`
`
`DETAILED DESCRIPTION OF ONE OR MORE
`
`
`
`
`
`EMBODIMENTS
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`The preferred operational environment 10 of the present
`
`
`
`
`
`
`
`
`invention is generally shown in FIG. 1. A network server
`
`
`
`
`
`
`
`
`
`
`system 12, operating as a data store and server of image data,
`
`
`
`
`
`
`
`
`
`
`
`is responsive to requests received through a communications
`
`
`
`
`
`
`
`network, such as the Internet 14 generally and various tiers of
`
`
`
`
`
`
`
`
`
`
`
`internet service providers (ISPs) including a wireless connec-
`
`
`
`
`
`
`tivity provider 16. Client systems, including conventional
`
`
`
`
`
`
`workstations and personal computers 18 and smaller, typi-
`
`
`
`
`
`
`
`cally dedicated function devices often linked through wire-
`
`
`
`
`
`
`
`less network connections, such as PDAs, webphones 20, and
`
`
`
`
`
`
`
`
`
`automobile navigation systems, source image requests to the
`
`
`
`
`
`
`
`
`network server 12, provide a client display and enable image
`
`
`
`
`
`
`
`
`
`navigational input by a user of the client system. Alternately,
`
`
`
`
`
`
`
`
`
`a dedicated function client system 20 may be connected
`
`
`
`
`
`
`
`
`
`through a separate or plug-in local network server 22, pref-
`
`
`
`
`
`
`
`
`erably implementing a small, embedded Web server, to a fixed
`
`
`
`
`
`
`
`
`or removable storage local image repository 24. Characteris-
`
`
`
`
`
`
`
`tically, the client system 18, 20 displays are operated at some
`
`
`
`
`
`
`
`
`
`
`fixed resolution generally dependent on the underlying dis-
`
`
`
`
`
`
`
`play hardware of the client systems 18, 20.
`
`
`
`
`
`
`
`
`The image navigation capability supported by the present
`
`
`
`
`
`
`
`
`invention encompasses a viewing frustum placed within a
`
`
`
`
`
`
`
`
`three-dimensional space over the imaged displayed on the
`
`
`
`
`
`
`
`
`client 18, 20. Client user navigational inputs are supported to
`
`
`
`
`
`
`
`
`
`
`control the x, y lateral, rotational and z height positioning of
`
`
`
`
`
`
`
`
`
`
`the viewing frustum over the image as well as the camera
`
`
`
`
`
`
`
`
`
`
`
`angle of incid