US009641645B2
`
`(12) United States Patent
`US 9,641,645 B2
`(10) Patent No.:
`Levanon et al.
`(45) Date of Patent:
`*May 2, 2017
`
`(54)
`
`OPTIMIZED IMAGE DELIVERY OVER
`LIMITED BANDWIDTH COMMUNICATION
`CHANNELS
`
`(71)
`
`Applicant: Bradium Technologies LLC, Suffem,
`NY (US)
`
`(72)
`
`Inventors:
`
`Isaac Levanon, Raanana (IL); Yonatan
`Lavi, Raanana (IL)
`
`(73)
`
`Assignee: BRADIUM TECHNOLOGIES LLC,
`Suffern, NY (US)
`
`(*)
`
`Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`
`This patent is subject to a terminal dis-
`claimer.
`
`(21)
`
`Appl. No.: 15/281,037
`
`(22)
`
`Filed:
`
`Sep. 29, 2016
`
`(65)
`
`(63)
`
`Prior Publication Data
`
`US 2017/0019501 A1
`
`Jan. 19, 2017
`
`Related US. Application Data
`
`Continuation of application No. 14/970,526, filed on
`Dec. 15, 2015, which is a continuation of application
`(Continued)
`
`Int. Cl.
`
`(51)
`
`G06F 15/16
`H04L 29/08
`
`US. Cl.
`
`(52)
`
`(2006.01)
`(2006.01)
`(Continued)
`
`CPC ............ .. H04L 67/327 (2013.01); G06F 3/14
`(2013.01); G06F 3/1454 (2013.01);
`(Continued)
`
`(58) Field of Classification Search
`CPC
`H04N 1/40068; H04N 1/4172; H04N 1/64;
`H04N 21/234345; H04N 21/234363;
`(Continued)
`
`(56)
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`4,682,869 A *
`
`4,972,319 A *
`
`7/1987 Itoh ....................... .. G06T 9/004
`358/426.12
`11/1990 Delorme ............ .. G09B 29/007
`340/990
`
`(Continued)
`
`OTHER PUBLICATIONS
`
`Declaration of Yonatan Lavi, Exhibit 1017 with exhibits (attach-
`ments) A-E filed in PTAB Case No. IPR2016-00448, all pages.
`
`Primary Examiner 7 David Lazaro
`(74) Attorney, Agent, or Firm iAnatoly S. Weiser, Esq.;
`Techlaw LLP.
`
`(57)
`
`ABSTRACT
`
`Large-scale images are retrieved over network communica-
`tions channels for display on a client device by selecting an
`update image parcel relative to an operator controlled image
`viewpoint to display via the client device. A request
`is
`prepared for the update image parcel and associated with a
`request queue for subsequent issuance over a communica-
`tions channel. The update image parcel is received from the
`communications channel and displayed as a discrete portion
`of the predetermined image. The update image parcel opti-
`mally has a fixed pixel array size, is received in a single and
`or plurality of network data packets, and were the fixed pixel
`array may be constrained to a resolution less than or equal
`to the resolution of the client device display.
`
`36 Claims, 5 Drawing Sheets
`
`e 5
`
`x\\\\\\\m\\\\\s\\~\n\\\\\\i\\\§
`1:;
`\h‘
`§*gw{~
`§\\
`“183;;
`R x3
`\\\\\\~m\\m~§xw~~~-~\~\\s
`\
`\~m~\\‘\\\‘~\\\\§\\\“n\~\\\\\\
`
`\\\\\ ‘-\\\\
`
`\
`
`.4} V
`q. }
`
`s'
`
`
`
`' M
`
`
`rm“
`
`i
`Jamaamm
`
`“‘x
`
`§\\sv-«v~~‘~
`w , “some
`3
`Y
`, 09:“ mm
`'
`':
`
`hm
`
`Microsoft Corp. Exhibit 1001
`
`x g
`
`~ “ \\\\s\\s\\«\\\§
`‘
`‘
`
`‘
`\
`
`§§ F§S§§§§§§¥
`?‘$‘
`
`i
`s‘
`§§
`{'3'
`gamma
`
`gm
`0”
`
`Microsoft Corp. Exhibit 1001
`
`

`

`US 9,641,645 B2
`Page 2
`
`Related U.S. Application Data
`
`No. 14/547,148, filed on NOV. 19, 2014, now Pat. No.
`9,253,239, which is a continuation of application No.
`13/027,929, filed on Feb. 15, 2011, now Pat. No.
`8,924,506, which is a continuation-in-part of appli-
`cation No. 12/619,643, filed on Nov. 16, 2009, now
`Pat. No. 7,908,343, which is a continuation of appli-
`cation No. 10/035,987, filed on Dec. 24, 2001, now
`Pat. No. 7,644,131.
`
`Provisional application No. 60/258,465, filed on Dec.
`27, 2000, provisional application No. 60/258,466,
`filed on Dec. 27, 2000, provisional application No.
`60/258,467,
`filed on Dec. 27, 2000, provisional
`application No. 60/258,468, filed on Dec. 27, 2000,
`provisional application No. 60/258,488, filed on Dec.
`27, 2000, provisional application No. 60/258,489,
`filed on Dec. 27, 2000.
`
`(60)
`
`Int. Cl.
`
`(51)
`
`(2006.01)
`(2006.01)
`(2006.01)
`(2006.01)
`
`5,995,903
`
`A*
`
`11/1999
`
`6,167,442
`
`A*
`
`12/2000
`
`6,212,301
`
`B1*
`
`4/2001
`
`Warner
`
`6,246,797
`
`B1*
`
`6/2001
`
`6,285,317
`
`B1*
`
`9/2001
`
`6,314,452
`
`B1*
`
`11/2001
`
`Dekel
`
`6,326,965
`
`B1*
`
`12/2001
`
`Castelli
`
`6,345,279
`6,346,938
`
`B1*
`B1*
`
`2/2002
`2/2002
`
`6,397,259
`
`B1*
`
`5/2002
`
`6,449,639
`
`B1*
`
`9/2002
`
`6,496,189
`
`B1*
`
`12/2002
`
`6,525,732
`
`B1*
`
`2/2003
`
`6,608,628
`
`B1*
`
`8/2003
`
`B1*
`
`8/2003
`
`Dowell
`
`Smith .................. .. G01C21/00
`340/995.26
`Sutherland ........ .. G06F 17/3028
`709/217
`................... .. G06T 9/00
`382/232
`Castor .................. .. H04N19/63
`375/E7.035
`Ong ................. .. G01C 21/3647
`340/995.2
`................ .. H04N 19/647
`375/E7.045
`........... .. G06F 17/30241
`345/420
`................... .. G06F 17/30905
`Li
`Chan ................. .. G06F 3/04815
`345/419
`Lincke .............. .. G06F 17/3089
`707/E17.116
`Blumberg ....... .. G06F 17/30905
`707/E17.118
`Yaron ................... .. G06T 15/40
`345/419
`Gadh .................... .. G06T 15/20
`345/428
`Ross ..................... .. G06T 17/20
`345/619
`................. .. G06T 9/007
`382/232
`......................... .. G06T 15/40
`345/418
`Robotham .............. .. G06F 3/ 14
`345/581
`Harris ................... .. G06T 15/20
`709/231
`Chang ..................... .. G06T 1/00
`375/E7.065
`Wen .................... .. G06T 1/0078
`382/100
`................ .. H04N19/70
`375/E7.056
`Silverstein ....... .. H04N 21/2662
`375/E7.011
`Whitehead ............. .. H04N1/64
`375/E7.184
`.................. .. H04N 19/ 176
`375/E7.056
`Levanon ................. .. G06F 3/ 14
`345/625
`Levanon ............. .. G06F 3/1454
`345/625
`
`Li
`
`Atsumi
`
`Chai
`
`(52)
`
`(58)
`
`(56)
`
`G06F 3/14
`G06T 3/40
`H04L 29/06
`G09G 5/00
`U.S. Cl.
`
`CPC ......... .. G06T 3/4092 (2013.01); G09G 5/003
`(2013.01); H04L 65/602 (2013.01); H04L
`67/42 (2013.01); G09G 2340/02 (2013.01);
`G09G 2350/00 (2013.01); G09G 2370/02
`(2013.01); G09G 2370/16 (2013.01)
`Field of Classification Search
`CPC ...... .. H04N 21/25825; G06F 17/30241; G06F
`3/04815; G06T 3/4092; G06T 19/00;
`G06T 19/003
`USPC ............. .. 709/202, 203, 217, 218, 230, 231;
`382/305, 232; 345/428, 581, 625
`See application file for complete search history.
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`5,559,936 A *
`
`5,613,051 A *
`
`5,929,860 A *
`
`9/1996 Poulter
`
`........... .. G06F17/30017
`345/428
`3/1997 Iodice ..................... .. G06F 3/14
`345/428
`7/1999 Hoppe .................. .. G06T 9/001
`345/419
`
`6,608,933
`
`6,625,309
`
`B1*
`
`9/2003
`
`6,704,024
`
`B2*
`
`3/2004
`
`6,704,791
`
`B1*
`
`3/2004
`
`6,711,297
`
`B1*
`
`3/2004
`
`6,754,365
`
`B1*
`
`6/2004
`
`6,801,665
`
`B1*
`
`10/2004
`
`6,882,755
`
`B2*
`
`4/2005
`
`6,898,311
`
`B2*
`
`5/2005
`
`6,970,604
`
`B1*
`
`11/2005
`
`7,644,131
`
`B2*
`
`1/2010
`
`8,924,506
`
`B2*
`
`12/2014
`
`* cited by examiner
`
`Microsoft Corp. Exhibit 1001
`
`Microsoft Corp. Exhibit 1001
`
`

`

`U.S. Patent
`
`May 2, 2017
`
`Sheet 1 of 5
`
`US 9,641,645 132
`
`fw‘\\
`~..-‘
`N
`\
`\
`§\x
`
`: §
`
`W‘:““.
`rfizafingw
`‘\.‘m\.}
`¢_
`\
`v
`-
`-
`jg?»
`
`“m \ ~x\“
`- ~ W}
`e,»
`RR“\
`
`w}v}
`«2a2
`
`“-
`2‘
`
` i
`,.v”~.»~¢~¢nmnrm
`“mummxxfiI"(wmnwaflt
`
`.«NWM
`’\x
`"
`Sse
`§§
`\“\“~««w
`\x\w»\\\\ v“
`pmw‘mwg
`\
`§
`S
`k WWW ‘
`s WNW“ 3»
`Q~M~w.\«w.9}
`\\\\\\\\\\\\\x“
`s
`i-
`E
`»
`.
`‘>=\
`6“. a
`
`Q 3
`w
`“\m \m
`§ § WSW“ M22
`§
`\
`\
`\
`S
`3
`x
`\
`$
`Q
`S
`3%
`§
`\\
`a
`§§
`u
`x
`§
`§
`5
`x
`2
`‘
`. M V m
`
`3
`§
`§
`§
`\
`*
`g
`\
`§
`\
`‘
`~‘.
`S
`
`
`
`\
`
`
`
`\s
`
`-
`
`s
`
`§
`
`:2 3::
`
`i
`
`9-"?
`x .\
`w ‘“‘”
`
`‘
`\'fi
`
`.5
`
`\\x\\“\\~\'~\~§o
`‘
`a N ‘
`
`G ,v
`~ \
`a“ $222
`m\\-~«-~i«~»»“-»»~
`
`M
`§§x§§x
`‘>
`2‘s
`\~“~.\“«~“«“wmu<~
`
`erfll’l-e/«mn’l1arrrzv5
`
`§§
`
`§
`S
`§
`
`4.,
`j/
`
`{3:}
`
`Microsoft Corp. Exhibit 1001
`
`r“\wmmm.mm“
`“
`‘
`‘ g
`u.
`'
`‘
`'
`§*.x‘\‘v§€:{
`‘ H
`,
`.
`mmsmi k
`x «
`u
`fiéfiw
`{"3 w;
`‘m-x'
`§
`Li‘i‘gihx‘Stbfi
`:“M~Mwwm..¢
`
`. ¢~mw0-\“‘.
`\~w‘\-‘h‘\s\w‘ “
`
`‘
`
`.5.2’5,9,:
`"'mvmauw/rqur/xvmm‘l
`/,,,,,W,,,.,W/ f2;
`
`.
`’v.
`:fi.
`
`
`[/JlI’I/JVIIIIMIVIIWI/
`’NWIWM/fitwm‘
`
`V .
`34
`
`5.3if
`
`’1
`
`54”"
`
`v»
`fig
`43'
`
`2I,g‘sg‘faa»2'42.If:*' ‘‘2,a,3" s’5[9/
`
`,’5.312‘
`
`“m
`5
`ww“
`“film 1 \‘m
`fix‘rm\\ .m¢\‘N':a “1m
`, WW“ ‘2“ .
`w z»
`\.
`
`“ ~
`mm» ¥"
`
`.
`
`e:
`
`,
`
` 6,, .91
`
`Microsoft Corp. Exhibit 1001
`
`

`

`U.S. Patent
`
`May 2, 2017
`
`Sheet 2 of 5
`
`US 9,641,645 132
`
`\
`i .u-
`. k}
`¢
`\
`§xxx\\\\\\\\\\\\\\\~\\\\A\\\\\\\\§
`r mm a mi:
`\ ‘e
`‘ u
`.
`§\
`5mm g \ V
`\\\\\\\«\§m~§~w~~~m~s-~s3
`s§{3
`
`u
`3
`S
`‘0'“ N231:
`\mmxmmw \u‘..‘.\““\‘
`“mung\vammwi‘“
`E
`g
`g
`S
`S
`u. m ‘H
`‘
`.
`\
`..
`é
`3 mm Q
`ucunv; ,- \» ~'~\-r
`\‘ .‘. ..
`_..._
`§ “
`gmmmi
`*5
`R\\\\\\v\
`‘9
`\
`‘
`g m .2
`R
`i mamas
`;
`i
`
`on
`
`.0“
`
`\\
`
`.~ \\._‘
`ms:
`~ \
`~.-
`‘xxn
`is
`
`:
`i
`3
`§m~mm§o
`\
`
`3‘
`figsgi‘yg
`
`w
`‘\{ fig“ ~‘
`
`
`
`§
`c§\::---~““l‘m§
`__
`i
`
`i
`§
`
`3
`§
`“N‘Wxxsmmwx‘m‘x
`wuss?» £333“
`
`gs}?!
`
`,H H
`
`
`
`M i
`
`i
`
`i
`
`
`
`«SR?
`
`“w s“
`~‘
`
`2
`
`\
`g
`~
`k
`
`u
`v } '
`. ‘
`\m\‘
`\\
`s
`\
`s
`v.
`\
`\
`\
`\‘-~\-\\-----\\-é
`§\\\\\\\\\\“%\\\\\s\\\\\\\w\\\\~§
`Ԥ
`i
`3;
`.
`\
`s
`\
`i
`S
`E
`S
`\.\‘\ \\.‘
`\
`x
`\
`nu"
`.
`\ A
`\
`\
`.
`.
`§
`: mix. ‘2 R2.
`3
`~
`SR 3.:
`§
`§
`'g'kke
`k
`eQNn‘“£“{‘V¢
`\{‘
`imwmw ‘
`i
`\
`g Naihm g
`3 “33%.
`3
`§
`\
`k
`§
`§
`§
`i
`s
`s
`i
`\
`y
`L
`i
`:
`. “WWW mmmmnz
`\.\.W.W..w‘w.\
`’
`flu”
`Y
`~
`‘
`s \
`\\
`‘Q
`s \
`' V
`x“
`.
`.
`.
`~.\\w«~““-~m-x-~»~»\~\~~»~»\“§
`i
`it {$39 m3“.
`2 K M “
`k-«~mw«mm“-m.mmmo
`“ ws
`§
`§
`‘ 2 » “
`
`7"‘2Wm.(~ /
`
`u
`(4.
`
`Microsoft Corp. Exhibit 1001
`
`\\
`-
`\
`
`
`\n~\\\\‘\\\\\\\\\\\\\\\\\-\\---\-\
`\“\\\\\\\\\\‘\\\\\\\\\\\\\\\\\w\\vv\v‘
`\vv‘ €\\“ “
`\
`v
`‘ -
`-~.\ ‘
`'
`x
`i
`V
`\
`‘
`\
`\
`\ M
`g
`x.
`.x
`v.
`umé‘
`}
`sR ~.\
`5!
`3
`in“ W“ H " ’ 3‘“
`.\
`-\
`.-
`23“
`\‘x e\
`‘
`\¢‘\
`
`-.
`
`’
`
`\
`\ .
`‘
`§m..\“...“.... «MM
`§
`“cu-"s: E. r
`zi2.,,525
`i,
`§
`3”: Rx-xs 3-§)§
`7,,
`
`x»
`\\
`
`‘4.)
`
`.
`
`f3
`
`is
`
`
`
`§$§§n 92::
`
`Microsoft Corp. Exhibit 1001
`
`

`

`US. Patent
`
`May 2, 2017
`
`Sheet 3 of 5
`
`US 9,641,645 B2
`
`82 ,
`
`~
`
`Bmmm
`
`‘
`
`
`
`,,,,,“53.2
`......
`% WWW ‘
`Mans
`fima‘ffifis
`
`m ........................
`.........................
`
`
`‘
`
`.2
`
`as
`
`Fig, §
`
`
`3mm E’aam
`»
`£55m. mom if?
`Wigwam
`
`
`
`
`
`Nm‘ém
`33
`
`
`Smizamizfifi
`
`
` ' mm mm
`"
`{Emma R:sz
`
`
`
`
`9i}
`92
`
`
` ’ Yam "
`
`
`
`{Skiffkii $9522;
`
`'
`
`
`
`
` M33 6
`103 j
`m;
`
`f
`
`
`355% Rwfifig
`
`
` m
`
`
`
`*
`
`
`
`I mum I
`Wm Fm:
`Wfiflfi‘é?
`mam “mam
`-
`
`
`a?
`u,
`.........
`mg
`~~~~~~~~~~~~~~~~~~~
`
`
`SELECFHEGRESE
`93mm“ gififigfi
`Mimi;
`
`" Wm "
`
`Microsoft Corp. Exhibit 1001
`
`Microsoft Corp. Exhibit 1001
`
`

`

`U.S. Patent
`
`May 2, 2017
`
`Sheet 4 of 5
`
`US 9,641,645 B2
`
`3»
`
`o.<.xxx8..AX..
`1‘s...a“
`
`.imm«\m\Lmk»‘6‘»m¢\\\WVn.5““mm“““.x:m“S.Ma&flu»““x«“{in“xx}5K“..“m\‘‘.xxthxtxkxxsxsssxxxi«SR\\\\\~\\\\§M\\\§§§!§xo\x\\xx1\\\~\\\§\\\\\\\!\§\t“R\\\\\\\A\\\\\\x\\\\
`
`
`
`
`
`
` \\U.\E§ERN«“x“é.%~1.m{1Rs\Ye.\V““x\“ux.\Axuh}“xK\¢\\{\»\\x.“\~1\\k0.x3\)3.(\wx5.33n«.wi‘T?{Au‘31V:“w.m“w\.3%mx!Sn.».gfi\..\es.1V.\..xLx»~1\t).\.5Q‘«“$353?wka?:33.anxxxvi3\LM»
`xxxxxxxxxx‘w~~.x)3.....ox\..‘\.,.\~\‘ax“r\:s~\.\\.\xx“K.veit\xx«““vi3m“.6.Sn“rt3:»{x\m.R.“N“asi.8“man:31w“e».“xwR»...sm1m‘i‘x.‘,6..vé\aux:
`\xx‘xé
`~3\§§\5§3\\\§\\\m\K1\s
`
`
`“1w..x\“x‘xxx\xN\x»\“xwxx\““x“»“‘x“F}!xx\xx~sx~x§xxxx\\\\\\\\\\\\\\\\\\\\\\xxx\\1xxxsxsxxxxsx\xxx..xxsss\\\\\\\\\\\\\\\\\\\\\M.\xxx\x\x»xxxxkxxx).\\2xtssssxx§§§t\\\\\\§\\¢§\\Q\\\\\\§
`
`53
`
`‘v»
`
`$\\\\\§3\\\\\\\§§\§\\§§
`
`,mm““333§SSR3§§§.xxxxxfi‘xii‘ktx
`
`xPK.»““R(.8w>3(3»“RX“.09\.“was\Y:“Q“.e\mka{EA“exmmNW3N:x‘‘3“x‘3~\3mex}wmo\\“mv3»
`
`
`4R3£3:“w?>3We“\\WVN“MmWW\‘WcmV3.vbAis&\..\§\-\.53?\\\\\\\\.\s\\.s.x}.u»,.e.\\W933
`
`
`
`vfixsssix‘skstxS.I:v.xNxC\.\\W«o9..«o\\“a“x\xo.fm\«u.w\E$333w§§§§«““m““smwem““uh“wea.mf»r3.u.‘g..\i.e. .x»EmNSA3s\m“.n\\\N:3%.\\»\\§{\\§\{X
`
`.ekx“mawn.‘V‘,mi»c.“w}
`W...&“S”cm«X.6:»xmKNfix..»fix:XVk“.
`.Y.\.ma“no8“9“35‘
`
`
`fix».\‘T.
`
`
`
`\§v
`
`m“Xm¢\\1
`
`1;xwm“S“.{Rwmamy.
`~\«ih.n?Wm.\\\\E.
`
`{k“Sam“km1%mfixV»m3.:vias.m“\«w.\i.\\\.\.w.&.m“MV:¢\‘\\§
`
`Microsoft Corp. Exhi
`
`it 1001
`
`Microsoft Corp. Exhibit 1001
`
`
`
`
`

`

`U.S. Patent
`
`Sheet 5 of 5
`
`\
`
`sxt»ss3~sxx§xi..xxxxxxxxxxxuuu\\\\\\\\\\\\\\\\\\\\\\\xxxxxx~sxssssxxs3‘x55‘:xxxI§\§\\\\5‘\\~\\\\\\\\\\\\\\\\>{){Esgfisssstti
`
`~
`
` ‘\,x\x\~\\.\\..\aw:9mrxNn~\\\x|\R“.0;mu\\»N3.\.‘_,t\6“““S:mx\a‘““M{333$\\\~\\\\\\\\\A\\\\\\\)\\\V“«(,3.m\x“w.\~““
`
`.x\\»§\\\\\““vxx.Y.)\.\x\~‘,“mtit.~{h““Y‘{e‘R:“NA1\~7\‘iZ“é“\r“k“~“am3&3x«R8%m0Q§\§§\\Exxx.mmmm“t“‘xm«w2m.s.NwMimwmn;m9.
`
`‘3~\“{$3\wk{at33$«ha:7“a‘2‘.
`.~\\\~\.{kx.\xyx“~.\~‘~V\Mm.»“w\\\m.mx:NV3rflv\uuuuxk\uh“is\§§§§§\\mv...nNa:3R.m.x.xv.SRx:i.3.§§3§e“G“mNXi>.“RuNMW«Kn,\wWK1n\‘x.u....‘..s:‘‘>3«x;uww“WwRanr“:xx.m,‘ow“w»\«».‘fi‘3\9Qs!»«.75KAn3‘x.iI“mkM?“xi2mR:an“«x\\\u3+‘\‘e\\\\§§§§x§§§§§§
`
`am3Nmmm“mmvzxw.2asmViv.“N.“Ezszzzazzi$¢§§§=§l¢§§W§§§F
`
`
`
`t\\\\\\\\\\\\\\\§§§\s-\s\\\\\\\\\\\\\\§\\\\.w\x.m\‘‘x...\\.RI.xx\xx\~\ix\\x.t.x}.x.\~\\\wmK.~»\s\W\\\
`\&\~\\\\§z\§3:~\x\“c“,m..i“Skwm\Ruf.g~to...
`_nxu“x\.uni:“avwv.»{w:9N....\\WNWm.ma.V..~WK\~\x,v?"x!\~,x\\.\~.~J~98““D»,
`v.“3.w3.:mWu.‘v3.~A}..\..
`
`~
`
`
`5t:43.«59“c3v3x§l\«nu».+w2....quw\n».3.6NMV.3){K.\\‘\‘..:.Ev
`“Q\\\x\\\\3§\\\§\\\§§\\\\\\\\\§§\\\§§§\\§§\\»9“RxNMQ“\m\WW“QR“~..\.u:1x.\“0).“{itvME.i:mVGA?m\h:mx2‘.‘“.‘.r..S§im:wmSm3“\“a?MAEx,“v)»_u.\w.5,"v:»«6AL“Vs
`.\.\..s\1xK\xR?xtxe4uN.xg@9k\\\Vfi\&\x\\\3‘\ans:6\xxx,xx_\~\.\xi.9xxxx
`
`
`is».o}\\«t,fix)vb“\A\\.\\39\it.{tm‘
`\cxuna:\Rx“v).4.
`\i‘k‘xw3»K\‘,.......‘»\{x\.S»Ix.Ky:it“.3nMk.Gm“rV«n»auQV“.‘\.,“vEv“2:xxx“x».A\“\..»z‘x.‘‘v‘\\u§t...2x3:.\\x‘‘~.$2C.\\.*xx»Cx:3.x..:2.nito,8m:{x*we.“3x8:“x».\N.“\vas3.“.Sxmu:MyvC:ei.\‘a:m»“MMV“ws.““R);“x!“\\\.SA“Rv“.3{LW.8.rs?\\5x)?«VKc““NWNWm{.1VHS“S.“(\Lm.o(xx.\m«h.\$\»2.v.tme,x3.“3..~~~\...a\m..«x5)“\~\..~390?Mm.
`“3xx.““.kn“u»m8‘.\N69w8.»{ie?\\G.“Bufin».
`
`
`.‘‘‘~:‘>..k?.‘.kA‘u‘.T‘»mw:8\333mRuw:WESM“R.{33“9“s“v0.33$§.EExSkim?“3\txtskws‘wn»W.\
`,“\zgsztskkkw.gxkkkkxt\m.w..mWW3“
`
`
`
`v3§§§§§§x§§z§$33.35::.$§x§x¥x§22§
`..x.xmo.\\m.-\N\“«“.
`\\\.§\s\sss\\s\\\\\\\\\\\\\\\\\\{\\xx!x\\\\)"5‘:3‘s!\giis~>l§\§§§§§sss>v§\§§§x§3§<§x§§§§tiix§xui~§x
`
`Microsoft Corp. Exhi
`
`it 1001
`
`Microsoft Corp. Exhibit 1001
`
`
`
`

`

`US 9,641,645 B2
`
`1
`OPTIMIZED IMAGE DELIVERY OVER
`LIMITED BANDWIDTH COMMUNICATION
`CHANNELS
`
`PRIORITY CLAIMS/RELATED APPLICATIONS
`
`This application is a continuation of and claims priority to
`US. patent application Ser. No. 14/970,526, filed Dec. 15,
`2015, entitled OPTIMIZED IMAGE DELIVERY OVER
`LIMITED BANDWIDTH COMMUNICATION CHAN-
`
`NELS; which is a continuation of and claims priority to US.
`patent application Ser. No. 14/547,148, filed Nov. 19, 2014,
`entitled OPTIMIZED IMAGE DELIVERY OVER LIM-
`ITED BANDWIDTH COMMUNICATION CHANNELS,
`now US. Pat. No. 9,253,239; which is a continuation of and
`claims priority to US. patent application Ser. No. 13/027,
`929, filed Feb. 15, 2011, entitled OPTIMIZED IMAGE
`DELIVERY OVER LIMITED BANDWIDTH COMMUNI-
`CATION CHANNELS, now US. Pat. No. 8,924,506; which
`is a continuation-in-part of and claims priority to US. patent
`application Ser. No. 12/619,643, filed on Nov. 16, 2009,
`entitled OPTIMIZED IMAGE DELIVERY OVER LIM-
`ITED BANDWIDTH COMMUNICATION CHANNELS,
`now US. Pat. No. 7,908,343; which is a continuation of and
`claims priority to US. patent application Ser. No. 10/035,
`987, filed on Dec. 24, 2001, entitled OPTIMIZED IMAGE
`DELIVERY OVER LIMITED BANDWIDTH COMMUNI-
`CATION CHANNELS, now US. Pat. No. 7,644,131; which
`claims the benefit under 35 U.S.C. §119(e) of US. Provi-
`sional Application Nos. 60/258,488, 60/258,489, 60/258,
`465, 60/258,468, 60/258,466, and 60/258,467, all filed Dec.
`27, 2000. The disclosures of all of the foregoing patent
`documents are incorporated herein by reference as if fully
`set forth herein, including Figures, Claims, and Tables. The
`present application is also related to US. patent application
`Ser. No. 10/035,981, filed on Dec. 24, 2001, entitled SYS-
`TEM AND METHODS FOR NETWORK IMAGE DELIV-
`ERY WITH DYNAMIC VIEWING FRUSTUM OPTI-
`MIZED
`FOR
`LIMITED
`BANDWIDTH
`COMMUNICATION CHANNELS, Levanon et al., now
`US. Pat. No. 7,139,794, issued on Nov. 21, 2006, which is
`assigned to the Assignee of the present Application.
`
`FIELD
`
`The disclosure is related to network based, image distri-
`bution systems and, in particular, to a system and methods
`for efficiently selecting and distributing image parcels
`through a narrowband or otherwise limited bandwidth com-
`munications channel to support presentation of high-reso-
`lution images subject to dynamic viewing frustums.
`
`BACKGROUND
`
`The Internet and or other network systems may provide a
`unique opportunity to transmit for example complex images,
`typically large scale bit-maps, particularly those approach-
`ing photo-realistic levels, over large area and or distances. In
`common application, the images may be geographic, topo-
`graphic, and or other highly detailed maps. The data storage
`requirements and often proprietary nature of such images
`could be such that conventional interests may be to transfer
`the images on an as-needed basis.
`In conventional fixed-site applications, the image data
`may be transferred over a relatively high-bandwidth network
`to client computer systems that in turn, may render the
`image. Client systems may typically implement a local
`
`2
`
`image navigation system to provide zoom and or pan
`functions based on user interaction. As well recognized
`problem with such conventional systems could be that full
`resolution image presentation may be subject to the inherent
`transfer latency of the network. Different conventional sys-
`tems have been proposed to reduce the latency affect by
`transmitting the image in highly compressed formats that
`support progressive resolution build-up of the image within
`the current client field of view. Using a transform com-
`pressed image transfer function increases the field of the
`image that can be transferred over a fixed bandwidth net-
`work in unit time. Progressive image resolution transmis-
`sion, typically using a differential resolution method, per-
`mits an approximate image to be quickly presented with
`image details being continuously added over time.
`Tzou, in US. Pat. No. 4,698,689, describes a two-dimen-
`sional data transform system that supports transmission of
`differential coefficients to represent an image. Subsequent
`transmitted coefficient sets are progressively accumulated
`with prior transmitted sets to provide a succeedingly refined
`image. The inverse-transform function performed by the
`client computer is, however, highly compute intensive. In
`order to simplify the transform implementation and further
`reduce the latency of presenting any portion of an approxi-
`mate image, images are subdivided into a regular array. This
`enables the inverse-transform function on the client, which
`is time-critical, to deal with substantially smaller coefficient
`data sets. The array size in Tzou is fixed, which leads to
`progressively larger coefficient data sets as the detail level of
`the image increases. Consequently, there is an inherently
`increasing latency in resolving finer levels of detail.
`An image visualization system proposed by Yap et al.,
`US. Pat. No. 6,182,114, overcomes some of the foregoing
`problems. The Yap et al. system also employs a progressive
`encoding transform to compress the image transfer stream.
`The transform also operates on a subdivided image, but the
`division is indexed to the encoding level of the transform.
`The encoded transform coefficient data sets are, therefore, of
`constant size, which supports a modest improvement in the
`algorithmic performance of the inverse transform operation
`required on the client.
`Yap et al. adds utilization of client image panning or other
`image pointing input information to support a foveation-
`based operator to influence the retrieval order of the subdi-
`vided image blocks. This two-dimensional navigation infor-
`mation is used to identify a foveal region that is presumed
`to be the gaze point of a client system user. The foveation
`operator defines the corresponding image block as the center
`point of an ordered retrieval of coefficient sets representing
`a variable resolution image. The gaze point image block
`represents the area of highest image resolution, with reso-
`lution reduction as a function of distance from the gaze point
`determined by the foveation operator. This technique thus
`progressively builds image resolution at the gaze point and
`succeedingly outward based on a relatively compute inten-
`sive function. Shifts in the gaze point can be responded to
`with relative speed by preferentially retrieving coefficient
`sets at and near the new foveal region.
`Significant problems remain in permitting the convenient
`and effective use of complex images by many different types
`of client systems, even with the improvements provided by
`the various conventional systems. In particular, the imple-
`mentation of conventional image visualization systems is
`generally unworkable for smaller, often dedicated or embed-
`ded, clients where use of image visualization would clearly
`be beneficial. Conventional approaches effectively presume
`that client systems have an excess of computing perfor-
`
`5
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`Microsoft Corp. Exhibit 1001
`
`Microsoft Corp. Exhibit 1001
`
`

`

`US 9,641,645 B2
`
`3
`mance, memory and storage. Small clients, however, typi-
`cally have restricted performance processors with possibly
`no dedicated floating-point support, little general purpose
`memory, and extremely limited persistent storage capabili-
`ties, particularly relative to common image sizes. A mobile
`computing device such as mobile phone, smart phone, tablet
`and or personal digital assistant (PDA) is a characteristic
`small client. Embedded, low-cost kiosk, automobile navi-
`gation systems and or Internet enabled I connected TV are
`other typical examples. Such systems are not
`readily
`capable, if at all, of performing complex, compute-intensive
`Fourier or wavelet transforms, particularly within a highly
`restricted memory address space.
`As a consequence of the presumption that the client is a
`substantial computing system, conventional image visual-
`ization systems also presume that the client is supported by
`a complete operating system.
`Indeed, many expect and
`require an extensive set of graphics abstraction layers to be
`provided by the client system to support the presentation of
`the delivered image data. In general, these abstraction layers
`are conventionally considered required to handle the map-
`ping of the image data resolution to the display resolution
`capabilities of the client system. That is, resolution resolved
`image data provided to the client is unconstrained by any
`limitation in the client system to actually display the corre-
`sponding image. Consequently, substantial processor per-
`formance and memory can be conventionally devoted to
`handling image data that is not or cannot be displayed.
`Another problem is that small clients are generally con-
`strained to generally to very limited network bandwidths,
`particularly when operating under wireless conditions. Such
`limited bandwidth conditions may exist due to either the
`direct technological constraints dictated by the use of a low
`bandwidth data channel or indirect constraints imposed on
`relatively high-bandwidth channels by high concurrent user
`loads. Cellular connected PDAs and webphones
`are
`examples of small clients that are frequently constrained by
`limited bandwidth conditions. The conventionally realizable
`maximum network transmission bandwidth for such small
`
`devices may range from below one kilobit per second to
`several tens of kilobits per second. While Yap et al. states
`that the described system can work over low bandwidth
`lines, little more than utilizing wavelet-based data compres-
`sion is advanced as permitting effective operation at low
`communications bandwidths. While reducing the amount of
`data that must be carried from the server to the client is
`
`significant, Yap et al. simply relies on the data packet
`transfer protocols to provide for an efficient transfer of the
`compressed image data. Reliable transport protocols, how-
`ever, merely mask packet losses and the resultant, some-
`times extended recovery latencies. When such covered
`errors occur, however, the aggregate bandwidth of the con-
`nection is reduced and the client system can stall waiting for
`further image data to process.
`Consequently, there remains a need for an image visual-
`ization system that can support small client systems, place
`few requirements on the supporting client hardware and
`software resources, and efficiently utilize low to very low
`bandwidth network connections.
`
`SUMMARY
`
`Thus, a general purpose of the present invention is to
`provide an efficient system and methods of optimally pre-
`senting image data on client systems with potentially limited
`processing performance, resources, and communications
`bandwidth.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`4
`
`This is achieved in the present invention by providing for
`the retrieval of large-scale images over network communi-
`cations channels for display on a client device by selecting
`an update image parcel relative to an operator controlled
`image viewpoint to display via the client device. A request
`is prepared for the update image parcel and associated with
`a request queue for subsequent issuance over a communi-
`cations channel. The update image parcel is received from
`the communications channel and displayed as a discrete
`portion of the predetermined image. The update image
`parcel optimally has a fixed pixel array size, is received in
`a single and or plurality of network data packets, and were
`the fixed pixel array may be constrained to a resolution less
`than or equal to the resolution of the client device display.
`An advantage of the present invention is that both image
`parcel data requests and the rendering of image data are
`optimized to address the display based on the display
`resolution of the client system.
`Another advantage of the present invention is that the
`prioritization of image parcel requests is based on an adapt-
`able parameter that minimizes the computational complexity
`of determining request prioritization and, in turn, the pro-
`gressive improvement in display resolution within the field
`of view presented on a client display.
`A further advantage of the present invention is that the
`client software system requires relatively minimal client
`processing power and storage capacity. Compute intensive
`numerical calculations are minimally required and image
`parcel data is compactly stored in efficient data structures.
`The client software system is very small and easily down-
`loaded to conventional computer systems or embedded in
`conventional dedicated function devices, including portable
`devices, such as PDAs, tablets and webphones.
`Still another advantage of the present invention is that
`image parcel data requests and presentation can be readily
`optimized to use low to very low bandwidth network con-
`nections. The software system of the present
`invention
`provides for re-prioritization of image parcel data requests
`and presentation in circumstances where the rate of point-
`of-view navigation exceeds the data request rate.
`Yet another advantage of the present invention is that
`image parcel data rendering is performed without requiring
`any complex underlying hardware or software display sub-
`system. The client software system of the present invention
`includes a bit-map rendering engine that draws directly to
`the video memory of the display,
`thus placing minimal
`requirements on any underlying embedded or disk operating
`system and display drivers. Complex graphics and anima-
`tion abstraction layers are not required.
`Still another advantage of the present invention is that
`image parcel block compression is used to obtain fixed size
`transmission data blocks. Image parcel data is recoverable
`from transmission data using a relatively simple client
`decompression algorithm. Using fixed size transmission data
`blocks enables image data parcels to be delivered to the
`client in bounded time frames.
`
`A yet further advantage of the present invention is that
`multiple data forms can be transferred to the client software
`system for concurrent display. Array overlay data, correlated
`positionally to the image parcel data and generally insensi-
`tive to image parcel resolution, can be initially or progres-
`sively provided to the client for parsing and parallel pre-
`sentation on a client display image view.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`These and other advantages and features of the present
`invention will become better understood upon consideration
`
`Microsoft Corp. Exhibit 1001
`
`Microsoft Corp. Exhibit 1001
`
`

`

`US 9,641,645 B2
`
`5
`of the following detailed description of the invention when
`considered in connection with the accompanying drawings,
`in which like reference numerals designate like parts
`throughout the figures thereof, and wherein:
`FIG. 1 depicts a preferred system environment within 5
`which various embodiments of the present invention can be
`utilized;
`FIG. 2 is a block diagram illustrating the preparation of
`image parcel and overlay data set that are to be stored by and
`served from a network server system in accordance with a
`preferred embodiment of the present invention;
`FIG. 3 is a block diagram of a client system image
`presentation system constructed in accordance with a pre-
`ferred embodiment of the present invention;
`FIG. 4 provides a data block diagram illustrating an
`optimized client image block processing path constructed in
`accordance with a preferred embodiment of the present
`invention;
`FIG. 5 is a process flow diagram showing a main pro-
`cessing thread implemented in a preferred embodiment of
`the present invention;
`FIG. 6 provides a process flow diagram showing a net-
`work request thread implemented in a preferred embodiment
`of the present invention;
`FIG. 7 provides a process flow diagram showing a display
`image rendering thread implemented in a preferred embodi-
`ment of the present invention;
`FIG. 8 provides a process flow diagram showing the
`parcel map processing performed preliminary to the render-
`ing of image data parcels in accordance with a preferred
`embodiment of the present invention;
`FIG. 9 provides a process flow diagram detailing the
`rendering and progressive prioritization of image parcel data
`download requests in accordance with a preferred embodi-
`ment of the present invention; and
`FIG. 10 provides a process flow diagram detailing the
`determination of an optimal detail level for image parcel
`presentation for a current viewing frustum in accordance
`with a preferred embodiment of the present invention.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`DETAILED DESCRIPTION OF ONE OR MORE
`EMBODIMENTS
`
`45
`
`The preferred operational environment 10 of the present
`invention is generally shown in FIG. 1. A network server
`system 12, operating as a data store and server of image data,
`is responsive to requests received through a communications
`network, such as the Internet 14 generally and various tiers
`of internet service providers (ISPs) including a wireless
`connectivity provider 16. Client systems, including conven-
`tional workstations and personal computers 18 and smaller,
`typically dedicated function devices often linked through
`wireless network connections, such as PDAs, webphones
`20, and automobile navigation systems,
`source image
`requests to the network server 12, provide a client display
`and enable image navigational input by a user of the client
`system. Alternately, a dedicated function client system 20
`may be connected through a separate or plug-in local
`network server 22, preferably implementing a small, embed-
`ded Web server, to a fixed or removable storage local image
`repository 24. Characteristically, the client system 18, 20
`displays are operated at some fixed resolution generally
`dependent on the underlying display hardware of the client
`systems 18, 20.
`The image navigation capability supported by the present 65
`invention encompasses a viewing frustum placed within a
`three-dimensional space over the imaged displayed on the
`
`50
`
`55
`
`60
`
`6
`client 18, 20. Client user navigational inputs are supported
`to control the x, y lateral, rotational and z height positioning
`of the viewing frustum over the image as well as the camera
`angle of incidence relative to the plane of the image. To
`effect these controls, the software implemented on the client
`systems 18, 20 supports a three-dimensional transform of
`the image data provided from the server 12, 22.
`In accordance with the preferred embodiments of the
`present
`invention, as generally illustrated in FIG. 2, a
`network image server system 30 stores a combination of
`source image data 32 and source overlay data 34. The source
`image data 32 is typically high-resolution bit-map raster
`map and or satellite imagery of geographic regions, which
`can be obtained from commercial suppliers. The overlay
`image data 34 is typically a discrete data file providing
`image annotation information at defined coordinates relative
`to the source image data 32. In the preferred embodiments
`of the present
`invention,
`image annotations include, for
`example, street, building and landmark names, as well as
`representative 2 and 3D objects, graphical icons, decals, line
`segments, and or text and or other characters, graphics and
`or other media.
`
`The network image server system 30 preferably pre-
`processes the source image data 32 and or source overlay
`data 34 to forms preferred for storage and serving by the
`network server 12, 22. The source image data 32 is prefer-
`ably pre-processed to obtain a series K.sub.l—N of deriva-
`tive images of progressively lower image resolution. The
`source image data 32, corresponding to the series image
`K.sub.O, is also subdivided into a r