USOO9253239B2
`
`(12) United States Patent
`US 9,253,239 B2
`(10) Patent N0.:
`Levanon et a1.
`
`(45) Date of Patent: *Feb. 2, 2016
`
`(54)
`
`OPTIMIZED IMAGE DELIVERY OVER
`LIMITED BANDWIDTH COMMUNICATION
`CHANNELS
`
`(71)
`
`Applicant: BRADIUM TECHNOLOGIES LLC,
`Suffem, NY (US)
`
`(72)
`
`Inventors:
`
`Isaac Levanon, Raanana (IL); Yonatan
`Lavi, Raanana (IL)
`
`(73)
`
`Assignee: BRADIUM TECHNOLOGIES LLC,
`Suffem, NY (US)
`
`(*)
`
`Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`
`This patent is subject to a terminal dis-
`claimer.
`
`(21)
`
`Appl. N0.: 14/547,148
`
`(22)
`
`Filed:
`
`Nov. 19, 2014
`
`(65)
`
`(63)
`
`Prior Publication Data
`
`US 2015/0180928 A1
`
`Jun. 25, 2015
`
`Related US. Application Data
`
`Continuation of application No. 13/027,929, filed on
`Feb. 15, 2011, now Pat. No. 8,924,506, which is a
`continuation-in-part of application No. 12/619,643,
`filed on Nov. 16, 2009, now Pat. No. 7,908,343, which
`
`(Continued)
`
`(2006.01)
`(2006.01)
`(Continued)
`
`Int. Cl.
`
`(51)
`
`G06F 15/16
`H04L 29/06
`
`US. Cl.
`
`(52)
`
`CPC ............ H04L 65/602 (2013.01); G06F 3/1454
`(2013.01); G06T3/4092 (2013.01); H04L
`67/42 (2013.01); G09G 2340/02 (2013.01);
`G09G 2350/00 (2013.01)
`
`(58) Field of Classification Search
`CPC ..... G06T 3/4092; G09G 2340/02; G06F 3/14
`USPC ................. 709/202, 203, 217, 230, 246, 247,
`345/625; 382/232, 305
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`4,682,869 A *
`4,972,319 A
`
`7/1987 Itoh et a1.
`11/1990 Delorme
`
`................. 358/42612
`
`(Continued)
`
`FOREIGN PATENT DOCUMENTS
`
`EP
`WO
`
`1070290 B1
`99/41675
`
`1/2001
`8/1999
`
`OTHER PUBLICATIONS
`
`Potrnesil Maps Alive with dArnaud Declaration (all pages).
`
`(Continued)
`
`Primary Examiner — David Lazaro
`(74) Attorney, Agent, or Firm iAnatoly S. Weiser, Esq.;
`Techlaw LLP
`
`(57)
`
`ABSTRACT
`
`Large-scale images are retrieved over network commtmica-
`tions channels for display on a client device by selecting an
`update image parcel relative to an operator controlled image
`viewpoint to display via the client device. A request is pre-
`pared for the update image parcel and associated with a
`request queue for subsequent issuance over a commtmica-
`tions channel. The update image parcel is received from the
`communications channel and displayed as a discrete portion
`of the predetermined image. The update image parcel opti-
`mally has a fixed pixel array size, is received in a single and or
`plurality of network data packets, and were the fixed pixel
`array may be constrained to a resolution less than or equal to
`the resolution of the client device display.
`
`25 Claims, 5 Drawing Sheets
`
`
`
`3
`ism-40a: 5”“ 3’2‘
`. ”Mag
`3i {filii’fififiia
`
`\\ “meg,
`
`§¥§r~§§$§§3§§§
`Magma“ 0m
`
`Page 1 of15
`
`Unified Patents Exhibit 1001
`
`Page 1 of 15
`
`Unified Patents Exhibit 1001
`
`

`

`US 9,253,239 B2
`Page 2
`
`Related U.S. Application Data
`is a continuation of application No. 10/035,987, filed
`on Dec. 24, 2001, now Pat. No. 7,644,131.
`(60) Provisional application No. 60/258,465, filed on Dec.
`27, 2000, provisional application No. 60/258,466,
`filed on Dec. 27, 2000, provisional application No.
`60/258,467, filed on Dec. 27, 2000, provisional appli(cid:173)
`cation No. 60/258,468, filed on Dec. 27, 2000, provi(cid:173)
`sional application No. 60/258,488, filed on Dec. 27,
`2000, provisional application No. 60/258,489, filed on
`Dec. 27, 2000.
`
`(51) Int. Cl.
`G06F 3/14
`G06T 3/40
`
`(2006.01)
`(2006.01)
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`.. 345/428
`........... 345/428
`
`. .... 701/470
`
`...... 709/217
`
`5,179,638 A
`1/1993 Dawson eta!.
`5,263,136 A
`11/1993 DeAguiaJ et al.
`6/1994 Inga et al.
`5,321,520 A
`5,559,936 A * 9/1996 Poulter et al.
`5,613,051 A * 3/1997 Iodice et al. """
`5,682,441 A
`10/1997 Ligtenberg et al.
`5,760,783 A
`6/1998 Migdal
`5,798,770 A
`8/1998 Baldwin
`5,940,117 A
`8/1999 Hassan et al.
`5,987,256 A
`11/1999 Wu et al.
`5,995,903 A * 11/1999 Smith et al. .......
`6,118,456 A
`912000 Cooper
`6,167,442 A * 12/2000 Sutherland et al.
`6,182,114 Bl
`1/2001 Yap et al.
`6,212,301 Bl * 4/2001 Warner et al. ........... ..... 382/232
`6,246,797 Bl * 6/2001 Castor et al.
`.. .... 382/232
`6,285,317 Bl * 9/2001 Ong .................. ...... 342/357.57
`6,314,452 Bl* 11/2001 Dekel et al. ............. ..... 709/203
`6,326,965 Bl * 12/2001 Castelli et al. ......
`...... 345/420
`6,345,279 Bl * 2/2002 Li et al.
`6,346,938 Bl * 212002 Chan et al. .............. ..... 345/419
`6,397,259 Bl*
`5/2002 Lincke et al.
`...... 709/236
`6,449,639 Bl * 912002 Blumberg ..................... 709/217
`6,496,189 Bl * 12/2002 Yaron et al .................... 345/428
`6,525,732 Bl*
`2/2003 Gadhet al.
`...... 345/428
`6,608,628 Bl * 8/2003 Ross et al. .................... 345/619
`6,608,933 Bl * 8/2003 Dowell et al. ................. 382/232
`6,625,309 Bl * 9/2003 Li et al.
`...... 382/173
`6,650,998 Bl
`11/2003 Rutledge et al.
`6,704,024 B2 * 3/2004 Robotham et al. ............ 345/581
`6,704,791 Bl*
`3/2004 Harris
`...... 709/231
`6,711,297 Bl*
`3/2004 Chang et al.
`382/240
`6/2004 Wen et al. ...
`382/100
`6.754.365 Bl*
`6,801,665 Bl* 10/2004 Atsumi et al.
`...... 382/239
`6,882,755 B2 * 4/2005 Silverstein et al ...
`382/282
`6,898.311 B2 * 5/2005 Whitehead.
`.. 382/166
`6,970,604 Bl * 11/2005 Chai
`...... 382/240
`7,908,343 B2 * 3/2011 Levanon et al.
`709/217
`8,924,506 B2 * 12/2014 Levanon et al.
`709/217
`
`OTHER PUBLICATIONS
`Michalson Declaration Appendices P-EE (all pages).
`Michalson Declaration with Appendices A-O (all pages).
`Michael Potmesil, "Maps Alive: Viewing Geo spatial Information on
`the WWW", Computer Networks and ISDN Systems vol. 29, issues
`8-13, pp. 1327-1342.
`Peter Lindstrom, David Koller, William Ribarsky, Larry F. Hodges,
`Augusto OP den Bosch, Nickk Faust, "An Integrated Global GIS and
`
`Visual Simulation System" by P. Lindstrom et al., Tech. Rep. GIT(cid:173)
`GVU-97-07, Mar. 1997 (all pages).
`Curriculum Vitae ofWilliam R. Michalson (all pages).
`Hanan Samet, The Design and Analysis of Spatial Data Strnctures,
`University ofMaiyland (1989) (all pages).
`B. Fuller and I. Richer, The Magic Project: From Vision to Reality,
`IEEE Network May/Jun. 1996, pp. 15-25.
`International Telegraph and Telephone Consultative Committee
`("CCIII") Recommendation T.81, Sep. 1992 (all pages).
`Ken Cabeen & Peter Gent, Image Compression and the Discrete
`Cosine Transform (all pages).
`M. Antonini, Image Coding Using Wavelet Transform, IEEE Trans(cid:173)
`actions on Image Processing, vol. 1, No. 2, Apr. 1992 (all pages).
`Lance Williams, Pyramidal Parametrics, Computer Graphics, vol.
`17, No. 3, Jul. 1983, (all pages).
`OpenGL Standard Version 1.1, Mar. 1997, available: https://www.
`opengl.org/documentation/specs/versionl .1/glspec l. l/node84.
`html#SECTION00681100000000000000, (all pages).
`H. Hoppe, Progressive Meshes, SIGGRAPH '96: Proceedings of the
`23rd annual conference on computer graphics and interactive tech(cid:173)
`niques, pp. 99-108.
`Boris Rabinovich and Craig Gotsman, Visualization of Large Ter(cid:173)
`rains in Resource-Limited Computing Environments (1997) (all
`pages).
`User Datagram Protocol (UDP) (Windows CE 5.0, Microsoft, Avail(cid:173)
`able:
`https:/ /msdn.microsoft.com/enus/libraiy/ms885773 .aspx
`[Accessed Apr. 28, 2015] (all pages).
`OpenGL Standard Version 1.2.1, Apr. 1999, available: https://www.
`opengl.org/documentation/specs/versionl .2/opengll .2. l .pdf , (all
`pages).
`George H. Forman and John Zahorjan, "The challenges ofmobile
`computing," Computer vol. 27, No. 4, pp. 38, 47 (Apr. 1994).
`K. Brown and S. Singh, A Network Architecture for Mobile Com(cid:173)
`puting, INFOCOM '96, Fifteenth Annual Joint Conference of the
`IEEE Computer Societies, Networking the Next Generation, Pro(cid:173)
`ceedings IEEE vol. 3, pp. 1388-139.
`B. Krell er et al "UMTS: a middleware architecture and mobile API
`approach," Personal Communications, IEEE, vol. 5, No. 2, pp. 32-38
`(Apr. 1998).
`J. Hansen et al, "Real-time synthetic vision cockpit display for gen(cid:173)
`eral aviation," AeroSense '99, International Society for Optics and
`Photonics, 1999, (all pages).
`Theresa-Marie Rhyne, A Commentaiy on Geo VRML: A Tool for 3 D
`Representation of GeoReferenced Data on the Web, International
`Journal of Geographic Information Sciences, issue 4 of vol. 13, 1999,
`(all pages).
`Declaration of Prof. William R. Michalson (all pages).
`Provisional Applications to which the 506 Patent claims priority (all
`pages).
`Numbering of Claim Elements of Challenged Claims of the 506
`Patent, (all pages).
`Claim chart illustrating teachings of Potmesil (Ex. 1002) and
`Hornbacker (Ex. 1003) pertinent to elements of Challenged Claims
`(all pages) .
`Claim chart illustrating teachings of Rutledge (Ex. 1006), Ligtenberg
`(Ex. 1005), and Cooper (Ex. 1007) pertinent to elements of Chal(cid:173)
`lenged Claims (all pages).
`Claim chart illustrating teachings of Rutledge (Ex. 1006), Ligtenberg
`(Ex. 1005), Cooper (Ex. 1007), and Hassan (Ex. 1008) pertinent to
`elements of Challenged Claims (all pages).
`Claim chart illustrating teachings of Fuller (App. E) and Hornbacker
`(Ex. 1003) pertinent to elements of Challenged Claims (all pages).
`Claim chart illustrating teachings of Yap (App. J) and Rabinovich
`(App. R) pertinent to elements of Challenged Claims (all pages).
`
`* cited by examiner
`
`Page 2 of 15
`
`Unified Patents Exhibit 1001
`
`

`

`U.S. Patent
`
`Feb. 2, 2016
`
`Sheet 1 of5
`
`US 9,253,239 B2
`
`”fig'shm ‘QMHMV‘
`“as:
`,5)
`an!“
`«.3
`\
`\y‘
`“in;
`..._
`
`\
`
`u'
`9‘
`X “M «“3”
`¢~M “ow
`w“
`w»
`
`°1v>
`{.x‘
`
`9'6
`a"
`
`“Ne
`o M‘
`
`\x w“ «NW
`
`
`. T v
`..
`.
`:
`g
`\
`§ 5
`1%
`
`35
`3
`i
`
`§
`
`ri’mmmumufl
`
`’11
`L\s\\\\\i“
`\\u\\\~\-\\\”.4:~14”
`is:x¢&ss::\
`i m
`3 M’
`
`\m‘“\m‘\.«
`.3
`
`gm
`“x.-
`
`$‘\.N.M..._....
`
`'\¢w§§ E‘Sfi
`mififiim § :i.
`"‘
`
`Sam
`
`.
`
`mama i
`‘Axx-nm-A~Nu\~uu-\\a\s
`
`‘
`
`-\
`
`
`f,aw,wa”’wwl
`'~»»WWMz/zr%
`
`'9
`
`i,
`
`I
`mmz‘iau 23m
`
`A,
`
`’s.
`
`Q A
`,a“ ‘3 at":
`n. \ .
`\-
`\
`gmwwi“““““““~Mg
`‘ ~
`u.
`\
`§ 11mm g
`§ km g
`1
`W
`\ ..
`‘.
`E
`*i‘
`{mm mm
`W
`
`‘
`
`\..w“..w_;§\
`§§N§§§
`
`\\‘\\\\\W-\\v~\\\\\~\¢erllwlzg'
`
`/
`
`4:2
`
`Egg; E
`
`Page 3 of 15
`
`Unified Patents Exhibit 1001
`
`Page 3 of 15
`
`Unified Patents Exhibit 1001
`
`

`

`U.S. Patent
`
`Feb. 2, 2016
`
`Sheet 2 of 5
`
`US 9,253,239 B2
`
`E E
`
`.E‘ ‘1‘
`5 RES}
`.E
`«‘EEEEEEJEEEEENWE E E E“
`
`2E
`
`‘
`
`E
`.
`nx\\\n-\\-u\1vv.\\\\n\“\\\\\\‘\2
`EEEEEE EEEEEE
`\
`EEEE EEEEE Ex
`E \\\\\\\M\\M§\v\\\~mvfi\v§
`n-
`\-~\\E\~\\\s\-§\\wNEW“?
`EUEEEE SEEE
`
`““w.
`
`““‘”“W
`
`Emma?
`
`
`
`ESEE‘EZEE ESE‘EE‘S‘E
`
`{9%
`SEE E
`
`EEEEEEEEEEE
`
`ium
`\Av»\~s\v«‘\
`§
`M\Wk
`5
`
`\
`
`iu
`E
`
`vl
`\3 3
`5: ‘\
`“M“gE.“
`EXEE EEK:
`‘E‘EEEEE EEEEEEE ‘
`
`WWW
`
`ESEEEE'EE
`«svwviwnmx“mmwmw.
`u».-
`
`.i":
`‘HEEL
`“u-m\\\-IW
`anEMF: :9;
`.1"
`SE43
`E
`”5‘
`”gr
`REEEEEEE‘
`
`“ gfikmm.“ _
`
`rim”?
`§EEMNM§WMMME
` 33m ENMHUE
`
`\S
`.
`3‘
`ES
`_ A“:
`i
`R" u!"
`“3“
`i
`3\
`E3“
`
`1
`
`.
`
`E.
`
`15¢““\\\\\\\
`\\
`
`-\.-\
`.-‘“ 5Q}
`___...\\.NE‘NN““W“
`
`\‘w‘swsfi‘smssvh
`
`r/r/rzxy/””IZ””M”r”,u
`
`WWIWW’W
`
`i”
`
`W
`
`3
`
`umr/I/d/flu/r
`
`ES§§€E§3
`
`SS?€S§E
`
`3i,,5
`
`I;§iE
`
`v:
`
`
`
`\\\\\\\\\\\\\u\x~u\~w\v‘
`
`E E
`
`
`
`2:?1:1"
`
`”(1,111”I
`E‘EE“EEEE EEEEE EEE‘EEE
`“mm“m.EEEE.“ ME“mm
`3
`.
`.3
`\
`K ‘
`\\\\\\V\V‘W\‘Vv‘§\
`\Wv»
`E
`
`ENE }\£
`EEEEEE E}
`RN
`
`EE E
`
`E
`
` I.”,4”,”4”,”,nunnuux
`
`wmmw“m
`V
`
`E\
`
`a: E
`t\
`. E}
`
`Page 4 of 15
`
`Unified Patents Exhibit 1001
`
`Page 4 of 15
`
`Unified Patents Exhibit 1001
`
`

`

`U.S. Patent
`
`Feb. 2, 2016
`
`Sheet 3 015
`
`US 9,253,239 B2
`
`82
`DETERMINE I
`
`
`VIEWABLE PARCELS
`
`Fox UPuAIE
`
`
`
`NAVIGATION
`
`Evans
`
`m
`
`80
`
`FIG. 5
`
`EXAMINE
`
`QUADTREES
`
`
`
`
`.
`
`n
`
`.__‘.-
`
`
` TRIM
`OUADTREE Nooss
`a
`
`94
`
`
`
`
`
`NETWORK
`
`
`
`
`ISSUE Palonmr
`
`REQUESTS
`
`
`REQUEST PARCEL
`DOWNLOADS
`
`
`88
`
`90
`
`
`
`SIORE PARCEL To
`QUADTREE NUDE
`
`
`92
`
`
`
`REQUEST
`
`WAITING?
`
`WAIT FOR
`
`
`SELECI HIGHEST
`
`PRIORITY REQUEST
`REQUEST IHREAD
`
`
`WAITING
`FREE
`
`
`
`
`102
`
`
`
`
`FIG. 6
`
`100 j
`
`106
`
`Page 5 of 15
`
`Unified Patents Exhibit 1001
`
`Page 5 of 15
`
`Unified Patents Exhibit 1001
`
`

`

`U.S. Patent
`
`Feb. 2, 2016
`
`Sheet 4 of 5
`
`US 9,253,239 B2
`
`“‘“WM««»~M-»»»»»“»w»»-.\
`
`»§§x
`
`1»
`
`.,.xx3
`
`as.»
`
`wife. 7?”
`
`a»
`
`»»»»»-.»»»»-.»»»»»»\
`
`§§§i§
`
`\w\\\\\n\\h\§‘h“
`
`
`
`
`‘\\vi.\\\\\\\\\\\\\M\\\\i\\\\\\\\\m.\\u.u.\\\\\\\!\\\\\\\\\\\\S\\\\\\\kM>\\\\\i\‘m.\\\\\\\\s.\5\§5\\3§\\5\|\\\\\\‘“
`.33?“W»mu«m..fix"}33$.A\»»n.»xw.".»\w».1.
`.3maMNmxmm»,.
`is:mmC;““R“$13..m»Ru«8aor{A$3».mmMK»1
`_..\..M$\»MWm»W»Kn“»uui.Rx»\3mm3mmrawf»\wMRR,91“
`.«z,1».mx»x.g\i§ixx:we»2.€\
`m...
`13533.N“xm8w»:
`"vi.\Sn\tm.»\Ea“Y).L:mmw«xmv.1.»m:m\\w“ana\».»»3k»“3.i,w333mA.»an1:3v.»“1..u\11..L\V.\x.xm1‘»E.»“WkmWm»a»»N»
`
`
`»{tx\E‘x\\\\\\\\x\\\\\\\\\§\\\\».\\\\\\\\§§§¥¥V§»\\\\»\xx»\x‘\».\.\\\§\\\\\\\\\\\§\\\.\
`xxx!\§\\\3§2\\i»>‘\\\\\s~\\»\\s\\\\\\\\\\\»\~§\\\\x\\\x\\\‘3
`
`\\\V~\\\\\\\\\\\\\\\X\V¢\\\\X\\\\\\\\\
`
`
`
`v1.13 »»“ww\m..»»»x..mR.w3“.“x3.“mu“.3“n»:m.1a»93.1,:Ex.t»,mmmm.»NasM»N.§9“,...wemiN111%;.m“5“an“
`
`Fox.»mAK“Hum“.3?K)“nvwx»fin»“V1“«a“e33m...\x3»1.m“$1m“.1mm\13....\m“xi»“3»»xug«“elumNe\\».mv.1w“»u...“\315313‘5‘“
`“.3m1N»wowmS.3vmm..433.3.?m3mma:1.fiMmab.3»w$5..3“$1.
`
`
`“3.33!3“811111111
`
`.»“\a»w»o»o\0.\ugg
`
`
`,»a»;
`
`wk»..«1{\xxivngMM\i..
`
`
`1x1.Is.»M»\rkV..&53.»:mRM..05..
`
`,3...m.13o.R\i.»1.R.\\..e»..x1
`ix“”.NA«u»
`«x».13.t.»1‘“
`ea“«gm“umi»xv\\»“VN
`
`“
`
`..x.ax
`
`
`
`\gggg.s§9§998\
`
`r\1....M3..
`
`
`i(\u<\\\\\}\\\\\\\ax\\\§§QEfiv.3».
`
`A...“En...»
`
`..x.QR“.$1..
`
`1...».
`
`Page 6 of 15
`
`Unified Patents Exhi
`
`it 1001
`
`Page 6 of 15
`
`Unified Patents Exhibit 1001
`
`
`
`

`

`U.S. Patent
`
`Feb. 2, 2016
`
`Sheet 5 of 5
`
`US 9,253,239 B2
`
`fu‘uu\\\\n\m\\\nu~“vu
`
`5%),
`
`u.M‘m
`
`\3.{i{V\\
`
`W\
`
`V\\\\\\\\\M\\\\\1¢.\1\\\x\\\\\\i\
`
`
`
`0}311x11{1.11
`
`5.60.vvx
`
`~1
`1~\
`.1\1.
`1.
`xxx
`\\
`\\\\x\\~\w‘i-\mmxx\m\
`
`‘x. -§ 53
`
`.Ssiiéiiié:.131333125533“
`
`551131§1§§§1§11k§§§§
`MF:.m.zmma.wmKmmmm.131
`Vt$333.3xa.‘.iii1.11:.re11155111111111Ei1§3\$\i§§1i§§i§i1\§i§
` 'vvnx-“mx
`
`
`xm“
`
`“Lint:
`
`«v.
`
`\wi\-sv~ss~vw«~«~\s~wvn~n
`
`1111..
`
`.\
`
`‘.m\
`
`1$5
`
`{—3111}
`
`Page 7 of 15
`
`Unified Patents Exhibit 1001
`
`Page 7 of 15
`
`Unified Patents Exhibit 1001
`
`
`
`
`
`
`

`

`US 9,253,239 B2
`
`1
`OPTIMIZED IMAGE DELIVERY OVER
`LIMITED BANDWIDTH COMMUNICATION
`CHANNELS
`
`PRIORITY CLAIMS/RELATED APPLICATIONS
`
`This application is a continuation of and claims priority to
`U.S. patent application Ser. No. 13/027,929, entitled OPTI-
`
`MIZED IMAGE DELIVERY OVER LIMITED BAND-
`
`
`
`WIDTH COMMUNICATION CHANNELS, filed on 15 Feb.
`2011, now allowed; which is a continuation in part of and
`claims priority under 35 USC 120 to U.S. paten application
`Ser. No. 12/619,643 filed on Nov. 16, 2009, now U.S. Pat. No.
`7,908,343 which in turn is a continuation of and claims pri-
`ority under 35 USC 120 to U.S. patent application Ser. No.
`10/035,987 filed on Dec. 24, 2001 and entitled “Optimized
`image delivery over limited bandwidth communication chan-
`nels” (that now issued on Jan. 5, 2010 as U.S. Pat. No. 7,644,
`131) which in turn claims the benefit under 35 USC 1 19(e) of
`U.S. Provisional Application Nos. 60/258,488, 60/258,489,
`60/258,465, 60/258,468, 60/258,466, and 60/258,467, all
`filed Dec. 27, 2000, all of which are incorporated herein by
`reference.
`
`FIELD
`
`The disclosure is related to network based, image distribu-
`tion systems and, in particular, to a system and methods for
`efficiently selecting and distributing image parcels through a
`narrowband or otherwise limited bandwidth communications
`
`channel to support presentation of high-resolution images
`subject to dynamic viewing frustums.
`
`BACKGROUND
`
`The Internet and or other network systems may provide a
`unique opportunity to transmit for example complex images,
`typically large scale bit-maps, particularly those approaching
`photo-realistic levels, over large area and or distances. In
`common application, the images may be geographic, topo-
`graphic, and or other highly detailed maps. The data storage
`requirements and often proprietary nature of such images
`could be such that conventional interests may be to transfer
`the images on an as-needed basis.
`In conventional fixed-site applications, the image data may
`be transferred over a relatively high-bandwidth network to
`client computer systems that in turn, may render the image.
`Client systems may typically implement a local image navi-
`gation system to provide zoom and or pan firnctions based on
`user interaction. As well recognized problem with such con-
`ventional systems could be that full resolution image presen-
`tation may be subject to the inherent transfer latency of the
`network. Different conventional systems have been proposed
`to reduce the latency affect by transmitting the image in
`highly compressed formats that support progressive resolu-
`tion build-up of the image within the current client field of
`View. Using a transform compressed image transfer function
`increases the field of the image that can be transferred over a
`fixed bandwidth network in unit time. Progressive image
`resolution transmission, typically using a differential resolu-
`tion method, permits an approximate image to be quickly
`presented with image details being continuously added over
`time.
`
`Tzou, in U.S. Pat. No. 4,698,689, describes a two-dimen-
`sional data transform system that supports transmission of
`differential coefficients to represent an image. Subsequent
`transmitted coefficient sets are progressively accumulated
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`2
`
`
`
`with prior transmitted sets to provide a succeedingly refined
`image. The inverse-transform function perfomied by the cli-
`ent computer is, however, highly compute intensive. In order
`to simplify the transform implementation and further reduce
`the latency of presenting any portion of an approximate
`image,
`images are subdivided into a regular array. This
`enables the inverse-transform function 011 the client, whic 1 is
`time-critical, to deal with substantially smaller coefficient
`data sets. The array size in Tzou is fixed, which leads to
`progressively larger coefficient data sets as the detail leve of
`the image increases. Consequently, there is an inherently
`increasing latency in resolving finer levels of detail.
`Animage visualization system proposed byYap et a1.,L..S
`Pat. No. 6,182,114, overcomes some of the foregoing prob-
`lems. TheYap et a1. system also employs a progressive encod-
`ing transform to compress the image transfer stream. The
`transform also operates on a subdivided image, but the d'vi-
`sion is indexed to the encoding level of the transform. The
`encoded transform coefficient data sets are, therefore, of con-
`stant size, which supports a modest improvement in the algo-
`rithmic performance of the inverse transform operation
`required on the client.
`Yap et al. adds utilization of client image panning or other
`image pointing input information to support a foveation-
`based operator to influence the retrieval order of the subdi-
`vided image blocks. This two-dimensional navigation infor-
`mation is used to identify a foveal region that is presumed to
`be the gaze point of a client system user. The foveation opera-
`tor defines the corresponding image block as the center point
`of an ordered retrieval of coefficient sets representing a vari-
`able resolution image. The gaze point image block represents
`the area of highest image resolution, with resolution reduc-
`tion as a function of distance from the gaze point determined
`by the foveation operator. This technique thus progressively
`builds image resolution at the gaze point and succeedingly
`outward based 011 a relatively compute intensive function.
`Shifts in the gaze point can be responded to with relative
`speed by preferentially retrieving coefficient sets at and near
`the new foveal region.
`Significant problems remain in permitting the convenient
`and effective use of complex images by many different types
`of client systems, even with the improvements provided by
`the various conventional systems. In particular, the imple-
`mentation of conventional image visualization systems is
`generally unworkable for smaller, often dedicated or embed-
`ded, clients where use ofimage visualization would clearly be
`beneficial. Conventional approaches effectively presume that
`client systems have an excess of computing performance,
`memory and storage. Small clients, however, typically have
`restricted performance processors with possibly no dedicated
`floating-point support, little general purpose memory, and
`extremely limited persistent storage capabilities, particularly
`relative to common image sizes. A mobile computing device
`such as mobile phone, smart phone, tablet and or personal
`digital assistant
`(PDA)
`is a characteristic small client.
`Embedded, low-cost kiosk, automobile navigation systems
`and or Internet enabled I connected TV are other typical
`examples. Such systems are not readily capable, if at all, of
`performing complex, compute-intensive Fourier or wavelet
`transforms, particularly within a highly restricted memory
`address space.
`As a consequence of the presumption that the client is a
`substantial computing system, conventional image visualiza-
`tion systems also presume that the client is supported by a
`complete operating system. Indeed, many expect and require
`an extensive set of graphics abstraction layers to be provided
`by the client system to support the presentation of the deliv-
`
`Page 8 of 15
`
`Unified Patents Exhibit 1001
`
`Page 8 of 15
`
`Unified Patents Exhibit 1001
`
`

`

`US 9,253,239 B2
`
`3
`
`4
`
`ered image data. In general, these abstraction layers are con-
`ventionally considered required to handle the mapping of the
`image data resolution to the display resolution capabilities of
`the client system. That is, resolution resolved image data
`provided to the client is unconstrained by any limitation in the
`client system to actually display the corresponding image.
`Consequently,
`substantial processor performance
`and
`memory can be conventionally devoted to handling image
`data that is not or cannot be displayed.
`Another problem is that small clients are generally con-
`strained to generally to very limited network bandwidths,
`oarticularly when operating under wireless conditions. Such
`limited bandwidth conditions may exist due to either the
`direct teclmological constraints dictated by the use of a low
`Jandwidth data chamiel or indirect constraints imposed on
`‘elatively high-bandwidth channels by high concurrent user
`loads. Cellular connected PDAs and webphones are examples
`of small clients that are frequently constrained by limited
`3andwidth conditions. The conventionally realizable maxi-
`num network transmission bandwidth for such small devices
`
`nay range from below one kilobit per second to several tens
`of kilobits per second. While Yap et al. states that
`the
`described system can work over low bandwidth lines, little
`nore than utilizing wavelet-based data compression is
`advanced as permitting effective operation at low communi-
`cations bandwidths. While reducing the amount of data that
`nust be carried from the server to the client is significant, Yap
`et al. simply relies on the data packet transfer protocols to
`3rovide for an efficient transfer ofthe compressed image data.
`Reliable transport protocols, however, merely mask packet
`losses and the resultant, sometimes extended recovery laten-
`cies. When such covered errors occur, however, the aggregate
`bandwidth of the connection is reduced and the client system
`can stall waiting for further image data to process.
`Consequently, there remains a need for an image visual-
`ization system that can support small client systems, place
`few requirements on the supporting client hardware and soft-
`ware resources, and efficiently utilize low to very low band-
`width network connections.
`
`
`
`SUMMARY
`
`Thus, a general purpose of the present invention is to pro-
`vide an efficient system and methods of optimally presenting
`image data on client systems with potentially limited process-
`ing performance, resources, and communications bandwidth.
`This is achieved in the present invention by providing for
`the retrieval of large-scale images over network communica-
`tions channels for display on a client device by selecting an
`update image parcel relative to an operator controlled image
`viewpoint to display via the client device. A request is pre-
`pared for the update image parcel and associated with a
`request queue for subsequent issuance over a communica-
`tions channel. The update image parcel is received from the
`communications channel and displayed as a discrete portion
`of the predetermined image. The update image parcel opti-
`mally has a fixed pixel array size, is received in a single and or
`plurality of network data packets, and were the fixed pixel
`array may be constrained to a resolution less than or equal to
`the resolution of the client device display.
`An advantage of the present invention is that both image
`parcel data requests and the rendering of image data are
`optimized to address the display based on the display resolu-
`tion of the client system.
`Another advantage of the present invention is that the pri-
`oritization of image parcel requests is based on an adaptable
`parameter that minimizes the computational complexity of
`
`determining request prioritization and, in turn, the progres-
`sive improvement in display resolution within the field of
`view presented on a client display.
`A further advantage of the present invention is that the
`client software system requires relatively minimal client pro-
`cessing power and storage capacity. Compute intensive
`ntunerical calculations are minimally required and image
`parcel data is compactly stored in efficient data structures.
`The client software system is very small and easily down-
`loaded to conventional computer systems or embedded in
`conventional dedicated function devices, including portable
`devices, such as PDAs, tablets and webphones.
`Still another advantage of the present invention is that
`image parcel data requests and presentation can be readily
`optimized to use low to very low bandwidth network connec-
`tions. The software system of the present invention provides
`for re-prioritization of image parcel data requests and presen-
`tation in circumstances where the rate of point-of—view navi-
`gation exceeds the data request rate.
`Yet another advantage ofthe present invention is that image
`parcel data rendering is performed without requiring any
`complex underlying hardware or software display sub system.
`The client software system ofthe present invention includes a
`bit-map rendering engine that draws directly to the video
`memory ofthe display, thus placing minimal requirements on
`any underlying embedded or disk operating system and dis-
`play drivers. Complex graphics and animation abstraction
`layers are not required.
`Still another advantage of the present invention is that
`image parcel block compression is used to obtain fixed size
`transmission data blocks. Image parcel data is recoverable
`from transmission data using a relatively simple client
`decompression algorithm. Using fixed size transmission data
`blocks enables image data parcels to be delivered to the client
`in bounded time frames.
`
`A yet further advantage of the present invention is that
`multiple data forms can be transferred to the client software
`system for concurrent display. Array overlay data, correlated
`positionally to the image parcel data and generally insensitive
`to image parcel resolution, can be initially or progressively
`provided to the client for parsing and parallel presentation 011
`a client display image view.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`These and other advantages and features of the present
`invention will become better understood upon consideration
`of the following detailed description of the invention when
`considered in connection with the accompanying drawings,
`in which like reference ntunerals designate like parts through-
`out the figures thereof, and wherein:
`FIG. 1 depicts a preferred system environment within
`which various embodiments of the present invention can be
`utilized;
`FIG. 2 is a block diagram illustrating the preparation of
`image parcel and overlay data set that are to be stored by and
`served from a network server system in accordance with a
`preferred embodiment of the present invention;
`FIG. 3 is a block diagram of a client system image presen-
`tation system constructed in accordance with a preferred
`embodiment of the present invention;
`FIG. 4 provides a data block diagram illustrating an opti-
`mized client image block processing path constructed in
`accordance with a preferred embodiment of the present
`invention;
`
`v.
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`Page 9 of 15
`
`Unified Patents Exhibit 1001
`
`Page 9 of 15
`
`Unified Patents Exhibit 1001
`
`

`

`US 9,253,239 B2
`
`5
`
`6
`
`FIG. 5 is a process flow diagram showing a main process-
`ing thread implemented in a preferred embodiment of the
`present invention;
`FIG. 6 provides a process flow diagram showing a network
`request thread implemented i 1 a preferred embodiment of the
`present invention;
`FIG. 7 provides a process 2low diagram showing a display
`image rendering thread implemented in a preferred embodi-
`ment of the present inventior;
`FIG. 8 provides a process flow diagram showing the parcel
`map processing performed preliminary to the rendering of
`image data parcels in accorc ance with a preferred embodi-
`ment of the present inventior;
`FIG. 9 provides a process flow diagram detailing the ren-
`dering and progressive priO‘itization of image parcel data
`download requests in accorcance with a preferred embodi-
`ment of the present invention; and
`FIG. 10 provides a process flow diagram detailing the
`determination of an optima detail level for image parcel
`presentation for a current viewing frustum in accordance with
`a preferred embodiment of the present invention.
`
`
`
`DETAILED DESCRIPTION OF ONE OR MORE
`EMBODIMENTS
`
`The preferred operational environment 10 of the present
`invention is generally shown in FIG. 1. A network server
`system 12, operating as a data store and server of image data,
`is responsive to requests received through a communications
`network, such as the Internet 14 generally and various tiers of
`internet service providers (ISPs) including a wireless connec-
`tivity provider 16. Client systems, including conventional
`workstations and personal computers 18 and smaller, typi-
`cally dedicated function devices often linked through wire-
`less network connections, such as PDAs, webphones 20, and
`automobile navigation systems, source image requests to the
`network server 12, provide a client display and enable image
`navigational input by a user of the client system. Alternately,
`a dedicated function client system 20 may be cmmected
`through a separate or plug-in local network server 22, pref-
`erably implementing a small, embedded Web server, to a fixed
`or removable storage local image repository 24. Characteris-
`tically, the client system 18, 20 displays are operated at some
`fixed resolution generally dependent on the underlying dis-
`play hardware of the client systems 18, 20.
`The image navigation capability supported by the present
`invention encompasses a viewing frustum placed within a
`three-dimensional space over the imaged displayed on the
`client 18, 20. Client user navigational inputs are supported to
`control the x, y lateral, rotational and 2 height positioning of
`the viewing frustum over the image as well as the camera
`angle of incidence relative to the plane of the image. To effect
`these controls, the software implemented on the client sys-
`tems 18, 20 supports a three-dimensional transform of the
`image data provided from the server 12, 22.
`In accordance with the preferred embodiments of the
`present invention, as generally illustrated in FIG. 2, a network
`image server system 30 stores a combination of source image
`data 32 and source overlay data 34. The source image data 32
`is typically high-resolution bit-map raster map and or satellite
`imagery of geographic regions, which can be obtained from
`commercial suppliers. The overlay image data 34 is typically
`a discrete data file providing image annotation information at
`defined coordinates relative to the source image data 32. In
`the preferred embodiments of the present invention, image
`annotations include, for example, street, building and land-
`mark names, as well as representative 2 and 3D objects,
`
`10
`
`15
`
`20
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`graphical icons, decals, line segments, and or text and or other
`characters, graphics and or other media.
`The network image server system 30 preferably pre-pro-
`cesses the source image data 32 and or source overlay data 34
`to forms preferred for storage and serving by the network
`server 12, 22. The source image data 32 is preferably pre-
`processed to obtain a series K.sub. l -N ofderivative images of
`progressively lower image resolution. The source image data
`32, corresponding to the series image K.sub.O, is also subdi-
`vided into a regular array such that each resulting image
`parcel ofthe array has for example a 64 by 64 pixel resolution
`where the image data has a color or bi