`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`I 1111111111111111 11111 1111111111 111111111111111 11111 111111111111111 IIII IIII
`
`US008924506B2
`
`02)United States Patent
`
`
`
`Levanon et al.
`
`US 8,924,506 B2
`
`(10)Patent No.:
`(45)Date of Patent:
`*Dec. 30, 2014
`
`(52)U.S. Cl.
`(54)OPTIMIZED IMAGE DELIVERY OVER
`
`CPC ............ G06F 3/1454 (2013.01);
`LIMITED BANDWIDTH COMMUNICATION
`G06T 3/4092
`
`(2013.01); G09G 2340102 (2013.01); G09G
`CHANNELS
`
`
`(2013.01) 2350/00
`
`
`
`
`USPC ........... 709/217; 709/203; 345/625; 382/232;
`
`382/305
`
`Isaac Levanon, Raanana, IL (US);(75)Inventors:
`
`
`
`
`
`Yonatan Lavi, Roanana, IL (US)
`
`(58)Field of Classification
`Search
`
`
`
`CPC ..... G06T 3/4092; G09G 2340/02; G06F 3/14
`
`
`(73)Assignee: Bradlum Technologies LLC, Suffern,
`
`
`USPC ................. 709/202, 203,217,230,246,247;
`NY(US)
`
`
`345/625; 382/232, 305
`( * ) Notice:Subject
`to any disclaimer, the term of this
`
`
`
`
`See application file for complete search history.
`
`
`
`patent is extended or adjusted under 35
`(56)
`
`U.S.C. 154(b) by 865 days.
`
`
`claimer.
`
`This patent is subject to a tenninal dis­
`
`
`
`References Cited
`
`U.S. PATENT DOCillvIENTS
`
`5,929,860 A 7/1999 Hoppe
`
`
`
`6,182,114 Bl 1/2001 Yap et al.
`(Continued)
`OTHER PUBLICATIONS
`
`(21)Appl. No.: 13/027,929
`
`(22)Filed:
`Feb. 15, 2011
`
`(65)
`
`Prior Publication Data
`
`US 2011/0175914 Al Jul. 21, 2011
`
`Hoppe, H. "Progressive Meshes", SIGGRAPH '%: Proceedings of
`
`
`
`
`
`
`
`
`the 23rd annual conference on Computer graphics and interactive
`
`techniques, 1996, pp. 99-108.
`- David Lazaro
`Primary E.,--.:aminer
`
`
`Related U.S. Application Data
`
`
`
`
`
`
`Continuation-in-part of application No. 12/619,643,
`(63)
`
`Acuity Law Group, P.C.
`
`filed on Nov. 16, 2009, now Pat. No. 7,908,343, which
`(57)
`ABSTR-\.CT
`
`
`
`is a continuation of application No. l 0/035,987, filed
`on Dec. 24, 2001, now Pat. No. 7,644,131.
`Large-scale images are retrieved over network communica­
`
`
`
`
`
`
`
`tions channels for display on a client device by selecting an
`
`
`
`
`(60) Provisional application No. 60/258,488, filed on Dec.
`
`
`
`
`
`update image parcel relative to an operator controlled image
`
`
`27, 2000, provisional application No. 60/258,489,
`
`
`
`
`
`viewpoint to display via the client device. A request is pre­
`
`
`filed on Dec. 27, 2000, provisional application No.
`
`
`pared for the update image parcel and associated with a
`
`60/258,465, filed on Dec. 27, 2000, provisional
`
`
`
`request queue for subsequent issuance over a commm1ica­
`
`
`application No. 60/258,468, filed on Dec. 27, 2000,
`
`
`
`
`tions charmel. The update image parcel is received from the
`
`
`provisional application No. 60/258,466, filed on Dec.
`
`
`
`
`communications channel and displayed as a discrete portion
`
`
`27, 2000, provisional application No. 60/258,467,
`
`
`
`
`of the predetermined image. The update image parcel opti­
`
`filed on Dec. 27, 2000.
`
`
`
`
`mally has a fixed. pixel array size, is received in a single and or
`
`
`
`plurality of network data packets, and were the fixed pixel
`
`
`array may be constrained to a resolution less than or equal to
`
`
`
`the resolution of the client device display.
`
`(51)
`Int. Cl.
`G06F 15116
`
`G06F 3/14
`
`G06T 3/40
`
`(2006.01)
`(2006.01)
`(2006.01)
`
`Esq.;S. Weiser, (74)Attorney, Agent, or Finn -Anatoly
`
`
`
`21 Claims, 5 Drawing Sheets
`
`l))w,1lMt v"' 313
`
`0AlA
`f(lHVil:Siri•
`
`12
`
`P11!:--l'lloms1:o
`PA!KUl!AAEf PAU
`
`
`
`Page 1 of 15
`
`Unified Patents Exhibit 1002
`
`

`

`
`
`
`US 8,924,506 B2
`
`
` Page 2
`
`
`
`(56)
`
`
`References Cited
`
`
`
`
`
`U.S. PATENT DOCUMENTS
`
`
`
`
`
`6,212,301 Bl
`
`
`6,314,452 Bl
`
`
`6,326,965 Bl
`
`
`
`
`
`4/2001 Warneret al.
`
`
`
`
`11/2001 Dekeletal.
`
`
`12/2001 Castelli et al.
`
`
`
`
`
`6,345,279 Bl
`
`
`6,346,938 Bl
`
`
`6,397,259 Bl
`
`
`6,608,933 Bl
`
`
`6,704,024 B2
`
`
`6,711,297 Bl
`
`
`6,801,665 Bl
`
`
`
`
`2/2002 Li etal.
`
`
`
`2/2002 Chan etal.
`
`
`
`
`§/2002 Linckeetal.
`
`
`
`
`8/2003 Dowell et al.
`
`
`3/2004 Robotham etal.
`
`
`
`3/2004 Changetal.
`
`
`
`
`10/2004 Atsumiet al.
`
`
`
`Page 2 of 15
`
`Unified Patents Exhibit 1002
`
`
`
`Page 2 of 15
`
`Unified Patents Exhibit 1002
`
`

`

`
`U.S. Patent
`
`
`
`
`
`Dec. 30, 2014
`
`
`
`Sheet 1 of 5
`
`
`
`
`
`
`US 8,924,506 B2
`
`
`
`*, ue
`‘
`X
`x
`
`x‘gx3.
`
`we
`
`i#
`
`ot s
`
`se, x
`we~
`
`
`aaner
`
`
`
`
`
`QOS
`ed ehA,“>
`*
`2a
`
`rn
`fey}
`us
`;
`nN
`§
`ee Neen|
`Reweae
`” a
`REA
`wens
`FIAF
`8
`*
`Raves ft
`Havas §
`Baan
`~
`as
`:
`:
`x
`st
`MeN
`£O}
`
`
`
`seceseag
`oy
`ary
`§ PRR PS
`3
`g
`&
`g
`z
`3
`3
`g
`z
`x
`RY
`3
`g
`g
`s
`sans
`:
`g
`s
`Fe,
`gS
`SN
`:
`:
`S$
`AS
`3
`3
`gS
`Ny
`:
`:
`u
`x
`S
`:
`s
`‘
`SEMAN
`3
`
`,
`
`>
`
`org
`fx
`
`:
`
`x
`:
`:
`
`nN<
`Sot
`”
`
`WANA
`
`“
`
`aay eyy
`INYERNEY
`
`
`
`
`aN 3id
`
`3
`ON
`ae nnn
`
`ae
`‘
`Sis 7
`8
`Sy
`3
`
`a
`:
`
`<
`ae ~ie
`ae
`
`bene
`NAAN
`
`Se,
`
`aor ot
`:
`:
`Senge
`s
`i EER
`pechemnnen
`x x
`S gpasnes
`$
`eS
`3
`a 3
`et
`sae 8 q
`FS oa
`i
`s
`7.
`alk
`has
`sosno200e
`QRSAipeenpensiy
`°
`\$ >Nep38
`
`perenne
`Basen gy PERE
`RasySl
`Ag
`Bays
`CORVERUN
`Boeernnnnnnnnnnd
`ee
`=
`Neenwe \
`en, NS.
`NO
`\:g
`we
`a
`oe
`ose
`:g:gggxgg
`
`
`RACCCAEPUISUNEPILEIEing,
`
`
`penal
`
`a4
`~ te
`asendionmnnnnny
`oy
`oy
`y
`3
`+ NihwaRe
`|
`8
`Some
`é
`§
`RRR
`ASSyrs
`g
`SEARS AgronBe
`Denetdenenedirs
`x.
`Days SENS
`
`i
`
`
`
`at
`
`o x
`
`
`o
`eaé
`
`
`
`
`.
`
`.
`
`'
`
`3 i
`
`e
`°sg
`
`‘ f
`
`¢
`
`we
`ceaneaeenennenw
`——
`
`eergee Ne anne
`
`SS ey,
`Yo Ae at
`~ Sa en *
`ene
`.
`PISS Ss
`Ne
`
`S wsceeee
`38
`
`SOULE
`a
`Tweet ay Dare
`UVERLAY DATA
`
`x
`
`so ~ee a
`
`Sue
`
`y
`
`
`
`
`
`
`
`
`
`
`See.Paani
`
`
`PancennsdDa
`
`Page 3 of 15
`
`Unified Patents Exhibit 1002
`
`
`
`Page 3 of 15
`
`Unified Patents Exhibit 1002
`
`

`

`
`U.S. Patent
`
`
`
`Dec. 30, 2014
`
`
`
`
`
`Sheet 2 of 5
`
`
`
`
`
`US 8,924,506 B2
`
`
`
`peennnrenreennnnemnnamnnsnans
`
`reosrapossocsieds
`
`33333333
`
`7
`
`3;3$43i
`
`,
`
`wa
`
`Lad
`
`:
`3
`;
`;
`
`833
`~ Sk
`= SS
`gessenedinnnnnnnnnnang
`Ievanaoatoneperceses
`Saneyr Pars
`PSRCES BATS
`content
`weeA weoe
`cob
`
`wn
`3
`3
`
`3
`peeSaxenannnneeeeennnnnny prinanesenneng
`;i a
`set RS STS
`Lae
`‘
`
`: ARK ATS
`BEE
`;
`pd peu,
`Feocsuas
`reo :
`a
`snantoneen
`‘
`of
`fea80
`‘Peary
`Ab ocamcey fee|YEMNG Freon| cneeseanne
`
`
`oT ROET
`RomoNAATCORRES
`secre
`
`ss fageh 3
`3
`2
`FONTSOL
`epMRSS
`Aaaeaennnannnnannnnnnd
`
` f
`
`‘
`3
`
`33
`
`
`
`woey
`iN
`
`rny prrsrennnnnenonny
`Ns
`:
`:
`:
`:
`:
`:
`:
`‘
`g
`:
`:
`:
`:
`3
`3
`g
`3
`g
`3
`g
`g
`3
`New:ROE th
`g
`$
`x
`x
`errs
`TASS
`3.
`3
`:
`Maren
`i
`g
`:
`3
`PACEEY
`:
`NETWORE MARA
`
`|
`pe Pee 3
`Xnmmnnnmnnnnnennupeinnnnnnennnannninn awwmrenreemnnnrd
`RSeasnay
`s
`POAEECS RS
`:
`2
`|
`-
`. MENGRY
`SESS
`t
`|
`ow
`x
`8
`3
`gsA ~
`g
`s
`3
`SSS
`g
`8
`3
`x
`s
`x
`5
`
`:
`:
`i
`:
`x
`‘
`;
`
`goesSBereunnenrgennrnnnnnnedl Kanmngrnnnnnnd
`x
`s
`s
`Fay
`y
`oy
`z
`:
`:
`S
` Baarss
`Ne
`:
`FARES Dara
`A
`:
`x
`
`:
`TA
`:
`ge
`:
`ceeeeeeneereeneereerunrerecreate
`~.
`qecmenmmnnnefoanennnnnnn
`son
`wu
`LY Data CRONE
`rmmmnnmntl
`ry
`i~ SA
`:
`Bawnnennnnnnninnnnnnnnnnd
`33
`
`HN,
`Sn,
`NN,
`
`.
`
`"Ss.
`#83
`
`*
`
`.
`
`‘
`
`\
`\
`
`:
`sf
`>
`
`wy
`8a
`
`GO
`
`FIG,
`
`eh,
`
`Page 4 of 15
`
`Unified Patents Exhibit 1002
`
`
`
`Page 4 of 15
`
`Unified Patents Exhibit 1002
`
`

`

`U.S. Patent Dec. 30, 2014 Sheet 3 of 5
`
`
`
`US 8,924,506 B2
`
`82
`
`DETERMIHE
`NAVIGATION VIEWABLE PARCELS
`
`fVEIITT FOR UPDATE
`
`�
`80
`
`FIG. 5
`
`84
`
`86
`
`REQUEST PARCEL
`DOWNLOADS
`
`• • • • • ♦
`
`NETWORk
`
`88
`
`STORE PARCfl To
`
`QUAD TREE NODE
`
`....
`.. . . .
`
`94
`
`90
`
`92
`
`WAJT FOR
`SEtECTHIGHESl
`
`
`REOUEST lHR£AO .....,__.,. PRIORITT REQUEST
`FREE
`WAmNG
`
`102
`
`104
`
`110
`
`FIG. 6
`100_;
`
`106
`
`
`
`Page 5 of 15
`
`Unified Patents Exhibit 1002
`
`

`

`U.S. Patent Dec. 30, 2014
`Sheet 4 of 5
`
`US 8,924,506
`B2
`
`FIG,. 1
`
`Ifs:fil)fR lMAGt
`� r'.>\Rm
`ro
`!0\SFUY �l)�r
`!
`
`0H%JIIrnt
`
`•144
`r.l ''. .. .,...0.._f,n_, fj-�""l ... D_n_�l_l ..4-� .. v•
`
`I _ trnt t
`RHlOffi 0Yf£'.l.AY
`o�rn ·fo
`: � \
`Ui�L\Y Mf.�mtr
`---·"""'!:=------
`
`
`
`Page 6 of 15
`
`Unified Patents Exhibit 1002
`
`

`

`
`U.S. Patent
`
`
`
`
`
`Dec. 30, 2014
`
`
`
`
`
`Sheet 5 of 5
`
`
`
`US 8,924,506 B2
`
`
`
`SSNS
`
`
`
`
`
`
`
`
`LLLPLELECORPOLOLLOELOLELOELOTOELEEEEEEIEEEBEDIDTOPEOEOLODEOLEELEELESOLLESELESELLESCESOLESEEEELESEIEOSESOSEDELSISIISESISOSILIODIDILSASIDPLSEVEDLESEDEUSEDESELEESEDECDESEDELELEELEEPEDISIPPESEDEEDSSEDIESESISIOPEDIIPLFLEEASESEESESEDESSESELELDIDODPOIIDODEOPELOLEOEOLLLELELOELELEDEELELIASELESLIES
` aleei“eeZope"bifyte¢code¢4iteihya7fea4netiaadwobiefy$FusZ4Ynap7Zzbad4;};$BEgemnftyeyif4wt,eedrecentSigewsEoyff4$fset4&&:ff.3as45Mb,ahwdiagte4jabWee.ai¢#Je,Tytth,pomyfgOSihe1:$;weed,3treatt4%2B;tYok3tageBe;$eoy;3%;ie4$atheeIeeccecvecerecsocsecsetttsVeseccereccnsteeettcetet.2B,eep1esgi:wsae5;EEScowe,.geshe4$Cee
`
`
`thtsZTo4s4fy,ntset$Se,ada4fentsg$fpioy3BaBePirit,BEgfmBABhBegegL saiaaaaanaaaaaiiaias
`*;BYjegerai;epiG;oe$proniesngfbani;feoe|0eei4ges2gaaeg%”;4areasBe1aronoitoransonsescersedabeeorececeeceesenesentssthiiediZa}:iéi;guij:q44gZ¢iZi3g$zg;i$¢s;Vaanenvinteniprasasinterl
`
`
`
`teaes eeveeceliterlIIhave,AeneOEE4thee,Beaccesses.Yen
`:Z;AirresdcseransevcesseseeeyYat
`¢g5gii‘ibPOOPOLELTELELLELELLEEELEL
`
`SELELEASESPLESEEEEESEEESSSELSODUIPIDOVIDEDIELELIOTODEDIOLEDEELEDOOLELALMALALELPEEELEREDEASAEIEEIETobboESoOnoroooeTTE
`
`
`
`
`
`
`
`
`eetPAPAPELOEIEtoehOPececennpcrctenSeamnnpeterooonenereelecocevccscccocesensttiels¢¢¢¢$¢¢$¢¢‘¢toeBFtoev4fh,iosHs4$iee33Zasihytge$honbeettokTeaeigeOg:yon;5ioeOt:;ers34Shioi,ceedZ;gaeaE3peesz¢.5od$Leesocssereodlbegs,BEpeweweelltge,tah2GeenemrenrlpeToeOOGpecrcenelitoeseedetYeeceecetty"$BO;peTAtethereDe+EeOF¢=tyGorttGeneG.5‘wotZ&oa;tat
`hase;5wensOoo;Bpwh$éai,tstouEftwt$ecoea1%#eBtyebete4he4BS3;Oeome4oeueHGtheefeat%.t4$ifh$tegofeabcetert
`
`
`
`
`ebee
`IPELISLESSLASIISSS5“h$$4;$d;:ah,i“ee
`ager.3geroe“ge}4;’;};f/affePEfefeif}iLOOOOOLSEPPEPLLLEPSELEDLLELLICLLSTSASSPPOOLLESLELELLSS
`
`
` 4BEfy6fesatethe“hepooceccliteies,ondtagveer
`hed3.yenagayPfs:“KC26sheees,ee
`thohZoaDAY
`
`
`
`
`X“PPPOOLEOLLALOLESUEDEEEEDEEEEESSEEASELESIOLIDEDLDPOPLDIOPEDPOOLLELEODEDIEOLIPOLELLIEEDEDEDLELSLEEDEDPLEDLEDPLEOLEDPOPOLPELEPEDLELELELDIDELLEDOLELLELELELELELEDELILE
`
`AR
`
`Pree
`
`ae
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`set
`
`¢
`
`ager
`
`edapo
`
`snntersecsgsttttetsieeed££ff
`
`Ao4Bn,
`
`RE
`8h
`seoreeeen
`
`Page 7 of 15
`
`Unified Patents Exhibit 1002
`
`
`
`Page 7 of 15
`
`Unified Patents Exhibit 1002
`
`
`
`
`
`

`

`
`
`
`
`US 8,924,506 B2
`
`
`1
`OPTIMIZED IMAGE DELIVERY OVER
`
`
`
`
`LIMITED BANDWIDTH COMMUNICATION
`
`
`CHANNELS
`
`
`
`
`PRIORITY CLAIMS/RELATED APPLICATIONS
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`This application is a continuation in part of and claims
`
`
`
`
`
`
`
`
`
`
`
`priority under 35 USC 120to U.S. patent application Ser. No.
`
`
`
`
`
`
`
`
`
`
`
`12/619,643 filed on Nov. 16, 2009 now U.S. Pat. No. 7,908,
`
`
`
`
`
`
`
`
`
`
`
`343 which in tum in a continuation of and claimspriority
`
`
`
`
`
`
`
`
`
`
`
`under 35 USC 120 to U.S. patent application Ser. No. 10/035,
`
`
`
`
`
`
`
`
`
`
`987 filed on Dec. 24, 2001 and entitled “Optimized image
`
`
`
`
`
`
`delivery over limited bandwidth communication channels”
`
`
`
`
`
`
`
`
`
`
`
`
`(that now issued on Jan. 5, 2010 as U.S. Pat. No. 7,644,131)
`
`
`
`
`
`
`
`
`
`whichin turn claimsthe benefit under 35 USC 119(e) of US.
`
`
`
`
`
`Provisional Application Nos. 60/258,488, 60/258,489,
`
`
`
`
`
`
`60/258,465, 60/258,468, 60/258,466, and 60/258,467, all
`
`
`
`
`
`
`
`
`
`
`
`filed Dec. 27, 2000, all of which are incorporated herein by
`
`
`
`
`
`
`
`
`
`reference. The present application is also related to the co-
`
`
`
`
`
`
`pending application Ser. No. 10/035,981 entitled “System
`
`
`
`
`
`
`
`and Methods for Network Image Delivery with Dynamic
`
`
`
`
`
`
`Viewing Frustum Optimized for Limited Bandwidth Com-
`
`
`
`
`
`
`
`
`
`munication Channels, Levanonetal., filed on Dec. 24, 2001
`
`
`
`
`
`
`
`
`
`
`
`(now USS. Pat. No. 7,139,794 issued on Nov. 21, 2006 and
`
`
`
`
`
`
`
`
`
`
`whichis assignedto the Assignee of the present Application.
`
`
`
`
`FIELD
`
`
`
`
`
`
`
`
`
`
`
`The disclosure is related to network based, imagedistribu-
`
`
`
`
`
`
`
`
`
`
`
`tion systems and,in particular, to a system and methodsfor
`
`
`
`
`
`
`
`efficiently selecting and distributing image parcels through a
`narrowbandor otherwise limited bandwidth communications
`
`
`
`
`
`
`
`
`
`
`
`
`
`channel to support presentation of high-resolution images
`
`
`
`
`
`subject to dynamic viewing frustums.
`
`BACKGROUND
`
`
`
`
`
`
`
`
`
`
`
`
`The Internet and or other network systems mayprovide a
`
`
`
`
`
`
`
`
`unique opportunity to transmit for example complex images,
`
`
`
`
`
`
`
`typically large scale bit-maps, particularly those approaching
`
`
`
`
`
`
`
`
`
`photo-realistic levels, over large area and or distances. In
`
`
`
`
`
`
`
`common application, the images may be geographic, topo-
`
`
`
`
`
`
`
`
`
`
`graphic, and or other highly detailed maps. The data storage
`
`
`
`
`
`
`
`
`requirements and often proprietary nature of such images
`
`
`
`
`
`
`
`
`
`
`could be such that conventional interests may be to transfer
`
`
`
`
`
`
`the images on an as-neededbasis.
`
`
`
`
`
`
`
`
`In conventionalfixed-site applications, the image data may
`
`
`
`
`
`
`
`
`be transferred over a relatively high-bandwidth network to
`
`
`
`
`
`
`
`
`
`
`client computer systems that in turn, may render the image.
`
`
`
`
`
`
`
`Client systems may typically implementa local image navi-
`
`
`
`
`
`
`
`
`
`
`gation system to provide zoom andor pan functions based on
`
`
`
`
`
`
`
`
`user interaction. As well recognized problem with such con-
`
`
`
`
`
`
`
`
`ventional systems could bethat full resolution image presen-
`
`
`
`
`
`
`
`
`
`
`tation may be subject to the inherent transfer latency of the
`
`
`
`
`
`
`network. Different conventional systems have been proposed
`
`
`
`
`
`
`
`
`
`to reduce the latency affect by transmitting the image in
`
`
`
`
`
`
`highly compressed formats that support progressive resolu-
`
`
`
`
`
`
`
`
`
`
`
`tion build-up of the image within the current client field of
`
`
`
`
`
`
`
`view. Using a transform compressed imagetransfer function
`
`
`
`
`
`
`
`
`
`
`
`increasesthe field of the image that can be transferred over a
`
`
`
`
`
`
`
`
`fixed bandwidth network in unit time. Progressive image
`
`
`
`
`
`resolution transmission, typically using a differential resolu-
`
`
`
`
`
`
`
`
`tion method, permits an approximate image to be quickly
`
`
`
`
`
`
`
`presented with image details being continuously added over
`time.
`
`
`
`
`
`
`
`
`
`Tzou, in U.S. Pat. No. 4,698,689, describes a two-dimen-
`
`
`
`
`
`
`
`sional data transform system that supports transmission of
`Page 8 of 15
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`20
`
`
`
`25
`
`
`30
`
`
`
`35
`
`
`
`40
`
`
`
`45
`
`
`
`50
`
`
`
`55
`
`
`
`60
`
`
`
`65
`
`
`
`2
`
`
`
`
`
`
`
`differential coefficients to represent an image. Subsequent
`
`
`
`
`
`
`transmitted coefficient sets are progressively accumulated
`
`
`
`
`
`
`
`
`with prior transmitted sets to provide a succeedingly refined
`
`
`
`
`
`
`
`image. The inverse-transform function performedbythecli-
`
`
`
`
`
`
`
`
`
`ent computeris, however, highly compute intensive. In order
`
`
`
`
`
`
`
`
`to simplify the transform implementation and further reduce
`
`
`
`
`
`
`
`
`
`the latency of presenting any portion of an approximate
`
`
`
`
`
`
`
`
`
`image,
`images are sub-divided into a regular array. This
`
`
`
`
`
`
`
`
`
`enablesthe inverse-transform function on the client, which is
`
`
`
`
`
`
`
`time-critical, to deal with substantially smaller coefficient
`
`
`
`
`
`
`
`
`
`
`
`
`data sets. The array size in Tzou is fixed, which leads to
`
`
`
`
`
`
`
`
`
`
`progressively larger coefficient data sets as the detail level of
`
`
`
`
`
`
`
`
`the image increases. Consequently, there is an inherently
`
`
`
`
`
`
`
`
`increasing latency in resolvingfiner levels of detail.
`
`
`
`
`
`
`
`
`
`An imagevisualization system proposed byYapet al., U.S.
`
`
`
`
`
`
`
`
`Pat. No. 6,182,114, overcomes someof the foregoing prob-
`
`
`
`
`
`
`
`
`lems. The Yapet al. system also employs a progressive encod-
`
`
`
`
`
`
`
`
`ing transform to compress the image transfer stream. The
`
`
`
`
`
`
`
`
`transform also operates on a subdivided image, but the divi-
`
`
`
`
`
`
`
`
`
`
`sion is indexed to the encoding level of the transform. The
`
`
`
`
`
`
`
`
`encoded transform coefficient data sets are, therefore, of con-
`
`
`
`
`
`
`
`
`stant size, which supports a modest improvementin the algo-
`
`
`
`
`
`
`rithmic performance of the inverse transform operation
`
`
`
`
`required on the client.
`
`
`
`
`
`
`
`
`
`Yap et al. adds utilization of client image panningor other
`
`
`
`
`
`
`
`image pointing input information to support a foveation-
`
`
`
`
`
`
`
`
`
`based operator to influence the retrieval order of the subdi-
`
`
`
`
`
`
`vided image blocks. This two-dimensional navigation infor-
`
`
`
`
`
`
`
`
`
`
`
`mationis used to identify a foveal region that is presumed to
`
`
`
`
`
`
`
`
`
`
`be the gaze point of a client system user. The foveation opera-
`
`
`
`
`
`
`
`
`
`tor defines the corresponding imageblock as the center point
`
`
`
`
`
`
`
`
`of an orderedretrieval of coefficient sets representing a vari-
`
`
`
`
`
`
`
`
`able resolution image. The gaze point image block represents
`
`
`
`
`
`
`
`
`the area of highest image resolution, with resolution reduc-
`
`
`
`
`
`
`
`
`
`
`tion as a function of distance from the gaze point determined
`
`
`
`
`
`
`
`
`by the foveation operator. This technique thus progressively
`
`
`
`
`
`
`
`
`
`builds image resolution at the gaze point and succeedingly
`
`
`
`
`
`
`
`
`outward based on a relatively compute intensive function.
`
`
`
`
`
`
`
`
`
`
`
`Shifts in the gaze point can be responded to with relative
`
`
`
`
`
`
`
`
`
`speed by preferentially retrieving coefficient sets at and near
`
`
`
`
`the new foveal region.
`
`
`
`
`
`
`
`Significant problems remain in permitting the convenient
`
`
`
`
`
`
`
`
`
`
`and effective use of complex images by manydifferent types
`
`
`
`
`
`
`
`
`
`of client systems, even with the improvements provided by
`
`
`
`
`
`
`
`the various conventional systems. In particular, the imple-
`
`
`
`
`
`
`mentation of conventional image visualization systems is
`
`
`
`
`
`
`
`generally unworkable for smaller, often dedicated or embed-
`
`
`
`
`
`
`
`
`
`
`ded, clients where use ofimagevisualization would clearly be
`
`
`
`
`
`
`beneficial. Conventional approacheseffectively presume that
`
`
`
`
`
`
`
`
`client systems have an excess of computing performance,
`
`
`
`
`
`
`
`
`memory and storage. Small clients, however, typically have
`
`
`
`
`
`
`
`restricted performanceprocessors with possibly no dedicated
`
`
`
`
`
`
`
`floating-point support, little general purpose memory, and
`
`
`
`
`
`
`extremely limited persistent storage capabilities, particularly
`
`
`
`
`
`
`
`
`
`relative to common imagesizes. A mobile computing device
`
`
`
`
`
`
`
`
`
`
`such as mobile phone, smart phone, tablet and or personal
`
`
`
`
`
`
`
`
`digital assistant
`(PDA)
`is a characteristic small client.
`
`
`
`
`
`
`Embedded, low-cost kiosk, automobile navigation systems
`
`
`
`
`
`
`
`
`and or Internet enabled/connected TV are other typical
`
`
`
`
`
`
`
`
`
`
`
`examples. Such systemsare not readily capable,if at all, of
`
`
`
`
`
`
`performing complex, compute-intensive Fourier or wavelet
`
`
`
`
`
`
`
`transforms, particularly within a highly restricted memory
`
`
`address space.
`
`
`
`
`
`
`
`
`
`
`As a consequence of the presumption that the client is a
`
`
`
`
`
`substantial computing system, conventional image visualiza-
`
`
`
`
`
`
`
`
`
`
`
`tion systems also presume that the client is supported by a
`
`
`
`
`
`
`
`
`complete operating system. Indeed, many expect and require
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Unified Patents Exhibit 1002
`
`
`
`Page 8 of 15
`
`Unified Patents Exhibit 1002
`
`

`

`
`3
`
`
`
`
`
`
`
`
`
`an extensive set of graphics abstraction layers to be provided
`
`
`
`
`
`
`
`
`
`
`by the client system to support the presentation of the deliv-
`
`
`
`
`
`
`
`
`
`ered imagedata. In general, these abstraction layers are con-
`
`
`
`
`
`
`
`
`
`ventionally considered required to handle the mapping of the
`
`
`
`
`
`
`
`
`
`imagedata resolutionto the display resolution capabilities of
`
`
`
`
`
`
`
`
`
`the client system. That is, resolution resolved image data
`
`
`
`
`
`
`
`
`
`
`
`providedto the client is unconstrained by anylimitation in the
`
`
`
`
`
`
`
`
`client system to actually display the corresponding image.
`
`
`
`
`
`Consequently,
`substantial processor performance
`and
`
`
`
`
`
`
`
`
`memory can be conventionally devoted to handling image
`
`
`
`
`
`
`
`
`data that is not or cannotbe displayed.
`
`
`
`
`
`
`
`
`Another problem is that small clients are generally con-
`
`
`
`
`
`
`
`
`strained to generally to very limited network bandwidths,
`
`
`
`
`
`
`
`particularly when operating under wireless conditions. Such
`
`
`
`
`
`
`
`
`
`limited bandwidth conditions may exist due to either the
`
`
`
`
`
`
`
`
`
`direct technological constraints dictated by the use of a low
`
`
`
`
`
`
`
`
`bandwidth data channel or indirect constraints imposed on
`
`
`
`
`
`
`
`relatively high-bandwidth channels by high concurrent user
`
`
`
`
`
`
`
`
`loads. Cellular connected PDAs and webphonesare examples
`
`
`
`
`
`
`
`
`
`of small clients that are frequently constrained by limited
`
`
`
`
`
`bandwidth conditions. The conventionally realizable maxi-
`mum network transmission bandwidth for such small devices
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`may range from below one kilobit per secondto several tens
`
`
`
`
`
`
`
`
`
`
`
`the
`of kilobits per second. While Yap et al. states that
`
`
`
`
`
`
`
`
`
`described system can work over low bandwidth lines, little
`
`
`
`
`
`
`
`more than utilizing wavelet-based data compression is
`
`
`
`
`
`
`
`advancedas permitting effective operation at low communi-
`
`
`
`
`
`
`
`
`
`cations bandwidths. While reducing the amountof data that
`
`
`
`
`
`
`
`
`
`
`
`
`mustbe carried from theserverto the client is significant, Yap
`
`
`
`
`
`
`
`
`
`
`
`et al. simply relies on the data packet transfer protocols to
`
`
`
`
`
`
`
`
`
`
`provideforanefficient transfer ofthe compressed image data.
`
`
`
`
`
`
`
`Reliable transport protocols, however, merely mask packet
`
`
`
`
`
`
`
`losses and the resultant, sometimes extended, recovery laten-
`
`
`
`
`
`
`
`
`cies. When such covered errors occur, however, the aggregate
`
`
`
`
`
`
`
`
`
`bandwidth of the connection is reduced andthe client system
`
`
`
`
`
`
`
`
`
`can stall waiting for further image data to process.
`
`
`
`
`
`
`
`
`Consequently, there remains a need for an image visual-
`
`
`
`
`
`
`
`
`ization system that can support small client systems, place
`
`
`
`
`
`
`
`
`few requirements on the supporting client hardware andsoft-
`
`
`
`
`
`
`
`
`
`ware resources, andefficiently utilize low to very low band-
`width network connections.
`
`
`
`
`
`
`
`
`
`
`
`SUMMARY
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Thus, a general purpose of the present inventionis to pro-
`
`
`
`
`
`
`
`
`vide an efficient system and methodsof optimally presenting
`
`
`
`
`
`
`
`
`imagedata on client systems with potentially limited process-
`
`
`
`
`
`
`ing performance, resources, and communications bandwidth.
`
`
`
`
`
`
`
`
`
`
`This is achieved in the present invention by providing for
`
`
`
`
`
`
`the retrieval of large-scale images over network communica-
`
`
`
`
`
`
`
`
`
`tions channels for display on a client device by selecting an
`
`
`
`
`
`
`
`
`update image parcel relative to an operator controlled image
`
`
`
`
`
`
`
`
`
`
`viewpoint to display via the client device. A request is pre-
`
`
`
`
`
`
`
`
`
`pared for the update image parcel and associated with a
`
`
`
`
`
`
`
`request queue for subsequent issuance over a communica-
`
`
`
`
`
`
`
`
`
`tions channel. The update image parcel is received from the
`
`
`
`
`
`
`
`communications channel and displayed as a discrete portion
`
`
`
`
`
`
`
`
`of the predetermined image. The update image parcel opti-
`
`
`
`
`
`
`
`
`
`
`
`
`mally hasa fixed pixelarray size, is received in a single and or
`
`
`
`
`
`
`
`
`
`
`plurality of network data packets, and were the fixed pixel
`
`
`
`
`
`
`
`
`
`
`
`
`array may be constrainedto a resolution less than or equal to
`
`
`
`
`
`
`
`the resolution ofthe client device display.
`
`
`
`
`
`
`
`
`
`An advantage of the present invention is that both image
`
`
`
`
`
`
`
`
`
`parcel data requests and the rendering of image data are
`
`
`
`
`
`
`
`
`
`optimized to address the display based on the display resolu-
`
`
`
`
`
`tion of the client system.
`Page 9 of 15
`
`
`
`
`
`
`
`
`20
`
`
`
`25
`
`
`30
`
`
`
`35
`
`
`
`40
`
`
`
`45
`
`
`
`50
`
`
`
`55
`
`
`
`60
`
`
`
`65
`
`
`
`
`US 8,924,506 B2
`
`
`
`
`4
`
`
`
`
`
`
`
`
`
`Another advantage of the present invention is that the pri-
`
`
`
`
`
`
`
`
`
`
`oritization of image parcel requests is based on an adaptable
`
`
`
`
`
`
`
`parameter that minimizes the computational complexity of
`
`
`
`
`
`
`
`determining request prioritization and, in turn, the progres-
`
`
`
`
`
`
`
`
`sive improvement in display resolution within the field of
`
`
`
`
`
`view presented on a client display.
`
`
`
`
`
`
`
`
`
`A further advantage of the present invention is that the
`
`
`
`
`
`
`
`client software system requires relatively minimalclient pro-
`
`
`
`
`
`
`
`cessing power and storage capacity. Compute intensive
`
`
`
`
`
`
`
`numerical calculations are minimally required and image
`
`
`
`
`
`
`
`
`
`parcel data is compactly stored in efficient data structures.
`
`
`
`
`
`
`
`
`
`The client software system is very small and easily down-
`
`
`
`
`
`
`
`loaded to conventional computer systems or embedded in
`
`
`
`
`
`conventional dedicated function devices, including portable
`
`
`
`
`
`
`
`devices, such as PDAs, tablets and webphones.
`
`
`
`
`
`
`
`
`
`Still another advantage of the present invention is that
`
`
`
`
`
`
`
`
`
`image parcel data requests and presentation can be readily
`
`
`
`
`
`
`
`
`
`optimized to use low to very low bandwidth network connec-
`
`
`
`
`
`
`
`
`tions. The software system of the present invention provides
`
`
`
`
`
`
`
`
`for re-prioritization of image parcel data requests and presen-
`
`
`
`
`
`
`
`
`tation in circumstances wherethe rate of point-of-view navi-
`
`
`
`
`
`
`gation exceeds the data requestrate.
`
`
`
`
`
`
`
`
`
`
`Yet another advantage ofthe present invention is that image
`
`
`
`
`
`
`
`
`parcel data rendering is performed without requiring any
`
`
`
`
`
`
`
`complex underlying hardware or software display subsystem.
`
`
`
`
`
`
`
`
`
`The client software system ofthe present invention includes a
`
`
`
`
`
`
`
`
`
`bit-map rendering engine that draws directly to the video
`
`
`
`
`
`
`
`
`
`memory ofthe display, thus placing minimal requirements on
`
`
`
`
`
`
`
`
`any underlying embeddedor disk operating system and dis-
`
`
`
`
`
`
`play drivers. Complex graphics and animation abstraction
`
`
`
`
`layers are not required.
`
`
`
`
`
`
`
`
`Still another advantage of the present invention is that
`
`
`
`
`
`
`
`
`
`image parcel block compression is used to obtain fixed size
`
`
`
`
`
`
`
`transmission data blocks. Image parcel data is recoverable
`
`
`
`
`
`
`
`from transmission data using a relatively simple client
`
`
`
`
`
`
`decompression algorithm. Usingfixed size transmission data
`
`
`
`
`
`
`
`
`
`
`blocks enables image data parcelsto be delivered to the client
`in bounded time frames.
`
`
`
`
`
`
`
`
`
`
`
`
`
`A yet further advantage of the present invention is that
`
`
`
`
`
`
`
`
`
`multiple data forms can be transferred to the client software
`
`
`
`
`
`
`
`system for concurrent display. Array overlay data, correlated
`
`
`
`
`
`
`
`
`positionally to the image parcel data and generally insensitive
`
`
`
`
`
`
`
`
`to image parcel resolution, can be initially or progressively
`
`
`
`
`
`
`
`
`
`providedto the client for parsing and parallel presentation on
`
`
`
`
`
`a client display image view.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`These and other advantages and features of the present
`
`
`
`
`
`
`
`invention will become better understood upon consideration
`
`
`
`
`
`
`
`
`
`of the following detailed description of the invention when
`
`
`
`
`
`
`
`considered in connection with the accompanying drawings,
`
`
`
`
`
`
`
`
`in which like reference numerals designate like parts through-
`
`
`
`
`
`
`out the figures thereof, and wherein:
`
`
`
`
`
`
`
`FIG. 1 depicts a preferred system environment within
`
`
`
`
`
`
`
`
`which various embodiments of the present invention can be
`
`utilized;
`
`
`
`
`
`
`
`
`
`
`FIG. 2 is a block diagram illustrating the preparation of
`
`
`
`
`
`
`
`
`
`
`
`
`
`imageparcel and overlay data set that are to be stored by and
`
`
`
`
`
`
`
`
`
`
`served from a network server system in accordance with a
`
`
`
`
`
`
`preferred embodimentofthe present invention;
`
`
`
`
`
`
`
`
`
`
`FIG.3 is a block diagram of a client system image presen-
`
`
`
`
`
`
`
`tation system constructed in accordance with a preferred
`
`
`
`
`
`embodimentof the present invention;
`
`
`
`
`
`
`Unified Patents Exhibit 1002
`
`
`
`Page 9 of 15
`
`Unified Patents Exhibit 1002
`
`

`

`
`6
`
`
`
`
`
`
`
`
`
`defined coordinates relative to the source image data 32. In
`
`
`
`
`
`
`
`the preferred embodiments of the present invention, image
`
`
`
`
`
`
`
`annotations include, for example, street, building and land-
`
`
`
`
`
`
`
`
`
`
`mark names, as well as representative 2 and 3D objects,
`
`
`
`
`
`
`
`
`
`
`
`graphical icons, decals, line segments, and or text and or other
`
`
`
`
`
`
`characters, graphics and or other media.
`
`
`
`
`
`
`
`The network image server system 30 preferably pre-pro-
`
`
`
`
`
`
`
`
`
`
`
`cesses the source imagedata 32 and or source overlay data 34
`
`
`
`
`
`
`
`
`
`to forms preferred for storage and serving by the network
`
`
`
`
`
`
`
`
`
`
`server 12, 22. The source image data 32 is preferably pre-
`
`
`
`
`
`
`
`
`
`processedto obtain a series K.sub.1-N ofderivative images of
`
`
`
`
`
`
`
`
`progressively lower imageresolution. The source image data
`
`
`
`
`
`
`
`
`
`32, corresponding to the series image K.sub.0, is also subdi-
`
`
`
`
`
`
`
`
`
`
`vided into a regular array such that each resulting image
`
`
`
`
`
`
`
`
`
`
`
`
`parcel ofthe array has for example a 64 by 64 pixel resolution
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`where the image data has a coloror bit per pixel depth of 16
`
`
`
`
`
`
`
`
`
`
`
`bits, which represents a data parcel size of 8K bytes. The
`
`
`
`
`
`
`
`
`
`resolution of the series K.sub.1-N of derivative images is
`
`
`
`
`
`
`
`
`
`
`
`preferably related to that of the source image data 32 or
`
`
`
`
`
`
`
`
`
`
`
`predecessor image inthe series by a factor of four. The array
`
`
`
`
`
`
`
`
`
`
`
`subdivision is likewise related by a factor of four such that
`
`
`
`
`
`
`
`
`
`each imageparcelis of a fixed 8K byte size.
`
`
`
`
`
`
`
`
`In the preferred embodimentof the present invention, the
`
`
`
`
`
`
`
`
`
`imageparcels are further compressed and stored by the net-
`
`
`
`
`
`
`
`
`work server 12, 22. The preferred compression algorithm
`
`
`
`
`
`
`
`
`
`may implements for example a fixed 4:1 compression ratio
`DETAILED DESCRIPTION OF ONE OR MORE
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`such that each compressed and stored image parcel has a fixed
`EMBODIMENTS
`
`
`
`
`
`
`
`
`
`
`
`
`2K byte size. The imageparcels are preferably stored inafile
`
`
`
`
`
`
`
`
`
`
`of defined configuration such that any image parcel can be
`
`
`
`
`
`
`
`
`located by specification of a K.sub.D, X, Y value, represent-
`
`
`
`
`
`
`
`
`
`ing the imageset resolution index D and corresponding image
`
`
`array coordinate.
`
`
`
`
`
`
`
`the image array dimensions
`In other implementations,
`
`
`
`
`
`
`
`
`
`
`
`
`(which as 64x64 above) may be powers of two so that the
`
`
`
`
`
`
`
`
`
`
`image array can be used in texture mappingefficiently. To
`
`
`
`
`
`
`
`
`
`accommodate different data parcel size than the 2 KByte
`
`
`
`
`
`
`
`associated with 64x64 pixel parcel dimension described
`
`
`
`
`
`
`
`above and other communication protocol and overhead
`
`
`
`
`
`
`requirements, to accommodate transmission through other
`
`
`
`
`
`
`
`
`
`than a 3 KByte per second transmission channel, the present
`
`
`
`
`
`
`
`
`
`invention may use larger compression ratios that takes, for
`
`
`
`
`
`
`
`
`
`example, a 128x128 or 256x256 pixel parcel dimension and
`
`
`
`
`
`
`
`
`
`
`compresses it to meet the 3 KByte per second transmission
`
`
`
`
`
`
`
`
`channel, or other communication bandwidth used to stream
`
`
`the parcel.
`
`
`
`
`
`
`
`
`The system may also accommodate different and larger
`
`
`
`
`
`
`
`
`data parcel sizes as transmission protocols, compressionratio
`
`
`
`
`
`
`
`achieved and micro-architectures of the client computers
`
`
`
`
`
`
`
`
`
`
`change. For purposes above, the data content was a pixel array
`
`
`
`
`
`
`
`
`
`representing image data. Where the data parcel content is
`
`
`
`
`
`
`
`
`
`
`
`vector, text or other data that may subject to different client
`
`
`
`
`
`
`
`
`
`system design factors, other parcel sizes may be used. Fur-
`
`
`
`
`
`
`
`
`
`thermore,the parcel sizes can be different between the server
`
`
`
`
`
`
`
`
`
`
`and the client. For example the server may create parcels or
`
`
`
`
`
`
`
`
`hold parcels, for streaming with 256x256 pixel parcel dimen-
`
`
`
`
`
`
`
`
`
`
`
`sion and the client my render them as 64x64. In addition,
`
`
`
`
`
`
`
`
`
`
`
`parcels sizes on different servers may vary from oneserver to
`
`