Doc Code: TRACK1.REQ
`
`Document Description: TrackOne Request
`
`PTO/AlA/424 (04-14)
`
`CERTIFICATION AND REQUEST FOR PRIORITIZED EXAMINATION
`
`UNDER 37 CFR 1.102(e) (Page 1 of 1)
`
`First Named
`
`Isaac —
`OPTIMIZED IMAGE DELIVERY OVER LIMITED BANDWIDTH COMMUNICATION CHANNELS
`APPLICANT HEREBY CERTIFIES THE FOLLOWING AND REQUESTS PRIORITIZED EXAMINATION FOR
`THE ABOVE-IDENTIFIED APPLICATION.
`
`Nonprovisional Application Number (if
`
`
`
`1. The processing fee set forth in 37 CFR 1.17(i)(1) and the prioritized examination fee set forth in
`37 CFR 1.17(c) have been filed with the request. The publication fee requirement is met
`because that fee, set forth in 37 CFR 1.18(d), is currently $0. The basic filing fee, search fee,
`and examination fee are filed with the request or have been already been paid.
`I understand
`that any required excess claims fees or application size fee must be paid for the application.
`
`I understand that the application may not contain, or be amended to contain, more than four
`independent claims, more than thirty total claims, or any multiple dependent claims, and that
`any request for an extension of time will cause an outstanding Track I request to be dismissed.
`
`3. The applicable box is checked below:
`
`Ori
`
`inal A lication Track One - Prioritized Examination under ~ 1.102 e 1
`
`i.
`
`(a) The application is an original nonprovisional utility application filed under 35 U.S.C. 111(a).
`This certification and request is being filed with the utility application via EFS—Web.
`___OR___
`
`(b) The application is an original nonprovisional plant application filed under 35 U.S.C. 111(a).
`This certification and request is being filed with the plant application in paper.
`
`ii. An executed inventor’s oath or declaration under 37 CFR 1.63 or 37 CFR 1.64 for each
`
`inventor, g the application data sheet meeting the conditions specified in 37 CFR 1.53(f)(3)(i) is
`filed with the application.
`
`Request for Continued Examination - Prioritized Examination under § 1.102(e)(2)
`
`.
`
`'
`
`A request for continued examination has been filed with, or prior to, this form.
`If the application is a utility application, this certification and request is being filed via EFS—Web.
`The application is an original nonprovisional utility application filed under 35 U.S.C. 111(a), or is
`a national stage entry under 35 U.S.C. 371.
`. This certification and request is being filed prior to the mailing of a first Office action responsive
`to the request for continued examination.
`No prior request for continued examination has been granted prioritized examination status
`under 37 CFR 1.102(e)(2).
`
`Signature/Anato|y S. Weiser/
`
`Name
`(PrInt/Typed)
`
`Anatoly S. Weiser
`
`Dateo3 November 2016
`
`“minim.”
`Reglstratlon Number
`
`43229
`
`This form must be signed in accordance with 37 CFR 1.33. See 37 CFR 1.4(d) for signature requirements and certifications.
`Note:
`Submit multile forms if more than one sinature is reuired. *
`
`D *Total of
`
`forms are submitted.
`
`1 of 150
`
`Microsoft Corp. Exhibit 1016
`
`1 of 150
`
`Microsoft Corp. Exhibit 1016
`
`

`

`Privacy Act Statement
`
`The Privacy Act of 1974 (P.L. 93-579) requires that you be given certain information in connection with your
`submission ofthe attached form related to a patent application or patent. Accordingly, pursuant to the requirements of
`the Act, please be advised that: (1) the general authority forthe collection ofthis information is 35 U.S.C. 2(b)(2); (2)
`furnishing ofthe information solicited is voluntary; and (3) the principal purpose for which the information is used by the
`US. Patent and Trademark Office is to process and/or examine your submission related to a patent application or
`patent.
`If you do not furnish the requested information, the US. Patent and Trademark Office may not be able to
`process and/or examine your submission, which may result in termination of proceedings or abandonment ofthe
`application or expiration ofthe patent.
`
`The information provided by you in this form will be subject to the following routine uses:
`
`1 . The information on this form will be treated confidentially to the extent allowed underthe Freedom of
`Information Act (5 U.S.C. 552) and the Privacy Act (5 U.S.C 552a). Records from this system of records may
`be disclosed to the Department of Justice to determine whether disclosure ofthese records is required by the
`Freedom of Information Act.
`
`A record from this system of records may be disclosed, as a routine use, in the course of presenting evidence
`to a court, magistrate, or administrative tribunal, including disclosures to opposing counsel in the course of
`settlement negotiations.
`A record in this system of records may be disclosed, as a routine use, to a Member of Congress submitting a
`request involving an individual, to whom the record pertains, when the individual has requested assistance from
`the Member with respect to the subject matter ofthe record.
`A record in this system of records may be disclosed, as a routine use, to a contractor ofthe Agency having
`need forthe information in orderto perform a contract. Recipients of information shall be required to comply
`with the requirements ofthe Privacy Act of 1974, as amended, pursuant to 5 U.S.C. 552a(m).
`A record related to an International Application filed underthe Patent Cooperation Treaty in this system of
`records may be disclosed, as a routine use, to the International Bureau ofthe World Intellectual Property
`Organization, pursuant to the Patent Cooperation Treaty.
`A record in this system of records may be disclosed, as a routine use, to another federal agency for purposes
`of National Security review (35 U.S.C. 181) and for review pursuant to the Atomic Energy Act (42 U.S.C.
`218(c)).
`A record from this system of records may be disclosed, as a routine use, to the Administrator, General
`Services, or his/her designee, during an inspection of records conducted by GSA as part ofthat agency’s
`responsibility to recommend improvements in records management practices and programs, under authority of
`44 U.S.C. 2904 and 2906. Such disclosure shall be made in accordance with the GSA regulations governing
`inspection of records forthis purpose, and any other relevant (i.e., GSA or Commerce) directive. Such
`disclosure shall not be used to make determinations about individuals.
`
`A record from this system of records may be disclosed, as a routine use, to the public after either publication of
`the application pursuant to 35 U.S.C. 122(b) or issuance of a patent pursuant to 35 U.S.C. 151. Further, a
`record may be disclosed, subject to the limitations of 37 CFR 1.14, as a routine use, to the public ifthe record
`was filed in an application which became abandoned or in which the proceedings were terminated and which
`application is referenced by either a published application, an application open to public inspection or an issued
`patent.
`A record from this system of records may be disclosed, as a routine use, to a Federal, State, or local law
`enforcement agency, ifthe USPTO becomes aware of a violation or potential violation of law or regulation.
`
`2 of 150
`
`Microsoft Corp. Exhibit 1016
`
`2 of 150
`
`Microsoft Corp. Exhibit 1016
`
`

`

`-1-
`
`OPTIMIZED IMAGE DELIVERY OVER LIMITED BANDWIDTH COMMUNICATION
`
`W
`
`Isaac Levanon
`
`Yonatan Lavi
`
`Priority Claims/Related Applications
`
`This application is a continuation of and claims priority to US. Patent Application Serial
`
`No. 14/970,526, filed December 15 , 2015, entitled OPTIMIZED IMAGE DELIVERY OVER
`
`LIMITED BANDWIDTH COMMUNICATION CHANNELS; this application is also a
`
`continuation of and claims priority to US. Patent Application Serial No. 15/281,037, filed
`
`September 29, 2016, entitled OPTIMIZED IMAGE DELIVERY OVER LIMITED
`
`BANDWIDTH COMMUNICATION CHANNELS; each of the US. Patent Applications Serial
`
`No. 14/970,526 and Serial No. 15/281,037 is a continuation of and claims priority to US. Patent
`
`Application Serial No. 14/547,148, filed November 19, 2014, entitled OPTIMIZED IMAGE
`
`DELIVERY OVER LIMITED BANDWIDTH COMMUNICATION CHANNELS, now US.
`
`Patent No. 9,253,239; which is a continuation of and claims priority to US. Patent Application
`
`Serial No. 13/027,929, filed February 15, 2011, entitled OPTIMIZED IMAGE DELIVERY
`
`OVER LIMITED BANDWIDTH COMMUNICATION CHANNELS, now US. Patent No.
`
`8,924,506; which is a continuation—in—part of and claims priority to US. Patent Application
`
`Serial No. 12/619,643, filed on November 16, 2009, entitled OPTIMIZED IMAGE DELIVERY
`
`OVER LIMITED BANDWIDTH COMMUNICATION CHANNELS, now US. Patent No.
`
`7,908,343; which is a continuation of and claims priority to US. Patent Application Serial No.
`
`10/035,987, filed on December 24, 2001, entitled OPTIMIZED IMAGE DELIVERY OVER
`
`LIMITED BANDWIDTH COMMUNICATION CHANNELS, now US. Patent No. 7,644,131;
`
`which claims the benefit under 35 U.S.C. §119(e) of US. Provisional Application Nos.
`
`60/258,488, 60/258,489, 60/258,465, 60/258,468, 60/258,466, and 60/258,467, all filed
`
`December 27, 2000. The disclosures of all the foregoing patent documents are incorporated
`
`herein by reference as if fully set forth herein, including Figures, Claims, and Tables. The
`
`present application is also related to application Serial No. 10/035,981, entitled SYSTEM AND
`
`METHODS FOR NETWORK IMAGE DELIVERY WITH DYNAMIC VIEWING FRUSTUM
`
`OPTIMIZED FOR LIMITED BANDWIDTH COMMUNICATION CHANNELS, Levanon et
`
`3 of 150
`
`Microsoft Corp. Exhibit 1016
`
`3 of 150
`
`Microsoft Corp. Exhibit 1016
`
`

`

`-2-
`
`al., filed on December 24, 2001, now U.S. Patent No. 7,139,794, issued on November 21, 2006,
`
`which is assigned to the Assignee of the present Application.
`
`
`Field
`
`The disclosure is related to network based, image distribution systems and, in particular,
`
`to a system and methods for efficiently selecting and distributing image parcels through a
`
`narrowband or otherwise limited bandwidth communications channel to support presentation of
`
`high—resolution images subject to dynamic viewing frustums.
`
`B ackground
`
`The Internet and or other network systems may provide a unique opportunity to transmit
`
`for example complex images, typically large scale bit—maps, particularly those approaching
`
`photo—realistic levels, over large area and or distances. In common application, the images may
`
`be geographic, topographic, and or other highly detailed maps. The data storage requirements
`
`and often proprietary nature of such images could be such that conventional interests may be to
`
`transfer the images on an as—needed basis.
`
`In conventional fixed—site applications, the image data may be transferred over a
`
`relatively high—bandwidth network to client computer systems that in turn, may render the
`
`image. Client systems may typically implement a local image navigation system to provide
`
`zoom and or pan functions based on user interaction. As well recognized problem with such
`
`conventional systems could be that full resolution image presentation may be subject to the
`
`inherent transfer latency of the network. Different conventional systems have been proposed to
`
`reduce the latency affect by transmitting the image in highly compressed formats that support
`
`progressive resolution build—up of the image within the current client field of view. Using a
`
`transform compressed image transfer function increases the field of the image that can be
`
`transferred over a fixed bandwidth network in unit time. Progressive image resolution
`
`transmission, typically using a differential resolution method, permits an approximate image to
`
`be quickly presented with image details being continuously added over time.
`
`4 of 150
`
`Microsoft Corp. Exhibit 1016
`
`4 of 150
`
`Microsoft Corp. Exhibit 1016
`
`

`

`-3-
`
`Tzou, in U.S. Pat. No. 4,698,689, describes a two—dimensional data transform system
`
`that supports transmission of differential coefficients to represent an image. Subsequent
`
`transmitted coefficient sets are progressively accumulated with prior transmitted sets to provide a
`
`succeedingly refined image. The inverse—transform function performed by the client computer
`
`is, however, highly compute intensive. In order to simplify the transform implementation and
`
`further reduce the latency of presenting any portion of an approximate image, images are sub—
`
`divided into a regular array. This enables the inverse—transform function on the client, which is
`
`time—critical, to deal with substantially smaller coefficient data sets. The array size in Tzou is
`
`fixed, which leads to progressively larger coefficient data sets as the detail level of the image
`
`increases. Consequently, there is an inherently increasing latency in resolving finer levels of
`
`detail.
`
`An image visualization system proposed by Yap et al., U.S. Pat. No. 6,182,114,
`
`overcomes some of the foregoing problems. The Yap et al. system also employs a progressive
`
`encoding transform to compress the image transfer stream. The transform also operates on a
`
`subdivided image, but the division is indexed to the encoding level of the transform. The
`
`encoded transform coefficient data sets are, therefore, of constant size, which supports a modest
`
`improvement in the algorithmic performance of the inverse transform operation required on the
`
`client.
`
`Yap et al. adds utilization of client image panning or other image pointing input
`
`information to support a foveation—based operator to influence the retrieval order of the
`
`subdivided image blocks. This two—dimensional navigation information is used to identify a
`
`foveal region that is presumed to be the gaze point of a client system user. The foveation
`
`operator defines the corresponding image block as the center point of an ordered retrieval of
`
`coefficient sets representing a variable resolution image. The gaze point image block represents
`
`the area of highest image resolution, with resolution reduction as a function of distance from the
`
`gaze point determined by the foveation operator. This technique thus progressively builds image
`
`resolution at the gaze point and succeedingly outward based on a relatively compute intensive
`
`function. Shifts in the gaze point can be responded to with relative speed by preferentially
`
`retrieving coefficient sets at and near the new foveal region.
`
`5 of 150
`
`Microsoft Corp. Exhibit 1016
`
`5 of 150
`
`Microsoft Corp. Exhibit 1016
`
`

`

`-4-
`
`Significant problems remain in permitting the convenient and effective use of complex
`
`images by many different types of client systems, even with the improvements provided by the
`
`various conventional systems. In particular, the implementation of conventional image
`
`visualization systems is generally unworkable for smaller, often dedicated or embedded, clients
`
`where use of image visualization would clearly be beneficial. Conventional approaches
`
`effectively presume that client systems have an excess of computing performance, memory and
`
`storage. Small clients, however, typically have restricted performance processors with possibly
`
`no dedicated floating—point support, little general purpose memory, and extremely limited
`
`persistent storage capabilities, particularly relative to common image sizes. A mobile computing
`
`device such as mobile phone, smart phone, tablet and or personal digital assistant (PDA) is a
`
`characteristic small client. Embedded, low—cost kiosk, automobile navigation systems and or
`
`Internet enabled I connected TV are other typical examples. Such systems are not readily
`
`capable, if at all, of performing complex, compute—intensive Fourier or wavelet transforms,
`
`particularly within a highly restricted memory address space.
`
`As a consequence of the presumption that the client is a substantial computing system,
`
`conventional image visualization systems also presume that the client is supported by a complete
`
`operating system. Indeed, many expect and require an extensive set of graphics abstraction
`
`layers to be provided by the client system to support the presentation of the delivered image data.
`
`In_general, these abstraction layers are conventionally considered required to handle the mapping
`
`of the image data resolution to the display resolution capabilities of the client system. That is,
`
`resolution resolved image data provided to the client is unconstrained by any limitation in the
`
`client system to actually display the corresponding image. Consequently, substantial processor
`
`performance and memory can be conventionally devoted to handling image data that is not or
`
`cannot be displayed.
`
`Another problem is that small clients are generally constrained to generally to very
`
`limited network bandwidths, particularly when operating under wireless conditions. Such
`
`limited bandwidth conditions may exist due to either the direct technological constraints dictated
`
`by the use of a low bandwidth data channel or indirect constraints imposed on relatively high—
`
`bandwidth channels by high concurrent user loads. Cellular connected PDAs and webphones are
`
`examples of small clients that are frequently constrained by limited bandwidth conditions. The
`
`6 of 150
`
`Microsoft Corp. Exhibit 1016
`
`6 of 150
`
`Microsoft Corp. Exhibit 1016
`
`

`

`-5-
`
`conventionally realizable maximum network transmission bandwidth for such small devices may
`
`range from below one kilobit per second to several tens of kilobits per second. While Yap et al.
`
`states that the described system can work over low bandwidth lines, little more than utilizing
`
`wavelet—based data compression is advanced as permitting effective operation at low
`
`communications bandwidths. While reducing the amount of data that must be carried from the
`
`server to the client is significant, Yap et al. simply relies on the data packet transfer protocols to
`
`provide for an efficient transfer of the compressed image data. Reliable transport protocols,
`
`however, merely mask packet losses and the resultant, sometimes extended recovery latencies.
`
`When such covered errors occur, however, the aggregate bandwidth of the connection is reduced
`
`and the client system can stall waiting for further image data to process.
`
`Consequently, there remains a need for an image visualization system that can support
`
`small client systems, place few requirements on the supporting client hardware and software
`
`resources, and efficiently utilize low to very low bandwidth network connections.
`
`Summary
`
`Thus, a general purpose of the present invention is to provide an efficient system and
`
`methods of optimally presenting image data on client systems with potentially limited processing
`
`performance, resources, and communications bandwidth.
`
`This is achieved in the present invention by providing for the retrieval of large—scale
`
`images over network communications channels for display on a client device by selecting an
`
`update image parcel relative to an operator controlled image viewpoint to display via the client
`
`device. A request is prepared for the update image parcel and associated with a request queue
`
`for subsequent issuance over a communications channel. The update image parcel is received
`
`from the communications channel and displayed as a discrete portion of the predetermined
`
`image. The update image parcel optimally has a fixed pixel array size, is received in a single and
`
`or plurality of network data packets, and were the fixed pixel array may be constrained to a
`
`resolution less than or equal to the resolution of the client device display.
`
`An advantage of the present invention is that both image parcel data requests and the
`
`rendering of image data are optimized to address the display based on the display resolution of
`
`the client system.
`
`7 of 150
`
`Microsoft Corp. Exhibit 1016
`
`7 of 150
`
`Microsoft Corp. Exhibit 1016
`
`

`

`-6-
`
`Another advantage of the present invention is that the prioritization of image parcel
`
`requests is based on an adaptable parameter that minimizes the computational complexity of
`
`determining request prioritization and, in turn, the progressive improvement in display resolution
`
`within the field of view presented on a client display.
`
`A further advantage of the present invention is that the client software system requires
`
`relatively minimal client processing power and storage capacity. Compute intensive numerical
`
`calculations are minimally required and image parcel data is compactly stored in efficient data
`
`structures. The client software system is very small and easily downloaded to conventional
`
`computer systems or embedded in conventional dedicated function devices, including portable
`
`devices, such as PDAs, tablets and webphones.
`
`Still another advantage of the present invention is that image parcel data requests and
`
`presentation can be readily optimized to use low to very low bandwidth network connections.
`
`The software system of the present invention provides for re—prioritization of image parcel data
`
`requests and presentation in circumstances where the rate of point—of—view navigation exceeds
`
`the data request rate.
`
`Yet another advantage of the present invention is that image parcel data rendering is
`
`performed without requiring any complex underlying hardware or software display subsystem.
`
`The client software system of the present invention includes a bit—map rendering engine that
`
`draws directly to the video memory of the display, thus placing minimal requirements on any
`
`underlying embedded or disk operating system and display drivers. Complex graphics and
`
`animation abstraction layers are not required.
`
`Still another advantage of the present invention is that image parcel block compression is
`
`used to obtain fixed size transmission data blocks. Image parcel data is recoverable from
`
`transmission data using a relatively simple client decompression algorithm. Using fixed size
`
`transmission data blocks enables image data parcels to be delivered to the client in bounded time
`
`frames.
`
`A yet further advantage of the present invention is that multiple data forms can be
`
`transferred to the client software system for concurrent display. Array overlay data, correlated
`
`positionally to the image parcel data and generally insensitive to image parcel resolution, can be
`
`8 of 150
`
`Microsoft Corp. Exhibit 1016
`
`8 of 150
`
`Microsoft Corp. Exhibit 1016
`
`

`

`-7-
`
`initially or progressively provided to the client for parsing and parallel presentation on a client
`
`display image view.
`
`Brief Description of the Drawings
`
`These and other advantages and features of the present invention will become better
`
`understood upon consideration of the following detailed description of the invention when
`
`considered in connection with the accompanying drawings, in which like reference numerals
`
`designate like parts throughout the figures thereof, and wherein:
`
`FIG. 1 depicts a preferred system environment within which various embodiments of the
`
`present invention can be utilized;
`
`FIG. 2 is a block diagram illustrating the preparation of image parcel and overlay data set
`
`that are to be stored by and served from a network server system in accordance with a preferred
`
`embodiment of the present invention;
`
`FIG. 3 is a block diagram of a client system image presentation system constructed in
`
`accordance with a preferred embodiment of the present invention;
`
`FIG. 4 provides a data block diagram illustrating an optimized client image block
`
`processing path constructed in accordance with a preferred embodiment of the present invention;
`
`FIG. 5 is a process flow diagram showing a main processing thread implemented in a
`
`preferred embodiment of the present invention;
`
`FIG. 6 provides a process flow diagram showing a network request thread implemented
`
`in a preferred embodiment of the present invention;
`
`FIG. 7 provides a process flow diagram showing a display image rendering thread
`
`implemented in a preferred embodiment of the present invention;
`
`FIG. 8 provides a process flow diagram showing the parcel map processing performed
`
`preliminary to the rendering of image data parcels in accordance with a preferred embodiment of
`
`the present invention;
`
`9 of 150
`
`Microsoft Corp. Exhibit 1016
`
`9 of 150
`
`Microsoft Corp. Exhibit 1016
`
`

`

`-8-
`
`FIG. 9 provides a process flow diagram detailing the rendering and progressive
`
`prioritization of image parcel data download requests in accordance with a preferred embodiment
`
`of the present invention; and
`
`FIG. 10 provides a process flow diagram detailing the determination of an optimal detail
`
`level for image parcel presentation for a current viewing frustum in accordance with a preferred
`
`embodiment of the present invention.
`
`Detailed Description of One or More Embodiments
`
`The preferred operational environment 10 of the present invention is generally shown in
`
`FIG. 1. A network server system 12, operating as a data store and server of image data, is
`
`responsive to requests received through a communications network, such as the Internet 14
`
`generally and various tiers of internet service providers (ISPs) including a wireless connectivity
`
`provider 16. Client systems, including conventional workstations and personal computers 18 and
`
`smaller, typically dedicated function devices often linked through wireless network connections,
`
`such as PDAs, webphones 20, and automobile navigation systems, source image requests to the
`
`network server 12, provide a client display and enable image navigational input by a user of the
`
`client system. Alternately, a dedicated function client system 20 may be connected through a
`
`separate or plug—in local network server 22, preferably implementing a small, embedded Web
`
`server, to a fixed or removable storage local image repository 24. Characteristically, the client
`
`system 18, 20 displays are operated at some fixed resolution generally dependent on the
`
`underlying display hardware of the client systems 18, 20.
`
`The image navigation capability supported by the present invention encompasses a
`
`viewing frustum placed within a three—dimensional space over the imaged displayed on the client
`
`18, 20. Client user navigational inputs are supported to control the X, y lateral, rotational and 2
`
`height positioning of the viewing frustum over the image as well as the camera angle of
`
`incidence relative to the plane of the image. To effect these controls, the software implemented
`
`on the client systems 18, 20 supports a three—dimensional transform of the image data provided
`
`from the server 12, 22.
`
`In accordance with the preferred embodiments of the present invention, as generally
`
`illustrated in FIG. 2, a network image server system 30 stores a combination of source image
`
`10 of 150
`
`Microsoft Corp. Exhibit 1016
`
`10 of 150
`
`Microsoft Corp. Exhibit 1016
`
`

`

`-9-
`
`data 32 and source overlay data 34. The source image data 32 is typically high—resolution bit—
`
`map raster map and or satellite imagery of geographic regions, which can be obtained from
`
`commercial suppliers. The overlay image data 34 is typically a discrete data file providing
`
`image annotation information at defined coordinates relative to the source image data 32. In the
`
`preferred embodiments of the present invention, image annotations include, for example, street,
`
`building and landmark names, as well as representative 2 and 3D objects, graphical icons, decals,
`
`line segments, and or text and or other characters, graphics and or other media.
`
`The network image server system 30 preferably pre—processes the source image data 32
`
`and or source overlay data 34 to forms preferred for storage and serving by the network server
`
`12, 22. The source image data 32 is preferably pre—processed to obtain a series K1-N of
`
`derivative images of progressively lower image resolution. The source image data 32,
`
`corresponding to the series image K0, is also subdivided into a regular array such that each
`
`resulting image parcel of the array has for example a 64 by 64 pixel resolution where the image
`
`data has a color or bit per pixel depth of 16 bits, which represents a data parcel size of 8K bytes.
`
`The resolution of the series K1-N of derivative images is preferably related to that of the source
`
`image data 32 or predecessor image in the series by a factor of four. The array subdivision is
`
`likewise related by a factor of four such that each image parcel is of a fixed 8K byte size.
`
`In the preferred embodiment of the present invention, the image parcels are further
`
`compressed and stored by the network server 12, 22. The preferred compression algorithm may
`
`implement for example a fixed 4:l compression ratio such that each compressed and stored
`
`image parcel has a fixed 2K byte size. The image parcels are preferably stored in a file of
`
`defined configuration such that any image parcel can be located by specification of a K13, X, Y
`
`value, representing the image set resolution index D and corresponding image array coordinate.
`
`In other implementations, the image array dimensions (which as 64 X 64 above) may be
`
`powers of two so that the image array can be used in texture mapping efficiently. To
`
`accommodate different data parcel size than the 2KByte associated with 64x64 pixel parcel
`
`dimension described above and other communication protocol and overhead requirements, to
`
`accommodate transmission through other than a 3KByte per second transmission channel, the
`
`present invention may use larger compression ratios that takes, for example, a 128x128 or
`
`11 of 150
`
`Microsoft Corp. Exhibit 1016
`
`11 of 150
`
`Microsoft Corp. Exhibit 1016
`
`

`

`-10-
`
`256x256 pixel parcel dimension and compresses it to meet the 3KByte per second transmission
`
`channel, or other communication bandwidth used to stream the parcel.
`
`The system may also accommodate different and larger data parcel sizes as transmission
`
`protocols, compression ratio achieved and micro—architectures of the client computers change.
`
`For purposes above, the data content was a pixel array representing image data. Where the data
`
`parcel content is vector, text or other data that may subject to different client system design
`
`factors, other parcel sizes may be used. Furthermore, the parcel sizes can be different between
`
`the server and the client. For example the server may create parcels or hold parcels, for
`
`streaming with 256x256 pixel parcel dimension and the client my render them as 64x64. In
`
`addition, parcels sizes on different servers may vary from one server to another and from the
`
`client side rendering. In the system, each grid is treated as a sparse data array that can be
`
`progressively revised to increase the resolution of the grid and thereby the level of detail
`
`presented by the grid.
`
`The source overlay data 34 is preferably pre—processed 36 into either an open XML
`
`format, such as the Geography Markup Language (GML), which is an XML based encoding
`
`standard for geographic information developed by the OpenGIS Consortium (OGC;
`
`www.opengis.org), or a proprietary binary representation. The XML/GML representation is
`
`preferred as permitting easier interchange between different commercial entities, while the
`
`binary representation is preferred as more compact and readily transferable to a client system 18,
`
`20. In both cases, the source overlay data 34 is pre—processed to contain the annotation data
`
`preferably in a resolution independent form associated with a display coordinate specification
`
`relative to the source image data 32. The XML, GML or binary overlay data may be compressed
`
`prior to storage on the network server 12, 22.
`
`The preferred architecture 40 of a client system 18, 20, for purposes of implementing the
`
`present invention, is shown in FIG. 3. The architecture 40 is preferably implemented by
`
`software plug—in or application executed by the client system 18, 20 and that utilizes basic
`
`software and hardware services provided by the client system 18, 20. A parcel request client 42
`
`preferably implements an HTML client that supports HTML—based interactions with the server
`
`12, 22 using the underlying network protocol stack and hardware network interface provided by
`
`the client systems 18, 20. A central parcel processing control block 44 preferably implements
`
`12 of 150
`
`Microsoft Corp. Exhibit 1016
`
`12 of 150
`
`Microsoft Corp. Exhibit 1016
`
`

`

`-11-
`
`the client process and control algorithms. The control block 44 directs the transfer of received
`
`image parcels and XML/GML/binary overlay data to a local parcel data store 46. Local parcel
`
`data store 46 may also act for example as local cache weather the entire data or part of it is in
`
`dynamic and/or static cache. Preferably image data parcels are stored in conventional quad—tree
`
`data structures, where tree nodes of depth D correspond to the stored image parcels of a
`
`derivative image of resolution KD. The XML/GML/binary overlay data is preferably stored as a
`
`data object that can be subsequently read by an XML/GML/binary parser implemented as part of
`
`the control block 44.
`
`The control block 44 is also responsible for de