Case: 1:21-cv-02666 Document #: 1 Filed: 05/17/21 Page 1 of 10 PageID #:1
`
`IN THE UNITED STATES DISTRICT COURT
`FOR THE NORTHERN DISTRICT OF ILLINOIS
`
`MYHERITAGE (USA), INC. AND
`MYHERITAGE LTD.,
`
`Plaintiffs,
`
`v.
`
`MEMORYWEB, LLC,
`
`Defendant.
`
`Case No.
`
`)
`)
`)
`)
`)
`)
`)
`)
`)
`)
`)
`)
`
`COMPLAINT FOR DECLARATORY JUDGMENT
`
`MyHeritage (USA), Inc. (“MyHeritage (USA)”) and MyHeritage Ltd. (collectively,
`
`“MyHeritage”), by and through its undersigned counsel, hereby alleges for its complaint against
`
`MemoryWeb, LLC (“MemoryWeb”) on personal knowledge as to its own activities and on
`
`information and belief as to the activities of others as follows:
`
`NATURE OF THE ACTION
`
`1.
`
`This is an action for a declaratory judgment of non-infringement arising
`
`under the patent laws of the United States, 35 U.S.C. § 1, et seq., and the Declaratory Judgment
`
`Act, 28 U.S.C. §§ 2201 and 2202. MyHeritage seeks relief because Defendant MemoryWeb has
`
`made clear through its statements and actions that it intends to pursue litigation on at least one
`
`claim of United States Patent Nos. 10,423,658 (the “’658 patent”), attached hereto as Exhibit A,
`
`and/or 10,621,228 (the “’228 patent”), attached hereto as Exhibit B, (collectively, “the Patents-in-
`
`Suit”) against MyHeritage’s products and services.
`
`2.
`
`MyHeritage has not infringed, and is not infringing, any claims of the
`
`Patents-in-Suit. MyHeritage thus seeks a declaratory judgment that, through its actions or through
`
`Petitioner Apple Inc. - Ex. 1046, p. 1
`
`

`

`Case: 1:21-cv-02666 Document #: 1 Filed: 05/17/21 Page 2 of 10 PageID #:2
`
`the normal, advertised and expected use of its products, services or technology, it has not infringed,
`
`induced others to infringe, or contributed to the infringement by others of any claim of the Patents-
`
`in-Suit.
`
`3.
`
`This relief is appropriate because MemoryWeb has alleged that MyHeritage
`
`has infringed, and allegedly continues to infringe, one or more claims from the ’658 and ’228
`
`patents. MemoryWeb has threatened MyHeritage with litigation and potential injunctions under
`
`these patents, while contemporaneously launching litigation against at least one other party on the
`
`same patents.
`
`4.
`
`These activities by MemoryWeb have placed a cloud over MyHeritage and
`
`its products, and have created a substantial, definite, concrete and immediate justiciable
`
`controversy between MyHeritage and MemoryWeb over whether any MyHeritage product or
`
`service infringes any of the patent claims.
`
`PARTIES
`
`5.
`
`Plaintiff MyHeritage (USA), Inc. is a corporation organized and existing
`
`under the laws of Delaware, with a principal place of business at 2975 Executive Parkway Suite
`
`310, Lehi, Utah 84043.
`
`6.
`
`Plaintiff MyHeritage Ltd. is a limited liability company organized and
`
`existing under the laws of Israel, with a principal place of business at 3 Ariel Sharon Blvd., Or
`
`Yehuda, Israel. MyHeritage Ltd. may be served with process by service in accordance with the
`
`Hague on the Service Abroad of Judicial and Extrajudicial Documents, in accordance with Fed. R.
`
`Civ. P. 4(f).
`
`7.
`
`On information and belief, Defendant MemoryWeb is a corporation existing
`
`and organized under the laws of Illinois with its principal place of business at 526 Crescent Blvd.
`
`2
`
`Petitioner Apple Inc. - Ex. 1046, p. 2
`
`

`

`Case: 1:21-cv-02666 Document #: 1 Filed: 05/17/21 Page 3 of 10 PageID #:3
`
`Suite 233, Glen Ellyn, Illinois 60137.
`
`JURISDICTION AND VENUE
`
`8.
`
`This action arises under the United States patent laws and includes a request
`
`for declaratory relief under 28 U.S.C. §§ 2201 and 2202.
`
`9.
`
`This Court has subject matter jurisdiction under 28 U.S.C. §§ 1331, 1338,
`
`2201, and 2202, and 35 U.S.C. § 1, et seq.
`
`10. MemoryWeb is subject to personal jurisdiction in this judicial district
`
`because, inter alia, MemoryWeb has its primary place of business located within this district, and
`
`it has sufficient business or contacts within the district to justify jurisdiction under the United
`
`States Constitution.
`
`11.
`
`Venue is proper under 28 U.S.C. §§ 1391(b)-(d) and 28 U.S.C. § 1400(b).
`
`EXISTENCE OF AN ACTUAL CONTROVERSY
`
`12. MyHeritage re-alleges and incorporates by reference each of Paragraphs 1-
`
`11 above.
`
`13.
`
`An actual controversy exists within the jurisdiction of this Court under 28
`
`U.S.C. §§ 2201 and 2202.
`
`14.
`
`Since at least 2007, MyHeritage has developed, marketed, used and offered
`
`for sale its genealogy software, MyHeritage Family Tree Builder. Since at least 2005, MyHeritage
`
`has also owned and operated a website at MyHeritage.com, which provides online genealogy
`
`services and is integrated with its genealogy software. MyHeritage’s software and website include
`
`a suite of features for family history research which, inter alia, allow users to upload, store and
`
`manage digital files, including photographs and videos, in the process of tracing and managing the
`
`users’ genealogy. Among these features are MyHeritage’s Photos, Family Tree and
`
`3
`
`Petitioner Apple Inc. - Ex. 1046, p. 3
`
`

`

`Case: 1:21-cv-02666 Document #: 1 Filed: 05/17/21 Page 4 of 10 PageID #:4
`
`PedigreeMapTM.
`
`15. MemoryWeb has explicitly and baselessly accused MyHeritage’s products
`
`and their features broadly of infringing the ’228 and ’658 patents, which MemoryWeb purports to
`
`own.
`
`16. MemoryWeb first accused MyHeritage and its products of infringement in
`
`a cease and desist letter sent to MyHeritage on April 20, 2021, which is attached hereto as Exhibit
`
`C. That April 20 letter explicitly accused MyHeritage of “infringing certain claims of at least the
`
`‘658 Patent and the ‘228 Patent ” and specifically identified the “functionality of [MyHeritage’s]
`
`FamilyTree, Photos/My Photos and PedigreeMapTM” as the basis of its assertions. The letter
`
`further threatened to “seek injunctive relief and recover monetary damages” and demanded that
`
`MyHeritage
`
`immediately discontinue use of
`
`its “Family Tree, Photos/MyPhotos and
`
`PedigreeMapTM technology.” Despite that threat, the letter also suggested some unspecified
`
`“business arrangement” and that it may serve the parties to agree to some “confidentiality
`
`agreement.” MemoryWeb also set a deadline of April 30, 2021 for MyHeritage to respond to its
`
`letter, stating that if MyHeritage did not respond by that date MemoryWeb “will assume”
`
`MyHeritage “is not willing to cooperate and will take the necessary steps to enforce MemoryWeb’s
`
`rights.”
`
`17.
`
`On April 26, 2021, well before MemoryWeb’s unilaterally imposed
`
`deadline, MyHeritage responded to MemoryWeb’s cease and desist letter. In its response, which
`
`is attached hereto as Exhibit D, MyHeritage explained that despite its demand that MyHeritage
`
`immediately cease operation, MemoryWeb’s April 20 letter did not provide information sufficient
`
`to enable meaningful evaluation of the merits of MemoryWeb’s assertions. It thus asked
`
`MemoryWeb to provide claim charts that identified the specific claims in the two patents
`
`4
`
`Petitioner Apple Inc. - Ex. 1046, p. 4
`
`

`

`Case: 1:21-cv-02666 Document #: 1 Filed: 05/17/21 Page 5 of 10 PageID #:5
`
`MemoryWeb was asserting, and to explain how the specific features or characteristics of
`
`MyHeritage products and services meet each limitation in each of any such claims. MyHeritage
`
`also indicated that MemoryWeb’s proposal of some sort of business arrangement and a
`
`confidentiality agreement did not make sense under the circumstances.
`
`18.
`
`On April 26, 2021, MemoryWeb filed an action against another party,
`
`Samsung, in the Western District of Texas alleging infringement of the ’228 and ’658 patents. See
`
`MemoryWeb, LLC v. Samsung Electronics Co., Ltd. and Samsung Electronics America, Inc., No.
`
`6:21-cv-00411-ADA (W.D. Tex.).
`
`19.
`
`On May 4, 2021, MemoryWeb responded to MyHeritage’s April 26, 2021
`
`letter with another letter, attached hereto as Exhibit E, again asserting infringement of the same
`
`two patents and including a claim chart that purported to show how MyHeritage’s products satisfy
`
`one claim in one of the two asserted patents (i.e., claim 1 of the ’228 patent). MemoryWeb also
`
`referred to “additional issued U.S. patents and pending applications that may be relevant to the
`
`same or similar MyHeritage features” but identified only a single patent application that has not
`
`been issued as a patent. MemoryWeb also attached a copy of its Complaint against Samsung as
`
`evidence that “MemoryWeb is serious about enforcing its valuable intellectual property rights.”
`
`MemoryWeb then set a May 17, 2021 deadline for MyHeritage to respond to its letter.
`
`20.
`
`Because MemoryWeb did not explain why it believed any claims in the ’228
`
`and ’658 patents (other than claim 1 of the ’228 patent) were infringed by MyHeritage products
`
`and services, MyHeritage promptly requested that MemoryWeb do so. MyHeritage did this in a
`
`letter sent on May 10, 2021, attached hereto as Exhibit F. MemoryWeb has not responded to this
`
`May 10, 2021 request for additional information.
`
`21.
`
`Despite MemoryWeb’s refusal to provide any further information,
`
`5
`
`Petitioner Apple Inc. - Ex. 1046, p. 5
`
`

`

`Case: 1:21-cv-02666 Document #: 1 Filed: 05/17/21 Page 6 of 10 PageID #:6
`
`MyHeritage responded to MemoryWeb’s assertions in a letter sent on May 14, 2021, attached
`
`hereto as Exhibit G. MyHeritage pointed out that based on the limited information that
`
`MemoryWeb had provided, there was no credible basis for MemoryWeb’s assertions of
`
`infringement of any claim of the ’228 and ’658 patents by any MyHeritage product or service.
`
`MyHeritage also indicated that it might reconsider MemoryWeb’s proposal for a non-disclosure
`
`agreement if doing that would facilitate further discussions, and invited MemoryWeb to provide a
`
`copy of its proposed NDA for MyHeritage to review. MemoryWeb has not responded to
`
`MyHeritage’s May 14, 2021 letter.
`
`22.
`
`Contrary to MemoryWeb’s assertions, none of MyHeritage’s products,
`
`services or technology infringe or have infringed, directly or indirectly, any claim of the Patents-
`
`in-Suit. Nonetheless, MemoryWeb’s claims, allegations and statements directed to photo
`
`management technology generally and MyHeritage’s products specifically have placed a cloud of
`
`uncertainty over MyHeritage and its products. This uncertainty has created a substantial, definite,
`
`concrete and immediate justiciable controversy between MyHeritage and MemoryWeb regarding
`
`the Patents-in-Suit. MemoryWeb’s actions manifest a threat and intent to assert the ’658 and ’228
`
`Patents against MyHeritage and/or MyHeritage products in litigation, as it already has asserted the
`
`patents and have threatened suit, including injunctive relief.
`
`23.
`
`Accordingly, MemoryWeb’s statements and actions show that there is a
`
`substantial controversy between MyHeritage and MemoryWeb of sufficient immediacy and reality
`
`conferring jurisdiction upon this Court pursuant to the Declaratory Judgment Act, 28 U.S.C. §§
`
`2201 and 2202.
`
`COUNT I
`NONINFRINGEMENT OF U.S. PATENT NO. 10,423,658
`
`24. MyHeritage re-alleges and incorporates the allegations of Paragraphs 1-23
`
`6
`
`Petitioner Apple Inc. - Ex. 1046, p. 6
`
`

`

`Case: 1:21-cv-02666 Document #: 1 Filed: 05/17/21 Page 7 of 10 PageID #:7
`
`as if set forth herein in their entirety.
`
`25.
`
`Neither MyHeritage nor its products, services or technology have infringed,
`
`directly or indirectly, any claim of the ’658 Patent. For example, on the MyHeritage.com website,
`
`before the user interacts with map elements in the PedigreeMap interface, the MyHeritage
`
`PedigreeMap™ does not display “thumbnail images” at locations on an interactive map. In
`
`addition, not all photos and videos in the MyHeritage.com user account that are associated with a
`
`geotag will be retrieved and displayed to the user.
`
`26.
`
`Based on at least the facts identified in paragraph 25, MyHeritage does not
`
`infringe claim 1 of the ’658 patent. Also, because every claim of the ’658 Patent depends from
`
`claim 1, MyHeritage does not infringe any claim of the ’658 Patent.
`
`27.
`
`A substantial, definite, concrete and immediate justiciable controversy
`
`exists between MyHeritage and MemoryWeb with respect to whether MyHeritage infringes any
`
`claim of the ’658 Patent.
`
`28. MyHeritage seeks a declaration that it does not infringe any claim of the
`
`’658 Patent.
`
`COUNT II
`NONINFRINGEMENT OF U.S. PATENT NO. 10,621,228
`
`29. MyHeritage re-alleges and incorporates the allegations of Paragraphs 1-28
`
`as if set forth herein in their entirety.
`
`30.
`
`Neither MyHeritage nor its products, services or technology have infringed,
`
`directly or indirectly, any claim of the ’228 Patent. For example, the MyHeritage.com website
`
`does not display a “first set of digital files” or a “second set of digital files” “being produced from
`
`outputs of one or more digital imaging devices,” which is required by all the claims of the ’228
`
`Patent. As another example, before the user interacts with map elements in the PedigreeMap
`
`7
`
`Petitioner Apple Inc. - Ex. 1046, p. 7
`
`

`

`Case: 1:21-cv-02666 Document #: 1 Filed: 05/17/21 Page 8 of 10 PageID #:8
`
`interface, the MyHeritage PedigreeMap™ does not include “a first location selectable thumbnail
`
`image at a first location on the interactive map” (or second or third “selectable thumbnail images”).
`
`31.
`
`Based on at least the facts identified in paragraph 30, MyHeritage does not
`
`infringe claim 1 of the ’228 patent. Also, because every claim of the ’228 Patent depends from
`
`claim 1, MyHeritage does not infringe any claim of the ’228 Patent.
`
`32.
`
`A substantial, definite, concrete and immediate justiciable controversy
`
`exists between MyHeritage and MemoryWeb with respect to whether MyHeritage infringes any
`
`claim of the ’228 Patent.
`
`33. MyHeritage seeks a declaration that it does not infringe any claim of the
`
`’228 Patent.
`
`
`
`PRAYER FOR RELIEF
`
`WHEREFORE, MyHeritage prays for an Order and entry of Judgment against
`
`MemoryWeb as follows:
`
`A.
`
`Declaring that MyHeritage does not infringe, directly or indirectly, the ’658
`
`patent;
`
`B.
`
`Declaring that MyHeritage does not infringe, directly or indirectly, the ’228
`
`patent;
`
`C.
`
`Declaring the case exceptional under 35 U.S.C. § 285 and awarding
`
`MyHeritage its reasonable attorneys’ fees in this action;
`
`Awarding MyHeritage its costs and expenses in this action; and
`
`Granting such other and further relief as the Court deems proper and just.
`
`D.
`
`E.
`
`
`
`
`
`8
`
`Petitioner Apple Inc. - Ex. 1046, p. 8
`
`

`

`Case: 1:21-cv-02666 Document #: 1 Filed: 05/17/21 Page 9 of 10 PageID #:9
`
`Dated: May 17, 2021
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`/s/ Richard A. Cederoth
`Richard A. Cederoth (Bar ID No. 6185199)
`SIDLEY AUSTIN LLP
`One South Dearborn
`Chicago, IL 60603
`(312) 853-7000
`
`Counsel for Plaintiffs MyHeritage (USA), Inc.
`and MyHeritage Ltd.
`
`
`
`
`
`9
`
`Petitioner Apple Inc. - Ex. 1046, p. 9
`
`

`

`Case: 1:21-cv-02666 Document #: 1 Filed: 05/17/21 Page 10 of 10 PageID #:10
`
`
`
`CERTIFICATE OF SERVICE
`
`I certify that on May 17, 2021, I electronically filed the foregoing with the Clerk of Court
`
`using the CM/ECF system, which will send notification of such filing to all counsel of record.
`
`
`
`
`
`/s/ Richard A. Cederoth
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`10
`
`Petitioner Apple Inc. - Ex. 1046, p. 10
`
`

`

`Case: 1:21-cv-02666 Document #: 1-1 Filed: 05/17/21 Page 1 of 72 PageID #:11
`
`
`
`
`
`
`EXHIBIT A
`
`
`
`Petitioner Apple Inc. - Ex. 1046, p. 11
`
`

`

`Case: 1:21-cv-02666 Document #: 1-1 Filed: 05/17/21 Page 2 of 72 PageID #:12
`I 1111111111111111 1111111111 111111111111111 111111111111111 IIIIII IIII IIII IIII
`US010423658B2
`
`c12) United States Patent
`Desmond et al.
`
`(IO) Patent No.: US 10,423,658 B2
`*Sep.24,2019
`(45) Date of Patent:
`
`(54) METHOD AND APPARATUS FOR
`MANAGING DIGITAL FILES
`
`(71) Applicant: MemoryWeb, LLC, Glen Ellyn, IL
`(US)
`
`(72)
`
`Inventors: Christopher J. Desmond, Glen Ellyn,
`IL (US); Nancy L. Desmond, Glen
`Ellyn, IL (US); L. Michael Taylor,
`Chicago, IL (US)
`
`(73) Assignee: NCM IP HOLDINGS, LLC, Glen
`Ellyn, IL (US)
`
`( *) Notice:
`
`Subject to any disclaimer, the term ofthis
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 518 days.
`
`This patent is subject to a terminal dis(cid:173)
`claimer.
`
`(21) Appl. No.: 15/375,927
`
`(22) Filed:
`
`Dec. 12, 2016
`
`(65)
`
`Prior Publication Data
`
`US 2017/0091225 Al Mar. 30, 2017
`
`(63)
`
`(51)
`
`(52)
`
`Related U.S. Application Data
`
`Continuation of application No. 14/193,426, filed on
`Feb. 28, 2014, now Pat. No. 9,552,376, which is a
`(Continued)
`
`Int. Cl.
`G06F 16190
`G06F 16158
`
`(2019.01)
`(2019.01)
`(Continued)
`
`U.S. Cl.
`CPC .......... G06F 1615866 (2019.01); G06F 16151
`(2019.01); G06F 161901 (2019.01); G06F
`161907 (2019.01); G06F 3/0481 (2013.01)
`
`(58) Field of Classification Search
`CPC ... G06F 16/5866; G06F 16/907; G06F 16/901
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`5,694,514 A
`6,629,104 Bl
`
`12/1997 Evans
`9/2003 Parulski
`(Continued)
`
`OTHER PUBLICATIONS
`
`Kustanowitz et al., "Motivating Annotation for Personal Digital
`Photo Libraries: Lowering Barriers while Raising Incentives," Tech.
`Report HCIL-2004-18, U. Maryland, 2005 (10 pages).
`(Continued)
`
`Primary Examiner - Loe Tran
`(74) Attorney, Agent, or Firm - Nixon Peabody LLP
`
`(57)
`
`ABSTRACT
`
`A computer-implemented method of associating digital tags
`with digital files comprises storing a plurality of digital files
`having embedded therein content data and metadata includ(cid:173)
`ing tags; receiving, via a user interface device of a client
`device, a first tag label containing alphanumeric text created
`and inputted by a user of the client device; modifying, using
`a controller device, a selected first one of the tags of the
`metadata in a first of the digital files to include the first tag
`label; receiving, via the user interface device or another user
`interface device, an instruction to search for all of the digital
`files having at least the first tag label; responsive to receiving
`the instruction, automatically searching for all of the digital
`files having at least the first tag label; and displaying, on a
`video display device associated with the client device, a first
`indication of the first tag label.
`
`15 Claims, 50 Drawing Sheets
`
`Select None
`Select All
`Delete Selected
`
`September 2, 2010 change
`Yellowstone, MT, USA change
`Christopher Desmond, Nancy Desmond
`Fly Fishing in Yellowstone
`Add
`Add
`Add
`
`1 File(s) Selected
`File Attributes:
`Date:
`Location:
`People:
`Event:
`Comments:
`Keywords:
`Album:
`Star Ranking:
`Sharing Rights: Add
`Recipe:
`Add
`DocumentType: Add
`File Name:
`IMG 123 change
`
`Petitioner Apple Inc. - Ex. 1046, p. 12
`
`

`

`Case: 1:21-cv-02666 Document #: 1-1 Filed: 05/17/21 Page 3 of 72 PageID #:13
`
`US 10,423,658 B2
`Page 2
`
`Related U.S. Application Data
`
`continuation-in-part of application No. 13/157,214,
`filed on Jun. 9, 2011, now Pat. No. 9,098,531.
`
`(51)
`
`Int. Cl.
`G06F 16151
`G06F 161901
`G06F 161907
`G06F 3/0481
`
`(2019.01)
`(2019.01)
`(2019.01)
`(2013.01)
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`7,372,976 B2
`7,480,669 B2
`7,694,236 B2
`8,150,844 B2
`8,484,223 B2
`2006/0165380 Al*
`
`5/2008 Rhoad
`1/2009 Lo
`4/2010 Gusmorino
`4/2012 Redstone
`7/2013 Ota
`7/2006 Tanaka .
`
`GllB 27/34
`386/227
`
`2007/0282908 Al
`2009/0ll3350 Al*
`
`2009/0265631 Al
`
`12/2007 Van Der Meulen
`4/2009 Hibino .................... G06F 16/41
`715/853
`
`10/2009 Sigurbjornsson
`
`OTHER PUBLICATIONS
`
`Miller et al., "Give and take: a study of consumer photo-sharing
`culture and practice," CHI ' 07 Proceedings of the SIGCHI Con(cid:173)
`ference on Human Factors in Computing Systems, pp. 347-356,
`2007 (10 pages).
`Ames et al., "Why we tag: motivations for annotation in mobile and
`online media," CHI '07 Proceedings of the SIGCHI Conference on
`Human Factors in Computing Systems, pp. 971-980, ACM, 2007
`(10 pages).
`
`Yee et al., "Faceted Metadata for Image Search and Browsing", CHI
`2003, pp. 401-408, 2003, ACM.
`Ferre, "CAMELIS: Organizing and Browsing a Personal Photo
`Collection with a Logical Information System", Int. Conf. Concept
`Lattices and Their Applications, pp. ll2-123, 2007, HAL.
`Tomasson et al., "PhotoCube: Effective and Efficient Multi(cid:173)
`Dimensional Browsing of Personal Photo Collections", ICMR 'll,
`20ll, ACM.
`Bartolini et al., "Integrating Semantic and Visual Facets for Brows(cid:173)
`ing Digital Photo Collections", SBED, pp. 65-72, 2009.
`Trattner et al., "Evaluating Tag-Based Information Access in Image
`Collections", Proceedings of the 23 rd ACM Conference on Hyper(cid:173)
`text and Social Media, pp. 113-122, 2012 ACM.
`Kang et al., "Capture, Annotate, Browse, Find, Share: Novel
`Interfaces for Personal Photo Management", International Journal
`of Human-Computer Interaction, 23(3), pp. 315-337, 2007, Law(cid:173)
`rence Eribaum Associates, Inc.
`Jaffe et al., "Generating Summaries and Visualization for Large
`Collections of GeoReferenced Photographs", MIR'06, pp. 89-98,
`2006 ACM.
`Tornial et al., "Sharing, Discovering and Browsing Geotagged
`Pictures on the Web", 2007, Hewlett-Packard Development Com(cid:173)
`pany, L.P., pp. 1-18.
`Snavely et al., "Photo Tourism: Exploring Photo Collection in 3D",
`SIGGRAPH '06 ACM Transactions on Graphics, vol. 25, Issue 3,
`pp. 835-846, 2006 ACM.
`Kisilevich et al., "Event-based analysis of People's Activities and
`Behavior Using Flickr and Panoramio Geotagged Photo Collec(cid:173)
`tions", 14th International Conference Information Visualization, pp.
`289-296, 2010 IEEE.
`Ahern et al., "World Explorer: Visualizing Aggregate Data From
`Unstructured Text in Geo-Referenced Collections", JCDL'07, pp.
`1-10, 2007, ACM.
`Kopf et al., "Deep photo: model-based photograph enhancement
`and viewing", ACM Transactions on Graphics, vol. 27, No. 5,
`Article ll6, Dec. 2008, ACM (10 pages).
`
`* cited by examiner
`
`Petitioner Apple Inc. - Ex. 1046, p. 13
`
`

`

`Case: 1:21-cv-02666 Document #: 1-1 Filed: 05/17/21 Page 4 of 72 PageID #:14
`
`U.S. Patent
`
`Sep.24,2019
`
`Sheet 1 of 50
`
`US 10,423,658 B2
`
`FIG. 1
`
`Select None
`Select All
`Delete Selected
`
`,:-,,
`
`\:: ·): ~:lij
`
`September 2, 2010 change
`Yellowstone, MT, USA change
`Christopher Desmond, Nancy Desmond
`Fly Fishing in Yellowstone
`Add
`Add
`Add
`
`1 File(s) Selected
`File Attributes:
`Date:
`Location:
`People:
`Event:
`Comments:
`Keywords:
`Album:
`Star Ranking:
`Sharing Rights: Add
`Recipe:
`Add
`Document Type: Add
`File Name:
`IMG 123 change
`
`•••••
`
`B '
`
`a
`
`.
`
`'
`
`Petitioner Apple Inc. - Ex. 1046, p. 14
`
`

`

`Case: 1:21-cv-02666 Document #: 1-1 Filed: 05/17/21 Page 5 of 72 PageID #:15
`
`U.S. Patent
`
`Sep.24,2019
`
`Sheet 2 of 50
`
`US 10,423,658 B2
`
`FIG. 2
`
`Comments:
`Suzanne and Anthony's Wedding Party where the cousins posed
`for a photo in the grass. Note, Jack with the lollipop and the
`photographer with his shoe in the photo
`
`Location:
`Historical Society
`Lisle, IL 60532
`
`People:
`Jack Wong
`CJ Wong
`Mary Firestone
`Zoe Peika
`Nick Persons
`
`Event: Suzanne & Anthony's Wedding Reception 2010
`
`Camera Details: more
`
`Petitioner Apple Inc. - Ex. 1046, p. 15
`
`

`

`Case: 1:21-cv-02666 Document #: 1-1 Filed: 05/17/21 Page 6 of 72 PageID #:16
`
`U.S. Patent
`
`Sep.24,2019
`
`Sheet 3 of 50
`
`US 10,423,658 B2
`
`FIG. 3
`
`r:·~:·t/!~··.···~·
`
`: ..
`~
`it\
`j:)t!
`
`Mary Bader and Clint Firestone
`Wedding October 11, 1945
`
`Capt. Clint Firestone
`1952
`
`Germany Trip 1955
`
`Father's Day 1966
`
`College Break 1992
`
`LA Trip 1993
`
`EIU College
`Graduation 1994
`
`Cairo, Egypt 1996
`
`Thumbnail I Table
`
`.
`
`.
`
`.
`
`:::::::
`
`}\~ I'•· ~~
`
`:\::::
`Christopher pt
`Year 1972
`
`Europe
`Backpacking
`1996
`
`\;:)rn)w]
`
`·~
`
`~
`
`.$.::::.~1
`
`Amazon Fishing Trip
`November 2009
`
`Nancy James and
`Christopher Desmond
`Wedding October 17, 1998
`
`Cousins Party 2010
`
`Fishing in Yellowstone
`Montana September 2010
`
`Petitioner Apple Inc. - Ex. 1046, p. 16
`
`

`

`Case: 1:21-cv-02666 Document #: 1-1 Filed: 05/17/21 Page 7 of 72 PageID #:17
`
`U.S. Patent
`
`Sep.24,2019
`
`Sheet 4 of 50
`
`US 10,423,658 B2
`
`FIG. 4
`
`Christopher Desmond and Nancy Desmond Wedding - October 17, 1998
`
`Event f}
`
`Captions On I Off
`
`◄
`
`Petitioner Apple Inc. - Ex. 1046, p. 17
`
`

`

`Case: 1:21-cv-02666 Document #: 1-1 Filed: 05/17/21 Page 8 of 72 PageID #:18
`
`U.S. Patent
`
`Sep.24,2019
`
`Sheet 5 of 50
`
`US 10,423,658 B2
`
`FIG. 5
`
`Petitioner Apple Inc. - Ex. 1046, p. 18
`
`

`

`Case: 1:21-cv-02666 Document #: 1-1 Filed: 05/17/21 Page 9 of 72 PageID #:19
`
`U.S. Patent
`
`Sep.24,2019
`
`Sheet 6 of 50
`
`US 10,423,658 B2
`
`FIG. 6
`
`Thumbnail I Table
`
`Clint Firestone
`
`Tim Dog
`
`Steve Fire
`
`Suzanne Lucky
`
`Steve Fwah
`
`Ben Sherman
`
`Joe Schmo
`
`◄
`
`►
`
`Mary Virginia
`
`Clint Dewitt
`
`Charlotte Killian
`
`Chris Jeep
`
`Peter LongHorn
`
`Tim Dog
`
`Suzanne Lucky
`
`Steve Fwah
`
`Ben Sherman
`
`Joe Schmo
`
`Mary Virginia
`
`Clint Dewitt
`
`Steve Fire
`
`I ~
`
`Charlotte Killian
`
`Petitioner Apple Inc. - Ex. 1046, p. 19
`
`

`

`Case: 1:21-cv-02666 Document #: 1-1 Filed: 05/17/21 Page 10 of 72 PageID #:20
`
`U.S. Patent
`
`Sep.24,2019
`
`Sheet 7 of 50
`
`US 10,423,658 B2
`
`FIG. 7
`
`Clinton Dewitt Firestone IV
`
`July 12, 1896
`Birth:
`April 29, 1971
`Death:
`Parents: Clinton Dewitt Firestone Ill and Viola Miller
`Comments: He was a WWII U.S. Air force pilot and POW in WWII and veteran
`honorably discharged in December of 1947. He worked for 44 years for the
`Firestone Tire and Rubber Company in retail, wholesale and original equipment
`sales, marketing and management. He was born in Akron, OH and is buried in
`Columbiana, OH.
`
`Edit bio
`
`Recipes
`
`Family Tree
`
`f:ifi~
`
`..................................................................................
`
`Petitioner Apple Inc. - Ex. 1046, p. 20
`
`

`

`Case: 1:21-cv-02666 Document #: 1-1 Filed: 05/17/21 Page 11 of 72 PageID #:21
`
`U.S. Patent
`
`Sep.24,2019
`
`Sheet 8 of 50
`
`US 10,423,658 B2
`
`FIG. 8
`
`Descendants of Charlotte Killian Halter
`
`Tree I Table
`
`Petitioner Apple Inc. - Ex. 1046, p. 21
`
`

`

`Case: 1:21-cv-02666 Document #: 1-1 Filed: 05/17/21 Page 12 of 72 PageID #:22
`
`U.S. Patent
`
`Sep.24,2019
`
`Sheet 9 of 50
`
`US 10,423,658 B2
`
`Year
`
`Month
`Day
`
`f
`
`◄
`
`FIG. 9
`
`•
`
`. )
`
`►
`
`Petitioner Apple Inc. - Ex. 1046, p. 22
`
`

`

`Case: 1:21-cv-02666 Document #: 1-1 Filed: 05/17/21 Page 13 of 72 PageID #:23
`
`U.S. Patent
`
`Sep.24,2019
`
`Sheet 10 of 50
`
`US 10,423,658 B2
`
`FIG. 10
`
`1 Zucchini (chopped)
`Thai Basil (8 leaves)
`Lemon Grass (in jar) 1 teaspoon
`Chinese Ginger Root (in jar) 1 teaspoon
`
`Rice
`Thai Rice (something that only takes 2 cups of water)
`Dice chicken in bowl and add two tablespoons of fish
`sauce. Let marinate for 20 minutes.
`Take thick part of coconut milk out into pan (about 4
`tablespoons), Curry paste, 1 spoon of lemon grass, 1
`spoon of ginger and garlic. Heat over high with boil and
`THEN stir for 1 minute. Add meat (uncooked) and fry
`until cooked over high heat
`Add milk, brown sugar and salt. Bring back to slight boil
`and constantly stir. Add veggies and soy sauce. Cook for
`about 10-14 minutes COVERED until veggies are cooked.
`Serve with a smile.
`
`Desmond's Yellow Thai Chicken Curry
`
`Curry Mix
`Coconut milk (400 ml)- DO NOT SHAKE IT UP
`800 gram of chicken (4 chicken breast)
`Fish sauce (Nam Pia) Thai Bamboo Garden - Bottle
`Garlic (2 cloves)
`Broccoli ( 2cups chopped)
`2 Peppers (chopped)
`2 Carrots (chopped)
`
`Chef: Barry Desmond
`
`Video on How to Make It
`
`Original Handwritten Recipe
`
`Petitioner Apple Inc. - Ex. 1046, p. 23
`
`

`

`Case: 1:21-cv-02666 Document #: 1-1 Filed: 05/17/21 Page 14 of 72 PageID #:24
`
`U.S. Patent
`
`Sep.24,2019
`
`Sheet 11 of 50
`
`US 10,423,658 B2
`
`FIG. 11
`
`Thumbnail I Table
`
`Album/Event
`
`Date
`
`Location
`
`# Photos
`
`# Videos
`
`# Docs
`
`Jack Monk's Arrival
`
`26-Dec-2003
`
`Chicago, IL
`
`Mike Testy's First Birthday
`
`13-Sep-1983 Minneapolis, MN
`
`Cubs Beat Cards 1998
`
`5-Aug-1998
`
`Chicago, IL
`
`Jack Monk's Arrival
`
`29-Dec-2003
`
`Chicago, IL
`
`Mike Testv's 2nd Birthday
`
`13-Sep-1983 Minneapolis, MN
`
`Cubs Beat Cards 1998
`
`5-Aug-1998
`
`Chicago, IL
`
`Jack Monk's Arrival
`
`29-Dec-2003
`
`Chicago, IL
`
`Mike Testv's 3rd Birthday
`
`13-Sep-1983 Minneapolis, MN
`
`Cubs Beat Cards 1998
`
`5-Aug-1998
`
`Chicago, IL
`
`Jack Wrigley Monk's Arrival
`
`29-Dec-2003
`
`Chicago, IL
`
`Mike Testy's 4th Birthday
`
`13-Sep-1983 Minneapolis, MN
`
`Cubs Beat Cards 1998
`
`5-Aug-1998
`
`Chicago, IL
`
`Nancv Learns How to Ride a Bike
`
`21-Jul-1978
`
`St. Louis, MO
`
`69
`
`54
`
`36
`
`69
`
`54
`
`36
`
`69
`
`54
`
`36
`
`69
`
`54
`
`36
`
`76
`
`4
`
`21
`
`2,199
`
`4
`
`21
`
`2,199
`
`4
`
`21
`
`2,199
`
`4
`
`21
`
`2,199
`
`2
`
`4
`
`0
`
`2
`
`4
`
`0
`
`2
`
`4
`
`0
`
`2
`
`4
`
`0
`
`2
`
`0
`
`Petitioner Apple Inc. - Ex. 1046, p. 24
`
`

`

`Case: 1:21-cv-02666 Document #: 1-1 Filed: 05/17/21 Page 15 of 72 PageID #:25
`
`U.S. Patent
`
`Sep.24,2019
`
`Sheet 12 of 50
`
`US 10,423,658 B2
`
`FIG. 12
`
`Thumbnail I Table
`
`Album/Event
`
`Date
`
`Location
`
`# Photos
`
`# Videos
`
`# Docs
`
`Jack Monk's Arrival
`
`26-Dec-2003
`
`Chicago, IL
`
`Mike Testy's First Birthday
`
`13-Sep-1983 Minneapolis, MN
`
`Cubs Beat Cards 1998
`
`5-Aug-1998
`
`Chicago, IL
`
`Jack Monk's Arrival
`
`29-Dec-2003
`
`Chicago, IL
`
`Mike Testv's 2nd Birthdav
`
`13-Sep-1983 Minneapolis, MN
`
`Cubs Beat Cards 1998
`
`5-Aug-1998
`
`Chicago, IL
`
`Jack Monk's Arrival
`
`29-Dec-2003
`
`Chicago, IL
`
`Mike Testy's 3rd Birthday
`
`13-Sep-1983 Minneapolis, MN
`
`Cubs Beat Cards 1998
`
`5-Aug-1998
`
`Chicago, IL
`
`Jack Wrigley Monk's Arrival
`
`29-Dec-2003
`
`Chicago, IL
`
`Mike Testv's 4th Birthdav
`
`13-Sep-1983 Minneapolis, MN
`
`Cubs Beat Cards 1998
`
`5-Aug-1998
`
`Chicago, IL
`
`Nancy Learns How to Ride a Bike
`
`21-Jul-1978
`
`St. Louis, MO
`
`69
`
`54
`
`36
`
`69
`
`54
`
`36
`
`69
`
`54
`
`36
`
`69
`
`54
`
`36
`
`76
`
`4
`
`21
`
`2,199
`
`4
`
`21
`
`2,199
`
`4
`
`21
`
`2,199
`
`4
`
`21
`
`2,199
`
`2
`
`4
`
`0
`
`2
`
`4
`
`0
`
`2
`
`4
`
`0
`
`2
`
`4
`
`0
`
`2
`
`0
`
`Petitioner Apple Inc. - Ex. 1046, p. 25
`
`

`

`Case: 1:21-cv-02666 Document #: 1-1 Filed: 05/17/21 Page 16 of 72 PageID #:26
`
`U.S. Patent
`
`Sep.24,2019
`
`Sheet 13 of 50
`
`US 10,423,658 B2
`
`FIG. 13
`
`Thumbnail I Table
`
`Last Name
`
`# People
`
`# Photos
`
`#Videos
`
`# Docs
`
`+ Alberts
`
`+ Annex
`
`+ Bade
`+ Bacon
`
`+ Bates
`
`+ Boone
`
`+ Danas
`
`+ Danes
`
`- Monk (All)
`
`I Monk,CJ
`~ Monk,Jack
`+ Firestone
`
`+ Moore
`
`+ Slythe
`+ Stein
`
`+ Testv
`
`2
`
`2
`
`3
`
`4
`
`5
`
`6
`
`7
`
`8
`
`2
`
`1
`
`1
`
`21
`
`1
`
`1
`
`2
`
`4
`
`8
`
`7
`
`8
`
`8
`
`7
`
`6
`
`5
`
`7
`
`499
`
`200
`
`199
`
`1249
`
`4
`
`9
`
`249
`
`788
`
`0
`
`0
`
`0
`
`0
`
`1
`
`2
`
`4
`
`3
`
`4
`
`2
`
`2
`
`17
`
`6
`
`0
`
`1
`
`2
`
`0
`
`0
`
`0
`
`0
`
`0
`
`2
`
`1
`
`2
`
`14
`
`7
`
`7
`
`39
`
`3
`
`9
`
`3
`
`12
`
`Petitioner Apple Inc. - Ex. 1046, p. 26
`
`

`

`Case: 1:21-cv-02666 Document #: 1-1 Filed: 05/17/21 Page 17 of 72 PageID #:27
`
`U.S. Patent
`
`Sep.24,2019
`
`Sheet 14 of 50
`
`US 10,423,658 B2
`
`FIG. 14
`
`Last Name
`
`Relationship
`
`# Photos # Videos
`
`# Docs
`
`Thumbnail I Table
`
`Alberts, John
`
`Killian, Jack
`
`Killian, Brian
`
`Killian, Kevin
`
`Killian, Sarah
`
`Killian, John
`
`Killian, Mark
`
`Killian, Louis
`
`Killian, John
`
`I Monk,CJ
`~ Monk,Jack
`Firestone, Mike
`
`Moore, Bertha
`
`Slvthe, Sarah
`
`Killian, John
`
`Killian, Mike
`
`Cousin
`
`Son
`
`Nephew
`
`Nephew
`
`Daughter-in-law
`
`Great Nephew
`
`Great Nephew
`
`Great Grandson
`
`Grandson
`
`Great Grandson
`
`Great Grandson
`
`Third Cousin
`
`Great Niece
`
`Sister
`
`Brother
`
`Brother
`
`8
`
`7
`
`8
`
`8
`
`7
`
`6
`
`5
`
`7
`
`499
`
`200
`
`199
`
`0
`
`0
`
`0
`
`0
`
`1
`
`2
`
`4
`
`3
`
`4
`
`2
`
`2
`
`1249
`
`17
`
`4
`
`9
`
`249
`
`788
`
`6
`
`0
`
`1
`
`2
`
`0
`
`0
`
`0
`
`0
`
`0
`
`2
`
`1
`
`2
`
`14
`
`7
`
`7
`
`39
`
`3
`
`9
`
`3
`
`12
`
`Petitioner Apple Inc. - Ex. 1046, p. 27
`
`

`

`Case: 1:21-cv-02666 Document #: 1-1 Filed: 05/17/21 Page 18 of 72 PageID #:28
`
`U.S. Patent
`
`Sep.24,2019
`
`Sheet 15 of 50
`
`US 10,423,658 B2
`
`FIG. 15
`
`Thumbnail I Table
`
`Location Name
`
`Address
`
`Citv
`
`State
`
`Countrv
`
`# Photos
`
`# Videos # Docs
`
`Dom
`
`Lucilla & Roberto
`
`Cologne
`
`Montalcino
`
`Lisle Home
`
`898 West St
`
`Lisle
`
`College
`
`545 Market
`
`Amazon Trio
`
`Akron
`
`Manus
`
`IL
`
`OH
`
`Cabin
`
`999 Pine
`
`Lake Geneva WI
`
`Grad School
`
`903 Plvmouth
`
`Charleston
`
`IL
`
`Germany
`
`ltalv
`
`USA
`
`USA
`
`Brazil
`
`USA
`
`USA
`
`USA
`
`3
`
`6
`
`45
`
`64
`
`235
`
`98
`
`1256
`
`12
`
`2
`
`1
`
`12
`
`2
`
`8
`
`2
`
`32
`
`0
`
`0
`
`0
`
`2
`
`0
`
`2
`
`0
`
`4
`
`0
`
`Griffith Park
`
`298 Glencarin
`
`Los Feliz
`
`LA Equestrian Ctr
`
`568 Horse Dr
`
`Glendale
`
`Del Coronado
`
`12 Coronado Dr Coronado
`
`CA
`
`CA
`
`CA
`
`Fenwav Park
`
`123 Yawke
`
`Boston
`
`MA
`
`Wrigley Field
`
`1190 W Addison
`
`Chicago
`
`Home
`
`444 Main
`
`Anvwhere
`
`GA Grill Party
`
`321 Silver
`
`Pike's Market
`
`786 Market
`
`Macon
`
`Seattle
`
`IL
`
`IL
`
`GA
`
`WA
`
`USA
`
`USA
`
`USA
`
`USA
`
`USA
`
`USA
`
`USA
`
`Raffels
`
`345 Fong
`
`Singapore
`
`Singapore
`
`4
`
`321
`
`57
`
`498
`
`4