a2) United States Patent
`US 10,423,658 B2
`(0) Patent No.:
`*Sep. 24, 2019
`(45) Date of Patent:
`Desmond etal.
`
`US010423658B2
`
`(54) METHOD AND APPARATUS FOR
`MANAGINGDIGITAL FILES
`
`(71) Applicant: MemoryWeb, LLC, Glen Ellyn, IL
`(US)
`
`(72)
`
`Inventors: Christopher J. Desmond, Glen Ellyn,
`IL (US); Nancy L. Desmond, Glen
`Ellyn, IL (US); L. Michael Taylor,
`Chicago, IL (US)
`
`(73) Assignee: NCM IP HOLDINGS, LLC, Glen
`Ellyn, IL (US)
`
`(*) Notice:
`
`Subject to any disclaimer, the term ofthis
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 518 days.
`
`This patent is subject to a terminal dis-
`claimer.
`
`(21) Appl. No.: 15/375,927
`
`(22)
`
`Filed:
`
`Dec. 12, 2016
`
`(65)
`
`Prior Publication Data
`
`US 2017/0091225 Al
`
`Mar. 30, 2017
`
`Related U.S. Application Data
`
`(63) Continuation of application No. 14/193,426, filed on
`Feb. 28, 2014, now Pat. No. 9,552,376, which is a
`(Continued)
`
`(51)
`
`Int. Cl.
`GO6F 16/90
`GO6F 16/58
`
`(2019.01)
`(2019.01)
`(Continued)
`
`(58) Field of Classification Search
`CPC ... GO6F 16/5866; GO6F 16/907; GO6F 16/901
`See application file for complete search history.
`
`(56)
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`5,694,514 A
`6,629,104 Bl
`
`12/1997 Evans
`9/2003 Parulski
`
`(Continued)
`
`OTHER PUBLICATIONS
`
`Kustanowitz et al., “Motivating Annotation for Personal Digital
`Photo Libraries: Lowering Barriers while Raising Incentives,” Tech.
`Report HCIL-2004-18, U. Maryland, 2005 (10 pages).
`(Continued)
`
`Primary Examiner — Loc Tran
`(74) Attorney, Agent, or Firm — Nixon Peabody LLP
`
`(57)
`
`ABSTRACT
`
`A computer-implemented method of associating digital tags
`with digital files comprises storing a plurality of digital files
`having embeddedtherein content data and metadata includ-
`ing tags; receiving, via a user interface device of a client
`device, a first tag label containing alphanumeric text created
`and inputted by a userofthe client device; modifying, using
`a controller device, a selected first one of the tags of the
`metadata in a first of the digital files to includethefirst tag
`label; receiving, via the user interface device or another user
`interface device, an instruction to searchfor all of the digital
`files havingat leastthe first tag label; responsive to receiving
`the instruction, automatically searching for all of the digital
`files having at least the first tag label; and displaying, on a
`video display device associated with the client device, a first
`indication ofthe first tag label.
`
`(52) U.S. Cl
`CPC veces GO6F 16/5866 (2019.01); GO6F 16/51
`(2019.01); GO6F 16/901 (2019.01); GO6F
`16/907 (2019.01); GO6F 3/0481 (2013.01)
`
`
`15 Claims, 50 Drawing Sheets
`
`&
`
`
`
`Select None
`Select All
`Delete Selected
`
`1 File(s} Selected
`File Attributes:
`Date:
`Location:
`People:
`Event:
`Comments:
`Keywords:
`Album:
`Star Ranking:
`Sharing Rights:
`Recipe:
`DocumentType:
`File Name:
`
`September2, 2010 change
`Yellowstone, MT, USA change
`Christopher Desmond, Nancy Desmond
`Fly Fishing in Yellowstone
`Add
`Add
`Add
`seoee
`Add
`Add
`Add
`IMG 123 change
`
`SAMSUNG 1001
`
`SAMSUNG 1001
`
`1
`
`

`

`US 10,423,658 B2
`Page 2
`
`Related U.S. Application Data
`
`continuation-in-part of application No. 13/157,214,
`filed on Jun. 9, 2011, now Pat. No. 9,098,531.
`
`(51)
`
`(56)
`
`Int. Cl.
`GO6F 16/51
`GO6F 16/901
`GO6F 16/907
`GOOF 3/0481
`
`(2019.01)
`(2019.01)
`(2019.01)
`(2013.01)
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`7,372,976 B2
`7,480,669 B2
`7,694,236 B2
`8,150,844 B2
`8,484,223 B2
`2006/0165380 Al*
`
`2007/0282908 Al
`2009/0113350 Al*
`
`5/2008 Rhoad
`1/2009 Lo
`4/2010 Gusmorino
`4/2012 Redstone
`7/2013 Ota
`7/2006 Tanaka........ GI1B 27/34
`386/227
`
`12/2007 Van Der Meulen
`4/2009 Hibino ou... GO6F 16/41
`715/853
`
`2009/0265631 Al
`
`10/2009 Sigurbjornsson
`
`OTHER PUBLICATIONS
`
`Miller et al., “Give and take: a study of consumer photo-sharing
`culture and practice,” CHI ’ 07 Proceedings of the SIGCHI Con-
`ference on Human Factors in Computing Systems, pp. 347-356,
`2007 (10 pages).
`Ameset al., “Why wetag: motivations for annotation in mobile and
`online media,” CHI ’07 Proceedings of the SIGCHI Conference on
`Human Factors in Computing Systems, pp. 971-980, ACM, 2007
`(10 pages).
`
`Yee et al., “Faceted Metadata for Image Search and Browsing”, CHI
`2003, pp. 401-408, 2003, ACM.
`Ferre, “CAMELIS: Organizing and Browsing a Personal Photo
`Collection with a Logical Information System”, Int. Conf. Concept
`Lattices and Their Applications, pp. 112-123, 2007, HAL.
`Tomasson et al., “PhotoCube: Effective and Efficient Multi-
`Dimensional Browsing of Personal Photo Collections”, ICMR ’11,
`2011, ACM.
`Bartolini et al., “Integrating Semantic and Visual Facets for Brows-
`ing Digital Photo Collections”, SBED, pp. 65-72, 2009.
`Trattner et al., “Evaluating Tag-Based Information Access in Image
`Collections”, Proceedings of the 23’ ACM Conference on Hyper-
`text and Social Media, pp. 113-122, 2012 ACM.
`Kang et al., “Capture, Annotate, Browse, Find, Share: Novel
`Interfaces for Personal Photo Management”, International Journal
`of Human-Computer Interaction, 23(3), pp. 315-337, 2007, Law-
`rence Eribaum Associates, Inc.
`Jaffe et al., “Generating Summaries and Visualization for Large
`Collections of GeoReferenced Photographs”, MIR’06, pp. 89-98,
`2006 ACM.
`Tornial et al., “Sharing, Discovering and Browsing Geotagged.
`Pictures on the Web”, 2007, Hewlett-Packard Development Com-
`pany, L.P., pp. 1-18.
`Snavely et al., “Photo Tourism: Exploring Photo Collection in 3D”,
`SIGGRAPH ’06 ACM Transactions on Graphics, vol. 25, Issue 3,
`pp. 835-846, 2006 ACM.
`Kisilevich et al., ““Event-based analysis of People’s Activities and.
`Behavior Using Flickr and Panoramio Geotagged Photo Collec-
`tions”, 14” International Conference Information Visualization, pp.
`289-296, 2010 IEEE.
`Ahern et al., “World Explorer: Visualizing Aggregate Data From
`Unstructured Text in Geo-Referenced Collections”, JCDL’07, pp.
`1-10, 2007, ACM.
`Kopf et al., “Deep photo: model-based photograph enhancement
`and viewing”, ACM Transactions on Graphics, vol. 27, No. 5,
`Article 116, Dec. 2008, ACM (10 pages).
`
`* cited by examiner
`
`2
`
`

`

`U.S. Patent
`
`Sep. 24, 2019
`
`Sheet 1 of 50
`
`US 10,423,658 B2
`
`
`
`FIG. 1
`
`Select None
`Select All
`Delete Selected
`
`IMG 123 change
`
`1 File(s) Selected
`File Attributes:
`September 2, 2010 change
`Date:
`Yellowstone, MT, USA change
`Location:
`Christopher Desmond, Nancy Desmond
`People:
`Fly Fishing in Yellowstone
`Event:
`Add
`Comments:
`Add
`Keywords:
`Add
`Album:
`eovc6e
`Star Ranking:
`Sharing Rights: Add
`Recipe:
`Add
`Document Type: Add
`File Name:
`
`3
`
`

`

`U.S. Patent
`
`Sep. 24, 2019
`
`Sheet 2 of 50
`
`US 10,423,658 B2
`
`FIG. 2
`
`Comments:
`Suzanne and Anthony’s Wedding Party where the cousins posed
`for a photo in the grass. Note, Jack with the lollipop and the
`photographerwith his shoe in the photo
`
`Location:
`Historical Society
`Lisle, IL 60532
`
`
`
`
`
`People:
`Jack Wong
`CJ Wong
`MaryFirestone
`Zoe Peika
`Nick Persons
`
`Event: Suzanne & Anthony’s Wedding Reception 2010
`
`Camera Details: more
`
`4
`
`

`

`U.S. Patent
`
`Sep. 24, 2019
`
`Sheet 3 of 50
`
`US 10,423,658 B2
`
`FIG. 3
`
`
`
`Thumbnail | Table
`
`Mary Bader and Clint Firestone
`Wedding October 11, 1945
`
`Capt. Clint Firestone
`1952
`
`GermanyTrip 1955
`
`Father’s Day 1966
`
`Christopher 1*
`Year 1972
`
`EIU College
`Graduation 1994
`
`.
`Cairo, Egypt 1996
`
`Europe
`Backpacking
`1996
`
`
`
`
` College Break 1992
`
` sae
`
`R
`Nancy James and
`Christopher Desmond
`Wedding October 17, 1998
`
`_ ;
`AmazonFishing Trip
`November 2009
`
`Cousins Party 2010
`
`Fishing in Yellowstone
`Montana September 2010
`
`5
`
`

`

`U.S. Patent
`
`Sep. 24, 2019
`
`Sheet 4 of 50
`
`US 10,423,658 B2
`
`FIG. 4
`
`Captions On | Off
`
`
`
`Christopher Desmond and Nancy Desmond Wedding — October 17, 1998
`
`6
`
`

`

`U.S. Patent
`
`Sep. 24, 2019
`
`Sheet 5 of 50
`
`US 10,423,658 B2
`
`FIG. 5
`
`7
`
`

`

`U.S. Patent
`
`Sep. 24, 2019
`
`Sheet 6 of 50
`
`US 10,423,658 B2
`
`FIG. 6
`
`
`
`Thumbnail | Table
`
`
`
`
`
`
`Suzanne Lucky
`
`Steve Fwah
`
`
`
`
`
` ARAETEARLEE
`
`SteveFire
`Bet i.
`
`
`:
`:
`ee
`3
`
`Steve Fire
`
`Joe Schmo
`tySSSSSSSSSSSSASAS
`NT
`
`SN
`Charlotte Killian
`
`ChrisJeep
`
`Peter LongHorn
`
`SN
`
`Ben Sherman
`
`Joe Schmo
`
`MaryVirginia
`
`Clint Dewitt
`
`Charlotte Killian
`
`8
`
`

`

`U.S. Patent
`
`Sep. 24, 2019
`
`Sheet 7 of 50
`
`US 10,423,658 B2
`
`
`
`FIG. 7
`
`SA SNaeAS
`
`
`
`Clinton Dewitt Firestone IV
`
`July 12, 1896
`Birth:
`April 29, 1971
`Death:
`Parents: Clinton Dewitt Firestone Ill and Viola Miller
`Comments: He was a WWII U.S. Air force pilot and POW in WWII and veteran
`honorably discharged in December of 1947. He worked for 44 years for the
`Firestone Tire and Rubber Companyin retail, wholesale and original equipment
`sales, marketing and management. He was bornin Akron, OH and is buried in
`Columbiana, OH.
`
`Edit bio
`
`Locations
`
`Timeline
`
`Family Tree
`
`Recipes
`
`9
`
`

`

`U.S. Patent
`
`Sep. 24, 2019
`
`Sheet 8 of 50
`
`US 10,423,658 B2
`
`FIG. 8
`
`
`
`Descendantsof Charlotte Killian Halter
`
`Tree | Table
`
`WTS,
`
`
`
`HERS
`
`-
`
`/
`
`
`
`EX nee FO EN
`ANAS piggy & ON
`
`> Powwow SANG
`pes
`S
`
`
`NS
`SCC gay
`SSRO
`
`CSSR
`
`Weerng
`
`
`
`
`NX EN
`FENN
`
`SESS RISKS RAYS
`
`AY SENAY SSE
`
`AFSSSN
`
`10
`
`10
`
`

`

`U.S. Patent
`
`Sep. 24, 2019
`
`Sheet 9 of 50
`
`US 10,423,658 B2
`
`
`
`FIG. 9
`
`Year
`Month
`
`:
`
`°
`
`"
`
`~
`
`es
`
`*
`
`°
`
`
`
`DayHeeeeecrc
`
` Cousins Party 2010
`
`
`Fishing in Yellowstone
`Montana September 2010
`
`11
`
`11
`
`

`

`U.S. Patent
`
`Sep. 24, 2019
`
`Sheet 10 of 50
`
`US 10,423,658 B2
`
`FIG. 10
`
` *
`
`1 Zucchini (chopped)
`Thai Basil (8 leaves)
`*
`Lemon Grass(in jar) 1 teaspoon
`*
`* Chinese Ginger Root(in jar) 1 teaspoon
`
`Rice
`*
`Thai Rice (something that only takes 2 cups of water)
`*
`Dice chicken in bowl and add twotablespoons offish
`sauce. Let marinate for 20 minutes.
`
`*
`
`Take thick part of coconut milk out into pan (about 4
`tablespoons), Curry paste, 1 spoon of lemongrass, 1
`.
`.
`spoonofginger and garlic. Heat over high with boil and
`THEN stir for 1 minute. Add meat (uncooked) and fry
`until cooked over high heat
`¢ Add milk, brown sugar and salt. Bring back to slight boil
`and constantly stir. Add veggies and soy sauce. Cook for
`about 10-14 minutes COVERED until veggies are cooked.
`Serve witha smile.
`
`Chef: Barry Desmond
`
`Video on How to MakeIt
`
`12
`
`Original Handwritten Recipe
`
` *
`
`Desmond's Yellow Thai Chicken Curry
`
`c
`*
`
`urry Mix
`.
`Coconutmilk (400 ml) - DO NOT SHAKE IT UP
`800 gram ofchicken (4 chicken breast)
`Fish sauce (Nam Pla) Thai Bamboo Garden — Bottle
`*
`* Garlic (2 cloves)
`*
`Broccoli { 2cups chopped)
`*
`2 Peppers (chopped)
`*
`2Carrots (chopped)
`
`12
`
`

`

`U.S. Patent
`
`Sep. 24, 2019
`
`Sheet 11 of 50
`
`US 10,423,658 B2
`
`
`
`FIG. 11
`
`Thumbnail | Table
`
`i v
`
`
`
`egeSanh
`Alburn/Event
`Hack Monk's Arrival|26-Dec-2003|chicago|oo|4|4|
`MikeTesty'sfirst Birthday|13-Sep-1983|Minneapolismn|54|21|0
`Cubs Beat Cards 1998
`
`
`Jack Monk's Arrival Chicago,IL|6|4[|lg|29-Dec-2003
`
`
`Mike Testy's 2nd Birthda 13-Sep-1983_}|Minneapolis, MNfoose|2|lo
`Cubs Beat Cards 1998
`5-Aug-1998
`Chicago,IL
`2,199
`Jack Monk's Arrival|29-Dec-2003|Chicago,|69
`
`
`MikeTesty's3rdBirthday|13-Sep-1983|Minneapolismn|sa|21|oo
`
`
`2iuki978_|stiovismo[|76|2|o| 333333333333333333333333333333333333333333333333.3yLy¥$.$
`
`
`
`lackWrigleyMonk'sArrival|29-Dec-2003|chicago,n_|69|4|4
`
`
`MikeTesty's4thBirthday|13-Sep-1983|Minneapolismn|sa|21|oo
`
`
`
`NancyLeasHowtoRideaBike|
`
`13
`
`13
`
`

`

`U.S. Patent
`
`Sep. 24, 2019
`
`Sheet 12 of 50
`
`US 10,423,658 B2
`
`
`
`FIG. 12
`
`Thumbnail | Table
`
`4&
`
`
`
`
`Ww Ss=> 2 =a>a wo zoowci]aToOwn
`
`
`©l=TsloS12l=ls©|=|g[2JEIgfe{Es|gnananaBie|=|FIs|eIFIsIzolal/s|[oJaIE[oeIA[E2|e|SIf|e|12|eIS=1212[612[kz[4[2JzBye[2|Ofw[21Dlw|eSsJw[>[a[I~[SF[5[am|SaIf1/5IaIFIsJaJ=15wjo121%Ja2[%14[2bbboIZ[2JoJZ12IoJo[&©|3©IA©|So[>ooJS |oaaay33 he
`
`
`Album/Event
`Location
`:|26-Dec-2003|chicagon_|69|4|4|
`
`'FirstBi|13-Sep-1983|Minneapolismn|sa|21|oo
`
`[29-Dec-2003|chicago_|69|4|4|
`[13-Sep-1983|Minneapolismn}54|21|oo_|
`
`:|29-dec-2003|chicago|oo|4|4|
`!|13-Sep-1983|Minneapotismn|5a|a1|on
`
`=@5~ = o = °5 q w >P==2
`ssArrival|29-Dec-2003|chicagow_|69[|4|4|
`= nao 4oWw4
`|13-Sep-1983|inneapotismn|5a|a1|on
`nN
`®16woaul 2310cex=laOwn
`=15+16216 2Qa o wo 2 nao
`ey¥$tz
`ideatike] 2iwuti97e|sttowismo|76[2|o|
`
`
`® —
`
`14
`
`14
`
`

`

`U.S. Patent
`
`Sep. 24, 2019
`
`Sheet 13 of 50
`
`US 10,423,658 B2
`
`FIG. 13
`
`Thumbnail | Table
`
`eS
`
`
`
`7
`
`ps
`
`15
`
`15
`
`

`

`U.S. Patent
`
`Sep. 24, 2019
`
`Sheet 14 of 50
`
`US 10,423,658 B2
`
`
`
`FIG. 14
`
`Thumbnail | Table
`
`[iitiantack|son||| ee
`
`
`[KilianBrian|Neprew|os|oo
`[kitian,Kevin|Neprew|os|oo
`[KitianSarah_|aughter-intow|7|1|oo|
`
`
`[Kilianohn|GreatNephew|6|2|2|
`
`
`[Kitian,Mark|GreatNephew|5|4|1|
`
`
`
`
`
`[Kilianjohn|Grandson|a9|a|ta
`
`
`lanke|ater[ree|2|a PODCOOOOOOLOLOLOOLELOLODLOLOLODLELELODLOLELODLELELODLOLELODDEDIDLLo
`
`bs Monk,Jack|GreatGrandson|19|2|7|
`
`Fete.tle7turdtoninfgfhre
`
`Moore, Bertha
`Slythe, Sarah
`Killian, John
`
`Great Niece
`Sister
`Brother
`
`16
`
`16
`
`

`

`U.S. Patent
`
`Sep. 24, 2019
`
`Sheet 15 of 50
`
`US 10,423,658 B2
`
`
`
`FIG. 15
`
`Thumbnail | Table
`
`pom|cotogne||Gerrmany|3|2||
`jucilaspoverto|_|Montatinof|tty|6fa|o|
`Lisleome|_goewestst_|tite|uw|usa|45|22|2|
`cotege|545Market_|Akron|on|usa|6a|2|o|
`mazontrip||Manus||erat|235|g|
`cabin|999PineflakeGeneval_wi|usa|os|2|o|
`radschool__|903plymouth|charleston]uw|usa|1256|32|a|
`
`critpark|298Glencarin|tosretz|ca|usa|a2|o|o|
`LAtquestrianctr|s68HorseDr_|Glendale|ca|usa[|4|a|o|
`Delcoronado_|12CoronadoDr|Coronado|_ca|usa[|321|a|o|
`
`e¥$tz
`
`4& a
`
`Home|__aaamain|anywhere|uw|usa|10987_|a9|9|
`caGritParty|_32asiiver_|Macon|ca|usa|25|o_|
`
`Pike'sMarket__|786Market_|seattle|wa|usa|is|a|o|
`fate[atsong—LSngagos||Sgn|2»P210
`
`17
`
`17
`
`

`

`Category | Card | Table~
`
`FIG. 16
`
`
`
`accactsatsdccddcadesdceddcddcadesdceddcddcadeadeeddcadeades
`
`U.S. Patent
`
`Sep. 24, 2019
`
`Sheet 16 of 50
`
`US 10,423,658 B2
`
`18
`
`18
`
`

`

`U.S. Patent
`
`Sep. 24, 2019
`
`Sheet 17 of 50
`
`US 10,423,658 B2
`
`FIG. 17
`
`
`
`
`
`Close (X)
`
`19
`
`19
`
`

`

`U.S. Patent
`
`Sep. 24, 2019
`
`Sheet 18 of 50
`
`US 10,423,658 B2
`
`
`
`FIG. 18
`
`
`
`
`
`AdvancedSearchFilter
`
`Keywords:
`Date:
`Location:
`
`Add
`Add
`
`Chicago, IL USA
`Yellowstone, MT USA
`Cologne, Germany
`
`Mike Cubbie
`Mary Lamb
`Christopher Monk
`Nancy Monk
`Dwight Schrut
`
`People:
`
`Event:
`
`Jack’s 1% Birthday
`Fly Fishing in Yellowstone
`Raking Leaves
`Christmas 2010
`Thanksgiving 2010
`July 4% Parade
`Add
`Album:
`Add
`Star Ranking:
`Sharing Rights: Add
`Document Type: Add
`
`20
`
`20
`
`

`

`U.S. Patent
`
`Sep. 24, 2019
`
`Sheet 19 of 50
`
`US 10,423,658 B2
`
`
`
`FIG. 19
`
`Captain Phil’s Memory-Webb
`
`Welcome, Captain Phil
`Last Login:
`11.18.2010
`
`Media
`# Photos
`# Videos
`# Documents
`
`Count
`1,342
`75
`
`173
`
`Count
`80% complete
`61% complete
`
`My recent memories:
`*
`123 Photos uploaded on 11.07.10
`*
` 2albums created 11.17.10
`PeopleStats:
`*
`12 visitors since last login date
`*
`123 Photos uploaded on 11.07.10
`Lastname|People|photos|videos|aDocs_|
`*
` 2albums created 11.17.10
`
`
`
`My recent Webb views:
`* Captain Phil 2010 (photo album)
`*
`Chicken PotPie (recipe)
`* Captain Phil (Timeline)
`
`
`St. Louis, MO
`
`EventStats:
`
`Updatesand Alerts:
`*
`License renewal due 1.15.2011
`vent|ate|tocation|#Media|
`Mike Testy’s 1%t Birthda 13-Sept-1988|Minneapolis, MN
`
`Cubs Beat Cards Aug 1998
`5-Aug-1998
`Chicago,IL
`2,199
`Nancy Learnsto Ride Bike
`21-July-1978
`
`
`
`21
`
`21
`
`

`

`U.S. Patent
`
`Sep. 24, 2019
`
`Sheet 20 of 50
`
`US 10,423,658 B2
`
`FIG. 20
`
`0050
`
`0501
`
`f
`
`4101
`
`1201
`
`Original Digital
`
`Third Party Facial
`
`file(s) upload to
`
`Recognition
`
`
`MemoryWeb
`Provider
`
`
`
`
`Third Party
`Geographical
`Mapping Provider
`
`CUESLnes eees
`Inside MW System
`
`Individual /
`Group of
`People
`
`MW Modified File
`exported from
`MemoryWeb to Storage
`System.
`
`
`Third Party Media Provider
`
`
`
`Share to Social
`Third Party Media
`Exchange) Third Party Facial
`NetworkProvider
`
`
`
`6300
`System (Data
`Recognition
`
`System
`System
`
`
`
`
`
`User Relationship
`
`System Reading
`System Interpreting and
`Table
`
`Adding Data to
`Phase
`
`Relationship Table Phase
`System Indexing
`
`
`Digital Files and
`Tag Data Blocks
`
`for a DigitalFile
`within User
`
`
`Relationship Table
`
`Application Export System
`
`0500
`
`
`
`4100
`
`Third Party Geographical
`Location Mapping
`System
`
`Share to in
`System
`
`Application Digital
`
`
`Tag Organizer
`System
`
`0600
`
`Continuous Link
`of Application
`
`0700
`
`Bot-Tags System
`
`Advanced Filters
`System
`0800
`
`Keyword Fast
`Search System
`9900
`
`22
`
`22
`
`

`

`U.S. Patent
`
`Sep. 24, 2019
`
`Sheet 21 of 50
`
`US 10,423,658 B2
`
`0100
`
`0200
`
`FIG. 21
`
`weeee eee nee eee ee eeeee
`System Reading Phase
`of
`Systern Interpreting and Adding Data to Relationship Table Phase
`
`{
`
`e104
`0401
`i
`ss
`een
`i
`Third Party Facial
`'
`
`Recognition Provider
`OriginalDigital file begins to
`{
`upload to MemoryWeb
`i
`Outside MW System
`
`inside MW System
`0400
`Third Party Facial Recagnition
`System
`
`I'I
`'''1
`
`
`t''III
`
`Upload original
`'t
`0201
`file into MW
`Locate EXIF
`''
`
`Image File
`Readand iterate through
`:
`Directories in
`
`each EXIF tag item
`vipat
`By
`'
`+
`Create duplicates
`'
`
`Digital File
`\
`oforiginal file in
`Storage of
`!
`1
`different file sizes
`Original File
`t
`
`i
`t
`}
`Save EXIF data to
`7
`"
`i
`User Relationship
`E
`
`!
`Table for Digital
`DuplicateFile
`'
`
`Small
`I
`File
`Duplicate File
`i
`'
`!
`
`
`
`Parse EXIF
`1
`DuplicateFile
`'
`
`data into
`:
`t
`Duplicate File
`MemoryWeb
`bee Re mm ee em em
`Create new
`
`
`
`EXIF Tag (Data
`
`MW tag
`
`
`
`data
`Blocks)
`
`
`Microsoft
`Look up MW
`
`
`
`
`
`tag data
`exists?
`Wa
`Windows Tag
`sssociate
`Data Blocks.
`
`
`existing Tag.
`‘MemoryWeb
`
`
`gata
`Tag (Data Blocks}
`
`MemoryWeb
`
`Person Tag (Data
`Blocks)
`
`
`System Indexing Digital Files and
`Tag Data Blocks for a Digital File
`within User Relationship Table
`3800
`
`H
`
`Outside MW System
`Inside MW System
`
`
`
`23
`
`
`
`
`0102
`
`Duplicate
`File?
`
`
`File not
`
`uploaded
`
`104
`
`
`II
`ttI
`IIIII
`
`t t
`
`23
`
`

`

`U.S. Patent
`
`Sep. 24, 2019
`
`Sheet 22 of 50
`
`US 10,423,658 B2
`
`FIG. 22
`
`0320
`
`EXIF Tags version 2.3 Image File Directories Zé
`0322
`(Data Blocks)
`
`Group Name
`Location
`OxScSb or
`Ox010e
`
`Tag Labels
`
`Description Title
`
`FDO
`
`MemoryWebTag (Data Blocks)
`
`0360
`
`MemoryWeb Tag
`
`IMediaAsset.Caption
`
`0362
`
`
`
`0324
`
` 0361
`
`
`
`
`
`0325
`MediaAssetStarRanking
`
`
`pesrpvonsatnga
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`0326
`
`rginowercouresTa
`
`0334
`
`0335
`
`iMediaAsset,DateCreated
`
`0364
`
`
`
`
`
`
`imageimage'o,oimension,widthHeightee)|utile
`
`
`0327
`Medi
`Wi
`diaAsset. Width
`fgoes|
`0328
`IMediaAsset Height
`feigntees|
`
`dvoncenPhoto(LensMokertensModelee)|tiple
`
`
`
`
`
`his is used to inject inforrnation that do not currently have
` 0367
`0329
`EXIF standardized tags including Collection, People, Location
`Name, Recipe Name, Person Tag Data Blocks (0380}, etc.
`
`
`
`
`
`
`po
`
`
`
`
`0368
`0369
`0370
`0372
`
`
`
`
`
`
`
`
`
`
`GPS Altitude
`orsavnsenssons
`
`0330
`0331
`0332
`0333
`
`EXifiFD
`
`Ox9286
`
`GPS Longitude
`
`GPS
`
`10x0004
`
`24
`
`24
`
`

`

`U.S. Patent
`
`Sep. 24, 2019
`
`Sheet 23 of 50
`
`US 10,423,658 B2
`
`FIG. 23
`
`0340
`
`0360
`
`0342
`
`0343
`
`Windows Imaging Component (WIC) Metadata
`(Data Blocks)
`
`7 Ol
`
`Owner, Computer, etc’
`
`.
`File Name
`
`File Item Type
`
`File Folder Path
`
`File Date Created
`
`File Date Modified
`
`File Size
`
`File Attributes
`File (Offline availability, Offline status, Shared with,
`
`MemoryWebTag (Data Blocks)
`
` 0361
`
`0372
`
`0373
`
`MemoryWebTag
`
`MediaAsset. Filename
`
`MediaAsset.UploadBatch.BatchName
`
`
`
`
`
`25
`
`25
`
`

`

`U.S. Patent
`
`Sep. 24, 2019
`
`Sheet 24 of 50
`
`US 10,423,658 B2
`
`FIG. 24
`
`0380
`
`MemoryWebPerson Tag (Data Blocks)
`
`[SS
`
`0395
`
`0381
`0382
`
`0383
`0384
`0385
`0386
`0387
`
`0388
`0389
`0390
`0391
`0392
`0393
`0394
`0396
`
`26
`
`26
`
`

`

`U.S. Patent
`
`Sep. 24, 2019
`
`Sheet 25 of 50
`
`US 10,423,658 B2
`
`FIG. 25
`
`
`
`Third Party Faciai Recognition Provider
`
`0402
`
`
`
`0400
`
`Outside MW System
`
`
`System parses
`identified faces
`System receives
`
`
`
`response from
`and corresponding
`
`Third Party Facial
`
`
`Facial Recognition
` System request for
`System sends
`
`
`
`
`Digital Files to
`Recognition
`data for each
`_thumbnaiis of all
`
`
`
`
`Digital Files that have
`Provider
`Digital File
`Third Party
`unconfirmed faces.
`Facial
`
`
`Recognition
`
` Provider
`
`System records and associates specific
`
`
`
`
`face for Digital File with a MemoryWeb
`
`
`Is Face Recognition
`Existing {Trained} Face ID
`Person ID and sends to MemoryWeb
`
`
`above score
`
`Relationship
`in User Relationship
`
`Person Tag (Data Blocks) in User
`
`threshold?
`Table
`Retrieves
`Table?
`Relationship Table
`
`Thumbnails of aff
`Digital Files that
`have unconfirmed
`
`Inside MW System
`
`faces
`
`System generates virtual
`System generates facial
`
`unconfirmed facial
`recognition record for unknown
`recognition record for person
`person and then sends
`
`information to MemoryWeb
`and then sendsinformation to
`Person Tag (Data Blocks)in User
`MemoryWeb Person Tag
`
`Relationship Table
`(Data Blocks) in User
`
`Relationship Table
`
`
`27
`
`27
`
`

`

`U.S. Patent
`
`Sep. 24, 2019
`
`Sheet 26 of 50
`
`US 10,423,658 B2
`
`FIG. 26
`
`Third Party Media Pra’
`
`0561
`
`Outside MW System
`
`
`9504
`
`0500
`
`1000
`
`
`
`Share to Third Party Media
`
`Provider System
`
`Inside MW System
`Systems sends
`registration
`
`0503
`
`confirmation 9502
`
`User selects sharing of
`their information with
`Third Party Media
`Provider with
`
`Systemsreceives
`registration
`confirmation, Third
`Party Media ID
`
`System sends
`registration
`information to
`UserSetting
`Table?
`
`System requests contact
`names and IDs from Third
`Party Media Provider
`(daily)
`
`9300
`
`0506
`
`System receives contact names
`and IDs from Third Party Media
`Provider {daify)
`
`0808
`
`
`Is Persona
`
`
`Nes
`
`0510
`
`System add new personto Tag
`Data Block of Relationship
`Table for DigitalFile
`
`0509
`
`System associates
`existing Person to Tag
`Data Block of
`Relationship Table for
`Digital File
`
`User Relationship
`Table
`
`
`Match, No
`
`
`
`MemoryWeb
`
`
`UserSettings
`Table
`
`28
`
`28
`
`

`

`U.S. Patent
`
`Sep. 24, 2019
`
`Sheet 27 of 50
`
`US 10,423,658 B2
`
`FIG. 27
`
`1900
`
`1901
`
`1902
`1903
`1904
`
`1905
`1906
`1907
`
`1908
`1909
`1910
`1911
`1912
`1913
`
`Item
`
`User’s Name
`
`Payment ID
`Password
`
`Payment Date
`
`Account Type
`User’s email
`
`Language preference
`Date format
`Email notifications
`
`Contacts (with third Party Social Media)
`Facebook ID
`API Token
`
`29
`
`29
`
`

`

`U.S. Patent
`
`Sep. 24, 2019
`
`Sheet 28 of 50
`
`US 10,423,658 B2
`
`FIG. 28
`
`Userselects
`Digital File in
`MW
`
`0601
`
`0600
`
`f
`
`
`
`>
`
`6300
`
`|
`
`30
`
`0602
`0608
` System sends
`Continuous Link
`User gelects
`
`
`of Application
`Application Dot-
`Tag
`Dot-Tag System
`
`0610
`
`
`
`System associates
`
`
`
`user selects
`
`
`new MemoryWeb tag
`to Tag Data Block of
`ser erect
`System will
`matching
`System retrieves
`
`
`
`Userselects add
`MemoryWebTag
`Relationship Table for
`display the
`Digital File and
`
`age
`Digital File
`Digital File and
`ieital
`Fil
`
`
`
`fora
`qT
`?
`Digital File Tag
`‘ag data exists?
`the
`
`
`MemoryWeb
`System creates new
`Data Blocks
`correspondiny
`
`
`
`from User
`Tag
`Digitatrite Toe
`User adds
`MemoryWebtag to nt
`
`
`Relationship
`Data Blocks in
`New MemoryWeb
`Tag Data Block of
`| gaa
`
`
`Table
`Relationship Table for
`the form of
`soteat
`Ci
`
`
`en
`Digital File
`
`
`Application Dot-
`
`
`System associates
`Tags
`Userselects
`
`
`existing MemoryWeb
`
`matching
`User selects edit
`Tag to Tag Data Block
`
`MemoryWeb Tag
`
`fora
`
`of Relationship Table
`
`MemoryWeb
`for Digital Fite
`Tag data exists?
`
`
`Application Bot-
`Syst
`"
`
`
` MemoryWebtag to
`Toe ystem creates new=aySTIR
`
`
`New MemorywWeb
`Tag
`Tag Data Block of
`0617
`
`Relationship Table for r
`
`User selects
`Digital File
`
`
`dsiste
`System deletes
`0619
`
`
`association of
`‘ora
`
`
`MemoryWebtag to
`MemoryWeb
`
`Tag Data Block of
`Application Dot-
`Tag
`
`
`Relationship Table for
`Digital
`File
`
`
`
`request to
`Relationship
`Table for specific
`Digital File
`
`User
`Relationship
`Table
`
`
`
`
`30
`
`

`

`U.S. Patent
`
`Sep. 24, 2019
`
`Sheet 29 of 50
`
`US 10,423,658 B2
`
`FIG. 29
`
`Structure
`
`0650
`
`
` peheg User Defined Tag Label
`
`Examples
`
`Within Character Umit for Lebals end Numbers
`Cologne Germany
`
`0654
`
`Exceeds Character Limit for Label and Murahers
`
`§ Gee’
`
`
`
`Holiday Photosfrom ...
`
` \ 0655
`
`31
`
`Dotted Application Dot-Tag denotespartial relationship.
`this example, personis a half-sibling to another person.
`~
`'1'it1'1tiiiT'1'1'i'1'ii1'i'ii1i1‘1'1'111'i'1'i'1'1'1'i'i'1'ix”
`gs Frank Smith
`!
`
`In
`
`31
`
`

`

`U.S. Patent
`
`Sep. 24, 2019
`
`Sheet 30 of 50
`
`US 10,423,658 B2
`
`FIG. 30
`
`
`
`
`
`Userselects
`Application Det-
`Tag
`
`Displays image
`in Slideshow
`view
`
`is Person
`lf Person Application Dot-Tag, system retrieves data
`
`
`
`for that person from User's User Relationship Table
`Dot-Tag?.
`
`
`
`anddisplays relationship data in People Profile View
`
`If Collection Application Dot-Tag, system retrieves
`Yes
`
`data from that collection from User's User
`Relationship Table and displays relationship data in
`
`Collection View
`
`is mage
`
`
`Selected?
`
`System sends
`If Location Application Dot-Tag, system retrieves data
`
`
`
`Is Location
`request to
`
`
`from that Location from User's User Relationship Table
`
`No
`
`and displays relationship data in individual Location
`Dot-Tag?,
`Relationship
`User
`.
`
`
`
`View
`-
`Table
`Relationship
`Table
`
`
`
`
`Userselects
`
`
`Application Dot-
` if date Application Dot-Tag, system retrieves data
`
`Tag
`from that date from User's UserRelationship Tabie
`
`and displays relationship data in Uploads View
`
`Table and displays relationship data in Recipe View
`
`0715
`
`0717
`
`ls Recipe
`
`Dot-Tagy
`
`If Recipe Application Dot-Tag, system retrieves data
`from that Recipe from User’s User Relationship
`
`
`
`
`32
`
`32
`
`

`

`U.S. Patent
`
`Sep. 24, 2019
`
`Sheet 31 of 50
`
`US 10,423,658 B2
`
`FIG. 31
`
`0750
`
`
`
`Share: mA Si“0761
`o760 ”
`cet
`ae
`
`Lg edit
`Bs. o759
`#
`
`0764bn
`HG incase Soran see ‘
`
`0763
`
`0780
`O765
`
`ore
`$B 3298203 A 766,
`Ro Letataitde767
`0768
`
`“0778
`
`
`Jackson and JC’s First Day of School
`No0755
`
`Media Type! MAMRERE mm 0776
`Orightal Fike Detatk
`
`33
`
`33
`
`

`

`U.S. Patent
`
`Sep. 24, 2019
`
`Sheet 32 of 50
`
`US 10,423,658 B2
`
`Multiple People Application View
`
`Lieaemme
`
`Locations Recipe
`
`FIG. 32
`
`2450
`
`4454
`
`1402
`
`FamilyTree
`
`ApplyFilters J i400
`
`[yl] ~~ 1402 ~~
`items Per Page:
`
`Sort By:|Newest to Oldest
`
`Jon Smith
`
`Jane Smith (Doe)
`
`Page:
`‘age
`Single People Profile Application View
`Uploads Collections
`Figgegmm
`Locations
`Recipe
`Family Tree
`
`Ley
`Jackson Smith
`
`AMAL
`_
`dC Jon Smith
`
`1430
`
`J
`Apply Filters
`
`Fast Search
`
`< View all People
`Ss 1432
`
`1453
`1454
`
`1448
`
`4449
`
`1431—JC Jon Smith
`.
`1433—eNicknames: Chip
`1434—-Born: (1/1/2004)
`1435—— Parents:(@lonsmith‘Jand
`1436-— Siblings:
`1437—Children:
`1438-Biogra phy: JC was namedafterhis great, great grandfather from German who's reali name was Johan Christoph.
`1452
`1439
`JCTags:
`Sf 4441 / 1442 fi Van
`J 1445
`Cia2 Collections
`2Facial Recognitions
`3Locations
`7 Family Relationships
`1. Recipe
`
`ChangeProfile Photo
`1440
`
`34
`
`34
`
`

`

`U.S. Patent
`
`Sep. 24, 2019
`
`Sheet 33 of 50
`
`US 10,423,658 B2
`
`a
`1501
`Multiple Collection View
`Uploads
`fieliGewuem™
`People
`Locations Recipe
`
`Family Tree
`
`Fast Search
`
`Apply Filters
`
`FIG. 33
`
`Sort By:
`
`our Fans
`
`[Newest to Oldest [| ~~ 1502 1500
`
`Smith Family
`Photos
`
`First Day of
`School
`
`Jackson and JC
`Photos 2013
`
`Baseball Games
`
`Individual Collection View
`Uploads
`MesiGarenimmm
`People
`Locations Recipe
`
`Family Tree
`
`< View all Collections — 1531
`
`Smith Family Photos —~ 5°?
`Owe Farad.|
`
`QO
`
`
`
`items Per Page: Ei
`
`35
`
`35
`
`

`

`U.S. Patent
`
`Sep. 24, 2019
`
`Sheet 34 of 50
`
`US 10,423,658 B2
`
`Multiple Location Application View
`Uploads Collections
`
`FIG. 34
`
`1605
`
`Fast Search
`
`Apply Filters
`
`1600
`
`
`
`Sort By:|Newest to Oldest yl] ——— 1606
`
`Abe Lincoin
`Elementary School
`
`Home Sweet
`Hame
`
`Stonehenge
`9
`items Per Page:
`
`1630
`
`Uploads Collections
`
`People MRIS Recipe
`
`Family Tree
`
`< ViewallLocations —— 4633
`
`Wrigley Field
`3651 N. Clark St. “~~ 1632
`Chicago illinois, United States
`
`1633
`
`YY
`1 Photos
`
`Sharethis Cplection:
`
`1636
`
`Apply Filters
`
`ai
`
`1637
`
`People Pecrecipe Family Tree
`
`items Per Page: Ea
`
`36
`
`36
`
`

`

`U.S. Patent
`
`Sep. 24, 2019
`
`Sheet 35 of 50
`
`US 10,423,658 B2
`
`FIG. 35
`
`1700
`
`Fast Search
`
`ApplyFilters
`
`1733 “ 1714
`1712
`-
`:
`ES
`&
`t Non
`aSelicted:
`Safeer Au
`Select None
`Es ums
`
`
`
`People —— 1716
`3733
`veseeeeveee
`
`Rt
`
`b
`
`
`
`
`
`Neer
`
`collections
`
`People
`
`Locations Recipe
`
`Family Tree
`
`1701
`“
`1702
`
`s.
`SB ve
`CUrrert peage25 ut
`1703
`serre|_ TsRebcmamme(Teh —
`
`File Folder: C:/2013/Family Fun
`
`1704
`
`1708
`1737
`
`4741
`
` 1707)
`
`
`
`P72do Acheok NSE
`
`1722-— tonaiony
`
`
` 1727mpm (IP aSs00 804 IC Ssoros
`
`1729stee ~ AS. Cosare Caton
`
`B vosenar
`
`1739
`owe: EI
`Be
`9 1738-7 emsPer Page:|50|
`
`
`37
`
`37
`
`

`

`U.S. Patent
`
`Sep. 24, 2019
`
`Sheet 36 of 50
`
`US 10,423,658 B2
`
`
`
`
`FIG. 36
`
`1800
`
`4813!
`
`. a 1809
`Ingredients
`
`
`
`Q) 1 Stick of butter
`(2) 1 cup offlour
`0% teaspoonof vanilla
`(J 1 cup of grand pecans
`2 tablespoons of sugar
`
`Directions 7 °°
`
`Mix all. Roll gently into 1 inch balls. 300
`
`and immediately roll in sugar.
`
`1811
`
`Comments
`
`This is one of my favorite recipes around the holidays and was
`one of Grandma Lori’s best!
`
`
`
`
`
`
`Uploads Collections
`People
`Locations
`iggehtammramily Tree
`Fast Search
`Apply Filters
`
`« Show all Recipes ~ **”
`Lori Doe Pecan Puffs~ "leon: Hd i
`1812!
`
`
`
`
`
`
`
`
`
`us
`
`Chet
`
`
`
`1816
`Lori Doe.
`+add/edit Chef ~
`ecipe
`otos
`Recipe Photos
`resent
`Poutree.
`ifioum
`
`
`3 -
`
`snannnnnnnannanantanannananaas aso?
`2:
`
`
`
`
`
`
`
`
`NS
`
`38
`
`38
`
`

`

`U.S. Patent
`
`Sep. 24, 2019
`
`Sheet 37 of 50
`
`US 10,423,658 B2
`
`FIG. 37
`
`0800
`
`User selects
`
`Userdesires
`to add
`
`0805
`
`9806,
`
`0807
`
`of Application
`apply if in People view, system retrieves data for applied
`displays relationship data 0812
`
`0803
`System sends
`request
`0804
`
`System
`produces
`
`0808
`System sends
`request
`
`ContinuousLink
`
`Dot-Tags System
`0700
`
`0814
`
`filter(s) from User's User Relationship Table and
`
`39
`
`AdvancedFilter
`
`
`
`
`
`
`
`additional
`ifin Uploadsview, system retrieves data for applied
`0802
`
`Usertypes
`filter(s) from User’s User Relationship Table and
`Advanced
`
`information inte
`Filter
`
`
`displays relationship data
`
`the text box
`
`
`
`
`
`
`
`User
`Relationship
`if in Collections view, system retrieves data for
`request
`Table
`
`
`applied filter(s) from User’s User Relationship Table
`5
`i
`i
`
`
`
`and displays relationship data
`
`
`0841
`i
`.
`:
`.
`.
`if in Locations view, system retrieves data for applied
`filter(s) from User's User Relationship Table and
`displays relationship data
`
`System displays
`possible
`atches of
`mat
`fi
`
`Userselects
`filter(s)
`
`
`
`User selects
`
`39
`
`

`

`U.S. Patent
`
`Sep. 24, 2019
`
`Sheet 38 of 50
`
`US 10,423,658 B2
`
`FIG. 38
`
`0831
`
`0830
`
`
`Apply Filters...
`
`
`Person: Jon Smith
`Person: Jane Smith
`
`
`Location: Smithsonian
`
`
`
`
`0839
`
`0837
`
`Available Fitters
`
`0834
`
`
`
`40
`
`
`
`Collection: Smithsonian Trip
`0833
`
`
`
`Apply Multiple Filters
`Rearad tor Fiters ueAde;
`
`Apply Multiple Filters
`
`0838
`
`Bearnh for Piters 10 Ade:
`Available Filters
`
`Smith
`
`Currant Active PHters
`
`
`
`
`
`
`
`40
`
`

`

`U.S. Patent
`
`Sep. 24, 2019
`
`Sheet 39 of 50
`
`US 10,423,658 B2
`
`FIG. 39
`
`0850
`
`0852
`
`o851
`
`"4C Smith*
`
`Application Dot-Tag in Uploads Application View
`LaisTerretd
`Collections
`People
`Locations Recipe
`Family Tree
`SinaSacer
`
`Tasrrent useage: 4.9 MB 4G
`
`Application Dot-Tag in Collections Application View
`Uploads
`BRSiceirmm
`People
`Locations Recipe
`Ffarnily Tree
`$C Smith”
`Claw Awe
`
`JacksonandKPROS2013
`
`Application Dot-Tag in People Application View ,/
`Uploads Collections
`$ig@aucmm
`Locations
`Recipe
`Family Tree
`44C Srnlth” Sear Bkers
`
`EuropeTip 0854
`Av recagnrions
`
`Baregs
`
`A1
`
`41
`
`

`

`U.S. Patent
`
`Sep. 24, 2019
`
`Sheet 40 of 50
`
`US 10,423,658 B2
`
`FIG. 40
`
`0860
`
`trom Collections
`
`People
`
`Locations
`
`Recipe
`
`Family Tree
`
`Apply Filters
`Fast Search
`
`
`OS vetertis|Currantunage: 1.9MS 7 0862 TSaimeved: Setecr AR Serecchore
`
`Son dy, Suawestie Stest
`Te! Viewing Photes from: 2080704 gacters su Gvienss
`Uploaded: 202013
`
`
`
`items Per Page: Ez
`
`
`~ beet wocames | fearon pod Mag
`Loftectocs,
`
`42
`
`42
`
`

`

`U.S. Patent
`
`Sep. 24, 2019
`
`Sheet 41 of 50
`
`US 10,423,658 B2
`
`FIG.
`
`41
`
`Uploads Collections
`
`People
`
`SiWremeaicm™e
`
`Recipe
`
`Farnily Tree
`
`0870
`
`Fast Search
`
`Apply Filters
`
`
`
`
`g View Locations as List — 0871
`Click on map to zoom in
`“IC Smith”—ClearFilters
`0872
`0873
`
`43
`
`43
`
`

`

`U.S. Patent
`
`Sep. 24, 2019
`
`Sheet 42 of 50
`
`US 10,423,658 B2
`
`FIG. 42
`
`0881
`0880
`
` apply Multiple Filters
`Search for iters tes Rae!
`Abe
`
` Available Fitters
`Currant Active Fiters
`
`Location: Abe Lincoln
`Porson: IC Smith
`
`
`Elementary School
`
`
`
`ARete Sarnned
`
`Apply Multiple Filters
`
`Search tor Filters webee:
`Abe
`
`Available Filters
`
`
`Location: Abe Lincoln
`
`Elementary School
`
`
`
`
`
`
`
`Current Active Futers
`Person: JC Smith
`
`Elementary School
`ES] Location: Abe Lincoin
`
`
`
`44
`
`44
`
`

`

`U.S. Patent
`
`Sep. 24, 2019
`
`Sheet 43 of 50
`
`US 10,423,658 B2
`
`FIG. 43
`
`
`
`Uploads Collections
`
`People
`
`Gi{tenenems
`
`Recipe
`
`Family Tree
`
`-§ View LocationsasList
`
`“IC Smith” “Abe Lincoin Elementary School” ClearFilters
`
`Fast Search
`
`Apply Filters
`
`Click on map to zoom in
`
`45
`
`45
`
`

`

`U.S. Patent
`
`Sep. 24, 2019
`
`Sheet 44 of 50
`
`US 10,423,658 B2
`
`FIG. 44
`
`0900
`
`0901
`
`User selects
`Search bar
`
`
`
`0902
`
`if user selects Person fast search tag, system retrieves
`User types
`data Person Profile from User's User Relationship Table
`
`
`information to
`and displays relationship data in Person Profile View
`search
`
`System sends
`request
`
`0904
`
`Relationship
`Hf user selects Collection fast search tag, system retrieves
` System displays
`
`
`Table
`Syst