(12)
`
`United States Patent
`Desmond et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 9,552,376 B2
`Jan. 24, 2017
`
`USO09552376B2
`
`(54) METHOD AND APPARATUS FOR
`MANAGING DIGITAL FILES
`
`(56)
`
`(71) Applicant: Meyorweb. LLC, Glen Ellyn, IL
`(US)
`(72) Inventors: Christopher J. Desmond, Glen Ellyn,
`IL (US); Nancy L. Desmond, Glen
`Ellyn, IL (US): L. Michael Taylor,
`Chicago, IL (US)
`s
`(73) Assignee: Memory Web, LLC, Chicago, IL (US)
`
`(*) Notice:
`
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 81 days.
`
`(21) Appl. No.: 14/193,426
`
`(22) Filed:
`(65)
`
`Feb. 28, 2014
`Prior Publication Data
`US 2014/O181089 A1
`Jun. 26, 2014
`
`O
`O
`Related U.S. Application Data
`(63) Continuation-in-part of application No. 13/157,214,
`filed on Jun. 9, 2011, now Pat. No. 9,098,531.
`(51) Int. Cl.
`G06F 7/00
`G06F 7/30
`(52) U.S. Cl
`CPC
`
`G06F 17/30268 (2013.01); G06F 17/3028
`•
`u. fs
`- - - - -
`(2013.01); G06F 1730. sites
`(
`.01)
`(58) Field of Classification Search
`None
`See application file for complete search history.
`
`(2006.01)
`(2006.01)
`
`References Cited
`U.S. PATENT DOCUMENTS
`5,694,514 A * 12/1997 Evans ................ GO6K 17/0022
`358,906
`6,629, 104 B1* 9/2003 Parulski ............ GO6F 17,30265
`348,231.2
`aSK
`5/2008 Rhoads et al.
`7,372,976 B2
`7,480,669 B2
`1/2009 Lo et al.
`7,694,236 B2 * 4/2010 Gusmorino ......... GO6F 3,04817
`715,763
`
`(Continued)
`
`OTHER PUBLICATIONS
`Kustanowitz et al., “Motivating Annotation for Personal Digital
`Photo Libraries: Lowering Barriers while Raising Incentives”. Tech.
`Report HCIL-2004-18, U. Maryland, 2005.*
`(Continued)
`Primary Examiner — Michael Hicks
`(74) Attorney, Agent, or Firm — Nixon & Peabody LLP
`(57)
`ABSTRACT
`A computer-implemented method of associating digital tags
`with digital files comprises storing a plurality of digital files
`having embedded therein content data and metadata includ
`ing tags; receiving, via a user interface device of a client
`device, a first tag label containing alphanumeric text created
`and inputted by a user of the client device; modifying, using
`a controller device, a selected first one of the tags of the
`metadata in a first of the digital files to include the first tag
`label; receiving, via the user interface device or another user
`interface device, an instruction to search for all of the digital
`files having at least the first tag label; responsive to receiving
`the instruction, automatically searching for all of the digital
`files having at least the first tag label; and displaying, on a
`video display device associated with the client device, a first
`indication of the first tag label.
`12 Claims, 50 Drawing Sheets
`
`Carments:
`Suzane arc Anthony's Wedding 3rty where the cousins posed
`for a photo in the grass. Note, jack with the olipop act the
`photographer with his shoe in the photo
`
`Location:
`historical Society
`Lisle, 80532
`
`People:
`Jack Wong
`CJ Wang
`wary firestone
`Zoe Peika
`Nickersons
`
`Event: Suzanne & Anthony's Wedding Reception 2010
`
`Camera Details: more
`
`
`
`
`
`MemoryWeb Ex. 2004
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`US 9,552,376 B2
`Page 2
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`4/2012 Redstone
`8, 150,844 B2
`8.484.223 B2 * 7/2013 Ota ................... GO6F 17,30247
`707/748
`2006/0165380 A1* 7/2006 Tanaka ................... G11B 27.34
`386.227
`
`2007/0282908 A1* 12, 2007 Van der Meulen
`et al. .......................... TO7 104.1
`2009/0113350 A1* 4, 2009 Hibino .............. GO6F 17,30061
`71.5/853
`2009/0265631 A1* 10/2009 Sigurbjornsson ...... G06Q 10/10
`715,716
`
`OTHER PUBLICATIONS
`
`Kisilevich et al., “Event-based analysis of people's activities and
`behavior using Flickr and Panoramio geotagged photo collections'.
`14th International Conference Information Visualization, pp. 289
`296, 2010, IEEE.*
`Ahern et al., “World Explorer: Visualizing Aggregate Data from
`Unstructured Text in Geo-Referenced Collections”, JCDL’07, pp.
`1-10, 2007, ACM.*
`Kang et al., "Capture, Annotate, Browse, Find, Share: Novel
`Interfaces for Personal Photo Management'. International Journal
`of Human-Computer Interaction, 23(3), pp. 315-337, 2007, Law
`rence Erlbaum Associates, Inc.
`Jaffe et al., “Generating Summaries and Visualization for Large
`Collections of GeoReferenced Photographs”. MIR'06, pp. 89-98,
`2006, ACM.*
`Torniai et al., "Sharing, Discovering and Browsing Geotagged
`Pictures on the Web”, 2007, Hewlett-Packard Development Com
`pany, L.P.*
`
`Snavely et al., “Photo Tourism: Exploring Photo Collections in 3D”.
`SIGGRAPH '06, ACM transactions on graphics, vol. 25, Issue 3,
`pp. 835-846, 2006, ACM.*
`Kopf et al., “Deep photo: model-based photograph enhancement
`and viewing”. ACM Transactions on Graphics, vol. 27. No. 5,
`Article 116, Dec. 2008, ACM.*
`Trattner et al., “Evaluating tag-based information access in image
`collections'. Proceedings of the 23rd ACM conference on Hyper
`text and social media, pp. 113-122, 2012, ACM.*
`Kustanowitz et al., “Motivating Annotation for Personal Digital
`Photo Libraries: Lowering Barriers while Raising Incentives.” Tech.
`Report HCIL-2004-18, U. Maryland, 2005 (10 pages).
`Miller et al., “Give and take: a study of consumer photo-sharing
`culture and practice.” CHI '07 Proceedings of the SIGCHI Confer
`ence on Human Factors in Computing Systems, pp. 347-356, 2007
`(10 pages).
`Ames et al., “Why we tag: motivations for annotation in mobile and
`online media,” CHI '07 Proceedings of the SIGCHI Conference on
`Human Factors in Computing Systems, pp. 971-980, ACM, 2007
`(10 pages).
`Yee et al., “Faceted Metadata for Image Search and Browsing”. CHI
`2003, pp. 401–408, 2003, ACM.
`Ferre, "CAMELIS: Organizing and Browsing a Personal Photo
`Collection with a Logical Information System”, Int. Conf. Concept
`Lattices and Their Applications, pp. 112-123, 2007. HAL.
`Tomasson et al., “PhotoCube: Effective and Efficient Multi-Dimen
`sional Browsing of Personal Photo Collections”, ICMR 11, 2011,
`ACM.
`Bartolini et al., “Integrating Semantic and Visual Facets for Brows
`ing Digital Photo Collections”, SBED, pp. 65-72, 2009.
`
`* cited by examiner
`
`MemoryWeb Ex. 2004
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jan. 24, 2017
`
`Sheet 1 of 50
`
`US 9,552,376 B2
`
`F.G. 1
`
`Select None
`Select All
`Delete Selected
`
`
`
`1. File(s) Selected
`File Attributes:
`Date:
`location:
`People:
`Event:
`Connents:
`Keywords:
`Album:
`Star Ranking:
`Sharing Rights:
`Recipe:
`Document Type:
`File Name:
`
`September 2, 2010 change
`Yellowstone, MT, USA change
`Christopher Desmond, Nancy Desmond
`Fly Fishing in Yellowstone
`Add
`Add
`Add
`
`Add
`Add
`Add
`MG 123 change
`
`MemoryWeb Ex. 2004
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jan. 24, 2017
`
`Sheet 2 of 50
`
`US 9,552,376 B2
`
`
`
`FG 2
`
`Comments:
`Suzanne and Anthony's Wedding Party where the cousins posed
`for a photo in the grass. Note, Jack with the lollipop and the
`photographer with his shoe in the photo
`
`location:
`Historical Society
`Lisle, it 60532
`
`People:
`Jack Wong
`C Wong
`Mary Firestone
`Zoe Peika
`Nick Persons
`
`Event: Suzanne & Anthony's Wedding Reception 2010
`
`Camera Details: more
`
`MemoryWeb Ex. 2004
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jan. 24, 2017
`
`Sheet 3 Of 50
`
`US 9,552,376 B2
`
`
`
`Thumbnail Table
`
`vary Bader and Cint Fifestone
`Wedding October 3, 1945
`
`Capt. Clint Firestone
`1952
`
`Germany trip 1955
`
`Father's Day 1966
`
`Christopher 35
`Year 39
`
`Co8ege Break 1992
`
`
`
`Aria 1993
`
`E3 College
`Sractation 34
`
`Cairo, Egypt 996
`
`Europe
`Backpacking
`996
`
`
`
`Nancy James and
`Christopher Dasmond
`Wedding October 17, 1998
`
`Amazon Fishing rip
`November 2009
`
`Cousins Party 2010
`
`Fishing in Yellowstone
`wortara September 2010
`
`MemoryWeb Ex. 2004
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jan. 24, 2017
`
`Sheet 4 of 50
`
`US 9,552,376 B2
`
`FG 4
`
`
`
`Christopher Desmond and Nancy Desmond Wedding - October 17, 1998
`
`&
`Event &
`
`Captions On Off
`
`MemoryWeb Ex. 2004
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jan. 24, 2017
`
`Sheet 5 of 50
`
`US 9,552,376 B2
`
`
`
`MemoryWeb Ex. 2004
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jan. 24, 2017
`
`Sheet 6 of 50
`
`US 9,552,376 B2
`
`F.G. 6
`
`Thumbnail Table
`
`
`
`
`
`it Festice
`
`mos
`
`Stewee Suzanne lucky
`
`Steve Fwah
`
`BerSerian
`
`ce Sching
`
`Marywirginia
`
`Citewitt
`
`Charlotte Kia
`
`Christeep
`
`Peter longhorn
`
`in Bog
`
`Steve Fire
`
`Suzannetucky
`
`Steve Fwah
`
`Ben Sierra
`
`de Sching
`
`wary Virginia
`
`C: Dewitt
`
`Charlotte Kisian
`
`MemoryWeb Ex. 2004
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jan. 24, 2017
`
`Sheet 7 Of 50
`
`US 9,552,376 B2
`
`FIG 7
`
`
`
`
`
`Clinton Dewitt Firestone V
`
`July 12, 1896
`Birth:
`Apri; 29, 1971
`Death:
`Parents: Cinton Dewitt Firestone and Viola Mier
`Comments. He was a WWII U.S. Air force pilot and POW in WWII and veteran
`honorably discharged in December of 1947. He worked for 44 years for the
`Firestone Tire and Rubber Company in retai, wholesale and original equipment
`sales, marketing and management. He was born in Akron, OH and is buried in
`Columbiana, Oh.
`
`Edit bio
`
`locations
`
`
`
`Tineine
`
`Family Tree
`
`Recipes
`
`MemoryWeb Ex. 2004
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jan. 24, 2017
`
`Sheet 8 of 50
`
`US 9,552,376 B2
`
`FG. 8
`
`Descendants of Charlotte Killian halter
`
`
`
`free Table
`
`MemoryWeb Ex. 2004
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jan. 24, 2017
`
`Sheet 9 Of 50
`
`US 9,552,376 B2
`
`FG 9
`
`Year
`
`
`
`
`
`Cousins Party 2010
`
`Fishing in Yakowstone
`Montana September 2030
`
`MemoryWeb Ex. 2004
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jan. 24, 2017
`
`Sheet 10 Of 50
`
`US 9,552,376 B2
`
`FIG 10
`
`v
`
`Zucchii chopped)
`hai Basi: 8 leaves
`* Lemon Grass in jar) 1 teaspoon
`v
`Chinese Ginger Root in jar 1 teaspoon
`
`"
`
`Rice
`•
`Ehai Rice (something that only takes 2 cups of water)
`•
`Dice chicker in bowl and add two tablespoons of fish
`sauce. et marinate for 20 minutes.
`Eake thick part of Coconut milk out into pan (about 4
`tablespoons), Curry paste, 1 spoon of lemon grass, 1.
`spoon of ginger and garlic. Heat over high with boil and
`EEN stir for 3 minute. Add neat (uncooked) and fry
`until cooked over high heat
`Add milk, brown Sugar and salt. Bring back to slight boil
`and constantly stir. Add veggies and soy sauce. Cook for
`about 10-14 minutes COWERED until veggies are cooked.
`Serve with a snie.
`
`Desmond's Ye:ow Thai Chicken Curry
`
`Curry Mix
`• Coconut milk (400 in) - DO NOT SHAKE FT UP
`800 gram of chicken (4 chicken breast
`Fish sauce (Nam Pahai Bamboo Garden - Bottle
`•
`• Garic (2 coves)
`v
`Brocco (2cups chopped)
`* 2 Peppers (chopped)
`• 2 Carrots (chopped
`
`Chef: Barry Desmond
`
`
`
`Video on how to Make it
`
`Original Handwritten Recipe
`
`MemoryWeb Ex. 2004
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jan. 24, 2017
`
`Sheet 11 of 50
`
`US 9,552,376 B2
`
`FG 11
`
`thumbnail Table
`
`Videos
`
`# Docs
`
`
`
`Album/Event
`ack Monk's Arrival
`Mike Testy's First Birthday
`Cubs Beat Cards 1998
`ack Monk's Arrival
`Mike festy's 2nd Birthda
`Cubs Beat Cards 1998
`ack Monk's Arrival
`Mike Testy's 3rd Birthday
`Cubs Beat Cards 1998
`ack Wrigley Monk's Arrival
`'s 4th Birthda
`Cubs Beat Cards 1998
`Nancy learns How to Ride a Bike
`
`Date
`
`26-Dec-2003
`13-Sep-1983
`5-Aug-1998
`29-Dec-2003
`3-SeO-933
`5-Aug-1998
`29-Oec-2003
`13-Sep-1983
`5-Aug-1998
`29-Dec-2003
`3-Sep-1983
`5-Aug-1998
`23-u-1978
`
`location
`Chicago, 8.
`Minneapolis, MN
`Chicago, it
`Chicago, i.
`Minneapolis, MN
`Chicago, 8.
`Chicago,
`Minneapolis, MN
`Chicago, 8.
`Chicago, 8.
`Minneapolis, MN
`Chicago, 8.
`St. Louis, MO
`
`Photos
`
`S4
`
`36
`
`S4
`36
`
`54
`
`36
`
`54.
`36
`76
`
`|
`
`2
`
`2 3.
`
`2
`
`2
`
`videos I HDoes
`4
`a
`21
`o
`2,199
`2
`a
`a
`21
`o
`2,199
`2
`a
`a
`21
`o
`2,199 ||
`2
`4
`4
`||
`21
`o
`2,199 ||
`2
`2
`o
`
`||
`
`||
`
`2
`
`2
`
`2
`
`MemoryWeb Ex. 2004
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jan. 24, 2017
`
`Sheet 12 of 50
`
`US 9,552,376 B2
`
`F.G. 12
`
`
`
`Album/Event
`ack Monk's Arriva
`Mike Testy's First Birthday
`Cubs Beat Cards 1998
`ack Monk's Arrival
`Mike Testy's 2nd Birthda
`Cubs Beat Cards 998
`ack Monk's Arrival
`Mike Testy's 3rd Birthday
`Cubs Beat Cards 998
`ack Wrigley Monk's Arriva
`Mike esty's 4th Birthda
`Cubs Beat Cards 998
`Nancy Learns How to Ride a Bike
`
`Date
`26-Dec-2003
`13-Sep-1983
`S-Aug-1998
`29-Dec-2003
`3-Sep-1983
`5-Aug-1998
`29-Dec-2003
`13-Sep-1983
`5-Aug-1998
`29-Dec-2003
`3-Sep-1983
`5-Aug-1998
`21--978
`
`location
`Chicago, 3
`Minneapolis, MN
`Chicago, it
`Chicago, it
`Winneapolis, VN
`Chicago, 3
`Chicago, 8.
`Minneapolis, MN
`Chicago, 3
`Chicago, it
`Minneapolis, MN
`Chicago, 3
`St. Louis, MO
`
`Thumbnail Table
`
`if Photos
`
`fi Videos
`
`if Docs
`
`2
`
`22
`
`2
`
`54
`
`36
`
`54
`
`36
`
`54
`
`36
`
`54
`36
`
`76
`
`2
`
`2
`
`MemoryWeb Ex. 2004
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jan. 24, 2017
`
`Sheet 13 Of 50
`
`US 9,552,376 B2
`
`F.G. 13
`
`
`
`||
`2
`|
`| Alberts
`||
`2
`|
`- Annex
`||
`3
`|
`Bade
`||
`4
`|
`- Bacon
`||
`5
`|
`- Bates
`||
`6
`|
`Boone
`one-T I
`
`8
`7
`8
`8
`7
`6
`
`||
`||
`||
`||
`||
`||
`I
`
`0
`0
`0
`0
`1
`2
`
`0
`0
`0
`0
`0
`2
`
`||
`||
`||
`||
`||
`||
`I
`
`8
`
`Monk, lack
`
`Thumbnail fable
`
`:
`
`MemoryWeb Ex. 2004
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jan. 24, 2017
`
`Sheet 14 of 50
`
`US 9,552,376 B2
`
`F.G. 14
`
`Thumbnail able
`
`
`
`Alberts, John
`Killian, Jack
`Killian, Brian
`killian, Kevin
`killian, Sarah
`killian, John
`killian, Mark
`killian, John
`
`: Monk, Jack
`Moore, Bertha
`Slythe, Sarah
`
`I
`|
`
`T.
`
`T.
`
`T
`
`|
`
`o
`8
`cousin
`0
`||
`7
`Son
`o
`8
`Nephew
`0
`||
`8
`Nephew
`1
`||
`7
`Daughter-in-law
`Great Nephew T G T 2
`Great Nephew
`5
`4
`Grandson
`T 499
`a
`
`||
`||
`||
`
`o
`0
`o
`0
`0
`2
`
`14
`
`Great Niece
`sister
`
`a
`9
`
`||
`
`6
`0
`
`||
`
`|
`
`3
`9
`
`|
`
`MemoryWeb Ex. 2004
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jan. 24, 2017
`
`Sheet 15 Of 50
`
`US 9,552,376 B2
`
`FG 15
`
`Thumbnail able
`
`
`
`cologne |
`Dom
`Montalcino
`Lucilla & Roberto
`oH
`Akron
`sas Market
`college
`Manus
`Amazon Trip
`lake Geneval w
`999 Pine
`cabin
`Charleston
`903 Plymouth
`Grad school
`los Feliz I ca
`298 Glencarin
`Griffith Park
`Glendale
`CA
`A Equestrian ctrl ses Horse Dr.
`Decoronado
`12 coronado Dr Coronado I CA
`
`| Germany |
`Italy
`|
`usa
`T
`Brazil
`USA
`USA
`usa
`USA
`USA
`
`|
`T
`|
`T
`
`3
`6
`64
`235
`98
`1256
`12
`4
`321
`
`| 0
`2
`||
`||
`0
`1
`||
`2 ||
`0
`||
`| 8 ||
`2
`||
`2 ||
`0
`32
`a
`o
`o
`4
`0
`a
`o
`
`||
`
`||
`
`Home
`GAgri Party
`Pike's Market
`Rafels
`
`444 Main
`321 silver
`786 Market
`345 Fong
`
`it
`Anywhere
`USA
`GA
`Macon
`USA
`Seattle
`I wa T
`usa
`Tsingapore |
`| Singapore
`
`I
`T
`
`10,987
`is
`18
`23
`
`49
`o
`1
`2
`
`||
`
`9
`o
`0
`o
`
`||
`
`MemoryWeb Ex. 2004
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jan. 24, 2017
`
`Sheet 16 of 50
`
`US 9,552,376 B2
`
`FG 16
`
`Category Card able
`
`
`
`MemoryWeb Ex. 2004
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jan. 24, 2017
`
`Sheet 17 Of 50
`
`US 9,552,376 B2
`
`F.G. 17
`
`
`
`Cose X)
`
`MemoryWeb Ex. 2004
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jan. 24, 2017
`
`Sheet 18 Of 50
`
`US 9,552,376 B2
`
`F.G. 18
`
`
`
`Advanced Search Filter
`Keywords:
`Add
`Add
`Date:
`location:
`
`Chicago, USA
`Yellowstone, MUSA
`Cologne, Germany
`
`People:
`
`Mike Cubbie
`Mary lamb
`Christopher Monk
`Nancy Monk
`Dwight Schrut
`
`Event:
`
`Jack's 1st Birthday
`Fly Fishing in Yellowstone
`Raking leaves
`Christmas 2030
`Shanksgiving 2010
`July 4th Parade
`Add
`Abun:
`Star Ranking:
`Add
`Sharing Rights:
`Add
`Document ype: Add
`
`MemoryWeb Ex. 2004
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jan. 24, 2017
`
`Sheet 19 Of 50
`
`US 9,552,376 B2
`
`F.G. 19
`
`
`
`Captain Phil's Memory-Webb
`
`Welcome, Captain Phi:
`last login:
`11.38.2010
`
`My recent memories:
`• 123 Photos uploaded on 11.07.10
`•
`2 albums created 13.17.10
`•
`12 visitors since last login date
`•
`123 Photos uploaded on 11.07.10
`v 2 abuns created 13.17.10
`My recent Webb views:
`v
`Captain Phil 2010 (photo album)
`
`•
`
`Captain Phil (Timeline)
`
`wedia
`Photos
`# Videos
`Octinents
`
`Count Archi
`1342
`AS
`73
`
`People Stats:
`
`Count
`80% complete
`61% complete
`
`i People
`11
`
`T
`
`AS9
`
`T 1249
`1,249
`
`Firestone
`Firestore
`
`17
`
`39
`39
`
`Updates and Alerts:
`•
`cense renewa due 1.15.2011
`
`Evert Stats:
`
`
`
`Mike festy's 1st Birthda
`Cubs Beat Cards Aug 1998
`Nancy Learns to Ride Bike
`
`13-Sept-1988
`5-Aug-1998
`21-July-1978
`
`winneapolis, MN
`Chicago,
`St. ouis, MO
`
`2,99
`
`MemoryWeb Ex. 2004
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jan. 24, 2017
`
`Sheet 20 Of 50
`
`US 9,552,376 B2
`
`FG 20
`
`Gs
`
`Original Digital
`files) pla; to
`wentyNet)
`
`this Party Faciel
`Recognior
`Rier
`
`third Pariwwedia Provic
`ird Party wedia Prowicer
`
`Sy
`inside MW System
`
`c. E.
`eographica
`tapping Provider
`
`individual
`Group of
`People
`
`
`
`1391
`wy Ficifies Fie
`exported from
`MemoryWeb to Storage
`
`
`
`
`
`
`
`
`
`third Party Pacial
`Recognition
`Syster
`
`third Party wiedia
`Syster, data
`Excharge)
`
`Share 3 Social
`Network Provider
`System
`
`System Reactins
`Phase
`
`system frterpreting and
`Adding pada to
`Relationship Table Phase
`
`
`
`User Ralationship
`Tale
`
`Systern Erdexing
`ligital Files and
`agata Slocks
`for a Digital Fila
`vii;ster
`Relationship fable
`
`Third party Geographical
`locatioi wapping
`System
`
`Application Export System
`
`Application Digital
`Tag organizer
`System
`
`Continuous tink
`of Application
`Do-ags System
`
`
`
`Ayarcadiers
`System
`Y
`
`Keyword Fast
`Search System
`
`MemoryWeb Ex. 2004
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jan. 24, 2017
`
`Sheet 21 of 50
`
`US 9,552,376 B2
`
`System Reading Phase
`
`y
`
`System interpreting and Adding data to Relationship fable Phase \
`
`FIG 21
`
`Original Digital file begins to
`upload to Merrorylie:
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`utside IWW Sysien
`inside MW Systems
`
`clais XF
`Enrage Fie
`firstories in
`Digital file
`
`uploadec
`
`is:
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`Recognition Provider
`
`40
`
`Shired Party Facial Recognitic:
`System
`
`Read and iterate through
`ear: XF tag item
`
`No
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`outside ww.system
`lnsidewWSystern
`
`Uploa: Griginal
`file ific WW
`
`Create uplicates
`of original file in
`different file sizes
`
`Says XS ista 3
`tiser Relationship
`abs for Digital
`File
`
`Parse Exif
`data into
`
`vernory Web
`
`took up MN
`tag data
`
`
`
`taggata
`exists
`
`Create new
`
`data
`
`Associate
`existing tag
`data
`
`Systery indexing digital Files and
`Tagbata Blocks for a digital File
`within User Relationship Table
`33
`
`:
`
`User Relationship table
`
`90
`
`(30.
`
`3rigine File
`
`bilicats Fis
`
`Clicats. Fis
`
`lite Firs
`ethnail
`Duplicate file
`
`locks)
`Airsoft
`Windows ?ag
`at Blacks
`Merrorisb
`Tag Data Blocks)
`Wenry Web
`Person Tag Data
`
`MemoryWeb Ex. 2004
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jan. 24, 2017
`
`Sheet 22 of 50
`
`US 9,552,376 B2
`
`FG, 22
`
`EXIF Tags version 2.3 image File Directories \
`(Data Blocks)
`32
`321
`EXF Family
`die
`x3e3 or
`xO1je
`
`Q32:
`
`
`
`3.25
`
`Oescription Title
`Oescription Subject
`
`32
`
`MemoryWeb Tag (Data Blocks)
`
`0360
`Y
`
`0323
`
`wediaAsset. Caption
`
`wediaAsset.Staraking
`
`O3.
`
`032
`
`(33
`
`32
`
`Origi Data faken
`
`327
`
`(328
`
`33,
`
`33
`
`832
`
`333
`
`frage indge iP. Dimensions, With Heig
`
`beneapnetter wore eme, e,
`
`sef Cornet
`Sattice
`SPSlatitude ef
`
`GPS longitude
`
`GPS logitude Ret
`
`GPs Altitude
`
`I
`
`Exit
`ses
`
`knee
`
`0x9236
`boo.
`
`is is used to inject information that do not curfety have
`EX stacardized tags including CoElectio, Pecipie, location
`iaine, Recipe Nahe, Personag Data Bocks (380, etc
`MadiaAssetocation.atitude
`wediaAsset.ocation, atitude
`
`wediaAsset. location. Longitude
`
`wediaAsset.location.org.tuce
`
`se
`8358
`C33
`
`030
`
`037
`
`MemoryWeb Ex. 2004
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jan. 24, 2017
`
`Sheet 23 Of 50
`
`US 9,552,376 B2
`
`
`
`34
`
`FG 23
`
`\
`Windows imaging Component (WIC) Metadata
`(Data Blocks)
`
`3.
`
`Tag Labe
`
`File Name
`
`File tem. Fype
`
`File Folder Path
`
`Feate Created
`
`File Date Modified
`
`FeSize
`
`File Attributes
`File (Offline availability, Offline status, Shared with,
`Owner, Computer, etc)
`
`y
`MemoryWeb Tag (Data Blocks)
`
`338
`
`MemoryWeb Tag
`
`MediaAsset, Fename
`
`38.
`
`372
`
`MediaAsset. UploadBatch, Batch Nare
`
`MemoryWeb Ex. 2004
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jan. 24, 2017
`
`Sheet 24 of 50
`
`US 9,552,376 B2
`
`FG 24
`
`038 y
`MemoryWeb Person Tag (Data Blocks
`Persor Nare
`
`
`
`Nickane
`
`Birthdate
`
`Date of Death
`Biography
`
`Mother
`
`Father
`
`Brother 1, Brother 2, ...
`
`Sister, Sister 2, ...
`Daughter 1, Daughter 2, ...
`
`Son 1 Son 2....
`
`Spouse 1, Spouse 2, ...
`Facial Recognition Data (faylor?)
`
`Facebook
`
`pets
`
`O395
`
`381
`0.382
`
`383
`
`O384
`
`O385
`
`0386
`
`O387
`
`O388
`
`0389
`
`O390
`
`39.
`
`O392
`
`O393
`
`O394
`
`O398
`
`MemoryWeb Ex. 2004
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jan. 24, 2017
`
`Sheet 25 Of 50
`
`US 9,552,376 B2
`
`FG, 25
`
`
`
`Recognition: Provider
`
`(At
`
`3.
`
`
`
`
`
`Outside MW System
`inside M W System
`
`System request for
`Art R
`'E',
`
`ges at We
`
`O330
`
`User
`Relationship
`able
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`System receives
`response for
`hir Party facial
`Fig.
`
`System parses
`idetified acas
`and correspocirg
`Facia Recognition
`data For each
`Eligital sile
`
`-
`
`C) --
`
`Sysian sends
`Digital Files to
`Third Party
`Facial
`Recognition
`rowice
`
`Existing trainec Face is
`is set Resters:g
`ac
`
`is Face recognition
`above score
`threshold
`
`Syster records and associates specific
`face for Oigital File with a warrioryWeb
`Person and sends to Memory Web
`Psrsonagata Blocks in User
`Relationship table
`
`Retrieves
`humb|3ils of a
`igital siles; at
`have incarfirine
`faces
`
`Syster generates facial
`recognitic record for unknown
`person and then sends
`information to Memory.web
`Person Tag Data Elaicks in Use
`Relationship table
`
`
`
`System generates wirtual
`?cofired facial
`recognition record for person
`and tenses information to
`AemoryWeb Person as
`Data elocks) in User
`Relicishto able
`
`MemoryWeb Ex. 2004
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jan. 24, 2017
`
`Sheet 26 of 50
`
`US 9,552,376 B2
`
`FG 26
`
`OSCO
`
`501
`
`Outside WW System
`
`inside MW System
`Systems sends
`registration
`confiniation
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`52
`
`ser selects, sharing of
`their information wish
`third Earty weria
`rywider with
`wernorweb
`
`Systems receives
`registration
`confirmation, Third
`Party Media IE
`
`Systein: sends
`registration
`information to
`User Setting
`sis
`
`System receives, contac: namas
`and Ds from Thirc. Party Mecia
`Provider daily
`
`System requests contact
`nates artisinthirt
`Party wedia Provider
`daily
`
`is Parsssf a
`&ai
`
`Syster ass}ciates
`existing Persson to fag.
`Data Block of
`Relationship Table for
`igital File
`
`System addrew person rotag
`Eata Bick of Relatianship
`able for Digital file
`
`user Relationship
`
`sef Set
`Table
`
`
`
`Share to Fhird Party Media
`Prowicer syster
`
`MemoryWeb Ex. 2004
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jan. 24, 2017
`
`Sheet 27 Of 50
`
`US 9,552,376 B2
`
`FG 27
`
`1900
`
`ter
`
`30
`
`902
`
`303
`
`
`
`User's Name
`
`Payment D
`
`Password
`
`94
`
`Account Type
`
`905
`1906
`
`1907
`
`1903
`
`1909
`
`1910
`
`User's email
`language preference
`
`Oate format
`
`Email notifications
`
`Contacts (with third Party Social Media)
`
`Facebook D
`
`9.
`
`APicken
`
`Payment Date
`
`192
`
`1913
`
`MemoryWeb Ex. 2004
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jan. 24, 2017
`
`Sheet 28 Of 50
`
`US 9,552,376 B2
`
`FG, 28
`
`ser selects
`80.
`Digitation 1
`
`system retrieves
`Digital File and
`Digital File a
`&
`Cata. Eocks
`for tise
`Relationship
`able
`
`system will
`cisplay the
`Digital File and
`the
`corresponding
`&ct
`Digits filesg
`3a socks in
`the for of
`Application in
`ags
`
`System sends
`request:
`Relationship
`able of spacific
`Eligita: File
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`ser
`m
`Relations it
`Table
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`User selects
`Application ot-
`ag
`
`Carios irk
`of Application
`Cot-tag System
`
`serspects add
`f
`O
`Memory Web
`ag
`
`ag data exists?
`
`Tag 3a exists
`
`sef selects ect
`for a
`MeroyWe
`Application ot
`
`iser selects
`delete
`ca
`wertory web
`Application ot-
`Tag
`
`
`
`OSOC)
`
`ser selects
`tracting
`Wieniorwises a
`y
`g
`
`set adds
`New Memories
`as:
`
`Sef sa:ects
`matching
`Ivermory wet as
`
`tiser adds
`New errory web
`g
`
`s
`System associates
`few wensory Web tag
`s Data Black of
`Relationship Table for
`initial ki
`igital sile
`Sysen, creates new
`MemoryWeb tag to
`agata 8oko
`3g 33 Biokoi
`Relationship table for
`M
`Digitalie
`Syster associates
`2xisting Wemory Web
`E.
`k
`Relationship able
`for Gigital file
`Systern creates new
`MerritoryWeb tag to
`agata Block of
`Rolationship tablo For
`fligital rile
`Systern dairies
`association of
`MenoryWeb tag to
`tag data Block of
`Relationship Table for
`
`MemoryWeb Ex. 2004
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jan. 24, 2017
`
`Sheet 29 Of 50
`
`US 9,552,376 B2
`
`FG, 29
`
`Structure
`
`St.
`
`SS2
`
`tifié
`
`seriefined Fagabe
`
`5.
`
`(653
`
`Examples
`8this
`a seriter sess
`
`ass) cologne Gerary
`
`8xees gracteriri free arc
`
`
`
`398 Holiday Photos from ...
`
`857
`
`Os38
`
`sers
`
`w OSS4
`series
`
`Dotted Application Dot-Tag denotes partial relationship. In
`this example, person is a half-sibling to another person,
`-----------------------------
`Frank Smith
`:
`
`- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - t
`Y 06ss
`
`MemoryWeb Ex. 2004
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jan. 24, 2017
`
`Sheet 30 Of 50
`
`US 9,552,376 B2
`
`
`
`user selects
`Application Dots
`Tag
`
`is Persin
`Dot-Tag
`
`is olection
`Ci-ag
`
`is location
`w
`ot-Tag
`
`sate
`Dot-Tag?
`
`is Recipe
`ci-age
`
`FG 30
`
`Person Application Pot-tag system retrieves cata
`for that person from user's User Relationship Table
`and displays relationship data in People Profile view
`
`Ef Collection, Application ot-ag, System retrieves
`gata from that collectign from User's 3sar
`Relationship Table and displays relationship data in
`Collection View.
`
`System, ssics
`request to
`Reghi
`
`38
`
`User
`
`Relationship
`able
`
`if tocation Application Gottag, system retrieves data
`from that location rosses user Relationship Fable
`a?id displays relatior ship stain individuallgation
`view
`
`date Application bot-tag. System etrieves data
`from that date from user's User Relationship Table
`and displays relationship 3.3 in ploads view
`
`if Recipe Application dot-Eag, system retrieves data
`from that Recipe frony tyser's Jser Relationship
`fable and displays relatio ship data in Recipe Wiew
`
`Displays image
`is Sideshow
`wavy
`
`Oat
`
`07
`
`User selects
`Abadication
`pp.
`ag
`
`-
`
`MemoryWeb Ex. 2004
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jan. 24, 2017
`
`Sheet 31 Of 50
`
`US 9,552,376 B2
`
`
`
`FIG 31
`
`h
`
`s
`
`Jackson and JC's First day of School
`ass
`
`oscities 78
`
`MemoryWeb Ex. 2004
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jan. 24, 2017
`
`Sheet 32 Of 50
`
`US 9,552,376 B2
`
`A1
`Multiple People Application View
`Year
`Uploads Collections
`Locations Recipe Faily Free
`Sort By: Newest to oldest
`7 - 1492
`
`FG 32
`
`SO
`
`14.
`
`436
`Apply Filters y
`
`Jon Smith
`
`Jane Smith (Doe)
`
`Y/
`
`an
`Jackson Smith
`
`Cori Srith
`
`
`
`Single People Profile Application View
`Uploads Co}lections
`locations Recipe Family ree
`
`\
`Apply Filters
`
`< View all People
`in 32
`
`153
`A54
`
`48
`
`29
`
`1431-Con Smith
`Chi
`433-Nicknames:
`0.
`1434-Born: Giao
`143S-arents:
`St
`1436-Siblings:
`t437-hirer:
`1438 Biagraphy: C was named after his great, great grandfather from German who's real name was Johar Christoph.
`445
`A39
`J.Crags:
`52/ 44. / L442
`- 14s
`at
`/
`3 Cotections 2Faciai Recognitians 3 locations 7Family Relationships
`Recipe
`
`and
`
`change Profile Photo
`
`12C
`
`items Per Page:
`
`MemoryWeb Ex. 2004
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jan. 24, 2017
`
`Sheet 33 Of 50
`
`US 9,552,376 B2
`
`581
`Multiple Collection View
`thploads Collections
`People locations Recipe Farity free
`
`Sort By: Newest to Oldest v - so
`
`FG 33
`
`FastSearch
`
`Adoly Filters
`
`Smith Family
`Phofs
`
`First Day of
`Schof
`
`Jackson and JC
`Photos 2013
`
`People locations Recipe Family free
`
`Fast Search
`
`Apply Filters
`
`< View all Collections -- iss
`Smith Family Photos - 1532
`
`Share this Collection: y
`
`1534
`
`1S)
`
`?
`
`S30
`
`
`
`
`
`Items Per Page,
`
`20
`
`MemoryWeb Ex. 2004
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jan. 24, 2017
`
`Sheet 34 of 50
`
`US 9,552,376 B2
`
`Multiple location Application View
`Uploads Collections
`People
`locations Recipe Family free
`
`sea
`
`Sort By: Newest to Oldest v - 1606
`
`FG 34
`805,
`
`Apply Filters
`
`6OO
`y
`
`Ahe Lincoln
`Elementary School
`
`Hone Sweet
`Horne
`
`Single location Application View
`Uploads CoEections. People
`locations Recipe Family ree
`
`
`
`K View all locations - 31
`Wrigley Field
`1832
`36S1 N, Clark St.
`Chicago Illinois, United States
`
`1633
`
`1 Photos
`
`1630
`
`Apply Filters
`
`Share this Collection:
`
`f
`
`items per Page 20
`
`so
`
`too
`
`MemoryWeb Ex. 2004
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jan. 24, 2017
`
`Sheet 35 of 50
`
`US 9,552,376 B2
`
`(Colactions People locations. Retire Fariy Free
`
`First Search
`
`Apply Filters
`
`FG 35
`
`1700
`
`173
`- 1701
`axis. currowess sow?
`AS
`1 /v
`sasy File Batch Name A-Z)
`34
`
`File Folder: Cf2013/general 1.
`
`File Folder: Cf2013/Family Fun 1 (563 1.O.
`
`
`
`
`
`
`
`
`
`a if:
`
`wissyfia's :
`s 73
`
`ch, Kia N 1732
`
`- &isis:- is:ss:
`
`
`
`-items Per Page: 20
`
`1787
`
`
`
`
`
`
`
`MemoryWeb Ex. 2004
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jan. 24, 2017
`
`Sheet 36 of 50
`
`US 9,552,376 B2
`
`FG. 36
`
`18
`
`JEoads Cotections People locations
`
`Recipe Famiy Tree
`
`FastSearch
`
`Apply Filters
`
`: Show all Recipes 1 1802
`
`Ori DOe Peca Puffs 1 1831
`
`Share this Collection
`1812
`
`Chef
`
`184
`
`1816
`
`ingredients
`
`- 1809
`
`1. Stick of butter
`C1 Cup of flour
`% teaspoon of vanila
`1 cup of grand pecans
`2 tablespoons of Sugar
`
`1810
`Directions 1
`
`80
`
`Mix all. Roll gently into 1 inch balls. 300
`and immediately roll in Sugar,
`
`Comments 1
`
`8ll
`
`:* - as x is re- . . . .e.
`is a w.
`
`
`
`c.
`
`This is one of my favorite recipes around the holidays and was
`one of Grandma Lori's best
`
`N.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`MemoryWeb Ex. 2004
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jan. 24, 2017
`
`Sheet 37 Of 50
`
`US 9,552,376 B2
`
`FG, 37
`
`ise selects
`Advanced iter
`
`O3O2
`
`iser types
`information into
`the textbox
`
`33
`
`if in Uploads wiew, system retrieves data for applied
`filters) from User's ser Relationship fable and
`displays relationship data
`
`080S sity
`p
`n
`matches of
`-
`filters
`
`Sys
`systein
`produces
`request
`
`User
`Relationship
`tale
`
`Ef in Collections wiew, syster retrieves data for
`applied filter(s) from User's User Relationship Tabl:
`and displays relationship data
`
`
`
`O3)
`
`80
`
`user selects
`filters
`ilterts
`
`User selects
`apply
`
`088
`
`System sends
`request
`
`081
`
`in locations view, System retrieves cata for applied
`-> filters from user's User Relationship table and
`displays relatiosip cata
`
`N
`if it. People view, syster retrieves data or applied
`filliers from User's User Relationship Table and
`displays relationship data
`
`a.
`
`O2
`
`83
`
`iser desires
`tad
`aciditional
`Advanced
`Fiter
`
`Continuous link
`of Application
`ot-tags System
`
`OGO
`
`MemoryWeb Ex. 2004
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jan. 24, 2017
`
`Sheet 38 of 50
`
`US 9,552,376 B2
`
`FG 38
`
`083
`
`83
`
`Apply littipe Filters
`
`G838
`
`saias siters
`
`screek 3rs
`
`Person:
`
`Stitt
`
`Persor: Jackson Smith
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Appy Msiris Filies
`
`assac fiers
`
`perssr: CSmith,
`
`Persi; a.kson Smith
`
`Pers; c. Sith
`
`Perso; ane Sath
`
`acation: Smithsonia
`
`Collectics: Sriitso?ia frin
`
`ret actities
`
`
`
`idersson::JSinit
`
`MemoryWeb Ex. 2004
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U.S. Patent
`
`Jan. 24, 2017
`
`Sheet 39 Of 50
`
`US 9,552,376