( 12 ) United States Patent
`Desmond et al .
`
`( * ) Notice :
`
`( 54 ) METHOD AND APPARATUS FOR
`MANAGING DIGITAL FILES
`( 71 ) Applicant : MemoryWeb , LLC , Glen Ellyn , IL
`( US )
`( 72 ) Inventors : Christopher J . Desmond , Glen Ellyn ,
`IL ( US ) ; Nancy L . Desmond , Glen
`Ellyn , IL ( US ) ; L . Michael Taylor ,
`Chicago , IL ( US )
`( 73 ) Assignee : NCM IP HOLDINGS , LLC , Glen
`Ellyn , IL ( US )
`Subject to any disclaimer , the term of this
`patent is extended or adjusted under 35
`U . S . C . 154 ( b ) by 518 days .
`This patent is subject to a terminal dis
`claimer .
`( 21 ) Appl . No . : 15 / 375 , 927
`( 22 ) Filed :
`Dec . 12 , 2016
`Prior Publication Data
`( 65 )
`US 2017 / 0091225 A1 Mar . 30 , 2017
`Related U . S . Application Data
`Continuation of application No . 14 / 193 , 426 , filed on
`Feb . 28 , 2014 , now Pat . No . 9 , 552 , 376 , which is a
`( Continued )
`
`( 63 )
`
`( 51 )
`
`Int . CI .
`G06F 16 / 90
`G06F 16 / 58
`
`( 52 )
`
`U . S . Cl .
`CPC
`
`( 2019 . 01 )
`( 2019 . 01 )
`( Continued )
`G06F 16 / 5866 ( 2019 . 01 ) ; G06F 16 / 51
`( 2019 . 01 ) ; G06F 16 / 901 ( 2019 . 01 ) ; G06F
`16 / 907 ( 2019 . 01 ) ; G06F 3 / 0481 ( 2013 . 01 )
`
`US010423658B2
`
`( 10 ) Patent No . : US 10 , 423 , 658 B2
`( 45 ) Date of Patent :
`* Sep . 24 , 2019
`( 58 ) Field of Classification Search
`CPC . . . GO6F 16 / 5866 ; GO6F 16 / 907 ; G06F 16 / 901
`See application file for complete search history .
`
`( 56 )
`
`References Cited
`U . S . PATENT DOCUMENTS
`5 , 694 , 514 A
`12 / 1997 Evans
`6 , 629 , 104 B1
`9 / 2003 Parulski
`( Continued )
`OTHER PUBLICATIONS
`Kustanowitz et al . , “ Motivating Annotation for Personal Digital
`Photo Libraries : Lowering Barriers while Raising Incentives , ” Tech .
`Report HCIL - 2004 - 18 , U . Maryland , 2005 ( 10 pages ) .
`( Continued )
`Primary Examiner — Loc Tran
`( 74 ) Attorney , Agent , or Firm — Nixon Peabody LLP
`ABSTRACT
`( 57 )
`A computer - implemented method of associating digital tags
`with digital files comprises storing a plurality of digital files
`having embedded therein content data and metadata includ
`ing tags ; receiving , via a user interface device of a client
`device , a first tag label containing alphanumeric text created
`and inputted by a user of the client device ; modifying , using
`a controller device , a selected first one of the tags of the
`metadata in a first of the digital files to include the first tag
`label ; receiving , via the user interface device or another user
`interface device , an instruction to search for all of the digital
`files having at least the first tag label ; responsive to receiving
`the instruction , automatically searching for all of the digital
`files having at least the first tag label ; and displaying , on a
`video display device associated with the client device , a first
`indication of the first tag label .
`
`15 Claims , 50 Drawing Sheets
`
`Select None
`Select All
`Delete Selected
`
`SUR
`
`09
`
`#
`
`448
`
`1 File ( s ) Selected
`File Attributes :
`September 2 , 2010 change
`Date :
`Yellowstone , MT , USA change
`Location :
`People :
`Christopher Desmond , Nancy Desmond
`Fly Fishing in Yellowstone
`Event :
`Add
`Comments :
`Add
`Keywords :
`Add
`Album :
`•••••
`Star Ranking :
`Sharing Rights : Add
`Recipe :
`Add
`Document Type : Add
`File Name :
`IMG 123 change
`
`WA
`
`MemoryWeb Ex. 2005
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`US 10 , 423 , 658 B2
`Page 2
`
`( 56 )
`
`Related U . S . Application Data
`continuation - in - part of application No . 13 / 157 , 214 ,
`filed on Jun . 9 , 2011 , now Pat . No . 9 , 098 , 531 .
`( 51 ) Int . CI .
`( 2019 . 01 )
`G06F 16 / 51
`G06F 16 / 901
`( 2019 . 01 )
`G06F 16 / 907
`( 2019 . 01 )
`G06F 3 / 0481
`( 2013 . 01 )
`References Cited
`U . S . PATENT DOCUMENTS
`7 , 372 , 976 B2
`5 / 2008 Rhoad
`7 , 480 , 669 B2
`1 / 2009 Lo
`4 / 2010 Gusmorino
`7 , 694 , 236 B2
`8 , 150 , 844 B2
`4 / 2012 Redstone
`8 , 484 , 223 B2
`7 / 2013 Ota
`2006 / 0165380 A1 *
`7 / 2006 Tanaka
`G11B 27 / 34
`386 / 227
`2007 / 0282908 A1
`12 / 2007 Van Der Meulen
`4 / 2009 Hibino . . . . . . . . . . . . . . . . . . . . . GO6F 16 / 41
`2009 / 0113350 A1 *
`715 / 853
`10 / 2009 Sigurbjornsson
`2009 / 0265631 Al
`
`OTHER PUBLICATIONS
`Miller et al . , " Give and take : a study of consumer photo - sharing
`culture and practice , " CHI ' 07 Proceedings of the SIGCHI Con
`ference on Human Factors in Computing Systems , pp . 347 - 356 ,
`2007 ( 10 pages )
`Ames et al . , “ Why we tag : motivations for annotation in mobile and
`online media , ” CHI ' 07 Proceedings of the SIGCHI Conference on
`Human Factors in Computing Systems , pp . 971 - 980 , ACM , 2007
`( 10 pages ) .
`
`Yee et al . , “ Faceted Metadata for Image Search and Browsing ” , CHI
`2003 , pp . 401 - 408 , 2003 , ACM .
`Ferre , “ CAMELIS : Organizing and Browsing a Personal Photo
`Collection with a Logical Information System ” , Int . Conf . Concept
`Lattices and Their Applications , pp . 112 - 123 , 2007 , HAL .
`Tomasson et al . , “ PhotoCube : Effective and Efficient Multi
`Dimensional Browsing of Personal Photo Collections ” , ICMR ’ 11 ,
`2011 , ACM .
`Bartolini et al . , “ Integrating Semantic and Visual Facets for Brows
`ing Digital Photo Collections ” , SBED , pp . 65 - 72 , 2009 .
`Trattner et al . , “ Evaluating Tag - Based Information Access in Image
`Collections ” , Proceedings of the 23 " ACM Conference on Hyper
`text and Social Media , pp . 113 - 122 , 2012 ACM .
`Kang et al . , “ Capture , Annotate , Browse , Find , Share : Novel
`Interfaces for Personal Photo Management ” , International Journal
`of Human - Computer Interaction , 23 ( 3 ) , pp . 315 - 337 , 2007 , Law
`rence Eribaum Associates , Inc .
`Jaffe et al . , “ Generating Summaries and Visualization for Large
`Collections of Geo Referenced Photographs ” , MIR ' 06 , pp . 89 - 98 ,
`2006 ACM .
`Tornial et al . , “ Sharing , Discovering and Browsing Geotagged
`Pictures on the Web ” , 2007 , Hewlett - Packard Development Com
`pany , L . P . , pp . 1 - 18 .
`Snavely et al . , “ Photo Tourism : Exploring Photo Collection in 3D ” ,
`SIGGRAPH ' 06 ACM Transactions on Graphics , vol . 25 , Issue 3 ,
`pp . 835 - 846 , 2006 ACM .
`Kisilevich et al . , “ Event - based analysis of People ' s Activities and
`Behavior Using Flickr and Panoramio Geotagged Photo Collec
`tions ” , 14th International Conference Information Visualization , pp .
`289 - 296 , 2010 IEEE .
`Ahern et al . , “ World Explorer : Visualizing Aggregate Data From
`Unstructured Text in Geo - Referenced Collections ” , JCDL ' 07 , pp .
`1 - 10 , 2007 , ACM .
`Kopf et al . , “ Deep photo : model - based photograph enhancement
`and viewing ” , ACM Transactions on Graphics , vol . 27 , No . 5 ,
`Article 116 , Dec . 2008 , ACM ( 10 pages ) .
`* cited by examiner
`
`MemoryWeb Ex. 2005
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U . S . Patent
`
`Sep . 24 , 2019
`
`Sheet 1 of 50
`
`US 10 , 423 , 658 B2
`
`FIG . 1
`
`Select None
`Select All
`Delete Selected
`
`X4
`
`Wh
`
`W * * * * *
`
`12
`
`1 File ( s ) Selected
`File Attributes :
`September 2 , 2010 change
`Date :
`Yellowstone , MT , USA change
`Location :
`Christopher Desmond , Nancy Desmond
`People :
`Fly Fishing in Yellowstone
`Event :
`Add
`Comments :
`Add
`Keywords :
`Album :
`Add
`. . . .
`Star Ranking :
`Sharing Rights : Add
`Recipe :
`Add
`Document Type : Add
`File Name :
`IMG 123 change
`
`MemoryWeb Ex. 2005
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U . S . Patent
`
`Sep . 24 , 2019
`
`Sheet 2 of 50
`
`US 10 , 423 , 658 B2
`
`FIG . 2
`
`11 : 1 : :
`
`-
`
`itir :
`
`.
`
`.
`
`.
`
`. .
`
`.
`
`.
`
`Location :
`Historical Society
`Lisle , IL 60532
`
`Comments :
`Suzanne and Anthony ' s Wedding Party where the cousins posed
`for a photo in the grass . Note , Jack with the lollipop and the
`photographer with his shoe in the photo
`People :
`Jack Wong
`CJ Wong
`Mary Firestone
`Zoe Peika
`Nick Persons
`Event : Suzanne & Anthony ' s Wedding Reception 2010
`
`Camera Details : more
`
`MemoryWeb Ex. 2005
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U . S . Patent
`
`Sep . 24 , 2019
`
`Sheet 3 of 50
`
`US 10 , 423 , 658 B2
`
`FIG . 3
`
`.
`
`.
`
`is
`
`WWW
`
`.
`
`TSU
`
`Mary Bader and Clint Firestone
`Wedding October 11 , 1945
`
`Capt . Clint Firestone
`1952
`
`Germany Trip 1955
`
`Father ' s Day 1966
`
`Thumbnail | Table
`
`Si
`
`DESTE
`
`VIUNI
`
`Christopher 1st
`Year 1972
`
`College Break 1992
`
`LA Trip 1993
`
`EIU College
`Graduation 1994
`
`Cairo , Egypt 1996
`
`Europe
`Backpacking
`1996
`
`Nancy James and
`Christopher Desmond
`Wedding October 17 , 1998
`
`Amazon Fishing Trip
`November 2009
`
`Cousins Party 2010
`
`Fishing in Yellowstone
`Montana September 2010
`
`MemoryWeb Ex. 2005
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U . S . Patent
`
`Sep . 24 , 2019
`
`Sheet 4 of 50
`
`US 10 , 423 , 658 B2
`
`FIG . 4
`
`Captions On Off
`Christopher Desmond and Nancy Desmond Wedding - October 17 , 1998
`
`Event
`
`MemoryWeb Ex. 2005
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U . S . Patent
`
`Sep . 24 , 2019
`
`Sheet 5 of 50
`
`US 10 , 423 , 658 B2
`
`FIG . 5
`
`Sons
`
`S
`
`V V
`
`W
`
`: 3 Ndg
`
`w
`
`w
`
`w
`
`MemoryWeb Ex. 2005
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U . S . Patent
`
`Sep . 24 , 2019
`
`Sheet 6 of 50
`
`US 10 , 423 , 658 B2
`
`FIG . 6
`
`Thumbnail Table
`
`Clint Firestone
`
`Tim Dog
`
`Steve Fire
`
`Suzanne Lucky
`
`Steve Fwah
`
`Ben Sherman
`
`Joe Schmo
`
`Mary Virginia
`
`Clint Dewitt
`
`Charlotte Killian
`
`Chris Jeep
`
`Peter Long Horn
`
`Tim Dog
`
`Steve Fire
`
`Suzanne Lucky
`
`Steve Fwah
`
`Ben Sherman
`
`Joe Schmo
`
`Mary Virginia
`
`Clint Dewitt
`
`Charlotte Killian
`
`MemoryWeb Ex. 2005
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U . S . Patent
`
`Sep . 24 , 2019
`
`Sheet 7 of 50
`
`US 10 , 423 , 658 B2
`
`FIG . 7
`
`Clinton Dewitt Firestone IV
`
`July 12 , 1896
`Birth :
`April 29 , 1971
`Death :
`Parents : Clinton Dewitt Firestone III and Viola Miller
`Comments : He was a WWII U . S . Air force pilot and POW in WWII and veteran
`honorably discharged in December of 1947 . He worked for 44 years for the
`Firestone Tire and Rubber Company in retail , wholesale and original equipment
`sales , marketing and management . He was born in Akron , OH and is buried in
`Columbiana , OH .
`
`Edit bio
`
`Locations
`
`Timeline
`
`Family Tree
`
`Recipes
`
`1
`
`. 04 . 1 . 0 . 1 . 100 . 000
`
`IRY 227 1882
`
`. .
`
`.
`
`EEEEEEEEEE
`
`MemoryWeb Ex. 2005
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U . S . Patent
`
`Sep . 24 , 2019
`
`Sheet 8 of 50
`
`US 10 , 423 , 658 B2
`
`FIG . 8
`
`Descendants of Charlotte Killian Halter
`
`Tree Table
`
`M
`
`NEW
`
`MemoryWeb Ex. 2005
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U . S . Patent
`
`Sep . 24 , 2019
`
`Sheet 9 of 50
`
`US 10 , 423 , 658 B2
`
`FIG . 9
`
`Year
`Month
`Dayrim w
`
`Cousins Party 2010
`
`Fishing in Yellowstone
`Montana September 2010
`
`22 12
`
`MemoryWeb Ex. 2005
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U . S . Patent
`
`Sep . 24 , 2019
`
`Sheet 10 of 50
`
`US 10 , 423 , 658 B2
`
`FIG . 10
`
`•
`
`•
`•
`
`1 Zucchini ( chopped )
`Thai Basil ( 8 leaves )
`Lemon Grass ( in jar ) 1 teaspoon
`Chinese Ginger Root ( in jar ) 1 teaspoon
`
`Rice
`•
`Thai Rice ( something that only takes 2 cups of water )
`•
`Dice chicken in bowl and add two tablespoons of fish
`sauce . Let marinate for 20 minutes .
`Take thick part of coconut milk out into pan ( about 4
`tablespoons ) , Curry paste , 1 spoon of lemon grass , 1
`spoon of ginger and garlic . Heat over high with boil and
`THEN stir for 1 minute . Add meat ( uncooked ) and fry
`until cooked over high heat
`Add milk , brown sugar and salt . Bring back to slight boil
`and constantly stir . Add veggies and soy sauce . Cook for
`about 10 - 14 minutes COVERED until veggies are cooked .
`Serve with a smile .
`
`Desmond ' s Yellow Thai Chicken Curry
`Curry Mix
`• Coconut milk ( 400 ml ) - DO NOT SHAKE IT UP
`•
`800 gram of chicken ( 4 chicken breast )
`Fish sauce ( Nam Pla ) Thai Bamboo Garden - Bottle
`• Garlic ( 2 cloves )
`Broccoli ( 2cups chopped )
`•
`•
`2 Peppers ( chopped )
`2 Carrots ( chopped )
`
`Chef : Barry Desmond
`WWW
`
`Video on How to Make It
`
`Original Handwritten Recipe
`
`MemoryWeb Ex. 2005
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U . S . Patent
`
`Sep . 24 , 2019
`
`Sheet 11 of 50
`
`US 10 , 423 , 658 B2
`
`FIG . 11
`
`Chicago , IL
`
`Album / Event
`Jack Monk ' s Arrival
`Mike Testy ' s First Birthday
`Cubs Beat Cards 1998
`Jack Monk ' s Arrival .
`Mike Testy ' s 2nd Birthday
`Cubs Beat Cards 1998
`Jack Monk ' s Arrival
`Mike Testy ' s 3rd Birthday
`Cubs Beat Cards 1998
`Jack Wrigley Monk ' s Arrival
`Mike Testy ' s 4th Birthday
`Cubs Beat Cards 1998
`Nancy Learns How to Ride a Bikel
`
`# Photos
`Location
`Date
`| 691
`Chicago , IL
`26 - Dec - 2003 |
`13 - Sep - 1983 Minneapolis , MN
`54
`36
`5 - Aug - 1998
`Chicago , IL
`29 - Dec - 2003
`Chicago , IL
`13 - Sep - 1983 Minneapolis , MN
`54
`5 - Aug - 1998
`Chicago , IL
`36
`69
`29 - Dec - 2003
`13 - Sep - 1983 | Minneapolis , MN 54
`5 - Aug - 1998
`Chicago , IL
`36
`29 - Dec - 2003
`Chicago , IL
`69
`54
`13 - Sep - 1983
`Minneapolis , MN
`5 - Aug - 1998
`Chicago , IL
`36
`21 - Jul - 1978
`St . Louis , MO |
`76
`
`GO
`
`1
`
`|
`
`|
`
`Thumbnail | Table
`
`# videos
`4
`21
`2 , 199
`4 .
`21
`2 , 199
`4 .
`21
`2 , 199
`4
`21
`2 , 199
`2
`
`# Docs
`
`|
`
`0
`
`EEEEE282
`
`PPPPPPPPPPmrrrrrrrrrrrrrrrrrrr ? 222222222222222222222222222222222222222222
`
`MemoryWeb Ex. 2005
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U . S . Patent
`
`Sep . 24 , 2019
`
`Sheet 12 of 50
`
`US 10 , 423 , 658 B2
`
`FIG . 12
`
`Thumbnail Table
`
`# Docs
`4
`
`# Videos
`4
`
`21
`2 , 199
`
`|
`
`# Photos
`69
`54
`
`36 .
`
`Album / Event
`Date
`26 - Dec - 2003
`Jack Monk ' s Arrival
`13 - Sep - 1983
`| Mike Testy ' s First Birthday
`5 - Aug - 1998
`Cubs Beat Cards 1998
`29 - Dec - 2003
`Jack Monk ' s Arrival
`13 - Sep - 1983
`Mike Testy ' s 2nd Birthday
`5 - Aug - 1998
`Cubs Beat Cards 1998
`29 - Dec - 2003
`Jack Monk ' s Arrival
`13 - Sep - 1983
`Mike Testy ' s 3rd Birthday
`5 - Aug - 1998
`Cubs Beat Cards 1998
`29 - Dec - 2003
`Jack Wrigley Monk ' s Arrival
`13 - Sep - 1983
`Mike Testy ' s 4th Birthday
`5 - Aug - 1998
`Cubs Beat Cards 1998
`Nancy Learns How to Ride a Bikel 21 - Jul - 1978
`
`Location
`Chicago , IL
`Minneapolis , MN
`Chicago , IL
`Chicago , IL
`|
`Minneapolis , MN
`|
`Chicago , IL
`Chicago , IL
`Minneapolis , MN |
`Chicago , IL
`Chicago , IL
`Minneapolis , MN
`Chicago , IL
`St . Louis , MO
`
`69
`54
`21
`2 , 199
`36
`4
`69
`21
`54
`362 , 199
`69
`54
`36
`76
`
`21
`2 , 199
`2
`
`L
`
`0
`
`MemoryWeb Ex. 2005
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U . S . Patent
`
`Sep . 24 , 2019
`
`Sheet 13 of 50
`
`US 10 , 423 , 658 B2
`
`FIG . 13
`
`Thumbnail Table
`
`# Videos
`
`# People
`2
`
`# Photos
`8
`
`Last Name
`
`+ Alberts
`+ Annex
`+ Bade
`+ Bacon
`+ Bates
`+ Boone
`+ Danas
`+ Danes
`- Monk ( All )
`Monk , CJ
`Monk , Jack
`+ Firestone
`+ Moore
`+ Slythe
`+ Stein
`+ Testy
`+ Testy
`
`I
`
`|
`|
`
`4
`5
`6
`
`8
`2
`1
`1
`21
`1
`
`2
`4
`
`|
`
`|
`
`8
`7
`6
`
`499
`200
`199
`1249
`4
`249
`7882
`788
`
`EMA
`
`rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr
`
`MemoryWeb Ex. 2005
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U . S . Patent
`
`Sep . 24 , 2019
`
`Sheet 14 of 50
`
`US 10 , 423 , 658 B2
`
`FIG . 14
`
`Last Name
`Alberts , John
`Killian , Jack
`Killian , Brian
`Killian , Kevin
`Killian , Sarah
`Killian , John
`Killian , Mark
`Killian , Louis
`Killian , John
`$ Monk , CJ
`Monk , Jack
`Firestone , Mike
`Moore , Bertha
`Slythe , Sarah
`Killian , John
`Killian , Mike
`
`Relationship
`Cousin
`Son
`Nephew
`Nephew
`Daughter - in - law
`Great Nephew
`Great Nephew
`Great Grandson
`Grandson
`Great Grandson
`Great Grandson
`Third Cousin
`Great Niece
`Sister
`Brother
`Brother
`
`Thumbnail | Table
`# Docs
`
`# Photos # Videos
`8
`0
`
`0
`7
`0
`0
`0
`8
`180
`1
`2
`
`6
`
`3
`4
`
`7
`499
`200
`1992
`1249 |
`17
`
`|
`
`90
`2491
`249
`| 788
`|
`2
`
`|
`
`/ / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / / /
`
`/
`
`MemoryWeb Ex. 2005
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U . S . Patent
`
`Sep . 24 , 2019
`
`Sheet 15 of 50
`
`US 10 , 423 , 658 B2
`
`FIG . 15
`
`Address
`
`Location Name 1
`Dom
`Lucilla & Roberto
`Lisle Home
`1898 West St
`College
`545 Market
`Amazon Trip
`Cabin
`Grad School
`Griffith Park
`LA Equestrian Ctr
`Del Coronado
`Fenway Park
`Wrigley Field
`Home
`GA Grill Party
`Pike ' s Market
`Raffels
`
`Manus
`Lake Geneva WI
`999 Pine
`Charleston
`903 Plymouth
`IL
`| CA
`Los Feliz
`298 Glencarin
`Glendale
`568 Horse Dr
`CA
`12 Coronado Dr Coronado
`CA
`123 Yawke
`Boston
`MA
`1190 W Addison
`444 Main
`321 Silver
`786 Market
`345 Fong
`
`Chicago
`Anywhere
`
`Macon
`
`Seattle
`Singapore
`
`|
`
`City
`Cologne
`Montalcino
`Lisle
`Akron
`
`State
`
`I
`
`IL
`OH
`
`Country
`Germany
`Italy
`USA
`USA
`
`Thumbnail | Table
`
`# Photos
`3
`
`6
`45
`
`# Videos # Docs
`2
`0
`1
`0
`122
`
`235
`98
`
`1256
`
`12
`
`321
`57
`498
`10 , 987
`15
`18
`23
`
`?????????????????????????????????
`
`herpressed
`
`49
`
`1
`2
`
`|
`
`|
`
`0
`
`Brazil
`USA
`
`USA
`
`USA
`USA
`USA
`USA
`
`USA
`USA
`USA
`USA
`Singapore |
`
`IL
`
`GA
`WA
`
`MemoryWeb Ex. 2005
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U . S . Patent
`
`Sep . 24 , 2019
`
`Sheet 16 of 50
`
`US 10 , 423 , 658 B2
`
`FIG . 16
`
`Category | Card | Table
`
`Recipe
`Blacks Yellow Thai Chicken Curry
`Skinny Germans
`KFC in a Bag
`Shit on a Shingle
`Mrs . Fields Cookies
`Chicken Pot Pie
`Roll Your Own Dough
`Pizza ala Franciscan
`Meatball Delight
`Almond Cookies
`Jumpin Jack Flap Jacks
`Vicki ' s Chow Mein
`Fat Steak
`Mud Pie
`Caesar Salad
`Daddio Pancakes
`
`Chef
`Jack Black
`Gerda
`
`The Kernal
`George James
`Mrs . Fields
`Jack Black
`Vito Spadavecchio
`Charles Faso
`Ben Delight
`Lori James
`Jack Jack
`Vicki Firestone
`Barry Monk
`Nancy Monk
`Christopher Monk
`Barry Monk
`
`Date
`31 Jan 2010
`29 Dec 2003
`13 Sept 1988
`5 Aug 1998
`21 July 1978
`31 Jan 2010
`29 Dec 2003
`13 Sept 1988
`5 Aug 1998
`21 July 1978
`31 Jan 2010
`29 Dec 2003
`13 Sept 1988
`5 Aug 1998
`21 July 1978
`2 March 2011
`
`Category
`Dinner
`Breakfast
`Anytime
`Anytime
`Dessert
`Dinner
`Dinner
`Dinner
`Dinner
`Dessert
`
`Breakfast
`Dinner
`Dinner
`Dessert
`Anytime
`Breakfast
`
`mmmmmrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr .
`
`MemoryWeb Ex. 2005
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U . S . Patent
`
`Sep . 24 , 2019
`
`Sheet 17 of 50
`
`US 10 , 423 , 658 B2
`
`FIG . 17
`
`Close ( X )
`
`WWWW
`
`MemoryWeb Ex. 2005
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U . S . Patent
`
`Sep . 24 , 2019
`
`Sheet 18 of 50
`
`US 10 , 423 , 658 B2
`
`FIG . 18
`
`Advanced Search Filter
`Keywords :
`Add
`Add
`Date :
`Location :
`
`Event :
`
`Jack ' s 1st Birthday
`Fly Fishing in Yellowstone
`Raking Leaves
`Christmas 2010
`Thanksgiving 2010
`July 4th Parade
`Add
`Album :
`Star Ranking :
`Add
`Sharing Rights : Add
`Document Type : Add
`
`People :
`
`Chicago , IL USA
`Yellowstone , MT USA
`Cologne , Germany
`
`Mike Cubbie
`Mary Lamb
`Christopher Monk
`Nancy Monk
`Dwight Schrut
`
`MemoryWeb Ex. 2005
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U . S . Patent
`
`Sep . 24 , 2019
`
`Sheet 19 of 50
`
`US 10 , 423 , 658 B2
`
`Captain Phil ' s Memory - Webb
`
`Welcome , Captain Phil
`Last Login :
`11 . 18 . 2010
`
`My recent memories :
`•
`123 Photos uploaded on 11 . 07 . 10
`2 albums created 11 . 17 . 10
`12 visitors since last login date
`•
`123 Photos uploaded on 11 . 07 . 10
`•
`2 albums created 11 . 17 . 10
`•
`My recent Webb views :
`•
`Captain Phil 2010 ( photo album )
`•
`Chicken Pot Pie ( recipe )
`•
`Captain Phil ( Timeline )
`Updates and Alerts :
`License renewal due 1 . 15 . 2011
`
`FIG . 19
`
`Archive Status
`
`Media
`# Photos
`# Videos
`# Documents
`
`Count
`1 , 342
`75
`173
`
`Count
`80 % complete
`61 % complete
`
`People Stats :
`
`Last Name
`
`Monk
`Firestone
`Testy
`
`|
`|
`
`# People
`7
`
`|
`111
`4
`|
`
`# Photos
`499
`, 249
`788
`
`# Videos
`4 .
`17
`17
`1
`
`T
`
`L
`
`# Docs
`14
`9
`39
`
`3
`
`1
`
`Event Stats :
`Event
`Mike Testy ' s 1st Birthday
`
`Cubs Beat Cards Aug 1998
`Nancy Learns to Ride Bike
`
`Date
`Date
`13 - Sept - 1988
`5 - Aug - 1998
`21 - July - 1978
`
`Location
`Minneapolis , MN |
`Chicago , IL
`St . Louis , MO
`
`# Media
`# Media
`21
`2 , 199
`
`MemoryWeb Ex. 2005
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U . S . Patent
`
`Sep . 24 , 2019
`
`Sheet 20 of 50
`
`US 10 , 423 , 658 B2
`
`FIG . 20
`
`0050
`
`0401
`
`0501
`
`Start
`
`1
`
`0101
`
`Original Digital
`file ( s ) upload to
`MemoryWeb
`
`Third Party Facial
`Recognition
`Provider
`
`Third Party Media Provider
`
`1101
`
`Third Party
`Geographical
`Mapping Provider
`
`1201
`
`individual /
`Group of
`People
`
`A
`
`1301
`MW Modified File
`exported from
`MemoryWeb to Storage
`System
`
`- - - - -
`
`
`
`wwwwwwwwwww " "
`
`"
`
`"
`
`"
`
`"
`
`"
`
`" wwwwww
`
`wwwwwwwwwww
`
`Outside MW System
`Inside MW System
`
`wwwmmmNwwwwwwwwwwwwwwwwwwwwwwwwwww NMMWWWWWW
`
`0400
`
`0500
`
`1000
`
`Third Party Facial
`Recognition
`System
`
`Third Party Media
`System ( Data
`Exchange )
`
`0300
`
`Share to Social
`Network Provider
`System
`
`1100
`
`System Reading
`Phase
`
`System Interpreting and
`Adding Data to
`Relationship Table Phase
`
`0100
`
`0200
`
`User Relationship
`Table
`System Indexing
`Digital Files and
`Tag Data Blocks
`for a Digital File
`Within User
`Relationship Table
`
`Third Party Geographical
`Location Mapping
`System
`
`1200
`
`Share to individual
`System
`
`1300
`
`Application Export System
`
`Application Digital
`Tag Organizer
`System
`
`Continuous Link
`of Application
`Dot - Tags System
`
`Advanced Filters
`System
`
`Keyword Fast
`Search System
`
`0700
`
`0800
`
`0900
`
`MemoryWeb Ex. 2005
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U . S . Patent
`
`Sep . 24 , 2019
`
`Sheet 21 of 50
`
`US 10 , 423 , 658 B2
`
`System Reading Phase
`
`3 Phase
`
`Start
`
`0100
`
`0101
`
`Original Digital file begins to
`upload to MemoryWeb
`
`Outside MW System
`Inside MW System
`
`0102
`
`Duplicate
`File ?
`
`Yes
`
`Locate EXIF
`Image File
`Directories in
`Digital File
`
`0103
`
`FIG . 21
`System Interpret
`- - System Interpreting and Adding Data to Relationship Table Phase
`
`-
`
`1
`
`??? ?? ?? ?? ?
`
`-
`
`?? ?? ???? ??
`
`*
`
`-
`
`*
`
`*
`
`-
`
`*
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`Third Party Facial
`Recognition Provider
`
`0401
`
`0400
`Third Party Facial Recognition
`System
`
`0201
`
`Read and iterate through
`each EX { F tag item
`
`is MW Tag ?
`
`No
`
`Outside MW System
`Inside MW System
`
`0211
`
`Upload origina !
`file into MW
`
`0203
`
`Create duplicates
`of original file in
`different file sizes
`
`0205
`
`0200
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`System Indexing Digital Files and
`Tag Data Blocks for a Digital File
`within User Relationship Table
`0300
`
`User Relationship Table
`1900
`
`User Settings
`
`0301
`
`File not
`uploaded
`
`0104
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`0204
`
`-
`
`L
`
`wwwwwww
`wwww
`
`Ves
`
`0207
`
`Save EXIF data to
`User Relationship
`Table for Digital
`File
`
`Parse EXIF
`data into
`MemoryWeb
`Tag data
`
`look up MW
`tag data
`
`0206
`
`0209
`
`-
`
`-
`
`-
`
`0208
`
`NO
`Tag data
`exists ?
`
`Create new
`MW tag
`data
`
`0210
`Associate
`existing Tag
`data
`
`Storage of
`Original File
`XL
`Dunlimate File
`Large
`Duplicate File
`Medium
`Duplicate File
`Small
`Duplicate File
`Thumbnail
`Duplicate File
`
`EXIF Tag ( Data
`Blocks )
`Microsoft
`Windows Tag
`( Data Blocks )
`MemoryWeb
`Tag ( Data Blocks )
`MemoryWeb
`Person Tag ( Data
`Blocks )
`
`0302
`0303
`
`0304
`0305
`0305
`0320
`
`0340
`
`03601
`
`0380
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`MemoryWeb Ex. 2005
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U . S . Patent
`
`Sep . 24 , 2019
`
`Sheet 22 of 50
`
`US 10 , 423 , 658 B2
`
`FIG . 22
`
`0320
`
`MemoryWeb Tag ( Data Blocks )
`
`0360
`
`03231
`Location
`Ox9c9b or
`0x010e
`0x9c9f
`
`EXIF Tags version 2 . 3 Image File Directories
`( Data Blocks )
`0322
`EXIF Family
`0321
`Group Name
`Tag Labels
`
`0324
`
`0325 —
`
`0326 -
`
`0327 -
`0328
`
`0329
`
`0330 -
`
`0331 -
`
`0332
`
`0333
`
`Description Title
`Description Subject
`Description Rating
`Description Tags
`Description Comments
`Origin Authors
`Origin Date Taken
`Origin Date Acquired
`Origin Copyright
`Jimage ( Image ID , Dimensions , Width Height , etc )
`Width
`Height
`Camera Camera Maker , Camera Model , etc )
`Advanced Photo Lens Maker , Lens Model , etc )
`
`User Comment
`GPS Latitude
`TGPS Latitude Ref
`GPS Longitude
`GPS Longitude Ref
`GPS Altitude
`
`www
`
`.
`
`IFDO
`IFDO
`
`LEDO
`1FDO
`IFDO
`ExifIFD
`
`JIEDO
`
`0334
`prentament
`
`ExifIFD
`GPS
`GPS
`GPS
`GPS
`GPS
`
`www . wwwwwwwwwwwwwwwww
`
`N / A
`ox9c9e
`Ox9c9c
`0x9c9d
`0x9003
`N / A
`0x8298
`Multiple
`
`Oxbc80
`Oxbc81
`Multiple
`Multiple
`
`Ox9286
`0x0002
`0x0003
`0x0004
`0x0005
`Ox0006
`
`MemoryWeb Tag
`
`MediaAsset . Caption
`
`MediaAsset . Star Ranking
`
`0335
`
`MediaAsset . DateCreated
`
`MediaAsset . Width
`MediaAsset . Height
`
`0361
`
`0362
`
`10363
`
`0364
`
`- 0365
`
`0366
`
`This is used to inject information that do not currently have
`EXIF standardized tags including Collection , People , location
`Name , Recipe Name , Person Tag Data Blocks ( 0380 ) , etc .
`MediaAsset . Location . Latitude
`MediaAsset . Location , Latitude
`MediaAsset . Location . Longitude
`MediaAsset . Location . Longitude
`
`0367
`pone 0368
`0369
`0370
`- 0371
`
`MemoryWeb Ex. 2005
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U . S . Patent
`
`Sep . 24 , 2019
`
`Sheet 23 of 50
`
`US 10 , 423 , 658 B2
`
`FIG . 23
`
`0340
`
`0360
`
`0342
`
`0343
`
`Windows Imaging Component ( WIC ) Metadata
`( Data Blocks )
`
`Tag Laber
`
`0341
`
`File Name
`File Item Type
`File Folder Path
`File Date Created
`File Date Modified
`File Size
`File Attributes
`File ( Offline availability , Offline status , Shared with ,
`Owner , Computer , etc ) .
`Other
`
`w
`
`ww
`
`. .
`
`MemoryWeb _ Tag ( Data Blocks )
`
`MemoryWeb Tag
`
`MediaAsset . Filename
`
`MediaAsset . Upload Batch . BatchName
`
`0361
`
`+
`
`0372
`
`0372
`
`0373
`
`MemoryWeb Ex. 2005
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U . S . Patent
`
`Sep . 24 , 2019
`
`Sheet 24 of 50
`
`US 10 , 423 , 658 B2
`
`FIG . 24
`
`0380
`
`MemoryWeb Person Tag ( Data Blocks )
`Person Name
`Nickname
`Birthdate
`Date of Death
`Biography
`Mother
`
`Father
`Brother 1 , Brother 2 , . . .
`Sister 1 , Sister 2 , . . .
`Daughter 1 , Daughter 2 , . . .
`Son 1 , Son 2 . . . .
`Spouse 1 , Spouse 2 , . . .
`Facial Recognition Data ( Taylor ? )
`Facebook ID
`pets
`
`0395
`- 0381
`0382
`- 0383
`0384
`
`- 0385
`0386
`0387
`0388
`
`0389
`
`0390
`
`0391
`
`0392
`+ 0393
`- 0394
`+ 0396
`
`MemoryWeb Ex. 2005
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U . S . Patent
`
`Sep . 24 , 2019
`
`Sheet 25 of 50
`
`US 10 , 423 , 658 B2
`
`FIG . 25
`
`Third Party Facial Recognition Provider
`
`0401
`
`0400
`
`Start
`
`0402
`V
`System request for
`thumbnails of all
`Digital Files that have
`unconfirmed faces
`
`0300
`
`User
`Relationship
`Table
`
`0404
`System sends
`Digital Files to
`Third Party
`Facia !
`Recognition
`Provider
`
`0403
`
`Retrieves
`Thumbnails of all
`Digital Files that
`have unconfirmed
`faces
`
`0405
`System receives
`response from
`Third Party Facial
`Recognition
`Provider
`
`0406
`
`System parses
`identified faces
`and corresponding
`Facial Recognition
`data for each
`Digital File
`
`Outside MW System
`Inside MW System
`
`0407
`
`0408
`
`Existing ( Trained ) Face ID
`in User Relationship
`Table ?
`
`Is Face Recognition
`above score
`threshold ?
`
`es . . . .
`
`0409
`
`System records and associates specific
`face for Digital File with a MemoryWeb
`Person ID and sends to MemoryWeb
`Person Tag ( Data Blocks ) in User
`Relationship Table
`
`0410
`
`System generates facial
`recognition record for unknown
`person and then sends
`information to MemoryWeb
`Person Tag Data Blocks ) in User
`Relationship Table
`
`No
`
`0411
`System generates virtual
`unconfirmed facial
`recognition record for person
`and then sends information to
`MemoryWeb Person Tag
`( Data Blocks ) in User
`. . . Relationship Table
`
`MemoryWeb Ex. 2005
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U . S . Patent
`
`Sep . 24 , 2019
`
`Sheet 26 of 50
`
`US 10 , 423 , 658 B2
`
`Outside MW System
`
`Inside MW System
`Systems sends
`registration
`confirmation
`
`0503
`
`| Start
`
`0502
`User selects sharing of
`their information with
`Third Party Media
`Provider with
`MemoryWeb
`
`FIG . 26
`
`Third Party Media Provider
`
`0501
`
`0507
`
`System receives contact names
`and IDs from Third Party Media
`Provider ( daily )
`
`0506
`
`System requests contact
`names and IDs from Third
`Party Media Provider
`( daily )
`
`0300
`
`me 0509
`System associates
`existing person to Tag
`Data Block of
`Relationship Table for
`Digital File
`
`Is Person a
`Match
`
`No
`
`0508
`Yes >
`
`0510
`
`0504
`
`V
`Systems receives
`registration
`confirmation , Third
`Party Media D
`0505
`System sends
`registration
`information to
`User Setting
`Table ?
`
`0500
`
`Wwwwwwwwwwww
`
`w wwwwwwwww . *
`
`User Relationship
`Table
`
`User Settings
`Table
`
`1900
`
`System add new person to Tag
`Data Block of Relationship
`Table for Digital File
`
`1000
`
`Share to Third Party Media
`Provider System
`
`MemoryWeb Ex. 2005
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U . S . Patent
`
`Sep . 24 , 2019
`
`Sheet 27 of 50
`
`US 10 , 423 , 658 B2
`
`FIG . 27
`
`1900
`
`Item
`
`1901
`
`1902
`1903
`
`User ' s Name
`Payment ID
`Password
`
`1904
`
`1905
`1906
`
`1907
`
`Account Type
`User ' s email
`Language preference
`Date format
`Email notifications
`Contacts ( with third Party Social Media )
`1910
`Facebook ID
`1911 — + API Token
`1912 –
`Payment Date
`1912
`
`1908
`1909
`
`1913
`
`MemoryWeb Ex. 2005
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U . S . Patent
`
`Sep . 24 , 2019
`
`Sheet 28 of 50
`
`US 10 , 423 , 658 B2
`
`Start
`
`User selects
`Digital File in
`
`0601
`
`0602
`
`System sends
`request to
`Relationship
`Table for specific
`Digital File
`
`0603
`System retrieves
`Digital File and
`Digital File Tag
`Data Blocks
`from User
`Relationship
`Table
`
`0604
`
`System will
`display the
`Digital File and
`the
`corresponding
`Digital File Tag
`Data Blocks in
`the form of
`Application Dot
`Tags
`
`User
`Relationship
`Table
`
`0300
`
`FIG . 28
`
`Continuous Link
`of Application
`Dot - Tag System
`
`0700
`
`Yes
`
`Tag data exists ?
`
`0607
`
`NO
`
`Yes
`
`Tag data exists ?
`
`0613
`
`0608
`User selects
`matching
`MemoryWeb Tag
`
`0609
`User adds
`New MemoryWeb
`Jag
`
`. . .
`
`0614
`User selects
`matching
`MemoryWeb Tas
`
`0615
`www User adds
`New MemoryWeb
`Tag
`
`0605
`
`User selects
`Application Dot
`Tag
`
`0606
`
`User selects add
`for a
`MemoryWeb
`Tag
`
`0612
`
`User selects edit
`for a
`MemoryWeb
`Application Dot
`Tag
`
`0618
`User selects
`delete
`for a
`MemoryWeb
`Application Dot
`Tag
`
`0610
`System associates
`new MemoryWeb tag
`to Tag Data Block of
`Relationship Table for
`Digital File
`System creates new
`MemoryWeb tag to
`Tag Data Block of
`Relationship Table for
`Digital File
`System associates
`existing MemoryWeb
`Tag to Tag Data Block
`of Relationship Table
`for Digital File
`System creates new
`MemoryWeb tag to
`Tag Data Block of
`Relationship Table for
`Digital File
`System deletes
`association of
`MemoryWeb tag to
`Tag Data Block of
`Relationship Table for
`Digital file
`
`0611
`
`0616
`
`0617
`
`0619
`
`MemoryWeb Ex. 2005
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U . S . Patent
`
`Sep . 24 , 2019
`
`Sheet 29 of 50
`
`US 10 , 423 , 658 B2
`
`FIG . 29
`
`Structure
`
`0650
`
`0652
`
`0551
`annetaan ennenstan
`User Defined Tag Laber
`
`# # # #
`
`0 653
`
`Examples
`
`Within Character Unit for labels and Numbers
`
`Dotted Application Dot - Tag denotes partial relationship . In
`this example , person is a half - sibling to another person .
`
`453 Cologne Germany
`
`45
`
`Frank Smith
`
`0654
`Exceeds Character limit for Label and Numbers
`
`2999 Holiday Photos from . . .
`
`0657
`
`0658
`
`0655
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`-
`
`0656
`
`MemoryWeb Ex. 2005
`Unified Patents v. MemoryWeb – IPR2021-01413
`
`

`

`U . S . Patent
`
`Sep . 24 , 2019
`
`Sheet 30 of 50
`
`US 10 , 423 , 658 B2
`
`Start
`
`User selects
`Application Dot
`Tag
`
`0701
`
`0702
`
`Is Person
`Dot - Tag ?
`
`Yes
`
`No
`
`0703
`Is Collection
`Dot - Tage
`
`No
`V
`
`0704
`Is Location
`Dot - Tag ?
`
`0705
`
`No
`V
`Is Date
`Dot - Tag ?
`
`Yes
`
`Yes
`
`Yes >
`Yes
`
`No
`
`V
`
`Yes
`
`0706
`
`Is Recipe
`Dot - Tag ?
`
`0707
`
`No
`V
`
`Ves
`
`0708
`System sends
`request to
`Relationship
`Table
`
`User
`Relationship
`Table
`
`0300
`
`FIG . 30
`
`0700
`
`0716
`
`Displays image
`in Slideshow
`view
`
`0709
`
`If Person Application Dot - Tag , system retrieves data
`for that person from User ' s User Relationship Table
`and displays relationship data in People Profile View
`
`0710
`If Collection Application Dot - Tag , system retrieves
`data from that collection from User ' s User
`Relationship Table and displays relationship data in
`Collection View
`
`0711
`If Location Application Dot - Tag , system retrieves data
`from that Location from User ' s User Relationship Table