I 1111111111111111 1111111111 111111111111111 111111111111111 IIIIII IIII IIII IIII
`US010423658B2
`
`c12) United States Patent
`Desmond et al.
`
`(IO) Patent No.: US 10,423,658 B2
`*Sep.24,2019
`(45) Date of Patent:
`
`(54) METHOD AND APPARATUS FOR
`MANAGING DIGITAL FILES
`
`(71) Applicant: MemoryWeb, LLC, Glen Ellyn, IL
`(US)
`
`(72)
`
`Inventors: Christopher J. Desmond, Glen Ellyn,
`IL (US); Nancy L. Desmond, Glen
`Ellyn, IL (US); L. Michael Taylor,
`Chicago, IL (US)
`
`(73) Assignee: NCM IP HOLDINGS, LLC, Glen
`Ellyn, IL (US)
`
`( *) Notice:
`
`Subject to any disclaimer, the term ofthis
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 518 days.
`
`This patent is subject to a terminal dis(cid:173)
`claimer.
`
`(21) Appl. No.: 15/375,927
`
`(22) Filed:
`
`Dec. 12, 2016
`
`(65)
`
`Prior Publication Data
`
`US 2017/0091225 Al Mar. 30, 2017
`
`(63)
`
`(51)
`
`(52)
`
`Related U.S. Application Data
`
`Continuation of application No. 14/193,426, filed on
`Feb. 28, 2014, now Pat. No. 9,552,376, which is a
`(Continued)
`
`Int. Cl.
`G06F 16190
`G06F 16158
`
`(2019.01)
`(2019.01)
`(Continued)
`
`U.S. Cl.
`CPC .......... G06F 1615866 (2019.01); G06F 16151
`(2019.01); G06F 161901 (2019.01); G06F
`161907 (2019.01); G06F 3/0481 (2013.01)
`
`(58) Field of Classification Search
`CPC ... G06F 16/5866; G06F 16/907; G06F 16/901
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`5,694,514 A
`6,629,104 Bl
`
`12/1997 Evans
`9/2003 Parulski
`(Continued)
`
`OTHER PUBLICATIONS
`
`Kustanowitz et al., "Motivating Annotation for Personal Digital
`Photo Libraries: Lowering Barriers while Raising Incentives," Tech.
`Report HCIL-2004-18, U. Maryland, 2005 (10 pages).
`(Continued)
`
`Primary Examiner - Loe Tran
`(74) Attorney, Agent, or Firm - Nixon Peabody LLP
`
`(57)
`
`ABSTRACT
`
`A computer-implemented method of associating digital tags
`with digital files comprises storing a plurality of digital files
`having embedded therein content data and metadata includ(cid:173)
`ing tags; receiving, via a user interface device of a client
`device, a first tag label containing alphanumeric text created
`and inputted by a user of the client device; modifying, using
`a controller device, a selected first one of the tags of the
`metadata in a first of the digital files to include the first tag
`label; receiving, via the user interface device or another user
`interface device, an instruction to search for all of the digital
`files having at least the first tag label; responsive to receiving
`the instruction, automatically searching for all of the digital
`files having at least the first tag label; and displaying, on a
`video display device associated with the client device, a first
`indication of the first tag label.
`
`15 Claims, 50 Drawing Sheets
`
`Select None
`Select All
`Delete Selected
`
`September 2, 2010 change
`Yellowstone, MT, USA change
`Christopher Desmond, Nancy Desmond
`Fly Fishing in Yellowstone
`Add
`Add
`Add
`
`1 File(s) Selected
`File Attributes:
`Date:
`Location:
`People:
`Event:
`Comments:
`Keywords:
`Album:
`Star Ranking:
`Sharing Rights: Add
`Recipe:
`Add
`DocumentType: Add
`File Name:
`IMG 123 change
`
`Petitioner Apple Inc. - Ex. 1012, p. 1
`
`

`

`US 10,423,658 B2
`Page 2
`
`Related U.S. Application Data
`
`continuation-in-part of application No. 13/157,214,
`filed on Jun. 9, 2011, now Pat. No. 9,098,531.
`
`Yee et al., "Faceted Metadata for Image Search and Browsing", CHI
`2003, pp. 401-408, 2003, ACM.
`Ferre, "CAMELIS: Organizing and Browsing a Personal Photo
`Collection with a Logical Information System", Int. Conf. Concept
`Lattices and Their Applications, pp. ll2-123, 2007, HAL.
`Tomasson et al., "PhotoCube: Effective and Efficient Multi(cid:173)
`Dimensional Browsing of Personal Photo Collections", ICMR 'll,
`20ll, ACM.
`Bartolini et al., "Integrating Semantic and Visual Facets for Brows(cid:173)
`ing Digital Photo Collections", SBED, pp. 65-72, 2009.
`Trattner et al., "Evaluating Tag-Based Information Access in Image
`Collections", Proceedings of the 23 rd ACM Conference on Hyper(cid:173)
`text and Social Media, pp. 113-122, 2012 ACM.
`Kang et al., "Capture, Annotate, Browse, Find, Share: Novel
`Interfaces for Personal Photo Management", International Journal
`of Human-Computer Interaction, 23(3), pp. 315-337, 2007, Law(cid:173)
`rence Eribaum Associates, Inc.
`Jaffe et al., "Generating Summaries and Visualization for Large
`Collections of GeoReferenced Photographs", MIR'06, pp. 89-98,
`2006 ACM.
`Tornial et al., "Sharing, Discovering and Browsing Geotagged
`Pictures on the Web", 2007, Hewlett-Packard Development Com(cid:173)
`pany, L.P., pp. 1-18.
`Snavely et al., "Photo Tourism: Exploring Photo Collection in 3D",
`SIGGRAPH '06 ACM Transactions on Graphics, vol. 25, Issue 3,
`pp. 835-846, 2006 ACM.
`Kisilevich et al., "Event-based analysis of People's Activities and
`Behavior Using Flickr and Panoramio Geotagged Photo Collec(cid:173)
`tions", 14th International Conference Information Visualization, pp.
`289-296, 2010 IEEE.
`Ahern et al., "World Explorer: Visualizing Aggregate Data From
`Unstructured Text in Geo-Referenced Collections", JCDL'07, pp.
`1-10, 2007, ACM.
`Kopf et al., "Deep photo: model-based photograph enhancement
`and viewing", ACM Transactions on Graphics, vol. 27, No. 5,
`Article ll6, Dec. 2008, ACM (10 pages).
`
`* cited by examiner
`
`(51)
`
`Int. Cl.
`G06F 16151
`G06F 161901
`G06F 161907
`G06F 3/0481
`
`(2019.01)
`(2019.01)
`(2019.01)
`(2013.01)
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`7,372,976 B2
`7,480,669 B2
`7,694,236 B2
`8,150,844 B2
`8,484,223 B2
`2006/0165380 Al*
`
`5/2008 Rhoad
`1/2009 Lo
`4/2010 Gusmorino
`4/2012 Redstone
`7/2013 Ota
`7/2006 Tanaka .
`
`GllB 27/34
`386/227
`
`2007/0282908 Al
`2009/0ll3350 Al*
`
`2009/0265631 Al
`
`12/2007 Van Der Meulen
`4/2009 Hibino .................... G06F 16/41
`715/853
`
`10/2009 Sigurbjornsson
`
`OTHER PUBLICATIONS
`
`Miller et al., "Give and take: a study of consumer photo-sharing
`culture and practice," CHI ' 07 Proceedings of the SIGCHI Con(cid:173)
`ference on Human Factors in Computing Systems, pp. 347-356,
`2007 (10 pages).
`Ames et al., "Why we tag: motivations for annotation in mobile and
`online media," CHI '07 Proceedings of the SIGCHI Conference on
`Human Factors in Computing Systems, pp. 971-980, ACM, 2007
`(10 pages).
`
`Petitioner Apple Inc. - Ex. 1012, p. 2
`
`

`

`U.S. Patent
`
`Sep.24,2019
`
`Sheet 1 of 50
`
`US 10,423,658 B2
`
`FIG. 1
`
`Select None
`Select All
`Delete Selected
`
`,:-,,
`
`\:: ·): ~:lij
`
`September 2, 2010 change
`Yellowstone, MT, USA change
`Christopher Desmond, Nancy Desmond
`Fly Fishing in Yellowstone
`Add
`Add
`Add
`
`1 File(s) Selected
`File Attributes:
`Date:
`Location:
`People:
`Event:
`Comments:
`Keywords:
`Album:
`Star Ranking:
`Sharing Rights: Add
`Recipe:
`Add
`Document Type: Add
`File Name:
`IMG 123 change
`
`•••••
`
`B '
`
`a
`
`.
`
`'
`
`Petitioner Apple Inc. - Ex. 1012, p. 3
`
`

`

`U.S. Patent
`
`Sep.24,2019
`
`Sheet 2 of 50
`
`US 10,423,658 B2
`
`FIG. 2
`
`Comments:
`Suzanne and Anthony's Wedding Party where the cousins posed
`for a photo in the grass. Note, Jack with the lollipop and the
`photographer with his shoe in the photo
`
`Location:
`Historical Society
`Lisle, IL 60532
`
`People:
`Jack Wong
`CJ Wong
`Mary Firestone
`Zoe Peika
`Nick Persons
`
`Event: Suzanne & Anthony's Wedding Reception 2010
`
`Camera Details: more
`
`Petitioner Apple Inc. - Ex. 1012, p. 4
`
`

`

`U.S. Patent
`
`Sep.24,2019
`
`Sheet 3 of 50
`
`US 10,423,658 B2
`
`FIG. 3
`
`r:·~:·t/!~··.···~·
`
`: ..
`~
`it\
`j:)t!
`
`Mary Bader and Clint Firestone
`Wedding October 11, 1945
`
`Capt. Clint Firestone
`1952
`
`Germany Trip 1955
`
`Father's Day 1966
`
`College Break 1992
`
`LA Trip 1993
`
`EIU College
`Graduation 1994
`
`Cairo, Egypt 1996
`
`Thumbnail I Table
`
`.
`
`.
`
`.
`
`:::::::
`
`}\~ I'•· ~~
`
`:\::::
`Christopher pt
`Year 1972
`
`Europe
`Backpacking
`1996
`
`\;:)rn)w]
`
`·~
`
`~
`
`.$.::::.~1
`
`Amazon Fishing Trip
`November 2009
`
`Nancy James and
`Christopher Desmond
`Wedding October 17, 1998
`
`Cousins Party 2010
`
`Fishing in Yellowstone
`Montana September 2010
`
`Petitioner Apple Inc. - Ex. 1012, p. 5
`
`

`

`U.S. Patent
`
`Sep.24,2019
`
`Sheet 4 of 50
`
`US 10,423,658 B2
`
`FIG. 4
`
`Christopher Desmond and Nancy Desmond Wedding - October 17, 1998
`
`Event f}
`
`Captions On I Off
`
`◄
`
`Petitioner Apple Inc. - Ex. 1012, p. 6
`
`

`

`U.S. Patent
`U.S. Patent
`
`Sep. 24, 2019
`Sep.24,2019
`
`Sheet 5 of 50
`Sheet 5 of 50
`
`US 10,423,658 B2
`US 10,423,658 B2
`
`FIG. 5
`
`FIG. 5
`
`Petitioner Apple Inc. - Ex. 1012, p. 7
`
`Petitioner Apple Inc. - Ex. 1012, p. 7
`
`

`

`U.S. Patent
`
`Sep.24,2019
`
`Sheet 6 of 50
`
`US 10,423,658 B2
`
`FIG. 6
`
`Thumbnail I Table
`
`Clint Firestone
`
`Tim Dog
`
`Steve Fire
`
`Suzanne Lucky
`
`Steve Fwah
`
`Ben Sherman
`
`Joe Schmo
`
`◄
`
`►
`
`Mary Virginia
`
`Clint Dewitt
`
`Charlotte Killian
`
`Chris Jeep
`
`Peter LongHorn
`
`Tim Dog
`
`Suzanne Lucky
`
`Steve Fwah
`
`Ben Sherman
`
`Joe Schmo
`
`Mary Virginia
`
`Clint Dewitt
`
`Steve Fire
`
`I ~
`
`Charlotte Killian
`
`Petitioner Apple Inc. - Ex. 1012, p. 8
`
`

`

`U.S. Patent
`
`Sep.24,2019
`
`Sheet 7 of 50
`
`US 10,423,658 B2
`
`FIG. 7
`
`Clinton Dewitt Firestone IV
`
`July 12, 1896
`Birth:
`April 29, 1971
`Death:
`Parents: Clinton Dewitt Firestone Ill and Viola Miller
`Comments: He was a WWII U.S. Air force pilot and POW in WWII and veteran
`honorably discharged in December of 1947. He worked for 44 years for the
`Firestone Tire and Rubber Company in retail, wholesale and original equipment
`sales, marketing and management. He was born in Akron, OH and is buried in
`Columbiana, OH.
`
`Edit bio
`
`Recipes
`
`Family Tree
`
`f:ifi~
`
`..................................................................................
`
`Petitioner Apple Inc. - Ex. 1012, p. 9
`
`

`

`U.S. Patent
`
`Sep.24,2019
`
`Sheet 8 of 50
`
`US 10,423,658 B2
`
`FIG. 8
`
`Descendants of Charlotte Killian Halter
`
`Tree I Table
`
`Petitioner Apple Inc. - Ex. 1012, p. 10
`
`

`

`U.S. Patent
`
`Sep.24,2019
`
`Sheet 9 of 50
`
`US 10,423,658 B2
`
`Year
`
`Month
`Day
`
`f
`
`◄
`
`FIG. 9
`
`•
`
`. )
`
`►
`
`Petitioner Apple Inc. - Ex. 1012, p. 11
`
`

`

`U.S. Patent
`
`Sep.24,2019
`
`Sheet 10 of 50
`
`US 10,423,658 B2
`
`FIG. 10
`
`1 Zucchini (chopped)
`Thai Basil (8 leaves)
`Lemon Grass (in jar) 1 teaspoon
`Chinese Ginger Root (in jar) 1 teaspoon
`
`Rice
`Thai Rice (something that only takes 2 cups of water)
`Dice chicken in bowl and add two tablespoons of fish
`sauce. Let marinate for 20 minutes.
`Take thick part of coconut milk out into pan (about 4
`tablespoons), Curry paste, 1 spoon of lemon grass, 1
`spoon of ginger and garlic. Heat over high with boil and
`THEN stir for 1 minute. Add meat (uncooked) and fry
`until cooked over high heat
`Add milk, brown sugar and salt. Bring back to slight boil
`and constantly stir. Add veggies and soy sauce. Cook for
`about 10-14 minutes COVERED until veggies are cooked.
`Serve with a smile.
`
`Desmond's Yellow Thai Chicken Curry
`
`Curry Mix
`Coconut milk (400 ml)- DO NOT SHAKE IT UP
`800 gram of chicken (4 chicken breast)
`Fish sauce (Nam Pia) Thai Bamboo Garden - Bottle
`Garlic (2 cloves)
`Broccoli ( 2cups chopped)
`2 Peppers (chopped)
`2 Carrots (chopped)
`
`Chef: Barry Desmond
`
`Video on How to Make It
`
`Original Handwritten Recipe
`
`Petitioner Apple Inc. - Ex. 1012, p. 12
`
`

`

`U.S. Patent
`
`Sep.24,2019
`
`Sheet 11 of 50
`
`US 10,423,658 B2
`
`FIG. 11
`
`Thumbnail I Table
`
`Album/Event
`
`Date
`
`Location
`
`# Photos
`
`# Videos
`
`# Docs
`
`Jack Monk's Arrival
`
`26-Dec-2003
`
`Chicago, IL
`
`Mike Testy's First Birthday
`
`13-Sep-1983 Minneapolis, MN
`
`Cubs Beat Cards 1998
`
`5-Aug-1998
`
`Chicago, IL
`
`Jack Monk's Arrival
`
`29-Dec-2003
`
`Chicago, IL
`
`Mike Testv's 2nd Birthday
`
`13-Sep-1983 Minneapolis, MN
`
`Cubs Beat Cards 1998
`
`5-Aug-1998
`
`Chicago, IL
`
`Jack Monk's Arrival
`
`29-Dec-2003
`
`Chicago, IL
`
`Mike Testv's 3rd Birthday
`
`13-Sep-1983 Minneapolis, MN
`
`Cubs Beat Cards 1998
`
`5-Aug-1998
`
`Chicago, IL
`
`Jack Wrigley Monk's Arrival
`
`29-Dec-2003
`
`Chicago, IL
`
`Mike Testy's 4th Birthday
`
`13-Sep-1983 Minneapolis, MN
`
`Cubs Beat Cards 1998
`
`5-Aug-1998
`
`Chicago, IL
`
`Nancv Learns How to Ride a Bike
`
`21-Jul-1978
`
`St. Louis, MO
`
`69
`
`54
`
`36
`
`69
`
`54
`
`36
`
`69
`
`54
`
`36
`
`69
`
`54
`
`36
`
`76
`
`4
`
`21
`
`2,199
`
`4
`
`21
`
`2,199
`
`4
`
`21
`
`2,199
`
`4
`
`21
`
`2,199
`
`2
`
`4
`
`0
`
`2
`
`4
`
`0
`
`2
`
`4
`
`0
`
`2
`
`4
`
`0
`
`2
`
`0
`
`Petitioner Apple Inc. - Ex. 1012, p. 13
`
`

`

`U.S. Patent
`
`Sep.24,2019
`
`Sheet 12 of 50
`
`US 10,423,658 B2
`
`FIG. 12
`
`Thumbnail I Table
`
`Album/Event
`
`Date
`
`Location
`
`# Photos
`
`# Videos
`
`# Docs
`
`Jack Monk's Arrival
`
`26-Dec-2003
`
`Chicago, IL
`
`Mike Testy's First Birthday
`
`13-Sep-1983 Minneapolis, MN
`
`Cubs Beat Cards 1998
`
`5-Aug-1998
`
`Chicago, IL
`
`Jack Monk's Arrival
`
`29-Dec-2003
`
`Chicago, IL
`
`Mike Testv's 2nd Birthdav
`
`13-Sep-1983 Minneapolis, MN
`
`Cubs Beat Cards 1998
`
`5-Aug-1998
`
`Chicago, IL
`
`Jack Monk's Arrival
`
`29-Dec-2003
`
`Chicago, IL
`
`Mike Testy's 3rd Birthday
`
`13-Sep-1983 Minneapolis, MN
`
`Cubs Beat Cards 1998
`
`5-Aug-1998
`
`Chicago, IL
`
`Jack Wrigley Monk's Arrival
`
`29-Dec-2003
`
`Chicago, IL
`
`Mike Testv's 4th Birthdav
`
`13-Sep-1983 Minneapolis, MN
`
`Cubs Beat Cards 1998
`
`5-Aug-1998
`
`Chicago, IL
`
`Nancy Learns How to Ride a Bike
`
`21-Jul-1978
`
`St. Louis, MO
`
`69
`
`54
`
`36
`
`69
`
`54
`
`36
`
`69
`
`54
`
`36
`
`69
`
`54
`
`36
`
`76
`
`4
`
`21
`
`2,199
`
`4
`
`21
`
`2,199
`
`4
`
`21
`
`2,199
`
`4
`
`21
`
`2,199
`
`2
`
`4
`
`0
`
`2
`
`4
`
`0
`
`2
`
`4
`
`0
`
`2
`
`4
`
`0
`
`2
`
`0
`
`Petitioner Apple Inc. - Ex. 1012, p. 14
`
`

`

`U.S. Patent
`
`Sep.24,2019
`
`Sheet 13 of 50
`
`US 10,423,658 B2
`
`FIG. 13
`
`Thumbnail I Table
`
`Last Name
`
`# People
`
`# Photos
`
`#Videos
`
`# Docs
`
`+ Alberts
`
`+ Annex
`
`+ Bade
`+ Bacon
`
`+ Bates
`
`+ Boone
`
`+ Danas
`
`+ Danes
`
`- Monk (All)
`
`I Monk,CJ
`~ Monk,Jack
`+ Firestone
`
`+ Moore
`
`+ Slythe
`+ Stein
`
`+ Testv
`
`2
`
`2
`
`3
`
`4
`
`5
`
`6
`
`7
`
`8
`
`2
`
`1
`
`1
`
`21
`
`1
`
`1
`
`2
`
`4
`
`8
`
`7
`
`8
`
`8
`
`7
`
`6
`
`5
`
`7
`
`499
`
`200
`
`199
`
`1249
`
`4
`
`9
`
`249
`
`788
`
`0
`
`0
`
`0
`
`0
`
`1
`
`2
`
`4
`
`3
`
`4
`
`2
`
`2
`
`17
`
`6
`
`0
`
`1
`
`2
`
`0
`
`0
`
`0
`
`0
`
`0
`
`2
`
`1
`
`2
`
`14
`
`7
`
`7
`
`39
`
`3
`
`9
`
`3
`
`12
`
`Petitioner Apple Inc. - Ex. 1012, p. 15
`
`

`

`U.S. Patent
`
`Sep.24,2019
`
`Sheet 14 of 50
`
`US 10,423,658 B2
`
`FIG. 14
`
`Last Name
`
`Relationship
`
`# Photos # Videos
`
`# Docs
`
`Thumbnail I Table
`
`Alberts, John
`
`Killian, Jack
`
`Killian, Brian
`
`Killian, Kevin
`
`Killian, Sarah
`
`Killian, John
`
`Killian, Mark
`
`Killian, Louis
`
`Killian, John
`
`I Monk,CJ
`~ Monk,Jack
`Firestone, Mike
`
`Moore, Bertha
`
`Slvthe, Sarah
`
`Killian, John
`
`Killian, Mike
`
`Cousin
`
`Son
`
`Nephew
`
`Nephew
`
`Daughter-in-law
`
`Great Nephew
`
`Great Nephew
`
`Great Grandson
`
`Grandson
`
`Great Grandson
`
`Great Grandson
`
`Third Cousin
`
`Great Niece
`
`Sister
`
`Brother
`
`Brother
`
`8
`
`7
`
`8
`
`8
`
`7
`
`6
`
`5
`
`7
`
`499
`
`200
`
`199
`
`0
`
`0
`
`0
`
`0
`
`1
`
`2
`
`4
`
`3
`
`4
`
`2
`
`2
`
`1249
`
`17
`
`4
`
`9
`
`249
`
`788
`
`6
`
`0
`
`1
`
`2
`
`0
`
`0
`
`0
`
`0
`
`0
`
`2
`
`1
`
`2
`
`14
`
`7
`
`7
`
`39
`
`3
`
`9
`
`3
`
`12
`
`Petitioner Apple Inc. - Ex. 1012, p. 16
`
`

`

`U.S. Patent
`
`Sep.24,2019
`
`Sheet 15 of 50
`
`US 10,423,658 B2
`
`FIG. 15
`
`Thumbnail I Table
`
`Location Name
`
`Address
`
`Citv
`
`State
`
`Countrv
`
`# Photos
`
`# Videos # Docs
`
`Dom
`
`Lucilla & Roberto
`
`Cologne
`
`Montalcino
`
`Lisle Home
`
`898 West St
`
`Lisle
`
`College
`
`545 Market
`
`Amazon Trio
`
`Akron
`
`Manus
`
`IL
`
`OH
`
`Cabin
`
`999 Pine
`
`Lake Geneva WI
`
`Grad School
`
`903 Plvmouth
`
`Charleston
`
`IL
`
`Germany
`
`ltalv
`
`USA
`
`USA
`
`Brazil
`
`USA
`
`USA
`
`USA
`
`3
`
`6
`
`45
`
`64
`
`235
`
`98
`
`1256
`
`12
`
`2
`
`1
`
`12
`
`2
`
`8
`
`2
`
`32
`
`0
`
`0
`
`0
`
`2
`
`0
`
`2
`
`0
`
`4
`
`0
`
`Griffith Park
`
`298 Glencarin
`
`Los Feliz
`
`LA Equestrian Ctr
`
`568 Horse Dr
`
`Glendale
`
`Del Coronado
`
`12 Coronado Dr Coronado
`
`CA
`
`CA
`
`CA
`
`Fenwav Park
`
`123 Yawke
`
`Boston
`
`MA
`
`Wrigley Field
`
`1190 W Addison
`
`Chicago
`
`Home
`
`444 Main
`
`Anvwhere
`
`GA Grill Party
`
`321 Silver
`
`Pike's Market
`
`786 Market
`
`Macon
`
`Seattle
`
`IL
`
`IL
`
`GA
`
`WA
`
`USA
`
`USA
`
`USA
`
`USA
`
`USA
`
`USA
`
`USA
`
`Raffels
`
`345 Fong
`
`Singapore
`
`Singapore
`
`4
`
`321
`
`57
`
`498
`
`4
`
`4
`
`3
`
`7
`
`10,987
`
`49
`
`15
`
`18
`
`23
`
`0
`
`1
`
`2
`
`0
`
`0
`
`5
`
`3
`
`9
`
`0
`
`0
`
`0
`
`Petitioner Apple Inc. - Ex. 1012, p. 17
`
`

`

`U.S. Patent
`
`Sep.24,2019
`
`Sheet 16 of 50
`
`US 10,423,658 B2
`
`FIG. 16
`
`Category I Card I Table
`
`Date
`
`Category
`
`31 Jan 2010
`
`Dinner
`
`29 Dec 2003
`
`Breakfast
`
`Anvtime
`
`Anytime
`
`Dessert
`
`Dinner
`
`Dinner
`
`Dinner
`
`Dinner
`
`13 Sept 1988
`
`5 Aug 1998
`
`21 Julv 1978
`
`Dessert
`
`31 Jan 2010
`
`Breakfast
`
`13 Sept 1988
`
`George James
`
`5 Aug 1998
`
`Mrs. Fields
`
`Jack Black
`
`21 July 1978
`
`31 Jan 2010
`
`Vito Spadavecchio
`
`29 Dec 2003
`
`Chef
`
`Jack Black
`
`Gerda
`
`The Kernal
`
`Charles Faso
`
`Ben Delight
`
`Lori James
`
`Jack Jack
`
`Vicki Firestone
`
`29 Dec 2003
`
`Barry Monk
`
`13 Sept 1988
`
`Nancv Monk
`
`5 Aug 1998
`
`Dinner
`
`Dinner
`
`Dessert
`
`Christopher Monk
`
`21 Julv 1978
`
`Anvtime
`
`Barry Monk
`
`2 March 2011
`
`Breakfast
`
`Recipe
`
`Blacks Yellow Thai Chicken Currv
`
`Skinny Germans
`
`KFC in a Bag
`
`Shit on a Shingle
`
`Mrs. Fields Cookies
`
`Chicken Pot Pie
`
`Roll Your Own Dough
`
`Pizza ala Franciscan
`
`Meatball Delight
`
`Almond Cookies
`
`Jumpin Jack Flap Jacks
`
`Vicki's Chow Mein
`
`Fat Steak
`
`Mud Pie
`
`Caesar Salad
`
`Daddio Pancakes
`
`Petitioner Apple Inc. - Ex. 1012, p. 18
`
`

`

`U.S. Patent
`
`Sep.24,2019
`
`Sheet 17 of 50
`
`US 10,423,658 B2
`
`FIG. 17
`
`Close (X)
`
`◄
`
`►
`
`Petitioner Apple Inc. - Ex. 1012, p. 19
`
`

`

`U.S. Patent
`
`Sep.24,2019
`
`Sheet 18 of 50
`
`US 10,423,658 B2
`
`FIG. 18
`
`Advanced Search Filter
`
`Keywords:
`Date:
`Location:
`
`Add
`Add
`
`Chicago, IL USA
`Yellowstone, MT USA
`Cologne, Germany
`
`Mike Cubbie
`Mary Lamb
`Christopher Monk
`Nancy Monk
`Dwight Sch rut
`
`Event:
`
`Jack's pt Birthday
`Fly Fishing in Yellowstone
`Raking Leaves
`Christmas 2010
`Thanksgiving 2010
`July 4th Parade
`Add
`Album:
`Add
`Star Ranking:
`Sharing Rights: Add
`DocumentType: Add
`
`Petitioner Apple Inc. - Ex. 1012, p. 20
`
`

`

`U.S. Patent
`
`Sep.24,2019
`
`Sheet 19 of 50
`
`US 10,423,658 B2
`
`FIG. 19
`
`Captain Phil's Memory-Webb
`
`Welcome, Captain Phil
`Last Login:
`11.18.2010
`
`My recent memories:
`123 Photos uploaded on 11.07.10
`2 albums created 11.17.10
`12 visitors since last login date
`123 Photos uploaded on 11.07.10
`2 albums created 11.17.10
`
`My recent Webb views:
`Captain Phil 2010 (photo album)
`Chicken Pot Pie (recipe)
`Captain Phil (Timeline)
`
`Updates and Alerts:
`License renewal due 1.15.2011
`
`Media
`# Photos
`
`Count
`Count Archive Status
`1,342
`,.·.·.·.·.·.·.· .. ·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.· .. ·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.· .. ·.·.·.·.·.·.·.·.·.·.·.·.·.·.·.· .. ·.·.·.·.·. 80% complete
`
`# Videos
`# Documents
`
`75
`173
`
`People Stats:
`
`61 % complete
`
`Last Name
`
`# People
`
`# Photos
`
`#Videos
`
`# Docs
`
`Monk
`
`Firestone
`
`Testy
`
`Event Stats:
`
`Event
`
`7
`
`11
`
`4
`
`499
`
`1,249
`
`788
`
`4
`
`17
`
`1
`
`14
`
`39
`
`12
`
`Date
`
`Location
`
`# Media
`
`Mike Testv's 1st Birthdav
`
`13-Sept-1988 Minneapolis, MN
`
`21
`
`Cubs Beat Cards Aue 1998
`
`5-Aue-1998
`
`Chicaeo, IL
`
`2,199
`
`Nancy Learns to Ride Bike
`
`21-July-1978
`
`St. Louis, MO
`
`2
`
`Petitioner Apple Inc. - Ex. 1012, p. 21
`
`

`

`U.S. Patent
`
`Sep.24,2019
`
`Sheet 20 of 50
`
`US 10,423,658 B2
`
`FIG. 20
`
`0050
`
`I
`
`0101
`
`0401
`
`0501
`
`Third Party Media Provider
`
`Outside MW S stem
`Inside MW System
`
`0100
`
`1000
`
`1100
`
`1200
`
`1300
`
`Application Export System
`
`0600
`
`0700
`
`0800
`
`0900
`
`Petitioner Apple Inc. - Ex. 1012, p. 22
`
`

`

`U.S. Patent
`
`Sep.24,2019
`
`Sheet 21 of 50
`
`US 10,423,658 B2
`
`0100
`System Reading Phase ____ / ______ ,
`
`Start
`
`FIG. 21
`
`System Interpreting and Adding Data to Relationship Table Phase _ /
`
`0200
`
`System Indexing Digital Files and
`Tag Data Blocks for a Digital File
`within User Relationship Table
`0300
`
`0101
`
`0401
`
`Orisin,11 Digit.ii filr: bt:gins to
`up!o1.d to M~moryWeb
`~ - - - - - - - -~ Outside MW System
`Inside MW System
`
`Locate EX!F
`Image File
`
`0103
`
`File not
`uploaded
`
`0104
`
`. I
`
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`I
`
`I .
`
`~ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - I
`
`Petitioner Apple Inc. - Ex. 1012, p. 23
`
`

`

`U.S. Patent
`
`Sep.24,2019
`
`Sheet 22 of 50
`
`US 10,423,658 B2
`
`FIG. 22
`
`0320
`
`EXIF Tags version 2.3 Image File Directories /
`(Data Blocks)
`/
`Tae Labels
`
`...........-0322
`
`EXIF Family
`Grouo Name
`
`/0323
`Location
`
`0321
`
`MemoryWeb_Tag (Data Blocks)
`
`MemoryWeb Tag
`
`o36o
`
`I
`- -o 361
`- -o 362
`
`- -o 363
`
`--
`
`0364
`
`- -
`- -
`
`0365
`
`0366
`
`--
`--
`--
`-- 0370
`--
`
`0367
`
`0368
`
`0369
`
`0371
`
`Med iaAsset. DateCreated
`
`MediaAsset.Width
`
`MediaAsset.Heie:ht
`
`his is used to inject information that do not currently have
`EXIF standardized tags including Collection, People, Location
`Name, Recipe Name, Person Tag Data Blocks (0380), etc.
`
`MediaAsset.Location.Latitude
`
`MediaAsset.Location.Latitude
`
`MediaAsset.Location.Lone:itude
`
`MediaAsset.Location.Lon itude
`
`IIFDO
`1Irno
`
`1IFD0
`1Irno
`IIFDO
`
`ExiflFD --- 0x9003 -
`
`0334
`
`0335
`
`0x9c9c
`
`0x9c9d
`
`N/A
`
`IFDO
`
`ExiflFD
`
`!GPS
`
`KiPS
`
`kips
`
`k;p5
`
`IGPS
`
`0x8298
`
`Multiple
`
`0xbc80
`
`0xbc81
`
`Multiple
`
`Multiole
`
`0x9286
`
`Dx0D02
`
`0x0003
`
`0x0004
`
`0x000S
`
`0x0006
`
`Description Title
`
`Description Subject
`
`lbescrlption Ratine-
`
`Descriotlon Tae:s
`
`Descrlotlon Comments
`
`Origin Authors
`
`brigin Date Taken
`
`Orie:in Date Acauired
`
`Ori in Convrie:ht
`
`fmaqe (Image ID, Dimensions, Width Heiqht, etc)
`
`Width
`
`Heie:ht
`
`Camera {Camera Maker, Camera Model, etc)
`
`lAdvanced Photo (Lens Maker, Lens Model etc!
`
`User Comment
`
`PS Latitude
`
`IGPS Latitude Ref
`
`GPS Lone:itude
`
`GPS Lon.Q"itude Ref
`
`GPS Altitude
`
`- -
`
`- -- - ----
`
`0324
`
`0325
`
`0326
`
`0327
`
`0328
`
`0329
`
`0330
`
`0331
`
`0332
`
`0333
`
`0x9c9b or
`0x010e
`
`0x9c9f
`
`N/A
`
`0x9c9e
`
`MediaAsset.Caotion
`
`MediaAsset.StarRankine:
`
`Petitioner Apple Inc. - Ex. 1012, p. 24
`
`

`

`U.S. Patent
`
`Sep.24,2019
`
`Sheet 23 of 50
`
`US 10,423,658 B2
`
`FIG. 23
`
`I
`Windows Imaging Component (WIC) Metadata
`(Data Blocks)
`
`0340
`
`I
`MemoryWeb_Tag (Data Blocks)
`
`0360
`
`/
`Tag Label
`
`0342
`
`--
`
`0341
`
`0343
`
`-
`
`File Name
`
`File Item Type
`
`Fi le Folder Path
`
`Fi le Date Created
`
`File Date Modified
`
`File Size
`
`File Attributes
`File (Offline availability, Offline status, Shared with,
`Owner, Computer, etc)
`
`Other
`
`MemoryWeb Tag
`
`MediaAsset.Filename
`
`MediaAsset.UploadBatch.BatchName
`
`-- 0361
`-- 0372
`
`-- 0373
`
`Petitioner Apple Inc. - Ex. 1012, p. 25
`
`

`

`U.S. Patent
`
`Sep.24,2019
`
`Sheet 24 of 50
`
`US 10,423,658 B2
`
`-0395 -----
`----
`--- -----
`- -----
`--------
`- -
`
`0381
`
`0382
`
`0383
`
`0384
`
`0385
`
`0386
`
`0387
`
`0388
`
`------
`
`0389
`
`0390
`
`0391
`
`0392
`
`0393
`
`0394
`
`0396
`
`FIG. 24
`
`MemoryWeb_Person Tag (Data Blocksl
`
`0380
`
`I
`
`Person Name
`
`Nickname
`
`Birthdate
`
`Date of Death
`
`Biography
`
`Mother
`
`Father
`
`Brother 1, Brother 2, ...
`
`Sister 1, Sister 2, ...
`
`Daughter 1, Daughter 2, ...
`
`Son 1, Son 2 ....
`
`Spouse 1, Spouse 2, ...
`
`Facial Recognition Data (Taylor?)
`
`Facebook ID
`
`pets
`...
`
`Petitioner Apple Inc. - Ex. 1012, p. 26
`
`

`

`U.S. Patent
`
`Sep.24,2019
`
`Sheet 25 of 50
`
`US 10,423,658 B2
`
`FIG. 25
`
`Third Party Facial Recognition Provider
`
`0401
`
`I 0400
`
`Start
`
`0402
`
`0404
`
`System request for
`thumbnails of all
`Digital Files that have
`
`System sends
`Digital Files to
`Third Party
`Facial
`Recognition
`
`0403
`
`Retrieves
`Thumbnails of a!!
`Digital Files that
`have unconfirmed
`faces
`
`0405
`
`System receives
`response from
`Third Party Facial
`Recognition
`Provider
`
`0406
`
`System pnrses
`identified faces
`and corresponding
`Facial Recognition
`data for each
`Digital File
`
`Outside MW System
`
`Inside MW System
`
`0409
`
`System records and associates specific
`face for Digital File with a MemoryWeb
`Person ID and sends to MemoryWeb
`Person Tag (Data Blocks) in User
`Rel.itionship T.ible
`
`r•cojnition r•cord for unknown
`person and then sends
`informnt1on to MemoryWeb
`Person Tag (Data Blocks) in User
`Relationship Table
`
`recognition record for person
`and then sends information to
`MemoryWeb Person Tag
`(Data Blocks) in User
`ReL.itionshi Table
`
`Petitioner Apple Inc. - Ex. 1012, p. 27
`
`

`

`U.S. Patent
`
`Sep.24,2019
`
`Sheet 26 of 50
`
`US 10,423,658 B2
`
`FIG. 26
`
`Third Party Media Provider
`
`0501
`
`0507
`
`0500
`
`I
`
`Outside MW System
`
`Inside MW System
`0503
`~-----'--"--~
`
`0504
`
`User selects sharing of
`their information with
`Third Party Media
`Provider with
`MemoryWeb
`
`System sends
`registration
`information to
`User Setting
`Table?
`
`0506
`
`System requests contact
`names and IDs from Third
`Party Media Provider
`(daily)
`
`0300
`
`System add new person to Tag
`Dnta Block of Relationship
`Table for Digital File
`
`1000
`
`Share to Third Party Media
`Provider System
`
`1900
`
`Petitioner Apple Inc. - Ex. 1012, p. 28
`
`

`

`U.S. Patent
`
`Sep.24,2019
`
`Sheet 27 of 50
`
`US 10,423,658 B2
`
`FIG. 27
`
`1900
`
`I
`
`Item
`
`User's Name
`
`Payment ID
`
`Password
`
`Account Type
`
`User's email
`
`Language preference
`
`Date format
`
`Email notifications
`
`Contacts (with third Party Social Media)
`
`Facebook ID
`
`API Token
`
`Payment Date
`
`I"'- ...
`
`--------
`
`--
`
`1901
`
`1902
`
`1903
`
`1904
`
`1905
`
`1906
`
`1907
`
`1908
`
`1909 -
`
`1910
`
`1911
`
`1912
`
`1913
`
`-
`-
`
`Petitioner Apple Inc. - Ex. 1012, p. 29
`
`

`

`U.S. Patent
`
`Sep.24,2019
`
`Sheet 28 of 50
`
`US 10,423,658 B2
`
`0601
`
`0602
`
`System sends
`request to
`Relationship
`Table for specific
`Digital File
`
`0604
`
`System wl!l
`display the
`Digital rl!e and
`the
`
`0603
`
`System retrieves
`Digital File and
`Digital File Tag
`Data Blocks
`from User
`Relationship
`Table
`
`User
`Relationship
`Table
`
`__..,...
`
`0300
`
`FIG. 28
`
`0605
`
`Continuous link
`of Application
`Dot~Tag System
`
`0600
`
`I
`
`User selects
`delete
`fora
`MemoryWeb
`Application Dot•
`Tag
`
`0611
`
`0616
`
`0610
`
`System associates
`new MemoryWeb tng
`to Tag Data Block of
`Relationship Table for
`Digital File
`
`System creates new
`MemoryWeb tag to
`Tag Data Block of
`Relationship Table for
`Digital File
`
`System associates
`existing MemoryWeb
`Tag to Tag Data Block
`of Relationship Table
`for Digital File
`
`System creates new
`MemoryWeb tag to
`Tag Data Block of
`Relationship Table for
`Digital File
`
`System deletes
`association of
`MemoryWeb tag to
`Tag Data Block of
`Relationship Table for
`
`Petitioner Apple Inc. - Ex. 1012, p. 30
`
`

`

`U.S. Patent
`
`Sep.24,2019
`
`Sheet 29 of 50
`
`US 10,423,658 B2
`
`Structure
`
`0650
`
`0652
`
`FIG. 29
`
`0653
`
`Examples
`
`u .... -c_+_i_l_co_1ogn __ e_G_er_m_a_nv __ .,,,.)
`
`,0654
`
`Dotted Application Dot-Tag denotes partial relationship. In
`this example, person is a half-sibling to another person.
`
`,------------------------------,
`: El Frank Smith
`:
`'~-----------------------------)
`
`,0656
`
`0658
`
`,0655
`
`Petitioner Apple Inc. - Ex. 1012, p. 31
`
`

`

`U.S. Patent
`
`Sep.24,2019
`
`Sheet 30 of 50
`
`US 10,423,658 B2
`
`FIG. 30
`
`0700
`
`I
`
`0709
`
`0716
`
`Displays Image
`in S11deshow
`view
`
`If Person Application Dot-Tag, system retrieves data
`for that person from User's User Relationship Table
`and displays relationship data in People Profile View
`
`0710
`
`If Col!ectlon Application Dot-Tag, system retrieves
`data from that collection from User's User
`Relationship Table and displays relationship data in
`Collection View
`
`If Location Application Dot-Tag, system retrieves data
`from that location from User's User Relationship Table
`and displays relationship data in individual Location
`View
`
`Yes
`
`0715
`
`0717
`
`!f date Application Dot-Tag, system retrieves data
`from that date from User's User Relationship Table
`and displays relationship data in Uploads VieN
`
`Tag
`
`0713
`
`If Recipe Application DotRTag, system retrieves data
`from that Recipe from User's User Relationship
`Table and displays relationship data i n~
`
`0714
`
`•••
`
`Petitioner Apple Inc. - Ex. 1012, p. 32
`
`

`

`U.S. Patent
`
`Sep.24,2019
`
`Sheet 31 of 50
`
`US 10,423,658 B2
`
`FIG. 31
`
`0750
`
`Share: 11 IJ
`
`/
`
`0760
`
`..... 0761
`
`Export •
`
`..,_ 0751
`
`◄ ..... 0752
`
`# :;..
`"
`_,31:.
`
`·~'¾i·.
`
`0762 -
`
`(2-"' edit
`
`····················-.. ····•···•··•
`• A<X' t,,•~t,!<" t°:«-'N•J;;;(•"Y S,J,,;,o: i
`
`0768
`
`0769
`
`0754
`
`(e Smith Family Photos .••••• !',Q781
`StorRaN(<>g:
`
`0773
`
`0774
`
`Rot.Ill<:
`
`Jackson and JC's First Day of School
`,0755
`
`edit
`'-0756
`
`(>rig:in.t fill:! Oe~t
`..... 0778
`
`Petitioner Apple Inc. - Ex. 1012, p. 33
`
`

`

`U.S. Patent
`
`Sep.24,2019
`
`Sheet 32 of 50
`
`US 10,423,658 B2
`
`FIG. 32
`
`Jon Smith
`
`Jane Smith (Doe)
`
`Jackson Smith
`
`Page:
`
`-
`
`1403
`
`Single People Profile Application View
`
`JC Jon Smith
`
`1405 '\
`ltemsPerPage: EIGG
`
`1430
`
`I
`i ! Apply Filters
`~
`\ 1447
`
`Uploads Collections
`
`Locations Recipe Family Tree
`
`Fast Search
`
`< View all People
`'1432
`~ - - - - -~
`
`1431-JC Jon Smith
`
`1453
`1454
`
`1434-Born: ~~;==~=,=!'---,r
`1448
`/
`1449
`and l©1ane Smith (Doe)
`Jon Smith
`
`Jackson Smith
`
`1438-Biography: JC was named after his great, great grandfather from German who's real name was Johan Christoph.
`
`1439
`
`JC Tags:
`
`1452
`
`1441
`
`/
`
`1442
`
`/1443
`
`/
`
`1444
`
`3 Collec(.s Z Facial Recognitions
`
`3 Locations
`
`7 Family Relationshlps
`
`1445
`
`/
`
`1 Recipe
`
`1446
`
`Items Per Page:
`
`EIGG
`
`Change Profile Photo
`
`\1440
`
`Page:
`
`-
`
`Petitioner Apple Inc. - Ex. 1012, p. 34
`
`

`

`U.S. Patent
`
`Sep.24,2019
`
`Sheet 33 of 50
`
`US 10,423,658 B2
`
`Multiple Collect=io==n==V ... ie;..w ___ __.=::..;;;;15.;;.;01--------------;::::::==========:::::;--;:::::======::::;-i
`I ! Apply Filters
`Fast Search
`People
`locations Recipe Family Tree
`
`FIG. 33
`
`,t
`
`\., / ·~
`.: .\
`
`1507
`
`1500
`
`I
`
`Baseball Games
`
`Smith Family
`Photos
`
`Europe Trip
`
`First Day of
`School
`
`Jackson and JC
`Photos 2013
`
`Page:
`
`-
`
`'1503
`
`'-1504
`
`1505
`
`1506/
`
`ltemsPerPage: l!I GB
`
`1530
`Individual Colle_c_ti~o_n_V_ie_w-:-----~-----------;:::::==========::::;-;::::=======:::;-i/
`11 Apply Filters
`Fast Search
`People
`locations Recipe Family Tree
`
`< View all Collections -
`
`Smith Family Photos -
`ov..c fir"'''/
`
`1531
`
`1532
`
`_share this Collection:
`
`1534
`
`mo @IO
`
`1535/
`
`Page:
`
`-
`
`Items Per Page: -
`
`0 B
`
`Petitioner Apple Inc. - Ex. 1012, p. 35
`
`

`

`U.S. Patent
`
`Sep.24,2019
`
`Sheet 34 of 50
`
`US 10,423,658 B2
`
`FIG. 34
`
`Sort By: ! Newest to Oldest
`
`!ill -
`
`1606
`
`1600
`
`I
`
`Wrigley Field
`
`Abe Lincoln
`Elementary School
`
`Home Sweet
`Home
`
`Page:
`
`-
`
`""1601
`
`"" 1602
`
`' - 1603
`
`1604
`
`Stonehenge
`
`Items Per Page: El 0 B
`
`1630
`
`Recipe Family Tree
`
`/1633
`
`1 Photos
`
`1634
`
`< View all Locations -
`Wrigley Field
`3651 N. Clark St. -
`1632
`Chicago Illinois, United States
`
`1631
`
`1635
`
`Page:
`
`-
`
`Fast Search
`
`I ! Apply Filters
`Share this Collection: mi IJ
`~ 1637
`
`163/
`
`Items Per Page: El 0 B
`
`Petitioner Apple Inc. - Ex. 1012, p. 36
`
`

`

`U.S. Patent
`
`Sep.24,2019
`
`Sheet 35 of 50
`
`US 10,423,658 B2
`
`Collections People
`
`locations Recipe Family Tree
`
`1701
`/
`(.,:rr~~,t .... ~~: 2:.0 u'tf""
`,A ~)FK•<>O i:,,,..\.
`[;JI
`~ - - - - - - -~ / 1703
`SM"te-t:I File Batch Name (A-Z)
`
`1702
`
`File Folder: C:/2013/Family Fun/
`
`1704 ~ J~,
`
`'-"
`
`{)i:J1~-t
`. /fl '.,:Jt')"~
`6===¥ 1735
`
`
`170
`,-F-ile_F_o_ld_e_cc_c_/2_0::,13;-/,Ge_n,re_ca_l _,,,,, _ _
`,:,
`,::
`
`1707
`
`FIG. 35
`
`Fast Search
`
`/
`
`1705
`
`FMSM-1715
`
`1700
`
`I
`
`! ! Apply Filters
`
`l'""Plt -
`f733',,
`
`1716
`
`(&w•.~"'''" }3!@-,ox ,,,.;'" .. Dr- 1731
`~""-«•>•~''" D1(P1<>"''"' D!.,.___ 1111
`
`1734
`.... A~C: ;~~C~i•~ .......... 1718
`
`1730
`
`(O i:t,$.< 1..~~.1 ~ ~ .. ;.:;'s),): •I
`(.0 l•""'"" •"" ,c·,""'" 2,,,, .. El] (jte•«b•;(:,_., ··Di
`
`1728
`
`171
`
`172
`
`1721
`
`172
`
`172
`
`1724
`
`172
`
`1726
`
`1727
`
`1729
`
`.I
`
`I
`.J
`
`i ~;J
`
`,:::,'
`
`--.:C-
`
`..,,..
`
`.
`
`File Folder: C:/2013/First Day of Schoo! /
`
`1709
`
`1708
`
`1737
`
`___..... 1739
`Page: - -
`
`1738/ttemsPerPage: El 0 B
`
`Petitioner Apple Inc. - Ex. 1012, p. 37
`
`

`

`U.S. Patent
`
`Sep.24,2019
`
`Sheet 36 of 50
`
`US 10,423,658 B2
`
`FIG. 36
`
`Uploads Collections People Locations
`
`, Show all Recipes/ 1802
`
`Lori Doe Pecan Puffs,,,,,. 1
`
`•
`
`01
`
`~ 1804
`
`~ 1 8 1 6
`
`Lon Doe :;::::::i--
`+add/editChef /
`
`1303
`
`/1809
`Ingredients
`
`□ 1 Stick of butter
`□ 1 cup of flour
`□ ½ teaspoon of vanilla
`□ 1 cup of grand pecans
`□ 2 tablespoons of sugar
`
`Directions /
`
`1810
`
`1800
`
`I
`
`! Fast Search
`! ! Apply Filters I
`Share this Collection: fll ii
`~
`
`1812/
`
`1813
`
`1806
`Recipe Photos/
`
`1807
`
`Mix all. Roll gently into 1 inch balls. 300
`and immediately roll in sugar.
`
`Comments/ 1211
`
`This is one of my favorite recipes around the holidays and was
`one of Grandma Lori's best!
`
`1815
`
`Petitioner Apple Inc. - Ex. 1012, p. 38
`
`

`

`U.S. Patent
`
`Sep.24,2019
`
`Sheet 37 of 50
`
`US 10,423,658 B2
`
`FIG. 37
`
`0809
`
`!fin Uploads view, system retrieves data for applied
`fi!ter(s) from User's User Relationship Table and
`displays relationship data
`
`0810
`
`!fin Collections vlew, system retrieves data for
`applied filter(s) from User's User Relationship Table
`and displays relationship data
`
`0811
`
`If in Locations view,