(12) United States Patent
`Tedesc0 et al.
`
`(10) Patent No.:
`(45) Date of Patent:
`
`US 8,296,686 B1
`Oct. 23, 2012
`
`USOO8296,686B1
`
`(54) PORTABLE PROMPTING AID FOR THE
`DEVELOPMENTALLY DISABLED
`
`289: A : g28 E. et al. ..................... 2,335
`6,056,549 A * 5/2000 Fletcher ........................ 434/112
`6,139,494 A 10/2000 Cairnes
`6,168,569 B1
`1/2001 McEwen et al.
`(75) Inventors: Daniel E. Tedesco, Shelton, CT (US);
`6,282,441 B1
`8/2001 Raymond et al.
`Carey M. Tedesco, Shelton, CT (US);
`6,314.405 B1
`1 1/2001 Richardson
`at St. S. 6,336,053 B1* 1/2002 Beatty ........................... TOOf 108
`6,449.460 B2 * 9/2002 Logan ...
`ames A. Jorasch, New York, NY (US);
`... 434,308
`Russell P. Sammon, San Francisco, CA
`6,585,516 B1* 7/2003 Alabaster .......
`... 434/127
`(US)
`6,710,791 B1* 3/2004 Kodama et al.
`T15,835
`
`W 4
`
`CIO ........
`
`(*) Notice:
`
`(73) Assignee: Handhold Adaptive, LLC. Shelton, CT
`US)
`(
`-
`Subject to any disclaimer, the term of this
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 351 days.
`(21) Appl. No.: 12/391,871
`(22) Filed:
`Feb. 24, 2009
`9
`Related U.S. Application Data
`(60) Eyal application No. 61/195,976, filed on Oct.
`s
`(51) Int. Cl.
`(2006.01)
`G06F 3/00
`(52) U.S. Cl. ......................... 715/865; 434/112:434/238
`(58) Field of Classification Search .................. 715/865.
`434/112, 237, 238
`See application file for complete search history.
`
`(56)
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`4,465,465 A * 8/1984 Nelson .......................... 434/112
`4,761,633. A * 8/1988 Leffet al. ................ 340,286.07
`5,323,314 A * 6/1994 Baber et al. .................. 705.7.19
`5,544,649 A
`8, 1996 David et al.
`5,576.946 A * 11, 1996 Bender et all
`7OOf 17
`5,580.254. A
`2, 1996 Ramsey. .434,236
`5,835,898 A * 1 1/1998 Borg et al. ................... 705/712
`5,890,905 A
`4, 1999 Bergman ...................... 434/118
`
`6,724,298 B2
`
`4, 2004 Smith .
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`.
`
`. 340/407.1
`
`6,828,989 B2 ck 12/2004 Cortright
`7,095,442 B2
`8/2006 van Zee ................... 348,333.01
`7, 177,235 B2
`2/2007 Rund ............................ 368,261
`7.217,133 B2
`5/2007 Thomas et al. ............... 434,236
`7,407,484 B2
`8/2008 Korman
`7,761,322 B2
`7/2010 Nakamura et al.
`(Continued)
`OTHER PUBLICATIONS
`Christopher Breen, “The iPhone Pocket Guide”, Sep. 20, 2007,
`Peachpit Press, p. 194.*
`
`ABSTRACT
`
`(Continued)
`Primary Examiner — William Bashore
`Assistant Examiner — Daeoo Lee
`(74) Attorney, Agent, or Firm — Withrow & Terranova,
`PLLC
`57
`(57)
`An aid for developmentally disabled individuals takes the
`form of a mobile terminal having software capable of creating
`graphical schedules, graphical countdowntimers, and graphi
`cal choice selections. A caregiver may dynamically program
`and save schedules which present a graphical sequence of
`events to the disabled individual to assist the disabled indi
`vidual in adjusting the changing events of the day. Likewise,
`the countdown timer may assist the individual in preparing
`for an upcoming transition in environment or activity. Still
`further, the graphical choice selections may help a nonverbal
`individual express themselves to a caregiver.
`15 Claims, 28 Drawing Sheets
`
`2
`
`O AT&T
`
`8:22A
`Digital Picture Prompts
`
`s Welcome to
`Digital Picture Prompts
`
`N.
`
`s
`
`Picture Schedule
`
`SetWisual Countdown
`
`58
`
`6.
`
`Offer Choices
`
`
`
`APPL-1007
`APPLE INC. / Page 1 of 44
`
`

`

`US 8,296,686 B1
`Page 2
`
`U.S. PATENT DOCUMENTS
`7,836,400 B2 11/2010 May et al.
`7,844.913 B2 * 1 1/2010 Amano et al. ................ 715,769
`2001/0020904 A1
`9/2001 Dayle .........
`... 341.21
`2002/0009696 A1
`1/2002 Lui et al. ....................... 434/171
`2002/009 1664 A1
`7/2002 Larder et al.
`2002/0142271 A1 10, 2002 Curtin ........................... 434,167
`2003/0050963 A1* 3/2003 Lamming et al.
`709f2O3
`2003, OO64758 A1* 4, 2003 Mizuta et al. ..
`... 455,566
`2003. O142227 A1* 7, 2003 Van Zee ......
`348,333.1
`2004/0096808 A1* 5, 2004 Price et al. .................... 434/112
`2004/O186772 A1
`9/2004 Kawatahara
`3/2005 Tyler et al.
`2005/0059034 A1
`2005/0090298 A1
`4/2005 Park et al.
`2005/0183026 A1* 8, 2005 Amano et al. ................ 715,764
`2006, O189278 A1
`8, 2006 Scott ...........
`455/90.3
`2006/026794.0 A1* 11, 2006 Groom et al.
`345,163
`2006/0279781 A1* 12, 2006 Kaneko .......
`358,115
`2007/01 17073 A1* 5, 2007 Walker et al.
`434,236
`2008.0031426 A1* 2, 2008 Weeks ...........
`... 379/45
`2008/0038698 A1
`2/2008 Meissner et al.
`434/112
`2008/01 15090 A1* 5/2008 Disbrow ....................... 71.5/865
`2008/O122796 A1
`5, 2008 JobS et al.
`2008/0294490 A1* 11/2008 Nuhaan et al. .................... 70.5/9
`2009/O125333 A1
`5/2009 Heywood et al.
`2009/O131758 A1
`5/2009 Heywood et al.
`2009, O144089 A1
`6/2009 Heywood et al.
`2009, O198511 A1
`8, 2009 Boehlke
`2010.0167255 A1
`7, 2010 Shane et al. .............. 434,307 R
`2010, O248775 A1* 9, 2010 Mikkelsen et al.
`455,556.1
`2011/0053129 A1
`3f2011 Basson et al. ................. 434.238
`
`
`
`OTHER PUBLICATIONS
`ABA Educational Resources Ltd., Message Board; 2 pp.
`Autism E-News—Apr. 2004; vol. 1, Issue 3: (http//www.ttac odu
`edu/ENewsArchives/Autism/autism e April 2004 toc
`htm)
`Jun. 16, 2009 10:24:22 AM; 8 pp.
`AMDI Tech/Touch. New Dynamic AAC Display Specification
`AMDI, Advanced Multimedia Devices, Inc.; (http flamdi net/
`TTFreedom htm) (Jun. 16, 2009 10:22:30 AM); 1 pg.
`BeeVisual: (http//www beevisual com/choiceworks html) (Jun. 16.
`2009 10:18:39 AM); 2 pp.
`planning';
`personal
`Picture
`Planner,
`"icon-based
`(http (ferinstitute 1001 qwestoffice net picture planner/detail html)
`Jun. 16, 2009 20:11:19 AM); 1 pg.
`DynaVox & Mayer-Johnson; M3-Features; “M3”; (http //www.
`dynavoxtech com/products/m3/features aspx) Jun. 16, 2009
`10:22:05 AM); 2 pp.
`Software;
`Products—Education
`DynaVox Mayer-Johnson;
`“Boardmaker Plus!”: (http//www dynavoxtech com/products/edu
`cation/bm-plus aspx) Jun. 18, 2009 12:30:01 PM); 2 pp.
`Ilene S. Schwartz et al., “Including Children with Autism in Inclusive
`Preschools: Strategies that Work”. (http //www newhorizons org/
`speneeds/inclusion/information/schwartz2 htm) (Jun. 16, 2009
`10:33:23 AM); 11 pp.
`“Welcome to JoeSchedule's VisualToolkit'. Teaching Tools for Kids
`With Autism; Introduction Video: (http //www.joeschedule com/
`js3col index01 htm) (Jun. 16, 2009 10:19:57 AM; 1 pg.
`“Picture Cue Dictionary: Spectronics—Inclusive Learning Tech
`nologies'; by Attainment Company: (http f/www spectronicsinoz
`com/product/21919) Jun. 16, 2009 10:07:40 AM; 4 pp.
`“Picture Cue Dictionary'. Attainment Products Menu: (http//www.
`synapseadaptive com/attainment special/calcd.html) Jun. 16, 2009
`10:08:52 AM); 1 pg.
`
`Overboard Communication Board, Designer by Gus Communica
`tion Devices, Inc.; (http//www.gusinc com/Overboard/index html)
`Jun. 16, 2009 10:10:28 AM; 4 pg.
`. . Series: (http //www.
`Silver Lining Multimedia Picture This .
`silverliningmm com/ptframe htm) (Jun. 18, 2009 12:32:56 PM; 3
`pp.
`Qcharm, “Portable & Visual Cuing System”; (http//www qcharm
`com/index.html) (Jun. 16, 2009 10:19:29AM); 1 pg.
`SourceForge.net: Community Discussion Forums; “Picture Sched
`ule Program (5 posts): (http//sourceforge net/community/forum/
`topic php?id=2214&page) Jun. 16, 2009 10:14:19 AM; 3 pp.
`"Super Symbols to Go'. Icon Talk Products; (http//www icontalk
`com/IT-2 html) (Jun. 16, 2009 10:21:24 AM); 1 pp.
`"Schedule Strips'. Super Symbols—IconTalk Products; (http//www.
`icontalk com/IT-17 html) (Jun. 16, 2009 10:21:06 AM); 1 pp.
`“Products', Time Timer-Tangible Time Management-Products;
`(http//www timetimer com/products php) (Jun. 16, 2009 10:23:47
`AM: 4 pp.
`"Time Timer CD'; Time Timer, Tangible Time Management,
`(http//wwwtimetimer com/product colphp) Jun. 16, 2009 10:24:05
`AM: 2 pp.
`Non-final Office Action mailed Oct. 7, 2011 for U.S. Appl. No.
`12/551, 186, 14 pages.
`Restriction Requirement mailed Aug. 25, 2011 for U.S. Appl. No.
`12/551, 186, 7 pages.
`Symtrend, “Electronic Diaries and graphic tools for health care and
`special education.” Accessed Dec. 13, 2011, 1 page, https://www.
`Symtrend.com.tw/publicf.
`Techcrunch, "AsthmaMD helps asthma Sufferers, gathers aggregate
`research data.” Jan. 10, 2010, 3 pages, http://techcrunch.com/2010/
`01/10/asthmamd-helps-asthma-Suffers-gathers-aggregate-research
`data.
`AsthmaMD, App Store, 1 page. Accessed Dec. 13, 2011, http://
`asthmamd.org/3resources?iphone chart.jpeg.
`Behaviortracker, “Behavior Tracker Pro Receives Coverage on
`NBC.” Jan. 29, 2010, 3 pages, www.behaviortrackerpro.com.
`Patients Like Me, “Patients Helping Patients Live Better Every Day.”
`Accessed Dec. 13, 2011, 1 page, www.patientslikeme.com.
`DDtrac, “DDtrac Data Collection and Progress Monitoring Soft
`ware.” 2007-2010, Accessed Apr. 9, 2010, 1 page, www.
`developingmindsSoftware.com/product.
`Mobile Thinking, “Introduction mTrial.” 2008, Accessed Apr. 9.
`2010, 1 page, www.mobilethinking.com/dtm.
`Nature, "Personal Technology: Phoning in Data.” Apr. 22, 2009, pp.
`959-961, www.nature.com/news/2009/090419/full/458959a.html.
`Autoworks, "About Autoworks.” 2004-2009, 2 pages, http://
`autoworks.hms.harvard.edu.
`Harvard, “The Wall Lab.” Accessed Dec. 13, 2011, 2 pages, http://
`wall.hms.harvards.edu/research.
`Abbott, "FDA Approves Abbott's Freestyle Navigator Continuous
`Glucose Monitoring System.” Mar. 13, 2008, 3 pages, www.ab
`bottoliabetescare.com/adc dotcom/url/content/en US 10:10/gen
`eral content/General Content 0000163.htm.
`Technology Review, “Cell phones that listen and learn.” Jun. 22.
`2009, 3 pages, www.technologyreview.com/communications/
`22.907.
`Final Office Action mailed Jan. 19, 2012, for U.S. Appl. No.
`12/551,186, 39 pages.
`Non-Final Rejection mailed Mar. 29, 2012, for U.S. Appl. No.
`12/706,020, 22 pages.
`* cited by examiner
`
`APPL-1007
`APPLE INC. / Page 2 of 44
`
`

`

`U.S. Patent
`
`Oct. 23, 2012
`
`Sheet 1 of 28
`
`US 8,296,686 B1
`
`
`
`14
`
`10
`1/
`
`12
`
`oDO AT&T
`
`8:22 AM
`
`D
`
`Calendar
`
`Photos
`
`Camera
`
`Text
`
`C (2
`
`Recorder
`
`O
`| Q 73
`33 ERA
`
`Stocks
`
`Maps
`
`Weather
`
`Calculator
`
`Notes
`
`KeyTasks
`
`FIG. 1
`
`APPL-1007
`APPLE INC. / Page 3 of 44
`
`

`

`U.S. Patent
`U.S. Patent
`
`Oct. 23, 2012
`
`Sheet 2 of 28
`
`US 8,296,686 B1
`US 8,296,686 B1
`
`OL
`
`
`
`
`
`
`a79oFverduvOaAayuanvadsANOHdONSINAVidsIdVaan
`do JOVIYSLNI
`
`
`
`¥3ASN
`
`SWdsHdldsdAMALLVALuOdAYMLINOUIDSYYMi4
`
`
`
`
`SNOSNVTISOSINQ3uIMYSAISZOSNVYL
`oPoew9
`¥3aLSWOYd1FIOV
`
`AYOWSM
`
`ve
`
`LuvYOId¢Old
`
`WA
`
`BE
`
`ce
`
`APPL-1007
`APPLEINC./ Page 4 of 44
`
`APPL-1007
`APPLE INC. / Page 4 of 44
`
`
`

`

`U.S. Patent
`
`Oct. 23, 2012
`
`Sheet 3 of 28
`
`US 8,296,686 B1
`
`1OB
`
`10A
`
`
`
`FIG. 3
`PRIOR ART
`
`APPL-1007
`APPLE INC. / Page 5 of 44
`
`

`

`U.S. Patent
`
`Oct. 23, 2012
`
`Sheet 4 of 28
`
`US 8,296,686 B1
`
`CARETAKERACQUIRES MOBILE TERMINAL
`
`career courtscasternal
`
`CARETAKERLOADS SOFTWARE ONTO MOBILE TERMINAL 102
`
`CARETAKER ACCESSES SOFTWARE ON MOBILE TERMINAL
`
`
`
`RECEIVE
`PICTURE
`SCHEDULE
`SELECTION
`
`
`
`
`
`
`
`106
`
`RECEIVE
`SET VISUAL
`COUNTDOWN
`SELECTION 108
`
`RECEIVE
`OFFER
`CHOICES
`SELECTION 110
`
`
`
`RECEIVE
`CREATE NEW
`SCHEDULE
`SELECTION 112
`
`
`
`
`
`OFG. 14
`
`TO FIG. 19
`
`PRESENT MENU OF
`SCHEDULE
`CATEGORIES 11
`
`RECEIVE CATEGORY
`SELECTION 18
`
`RECEIVE
`ACCESSSAVED
`SCHEDULE
`SELECTION
`
`114
`
`
`
`
`
`PRESENT MENU OF
`THEMATICALLY
`LINKED CONS
`120
`
`
`
`RECEIVE IMAGE
`SELECTION 122
`
`DISPLAY
`PARTIALLY-FINISHED
`SCHEDULE 124
`
`RECEIVE COMMAND
`TO SET TIME 126
`
`PRESENT MENU OF
`SAVED SCHEDULES
`134
`
`RECEIVE SELECTION
`OF SCHEDULE
`136
`
`DISPLAY SELECTED
`SCHEDULE 132
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`NO
`
`128
`
`
`
`
`
`SAVE OR SHOW
`130
`FIG. 4
`
`APPL-1007
`APPLE INC. / Page 6 of 44
`
`

`

`U.S. Patent
`
`Oct. 23, 2012
`
`Sheet 5 of 28
`
`US 8,296,686 B1
`
`
`
`oOO AT&T
`
`8:22 AM
`Digital Picture Prompts
`
`O
`
`s Welcome to
`Digital Picture Prompts
`
`Picture Schedule
`
`Offer Choices
`
`FIG. 5
`
`APPL-1007
`APPLE INC. / Page 7 of 44
`
`

`

`U.S. Patent
`
`Oct. 23, 2012
`
`Sheet 6 of 28
`
`US 8,296,686 B1
`
`
`
`goO AT&T
`
`8:22AM
`Digital Picture Prompts
`
`C
`
`Create a
`Picture Schedule
`
`Create New Schedule
`
`ACCeSS Saved Schedule
`
`FIG. 6
`
`APPL-1007
`APPLE INC. / Page 8 of 44
`
`

`

`U.S. Patent
`
`Oct. 23, 2012
`
`Sheet 7 of 28
`
`US 8,296,686 B1
`
`
`
`oDO AT&T
`
`8:22 AM
`Digital Picture Prompts
`
`Select Your
`First Image
`
`68A
`
`Emotions I Feelings
`
`Therapy
`
`Take I Select Photo
`
`ShOW All
`
`FIG. 7
`
`APPL-1007
`APPLE INC. / Page 9 of 44
`
`

`

`U.S. Patent
`
`Oct. 23, 2012
`
`Sheet 8 of 28
`
`US 8,296,686 B1
`
`
`
`goO AT&T
`8:22 AM
`{BACK Digital Picture Prompts
`
`O
`
`O) Select Your
`First Image
`
`.
`
`70
`
`70
`
`72
`
`1.
`Fork
`
`70
`
`More
`
`APPL-1007
`APPLE INC. / Page 10 of 44
`
`

`

`U.S. Patent
`
`Oct. 23, 2012
`
`Sheet 9 of 28
`
`US 8,296,686 B1
`
`
`
`ooD AT&T
`
`8:22AM
`Digital Picture Prompts
`
`Schedule Under
`Construction
`
`Add Another
`image
`
`80
`
`Finish and
`Show Schedule 82
`
`APPL-1007
`APPLE INC. / Page 11 of 44
`
`

`

`U.S. Patent
`
`Oct. 23, 2012
`
`Sheet 10 of 28
`
`US 8,296,686 B1
`
`
`
`oOO AT&T
`
`8:22AM
`
`FIG 10
`
`APPL-1007
`APPLE INC. / Page 12 of 44
`
`

`

`U.S. Patent
`
`Oct. 23, 2012
`
`Sheet 11 of 28
`
`US 8,296,686 B1
`
`
`
`go AT&T
`
`8:22 AM
`Digital Picture Prompts
`
`Save This Schedule?
`Name it as:
`
`(a)(W)(E)(R) (TDCY) (UDO (o) (P)
`(A)(s) (D) (F) (G) (HDGJD (KCL)
`(f) (z) CX)(C)(V)(B)(N) (M) Ce)
`
`FIG. 11
`
`APPL-1007
`APPLE INC. / Page 13 of 44
`
`

`

`U.S. Patent
`U.S. Patent
`
`Oct. 23, 2012
`
`Sheet 12 of 28
`
`US 8,296,686 B1
`US 8,296,686 B1
`
`
`
`Q06
`
`Aepu4
`
` 0}
`[JL]Lsa,aeenamguorg
`966Ve6a69688
`
`juaung[isn>yp|Cooooo
`
`aiNpeyIsalowJames
`
`8h
`
`cbSls
`
`APPL-1007
`APPLEINC./ Page 14 of 44
`
`APPL-1007
`APPLE INC. / Page 14 of 44
`
`
`
`
`

`

`U.S. Patent
`
`Oct. 23, 2012
`
`Sheet 13 of 28
`
`US 8,296,686 B1
`
`
`
`oOO AT&T
`
`8:22AM
`
`D
`
`Access a Saved
`Picture Schedule
`
`Trip to Airport
`
`Medicine ROutine
`
`Preparing for School
`
`Wednesday Play Group
`
`October 25
`
`APPL-1007
`APPLE INC. / Page 15 of 44
`
`

`

`U.S. Patent
`
`Oct. 23, 2012
`
`Sheet 14 of 28
`
`US 8,296,686 B1
`
`DISPLAY CREATE COUNTDOWN SCREEN
`
`RECEIVE TIME FOR COUNTDOWN
`
`RECEIVE ALERT SELECTION
`
`RECEIVE IMAGE COMMAND
`
`DISPLAY IMAGE CATEGORIES
`
`150
`
`152
`
`154
`
`156
`
`158
`
`RECEIVE CATEGORY SELECTION
`
`160
`
`DISPLAY MAGES
`
`RECEIVE IMAGE SELECTION
`
`162
`
`164
`
`168
`
`SAVE
`
`(ED)
`
`SAVE OR START
`166
`
`START
`
`DISPLAY COUNTDOWN
`
`170
`
`FIG. 14
`
`APPL-1007
`APPLE INC. / Page 16 of 44
`
`

`

`U.S. Patent
`
`Oct. 23, 2012
`
`Sheet 15 of 28
`
`US 8,296,686 B1
`
`
`
`noG AT&T
`
`8:22 AM
`Digital Picture Prompts
`
`Set a
`Visual Countdown
`
`%
`2%
`
`APPL-1007
`APPLE INC. / Page 17 of 44
`
`

`

`U.S. Patent
`
`Oct. 23, 2012
`
`Sheet 16 of 28
`
`US 8,296,686 B1
`
`
`
`208
`
`
`
`
`
`210
`
`oOO AT&T
`
`8:22 AM
`Digital Picture Prompts
`
`O
`
`Select Image for
`When Timer Ends
`
`
`
`O Safety D
`Activities
`CDays sweather)
`Meal Time
`Home & School C seasonal D
`
`
`
`
`
`
`
`
`
`Emotions I Feelings
`
`Therapy
`
`Take I Select Photo
`
`ShOW All
`
`16
`
`18
`
`FIG. 16
`
`APPL-1007
`APPLE INC. / Page 18 of 44
`
`

`

`U.S. Patent
`
`Oct. 23, 2012
`
`Sheet 17 of 28
`
`US 8,296,686 B1
`
`
`
`212
`
`214
`
`goO AT&T
`
`8:22 AM
`Digital Picture Prompts
`
`O
`
`Select Image for
`When Timer Ends
`
`Wash Hands
`
`N.E.
`
`FIG. 17
`
`APPL-1007
`APPLE INC. / Page 19 of 44
`
`

`

`U.S. Patent
`
`Oct. 23, 2012
`
`Sheet 18 of 28
`
`US 8,296,686 B1
`
`10
`
`goO AT&T
`
`8:22 AM
`Digital Picture Prompts
`
`O
`
`16
`
`
`
`220
`
`Brush Teeth 21
`
`APPL-1007
`APPLE INC. / Page 20 of 44
`
`

`

`U.S. Patent
`
`Oct. 23, 2012
`
`Sheet 19 Of 28
`
`US 8,296,686 B1
`
`DISPLAY CATEGORIES
`
`2 5 O
`
`RECEIVE FIRST CATEGORY SELECTION
`
`RECEIVE FIRST IMAGE SELECTION
`
`DISPLAY CATEGORIES
`
`SAVE
`
`268
`
`ACCESSLATER
`270
`
`ANOTHER CHOICE
`
`
`
`RECEIVE SHOW CHOICES COMMAND
`
`DISPLAY CHOICES
`
`RECEIVE CHOICE SELECTION
`
`CONFIRM SELECTION VIA MEDIA OUTPUT
`
`FIG. 19
`
`272
`
`274
`
`276
`
`278
`
`APPL-1007
`APPLE INC. / Page 21 of 44
`
`

`

`U.S. Patent
`
`Oct. 23, 2012
`
`Sheet 20 of 28
`
`US 8,296,686 B1
`
`
`
`302
`
`Digital Picture Prompts
`
`Offer Choices
`Select First Choice
`
`Emotions I Feelings
`
`Therapy
`
`Take | Select Photo
`
`Show All
`
`FIG. 20
`
`APPL-1007
`APPLE INC. / Page 22 of 44
`
`

`

`U.S. Patent
`
`Oct. 23, 2012
`
`Sheet 21 of 28
`
`US 8,296,686 B1
`
`
`
`308
`
`Digital Picture Prompts
`
`Offer Choices
`Select First Choice
`
`FIG. 21
`
`APPL-1007
`APPLE INC. / Page 23 of 44
`
`

`

`U.S. Patent
`
`Oct. 23, 2012
`
`Sheet 22 of 28
`
`US 8,296,686 B1
`
`go AT&T
`
`8:22 AM
`Digital Picture Prompts
`
`Offer Choices
`Select Second Choice
`
`
`
`308
`
`APPL-1007
`APPLE INC. / Page 24 of 44
`
`

`

`U.S. Patent
`
`Oct. 23, 2012
`
`Sheet 23 of 28
`
`US 8,296,686 B1
`
`
`
`goO AT&T
`
`8:22 AM
`Digital Picture Prompts
`
`318
`
`Offer Choices
`Your Selected Choices
`
`FIG. 23
`
`APPL-1007
`APPLE INC. / Page 25 of 44
`
`

`

`U.S. Patent
`
`Oct. 23, 2012
`
`Sheet 24 of 28
`
`US 8,296,686 B1
`
`
`
`APPL-1007
`APPLE INC. / Page 26 of 44
`
`

`

`U.S. Patent
`
`Oct. 23, 2012
`
`Sheet 25 of 28
`
`US 8,296,686 B1
`
`
`
`APPL-1007
`APPLE INC. / Page 27 of 44
`
`

`

`U.S. Patent
`
`Oct. 23, 2012
`
`
`
`
`
`
`
`
`
`3O1OHO NI CEST)NMOCIIN?IOONI CEST)ETTOEHOS NI CEST)Å809EIWOEWWN ESOWW]
`
`?J?§§§§§§???
`
`Ž??
`
`Sheet 26 of 28
`
`
`
`
`
`
`
`OOOOO
`
`US 8,296,686 B1
`
`O
`
`O
`
`OOOOO O
`
`O
`
`O
`
`APPL-1007
`APPLE INC. / Page 28 of 44
`
`

`

`U.S. Patent
`U.S. Patent
`
`
`
`Sheet 27 of 28
`
`US 8,296,686 B1
`US 8,296,686 B1
`
`adAL
`
`ore
`
`998
`
`JINGSHOS
`
`JOIOHOD
`
`Oct. 23, 2012
`
`SAWILSawSWYN314
`AINGSHOSWd00:2AINGAHSSYAddNs
`
`
`
`
`WY00:2HSNYSHLOOLONINSOW
`
`Zé‘Old
`
`WV3Y89301
`
`HSNYaHLool
`
`
`
`SONVHHSVM
`
`
`
`SONVHHSVM
`
`YIVHSWOD
`
`aVvaMysaNn
`
`SINVd
`
`LYIHS
`
`SH90$
`
`S3OHS
`
`APPL-1007
`APPLEINC./ Page 29 of 44
`
`APPL-1007
`APPLE INC. / Page 29 of 44
`
`

`

`U.S. Patent
`
`Oct. 23, 2012
`
`Sheet 28 of 28
`
`US 8,296,686 B1
`
`
`
`330
`
`ooD AT&T
`
`8:22AM
`Digital Picture Prompts
`
`O
`
`Select Your
`First image
`
`Juice 70
`
`Ice Cream 70
`
`More
`
`72
`
`APPL-1007
`APPLE INC. / Page 30 of 44
`
`

`

`1.
`PORTABLE PROMPTING AD FOR THE
`DEVELOPMENTALLY DISABLED
`
`PRIORITY
`
`The present application claims priority to and the benefit of
`U.S. Provisional Patent Application Ser. No. 61/195,976,
`filed 14 Oct. 2008, which is hereby incorporated by reference
`in its entirety.
`
`10
`
`FIELD OF THE INVENTION
`
`The present disclosure is directed to providing devices,
`systems and methods for providing a sensory aid for the
`developmentally disabled.
`
`15
`
`BACKGROUND
`
`With a growing population, the number of developmen
`tally disabled children grows. Additionally, the rates at which
`children have been diagnosed as developmentally disabled,
`and particularly diagnosed with autism, have steadily
`increased. While there is a continuum of disabilities that fall
`under the general heading of developmentally disabled, many
`individuals that are developmentally disabled experience dif
`ficulty in developing normal speech and language abilities
`and may also suffer from sensory processing dysfunction
`(SPD). SPD encompasses those situations where a person has
`one or more senses that are hyposensitive or hypersensitive.
`These dysfunctions and their manifestations make it diffi
`cult for such individuals to adapt to dynamic settings and in
`particular they may have difficulty transitioning between
`activities. This difficulty is compounded by the aforemen
`tioned language barriers that may exist.
`Conventional approaches have not been able to provide a
`convenient tool that allows caregivers to communicate effec
`tively with such individuals and prepare such individuals for
`known upcoming changes, especially in contexts such as
`ordinary life in the community, which is markedly more
`dynamic and rich with sensory stimuli as compared to rela
`tively sterile and controlled therapeutic settings.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIG. 1 illustrates a front elevational view of a mobile ter
`minal.
`FIG. 2 illustrates a block diagram of the components of a
`conventional mobile terminal.
`FIG. 3 illustrates a variety of networks in which a mobile
`terminal may operate.
`FIG. 4 illustrates a first flow chart showing a portion of an
`exemplary embodiment of the present disclosure.
`FIG. 5 illustrates a first screen shot showing a screen shot
`associated with the flow chart of FIG. 4.
`FIG. 6 illustrates a second screen shot associated with the
`flow chart of FIG. 4.
`FIG. 7 illustrates a third screen shot associated with the
`flow chart of FIG. 4.
`FIG. 8 illustrates a fourth screen shot associated with the
`flow chart of FIG. 4.
`FIG.9 illustrates a fifth screen shot associated with the flow
`chart of FIG. 4.
`FIG. 10 illustrates a sixth screen shot associated with the
`flow chart of FIG. 4.
`FIG.11 illustrates a seventh screen shot associated with the
`flow chart of FIG. 4.
`
`25
`
`30
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`US 8,296,686 B1
`
`2
`FIG. 12 illustrates an eighth screen shot associated with the
`flow chart of FIG. 4.
`FIG. 13 illustrates a ninth screen shot associated with the
`flow chart of FIG. 4.
`FIG. 14 illustrates a second flow chart showing a second
`portion of an exemplary embodiment of the present disclo
`SUC.
`FIG. 15 illustrates a first screen shot associated with the
`flow chart of FIG. 14.
`FIG.16 illustrates a second screen shot associated with the
`flow chart of FIG. 14.
`FIG. 17 illustrates a third screen shot associated with the
`flow chart of FIG. 14.
`FIG. 18 illustrates a fourth screen shot associated with the
`flow chart of FIG. 14.
`FIG. 19 illustrates a third flow chart showing a third portion
`of an exemplary embodiment of the present disclosure.
`FIG. 20 illustrates a first screen shot associated with the
`flow chart of FIG. 19.
`FIG.21 illustrates a second screen shot associated with the
`flow chart of FIG. 19.
`FIG. 22 illustrates a third screen shot associated with the
`flow chart of FIG. 19.
`FIG. 23 illustrates a fourth screen shot associated with the
`flow chart of FIG. 19.
`FIG. 24 illustrates a fifth screen shot associated with the
`flow chart of FIG. 19.
`FIG. 25 illustrates a sixth screen shot associated with the
`flow chart of FIG. 19.
`FIG. 26 illustrates a first exemplary database suitable for
`use with the present disclosure.
`FIG. 27 illustrates a second exemplary database suitable
`for use with the present disclosure.
`FIG. 28 illustrates a screen shot associated with an alter
`nate embodiment of the present disclosure.
`
`DETAILED DESCRIPTION
`
`A mobile terminal with a display screen may be used with
`the software described herein to allow caregivers to create a
`schedule using appropriate graphical elements, which may be
`presented to the disabled individual so that the disabled indi
`vidual may be graphically prepared for upcoming transitions
`or changes. Likewise, the caregiver may program graphical
`timers for use with the disabled individualso that the disabled
`individual has a graphical indication of an amount of time
`remaining until a change in activity is to occur. Such timers
`may be incorporated into the schedule so that as a transition
`nears, a timer may be displayed to the disabled individual.
`Still further the caregiver may program the mobile terminal to
`display two graphical elements for display to the disabled
`individual from which the disabled individual may select a
`choice. During creation of these graphical tools, the caregiver
`may be given the opportunity to select the images by selecting
`images from predefined categories of thematically-linked
`images. The image grouping may facilitate dynamic creation
`of the graphical tools and/or allow the disabled individual to
`participate in the creation of the graphical tools.
`Still other graphical prompts may be programmed by the
`caregiver to facilitate graphical interaction rather than oral
`interaction with the disabled individual. Likewise, new
`graphical elements may be captured by a camera either inter
`nal to the mobile terminal or external to the mobile terminal
`and uploaded to the mobile terminal. Likewise, video, audio
`or other elements (e.g., vibration output) may be incorporated
`into the schedule, timers, or choices so as to facilitate com
`munication with the disabled individual.
`
`APPL-1007
`APPLE INC. / Page 31 of 44
`
`

`

`3
`The use of a mobile terminal provides a highly portable,
`stylish, and malleable tool through which caregivers may
`adjust to the dynamic Surroundings of the disabled individual
`and provide customized graphical elements to assist in com
`municating with the disabled individual.
`An overview of some of the hardware elements is provided
`before addressing some of the methods and potential Screen
`configurations associated with embodiments of the present
`disclosure. A mobile terminal 10 is illustrated in FIG.1. An
`exemplary suitable mobile terminal 10 may be the APPLE(R)
`10
`IPHONETM (or IPODTM) and may include a user interface 12
`that includes a speaker 14, a microphone 26 (FIG. 2), a touch
`screen display 16, and a command entry button 18. One or
`more icons 20 may be displayed on the touch screen display
`16. The user may enter commands by touching an icon on the
`touchscreen display 16 or by using the command entry button
`18 as is well understood. One such icon 22 may launch
`software that effectuates embodiments of the present disclo
`sure. In one embodiment, the software is developed by a third
`party. The practice of third-party software development on
`the IPHONETM platform is commonplace today, with tens of
`thousands of applications available for sale through the
`APPLE(RAPP STORE, and other mobile terminal manufac
`turers are embracing Such third-party development of mobile
`software applications. Thus, while the IPHONETM is particu
`larly contemplated, the disclosure is not so limited, and any
`readily portable handheld computing device may be appro
`priate including personal digital assistants, cellular tele
`phones, WINDOWS MOBILETM enabled devices (e.g., the
`HTC TOUCH DIAMONDTM), or the like. Further specific
`30
`examples of such mobile terminals include the BLACK
`BERRYTM line of wireless devices manufactured by
`Research in Motion of Waterloo, Ontario, Calif.; the T-MO
`BILE G1 or other mobile terminals operating on the
`ANDROID platform developed by Google, Inc. of Mountain
`35
`View, Calif.; the PRADATM or other mobile terminals manu
`factured by LG ELECTRONICS MOBILECOMM U.S.A.,
`INC. of San Diego, Calif.; the INSTINCTTM or other mobile
`terminals manufactured by SAMSUNG TELECOMMUNI
`CATIONS AMERICA, LLC; the XPERIATM X1 or other
`40
`mobile terminals manufactured by SONY ERICSSON
`MOBILE COMMUNICATIONSAB of Sweden; the N96 or
`other mobile terminals manufactured by NOKIA GROUP of
`Finland; the PRETM or other mobile terminals manufactured
`by PALM, INC. of Sunnyvale, Calif., or the like. Software of
`45
`the present disclosure may operate in conjunction with any of
`these hardware platforms, or in conjunction with any plat
`forms similar to these.
`A block diagram of the mobile terminal 10 is illustrated in
`FIG. 2. The mobile terminal 10 includes the elements already
`illustrated and further includes, within the user interface 12, a
`camera 24 and the aforementioned microphone 26. The user
`interface 12 is operatively coupled to a central processor
`(CPU) 28. The CPU 28 is also operatively coupled to memory
`30 with software 32 stored therein, an accelerometer 34,
`transceiver circuitry 36, which in turnis coupled to an antenna
`38, one or more wired ports 40, a battery 42, and any other
`miscellaneous peripherals 44 as is common in the mobile
`terminal industry (e.g., an internal clock, a vibration unit, a
`GPS receiver, a thermometer, pressure sensor, a light sensor,
`an infrared sensor, a temperature sensor, a motion detector, an
`altimeter, a radio frequency identification (RFID) transpon
`der or receiver, or the like).
`FIG. 3 illustrates a variety of networks through which
`mobile terminals 10 may communicate. For example, a first
`mobile terminal 10A may communicate with a second mobile
`terminal 10B through a local wireless network such as a
`
`50
`
`55
`
`60
`
`65
`
`US 8,296,686 B1
`
`15
`
`25
`
`4
`BLUETOOTHTM, infrared, or other short range communica
`tion protocol. Additionally, or alternatively, the second
`mobile terminal 10B may communicate with a cellular net
`work such as the Public Land Mobile Network (PLMN) 46
`through a cellular base station 48. The PLMN 46 may com
`municate with the Public Switched Telephone Network
`(PSTN) 50 and/or the Internet 52. One or more computers 54
`may be communicatively coupled to the Internet 52 and may
`include a wireless transceiver to communicate wirelessly
`with a third mobile terminal 10C or a docking station 56 to
`communicate via a wired line with a fourth mobile terminal
`10D. More information on networks and communication pro
`tocols can be found in the Rules of Interpretation and General
`Definitions set forth below. Suffice to say that the particular
`network or communication protocol is not central to embodi
`ments of the present disclosure.
`Against this backdrop of hardware, an overview of an
`exemplary method is presented Starting with reference to
`FIG. 4 as a flow diagram interspersed with additional Figures
`for Supporting explanatory and exemplary Screen shots. Ini
`tially, the caregiver acquires a mobile terminal 10 (block 100).
`This acquisition may be a gift, a purchase, a lease, borrowing,
`or otherwise. For example, the disabled individual may pur
`chase the mobile terminal 10 and loan it to the caregiver for
`the purposes of carrying out embodiments of the present
`disclosure. Alternatively, the caregiver may purchase the
`mobile terminal 10 for use by and/or with the disabled indi
`vidual. The caregiver may load the software embodying the
`present disclosure onto the mobile terminal 10 (block 102). If
`the mobile terminal 10 has a disk drive or other data storage
`device or functionality (e.g. flash memory), the Software may
`be loaded from a disk or other source. Otherwise, the software
`may be loaded onto a computer 54 (e.g., via disk transfer or
`download) and transferred to the mobile terminal 10 through
`the docking station 56 or wirelessly. As still another alterna
`tive, the mobile terminal 10 may download the software
`through the PLMN 46 or other technique as desired. The
`software could be preloaded on the mobile terminal 10 such
`that when the lease or purchase is made, the software is
`immediately accessible. The software itself may be free, paid
`or subsidized in whole or part by a third party (such as a
`School district or other governmental agency or private foun
`dation). Note that while it is contemplated that the caregiver
`performs these and other steps, it is also possible that the
`disabled individual performs these and the other steps.
`The caregiver accesses the Software through the user inter
`face 12 of the mobile terminal 10 (block 104), such as by
`tapping the appropriate icon 22 on the display 16. Activating
`the Software may call up a screen Such as that illustrated in
`FIG. 5, where the software causes a number of choices to be
`displayed to the user. The choices, in this exemplary embodi
`ment, comprise picture schedule (block 106, FIG. 4; icon 58
`FIG. 5); set a visual countdown (block 108, FIG. 4; icon 60
`FIG.5); and offer choices (block 110, FIG. 4; icon 62, FIG. 6).
`The user may select an appropriate icon by touching the
`display 16 at an appropriate location, Scrolling through the
`prompts with the command button 18, or other technique as
`desired.
`To continue the example, assume the user selects picture
`schedule at block 106. A screenshot, such as that illustrated in
`FIG. 6 may appear. The user may be offered the choices of
`creating a new schedule (block112, FIG.4, icon 64 FIG. 6) or
`accessing a saved schedule (block 114, FIG. 4, icon 66, FIG.
`6). If the user selects create a new schedule, the mobile
`terminal 10 may display a screen shot such as that illustrated
`in FIG. 7, wherein icons 68 for various schedule categories
`are presented (block 116, FIG. 4). The user may select an
`
`APPL-1007
`APPLE INC. / Page 32 of 44
`
`

`

`US 8,296,686 B1
`
`5
`
`10
`
`15
`
`5
`appropriate icon 68, (causing the control system to receive the
`selection (block 118)) such as icon 68 for meal time to start
`creating a schedule that is thematically linked to the category
`of a meal time or at a minimum view images that are themati
`cally linked to the category for inclusion in a schedule. An
`alternate approach, wherein the first user indicates a time and
`Subsequently selects an associated icon (perhaps from a cat
`egory of icons) is also contemplated. Additionally, rather than
`select a stored image, the user may elect a “take photo now
`(or similarly labeled)