United States Patent
`
`[19]
`
`Hoflberg et al.
`
`[11]
`
`[45]
`
`Patent Number:
`
`Date of Patent:
`
`6,081,750
`
`Jun. 27, 2000
`
`US006081750A
`
`[54]
`
`[76]
`
`ERGONOMIC MAN-MACHINE INTERFACE
`INCORPORATING ADAPTIVE PATTERN
`RECOGNITION BASED CONTROL SYSTEM
`
`Inventors: Steven Mark Hoflberg, 20 Greystone
`Ter., Yonkers, N.Y. 10701-1705; Linda
`Irene Holfberg-Borghesani, 40 Jackson
`Dr., Acton, Mass. 01720
`
`Appl. No.: 08/471,213
`
`Filed:
`
`Jun. 6, 1995
`
`Related U.S. Application Data
`
`Continuation—in—part of application No. 07/812,805, Dec.
`23, 1991, Pat. No. 5,903,454.
`
`Int. Cl.7 ................................................... .. G05B 11/01
`US. Cl.
`700/17, 700/56; 700/11,
`00/83; 700/86; 345/520; 345/521
`Field of Search ................................... .. 364/140—147,
`364/148, 518, 188-190; 382/1, 8, 27, 30,
`34, 56, 16, 36, 42, 249, 700/11, 17,28,
`56, 47, 48, 49, 83, 84, 85; 345/520, 521
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`4,208,652
`4,697,209
`4,789,933
`4,841,575
`4,908,713
`4,963,994
`5,031,228
`5,060,277
`5,065,447
`5,076,662
`5,103,498
`5,123,057
`5,123,087
`5,136,659
`
`6/1980
`9/1987
`12/1988
`6/1989
`3/1990
`10/1990
`7/1991
`10/1991
`11/1991
`12/1991
`4/1992
`6/1992
`6/1992
`8/1992
`
`...... ..
`Marshall
`Kiewit et al.
`Chen et al.
`Welsh et al.
`
`.
`
`Barnsley et al.
`Shih et al.
`..
`lanier et al.
`.
`Verly et al.
`Newell et al.
`Kaneko et al.
`
`340/146.3Y
`358/84
`. 364/413.13
`.. 704/260
`........ .. 358/335
`........ .. 358/335
`382/38
`382/160
`382/56
`359/36
`395/68
`.. 382/156
`.. 345/348
`.................... .. 382/190
`
`OTHER PUBLICATIONS
`
`Hoffberg, Linda I., Thesis “An Improved Human Factored
`Interface for Programmable Devices.‘ A Case Study of the
`VCR”, Tufts University, Master of Sciences in Engineering
`Design.
`Micromint, “The Catalog of Embedded Controllers”, (Win-
`ter 1991):1—28.
`Didier LeGall, “MPEG: AVideo Compression Standard for
`Multimedia Applications”, Communications of the ACM.
`Shepard, Jeffrey D, “Tapping the Potential of Data Com-
`pression”, Military & Aerospace Electronics,
`(May 17,
`1993): 25-28.
`
`(List continued on next page.)
`Primary Examiner—Paul P. Gordon
`Assistant Examiner—Ramesh Patel
`Attorney, Agent, or Firm—Milde, Hoifberg & Macklin, LLP
`
`[57]
`
`ABSTRACT
`
`An adaptive interface for a programmable system, for pre-
`dicting a desired user function, based on user history, as Well
`as machine internal status and context. The apparatus
`receives an input from the user and other data. A predicted
`input is presented for confirmation by the user, and the
`predictive mechanism is updated based on this feedback.
`Also provided is a pattern recognition system for a multi-
`media device, wherein a user input is matched to a video
`stream on a conceptual basis, allowing inexact programming
`of a multimedia device. The system analyzes a data stream
`for correspondence with a data pattern for processing and
`storage. The data stream is subjected to adaptive pattern
`recognition to extract features of interest to provide a highly
`compressed representation which may be efficiently pro-
`cessed to determine correspondence. Applications of the
`interface and system include a VCR, medical device, vehicle
`control system, audio device, environmental control system,
`securities trading terminal, and smart house. The system
`optionally includes an actuator for effecting the environment
`of operation, allowing closed-loop feedback operation and
`automated learning.
`
`(List continued on next page
`
`24 Claims, 28 Drawing Sheets
`
`211“)
`lnlerlace
`‘
`mp“
`1 D1-v11'eJ
`
`311,5
`El l8\
`,
`l
`c1111111,11-
`"‘
`‘1lL.\ll4)l\ ‘m .
`Spcul
`111111111
`L111l<
`L‘“‘3°“"
`Purl‘
`M; 7 Optn1Ix7aL1011
`t1{1;11111; 111
`Stltt 1 10 1
`L/T
`
`1'09‘
`2110.. 7
`
`2 101
`Z l 01
`Current
`AL 2
`User Level
`
`
`
`AdaznvcWalglitmgI‘tr1wn‘1'
`
`71 UH/T_Lev1~1
`41,41. 11-61111211111
`3111 /« S1l:t1.1lCm11pm1cr.r
`Accu1ur‘1'o[ " T1
`l‘/117'./Qlfltllf 4
`2ll3—V“
`1
`Pl1y51o—Dyna1:1T7
`2l14\,
`71
`I)pnnu'1'.auon 7
`£11711-11141111,:
`Request
`Help Level15%.
`Current
`IJJS 'ay
`l
`,
`,
`
`Snupli irafion <‘¥113<
`
`Petitioners HTC and LG - Exhibit 1040, p. 1
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`6,081,750
`Page 2
`
`U.S. PATENT DOCUMENTS
`
`9/1992 Okazaki et al.
`....... .. 345/358
`5,148,522
`358/194.1
`9/1992 Young .......... ..
`5,151,789
`358/85
`3/1993 Graczyk et al.
`5,192,999
`395/600
`10/1993 Prager
`5,255,386
`.. 382/1
`.
`1/1994 Trew et al.
`5,280,530
`379/90
`.
`2/1994 Glick et al.
`5,283,819
`382/56
`9/1994 Barnsley et al.
`5,347,600
`2/1995 Luciw et al.
`......................... .. 395/12
`5,390,281
`3/1995 Remillard . . . . . . . .. . .
`. . . . .. . . . . .. 379/96
`5,396,546
`5/1995 Carlucci et al.
`.. 395/156
`5,412,773
`7/1995 Barnsley et al.
`.. 382/235
`5,430,812
`7/1995 Ellis et al.
`..
`.. 348/2
`5,436,653
`9/1995 Jensen et al.
`.. 380/6
`5,450,490
`5,477,447 12/1995 Luciw et al.
`0/119.08
`5,500,920
`3/1996 Kupiec
`_ 395/2.79
`5,508,815
`4/1996 Levine
`.. 358/335
`. .. .
`5,544,358
`8/1996 Capps et al.
`. .. . . . . .. 395/600
`5,560,011
`9/1996 Uyama .. . .. . . . . . .
`. . . . .. . . .. 395/700
`5,572,246
`11/1996 Ellis et al.
`.. 348/2
`5,583,966
`12/1996 Nakajima
`. 395/51
`5,584,050 12/1996 Lyons ..
`. 455/67.1
`5,586,317 12/1996 Smith
`.. 395/683
`OTHER PUBLICATIONS
`
`(Oct.,
`
`Green, Lee, “Thermo-Tech”, Popular Mechanics,
`1985): 155-160.
`Sperling, Barbra & Tullis, Thomas, “Are You A Better-
`“Mouser” or “Trackballer”? A Comparison of Cursor-Posi-
`tioning Performance”, McDonnell Douglas Astronautics
`Company : 1-1.
`Abedini, Kamran, “An Ergonomically-Improved Control
`Unit Design”, Proceedings of Interface 87, (1987): 375-380.
`Schmitt, Lee & Olson, Dean, “Let’s Discuss Programmable
`Controllers”, Modern Machine Shop (May 1987): 90-99.
`Carlson, Mark A., “Design Goals for an Effective User
`Interface”, Human Interfacing with Instruments,
`: 1-1.
`Wilke, William, “Easy Operation of Instruments by Both
`Man and Machine”, Human Interfacing with Instruments:
`1-4.
`
`Kriefeldt, John, “Human Factors Approach to Medical
`Instrument Design”, Human Interfacing with Instruments:
`1-6.
`
`Atkinson, Terry, “VCR Programming Making Life Easier
`Using Bar Codes”, The Globe.
`for Designing Better
`Abedine, Kamran,
`“Guidelines
`VCRS”, California State Polytechnic University, Pomona,
`Report No. IME 46.2, (Feb. 4, 1987).
`Card, Smart, “A Method for Calculating Performance Times
`for Users of Interactive Computing Systems”, IEEE CH
`1424, (Jan., 1979):653-658.
`Meads, Jon, “Friendly or Frivolous‘.”’, Hardware Gimmicks
`and Software, (Jan., 1988): 95-100.
`Kreifeldt, J.G., “A Methodology for Consumer Product
`Safety Analysis”, Dept of Engineering Design-Tufts Uni-
`versity,
`: 175-184.
`Carroll, Paul, “High Tech Gear Draws Ones of Uncle” Wall
`Street Journal, (Apr. 27, 1988): 29.
`Kolson, Ann, “Computer Wimps Drown in a Raging Sea of
`Technology”, Globe, (May 24, 1989).
`W'iedlnbeck, Susan, et al, “Using Protocol Analysis to Study
`the User Interface”, Bulletin of The American Society for
`Information Science, (Jul. 1989): 25-26.
`“The Highs and Lows of Nielsen HomeVideo Index,” Mar-
`keting & Media Decisions, (Nov. 1985): 84-86+.
`
`Tello, Ernest R, “Between Man and Machine”, Byte, (Sep.
`1988): 288-293.
`“Voice Recognition: Understanding the Master’s Voice”,PC
`Magazine, (Oct. 27, 1987): 261-308.
`Hawkins, William J ., “Super Remotes”, Popular Science,
`(Feb. 1989): 76-77.
`Norman, Donald A., “Infuriating by Design”, Psychology
`Today, Vol. 22, (Mar. 1988): 52-56.
`The Quest for ‘User Friendly’, US News & World Report,
`(Jun. 13, 1988): 54-56.
`Trachtenberg, Jeffrey, “How Do We Confuse Thee? Let Us
`Count the Ways”, Forbes, (Mar. 21, 1988): 155-160.
`Sharpe, Lora, “Teen Havens”, The Globe,
`:12.
`Cobb, Nathan, “I Dont Get It”, The Globe Magazine, (Mar.
`25, 1990): 22-29.
`Weiss, Ray, “32-Bit Floating Point DSP Processors”, EDN,
`(Nov. 7, 1997):128-146.
`Hoffberg, Linda I, “Designing AProgrammable Interface for
`a Video Cassette Recorder (VCR) to Meet
`the User’s
`Needs”, Interface ’9I .'346-35 1.
`Hoffberg, Linda I., “Designing User Interface Guidelines for
`Time-Shift Programming on a Video Cassette Recorder
`(VCR)” : 501-504.
`Hoban, Phoebe, “Stacking the Decks”, New York V 20: 14.
`Platte, Hans. Joachim et al “A New Intelligent Remote
`Control Unit for Consumer Electronic Devices”, IEEE,
`(1985): 59-68.
`Zeisel, Gunther et al, “An Interactive Menu-Driven Remote
`Control Unit for TV-Receivers and VC-Recorders”, IEEE,
`(1988):814—818.
`Bensch, U, “VPV-Videotext Programs Videorecorder”,
`IEEE, (1988): 788-792.
`Moore, T.G. & Dartnall, A., “Human Factors of a Micro-
`electronic Prod11ct:'l'he Central Heating 'l‘imer/Program-
`mer",Applied Ergonomics, (1982)13.1 : 15-23.
`Kraiss, K F, “Alternative Input Devices for Human Com-
`puter Interaction”, Preprint.
`Verplank, William, “Graphics in Human-Computer Com-
`munication:
`Principles
`of Graphical User-Interface
`Design”, Preprint.
`“Digital Compression Hikes Cable
`Doherty, Richard,
`Capacity”, Electronic Engineering, (Dec. 2, 1991): 1-16.
`Moser, Karen D., “REXX/Windows Shortens GUI Design
`Time”, PC Week, (1991).
`Davis, Frederic E, “A Scripting Language for the Mac One
`Man’Odyssey”, PC Week, (Nov. 11, 1991):142.
`McNamara, George, “Multimedia the Rainbow Pot-or Pot-
`hole?” Computer Technology, (1991).
`Costlow, Terry, “IBM Points a New Way”, Electronic Engi-
`neering Times, (Oct. 28, 1991):62.
`Casasent, David, “Optical Pattern Recognition: For Inspec-
`tion, Image Enhancement”, Photonics Spectra,
`(1991)
`:
`130-140.
`“Microway”, AD, 1991.
`“Scene Locator”, New Media (Nov./Dec. 1991).
`Kim, Yongmin, “Chips Deliver Multimedia”, Byte, (Dec.
`1991) : 163-173.
`“Frame Grabber/Imager Ilas TMS340Z0 IC”, Unknown.
`Bindra, Ashor et al, “TI Leads Five-Firm Parallel-Process-
`ing Effort”, Electronic Engineering Times, (Dec. 2, 1991):
`21-22.
`
`“Compression lCS Target Digital Cameras”,EDN, (Nov. 28,
`1991).
`Yoshida, Junko, “Battle Brewing Over Digital-Video For-
`mats”, Electronic Engineering Times, (Dec. 2, 1991): 20-21.
`
`Petitioners HTC and LG - Exhibit 1040, p. 2
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`6,081,750
`Page 3
`
`“Fractal Geometry Compresses Video Images That Have
`Independent Resolution”, EDN, (Nov. 7, 1991):122-123.
`Mera, Narciso, “DSP and Open Real Time OS Target
`Multimedia Applications”, Computer Technology Review,
`(Fall 1991) 2 14-17.
`Guglielmo, Connie, “MPEG Standard Aims to Squeeze
`Digital Video Into Mainstream”, lilac Week (Dec. 3, 1991
`vol. 5, No. 41): 31-32.
`Doherty, Richard, “MPEG Group Reveals Audiovisual Code
`Data”, Electronic Engineering Times 199], (Dec. 2, 1991) :
`97.
`Quinnell, Richard A, “Gyroscope Allows 3-D Motion Sens-
`ing for Robotics and Desktop Computers”, EDN, (Nov. 7,
`1991): 120.
`Zook, Chris, “8mm Incorporates Arthmetic Encoding for
`Data Compression”, Computer Technology Review,
`(Fall
`1991) : 81-85.
`Conway, William “New Modem Standards Challenge Inte-
`grators with Multiple Choices”, Computer Technology
`Review, (Fall 1991) 23-28.
`NBC TV News “Radio TV Reports” (Jul. 17, 1990) 1-37.
`“The “Smart” House: Human Factors in Home Automa-
`
`tion”, Human Factors in Practice (Dec. 1990) 3-36.
`LaGale, Didier, “MPEG: AVideo Compression Standard for
`Multimedia Applications”, Communications of the ACM
`(Apr. 1991/vol. 34, No. 4) 47-58.
`Yoshida, Junko, “EMC2 Pushes Video Rental by Satellite”,
`Electronic Engineering Times (Dec. 2, 1991) 97-98.
`
`Erickson, Thomas and Salomon, Gitta “Designing a Desktop
`Information System: Observations and Issues”, CHI ’91
`Proceedings, (1991) ACM 0—89791-383-3/91/0004/0049,
`pp. 49-54.
`Shepard, Jeffrey, “Trapping the Potential of Data Compres-
`sion”, Military &Aerospace Electronics, May 17, 1993, pp.
`25-28.
`
`Cypher, Allen, “Video Presentation Eager: Programming
`Repetitive Tasks by Example”, CHI
`’91 Proceedings,
`(1991), pp. 445-446.
`Ueda, Hirotada et al, “Impact: An Interactive Natural—Mo-
`tion-Picture Dedicated Multimedia Authoring System”,
`CHI
`’91 Proceedings,
`(1991) ACM 0-89791-383-3/91/
`0004/0343, pp. 343-350.
`Siochi, Antonio C. and HiX, Deborah, “A Study of Com-
`puter-Supported User Interface Evaluation Using Maximal
`Repeating Pattern Analysis”, C111 ’91 Proceedings, (1991),
`ACM (P89791-383-3/91/0004/0301, pp. 301-304.
`Smith, Sidney L. and Mosier, Jane N., “Guidelines for
`Designing User
`Interface Software", ESD-TR-86-278,
`MTR 10090, Mitre Corporation, Bedford, Massachusetts,
`(A11g., 1986), (pp. 1-10, 401-418 provided) NTIS AD A177
`198.
`
`Fox, Jelfrey, A. and Smith, Sydney L., “Dynamic Rules for
`User Interface Design" (Druid), M89-22, Mitre Corpora-
`tion, Bedford, Massachusetts, (May 1989), (pp. 1-2, 40-42
`provided).
`
`Petitioners HTC and LG - Exhibit 1040, p. 3
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`Jun. 27, 2000
`
`Sheet 1 0f 28
`
`6,081,750
`
`CLOCK SET
`
`I ¢ = Search Time
`
`~ Required
`k\\\‘ Keypresses
`
`
`
`AverageNumberofKeypresses
`
`U10O
`
`Petitioners HTC and LG - Exhibit 1040, p. 4
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`U.S. Patent
`
`Jun. 27,2000
`
`Sheet 2 0f 28
`
`6,081,750
`
`—
`W///J//'///1.
`
`—
`
`;
`
` .
`
`
`
`
`
`7/III/.7///////////'/.
`
` JIJ/A
`
`7//ZJ//////I//////9678910111213141516
`
`ETotalTime E]Total-Tr
`
`V
`
`Fig.3
`
`SubjectNumber
`
`?
`
`'I//////I///////////////J.
`
`
`
`S
`
`4
`
` d
`
`
`
`V//J////////////////A
`
`G
`
`1000
`
`S00
`
`SDUODBS III BUILL
`
`Petitioners HTC and LG - Exhibit 1040, p. 5
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`U.S. Patent
`
`Jun. 27,2000
`
`Sheet 3 0f 28
`
`6,081,750
`
`’////l///////////
`
`§§ kg
`
`7////////////////
`
`7////I////////////A
`
`V/////////Z
`
`VIIIIIIIIIIIIIIIIIIIII
`
`
`
` .§
`
`0lK
`
`§TotalTime DTotal-Tr Il
` SubjectNumber
`
`
`
`7///////////I/////A
`
`
`
`VIIIIIIIIIIIIII
`|§
`7//I//I////////////I
`
` '
`
`
`
` 4
`
` 4
`
`spuoaag LII 9U.II]_
`
`Petitioners HTC and LG - Exhibit 1040, p. 6
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`Jun. 27, 2000
`
`Sheet 4 0f 28
`
`6,081,750
`
`‘ P3
`
`Critical Steps
`
`Fig. 5
`
`Petitioners HTC and LG - Exhibit 1040, p. 7
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`U.S. Patent
`
`Jun. 27,2000
`
`Sheet 5 0f 28
`
`6,081,750
`
`:
`W///I///A
`—
`7////7///////4
`
`—
`V//////////I////.
`I
`'////////7/A.
`
`—
`
`7/7//I////////A78910111213141516
`RExistingInterface DNewInterface —
`
` SubjectNumber
`
`V//////I////I////A
`I
`VI////////////I/I4
`
`
`
`— —
`
`’II//////I///I//////I4
`
`7/////17//I/J//////4
`
`
`
`'//////////I//.
`
`VIIIIIIJIIIIIIIIII
`
`spuoaag (II 31111;
`
`Petitioners HTC and LG - Exhibit 1040, p. 8
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`U.S. Patent
`
`00027:2n.uJ
`
`Sheet 6 of 28
`
`6,081,750
`
`mumouomEEHHCL
`
`T1!!!‘
`
`I: Avg
`Min
`Max @ Std
`
` P
`
`Critical Steps
`
`rO»0.0oOyO»..>.I?////////;S
`
`Petitioners HTC and LG - Exhibit 1040, p. 9
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`U.S. Patent
`
`00027:2n.uJ
`
`Sheet 7 of 28
`
`6,081,750
`
`Min D Avg
`Max @ Std
`
`765
`
`000
`
`000
`
`munoummEEEH
`
`wO1O1O<O1O4O491O4O1§4O19.OOOOOOOOOO.w.~»§.§.~.§h
`
`P2
`
`Critical Steps
`
`Petitioners HTC and LG - Exhibit 1040, p. 10
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`U.S. Patent
`
`Jun. 27, 2000
`
`Sheet 8 0f 28
`
`6,081,750
`
`'7///J/I/////////4
`
`7////// III .
`
`
`
`V/I//I///II/II////I
`
`FIIIIIIIWIIIJIIIIIIIIJIVIII .
`
`7/////////////////////A
`
` JJ/WJVA
`
` J/J 678910111213141S16
`
`
`
`
`
`3%‘ExistingInterface DNewInterface
`
`
`
`SubjectNumber
`
` J
`
`V/////////I//////fl/fl///////A
`
`'
`:
`
` 4W/Jflfl 4‘5
`
` mm
`
`j
`7/ll////////////////////////////1//AW/Arr?///M1!////A m
`e
` mmwm N
`
`O
`O
`xn
`
`O
`O
`V)‘
`
`1ZZ7"7’/ /[IA H
`
`G
`
`1
`O
`O
`N
`
`SBS‘3E)JCM9)] JO 1aqu,mN
`
`Petitioners HTC and LG - Exhibit 1040, p. 11
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`Jun. 27, 2000
`
`Sheet 9 0f 28
`
`6,081,750
`
`Theoretical
`
`L
`
`‘ Actual
`
`k\\.
`
`Fxisting Interface
`
`New Interface
`
`Fig. 10
`
`TimeinSeconds
`
`0
`
`§
`
`.\\\\\\\\\\\\\\\\\\\\\\‘
`
`I: Theoretical
`
`Actual
`
`.
`
`Existing Interfa
`Times for the Minimum,
`
`New Interface
`oretical and Actual Keypresses
`
`Fig. 1 1
`
`Petitioners HTC and LG - Exhibit 1040, p. 12
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`U.S. Patent
`
`Jun. 27, 2000
`
`Sheet 10 0f 28
`
`6,081,750
`
`. v "9'
`.,v,;o;¢:o:;:o’o‘9 4
`,,'o’3’o’¢‘o’o‘o‘¢:o:<
`.«:‘:%(¥3i§I93t‘:‘
`‘~'¢’««¢««’o‘
`‘ °o’¢’¢’o’o’¢‘o’o’o’o‘
`“’°2°$°$°$°3°3°2°1
`~.g,o,««
`
`Existing Interface
`
`o'o'¢'6’6‘
`.
`o;o.o,¢,o,o«
`“$30 99 00o‘«
`
`0.‘
`~««««’o‘
`\o:o’o’¢:o‘o:¢:oj
`M 9
`90 o’o‘o‘o’¢’<
`9.0.9.0,”
`9,0 9 9,0,
`* ~to’«.
`
`Clock Set (106)
`
`New Interface
`
`Fig. 12b
`
`Petitioners HTC and LG - Exhibit 1040, p. 13
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`U.S. Patent
`
`Jun. 27, 2000
`
`Sheet 11 0f 28
`
`6,081,750
`
`Search (240) A
`,
`'O‘\
`,;:v:»:;.\,,,%
`0 0%‘?9 9 9
`oooooooo
`’o"o‘o’o’o’o’o’o"o‘s
`oooooooooo.
`,:,¢«>o¢¢¢oo,::;;.g‘
`O O
`X*3°:*23‘~‘:
`:.:.:2:.:.:.:.:.:.:.:::§:¢o?»'§7'
`, .
`-§N&flfifi9*
`‘\
`
`d&fl©“
`
`‘-'-Q P1(432)
`
`.
`J???
`oqgggfiggv,
`
`W$}““““‘§‘*’
`‘ ’ ooooo:£3”
`\
`§$fi$9’
`qggp
`
`Petitioners HTC and LG - Exhibit 1040, p. 14
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`U.S. Patent
`
`Jun. 27, 2000
`
`Sheet 12 0f 28
`
`6,081,750
`
`Input Instruction
`
`Display Error
`Message
`
`<j-‘ Verify Instruction
`As Being Consistent
`
`Append Instruction to
`Program Sequence
`
`I
`Analyze Program
`
`Uequence to Predict
`
`Next Action
`
`Display Predicted
`Next Action on
`Display
`
`User Accept
`Confirmation
`Screen
`
`Manual Input
`Option
`
`Display Complete
`Program Sequence
`to User
`
`User Accept
`
`No
`
`Yes
`
`Petitioners HTC and LG - Exhibit 1040, p. 15
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`U.S. Patent
`
`Jun. 27, 2000
`
`Sheet 13 0f 28
`
`6,081,750
`
`Input
`Instruction
`
`Assign Identifier
`n = x
`
`Delete
`Instruction n
`
`Verify
`Confirmation
`Screen
`
`Petitioners HTC and LG - Exhibit 1040, p. 16
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`U.S. Patent
`
`Jun. 27, 2000
`
`Sheet 14 0f 28
`
`6,081,750
`
`Input User
`Identification
`
`Personalized
`
`Startup Based on
`User Profile
`
`Update
`Preference
`Matrix
`
`Present Most
`likely Choices
`To User
`
`Update
`Preference
`Matrix
`
`Manual Input
`Op-L-ion
`
`Accept
`Confirmation
`Screen
`
`Petitioners HTC and LG - Exhibit 1040, p. 17
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`U.S. Patent
`
`00027:2n.uJ
`
`Sheet 15 of 28
`
`6,081,750
`
`25.20
`
`coE..NEofi:<835
`
`
`
`>mEm58293.8
`
`
`
`:33Em:ubsmum
`
`
`
`EbqouHobmbxm
`
`
`
`aowusbmfiE8325
`
`
`
`S9:Hobmbxm
`
`Petitioners HTC and LG - Exhibit 1040, p. 18
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`U.S. Patent
`
`00027:2muJ
`
`Sheet 16 of 28
`
`6,081,750
`
`S.3
`
`
`
`:932R82BEEBE$282N0933'
`
`
`
`
`mwuou:o:§E8E55:52.898:
`
`
`
`mfitutouSimu>:&.a$QSam35gmEfimm:EE_%_
`
`
`w>uS5m83m3585confineSEHtfim32.
`35%3:252
`
`Petitioners HTC and LG - Exhibit 1040, p. 19
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`U.S. Patent
`
`00027:2n.uJ
`
`Sheet 17 of 28
`
`6,081,750
`
`3&5
`
`
`
`qoumfihofiawanna
`
`
`wnuzazfimufiobuflm
`
`
`\wEU8uQ8E>>\u$m
`
`zsuhu\$a:mE<mime
`
`8.53.0293:
`
`
`
`EfiflsuofiwmEouwmxwmwm
`
`mufiobuflm
`
`3w,$mmw§8838
`
`NoamSow
`
`wk
`
`
`
`Ebqou.HOuO2
`
`882gummane
`
`Petitioners HTC and LG - Exhibit 1040, p. 20
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`U.S. Patent
`
`Jun. 27, 2000
`
`Sheet 18 0f 28
`
`6,081,750
`
`2119
`?
`Interface
`
`2118
`
`2llZ
`*1
`¢
`C<)mmu—‘
`i
`
`énput 1{ lllE:dl,ll0l1 {¥
`
`evice
`
`rm
`
`;
`
`.
`
`2 105]
`
`Optimization
`
`‘
`
`J\
`
`Quality of W
`
`Selection
`
`Past User
`
`]\
`
`History
`
`N
`
`X
`
`Current ‘if
`
`2108 JT Programming Status
`
`2
`
`2 1 1 0
`
`I
`
`Number of
`
`Errors
`
`Number ol’
`Cancels
`
`l
`
`I
`
`
`
`
`
`AdaptiveWeightingNetwork
`
`2111_/T
`
`I
`
`N
`
`__
`
`Explicit User
`Level Choice
`
`High Frequency
`
`211 2_/\ Signal Componen I‘
`
`211 3
`
`'
`2114
`
`Accuracy of
`MOVEIHEHI
`
`Physio-Dynamic
`OptiI1'1iZ£1tiO1’1
`
`2 115___/T
`
`Explicit Help
`Request
`
`_l
`2101
`|_ User Level
`I
`fl Help Level
`J
`[flmplification P105
`
`Current
`
`Current
`
`Display
`
`'
`
`,
`
`Petitioners HTC and LG - Exhibit 1040, p. 21
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`U.S. Patent
`
`00027:2n.uJ
`
`Sheet 19 of 28
`
`6,081,750
`
`........_
`
`u>u9%<
`
`game:
`
`vEO>fiwZ
`
`.HO..m_w.COUfSS
`
`.§mE:ouIRomm
`
`Bamboo
`
`Egommsmfip8%
`
`PENN
`
`EEEEH
`
`f¢¢$
`
`Smmomm8%3mm
`
`355$EmQ82»89>
`
`
`
`Sfiwbxm@223EtsmE9:
`
`Bmfifime
`
`mmmnfimm2m6wE$E_
`wfiwmwuoéKHmm
`
`83383::Smm
`
`omfoeu,./.
`
`Petitioners HTC and LG - Exhibit 1040, p. 22
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`U.S. Patent
`
`Jun. 27, 2000
`
`Sheet 20 0f 28
`
`6,081,750
`
`2302
`
`\
`
`Microprocessor
`
`Data
`Seriahzer
`
`Pattern
`Recognition
`System
`S
`
`2304
`
`Interface &
`Programming
`Console
`S
`
`2305
`
`Fig. 23
`
`Petitioners HTC and LG - Exhibit 1040, p. 23
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`U.S. Patent
`
`00027:2n.uJ
`
`Sheet 21 of 28
`
`6,081,750
`
`_23.33
`
`0,
`
`mm.::SSmo$
`
`mu.w:3H\
`
`Em>m2:?RamNwmmmnfiwm
`
`
`Hmw>b$mmo.E50,33Em:
`
` |
`Beam
`
`3%:_
`
`Til‘
`
`/Howm
`
`
`
`
`
`fimuodEQUwN.SwuUm.Hm£Umuummgflmfim
`
`mmmgfimm
`
`82
`
`Wk
`
`
`
`Eimumwflmmmnfimaqouamousr
`_
`25.20@8325
`.Nw§BomE>
`
`Nm35832;
`
`§m>m
`
`5:Embou
`
`Petitioners HTC and LG - Exhibit 1040, p. 24
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`U.S. Patent
`
`00027:2n.uJ
`
`Sheet 22 of 28
`
`6,081,750
`
`wommf
`
`SEf
`
`Sam
`
`nofimmzqfiou
`
`xaofiafiunm
`
`fmocnfibom
`
`umfifim
`
`
`
` Emé«EH35f
`
`owfiopm
`
`momm
`
`3m:uwHfiHw...fiHf
`
`wmfioam
`
`momm
`
`Em:
`
`82f8£$..£
`
`585$
`
`1moufiwoumm
`
`BE
`
`mm.3
`
`Petitioners HTC and LG - Exhibit 1040, p. 25
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`U.S. Patent
`
`Jun. 27, 2000
`
`Sheet 23 0f 28
`
`6,081,750
`
`2603
`
`H P
`
`lant
`Actuator
`
`Intermediate
`Sensor Data
`Storage I
`
`261 1
`
`Transform
`
`Input
`-
`DEVICE
`
`Characterization
`2 607K Network
`
`26 1 0
`
`2609
`
`Pattern
`Storage
`Memory
`
`Correlator
`
`1 2 608
`
`Fig. 26
`
`Petitioners HTC and LG - Exhibit 1040, p. 26
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`U.S. Patent
`
`Jun. 27, 2000
`
`Sheet 24 of 28
`
`6,081,750
`
`Store Templates
`
`Generate Image
`Domains
`
`Create Mapped Ranges
`
`Assign Identifiers
`
`Select Matching
`Ranges for
`Each Domain
`
`Represent Image as
`Identifiers
`
`Select Template That
`Matches Image From
`Combination of
`Identifiers
`
`Petitioners HTC and LG - Exhibit 1040, p. 27
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`U.S. Patent
`
`Jun. 27, 2000
`
`Sheet 25 0f 28
`
`6,081,750
`
`Input Templates
`
`Store Templates
`
`Input Image Data
`
`Store Image Data
`
`Create Domains of
`Stored Image Data
`
`Create Mapping
`Ranges of Image Data
`
`Process a
`Subset Range
`
`Yes
`
`Assign Identifiers
`
`Transform Domain
`
`Select Mapping
`Range by Criteria
`
`Represent Image
`As Identified
`
`Select
`0 Matching Template
`
`Stop
`
`Fig. 28
`
`Petitioners HTC and LG - Exhibit 1040, p. 28
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`U.S. Patent
`
`Jun. 27, 2000
`
`Sheet 26 0f 28
`
`6,031,750
`
`Input Templates
`
`Store Templates
`
`Input Motion Image Data
`
`Store Motion Image Data
`
`Create Domains of
`Stored Motion Image Data
`
`Create Mapping Ranges of Data
`With Corresponding Data
`
`I Process a Subset Range
`®
`
`YES
`
`Assign Identifiers
`
`Compute Motion Vector
`
`Select Mapping
`Range by Criteria
`
`Represent
`Motion Vectors
`
`Scene Change Detection
`
`Select
`Matching Template
`
`Fig. 29
`
`Petitioners HTC and LG - Exhibit 1040, p. 29
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`U.S. Patent
`
`Jun. 27, 2000
`
`Sheet 27 0f 28
`
`6,081,750
`
`Transform
`and Subtract
`
`;-----------------------u;///3002
`: Window
`§ Around Object
`=
`of Interest
`
`Create First
`
`Region
`
`Compare Codes
`With Templates
`
`Transform
`and Subtract
`
`Match
`Pr9ba_bihFY
`Maxmuzatlon
`
`Output
`Recognition
`
`Petitioners HTC and LG - Exhibit 1040, p. 30
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`U.S. Patent
`
`Jun. 27, 2000
`
`Sheet 28 0f 28
`
`6,031,750
`
`———>
`
`System Status
`
`Past History
`——>
`——.__;_+_—_J
`
`Adaptive
`Prediction System
`
`Contextuat
`
`Analysis
`
`Output Predicted
`Choices
`
`1
`
`C
`
`User Feedback
`Input
`
`[
`
`User Interface
`Control
`
`C
`
`C ‘T
`
`Control System
`
`Control Output
`
`l__,___o
`
`Petitioners HTC and LG - Exhibit 1040, p. 31
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`6,081,750
`
`1
`ERGONOMIC MAN-MACHINE INTERFACE
`INCORPORATING ADAPTIVE PATTERN
`RECOGNITION BASED CONTROL SYSTEM
`
`5
`
`The present application is a Continuation—in—Part of U.S.
`patent application Ser. No. 07/812,805, now U.S. Pat. No.
`5,903,454, filed Dec. 23, 1991.
`A portion of the disclosure of this patent document and
`appendices incorporated by the reference, contain material
`which is subject
`to copyright protection. The copyright
`owner has no objection to the facsimile reproduction by
`anyone of this patent document or the patent disclosure, as
`it appears in the US. Patent and Trademark Office patent file
`or records, but otherwise reserves all copyright rights what-
`soever.
`
`FIELD OF THE INVENTION
`
`The present invention relates to the field of programmable
`man-machine interfaces,
`for general or special purpose
`computing or sequencing devices. The present
`interface
`system is ergonomically adapted to provide an optimized
`environment for huma11 interaction with the device. The
`present system preferably employs a graphical direct-
`manipulation style interface. The present invention provides
`an enhanced interface for facilitating human input of a
`desired information and for modifying information previ-
`ously entered information.
`BACKGROUND OF THE INVENTION
`
`Significant difliculties are experienced by users when
`complex programmable devices having multiple commands
`which are infrequently used or progammed by those users.
`Further, when a user attempts to use an uncommon or rarely
`used function of these devices, which may be, for example
`video cassette recorders (hereinafter “VCRs”) difliculties are
`also encountered. For example, studies have concluded that
`80% of users cannot correctly program their VCRs. This has
`been due, in part, to the fact that manufacturers continue to
`add more features to existing devices, without simplifying
`those which already exist. Another problem has been the
`failure of manufacturers to design products in which the
`control interface adapts to the behavior of the user or to
`allow a system to be adaptive to the behaviors of a plurality
`of users.
`
`People learn most efliciently through the interactive expe-
`riences of doing, thinking, and knowing. Learning may often
`be facilitated by observing an experienced teacher. For
`ease—of—use, efficiency, and lack of frustration of the user,
`utilizing the device should be intuitive. Users should be able _
`to operate the device without referring to an instruction
`manual. Often, actual working examples are helpful. Well-
`designed products should contain visual clues which prompt
`and convey their meanings, however, prior art devices do not
`always live up to this ideal. This problem of insufficient
`perceptual clues to cue the user as to the identity and nature
`of available choices is accentuated by various manufacturers
`and designers who focus on the production and design of
`feature-rich systems, rather than on ones which are also
`“User Friendly” and th11s easier to use. Therefore, many
`products are extremely complex and thus diflicult to use,
`thereby preventing all but the most technically advanced
`people from using them. Other products are simple to use for
`a low level of functionality, but make it extremely diflicult
`to transcend an arbitrary barrier set by the interface designer.
`Some display systems have a higher available resolution
`than others, and the interface is preferably arranged to
`
`2
`optimize the intended display for the resolution limits and
`display format of the intended or available display device.
`Further, even with sufficient resolution, certain displays are
`of small size, and thus the visibflity of the information may
`also be optimized by taking into consideration the size,
`resolution, contrast, brightness of the display, ambient
`conditions, characteristics of the human visual system, fac-
`tors specific for a known user, and the available options of
`the apparatus. Thus, the interface may employ a number of
`methods to optimize the visibility of the information for a
`variety of display devices, storage formats and transmission
`standards. Known display standards and types include:
`NTSC, PAL, SECAM, CCIR—601, HDTV, MUSE, IDTV,
`VHS, S-VHS, Beta, SuperBeta, 8 mm, Hi-8 mm, videotel or
`picturephone (PX64), H.261, MPEG-1, MPEG-2. JPEG,
`computer display standards (CGA, HGC, EGA, PGA, VGA,
`SVGA, XGA, Macintosh (TM), 8514, Private Eye (TM) (a
`small reflection-scanned LET) line array for projecting a
`virtual display image in front of the eye, available from
`Reflection Technology, Inc.), LCD, etc., each of which may
`have a number of size ranges, e.g. about 1 cm2 to about 10
`m2, with a resolution range including displays having about
`16 dot matrix characters or more or about 16 by 64 pixels to
`about 2,048 by 2,048 pixels. Techniques such as
`antialiasing, font substitution, hinting, precompensating for
`expected distortion, etc., are all known employed to improve
`the readability of the display under various circumstances.
`
`PRIOR ART
`
`The prior art details a number of components of the
`present invention, and in fact,
`in a number of areas the
`present invention builds upon the prior art by adding novel
`aspects disclosed herein to result
`in improvements.
`Therefore, as set forth below, and in the attached appendix
`C of references (including abstracts), incorporated herein by
`reference, a significant number of references detail funda-
`mental technologies which may be improved according to
`the present invention. To the extent necessary, these tech-
`nologies are disclosed and are expressly incorporated herein
`by reference to avoid duplication of prior art teachings.
`Recitation hereinbelow of these teachings or reference to
`these teachings is not meant to imply that the inventors
`hereof were necessarily in any way involved in these
`references, nor that
`the particular improvements recited
`herein were made or conceived after the publication of these
`references. Thus, prior art cited herein is intended to (1)
`disclose information related to the application published
`before the filing hereof; (2) define the problem in the art to
`which the present invention is directed, (3) define prior art
`methods of solving various problems also addressed by the
`present invention; (4) define the state of the art with respect
`to methods disclosed or referenced herein; and/or (5) detail
`technologies used to implement methods or apparatus in
`accordance with the present invention.
`
`Human Interface
`
`One aspect of the present invention relates to a program-
`mable device that comprises a menu-driven interface in
`which the user enters information using a direct manipula-
`tion input device. Such a type of interface scheme is
`disclosed in Verplank, William I.., “Graphics in Human-
`Computer Communication: Principles of Graphical User-
`Interface Design”, Xerox Office Systems, and the references
`cited therein: Foley, J. D., Wallace, V. I.., (Chan, P., “The
`Human Factor of Computer Graphics Interaction
`Techniques”, IEEE CG&A, November 1984, pp. 13-48,
`
`Petitioners HTC and LG - Exhibit 1040, p. 32
`HTC and LG v. PUMA, IPR2015-01502
`
`

`
`6,081,750
`
`3
`Koch, H., “Ergonomische Betrachtung Von
`Schreibtastaturen”, Humane Production, 1, pp. 12-15
`(1985); Norman, D. A., Fisher, D., “Why Alphabetic Key-
`boards Are Not Easy To Use: Keyboard Layout Doesn’t
`Much Matter”, Human Factors 24(5), pp. 509-519 (1982);
`Perspectives: High Technology 2, 1985; Knowlton, K.,
`“Virtual Pushbuttons as a Means of Person-Machine
`Interaction”, Proc of Conf. Computer Graphics, Pattern
`Recognition and Data Structure, Beverly Hills, Calif., May
`1975, pp. 350-352; “Machine Now Reads, enters Informa-
`tion 25 Times Faster Than Human Keyboard Operators”,
`Information Display 9, p. 18 (1981); “Scanner Converts
`Materials to Electronic Files for PCs”,
`IEEE CG&A,
`December 1984, p. 76; “New Beetle Cursor Director
`Escapes All Surface Constraints”, Information Display 10,
`p. 12, 1984; Lu, C., “Computer Pointing Devices: Living
`VVith Mice”, High Technology, January 1984, pp. 61-65;
`“Finger Painting”, Information Display 12, p. 18, 1981;
`Kraiss, K. F., “Neuere Methoden der Interaktion an der
`Schnittstelle Mensch—Maschine”, Z. F. Arbeitswissenschaft,
`2, pp. 65-70, 1978, Hirzinger, G., Landzettel, K., “Sensory
`Feedback Structures for Robots with Supervised Learning”,
`IEEE Co11f. on Robotics and Automation, St. Louis, March
`1985; Horgan, H., “Medical Electronics”, IEEE Spectrum,
`January 1984, pp. 90-93.
`The following references are relevant to the interface
`aspects of the present invention, certain of which are incor-
`porated by reference in U.S. patent application Ser. No.
`07/812,805, incorporated herein by reference:
`Hoffberg, Linda 1, “AN IMPROVED HUMAN FAC-
`TORED INTERFACE FOR PROGRAMMABLE
`DEVICES: A CASE STUDY OF THE VCR” Master’s
`Thesis, Tufts University (Master of Sciences ir1 Engir1eer-
`ing Design, November, 1990).
`“Bar Code Programs VCR”, Design News, Feb. 1, 1988, 26.
`“The Highs and Lows of Nielsen Homevideo Index”, Mar-
`keting & Media Decisions, November 1985, 84-86+.
`“The Quest for “User Friendly’, U.S. News & World Report,
`Jun. 13, 1988. 54-56.
`“The Smart House: Human Factors in Home Automation”,
`Human Factors in Practice, December 1990, 1-36.
`“VCR, Camcorder Trends”, Television Digest, Vol. 29:16
`(Mar. 20, 1989).
`Abedini, Kamran, “An Ergonomically-improved Remote
`Control Unit Design”,
`Interface ’87 Proceedings,
`375-380.
`Abedini, Kamran, and Hadad, George, “Guidelines For
`Designing Better VCRs”, Report No. IME 462, Feb. 4,
`1987.
`Bensch, U., “VPV—VIDEOTEXT PROGRAMS
`VIDEORECORDER”,
`IEEE Transactions or1 Consumer
`Electronics, 34(3):788-792.
`Berger, Ivan, “Secrets of the Universals”, Video, February
`1989, 45—47+.
`Beringer, D. B., “A Comparative Evaluation of Calculator
`Watch Data Entry Technologies: Keyboards to
`Chalkboards”, Applied Ergonomics, December 1985,
`275-278.
`Bishop, Edward W., and Guinness, G. Victor Jr., “Human
`Factors Interaction with Industrial Design”, Human Factors,
`8(4):279-289 (August 1966).
`Brown, Edward, “Human Factors Concepts For
`Management”, Proceedings of the Human Factors
`Society, 1973, 372-375.
`Bulkeley, Debra, “The Smartest H