(12) United States Patent
`Dara-Abrams et al.
`
`USOO6456892B1
`(10) Patent No.:
`US 6,456,892 B1
`(45) Date of Patent:
`Sep. 24, 2002
`
`(54) DATA DRIVEN INTERACTION FOR
`NETWORKED CONTROL OF ADDITARGET
`DEVICE OVER A HOME ENTERTAINMENT
`NETWORK
`
`FOREIGN PATENT DOCUMENTS
`2767795
`3/1989
`OTHER PUBLICATIONS
`
`(75) Inventors: REEEEE, Sony et al., Specification of the Home Audio/Video Interop
`Edward Birmingham, both of San
`erability (HAVii) Architecture, May 11, 1998.
`Jose, all of CA (US); Neil David
`Matthews, Tadley (GB); Yoshifumi
`Yanagawa, Kyoto (JP); Wim
`Bronnenberg, Geldrop (NL)
`(73) Assignees: Sony Electronics, Inc., Park Ridge, NJ
`(US); Sony Corporation, Tokyo (JP)
`Subject t
`disclai
`the t
`f thi
`ubject to any disclaimer, the term of uns
`patent is extended or adjusted under 35
`U.S.C. 154(b) by 0 days.
`
`c:
`(*)
`
`Notice:
`OCC.
`
`(21) Appl. No.: 09/183,163
`1-1.
`(22) Filed:
`Oct. 30, 1998
`Related U.S. Application Data
`(63) Continuation-in-part of application No. 09/108,265, filed on
`Jul. 1, 1998, now Pat. No. 6,148.241.
`(51) Int. Cl. .......................... G05B 15700; G06F 13/00
`s
`s
`s
`(52) U.S. Cl. ........................... 700/83; 700/17; 709:28,
`710/10; 710/72
`(58) Field of Search ............................... 700/11, 12, 17,
`700/83; 710/8, 10, 12, 72; 709/201, 203,
`217, 218, 220, 321, 328; 34.5/173, 326,
`339, 327, 718, 746, 764, 810; 712/1
`
`(56)
`
`References Cited
`U.S. PATENT DOCUMENTS
`
`1/1990 Okada ......................... 360/48
`4,893,199 A
`5,086,385 A 2/1992 Launey et al. .............. 364/188
`(List continued on next page.)
`
`Primary Examiner Leo Picard
`ASSistant Examiner Steven R. Garland
`E. Attorney, Agent, or Firm Wagner, Murabito & Hao
`
`ABSTRACT
`(57)
`A method and System for providing a user interface for a
`networked target device within a home audio/visual net
`work. The DDI allows any DDI target to describe its
`physical appearance including controls and displays, etc., to
`a DDI controller. The controller can be a remote physical
`device or can be a Software program resident on the same or
`Separate device as the target. The controller interfaces with
`the target to obtain the DDI data and generates a user
`interface for the target including: 1) interactive controls; and
`2) user display information pertinent to the target. The DDI
`allows the controller tO trigger actions on the target as if a
`user had physically manipulated the controls of the target.
`The controller and the target are connected to the same
`communication network. A particular embodiment operates
`within the HAVii architecture. The controller communicates
`with the user by using the input and output devices of
`(typically) the device upon which the controller is executing.
`This communication can be done in a controller
`implementation-dependent manner. The target can be a
`DCM that controls its device in a implementation-dependent
`manner. The controller may be written in a generic manner
`that does not need to be implemented with knowledge of a
`particular target in mind; all target-dependencies are repre
`sented in the DDI data provided by the target to the
`controller.
`
`24 Claims, 16 Drawing Sheets
`
`ENTER
`-Fe
`
`RECEIVE USERACTION FROD CONTROLLER DEVICE OR FROM
`NRESCIPUDENICE
`942
`
`—- V -
`ORARD CHARGED ELEMENTSTODD CONTROLLERDEVICET)
`REPRESENT CHANGEN GUION DISPLAY DEFICE
`944
`
`process THE USERACTION TO CHANGE INTERNAL STATE OF
`TEDDITRGE
`946
`
`y
`FORWARD CHANGEREEMENTSODDIONTROLLER BASED OR
`THE CHARGE OF INTERNAL STATE OFDDITARGET DEVICE
`948
`
`
`
`--- YES
`
`STSUBSCRIBE
`950
`
`RO
`
`APPL-1010 / Page 1 of 39
`Apple v. Uniloc
`
`

`

`US 6,456,892 B1
`Page 2
`
`U.S. PATENT DOCUMENTS
`
`5,420,573 A 5/1995 Tanaka et al. ......... 340/825.24
`5,500,794. A 3/1996 Fujita ......................... 364/188
`5,519,878 A
`5/1996 Dolin ......................... 395/800
`5,537,605. A
`7/1996 Teece ......................... 395/800
`5,570,085. A 10/1996 Bertsch ................. 340/825.07
`
`
`
`5,621,662 A 4/1997 Humprhies et al.......... 364/550
`5,657,221. A * 8/1997 Warman et al................ 700/83
`5,706,191 A
`1/1998 Bassett et al. .............. 364/138
`5,796,607 A 8/1998 Le Van Suu ........... 364/140.01
`6,020,881. A * 2/2000 Naughton et al. .......... 345/327
`* cited by examiner
`
`APPL-1010 / Page 2 of 39
`
`

`

`U.S. Patent
`
`Sep. 24, 2002
`
`Sheet 1 of 16
`
`US 6,456,892 B1
`
`w
`w
`
`s
`wa
`
`sue
`
`se
`
`se
`
`APPL-1010 / Page 3 of 39
`
`

`

`U.S. Patent
`
`Sep. 24, 2002
`
`Sheet 2 of 16
`
`US 6,456,892 B1
`
`20]
`
`[euondg
`
`TOWUOLSLUC
`
`TAI
`
`afTE[OA
`
`a[tefo4-Uoyy
`
`TWaLNd)
`
`YOSSII0Nd
`
`
`
`THNOIS
`
`ONTLYEING)
`
`dOIAdC
`
`jeuorydg
`
`wosan)
`
`
`
`ZIAITTOULNOD
`
`jeuondg
`
`JRINAN-WHdTY
`
`
`
`QOIAdCLAdNI
`
`
`
`AQIAICAVTdSIC
`
`Ol
`
`101
`
`901
`
`01
`
`¢AYNOSIS
`
`APPL-1010 / Page 4 of 39
`
`APPL-1010 / Page 4 of 39
`
`
`
`
`
`
`
`
`
`
`
`

`

`U.S. Patent
`
`Sep. 24, 2002
`
`Sheet 3 of 16
`
`US 6,456,892 B1
`
`
`
`
`
`
`
`?ŽŽ ? ÄITIOXILNOJ
`
`[[[[[
`
`APPL-1010 / Page 5 of 39
`
`

`

`U.S. Patent
`
`Sep. 24, 2002
`
`Sheet 4 of 16
`
`US 6,456,892 B1
`
`21
`
`212
`
`FIGURE 4A
`
`APPL-1010 / Page 6 of 39
`
`

`

`U.S. Patent
`
`Sep. 24, 2002
`
`Sheet S of 16
`
`US 6,456,892 B1
`
`21
`
`
`
`FIGURE 4B
`
`APPL-1010 / Page 7 of 39
`
`

`

`U.S. Patent
`
`Sep. 24, 2002
`
`Sheet 6 of 16
`
`US 6,456,892 B1
`
`
`
`
`
`
`
`(HOIMHQ) LINII HOA
`
`APPL-1010 / Page 8 of 39
`
`

`

`U.S. Patent
`
`Sep. 24, 2002
`
`Sheet 7 of 16
`
`US 6,456,892 B1
`
`USER INPUT
`DEVICE
`106/10
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`DDI
`CONTROLLER
`N
`220
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`SUBSCRIBE
`GUI
`
`410
`
`ROOT GUI
`415 ELEMENT ID
`GET GUI
`420 ELEMENT ID
`GUI
`ELEMENTS
`
`425
`
`USEREVENT
`430 (ELEMENT ID,
`ACTION)
`
`
`
`CHANGED ELEMENT
`435
`(IDS)
`
`NOTIFY GUI CHANGE
`(CHANGED ELE (IDS)
`440
`
`UNSUBSCRIBE
`GUI
`
`DDI TARGET
`210
`
`USER
`DISPLAY
`DEVICE
`105
`
`
`
`DDI TARGETS
`PHYSICAL
`PANE
`204
`
`
`
`FIGURE 6
`
`APPL-1010 / Page 9 of 39
`
`

`

`U.S. Patent
`U.S. Patent
`
`Sep. 24, 2002
`Sep. 24, 2002
`
`Sheet 8 of 16
`Sheet 8 of 16
`
`US 6,456,892 B1
`US 6,456,892 B1
`
`q0rs
`
`
`
`5016
`
`JOYNOI
`
`#IG
`
`Pols
`
`7AXNDIA
`
`POIs
`
`APPL-1010 / Page 10 of 39
`
`APPL-1010 / Page 10 of 39
`
`

`

`U.S. Patent
`
`Sep. 24, 2002
`
`Sheet 9 of 16
`
`US 6,456,892 B1
`
`
`
`\/8 ERH[15)|–|
`
`GOED
`
`
`
`GÊD GOED
`
`APPL-1010 / Page 11 of 39
`
`

`

`U.S. Patent
`
`Sep. 24, 2002
`
`Sheet 10 of 16
`
`US 6,456,892 B1
`
`
`
`NOII HININH
`
`APPL-1010 / Page 12 of 39
`
`

`

`U.S. Patent
`
`Sep. 24, 2002
`
`Sheet 11 of 16
`
`US 6,456,892 B1
`
`O
`
`START
`
`DDI CONTROLIER SUBSCRIBES TO A PARTICULAR DDI TARGET
`DEVICE
`912
`
`DDI TARGET FORWARDS ROOT GUI ELEMENT ID TO THE DDI
`CONTROLLER THAT IS SUBSCRIBINC
`914
`
`DDI CONTROLLER REQUESTS GUI ELEMENT ID BASED ON
`RECEIVED INFORMATION
`916
`
`GUI ELEMENTS ARE FORWARDED FROM THE DDI TARGET TO THE
`DD CONTROLLER
`918
`
`
`
`
`
`
`
`YES
`
`
`
`MORE GUI
`ELEMENT TO CET
`920
`
`NO
`
`
`
`
`
`DISPLAY GRAPHICAL, USER INTERFACE ON DISPLAY DEVICE OF
`DDI CONTROLLER AND INTERFACE WITH USER
`922
`
`FIGURE 9A
`
`APPL-1010 / Page 13 of 39
`
`

`

`U.S. Patent
`
`Sep. 24, 2002
`
`Sheet 12 of 16
`
`US 6,456,892 B1
`
`22
`
`ENTER D
`
`USER INTERACTS WITH GUI ON DDI CONTROLLER DEVICE
`924
`
`DDI CONTROLLER DEVICE FORWARDS USER ACTION MESSAGE TO
`DDI TARGET DEVICE
`926
`
`NOTIFICATION OF CHANGED ELEMENTS AND CHANGED ELEMENT
`MESSAGES ARE RECEIVED FROM DDI TARGET
`928
`
`DDI CONTROLLER UPDATES THE GUI OF THE DISPLAY DEVICE
`BASED ON THE CHANCES FROM THE DDI TARGET
`930
`
`
`
`
`
`
`
`
`
`STILL SUBSCRIBED
`932
`
`
`
`
`
`NO
`
`RETURN
`
`FIGURE 9B
`
`APPL-1010 / Page 14 of 39
`
`

`

`U.S. Patent
`
`Sep. 24, 2002
`
`Sheet 13 of 16
`
`US 6,456,892 B1
`
`940
`
`ENTER
`
`RECEIVE USER ACTION FROM DDI CONTROLLER DEVICE OR FROM
`OWN PHYSICAL INPUT DEVICE
`942
`
`
`
`FORWARD CHANGED ELEMENTS TO DDI CONTROLLER DEVICE TO
`REPRESENT CHANGE IN CUI ON DISPLAY DEVICE
`944
`
`PROCESS THE USER ACTION TO CHANGE INTERNAL STATE OF
`THE DDITARCET
`946
`
`FORWARD CHANGE IN ELEMENTS TO DDI CONTROLLER BASED ON
`THE CHANGE OF INTERNAL STATE OFDDI TARGET DEVICE
`948
`
`
`
`YES
`
`
`
`
`
`STILL SUBSCRIBED
`950
`
`NO
`
`RETURN
`
`FIGURE 9C
`
`APPL-1010 / Page 15 of 39
`
`

`

`U.S. Patent
`
`Sep. 24, 2002
`
`Sheet 14 of 16
`
`US 6,456,892 B1
`
`
`
`DISPLAY A PANE
`ON ONE PAGE
`
`
`
`
`
`
`
`
`
`PANE
`REPRESENTATION
`640
`
`
`
`
`
`
`
`
`
`
`
`
`
`HAS GROUP
`
`CHOOSE A GROUP
`USING GROUP D
`RULE
`
`EEMENTS
`REPRESENTATION REPRESENTATION
`650
`
`EEMENTS
`REPRESENTATION
`655
`
`FINISH THIS
`GROUP
`660
`
`
`
`FINISH
`EEMENTS
`
`FINISH L.
`EEMENTS
`
`YES
`
`FIGURE 10
`
`APPL-1010 / Page 16 of 39
`
`

`

`U.S. Patent
`
`Sep. 24, 2002
`
`Sheet 15 of 16
`
`US 6,456,892 B1
`
`O
`
`
`
`
`
`
`
`
`
`
`
`HAS POSITION
`ATTRIBUTE
`720
`
`DISPLAY ALL ELEMENTS
`USING POSITION RULE
`730
`
`
`
`
`
`
`
`
`
`DISPLAY ALL ELEMENTS
`USING ENTRY ORDER RULE
`AND NON-POSITION RULE
`725
`
`FIGURE 11
`
`APPL-1010 / Page 17 of 39
`
`

`

`U.S. Patent
`
`Sep. 24, 2002
`
`Sheet 16 of 16
`
`US 6,456,892 B1
`
`O
`
`START
`
`CHOOSE AN ELEMENT USINC
`ENTRY ORDER RULE
`815
`
`HAS RELATION
`ATTRIBUTE
`820
`
`NO
`
`
`
`
`
`YES
`— —
`CHOOSE THE NEXTELEMENT
`USING RELATION RULE
`825
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`DISPLAY THESE TWO
`ELEMENTS USING
`NON-POSITION RULE
`830
`
`
`
`DISPLAYING THE ELEMENT
`USING NON-POSITION RULE
`835
`
`
`
`- NO
`
`FILLED UP ONE
`PACE
`840
`
`
`
`FIGURE 12
`
`APPL-1010 / Page 18 of 39
`
`

`

`1
`DATA DRIVEN INTERACTION FOR
`NETWORKED CONTROL OF ADDITARGET
`DEVICE OVER A HOME ENTERTAINMENT
`NETWORK
`
`RELATED UNITED STATES PATENT
`APPLICATION
`The present application is a continuation-in-part of patent
`application Ser. No. 09/108,265, now U.S. Pat. No. 6,148,
`241, filed on Jul. 1, 1998 entitled “A Method and System for
`Providing a User Interface for a Networked Device Using
`DDI Descriptor Information,” by Harold Aaron Ludtke,
`Harumi Kawamura and Hisato Shima and assigned to the
`assignee of the present application and incorporated herein
`by reference.
`
`BACKGROUND OF THE INVENTION
`
`1. Field of the Invention
`The present invention relates to the field of consumer
`electronic devices. More Specifically, the present invention
`relates to methods and Systems for providing user interfaces
`for networked electronic devices including remote devices.
`2. Related Art
`The typical home entertainment System today consists of
`a variety of different consumer electronic devices which
`present and record audio/visual media in different ways. In
`the field of media devices, there is a spectrum of features for
`products of a given class (VCRS, video camera, etc.). Most
`of the features are represented by physical controls or
`elements on a control panel on the device which can be
`manipulated by a human user.
`For instance, typical home audio-visual (AV) equipment
`includes a number of components Such as a radio receiver or
`“tuner, a compact disk (CD) player and/or a digital video
`disc player (DVD), a number of speakers, a television, a
`Video cassette recorder (VCR), a tape deck, and the like.
`Each of these components is connected to the others via a Set
`of wires. One component is usually the central component of
`the home AV System; for example, the tuner. The tuner has
`a number of Specific inputs for coupling the other compo
`nents. Tuner-based control is often limited to connecting/
`disconnecting another component's AV input or output. The
`tuner has a corresponding number of control buttons or
`control Switches which provide a limited degree of control
`lability and interoperability for the components. However,
`only rarely does a tuner control the component's operation
`(e.g., rewind, play, etc.). The control buttons and control
`Switches are usually located on the front of the tuner
`providing the user with a So-called hard front panel inter
`face. In many cases, Some, or all, of these buttons and
`Switches are duplicated on a hand-held remote control unit,
`another type of hard front panel interface. A user controls the
`home AV System by manipulating the buttons and Switches
`on the front of the tuner or, alternatively, by manipulating
`buttons on the hand-held remote control unit. This conven
`tional home AV System paradigm has become quite popular.
`AS consumer electronic devices become more capable and
`more complex, the demand for the latest and most capable
`devices has increased. AS new devices emerge and become
`popular, the devices are purchased by consumers and
`“plugged' into their home AV Systems. As a consumer
`purchases new devices (e.g., digital audio tape recorders,
`digital video disk players, digital camcorders, and the like),
`most often the new device is simply plugged into the System
`alongside the pre-existing, older devices (e.g., cassette tape
`
`15
`
`25
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`US 6,456,892 B1
`
`2
`deck, CD player, and the like). The new device is plugged
`into an open input on the back of the tuner or into Some other
`device coupled to the tuner. The consumer (e.g., the user)
`controls the new device via the control buttons on the tuner,
`via the control buttons and control Switches on the front of
`the new device itself, or via an entirely new and Separate
`remote control unit for the new device.
`AS the number of new consumer electronic devices for the
`home AV System have grown and as the Sophistication and
`capabilities of these devices have increased, a number of
`problems with the conventional paradigm have emerged.
`One such problem is incompatibility between devices in the
`home AV System. Consumer electronic devices from one
`manufacturer often couple to an AV System in a different
`manner than Similar devices from another manufacturer. For
`example, a tuner made by one manufacturer may not prop
`erly couple with a television made by another manufacturer.
`In addition, where one device is much newer than another
`device, additional incompatibilities may exist. For example,
`a new device might a incorporate hardware (e.g., specific
`inputs and outputs) which enables more Sophisticated
`remote control functions. This hardware may be unusable
`with older devices within the AV system. Also, for example,
`older tunerS may lack Suitable inputs for Some newer devices
`(e.g., mini-disc players, VCRs, etc.), or may lack enough
`inputs for all devices of the system. Another problem is the
`lack of functional support for differing devices within an AV
`System. For example even though a television may Support
`advanced Sound formats (e.g., Surround Sound, Stereo, etc.),
`if an older and less capable tuner does not Support Such
`functionality, the benefits of the advanced Sound formats can
`be lost.
`Another problem is the proliferation of controls for the
`new and differing devices within the home AV system. For
`example, Similar devices from different manufacturers can
`each have different control buttons and control Switch for
`mats for accomplishing Similar tasks (e.g., Setting the clock
`on a VCR, programming a VCR record a later program, and
`the like). In addition, each new device coupled to the AV
`System often leads to another dedicated remote control unit
`for the user to keep track of and learn to operate.
`To address the problems above, a home AV network has
`been proposed for consumer electronic devices, including
`computing devices (e.g., personal computers and peripheral
`devices such as printers). The home AV network would
`provide a set of Services which facilitate device interoper
`ability and allow the devices to cooperate to perform appli
`cation tasks. By connecting consumer electronic devices in
`a home AV network, it is also possible to share processing
`and Storage resources. In addition, a home AV network
`allows consumer electronic devices to be coordinated and
`Simultaneously controlled by a user, thereby simplifying that
`aspect of operation from the user's perspective. The under
`lying Structure of a home AV network consists of a Set of
`interconnected clusters of devices. Typically, there may be
`more than one cluster in a home, perhaps one per floor or one
`per room. Each cluster functions as a set of interconnected
`devices to provide Services to a Single user or to multiple
`Simultaneous users.
`Recently, a class of consumer electronic media devices
`has been proposed that can be networked together using a
`particular type of network using a Standard communication
`protocol layer, e.g., IEEE 1394 communication Standard.
`The IEEE 1394 standard is an international standard for
`implementing an inexpensive high-speed Serial bus archi
`tecture which Supports both asynchronous and isochronous
`
`APPL-1010 / Page 19 of 39
`
`

`

`25
`
`3
`format data transfers. The IEEE 1394 standard provides a
`high-Speed Serial bus for interconnecting digital devices
`thereby providing universal input/output connection. The
`IEEE 1394 standard defines a digital interface for applica
`tions thereby eliminating the need for an application to
`covert digital data to an analog form before it is transmitted
`acroSS the bus. Correspondingly, a receiving application will
`receive digital data from the bus, not analog data and will
`therefore not be required to convert analog data to digital
`form. The IEEE 1394 is ideal for consumer electronics
`communication in part because devices can be added to or
`removed from the serial bus while the bus is active. If a
`device is So added or removed, the bus automatically recon
`figures itself for transmitting data between the then existing
`devices. Each device on the bus is a “node” that has its own
`address on the bus and contains its own address Space. Each
`node connected to the bus is capable of communicating with
`any other node also on the bus.
`The provision of the IEEE 1394 serial communication bus
`for networking consumer electronic devices has introduced
`a powerful new platform on which device functionality and
`inter-operability can be built. For instance, in Such a System,
`complex operations involving media transfers, media
`recordings and media presentation can be performed that
`involve two or more devices acting in concert. However,
`interaction between these devices can be complex, error
`prone and laborious if it is required that each device be
`directly touched in order to properly configure the desired
`operation. The problems associated with properly configur
`ing the media operation can be exacerbated if one or more
`of the devices are remotely located and/or need constant
`attention and or adjustment. What is needed is an effective
`mechanism for interfacing with networked consumer elec
`tronic devices to facilitate media operations between two or
`more devices.
`35
`The physical Separation between devices in a home AV
`network introduces a problem regarding control of a par
`ticular device. A user may be trying to remotely control a
`device that is physically Separated from the user and thus is
`not able to be seen. For example, a VCR on the home AV
`40
`network may be in one room, while the user is in another
`room containing the television. Therefore, the user is not
`Sure whether he/she entered in the proper commands and
`that the device is correctly performing the desired function.
`Another problem is introduced when there are multiple
`users Simultaneously on the home AV network, each user
`working from a separate location. Thus, continuing the
`example above, more than one user may be attempting to
`control the VCR. Each user, not aware of the fact that there
`is another user also trying to use the VCR, will be entering
`commands that are perhaps conflicting and that are not being
`carried out properly. In addition, each user will not have
`information regarding the status of the VCR (e.g., whether
`the VCR is recording or playing), and thus will not be aware
`of whether they Successfully entered the proper commands
`and whether the device is performing the desired function.
`In addition, it is often desired to provide remote control
`access to the features performed by a consumer electronic
`device So that these devices can be accessed from a central
`location within a home or office. However, not all consumer
`electronic devices are able to provide a Sophisticated display
`System for remote interaction, e.g., Some devices offer only
`a Small liquid crystal display (LCD) or a Small collection of
`light emitting diodes (LEDs) as display devices. What is
`needed is a mechanism for interfacing remotely with devices
`that provides a Sophisticated level of user interaction for
`devices of differing local display capability. Also, in many
`
`45
`
`50
`
`55
`
`60
`
`65
`
`US 6,456,892 B1
`
`15
`
`4
`consumer electronic Systems, each device can have its own
`remote control unit. In Systems having even a modest
`amount of devices, the user discovers that three or more
`different and bulky remote control devices are required to
`perform media operations. What is needed is a mechanism
`for interfacing with electronic devices that reduces the
`problems of having different remote control units for dif
`ferent devices. What is needed further is a mechanism for
`interfacing with electronic devices that is flexible and can
`adapt to new devices and device types within the consumer
`electronicS market.
`
`SUMMARY OF THE INVENTION
`Accordingly, the present invention provides an effective
`mechanism for interfacing with networked consumer elec
`tronic devices to facilitate media operations between two or
`more devices. The present invention also provides a mecha
`nism for interfacing remotely with devices and that provides
`a Sophisticated level of user interaction for many devices
`that themselves may have limited display capability. The
`present invention also provides a mechanism for interfacing
`with electronic devices that operates using a remote con
`troller and a central display for instructing different elec
`tronic devices and for receiving Status information regarding
`the different electronic devices. The present invention pro
`vides an interface mechanism for interfacing with electronic
`devices that is additionally flexible and can adapt to new
`devices and device types within the consumer electronics
`market. The present invention provides the above advanta
`geous features within one embodiment that is compliant
`with the Home Audio/Visual Interoperability (HAVi) archi
`tecture. These and other advantages of the present invention
`not specifically mentioned above will become clear within
`discussions of the present invention presented herein.
`A method and System are described herein for providing
`a user interface for a networked electronic device using a
`data driven interface (DDI) including a data structure of
`information maintained between a DDI target device and a
`DDI controller. The present invention utilizes the DDI to
`allow any compliant device (e.g., a DDI target device) to
`have the physical appearance of its controls and displayS
`(e.g., elements) represented by a data structure used by
`another device (e.g., a DDI controller) and allows the DDI
`controller to thereby trigger actions on the DDI target device
`as if a user had directly physically manipulated the controls
`on the DDI target device. The present invention operates
`within a network of consumer electronic devices, e.g.,
`television (TV), set-top-box, video cassette recorder (VCR),
`compact disk (CD) device, personal computer System (PC),
`etc., that are coupled together using a Standard communi
`cation protocol layer, e.g., the IEEE 1394 Serial communi
`cation standard. ADDI controller (e.g., a TV or set-top-box
`or computer System or other intelligent electronic device)
`monitors the network to discover the units coupled thereto
`and is programmed to locate DDIs that are defined for a DDI
`target device. The DDI target device may be remotely
`located. In one embodiment, the network is compliant with
`the HAVii architecture.
`In general, the DDI provides the DDI controller with
`information for rendering a depiction of the controls and
`displays of the DDI target device and this depiction can be
`interacted with by a user for remotely triggering actions by
`the DDI target device. The DDI is defined such that com
`mand interpretations and image alterations are controlled by
`the DDI target device thereby reducing the responsibility of
`the DDI controller in managing the user interface with
`generic user events. This approach allows increased flex
`
`APPL-1010 / Page 20 of 39
`
`

`

`S
`ibility to expand the user interface and adjust to future
`control types and display information protocols.
`The DDI uses graphical user interface (GUI) "elements”
`Stored in a data Structure to define the physical controls of
`the DDI target device. The GUI elements include standard
`types of controls and displays (e.g., push buttons, toggles,
`Sliders, dials, LCD Screens, alphanumeric inputs, etc.) that
`are commonly found on consumer electronic devices. The
`control types have well defined behaviors (e.g., buttons are
`pushed, dials are moved, values are input, etc.). In addition,
`the DDI defines a set of “user events” which can be applied
`to any of these controls by the user. The user events are
`defined to be Sufficiently generic So that they apply to most
`types of controls. The purpose of these generic user events
`is to encapsulate the typical user manipulated action for the
`GUI elements and to let the DDI target device decide the
`manner in which to interpret the user actions on its own
`thereby relieving the DDI controller of these responsibili
`ties. Related elements of a DDI target device can be orga
`nized together into logical groups which the DDI allows to
`be displayed or processed together in Special ways. The DDI
`controller is responsible for the exact visual appearance of
`the GUI elements and their exact layout on its display.
`The internal state of the DDI target is maintained by the
`DDI target which keeps the DDI controller informed of its
`Status. The DDI target forwards Status update messages
`(upon changes of State) to the DDI controller to update its
`graphical display for the user. A Status notification com
`mands also allows the DDI controller to poll the DDI target
`for changes in State.
`Aparticular embodiment of the present invention operates
`within the HAViarchitecture. Using the DDI mechanism, a
`HAViSoftware element can provide a user with the ability to
`control another Software element. Within this interaction, the
`first Software element is termed the DDI controller and the
`second software element the DDI target. The DDI controller
`uses a description of the User Interface (UI) to be presented
`to the user and is obtained from the DDI target. The means
`by which this control is accomplished is described herein.
`In this particular embodiment, the DDI controller and its
`DDI target are both HAVii components each executing on an
`IAV or FAV device. The components may be on the same or
`different devices, implemented using native code or Java
`bytecode; in all cases, though, they interact by Sending HAVi
`messages to each other. The DDI controller communicates
`with the user by using the input and output devices of
`(typically) the device upon which the controller is executing.
`This input/output communication can be done in a
`controller-implementation-dependent manner. The DDI tar
`get can be a DCM that controls its device in a
`implementation-dependent manner. The HAViDDI protocol
`described herein is structured so that a DDI controller may
`be written in a generic manner. That is, a DDI controller does
`not have to be implemented with knowledge of a particular
`DDI target in mind; all target-dependencies are represented
`in the DDI data provided by the target to the controller.
`BRIEF DESCRIPTION OF THE DRAWINGS
`FIG. 1 illustrates an exemplary networked system of
`electronic devices including a Video camera, a Video cassette
`recorder, a computer, a Set-top-box, a television and a
`compact disk changer.
`FIG. 2 illustrates exemplary components of a DDI con
`troller device in accordance with the present invention.
`FIG. 3 is a logical diagram of a networked System in
`accordance with the present invention including one or more
`DDI controllers and a DDI target device.
`
`6
`FIG. 4A is a perspective view of a DDI target device
`having a physical panel display elements and panel control
`elements.
`FIG. 4B is a perspective view of the DDI target device of
`FIG. 4A having a front panel in the flipped-open State to
`expose more physical control and display elements.
`FIG. 5 illustrates a logical block diagram of the logical
`partitions or “subunits' located within a DDI target device
`(a VCR) including the DDI of the present invention.
`FIG. 6 is an illustration of the communication interface
`between a DDI target and a DDI controller in accordance
`with the present invention.
`FIG. 7 is a diagram illustrating an exemplary GUI dis
`played on a DDI controller in response to a target's DDI
`defined in accordance with the present invention.
`FIG. 8A illustrates an exemplary generic DDI hierarchy,
`including panels, objects and non-organizational GUI
`elements, defined in accordance with the present invention.
`FIG. 8B illustrates an exemplary specific DDI hierarchy,
`including panels, objects and Specific non-organizational
`GUI elements, defined in accordance with the present inven
`tion.
`FIG. 9A is a flow diagram illustrating a process used by
`a DDI controller and a DDI target for rending a GUI on the
`DDI controller in response to the target's DDI.
`FIG.9B is a flow diagram illustrating the steps performed
`by the DDI controller of FIG. 9A in response to a user
`interaction with the controller's GUI and in response to the
`DDI target's State change messages.
`FIG.9C is a flow diagram illustrating the steps performed
`by the DDI target of FIG. 9A in response to user event
`messages from the DDI controller and in response to inter
`action with the targets own physical input device (if
`present).
`FIG. 10 is a flow diagram illustrating the steps performed
`by the present invention for displaying and presenting a
`representation of a panel as defined by the DDI of the
`present invention.
`FIG. 11 is a flow diagram of the panel and group repre
`Sentation in accordance with the present invention.
`FIG. 12 is a flow diagram of the element representation in
`accordance with the present invention.
`
`DETAILED DESCRIPTION OF THE
`INVENTION
`In the following detailed description of the present
`invention, a data driven interface (DDI) for allowing a DDI
`controller device to remotely interface with a DDI target
`device within a network of consumer electronic devices,
`numerous Specific details are Set forth in order to provide a
`thorough understanding of the present invention. However,
`it will be recognized by one skilled in the art that the present
`invention may be practiced without these specific details or
`with equivalents thereof. In other instances, well known
`methods, procedures, components, and circuits have not
`been described in detail as not to unnecessarily obscure
`aspects of the present invention.
`The present invention is drawn to a DDI that allows any
`compliant device (e.g., “DDI target device') to describe its
`physical appearance including controls and displays, etc., to
`a DDI controller. The DDI controller can be a remote
`physical device or can be a Software program resident on the
`same or separate device as the DDI target. The DDI con
`troller interfaces with the DDI target to obtain the DDI data
`
`US 6,456,892 B1
`
`15
`
`25
`
`35
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`APPL-1010 / Page 21 of 39
`
`

`

`US 6,456,892 B1
`
`7
`and generates a user interface for the DDI target device
`including: 1) interactive controls; and 2) user display infor
`mation pertinent to the DDI target device. The DDI allows
`the DDI controller to trigger actions on the DDI target
`device as if a user had physically manipulated the controls
`of the DDI target device. The DDI controller and the DDI
`target device are connected to the same communication
`network (e.g., using the IEEE 1394 serial standard). The
`present invention can also be used in one embodiment to
`promote high levels of inter-operability between any of the
`devices.
`Generally, the present invention allows the DDI controller
`to communicate with the DDI of the DDI target device and
`inquire about the various types of controls which it has, e.g.,
`buttons, dials, sliders, etc., which are represented as graphi
`cal user interface (GUI) elements within the DDI. The DDI
`controller then generates, on its display, a user interface
`based on these control object descriptions, and when the user
`manipulates this user interface, the controller Sends Special
`user events to the DDI target device. The information that is
`capable of being displayed by the DDI controller can be
`made scaleable by the DDI target device so that the human
`inter