`
`
`
`
`
`
`
`
`
`I IIIII IIIIIIII Ill lllll lllll lllll lllll lllll lllll lllll lllll 111111111111111111
`
`
`
`
`
`
`
`
`
`
`
`US007158ll8B2
`
`02)United States Patent
`
`
`Liberty
`
`(10)Patent No.:US 7,158,118 B2
`
`(45) Date of Patent:
`*Jan.2,2007
`
`(54)30 POINTING DEVICES WITH
`
`IMPROVED USABILITY
`
`ORIENTATION COMPENSATION AND
`
`(75)Inventor:
`
`(US)
`
`
`
`611989 LaBicbe et al.
`
`4,839,838
`A
`
`5,045,843 A
`9/ 1991 Hansen
`8/1992 Hotelling
`5,138,154 A
`
`1/1993 Glynn ........................ 702/141
`
`5,181,181 A *
`5,359,348
`
`1011994 Pilcher et al.
`A
`
`5,440,326 A
`8/1995 Quinn
`4/1996 Paley ......................... 345/163
`
`
`
`5,506,605 A *
`
`5,698,784 A
`
`
`1211997 Hotelling et al.
`
`
`Hillcrest Laboratories, Inc., Rockville,(73)Assignee:
`
`12/1997 Hall et aJ.
`5,703,623
`A
`MD (US)
`I0/ 1998 Case, Jr. et al.
`5,825,350 A
`
`
`11/1998 Blonstein et al.
`5,835,156
`A
`4/ 1999 Quinn
`
`5,898,421 A
`
`5,912,612 A
`6/1999 DeVolpi
`
`9/ 1999 Blonstein et al.
`
`5,955,988 A
`
`
`12/1999 Schein et al.
`
`6,002,394 A
`
`1/2000 Blonstein et al.
`
`6,016,144 A
`4/2000 Hwang
`
`6,049,823 A
`
`9/2000 Balakrishnan et al. .. .. .. 34 5/15 7
`
`6,115,028 A •
`
`
`
`12/2000 Shibata et al. ................ 700/85
`
`6,164,808 A *
`
`
`
`Matthew G. L iberty, Potomac, MD
`
`( *) Notice: Subject
`to any disclaimer, the tenn of this
`
`
`
`
`patent is extended or adjusted under 35
`
`U.S.C. 154(b) by O days.
`
`This patent is subject to a terminal dis­
`
`
`
`
`claimer.
`
`(21)Appl. No.: 11/119,719
`
`
`
`(22)Filed: May 2, 2005
`
`(Continued)
`
`(65)
`
`
`OTHER PUBLICATIONS
`
`
`Prior Publication Data
`Navarrete, P., et al., "Eigensapce-based Recognition of Faces:
`
`
`
`US 2005/0243062 Al Nov. 3, 2005
`
`
`
`Comparisons and a new Approach," Image Analysis and Processing,
`2001, pp. 1-6.
`
`
`
`Related U.S. Application Data
`
`
`
`
`on Jan.filed (60)Provisional application No. 60/641,410,
`
`
`
`5, 2005, provisional application No. 60/566,444, filed
`Primary Examiner-Vijay Shankar
`
`
`
`
`on Apr. 30, 2004, provisional application No. 60/612,
`
`(74)Attorney, Agent, or Firm-Potomac Patent Group
`571, filed on Sep. 23, 2004.
`PLLC
`
`(Continued)
`
`(57)
`
`ABSTRACT
`
`(51)Int. Cl.
`G09G 5/00 (2006.01)
`G09G 5/08 (2006.01)
`Systems and methods according to the present invention
`
`
`
`
`
`345/158; (52)U.S. Cl ....................... 345/156; 345/157;
`
`
`
`
`
`
`
`describe 3D pointing devices which enhance usability by
`345/163
`
`transfonning sensed motion data from a first frame of
`
`(58)Field of Classification Search ........ 345/156-169,
`
`
`
`
`
`reference (e.g., the body of the 3D pointing device) into a
`
`
`345/173-179; 178/18.01-18.07, 19.01-19.06
`
`
`
`second frame of reference (e.g., a user's frame of reference).
`
`
`See application file for complete search history.
`
`
`
`One exemplary embodiment of the present invention
`
`
`
`
`removes effects associated with a tilt orientation in which the
`
`
`3D pointing device is held by a user.
`
`(56)
`
`
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`
`
`4,787,051 A 11/1988 Olson
`
`
`
`17 Claims, 9 Drawing Sbeets
`
`001
`
`
`
` LG 1006
`
`

`

`
`
`US 7,158,118 B2
`
` Page 2
`
`U.S. PATENT DOCUMENTS
`
`
`
`
`
`2006/0092133 Al*
`
`
`
`
`5/2006 Touma etal... 345/158
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`6,466,831 B1* 10/2002 Shibata et al. .....ccce 700/85
`
`
`
`
`
`6,492,981 Bl
`12/2002 Stork et al.
`
`
`
`
`
`6,753,849 Bl
`6/2004 Curran etal.
`
`
`
`
`
`6,757,446 BI1*
`6/2004 Lietal. we 382/293
`
`
`
`
`
`6,933,923 Bl
`8/2005 Feinstein
`
`
`
`
`6,990,639 Bl
`1/2006 Wilson
`
`
`
`
`6,998,966 BI
`2/2006 Pedersenetal.
`
`
`
`
`
`2003/0107551 Al
`6/2003 Dunker
`
`
`
`
`2004/0095317 Al
`5/2004 Zhanget al.
`
`
`
`
`2004/0239626 Al
`12/2004 Noguera
`
`
`
`
`
`2004/0268393 Al
`12/2004 Hunleth et al.
`
`
`
`
`
`2005/0174324 Al
`8/2005. Liberty etal.
`
`
`
`
`
`
`2005/0212767 Al
`9/2005 Marvit et al.
`
`
`
`
`
`2005/0243061 Al
`11/2005 Liberty et al.
`
`
`
`
`
`
`2005/0253806 Al
`11/2005 Liberty et al.
`
`
`
`
`
`2006/0028446 Al
`2/2006 Liberty etal.
`
`
`
`
`
`
`
`
`
`
`
`OTHER PUBLICATIONS
`
`
`
`
`
`
`
`
`Jakubowski, J., et al., “Higher Order Statistics and Neural Network
`
`
`
`
`
`
`
`for Tremor Recognition,” IEEE Transactions on Biomedical Engi-
`
`
`
`
`
`
`
`
`neering, vol. 49, No. 2, Feb. 2002, pp. 152-159.
`
`
`
`
`
`
`
`
`
`Liu, C., et al., “Enhanced Fisher Linear Discriminant Models for
`
`
`
`
`
`
`Face Recognition,” Proc. 14 International Conference on Pattern
`
`
`
`
`
`
`
`
`Recognition, Queensland, Australia, Aug. 17-20, 1998, pp. 1-5.
`
`
`
`
`
`
`
`International Search Report for PCT/US05/15096, mailed May 16,
`
`2006.
`
`
`
`
`
`
`
`Written Opinion for PCT/US05/15096, mailed May 15, 2006.
`
`
`
`
`
`
`
`International Search Report for PCT/US04/35369, mailed May11,
`
`2006.
`
`
`
`
`
`
`
`Written Opinion for PCT/US04/35369, mailed May 11, 2006.
`
`
`
`
`
`
`Green, J., et al., “New iMEMS Angular-Rate-Sensing Gyroscope,”
`
`
`
`
`
`
`Analog Dialogue, 37-03 (2003), pp. 1-4.
`
`
`
`* cited by examiner
`
`
`
`
`
`
`002
`
`002
`
`

`

`
`U.S. Patent
`
`
`
`
`Jan. 2, 2007
`
`
`
`
`
`Sheet 1 of 9
`
`
`
`US 7,158,118 B2
`
`
`FIG.
`7
`
`
`(Prior Art)
`
`
`
`
`
`
`
`
`
`OOO
`
`——TUNCTION ———
`
`
`
`_IV___—DSS
`CABLE
`(CD)©C)
`
`
`
`CODE SET ee
`
`
`
`
`
`TV/VIDEO TV/DSS
`JUMP
`
`
`
`
`003
`
`003
`
`

`

`U.S. Patent
`
`Jan. 2, 2007
`
`Sheet 2 of 9
`
`US 7,158,118 B2
`
`
`
`auoydsjaifa/qD4))
`
`
`
`ynduyWwiapopy
`
`¢Ola
`
`ynduy
`
`vee[REaz
`
`SNGO/I
`
`004
`
`
`
`capi,|/AL
`
`
`
`DJBWUD)4OJIUOW
`
`01g
`
`4X4CLEOf
`
`812912blZ820
`
`
`
`oapl\/olpnyO2piAsa}SKS
`
`
`4ausn|ayasson4JaJ0.NUOD
`
`
`
`
`
`jnduyjnduyjnduy
`
`
`
`SHVIMA23/118}}S02p001g
`
`3/90)
`
`SS
`
`004
`
`
`

`

`
`U.S. Patent
`
`
`
`
`Jan. 2, 2007
`
`
`
`
`Sheet 3 of 9
`
`
`
`US 7,158,118 B2
`
`
`
`408
`
`410
`
`005
`
`005
`
`

`

`U.S. Patent
`
`Jan. 2, 2007
`
`Sheet 4 of 9
`
`US 7,158,118 B2
`
`Buljo2S
`
`92DlL
`
`819
`
`619
`
`[ues
`JOSUSS
`
`GOld
`
`feat}
`rae1=|
`‘29Ru0/}28}9qmAIDUO!]OIS
`
`
`
`ulog
`
`909
`
`006
`
`XLOW
`
`UO!}O1QIIOD
`
`C09
`
`v09
`
`UOISJBAUOD
`
`n
`
`NGIND‘Ip
`
`pajdwos
`
`006
`
`
`
`
`
`
`
`
`

`

`
`U.S. Patent
`
`
`
`
`Jan. 2, 2007
`
`
`
`
`Sheet 5 of 9
`
`
`
`US 7,158,118 B2
`
`
`
`
`FIG. 6B
`
`
`
`
`
`CE 410
`
`408
`
`
`
`007
`
`007
`
`

`

`
`U.S. Patent
`
`
`
`
`Jan. 2, 2007
`
`
`
`
`Sheet 6 of 9
`
`
`
`US 7,158,118 B2
`
`
`FIG. 6C
`
`
`
`400 —*
`
`((/
`
`»
`
`(
`
`
`FIG. 6D
`
`
`
`
`
`410
`
`408
`
`
`
`
`
`
`
`008
`
`008
`
`

`

`
`U.S. Patent
`
`
`
`
`Jan. 2, 2007
`
`
`
`
`Sheet 7 of 9
`
`
`
`US 7,158,118 B2
`
`
`FIG. 7
`
`302
`
`
`
`804
`
`
`
`806
`
`
`
`
`68&7
`
`
`
`
`
`
`
`800
`
`
`
`
`
`Scroll Wheel
`
`
`
`
`
`
`
`
`
`
`
`Accelerometer
`
`812
`
`
`
`
`
`
`
`
`
`
`
`Processor
`Rotational
`
`
`Sensors
`
`
` 810
`
`
`
` 814
`
`
`
`
`
`
`IR
`Photodetector
`
`
`Switch Matrix
`
`
`
`808
`
`
`
`009
`
`009
`
`

`

`
`U.S. Patent
`
`
`
`
`Jan. 2, 2007
`
`
`
`
`Sheet 8 of 9
`
`
`
`US 7,158,118 B2
`
`
`
`
`
`
`
` force sleep
`
`
`FIG. 8
`
`010
`
`010
`
`

`

`
`U.S. Patent
`
`
`
`
`Jan. 2, 2007
`
`
`
`
`Sheet 9 of 9
`
`
`
`US 7,158,118 B2
`
`
`FIG. 9
`
`
`
`
`903
`
`
`
`Sensors
`
`
`
`
`Interpret
`
`Sensors
`
`Convert
`to User
`Frame
`
`
`Map
`Movement
`
`
`
`
`Produce
`
`Action
`
`
`FIG. 10
`
`
`
`—Gravity—>
`
`
`Body Frame
`
`
`
`ev
`
`
`User Frame
`
`
`
`011
`
`011
`
`

`

`RELATED APPLICATIONS
`
`
`
`
`BACKGROUND
`
`
`
`
`
`US 7,158,118 B2
`
`
`1
`3D POINTING DEVICES WITH
`
`
`
`
`ORIENTATION COMPENSATION AND
`
`
`IMPROVED USABILITY
`
`
`
`
`
`
`
`
`
`
`
`
`
`This application is related to, and claimspriority from,
`
`
`
`
`
`
`
`U.S. Provisional Patent Application Ser. No. 60/566,444
`
`
`
`
`
`
`
`
`filed on Apr. 30, 2004, entitled “Freespace Pointing Device”,
`
`
`
`
`
`
`the disclosure of which is incorporated here by reference.
`
`
`
`
`
`
`
`
`
`This application is also related to, and claimspriority from,
`
`
`
`
`
`
`
`U.S. Provisional Patent Application Ser. No. 60/612,571,
`
`
`
`
`
`
`
`
`
`filed on Sep. 23, 2004, entitled “Free Space Pointing
`
`
`
`
`
`
`
`Devices and Methods”, the disclosure of which is incorpo-
`
`
`
`
`
`
`
`
`rated here by reference. This application is also related to,
`
`
`
`
`
`
`
`
`and claims priority from, U.S. Provisional Patent Applica-
`
`
`
`
`
`
`
`
`
`
`tion Ser. No. 60/641,410, filed on Jan. 5, 2005, entitled
`
`
`
`
`
`
`
`“Freespace Pointing Devices and Methods for Using Same”,
`
`
`
`
`
`
`the disclosure of which is incorporated here by reference.
`
`
`
`
`
`
`
`This application is also related to U.S. patent applications
`
`
`
`
`
`
`
`Ser. Nos. 11/119,987, 11/119,688, and 11/119,663, entitled
`
`
`
`
`
`
`
`“Methods and Devices for Removing Unintentional Move-
`
`
`
`
`
`
`
`
`ment in 3D Pointing Devices”, “Methods and Devices for
`
`
`
`
`
`
`
`
`Identifying Users Based on Tremor”, and “3D Pointing
`
`
`
`
`
`
`
`Devices and Methods”, all of which werefiled concurrently
`
`
`
`
`
`
`
`
`
`here with and all of which are incorporated here by refer-
`ence.
`
`
`
`2
`
`
`
`
`
`
`
`
`
`being transmitted on those channels and (3) date and time.
`
`
`
`
`
`
`
`
`The television was tuned to the desired channel by adjusting
`
`
`
`
`
`
`
`
`
`
`a tuner knob and the viewer watched the selected program.
`
`
`
`
`
`
`
`
`Later, remote control devices were introducedthat permitted
`viewers to tune the television from a distance. This addition
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`to the user-television interface created the phenomenon
`
`
`
`
`
`
`
`knownas “channel surfing” whereby a viewer could rapidly
`
`
`
`
`
`
`
`view short segments being broadcast on a number of chan-
`
`
`
`
`
`
`
`
`nels to quickly learn what programs were available at any
`
`
`given time.
`
`
`
`
`
`
`
`
`
`Despite the fact that the number of channels and amount
`
`
`
`
`
`
`
`
`of viewable content has dramatically increased, the gener-
`
`
`
`
`
`
`
`
`ally available user interface, control device options and
`
`
`
`
`
`
`
`
`
`frameworks for televisions has not changed much over the
`
`
`
`
`
`
`
`
`
`
`last 30 years. Printed guides are still the most prevalent
`
`
`
`
`
`
`mechanism for conveying programming information. The
`
`
`
`
`
`
`
`
`multiple button remote control with up and downarrowsis
`
`
`
`
`
`
`
`still the most prevalent channel/content selection mecha-
`
`
`
`
`
`
`
`
`
`nism. The reaction of those who design and implement the
`TVuser interface to the increase in available media content
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`has been a straightforward extension of the existing selec-
`
`
`
`
`
`
`
`
`tion procedures and interface objects. Thus, the number of
`
`
`
`
`
`
`
`
`rowsin the printed guides has been increased to accommo-
`date more channels. The number of buttons on the remote
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`control devices has been increased to support additional
`
`
`
`
`
`
`
`functionality and content handling, e.g., as shown in FIG. 1.
`
`
`
`
`
`
`
`
`However, this approach has significantly increased both the
`
`
`
`
`
`
`
`
`time required for a viewer to review the available informa-
`
`
`
`
`
`
`
`tion and the complexity of actions required to implementa
`
`
`
`
`
`
`
`selection. Arguably, the cumbersomenature of the existing
`
`
`
`
`
`
`
`
`
`
`
`
`
`invention relates generally to handheld,
`The present
`interface has hampered commercial implementation of some
`
`
`
`
`
`
`
`
`
`
`
`
`
`pointing devices and, more specifically to three-dimensional
`services, e.g., video-on-demand, since consumersare resis-
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`(hereinafter “3D”) pointing devices and techniques for tilt
`tant to new services that will add complexity to an interface
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`compensation and improved usability associated therewith.
`that they view as already too slow and complex.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`In addition to increases in bandwidth and content, the user
`Technologies associated with the communication of infor-
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`mation have evolved rapidly over the last several decades.
`interface bottleneck problem is being exacerbated by the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Television, cellular telephony, the Internet and optical com-
`aggregation of technologies. Consumers are reacting posi-
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`munication techniques (to name just a few things) combine
`tively to having the option of buying integrated systems
`
`
`
`
`
`
`to inundate consumers with available information and enter-
`
`
`
`
`
`
`
`
`rather than a numberof segregable components. An example
`
`
`
`
`
`
`
`of this trend is the combination television/VCR/DVD in
`
`
`
`
`
`
`tainment options. Taking television as an example, the last
`
`
`
`
`
`
`
`three decades have seen the introduction of cable television
`
`
`
`
`
`
`
`
`which three previously independent components are fre-
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`service, satellite television service, pay-per-view movies and
`quently sold today as an integrated unit. This trend is likely
`
`
`
`
`
`
`
`
`
`video-on-demand. Whereastelevision viewers of the 1 960s
`
`
`
`
`
`
`to continue, potentially with an endresult that most if notall
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`could typically receive perhaps four or five over-the-air TV
`of the communication devices currently found in the house-
`45
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`channels on their television sets, today’s TV watchers have
`hold will be packaged together as an integrated unit, e.g., a
`
`
`
`
`
`
`
`
`
`television/VCR/DVD/internet access/radio/stereo—_unit.
`
`
`
`the opportunity to select from hundreds,
`thousands, and
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`potentially millions of channels of shows and information.
`Even those who continue to buy separate components will
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`Video-on-demand technology, currently used primarily in
`likely desire seamless control of, and interworking between,
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`hotels and the like, provides the potential for in-home
`the separate components. With this increased aggregation
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`entertainment selection from among thousands of movie
`comes the potential for more complexity in the user inter-
`titles.
`
`
`
`
`
`
`
`
`
`face. For example, when so-called “universal” remote units
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`The technological ability to provide so much information
`were introduced, e.g., to combine the functionality of TV
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`remote units and VCR remote units, the number of buttons
`and content to end users provides both opportunities and
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`challenges to system designers and service providers. One
`on these universal remote units was typically more than the
`
`
`
`
`
`
`
`
`number of buttons on either the TV remote unit or VCR
`
`
`
`
`
`
`
`
`
`challenge is that while end users typically prefer having
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`more choices rather than fewer, this preference is counter-
`remote unit individually. This added number of buttons and
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`weighted by their desire that the selection process be both
`functionality makes it very difficult to control anything but
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`fast and simple. Unfortunately,
`the development of the
`the simplest aspects of a TV or VCR without hunting for
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`systems and interfaces by which end users access media
`exactly the right button on the remote. Manytimes, these
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`items has resulted in selection processes which are neither
`universal remotes do not provide enough buttons to access
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`fast nor simple. Consider again the example of television
`many levels of control or features unique to certain TVs. In
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`programs. Whentelevision was in its infancy, determining
`these cases, the original device remote unit is still needed,
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`which program to watch wasa relatively simple process
`and the original hassle of handling multiple remotes remains
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`primarily due to the small number of choices. One would
`due to user interface issues arising from the complexity of
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`consult a printed guide which was formatted, for example, as
`aggregation. Some remote units have addressed this problem
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`series of columns and rows which showed the correspon-
`by adding “soft” buttons that can be programmed with the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`dence between (1) nearby television channels, (2) programs
`expert commands. These soft buttons sometimes have
`012
`
`
`
`
`
`
`
`20
`
`25
`
`30
`
`
`
`35
`
`
`
`40
`
`
`
`50
`
`
`
`55
`
`
`
`60
`
`
`
`65
`
`
`
`012
`
`

`

`
`3
`
`
`
`
`
`
`
`accompanying LCD displays to indicate their action. These
`
`
`
`
`
`
`
`
`
`
`
`too have the flaw that they are difficult
`to use without
`
`
`
`
`
`
`
`
`
`looking away from the TV to the remote control. Yet another
`
`
`
`
`
`
`
`
`flaw in these remote units is the use of modes in an attempt
`to reduce the numberof buttons. In these “moded”universal
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`remote units, a special button exists to select whether the
`
`
`
`
`
`
`
`
`
`remote should communicate with the TV, DVD player, cable
`
`
`
`
`
`
`
`
`
`set-top box, VCR, etc. This causes many usability issues
`
`
`
`
`
`
`
`including sending commands to the wrong device, forcing
`the user to look at the remote to make sure that it is in the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`right mode, andit does not provide any simplification to the
`
`
`
`
`
`
`
`integration of multiple devices. The most advancedof these
`
`
`
`
`
`
`
`universal remote units provide someintegration by allowing
`
`
`
`
`
`
`
`the user to program sequences of commands to multiple
`devices into the remote. This is such a difficult task that
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`many users hire professional
`installers to program their
`universal remote units.
`
`
`
`
`
`
`
`
`
`
`
`Some attempts have also been made to modernize the
`
`
`
`
`
`
`
`
`screen interface between end users and media systems.
`
`
`
`
`
`
`
`
`However, these attempts typically suffer from, among other
`
`
`
`
`
`
`
`
`drawbacks, an inability to easily scale between large col-
`lections of media items and small collections of media items.
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`For example, interfaces which rely on lists of items may
`
`
`
`
`
`
`
`
`
`
`work well for small collections of media items, but are
`
`
`
`
`
`
`
`
`
`tedious to browse for large collections of media items.
`
`
`
`
`
`
`
`
`Interfaces which rely on hierarchical navigation (e.g., tree
`
`
`
`
`
`
`
`
`structures) may be speedierto traverse than list interfaces for
`
`
`
`
`
`
`
`
`
`large collections of media items, but are not readily adapt-
`
`
`
`
`
`
`
`able to small collections of media items. Additionally, users
`
`
`
`
`
`
`
`
`tend to lose interest in selection processes wherein the user
`
`
`
`
`
`
`
`
`has to move through three or more layers in a tree structure.
`
`
`
`
`
`
`
`
`
`
`For all of these cases, current remote units make this
`
`
`
`
`
`
`
`
`selection processor even more tedious by forcing the user to
`
`
`
`
`
`
`
`
`repeatedly depress the up and down buttons to navigate the
`
`
`
`
`
`
`
`
`list or hierarchies. When selection skipping controls are
`
`
`
`
`
`
`
`
`
`available such as page up and page down,the user usually
`
`
`
`
`
`
`
`
`has to look at the remote to find these special buttons or be
`
`
`
`
`
`
`
`
`trained to know that they even exist. Accordingly, organiz-
`
`
`
`
`
`
`
`
`ing frameworks, techniques and systems which simplify the
`
`
`
`
`
`
`
`
`
`control and screen interface between users and media sys-
`
`
`
`
`
`
`
`
`tems as well as accelerate the selection process, while at the
`
`
`
`
`
`
`
`same time permitting service providers to take advantage of
`
`
`
`
`
`
`
`the increases in available bandwidth to end user equipment
`
`
`
`
`
`
`
`
`by facilitating the supply of a large number of media items
`
`
`
`
`
`
`
`
`
`and new services to the user have been proposed in U.S.
`
`
`
`
`
`
`
`
`
`patent application Ser. No. 10/768,432, filed on Jan. 30,
`
`
`
`
`
`
`
`2004, entitled “A Control Framework with a Zoomable
`
`
`
`
`
`
`
`Graphical User Interface for Organizing, Selecting and
`
`
`
`
`
`
`
`Launching Media Items”, the disclosure of which is incor-
`
`
`
`porated here by reference.
`
`
`
`
`
`
`
`
`
`Ofparticular interest for this specification are the remote
`
`
`
`
`
`
`
`devices usable to interact with such frameworks, as well as
`
`
`
`
`
`
`
`other applications and systems. As mentioned in the above-
`
`
`
`
`
`
`incorporated application, various different types of remote
`
`
`
`
`
`
`
`
`devices can be used with such frameworks including, for
`
`
`
`
`
`
`example, trackballs, “mouse”-type pointing devices,
`light
`
`
`
`
`
`
`
`
`pens, etc. However, another category of remote devices
`
`
`
`
`
`
`
`
`
`which can be used with such frameworks (and other appli-
`
`
`
`
`
`
`
`
`cations) is 3D pointing devices. The phrase “3D pointing”is
`
`
`
`
`
`
`
`
`used in this specification to refer to the ability of an input
`
`
`
`
`
`
`
`
`device to move in three (or more) dimensionsin the air in
`
`
`
`
`
`
`
`
`front of, e.g., a display screen, and the corresponding ability
`
`
`
`
`
`
`
`
`of the user interface to translate those motions directly into
`
`
`
`
`
`
`
`user interface commands, e.g., movementof a cursor on the
`
`
`
`
`
`
`
`
`display screen. Thetransfer of data between the 3D pointing
`
`
`
`
`
`
`
`device may be performed wirelessly or via a wire connecting
`
`
`
`
`
`
`
`
`the 3D pointing device to another device. Thus “3D point-
`013
`
`40
`
`
`4
`
`
`
`
`
`
`
`
`ing”differs from, e.g., conventional computer mouse point-
`
`
`
`
`
`
`
`
`ing techniques which use a surface, e.g., a desk surface or
`
`
`
`
`
`
`
`mousepad, as a proxy surface from which relative move-
`mentof the mouseis translated into cursor movement on the
`
`
`
`
`
`
`
`
`
`
`
`
`
`computer display screen. An example of a 3D pointing
`
`
`
`
`
`
`
`device can be found in U.S. Pat. No. 5,440,326.
`
`
`
`
`
`
`
`
`The 326 patent describes, among other things, a vertical
`
`
`
`
`
`
`
`
`gyroscope adapted for use as a pointing device for control-
`
`
`
`
`
`
`
`ling the position of a cursor on the display of a computer. A
`
`
`
`
`
`
`
`motorat the core of the gyroscope is suspended by twopairs
`
`
`
`
`
`
`
`of orthogonal gimbals from a hand-held controller device
`
`
`
`
`
`
`
`
`
`and nominally oriented with its spin axis vertical by a
`
`
`
`
`
`
`
`pendulous device. Electro-optical shaft angle encoders sense
`the orientation of a hand-held controller device as it is
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`manipulated by a user and the resulting electrical output is
`
`
`
`
`
`
`
`converted into a format usable by a computer to control the
`
`
`
`
`
`
`
`movementof a cursor on the screen of the computer display.
`
`
`
`
`
`
`However, the freedom of use associated with 3D pointers
`
`
`
`
`
`
`
`creates additional challenges. For example, since there is
`
`
`
`
`
`
`
`generally no proxy surface on which a 3D pointing device
`
`
`
`
`
`
`
`
`
`rests, the orientation of the handheld control device may
`
`
`
`
`
`
`
`
`vary considerably from user to user or even use to use. Ifa
`
`
`
`
`
`
`
`
`
`3D pointing device is used to, for example, control the
`
`
`
`
`
`
`
`movement of a cursor displayed on a screen, then some
`
`
`
`
`
`
`mapping is performed between the detected movement of
`the handheld device and the movementof the cursor on the
`
`
`
`
`
`
`
`
`screen.
`
`
`
`
`
`
`
`
`
`
`One technique for performing this mapping is to use the
`
`
`
`
`
`
`
`
`
`
`body frame of the device as the frame of reference for
`
`
`
`
`
`
`
`
`
`mapping detected motion of the 3D pointing device into
`
`
`
`
`
`
`
`
`intended motion of the cursor. The term “body frame”refers
`
`
`
`
`
`
`
`
`to a set of axes associated with the body ofthe object being
`
`
`
`
`
`
`
`
`moved as described in more detail below. Using the body
`
`
`
`
`
`
`
`frame of reference to perform the mapping, however, has
`
`
`
`
`
`
`
`
`certain drawbacks. For example, it requires the user to hold
`the device in a certain orientation in order to obtain the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`cursor movementhe or she desires. For example, if the user
`holds the device on its side and moves the device left to
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`right, the cursor will movevertically, not horizontally, on the
`screen.
`
`
`
`
`
`
`
`
`Accordingly, the present invention describes methods and
`
`
`
`
`
`
`
`
`devices for processing the data received from sensor(s) in a
`
`
`
`
`
`
`
`
`manner which addresses these and other problems associ-
`
`
`
`
`
`ated with conventional 3D pointing devices.
`SUMMARY
`
`
`
`
`
`
`
`
`
`
`Systems and methods according to the present invention
`
`
`
`
`
`
`
`describe 3D pointing devices which enhance usability by
`
`
`
`
`
`
`
`
`transforming sensed motion data from a first frame of
`
`
`
`
`
`
`
`reference (e.g., the body of the 3D pointing device) into a
`
`
`
`
`
`
`
`second frameof reference (e.g., a user’s frame of reference).
`
`
`
`
`
`
`
`One exemplary embodiment of the present
`invention
`removeseffects associated with a tilt orientation in which the
`
`
`
`
`
`
`
`
`
`
`
`
`3D pointing device is held by a user.
`
`
`
`
`
`
`According to an exemplary embodiment of the present
`
`
`
`
`
`
`
`invention, a handheld, pointing device includesa first rota-
`
`
`
`
`
`
`
`tional sensor for determining rotation of the pointing device
`
`
`
`
`
`
`
`
`about a first axis and generating a first rotational output
`
`
`
`
`
`
`
`associated therewith, a second rotational sensor for deter-
`
`
`
`
`
`
`
`
`mining rotation of the pointing device about a second axis
`
`
`
`
`
`
`
`and generating a second rotational output associated there-
`
`
`
`
`
`with, an accelerometer for determining an acceleration of the
`
`
`
`
`
`
`
`pointing device and outputting an acceleration output asso-
`
`
`
`
`
`
`
`
`
`ciated therewith and a processing unit for receiving thefirst
`
`
`
`
`
`
`
`
`and second rotational outputs and the acceleration output
`
`
`
`
`
`
`
`
`
`
`and for: (a) converting the first and secondrotational outputs
`
`
`
`US 7,158,118 B2
`
`
`
`
`
`20
`
`
`
`25
`
`
`
`30
`
`
`
`35
`
`
`
`
`
`45
`
`
`
`50
`
`
`
`55
`
`
`
`60
`
`
`
`65
`
`
`
`013
`
`

`

`
`
`US 7,158,118 B2
`
`
`5
`
`
`
`
`
`
`
`
`and the acceleration output from a body frame of reference
`
`
`
`
`
`
`
`
`associated with the handheld pointing device into a user’s
`frame of reference in order to remove the effects oftilt
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`associated with the manner in which a user is holding the
`
`
`
`
`
`
`
`handheld, pointing device; and (b) determining data associ-
`
`
`
`
`
`
`
`
`ated with x and y coordinates which are in turn associated
`
`
`
`
`
`
`
`
`with movement of a screen cursor, the data based on the
`
`
`
`
`
`
`
`
`
`converted first and second rotational outputs and the con-
`
`
`
`
`
`
`
`
`verted acceleration output, wherein the step of converting
`
`
`
`
`
`
`
`renders the movement of the screen cursor substantially
`
`
`
`
`
`
`
`
`independent of an orientation in which a user holds the
`handheld device.
`
`
`
`
`
`
`
`
`According to another exemplary embodiment of the
`
`
`
`
`
`
`
`present invention, a method for using a 3D pointing device
`
`
`
`
`
`
`includesthe steps of detecting movementof the 3D pointing
`
`
`
`
`
`
`
`device and compensating the detected movementbytrans-
`
`
`
`
`
`
`
`
`forming the detected movement from a body frame of
`
`
`
`
`
`
`
`
`reference associated with the 3D pointing device into an
`inertial frame of reference.
`
`
`
`
`
`
`
`
`
`According to yet another exemplary embodiment of the
`
`
`
`
`
`
`
`present invention, a 3D, handheld device includes at least
`
`
`
`
`
`
`
`
`one sensor for detecting movement of the 3D pointing
`
`
`
`
`
`
`
`
`device and a processing unit for compensating the detected
`
`
`
`
`
`
`movementby transforming the detected movement from a
`
`
`
`
`
`
`
`
`body frame of reference associated with the 3D pointing
`device into an inertial frame of reference.
`
`
`
`
`
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`
`
`
`
`
`
`
`
`
`
`
`20
`
`25
`
`
`
`30
`
`
`
`35
`
`
`
`
`
`
`
`
`
`
`40
`
`
`
`45
`
`
`
`
`
`
`6
`
`
`
`
`
`
`
`
`
`Also, the following detailed description does not limit the
`
`
`
`
`
`
`invention. Instead, the scope of the invention is defined by
`
`
`
`the appendedclaims.
`
`
`
`
`
`
`
`
`In order to provide some context for this discussion, an
`
`
`
`
`
`
`
`exemplary aggregated media system 200 in which the
`
`
`
`
`
`
`
`present invention can be implementedwill first be described
`
`
`
`
`
`
`
`
`
`with respect to FIG. 2. Those skilled in the art will appre-
`
`
`
`
`
`
`
`
`ciate, however, that the present invention is notrestricted to
`
`
`
`
`
`
`
`
`implementation in this type of media system and that more
`
`
`
`
`
`
`
`or fewer components can be included therein. Therein, an
`
`
`
`
`
`
`
`
`input/output (1/O) bus 210 connects the system components
`
`
`
`
`
`
`
`
`
`
`in the media system 200 together. The I/O bus 210 represents
`
`
`
`
`
`
`any of a numberofdifferent of mechanisms and techniques
`
`
`
`
`
`
`
`
`for routing signals between the media system components.
`
`
`
`
`
`
`
`
`
`For example, the I/O bus 210 may include an appropriate
`
`
`
`
`
`
`
`
`numberof independentaudio “patch”cables that route audio
`
`
`
`
`
`
`
`
`signals, coaxial cables that route video signals, two-wire
`
`
`
`
`
`
`
`
`serial lines or infrared or radio frequency transceivers that
`
`
`
`
`
`
`
`
`
`route control signals, optical fiber or any other routing
`
`
`
`
`
`
`mechanisms that route other types of signals.
`
`
`
`
`
`
`
`
`In this exemplary embodiment, the media system 200
`
`
`
`
`
`
`includes a television/monitor 212, a video cassette recorder
`
`
`
`
`
`
`
`(VCR) 214, digital video disk (DVD) recorder/playback
`
`
`
`
`
`
`
`
`
`device 216, audio/video tuner 218 and compact disk player
`
`
`
`
`
`
`
`
`
`
`220 coupledto the I/O bus 210. The VCR 214, DVD 216 and
`
`
`
`
`
`
`
`
`
`compact disk player 220 may be single disk or single
`
`
`
`
`
`
`
`cassette devices, or alternatively may be multiple disk or
`
`
`
`
`
`
`
`multiple cassette devices. They may be independentunits or
`
`
`
`
`
`
`
`
`integrated together.
`In addition,
`the media system 200
`
`
`
`
`
`
`includes a microphone/speaker system 222, video camera
`
`
`
`
`
`
`
`
`224 and a wireless I/O control device 226. According to
`
`
`
`
`
`
`exemplary embodiments of the present invention, the wire-
`
`
`
`
`
`
`
`
`less I/O control device 226 is a 3D pointing device according
`
`
`
`
`
`
`
`to one of the exemplary embodiments described below. The
`wireless I/O control device 226 can communicate with the
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`entertainment system 200 using, e.g., an IR or RF transmit-
`
`
`
`
`
`
`
`
`ter or transceiver. Alternatively, the I/O control device can be
`
`
`
`
`
`
`
`connected to the entertainment system 200 via a wire.
`
`
`
`
`
`
`
`The entertainment system 200 also includes a system
`
`
`
`
`
`
`controller 228. According to one exemplary embodimentof
`
`
`
`
`
`
`
`
`the present invention, the system controller 228 operates to
`
`
`
`
`
`
`
`
`store and display entertainment system data available from
`
`
`
`
`
`
`
`
`a plurality of entertainment system data sources and to
`
`
`
`
`
`
`
`control a wide variety of features associated with each of the
`
`
`
`
`
`
`system components. As shown in FIG. 2, system controller
`
`
`
`
`
`
`
`228 is coupled, either directly or indirectly, to each of the
`
`
`
`
`
`
`
`system components, as necessary, through I/O bus 210. In
`
`
`
`
`
`
`one exemplary embodiment, in additionto or in place of I/O
`
`
`
`
`
`
`
`
`bus 210, system controller 228 is configured with a wireless
`
`
`
`
`
`
`communication transmitter
`(or
`transceiver), which is
`
`
`
`
`
`
`
`capable of communicating with the system components via
`
`
`
`
`
`
`IR signals or RF signals. Regardless of the control medium,
`
`
`
`
`
`
`
`
`the system controller 228 is configured to control the media
`
`
`
`
`
`
`
`
`components of the media system 200 via a graphical user
`interface described below.
`
`
`
`
`
`
`
`
`
`
`
`Asfurtherillustrated in FIG. 2, media system 200 may be
`
`
`
`
`
`
`
`
`configured to receive media items from various media
`
`
`
`
`
`
`
`
`sources and service providers. In this exemplary embodi-
`
`
`
`
`
`
`
`
`
`ment, media system 200 receives media input from and,
`
`
`
`
`
`
`optionally, sends information to, any orall of the following
`
`
`
`
`
`
`
`
`sources: cable broadcast 230, satellite broadcast 232 (e.g.,
`
`
`
`
`
`
`
`
`
`via a satellite dish), very high frequency (VHF)orultra high
`
`
`
`
`
`
`
`frequency (UHF) radio frequency communication of the
`
`
`
`
`
`
`
`
`broadcast
`television networks 234 (e.g., via an aerial
`
`
`
`
`
`
`
`
`
`antenna), telephone network 236 and cable modem 238 (or
`
`
`
`
`
`
`
`
`another source of Internet content). Those s