United States Patent [19]
`Gordon
`
`US
`[11]
`Patent Number:
`[45] Date of Patent:
`
`5,786,804
`Jul. 28, 1998
`
`[54] METHOD AND SYSTEM FOR TRACKING
`ATTITUDE
`
`3/1995 European Pat. Off. ..... .. 606K 11/18
`0645732A1
`WO95/l903l 7/1995 WIPO ............................ .. 6096 3/02
`
`_
`_
`[75] Inventor. Gary B. Gordon. Saratoga. Calif.
`[73] Assignee: Hewlett-Packard Company. Palo Alto.
`Ca|jf_
`
`Primary Examiner-Richard Hjerpe
`Asst-Slam Emminer_Ronald Lancau
`Attorney’ Agent’ or Fl
`Tummy Rex Crou
`[57]
`ABSTRACT
`
`[21] Appl' No" 540’355
`[22] Filcd;
`Oct 6, 1995
`
`6
`
`..................................................... ..
`Int.
`[52] US. Cl. . .......................................... .. 345/158; 345/145
`[58] Field of Search ................................... .. 345/145. 157.
`345/ 158. 160
`
`[56]
`
`References Cited
`
`us‘ PATENT DOCUMENTS
`345/153
`9/1991 Hansen
`5,045,843
`345/158
`5,363,120 11/1994 Drumm .
`364/559
`5,333,059
`2/1995 DeM?'nllion
`343/362
`$546,123 3/1996 Nakagahucl" -
`5,574,479 11/1996 Odell ..................................... .. 345/158
`FOREIGN PATENT DOCUMENTS
`
`A method and system for tracking attitude of a device
`includes ?xing a two-dimensional array of photosensors to
`the device and using the array to form a reference frame and
`a sample frame of images. The ?elds of view of the sample
`and refcrence frames
`0v¢rlap_ so that fhcrg are
`common imaged features from frame to frame. Sample
`frames are correlated with the reference name to detect
`differences in location of the common features. Based upon
`detection of correlations of features. an attitudinal signal
`indicative of pitch. yaw. and/or roll is generated In the
`preferred embodiment. the attitudinal signal is used to
`manipulate a screen cursor of a display system. such as a
`remote interactive video system (RIVS). However. attitudi
`nal tracking using the invention may be employed in other
`applications. Another aspect of the invention is that the
`two-dimensional array is con?gured to compensate for any
`curvilinear distortions introduced by a kins systcm for
`imaging the features within the ?eld of view of the array.
`
`0596594A1 5/1994 European Pat. Off, ..... .. 606K 11/18
`
`16 Claims, 8 Drawing Sheets
`
`[38
`
`EXPOSURE
`CWTROL
`[511s Mn. 5mS Max)
`
`32x32
`
`1
`/
`16
`
`5ms
`
`f4!)
`(42
`’
`
`52
`
`AX AY (HEXI
`[BQ.AX=*0 75PM’,
`AY=-31Prx)
`
`{-54
`
`r
`ACCUMULATOR
`A X=+15 2
`AY,_33
`
`r 56
`1
`MODULO 1
`H
`
`Sms
`
`36
`200Hz CLOCK H
`
`I
`
`56
`
`52
`
`AX=+15. AY=-3
`
`(L-BITS e0. PLUS SIGNSI
`
`5‘
`
`f 50
`'
`PULSE CODE
`MODULATOR
`
`IR
`IR
`SENSOR
`EMITTER
`N /\/
`N wink-[MODULATOR
`68B
`66
`
`TEN-BIT WORD. EVERY LUms (25 WORDS/SEC)
`
`70
`1
`CURSOF;
`CONTROL
`
`SCREEN
`(64x64
`CURSOR
`ADDRE SSES)
`
`Page 1 of 17
`
`HANDI QUILTER, ET AL.
`EXHIBIT 1016
`
`

`
`US. Patent
`
`Jul. 28, 1998
`
`Sheet 1 of 8
`
`5,786,804
`
`is...‘
`
`Page 2 of 17
`
`

`
`U.S. Patent
`
`Jul. 23, 1993
`
`Sheet 2 of 3
`
`5,786,804
`
`mm8
`
`.”5a_mmaEvaodNISN
`
`mmifi
`
`.ca;><xqBEmmmy
`
`
`
`2n=m_-£<E3:82mmtuxd_x_%.\.o:x<...§mE<dm%o»<mm~,Nmm_~n.wzmm
`moE52:8<
`
`mm9
`
`azommin..8mcméT2.m:"x<
`
`3
`
`mm
`
`mmamoaxm
`
`._om._.zoo
`
`.xo_>_mam£2mam.
`
`a__E.58
`
`
`
`momzmmmmzmczm
`
`mommao
`
`am.200
`
`mozgaoozmo
`
`),«.0258:@.'%88mm?
`
`«€39
`
`.mmm$%o<.9mommaoNLx
`
`we
`
`
`
`zmmmomamm\mEo;me33EmaQ5;:m_-zE
`
`Page 3 of 17
`
`Page 3 of 17
`
`
`
`
`

`
`US. Patent
`
`Jul. 28, 1998
`
`Sheet 3 of 8
`
`5,786,804
`
`5 mm Q.
`
`s\T mm
`
`1. mm
`
`$\,L ME
`
`1.\1 ME
`
`mm
`
`1. mm.
`
`1.\T ME
`
`mm
`
`mm
`
`mm
`
`mm
`
`1. ME
`
`1.\T E
`
`ME
`
`a\T NE
`
`Page 4 of 17
`
`

`
`US. Patent
`
`Jul. 28, 1998
`
`Sheet 4 of 8
`
`5,786,804
`
`
`
`$3,; S56E18
`
`no; 00
`
`com
`
`
`
`\\zoc<mm$o E828
`
`no... 00
`
`v at
`
`@
`
`w2<wE mum
`
`mm;
`
`Page 5 of 17
`
`

`
`US. Patent
`
`Jul. 28, 1998
`
`Sheet 5 of 8
`
`5,786,804
`
`m ZOEmOQ
`
`N ZOEmOQ
`
`wow
`
`wow
`
`o ZOEmOQ
`
`m zoEmoa
`
`m ZOEWOQ
`
`N zoEmoa
`
`m .mt
`
`Page 6 of 17
`
`

`
`U.S. Patent
`
`Jul. 23, 1993
`
`Sheet 6 of 8
`
`5,786,804
`
`m:m:>om
`
`
`
`omKmcmwmoo.2:..1.222.3mg
`:E<Qs//353;AE222._5%Em:N:508AN983PAm9.222
`OAII§Ezq:<2.
`
`HDO¢mm_o§
`E22Aas3%,AHlam/3
`
`
`
`wmr.\.m.3.22
`
`_Emmm
`
`
`‘cmAII522.mmw..m.._
`
`
`
`
`
`zmnmm§<aH._mweImamSn.z_zz..m:
`
`I_Em
`
`V_B.9:am.
`
`B.<E223mm._Em%4M522,.E.I.
`..,.~.5.5m.222.Tue?NW.5o.<<ZEmmEm
`
`Q:S.,N:
`
`
`
`m.m.Q\aam3,
`
`Page 7 of 17
`
`Page 7 of 17
`
`
`

`
`US. Patent
`
`Jul. 28, 1998
`
`Sheet 7 of 8
`
`5,786,804
`
`Fig. 7
`
`/222
`
`Fig. 8
`
`Page 8 of 17
`
`

`
`US. Patent
`
`Jul. 28, 1998
`
`Sheet 8 of 8
`
`5,786,804
`
`/224
`
`Fig.9
`
`228
`
`Illllllllll
`
`Fig. 11
`
`Page 9 of 17
`
`

`
`5.786.804
`
`1
`METHOD AND SYSTEM FOR TRACKING
`ATTITUDE
`
`TECHNICAL FIELD
`The invention relates generally to methods and systems
`for tracking attitude of a device and more particularly to
`tracking the attitude of a device in order to control a device
`or process. such as a cursor of a video display.
`
`2
`with a controller. These devices may measure the attitude.
`i.e. pitch. yaw. and possibly roll. of the controller. A ?rst
`category of such an approach employs light beams to
`measure attitude. PCI‘ International Publication Number
`W0 95/ 19031 describes a system for determining the point
`ing orientation of a remote unit relative to a ?xed base unit.
`The ?xed base unit includes one or more light sources for
`emitting a light beam. The emitted light is polarized in at
`least one predetermined orientation. The movable remote
`unit includes a photodetector for detecting the polarized
`emitted light. The attitude of the movable remote unit may
`be determined by measuring the intensity of received light
`from various directions.
`Another implementation of the emitted-light category of
`measuring attitude is one in which an infrared (IR) signal is
`beamed from the area of the video display. The IR signal is
`defocused and is imaged onto a quad photodiode array in the
`controller. The relative signal amplitudes from the four
`photodiodes may be used to determine the relative orienta
`tion of the controller to a line drawn from the display. One
`concern is that the system may undesirably ?ood the room
`with intense IR. rendering other nearby IR-coupled appli~
`ances (e. g.. a VCR controller) inoperative. A second concern
`is that the limited range of transmission of defocused 1R
`signals may render this system of measuring attitude unre
`liable when the controller is more than a relatively short
`distance from the video display.
`A second category of devices that measure attitude of the
`controller is one in which inertial navigation principles are
`employed. Gyroscopes or encoded gimballed masses estab~
`lish inertial frames in the controllers. against which attitude
`changes can be measured. The attitude information may then
`be transmitted to the video display via a radio-frequency link
`to a small dipole antenna a?ixed atop the video display.
`The third category is related to the ?rst category. A
`hand-held object that provides cursor control has a number
`of light sources mounted on one surface. A single elecnonic
`camera is directed to capture images of the light sources
`mounted on a hand-held object. Locations of the images of
`the light sources are detected in each camera image. and a
`computer is used to determine the attitude of the light
`emitting hand-held object. Such a device is described in U.S.
`Pat. No. 5.338.059 to DeMenphon.
`A closely related need exists in the ?eld of virtual reality.
`In games. simulations. and other visualization situations. it
`is often necessary to encode the attitude of a user's head. or
`other body part In many cases. systems for encoding head
`pitch and yaw may be applied to RIVS controllers. and vice
`versa. One known virtual reality system encodes pitch and
`yaw by means of instrumented compasses and gravimeters.
`While the known cursor control devices and attitude
`deterrnining systems operate adequately for their intended
`purposes, each is associated with a concern or a problem.
`Operation may be slow or tedious. or may require use of a
`speci?c operating surface. Devices and systems that include
`IR radiation may adversely atfect operation of other devices.
`Attitude-sensing devices that are based on gravity may have
`di?iculty in distinguishing tilting from transverse
`acceleration. thereby rendering control erratic. This last
`problem conceivably could be solved by gyro stabilization.
`but the cost and power consumption make this solution
`unattractive. Known systems that utilize light detection
`require adding a second contrivance at the display. again
`adding additional cost.
`What is needed is a method and a system for reliably
`tracking attitude of a device. What is further needed is such
`
`15
`
`25
`
`35
`
`BACKGROUND ARI‘
`There are applications in which video systems require that
`a person interact with information presented on a display
`screen. At times. the interaction is to occur while the person
`is situated at a distance from the display screen. As will be
`described more fully below. the interaction may be accom
`plished by remotely controlling a screen cursor in one of a
`variety of manners. The interactions may include selecting
`from a variety of choices presented as a screen menu. or
`“typing” text using an on-screen keyboard. Examples of
`remote interactive video systems (RIV S) include interactive
`television (ITV). TV-style Internet browsers. and
`conference-room video projectors.
`One key component of a RIVS is the “pointing” device for
`controlling the on- screen cursor. The pointing device ful?lls
`a function analogous to that which mice. trackballs. and
`graphic tablets perform for computers. However. the envi
`ronment for RIVS presents di?iculties that are typically not
`encountered in operation of a computer. For example. an
`operator of a RIVS is typically further away from the
`controlled device than is the operator of a computer. As
`another example. the operator of a RIVS is more likely to be
`in an unstructln'ed immediate environment. e.g.. an lTV
`operator seated across a living room from a television set. In
`many situations. the environment precludes use of conven
`tional computer pointing devices. such as mice. Moreover. a
`RIVS is rarely equipped with a keyboard. so that the
`pointing device may have to accommodate the extra burden
`of providing a text entry.
`There are a number of known pointing devices for a
`RIVS. Most of the known pointing devices implement some
`variation of a four-key cursor pad on a hand-held controller.
`The four-key cursor pad is manipulated to step the screen
`cursor up. down. left or right among various menu choices.
`Such interfaces emulate the computer keyboard cursor keys
`used with old-style textural interfaces. However. these inter
`faces are typically much slower and less intuitive to use than
`computer mice and other pointing devices developed for
`modern graphical software interfaces.
`In an effort to improve upon cursor control within the
`RIVS environment. more advanced computer pointing
`devices of mice and trackballs have been adapted. In one
`adaptation. a miniature trackball is mounted atop a
`controller. with the trackball being operated by the person’s
`thumb. The trackball controller is faster than the use of
`cursor keys and facilitates diagonal moves. Unfortunately.
`the trackball may require repeated strokes to accomplish
`large cm'sor movements and. in general. thumb control taxes
`the user’ 5 thumb dexterity. For example. it is di?icult to trace
`the cursor in a circle on the display screen.
`The use of a mouse for ITV cursor control has been
`demonstrated. The advantage of the mouse is that it provides
`excellent and intuitive cursor control The concern is that
`there may not be a suitable planar operating surface that is
`convenient to the operator.
`A further re?nement in the RIVS pointing art is the use of
`devices that enable control of a cursor by merely gesturing
`
`45
`
`50
`
`65
`
`Page 10 of 17
`
`

`
`5.786.804
`
`20
`
`25
`
`35
`
`3
`a method and system that is cost e?icient when used in
`controlling a screen cursor or when used in other remote
`interactive video applications.
`SUMMARY OF THE INVENTION
`Correlation of successive images acquired by means of a
`two-dimensional array of photosensors is used as a basis for
`tracldng attitude of a device to which the array is affixed In
`the preferred embodiment. the device is a hand-holdable
`member. such as a controller for maneuvering a cursor on a
`display screen of a video set. Based upon the step of
`correlating images to detect differences in location of
`imaged features that are common to a succession of images.
`the system generates an attitudinal signal indicative of any
`changes in angular orientation during the time period of
`acquiring the images. That is. the attitudinal signal is deter
`mined by the pitch and yaw. and optionally the roll. of the
`device that bears the array of photosensors. Since the
`acquired images need not be related to that which is being
`controlled. e.g. a screen cursor. the device can face in any
`direction during the control process. Moreover. it is not
`necessary to provide a dimensional one-to-one correspon
`dence of angular displacement of the device and travel of
`that which is being controlled. Within cursor control. for
`example. the controller may be directed arbitrarily and
`relationships of degrees of pitch and yaw to lengths of cursor
`movement may be user-adjustable.
`The two-dimensional array of photosensors is used to
`acquire a reference frame for tracking the attitude of the
`device. The reference frame is stored and a second image of
`features within a ?eld of view of the array is acquired. The
`second image may be considered to be a sample image. and
`the ?elds of view of the two images should be largely
`overlapping. so that the reference and sample frames include
`a number of common features. While not critical. the device
`includes optics which provide a focus nominally at in?nity.
`intentionally presenting an off-sharp image to the array of
`photosensors. In the application of the device for controlling
`a screen cursor. the representative imaged features will
`typically include windows. lamps. furniture and the display
`screen itself. In any application of the invention. one or more
`stationary sources of light may be speci?cally added within
`the environment to be imaged. so that successive images of
`the ?xed light are used for the purpose of correlation. In one
`implementation of such an embodiment. the source of light
`is an IR emitter and the imaging array on the device is
`provided with IR ?ltering to permit tracking of the attitude
`of the device.
`Conceptually. the step of correlating the reference frame
`with a sample frame is one in which one of the frames is
`?xed in position and the other frame is repeatedly shifted to
`determine which shifted position best approximates an
`alignment of the imaged features that are common to the two
`frames. thereby allowing the determination of the pitch and
`yaw of the imaging array during the interval between
`acquiring the two frames. In practice. the shifts are per
`formed computationally and are shifts of pixel values in
`which each pixel value is indicative of light energy received
`at a particular photosensor at a. speci?c time. The correla
`tions may be limited to computational shifts of only one
`pixel for nearest-neighbor correlations. or may be multi
`pixel computational shifts. The nearest-neighbor correlation
`process is often preferred. since it is less computationally
`complex. with only the original position and eight compu
`tational shifts being necessary. Interpolations are then per
`formed to determine angular displacements that are less than
`a full pixel. Angular displacement of the device about a
`
`4
`horizontal axis. i.e. pitch. will result in the arrangement of
`pixel values of the reference frame being moved upwardly
`or downwardly. Angular displacement of the device about a
`vertical axis. i.e. yaw. will result in the pixel value arrange
`ment being moved to the left or to the right. The system
`detects pitch. yaw and combinations of pitch and yaw. The
`attitudinal signal that is generated by the system is respon
`sive to the detection of such angular displacements.
`Optionally. roll may also be considered.
`In the application in which the attitudinal signal is gen
`erated in order to control a screen cursor. the device pref
`erably includes a transmitter for wireless transmission of a
`cursor-control signal. For example. the signal may be trans
`mitted via an infrared beam. Changes in the pitch of the
`hand-holdable device are then translated into vertical move—
`ments of the screen cursor. while changes in device yaw will
`move the screen cursor laterally. In this embodiment. trans
`lational movement of the device may also be detected and
`utilized. so that vertical or horizontal movement of the
`device translates to a corresponding vertical or horizontal
`movement of the screen cursor.
`One concern in the implementation of the method and
`system is the effect of the phenomena known in the lens
`design as curvilinear distortions. Curvilinear distortions are
`also referred to as pin-cushion. barrel. and perspective
`distortions. Rectilinear detail is compressed at the outer
`edges of the ?eld by such distortion. Curvilinear distortion
`is particularly pronounced in simple lenses with wide fields
`of view. such as the lens contemplated for use with the
`present invention. In the invention. the ?eld of view is
`preferably approximately 64°. so that curvilinear distortions
`will inevitably occur.
`In the preferred embodiment. the photosensors of the
`array vary dimensionally in order to de?ne an array that is
`curvilinear. i.e.. includes an arcuate outer edge. The curvi
`linear array is dimensioned to compensate for the curvilinear
`distortion introduced by the lens system. The imaging by the
`optics is evaluated to characterize the curvilinear distortion.
`with the array then being patterned to o?‘set the distortion.
`In this manner. the arrangement of the photosensor array and
`the optics greatly reduces adverse eifects of curvilinear
`distortion.
`An advantage of the invention is that device attitude may
`be tracked in a reliable and cost-el?cient manner. For those
`applications in which the array-bearing device is a hand
`holdable device. control of a screen cursor or the like is
`economically achieved without a premium on dexterity.
`Moreover. the device does not require operation on a suit
`able surface.
`
`45
`
`50
`
`55
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`FIG. 1 is a perspective view of a hand-holdable device
`that enables attitude tracking of the device. in accordance
`with the invention.
`FIG. 2 is a block diagram of components for controlling
`a screen cursor in accordance with the invention.
`FIG. 3 is a block diagram of an array of photosensors and
`transfer ampli?ers of the correlator of FIG. 2.
`FIG. 4 is an operation view of the steps for correlating
`successive images in accordance with the invention.
`FIG. 5 is a conceptual view of the steps of FIG. 4.
`FIG. 6 is an embodiment of a computation cell of an array
`of cells for carrying out the steps of FIG. 4.
`FIGS. 7-11 illustrate arrays of photosensors and compu
`tation cells for demonstrating and then reducing curvilinear
`distortions.
`
`Page 11 of 17
`
`

`
`5,786,804
`
`15
`
`20
`
`25
`
`35
`
`5
`BEST MODE FOR CARRYING OUT THE
`INVENTION
`With reference to FIG. 1. a hand-holdable controller
`device 10 is shown as including an integrated circuit pack
`age 12 for acquiring and processing image data. On the
`surface of the package is an optoelectric integrated circuit
`chip 14 having a two-dimensional array 16 of photosensors
`and having processing circuitry 18. The array and processing
`circuitry are used to acquire successive frames of image
`data. permitting the attitude of the device to be tracked.
`Thus. angular displacement about a horizontal axis 20. Le.
`pitch. and angular displacement about a vertical axis 22. Le.
`yaw. can be determined. Optionally. roll about the longitu
`dinal axis 24 of the device 10 can also be monitored.
`The invention will be described primarily with reference
`to providing control of a remote element. such as controlling
`a screen cursor of a video display. However. it is understood
`that the approach of tracking attitude by means of correlat
`ing images formed by a photosensor array attached to the
`device has applications beyond that of controlling a screen
`cursor or the like.
`The sensor array 16 and the processing circuitry 18 are
`mounted on the front of the controller device 10 with the
`array facing outwardly. Alens 26 is shown in FIG. 1 as being
`partially cutaway in order to expose the array and circuitry.
`The lens may be an inexpensive plastic lens that is molded
`to the integrated circuit package 12. The focus of the lens is
`set nominally at in?nity. The lens provides an image of a
`region of the surrounding environment to the sensor array
`16. A short focal-length lens is preferred. since a greater
`number of features within the environment will be imaged
`and used for the purpose of tracking attitude. The image
`quality demands are sut?ciently lax. such that the demands
`on the optical arrangement are not stringent. In fact. a slight
`blurring is desirable. so that the spatial frequencies of the
`image are coarser than the size of the array elements.
`The width of the ?eld of view for imaging the environ
`ment is a matter of balancing the desire of capturing as much
`visible detail as possible with the requirement of avoiding
`excessive distortion. A 64° ?eld of view provides a reason
`able compromise between these two concerns. The dashed
`lines 28 in FIG. 1 represent the ?eld of view for the sensor
`array 16. The arrangement of the optics and the sensor array
`may be selected to reduce any adverse effects of curvilinear
`distortion on the attitude-tracking approach of the invention.
`The optic/sensor arrangement will be described in greater
`detail below.
`On the top surface of the controller device 10 are a cursor
`control key 30 and a return key 32. In operation. the screen
`cursor of a video display may be captured by depression of
`the control key 30. With the control key in a depressed
`position. the angular displacement of the device 10 will be
`translated into movement of the cursor along the display
`screen. For example. in selecting a particular movie from a
`menu of movies presented by an interactive television
`system. key 30 is depressed and pitch and/or yaw of the
`device 10 is employed to drag the screen cursor to the
`desired movie. The control key 30 is then released to release
`control of the screen cursor. With the cursor indicating the
`desired menu choice. the return key 32 is depressed to
`activate the selection. Alternatively. a single selection key
`similar in operation to that of a computer mouse may be
`utilized. so that cursor control is continuously enabled.
`Remote communication is provided by infrared transmis
`sion via an emitter 34. Infrared emission is well known in the
`art. For cursor control. signal vectors are generated and
`
`45
`
`55
`
`6
`transmitted. The vectors are sent as increments in order to
`reduce the burden of the IR communications channel by
`keeping the data bytes shorter. The rate at which vectors are
`transmitted should be su?iciently fast to achieve the neces
`sary cursor responsiveness. but not so fast as to tax the
`limited bandwidth of the 1R link to the controlled set. As will
`be explained more fully below. in one embodiment the
`transmission is a lO-bit word that is emitted every 40 ms. as
`needed. When neither of the keys 30 and 32 is depressed. no
`transmission occurs. This frees the 1R emitter for other
`functions during non-cursor events. Thus. the controller
`device 10 may include keys to control other events. such as
`the changing of a television channel.
`Referring now to FIGS. 1 and 2. a control clock 36
`determines the timing of operations for the controller device
`10. The image-capture rate is at least partially determined by
`the features that are to be imaged. Where the ?eld of view
`28 is a living room of a person engaged in operation of an
`lTV system. the image capture rate mu st be su?‘lciently great
`to ensure that the image data is su?iciently intelligible to
`permit correlation between successive images. The circuitry
`of FIG. 2 includes an exposure control 38 which shows a
`minimum exposure of 5 us and a maximum exposure of 5
`ms. Electronically. the “exposure time" is the interval during
`which charge generated by the individual photosensors is
`allowed to integrate during the 5 ms period between suc
`cessive image captures. While the exposure control may be
`manually adjusted. the preferred embodiment is one in
`which there is a feedback line 40 from image correlator
`circuitry 42 to establish automatic gain control. Thus. if the
`image data is consistently off-scale for performing correla
`tion processing. the exposure control 38 increases the inter
`val for integrating charge by the photosensors.
`As previously noted. the ?eld of view 28 is contemplated
`to be approximately 64°. In a sensor array of 32x32
`photosensors. a single pixel will have an approximately 2°
`?eld of view (64°/32 pixels). In the preferred embodiment to
`be described below. correlations are limited to nearest
`neighbor pixel cells. This establishes a maximum velocity
`for movement of the controller device 12. In the preferred
`embodiment. the maximum velocity should not exceed one
`pixel length between successive image captures. If a frame
`of image data is to be acquired at the correlator 42 every 5
`ms. the maximum velocity will be 200 pixels per second
`Since the ?eld of view of each pixel is 2°. the maximum
`velocity is 400° per second.
`The operation of the correlator 42 will be explained in
`detail below. Its basic function is to compare the arrange
`ment of pixel values in a ?rst frame of image data to the
`location of a similar arrangement in a subsequent frame. The
`sensor array 16 is used to acquire a frame of 32x32 pixel
`values. The ?rst captured frame is referred to as a reference
`frame. The first reference frame is captured upon depression
`of the cursor control key 30 of FIG. 1. A subsequently
`captured frame of pixel values. referred to as a sample
`frame, is compared with the reference frame to detect
`changes in attitude of the controller device 10. A new
`reference frame is then acquired. The reference frame may
`be replaced by the sample frame after each correlation.
`Alternatively. the reference frame may be replaced with a
`sample frame each time a threshold is reached. such as a
`time threshold or a threshold based upon movement of the
`device.
`Correlation will be described in detail below.
`Conoeptually. the process may be considered as ?xing the
`two-dimensional reference frame in position and shifting a
`transparency of the two-dimensional sample frame to vari
`
`Page 12 of 17
`
`

`
`5.786.804
`
`7
`ous positions atop the reference frame in order to detect
`alignment of common features of the two frames. This
`conceptual view is shown in FIG. 5 and is computationally
`carried out by an array of cells such as the pixel cell shown
`in FIG. 6.
`The correlation processing generates an attitudinal signal
`indicative of any changes in angular orientation of the
`controller device 10. In FIG. 2. the correlator 42 is shown as
`generating a signal in which change along the X axis is
`+0.75 pixels. while the change along the Y axis is -0.31
`pixels. In FIG. 1. positive movement along the X axis may
`be indicated as a result of angular displacement of the device
`10 such that the ?eld of view 28 is moved in the direction
`of arrow 44. while a negative movement along the X axis is
`one in which a component of displacement of the ?eld of
`view 28 is in the direction indicated by arrow 46. Regarding
`the Y axis. a positive value at the attitudinal signal indicates
`manipulation of the device 10 to rotate the ?eld of view as
`shown by arrow 48. while a negative value of AY indicates
`displacement in the direction shown by arrow 50.
`The preferred approach for operation of the correlator 42
`is one in which nearest-neighbor pixel shifts are executed in
`order to track changes in attitude of the controller device.
`This approach will be detailed when referring to FIGS. 4. 5
`and 6. However. the attitude tracking alternatively may be
`carried out using correlations requiring shifts of greater than
`a single pixel. For either approach. interpolations are com
`putationally performed to identify fractional pixel values.
`Such image correlating interpolating is well understood by
`persons skilled in the art.
`The output from the corre1ato1'42 is received at a multi
`plier 52 that increases the sensitivity of the cursor control. In
`the embodiment of FIG. 2. the multiplier doubles the AX and
`AY values from the correlator. The signal is then conducted
`to an accumulator 54. A divide-by-eight circuit 56 dictates
`operation of the accumulator 54. Eight samples will be
`received from the multiplier 52 before the accumulator
`outputs a sample to a modulo function block 58. The output
`of the modulo block will be the integer portion of the
`accumulation of the eight samples from the multiplier 52. As
`previously noted. the controller device transmits a cursor
`control signal by means of an IR emitter 34. IR transmission
`links have limited bandwidth. The use of the accumulator 54
`reduces the demands on the communications link.
`The exemplary sample from the accumulator 54 in FIG.
`2 is shown to be one in which AX=+15.2 pixels and
`AY=—-3.3 pixels. The modulo block 58 passes only the whole
`numbers to a pulse code modulator 60. Each of the AX and
`AY values is contained within 5 bits. with 4 bits relating to
`the numerical value and the remaining bit relating to
`whether the numerical value is positive or negative.
`The values output from the modulo block 58 to the
`modulator 60 are whole number values. while the remainder
`of the sample from the accumulator 54 is conducted to a
`residue circuit 62. The residue values are saved at the residue
`circuit and added to subsequent values from the accumula
`tor.
`The remaining arrangement of‘ components is well known
`in the art. A 10-bit word is output from the pulse code
`modulator 60 to an ampli?er 64 every 40 ms. Thus. the 1R
`transmission from the emitter 34 is only 25 words per
`second. The IR transmission is received by a sensor 66 of a
`display system. such as a RIVS. The signal from the IR
`sensor 66 is decoded at a demodulator 68 and transmitted to
`a cursor control circuit 70 for manipulating a screen cursor
`of a video screen 72. Merely for example. the screen may
`
`20
`
`25
`
`30
`
`35
`
`45
`
`50
`
`55
`
`8
`have 64x64 cursor addresses and the screen cursor may
`move l5><25=375 address states/ second. In the example
`given above in which the ?eld of view 28 of FIG. 1 is 64°.
`the screen can be traversed in 0.17 second (64/315).
`The arrangement and interaction of the components of
`FIG. 2 permit cursor control based upon tracking of the
`attitude of the controller device 10 of FIG. 1. The controller
`encodes the device pitch. yaw and optionally roll by tracking
`the apparent motion of an image of the environment in
`which the device resides. It is not necessary to point the
`controller in any one direction. other than to establish the IR
`link between the emitter 34 and the sensor 66. With the
`possible exception of the correlator 42. the individual com
`ponents of FIG. 2 are readily understood by persons skilled
`in the art and need not be explained in detail. However. the
`preferred embodiment of the correlator 42 will be set forth
`below. The preferred arrangement of the two-dimensional
`sensor array 16 and the optics 26 will also be set forth.
`
`Column Transfers of Photosensor Signals
`As previously noted. the embodiment of FIGS. 1 and 2
`includes a sensor array 16 having thirty-two columns and
`thirty-two rows of photosensors. Referring now to FIG. 3.
`?ve columns 74. 75. 76. 77 and 78 of the thirty-two columns
`of photosensors are shown. Also shown are six of the
`thirty-two rows 79. 80. 81. 82. 83 and 84. Each column is
`operatively associated with a separate transfer ampli?er 85.
`A photosensor in a column is connected to the operatively
`associated transfer ampli?er for closing a read switch 86. In
`the operation of the circuitry of FIG. 3. no two photosensors
`are connected to the same transfer ampli?er simultaneously.
`Each transfer ampli?er 85 includes an input 87 that is
`connected to a source of a ?xed voltage. A second input 88
`is capacitively connected to the output 90 of the transfer
`ampli?er by a transfer capacitor 89.
`In the operation of the circuit of FIG. 3.