United States Patent [191
`Gerhardt et a].
`
`lllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllll
`5,481,622
`Jan. 2, 1996
`
`USOO5481622A
`[11] Patent Number:
`[45] Date of Patent:
`
`[54]
`
`EYE TRACKING APPARATUS AND
`METHOD EMPLOYING GRAYSCALE
`THRESHOLD VALUES
`
`[75]
`
`Inventors: Lester A. Gerhardt, Clifton Park, N.Y.;
`Ross M. Sabolcik, Austin, Tex.
`
`[73]
`
`Assignee: Rensselaer Polytechnic Institute, Troy,
`NY.
`
`[21]
`
`Appl. No.: 204,008
`
`[22]
`[51]
`[52]
`
`[58]
`
`[56]
`
`Filed:
`
`Mar. 1, 1994
`
`Int. Cl.6 ..................................................... .. G06K 9/00
`US. Cl. ........................ .. 382/103; 382/171; 382/291;
`345/158; 364/709.11
`Field of Search ........................... .. 382/1, 9, 48, 100,
`382/103, 117, 171, 173, 291; 348/78; 345/8,
`157, 158; 364/7091, 709.11; 351/206, 209,
`210, 245
`
`References Cited
`
`U.S. PATENT DOCUMENTS
`
`2/1966 Mackworth et al. ..................... .. 351/7
`3,236,578
`3,542,457 1l/1970 Balding et a1.
`351/7
`
`4,102,564
`
`7/1978 Michael . . . . . . . . .
`
`. . . .. 351/7
`
`364/518
`6/1986 Garwin et a1. ..
`4,595,990
`382/1
`4,625,329 11/1986 Ishikawa et al.
`364/550
`4,648,052
`3/1987 Friedman et a1. ..
`.. 358/93
`4,748,502
`5/1988 Friedman et a1. ..
`. 351/210
`4,815,839
`3/1989 Waldorf ....... ..
`351/210
`4,836,670
`6/1989 Hutchinson
`351/210
`4,852,988
`8/1989 Velez et a1.
`351/210
`4,988,183
`l/199l Kasahara et a1.
`5,002,385
`3/1991 Kasahara et a1. ..................... .. 351/210
`
`(List continued on next page.)
`
`FOREIGN PATENT DOCUMENTS
`
`OTHER PUBLICATIONS
`Cunningham, R., “segmenting Binary Images”, Robotic
`Age, Jul/Aug. 1981, pp. 4-19.
`Kitter, J., Illingworth, J ., & Foglein, J., “Threshold Selection
`Based on a Simple Image Statistic”, Computer Vision,
`Graphics, and Image Processing, 1985, vol. 30, pp.
`l25—147.
`Haralick, R. M. & Shapiro, L. 6., “Survey: Image Segmen~
`tation Techniques”, Computer Vision, Graphics and Image
`Processing, 1985, vol. 29, pp. 100-132.
`“The Eyegaze Computer System”, LC Technologies, Inc.,
`Product Brochure, Aug. 1991, (13 pages).
`Haralick, R. M., Sternberg, S. R. & Zhuang, X., “Image
`Analysis Using Mathematical Morphology”, IEEE Transac
`tions on Pattern Analysis and Machine Intelligence, vol.
`PAMI-9, No. 4, Jul. 1987, pp. 532-550.
`
`Primary Examiner—LeO H. Boudreau
`Assistant Examiner—Andrew W. Johns
`Attorney, Agent, or Firm—Heslin & Rothenberg
`
`[57]
`
`ABSTRACT
`
`An eye-tracking system determines the position of a user’s
`pupil and maps this position into a point of regard of the user
`on an interface device, such as a display screen, or other
`real-world object by a system comprising a camera for
`acquiring a video image of the pupil; a frame grabber
`coupled to the camera for accepting and converting analog
`video data from the camera to digital pixel data; a computer
`coupled to the frame grabber for processing the digital pixel
`data to substantially determine the position of the pupil; a
`display screen coupled to the computer; and a support
`connected to the camera and display screen for ?xing the
`relative physical positions thereof relative to the user’s
`pupil. The processing performed by the computer may
`include the selection of a ?rst pixel intensity threshold for
`the segmentation of the digital pixel data into ?rst and
`second groups, where the total pixel area of the ?rst group
`is selected to be substantially equal to a pre-determined
`value expected to correspond to the area of a user’s pupil.
`The system may be calibrated by the user’s following a
`cursor on the display screen while the system measures the
`pupil position for known locations of the cursor.
`
`351/209
`.
`.
`0456166 11/1991 European Pat. Off.
`1090333 5/1985 U.S.S.R. ........................ .. A61B 3/14
`
`35 Claims, 16 Drawing Sheets
`
`Page 1 of 32
`
`SAMSUNG EXHIBIT 1013
`Samsung v. Image Processing Techs.
`
`

`

`5,481,622
`Page 2
`
`US. PATENT DOCUMENTS
`
`5,093,567
`
`3/1992 Staveley ................................ .. 250/221
`
`5,016,282
`
`5/1991 Tomono e161. .......................... .. 382/2
`
`7/1991 Akeel et a1. .......................... .. 250/561
`5,034,618
`9/1991 Danon ..................................... .. 606/10
`5,049,147
`5,070,883 12/1991 Kasahara ............................... .. 128/745
`
`2332:;
`
`1
`1
`5,189,512
`5,325,133
`
`ids“ et a1‘
`
`" 351/210
`
`awm“ - - - - - - - - - -
`- - - - -' 382/1
`2/1993 Cameron et a1- ----------------------- -- 358/93
`6/1994 Adachi .................................. .. 351/209
`
`SAMSUNG EXHIBIT 1013
`Page 2 of 32
`
`

`

`US. Patent
`
`Jan. 2, 1996
`
`Sheet 1 of 16
`
`5,481,622
`
`SAMSUNG EXHIBIT 1013
`Page 3 of 32
`
`

`

`US. Patent
`
`Jan. 2, 1996
`
`Sheet 2 0f 16
`
`5,481,622
`
`64
`
`/./
`
`SPEECH
`SYNTHESIZER
`
`7 T
`
`COMPUTER Q62
`
`14
`
`DISPLAY
`SCREEN
`
`12
`
`CAMERA
`
`FRAME
`GRABBER —2/60
`
`fig. 3
`
`SAMSUNG EXHIBIT 1013
`Page 4 of 32
`
`

`

`US. Patent
`
`Jan. 2, 1996
`
`Sheet 3 0f 16
`
`5,481,622
`
`
`
`PIXEL COUNT
`
`fig. 5
`
`PIXEL INTENSITY
`
`SAMSUNG EXHIBIT 1013
`Page 5 of 32
`
`

`

`US. Patent
`
`Jan. 2, 1996
`
`Sheet 4 of 16
`
`5,481,622
`
`
`
`P2300 .?xE
`
`10000
`
`8000
`
`6000
`
`000
`
`2000
`
`PIXEL INTENSITY
`
`fig. 7b
`
`SAMSUNG EXHIBIT 1013
`Page 6 of 32
`
`

`

`US. Patent
`
`Jan. 2, 1996
`
`Sheet 5 of 16
`
`5,481,622
`
`18000
`
`16000
`
`14000
`
`
`
`E38 .55 12000
`
`10000
`
`O O o 8
`
`o O o 6
`
`4000
`
`2000
`
`PIXEL INTENSITY
`
`SAMSUNG EXHIBIT 1013
`Page 7 of 32
`
`

`

`US. Patent
`
`Jan. 2, 1996
`
`Sheet 6 of 16
`
`5,481,622
`
`104
`102
`
`.
`f 71g- 9
`
`116
`
`,
`
`106‘
`I
`1
`*2/08
`
`2,100
`'1 110
`"2,11
`2
`
`128 (Xmin - Ymin )
`
`124
`
`128
`
`SAMSUNG EXHIBIT 1013
`Page 8 of 32
`
`

`

`US. Patent
`
`Jan. 2, 1996
`
`Sheet 7 of 16
`
`5,481,622
`
`154
`
`fig. 13
`
`fig. 74
`
`SAMSUNG EXHIBIT 1013
`Page 9 of 32
`
`

`

`US. Patent
`
`Jan. 2, 1996
`
`Sheet 8 of 16
`
`5,481,622
`
`T
`
`ACQUIRE
`"__* EYE IMAGE "'__'
`
`.
`RELAX
`COMPARISON
`CONSTRAINTS
`
`A
`LOCATE
`PUPIL
`
`MAP PUPIL LOCATION
`TO SCREEN LOCATION
`T
`SEND SCREEN LOCATION
`TO USER INTERFACE
`T
`UPDATE PUPIL
`STATISTICS
`
`fig. 15
`
`SAMSUNG EXHIBIT 1013
`Page 10 of 32
`
`

`

`US. Patent
`
`Jan. 2, 1996
`
`Sheet 9 0f 16
`
`5,481,622
`
`IF MISS .2
`MAXIMUM
`LIMIT, THEN
`REPORT ERRoR
`I
`
`IF MISS 2
`UPPER LIMIT
`2N0 MISS <T
`M ‘MUM L'M' -
`THEN RELAX
`COMPAR'SON
`CONSTRAINTS
`
`MISS = MISS + I
`
`.
`f'l/g. 16
`
`6» ACQUIRE EYE IMAGE
`
`THRESHOLD IMAGE
`I
`CONSTRUCT IMAGE BLOBS
`
`COMPUTE BLOB STATISTICS
`I
`COMPARE BLOB STATISTICS
`To RUNNING AVERAGES OF
`PUPIL STATISTICS TO
`CALCULATE DIFFERENCE ERRoR
`
`DIFFERENCE
`ERROR <
`TOLERANCE
`
`MISS = MISS — 1
`
`I
`
`IF MISS = 0, THEN TIGHTEN
`COMPARISON CONSTRAINTS
`
`I
`STORE PUPIL
`CENTER = CENTROID
`
`SAMSUNG EXHIBIT 1013
`Page 11 of 32
`
`

`

`US. Patent
`
`Jan. 2, 1996
`
`Sheet 10 of 16
`
`5,481,622
`
`CREATE IMAGE HISTOGRAM
`I
`SET DESIRED_AREA =
`EXPECTED PUPIL SIZE
`I
`SETX=O
`I
`
`SET AREA_FOUND = O
`
`AREA_FOUND
`< DESIRED_AREA
`
`X = X+1
`
`THRESHOLD_LEVEL = x
`
`AREA_FOUND = AREA_FOUND +
`AREA OF HISTOGRAM BIN X
`
`fig. 77
`
`SAMSUNG EXHIBIT 1013
`Page 12 of 32
`
`

`

`US. Patent
`
`Jan. 2, 1996
`
`Sheet 11 of 16
`
`5,481,622
`
`‘I
`
`ACOUIRE
`’ EYE IMAGE ‘—‘
`
`RELAX PUPIL
`COMPARISON
`CONSTRAINTS
`
`AT
`Lg8p|LE
`
`RELAX
`HIGHLIGHT
`COMPARISON
`CONSTRAINTS
`
`LOCATE
`HIGHLIGHT(S)
`
`REFERENCE
`HIGHLIGHT
`FOIaIND
`
`Xdeltc: = Xeye — Xhighlight
`Ydeltcl = Y eye — Yhighlight
`I
`MAP (XdeIto, Ydelto) TO
`SCREEN COORDINATES
`I
`SEND SCREEN LOCATION
`TO USER INTERFACE
`I
`UPDATE PUPIL
`AND
`HIGHLIGHT STATISTICS
`
`fig. 18
`
`SAMSUNG EXHIBIT 1013
`Page 13 of 32
`
`

`

`US. Patent
`
`Jan. 2, 1996
`
`Sheet 12 of 16
`
`5,481,622
`
`PLACE CURSOR AT
`KNOWN SCREEN LOCATION
`
`I
`
`MEASURE PUPIL LOCATION
`
`CONTINUE
`
`REPEAT N
`TIMES
`9
`DONE
`
`FIT FUNCTION TO N
`COLLECTED LOCATION PAIRS
`
`fig. 19
`
`SAMSUNG EXHIBIT 1013
`Page 14 of 32
`
`

`

`US. Patent
`
`Jan. 2, 1996
`
`Sheet 13 of 16
`
`5,481,622
`
`180
`
`EYE IMAGE
`(Xeye ' Yeye )
`
`IMAGE TO SCREEN MAPPING
`
`fig. 200
`
`El
`
`183
`
`DISPLAY SCREEN
`(X screen- Yscreen )
`
`SAMSUNG EXHIBIT 1013
`Page 15 of 32
`
`

`

`US. Patent
`
`Jan. 2, 1996
`
`Sheet 14 of 16
`
`5,481,622
`
`SAMSUNG EXHIBIT 1013
`Page 16 of 32
`
`

`

`US. Patent
`
`Jan. 2, 1996
`
`Sheet 15 of 16
`
`5,481,622
`
`
`
`ZOC<0O4 zummom x Duh/330.20
`
`0 5 050 505 6 5 5 4. 4. 332
`
`2%
`
`ACTUAL X SCREEN LOCATION
`
`fig. 23a
`
`0 7
`
`ACTUAL Y SCREEN LOCATION
`
`fig. 23b
`
`SAMSUNG EXHIBIT 1013
`Page 17 of 32
`
`

`

`US. Patent
`
`Jan. 2, 1996
`
`Sheet 16 0f 16
`
`5,481,622
`
`700
`
`. ~ ‘ . . . ~ » ~ ¢ . . . . . . . 1 . o . . . ~ . . . \ . . \ 6 \ \ ~ . ~ . . \ . . . . ~ . . z
`
`
`
`ZO;<UO|_ zmmmom x cub/‘4304640
`
`6 5 4 3 2 1
`O O O o o O
`O 0 O 0 O o 0
`
`. . . . . . _ . . . 1 . _ . . . . . . . . . 6 . . . . . . . L . . . . . _ . . 1 . 6 . . _ . . I
`
`. ~ 6 v 1 ~ . 1 r 1 . ~ . . ~ . . ~ ~ . . 6 . ~ . ~ - - - . 1 6 - . . . . . . . . . . . > . . 1 ~ \ . ~ 1 . . . . . . . . _ v 6 _ 6.
`
`ACTUAL X SCREEN LOCATION
`
`fig. 24a
`
`. . . . . . 6 . . . . . - p . \ _ . ~ . . . . _ - . . - . 1 _ . . . \ . . . ~ . . . . ~ . 6 \ . . \ . \ 1 . . . _ . \ . \ _ _ x . . 1 . ..
`
`. . . ~ . . \ . . 1 . . . \ . . . ~ . 1 \ ~ ~ . - . . 1 ~ . . . ~ _ . . _ ~ . 6 6 \ . z . _ . . . . v . . _ . . . 6 ~ - 1 _ .
`
`. \ v \ 6 . . . 6 . . . 6 . . . _ . . . 1 . . ~ . 6 x . . . . . . ~ . . . ‘ . ~ \ ‘ 1 \ 6 6.
`
`200
`
`m .6 m m w a e 4
`6 6 6 6 6 6 6 6
`
`
`
`ZO;<0O|_ zmmmow > owh<|504<0
`
`r
`
`2
`O O
`
`o 2
`
`40
`
`6O
`140
`ACTUAL Y SCREEN LOCATION
`
`160
`
`180
`
`fig. 24b
`
`SAMSUNG EXHIBIT 1013
`Page 18 of 32
`
`

`

`5,481,622
`
`1
`EYE TRACKING APPARATUS AND
`METHOD EMPLOYING GRAYSCALE
`THRESHOLD VALUES
`
`FIELD OF THE INVENTION
`
`The present invention relates generally to a vision-track
`ing system and more particularly to an eye-tracking system
`that uses digital processing of an image of an eye to locate
`the eye’s viewing direction.
`
`10
`
`DESCRIPTION OF THE PRIOR ART
`
`Often there is a need to interact with a computer without
`the use of one’s hands. This need may arise because one’s
`hands are occupied while executing some task or as the
`result of a physical disability which prevents one from
`having su?icient physical control over one’s hands to
`manipulate a traditional input device such as a keyboard.
`Interaction with a computer through the use of one’s eyes is
`one way to satisfy this need. For example, where a person is
`afflicted with severe physical disabilities, eye movements
`can represent one of the few remaining motions that can be
`readily controlled. A physically-disabled person could inter
`act with the computer through a system able to track and
`respond to the motion of one or both of his eyes.
`Several prior eye-tracking systems have been built to
`track the motion of the eye in its viewing of objects. Earlier
`systems determined the eye’s motion directly by physical
`contact with the eye, while current systems determine its
`motion indirectly by sensing light re?ected from the eye’s
`surface. Applications of prior eye-tracking systems have
`included the determination of the parts of advertising that
`catch a viewer’s attention, and the evaluation of the layout
`of automotive dashboards and cockpit displays to determine
`their effectiveness for drivers and pilots.
`Some recent eye-tracking systems have permitted a user
`to use the eye as a control input to a computer. In one
`example of such an application, a user selects words from a
`menu on a video screen to produce synthesized speech. This
`system operates by determining the intersection of the eye’s
`line of sight with the plane of the screen to determine a
`so-called “point of regard”, which is the point which the user
`is looking at on the screen and corresponds in this case to a
`menu selection. In other applications, however, the eye’s
`point of regard generally corresponds to the physical point
`at which the eye is looking, whether on a display screen or
`elsewhere in three-dimensional space. The location of the
`point of regard is determined by the eye-tracking system and
`used as a control input for interactive control by the user.
`Although certain prior systems permit a user to have some
`interactive control of a computer, these systems exhibit
`several disadvantages. In determining the eye’s point of
`regard it is necessary to know the relative positions of the
`sensing camera, the display screen, and the user’s eye. One
`of the more recent interactive systems ?xes the position of
`the display screen and the sensing camera relative to one
`another, but not relative to the user. Thus, the user’ s physical
`position must be restrained for proper functioning. This is a
`disadvantage because the user’ s head must remain stationary
`for long periods of time leading to increased fatigue.
`Another interactive system places the sensing camera on
`a helmet worn by the user. Although the camera’s position
`relative to the user is ?xed, the display screen’s relative
`position is not. Instead, the display screen is mounted in a
`?xed position apart from the helmet. Therefore, an addi
`tional helmet sensor is required to track the position and
`
`25
`
`35
`
`45
`
`55
`
`60
`
`65
`
`2
`orientation of the head. As a result, in this system the
`positions of the head and the eye must both be calculated to
`determine a point of regard. This second calculation for the
`head position increases the computational requirements of
`the system. In addition, both this and the aforementioned
`systems suffer from large physical size, lack of portability,
`and excessive cost.
`Thus, there is a need for an eye-tracking system that will
`not restrict the mobility of the user, is portable, is more
`affordable, and avoids the additional computational over
`head associated with tracking the relative positions of sys~
`tem components (or of the user), other than that of the eye
`itself.
`
`SUMMARY OF THE INVENTION
`
`This need is satis?ed, the limitations of the prior art
`overcome, and other bene?ts realized in accordance with the
`principles of the present invention by a vision-tracking
`system for determining a point of regard. In one approach,
`the vision-tracking system determines the point of regard by
`determining the position of a pupil of a ?rst vision means by
`digital image processing and then relating this position to a
`point of regard in the real-world of the ?rst or a second
`vision means (e. g. the point of regard on a display screen or
`on a selected object in three-dimensional space). Although
`the pupil position being determined is that of a ?rst vision
`means, the point of regard being determined can be that of
`either the ?rst or the second vision means. The point of
`regard may be that of the second vision means in situations
`where the pupil position of the ?rst vision means substan
`tially corresponds to the point of regard of the second vision
`means. This situation occurs, for example, in a typical pair
`of human eyes in which the left eye’s pupil position closely
`tracks the right eye’s pupil position.
`' In one aspect of the present invention, the vision-tracking
`system comprises:
`,
`a camera means for acquiring a video image of a vision
`means, wherein the video image comprises a pupil
`image;
`a frame grabber means, coupled to the camera means, for
`accepting video data corresponding to the video image
`from the camera means and converting the video data
`to digital pixel data;
`a computer means, coupled to the frame grabber means,
`for processing the digital pixel data to substantially
`determine the position of the pupil;
`a feedback means, coupled to the computer means, for
`accepting feedback data corresponding to the pupil
`position from the computer means; and
`a support means, connected to the camera and feedback
`means, for ?xing the relative physical positions of the
`camera and feedback means.
`The processing performed by the computer means of the
`vision tracking system may further comprise the selection of
`a ?rst pixel intensity threshold for the segmentation of the
`pixel data into ?rst and second groups. This processing may
`also comprise the following steps:
`grouping individual pixels from one of the ?rst or second
`groups into a ?rst set having at least one pixel blob
`(note: a blob is a region of connected pixels belonging
`to the same group); and
`selecting from the ?rst set one of the pixel blobs corre
`sponding to the pupil image.
`The feedback means may be a display screen, and the
`processing by the computer means may further comprise
`determining the position of the pupil image in image coor
`
`SAMSUNG EXHIBIT 1013
`Page 19 of 32
`
`

`

`5,481,622
`
`3
`dinate space, and mapping the position of the pupil image in
`image coordinate space into a position in display screen
`coordinate space.
`In another aspect of the present invention, the ?rst pixel
`intensity threshold is selected so that the total pixel area of
`the ?rst group is substantially equal to a pre—determined
`expected pupil area, and the step of selecting one of the pixel
`blobs corresponding to the pupil image comprises the steps
`of:
`calculating one or more statistics for each of the pixel
`blobs;
`comparing the statistic for each pixel blob with an
`expected value corresponding to the pupil image to
`calculate a difference error; and
`selecting the pixel blob corresponding to the pupil image
`where the difference error is less than a pre—determined
`tolerance.
`In a further aspect of the present invention, the vision
`tracking system comprises at least one light source mounted
`on the support means that illuminates the vision means and
`creates a radiation intensity highlight on the vision means. In
`this aspect, the processing by the computer means may also
`comprise the steps of:
`selecting a second pixel intensity threshold, greater in
`intensity than the ?rst pixel intensity threshold, for the
`segmentation of the pixel data into third and fourth
`groups, the second pixel intensity threshold being
`selected so that the total pixel area of the fourth group
`is substantially equal to a predetermined expected area
`for all highlights of the light sources illuminating the
`vision means;
`grouping individual pixels from the fourth group into a
`second set having at least one pixel blob;
`selecting from the second set one of the pixel blobs
`corresponding to a ?rst highlight; and comparing the
`relative positions of the pixel blob corresponding to the
`pupil image and the pixel blob corresponding to the
`?rst highlight to determine the point of regard of the
`vision means.
`Yet another aspect of the present invention is realized in
`a digital vision-tracking system by a tracking method for
`determining a point of regard. This method comprises the
`steps of:
`acquiring video data from a camera corresponding to a
`video image of a vision means having a pupil, wherein
`the position of the pupil corresponds to the point of
`regard and the video image comprises a pupil image;
`converting the video data to digital pixel data correspond
`ing to the video image using an analog-to-digital inter
`face coupled to the camera;
`processing the pixel data in a computer coupled to the
`analog-to-digital interface to substantially determine
`the position of the pupil by a processing method
`comprising the step of selecting a pixel intensity thresh
`old for the segmentation of the pixel data into ?rst and
`second groups; and
`providing feedback data corresponding to the pupil posi
`tion.
`The feedback data may be provided by a display screen,
`and the pixel intensity threshold may be selected so that the
`total pixel area of the ?rst group is substantially equal to a
`pre—determined expected pupil area. The processing by the
`computer may also further comprise the steps of:
`grouping pixels from the ?rst group into a ?rst set having
`at least one pixel blob;
`
`25
`
`35
`
`45
`
`55
`
`60
`
`65
`
`4
`selecting from the ?rst set one of the pixel blobs as
`corresponding to the pupil image;
`determining the position of the pupil image by a calcu
`lated value based on a property of the selected pixel
`blob; and
`mapping the position of the pupil image in image coor
`dinate space into a position in display screen coordinate
`space,
`An advantage of the present invention is that all system
`components may be carried on one’s person, including a
`portable personal computer and a power supply in a small
`backpack. Other advantages include hands-free operation,
`reduced cost due to the elimination of head-position sensing,
`reduced computational complexity, and robust tolerance to
`head and body movements and variations in lighting. Fur
`ther, in one embodiment both the camera and feedback
`means are directed to a single eye to reduce the error in
`eye-tracking. Alternatively, in a different embodiment the
`camera is used to sense the pupil of one eye while the other
`eye is free to view a monitor or the surrounding environ
`ment. The foregoing and other objects, features, and advan
`tages of the invention will be apparent from the following
`more particular description of a preferred embodiment of the
`invention, as illustrated in the accompanying drawings.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`
`FIGS. 1 and 2 are front and side views, respectively. of the
`helmet-mounted components of one embodiment of a
`vision-tracking system according to the present invention.
`FIG. 3 is a system architecture block diagram for one
`embodiment of the vision-tracking system of the present
`invention.
`FIG. 4 is a typical eye image as captured by a video
`camera in a vision-tracking system using two LED light
`sources.
`FIG. 5 is a histogram classi?ed by the number of pixels
`per class for the captured eye image of FIG. 4.
`FIG. 6 is a thresholded binary image of the eye image of
`FIG. 4.
`FIGS. 7a-7c are an image of an eye, a pixel histogram,
`and a thresholded binary image corresponding thereto, for
`lighting conditions having a greater intensity than that for
`FIGS. 4—6.
`FIGS. 8a—8c are an eye image, a corresponding pixel
`histogram, and a thresholded binary image, for lighting
`conditions having an intensity less than that for FIGS. 4—6.
`FIG. 9 illustrates the pixels examined for connectivity
`during the scanning of an eye image during blob formation.
`FIG. 10 illustrates a pair of pixel blobs, showing a
`centroid of one blob and the rectangular boundaries of both
`blobs.
`FIG. 11 illustrates a typical eye image following image
`thresholding and blob de?nition.
`FIGS. 12—14 illustrate blob selection results for three
`different cases, including one case shown in FIG. 18 involv
`ing a blob selection failure.
`FIGS. 15-19 are ?owcharts illustrating the operation of
`the vision-tracking system of the present invention.
`FIGS. 20a-20e illustrate the mapping of an image from
`image coordinate space to display screen coordinate space.
`FIG. 21 illustrates the relationship between the x coordi
`nate of the pupil in image coordinate space and the (x,y)
`coordinates in display screen coordinate space.
`
`SAMSUNG EXHIBIT 1013
`Page 20 of 32
`
`

`

`5,481,622
`
`5
`FIG. 22 illustrates the relationship between the y coordi-
`nate of the pupil in image coordinate space and the (x,y)
`coordinates in display screen coordinate space.
`FIGS. 23a, 23b, 24a, and 24b are graphs illustrating
`calculated screen location versus actual screen location for
`both the x and y coordinates for two different test cases.
`
`BEST MODE FOR CARRYING OUT THE
`INVENTION
`
`In the operation of a vision-tracking system there are two
`primary functions of interest: acquisition of position infor-
`mation aboutthe pupil of a vision means, and determination
`of the point of regard of the vision means (or a correspond-
`ing second vision means) basedonthis position information.
`In one particular case, the vision means is a human eye and
`the point of regard is the object in the real world which the
`user is looking at. A specific system implementing these two
`primary functions is described below as one embodiment of
`a vision-tracking system according to the present invention.
`The physical structure of the system (also hereinafter
`referred to as an “eye-tracking system”) is described first,
`and then the architecture and operation of the system are
`next described. Finally, modifications and other embodi-
`ments of the present invention are described.
`Although the following description of the present inven-
`tion discusses a system for use with a human eye,it is not
`intended that the present
`invention be limited as such.
`Instead, as will be recognized by oneskilled in the art, the
`present invention may be used with other vision means, such
`as artificial or robotic viewing mechanismsor the eyes of
`animals. In general, such a vision means need only comprise
`a pupil having an external shape that can be captured in a
`video image and related to a point of regard by image
`processing according to the present invention.
`
`20
`
`25
`
`30
`
`35
`
`Physical Structure
`
`6
`Although in this particular application camera 12 and
`display screen 14 are mounted in front of separate eyes, in
`other applications these components are preferably directed
`to a single eye so that the other eye is free to view the
`surrounding scene or other real-world objects. Directing
`both the camera and the display screen to the same eyeis
`also advantageousin situations where the userhas a physical
`impairment(such as in a handicapped person)that prevents
`one eye from repeatably tracking the other eye. The eye-
`tracking system as described herein can be used with such a
`system directed to a single eye, as can be recognized by one
`of skill in the art.
`
`For example, a preferred alternative embodimentfor the
`physical structure of the eye-tracking system has a camera
`mounted on the side of the helmet (or on the side of a sturdy
`pair of glasses) looking at the eye through a prism mounted
`appropriately on the helmet. A display screen is mounted on
`top of the helmet (or on top of the glasses) and projectsits
`image onto, say, a half-silvered mirror surface. This mir-
`rored surfaceis a part of the prism through which the camera
`looks and permits the use of the same eye for both the
`camera and the display screen. Such a system provides
`greater eye-tracking accuracy, improved field of view, and
`reduced eyestrain.
`Also, in other embodimentsit is preferred that the camera
`and LEDs not be mounteddirectly in front of the user’s eye
`so that the user’s view of the real world is not obscured and
`so that the torque produced on the helmet by the weight of
`these components is reduced. Sometimes, excessive torque
`results in neck strain, particularly for physically-handi-
`capped users.
`Referring again to FIG. 1, display 14 may be, for example,
`the PC Private Eye display manufactured by Reflection
`Technology of Waltham, Mass. The Private Eye is a small (3
`emx3.3 cmx8.9 cm),
`light-weight (106 g) screen which
`produces a 30.5 cm virtual image at a distance of 61 cm.
`Also, the Private Eye is a PC-compatible display which can
`be operated as a monochrome CGAvideo adapter. The small
`size and light weight of the Private Eye makeit suitable for
`use as display screen 14. However, in other embodiments
`display screen 14 may be a color display and/or may be
`directed to more than one eye.
`Plates 22 and 24 are manufacturedof a translucentplastic
`so that the user may see through these plates into the
`surrounding world. Plates 22 and 24 are used here to permit
`the adjustmentof the camera and LEDsrelative to the user’s
`eye. However, in other embodiments different means could
`be used to permit this adjustment.
`Referring now to FIG.2, a side view of helmet 10 andits
`mounted components is shown. Video camera 12 is pointed
`so that eye 26 falls within its field of view, and LED 16 (not
`shown in FIG. 2) and LED 18 are pointed so that eye 26 is
`evenly illuminated. Display screen 14 is positioned for ready
`viewing by eye 28 (not shown in FIG. 2). The position of
`plate 24 relative to plate 22 can be adjusted via wingnuts 30
`as described aboveto adjust the aim of camera 12 and LEDs
`16 and 18.
`
`SAMSUNG EXHIBIT 1013
`Page 21 of 32
`
`First, the physical structure of one particular embodiment
`of an eye-tracking system is described. Referring to FIG. 1,
`a helmet 10 supports a video camera 12, a video display
`screen 14, and two LEDlight sources 16 and 18. Camera 12
`and LEDs16 and 18 are supported on a mounting block 20.
`Twoplastic plates 22 and 24 connect mounting block 20 to
`helmet 10. Mounting block 20 is positioned relative to
`helmet 10 such that camera 12 and LEDs 16 and 18 are
`positioned substantially in front of a user’s eye 26. Display
`screen 14, on the other hand, is positioned substantially in
`front of a second user’s eye 28. Plate 22 is firmly mounted
`to helmet 10. However, plate 24 can be adjusted relative to
`plate 22 via wing nuts 30. Similarly, mounting block 20 can
`be adjusted relative to plate 24 via wing nut 31. A standard
`NISC video cable 32 is connected to camera 12 for trans-
`mitting an analog video signal to the rest of the eye-tracking
`system.
`Two power-supply cables (not shown) are connected to
`LEDs 16 and 18, and a standard video display cable (not
`shown) is connected to display screen 14. LEDs 16 and 18
`Although two LEDsare shownin FIGS. 1 and 2, depend-
`are preferably infrared LEDs, and helmet 10 preferably fits
`ing upon the particular embodiment selected for use, the
`firmly to the user’s head and substantially prevents any
`eye-tracking system will work with only a single LED or
`motion of camera 12 and display screen 14 relative to eyes
`otherlight source. Two LEDsare preferred so that eye 26 is
`26 and 28. Relative motion of these components will result
`evenly illuminated. In other embodiments even more than
`in errors in the mapping of the pupil position of eye 26 into
`two LEDs may be used, depending upon illumination
`a position on display screen 14. A preferred example of
`requirements. Also, because helmet 10 mounts both camera
`helmet 10 is a hockey helmet, but other helmets suchas hats,
`
`goggles, headbands, or masks may be used in other embodi- 12 and display screen 14 inafixed position relative to eyes
`ments.
`26 and 28,there is no need to track the location of the user’s
`
`40
`
`45
`
`50
`
`55
`
`60
`
`65
`
`SAMSUNG EXHIBIT 1013
`Page 21 of 32
`
`

`

`5,481,622
`
`7
`head in space. This is advantageous becauseit eliminates
`costly computational overhead associated with either head
`sensing or additional image processing.
`Camera 12 maybe either a black and white camera or a
`color camera. A black and white camera is advantageous for
`reducing image processing complexity in some applications.
`However, in other embodiments of the present invention a
`color video camera may be used depending upon speed
`requirements. Also,
`in other embodiments of the present
`invention, a light source(s) other than an LED maybeused.
`In general it is sufficient that the light source provide even
`illumination of the eye to be imaged by camera 12.Toolittle
`illumination will result in a compression of the image gray
`scale, failing to take advantage of the full dynamic range of
`camera 12. On the other hand, too much illumination will
`cause the imaging element of camera 12 to become satu-
`rated, which will result in a compression ofthe gray scale at
`the higherintensity range. Also, using ambient light may not
`be acceptable in some situations due to its often changing
`and poorilluminating qualities. It is preferred that a constant
`light source be used to ensure consistently good image data.
`Even illumination is advantageous to avoid any shadows
`which might be mistaken for the pupil of the eye. Light
`sources which may be used depending uponthe application
`include incandescent
`lights,
`lighting through fiber optic
`cables, visible-light LEDs, and infrared-light LEDs. How-
`ever, because charge-coupled-device video cameras are
`extremely sensitive to infrared illumination, it is preferred
`that infrared LEDs be used as the light source. Infrared
`LEDsare also valuable because IRlightis not visible to the
`user,
`
`One consideration in selecting IR LEDs is that thermal
`damage to the human tissue of the eye needs to be avoided.
`An accepted limit for exposure to IR laser sources for a
`period exceeding 10 seconds is about 100 mW/cm”. The IR
`LEDsused should beless than this limit, for example, about
`3.4 mW/cm”. Another consideration which affects the num-
`ber of LEDs required to evenly illuminate the entire eye is
`the cone angle of light emittance from the LED. For
`example, an IR LED may have a 20° cone at a wavelength
`of 880 nm,and given this particular cone angle along with
`the positioning of the LEDsrelative to the eye as shown in
`FIGS. 1 and 2, two LEDsare preferably used.
`
`System Architecture
`
`FIG. 3 is a system architecture block diagram for one
`embodiment of the eye-tracking system according to the
`present
`invention. Camera 12 provides an analog video
`output to a frame grabber 60, which converts the analog
`video data to digital pixel data