llIIlIlllIlIlIllIlIllIllIllIIlIlllIlIllIllIIllIllIIllIllllIIlIllIlllIIllIlIIIIIIIIIIIIIIII
`US 20120287274A1
`
`119~ United States
`112) Patent Application Publication
`Bevirt
`
`110) Pub. No. : US 2012/0287274 A1
`Nov. 15, 2012
`(43) Pub. Date:
`
`154) TRACKING OF DYNAMIC OB JECT OF
`INTEREST AND ACTIVE STABILIZATION OF
`AN AUTONOMOUS AIRBORNE PLATFORM
`MOUNTED CAMERA
`
`Publication Classification
`
`151) Int. Cl.
`H04N 7/18
`152) U. S. Cl. . . . .
`
`12006. 01)
`. . . . . . . . . . . . . . 348/144; 348/E07. 085
`
`176)
`
`Inventor:
`
`JoeBen Bevirt, Santa Cruz, CA
`1US)
`
`157)
`
`ABSTRACT
`
`121) Appl. No. :
`
`13/451, 509
`
`122) Filed:
`
`ApF. 19, 2012
`
`Related U. S. Application Data
`
`160) Provisional application No. 61/476, 767, filed on Apr.
`19, 2011.
`
`A system for tracking and video surveillance using an aerial
`videographic platform. The system may be adapted to follow
`a user possessing a GPS device with a transmitter,
`such as an
`telephone. The aerial platform may be
`advanced cellular
`commanded by the user to follow the user during a course of
`the completion of an outdoor activity.
`time, such as during
`The aerial plat form may use a video camera to record the user
`while the user is engaged in the activity.
`
`100 ~
`
`~103
`
`Yuneec Exhibit 1004 Page 1
`
`

`
`Patent Application Publication
`
`Nov. 15, 2012 Sheet 1 of 3
`
`US 2012/0287274 A1
`
`100 ~
`
`+ 103
`
`FIG. 1
`
`Yuneec Exhibit 1004 Page 2
`
`

`
`Patent Application Publication
`
`Nov. 15, 2012 Sheet 2 of 3
`
`US 2012/0287274 A1
`
`102
`
`106
`
`c
`
`108
`
`, s'
`
`~ ~"~103
`
`FIG. 3
`
`Yuneec Exhibit 1004 Page 3
`
`

`
`Patent Application Publication
`
`Nov. 15, 2012 Sheet 3 of 3
`
`US 2012/0287274 A1
`
`FIG. 5
`
`Yuneec Exhibit 1004 Page 4
`
`

`
`US 2012/0287274 A1
`
`Nov. 15, 2012
`
`TRACKING OF DYNAMIC OBJECT OF
`INTEREST AND ACTIVE STABILIZATION OF
`AN AUTONOMOUS AIRBORNE PLATFORM
`MOUNTED CAMERA
`
`CROSS-REFERENCE TO RELATED
`APPLICATIONS
`[0001] This application claims priority to U. S. Provisional
`Patent Application No. 61/476, 767 to Bevirt, filed Apr. 19,
`2011, which
`by reference
`incorporated
`is hereby
`in its
`entirety.
`
`FIELD OF THE INVENTION
`
`relates to aerial vehicles, namely a
`[0002] This invention
`system with an aerial platform adapted for tracking and video
`target. The platform may be radio con-
`recording a moving
`trolled and may be actively stabilized by on-board controllers
`in various modes of operation.
`
`BRIEF DESCRIPTION OF THE DRAWINGS
`[0003] FIG. 1 is a sketch of an aerial platform
`tracking a
`user according to some embodiments of the present invention.
`[0004] FIG. 2 is a perspective view of an aerial platform in
`a quad rotor configuration according to some embodiments of
`the present invention.
`[0005] FIG. 3 is a front view of an aerial platform
`in a quad
`to some embodiments of the
`rotor configuration
`according
`present invention.
`[0006] FIG. 4 is a top view of an aerial platform
`in a quad
`to some embodiments of the
`rotor configuration
`according
`present invention.
`[0007] FIG. 5 is a side view of an aerial platform
`in a quad
`to some embodiments of the
`rotor configuration
`according
`present invention.
`
`SUMMARY
`
`[000S] A system for tracking, video imaging, and photo-
`graphing a dynamic object using an autonomous aerial video-
`graphic platform. The system may be adapted to follow a user
`a GPS device with a transmitter,
`such as an
`possessing
`telephone. The aerial platform may be
`advanced cellular
`commanded by the user to follow an object of interest, which
`may the user, during a course of time, such as during
`the
`completion of an outdoor activity, either from a given posi-
`tion, or on a prescribed trajectory pattern, relative to the user.
`The aerial platform may use a video camera to record the user
`while the user is engaged
`in the activity.
`
`DETAILED DESCRIPTION
`
`In some embodiments of the present invention,
`[0009]
`the
`is comprised of a handheld
`control device with a
`system
`graphical user interface for sending high level mission com-
`mands and information
`its position, an autonomous
`about
`aerial platform with vertical take-off and landing
`(VTOL)
`capability and an actuated gimbal mounted camera. Both the
`handheld control device, which could be implemented on an
`and the airborne vehicle are
`advanced cellular
`telephone,
`equipped with GPS and inertial navigation sensors. Estima-
`tion and control algorithms
`running onboard the vehicle use
`taken from the sensors onboard both the
`the measurements
`vehicle and the handheld device for flight and camera stabi-
`lization, as well as for ongoing correction of the relative
`
`position/trajectory of the vehicle and camera assembly with
`respect to the OoI. Computer vision algorithms
`are imple-
`the detection and
`the vehicle
`to enhance
`mented onboard
`tracking of the OoI, and the software- and hardware-based
`image stabilization.
`In some embodiments of the present
`[0010]
`invention, as
`seen in FIG. 1, a tracking and surveillance system 100 utilizes
`an aerial platform 101. The aerial platform 101 may have a
`video camera adapted to record video of an Object of Interest
`(Ooi), which may be the user 102. The user 102 may have an
`electronic device, such as a cellular telephone with GPS capa-
`bility, which emits a signal 104. The aerial platform 101 may
`have a video camera 103 adapted to take video and/or still
`images.
`as seen in FIGS. 2-5, the
`[0011] In some embodiments,
`aerial platform 101 may have a main body 106 which has
`support arms 10S. The support arms 10S may support motors
`105 which drive rotors 102. The camera 103 may be attached
`to the main body 106 with a positioning mechanism 104. The
`aerial platform may be adapted to engage in horizontal
`flight
`using the four rotors as seen. The platform may be tipped in
`order to provide thrust with a horizontal component. Control
`electronics 107 may be mounted on the main body 106.
`the aerial platform may con-
`[0012]
`In some embodiments,
`sist of a platform without a wing wherein
`the thrust
`is typi-
`downward, with the use of three, four, or
`cally predominantly
`more spaced rotors. The platform may be adapted
`to hover
`and engage in flight along a horizontal plane.
`[0013] In some embodiments,
`a
`the platform
`supports
`video camera. In some embodiments,
`the video camera
`is
`to allow for
`supported with an articulated
`system adapted
`position control of the camera. Thus, when tracking a moving
`the flight path of the platform and the
`object on the ground,
`view angle of the camera may be independent,
`or quasi-
`In some embodiments, when coupled with an
`independent.
`system and an estimation system, the
`inertial measurement
`system may be actively controlled
`to isolate the
`articulated
`camera from vehicle jitters.
`[0014]
`In some embodiments,
`to
`is adapted
`the system
`allow the user to use an electronic device, which may be a
`handheld electronic controller, or a cellular telephone,
`to set
`up the operation of the aerial platform and the video camera
`mounted thereon. The handheld controller may be adapted to
`receive a signal
`from the camera such that the user may
`observe the image that is captured by the camera. In some
`the video camera may transmit relatively
`embodiments,
`low
`quality images to the handheld unit, and may record relatively
`high quality video stream.
`[0015]
`In some embodiments,
`to
`is adapted
`the system
`allow the user to select from a variety of operational modes.
`Multiple operational modes may be strung together, with user
`to pre-plan a scene recording mission
`specified parameters,
`for the vehicle (e. g. take-off location, relative distance from
`user, segments of trajectory to follow, still photography
`fre-
`quency and/or trigger times, landing location, etc. ). Instances
`of user selected and specified video recording sequences are
`for the aerial vehicle to record from a prescribed position, or
`on a given trajectory pattern, either relative to the OoI loca-
`tion, or relative to a fixed reference location. For example, the
`user may desire to have the aerial platform
`record video
`images of the user at an altitude of 15 m higher
`than the
`fromaposition30mnorthoftheuser.
`altitudeoftheuser,
`The
`relative position may also be specified with respect to the OoI
`current direction of travel averaged over a specified
`time
`
`Yuneec Exhibit 1004 Page 5
`
`

`
`US 2012/0287274 A1
`
`Nov. 15, 2012
`
`interval. The position of the aerial platform may be relative to
`direction of the user, such that the platform maintains a posi-
`tion of 30 m behind
`the user along the user's direction of
`travel, or 30 m to the left of the user on a parallel trajectory, for
`example. Prescribed flight trajectories may be composed by
`the user by selecting basic flight pattern primitives from a list
`line, spiral, circular and polygonal
`(including hover, straight
`patterns) and specifying their parameters. Spirals and straight
`lines may be in the horizontal plane, or vertical plane, or
`ascending or descending at a specified rate.
`[0016]
`the mission planning may be
`In some embodiment,
`facilitated using a specially designed graphical user interface
`telephone, or a dedicated handheld control
`on an advanced
`device. The handheld control device may transmit
`tracking
`assistance data to the aerial vehicle. This data may include
`position, velocity, acceleration, orientation,
`and heading.
`information may be fused with sensor data on the
`This
`vehicle.
`[0017]
`In some embodiments,
`the handheld controller and
`the aerial platform are both equipped with GPS capability
`which allows both the absolute and the relative positions of
`the user and the platform to be ascertained and used in posi-
`to a moving user. In some
`the platform
`relative
`tioning
`the handheld controller may be an application
`embodiments,
`running on a cellular
`and the communication
`telephone,
`between the handheld controller and the aerial platform may
`be using a wireless cellular
`In some
`telephone network.
`controller may communicate
`embodiments,
`the handheld
`directly with the aerial platform.
`[001S] In some embodiments,
`the system maybe adapted to
`have the aerial platform hover, or follow a pre-determined
`flight path independent o f the motion o f the user, and have the
`camera track the user using an articulated
`sub-system and
`interpretation of the image generated by the
`image
`using
`the aerial platform may con-
`camera. In some embodiments,
`tain electronics adapted to view the user, and to track the user
`via articulation of the camera, based upon the taken images.
`the aerial platform may track the user
`In some embodiments,
`using a GPS offset, as described above, while
`the camera
`tracks the user using image interpretation. The location of the
`the field of view may be maintained while the
`OoI within
`aerial vehicle is in flight.
`[0019] In some embodiments,
`the user may command
`the
`to lift off of the ground using the handheld
`aerial platform
`controller, and then may view an image taken by the camera
`on the aerial platform. The user may then adjust the camera
`the change in
`such as the zoom, while reviewing
`parameters,
`view of the camera. The handheld controller may also allow
`for selection of a variety of parameters
`to a subse-
`relating
`quent use of the system wherein the aerial platform tracks the
`user and records a video of the user. The user may set the
`aerial platform to run for a certain length of time, and then to
`location. The user may set the aerial
`land in a pre-selected
`to stop and land. In some
`to run until commanded
`platform
`the aerial platform may be programmed
`to
`embodiments,
`safely land at either its current location, or at a pre-determined
`location, should the aerial platform or the camera lose track of
`the user for an extended period of time and/or when
`the
`battery charge has decreased below a certain threshold.
`there may be a high resolu-
`[0020]
`In some embodiments,
`tion camera used to record the images, with a field of view
`selected by the user, and a lower resolution wide angle camera
`used with a visual tracking portion of the system.
`
`[0021]
`the user may use the hand-
`In some embodiments,
`held device to select an Object of Interest (Ooi), which may
`be the person possessing the GPS based phone, and a tracking
`capability may be used to track the selected target. In some
`the tracking capability may be blended with
`embodiments,
`the tracking of the
`the GPS tracking capability
`to augment
`OoI.
`[0022] Algorithms and Principles for Tracking of Dynamic
`Object of Interest andActive Stabilization ofanAutonomous
`Airborne Platform Mounted Camera
`[0023] Computer Vision:
`a combination of novel and
`In some embodiments,
`[0024]
`documented computer vision algorithms are used to process
`images acquired with the camera to detect the OoI and deter-
`mine its position and orientation with respect to the camera.
`These measurements
`are then passed to the estimation and
`the accuracy of the OoI position
`filter to improve
`tracking
`estimate.
`[0025] A. I Detection/Recognition of OoI
`[0026] Featureless
`[0027] Point based
`[0 02S] General
`(SIFT): This
`[0029] Scale Invariant Feature Transform
`algorithm gives the ability to extract distinctive visual key-
`to location, scale and rotation.
`points that are invariant
`[0030] Principal Component Analysis (PCA)-SIFT
`[0031] Speeded Up Robust Features (SURF)
`[0032] Optical flow
`[0033] Hidden Markov Model (HMM)
`[0034] Predefined Types
`The object of interest might be selected from a predefined
`of objects.
`[0035] Face
`[0036] Human
`[0037] Bird
`[003S] A. 2 Object learning; The OoI may be identified
`the object
`by the user on different
`images containing
`using tools on a graphical user interface. Machine learn-
`ing algorithms may be used to improve the model of the
`object to be tracked (predefined type or user defined) and
`hence improve the robustness of the algorithm.
`[0039] A3 Visual Tracking; Estimation
`filtering algo-
`rithm which use object position measurements
`resulting
`from the object detection algorithm(s).
`[0040] B) Estimation
`[0041] An estimator
`the
`is an algorithm which combines
`from the different localization sensors to
`noisy measurements
`determine over time the best possible estimate of an object
`state (e. g. position, orientation, color, texture, etc. ).
`[0042] B. l Localization: A localization
`is
`estimator
`implemented on each the OoI localization device and the
`aerial platform.
`[0043] Strap-down IMU
`[0044] Loosely coupled GPS aided IMU
`[0045] Tightly coupled GPS aided IMU
`[0046] Handheld control device
`[0047] Airborne vehicle
`[004S] B. 2 Relative position estimation; An estimator is
`the relative positions of the
`to determine
`implemented
`aerial platform and the OoI given the estimates from the
`respective localization estimator.
`[0049] B. 3 Tracking:
`[0050] Extended Kalman Filter flavors (EKF, UKF, Sigma
`Point)
`[0051]
`Inertial and GPS sensing only
`[0052] Enhanced with computer vision
`
`list
`
`Yuneec Exhibit 1004 Page 6
`
`

`
`US 2012/0287274 A1
`
`Nov. 15, 2012
`
`3
`
`[0053] Particle Filter
`Inertial and GPS sensing only
`[0054]
`[0055] Enhanced with computer vision
`[0056] Vehicle Control
`[0057] A controller is an algorithm developed using feed-
`back control theory and optimal control theory. It uses the
`current best estimate of an object state as an input and deter-
`mines the best actuator commands to affect the state and move
`it to the actual target or desired state.
`[005S] C. I Multi-layer
`controller: Low-level
`flight
`vehicle control, mid-level
`trajectory
`and
`following,
`high-level mission control
`[0059] C. 2 Flight pattern primitives
`[0060] C3 Station keeping and trajectory following
`take-o ff and landing
`[0061] C. 4 Autonomous
`[0062] C. 5 Computer aided manual control
`[0063] Camera Control
`[0064] D. I Camera aiming
`[0065] D. 2 Hardware-based
`[0066] D3 Software-based
`[0067] User Interface
`[006S] E. I For computer aided manual control
`[0069] E. 2 For planning of autonomous mission. As evi-
`a wide variety of
`dent from
`the above description,
`embodiments may be configured
`from the description
`and modifica-
`given herein and additional advantages
`tions will readily occur to those skilled in the art. The
`in its broader aspects is, therefore, not limited
`invention
`to the specific details and illustrative examples shown
`and described. Accordingly,
`from
`such
`departures
`details may be made without departing from the spirit or
`scope of the applicant's general invention.
`
`image stabilization
`image stabilization
`
`What is claimed is:
`1. A system for the video recording of a user, said system
`comprising:
`an aerial platform, said aerial platform comprising control
`electronics;
`
`a video camera mounted on said aerial platform, said video
`camera coupled to said control electronics;
`a ground unit controller, said ground unit controller in radio
`communication with said control electronics;
`is adapted to track said ground
`wherein said aerial platform
`unit controller and record video images of the location of said
`ground unit controller.
`2. The system of claim 1 wherein said ground unit control-
`ler comprises a graphical user interface, and wherein said
`is adapted to allow the user to plan a
`ground unit controller
`video recording mission from a set of user selectable flight
`pattern primitives.
`3. The system of claim 2 wherein said ground controller is
`adapted to allow the used to track the ground unit controller
`based upon GPS information
`sent by the ground controller
`unit to the control electronics on the aerial platform.
`4. A method of automated video capture of a moving sub-
`the steps of:
`ject, said method comprising
`selecting a flight profile for an aerial platform, said aerial
`platform comprising a video camera;
`for the recording of a video
`selecting a set of parameters
`stream of a moving subject; and
`recording video of said moving subject.
`5. The method of claim 4 wherein said moving subject
`comprises a user with a ground unit controller.
`6. The method of claim 5 wherein said ground unit con-
`troller is in radio communication with said aerial platform.
`7. The method of claim 6 wherein said ground unit con-
`troller sends GPS locations of the ground unit controller
`to
`said aerial platform.
`the step of
`S. The method of claim 7 further comprising
`the video camera such that said video camera
`positioning
`takes images of the moving subject.
`9. The method of claim S further comprising
`the step of
`setting articulating said video camera during the flight of the
`aerial platform such that the video camera tracks the moving
`subject while said aerial platform flies in its flight profile.
`
`Yuneec Exhibit 1004 Page 7