`May 28-31, 2013, Grand Hyatt Atlanta, Atlanta, GA
`
`978-1-4799-0817-2/13/$31.00 ©2013 IEEE
`
`1165
`
`
`
`
`
`A Vision-based Target Tracking Control System
`of a Quadrotor by using a Tablet Computer
`
`JeongWoon Kim and David Hyunchul Shim
`
`Abstract — This paper proposes a vision-based target
`tracking system of a quadrotor. The system consists of a
`vision-based target detection algorithm using color and
`image moment of a target candidate. A flight control layer
`working with the vision layer and the low level quadrotor
`control layer is designed using the offset of the target from the
`center of the image frame. The image processing and control
`algorithms are all implemented on a latest tablet computer,
`which was capable of running those algorithms in real-time.
`The proposed system was validated on a commercially
`available quadrotor platform and demonstrated satisfactory
`target tracking performance.1
`
`I. INTRODUCTION
`In recent few years, interests in the research of
`rotary-wing aircraft have been growing rapidly due
`to the demand on their potential civilian and
`military applications. The quadrotor, one of the
`simplest
`rotorcraft,
`flies by
`independently
`controlling the rotor speed of the four rotors.
`Thanks to its simple structure and symmetric
`dynamics, quadrotors are getting more popular and
`being used for various purposes such as a flying toy
`to take picture in the air.
`One of the crucial advantages of a rotorcraft
`compared with a fixed-wing aircraft is its hovering
`capability. Thus, a rotorcraft, particularly a small
`sized quadrotor, can take pictures or videos with
`on-board cameras at low altitude without any
`collision in urban or indoor environment. A large
`portion of studies on vision-based tracking control
`of multi-rotors have been performed about on-
`board cameras faced toward ground. [1-4]
` However, ground faced cameras are not suitable
`to take close-up pictures of targets from the
`sidelines to detect a target or obstacles. Therefore,
`the need of target detection and collision avoidance
`
`Fig. 1. A UAV controlled by a tablet PC and a military
`grade tablet capable of UAV control (Harris Inc.)
`
`should be fulfilled with a vision-based tracking
`control using a forward-facing camera. [5]
`Generally, vision-based target recognition or
`obstacle detection require processors with high
`computational power that are typically found in
`desktop or
`laptop computers with graphic
`accelerators. However, these devices are usually
`too heavy to be carried on small-size UAVs.
`Therefore,
`the vision processing should be
`performed either onboard by solving a simplified
`problem (therefore less capable) or on a remote
`computer by wirelessly sending the video stream.
`During the last few years, a new generation of
`portable PC has emerged as a powerful mobile
`computing platform, which is primarily operated by
`touchscreen without mouse and keyboard. Known
`as “tablet”, they are light and typically equipped
`with high-speed wireless communication modem
`with ever-increasing computing power. These
`features encouraged UAV researchers to use them
`as potent mobile ground stations.
`In Section II target detection and classification
`algorithm is presented. Based on the relative
`position of a specified target in image frame,
`control method and controller layout diagram is
`formulated explained in Section III.
`
`
`
`1 This work was supported by the “High-Risk and High-Return (HRHR)”
`project program from the Korea Advanced Institute of Science and
`Technology (KAIST).
`
`J. Kim is with the Department of Aerospace Engineering, KAIST,
`Daejeon, South Korea (phone: +82-42-350-3764; fax: +82-42-350-3710; e-
`mail: yawh03@kaist.ac.kr).
`D.H. Shim is with the Department of Aerospace Engineering, KAIST,
`Daejeon, South Korea (e-mail: hcshim@kaist.ac.kr).
`
`Yuneec Exhibit 1022 Page 1
`
`
`
`1166
`
`
`
`The whole system including a quadrotor and a
`tablet computer is described in the following part.
`Implementation of a tablet computer application is
`briefly described as well. In Section V, the
`experiment results using a commercially available
`quadrotor and a tablet PC are presented and
`discussed.
`
`II. DETECTION OF A TARGET
`In this paper, we consider a red circular target for
`vision-based tracking of a quadrotor. The adopted
`methods to detect the specified target in the image
`frame is proposed in first.
`A. Color filter Algorithm
`The color filter filters out red pixels in original
`color image and convert it to a binary image.
`Assuming that an acquired image is in 8-bit and 3-
`channel RGB format, Algorithm 1 filters out pixels
`having one major specific color compared with
`other two colors.
`
`Algorithm 1.
`
`a 8-bit, 1 channel image frame.
`Output: A filtered binary image frame matrix,
`
`
`
`
`
`
`
`
`
`
` Else
`
`
`endIf
`endFor
`
`Input: (𝐈𝐈1,𝐈𝐈2,𝐈𝐈3), where each 𝐈𝐈k is a matrix of
`𝐈𝐈1b
`getBinaryImageofColor(𝐈𝐈1,𝐈𝐈2,𝐈𝐈3)
`For every pixel value 𝐈𝐈k,ij of i-th row and
`j-th column in the matrix 𝐈𝐈k
` If 𝐈𝐈1,ij≥0.5×�𝐈𝐈2,ij+𝐈𝐈3,ij�+40 then
` 𝐈𝐈1b,ij←1
` 𝐈𝐈1b,ij←0
`return 𝐈𝐈1b
`
`Algorithm 1. Pseudocode to filter out a 1-channel binary
`image from a 3-channel image respective to a specific color.
`Executing Algorithm 1 three times with respect
`to the three colors sequentially, the resulted three
`images are put into several binary operations to find
`“reddish” pixels in the entire RGB input image
`frame. This whole procedure is described in
`Algorithm 2. The “reddish” region may belong to
`
`Algorithm 2.
`
`matrices of 8-bit, 1 channel image
`frames of red, blue and green color from
`the 3-channel RGB image.
`Output: A filtered binary image frame matrix,
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`
`else
`
`endIf
`
`endFor
`
`Input: (𝐈𝐈R,𝐈𝐈G,𝐈𝐈B), where 𝐈𝐈R,𝐈𝐈G and 𝐈𝐈B are split
`𝐈𝐈CF
`colorFilter(𝐈𝐈R,𝐈𝐈G,𝐈𝐈B)
` 𝐈𝐈RR= getBinaryImageofColor(𝐈𝐈R,𝐈𝐈G,𝐈𝐈B)
` 𝐈𝐈GG= getBinaryImageofColor(𝐈𝐈G,𝐈𝐈B,𝐈𝐈R)
` 𝐈𝐈BB= getBinaryImageofColor(𝐈𝐈B,𝐈𝐈R,𝐈𝐈G)
`For every pixel value 𝐈𝐈k,ij
` 𝐈𝐈CF,ij←𝐈𝐈RR,ij−𝐈𝐈GG,ij
` 𝐈𝐈CF,ij←𝐈𝐈F,ij−𝐈𝐈BB,ij
` If 0.6𝐈𝐈R,ij≥𝐈𝐈G,ij then
` 𝐈𝐈Temp,ij←1
` 𝐈𝐈Temp,ij←0
`𝐈𝐈CF,ij←𝐈𝐈F,ij−𝐈𝐈Temp,ij
`return 𝐈𝐈CF
`
`Algorithm 2. Pseudocode for filtering out the most
`“reddish” pixels in an input RGB image frame.
`
`an object with red tint to human eyes.
`
`B. Shape filter Algorithm
`The target specified in this paper is assumed to be
`a circular shape. The image filtered through the
`procedure described in Section II-A may have some
`noise or small pixel lumps in the frame. They are
`not necessarily exist in the frame and in such a case,
`the algorithm runs the whole shape filter algorithm
`to waste processor resource and make fault
`detection. Following (1) and (2) are applied to
`every pixel of the color-filtered image frame to
`eliminate such noises.
`
`
`𝑰𝑰𝑆𝑆𝑆𝑆1,𝑖𝑖𝑖𝑖= min(𝛥𝛥𝑖𝑖,𝛥𝛥𝑖𝑖)𝑰𝑰𝐶𝐶𝑆𝑆,(𝑖𝑖+𝛥𝛥𝑖𝑖)(𝑖𝑖+𝛥𝛥𝑖𝑖)
`
`𝑰𝑰𝑆𝑆𝑆𝑆2,𝑖𝑖𝑖𝑖=max(𝛥𝛥𝑖𝑖,𝛥𝛥𝑖𝑖)𝑰𝑰𝑆𝑆𝑆𝑆1,(𝑖𝑖+𝛥𝛥𝑖𝑖)(𝑖𝑖+𝛥𝛥𝑖𝑖)
`|𝛥𝛥𝛥𝛥|≤𝑘𝑘 and |𝛥𝛥𝛥𝛥|≤𝑙𝑙 , where 𝛥𝛥𝛥𝛥,𝛥𝛥𝛥𝛥∈ℤ. k and l
`
`
`
`
`
`
`
`
`
`(1)
`(2)
`
`denote sizes of these morphological operations.
`
`Yuneec Exhibit 1022 Page 2
`
`
`
`1167
`
`
`The image is eroded by using (1) to eliminate noise
`and then dilated by using (2) to recover shrunken
`objects to real size in the frame. The size of the
`operation has to be determined carefully because
`our target can be erased if the size is set too large.
`In the next step, contours in the resultant binary
`
`proposed in [6]. Each of the polygon surrounded by
`detected border lines is numbered and temporarily
`saved. Generally, polygons in the image frame are
`checked whether its shape resembles a circle by
`calculating circularity ratio F in (3).
`
`
`
`(3)
`
`
`
`images 𝑰𝑰𝑆𝑆𝑆𝑆2,𝑖𝑖𝑖𝑖 are retrieved using the algorithm
`𝐹𝐹=4𝜋𝜋𝜋𝜋𝑃𝑃2
`0≤𝐹𝐹≤1 is rotation- and scale-invariant. For
`circle, 𝐹𝐹=1.
`The central moment of order (𝑝𝑝+𝑞𝑞) for polygon
`Q is defined
`𝜇𝜇𝑐𝑐𝑝𝑝=1𝜋𝜋�(𝑥𝑥−𝑥𝑥𝑐𝑐)𝑐𝑐
`(𝑦𝑦−𝑦𝑦𝑐𝑐)𝑝𝑝
`𝑥𝑥,𝑦𝑦∈𝑄𝑄
`Here, 𝑥𝑥𝑐𝑐=1𝑆𝑆∑
` and 𝑦𝑦𝑐𝑐=1𝑆𝑆∑
`𝑦𝑦
`𝑥𝑥
`𝑥𝑥,𝑦𝑦∈𝑄𝑄
`𝑥𝑥,𝑦𝑦∈𝑄𝑄
`normalized feature of compactness 𝑀𝑀𝑐𝑐𝑐𝑐𝑐𝑐 and
`eccentricity 𝑀𝑀𝑒𝑒𝑐𝑐𝑒𝑒 are calculated from second-order
`𝑀𝑀𝑐𝑐𝑐𝑐𝑐𝑐= 12𝜋𝜋∙
`𝜋𝜋
`𝜇𝜇20+𝜇𝜇02
`𝑀𝑀𝑒𝑒𝑐𝑐𝑒𝑒=�(𝜇𝜇20+𝜇𝜇02)2−4𝜇𝜇11
`𝜇𝜇20+𝜇𝜇02
`Here, we set 0≤𝑀𝑀𝑐𝑐𝑐𝑐𝑐𝑐≤1 , 0≤𝑀𝑀𝑒𝑒𝑐𝑐𝑒𝑒≤1 and
`𝑀𝑀𝑐𝑐𝑐𝑐𝑐𝑐=1, 𝑀𝑀𝑒𝑒𝑐𝑐𝑒𝑒=0 for the disk. The parameters
`circularity for robustness. Polygons with 𝜇𝜇𝑐𝑐𝑝𝑝>
`0.85, 𝑀𝑀𝑐𝑐𝑐𝑐𝑐𝑐>0.8 and 𝑀𝑀𝑒𝑒𝑐𝑐𝑒𝑒<0.55 are chosen to
`
`Fig. 2. An image from camera (upper) and a filtered
`image (lower). Contours of the red polygons are drawn
`and the detected target is denoted in green lined
`rectangle.
`
`the biggest
`targets. Finally,
`tentative
`detect
`candidate among them is selected as our tracking
`target.
`
`III. TRACKING CONTROL STRATEGY
`A. Quadrotor UAV Platform
`In
`this
`research, a commercial quadrotor
`AR.Drone 2.0 of Parrot Inc. is chosen for flight. A
`forward facing High-Definition camera is equipped
`on this rotary-wing aircraft. It has Wi-Fi access
`point to transmit control commands and receive
`flight data and video via wireless LAN. This
`platform has Attitude and Heading Reference
`System (AHRS) onboard, which indicates it has
`three-axis accelerometers, gyroscopes and a
`magnetometer. One convenient characteristic of the
`aircraft is that its attitude stabilization/control loop
`is implemented so it can hover at the same position.
`However,
`the
`inner stabilization/control
`loop
`cannot be accessed by a user. Instead, the user can
`send roll, pitch, and yaw angle commands to
`control the motion of the quadrotor. The commands
`are not in degrees or radians, instead they are
`floating point values varying between -1 and 1. The
`
`
`The parameters, S and P are an area and perimeter
`of a polygon, respectively. Note that the shape with
`
`The other method to check the circularity of a
`polygon is to use second-order moment of the shape.
`
`
`
`(4)
`
`
`
`
`
` are
`coordinates of a centroid of the shape. Then the
`
`central moments.
`
`
`
`
`
`(5)
`
`
`
`
`calculated from (3) and (5) are both used to check
`
`Yuneec Exhibit 1022 Page 3
`
`
`
`1168
`
`
`
`Fig. 3. AR.Drone of Parrot Inc.
`
`available commands are roll, pitch, yaw rate and
`vertical velocity and these floating point values also
`vary in the range [-1, 1] as well.
`
`
`B. Color Tracking Command Loop
`The camera is attached on the quadrotor and
`aligned with the heading of the quadrotor. Thus, the
`main strategy to track the target from a fixed
`distance is to move the quadrotor so that the
`detected circle appear at the center of the camera
`image with a specified radius. This control strategy
`is formulated into the following three steps.
`First step is to control pitch angle by calculating
`
`the difference of the radii of the shape (𝑟𝑟𝑐𝑐−𝑟𝑟𝑜𝑜),
`where 𝑟𝑟𝑐𝑐=�𝜋𝜋/2𝜋𝜋 is a radius of the specified
`target and ro is the reference radius. The quadrotor
`have to pitch down(up) if (𝑟𝑟𝑐𝑐−𝑟𝑟𝑜𝑜) is less (greater)
`A horizontal position difference (𝑥𝑥𝑐𝑐−𝑥𝑥𝑜𝑜) of the
`
`than zero to maintain uniform distance to the target.
`
`centroid of the target polygon with respect to center
`of the image frame generates an angular velocity
`
`Fig. 4. Illustration of the target in the image frame.
`
`Fig. 5. Control Procedure of AR.Drone from filtered
`
`image. 𝐮𝐮𝐨𝐨, 𝐮𝐮𝐫𝐫̇ and 𝐮𝐮𝒛𝒛̇𝑩𝑩 are pitch, heading rate and vertical
`
`velocity commands, respectively.
`
`input command in yaw direction. The aircraft
`should turn right if the value is greater than zero
`and vice versa.
`The last step to track the red circle in the image
`frame is to move vertically to make vertical
`
`Position difference (𝑧𝑧𝑐𝑐−𝑧𝑧𝑜𝑜) to zero. This is
`
`handled by sending proper vertical velocity
`command to the quadrotor according to a sign and
`absolute value of difference.
`A proportional-differential controller is designed
`to process the whole three steps described so far.
`Since
`the control
`inputs may
`include high-
`frequency noise due to the reflection of irregular
`illumination, a low pass filter is designed at the
`input side of the controller. Also, the output of the
`controller is limited within the range [-1, 1] of input
`commands to the drone. This whole procedure is
`described in Fig. 5.
`
`
`IV. SYSTEM DESCRIPTION
`A. The iOS Application Layout
`The AR.Drone 2.0 is compatible with the iPad, a
`tablet computer of Apple Inc. The 4th generation
`iPad
`is used for running
`the whole
`image
`processing algorithms and the control algorithm
`proposed in this paper. The program has a form of
`a view-based iOS application implemented with
`Objective-C.
`Screen layout of the application in Fig. 6 includes
`two screen outputs, an acquired image frame from
`
`the quadrotor (①), and a processed image frame of
`the previous one (②). Several buttons are provided
`and landing, etc. (③, ⑤) Small interface under
`gains of the proposed PD controller(④). By
`
`for initializing wireless LAN connection to the
`access point, starting the image processing, take-off
`
`processed image frame is for tuning the control
`
`dragging a round button at the slider interface, the
`
`Yuneec Exhibit 1022 Page 4
`
`
`
`1169
`
`
`
`can handle this procedure. This framework is built
`for armv7s, the architecture of iPad A6X processor.
`It is acceptable that the image processing
`algorithms used in this paper are relatively simple
`and common. Recently, open-source
`image
`processing framework called OpenCV became very
`popular for their powerful and easy-to-use image
`acquisition and processing routines. An
`iOS
`version of OpenCV framework is used together
`with FFMPEG framework in this paper.
`
`
`Fig. 6. The Interface of an iPad application (Left). UAV and a tablet computer application system architecture (Right).
`
`corresponding controller gain value changes and
`applied when “Apply” button pressed. This
`function prevents users from wasting time to tune
`the gain values by recompiling, rebuilding, and
`downloading whole application every time.
`
`B. Application Architecture
`The main thread of an iOS application is in charge
`of operating touch interfaces and updating view
`events. Since this thread is busy interacting with the
`operator, another thread is added to handle image
`data. The
`“image-processing
`thread”
`is
`implemented with Grand Central Dispatch (GCD),
`which is a technology developed by Apple Inc. to
`optimize support of an application for systems with
`multi-core processors and other multiprocessing
`systems. While
`distributing
`heavy
`image
`processing to dual-core A6X chips and quad-core
`graphics, the application can run much faster.
`Since image processing is still heavy for latest
`tablet computers, the image is shrunken to a half of
`its original resolution. This results in a notable
`performance improvement, close to 2.5 times faster
`than before.
`The application communicates with the quadrotor
`using wireless Ethernet interface. Data received
`from the aircraft is HD camera video stream that
`
`V. FLIGHT TESTS
`Indoor experiments are performed to avoid
`disturbance of gust and irregular illumination of the
`sun. AR.Drone 2.0 and 4th generation iPad are used
`for this research.
`A. Fixed Target Tracking Test
`A red circle was attached on the wall to serve as
`our
`fixed
`target. The quadrotor was
`then
`commanded to detect the circle, and track with
`
`has 640×360 resolution encoded in H.264 codec.
`
`This has to be decoded to convert it into UIImage,
`which is an image object format to show image in
`iOS devices. An open-source FFMPEG framework
`
`Fig. 7. Fixed target tracking test
`
`Yuneec Exhibit 1022 Page 5
`
`
`
`1170
`
`
`
`Fig. 9. Moving target tracking test
`
`
`VI. CONCLUSION
`This paper introduced a target tracking algorithm
`for a commercially available quadrotor whose
`camera is facing frontward. A set of algorithms are
`developed for detecting the target based on the
`shape and color appearing on the image. The target
`shape is classified using the circularity and the
`moment of the polygon in the image frame.
`After detecting and locating the desired target in
`the image frame, the quadrotor is commanded to
`track the target through the provided user interface
`of the quadrotor control software. The strategy of
`the algorithm was to maintain its relative position
`with respect to the target using measurements from
`captured image frame.
`These procedures were implemented as an
`application for a tablet computer. The tablet
`computer was configured to communicate with the
`aircraft platform over a wireless LAN. It could
`receive streaming video from
`the quadrotor,
`process it and send tracking commands via wireless
`LAN.
`The application and the quadrotor system was
`tested with both fixed and moving target. As a
`consequence, it found the target and followed very
`well.
`The proposed target observation and tracking
`algorithm can be used to follow or avoid specific
`objects for rotary-wing UAVs which perform low-
`altitude missions. Furthermore, this paper showed
`the possibilities to use commercially available
`
`Fig. 8. Result graph from fixed target tracking test.
`
`proposed the algorithms presented in this paper.
`The result from the experiments are given in Fig. 8.
`The quadrotor was guided to have zero errors in the
`target’s position and size in image frame. The
`vehicle converged to the steady state after 10
`seconds in this graph.
`
`B. Moving Target Tracking Test
`With using the same control gains in previous
`chapter, moving target was also tested in the
`identical experiment environment. An assisting
`person held the target, a paper with a red circle and
`showed it to the quadrotor. Then he moved around
`while showing the paper to the aircraft. The vehicle
`could follow the walking person with the target
`smoothly without any difficulty.
`
`
`
`Yuneec Exhibit 1022 Page 6
`
`
`
`1171
`
`
`products such as the quadrotor and the tablet PC as
`a viable UAV and a potent computing platform for
`realistic application.
`
`ACKNOWLEDGMENT
`This work was supported by National Research
`Foundation of Korea (NRF) grant funded by the
`Korean government (Ministry of Science, ICT &
`Future Planning)
`(No. 20110015377 & No. 2012033464).
`
`
`
`
`
`
`Yuneec Exhibit 1022 Page 7
`
`
`
`1172
`
`[10] A. Proctor, E. Johnson and T. Apker, “Vision-
`only Control and Guidance for Aircraft,”
`Journal of Field Robotics, Vol. 23, No. 10,
`2006.
`[11] J. Lai, L. Mejias and J. J. Ford, “Airborne
`Vision-Based Collision-Detection System”,
`Journal of Field Robotics, Vol. 28, No. 2, 2011
`[12] E. Altug, J. Ostrowski and R. Mahony,
`“Control of a quadrotor helicopter using visual
`feedback,” IEEE International Conference on
`Robotics and Automation, May 2002
`AR.Drone
`[13] Parrot
`http://ardrone2.parrot.com/
`[14] P.-J. Bristeau, F. Callou, D. Vissiere, and N.
`Petit, “The Navigation and Control Technology
`Inside the AR.Drone Micro UAV,” 18th IFAC
`World Congress, Milano, Italy, 2011, pp. 1477–
`1484
`
`2,
`
`
`
`REFERENCES
`[1] S.G. Fowers, D.-J. Lee, B.J. Tippetts, K.D.
`Lillywhite, A.W. Dennis and J.K. Archibald,
`“Vision
`aided
`stabilization
`and
`the
`development of a quad-rotor micro UAV,”
`International Symposium of Computational
`Intelligence in Robotics and Automation, pp.
`143-148, June 20-23, 2007.
`[2] S. Azrad, F. Kendoul and K. Nonami, “Visual
`Servoing of Quadrotor Micro-Air Vehicle
`Using Color-Based Tracking Algorithm,”
`Journal of System Design and Dynamics, Vol.
`4, No. 2, pp. 255-268, 2010.
`[3] C. Teuliere, L. Eck and E. Marchand, “Chasing
`a moving target from a flying UAV,” IEEE/RSJ
`International Conference on Intelligent Robots
`and Systems, pp.4929-4934, Sep 25-30. 2011.
`[4] M. Blosch, S. Weiss, D. Scaramuzza and R.
`Siegwart, “Vision Based MAV Navigation in
`Unknown and Unstructured Environments,”
`IEEE International Conference on Robotics
`and Automation, pp.21-28, May 3-8. 2010.
`[5] S. Ahrens, D. Levine, G. Andrews, and J. P.
`How, “Vision-Based Guidance and Control of
`a Hovering Vehicle in Unknown, GPS-denied
`Environments,”
`IEEE
`International
`Conference on Robotics and Automation, pp.
`2643-2648, May 12-17. 2009.
`[6] Suzuki, S. and Abe, K., “Topological Structural
`Analysis of Digitized Binary Images by Border
`Following”. Computer Vision Graphics Image
`Process Vol. 1, No. 30, pp. 32-46, 1985.
`[7] S. Huh and D. H. Shim, “A Vision-Based
`Landing System for Small Unmanned Aerial
`an Airbag”, Control
`Vehicles
`using
`Engineering Practice, Vol. 18, Issue 7, pp.812-
`823, 2010.
`[8] Sungwook Cho, Sungsik Huh, David Hyunchul
`Shim, Hyoung Sik Choi, “Vision-Based
`Detection and Tracking of Airborne Obstacles
`in a Cluttered Environment”, Journal of
`Intelligent and Robotic Systems, Vol. 69, Issue
`1-4, pp. 475-488, Jan. 2013.
`[9] S. Saripalli, J. Montgomery and G. Sukhtame,
`“Visually-Guided Landing of an Unmanned
`IEEE Transactions on
`Aerial Vehicle,”
`Robotics and Automation, Vol. 19, No. 3, pp.
`371-381, June 2003.
`
`Yuneec Exhibit 1022 Page 8