To overcome the difficulties in tracking flying targets in the sky under different conditions, a combined scheme is proposed to get a long-term tracklet: (1) point object is efficiently detected via a fast hypothesis test for background extraction, and an improved correlation filtering algorithm is utilized for possibly near object with much texture; (2) tracker is initialized and managed by estimating continuous motion using an acceleration Kalman filter; and (3) prior knowledge of object is further incorporated to remove false object in tracklet association process. Outdoor experiments prove that the proposed techniques improve the accuracy and reliability for target objects undergoing significant appearance variation due to cloud shift, abrupt motion, and temporary occlusion, and it also extends the validity of our strategy for further valuable applications.