In visual tracking, sometimes the target response value is high, but it is not the tracking result, which can result in the wrong judgment. Moreover, the threshold to decide the tracking result needs to be set artificially in the traditional discriminative methods. We propose a deep learning-based target drift discriminative network to judge whether the target is lost. We design a lightweight network without the threshold, using four convolutional layers, three full connection layers, and the Softmax function to judge the tracking results. When training the network, the established positive and negative samples are used, and we select difficult samples for further training to achieve a better target discriminative effect. Finally, a target drift discriminative network is introduced into the accurate tracking by overlap maximization. When it is judged that the target is lost, another search area is selected to quickly find the target. Numerous experiments show that our method achieves the best performance on datasets UAV123, UAV20L, and VOT2018-LT, especially on the UAV20L dataset, for which the tracking precision and tracking success rate are improved by 3.7% and 2.8%. Compared with several other classical threshold discriminative criteria, we do not need to set the threshold artificially and have better judgment performance.