It is always a challenging task to detect a small target with low signal-to-noise ratio under complex background in infrared images. To address this problem, an effective algorithm based on background subtraction is proposed. First, we add the gradient feature into the kernel regression model to acquire an edge-preserving background estimation. The smoothing matrix of the kernel function is reestablished by a rotation angle and an elongation scale. Further, a multiscale first-order directional derivative filter is presented to calculate these factors adaptively. Second, to segment the real target from the subtracted image, we model the imaging process of the small target using the point spread function of the optical system. According to the analysis of the imaging size and the energy distribution of target, an energy concentration criterion is constructed and used for target extraction. Finally, comparison of experimental results demonstrates that the proposed algorithm achieves robust performances on background suppression and extracts the target accurately with a high detection probability and low false alarm rate.