Visual ship detection is a crucial component in recognition systems for detecting objects at different scales. However, there remains a challenge in detecting multiscale ships with the influence of various sizes of ships and complex backgrounds. We propose an efficient multilayer attention receptive fusion network (MARN) that is based on the well-established YOLOv4. MARN is a one-stage algorithm that essentially extracts semantic information from feature maps at different scales to highlight salient features of ships, thereby improving the detection performance of multiscale ships. Moreover, a multilayer information interactive fusion module (MIFM) and attention receptive field block (ARFB) are applied and combined reasonably to build a bidirectional fine-grained feature pyramid. Specifically, MIFM is used to fuse features at different scales, which not only concatenates high-level semantic features from deep layers but also reshapes richer features from shallower layers. Meanwhile, ARFB consists of a receptive field block, spatial attention, and channel attention, effectively emphasizing the most important features and suppressing unnecessary ones. Finally, experiments on the Singapore maritime dataset demonstrate that MARN can effectively solve the detection problem of multiscale ships in the complex marine environment.