In the last few years, consumer Graphics Processor Units (GPUs) have been evolving from fixed-function display generators into general purpose parallel computers. We have explored the potential uses and limitations of this emerging technology as a video coprocessor for real-time image processing applications such as video enhancement, tracking, video stabilisation and multi-sensor fusion. We show how a GPU can be used to implement and accelerate some of these common tasks and show our results. We also address the problem of integrating a GPU into a rugged system in order to deploy this capability into the environments encountered in many defence and security applications.
A significant capability of unmanned airborne vehicles (UAV's) is that they can operate tirelessly and at maximum
efficiency in comparison to their human pilot counterparts. However a major limiting factor preventing ultra-long
endurance missions is that they require landing to refuel. Development effort has been directed to allow UAV's to
automatically refuel in the air using current refueling systems and procedures. The 'hose & drogue' refueling system
was targeted as it is considered the more difficult case. Recent flight trials resulted in the first-ever fully autonomous
airborne refueling operation.
Development has gone into precision GPS-based navigation sensors to maneuver the aircraft into the station-keeping
position and onwards to dock with the refueling drogue. However in the terminal phases of docking, the accuracy of the
GPS is operating at its performance limit and also disturbance factors on the flexible hose and basket are not predictable
using an open-loop model. Hence there is significant uncertainty on the position of the refueling drogue relative to the
aircraft, and is insufficient in practical operation to achieve a successful and safe docking.
A solution is to augment the GPS based system with a vision-based sensor component through the terminal phase to
visually acquire and track the drogue in 3D space. The higher bandwidth and resolution of camera sensors gives
significantly better estimates on the state of the drogue position. Disturbances in the actual drogue position caused by
subtle aircraft maneuvers and wind gusting can be visually tracked and compensated for, providing an accurate
estimate.
This paper discusses the issues involved in visually detecting a refueling drogue, selecting an optimum camera
viewpoint, and acquiring and tracking the drogue throughout a widely varying operating range and conditions.
Distributed aperture sensor (DAS) systems can enhance the situational awareness of operators in both manned and
unmanned platforms. In such a system, images from multiple sensors must be registered and fused into a seamless
panoramic mosaic in real time, whilst being displayed with very low latency to an operator. This paper describes an
algorithm for solving the multiple-image alignment problem and an architecture that leverages the power of consumer
graphics processing units (GPU) to provide a live panoramic mosaic display. We also describe other developments
aimed at integrating high resolution imagery from an independently steerable fused TV/IR sensor into the mosaic,
panorama stabilisation and automatic target detection.
Distributed Aperture Sensor (DAS) systems employ multiple sensors to obtain high resolution, wide angle video coverage of their local environment in order to enhance the situational awareness of manned and unmanned platforms. The images from multiple sensors must be presented to an operator in an intuitive manner and with minimal latency if they are to be rapidly interpreted and acted upon. This paper describes a display processor that generates a real-time panoramic video mosaic from multiple image streams and the algorithms for calibrating the image alignments. The architecture leverages the power of commercial graphics processing units (GPUs) in order to accelerate the image warping and display rendering, providing the operator with real-time virtual environment viewed through a virtual camera. The possibility of integrating high resolution imagery from a zoom sensor on a pan-tilt mount directly into the mosaic, introducing a 'foveal' region of high fidelity into the panoramic image is also possible.
Optical flow fields can be used to recover some components of the camera ego-motion such as velocity and angular velocity. In this paper, we discuss the use of optical flow fields to estimate the relative orientation of two imagers with non-overlapping fields of view. The algorithms proposed are based on a spherical alignment technique which is closely related to rapid transfer alignment methods used to align aircraft inertial navigation systems. Of particular importance is the relationship between the accuracy of the optical flow field (which is dependent upon the complexity of the scene and the resolution of the cameras) and the accuracy of the resultant alignment process.
Many Command-to-Line-of-Sight missile systems use ground-based electro-optic sensors to track their targets. Both optical and Infra-Red systems can be affected by launch effects, which can include camera shake on launch and target obscuration due to the missile exhaust plume. Further effects can be encountered during flight including aimpoint disturbance, launch debris and countermeasures.
An automatic video tracking system (AVT) is required to cope with all of these distractions, whilst maintaining track on the primary target. If track is broken during the engagement, the AVT needs to employ a strategy that will enable reacquisition of the primary target with the minimum of delay. This task can be significantly more complicated in a cluttered scene.
This paper details such a reacquisition algorithm, the primary purpose of which is to correctly identify the primary target whilst reducing the reacquisition timeline. Results are presented against synthetic imagery and actual missile firings.
Multiple camera systems have been considered for a number of applications, including infrared (IR) missile detection in modern fast jet aircraft, and soldier-aiding data fusion systems. This paper details experimental work undertaken to test image-processing and harmonisation techniques that were developed to align multiple camera systems. This paper considers systems where the camera properties are significantly different and the camera fields of view do not necessarily overlap. This is in contrast to stereo calibration alignment techniques that rely on similar resolution, fields of view and overlapping imagery. Testing has involved the use of two visible-band cameras and attempts to harmonise a narrow field of view camera with a wide field of view camera. In this paper, consideration has also been given to the applicability of the algorithms to both visual-band and IR based camera systems, the use of supplementary motion information from inertial measurement systems and consequent system limitations.
The ability to automatically detect and track moving targets whilst stabilizing and enhancing the incoming video would be highly beneficial in a range of aerial reconnaissance scenarios. We have implemented a number of image-processing algorithms on our ADEPT hardware to perform these and other useful tasks in real-time. Much of this functionality is currently being migrated onto a smaller PC104 form-factor implementation that would be ideal for UAV applications. In this paper, we show results from both software and hardware implementations of our current suite of algorithms using synthetic and real airborne video. We then investigate an image processing architecture that integrates mosaic formation, stabilisation and enhancement functionality using micro-mosaics, an architecture which yields benefits for all the processes.
Many modern imaging and surveillance systems contain more than one sensor. For example, most modern airborne imaging pods contain at least visible and infrared sensors. Often these systems have a single display that is only capable of showing data from either camera, and thereby fail to exploit the benefit of having simultaneous multi-spectral data available to the user. It can be advantageous to capture all spectral features within each image and to display a fused result rather than single band imagery. This paper discusses the key processes necessary for an image fusion system and then describes how they were implemented in a real-time, rugged hardware system. The problems of temporal and spatial misalignment of the sensors and the process of electronic image warping must be solved before the image data is fused. The techniques used to align the two inputs to the fusion system are described and a summary is given of our research into automatic alignment techniques. The benefits of different image fusion schemes are discussed and those that were implemented are described. The paper concludes with a summary of the real-time implementation of image alignment and image fusion by Octec and Waterfall Solutions and the problems that have been encountered and overcome.
A number of algorithms including moving target detection, video stabilisation and image enhancement have been described in the literature as useful in aerial reconnaissance scenarios. These algorithms are often described in isolation and require a base station for off-line processing. We consider the problem of designing a single image processing architecture capable of supporting these and other useful tasks in an embedded real-time system such as a semi-autonomous UAV.
This paper describes our current algorithm suite and a versatile new architecture in development based on the formation of mosaic images. We show how these mosaics can be generated in real-time through fast image registration techniques and then exploited to accomplish typical aerial reconnaissance tasks. We also illustrate how they can be used to compress the video sequence.
We show results from synthetic and real video using both software and hardware implementations. Our embedded hardware solution, its current algorithm suite and future developments are discussed.
Many modern imaging and surveillance systems contain more than one sensor. For example, most modern airborne imaging pods contain at least visible and infrared sensors. Often these systems have a single display that is only capable of showing data from either camera, and thereby fail to exploit the benefit of having simultaneous multi-spectral data available to the user. It can be advantageous to capture all spectral features within each image and to display a fused result rather than single band imagery. This paper discusses the key processes necessary for an image fusion system and then describes how they were implemented in a real-time, rugged hardware system. The problems of temporal and spatial misalignment of the sensors and the process of electronic image warping must be solved before the image data is fused. The techniques used to align the two inputs to the fusion system are described and a summary is given of our research into automatic alignment techniques. The benefits of different image fusion schemes are discussed and those that were implemented are described. The paper concludes with a summary of the real-time implementation of image alignment and image fusion by Octec and Waterfall Solutions and the problems that have been encountered and overcome.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.