Label-free tissue identification is the new frontier of image guided surgery. One of the most promising modalities is hyperspectral imaging (HSI). Until now, the use of HSI has, however, been limited due to the challenges of integration into the existing clinical workflow. Research to reduce the implementation effort and simplifying the clinical approval procedure is ongoing, especially for the acquisition of feasibility datasets to evaluate HSI methods for specific clinical applications. Here, we successfully demonstrate how an HSI system can interface with a clinically approved surgical microscope making use of the microscope’s existing optics. We outline the HSI system adaptations, the data pre-processing methods, perform a spectral and functional system level validation and integration into the clinical workflow. Data were acquired using an imec snapscan VNIR 150 camera enabling hyperspectral measurement in 150 channels in the 470-900 nm range, assembled on a ZEISS OPMI Pentero 900 surgical microscope. The spectral range of the camera was adapted to match the intrinsic illumination of the microscope resulting in 104 channels in the range of 470-787 nm. The system’s spectral performance was validated using reflectance wavelength calibration standards. We integrated the HSI system into the clinical workflow of a brain surgery, specifically for resections of low-grade gliomas (LGG). During the study, but out of scope of this paper, the acquired dataset was used to train an AI algorithm to successfully detect LGG in unseen data. Furthermore, dominant spectral channels were identified enabling the future development of a real-time surgical guidance system.
This paper presents the latest advances on Imec snapshot multispectral imagers based on either 3x3, 4x4 and 5x5 mosaic filter patterning on industry ready VIS/NIR and SWIR detectors. The mosaic patterns are implemented by means of high-transmission Fabry-Pérot interferometers processed using thin-film technology. Our snapshot multispectral imagers offer a spatial resolution of 640x480 pixels (SWIR) and 2048x1088 (VIS/NIR) down sampled according to the mosaic pattern to acquire data in nine (3x3), 16 (4x4) of 25 (5x5) spectral bands respectively. To achieve imaging at the native spatial resolution of the sensor, super resolution methods are available post-acquisition. Moreover, our compact USB-3 cameras of 260 gr (SWIR) and 27 gr (VNIR), without lens, reach an acquisition speed of up to 120 multispectral cubes/second and are therefore suitable for dynamic applications, high-speed inspection such as a conveyor belt or UAV inspection. The potential for snapshot cameras in a wide range of applications is showcased in this paper. We first show how applications on industrial quality inspection (chocolate gloss estimation) and precision agriculture (plant disease detection) achieve good discrimination potential in the VNIR range. Specifically, UAV inspection benefits from our compact camera size, low weight, and video capabilities. We then demonstrate the potential for plastic and textile recycling in the SWIR range and the benefit brought by both VNIR and SWIR ranges for people tracking under low visibility conditions. Finally, an application involving the joint use of a microscope and a multispectral camera system is presented for particle contamination exposure assessment. The suitable range is in this case application dependent.
KEYWORDS: Cameras, Imaging systems, Sensors, Unmanned aerial vehicles, Control systems, System integration, Data processing, Visualization, Data acquisition, UAV imaging systems
Multispectral imaging technology analyzes for each pixel a wide spectrum of light and provides more spectral
information compared to traditional RGB images. Most current Unmanned Aerial Vehicles (UAV) camera systems are
limited by the number of spectral bands (≤10 bands) and are usually not fully integrated with the ground controller to
provide a live view of the spectral data.
We have developed a compact multispectral camera system which has two CMV2K 4x4 snapshot mosaic sensors
internally, providing 31 bands in total covering the visible and near-infrared spectral range (460-860nm). It is compatible
with (but not limited to) the DJI M600 and can be easily mounted to the drone. Our system is fully integrated with the
drone, providing stable and consistent communication between the flight controller, the drone/UAV, and our camera
payload. With our camera control application on an Android tablet connected to the flight controller, users can easily
control the camera system with a live view of the data and many useful information including histogram, sensor
temperature, etc. The system acquires images at a maximum framerate of 2x20 fps and saves them on an internal storage
of 1T Byte. The GPS data from the drone is logged with our system automatically. After the flight, data can be easily
transferred to an external hard disk. Then the data can be visualized and processed using our software into single
multispectral cubes and one stitched multispectral cube with a data quality report and a stitching report.
Multispectral and hyperspectral cameras have long been large and expensive lab instruments which need to be operated by expert users. At imec, our mission has always been to build compact, affordable and easy-to-use systems for high fidelity spectral imaging. In this presentation, we will explain our latest developments in data processing. This already opened the door to many new applications like outdoor surveillance, orchard inspection, UAV, medical microscopy, under water imaging, ... with high spectral radiance and reflectance accuracy.
KEYWORDS: Cameras, Line scan image sensors, Sensors, Image sensors, Photonic devices, Image processing, Hyperspectral imaging, CMOS sensors, Signal to noise ratio, Imaging systems, High dynamic range imaging, Data acquisition
Imec has developed a process for the monolithic integration of optical filters on top of CMOS image sensors, leading to compact, cost-efficient and faster hyperspectral cameras. Linescan cameras are typically used in remote sensing or for conveyor belt applications. Translation of the target is not always possible for large objects or in many medical applications. Therefore, we introduce a novel camera, the Snapscan (patent pending), exploiting internal movement of a linescan sensor enabling fast and convenient acquisition of high-resolution hyperspectral cubes (up to 2048x3652x150 in spectral range 475–925 nm). The Snapscan combines the spectral and spatial resolutions of a linescan system with the convenience of a snapshot camera.
Imec has developed a process for the monolithic integration of optical filters on top of CMOS image sensors, leading to compact, cost-efficient and faster hyperspectral cameras. Different prototype sensors are available, most notably a 600- 1000 nm line-scan imager, and two mosaic sensors: a 4x4 VIS (470-620 nm range) and a 5x5 VNIR (600-1000 nm). In response to the users’ demand for a single sensor able to cover both the VIS and NIR ranges, further developments have been made to enable more demanding applications. As a result, this paper presents the latest addition to imec’s family of monolithically-integrated hyperspectral sensors: a line scan sensor covering the range 470-900 nm. This new prototype sensor can acquire hyperspectral image cubes of 2048 pixels over 192 bands (128 bands for the 600- 900 nm range, and 64 bands for the 470-620 nm range) at 340 cubes per second for normal machine vision illumination levels.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.