Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Multispectral single chip reconstruction using DNNs with application to open neurosurgery

Multispectral single chip reconstruction using DNNs with application to open neurosurgery DE GRUYTER Current Directions in Biomedical Engineering 2021;7(2): 37-40 Stephan Göb*, Theresa Ida Götz, Thomas Wittenberg Multispectral single chip reconstruction using DNNs with application to open neurosurgery Abstract: Multispectral imaging devices incorporating up to Keywords: Spectral Reconstruction, Debayering, 256 different spectral channels have recently become avail- Demosaicing, DCNN, open neurosurgery able for various healthcare applications, as e.g. laparoscopy, https://doi.org/10.1515/cdbme-2021-2010 gastroscopy, dermatology or perfusion imaging for wound analysis. Currently, the use of such devices is limited due to very high investment costs and slow capture times. To compensate these shortcomings, single sensors with spectral 1 Introduction masking on the pixel level have been proposed. Hence, adequate spectral reconstruction methods are needed. Within Multispectral imaging (MSI) or hyperspectral imaging (HSI) this work, two deep convolutional neural networks (DCNN) devices incorporating nine, sixteen or even 128 or 256 diffe- architectures for spectral image reconstruction from single rent spectral channels across wavelengths within and beyond sensors are compared with each other. Training of the net- the visual range have in the past years become available for works is based on a huge collection of different MSI image- various applications in the field of healthcare and biomedicine. stacks, which have been subsampled, simulating 16-channel Some typical clinical examples for such applications based on single sensors with spectral masking. We define a training, MSI and HSI are e.g., diagnostic laparoscopy [1] or gastrosco- validation and test set (‘HITgoC’) resulting in 351 training py [2], screening dermatology [3], or perfusion imaging for (631.128 sub-images), 99 validation (163.272 sub-images) wound analysis [4], or open neurosurgery [6,7]. An overview and 51 test images. For the application in the field of of the possibilities for MSI and HSI possibilities for digital and neurosurgery an additional testing set of 36 image stacks computational pathology has recently been presented by Orte- from the Nimbus data collection is used, depicting MSI brain ga et al. [5]. data during open surgery. Two DCNN architectures were Nevertheless, currently routine use of such MSI or HSI compared to bilinear interpolation (BI) and an intensity devices is limited due to very high investment-costs as well as difference (ID) algorithm. The DCNNs (ResNet-Shinoda) slow capture times, as e.g., a brush-broom sensor with 256 were trained on HITgoC and consist of a preprocessing step spectral lines has to be moved mechanically over the field of using BI or ID and a refinement part using a ResNet struc- view while scanning and concatenating the different spectral ture. Similarity measures used were PSNR, SSIM and MSE data. Similar, steerable spectral filters need time to integrate a between predicted and reference images. We calculated the sufficient number of spectral images. In order to compensate similarity measures for HitgoC and Nimbus data and determi- these named shortcomings, alternative approaches have re- ned differences of the mean similarity measure values achie- cently proposed. One suggested method is based on spectral ved with the ResNet-ID and baseline algorithms such as BI masking on the pixel level using only one single sensor plane algorithm and ResNet-Shinoda. The proposed method achie- (similar to the well-known Bayer pattern for single-chip RGB ved better results against BI in SSIM (.0644 vs. .0252), cameras) and successive interpolation of the missing spectral PSNR (15.3 dB vs. 9.1 dB) and 1-MSE*100 (.0855 vs. values based on image processing or image analysis approa- .0273) and compared to ResNet-Shinoda in SSIM (.0103 vs. ches. Typical such image processing methods for spectral re- .0074), PSNR (3.8 dB vs. 3.6 dB) and 1-MSE*100 (.0075 vs. construction include BI and nonlinear filters. The advantages .0047) for HITgoC/Nimbus. In this study, significantly better of such filters are fast and real-time implementation as well as results for spectral reconstruction in MSI images of open the independence from any application. However, if a spectral neurosurgery was achieved using a combination of ID-inter- reconstruction with a focus on specific applications is needed, polation and ResNet structure compared to standard methods. higher-level reconstruction approaches are desirable. To this end, recently deep convolutional neural networks (DCNN) have been suggested. ______ *Corresponding author: Stephan Göb: Fraunhofer Institute for Integrated Circuits IIS, Am Wolfsmantel 33, Erlangen, Germany, goebsn@iis.fraunhofer.de, T Götz, T Wittenberg, Fraunhofer IIS Open Access. © 2021 The Author(s), published by De Gruyter. This work is licensed under the Creative Commons Attribution 4.0 International License. 37 Multispectral single chip reconstruction using DNNs with application to open neurosurgery images. We refer to this data collection as ‘HITgoC’. Additio- nally, the Nimbus [6,7] data set was used to test the proposed DCNN, including 36 images from open neurosurgery (see Fig. 1). All datasets have different local resolutions and a spectral resolution in the range of 400-1000 nm, with approximately Figure 1: Images from the hyperspectral Nimbus data collection every 1 nm occupied. [6,7] as synthetic red-green-blue (RGB) representation. To work with HSI data of the HITgoC and Nimbus data sets, a multi spectral filter array (MSFA) had to be simulated. Within this work, two DCNN architectures with the goal Thus, we selected 16 wavelengths in the range of 480-630 nm of spectral reconstruction from single sensor MSI are with a delta of 10 nm from the HSI data to yield MSI images. compared with each other, and furthermore some extensions The used wavelengths and the pattern of the MSFA are shown for improvements are introduced and evaluated. The training in Figure 2. During pre-processing the training data consisted of the networks is based on a large collection of different MSI of MSFA data and related known original spectral channels. image-stacks, which have been subsampled, simulating a 16- For the analytical algorithms, we implemented BI and channel spectral pixel. For testing and evaluation of the ID suggested by Mihoubi et al. [8]. Furthermore, we proposed approach a set of 36 image stacks (and 572 sub- implemented a DCNN architecture suggested by the Shinoda images) from open neurosurgery is used, related to the et al. [10] and trained it on the joined HITgoC data collection. hyperspectral Nimbus data collection [6,7], see Figure 1. This approach (referred to as ‘ResNet-Shinoda’) consists of two parts: A pre-processing step using BI and a refinement step using a ResNet based DCNN structure (Fig. 2), where the 2 Related Work refinement is made up of 5 ResNet blocks (Fig. 3). Procedures for spectral reconstruction can coarsely be di- vided into two groups, namely analytical algorithms and the application of DCNNs. Most cameras make use of standard analytical spectral reconstruction methods, such as BI. Mihoubi et al. [8] have further developed a new algorithm cal- led Intensity Difference (ID). It is based on an analytical algo- rithm to detect the intensity of a multispectral image combined with the weighted bilinear algorithm. The algorithm was eva- luated on the CAVE [9] database and achieved an average im- provement of 1 dB compared to the common bilinear algo- rithm. Shinoda et al. [10] examined the spectral reconstruction process using analytical algorithms for pre-processing and a Figure 2: ResNet-based refinement of ResNet-Shinoda. DCNN-ResNet structure for refinement. The network was trained and tested on the CAVE data set with using 8-fold cross The starting number of filters is 8 and is doubled in each ResNet block except for the last one with 64 filters. The conv- validation. The reconstruction of the 16-band multi spectral layers use a 3 kernel and a ReLU activation function. To filter array (MSFA) with the proposed method improved by 8.47 dB compared to the conventional method using BI. improve this architecture, we replaced the BI in the ResNet- Shinoda by the above-mentioned ID-Interpolation. The new processing pipeline is also shown in Figure 2. 3 Material and Methods In order to train and validate the networks proposed in this work, a large HSI data set was built from different data collections, including CAVE [9], TokyoTech [11], ICVL [12], HyTexiLa [13] and google [14] data sets. This data collection was broken down into 70 percent training, 20 percent validati- on and 10 percent test data, resulting in 351 training (631.128 Figure 3: Processing pipeline for ResNet-Shinoda and ResNet-ID. sub images), 99 validation (163.272 sub images) and 51 test ResNet-Shinoda use the bilinear interpolation (blue) and ResNet- ID use the intensity difference interpolation (red). 38 Multispectral single chip reconstruction using DNNs with application to open neurosurgery To make comparisons between the results of the networks and the analytic algorithms, 3 similarity measures were selec- ted. The image reconstruction quality was determined by the Peak-Signal-to-Noise Ratio (PSNR), the Structural Similarity (SSIM) and the Mean-Squared Error (MSE) of the restored MSFA and MSI data. To analyze differences between the si- milarity measures of the proposed methods, the Welch-t-Test was used which (according to Rasch et al. [15]) is preferred to the Student-t-Test. If the P value is smaller than P< .05 the mean of two samples is seen to be significantly different. Figure 5: SSIM for 16 channel MSFA and the datasets HITgoC and Nimbus for the algorithms BI and ID and the neural networks ResNet-Shinoda and ResNet-ID. 4 Results We calculated the similarity measures for the HitgoC and Nimbus data and determined the differences of the mean similarity measure values achieved with the ResNet-ID and baseline algorithms such as BI and ResNet-Shinoda. The re- sulting similarity measures are of all four methods are present- ed in box plots in Figures 4-6. T-tests are included in the graphs as lines between methods. The Figures are separated in two columns (left HITgoC and right Nimbus). Results for ana- Figure 6: MSE for 16 channel MSFA and the datasets HITgoC and lytical algorithms are separated in the left and for the neuronal Nimbus for the algorithms BI and ID and the neural networks networks in the right of the column. ResNet-Shinoda and ResNet-ID. The proposed method achieved better results against BI in Table 1: PSNR values of beads image from CAVE of BI and the SSIM (.0644 vs. .0252), PSNR (15.34 dB vs. 9.12 dB) and in ResNet-Shinoda according to Shinoda et al.. Additionally, the the 1-MSE*100 (.0855 vs. .0273) as well as compared to the PSNR improvement to BI is evaluated. ResNet-Shinoda in SSIM (.0103 vs. .0074), PSNR (3.87 dB vs. 3.66 dB) and 1-MSE*100 (.0075 vs. .0047) for HITgoC/ Nim- Algorithm PSNR in dB Improvement to BI in dB bus. The proposed network ResNet-ID is significantly better BI 26.81 --- than the compared ResNet-Shinoda, in the SSIM (HITgoC: t = ResNet-Shinoda 32.69 5.88 -3.387, P <.001; Nimbus: t = -5.44, P < .001), in the PSNR (HITgoC: t = -4.34, P <.001; Nimbus: t = -3.74, P < .001) and For qualitative evaluation, single image from CAVE and in the MSE (HITgoC: t = -2.30, P = .0232; Nimbus: t = -3.21, from Nimbus (see Fig. 7) were extracted, and the spectral re- P = .003). Both neural networks delivered significantly better construction for the algorithms was carried out on the images. results than all of the analytical algorithms (see Fig. 4-6). Reconstruction and an error-image were calculated with L1- Norm of ID-interpolation, ResNet-Shinoda and ResNet-ID. PSNRs, the improvement to BI calculated by Shinoda et al. (Table 1) and by us for the beads image are summarized in Table 2. Table 2: PSNR values of beads image from CAVE of BI, the Res- Net-Shinoda and ResNet-ID according to our implementation. Ad- ditionally, the PSNR improvement to BI is evaluated. Figure 4: PSNR for 16 channel MSFA and the datasets HITgoC Algorithm PSNR in dB Improvement to BI in dB and Nimbus for the algorithms BI and ID and the neural networks ResNet-Shinoda and ResNet-ID. BI 23.33 --- ResNet-Shinoda 28.05 4.72 ResNet-ID 31.71 8.38 39 Multispectral single chip reconstruction using DNNs with application to open neurosurgery 6 Conclusion In this study, a new spectral reconstruction method is pre- sented which combines two existing methods from literature, the ID algorithm and a ResNet architecture. Two analytical standard algorithms as well as the neural networks are evaluated in PSNR, MSE and SSIM on the datasets HITgoC and Nimbus. The proposed method – ResNet-ID – performs best in all tests. Acknowledgment The deep learning cluster were funded by the Federal Ministry of Education and Research under the project reference numbers 16FMD01K, 16FMD02 and 16FMD03. Figure 7: Prediction and error image of the reconstructions from the selected picture of CAVE database (error image scaled to .5) References and Nimbus database (error image scaled to .1) for ID-interpolation, [1] Zhang et al.Tissue classification for laparoscopic image un- ResNet-Shinoda and ResNet-ID. derstanding based on multispectral texture analysis. J Med Imaging. 2017 [2] Yoon et al. A clinically translatable hyperspectral endoscopy 5 Discussion system for imaging the gastrointestinal tract. Nat Commun 10, 2019 In this contribution, two analytical algorithms and two [3] Vasefi et al. Hyperspectral and multispectral imaging in der- matology. In: Hamblin MR Pinar A, Gupta GK (eds): Imaging neural networks for spectral reconstruction have been compa- in dermatology. 2016, pp. 187-201 red. In the work by Shinoda et al. a network was validated and [4] Holmer et al. Hyperspectral imaging in perfusion & wound tested on CAVE with 8-fold cross-validation and they yield an diagnostics - methods and algorithms for the determination of average PSNR of 43.05 dB for all 32 images, being an impro- tissue parameters. Biomed Tech 25;63(5):547-556, 2018 [5] Ortega et al. Hyperspectral and multispectral imaging in di- vement of 8.47 dB to BI. By reimplementation of the ResNet- gital and computational pathology: a systematic review. Bio- Shinoda our results of 28.05 dB in PSNR for the beads image med Opt Express. 2020, 11(6):3195-3233. differ from the results of 32.69 dB in PSNR from the original [6] Fabelo et al. Spatio-spectral classification of hyperspectral work (see Tables 1 & 2). In the evaluation of the BI algorithm, images for brain cancer detection during surgical operations. we yielded 23.33 dB in PSNR compared to 26.81 dB from Shi- PLoS ONE 13(3), 2018 [7] Fabelo et al., In-vivo hyperspectral human brain image data- noda. It should be noted that Shinoda's paper does not describe base for brain cancer detection. IEEE Access 7, pp. 39098- exactly how their PSNR was calculated. Presumably, the 39116, 2019 PSNR in Shinoda's paper is not based on the 16 spectral data [8] Mihoubi et al. Multispektral demosaicing using intensity-ba- cube, but on the reconstruction of the RGB image derived by sed spectral correlation. Proc’s Int. Conf. on Image Proces- sing Theory, Tools & Applications (IPTA), pp. 461-6, 2015 CIE D65. Therefore, our values are smaller because we are [9] Yasuma et al., Generalized Assorted Pixel Camera: Post- comparing the entire data cube. While Shinoda achieves an Capture Control of Resolution, Dynamic Range and Spec- improvement of 5.88 dB, we get an improvement with our pro- trum, Tech. Rep., Dept. Comp. Science, Columbia U. 2008. posed method of 8.38 dB over the respective implemented va- [10] Shinoda et al. Deep demosaicking for multispectral filter arrays, in arXiv, 2018 riant of the bilinear interpolation. The smaller improvement of [11] Monno et al., A practical one-shot multispectral imaging sys- 4.77dB in PSNR between BI and our ResNet-Shinoda is due tem using a single image sensor. IEEE Trans. Image to the small number of training epochs, which was chosen due Processing, pp 3048-3059, 2015 to a lack of time and resources. The prediction and the error [12] Arad et al. Sparse Recovery of Hyperspectral Signal from pattern clearly show that the ResNet-ID yields better results, Natual RGB Images, in Europ. Conf. on Comp. Vision, 2016 [13] Khan et al., HyTexiLa: High Resolution Visible and Near i.e., a lower deviation from the original image, especially in Infrared Hyperspectral Texture Images. Sensors 2018 the area of the edges. [14] Prasad et al. Metrics and statistics of frequency of occurren- ce of metamerism in consumer cameras for natural scenes, in J. Optical Soc. America, pp. 1390-1402, 2015 [15] Rasch et al., The two-sample t test: pre-testing its assump- tions does not pay off, Statistical Papers 52, p. 219-31, 2011 http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Current Directions in Biomedical Engineering de Gruyter

Multispectral single chip reconstruction using DNNs with application to open neurosurgery

Loading next page...
 
/lp/de-gruyter/multispectral-single-chip-reconstruction-using-dnns-with-application-SFfG1K20nk

References

References for this paper are not available at this time. We will be adding them shortly, thank you for your patience.

Publisher
de Gruyter
Copyright
© 2021 The Author(s), published by Walter de Gruyter GmbH, Berlin/Boston
eISSN
2364-5504
DOI
10.1515/cdbme-2021-2010
Publisher site
See Article on Publisher Site

Abstract

DE GRUYTER Current Directions in Biomedical Engineering 2021;7(2): 37-40 Stephan Göb*, Theresa Ida Götz, Thomas Wittenberg Multispectral single chip reconstruction using DNNs with application to open neurosurgery Abstract: Multispectral imaging devices incorporating up to Keywords: Spectral Reconstruction, Debayering, 256 different spectral channels have recently become avail- Demosaicing, DCNN, open neurosurgery able for various healthcare applications, as e.g. laparoscopy, https://doi.org/10.1515/cdbme-2021-2010 gastroscopy, dermatology or perfusion imaging for wound analysis. Currently, the use of such devices is limited due to very high investment costs and slow capture times. To compensate these shortcomings, single sensors with spectral 1 Introduction masking on the pixel level have been proposed. Hence, adequate spectral reconstruction methods are needed. Within Multispectral imaging (MSI) or hyperspectral imaging (HSI) this work, two deep convolutional neural networks (DCNN) devices incorporating nine, sixteen or even 128 or 256 diffe- architectures for spectral image reconstruction from single rent spectral channels across wavelengths within and beyond sensors are compared with each other. Training of the net- the visual range have in the past years become available for works is based on a huge collection of different MSI image- various applications in the field of healthcare and biomedicine. stacks, which have been subsampled, simulating 16-channel Some typical clinical examples for such applications based on single sensors with spectral masking. We define a training, MSI and HSI are e.g., diagnostic laparoscopy [1] or gastrosco- validation and test set (‘HITgoC’) resulting in 351 training py [2], screening dermatology [3], or perfusion imaging for (631.128 sub-images), 99 validation (163.272 sub-images) wound analysis [4], or open neurosurgery [6,7]. An overview and 51 test images. For the application in the field of of the possibilities for MSI and HSI possibilities for digital and neurosurgery an additional testing set of 36 image stacks computational pathology has recently been presented by Orte- from the Nimbus data collection is used, depicting MSI brain ga et al. [5]. data during open surgery. Two DCNN architectures were Nevertheless, currently routine use of such MSI or HSI compared to bilinear interpolation (BI) and an intensity devices is limited due to very high investment-costs as well as difference (ID) algorithm. The DCNNs (ResNet-Shinoda) slow capture times, as e.g., a brush-broom sensor with 256 were trained on HITgoC and consist of a preprocessing step spectral lines has to be moved mechanically over the field of using BI or ID and a refinement part using a ResNet struc- view while scanning and concatenating the different spectral ture. Similarity measures used were PSNR, SSIM and MSE data. Similar, steerable spectral filters need time to integrate a between predicted and reference images. We calculated the sufficient number of spectral images. In order to compensate similarity measures for HitgoC and Nimbus data and determi- these named shortcomings, alternative approaches have re- ned differences of the mean similarity measure values achie- cently proposed. One suggested method is based on spectral ved with the ResNet-ID and baseline algorithms such as BI masking on the pixel level using only one single sensor plane algorithm and ResNet-Shinoda. The proposed method achie- (similar to the well-known Bayer pattern for single-chip RGB ved better results against BI in SSIM (.0644 vs. .0252), cameras) and successive interpolation of the missing spectral PSNR (15.3 dB vs. 9.1 dB) and 1-MSE*100 (.0855 vs. values based on image processing or image analysis approa- .0273) and compared to ResNet-Shinoda in SSIM (.0103 vs. ches. Typical such image processing methods for spectral re- .0074), PSNR (3.8 dB vs. 3.6 dB) and 1-MSE*100 (.0075 vs. construction include BI and nonlinear filters. The advantages .0047) for HITgoC/Nimbus. In this study, significantly better of such filters are fast and real-time implementation as well as results for spectral reconstruction in MSI images of open the independence from any application. However, if a spectral neurosurgery was achieved using a combination of ID-inter- reconstruction with a focus on specific applications is needed, polation and ResNet structure compared to standard methods. higher-level reconstruction approaches are desirable. To this end, recently deep convolutional neural networks (DCNN) have been suggested. ______ *Corresponding author: Stephan Göb: Fraunhofer Institute for Integrated Circuits IIS, Am Wolfsmantel 33, Erlangen, Germany, goebsn@iis.fraunhofer.de, T Götz, T Wittenberg, Fraunhofer IIS Open Access. © 2021 The Author(s), published by De Gruyter. This work is licensed under the Creative Commons Attribution 4.0 International License. 37 Multispectral single chip reconstruction using DNNs with application to open neurosurgery images. We refer to this data collection as ‘HITgoC’. Additio- nally, the Nimbus [6,7] data set was used to test the proposed DCNN, including 36 images from open neurosurgery (see Fig. 1). All datasets have different local resolutions and a spectral resolution in the range of 400-1000 nm, with approximately Figure 1: Images from the hyperspectral Nimbus data collection every 1 nm occupied. [6,7] as synthetic red-green-blue (RGB) representation. To work with HSI data of the HITgoC and Nimbus data sets, a multi spectral filter array (MSFA) had to be simulated. Within this work, two DCNN architectures with the goal Thus, we selected 16 wavelengths in the range of 480-630 nm of spectral reconstruction from single sensor MSI are with a delta of 10 nm from the HSI data to yield MSI images. compared with each other, and furthermore some extensions The used wavelengths and the pattern of the MSFA are shown for improvements are introduced and evaluated. The training in Figure 2. During pre-processing the training data consisted of the networks is based on a large collection of different MSI of MSFA data and related known original spectral channels. image-stacks, which have been subsampled, simulating a 16- For the analytical algorithms, we implemented BI and channel spectral pixel. For testing and evaluation of the ID suggested by Mihoubi et al. [8]. Furthermore, we proposed approach a set of 36 image stacks (and 572 sub- implemented a DCNN architecture suggested by the Shinoda images) from open neurosurgery is used, related to the et al. [10] and trained it on the joined HITgoC data collection. hyperspectral Nimbus data collection [6,7], see Figure 1. This approach (referred to as ‘ResNet-Shinoda’) consists of two parts: A pre-processing step using BI and a refinement step using a ResNet based DCNN structure (Fig. 2), where the 2 Related Work refinement is made up of 5 ResNet blocks (Fig. 3). Procedures for spectral reconstruction can coarsely be di- vided into two groups, namely analytical algorithms and the application of DCNNs. Most cameras make use of standard analytical spectral reconstruction methods, such as BI. Mihoubi et al. [8] have further developed a new algorithm cal- led Intensity Difference (ID). It is based on an analytical algo- rithm to detect the intensity of a multispectral image combined with the weighted bilinear algorithm. The algorithm was eva- luated on the CAVE [9] database and achieved an average im- provement of 1 dB compared to the common bilinear algo- rithm. Shinoda et al. [10] examined the spectral reconstruction process using analytical algorithms for pre-processing and a Figure 2: ResNet-based refinement of ResNet-Shinoda. DCNN-ResNet structure for refinement. The network was trained and tested on the CAVE data set with using 8-fold cross The starting number of filters is 8 and is doubled in each ResNet block except for the last one with 64 filters. The conv- validation. The reconstruction of the 16-band multi spectral layers use a 3 kernel and a ReLU activation function. To filter array (MSFA) with the proposed method improved by 8.47 dB compared to the conventional method using BI. improve this architecture, we replaced the BI in the ResNet- Shinoda by the above-mentioned ID-Interpolation. The new processing pipeline is also shown in Figure 2. 3 Material and Methods In order to train and validate the networks proposed in this work, a large HSI data set was built from different data collections, including CAVE [9], TokyoTech [11], ICVL [12], HyTexiLa [13] and google [14] data sets. This data collection was broken down into 70 percent training, 20 percent validati- on and 10 percent test data, resulting in 351 training (631.128 Figure 3: Processing pipeline for ResNet-Shinoda and ResNet-ID. sub images), 99 validation (163.272 sub images) and 51 test ResNet-Shinoda use the bilinear interpolation (blue) and ResNet- ID use the intensity difference interpolation (red). 38 Multispectral single chip reconstruction using DNNs with application to open neurosurgery To make comparisons between the results of the networks and the analytic algorithms, 3 similarity measures were selec- ted. The image reconstruction quality was determined by the Peak-Signal-to-Noise Ratio (PSNR), the Structural Similarity (SSIM) and the Mean-Squared Error (MSE) of the restored MSFA and MSI data. To analyze differences between the si- milarity measures of the proposed methods, the Welch-t-Test was used which (according to Rasch et al. [15]) is preferred to the Student-t-Test. If the P value is smaller than P< .05 the mean of two samples is seen to be significantly different. Figure 5: SSIM for 16 channel MSFA and the datasets HITgoC and Nimbus for the algorithms BI and ID and the neural networks ResNet-Shinoda and ResNet-ID. 4 Results We calculated the similarity measures for the HitgoC and Nimbus data and determined the differences of the mean similarity measure values achieved with the ResNet-ID and baseline algorithms such as BI and ResNet-Shinoda. The re- sulting similarity measures are of all four methods are present- ed in box plots in Figures 4-6. T-tests are included in the graphs as lines between methods. The Figures are separated in two columns (left HITgoC and right Nimbus). Results for ana- Figure 6: MSE for 16 channel MSFA and the datasets HITgoC and lytical algorithms are separated in the left and for the neuronal Nimbus for the algorithms BI and ID and the neural networks networks in the right of the column. ResNet-Shinoda and ResNet-ID. The proposed method achieved better results against BI in Table 1: PSNR values of beads image from CAVE of BI and the SSIM (.0644 vs. .0252), PSNR (15.34 dB vs. 9.12 dB) and in ResNet-Shinoda according to Shinoda et al.. Additionally, the the 1-MSE*100 (.0855 vs. .0273) as well as compared to the PSNR improvement to BI is evaluated. ResNet-Shinoda in SSIM (.0103 vs. .0074), PSNR (3.87 dB vs. 3.66 dB) and 1-MSE*100 (.0075 vs. .0047) for HITgoC/ Nim- Algorithm PSNR in dB Improvement to BI in dB bus. The proposed network ResNet-ID is significantly better BI 26.81 --- than the compared ResNet-Shinoda, in the SSIM (HITgoC: t = ResNet-Shinoda 32.69 5.88 -3.387, P <.001; Nimbus: t = -5.44, P < .001), in the PSNR (HITgoC: t = -4.34, P <.001; Nimbus: t = -3.74, P < .001) and For qualitative evaluation, single image from CAVE and in the MSE (HITgoC: t = -2.30, P = .0232; Nimbus: t = -3.21, from Nimbus (see Fig. 7) were extracted, and the spectral re- P = .003). Both neural networks delivered significantly better construction for the algorithms was carried out on the images. results than all of the analytical algorithms (see Fig. 4-6). Reconstruction and an error-image were calculated with L1- Norm of ID-interpolation, ResNet-Shinoda and ResNet-ID. PSNRs, the improvement to BI calculated by Shinoda et al. (Table 1) and by us for the beads image are summarized in Table 2. Table 2: PSNR values of beads image from CAVE of BI, the Res- Net-Shinoda and ResNet-ID according to our implementation. Ad- ditionally, the PSNR improvement to BI is evaluated. Figure 4: PSNR for 16 channel MSFA and the datasets HITgoC Algorithm PSNR in dB Improvement to BI in dB and Nimbus for the algorithms BI and ID and the neural networks ResNet-Shinoda and ResNet-ID. BI 23.33 --- ResNet-Shinoda 28.05 4.72 ResNet-ID 31.71 8.38 39 Multispectral single chip reconstruction using DNNs with application to open neurosurgery 6 Conclusion In this study, a new spectral reconstruction method is pre- sented which combines two existing methods from literature, the ID algorithm and a ResNet architecture. Two analytical standard algorithms as well as the neural networks are evaluated in PSNR, MSE and SSIM on the datasets HITgoC and Nimbus. The proposed method – ResNet-ID – performs best in all tests. Acknowledgment The deep learning cluster were funded by the Federal Ministry of Education and Research under the project reference numbers 16FMD01K, 16FMD02 and 16FMD03. Figure 7: Prediction and error image of the reconstructions from the selected picture of CAVE database (error image scaled to .5) References and Nimbus database (error image scaled to .1) for ID-interpolation, [1] Zhang et al.Tissue classification for laparoscopic image un- ResNet-Shinoda and ResNet-ID. derstanding based on multispectral texture analysis. J Med Imaging. 2017 [2] Yoon et al. A clinically translatable hyperspectral endoscopy 5 Discussion system for imaging the gastrointestinal tract. Nat Commun 10, 2019 In this contribution, two analytical algorithms and two [3] Vasefi et al. Hyperspectral and multispectral imaging in der- matology. In: Hamblin MR Pinar A, Gupta GK (eds): Imaging neural networks for spectral reconstruction have been compa- in dermatology. 2016, pp. 187-201 red. In the work by Shinoda et al. a network was validated and [4] Holmer et al. Hyperspectral imaging in perfusion & wound tested on CAVE with 8-fold cross-validation and they yield an diagnostics - methods and algorithms for the determination of average PSNR of 43.05 dB for all 32 images, being an impro- tissue parameters. Biomed Tech 25;63(5):547-556, 2018 [5] Ortega et al. Hyperspectral and multispectral imaging in di- vement of 8.47 dB to BI. By reimplementation of the ResNet- gital and computational pathology: a systematic review. Bio- Shinoda our results of 28.05 dB in PSNR for the beads image med Opt Express. 2020, 11(6):3195-3233. differ from the results of 32.69 dB in PSNR from the original [6] Fabelo et al. Spatio-spectral classification of hyperspectral work (see Tables 1 & 2). In the evaluation of the BI algorithm, images for brain cancer detection during surgical operations. we yielded 23.33 dB in PSNR compared to 26.81 dB from Shi- PLoS ONE 13(3), 2018 [7] Fabelo et al., In-vivo hyperspectral human brain image data- noda. It should be noted that Shinoda's paper does not describe base for brain cancer detection. IEEE Access 7, pp. 39098- exactly how their PSNR was calculated. Presumably, the 39116, 2019 PSNR in Shinoda's paper is not based on the 16 spectral data [8] Mihoubi et al. Multispektral demosaicing using intensity-ba- cube, but on the reconstruction of the RGB image derived by sed spectral correlation. Proc’s Int. Conf. on Image Proces- sing Theory, Tools & Applications (IPTA), pp. 461-6, 2015 CIE D65. Therefore, our values are smaller because we are [9] Yasuma et al., Generalized Assorted Pixel Camera: Post- comparing the entire data cube. While Shinoda achieves an Capture Control of Resolution, Dynamic Range and Spec- improvement of 5.88 dB, we get an improvement with our pro- trum, Tech. Rep., Dept. Comp. Science, Columbia U. 2008. posed method of 8.38 dB over the respective implemented va- [10] Shinoda et al. Deep demosaicking for multispectral filter arrays, in arXiv, 2018 riant of the bilinear interpolation. The smaller improvement of [11] Monno et al., A practical one-shot multispectral imaging sys- 4.77dB in PSNR between BI and our ResNet-Shinoda is due tem using a single image sensor. IEEE Trans. Image to the small number of training epochs, which was chosen due Processing, pp 3048-3059, 2015 to a lack of time and resources. The prediction and the error [12] Arad et al. Sparse Recovery of Hyperspectral Signal from pattern clearly show that the ResNet-ID yields better results, Natual RGB Images, in Europ. Conf. on Comp. Vision, 2016 [13] Khan et al., HyTexiLa: High Resolution Visible and Near i.e., a lower deviation from the original image, especially in Infrared Hyperspectral Texture Images. Sensors 2018 the area of the edges. [14] Prasad et al. Metrics and statistics of frequency of occurren- ce of metamerism in consumer cameras for natural scenes, in J. Optical Soc. America, pp. 1390-1402, 2015 [15] Rasch et al., The two-sample t test: pre-testing its assump- tions does not pay off, Statistical Papers 52, p. 219-31, 2011

Journal

Current Directions in Biomedical Engineeringde Gruyter

Published: Oct 1, 2021

Keywords: Spectral Reconstruction; Debayering; Demosaicing; DCNN; open neurosurgery

There are no references for this article.