Access the full text.
Sign up today, get DeepDyve free for 14 days.
Robust Registration of Multi-Modal Medical Images: Towards Real-Time Clinical Applications
S. Wright, Victoria Centonze, Stephen Stricker, Peter DeVries, Stephen Paddock, Gerald Schatten (1993)
Introduction to confocal microscopy and three-dimensional reconstruction.Methods in cell biology, 38
Michael Gué, C. Messaoudi, Jian Sun, T. Boudier (2005)
Smart 3D‐fish: Automation of distance analysis in nuclei of interphase cells by image processingCytometry Part A, 67A
M. Tanas, B. Rubin, R. Tubbs, S. Billings, E. Downs‐Kelly, J. Goldblum (2010)
Utilization of fluorescence in situ hybridization in the diagnosis of 230 mesenchymal neoplasms: an institutional experience.Archives of pathology & laboratory medicine, 134 12
(S.R.K. Gihan Kuruppu, U.A.J. Pinidiyaarachchi, High speed motion tracking for weightlifting based on correlation coefficient template matching. Int. J. Soft. Comput. Eng. 2 (2013))
S.R.K. Gihan Kuruppu, U.A.J. Pinidiyaarachchi, High speed motion tracking for weightlifting based on correlation coefficient template matching. Int. J. Soft. Comput. Eng. 2 (2013)S.R.K. Gihan Kuruppu, U.A.J. Pinidiyaarachchi, High speed motion tracking for weightlifting based on correlation coefficient template matching. Int. J. Soft. Comput. Eng. 2 (2013), S.R.K. Gihan Kuruppu, U.A.J. Pinidiyaarachchi, High speed motion tracking for weightlifting based on correlation coefficient template matching. Int. J. Soft. Comput. Eng. 2 (2013)
Pierre Sesques, N. Johnson (2017)
Approach to the diagnosis and treatment of high-grade B-cell lymphomas with MYC and BCL2 and/or BCL6 rearrangements.Blood, 129 3
Chun-yan Li, J. Bai, Xiaomeng Hao, S. Zhang, Yunhui Hu, Xiaobei Zhang, W. Yuan, Linping Hu, T. Cheng, A. Zetterberg, Mong-Hong Lee, J. Zhang (2014)
Multi-gene fluorescence in situ hybridization to detect cell cycle gene copy number aberrations in young breast cancer patientsCell Cycle, 13
Xiujun Fu, J. Lennerz, Maristela Onozato, A. Iafrate, Y. Yagi (2017)
Evaluation of a confocal WSI scanner for FISH slide imaging and image analysisDiagnostic Pathology, 3
A. Gozzetti, M. Beau (2000)
Fluorescence in situ hybridization: uses and limitations.Seminars in hematology, 37 4
L. Stefano, S. Mattoccia (2003)
Fast template matching using bounded partial correlationMachine Vision and Applications, 13
J. Pawley (2002)
Confocal and two‐photon microscopy: Foundations, applications and advancesMicroscopy Research and Technique, 59
Guang-Zhong Yang, D. Hawkes, D. Rueckert, J. Noble, C. Taylor (2009)
Medical Image Computing and Computer-Assisted InterventionSpringer US
Toby Cornish, Ryan Swapp, K. Kaplan (2012)
Whole-slide Imaging: Routine Pathologic DiagnosisAdvances In Anatomic Pathology, 19
Shou-Der Wei, S. Lai (2008)
Fast Template Matching Based on Normalized Cross Correlation With Adaptive Multilevel Winner UpdateIEEE Transactions on Image Processing, 17
A. Watters, J. Bartlett (2002)
Fluorescence in situ hybridization in paraffin tissue sectionsMolecular Biotechnology, 21
C. Laurent, Maxime Guérin, F. Frenois, V. Thuriès, Laurence Jalabert, P. Brousset, S. Valmary-Degano (2013)
Whole-slide imaging is a robust alternative to traditional fluorescent microscopy for fluorescence in situ hybridization imaging using break-apart DNA probes.Human pathology, 44 8
D. Alpár, Judit Hermesz, L. Pótó, R. László, L. Kereskai, P. Jáksó, G. Pajor, L. Pajor, B. Kajtár (2008)
Automated FISH analysis using dual‐fusion and break‐apart probes on paraffin‐embedded tissue sectionsCytometry Part A, 73A
G. Kuruppu, S. Kodituwakku, U. Pinidiyaarachchi (2013)
High Speed Motion Tracking for Weightlifting based on Correlation Coefficient Template Matching
E. Brachtel, Y. Yagi (2012)
Digital imaging in pathology – current applications and challengesJournal of Biophotonics, 5
Linping Hu, K. Ru, Li Zhang, Yuting Huang, Xiaofan Zhu, Hanzhi Liu, A. Zetterberg, T. Cheng, W. Miao (2014)
Fluorescence in situ hybridization (FISH): an increasingly demanded tool for biomarker research and personalized medicineBiomarker Research, 2
MR Tanas (2010)
10.5858/2009-0571-OAR.1Arch. Pathol. Lab. Med., 134
B. Kajtár, G. Méhes, T. Lörch, L. Deák, Marika Kneifné, D. Alpár, L. Pajor (2006)
Automated fluorescent in situ hybridization (FISH) analysis of t(9;22)(q34;q11) in interphase nucleiCytometry Part A, 69A
G. Hildenbrand, A. Rapp, U. Spöri, C. Wagner, C. Cremer, M. Hausmann (2005)
Nano-sizing of specific gene domains in intact human cell nuclei by spatially modulated illumination light microscopy.Biophysical journal, 88 6
Jiao Li, W. Su, S. Zhang, Yunhui Hu, Jingjing Liu, Xiaobei Zhang, J. Bai, W. Yuan, Linping Hu, T. Cheng, A. Zetterberg, Z. Lei, Jin Zhang (2015)
Epidermal growth factor receptor and AKT1 gene copy numbers by multi-gene fluorescence in situ hybridization impact on prognosis in breast cancerCancer Science, 106
E. Pike (2002)
Confocal and Two-Photon Microscopy: Foundations, Applications, and Advances
A. Kikuchi, T. Sawamura, O. Daimaru, M. Horie, Kazutoshi Sasaki, Noriyuki Okita (2016)
Improved protocol for extraction of genomic DNA from formalin-fixed paraffin-embedded tissue samples without the use of xyleneClinical Chemistry and Laboratory Medicine (CCLM), 54
Publisher's Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations
J. Roix, P. McQueen, P. Munson, L. Parada, T. Misteli (2003)
Spatial proximity of translocation-prone gene loci in human lymphomasNature Genetics, 34
Fluorescence in situ hybridization (FISH) is a technique to visualize specific DNA/RNA sequences within the cell nuclei and provide the presence, location and structural integrity of genes on chromosomes. A confocal Whole Slide Imaging (WSI) scanner technology has superior depth resolution compared to wide-field fluorescence imaging. Confocal WSI has the ability to perform serial optical sections with specimen imaging, which is critical for 3D tissue reconstruction for volumetric spatial analysis. The standard clinical manual scoring for FISH is labor- intensive, time-consuming and subjective. Application of multi-gene FISH analysis alongside 3D imaging, significantly increase the level of complexity required for an accurate 3D analysis. Therefore, the purpose of this study is to establish automated 3D FISH scoring for z-stack images from confocal WSI scanner. The algorithm and the application we developed, SHIMARIS PAFQ, successfully employs 3D calculations for clear individual cell nuclei segmentation, gene signals detection and distribution of break-apart probes signal patterns, including standard break-apart, and variant patterns due to truncation, and deletion, etc. The analysis was accurate and precise when compared with ground truth clinical manual counting and scoring reported in ten lymphoma and solid tumors cases. The algorithm and the application we developed, SHIMARIS PAFQ, is objective and more efficient than the conventional procedure. It enables the automated counting of more nuclei, precisely detecting additional abnormal signal variations in nuclei patterns and analyzes gigabyte multi-layer stacking imaging data of tissue samples from patients. Currently, we are developing a deep learning algorithm for automated tumor area detection to be integrated with SHIMARIS PAFQ. Keywords: Fluorescence in situ hybridization (FISH), Confocal whole slide imaging (WSI) scanner, Automated, Segmentation, Algorithm Introduction structural integrity of genes on chromosomes. Applica- Fluorescence in situ hybridization (FISH) is a technique tions of FISH assay together with imaging techniques, employed fluorescently labeled probes to specifically such as confocal and wide-field fluorescence are com- bind a target genome sequence and it is in research and monly in use. However, confocal imaging provides im- clinical use (Gozzetti and Le Beau 2000; Kajtar et al. ages with higher quality in terms of sharpness, contrast, 2006; Tanas et al. 2010; Hu et al. 2014). The technique and noise when compared to wide-field fluorescence im- enables spatial localization of multiple signals within the aging (Xiujun Fu et al. 2017). Confocal imaging technol- cell nuclei to provide the presence, location and ogy increases optical resolution compared to traditional wide-field fluorescent imaging by means of adding a spatial pinhole placed at the focal plane of the lens to * Correspondence: frankenz@mskcc.org Department of Pathology, Memorial Sloan Kettering Cancer Center, New eliminate the out-of-focus light (Wright et al. 1993). York, NY 10065, USA © The Author(s). 2021 Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. Frankenstein et al. Applied Microscopy (2021) 51:4 Page 2 of 12 However, in wide-field fluorescence the entire specimen fluorescence probes cannot be interpreted accurately by of interest is exposed to the light source and the speci- 2D imaging strategy (due to missing information of the men axial dimension should be less than the wave- Z-axis) when taking into account the various spatial lo- optical depth to satisfy in-focus condition. This condi- cations of gene signals inside the cell nuclei. For ex- tion limits the portion of the tumor that can be scanned. ample, the diagnosis of gene translocation or fusion Other advantages of confocal fluorescence imaging using FISH break-apart probe requires to measure the over wide-field fluorescence imaging are the elimin- spatial distance between gene signals from different ation/reduction of background information from focal channels to determine break-apart or co-localization (fu- plane and lower excitation energy as well as the ability sion) of signals (Alpar et al. 2008; Cornish et al. 2012). to perform serial optical sections with thick specimen This spatial distance between different gene signals is es- (which is critical for 3D tissue reconstruction). Applica- sential for the diagnosis. Here, we use confocal WSI in tion of multi-layer Z-stack for 3D tissue reconstruction analysis of FISH signals across Z-stack volume, which al- with FISH assay enables the volumetric spatial lows to precisely localize and detect the spatial distance visualization of multiple genes signals with different between gene signals inside the cell nuclei volume. Yet, colors within the cell nuclei (Diaspro 2001; Xiujun Fu it is impossible to precisely determine by eye the dis- et al. 2017). Since image acquisition is time-consuming tance between gene signals within individual cell nuclei. and subjective, whole slide imaging (WSI) technology Even though FISH analysis is complicated, clinical cyto- has been applied to automate the digital image acquisi- geneticists perform counting rely on their experience for tion from glass slide (Brachtel and Yagi 2012; Laurent gene signal patterns detection and individual nuclei et al. 2013) with confocal scanner for FISH slide imaging morphology identification. Automated signal quantifica- (Xiujun Fu et al. 2017). tion is objective and may improve productivity with Hybridized gene signals on FISH slides have extremely plentiful information. Therefore, we established auto- small size and occupy tiny volumes inside the nuclei, mated 3D FISH scoring of z-stack images from confocal with average diameter of several hundred nanometers WSI scanner. Our algorithm and application, SHIMAR (Hildenbrand et al. 2005; Xiujun Fu et al. 2017). Micros- IS PAFQ, successfully employs 3D calculations for seg- copy with high magnification objective is required in menting clear individual nuclei shapes, gene signals de- order to visualize and distinguish these extremely small tection, distribution of break-apart probe signal patterns, size signals. The epifluorescence microscopy (wide-field including standard break-apart, and variant patterns due fluorescence microscopy), is commonly used to view to truncation, deletion, etc. The analysis was accurate FISH slides to count the fluorescent signals for scoring and precise when compared with grand truth clinical and diagnosis. Most WSI scanners designed for bright manual counting and scoring reported in ten lymphoma field imaging and have 20× or 40× objectives for digital and solid tumors cases. Where EWSR1, MYC, BCL2 and fluorescence imaging with digitize slide optics equivalent BCL6 break apart FISH probes used as a diagnostic to epifluorescence microscopy (Cornish et al. 2012; guide to determine treatment of lymphoma or solid tu- Laurent et al. 2013). A WSI fluorescence scanner has mors patients (Sesques and Johnson 2017). The algo- been used with FISH slide of diffuse large B cells lymph- rithm we developed is objective and more efficient than oma cases with break-apart probes to detect MYC re- the current standard clinical procedure. It enables the arrangement (Laurent et al. 2013). It shown to be rapid, automated counting of more cell nuclei and detect add- robust, and highly sensitive. However, these scanners en- itional variations in gene signals abnormal patterns counter difficulty in capturing the miniscule fluores- within the nuclei than the conventional clinical counting cence signals from the nuclei when digitizing FISH slide. method. As well as accurately retrieve gene signals num- A confocal WSI scanner recently used with high magni- ber and calculate 3D vector lengths between different fication of 40× objective, producing final image with gene signals for each individual nuclei together with nu- high pixel resolution of 0.16 μm/pixel, shown to be cap- clei patterns classification. able of acquiring each of the fluorescence signals from the FISH slide (Xiujun Fu et al. 2017). Moreover, the ex- Materials and methods tremely small size fluorescent signals which carry the This study involves human subjects and is therefore ap- specific genetic information on FISH slides are distrib- proved by the institutional review board of Memorial uted spatially inside the nuclei volume (Xiujun Fu et al. Sloan Kettering Cancer Center (MSKCC), New York, 2017); and therefore, could not be completely detected NY, USA (IRB No. 18–216). by a single-layer scanning method. The spatial arrangement of genes may reflect normal Tissue sectioning and FISH slides preparation or rearrangements in chromosomes (Roix et al. 2003; Information concerning lymphoma and solid tumors pa- Gue et al. 2005). However, genes visualization by tients has been retrieved from MSKCC (Table 1). Tissue Frankenstein et al. Applied Microscopy (2021) 51:4 Page 3 of 12 Table 1 Dataset for FISH diagnosis of lymphoma and solid tumors patients Case Diagnosis Break-apart probe Clinical result 1 Diffuse large B-cell lymphoma BCL6 Negative (−) 2 Follicular lymphoma BCL2 Negative (−) 3 Ewing’s sarcoma EWSR1 Positive (+) 4 Diffuse large B-cell lymphoma with plasmacytic differentiation MYC Negative (−) 5 In situ follicular neoplasia BCL2 Positive (+) 6 Focal diffuse large B-cell lymphoma and follicular lymphoma BCL6 Negative (−) 7 Diffuse large B-cell lymphoma BCL6 Positive (+) 8 Diffuse large B-cell lymphoma MYC Positive (+) 9 Diffuse large B-cell lymphoma MYC Negative (−) 10 Diffuse large B-cell lymphoma MYC Negative (−) samples included in this analysis were formalin-fixed sectioning machine was used. A robotic arm that is paraffin-embedded (FFPE) blocks. FFPE tissue blocks is guided by a sensor, picks the tissue block to be sec- suitable for clinical diagnostics due to the preservation tioned. The tissue block is charged positively, cooled, procedure were the morphology retain relatively intact and then humidified before it is sectioned. The positively (Watters and Bartlett 2002; Kikuchi et al. 2016). There- charged tissue block attaches to a negatively charged fore, FISH pretreatment protocol reduces formalin effect carrier tape that transports and deposits it on a glass to optimize the access of FISH probes to target DNA slide moistened with water droplets (help spread the tis- (Watters and Bartlett 2002). Serial sectioning of FFPE sue). The tissue slide heated (to minimize, if not totally tissue blocks (Fig. 1) was used for Hematoxylin and remove, wrinkles) and then drying. This H&E or IHC Eosin (H&E) or immunohistochemistry (IHC) staining in slide was used for tissue orientation to ensure that the order to characterize the region of interest (ROIs). The correct area in an adjacent slide was selected for FISH AS-410 (Dainippon Seiki Co. LTD., Japan) automated scoring. FISH analysis was performed on 4.0 μm section Fig. 1 Workflow of tissue sectioning, staining and scanning. Serial sectioning of FFPE tissue blocks was used for H&E or IHC staining in order to characterize the region of interest. H&E and IHC slides were scanned at wide-field mode with 20× water immersion objective at a single layer. ROIs on FISH slides were scanned at confocal mode with multiple layers (N = 7 layers at 0.6 μm interval) with 40× water immersion objective and a final image resolution of 0.16 μm/pixel. Three filters were chosen: DAPI (blue), FITC (green) and TRITC (red). Showing in a dashed line frame: we are currently developing a deep learning algorithm for an automated tumor area detection Frankenstein et al. Applied Microscopy (2021) 51:4 Page 4 of 12 were tissue hybridized with break-apart probe to detect fully automated scanning to define the focus maps as gene rearrangements. FISH slides were prepared for ten well as detection of the tissue regions were performed patients as follows. Slides were pretreated with buffer so- by a control software. However, the semi-automated lution as well as with hydrochloric acid to solubilizing scanning allows user to define the focus map as well as basic nuclear proteins, improving the accessibility of the the tissue regions to be scanned. DNA. This method extracts the extracellular matrix of H&E and IHC slides were scanned at wide-field mode proteins to improve accessibility of the probe to the cells with 20× water immersion objective at a single layer. and preventing tissue autofluorescence (Watters and Scanned wide-field images were viewed, and several Bartlett 2002). Pretreated tissue was digested with buffer ROIs from each slide were selected within the tumor and protease for the purpose of breaking of peptide area of the tissue and reviewed by a pathologist. ROIs on bonds to affect signal quality by allowing access of the H&E and IHC slides were used in semi-automated mode FISH probes to the genomic target DNA and reduces to define the ROIs on FISH slides. ROIs on FISH slides autofluorescence generated by intact proteins (Watters were scanned at confocal mode with multiple layers for and Bartlett 2002; Kikuchi et al. 2016). Protease diges- both targeting genes and nuclei visualization. Multi-layer tion was terminated by dehydrating slides in an alcohol scanning of N = 7 layers at 0.6 μm interval were per- series and air-dried. FISH probes directly labeled with formed. Exposure time of the scans was set based on the fluorochromes are commercially available and ready to signal intensities of each channel with 40× water use in red, green and blue fluorophores. Probes were ap- immersion objective and a final image resolution of plied to the tissue slide, cover slipped, sealed and de- 0.16 μm/pixel (has a numerical aperture of 1.2). Three naturation was conducted and hybridized in a filters were chosen in accordance with their fluorescent humidified ThermoBrite system. Post hybridization excitation and the emission wavelengths of the probes washing preformed at preheated temperature in order to (Supplemental Table 1). The three filters are DAPI, avoid hybrids of low homology. Slides were dehydrated FITC and TRITC (as described above). The source of in an alcohol series. Air dried slides were counterstained the scanner excitation light is the Lumencor LED light using Vectashield with 4′,6-diamidino-2-phenylindole engine for the highest possible illumination power and (DAPI) medium and cover slipped. Slides were stored at PCO edge cooled scientific CMOS camera combining − 20 °C. DAPI targeting the DNA in the cell nucleus high sensitivity and low noise. with blue fluorophore. The FISH dataset includes patients who had been diag- Image evaluation and analysis nosed with diffuse large B-cell lymphoma (DLBCL), fol- WSI images were visually assessed and annotated in licular lymphoma, diffuse large B-cell lymphoma (DLBCL) CaseViewer provided by 3DHISTECH. The tumor areas with plasmacytic differentiation, in situ follicular neoplasia were semi-automatically detected on FISH slides as de- and focal diffuse large B-cell lymphoma (DLBCL) with fol- scribed above. The annotated regions were exported into licular lymphoma. Patients were analyzed with BCL6, tiled TIFFs (representation of each layer of the multi- BCL2, MYC, EWSR1 and MYC break apart probes as a layer scanning). The exported tiled TIFFs were imported diagnostic guide to determine treatment. This probes set into our algorithm for clear individual cell nuclei seg- includes the combinations of the following fluorophores: mentation and gene signals detection, quantification, co- fluorescein isothiocyanate (FITC) green fluorescent pro- localization and 3D analysis as described below. Analysis teins (GFP and EGFP), paired with tetramethylrhodamine of gene signals corresponding to individual cell nuclei isothiocyanate (TRITC) red fluorescent protein (DsRed). were performed using our algorithm. The accuracy of the analysis was compared with manual investigation as FISH slides scanning assessed clinically by pathologist and cytogeneticists, As shown in Fig. 1, H&E as well as IHC were used for where overlapping red and green or fused yellow signal the propose of tumor area detection. IHC interpretation represents co-localization, and separate red and green (demonstrates coexpression of markers) was useful when signals indicate break-apart. Unlike the clinical manual follicular lymphoma/neoplasia were part of the differen- investigation, our new algorithm calculates the 3D vector tial diagnosis (Sesques and Johnson 2017): cases 2, 5 and length between different channels. Thus, the diameters 6. Slides were digitized with the pannoramic confocal of gene signals spots both in FITC and TRITC channels scanner (3DHISTECH Ltd., Budapest, Hungary). The were set as 0.6 μm and the cut-off 3D distance to define optical components of this scanner allow both bright break-apart gene signals was set to 1.2 μm (twice or field and fluorescence imaging as well as both wide-field more than the gene signal spot diameter). The negative and confocal modes are provided for fluorescence im- diagnosis of patient relies on the mentioned relationship aging. The scanner supports fully automated scanning of gene signals inside each individual nuclei, where 10% and semi-automated scanning. All the calculations in the Frankenstein et al. Applied Microscopy (2021) 51:4 Page 5 of 12 or less in counted individual nuclei shows abnormal sig- bellow, where x and y are the source pixel position and nal patterns. u and v are variable, shift component along x-direction and y-direction respectively. Algorithm description for 3D scoring of FISH using a Square difference matching is defined as: confocal WSI scanner X R ðÞ x; y ¼ðÞ TuðÞ ; v −IxðÞ þ u; y þ v SD The algorithm is described in Fig. 2 and illustrated in u;v Fig. 3. 3D information of Z-stack images were exported into tiled TIFFs data. Exported tiled TIFFs were Normalized square difference is defined as: imported into our algorithm. We employed Gaussian fil- X , ter to reduce noise. This is a non-linear low-pass filter ðÞ TuðÞ ; v −IxðÞ þ u; y þ v u;v R ðÞ x; y ¼ NSD that removes high-frequency components. Gaussian function is given as: qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi X X 2 2 . TuðÞ ; v IxðÞ þ u; y þ v u;v u;v 1 ðÞ x−μ pffiffiffiffiffiffi fxðÞ ¼ exp σ 2π 2σ Cross correlation matching is defined as: where μ is mean and σ is variance. R ðÞ x; y ¼ðÞ TuðÞ ; v IxðÞ þ u; y þ v In addition, we operated morphological opening and u;v closing transformation for noise removing, isolation of individual elements and joining disparate elements as Normalized cross correlation matching is defined as: well as finding of intensity bumps or holes. Opening ob- X , tained by erosion followed by a dilation which results in ðÞ TuðÞ ; v IxðÞ þ u; y þ v u;v R ðÞ x; y ¼ NC removing small objects on the foreground: qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi X X dst ¼ openðÞ src; element 2 2 TuðÞ ; v IxðÞ þ u; y þ v u;v u;v ¼ dilateðÞ erodeðÞ src; element Correlation coefficient matching is defined as: Closing is reverse of opening and obtained by dilation followed by erosion. It is useful in closing small holes in- 0 0 R ðÞ x; y ¼ðÞ TðÞ u; v IðÞ x þ u; y þ v CC side the objects: u;v dst ¼ closeðÞ src; element where ¼ erodeðÞ dilateðÞ src; element 0 1 TðÞ u; v ¼ TuðÞ ; v − TuðÞ ; v A template matching technique was employed to seg- ðÞ w h u;v ment DAPI stained clear individual cell nuclei. We used the technique to find the statistically significant match between an individual nuclei templates and the target 0 1 image (Gihan Kuruppu and Pinidiyaarachchi 2013). The IðÞ x þ u; y þ v¼ IxðÞ þ u; y þ v − ðÞ w h size of the source image I is W × H where W and H IxðÞ þ u; y þ v representing the width and height, respectively. The u;v source image I was compared with the overlapped patches of the template image T (with width “w” and Normalized correlation coefficient matching is defined height “h”). The template moves one pixel in the hori- as: zontal or vertical direction on the image to be tested X , ðÞ TuðÞ ; v IxðÞ þ u; y þ v and performs a comparison calculation. All possible lo- u;v R ðÞ x; y ¼ NcC cations to be matched with the template are stored in a resultant matrix R given by (W – w + 1) × (H – h+1) qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi X X 2 2 which stores the coefficient value for each matched loca- TuðÞ ; v IxðÞ þ u; y þ v u;v u;v tion in pixel. We tested different approaches for nuclei segmentation, some based on pixel by pixel intensity dif- We found no significant differences between the six ferences to calculate the summation of squared (Ourse- template matching algorithms. Normalized correlation lin et al. 2001; Di Stefano and Mattoccia 2003). Other coefficient and correlation coefficient methods are al- approaches are more complex as they involve numerous most perfectly matching with the ground truth segmen- multiplication, division and square root operations (Wei tation. Normalized square difference, square difference, and Lai 2008). The different approaches described normalized cross correlation and cross correlation Frankenstein et al. Applied Microscopy (2021) 51:4 Page 6 of 12 Fig. 2 Algorithm description for 3D scoring of FISH using confocal WSI scanner. Steps are described for 3D calculations for clear individual cell nuclei segmentation, gene signals detection and distribution of break-apart probes signal patterns, including standard break-apart, and variant patterns due to truncation, and deletion, etc. Frankenstein et al. Applied Microscopy (2021) 51:4 Page 7 of 12 Fig. 3 Segmentation and coordinates representation of signals to determine 3D co-localization, break-apart and other variations in individual cell nuclei patterns. a Volume and segmentation representation. Showing Z-stack images scanned with 7-layer and 0.6 μm interval at the same area. Blue is DAPI channel for stained nuclei, green is FITC channel, and red is TRITC channel. b 3D representation of selected volume from panel A. c Segmentation of clear individual cell nuclei (shown in gray) found at the volume. d Coordinates representation of segmented signals from the 7- layers Z-stack. e 3D vector length calculation using the X, Y and Z coordinates extracted from the 7-layers Z-stack (to determine co-localization and break-apart between FITC and TRITC signals) and classification of cell nuclei pattern. f Collection of segmented individual cell nuclei to show variations in signals patterns (normal in orange frame, break in purple frame and other patterns in gray frame) method have minor variations compared to ground truth Therefore, we selected the normalized correlation coeffi- segmentation. Also, there is no significant difference be- cient matching approach for clear individual nuclei tween the six template matching algorithms on process- shapes segmentation (Fig. 3). In order to distinguish be- ing time. The normalized correlation coefficient method tween the different segmented nuclei by different unique performs slightly better in terms of processing time identifier we used connected components of a hyper- compared to the correlation coefficient method. graph method. A connected component of a hypergraph Frankenstein et al. Applied Microscopy (2021) 51:4 Page 8 of 12 is defined as any maximal set of vertices which are pair- of which was saved as 3D interaction and until list of all wise connected by a non-trivial path. A vertex of a possible 3D vector lengths was completed. Distribution hypergraph considered to be an isolated vertex if it is of gene signals number and the calculated 3D vector not contained in any edge of the hypergraph. If a vertex lengths were output together with nuclei patterns classi- of the hypergraph is contained in an edge of a particular fication. Individual nuclei patterns were classified based size, then it is not considered isolated from a specific de- on number of co-localization and break-apart cases as scription of a nuclei. Nuclei coordinates were extracted well as copy number of signals (non interacting signals). and compared to assure 3D representation of individual cell nuclei across the layers. The FITC and TRITC chan- Application description for 3D scoring of FISH using a nels located inside each individual nuclei were converted confocal WSI scanner into coordinates representation and the high intensity The application we developed, SHIMARIS PAFQ (Fig. 4), coordinate for each 3D gene signal was extracted by includes several functions, such as 3D data uploading, comparing coordinate’s intensities. Figure 2 illustrates 3D data deletion, viewing of counting and scoring results the 3D coordinates comparisons we employed. Any se- (break-apart ratio, normal and multiple ratio, total num- lected coordinate consisting of up to 24 neighboring (co- ber of counted nuclei, number of counted nuclei for ordinates) were compared with the selected one for our each pattern and number of discard nuclei), selecting data representation decisions. and removing an individual nuclei from the calculations, Co-localization and break-apart gene signals where z-stack image zooms and translations view (with the op- calculated using network representation of 3D vector tion to move across layers), export and view of clinical lengths (between different gene signals) followed by report, quit the software, as well as assistance through multiply comparisons of the 3D vector lengths (Fig. 2). manual view. This approach was successful for analyzing The network is a weighted network, with each edge gigabyte multi-layer stacking imaging data of tissue sam- assigned a score, representing the 3D vector lengths of ples. This application allows users to analyze the data by the physical interaction between the two signals. Higher pressing optional buttons. In addition, the application score indicates larger 3D distance for the interaction. responds to a successful function through message, and The distance between two signals with a link in the net- if the application detects failure or an error, it provides useful messages that assist users to make the necessary work is defined as FITC TRITC , so that smaller ! correction. The application provides a friendly user in- FITC TRITC would correspond to shorter 3D distance terfaces to analyze the data. for the interaction. We calculated all possible pairs of different signals directly linked in the network. A key Results step is the calculations of the network distances. Given FISH diagnosis with two or more different fluorescence the network distance between FITC signal and TRITC probes can be applied to one sample (Li et al. 2014, signal, we then computed their 3D distance as: 2015) and relies on the number or the local relationship vffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi of gene signals within an individual cell nuclei. The ! u 2 2 FITC TRITC ¼ðÞ X −X þðÞ Y −Y FITC TRITC FITC TRITC current clinical analysis to interpret FISH signals by þðÞ Z −Z manually counting and scoring of individual cell nuclei FITC TRITC under fluorescence microscope is time-consuming and where (X , Y , Z ) are the coordinates represen- subjective. Clinical manual analysis is especially compli- FITC FITC FITC tation of FITC signal at layer N and (X , Y , cated when applying multi-gene FISH assay together FITC TRITC TRITC Z ) are the coordinates representation of TRITC sig- with confocal WSI scanning (Z-stack information). Due TRITC nal at layer N . Given a list of all possible 3D vector to the high level of complexity required for the 3D ana- TRITC lengths for each individual cell nuclei, we made the se- lysis we have developed an algorithm for quantification lection for interacting signals based on ranking distances and co-localization analysis of confocal WSI scanned with non-repetitive signals (across selected paths). For a FISH images. The time to analyze volume of the entire given nuclei, the 3D vector lengths can be compared tumor is 3.4 min. The algorithm allows 3D analysis of with each other, but not across different nuclei. To make FISH Z-stack images with 2 distinct channels or more. them comparable across each individual nuclei, we first Number of clear individual cell nuclei and number of sorted all 3D distances by ranking in increasing order. gene signals in each channel were quantified automatic- Then, the top rank (sorter 3D distance) was selected as ally, as well as the 3D vector lengths between the differ- an interaction, where we removed the lower weights ent channels. Distribution of data was output together (longer 3D distances) calculated based on at least one with individual nuclei patterns classification. The algo- same signal as found in the top rank. The sorting rithm was validated against ten clinical cases that were process continued without the top rank at each iteration analyzed manually by pathologist and cytogeneticists Frankenstein et al. Applied Microscopy (2021) 51:4 Page 9 of 12 Fig. 4 Application description for 3D scoring of FISH using confocal WSI scanner. Flowchart showing application functions, such as 3D data uploading, 3D data deletion, viewing of counting and scoring results, selecting and removing an individual nuclei from the calculations, z-stack image zooms and translations view, export and view of clinical report, quit the software, as well as assistance through manual view (Table 1). Figure 5 shows that the automated analysis is rearmament as shown in Fig. 5). However, the automatic significantly correlated with the validation procedures in procedure can detect more variations in nuclei patterns detecting the expected outcome of positive or negative for rearrangement than the clinical manual scoring. Such diagnosis for all the lymphoma and solid tumors pa- as one co-localization and an additional two or more tients. Moreover, the algorithm produces extensively FITC gene signals with additional one TRITC gene sig- more information that the clinical manual scoring. The nal or more than one co-localization and an additional algorithm performed the fastest calculations in a signifi- one FITC gene signal with additional two or more TRIT cantly short time (3.4 min for entire tumor area) than C gene signals (non-typical rearmament as shown in Fig. the procedure used for validation. The clinical manual 5). Non-typical rearmament patterns show features that counting is of 100 individual nuclei for each patient, are different from the typical rearmament pattern. Non- while the automatic procedure was several times more typical rearmament patterns seem to be important to de- of individual nuclei for each patient. The automatic diag- termine diagnosis, since in most of the cases the fraction nosis procedure and the procedure used for validation of nuclei counting in that group is significantly higher are significantly correlated in detecting nuclei pattern than the typical rearmament nuclei counting. for rearrangement with one co-localization and an add- Concerning the normal and multiple copy nuclei pat- itional one FITC and one TRITC gene signals (typical terns, automatic diagnosis procedure and the procedures Frankenstein et al. Applied Microscopy (2021) 51:4 Page 10 of 12 Fig. 5 3D FISH counting and scoring of individual cell nuclei with FITC (green) and TRITC (red) channels. Nuclei patterns are illustrated (left column: normal, multiple copy, break apart or others) and results are shown for the automatic and the clinical manual procedure (nuclei counting with percentage from total). Outcome of positive or negative diagnosis for the lymphoma and solid tumors patients are shown as well used for validation are significantly correlated. Individual since many nuclei counted to be in that group compared nuclei with normal pattern is characterized with only 2 with the normal, multiple copy and break apart inci- co-localizations, while multiple copy nuclei pattern is dences. While both the automatic procedure and the characterized with more than 2 co-localizations (where procedures used for validation determined the same out- all signals are infusion). In most cases, normal pattern come of positive or negative diagnosis, we found specific counting is significantly higher than the multiple copy differences in counting. This is due to differences in the pattern. Also, when compared with the clinical manual techniques used for counting and scoring. scoring, the automatic procedure can detect other nuclei patterns that cannot be classified as normal, multiple Discussion copy or break apart. Other nuclei patterns show features As described above, the 3D analysis for the organization such as one or more co-localization and an additional and alteration of chromosomes and genes by FISH using two or more FITC gene signals with no TRITC gene sig- a confocal WSI scanner is significant. The individual nals. The other nuclei patterns seem to be significant, genes visualized by fluorescence probes localized in Frankenstein et al. Applied Microscopy (2021) 51:4 Page 11 of 12 various locations within the cell nuclei, cannot be inter- We are currently developing a deep learning algorithm preted accurately by 2D imaging strategy, such as deter- for automated tumor area detection to be integrated mine rearrangements, while the relative 3D position of with SHIMARIS PAFQ (Fig. 1). The deep learning algo- genes permits precise localization. Moreover, the current rithm is trained to identify tumor region compared with clinical manual FISH counting and scoring under fluor- that of the nontumor area. escence microscope is time-consuming and subjective. Application of multi-gene FISH analysis (with two or Conclusion more different fluorescence probes in one sample (Li We established automated 3D FISH scoring (multi-gene) et al. 2014, 2015) together with 3D imaging, significantly for z-stack images from confocal WSI scanner. The increase the level of complexity required for an accurate standard clinical manual scoring for FISH is labor- 3D analysis. Hence, we developed an automated algo- intensive, time-consuming and subjective. Application of rithm and application, SHIMARIS PAFQ, for 3D quanti- multi-gene FISH analysis alongside 3D imaging, signifi- fication, co-localization and abnormal signal patterns cantly increase the level of complexity required for an analysis of confocal WSI scanned FISH z-stack images accurate 3D analysis. Therefore, the procedure we devel- with 2 distinct channels or more. The algorithm per- oped successfully employs 3D calculations for individual forms 3D automatic analysis of FISH Z-stack images to cell nuclei segmentation, gene signals detection and dis- count the number of clear individual cell nuclei, the tribution of break-apart probes signal patterns, including number of gene signals and the 3D vector length be- standard break-apart, and variant patterns due to trunca- tween the different channels in each cell nuclei. Distri- tion, and deletion, etc. The procedure enables the auto- bution of signals and the 3D vector lengths were output mated counting of more nuclei, precisely detecting together with individual cell nuclei patterns classifica- additional abnormal signal variations in nuclei patterns tion. Automatic calculations was conducted in a signifi- than the conventional clinical counting method. As well cantly shorter time (3.4 min for the entire tumor area) as analyzes gigabyte multi-layer stacking imaging data of than the procedure used for validation, clinical manual tissue samples from patients. scoring. For all lymphoma and solid tumors patients, the algorithm detected the same outcome of positive or Supplementary Information negative diagnosis as detected using the validation pro- The online version contains supplementary material available at https://doi. org/10.1186/s42649-021-00053-y. cedure. While nuclei patterns counting classified as nor- mal is significantly higher than the multiple copy Additional file 1: Supplemental Table 1. Fluorescent excitation and pattern. The multiple copy pattern requires further in- the emission wavelengths of the probes. DAPI, SpGold and FITC were vestigation concerning the number of co-localizations used in this study. found within each cell nuclei. Yet, the algorithm counted several times more of individual cell nuclei for each pa- Acknowledgments The authors would like to thank the National Institutes of Health/National tient than the clinical manual counting. Since the Cancer Institute, the Warren Alpert Foundation, 3DHISTECH for the technical algorithm produced a relatively larger amount of infor- support and all the pathologists who provided their time and expertise in mation than the clinical manual procedure, there are evaluating the images and providing useful comments. specific differences in counting and the patterns de- Authors’ contributions tected. The algorithm detected more variations in nuclei Z.F. and Y.Y. developed the algorithm and the application. Z.F., N.U., Y.Z. and patterns classified as rearrangement, while the combin- Y. Y designed this study. N.U., U.A., R.A., M.R. and M.H. performed the tissue ation between gene signals is open to any break apart sectioning, slides preparation, scanning, image evaluation and manual analysis. Z.F. preformed the automated analysis. The author(s) read and feature (Fig. 5, non-typical rearrangement). For example, approved the final manuscript. more than one co-localization and an additional several FITC gene signal with additional of several TRITC gene Funding signals. That variations in the break apart features seem Research reported in this publication was supported in part by the Cancer Center Support Grant of the National Institutes of Health/National Cancer to be important, since in most cases the fraction of that Institute under award number P30CA008748. The content is solely the group is significant when a positive diagnosis is deter- responsibility of the authors and does not necessarily represent the official mined. Also, the algorithm detected other nuclei pat- views of the National Institutes of Health. The Warren Alpert Foundation. terns that cannot be classified as normal, multiple copy or break apart and many nuclei were counted to be in Availability of data and materials this group which makes it significant. For example, only Please contact author for data requests. one co-localization and no additional FITC gene signals Declarations or TRITC gene signals. The variations found automatic- ally in nuclei patterns requires further investigation that Competing interests may improve diagnosis. Authors declare no conflict of interest. Frankenstein et al. Applied Microscopy (2021) 51:4 Page 12 of 12 Received: 16 February 2021 Accepted: 29 March 2021 J.K.L. Xiujun Fu, M. Onozato, A. Iafrate, Y. Yagi, Evaluation of a confocal WSI scanner for FISH slide imaging and image analysis. Diagn. Pathol. 3, 2364– 4893 (2017) References Publisher’sNote D. Alpar, J. Hermesz, L. Poto, R. Laszlo, L. Kereskai, P. Jakso, G. Pajor, L. Pajor, B. Springer Nature remains neutral with regard to jurisdictional claims in Kajtar, Automated FISH analysis using dual-fusion and break-apart probes on published maps and institutional affiliations. paraffin-embedded tissue sections. Cytometry A 73, 651–657 (2008) E. Brachtel, Y. Yagi, Digital imaging in pathology--current applications and challenges. J. Biophotonics 5, 327–335 (2012) T.C. Cornish, R.E. Swapp, K.J. Kaplan, Whole-slide imaging: routine pathologic diagnosis. Adv. Anat. Pathol. 19, 152–159 (2012) L. Di Stefano, S. Mattoccia, Fast template matching using bounded partial correlation. Mach. Vis. Appl. 13, 213–221 (2003) A. Diaspro, Confocal and two-photon microscopy: foundations, applications and advances (Wiley-Liss, 2001) S.R.K. Gihan Kuruppu, U.A.J. Pinidiyaarachchi, High speed motion tracking for weightlifting based on correlation coefficient template matching. Int. J. Soft. Comput. Eng. 2 (2013) A. Gozzetti, M.M. Le Beau, Fluorescence in situ hybridization: uses and limitations. Semin. Hematol. 37, 320–333 (2000) M. Gue, C. Messaoudi, J.S. Sun, T. Boudier, Smart 3D-FISH: automation of distance analysis in nuclei of interphase cells by image processing. Cytometry A 67, 18–26 (2005) G. Hildenbrand, A. Rapp, U. Spori, C. Wagner, C. Cremer, M. Hausmann, Nano- sizing of specific gene domains in intact human cell nuclei by spatially modulated illumination light microscopy. Biophys. J. 88, 4312–4318 (2005) L. Hu, K. Ru, L. Zhang, Y. Huang, X. Zhu, H. Liu, A. Zetterberg, T. Cheng, W. Miao, Fluorescence in situ hybridization (FISH): an increasingly demanded tool for biomarker research and personalized medicine. Biomark. Res. 2, 3 (2014) B. Kajtar, G. Mehes, T. Lorch, L. Deak, M. Kneifne, D. Alpar, L. Pajor, Automated fluorescent in situ hybridization (FISH) analysis of t (9;22)(q34;q11) in interphase nuclei. Cytometry A 69, 506–514 (2006) A. Kikuchi, T. Sawamura, O. Daimaru, M. Horie, K. Sasaki, N. Okita, Improved protocol for extraction of genomic DNA from formalin-fixed paraffin- embedded tissue samples without the use of xylene. Clin. Chem. Lab. Med. 54, e375–e377 (2016) C. Laurent, M. Guerin, F.X. Frenois, V. Thuries, L. Jalabert, P. Brousset, S. Valmary- Degano, Whole-slide imaging is a robust alternative to traditional fluorescent microscopy for fluorescence in situ hybridization imaging using break-apart DNA probes. Hum. Pathol. 44, 1544–1555 (2013) C. Li, J. Bai, X. Hao, S. Zhang, Y. Hu, X. Zhang, W. Yuan, L. Hu, T. Cheng, A. Zetterberg, et al., Multi-gene fluorescence in situ hybridization to detect cell cycle gene copy number aberrations in young breast cancer patients. Cell Cycle 13, 1299–1305 (2014) J. Li, W. Su, S. Zhang, Y. Hu, J. Liu, X. Zhang, J. Bai, W. Yuan, L. Hu, T. Cheng, et al., Epidermal growth factor receptor and AKT1 gene copy numbers by multi- gene fluorescence in situ hybridization impact on prognosis in breast cancer. Cancer Sci. 106, 642–649 (2015) S. Ourselin, X. Pennec, R. Stefanescu, G. Malandain, N. Ayache, Robust registration of multi-modal medical images: toward real-time clinical applications, medical image computing and computer-assisted intervention (2001) J.J. Roix, P.G. McQueen, P.J. Munson, L.A. Parada, T. Misteli, Spatial proximity of translocation-prone gene loci in human lymphomas. Nat. Genet. 34, 287–291 (2003) P. Sesques, N.A. Johnson, Approach to the diagnosis and treatment of high- grade B-cell lymphomas with MYC and BCL2 and/or BCL6 rearrangements. Blood 129, 280–288 (2017) M.R. Tanas, B.P. Rubin, R.R. Tubbs, S.D. Billings, E. Downs-Kelly, J.R. Goldblum, Utilization of fluorescence in situ hybridization in the diagnosis of 230 mesenchymal neoplasms: An institutional experience. Arch. Pathol. Lab. Med. 134, 1797–1803 (2010) A.D. Watters, J.M. Bartlett, Fluorescence in situ hybridization in paraffin tissue sections: Pretreatment protocol. Mol. Biotechnol. 21, 217–220 (2002) S.D. Wei, S.H. Lai, Fast template matching based on normalized cross correlation with adaptive multilevel winner update. IEEE Trans. Image Process. 17, 2227– 2235 (2008) S.J. Wright, V.E. Centonze, S.A. Stricker, P.J. DeVries, S.W. Paddock, G. Schatten, Introduction to confocal microscopy and three-dimensional reconstruction. Methods Cell Biol. 38,1–45 (1993)
Applied Microscopy – Springer Journals
Published: Apr 9, 2021
You can share this free article with as many people as you like with the url below! We hope you enjoy this feature!
Read and print from thousands of top scholarly journals.
Already have an account? Log in
Bookmark this article. You can see your Bookmarks on your DeepDyve Library.
To save an article, log in first, or sign up for a DeepDyve account if you don’t already have one.
Copy and paste the desired citation format or use the link below to download a file formatted for EndNote
Access the full text.
Sign up today, get DeepDyve free for 14 days.
All DeepDyve websites use cookies to improve your online experience. They were placed on your computer when you launched this website. You can change your cookie settings through your browser.