Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Patch-based field-of-view matching in multi-modal images for electroporation-based ablations

Patch-based field-of-view matching in multi-modal images for electroporation-based ablations Patch-based field-of-view matching in multi-modal images for electroporation-based ablations 1 2 3 4 Luc Lafitte , R´emi Giraud , Cornel Zachiu , Mario Ries , 5,6 5,6 5,6 Olivier Sutter , Antoine Petit , Olivier Seror , Clair 1 1,3 Poignard , Baudouin Denis de Senneville University of Bordeaux, IMB, UMR CNRS 5251, INRIA Project team Monc, Talence, France, F-33405 Talence Cedex, France University of Bordeaux, IMS, CNRS UMR 5218, F-33405 Talence Cedex, France Department of Radiotherapy, UMC Utrecht, Heidelberglaan 100, 3584 CX, Utrecht, The Netherlands Imaging Division, UMC Utrecht, Heidelberglaan 100, 3584 CX, Utrecht, The Netherlands Interventional radiology unit, Hoˆpitaux Universitaires Paris Seine Saint Denis, Hoˆpital Avicenne, Assistance Publique Hoˆpitaux de Paris, Bobigny France University of Paris 13, “Sciences M´edicale et Biologie Humaine”, Bobigny, France Abstract. Various multi-modal imaging sensors are currently involved at different steps of an interventional therapeutic work-flow. Cone beam computed tomography (CBCT), computed tomography (CT) or Magnetic Resonance (MR) images thereby provides complementary functional and/or structural information of the targeted region and organs at risk. Merging this information relies on a correct spatial alignment of the observed anatomy between the acquired images. This can be achieved by the means of multi-modal deformable image registration (DIR), demonstrated to be capable of estimating dense and elastic deformations between images acquired by multiple imaging devices. However, due to the typically different field-of-view (FOV) sampled across the various imaging modalities, such algorithms may severely fail in finding a satisfactory solution. In the current study we propose a new fast method to align the FOV in multi-modal 3D medical images. To this end, a patch-based approach is introduced and combined with a state-of-the-art multi-modal image similarity metric in order to cope with multi- modal medical images. The occurrence of estimated patch shifts is computed for each spatial direction and the shift value with maximum occurrence is selected and used to adjust the image field-of-view. The performance of the proposed method — in terms of both registration accuracy and computational needs — is analyzed in the practical case of on-line irreversible electroporation procedures. In total, 30 pairs of pre-/per- operative IRE images are considered to illustrate the efficiency of our algorithm. We show that a regional registration approach using voxel patches provides a good structural compromise between the voxel-wise and “global shifts” approaches. The method was thereby beneficial for CT to CBCT and MRI to CBCT registration tasks, especially when highly different image FOVs are involved. Besides, the benefit of the method for CT to CBCT and MRI to CBCT image registration is analyzed, including the impact of artifacts generated by percutaneous needle insertions. Additionally, the computational needs using commodity hardware are demonstrated to be compatible with clinical constraints in the practical case of on-line procedures. The proposed arXiv:2011.11759v1 [eess.IV] 9 Nov 2020 Patch-based field-of-view matching in multi-modal images for electroporation-based ablations2 patch-based workflow thus represents an attractive asset for DIR at different stages of an interventional procedure. Keywords: Multi-modal image registration, patch-based matching, interventional procedures Patch-based field-of-view matching in multi-modal images for electroporation-based ablations3 1. Introduction Multiple imaging devices can be involved at different stages of an interventional procedure, such as image-guided radiotherapy (IGRT) (Guckenberger et al. 2012), irreversible electroporation (IRE) (Gallinato et al. 2019) or hyperthermia ablation (Holbrook et al. 2009) (Mougenot et al. 2009). In particular, cone-beam computed tomography (CBCT), computed tomography (CT) or Magnetic Resonance (MR) images are recently being employed at nearly all stages of the therapy: i.e., pre-, intra- and post-operatively. One of the benefits that employing multiple imaging sensors provides, is the ability to extract complementary functional and/or structural information of the targeted region and organs-at-risk. For example, as shown in (Hocquelet et al. 2016), novel diagnostic indicators can thereby be calculated by fusing pre- and post-operative image data. Similarly, multi-modal imaging may also be beneficial during the interventional procedure itself (Zachiu, Denis de Senneville, Dmitriev, Moonen & Ries 2017). Of note is that the quality and the amount of data that can be acquired intra- operatively is generally limited by practical clinical considerations: CBCT guidance is for example particularly beneficial due to the low amount of imaging-related radiation delivered to the patient compared to a conventional CT scan. However, this often leads to the intra-operative images being subject to low contrast, low signal-to-noise ratio and artifacts. Thus, it would be of clinical benefit if such images would be augmented by pre- operative data (Gallinato et al. 2019). A common pre-requisite is that organ locations must be set in a common frame of reference. To this end, previous studies propose several multi-modal deformable image registration (DIR) algorithms dedicated to the estimation of dense and elastic deformations between images (Heinrich et al. 2012, Rivaz et al. 2014, Denis de Senneville et al. 2016). This remains a challenging task since such algorithms have to be fast (to meet clinically acceptable durations) and automatic (the use must not be limited to a case-by-case basis and a manual recalibration is not preferable), especially when the patient is on the interventional table (Rubeaux et al. 2013) (Zachiu, Denis de Senneville, Tijssen, Kotte, C., Kerkmeijer, Lagendijk, Moonen & Ries 2017). A particular challenge arises when highly different fields-of-view (FOV) are sampled within the images. For example, while the FOV within intra-operative CBCT images is typically restricted to the targeted organ and its immediate surroundings, the corresponding pre-operative high-resolution CT image generally covers the entire abdomen and part of the thorax. A similar situation arises when a patient is screened via both CT/CBCT and MR imaging, with the resulting acquisitions typically having considerably different FOVs. This can severely hamper the performance of image registration algorithms, especially when iterative optimization strategies are employed: the algorithm is likely to get trapped into local optima if the apparent location of the anatomy-of-interest is too far apart within the two images. In such a case, a direct employment of DIR methods may be hardly feasible and a preliminary matching of the image FOVs (i.e., compensation of the 3D global shift between images) is necessary. Patch-based field-of-view matching in multi-modal images for electroporation-based ablations4 While registration solutions optimizing a translational model may perform well for estimating rigid displacements, they may also become sub-optimal when elastic deformations are present between the images. Moreover, such methods typically imply high computational demands and manual tuning of several input parameters, which limits their use in a clinical setting (Klein et al. 2010). Alternatively, a regional registration approach using pixel/voxel patches, may provide a good structural compromise between the voxel-wise and “global shifts” approaches. Several patch or block-matching algorithms have been previously proposed, dedicated to various applications (Jakubowski & Pastuszak 2013). The aim of these approaches is to consider each pixel by its square neighborhood, to characterize its local context. Matching algorithms may then be used to find local correspondences between images. Nevertheless, these methods are highly time consuming, especially when dealing with an important number of patches and when the search for correspondences must be performed in a large window search (i.e., searching for large patch displacements). A significant breakthrough has been obtained with the so-called “PatchMatch” algorithm (Barnes et al. 2009), which was initially proposed for finding pixel patch correspondences between 2D images in digital photography. The idea behind this approach is that some good patch matches can be found by random sampling, which can subsequently be allocated to surrounding areas as well, relying on the assumption that neighboring areas typically have similar displacements. The fast convergence of the process enables to quickly find good matches, even when these are located far from each other in the image spaces. This approach has been successfully employed to achieve numerous image analysis and editing tasks such as: stereo matching (Bleyer et al. 2011), optical flow computation (Bao et al. 2014), region inpainting (Newson et al. 2014), or 3D medical image segmentation (Giraud et al. 2016). In the current study, our contribution is four-fold: (i) A new fast method — using as a starting-point a 3D modified PatchMatch algorithm — is proposed to align the FOV in medical 3D images. A user-defined mask surrounding a region/organ of interest in one of the images can be provided as an input so that a global shift can be estimated relying on image information from this specific region. (ii) The modified PatchMatch algorithm is combined with a well-adapted multi-modal image similarity metric in order to cope with multi-modal medical images. (iii) The performance — in terms of registration accuracy and computational needs — is analyzed, and demonstrated to be compatible with clinical constraints in the practical case of on-line irreversible electroporation procedures. In total, 30 pairs of pre-/per- operative IRE images are considered to illustrate the efficiency of our algorithm. (iv) The benefit of proposed approach is evaluated for a potential pre-conditioning of a more complex multi-modal DIR algorithm. Patch-based field-of-view matching in multi-modal images for electroporation-based ablations5 2. Materials and Methods 2.1. Proposed method Let I and J be two 3D images. In the scope of this study, let I and J be a pre- and an intra-operative image, respectively. We seek X-, Y- and Z- translation components between I and J in order to match the position an organ of interest manually delineated in I. We recall that the estimation of elastic organ deformations by itself is outside the scope of the study: the proposed workflow is solely intend to standardize field-of-view in multi-modal images for a potential pre-conditioning a more complex multi-modal elastic registration algorithm. The proposed method (detailed in Figure 1) includes the following three main successive steps: (i) The PatchMatch (PM) algorithm is adapted and combined with a multi-modal metric in order to compute patch correspondences between I and J (see section 2.1.3). The multi-modal metric aims at evaluating edge alignments (EA) within patches. (ii) The occurrence of estimated patch shifts, i.e., the displacement between the patch positions in I and their correspondence in J, is computed for each spatial direction (see section 2.1.4). (iii) For each spatial direction, the shift value with maximum occurrence is selected and used to adjust the image field-of-view (see section 2.1.5). The proposed method is referred to as “PM-EA” (PatchMatch-Edge Alignment) in the scope of this study. Figure 1: Data processing sequence designed for the fast standardization of field-of-view in multi-modal images using the proposed patch-based framework. Patch-based field-of-view matching in multi-modal images for electroporation-based ablations6 2.1.1. Manual delineation of the organ of interest. The pre-operative image I is first used to manually segment the targeted region of interest. A binary mask (denoted by M) is constructed. Voxels of the image inside the mask have a value of one, and outside a value of zero. We underline that, in the scope of this study, this process was done using the pre-operative image I only, in order to demonstrate that the method is compatible with an automatic use during an intra-operative session. Note that this delineation step is often performed anyway during the planning session of the therapy and thus does not put extra burden on the medical staff. 2.1.2. Preprocessing of input data. I, J and M were resampled onto a common grid with a voxel size of 1 × 1 × 1 millimeters using a trilinear interpolation. 2.1.3. Implemented PatchMatch algorithm. PatchMatch is an iterative algorithm designed to quickly estimate patch correspondences between two given 2D images (Barnes et al. 2009). In our study, we first extend this algorithm for the matching of 3D patches. Hence, a patch consists in a cubic subset of the image domain, denoted by Γ, centered on one single voxel. Let ~r = (x, y, z) ∈ Ω be the spatial location of the center voxel, Ω the image domain and (x, y, z) the voxel coordinates. • Initialization. An initial guess is first computed: each patch from image I is initially randomly matched with a patch from image J. Subsequently, at each iteration of PatchMatch, voxels are scanned from left to right (X-axis), head to foot (Y- axis), front to back (Z-axis). For each voxel examination, the corresponding patch undergoes a “propagation step” followed by a “random search step”, as described in the seminal paper (Barnes et al. 2009): The output of our algorithm is a patch shift map V , defined for each voxel in Ω. • Propagation step. During this step, in order to find better correspondences, the patch shift of the current voxel ~r in I in I is considered to be similar to the ones of its three already examined neighboors in each direction (6-connexity) (i.e., the three voxels at locations ~r = (x − 1, y, z), ~r = (x, y − 1, z) and (−1,0,0) (0,−1,0) ~r = (x, y, z − 1)). (0,0,−1) Let V = (u, v, w) be the patch shifts that we seek ((u, v, w) being the voxelwise patch shift coordinates). Let ~r and ~r be two given spatial location in Ω and 1 2 D(~r , V (~r )) the distance between the patch at location ~r in I and the patch at 1 2 1 ~ ~ location r~ + V (~r ) in J. V (~r) is updated as follows: 1 2 ~ ~ ~ ~ ~ V (~r) = argmin {D(~r, V (~r)), D(~r, V (~r )), D(~r, V (~r )), D(~r, V (~r ))} (−1,0,0) (0,−1,0) (0,0,−1) (1) • Random search step. This step attempts to improve V (~r) by computing a set of candidate shifts (noted V (~r)) at an exponentially decreasing spatial distance from V (~r): Patch-based field-of-view matching in multi-modal images for electroporation-based ablations7 ~ ~ V (~r) = V (~r) + wα R (2) i i R being a uniform random in [−1, 1] , w the maximum search distance (set to the maximum image dimension), and α a fixed ratio between search window sizes (we took α = 0.5, as suggested in (Barnes et al. 2009)). Patches for i = 0, 1, 2 and so on are examined until the current search distance wα falls below one voxel. As reported in Barnes et al seminal paper, PatchMatch provides satisfactory results using a fixed number of iterations (5 iterations max) and that the algorithm converges most rapidly in the first iterations (Barnes et al. 2009). In the current study we used two iterations, since it was found to be a good compromise between accuracy and computational costs. In our implementation, the input images are down-sampled before PatchMatch: while lower computation times are expected using down-sampled versions of input images, it should also impact overall registration results. The down-sampling factor is thus an important input parameter for the algorithm and its impact will be carrefully analysed and discussed below. To further reduce the computational burden, the search window were a rectangular bounding box including both J and voxels with a value of one in M. • Aggregation of multiple PM estimations. As for exemplar-based segmentation (Giraud et al. 2016), our method can benefit from multiple PM estimations. Patch- shift estimates indeed rely on random candidate selection and several independent processes may provide different correspondences. Although PatchMatch inherently relies on serial operations and cannot benefit from parallel architectures in its current form, multiple realisations of PatchMatch can easily be calculated using j j j j separate CPU threads. Let V (~r) = (u (~r), v (~r), w (~r)) be one realisation at spatial location ~r, j being the realisation index. For each voxel, the obtained 3D shift realisations were then combined into a single 3D median vector V (~r) (Astola et al. 1990) as follows: V (~r) = argmin V (~r) − V (~r) (3) V (~ r)∈{V (~ r)} In this manner, outliers realisations were discarded without any additional penalty on the computational burden. 2.1.4. Proposed multi-modal metric. In the original PatchMatch paper, the distance between patches D(.) is the Sum of Squared Differences (SSD, also referred to as L2- norm) applied on the image intensity. While for mono-modal registration algorithms the SSD applied directly on the images might be sufficient (Horn & Schunck 1981) (Denis de Senneville et al. 2011), such a measure is unsuitable for registering across modalities. A modality independent similarity measure is thus necessary. In the current paper we Patch-based field-of-view matching in multi-modal images for electroporation-based ablations8 used an existing multi-modal metric which favors edge alignments (EA) in both patches (Irani & Anandan 1998) (Sutour et al. 2015). ~ ~ Let ∇ and ∇ be the gradient of the reference image I and the image to register I J J, respectively. The distance D(~r , V (~r )) between a patch of interest Γ in I (centered 1 2 on the voxel located in ~r ) and its potential correspondence in J (centered on the voxel located in ~r + V (~r )) was defined as follows: 1 2 ~ ~ ~ ∇ (~r ) · ∇ ~r + V (~r ) d~r I 1 J 1 2 D(~r , V (~r )) = − (4) 1 2 ~ ~ ~ ∇ (~r ) ∇ ~r + V (~r ) d~r I 1 J 1 2 2 2 where k · k is the Euclidean norm. Practically, the scalar product in the numerator is maximized when the edges in Γ are aligned with edges in the potential corresponding patch in J. Note that the numerator is maximized regardless any possible contrast reversals: due to the absolute value, the numerator is maximized for both parallel and anti-parallel edges. In addition, the scalar product in the numerator favors strong edges present in both modalities. The denominator, for its part, acts as a normalisation factor. Ultimately, since a minimization of D is required to compute patch correspondences in Eq. (1), a negative sign has been set behing the fractional term in Eq. (4). 2.1.5. Matching image FOVs from estimated patch correspondences. At this point we ~ ~ have a voxelwise 3D shift maps V (~r), ~r ∈ Ω. The objective is now to simplifiy V down to a single 3D image shift. For this purpose, we individually analysed shift occurrences in each component of V (i.e., u, v and w) and within the binary mask M encompassing the target organ: for each component, a histogram was calculated and the shift value with the highest occurrence was selected. The obtained 3D image shift was subsequently used to adjust the FOV of J with respect to the one of I. Note that lower and upper bounds on estimated shift values as well as a number of bins needs to be determined for the construction of the histograms. The lower and upper bounds were set to -100 and 100 millimeters, respectively, which was found to be sufficient in our tests. The number of bins is an input parameter for the algorithm that will be analysed below. 2.2. Experimental evaluation 2.2.1. Data sets. For the evaluation of the method we used data acquired during IRE procedures which are routinely performed at the University Hospital Jean Verdier at Bondy in France. This retrospective study is in accordance with ethical principals of the Declaration of Helsinki and has been approved by the local committee on human research of the University Hospital J Verdier. The clinical workflow included the following sessions: Patch-based field-of-view matching in multi-modal images for electroporation-based ablations9 • Pre-operative session. This session, performed several days before the interventional procedure, allowed identifying the tumor and the main liver structures using either a CT-scan (voxel size=[0.67 − 0.88] × [0.67 − 0.88] × [1.25 − 2] mm , FOV=[341 − 450] × [341 − 450] × [182 − 506] mm ) or a MR-scan (T1-weighting, 3 3 voxel size=1.72 × 1.72 × [2.5 − 3] mm , FOV=440 × 440 × [180 − 200] mm ). • Interventional session. The day of the procedure, an IRE ablation was performed under general anesthesia. The needles are percutaneously inserted around the tumor by the interventional radiologist with a free-hand technique under combination of real-time ultrasound (US) and 3D Virtual Target Fluoroscopic Display such that the electric field covers the target region (Sutter et al. 2018). A 3D CBCT (voxel size=0.45 × 0.45 × 0.45 mm , FOV=230 × 230 × [192 − 256] mm ) imaging was performed to visualise liver and needle locations. During the interventional session, a registration with pre-operative data is intend to augment the CBCT with tumor/liver structures segmentations. The pursued objective is to improve the targeting and to allow dose modeling, as described in (Gallinato et al. 2019). In total, we analysed a set of 30 pairs of pre-/intra-operative images distributed over the four following groups: (i) 8 pairs of CT/CBCT images obtained on 8 patients, respectively. CBCTs were acquired before needle insertion. (ii) 8 pairs of CT/CBCT images. Patients and CTs were those used in (i). CBCTs were acquired after needle insertion. (iii) 7 pairs of MR/CBCT images obtained on 8 patients, respectively. CBCTs were acquired before needle insertion. (iv) 7 pairs of MR/CBCT images. Patients and MRs were those used in (iii). CBCTs were acquired after needle insertion. 2.2.2. Performance assessment. The Dice Similarity Coefficient (DSC) was employed to determine the contour overlap of the liver: 2 |A ∩ B| DSC = (5) |A| + |B| where A and B are two manually defined ROIs encompassing the liver in the reference and the corrected image, respectively. A ∩ B is their intersection and |·| denotes the cardinality of a set (i.e., the number of voxels). DSC mean and standard deviation were computed over the 4 sets of image pairs individually (i.e., CT/CBCT no needle, CT/CBCT needles, MR/CBCT no needle, MR/CBCT needles, as defined in section 2.2.1). A Wilcoxon paired test was carried out in order to study whether DSC differences are statistically significant. A significance threshold of p = 0.05 was used. Patch-based field-of-view matching in multi-modal images for electroporation-based ablations10 2.2.3. Calibration of the proposed PM-EA algorithm. At this point it is important to underline that four main input parameters may influence the performance of the proposed approach. The proposed PM-EA algorithm was challenged against various modifications applied to these calibration parameters: Input parameter #1: down-sampling of input images (section 2.1.3). DSC and computation time were calculated using I, J and M at original image dimension and using down-sampling (DS) factors 2×, 4× and 8× in each spatial direction. A default down-sampling factor of 8× was used. Input parameter #2: patch size (section 2.1.4). DSC and computation time were calculated using patches of size 3 × 3 × 3, 5 × 5 × 5, 7 × 7 × 7, 9 × 9 × 9 and 11 × 11 × 11 voxels. A default size of 9 × 9 × 9 voxels was used. Input parameter #3: number of histogram bins (section 2.1.5). DSC were calculated for number of histogram bins of 10, 30, 50, 70 and 90. The computation time was not evaluated since a marginal impact is expected here. A default value of 50 was used. Input parameter #4: manual delineation errors of the targeted organ (section 2.1.1). To analyse potential errors arising from the manual delineation process, M was iteratively eroded (resp. dilated) using a 5 × 5 × 5 kernel. At each iteration, the volume of the eroded (resp. dilated) mask was calculated as well as the corresponding DSC after image alignement. Here again, the computation time was not evaluated since a marginal impact is expected. 2.2.4. Tested algorithms. The PM-EA’s ability to estimate a global 3D translation between images was challenged against two competing approaches. We also evaluated the benefit of using PM-EA as a starting point for an existing more complex multi-modal elastic registration algorithm. The above-mentioned 30 pairs of images were processed using PM-EA and using the following selection of image registration solutions: Elastix. We have selected in the Open source Elastix registration software (Klein et al. 2010) (Shamonin 2013) a registration solution which employs a 3D translation transformation model and which maximizes the normalized mutual information (Studholme et al. 1999) between the images to be registered. This registration solution is referred to as “Elastix” in the following. PM-L2. The proposed PatchMatch registration framework is here employed using a L2-norm for patch comparison, as it was introduced in the original paper (Barnes et al. 2009). This registration solution is referred to as “PM-L2” hereafter. Patch-based field-of-view matching in multi-modal images for electroporation-based ablations11 Evo. The EVolution algorithm (which is abbreviated as “Evo” in the scope of this study) was employed to estimate the elastic deformation V as the minimizer of the following energy E: 2 2 2 ~ ~ ~ ~ ~ E(V ) = exp(D(V )) + k ∇u k + k ∇v k + k ∇w k d~r, (6) 2 2 2 D(V ) being the multi-modal metric of Eq. (4) and α a weighting factor designed to link both the data fidelity term (left part of Eq. (6)) and the motion field regularity (right part of Eq. (6)). Note that D(V ) is composed with an exponential function in order to ensure that the data fidelity term is a positive-definite function. Additional details concerning the manner in which the Evo functional is minimized together with a detailed analysis of the algorithm performance can be found in (Denis de Senneville et al. 2016). A numerical implementation designed for the reduction of computational costs can be found in (Lafitte et al. 2018). PM-EA+Evo. The above-mentioned multi-modal registration algorithm Evo is here employed after FOV standardization using proposed PM-EA algorithm. The registration workflow, comprised of PM-EA followed by Evo, is referred to as “PM- EA+Evo” throughout the rest of the manuscript. Each of Elastix, PM-L2 and PM-EA aims to estimate a global 3D translation between I and J within the shortest possible time. A common factor of 8× in each spatial direction (i.e. the default value for PM-EA given in section 2.2.3) was used to down-sample the input data (i.e., I, J and M). On the other hand, elastic registration is known to be a complex task. Using Evo, a down-sampling of input data (i.e., I, J and M) by a factor 4× was used in order to maintain registration accuracy of the outputs and computation times compatible with our clinical constraints (below 30 seconds). 2.3. Hardware and implementation Our test platform was an Intel 2.5 GHz i7 workstation (8 cores) with 32 GB of RAM. The implementation was performed in C++ and parallelized through multi-threading (one thread per core). 3. Results Figure 2 provides a visual assessment of challenges arising from the use of CBCT during the intra-operative IRE session. Middle transversal, sagittal and coronal slices are illustrated for CBCT images acquired on the same patient before and after insertion of 4 needles. First, only voxels contained within a cylinder have non-zero values: a partial circular FOV in the transveral plane is observable in the first column (see 2a Patch-based field-of-view matching in multi-modal images for electroporation-based ablations12 and 2d). In addition, a low contrast-to-noise ratio is observable on both images. Also, an increasing amount of streaking artifacts is observable after needle insertions. An example of CT/CBCT and MR/CBCT registration results are shown in figures 3 and 4, respectively. For both cases, the CBCT image (first column) was acquired intra-operatively after needle insertions and was employed as a reference for image registration. The pre-operative image is displayed before registration (second column), after PM-EA (third column) and after PM-EA+Evo (fourth column). The occurrence of patch shifts is reported for each spatial direction in panels (m–o): for each histogram, the shift with maximal occurrence is shown by the red dashed line. For panels (a–l), a ROI — manually defined on the CBCT image/encompassing the liver — is shown using red dash lines. Our visualization shows an improved correspondence of the contour of the liver with the manually defined liver boundary when the PM-EA solution is employed (see 3(c,g,k) and 4(c,g,k)). Moreover, an even better correspondence of the contour is observable using the PM-EA+Evo solution (see 3(d,h,l) and 4(d,h,l)). Sagittal Coronal Transversal Before needle insertion (a) (b) (c) After needle insertion (d) (e) (f) Figure 2: Typical CBCT images obtained during an IRE procedure. Compared to the image acquired before needle insertion (top row), the image acquired after the insertion of four needles (bottom row) is visibly altered by streaking artifacts. The latter introduce intensity variations which obstructs and degrades finer details of the anatomy. The partial image FOV is also observable: only data contained within a cylinder are available (see the circular FOV in the transveral plane in the first column). Figure 5 analyzes the sensitivity to the down-sampling parameter (i.e., the input parameter #1 of the proposed PM-EA method, see section 2.2.3). A great DSC Patch-based field-of-view matching in multi-modal images for electroporation-based ablations13 CBCT CT / No registration CT / PM-EA CT / PM-EA+Evo Trans. [X-Y] (a) (b) (c) (d) Sag. [X-Z] (e) (f) (g) (h) Cor. [Y-Z] (i) (j) (k) (l) (m) (n) (o) Figure 3: Example of a CT/CBCT registration results. The CBCT image, used as a reference for registration, was acquired immediately after insertion of 4 needles. Transversal (a-d), sagittal (e-h) and coronal (i-l) cross-sections are reported for: CBCT (first column), CT before (second column) and after registration using PM-EA (third column) and PM-EA+Evo (fourth column). Histograms of X-, Y- and Z-shifts are reported in (m), (n) and (o), respectively (maximum occurrence in red dashed line). Patch-based field-of-view matching in multi-modal images for electroporation-based ablations14 improvement together with a huge speed-up of the algorithm was obtained for increasing down-sampling factors. This tendancy was observed for both CT/CBCT (5a) and MR/CBCT (5b). Best results were obtained using the default down-sampling factor of 8×. For all tested scenarios, differences between DSC obtained before and after needle insertions were not statistically significant (p ≥ 0.2 for CT/CBCT, p ≥ 0.3 for MR/CBCT). Figure 6 analyzes the impact of the patch size (i.e., the input parameter #2 of PM- EA). An improved registration accuracy with minimal losses in terms of computation time (several tenth of seconds) was obtained for an increasing patch size. This tendancy was observed for both CT/CBCT (6a) and MR/CBCT (6b). Best results were obtained using the default patch size of 9×9×9 voxels. For all tested scenarios, differences between DSC obtained before and after needle insertions were not statistically significant (p ≥ 0.2 for CT/CBCT, p ≥ 0.22 for MR/CBCT). Differences between DSC for each pair of tested numbers of histogram bins (i.e., the input parameter #3 of PM-EA) were not statistically significant (p ≥ 0.08 for both CT/CBCT and MR/CBCT) (see figure 7). This was observable for both CT/CBCT (7a) and MR/CBCT (7b). Here again, differences between DSC obtained before and after needle insertions were not statistically significant (p ≥ 0.11 for CT/CBCT, p ≥ 0.16 for MR/CBCT). Figure 8 analyzes the sensitivity of the proposed PM-EA algorithm against manual delineation errors of the organ of interest (i.e., the input parameter #4 of PM-EA). A significant negative impact is observable for eroded versions of M, especially for MR/CBCT (8b) (p ≤ 0.01). Using dilated versions of M, DSC differences were not statistically significant (p ≥ 0.3). Here again, differences between DSC obtained before and after needle insertions were not statistically significant (p ≥ 0.46 for CT/CBCT, p ≥ 0.3 for MR/CBCT). Figure 9 compares the registration accuracy obtained using all tested algorithms (see section 2.2.4 for details). Regarding solutions designed to estimate the global 3D shift of the liver (i.e., Elastix, PM-L2 and PM-EA), best results where achieved using the proposed PM-EA approach for both CT/CBCT (9a) and MR/CBCT (9b). PM- EA outperformed significantly a standard registration strategy implemented using the Elastix toolbox (p = 0.02). The use of the proposed multi-modal metric improved significantly the DSC obtained using the L2-norm used in the original PatchMatch paper (Barnes et al. 2009) (p = 0.01). Regarding solutions designed to estimate the elastic deformation of the liver (i.e., Evo and PM-EA+Evo), the use of PM-EA improved significantly the performance of the tested multi-modal elastic registration algorithm Evo (p = 0.01). For all tested solutions, differences between DSC obtained before and after needle insertions were not statistically significant (p ≥ 0.2 for CT/CBCT, p ≥ 0.08 for MR/CBCT). It is interesting to note that the computational demand remained here below 30 seconds for the successive achievement of PM-EA and Evo algorithms with the used hardware for both CT/CBCT and MR/CBCT pairs. Patch-based field-of-view matching in multi-modal images for electroporation-based ablations15 4. Discussion In the current study, we designed a novel method to estimate the global 3D translation between two multi-modal images. We retrospectively evaluate the proposed PM-EA algorithm under a realistic clinical scenarios. We focus on the specific interventional procedure of irreversible electroporation (IRE) ablation for liver tumors. IRE technique provides an interesting alternative to standard ablative techniques, especially for tumor located near vital structures as detailed in (Gallinato et al. 2019). Moreover, it gathers the main computational challenges in terms of medical image registrations, that have to be addressed to improve the procedures. Indeed, the procedures rely upon multimodal medical imaging: preoperative CT-scan or MRI to detect the target ablation region, and preoperative CBCT without and with needles to position the needles and to verify the positioning. Importantly, the needles positioning generate an elastic deformation of the liver, that has be accounted for as previously shown in (Gallinato et al. 2019), and thus nonrigid multimodal algorithm as EVolution (Denis de Senneville et al. 2016) has to be used. As far as we know, the current non rigid registration algorithm needed an initial manual tuning step to superimpose rouglhy the FOVs of two images of different modality. Importantly, the registration has be performed during the procedure, as demonstrated in (Gallinato et al. 2019) in order to provide a numerical assessment of the therapy to the physicians. There is therefore a crucial need to automatize the image preprocessing of FOV alignement in any electroporation ablation procedures. In the current study, the use of various imaging sensors (CT/CBCT, MR/CBCT image pairs, CBCTs being acquired intra-operatively) is analysed as well as the impact of needle insertions during IRE procedures. Using the proposed experimental setup, the registration process is hampered by the use of different image FOVs, especially when using CT during the pre-operative session (see the low DCS obtained in figure 9 before registration when using CT instead of MRI). Moreover, partial FOVs, cross-contrast variations, appearing/disappearing (anatomical or not) structures are also involved between the image to register and the reference one. Using such data sets, optimizing a simple translational model, as implemented in the Elastix toolbox, was found to be insufficient. The proposed regional registration approach using voxel patches provided a good structural compromise between the voxel- wise (as done with Evo) and “global shifts” (as done with Elastix in the scope of this study) approaches. Moreover, contrary to optimization methods which are inherently sensitive to local minima, PM-EA is able to deal with large translation amplitude, since potential patch matches are considered within the complete FOV, as described in section 2.1.3. We have also shown that the proposed multi-modal image similarity metric, which favors edge alignements irrespective of the gradient direction, outperforms the L2-norm proposed in the original PatchMatch paper (Barnes et al. 2009). Ultimately, we have shown that PM-EA may greatly improve the performance of an existing multi-modal elastic registration algorithm (Evo in the scope of this study). As expected, computation times were greatly reduced using down-sampled versions Patch-based field-of-view matching in multi-modal images for electroporation-based ablations16 of input data (see figure 5). This down-sampling step also acts as an inherent low-pass filter applied on I and J which improved the registration accuracy in our tests. Using the proposed default user-defined parameter (i.e., down-sampling factor of 8×), the average DSC with PM-EA exceeded 0.6 for both CT/CBCT, MR/CBCT image pairs together with a computation time cost below 3 seconds on a commodity hardware. The proposed multi-modal metric of Eq. (4) performs a weighted average over patches of the edge alignement score. Consequently, the patch size is an input parameter which can be increased in order to mitigate the fact that the observed image features might not be discriminative enough. Increasing the patch size may thus improve the robustness against anatomical structure without counterpart between the image to register and the reference one. This benefit was achievable with a moderate negative impact on the computation time. However, to some extent, increasing the patch size may be unable to cope with complex local tissue deformations. A good compromise in the choice of the patch size is thus essential for a reliable and accurate patch matching. Using the proposed default user-defined parameter (i.e., patch size of 9 × 9 × 9 voxels), PM-EA attained the best results in terms of DSC for all tested experimental conditions (CT/CBCT, MR/CBCT registration, needle insertions), as shown in figure 6. It can be noticed that the number of histogram bins had no impact on the overall results, as shown in figure 7. The default user-defined parameter (i.e., 50 bins) was thus well suited for all presented results. In our data, the liver undergoes complex deformations between the images being registered. The registration accuracy decreased for eroded versions of M, as shown in figure 8. Liver boundaries, which are needful contrast regions, are not taken into account in such a case. Moreover, the overall shift estimate is likely to differ from the global liver displacement if the manually defined mask M only includes a subset of the liver. Alternatively, no significant impact on the registration accuracy was observed for dilated versions of M in our tests. Therefore, the guideline is that M must include at least the targeted organ. 5. Conclusion The successfull completion of an interventional therapeutic workflow often relies on establishing a spatial coherence between images acquired by various sensors at different stages. The proposed PM-EA algorithm was validated in several complementary experiments. It was demonstrated that it outperforms existing registration solutions for the estimation of a global 3D translation between two multi-modal images. The method can be used as a pre-conditioning step for a more complex multi-modal elastic registration algorithm. The method was thereby beneficial for CT to CBCT and MRI to CBCT registration tasks, especially when highly different image FOVs are involved. In addition, this was achievable together with a computation time cost below 3 seconds on a commodity hardware using our experimental protocol. The proposed patch- based workflow thus represents an attractive asset for DIR at different stages of an Patch-based field-of-view matching in multi-modal images for electroporation-based ablations17 interventional procedure. Acknowledgment Experiments presented in this paper were carried out using the PlaFRIM experimental testbed, supported by Inria, CNRS (LABRI and IMB), Universit´e de Bordeaux, Bordeaux INP and Conseil R´egional d’Aquitaine (see https://www.plafrim.fr/). The authors thank the Laboratory of Excellence TRAIL ANR-10-LABX-57 for funding. This study has been carried out with the financial support of the French National Research Agency (ANR) in the frame of the “Investments for the future” Programme IdEx Bordeaux-CPU (ANR-10-IDEX-03-02). This research has been partly granted by the Plan Cancer project NUMEP (Inserm 11099), led by C.P. References Astola, J., Haavisto, P. & Neuvo, Y. (1990). Vector median filters, Proceedings of the IEEE 78(4): 678– Bao, L., Yang, Q. & Jin, H. (2014). Fast edge-preserving patchmatch for large displacement optical flow, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3534– Barnes, C., Shechtman, E., Finkelstein, A. & Goldman, D. B. (2009). PatchMatch: A randomized correspondence algorithm for structural image editing, ACM Transactions on Graphics (Proc. SIGGRAPH) 28(3). Bleyer, M., Rhemann, C. & Rother, C. (2011). Patchmatch stereo-stereo matching with slanted support windows, Bmvc, Vol. 11, pp. 1–11. Denis de Senneville, B., Ries, M., Maclair, G. & Moonen, C. (2011). MR-guided thermotherapy of abdominal organs using a robust PCA-based motion descriptor, IEEE Transactions on Medical Imaging 30(11): 1987–1995. Denis de Senneville, B., Zachiu, C., Ries, M. & Moonen, C. T. W. (2016). Evolution: an edge- based variational method for non-rigid multi-modal image registration, Physics in Medicine and Biology 61(20): 7377. Gallinato, O., Denis de Senneville, B., Seror, O. & Poignard, C. (2019). Numerical workflow of irreversible electroporation for deep-seated tumor, Physics in Medicine and Biology 64(5): 055016. Giraud, R., Ta, V.-T., Papadakis, N., Manj´on, J. V., Collins, D. L., Coup´e, P. & the Alzheimer’s Disease Neuroimaging Initiative (2016). An optimized PatchMatch for multi-scale and multi- feature label fusion, NeuroImage 124: 770–782. Guckenberger, M., Richter, A., Boda-Heggemann, J. & Lohr, F. (2012). Motion compensation in radiotherapy, Critical Reviews and trade; in Biomedical Engineering 40(3): 187–197. Heinrich, M., Jenkinson, M., Bhushan, M., Matin, T., Gleeson, F., Brady, S. & Schnabel, J. (2012). MIND: Modality independent neighbourhood descriptor for multi-modal deformable registration, Medical Image Analysis 16(7): 1423–1435. Hocquelet, A., Trillaud, H., Frulio, N., Papadopoulos, P., Balageas, P., Salut, C., Meyer, M., Blanc, J. F., Montaudon, M. & Denis de Senneville, B. (2016). Three-dimensional measurement of hepatocellular carcinoma ablation zones and margins for predicting local tumor progression, Journal of Vascular and Interventional Radiology 27(7): 1038 –1045.e2. Holbrook, A. B., Santos, J. M., Kaye, E., Rieke, V. & Butts Pauly, K. (2009). Real-time MR thermometry for monitoring HIFU ablations of the liver, Magnetic Resonance in Medicine 63(2): 365–373. Patch-based field-of-view matching in multi-modal images for electroporation-based ablations18 Horn, B. & Schunck, B. (1981). Determining optical flow, Artificial intelligence 17: 185–203. Irani, M. & Anandan, P. (1998). Robust multi-sensor image alignment, IEEE Computer Vision, Sixth International Conference, pp. 959–966. Jakubowski, M. & Pastuszak, G. (2013). Block-based motion estimation algorithms — a survey, Opto- Electronics Review 21(1). Klein, S., Staring, M., Murphy, K., Viergever, M. & Pluim, J. (2010). elastix: A toolbox for intensity- based medical image registration, IEEE Transactions on Medical Imaging 29(1): 196–205. Lafitte, L., Zachiu, C., Kerkmeijer, L. G. W., Ries, M. & Denis de Senneville, B. (2018). Accelerating multi-modal image registration using a supervoxel-based variational framework, Physics in Medicine and Biology 63(23): 235009. Mougenot, C., Quesson, B., Denis de Senneville, B., de Oliveira, P., Sprinkhuizen, S., Palussiere, J., Grenier, N. & Moonen, C. T. W. (2009). Three-dimensional spatial and temporal temperature control with MR thermometry-guided focused ultrasound (MRgHIFU), Magnetic Resonance in Medicine 61: 603–614. Newson, A., Almansa, A., Fradet, M., Gousseau, Y. & P´erez, P. (2014). Video inpainting of complex scenes, SIAM Journal on Imaging Sciences 7(4): 1993–2019. Rivaz, H., Karimaghaloo, Z. & Collins, D. L. (2014). Self - similarity weighted mutual information: A new nonrigid image similarity metric, Med Image Anal 18(2): 343 – 358. Rubeaux, M., Simon, A., Gnep, K., Colliaux, J., Acosta, O., de Crevoisier, R. & Haigron, P. (2013). Evaluation of non-rigid constrained CT/CBCT registration algorithms for delineation propagation in the context of prostate cancer radiotherapy, Medical Imaging 2013: Image-guided procedures, robotic interventions and modeling, Vol. 8671, SPIE Proceedings. Shamonin, D. (2013). Fast parallel image registration on cpu and gpu for diagnostic classification of alzheimer’s disease, Frontiers in Neuroinformatics 7. Studholme, C., Hill, D. & Hawkes, D. (1999). An overlap invariant entropy measure of 3d medical image alignment, pattern recognition, 32(1): 71–86. Sutour, C., Aujol, J. F., Deledalle, C. A. & Denis de Senneville, B. (2015). Edge-based multi-modal registration and application for night vision devices, Journal of Mathematical Imaging and Vision 53: 131–150. Sutter, O., Fihri, A., Ourabia-Belkacem, R., Sellier, N., Diallo, A. & Seror, O. (2018). Real-time 3D virtual target fluoroscopic display for challenging hepatocellular carcinoma ablations using cone beam CT, Technology in cancer research & treatment 17: 1533033818789634. Zachiu, C., Denis de Senneville, B., Dmitriev, I. D., Moonen, C. T. W. & Ries, M. (2017). A framework for continuous target tracking during MR-guided high intensity focused ultrasound thermal ablations in the abdomen, Journal of Therapeutic Ultrasound 5(1): 27. Zachiu, C., Denis de Senneville, B., Tijssen, R. H. N., Kotte, A. N. T. J., C., H. A., Kerkmeijer, L. G. W., Lagendijk, J. J. W., Moonen, C. T. W. & Ries, M. G. (2017). Non-rigid CT/CBCT to CBCT registration for online external beam radiotherapy guidance, Physics in Medicine and Biology 63(1): 015027. Patch-based field-of-view matching in multi-modal images for electroporation-based ablations19 CBCT MRI / No registration MRI / PM-EA MRI / PM-EA+Evo Trans. [X-Y] (a) (b) (c) (d) Sag. [X-Z] (e) (f) (g) (h) Cor. [Y-Z] (i) (j) (k) (l) (m) (n) (o) Figure 4: Example of a MR/CBCT registration results. The CBCT image, used as a reference for registration, was acquired immediately after insertion of 3 needles. Transversal (a-d), sagittal (e-h) and coronal (i-l) cross-sections are reported for: CBCT (first column), MRI before (second column) and after registration using PM-EA (third column) and PM-EA+Evo (fourth column). Histograms of X-, Y- and Z-shifts are reported in (m), (n) and (o), respectively (maximum occurrence in red dashed line). Patch-based field-of-view matching in multi-modal images for electroporation-based ablations20 CT/CBCT registration MR/CBCT registration No needle Needles No needle Needles Image down-sampling Image down-sampling (a) (b) Figure 5: Analysis of the impact of the down-sampling of input data on the performance of the proposed PM-EA method. DSC (left Y-axis) and computation times (red dashed line/right Y-axis) are reported for the registration of CT/CBCT (a) and MR/CBCT (b) pairs for down-sampling factors 2× (DS-2), 4× (DS-4) and 8× (DS-8). We recall that the patch size was here fixed to 9 × 9 × 9 voxels. The number of histogram bins was fixed to a value of 50. CT/CBCT registration MR/CBCT registration 1 4 1 4 No needle Needles No needle Needles 0.8 0.8 3.5 3.5 0.6 0.6 3 3 0.4 0.4 2.5 2.5 0.2 0.2 0 2 0 2 3x3x3 5x5x5 7x7x7 9x9x9 11x11x11 3x3x3 5x5x5 7x7x7 9x9x9 11x11x11 Patch size Patch size (a) (b) Figure 6: Analysis of the impact of the patch size on the performance of the proposed PM-EA method. DSC (left Y-axis) and computation times (red dashed line/right Y- axis) are reported for the registration of CT/CBCT (a) and MR/CBCT (b) pairs. We recall that the down-sampling factor of input data was here fixed to 4×. The number of histogram bins was fixed to a value of 50. DSC [a.u] DSC [a.u] Computation time [s] Computation time [s] DSC [a.u] DSC [a.u] Computation time [s] Computation time [s] Patch-based field-of-view matching in multi-modal images for electroporation-based ablations21 CT/CBCT registration MR/CBCT registration No needle Needles No needle Needles Histogram bins [#] Histogram bins [#] (a) (b) Figure 7: Analysis of the impact of the number of histogram bins on the performance of the proposed PM-EA method. DSC (left Y-axis) and computation times (red dashed line/right Y-axis) are reported for the registration of CT/CBCT (a) and MR/CBCT (b) pairs. We recall that the down-sampling factor of input data was here fixed to 4×. The patch size was here fixed to 9 × 9 × 9. CT/CBCT registration MR/CBCT registration No needle Needles No needle Needles Masking sensibility [%] Masking sensibility [%] (a) (b) Figure 8: Analysis of the impact of errors occurred in the targeted organ delineation process performed on the pre-operative image I. DSC [a.u] DSC [a.u] DSC [a.u] DSC [a.u] Patch-based field-of-view matching in multi-modal images for electroporation-based ablations22 CT/CBCT registration MR/CBCT registration No needle Needles No needle Needles (a) (b) Figure 9: Summary of DSC scores obtained for the registration of CT/CBCT (a) and MR/CBCT (b) images, using tested solutions detailed in section 2.2.4. Standard deviations over the patients are given by the size of the black error bars. We recall that the down-sampling factor of input data was here fixed to 4×. The patch size was fixed to 9 × 9 × 9 voxels. The number of histogram bins was fixed to a value of 50. DSC [a.u] DSC [a.u] http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Electrical Engineering and Systems Science arXiv (Cornell University)

Patch-based field-of-view matching in multi-modal images for electroporation-based ablations

Download PDF
22 pages

Loading next page...
 
/lp/arxiv-cornell-university/patch-based-field-of-view-matching-in-multi-modal-images-for-e2FhDPT8Hh

References (36)

ISSN
0895-6111
eISSN
ARCH-3348
DOI
10.1016/j.compmedimag.2020.101750
Publisher site
See Article on Publisher Site

Abstract

Patch-based field-of-view matching in multi-modal images for electroporation-based ablations 1 2 3 4 Luc Lafitte , R´emi Giraud , Cornel Zachiu , Mario Ries , 5,6 5,6 5,6 Olivier Sutter , Antoine Petit , Olivier Seror , Clair 1 1,3 Poignard , Baudouin Denis de Senneville University of Bordeaux, IMB, UMR CNRS 5251, INRIA Project team Monc, Talence, France, F-33405 Talence Cedex, France University of Bordeaux, IMS, CNRS UMR 5218, F-33405 Talence Cedex, France Department of Radiotherapy, UMC Utrecht, Heidelberglaan 100, 3584 CX, Utrecht, The Netherlands Imaging Division, UMC Utrecht, Heidelberglaan 100, 3584 CX, Utrecht, The Netherlands Interventional radiology unit, Hoˆpitaux Universitaires Paris Seine Saint Denis, Hoˆpital Avicenne, Assistance Publique Hoˆpitaux de Paris, Bobigny France University of Paris 13, “Sciences M´edicale et Biologie Humaine”, Bobigny, France Abstract. Various multi-modal imaging sensors are currently involved at different steps of an interventional therapeutic work-flow. Cone beam computed tomography (CBCT), computed tomography (CT) or Magnetic Resonance (MR) images thereby provides complementary functional and/or structural information of the targeted region and organs at risk. Merging this information relies on a correct spatial alignment of the observed anatomy between the acquired images. This can be achieved by the means of multi-modal deformable image registration (DIR), demonstrated to be capable of estimating dense and elastic deformations between images acquired by multiple imaging devices. However, due to the typically different field-of-view (FOV) sampled across the various imaging modalities, such algorithms may severely fail in finding a satisfactory solution. In the current study we propose a new fast method to align the FOV in multi-modal 3D medical images. To this end, a patch-based approach is introduced and combined with a state-of-the-art multi-modal image similarity metric in order to cope with multi- modal medical images. The occurrence of estimated patch shifts is computed for each spatial direction and the shift value with maximum occurrence is selected and used to adjust the image field-of-view. The performance of the proposed method — in terms of both registration accuracy and computational needs — is analyzed in the practical case of on-line irreversible electroporation procedures. In total, 30 pairs of pre-/per- operative IRE images are considered to illustrate the efficiency of our algorithm. We show that a regional registration approach using voxel patches provides a good structural compromise between the voxel-wise and “global shifts” approaches. The method was thereby beneficial for CT to CBCT and MRI to CBCT registration tasks, especially when highly different image FOVs are involved. Besides, the benefit of the method for CT to CBCT and MRI to CBCT image registration is analyzed, including the impact of artifacts generated by percutaneous needle insertions. Additionally, the computational needs using commodity hardware are demonstrated to be compatible with clinical constraints in the practical case of on-line procedures. The proposed arXiv:2011.11759v1 [eess.IV] 9 Nov 2020 Patch-based field-of-view matching in multi-modal images for electroporation-based ablations2 patch-based workflow thus represents an attractive asset for DIR at different stages of an interventional procedure. Keywords: Multi-modal image registration, patch-based matching, interventional procedures Patch-based field-of-view matching in multi-modal images for electroporation-based ablations3 1. Introduction Multiple imaging devices can be involved at different stages of an interventional procedure, such as image-guided radiotherapy (IGRT) (Guckenberger et al. 2012), irreversible electroporation (IRE) (Gallinato et al. 2019) or hyperthermia ablation (Holbrook et al. 2009) (Mougenot et al. 2009). In particular, cone-beam computed tomography (CBCT), computed tomography (CT) or Magnetic Resonance (MR) images are recently being employed at nearly all stages of the therapy: i.e., pre-, intra- and post-operatively. One of the benefits that employing multiple imaging sensors provides, is the ability to extract complementary functional and/or structural information of the targeted region and organs-at-risk. For example, as shown in (Hocquelet et al. 2016), novel diagnostic indicators can thereby be calculated by fusing pre- and post-operative image data. Similarly, multi-modal imaging may also be beneficial during the interventional procedure itself (Zachiu, Denis de Senneville, Dmitriev, Moonen & Ries 2017). Of note is that the quality and the amount of data that can be acquired intra- operatively is generally limited by practical clinical considerations: CBCT guidance is for example particularly beneficial due to the low amount of imaging-related radiation delivered to the patient compared to a conventional CT scan. However, this often leads to the intra-operative images being subject to low contrast, low signal-to-noise ratio and artifacts. Thus, it would be of clinical benefit if such images would be augmented by pre- operative data (Gallinato et al. 2019). A common pre-requisite is that organ locations must be set in a common frame of reference. To this end, previous studies propose several multi-modal deformable image registration (DIR) algorithms dedicated to the estimation of dense and elastic deformations between images (Heinrich et al. 2012, Rivaz et al. 2014, Denis de Senneville et al. 2016). This remains a challenging task since such algorithms have to be fast (to meet clinically acceptable durations) and automatic (the use must not be limited to a case-by-case basis and a manual recalibration is not preferable), especially when the patient is on the interventional table (Rubeaux et al. 2013) (Zachiu, Denis de Senneville, Tijssen, Kotte, C., Kerkmeijer, Lagendijk, Moonen & Ries 2017). A particular challenge arises when highly different fields-of-view (FOV) are sampled within the images. For example, while the FOV within intra-operative CBCT images is typically restricted to the targeted organ and its immediate surroundings, the corresponding pre-operative high-resolution CT image generally covers the entire abdomen and part of the thorax. A similar situation arises when a patient is screened via both CT/CBCT and MR imaging, with the resulting acquisitions typically having considerably different FOVs. This can severely hamper the performance of image registration algorithms, especially when iterative optimization strategies are employed: the algorithm is likely to get trapped into local optima if the apparent location of the anatomy-of-interest is too far apart within the two images. In such a case, a direct employment of DIR methods may be hardly feasible and a preliminary matching of the image FOVs (i.e., compensation of the 3D global shift between images) is necessary. Patch-based field-of-view matching in multi-modal images for electroporation-based ablations4 While registration solutions optimizing a translational model may perform well for estimating rigid displacements, they may also become sub-optimal when elastic deformations are present between the images. Moreover, such methods typically imply high computational demands and manual tuning of several input parameters, which limits their use in a clinical setting (Klein et al. 2010). Alternatively, a regional registration approach using pixel/voxel patches, may provide a good structural compromise between the voxel-wise and “global shifts” approaches. Several patch or block-matching algorithms have been previously proposed, dedicated to various applications (Jakubowski & Pastuszak 2013). The aim of these approaches is to consider each pixel by its square neighborhood, to characterize its local context. Matching algorithms may then be used to find local correspondences between images. Nevertheless, these methods are highly time consuming, especially when dealing with an important number of patches and when the search for correspondences must be performed in a large window search (i.e., searching for large patch displacements). A significant breakthrough has been obtained with the so-called “PatchMatch” algorithm (Barnes et al. 2009), which was initially proposed for finding pixel patch correspondences between 2D images in digital photography. The idea behind this approach is that some good patch matches can be found by random sampling, which can subsequently be allocated to surrounding areas as well, relying on the assumption that neighboring areas typically have similar displacements. The fast convergence of the process enables to quickly find good matches, even when these are located far from each other in the image spaces. This approach has been successfully employed to achieve numerous image analysis and editing tasks such as: stereo matching (Bleyer et al. 2011), optical flow computation (Bao et al. 2014), region inpainting (Newson et al. 2014), or 3D medical image segmentation (Giraud et al. 2016). In the current study, our contribution is four-fold: (i) A new fast method — using as a starting-point a 3D modified PatchMatch algorithm — is proposed to align the FOV in medical 3D images. A user-defined mask surrounding a region/organ of interest in one of the images can be provided as an input so that a global shift can be estimated relying on image information from this specific region. (ii) The modified PatchMatch algorithm is combined with a well-adapted multi-modal image similarity metric in order to cope with multi-modal medical images. (iii) The performance — in terms of registration accuracy and computational needs — is analyzed, and demonstrated to be compatible with clinical constraints in the practical case of on-line irreversible electroporation procedures. In total, 30 pairs of pre-/per- operative IRE images are considered to illustrate the efficiency of our algorithm. (iv) The benefit of proposed approach is evaluated for a potential pre-conditioning of a more complex multi-modal DIR algorithm. Patch-based field-of-view matching in multi-modal images for electroporation-based ablations5 2. Materials and Methods 2.1. Proposed method Let I and J be two 3D images. In the scope of this study, let I and J be a pre- and an intra-operative image, respectively. We seek X-, Y- and Z- translation components between I and J in order to match the position an organ of interest manually delineated in I. We recall that the estimation of elastic organ deformations by itself is outside the scope of the study: the proposed workflow is solely intend to standardize field-of-view in multi-modal images for a potential pre-conditioning a more complex multi-modal elastic registration algorithm. The proposed method (detailed in Figure 1) includes the following three main successive steps: (i) The PatchMatch (PM) algorithm is adapted and combined with a multi-modal metric in order to compute patch correspondences between I and J (see section 2.1.3). The multi-modal metric aims at evaluating edge alignments (EA) within patches. (ii) The occurrence of estimated patch shifts, i.e., the displacement between the patch positions in I and their correspondence in J, is computed for each spatial direction (see section 2.1.4). (iii) For each spatial direction, the shift value with maximum occurrence is selected and used to adjust the image field-of-view (see section 2.1.5). The proposed method is referred to as “PM-EA” (PatchMatch-Edge Alignment) in the scope of this study. Figure 1: Data processing sequence designed for the fast standardization of field-of-view in multi-modal images using the proposed patch-based framework. Patch-based field-of-view matching in multi-modal images for electroporation-based ablations6 2.1.1. Manual delineation of the organ of interest. The pre-operative image I is first used to manually segment the targeted region of interest. A binary mask (denoted by M) is constructed. Voxels of the image inside the mask have a value of one, and outside a value of zero. We underline that, in the scope of this study, this process was done using the pre-operative image I only, in order to demonstrate that the method is compatible with an automatic use during an intra-operative session. Note that this delineation step is often performed anyway during the planning session of the therapy and thus does not put extra burden on the medical staff. 2.1.2. Preprocessing of input data. I, J and M were resampled onto a common grid with a voxel size of 1 × 1 × 1 millimeters using a trilinear interpolation. 2.1.3. Implemented PatchMatch algorithm. PatchMatch is an iterative algorithm designed to quickly estimate patch correspondences between two given 2D images (Barnes et al. 2009). In our study, we first extend this algorithm for the matching of 3D patches. Hence, a patch consists in a cubic subset of the image domain, denoted by Γ, centered on one single voxel. Let ~r = (x, y, z) ∈ Ω be the spatial location of the center voxel, Ω the image domain and (x, y, z) the voxel coordinates. • Initialization. An initial guess is first computed: each patch from image I is initially randomly matched with a patch from image J. Subsequently, at each iteration of PatchMatch, voxels are scanned from left to right (X-axis), head to foot (Y- axis), front to back (Z-axis). For each voxel examination, the corresponding patch undergoes a “propagation step” followed by a “random search step”, as described in the seminal paper (Barnes et al. 2009): The output of our algorithm is a patch shift map V , defined for each voxel in Ω. • Propagation step. During this step, in order to find better correspondences, the patch shift of the current voxel ~r in I in I is considered to be similar to the ones of its three already examined neighboors in each direction (6-connexity) (i.e., the three voxels at locations ~r = (x − 1, y, z), ~r = (x, y − 1, z) and (−1,0,0) (0,−1,0) ~r = (x, y, z − 1)). (0,0,−1) Let V = (u, v, w) be the patch shifts that we seek ((u, v, w) being the voxelwise patch shift coordinates). Let ~r and ~r be two given spatial location in Ω and 1 2 D(~r , V (~r )) the distance between the patch at location ~r in I and the patch at 1 2 1 ~ ~ location r~ + V (~r ) in J. V (~r) is updated as follows: 1 2 ~ ~ ~ ~ ~ V (~r) = argmin {D(~r, V (~r)), D(~r, V (~r )), D(~r, V (~r )), D(~r, V (~r ))} (−1,0,0) (0,−1,0) (0,0,−1) (1) • Random search step. This step attempts to improve V (~r) by computing a set of candidate shifts (noted V (~r)) at an exponentially decreasing spatial distance from V (~r): Patch-based field-of-view matching in multi-modal images for electroporation-based ablations7 ~ ~ V (~r) = V (~r) + wα R (2) i i R being a uniform random in [−1, 1] , w the maximum search distance (set to the maximum image dimension), and α a fixed ratio between search window sizes (we took α = 0.5, as suggested in (Barnes et al. 2009)). Patches for i = 0, 1, 2 and so on are examined until the current search distance wα falls below one voxel. As reported in Barnes et al seminal paper, PatchMatch provides satisfactory results using a fixed number of iterations (5 iterations max) and that the algorithm converges most rapidly in the first iterations (Barnes et al. 2009). In the current study we used two iterations, since it was found to be a good compromise between accuracy and computational costs. In our implementation, the input images are down-sampled before PatchMatch: while lower computation times are expected using down-sampled versions of input images, it should also impact overall registration results. The down-sampling factor is thus an important input parameter for the algorithm and its impact will be carrefully analysed and discussed below. To further reduce the computational burden, the search window were a rectangular bounding box including both J and voxels with a value of one in M. • Aggregation of multiple PM estimations. As for exemplar-based segmentation (Giraud et al. 2016), our method can benefit from multiple PM estimations. Patch- shift estimates indeed rely on random candidate selection and several independent processes may provide different correspondences. Although PatchMatch inherently relies on serial operations and cannot benefit from parallel architectures in its current form, multiple realisations of PatchMatch can easily be calculated using j j j j separate CPU threads. Let V (~r) = (u (~r), v (~r), w (~r)) be one realisation at spatial location ~r, j being the realisation index. For each voxel, the obtained 3D shift realisations were then combined into a single 3D median vector V (~r) (Astola et al. 1990) as follows: V (~r) = argmin V (~r) − V (~r) (3) V (~ r)∈{V (~ r)} In this manner, outliers realisations were discarded without any additional penalty on the computational burden. 2.1.4. Proposed multi-modal metric. In the original PatchMatch paper, the distance between patches D(.) is the Sum of Squared Differences (SSD, also referred to as L2- norm) applied on the image intensity. While for mono-modal registration algorithms the SSD applied directly on the images might be sufficient (Horn & Schunck 1981) (Denis de Senneville et al. 2011), such a measure is unsuitable for registering across modalities. A modality independent similarity measure is thus necessary. In the current paper we Patch-based field-of-view matching in multi-modal images for electroporation-based ablations8 used an existing multi-modal metric which favors edge alignments (EA) in both patches (Irani & Anandan 1998) (Sutour et al. 2015). ~ ~ Let ∇ and ∇ be the gradient of the reference image I and the image to register I J J, respectively. The distance D(~r , V (~r )) between a patch of interest Γ in I (centered 1 2 on the voxel located in ~r ) and its potential correspondence in J (centered on the voxel located in ~r + V (~r )) was defined as follows: 1 2 ~ ~ ~ ∇ (~r ) · ∇ ~r + V (~r ) d~r I 1 J 1 2 D(~r , V (~r )) = − (4) 1 2 ~ ~ ~ ∇ (~r ) ∇ ~r + V (~r ) d~r I 1 J 1 2 2 2 where k · k is the Euclidean norm. Practically, the scalar product in the numerator is maximized when the edges in Γ are aligned with edges in the potential corresponding patch in J. Note that the numerator is maximized regardless any possible contrast reversals: due to the absolute value, the numerator is maximized for both parallel and anti-parallel edges. In addition, the scalar product in the numerator favors strong edges present in both modalities. The denominator, for its part, acts as a normalisation factor. Ultimately, since a minimization of D is required to compute patch correspondences in Eq. (1), a negative sign has been set behing the fractional term in Eq. (4). 2.1.5. Matching image FOVs from estimated patch correspondences. At this point we ~ ~ have a voxelwise 3D shift maps V (~r), ~r ∈ Ω. The objective is now to simplifiy V down to a single 3D image shift. For this purpose, we individually analysed shift occurrences in each component of V (i.e., u, v and w) and within the binary mask M encompassing the target organ: for each component, a histogram was calculated and the shift value with the highest occurrence was selected. The obtained 3D image shift was subsequently used to adjust the FOV of J with respect to the one of I. Note that lower and upper bounds on estimated shift values as well as a number of bins needs to be determined for the construction of the histograms. The lower and upper bounds were set to -100 and 100 millimeters, respectively, which was found to be sufficient in our tests. The number of bins is an input parameter for the algorithm that will be analysed below. 2.2. Experimental evaluation 2.2.1. Data sets. For the evaluation of the method we used data acquired during IRE procedures which are routinely performed at the University Hospital Jean Verdier at Bondy in France. This retrospective study is in accordance with ethical principals of the Declaration of Helsinki and has been approved by the local committee on human research of the University Hospital J Verdier. The clinical workflow included the following sessions: Patch-based field-of-view matching in multi-modal images for electroporation-based ablations9 • Pre-operative session. This session, performed several days before the interventional procedure, allowed identifying the tumor and the main liver structures using either a CT-scan (voxel size=[0.67 − 0.88] × [0.67 − 0.88] × [1.25 − 2] mm , FOV=[341 − 450] × [341 − 450] × [182 − 506] mm ) or a MR-scan (T1-weighting, 3 3 voxel size=1.72 × 1.72 × [2.5 − 3] mm , FOV=440 × 440 × [180 − 200] mm ). • Interventional session. The day of the procedure, an IRE ablation was performed under general anesthesia. The needles are percutaneously inserted around the tumor by the interventional radiologist with a free-hand technique under combination of real-time ultrasound (US) and 3D Virtual Target Fluoroscopic Display such that the electric field covers the target region (Sutter et al. 2018). A 3D CBCT (voxel size=0.45 × 0.45 × 0.45 mm , FOV=230 × 230 × [192 − 256] mm ) imaging was performed to visualise liver and needle locations. During the interventional session, a registration with pre-operative data is intend to augment the CBCT with tumor/liver structures segmentations. The pursued objective is to improve the targeting and to allow dose modeling, as described in (Gallinato et al. 2019). In total, we analysed a set of 30 pairs of pre-/intra-operative images distributed over the four following groups: (i) 8 pairs of CT/CBCT images obtained on 8 patients, respectively. CBCTs were acquired before needle insertion. (ii) 8 pairs of CT/CBCT images. Patients and CTs were those used in (i). CBCTs were acquired after needle insertion. (iii) 7 pairs of MR/CBCT images obtained on 8 patients, respectively. CBCTs were acquired before needle insertion. (iv) 7 pairs of MR/CBCT images. Patients and MRs were those used in (iii). CBCTs were acquired after needle insertion. 2.2.2. Performance assessment. The Dice Similarity Coefficient (DSC) was employed to determine the contour overlap of the liver: 2 |A ∩ B| DSC = (5) |A| + |B| where A and B are two manually defined ROIs encompassing the liver in the reference and the corrected image, respectively. A ∩ B is their intersection and |·| denotes the cardinality of a set (i.e., the number of voxels). DSC mean and standard deviation were computed over the 4 sets of image pairs individually (i.e., CT/CBCT no needle, CT/CBCT needles, MR/CBCT no needle, MR/CBCT needles, as defined in section 2.2.1). A Wilcoxon paired test was carried out in order to study whether DSC differences are statistically significant. A significance threshold of p = 0.05 was used. Patch-based field-of-view matching in multi-modal images for electroporation-based ablations10 2.2.3. Calibration of the proposed PM-EA algorithm. At this point it is important to underline that four main input parameters may influence the performance of the proposed approach. The proposed PM-EA algorithm was challenged against various modifications applied to these calibration parameters: Input parameter #1: down-sampling of input images (section 2.1.3). DSC and computation time were calculated using I, J and M at original image dimension and using down-sampling (DS) factors 2×, 4× and 8× in each spatial direction. A default down-sampling factor of 8× was used. Input parameter #2: patch size (section 2.1.4). DSC and computation time were calculated using patches of size 3 × 3 × 3, 5 × 5 × 5, 7 × 7 × 7, 9 × 9 × 9 and 11 × 11 × 11 voxels. A default size of 9 × 9 × 9 voxels was used. Input parameter #3: number of histogram bins (section 2.1.5). DSC were calculated for number of histogram bins of 10, 30, 50, 70 and 90. The computation time was not evaluated since a marginal impact is expected here. A default value of 50 was used. Input parameter #4: manual delineation errors of the targeted organ (section 2.1.1). To analyse potential errors arising from the manual delineation process, M was iteratively eroded (resp. dilated) using a 5 × 5 × 5 kernel. At each iteration, the volume of the eroded (resp. dilated) mask was calculated as well as the corresponding DSC after image alignement. Here again, the computation time was not evaluated since a marginal impact is expected. 2.2.4. Tested algorithms. The PM-EA’s ability to estimate a global 3D translation between images was challenged against two competing approaches. We also evaluated the benefit of using PM-EA as a starting point for an existing more complex multi-modal elastic registration algorithm. The above-mentioned 30 pairs of images were processed using PM-EA and using the following selection of image registration solutions: Elastix. We have selected in the Open source Elastix registration software (Klein et al. 2010) (Shamonin 2013) a registration solution which employs a 3D translation transformation model and which maximizes the normalized mutual information (Studholme et al. 1999) between the images to be registered. This registration solution is referred to as “Elastix” in the following. PM-L2. The proposed PatchMatch registration framework is here employed using a L2-norm for patch comparison, as it was introduced in the original paper (Barnes et al. 2009). This registration solution is referred to as “PM-L2” hereafter. Patch-based field-of-view matching in multi-modal images for electroporation-based ablations11 Evo. The EVolution algorithm (which is abbreviated as “Evo” in the scope of this study) was employed to estimate the elastic deformation V as the minimizer of the following energy E: 2 2 2 ~ ~ ~ ~ ~ E(V ) = exp(D(V )) + k ∇u k + k ∇v k + k ∇w k d~r, (6) 2 2 2 D(V ) being the multi-modal metric of Eq. (4) and α a weighting factor designed to link both the data fidelity term (left part of Eq. (6)) and the motion field regularity (right part of Eq. (6)). Note that D(V ) is composed with an exponential function in order to ensure that the data fidelity term is a positive-definite function. Additional details concerning the manner in which the Evo functional is minimized together with a detailed analysis of the algorithm performance can be found in (Denis de Senneville et al. 2016). A numerical implementation designed for the reduction of computational costs can be found in (Lafitte et al. 2018). PM-EA+Evo. The above-mentioned multi-modal registration algorithm Evo is here employed after FOV standardization using proposed PM-EA algorithm. The registration workflow, comprised of PM-EA followed by Evo, is referred to as “PM- EA+Evo” throughout the rest of the manuscript. Each of Elastix, PM-L2 and PM-EA aims to estimate a global 3D translation between I and J within the shortest possible time. A common factor of 8× in each spatial direction (i.e. the default value for PM-EA given in section 2.2.3) was used to down-sample the input data (i.e., I, J and M). On the other hand, elastic registration is known to be a complex task. Using Evo, a down-sampling of input data (i.e., I, J and M) by a factor 4× was used in order to maintain registration accuracy of the outputs and computation times compatible with our clinical constraints (below 30 seconds). 2.3. Hardware and implementation Our test platform was an Intel 2.5 GHz i7 workstation (8 cores) with 32 GB of RAM. The implementation was performed in C++ and parallelized through multi-threading (one thread per core). 3. Results Figure 2 provides a visual assessment of challenges arising from the use of CBCT during the intra-operative IRE session. Middle transversal, sagittal and coronal slices are illustrated for CBCT images acquired on the same patient before and after insertion of 4 needles. First, only voxels contained within a cylinder have non-zero values: a partial circular FOV in the transveral plane is observable in the first column (see 2a Patch-based field-of-view matching in multi-modal images for electroporation-based ablations12 and 2d). In addition, a low contrast-to-noise ratio is observable on both images. Also, an increasing amount of streaking artifacts is observable after needle insertions. An example of CT/CBCT and MR/CBCT registration results are shown in figures 3 and 4, respectively. For both cases, the CBCT image (first column) was acquired intra-operatively after needle insertions and was employed as a reference for image registration. The pre-operative image is displayed before registration (second column), after PM-EA (third column) and after PM-EA+Evo (fourth column). The occurrence of patch shifts is reported for each spatial direction in panels (m–o): for each histogram, the shift with maximal occurrence is shown by the red dashed line. For panels (a–l), a ROI — manually defined on the CBCT image/encompassing the liver — is shown using red dash lines. Our visualization shows an improved correspondence of the contour of the liver with the manually defined liver boundary when the PM-EA solution is employed (see 3(c,g,k) and 4(c,g,k)). Moreover, an even better correspondence of the contour is observable using the PM-EA+Evo solution (see 3(d,h,l) and 4(d,h,l)). Sagittal Coronal Transversal Before needle insertion (a) (b) (c) After needle insertion (d) (e) (f) Figure 2: Typical CBCT images obtained during an IRE procedure. Compared to the image acquired before needle insertion (top row), the image acquired after the insertion of four needles (bottom row) is visibly altered by streaking artifacts. The latter introduce intensity variations which obstructs and degrades finer details of the anatomy. The partial image FOV is also observable: only data contained within a cylinder are available (see the circular FOV in the transveral plane in the first column). Figure 5 analyzes the sensitivity to the down-sampling parameter (i.e., the input parameter #1 of the proposed PM-EA method, see section 2.2.3). A great DSC Patch-based field-of-view matching in multi-modal images for electroporation-based ablations13 CBCT CT / No registration CT / PM-EA CT / PM-EA+Evo Trans. [X-Y] (a) (b) (c) (d) Sag. [X-Z] (e) (f) (g) (h) Cor. [Y-Z] (i) (j) (k) (l) (m) (n) (o) Figure 3: Example of a CT/CBCT registration results. The CBCT image, used as a reference for registration, was acquired immediately after insertion of 4 needles. Transversal (a-d), sagittal (e-h) and coronal (i-l) cross-sections are reported for: CBCT (first column), CT before (second column) and after registration using PM-EA (third column) and PM-EA+Evo (fourth column). Histograms of X-, Y- and Z-shifts are reported in (m), (n) and (o), respectively (maximum occurrence in red dashed line). Patch-based field-of-view matching in multi-modal images for electroporation-based ablations14 improvement together with a huge speed-up of the algorithm was obtained for increasing down-sampling factors. This tendancy was observed for both CT/CBCT (5a) and MR/CBCT (5b). Best results were obtained using the default down-sampling factor of 8×. For all tested scenarios, differences between DSC obtained before and after needle insertions were not statistically significant (p ≥ 0.2 for CT/CBCT, p ≥ 0.3 for MR/CBCT). Figure 6 analyzes the impact of the patch size (i.e., the input parameter #2 of PM- EA). An improved registration accuracy with minimal losses in terms of computation time (several tenth of seconds) was obtained for an increasing patch size. This tendancy was observed for both CT/CBCT (6a) and MR/CBCT (6b). Best results were obtained using the default patch size of 9×9×9 voxels. For all tested scenarios, differences between DSC obtained before and after needle insertions were not statistically significant (p ≥ 0.2 for CT/CBCT, p ≥ 0.22 for MR/CBCT). Differences between DSC for each pair of tested numbers of histogram bins (i.e., the input parameter #3 of PM-EA) were not statistically significant (p ≥ 0.08 for both CT/CBCT and MR/CBCT) (see figure 7). This was observable for both CT/CBCT (7a) and MR/CBCT (7b). Here again, differences between DSC obtained before and after needle insertions were not statistically significant (p ≥ 0.11 for CT/CBCT, p ≥ 0.16 for MR/CBCT). Figure 8 analyzes the sensitivity of the proposed PM-EA algorithm against manual delineation errors of the organ of interest (i.e., the input parameter #4 of PM-EA). A significant negative impact is observable for eroded versions of M, especially for MR/CBCT (8b) (p ≤ 0.01). Using dilated versions of M, DSC differences were not statistically significant (p ≥ 0.3). Here again, differences between DSC obtained before and after needle insertions were not statistically significant (p ≥ 0.46 for CT/CBCT, p ≥ 0.3 for MR/CBCT). Figure 9 compares the registration accuracy obtained using all tested algorithms (see section 2.2.4 for details). Regarding solutions designed to estimate the global 3D shift of the liver (i.e., Elastix, PM-L2 and PM-EA), best results where achieved using the proposed PM-EA approach for both CT/CBCT (9a) and MR/CBCT (9b). PM- EA outperformed significantly a standard registration strategy implemented using the Elastix toolbox (p = 0.02). The use of the proposed multi-modal metric improved significantly the DSC obtained using the L2-norm used in the original PatchMatch paper (Barnes et al. 2009) (p = 0.01). Regarding solutions designed to estimate the elastic deformation of the liver (i.e., Evo and PM-EA+Evo), the use of PM-EA improved significantly the performance of the tested multi-modal elastic registration algorithm Evo (p = 0.01). For all tested solutions, differences between DSC obtained before and after needle insertions were not statistically significant (p ≥ 0.2 for CT/CBCT, p ≥ 0.08 for MR/CBCT). It is interesting to note that the computational demand remained here below 30 seconds for the successive achievement of PM-EA and Evo algorithms with the used hardware for both CT/CBCT and MR/CBCT pairs. Patch-based field-of-view matching in multi-modal images for electroporation-based ablations15 4. Discussion In the current study, we designed a novel method to estimate the global 3D translation between two multi-modal images. We retrospectively evaluate the proposed PM-EA algorithm under a realistic clinical scenarios. We focus on the specific interventional procedure of irreversible electroporation (IRE) ablation for liver tumors. IRE technique provides an interesting alternative to standard ablative techniques, especially for tumor located near vital structures as detailed in (Gallinato et al. 2019). Moreover, it gathers the main computational challenges in terms of medical image registrations, that have to be addressed to improve the procedures. Indeed, the procedures rely upon multimodal medical imaging: preoperative CT-scan or MRI to detect the target ablation region, and preoperative CBCT without and with needles to position the needles and to verify the positioning. Importantly, the needles positioning generate an elastic deformation of the liver, that has be accounted for as previously shown in (Gallinato et al. 2019), and thus nonrigid multimodal algorithm as EVolution (Denis de Senneville et al. 2016) has to be used. As far as we know, the current non rigid registration algorithm needed an initial manual tuning step to superimpose rouglhy the FOVs of two images of different modality. Importantly, the registration has be performed during the procedure, as demonstrated in (Gallinato et al. 2019) in order to provide a numerical assessment of the therapy to the physicians. There is therefore a crucial need to automatize the image preprocessing of FOV alignement in any electroporation ablation procedures. In the current study, the use of various imaging sensors (CT/CBCT, MR/CBCT image pairs, CBCTs being acquired intra-operatively) is analysed as well as the impact of needle insertions during IRE procedures. Using the proposed experimental setup, the registration process is hampered by the use of different image FOVs, especially when using CT during the pre-operative session (see the low DCS obtained in figure 9 before registration when using CT instead of MRI). Moreover, partial FOVs, cross-contrast variations, appearing/disappearing (anatomical or not) structures are also involved between the image to register and the reference one. Using such data sets, optimizing a simple translational model, as implemented in the Elastix toolbox, was found to be insufficient. The proposed regional registration approach using voxel patches provided a good structural compromise between the voxel- wise (as done with Evo) and “global shifts” (as done with Elastix in the scope of this study) approaches. Moreover, contrary to optimization methods which are inherently sensitive to local minima, PM-EA is able to deal with large translation amplitude, since potential patch matches are considered within the complete FOV, as described in section 2.1.3. We have also shown that the proposed multi-modal image similarity metric, which favors edge alignements irrespective of the gradient direction, outperforms the L2-norm proposed in the original PatchMatch paper (Barnes et al. 2009). Ultimately, we have shown that PM-EA may greatly improve the performance of an existing multi-modal elastic registration algorithm (Evo in the scope of this study). As expected, computation times were greatly reduced using down-sampled versions Patch-based field-of-view matching in multi-modal images for electroporation-based ablations16 of input data (see figure 5). This down-sampling step also acts as an inherent low-pass filter applied on I and J which improved the registration accuracy in our tests. Using the proposed default user-defined parameter (i.e., down-sampling factor of 8×), the average DSC with PM-EA exceeded 0.6 for both CT/CBCT, MR/CBCT image pairs together with a computation time cost below 3 seconds on a commodity hardware. The proposed multi-modal metric of Eq. (4) performs a weighted average over patches of the edge alignement score. Consequently, the patch size is an input parameter which can be increased in order to mitigate the fact that the observed image features might not be discriminative enough. Increasing the patch size may thus improve the robustness against anatomical structure without counterpart between the image to register and the reference one. This benefit was achievable with a moderate negative impact on the computation time. However, to some extent, increasing the patch size may be unable to cope with complex local tissue deformations. A good compromise in the choice of the patch size is thus essential for a reliable and accurate patch matching. Using the proposed default user-defined parameter (i.e., patch size of 9 × 9 × 9 voxels), PM-EA attained the best results in terms of DSC for all tested experimental conditions (CT/CBCT, MR/CBCT registration, needle insertions), as shown in figure 6. It can be noticed that the number of histogram bins had no impact on the overall results, as shown in figure 7. The default user-defined parameter (i.e., 50 bins) was thus well suited for all presented results. In our data, the liver undergoes complex deformations between the images being registered. The registration accuracy decreased for eroded versions of M, as shown in figure 8. Liver boundaries, which are needful contrast regions, are not taken into account in such a case. Moreover, the overall shift estimate is likely to differ from the global liver displacement if the manually defined mask M only includes a subset of the liver. Alternatively, no significant impact on the registration accuracy was observed for dilated versions of M in our tests. Therefore, the guideline is that M must include at least the targeted organ. 5. Conclusion The successfull completion of an interventional therapeutic workflow often relies on establishing a spatial coherence between images acquired by various sensors at different stages. The proposed PM-EA algorithm was validated in several complementary experiments. It was demonstrated that it outperforms existing registration solutions for the estimation of a global 3D translation between two multi-modal images. The method can be used as a pre-conditioning step for a more complex multi-modal elastic registration algorithm. The method was thereby beneficial for CT to CBCT and MRI to CBCT registration tasks, especially when highly different image FOVs are involved. In addition, this was achievable together with a computation time cost below 3 seconds on a commodity hardware using our experimental protocol. The proposed patch- based workflow thus represents an attractive asset for DIR at different stages of an Patch-based field-of-view matching in multi-modal images for electroporation-based ablations17 interventional procedure. Acknowledgment Experiments presented in this paper were carried out using the PlaFRIM experimental testbed, supported by Inria, CNRS (LABRI and IMB), Universit´e de Bordeaux, Bordeaux INP and Conseil R´egional d’Aquitaine (see https://www.plafrim.fr/). The authors thank the Laboratory of Excellence TRAIL ANR-10-LABX-57 for funding. This study has been carried out with the financial support of the French National Research Agency (ANR) in the frame of the “Investments for the future” Programme IdEx Bordeaux-CPU (ANR-10-IDEX-03-02). This research has been partly granted by the Plan Cancer project NUMEP (Inserm 11099), led by C.P. References Astola, J., Haavisto, P. & Neuvo, Y. (1990). Vector median filters, Proceedings of the IEEE 78(4): 678– Bao, L., Yang, Q. & Jin, H. (2014). Fast edge-preserving patchmatch for large displacement optical flow, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3534– Barnes, C., Shechtman, E., Finkelstein, A. & Goldman, D. B. (2009). PatchMatch: A randomized correspondence algorithm for structural image editing, ACM Transactions on Graphics (Proc. SIGGRAPH) 28(3). Bleyer, M., Rhemann, C. & Rother, C. (2011). Patchmatch stereo-stereo matching with slanted support windows, Bmvc, Vol. 11, pp. 1–11. Denis de Senneville, B., Ries, M., Maclair, G. & Moonen, C. (2011). MR-guided thermotherapy of abdominal organs using a robust PCA-based motion descriptor, IEEE Transactions on Medical Imaging 30(11): 1987–1995. Denis de Senneville, B., Zachiu, C., Ries, M. & Moonen, C. T. W. (2016). Evolution: an edge- based variational method for non-rigid multi-modal image registration, Physics in Medicine and Biology 61(20): 7377. Gallinato, O., Denis de Senneville, B., Seror, O. & Poignard, C. (2019). Numerical workflow of irreversible electroporation for deep-seated tumor, Physics in Medicine and Biology 64(5): 055016. Giraud, R., Ta, V.-T., Papadakis, N., Manj´on, J. V., Collins, D. L., Coup´e, P. & the Alzheimer’s Disease Neuroimaging Initiative (2016). An optimized PatchMatch for multi-scale and multi- feature label fusion, NeuroImage 124: 770–782. Guckenberger, M., Richter, A., Boda-Heggemann, J. & Lohr, F. (2012). Motion compensation in radiotherapy, Critical Reviews and trade; in Biomedical Engineering 40(3): 187–197. Heinrich, M., Jenkinson, M., Bhushan, M., Matin, T., Gleeson, F., Brady, S. & Schnabel, J. (2012). MIND: Modality independent neighbourhood descriptor for multi-modal deformable registration, Medical Image Analysis 16(7): 1423–1435. Hocquelet, A., Trillaud, H., Frulio, N., Papadopoulos, P., Balageas, P., Salut, C., Meyer, M., Blanc, J. F., Montaudon, M. & Denis de Senneville, B. (2016). Three-dimensional measurement of hepatocellular carcinoma ablation zones and margins for predicting local tumor progression, Journal of Vascular and Interventional Radiology 27(7): 1038 –1045.e2. Holbrook, A. B., Santos, J. M., Kaye, E., Rieke, V. & Butts Pauly, K. (2009). Real-time MR thermometry for monitoring HIFU ablations of the liver, Magnetic Resonance in Medicine 63(2): 365–373. Patch-based field-of-view matching in multi-modal images for electroporation-based ablations18 Horn, B. & Schunck, B. (1981). Determining optical flow, Artificial intelligence 17: 185–203. Irani, M. & Anandan, P. (1998). Robust multi-sensor image alignment, IEEE Computer Vision, Sixth International Conference, pp. 959–966. Jakubowski, M. & Pastuszak, G. (2013). Block-based motion estimation algorithms — a survey, Opto- Electronics Review 21(1). Klein, S., Staring, M., Murphy, K., Viergever, M. & Pluim, J. (2010). elastix: A toolbox for intensity- based medical image registration, IEEE Transactions on Medical Imaging 29(1): 196–205. Lafitte, L., Zachiu, C., Kerkmeijer, L. G. W., Ries, M. & Denis de Senneville, B. (2018). Accelerating multi-modal image registration using a supervoxel-based variational framework, Physics in Medicine and Biology 63(23): 235009. Mougenot, C., Quesson, B., Denis de Senneville, B., de Oliveira, P., Sprinkhuizen, S., Palussiere, J., Grenier, N. & Moonen, C. T. W. (2009). Three-dimensional spatial and temporal temperature control with MR thermometry-guided focused ultrasound (MRgHIFU), Magnetic Resonance in Medicine 61: 603–614. Newson, A., Almansa, A., Fradet, M., Gousseau, Y. & P´erez, P. (2014). Video inpainting of complex scenes, SIAM Journal on Imaging Sciences 7(4): 1993–2019. Rivaz, H., Karimaghaloo, Z. & Collins, D. L. (2014). Self - similarity weighted mutual information: A new nonrigid image similarity metric, Med Image Anal 18(2): 343 – 358. Rubeaux, M., Simon, A., Gnep, K., Colliaux, J., Acosta, O., de Crevoisier, R. & Haigron, P. (2013). Evaluation of non-rigid constrained CT/CBCT registration algorithms for delineation propagation in the context of prostate cancer radiotherapy, Medical Imaging 2013: Image-guided procedures, robotic interventions and modeling, Vol. 8671, SPIE Proceedings. Shamonin, D. (2013). Fast parallel image registration on cpu and gpu for diagnostic classification of alzheimer’s disease, Frontiers in Neuroinformatics 7. Studholme, C., Hill, D. & Hawkes, D. (1999). An overlap invariant entropy measure of 3d medical image alignment, pattern recognition, 32(1): 71–86. Sutour, C., Aujol, J. F., Deledalle, C. A. & Denis de Senneville, B. (2015). Edge-based multi-modal registration and application for night vision devices, Journal of Mathematical Imaging and Vision 53: 131–150. Sutter, O., Fihri, A., Ourabia-Belkacem, R., Sellier, N., Diallo, A. & Seror, O. (2018). Real-time 3D virtual target fluoroscopic display for challenging hepatocellular carcinoma ablations using cone beam CT, Technology in cancer research & treatment 17: 1533033818789634. Zachiu, C., Denis de Senneville, B., Dmitriev, I. D., Moonen, C. T. W. & Ries, M. (2017). A framework for continuous target tracking during MR-guided high intensity focused ultrasound thermal ablations in the abdomen, Journal of Therapeutic Ultrasound 5(1): 27. Zachiu, C., Denis de Senneville, B., Tijssen, R. H. N., Kotte, A. N. T. J., C., H. A., Kerkmeijer, L. G. W., Lagendijk, J. J. W., Moonen, C. T. W. & Ries, M. G. (2017). Non-rigid CT/CBCT to CBCT registration for online external beam radiotherapy guidance, Physics in Medicine and Biology 63(1): 015027. Patch-based field-of-view matching in multi-modal images for electroporation-based ablations19 CBCT MRI / No registration MRI / PM-EA MRI / PM-EA+Evo Trans. [X-Y] (a) (b) (c) (d) Sag. [X-Z] (e) (f) (g) (h) Cor. [Y-Z] (i) (j) (k) (l) (m) (n) (o) Figure 4: Example of a MR/CBCT registration results. The CBCT image, used as a reference for registration, was acquired immediately after insertion of 3 needles. Transversal (a-d), sagittal (e-h) and coronal (i-l) cross-sections are reported for: CBCT (first column), MRI before (second column) and after registration using PM-EA (third column) and PM-EA+Evo (fourth column). Histograms of X-, Y- and Z-shifts are reported in (m), (n) and (o), respectively (maximum occurrence in red dashed line). Patch-based field-of-view matching in multi-modal images for electroporation-based ablations20 CT/CBCT registration MR/CBCT registration No needle Needles No needle Needles Image down-sampling Image down-sampling (a) (b) Figure 5: Analysis of the impact of the down-sampling of input data on the performance of the proposed PM-EA method. DSC (left Y-axis) and computation times (red dashed line/right Y-axis) are reported for the registration of CT/CBCT (a) and MR/CBCT (b) pairs for down-sampling factors 2× (DS-2), 4× (DS-4) and 8× (DS-8). We recall that the patch size was here fixed to 9 × 9 × 9 voxels. The number of histogram bins was fixed to a value of 50. CT/CBCT registration MR/CBCT registration 1 4 1 4 No needle Needles No needle Needles 0.8 0.8 3.5 3.5 0.6 0.6 3 3 0.4 0.4 2.5 2.5 0.2 0.2 0 2 0 2 3x3x3 5x5x5 7x7x7 9x9x9 11x11x11 3x3x3 5x5x5 7x7x7 9x9x9 11x11x11 Patch size Patch size (a) (b) Figure 6: Analysis of the impact of the patch size on the performance of the proposed PM-EA method. DSC (left Y-axis) and computation times (red dashed line/right Y- axis) are reported for the registration of CT/CBCT (a) and MR/CBCT (b) pairs. We recall that the down-sampling factor of input data was here fixed to 4×. The number of histogram bins was fixed to a value of 50. DSC [a.u] DSC [a.u] Computation time [s] Computation time [s] DSC [a.u] DSC [a.u] Computation time [s] Computation time [s] Patch-based field-of-view matching in multi-modal images for electroporation-based ablations21 CT/CBCT registration MR/CBCT registration No needle Needles No needle Needles Histogram bins [#] Histogram bins [#] (a) (b) Figure 7: Analysis of the impact of the number of histogram bins on the performance of the proposed PM-EA method. DSC (left Y-axis) and computation times (red dashed line/right Y-axis) are reported for the registration of CT/CBCT (a) and MR/CBCT (b) pairs. We recall that the down-sampling factor of input data was here fixed to 4×. The patch size was here fixed to 9 × 9 × 9. CT/CBCT registration MR/CBCT registration No needle Needles No needle Needles Masking sensibility [%] Masking sensibility [%] (a) (b) Figure 8: Analysis of the impact of errors occurred in the targeted organ delineation process performed on the pre-operative image I. DSC [a.u] DSC [a.u] DSC [a.u] DSC [a.u] Patch-based field-of-view matching in multi-modal images for electroporation-based ablations22 CT/CBCT registration MR/CBCT registration No needle Needles No needle Needles (a) (b) Figure 9: Summary of DSC scores obtained for the registration of CT/CBCT (a) and MR/CBCT (b) images, using tested solutions detailed in section 2.2.4. Standard deviations over the patients are given by the size of the black error bars. We recall that the down-sampling factor of input data was here fixed to 4×. The patch size was fixed to 9 × 9 × 9 voxels. The number of histogram bins was fixed to a value of 50. DSC [a.u] DSC [a.u]

Journal

Electrical Engineering and Systems Science – arXiv (Cornell University)

Published: Nov 9, 2020

There are no references for this article.