Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Learning-based classification of multispectral images for deterioration mapping of historic structures

Learning-based classification of multispectral images for deterioration mapping of historic... The conservation of historic structures requires detailed knowledge of their state of preservation. Documentation of dete- rioration makes it possible to identify risk factors and interpret weathering mechanisms. It is usually performed using non- destructive methods such as mapping of surface features. The automated mapping of deterioration is a direction not often explored, especially when the investigated architectural surfaces present a multitude of deterioration forms and consist of heterogeneous materials, which significantly complicates the generation of thematic decay maps. This work combines reflec- tance imaging and supervised segmentation, based on machine learning methods, to automatically segment deterioration patterns on multispectral image composites, using a weathered historic fortification as a case study. Several spectral band combinations and image classification techniques (regression, decision tree, and ensemble learning algorithmic implementa- tions) are evaluated to propose an accurate approach. The automated thematic mapping facilitates the spatial and semantic description of the deterioration patterns. Furthermore, the utilization of low-cost photographic equipment and easily operable digital image processing software adds to the practicality and agility of the presented methodology. Keywords Built heritage · Deterioration mapping · Multispectral reflectance imaging · Thermal infrared imaging · Supervised image segmentation · Machine learning 1 Introduction Recording the preservation state of a historic structure is a crucial prerequisite for pathology diagnosis. Document- Architectural heritage possesses outstanding value while ing in detail the condition of the structure's elements is the concomitantly comprises a fundamental manifestation first step towards qualitatively interpreting its condition and of sociocultural identity. The historic build environment identifying mechanisms of deterioration. Therefore defin- is a vital aspect of a place's culture, history, and land- ing the data recording techniques that will provide rich and scape, which necessitates measures to ensure its preser- suitable information about the extent and forms of deteriora- vation through time. However, environmental pressures tion is essential for condition documentation. To the greatest and anthropogenic factors cause constant alterations and extent possible, recording should be non-destructive, mean- impose significant risks. Planning appropriate and compat- ing that it should encompass those nonintrusive inspection ible conservation and restoration interventions to tackle the and sensing techniques that do not cause further damage deterioration of historic structures requires a comprehensive to, nor impair the future usefulness of the structure and the knowledge of the preservation state. Thus, the need for the historic materials. historic structures' recording emerges, which will provide Mapping is widely recognized as an effective non- the detailed information needed to support required preser- destructive method useful for condition documentation and vation interventions. can be applied to all materials at different scales. It registers information about the surface patterns of historic structures that can be later analyzed through computational systems. Mapping is frequently performed as a manual process in * Efstathios Adamopoulos a computer-aided design (CAD) or geographic informa- efstathios.adamopoulos@unito.it tion system (GIS) environment using as background color Department of Computer Science, University of Turin, photos. Turin, Italy Vol.:(0123456789) 1 3 41 Page 2 of 15 Journal of Building Pathology and Rehabilitation (2021) 6:41 Progress in automated mapping for historic structures 2.1 Mapping the preservation state has primarily concentrated on identifying and classifying building elements, materials, and additionally deterioration Mapping is a valuable non-destructive method as it facili- as a binary concept—considering the presence and absence tates the description, registration, and quantification of the, of deterioration solely. The segmentation algorithms that often overlapping, multitude of surface patterns on historic have been considered are mainly based on dimensional- structures. When performed in a digital, computerized man- ity reduction, unsupervised clustering, and deep learning ner, it produces spatial information, entities with geometric approaches, occasionally considering spectral bands at the attributes that can be correlated, compared, used to pro- infrared range. duce statistical information, and allow for the attribution of semantic data about the characteristics of materials and 1.1 Aims and scope their decay. Traditionally, mapping is a technique manually performed inside CAD and GIS platforms by describing the This work delves into the fields of imaging science and pat- shape of surface patterns and organizing them into thematic tern recognition to identify a novel and accurate methodol- layers [1–6]. An alternative way of mapping deterioration ogy for classifying different deterioration forms on historic is the visualization of damage levels/indexes, which can be structures. Reflecting on the potential of multispectral imag- either accomplished directly or indirectly by analyzing the ing and learning-based image analysis for defect detection, mapped deterioration patterns [7–15]. The additional spatial the classification of multispectral composites synthesized annotation of lithotypes facilitates the association between from reflectance images captured at the visible (RGB), near- materials and alteration [1, 11–13, 15–23]. Mapping sup- ultraviolet (NUV), near-infrared (NIR), and thermal infrared ports the interpretation of weathering phenomena when (TIR) spectra, with supervised segmentation methods based combined with data from non-destructive testing (NDT) [8, on random decision trees, ensemble learning, and regression 9, 17–20, 22, 24, 25], laboratory mineralogical, chemical algorithmic implementations, is thoroughly evaluated. and physical characterization [15, 18, 19, 23, 26], and envi- ronmental measurements [8, 12, 18, 19, 25, 27]. 1.2 Article structure 2.2 Generating base‑maps for deterioration This article is structured into six sections. Section 2 delivers mapping an overview of the background for the presented work and discusses the related research. Section 3 describes the meth- Mapping is typically a photo-based approach where a color odology followed, including the instrumentation, data col- image, an orthorectified image, or an orthoimage-mosaic is lection and preparation, algorithmic implementations, and used as a base-map for designing the geometrical shape of approaches followed to evaluate the segmentation results. surface patterns [27]. The metric (accuracy, scale-dependent Section 4 presents the application and results for the case spatial resolution) and chromatic quality of this background study of a historic fortification, while Sect.  5 discusses the are essential for identifying deterioration [25, 28, 29]. Thus, accuracy and interpretation of the results. The concluding acquiring suitable images is crucial for successful deterio- remarks are presented in Sect. 6. ration mapping. However, not only true color images have been considered as base maps, but also images captured at portions of the electromagnetic spectrum beyond the visible. 2 Background and related work 2.3 Multispectral imaging and data Architectural surfaces of historic structures are subjected complementarity to continuous alterations due to exposure to environmental conditions, microorganisms, pollution, anthropogenic dam- The reciprocity of mapping and infrared reflectance imag- ages; their susceptibility to decay also depends on (incom- ing—especially thermography—has often been considered patible) conservation interventions of the past and the inher- essential for detecting weathering on historic structures [5, ent characteristics of historic materials. Particularly when 20, 24, 30, 31]. Besides, thermography is being extensively several different materials are present (such as in masonry used in built heritage structural diagnostics [32–34] and has structures), the architectural surfaces consist of an intricate also been explored to detect different historic materials on mosaic of deterioration forms. Consequently, documentation building façades [35, 36]. The additional inclusion of NIR methods for describing these complicated conditions in a reflectance images enhances the identifiability of deterio- non-destructive way become pertinent and often necessary. ration, mainly when there is a presence of vegetation and biogenic crusts, which present vastly different near-infrared 1 3 Journal of Building Pathology and Rehabilitation (2021) 6:41 Page 3 of 15 41 Fig. 1 Overall research methodology Table 1 Specifications of digital cameras used for acquiring multi- is cut off by an internal blocking filter. Removing this filter spectral reflectance data implies that the camera can be used for imaging at a wider than visible range, and external wavelength-specific filters Camera model Canon EOS Rebel SL1 FLIR ONE Pro can be utilized. Detection in the long-wavelength infrared Spectral range 0.3–1.1 μm 8–14 μm (LWIR) range has usually been performed with uncooled Resolution 5184 × 3456 pixels 160 × 120 pixels microbolometer detectors for building inspections. The Pixel pitch 4.3 μm 12 μm spatial resolution of thermography cameras is considerably FOV – 43° ± 1° lower than that of DSLR, and their relative cost is higher. NETD – 70 mK Recently, more affordable thermography camera models Measurement accuracy – ± 5% have come into the market, including smartphone-adjusta- ble low-resolution instruments. However, these inexpensive Field-of-view cameras provide lower accuracy, which makes them unus- Noise equivalent temperature difference (thermal sensitivity) c able for some applications. Typical percentage of the difference between ambient and scene tem- perature 2.4 Digital image processing The need for more efficient inspection [44] and intelligent reflectance characteristics compared with construction mate- rials [37, 38]. However, the decision to include recorded data identification of conservation needs [45] has led to the adop- tion of image processing approaches to generate the thematic from multiple bands comes with the realization that suitable sensing techniques have to be selected. data needed for deterioration mapping. Digital image pro- cessing (DIP) refers to the manipulation of the digital images Spectral collection in the infrared is connected with various sensing techniques that depend on the wavelength to extract features and recognize patterns, which, after hav- ing acquired the suitable base-maps, can be performed choice. Detection in the wavelength range between 400 and 1100 nm has been performed with multispectral configura- with techniques as simple as thresholding, edge detection, or information reduction to obtain the required results [33, tions that involve multiple single-band cameras recording at 4–12 different narrow spectral bands. The resolution of these 46–48]. However, these approaches still largely depend on the human factor since many parameters have to be tuned instruments is usually low, and the collected imagery has to be meticulously checked to correct sensors' errors [39, 40]. differently for each application, and often deterioration pat- terns have to be identified and extracted one at a time. The The introduction, or rather repurposing, of commercial digi- tal single-lens reflex (DSLR) cameras with charge-coupled current rise of deep learning-based pattern recognition has delivered powerful tools for fully automated detection of device (CCD) and complementary metal–oxide–semicon- ductor (CMOS)-based detectors, for spectral imaging at the deterioration (often through convolutional neural networks), even when a plethora of surface patterns can be observed same range, however, provides more affordable and agile solutions that retain the user-friendly features and the inter- [49–52]. Nevertheless, deep-learning implementations require large image datasets to be efficiently trained, which faces to a wide variety of photographic software and acces- sories, and have high spatial resolution [41–43]. Commercial is often impractical for built heritage applications. They may also underperform considering the uniqueness of each off-the-shelf (COTS) DSLR camera detectors are generally sensitive in a portion of the NIR range up to 1100 nm, which heritage asset, many of which present a distinctive mixture 1 3 41 Page 4 of 15 Journal of Building Pathology and Rehabilitation (2021) 6:41 of historic materials. Therefore, other more easily execut- acquiring appropriate images and then continues with their able supervised learning-based approaches are sometimes radiometric correction. The multispectral composites are considered for deterioration detection through classification digitally synthesized from the band-specific reflectance and regression. images and subsequently segmented into deterioration cat- Multiband and multispectral image segmentation for built egories following a visual identification of training regions. heritage inspection purposes has been applied via a range The results are evaluated with metrics deriving from the of clustering algorithms, some of the most common being field of remote sensing. The output of the deterioration maximum-likelihood, minimum-distance, and k-means [36, classification can be optionally transferred to an environ- 37, 39, 48, 53–56]. However, most of the relevant works ment appropriate for spatial information management. The aim at segmenting the materials and elements of historic principle of using low-cost equipment and software was fol- façades, and when deterioration is considered, it is deter- lowed throughout this work as it is an essential factor for the mined as present or not present. To be specific, many works inspection of historic buildings. consider the altered and unaltered areas of a historic material as two categories rather than identifying the different dete- 3.1 Sensors and data acquisition rioration typologies, which is also partly a result of the state of preservation of the heritage assets involved. Alternative The selection of the instruments employed in this work multi-sensor approaches, involving terrestrial LiDAR for considers the complementarity of data captured at differ - NIR recording, have been reported to produce high-accuracy ent spectral bands and the flexibility requirements of sens- thematic mapping results for damaged historic structures ing techniques used for built heritage condition monitoring. [57, 58]. However, they introduce significant instrumenta- Affordable, portable sensors are utilized to obtain the neces- tion costs, and require rigorous radiometric calibrations and sary multispectral data that will constitute the background optimal data gathering conditions. for the deterioration pattern analysis, contributing to a sim- ple to implement methodology. The characteristics of the instrumentation are presented in Table 1. The images are 3 Methods and materials taken with two sensors, an EOS Rebel SL1 (Canon Inc., Tokyo, Japan) digital single-lens reflex camera with an EF-S The rationale behind this work is set on the identified lack 18-55 mm f/3.5–5.6 IS II lens, and a FLIR ONE Pro (Tel- of image-based methods for automatic mapping of weath- edyne FLIR LLC, Wilsonville, OR, USA) thermographic ered historic structures. The methods tested aim to tackle the camera attached to a smartphone. The internal hot mirror problematics of mapping the preservation state when various filter of the SL1 camera has been removed to allow imag- surface deterioration forms are present. Instead of following ing beyond the visible range. Three low-cost external filters unsupervised segmentation techniques and then interpret- are employed to allow RGB, NUV, and NIR photo shoot- ing each classified category of weathering-caused alteration, ing. The images are acquired as parallel as possible to the supervised algorithmic approaches are implemented using as architectural surfaces to avoid occlusions, and with small input the already identified deterioration categories. Com- focal lengths to avoid large distortions that can affect image binations of different spectral band composite images and quality during the resampling phase of distortion correction. supervised segmentation algorithms are evaluated to distin- Furthermore, the images are acquired under homogeneous guish an optimal solution in terms of accuracy—based on illumination conditions and without shadows, improving reference data. their radiometric potential and with a steady tripod, thus Figure  1 depicts the implemented research design in preventing image blur. Since low-cost sensors are more this work. As already highlighted, the quality of available likely to be affected by noise sources, the camera sensor is imagery upon which the pattern recognition will be per- checked to estimate the vignetting and background noise formed is essential for ensuring the accuracy and inter- levels, and the images are corrected to ensure their quality. pretability of results. Therefore, the workflow starts from Table 2 Multispectral image Multispectral image Red band Green band Blue band composition G-B-NUV Green Blue Near-ultraviolet R-G-B Red Green Blue NIR-R-G Near-infrared Red Green TIR-NIR-R Thermal infrared Near-infrared Red NIR-M-NUV Near-infrared RGB monochromatic Near-ultraviolet 1 3 Journal of Building Pathology and Rehabilitation (2021) 6:41 Page 5 of 15 41 The thermographic data are acquired at sets of burst images 3.3 Machine learning‑based segmentation to increase digitally later their spatial resolution. of deterioration patterns 3.2 Multispectral data preparation The classification of deterioration patterns is performed via a supervised segmentation procedure using the Train- Pre-processing the imagery data involves the preparation able WeKa Segmentation 3D plugin [63] of ImageJ2. The of multispectral image composites for the subsequent seg- machine learning-based image segmentation techniques fol- mentation. At first, the radiance images acquired with the low decision tree [64], ensemble learning [65], and regres- SL1 camera are downloaded in the RawDigger (LibRaw sion approaches. Specifically, the Random Tree, Random LLC, Maryland, USA) software, where the color filter array Forest, Fast Random Forest, and LogitBoost classifiers are conversion is reversed to acquire raw radiance images, and employed. The supervised approach presupposes the annota- RGB images are color balanced. Non-visible spectrum tion of image regions of interest (ROIs), corresponding to images should also be converted to reflectance images based each semantic deterioration category to be segmented, that on pixel values of a reference surface. The uncompressed will train the algorithmic model into providing a semantic images are then corrected from distortion [59] in ImageJ2 classification of the entire image. [60]. Thermal infrared burst mode images acquired with the The decision tree model is a machine learning algorithm FLIR ONE Pro camera are used to create high-resolution that can be used for both supervised classification and thermal images [61]. regression problems. A decision tree simply consists of a The manual matching of band-specific images is done series of sequential decisions made to reach a specific result using the HyperCube software [62] (projective transfor- of distinct data classes. The classes are mutually exclusive mation, nearest-neighbor interpolation). Subsequently, the and represented by specific attributes. The learning input, image composites are constructed using different multispec- which consists of sets of pixels belonging to known classes, tral combinations, as described in Table 2. The images are assists the accurate classification of both annotated pixels resampled to match the resolution of all bands, and the sky and not annotated pixels. Each node of the decision tree and ground are trimmed from all multispectral composites decides an outcome based on the attribute values and leads to reduce potential misclassifications. The synthesis of the either to another node, using an appropriate subtree, or to a multiband composites also considers the same principle of leaf, which gives the predicted class of the pixel [66]. The using low-cost equipment, and thus all composites consist Random Tree classifier is based on a decision tree learning of three bands so that segmentation can be performed in method. Single decision trees are easy to conceptualize but ImageJ2—avoiding the use of commercial specialized spa- usually suffer from high variance, making them not competi- tial analysis software. tive in terms of accuracy. Fig. 2 Fort of Karababa, bird’s- eye view 1 3 41 Page 6 of 15 Journal of Building Pathology and Rehabilitation (2021) 6:41 LogitBoost is a boosting algorithm that performs classifi- cation using a regression scheme as the base learner and can handle multi-class problems. It can be seen as a convex opti- mization; it applies the cost function of logistic regression on a generalized additive model. This classifier determines the appropriate number of iterations by performing efficient internal cross-validation [68]. 3.4 Accuracy metrics The performance of the machine learning classification implementations, and of the different multispectral combina- tions, are quantitatively evaluated using manually produced degradation maps as the ground truth. Different parameters are used to assess the classification efficiency of the intel - ligent feature extraction techniques based on accuracy met- rics common for thematic mapping. More specifically, the evaluation relies on the precision (fraction of appropriate classification among the classified instances) and F1-score (harmonic mean of precision and sensitivity) calculated for each class (Eqs. 1,2), and on the overall accuracy (Eq. 3)— useful to estimate the overall performance of the classifiers. TP Precision = (1) TP + FP 2TP F1 Score = (2) TP + FP + TP + FN Sum of correctly classified units Overall accuracy = (3) Total number units where, for each class the TP (true positive), FP (false posi- tive), and FN (false negative) come from the error matrix, a square array of numbers, which express the number of pixels assigned to a particular class in one classification relative to the number of pixels assigned to a particular class in the reference data [69, 70]. Fig. 3 Fort of Karababa north side, façades selected for evaluating 3.5 Case study the mapping methodology; from top to bottom: A (westernmost), B, and C (easternmost) The historic structure selected as a case study for the appli- cation and assessment of the proposed methodology is a for- A random forest classifier combines ensemble classifica- tification in Euboea, Greece (Fig.  2). The Fort of Karababa is tion machine learning algorithms and decision trees. Each an Ottoman fortification constructed in 1684 on the homony - tree classifier is independently generated from the input mous hill which dominates the Boeotian coast across the city training data using a random sample like in bagging. When of Chalcis. The construction of the fortress was part of the growing a tree, the best possible split is computed for a ran- effort to protect the city of Chalcis from impending Venetian dom subset, instead of always computing the best split for attacks. The architectural style of the fort is more European each node. In this way, tree diversity is generated using two than Turkish. It is oblong in plan, with a rampart on the ways of randomization. Aggregating predictions make the north side, three bastions, and a large tower. Several parts class prediction of the ensemble. Random forest generally of the fortification walls have ancient spolia built-in, while overcomes the accuracy limitations of single decision trees the south part is preserved in poor condition. The weathered [65, 67]. 1 3 Journal of Building Pathology and Rehabilitation (2021) 6:41 Page 7 of 15 41 Fig. 4 Multispectral data preparation for façade C. Note: NUV near-ultraviolet; R red; B blue; G green; NIR near infrared; TIR thermal infrared; M monochromatic color image masonry surfaces selected for evaluating the methodology 4 Results are presented in Fig. 3. They are on the north side, and for abbreviation purposes, they have been named A, B, and C, Following the described methodologies, after the compo- starting with the westernmost (on the north bastion). sition of multispectral images was completed (Fig. 4), 60 classifications were performed. The generation of reference 1 3 41 Page 8 of 15 Journal of Building Pathology and Rehabilitation (2021) 6:41 Table 3 Overall accuracy statistics by image and classifier especially for classifications performed with ensemble learning-based algorithmic implementations. Furthermore, A B C the inclusion of different spectral bands improved the clas- Overall accuracy (%) sification potential, subject to the categories of deterioration G-B-NUV present.  LogitBoost 82.1 66.9 58.4  Random tree 71.0 66.3 50.9  Random forest 84.9 69.9 57.3 5 Discussion  Fast random forest 84.6 70.2 58.4 R-G-B The inclusion of the NIR spectral band fairly improved the  LogitBoost 77.3 67.9 78.5 classification results for all deterioration forms. The seg -  Random tree 73.7 63.7 74.3 mentation of a NIR-R-G multispectral image and the Fast  Random forest 84.4 69.8 81.9 Random Forest classifier proved to be the most consistent  Fast random forest 84.4 69.9 83.1 solution overall (79 ≤ overall accuracy%). Figure 5 presents NIR-R-G a comparison between the reference maps and the NIR-  LogitBoost 80.4 71.3 80.1 R-G composites segmented with the Fast Random Forest.  Random tree 72.6 71.0 77.4 Using NUV reflectance data generally did not provide any  Random forest 85.4 75.8 83.8 improvement to the quality of the classifications. Including  Fast random forest 86.3 79.0 84.6 the TIR band also did not improve the deterioration patterns' TIR-NIR-R classification. Furthermore, the fusion of visible with ther -  LogitBoost 74.6 76.3 58.4 mal data significantly decreased the accuracy of detecting  Random tree 64.9 71.3 50.9 deterioration when dampness was present, which contradicts  Random forest 76.8 76.3 57.3 that thermal images are helpful in detecting moisture on his-  Fast random forest 78.5 77.7 58.4 toric masonry, as evident by Fig. 6. NIR-M-NUV According to the overall accuracy results, the Fast Ran-  LogitBoost 83.2 71.6 75.3 dom Forest classifier was the most accurate learning-based  Random tree 75.4 66.8 66.6 method for deterioration classification for all multispec-  Random forest 85.5 69.3 77.0 tral images, not including the TIR band (70% < overall  Fast random forest 86.8 69.8 78.2 accuracy < 87%). Implementing the random tree classifier resulted in more inconsistent and less accurate classifications (60% < overall accuracy < 77%). LogitBoost outperformed the Random Tree classifier. According to the precision and F1-score values, moss and maps considered the Illustrated Glossary on Stone Dete- lichens were the most misclassified surface patterns, even rioration Patterns [71] as a guide during visual inspection. though both random forest approaches improved their clas- The observed categories of deterioration were vegetation, sification. The results prove that the distinction among non- moss, black crusts, lichens, missing material (including loss deteriorated material, dampness, black crusts/discoloration, of components, large cracks, and windows), and dampness. and plants is much more easily detectable (and therefore These constituted all the categories of surface pathology that classifiable) than biogenic colonization of any form. There- altered the surface reflectance characteristics of the masonry fore, surface alterations of the historic materials—which façades. Patterns that caused slight geometrical surface alter the reflectance characteristics— can be more accurately alterations, such as minor cracks, superficial cracking due mapped using multispectral images in comparison with the to biogenic deterioration, disintegration, or other shape fea- deterioration forms that completely cover them as an addi- tures induced by material loss, insignificant concerning the tional layer. considered scale and the reflectance contract comparing with healthy historic materials could not be considered. The the- matic comparisons were performed using the full reference 6 Conclusions maps and not sampled patch areas. Overall accuracy statis- tics calculated from the confusion matrixes are presented in In this work, a novel methodology for the automatic clas- Table 3. The precision and F1-score results are presented in sification of damage on built cultural heritage was proposed detail in “Appendix A”. that uses low-cost photographic equipment for multispec- The deterioration maps produced for all the studied archi- tral data acquisition and supervised machine learning-based tectural surfaces were of generally high thematic accuracy, image segmentation to map deterioration patterns. It was 1 3 Journal of Building Pathology and Rehabilitation (2021) 6:41 Page 9 of 15 41 Fig. 5 Reference deterioration maps (left), and corresponding deterioration maps produced with a NIR-R-G multispectral image using the Fast Random Forest Classifier (right); façades A, B, and C (from top to bot- tom) Fig. 6 Thermograms of façades A (left), and B (right) confirmed that including near-infrared reflectance intensi- The segmentation of multispectral composites (synthe- ties in the employed methods improved the classification of sized with visible and near-infrared reflectance images), alterations on the historic masonry façades. with classifiers combining random trees and ensemble learn- ing, performed particularly well even were a high number 1 3 41 Page 10 of 15 Journal of Building Pathology and Rehabilitation (2021) 6:41 of surface patterns was present. However, the coexistence category. Furthermore, there is a clear advantage over deep of different overlapping categories of biogenic colonization learning-based methods, that require large image datasets, complicated the mapping procedure significantly. It should for rapid monitoring purposes of monumental heritage struc- be highlighted that the accuracy evaluation considered some tures. A direct outlook of the proposed framework is the level of bias since the manually produced reference thematic combination with 3D recording technologies to enhance the maps cannot consider the overlapping surface patterns. capability of detecting and mapping the geometric altering The proposed methodology has the limitation that it can (material loss) of historic monuments. map only the pathologies that have been previously recog- nized through visual inspection (or analytical techniques) because regions of interest have to be annotated to train the Appendix A intelligent algorithms. However, a crucial advantage is that it produces easily interpretable mapping results, in contra- See Tables 4, 5, 6. diction to unsupervised methods where each mapped pat- tern class has to be a posteriori assigned to a deterioration Table 4 Accuracy statistics calculated for façade A Leafy vegetation No deterioration Black crusts Missing material Dampness G-B-NUV  LB 44.4 60.3 87.5 82.1 56.7 60.4 39.0 53.4 91.7 90.4  RT 28.7 42.6 80.1 77.0 46.1 48.3 6.8 12.3 86.0 79.1  RF 46.3 61.4 86.3 87.0 75.5 64.1 37.8 53.2 89.8 91.7  FRF 41.8 55.8 85.3 87.7 81.8 60.2 45.4 60.5 88.5 91.0 R-G-B  LB 27.7 42.6 85.8 80.4 49.9 51.2 26.0 39.5 87.6 88.5  RT 27.9 42.0 77.7 78.6 57.0 53.0 13.8 23.5 85.4 81.3  RF 39.0 54.6 87.4 87.5 74.7 64.3 40.8 56.0 89.2 90.3  FRF 40.5 55.0 85.5 88.0 83.0 58.6 45.7 60.7 87.8 90.0 NIR-R-G  LB 41.4 57.5 87.0 81.2 51.4 56.6 31.4 45.0 91.4 92.1  RT 26.6 40.1 79.0 77.4 50.8 45.7 11.6 20.2 83.4 81.5  RF 48.6 63.9 88.0 87.3 76.7 67.9 28.6 43.2 89.8 91.6  FRF 54.3 68.0 86.4 88.3 84.6 64.8 43.1 58.0 89.6 92.2 TIR-NIR-R  LB 38.9 55.0 77.5 71.4 45.4 52.3 34.5 48.4 90.5 89.6  RT 16.2 27.3 77.2 59.3 39.5 51.2 10.0 17.7 91.8 85.7  RF 31.1 46.5 80.4 75.1 51.0 58.4 39.9 52.9 92.5 90.0  FRF 32.2 47.6 79.8 77.6 56.0 61.0 43.8 55.4 93.4 89.6 NIR-M-NUV  LB 40.5 56.5 88.9 83.9 59.3 63.8 35.1 48.4 92.6 92.5  RT 17.6 29.4 82.9 78.6 59.8 59.1 15.9 26.8 90.0 86.5  RF 38.9 55.0 87.6 87.5 76.3 70.2 26.6 41.0 92.9 92.2  FRF 44.5 59.3 87.3 88.6 84.5 69.4 43.0 58.1 90.9 92.4 Precision F1-score Precision F1-score Precision F1-score Precision F1-score Precision F1-score LB LogitBoost; RT random tree; RF random forest; FRF fast random forest 1 3 Journal of Building Pathology and Rehabilitation (2021) 6:41 Page 11 of 15 41 1 3 Table 5 Accuracy statistics calculated for façade B Leafy vegetation No deterioration Black crusts Lichens G-B-NUV  LB 52.2 62.0 66.4 75.7 92.8 69.0 30.0 41.9  RT 47.7 60.2 66.2 70.8 90.3 72.0 30.5 43.0  RF 47.4 61.6 66.8 75.0 93.5 73.6 38.2 50.8  FRF 47.2 61.0 62.5 72.5 93.4 73.9 46.3 56.0 R-G-B  LB 42.1 56.4 64.2 72.5 91.4 72.4 36.1 46.3  RT 39.6 54.8 64.2 69.5 89.6 68.5 29.9 42.1  RF 47.7 62.0 64.3 73.5 92.9 73.5 41.2 52.1  FRF 50.4 63.8 59.1 70.2 92.8 73.2 51.0 57.1 NIR-R-G  LB 55.3 65.1 65.3 72.7 92.1 77.2 40.2 51.5  RT 47.9 60.8 70.5 74.2 88.3 77.0 36.4 47.7  RF 43.3 58.0 75.4 78.7 92.4 82.1 43.4 54.6  FRF 43.5 57.9 74.7 78.3 89.9 85.9 63.3 62.0 TIR-NIR-R  LB 37.1 45.5 80.3 82.5 92.6 83.5 36.2 47.3  RT 32.8 43.8 83.3 84.7 91.0 75.8 28.8 41.0  RF 37.6 49.8 90.0 84.9 93.5 83.8 33.1 46.9  FRF 37.2 47.2 89.2 86.6 92.8 84.1 35.8 49.3 NIR-M-NUV  LB 49.3 55.6 68.2 74.5 92.7 77.6 40.5 51.7  RT 36.4 48.3 66.7 70.2 89.8 74.7 29.2 39.4  RF 39.6 54.1 65.5 72.4 93.9 74.9 39.3 51.8  FRF 42.0 55.8 59.3 69.2 93.7 75.5 53.0 58.9 Precision F1-score Precision F1-score Precision F1-score Precision F1-score LB LogitBoost; RT random tree; RF random forest; FRF fast random forest 41 Page 12 of 15 Journal of Building Pathology and Rehabilitation (2021) 6:41 Table 6 Accuracy statistics calculated for façade C Leafy vegetation No deterioration Black crusts Missing material Plant Dampness G-B-NUV  LB 21.6 35.4 91.8 75.2 26.8 37.1 30.7 46.1 72.4 38.3 87.3 88.4  RT 24.1 38.7 92.3 74.5 23.9 36.0 8.1 14.7 67.7 38.4 88.4 84.7  RF 47.3 63.7 89.9 78.1 32.1 46.7 25.2 39.7 63.4 38.2 92.1 91.1  FRF 47.3 63.1 88.5 78.9 37.2 51.8 29.4 45.1 57.3 37.4 91.9 91.3 R-G-B  LB 29.5 45.3 78.8 77.8 29.3 41.7 29.9 45.2 67.4 37.8 90.6 83.9  RT 11.4 19.9 76.7 73.1 19.8 29.9 22.2 35.5 63.8 38.2 84.1 74.1  RF 52.6 68.2 79.6 76.3 34.7 50.2 27.7 42.6 57.1 37.3 90.0 85.1  FRF 61.4 74.6 78.3 76.0 44.6 58.2 27.0 42.0 53.4 36.5 88.4 86.0 NIR-R-G  LB 21.1 34.9 90.1 78.6 32.1 43.3 31.7 47.0 68.5 38.0 89.2 88.7  RT 26.6 40.8 88.7 78.3 23.0 33.8 25.9 40.1 70.1 37.4 89.4 86.6  RF 64.4 78.4 89.2 79.3 38.9 54.4 28.5 43.8 53.8 36.7 91.9 91.5  FRF 67.7 80.7 88.1 80.2 44.0 58.7 31.1 46.9 54.7 36.8 91.5 91.6 TIR-NIR-R  LB 18.6 31.3 54.9 61.1 25.6 38.8 25.2 39.3 49.9 34.6 83.3 62.9  RT 8.5 15.5 53.8 56.2 18.2 28.7 24.6 37.8 28.4 28.6 80.2 57.1  RF 21.4 35.0 50.7 57.9 31.3 45.7 31.4 46.8 38.5 32.7 80.9 61.1  FRF 23.8 37.9 48.8 56.2 40.7 54.2 33.4 49.3 37.4 32.3 78.7 62.6 NIR-M-NUV  LB 24.0 38.5 80.7 72.6 23.5 32.0 26.9 41.2 68.8 38.0 86.5 85.1  RT 13.1 22.7 78.9 70.6 20.2 30.5 13.2 22.8 56.6 35.6 82.2 75.2  RF 36.0 52.4 80.1 73.8 36.7 52.1 25.1 39.7 59.4 37.7 87.4 84.5  FRF 43.9 59.3 78.9 73.8 42.4 56.4 25.7 40.5 54.1 36.7 86.7 85.1 Precision F1-score Precision F1-score Precision F1-score Precision F1-score Precision F1-score Precision F1-score LB LogitBoost; RT random tree; RF random forest; FRF fast random forest Acknowledgements The author acknowledges the Hellenic Ministry Declarations of Culture and Sports/Archaeological Resources Fund. All copyrights to the depicted monuments belong to the Hellenic Ministry of Cul- Conflict of interest The author declares no conflict of interest. The ture and Sports (law 3028/2002). The fort of Karababa in Chalcis falls funders had no role in the study's design, in the collection, analyses, or within the jurisdiction of the Ephorate of Antiquities of Euboea. The interpretation of data, in the writing of the manuscript, or in the deci- author extends his gratitude to the Ephorate of Antiquities of Euboea sion to publish the results. for granting permission to capture, reproduce and disseminate for research purposes images of archaeological content regarding the Open Access This article is licensed under a Creative Commons Attri- depicted monument. bution 4.0 International License, which permits use, sharing, adapta- tion, distribution and reproduction in any medium or format, as long Funding Open access funding provided by Università degli Studi di as you give appropriate credit to the original author(s) and the source, Torino within the CRUI-CARE Agreement. This study was funded by provide a link to the Creative Commons licence, and indicate if changes the European Union’s Framework Program for Research and Innova- were made. The images or other third party material in this article are tion Horizon 2020 (H2020) under the Marie Skłodowska-Curie Actions included in the article's Creative Commons licence, unless indicated (MSCA) COFUND scheme (grant number 754511), and by the founda- otherwise in a credit line to the material. If material is not included in tion Compagnia di San Paolo. the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will Availability of data and material Restrictions apply to the availability need to obtain permission directly from the copyright holder. To view a of the images used for this project. Images were acquired with the copy of this licence, visit http://cr eativ ecommons. or g/licen ses/ b y/4.0/ . permission of the Ephorate of Antiquities of Euboea and are available from the author only if proper authorization can be obtained from the Ephorate of Antiquities of Euboea. 1 3 Journal of Building Pathology and Rehabilitation (2021) 6:41 Page 13 of 15 41 16. McCabe S, Smith BJ, Warke PA (2007) An holistic approach References to the assessment of stone decay: Bonamargy Friary, Northern Ireland. Geol Soc Spec Publ 271:77–86. https:// doi. org/ 10. 1144/ 1. Fitzner B, Heinrichs K (2001) Damage diagnosis on stone monu- GSL. SP. 2007. 271. 01. 09 ments–weathering forms, damage categories and damage indices. 17. Delegou ET, Tsilimantou E, Oikonomopoulou E, Sayas J, Ioan- Acta Univ Carol Geol 45(1):12–13 nidis C, Moropoulou A (2013) Mapping of building materials and 2. Inkpen R, Duane B, Burdett J, Yates T (2008) Assessing stone conservation interventions using GIS: the case of Sarantapicho degradation using an integrated database and geographical infor- Acropolis and Erimokastro Acropolis in Rhodes. Int J Herit Digit mation system (GIS). Environ Geol 56:789–801. https:// doi. org/ Era 2(4):631–653. https:// doi. org/ 10. 1260/ 2047- 4970.2. 4. 631 10. 1007/ s00254- 008- 1309-x 18. Adamopoulos E, Tsilimantou E, Keramidas V, Apostolopou- 3. Brunetaud X, Luca LD, Janvier-Badosa S, Beck K, Al-Mukhtar lou M, Karoglou M, Tapinaki S, Ioannidis C, Georgopoulos A, M (2012) Application of digital techniques in monument preser- Moropoulou A (2017) Multi-sensor documentation of metric and vation. Eur J Environ Civ Eng 16(5):543–556. https://doi. or g/10. qualitative information of historic stone structures. ISPRS Ann 1080/ 19648 189. 2012. 676365 Photogramm Remote Sens Spatial Inf Sci. https://doi .or g/10. 519 4/ 4. Janvier-Badosa S, Beck K, Brunetaud X, Al-Mukhtar M (2013) isprs- annals- IV-2- W2-1- 2017 Historical study of Chambord Castle: basis for establishing the 19. Martínez-Martínez J, Benavente D, Jiménez Gutiérrez S, García- monument health record. Int J Archit Herit 7(3):247–260. https:// del-Cura MA, Ordóñez S (2017) Stone weathering under Mediter- doi. org/ 10. 1080/ 15583 058. 2011. 634959 ranean semiarid climate in the fortress of Nueva Tabarca island 5. Mileto C, Vegas F, Lerma JL (2015) Multidisciplinary studies, (Spain). Build Environ 121:262–276. https:// doi. org/ 10. 1016/j. crossreading and transversal use of thermography: the Castle of build env. 2017. 05. 034 Monzón (Huesca) as a case study. In: Rodríguez-Navarro P (ed) 20. İnce İ, Bozdağ A, Tosunlar MB, Hatır ME, Korkanç M (2018) Defensive architecture of the mediterranean. Editorial Universitat Determination of deterioration of the main facade of the Ferit Paşa Politècnica de València, Valencia, pp 405–412 Cistern by non-destructive techniques (Konya, Turkey). Environ 6. Janvier-Badosa S, Brunetaud X, Beck K, Al-Mukhtar M (2016) Earth Sci 77(11):420. https://doi. or g/10. 1007/ s12665- 018- 7595-z Kinetics of stone degradation of the castle of Chambord in France. 21. Gulotta D, Toniolo L (2019) Conservation of the built heritage: Int J Archit Herit 10(1):96–105. https:// doi. org/ 10. 1080/ 15583 pilot site approach to design a sustainable process. Herit 2(1):797– 058. 2014. 971194 812. https:// doi. org/ 10. 3390/ herit age20 10052 7. Fitzner B, Heinrichs K, Bouchardiere DL (2003) Weathering dam- 22. Hatır ME, Korkanç M, Başar ME (2019) Evaluating the deteriora- age on Pharaonic sandstone monuments in Luxor-Egypt. Build tion effects of building stones using NDT: the Küçükköy Church, Environ 38(9–10):1089–1103. https:// doi. org/ 10. 1016/ S0360- Cappadocia Region, central Turkey. Bull Eng Geol Environ 1323(03) 00086-6 78(5):3465–3478. https:// doi. org/ 10. 1007/ s10064- 018- 1339-x 8. Heinrichs K (2008) Diagnosis of weathering damage on rock- 23. Iandelli N, Coli M, Donigaglia T, Ciuffreda AL (2021) An uncon- cut monuments in Petra. Jordan Environ Geol 56(3–4):643–675. ventional e fi ld mapping application: a complete opensource work - https:// doi. org/ 10. 1007/ s00254-008- 1358-1 flow solution applied to lithological mapping of the coatings of 9. Heinrichs K, Fitzner B (2011) Assessment of weathering damage cultural heritage. IJGI 10(6):357. https:// doi. org/ 10. 3390/ ijgi1 on the Petroglyphs of Cheonjeon-ri, Ulsan, Republic of Korea. 00603 57 Environ Earth Sci 63(7–8):1741–1761. https:// doi. org/ 10. 1007/ 24. Kilic G (2015) Using advanced NDT for historic buildings: s12665- 010- 0828-4 Towards an integrated multidisciplinary health assessment strat- 10. Delgado Rodrigues J (2015) Defining, mapping and assessing egy. J Cult Herit 16(4):526–535. https://doi. or g/10. 1016/j. culher . deterioration patterns in stone conservation projects. J Cult Herit 2014. 09. 010 16:267–275. https:// doi. org/ 10. 1016/j. culher. 2014. 06. 007 25. Silveira da Costa V, Montagna da Silveira A, da Silva TA (2021) 11. Franković M, Novaković N, Matović V (2015) Damage quan- Evaluation of degradation state of historic building facades tification of built stone on Dark Gate (Belgrade, Serbia): sam- through qualitative and quantitative indicators: case study in Pelo- ple of damage index application for decay rate evaluation. tas Brazil. Int J Archit Herit. https://doi. org/ 10. 1080/ 15583 058. Environ Earth Sci 73(10):6181–6193. https:// doi. org/ 10. 1007/ 2021. 19011 61 s12665- 014- 3843-z 26. Lezzerini M, Antonelli F, Columbu S, Gadducci R, Marradi A, 12. Gizzi FT, Sileo M, Biscione M, Danese M, Alvarez de Buergo M Miriello D, Parodi L, Secchiari L, Lazzeri A (2016) Cultural her- (2016) The conservation state of the Sassi of Matera site (South- itage documentation and conservation: three-dimensional (3D) ern Italy) and its correlation with the environmental conditions laser scanning and geographical information system (GIS) tech- analysed through spatial analysis techniques. J Cult Herit 17:61– niques for thematic mapping of facade stonework of St. Nicholas 74. https:// doi. org/ 10. 1016/j. culher. 2015. 05. 002 Church (Pisa, Italy). Int J Archit Herit 10(1):9–19. https://doi. or g/ 13. Germinario C, Gorrasi M, Izzo F, Langella A, Limongiello M, 10. 1080/ 15583 058. 2014. 924605 Mercurio M, Musmeci D, Santoriello A, Grifa C (2020) Damage 27. Gutiérrez-Carrillo ML, Bestué Cardiel I, Molero Melgarejo diagnosis of Ponte Rotto, a Roman bridge along the ancient Appia. E, Marcos Cobaleda M (2020) Pathologic and risk analysis of Int J Conserv Sci 11(1):277–290 the Lojuela Castle (Granada-Spain): methodology and preven- 14. Randazzo L, Collina M, Ricca M, Barbieri L, Bruno F, Arcudi tive conservation for medieval earthen fortifications. Appl Sci A, La Russa MF (2020) Damage indices and photogrammetry for 10(18):6491. https:// doi. org/ 10. 3390/ app10 186491 decay assessment of stone-built cultural heritage: the case study of 28. Thornbush MJ, Viles HA (2007) Photo-based decay mapping of the San Domenico church main entrance Portal (South Calabria, replaced stone blocks on the boundary wall of Worcester college. Italy). Sustainability 12(12):5198. https:// doi. org/ 10. 3390/ su121 Oxford Geol Soc Spec Publ 271:69–75. https:// doi. org/ 10. 1144/ GSL. SP. 2007. 271. 01. 08 15. Puy-Alquiza MJ, Ordaz Zubia VY, Aviles RM, MaDC S-H (2021) 29. Russo M, Carnevali L, Russo V, Savastano D, Taddia Y (2019) Damage detection historical building using mapping method in Modeling and deterioration mapping of façades in historical urban music school of the University of Guanajuato. Mexico Mech Adv context by close-range ultra-lightweight UAVs photogrammetry. Mater Struct 28(10):1049–1060. https:// doi. org/ 10. 1080/ 15376 Int J Archit Herit 13(4):549–568. https://doi. org/ 10. 1080/ 15583 494. 2019. 16290 49 058. 2018. 14400 30 1 3 41 Page 14 of 15 Journal of Building Pathology and Rehabilitation (2021) 6:41 30. Jo YH, Lee CH (2014) Quantitative modeling and mapping of a tool for mapping and evaluation of stone decay. Constr Build blistering zone of the Magoksa Temple stone pagoda (13th cen- Mater 25(4):1603–1611. https:// doi. org/ 10. 1016/j. conbu ildmat. tury, Republic of Korea) by graduated heating thermography. 2010. 10.003 Infrared Phys Technol 65:43–50. https:// doi. org/ 10. 1016/j. infra 48. Adamopoulos E, Rinaudo F (2021) Documenting the state of pres- red. 2014. 02. 011 ervation of historical stone sculptures in three dimensions with 31. Napolitano R, Hess M, Glisic B (2020) Quantifying the differ - digital tools. In: Del Bimbo A, Cucchiara R, Sclaroff S, Farinella ences in documentation and modeling levels for building pathol- GM, Mei T, Bertini M, Escalante HJ, Vezzani R (eds) Pattern rec- ogy and diagnostics. Arch Computat Methods Eng 27(4):1135– ognition. ICPR international workshops and challenges. Springer 1152. https:// doi. org/ 10. 1007/ s11831-019- 09350-y International Publishing, Cham, pp 666–673 32. Maierhofer C, Röllig M, Krankenhagen R (2010) Integration 49. Hatir ME, Barstuğan M, İnce İ (2020) Deep learning-based weath- of active thermography into the assessment of cultural heritage ering type recognition in historical stone monuments. J Cult Herit buildings. J Mod Opt 57(18):1790–1802. https://doi. or g/10. 1080/ 45:193–203. https:// doi. org/ 10. 1016/j. culher. 2020. 04. 008 09500 34100 37034 97 50. Mansuri LE, Patel DA (2021) Artificial intelligence-based auto- 33. Moropoulou A, Labropoulos KC, Delegou ET, Karoglou M, matic visual inspection system for built heritage. SASBE. https:// Bakolas A (2013) Non-destructive techniques as a tool for the doi. org/ 10. 1108/ SASBE- 09- 2020- 0139 protection of built cultural heritage. Constr Build Mater 48:1222– 51. Hatır ME, İnce İ, Korkanç M (2021) Intelligent detection of dete- 1239. https:// doi. org/ 10. 1016/j. conbu ildmat. 2013. 03. 044 rioration in cultural stone heritage. J Build Eng 44:102690. https:// 34. Brooke C (2018) Thermal Imaging for the archaeological investi- doi. org/ 10. 1016/j. jobe. 2021. 102690 gation of historic buildings. Remote Sens 10(9):1401. https:// doi. 52. Pathak R, Saini A, Wadhwa A, Sharma H, Sangwan D (2021) An org/ 10. 3390/ rs100 91401 object detection approach for detecting damages in heritage sites 35. Lerma JL (2001) Multiband versus multispectral supervised clas- using 3-D point clouds and 2-D visual data. J Cult Herit 48:74–82. sification of architectural images. Photogramm Rec 17(97):89– https:// doi. org/ 10. 1016/j. culher. 2021. 01. 002 101. https:// doi. org/ 10. 1111/ 0031- 868X. 00169 53. Meroño JE, Perea AJ, Aguilera MJ, Laguna AM (2015) Recogni- 36. Lerma JL (2005) Automatic plotting of architectural facades with tion of materials and damage on historical buildings using digital multispectral images. J Surv Eng 131(3):73–77. https:// doi. org/ image classification. S Afr J Sci 111(1/2):1– 9. https://doi. or g/10. 10. 1061/ (ASCE) 0733- 9453(2005) 131: 3(73) 17159/ sajs. 2015/ 20140 001 37. Lerma JL, Cabrelles M, Akasheh TS, Haddad NA (2012) Docu- 54. Sánchez-Aparicio LJ, Del Pozo S, Ramos LF, Arce A, Fernandes mentation of weathered architectural heritage with visible, near FM (2018) Heritage site preservation with combined radiometric infrared, thermal and laser scanning data. Int J Herit Digit Era and geometric analysis of TLS data. Autom Constr 85:24–39. 1(2):251–275. https:// doi. org/ 10. 1260/ 2047- 4970.1. 2. 251 https:// doi. org/ 10. 1016/j. autcon. 2017. 09. 023 38. Armesto-González J, Riveiro-Rodríguez B, González-Aguilera D, 55. Grilli E, Remondino F (2019) Classic fi ation of 3D digital heritage. Rivas-Brea MT (2010) Terrestrial laser scanning intensity data Remote Sens 11(7):847. https:// doi. org/ 10. 3390/ rs110 70847 applied to damage detection for historical buildings. J Archaeol 56. Adamopoulos E, Rinaudo F (2021) Combining multiband imag- Sci 37(12):3037–3047. https:// doi. org/10. 1016/j. jas. 2010. 06. 031 ing, photogrammetric techniques, and FOSS GIS for affordable 39. Del Pozo S, Herrero-Pascual J, Felipe-García B, Hernández-López degradation mapping of stone monuments. Buildings 11(7):304. D, Rodríguez-Gonzálvez P, González-Aguilera D (2016) Multi- https:// doi. org/ 10. 3390/ build ings1 10703 04 spectral radiometric analysis of façades to detect pathologies from 57. Sánchez-Aparicio LJ, Del Pozo S, Rodriguez-Gonzalvez P, active and passive remote sensing. Remote Sens 8(1):80. https:// Herrero-Pascual J, Muñoz-Nieto A, Gonzalez-Aguilera D (2016) doi. org/ 10. 3390/ rs801 0080 Practical use of multispectral techniques for the detection of 40. Nocerino E, Dubbini M, Menna F, Remondino F, Gattelli M, pathologies in constructions. In: Riveiro B, Solla M (eds) Non- Covi D (2017) Geometric calibration and radiometric correc- destructive techniques for the evaluation of structures and infra- tion of the MAIA multispectral camera. Int Arch Photogramm structure, 1st edn. CRC Press, London, pp 253–271 Remote Sens Spat Inf Sci. https:// doi. org/ 10. 5194/ isprs- archi 58. Pozo-Antonio JS, Puente I, Pereira MFC, Rocha CSA (2019) ves- XLII-3- W3- 149- 2017 Quantification and mapping of deterioration patterns on granite 41. Verhoeven G (2008) Imaging the invisible using modified digital surfaces by means of mobile LiDAR data. Measurement 140:227– still cameras for straightforward and low-cost archaeological near- 236. https:// doi. org/ 10. 1016/j. measu rement. 2019. 03. 066 infrared photography. J Archaeol Sci 35(12):3087–3100. https:// 59. Kaynig V, Fischer B, Müller E, Buhmann JM (2010) Fully auto- doi. org/ 10. 1016/j. jas. 2008. 06. 012 matic stitching and distortion correction of transmission electron 42. Falco CM (2009) Invited Article: High resolution digital camera microscope images. J Struct Biol 171(2):163–173. https://doi. or g/ for infrared reflectography. Rev Sci Instrum 80(7):071301. https:// 10. 1016/j. jsb. 2010. 04. 012 doi. org/ 10. 1063/1.31744 31 60. Rueden CT, Schindelin J, Hiner MC, DeZonia BE, Walter AE, 43. Webb EK, Robson S, MacDonald L, Garside D, Evans R (2018) Arena ET, Eliceiri KW (2017) Image J2: ImageJ for the next Spectral and 3D cultural heritage documentation using a modi- generation of scientific image data. BMC Bioinform 18(1):529. fied camera. Int Arch Photogramm Remote Sens Spatial Inf Sci. https:// doi. org/ 10. 1186/ s12859- 017- 1934-z https:// doi. org/ 10. 5194/ isprs- archi ves- XLII-2- 1183- 2018 61. Shepard D (1968) A two-dimensional interpolation function 44. Dias IS, Flores-Colen I, Silva A (2021) Critical analysis about for irregularly-spaced data. Proceedings of the 1968 23rd ACM emerging technologies for building’s Façade inspection. Buildings national conference. ACM Press, London, pp 517–524. https:// 11(2):53. https:// doi. org/ 10. 3390/ build ings1 10200 53 doi. org/ 10. 1145/ 800186.810616 45. Marzouk M, Zaher M (2020) Artificial intelligence exploita- 62. HyperCube: Spectral Imagery Analysis Software (2021). https:// tion in facility management using deep learning. Constr Innov www. erdc. usace. army. mil/ Media/ Fact- Sheets/ Fact- Sheet- Artic 20(4):609–624. https:// doi. org/ 10. 1108/ CI- 12- 2019- 0138 le- View/ Artic le/ 610433/ hyper cube. Accessed 21 June 2021 46. Cossu R, Chiappini L (2004) A color image segmentation method 63. Arganda-Carreras I, Kaynig V, Rueden C, Eliceiri KW, Schindelin as used in the study of ancient monument decay. J Cult Herit J, Cardona A, Sebastian Seung H (2017) Trainable Weka Segmen- 5(4):385–391. https:// doi. org/ 10. 1016/j. culher. 2004. 03. 005 tation: a machine learning tool for microscopy pixel classification. 47. Vázquez MA, Galán E, Guerrero MA, Ortiz P (2011) Digital Bioinformatics 33(15):2424–2426. https:// doi. org/ 10. 1093/ bioin image processing of weathered stone caused by efflorescences: forma tics/ btx180 1 3 Journal of Building Pathology and Rehabilitation (2021) 6:41 Page 15 of 15 41 64. Quinlan JR (1993) C4.5: programs for machine learning, 1st edn. 69. Congalton RG, Green K (2019) Assessing the accuracy of Morgan Kaufmann, San Mateo remotely sensed data: principles and practices, 3rd edn. Taylor & 65. Breiman L (2001) Random forests. Mach Learn 45:5–23 Francis Group, Boca Raton 66. Quinlan JR (1996) Learning decision tree classifiers. ACM Com- 70. Das R (2020) Content-based image classification: efficient put Surv 28(1):71–72 machine learning using robust feature extraction techniques, 1st 67. Zhao Y, Zhang Y (2008) Comparison of decision tree methods for edn. Chapman and Hall/CRC, Boca Raton finding active objects. Adv Space Res 41(12):1955– 1959. https:// 71. ICOMOS-ISCS (2008) Illustrated glossary on stone deterioration doi. org/ 10. 1016/j. asr. 2007. 07. 020 patterns, 1st edn. Ateliers 30 Impression, Champigny-sur-Marne 68. Friedman J, Hastie T, Tibshirani R (2000) Additive Logis- tic Regression: a Statistical View of Boosting. Ann Stat Publisher's Note Springer Nature remains neutral with regard to 28(2):337–374 jurisdictional claims in published maps and institutional affiliations. 1 3 http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Journal of Building Pathology and Rehabilitation Springer Journals

Learning-based classification of multispectral images for deterioration mapping of historic structures

Download PDF
15 pages

Loading next page...
 
/lp/springer-journals/learning-based-classification-of-multispectral-images-for-7MGgd1j40U

References (78)

Publisher
Springer Journals
Copyright
Copyright © The Author(s) 2021
ISSN
2365-3159
eISSN
2365-3167
DOI
10.1007/s41024-021-00136-z
Publisher site
See Article on Publisher Site

Abstract

The conservation of historic structures requires detailed knowledge of their state of preservation. Documentation of dete- rioration makes it possible to identify risk factors and interpret weathering mechanisms. It is usually performed using non- destructive methods such as mapping of surface features. The automated mapping of deterioration is a direction not often explored, especially when the investigated architectural surfaces present a multitude of deterioration forms and consist of heterogeneous materials, which significantly complicates the generation of thematic decay maps. This work combines reflec- tance imaging and supervised segmentation, based on machine learning methods, to automatically segment deterioration patterns on multispectral image composites, using a weathered historic fortification as a case study. Several spectral band combinations and image classification techniques (regression, decision tree, and ensemble learning algorithmic implementa- tions) are evaluated to propose an accurate approach. The automated thematic mapping facilitates the spatial and semantic description of the deterioration patterns. Furthermore, the utilization of low-cost photographic equipment and easily operable digital image processing software adds to the practicality and agility of the presented methodology. Keywords Built heritage · Deterioration mapping · Multispectral reflectance imaging · Thermal infrared imaging · Supervised image segmentation · Machine learning 1 Introduction Recording the preservation state of a historic structure is a crucial prerequisite for pathology diagnosis. Document- Architectural heritage possesses outstanding value while ing in detail the condition of the structure's elements is the concomitantly comprises a fundamental manifestation first step towards qualitatively interpreting its condition and of sociocultural identity. The historic build environment identifying mechanisms of deterioration. Therefore defin- is a vital aspect of a place's culture, history, and land- ing the data recording techniques that will provide rich and scape, which necessitates measures to ensure its preser- suitable information about the extent and forms of deteriora- vation through time. However, environmental pressures tion is essential for condition documentation. To the greatest and anthropogenic factors cause constant alterations and extent possible, recording should be non-destructive, mean- impose significant risks. Planning appropriate and compat- ing that it should encompass those nonintrusive inspection ible conservation and restoration interventions to tackle the and sensing techniques that do not cause further damage deterioration of historic structures requires a comprehensive to, nor impair the future usefulness of the structure and the knowledge of the preservation state. Thus, the need for the historic materials. historic structures' recording emerges, which will provide Mapping is widely recognized as an effective non- the detailed information needed to support required preser- destructive method useful for condition documentation and vation interventions. can be applied to all materials at different scales. It registers information about the surface patterns of historic structures that can be later analyzed through computational systems. Mapping is frequently performed as a manual process in * Efstathios Adamopoulos a computer-aided design (CAD) or geographic informa- efstathios.adamopoulos@unito.it tion system (GIS) environment using as background color Department of Computer Science, University of Turin, photos. Turin, Italy Vol.:(0123456789) 1 3 41 Page 2 of 15 Journal of Building Pathology and Rehabilitation (2021) 6:41 Progress in automated mapping for historic structures 2.1 Mapping the preservation state has primarily concentrated on identifying and classifying building elements, materials, and additionally deterioration Mapping is a valuable non-destructive method as it facili- as a binary concept—considering the presence and absence tates the description, registration, and quantification of the, of deterioration solely. The segmentation algorithms that often overlapping, multitude of surface patterns on historic have been considered are mainly based on dimensional- structures. When performed in a digital, computerized man- ity reduction, unsupervised clustering, and deep learning ner, it produces spatial information, entities with geometric approaches, occasionally considering spectral bands at the attributes that can be correlated, compared, used to pro- infrared range. duce statistical information, and allow for the attribution of semantic data about the characteristics of materials and 1.1 Aims and scope their decay. Traditionally, mapping is a technique manually performed inside CAD and GIS platforms by describing the This work delves into the fields of imaging science and pat- shape of surface patterns and organizing them into thematic tern recognition to identify a novel and accurate methodol- layers [1–6]. An alternative way of mapping deterioration ogy for classifying different deterioration forms on historic is the visualization of damage levels/indexes, which can be structures. Reflecting on the potential of multispectral imag- either accomplished directly or indirectly by analyzing the ing and learning-based image analysis for defect detection, mapped deterioration patterns [7–15]. The additional spatial the classification of multispectral composites synthesized annotation of lithotypes facilitates the association between from reflectance images captured at the visible (RGB), near- materials and alteration [1, 11–13, 15–23]. Mapping sup- ultraviolet (NUV), near-infrared (NIR), and thermal infrared ports the interpretation of weathering phenomena when (TIR) spectra, with supervised segmentation methods based combined with data from non-destructive testing (NDT) [8, on random decision trees, ensemble learning, and regression 9, 17–20, 22, 24, 25], laboratory mineralogical, chemical algorithmic implementations, is thoroughly evaluated. and physical characterization [15, 18, 19, 23, 26], and envi- ronmental measurements [8, 12, 18, 19, 25, 27]. 1.2 Article structure 2.2 Generating base‑maps for deterioration This article is structured into six sections. Section 2 delivers mapping an overview of the background for the presented work and discusses the related research. Section 3 describes the meth- Mapping is typically a photo-based approach where a color odology followed, including the instrumentation, data col- image, an orthorectified image, or an orthoimage-mosaic is lection and preparation, algorithmic implementations, and used as a base-map for designing the geometrical shape of approaches followed to evaluate the segmentation results. surface patterns [27]. The metric (accuracy, scale-dependent Section 4 presents the application and results for the case spatial resolution) and chromatic quality of this background study of a historic fortification, while Sect.  5 discusses the are essential for identifying deterioration [25, 28, 29]. Thus, accuracy and interpretation of the results. The concluding acquiring suitable images is crucial for successful deterio- remarks are presented in Sect. 6. ration mapping. However, not only true color images have been considered as base maps, but also images captured at portions of the electromagnetic spectrum beyond the visible. 2 Background and related work 2.3 Multispectral imaging and data Architectural surfaces of historic structures are subjected complementarity to continuous alterations due to exposure to environmental conditions, microorganisms, pollution, anthropogenic dam- The reciprocity of mapping and infrared reflectance imag- ages; their susceptibility to decay also depends on (incom- ing—especially thermography—has often been considered patible) conservation interventions of the past and the inher- essential for detecting weathering on historic structures [5, ent characteristics of historic materials. Particularly when 20, 24, 30, 31]. Besides, thermography is being extensively several different materials are present (such as in masonry used in built heritage structural diagnostics [32–34] and has structures), the architectural surfaces consist of an intricate also been explored to detect different historic materials on mosaic of deterioration forms. Consequently, documentation building façades [35, 36]. The additional inclusion of NIR methods for describing these complicated conditions in a reflectance images enhances the identifiability of deterio- non-destructive way become pertinent and often necessary. ration, mainly when there is a presence of vegetation and biogenic crusts, which present vastly different near-infrared 1 3 Journal of Building Pathology and Rehabilitation (2021) 6:41 Page 3 of 15 41 Fig. 1 Overall research methodology Table 1 Specifications of digital cameras used for acquiring multi- is cut off by an internal blocking filter. Removing this filter spectral reflectance data implies that the camera can be used for imaging at a wider than visible range, and external wavelength-specific filters Camera model Canon EOS Rebel SL1 FLIR ONE Pro can be utilized. Detection in the long-wavelength infrared Spectral range 0.3–1.1 μm 8–14 μm (LWIR) range has usually been performed with uncooled Resolution 5184 × 3456 pixels 160 × 120 pixels microbolometer detectors for building inspections. The Pixel pitch 4.3 μm 12 μm spatial resolution of thermography cameras is considerably FOV – 43° ± 1° lower than that of DSLR, and their relative cost is higher. NETD – 70 mK Recently, more affordable thermography camera models Measurement accuracy – ± 5% have come into the market, including smartphone-adjusta- ble low-resolution instruments. However, these inexpensive Field-of-view cameras provide lower accuracy, which makes them unus- Noise equivalent temperature difference (thermal sensitivity) c able for some applications. Typical percentage of the difference between ambient and scene tem- perature 2.4 Digital image processing The need for more efficient inspection [44] and intelligent reflectance characteristics compared with construction mate- rials [37, 38]. However, the decision to include recorded data identification of conservation needs [45] has led to the adop- tion of image processing approaches to generate the thematic from multiple bands comes with the realization that suitable sensing techniques have to be selected. data needed for deterioration mapping. Digital image pro- cessing (DIP) refers to the manipulation of the digital images Spectral collection in the infrared is connected with various sensing techniques that depend on the wavelength to extract features and recognize patterns, which, after hav- ing acquired the suitable base-maps, can be performed choice. Detection in the wavelength range between 400 and 1100 nm has been performed with multispectral configura- with techniques as simple as thresholding, edge detection, or information reduction to obtain the required results [33, tions that involve multiple single-band cameras recording at 4–12 different narrow spectral bands. The resolution of these 46–48]. However, these approaches still largely depend on the human factor since many parameters have to be tuned instruments is usually low, and the collected imagery has to be meticulously checked to correct sensors' errors [39, 40]. differently for each application, and often deterioration pat- terns have to be identified and extracted one at a time. The The introduction, or rather repurposing, of commercial digi- tal single-lens reflex (DSLR) cameras with charge-coupled current rise of deep learning-based pattern recognition has delivered powerful tools for fully automated detection of device (CCD) and complementary metal–oxide–semicon- ductor (CMOS)-based detectors, for spectral imaging at the deterioration (often through convolutional neural networks), even when a plethora of surface patterns can be observed same range, however, provides more affordable and agile solutions that retain the user-friendly features and the inter- [49–52]. Nevertheless, deep-learning implementations require large image datasets to be efficiently trained, which faces to a wide variety of photographic software and acces- sories, and have high spatial resolution [41–43]. Commercial is often impractical for built heritage applications. They may also underperform considering the uniqueness of each off-the-shelf (COTS) DSLR camera detectors are generally sensitive in a portion of the NIR range up to 1100 nm, which heritage asset, many of which present a distinctive mixture 1 3 41 Page 4 of 15 Journal of Building Pathology and Rehabilitation (2021) 6:41 of historic materials. Therefore, other more easily execut- acquiring appropriate images and then continues with their able supervised learning-based approaches are sometimes radiometric correction. The multispectral composites are considered for deterioration detection through classification digitally synthesized from the band-specific reflectance and regression. images and subsequently segmented into deterioration cat- Multiband and multispectral image segmentation for built egories following a visual identification of training regions. heritage inspection purposes has been applied via a range The results are evaluated with metrics deriving from the of clustering algorithms, some of the most common being field of remote sensing. The output of the deterioration maximum-likelihood, minimum-distance, and k-means [36, classification can be optionally transferred to an environ- 37, 39, 48, 53–56]. However, most of the relevant works ment appropriate for spatial information management. The aim at segmenting the materials and elements of historic principle of using low-cost equipment and software was fol- façades, and when deterioration is considered, it is deter- lowed throughout this work as it is an essential factor for the mined as present or not present. To be specific, many works inspection of historic buildings. consider the altered and unaltered areas of a historic material as two categories rather than identifying the different dete- 3.1 Sensors and data acquisition rioration typologies, which is also partly a result of the state of preservation of the heritage assets involved. Alternative The selection of the instruments employed in this work multi-sensor approaches, involving terrestrial LiDAR for considers the complementarity of data captured at differ - NIR recording, have been reported to produce high-accuracy ent spectral bands and the flexibility requirements of sens- thematic mapping results for damaged historic structures ing techniques used for built heritage condition monitoring. [57, 58]. However, they introduce significant instrumenta- Affordable, portable sensors are utilized to obtain the neces- tion costs, and require rigorous radiometric calibrations and sary multispectral data that will constitute the background optimal data gathering conditions. for the deterioration pattern analysis, contributing to a sim- ple to implement methodology. The characteristics of the instrumentation are presented in Table 1. The images are 3 Methods and materials taken with two sensors, an EOS Rebel SL1 (Canon Inc., Tokyo, Japan) digital single-lens reflex camera with an EF-S The rationale behind this work is set on the identified lack 18-55 mm f/3.5–5.6 IS II lens, and a FLIR ONE Pro (Tel- of image-based methods for automatic mapping of weath- edyne FLIR LLC, Wilsonville, OR, USA) thermographic ered historic structures. The methods tested aim to tackle the camera attached to a smartphone. The internal hot mirror problematics of mapping the preservation state when various filter of the SL1 camera has been removed to allow imag- surface deterioration forms are present. Instead of following ing beyond the visible range. Three low-cost external filters unsupervised segmentation techniques and then interpret- are employed to allow RGB, NUV, and NIR photo shoot- ing each classified category of weathering-caused alteration, ing. The images are acquired as parallel as possible to the supervised algorithmic approaches are implemented using as architectural surfaces to avoid occlusions, and with small input the already identified deterioration categories. Com- focal lengths to avoid large distortions that can affect image binations of different spectral band composite images and quality during the resampling phase of distortion correction. supervised segmentation algorithms are evaluated to distin- Furthermore, the images are acquired under homogeneous guish an optimal solution in terms of accuracy—based on illumination conditions and without shadows, improving reference data. their radiometric potential and with a steady tripod, thus Figure  1 depicts the implemented research design in preventing image blur. Since low-cost sensors are more this work. As already highlighted, the quality of available likely to be affected by noise sources, the camera sensor is imagery upon which the pattern recognition will be per- checked to estimate the vignetting and background noise formed is essential for ensuring the accuracy and inter- levels, and the images are corrected to ensure their quality. pretability of results. Therefore, the workflow starts from Table 2 Multispectral image Multispectral image Red band Green band Blue band composition G-B-NUV Green Blue Near-ultraviolet R-G-B Red Green Blue NIR-R-G Near-infrared Red Green TIR-NIR-R Thermal infrared Near-infrared Red NIR-M-NUV Near-infrared RGB monochromatic Near-ultraviolet 1 3 Journal of Building Pathology and Rehabilitation (2021) 6:41 Page 5 of 15 41 The thermographic data are acquired at sets of burst images 3.3 Machine learning‑based segmentation to increase digitally later their spatial resolution. of deterioration patterns 3.2 Multispectral data preparation The classification of deterioration patterns is performed via a supervised segmentation procedure using the Train- Pre-processing the imagery data involves the preparation able WeKa Segmentation 3D plugin [63] of ImageJ2. The of multispectral image composites for the subsequent seg- machine learning-based image segmentation techniques fol- mentation. At first, the radiance images acquired with the low decision tree [64], ensemble learning [65], and regres- SL1 camera are downloaded in the RawDigger (LibRaw sion approaches. Specifically, the Random Tree, Random LLC, Maryland, USA) software, where the color filter array Forest, Fast Random Forest, and LogitBoost classifiers are conversion is reversed to acquire raw radiance images, and employed. The supervised approach presupposes the annota- RGB images are color balanced. Non-visible spectrum tion of image regions of interest (ROIs), corresponding to images should also be converted to reflectance images based each semantic deterioration category to be segmented, that on pixel values of a reference surface. The uncompressed will train the algorithmic model into providing a semantic images are then corrected from distortion [59] in ImageJ2 classification of the entire image. [60]. Thermal infrared burst mode images acquired with the The decision tree model is a machine learning algorithm FLIR ONE Pro camera are used to create high-resolution that can be used for both supervised classification and thermal images [61]. regression problems. A decision tree simply consists of a The manual matching of band-specific images is done series of sequential decisions made to reach a specific result using the HyperCube software [62] (projective transfor- of distinct data classes. The classes are mutually exclusive mation, nearest-neighbor interpolation). Subsequently, the and represented by specific attributes. The learning input, image composites are constructed using different multispec- which consists of sets of pixels belonging to known classes, tral combinations, as described in Table 2. The images are assists the accurate classification of both annotated pixels resampled to match the resolution of all bands, and the sky and not annotated pixels. Each node of the decision tree and ground are trimmed from all multispectral composites decides an outcome based on the attribute values and leads to reduce potential misclassifications. The synthesis of the either to another node, using an appropriate subtree, or to a multiband composites also considers the same principle of leaf, which gives the predicted class of the pixel [66]. The using low-cost equipment, and thus all composites consist Random Tree classifier is based on a decision tree learning of three bands so that segmentation can be performed in method. Single decision trees are easy to conceptualize but ImageJ2—avoiding the use of commercial specialized spa- usually suffer from high variance, making them not competi- tial analysis software. tive in terms of accuracy. Fig. 2 Fort of Karababa, bird’s- eye view 1 3 41 Page 6 of 15 Journal of Building Pathology and Rehabilitation (2021) 6:41 LogitBoost is a boosting algorithm that performs classifi- cation using a regression scheme as the base learner and can handle multi-class problems. It can be seen as a convex opti- mization; it applies the cost function of logistic regression on a generalized additive model. This classifier determines the appropriate number of iterations by performing efficient internal cross-validation [68]. 3.4 Accuracy metrics The performance of the machine learning classification implementations, and of the different multispectral combina- tions, are quantitatively evaluated using manually produced degradation maps as the ground truth. Different parameters are used to assess the classification efficiency of the intel - ligent feature extraction techniques based on accuracy met- rics common for thematic mapping. More specifically, the evaluation relies on the precision (fraction of appropriate classification among the classified instances) and F1-score (harmonic mean of precision and sensitivity) calculated for each class (Eqs. 1,2), and on the overall accuracy (Eq. 3)— useful to estimate the overall performance of the classifiers. TP Precision = (1) TP + FP 2TP F1 Score = (2) TP + FP + TP + FN Sum of correctly classified units Overall accuracy = (3) Total number units where, for each class the TP (true positive), FP (false posi- tive), and FN (false negative) come from the error matrix, a square array of numbers, which express the number of pixels assigned to a particular class in one classification relative to the number of pixels assigned to a particular class in the reference data [69, 70]. Fig. 3 Fort of Karababa north side, façades selected for evaluating 3.5 Case study the mapping methodology; from top to bottom: A (westernmost), B, and C (easternmost) The historic structure selected as a case study for the appli- cation and assessment of the proposed methodology is a for- A random forest classifier combines ensemble classifica- tification in Euboea, Greece (Fig.  2). The Fort of Karababa is tion machine learning algorithms and decision trees. Each an Ottoman fortification constructed in 1684 on the homony - tree classifier is independently generated from the input mous hill which dominates the Boeotian coast across the city training data using a random sample like in bagging. When of Chalcis. The construction of the fortress was part of the growing a tree, the best possible split is computed for a ran- effort to protect the city of Chalcis from impending Venetian dom subset, instead of always computing the best split for attacks. The architectural style of the fort is more European each node. In this way, tree diversity is generated using two than Turkish. It is oblong in plan, with a rampart on the ways of randomization. Aggregating predictions make the north side, three bastions, and a large tower. Several parts class prediction of the ensemble. Random forest generally of the fortification walls have ancient spolia built-in, while overcomes the accuracy limitations of single decision trees the south part is preserved in poor condition. The weathered [65, 67]. 1 3 Journal of Building Pathology and Rehabilitation (2021) 6:41 Page 7 of 15 41 Fig. 4 Multispectral data preparation for façade C. Note: NUV near-ultraviolet; R red; B blue; G green; NIR near infrared; TIR thermal infrared; M monochromatic color image masonry surfaces selected for evaluating the methodology 4 Results are presented in Fig. 3. They are on the north side, and for abbreviation purposes, they have been named A, B, and C, Following the described methodologies, after the compo- starting with the westernmost (on the north bastion). sition of multispectral images was completed (Fig. 4), 60 classifications were performed. The generation of reference 1 3 41 Page 8 of 15 Journal of Building Pathology and Rehabilitation (2021) 6:41 Table 3 Overall accuracy statistics by image and classifier especially for classifications performed with ensemble learning-based algorithmic implementations. Furthermore, A B C the inclusion of different spectral bands improved the clas- Overall accuracy (%) sification potential, subject to the categories of deterioration G-B-NUV present.  LogitBoost 82.1 66.9 58.4  Random tree 71.0 66.3 50.9  Random forest 84.9 69.9 57.3 5 Discussion  Fast random forest 84.6 70.2 58.4 R-G-B The inclusion of the NIR spectral band fairly improved the  LogitBoost 77.3 67.9 78.5 classification results for all deterioration forms. The seg -  Random tree 73.7 63.7 74.3 mentation of a NIR-R-G multispectral image and the Fast  Random forest 84.4 69.8 81.9 Random Forest classifier proved to be the most consistent  Fast random forest 84.4 69.9 83.1 solution overall (79 ≤ overall accuracy%). Figure 5 presents NIR-R-G a comparison between the reference maps and the NIR-  LogitBoost 80.4 71.3 80.1 R-G composites segmented with the Fast Random Forest.  Random tree 72.6 71.0 77.4 Using NUV reflectance data generally did not provide any  Random forest 85.4 75.8 83.8 improvement to the quality of the classifications. Including  Fast random forest 86.3 79.0 84.6 the TIR band also did not improve the deterioration patterns' TIR-NIR-R classification. Furthermore, the fusion of visible with ther -  LogitBoost 74.6 76.3 58.4 mal data significantly decreased the accuracy of detecting  Random tree 64.9 71.3 50.9 deterioration when dampness was present, which contradicts  Random forest 76.8 76.3 57.3 that thermal images are helpful in detecting moisture on his-  Fast random forest 78.5 77.7 58.4 toric masonry, as evident by Fig. 6. NIR-M-NUV According to the overall accuracy results, the Fast Ran-  LogitBoost 83.2 71.6 75.3 dom Forest classifier was the most accurate learning-based  Random tree 75.4 66.8 66.6 method for deterioration classification for all multispec-  Random forest 85.5 69.3 77.0 tral images, not including the TIR band (70% < overall  Fast random forest 86.8 69.8 78.2 accuracy < 87%). Implementing the random tree classifier resulted in more inconsistent and less accurate classifications (60% < overall accuracy < 77%). LogitBoost outperformed the Random Tree classifier. According to the precision and F1-score values, moss and maps considered the Illustrated Glossary on Stone Dete- lichens were the most misclassified surface patterns, even rioration Patterns [71] as a guide during visual inspection. though both random forest approaches improved their clas- The observed categories of deterioration were vegetation, sification. The results prove that the distinction among non- moss, black crusts, lichens, missing material (including loss deteriorated material, dampness, black crusts/discoloration, of components, large cracks, and windows), and dampness. and plants is much more easily detectable (and therefore These constituted all the categories of surface pathology that classifiable) than biogenic colonization of any form. There- altered the surface reflectance characteristics of the masonry fore, surface alterations of the historic materials—which façades. Patterns that caused slight geometrical surface alter the reflectance characteristics— can be more accurately alterations, such as minor cracks, superficial cracking due mapped using multispectral images in comparison with the to biogenic deterioration, disintegration, or other shape fea- deterioration forms that completely cover them as an addi- tures induced by material loss, insignificant concerning the tional layer. considered scale and the reflectance contract comparing with healthy historic materials could not be considered. The the- matic comparisons were performed using the full reference 6 Conclusions maps and not sampled patch areas. Overall accuracy statis- tics calculated from the confusion matrixes are presented in In this work, a novel methodology for the automatic clas- Table 3. The precision and F1-score results are presented in sification of damage on built cultural heritage was proposed detail in “Appendix A”. that uses low-cost photographic equipment for multispec- The deterioration maps produced for all the studied archi- tral data acquisition and supervised machine learning-based tectural surfaces were of generally high thematic accuracy, image segmentation to map deterioration patterns. It was 1 3 Journal of Building Pathology and Rehabilitation (2021) 6:41 Page 9 of 15 41 Fig. 5 Reference deterioration maps (left), and corresponding deterioration maps produced with a NIR-R-G multispectral image using the Fast Random Forest Classifier (right); façades A, B, and C (from top to bot- tom) Fig. 6 Thermograms of façades A (left), and B (right) confirmed that including near-infrared reflectance intensi- The segmentation of multispectral composites (synthe- ties in the employed methods improved the classification of sized with visible and near-infrared reflectance images), alterations on the historic masonry façades. with classifiers combining random trees and ensemble learn- ing, performed particularly well even were a high number 1 3 41 Page 10 of 15 Journal of Building Pathology and Rehabilitation (2021) 6:41 of surface patterns was present. However, the coexistence category. Furthermore, there is a clear advantage over deep of different overlapping categories of biogenic colonization learning-based methods, that require large image datasets, complicated the mapping procedure significantly. It should for rapid monitoring purposes of monumental heritage struc- be highlighted that the accuracy evaluation considered some tures. A direct outlook of the proposed framework is the level of bias since the manually produced reference thematic combination with 3D recording technologies to enhance the maps cannot consider the overlapping surface patterns. capability of detecting and mapping the geometric altering The proposed methodology has the limitation that it can (material loss) of historic monuments. map only the pathologies that have been previously recog- nized through visual inspection (or analytical techniques) because regions of interest have to be annotated to train the Appendix A intelligent algorithms. However, a crucial advantage is that it produces easily interpretable mapping results, in contra- See Tables 4, 5, 6. diction to unsupervised methods where each mapped pat- tern class has to be a posteriori assigned to a deterioration Table 4 Accuracy statistics calculated for façade A Leafy vegetation No deterioration Black crusts Missing material Dampness G-B-NUV  LB 44.4 60.3 87.5 82.1 56.7 60.4 39.0 53.4 91.7 90.4  RT 28.7 42.6 80.1 77.0 46.1 48.3 6.8 12.3 86.0 79.1  RF 46.3 61.4 86.3 87.0 75.5 64.1 37.8 53.2 89.8 91.7  FRF 41.8 55.8 85.3 87.7 81.8 60.2 45.4 60.5 88.5 91.0 R-G-B  LB 27.7 42.6 85.8 80.4 49.9 51.2 26.0 39.5 87.6 88.5  RT 27.9 42.0 77.7 78.6 57.0 53.0 13.8 23.5 85.4 81.3  RF 39.0 54.6 87.4 87.5 74.7 64.3 40.8 56.0 89.2 90.3  FRF 40.5 55.0 85.5 88.0 83.0 58.6 45.7 60.7 87.8 90.0 NIR-R-G  LB 41.4 57.5 87.0 81.2 51.4 56.6 31.4 45.0 91.4 92.1  RT 26.6 40.1 79.0 77.4 50.8 45.7 11.6 20.2 83.4 81.5  RF 48.6 63.9 88.0 87.3 76.7 67.9 28.6 43.2 89.8 91.6  FRF 54.3 68.0 86.4 88.3 84.6 64.8 43.1 58.0 89.6 92.2 TIR-NIR-R  LB 38.9 55.0 77.5 71.4 45.4 52.3 34.5 48.4 90.5 89.6  RT 16.2 27.3 77.2 59.3 39.5 51.2 10.0 17.7 91.8 85.7  RF 31.1 46.5 80.4 75.1 51.0 58.4 39.9 52.9 92.5 90.0  FRF 32.2 47.6 79.8 77.6 56.0 61.0 43.8 55.4 93.4 89.6 NIR-M-NUV  LB 40.5 56.5 88.9 83.9 59.3 63.8 35.1 48.4 92.6 92.5  RT 17.6 29.4 82.9 78.6 59.8 59.1 15.9 26.8 90.0 86.5  RF 38.9 55.0 87.6 87.5 76.3 70.2 26.6 41.0 92.9 92.2  FRF 44.5 59.3 87.3 88.6 84.5 69.4 43.0 58.1 90.9 92.4 Precision F1-score Precision F1-score Precision F1-score Precision F1-score Precision F1-score LB LogitBoost; RT random tree; RF random forest; FRF fast random forest 1 3 Journal of Building Pathology and Rehabilitation (2021) 6:41 Page 11 of 15 41 1 3 Table 5 Accuracy statistics calculated for façade B Leafy vegetation No deterioration Black crusts Lichens G-B-NUV  LB 52.2 62.0 66.4 75.7 92.8 69.0 30.0 41.9  RT 47.7 60.2 66.2 70.8 90.3 72.0 30.5 43.0  RF 47.4 61.6 66.8 75.0 93.5 73.6 38.2 50.8  FRF 47.2 61.0 62.5 72.5 93.4 73.9 46.3 56.0 R-G-B  LB 42.1 56.4 64.2 72.5 91.4 72.4 36.1 46.3  RT 39.6 54.8 64.2 69.5 89.6 68.5 29.9 42.1  RF 47.7 62.0 64.3 73.5 92.9 73.5 41.2 52.1  FRF 50.4 63.8 59.1 70.2 92.8 73.2 51.0 57.1 NIR-R-G  LB 55.3 65.1 65.3 72.7 92.1 77.2 40.2 51.5  RT 47.9 60.8 70.5 74.2 88.3 77.0 36.4 47.7  RF 43.3 58.0 75.4 78.7 92.4 82.1 43.4 54.6  FRF 43.5 57.9 74.7 78.3 89.9 85.9 63.3 62.0 TIR-NIR-R  LB 37.1 45.5 80.3 82.5 92.6 83.5 36.2 47.3  RT 32.8 43.8 83.3 84.7 91.0 75.8 28.8 41.0  RF 37.6 49.8 90.0 84.9 93.5 83.8 33.1 46.9  FRF 37.2 47.2 89.2 86.6 92.8 84.1 35.8 49.3 NIR-M-NUV  LB 49.3 55.6 68.2 74.5 92.7 77.6 40.5 51.7  RT 36.4 48.3 66.7 70.2 89.8 74.7 29.2 39.4  RF 39.6 54.1 65.5 72.4 93.9 74.9 39.3 51.8  FRF 42.0 55.8 59.3 69.2 93.7 75.5 53.0 58.9 Precision F1-score Precision F1-score Precision F1-score Precision F1-score LB LogitBoost; RT random tree; RF random forest; FRF fast random forest 41 Page 12 of 15 Journal of Building Pathology and Rehabilitation (2021) 6:41 Table 6 Accuracy statistics calculated for façade C Leafy vegetation No deterioration Black crusts Missing material Plant Dampness G-B-NUV  LB 21.6 35.4 91.8 75.2 26.8 37.1 30.7 46.1 72.4 38.3 87.3 88.4  RT 24.1 38.7 92.3 74.5 23.9 36.0 8.1 14.7 67.7 38.4 88.4 84.7  RF 47.3 63.7 89.9 78.1 32.1 46.7 25.2 39.7 63.4 38.2 92.1 91.1  FRF 47.3 63.1 88.5 78.9 37.2 51.8 29.4 45.1 57.3 37.4 91.9 91.3 R-G-B  LB 29.5 45.3 78.8 77.8 29.3 41.7 29.9 45.2 67.4 37.8 90.6 83.9  RT 11.4 19.9 76.7 73.1 19.8 29.9 22.2 35.5 63.8 38.2 84.1 74.1  RF 52.6 68.2 79.6 76.3 34.7 50.2 27.7 42.6 57.1 37.3 90.0 85.1  FRF 61.4 74.6 78.3 76.0 44.6 58.2 27.0 42.0 53.4 36.5 88.4 86.0 NIR-R-G  LB 21.1 34.9 90.1 78.6 32.1 43.3 31.7 47.0 68.5 38.0 89.2 88.7  RT 26.6 40.8 88.7 78.3 23.0 33.8 25.9 40.1 70.1 37.4 89.4 86.6  RF 64.4 78.4 89.2 79.3 38.9 54.4 28.5 43.8 53.8 36.7 91.9 91.5  FRF 67.7 80.7 88.1 80.2 44.0 58.7 31.1 46.9 54.7 36.8 91.5 91.6 TIR-NIR-R  LB 18.6 31.3 54.9 61.1 25.6 38.8 25.2 39.3 49.9 34.6 83.3 62.9  RT 8.5 15.5 53.8 56.2 18.2 28.7 24.6 37.8 28.4 28.6 80.2 57.1  RF 21.4 35.0 50.7 57.9 31.3 45.7 31.4 46.8 38.5 32.7 80.9 61.1  FRF 23.8 37.9 48.8 56.2 40.7 54.2 33.4 49.3 37.4 32.3 78.7 62.6 NIR-M-NUV  LB 24.0 38.5 80.7 72.6 23.5 32.0 26.9 41.2 68.8 38.0 86.5 85.1  RT 13.1 22.7 78.9 70.6 20.2 30.5 13.2 22.8 56.6 35.6 82.2 75.2  RF 36.0 52.4 80.1 73.8 36.7 52.1 25.1 39.7 59.4 37.7 87.4 84.5  FRF 43.9 59.3 78.9 73.8 42.4 56.4 25.7 40.5 54.1 36.7 86.7 85.1 Precision F1-score Precision F1-score Precision F1-score Precision F1-score Precision F1-score Precision F1-score LB LogitBoost; RT random tree; RF random forest; FRF fast random forest Acknowledgements The author acknowledges the Hellenic Ministry Declarations of Culture and Sports/Archaeological Resources Fund. All copyrights to the depicted monuments belong to the Hellenic Ministry of Cul- Conflict of interest The author declares no conflict of interest. The ture and Sports (law 3028/2002). The fort of Karababa in Chalcis falls funders had no role in the study's design, in the collection, analyses, or within the jurisdiction of the Ephorate of Antiquities of Euboea. The interpretation of data, in the writing of the manuscript, or in the deci- author extends his gratitude to the Ephorate of Antiquities of Euboea sion to publish the results. for granting permission to capture, reproduce and disseminate for research purposes images of archaeological content regarding the Open Access This article is licensed under a Creative Commons Attri- depicted monument. bution 4.0 International License, which permits use, sharing, adapta- tion, distribution and reproduction in any medium or format, as long Funding Open access funding provided by Università degli Studi di as you give appropriate credit to the original author(s) and the source, Torino within the CRUI-CARE Agreement. This study was funded by provide a link to the Creative Commons licence, and indicate if changes the European Union’s Framework Program for Research and Innova- were made. The images or other third party material in this article are tion Horizon 2020 (H2020) under the Marie Skłodowska-Curie Actions included in the article's Creative Commons licence, unless indicated (MSCA) COFUND scheme (grant number 754511), and by the founda- otherwise in a credit line to the material. If material is not included in tion Compagnia di San Paolo. the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will Availability of data and material Restrictions apply to the availability need to obtain permission directly from the copyright holder. To view a of the images used for this project. Images were acquired with the copy of this licence, visit http://cr eativ ecommons. or g/licen ses/ b y/4.0/ . permission of the Ephorate of Antiquities of Euboea and are available from the author only if proper authorization can be obtained from the Ephorate of Antiquities of Euboea. 1 3 Journal of Building Pathology and Rehabilitation (2021) 6:41 Page 13 of 15 41 16. McCabe S, Smith BJ, Warke PA (2007) An holistic approach References to the assessment of stone decay: Bonamargy Friary, Northern Ireland. Geol Soc Spec Publ 271:77–86. https:// doi. org/ 10. 1144/ 1. Fitzner B, Heinrichs K (2001) Damage diagnosis on stone monu- GSL. SP. 2007. 271. 01. 09 ments–weathering forms, damage categories and damage indices. 17. Delegou ET, Tsilimantou E, Oikonomopoulou E, Sayas J, Ioan- Acta Univ Carol Geol 45(1):12–13 nidis C, Moropoulou A (2013) Mapping of building materials and 2. Inkpen R, Duane B, Burdett J, Yates T (2008) Assessing stone conservation interventions using GIS: the case of Sarantapicho degradation using an integrated database and geographical infor- Acropolis and Erimokastro Acropolis in Rhodes. Int J Herit Digit mation system (GIS). Environ Geol 56:789–801. https:// doi. org/ Era 2(4):631–653. https:// doi. org/ 10. 1260/ 2047- 4970.2. 4. 631 10. 1007/ s00254- 008- 1309-x 18. Adamopoulos E, Tsilimantou E, Keramidas V, Apostolopou- 3. Brunetaud X, Luca LD, Janvier-Badosa S, Beck K, Al-Mukhtar lou M, Karoglou M, Tapinaki S, Ioannidis C, Georgopoulos A, M (2012) Application of digital techniques in monument preser- Moropoulou A (2017) Multi-sensor documentation of metric and vation. Eur J Environ Civ Eng 16(5):543–556. https://doi. or g/10. qualitative information of historic stone structures. ISPRS Ann 1080/ 19648 189. 2012. 676365 Photogramm Remote Sens Spatial Inf Sci. https://doi .or g/10. 519 4/ 4. Janvier-Badosa S, Beck K, Brunetaud X, Al-Mukhtar M (2013) isprs- annals- IV-2- W2-1- 2017 Historical study of Chambord Castle: basis for establishing the 19. Martínez-Martínez J, Benavente D, Jiménez Gutiérrez S, García- monument health record. Int J Archit Herit 7(3):247–260. https:// del-Cura MA, Ordóñez S (2017) Stone weathering under Mediter- doi. org/ 10. 1080/ 15583 058. 2011. 634959 ranean semiarid climate in the fortress of Nueva Tabarca island 5. Mileto C, Vegas F, Lerma JL (2015) Multidisciplinary studies, (Spain). Build Environ 121:262–276. https:// doi. org/ 10. 1016/j. crossreading and transversal use of thermography: the Castle of build env. 2017. 05. 034 Monzón (Huesca) as a case study. In: Rodríguez-Navarro P (ed) 20. İnce İ, Bozdağ A, Tosunlar MB, Hatır ME, Korkanç M (2018) Defensive architecture of the mediterranean. Editorial Universitat Determination of deterioration of the main facade of the Ferit Paşa Politècnica de València, Valencia, pp 405–412 Cistern by non-destructive techniques (Konya, Turkey). Environ 6. Janvier-Badosa S, Brunetaud X, Beck K, Al-Mukhtar M (2016) Earth Sci 77(11):420. https://doi. or g/10. 1007/ s12665- 018- 7595-z Kinetics of stone degradation of the castle of Chambord in France. 21. Gulotta D, Toniolo L (2019) Conservation of the built heritage: Int J Archit Herit 10(1):96–105. https:// doi. org/ 10. 1080/ 15583 pilot site approach to design a sustainable process. Herit 2(1):797– 058. 2014. 971194 812. https:// doi. org/ 10. 3390/ herit age20 10052 7. Fitzner B, Heinrichs K, Bouchardiere DL (2003) Weathering dam- 22. Hatır ME, Korkanç M, Başar ME (2019) Evaluating the deteriora- age on Pharaonic sandstone monuments in Luxor-Egypt. Build tion effects of building stones using NDT: the Küçükköy Church, Environ 38(9–10):1089–1103. https:// doi. org/ 10. 1016/ S0360- Cappadocia Region, central Turkey. Bull Eng Geol Environ 1323(03) 00086-6 78(5):3465–3478. https:// doi. org/ 10. 1007/ s10064- 018- 1339-x 8. Heinrichs K (2008) Diagnosis of weathering damage on rock- 23. Iandelli N, Coli M, Donigaglia T, Ciuffreda AL (2021) An uncon- cut monuments in Petra. Jordan Environ Geol 56(3–4):643–675. ventional e fi ld mapping application: a complete opensource work - https:// doi. org/ 10. 1007/ s00254-008- 1358-1 flow solution applied to lithological mapping of the coatings of 9. Heinrichs K, Fitzner B (2011) Assessment of weathering damage cultural heritage. IJGI 10(6):357. https:// doi. org/ 10. 3390/ ijgi1 on the Petroglyphs of Cheonjeon-ri, Ulsan, Republic of Korea. 00603 57 Environ Earth Sci 63(7–8):1741–1761. https:// doi. org/ 10. 1007/ 24. Kilic G (2015) Using advanced NDT for historic buildings: s12665- 010- 0828-4 Towards an integrated multidisciplinary health assessment strat- 10. Delgado Rodrigues J (2015) Defining, mapping and assessing egy. J Cult Herit 16(4):526–535. https://doi. or g/10. 1016/j. culher . deterioration patterns in stone conservation projects. J Cult Herit 2014. 09. 010 16:267–275. https:// doi. org/ 10. 1016/j. culher. 2014. 06. 007 25. Silveira da Costa V, Montagna da Silveira A, da Silva TA (2021) 11. Franković M, Novaković N, Matović V (2015) Damage quan- Evaluation of degradation state of historic building facades tification of built stone on Dark Gate (Belgrade, Serbia): sam- through qualitative and quantitative indicators: case study in Pelo- ple of damage index application for decay rate evaluation. tas Brazil. Int J Archit Herit. https://doi. org/ 10. 1080/ 15583 058. Environ Earth Sci 73(10):6181–6193. https:// doi. org/ 10. 1007/ 2021. 19011 61 s12665- 014- 3843-z 26. Lezzerini M, Antonelli F, Columbu S, Gadducci R, Marradi A, 12. Gizzi FT, Sileo M, Biscione M, Danese M, Alvarez de Buergo M Miriello D, Parodi L, Secchiari L, Lazzeri A (2016) Cultural her- (2016) The conservation state of the Sassi of Matera site (South- itage documentation and conservation: three-dimensional (3D) ern Italy) and its correlation with the environmental conditions laser scanning and geographical information system (GIS) tech- analysed through spatial analysis techniques. J Cult Herit 17:61– niques for thematic mapping of facade stonework of St. Nicholas 74. https:// doi. org/ 10. 1016/j. culher. 2015. 05. 002 Church (Pisa, Italy). Int J Archit Herit 10(1):9–19. https://doi. or g/ 13. Germinario C, Gorrasi M, Izzo F, Langella A, Limongiello M, 10. 1080/ 15583 058. 2014. 924605 Mercurio M, Musmeci D, Santoriello A, Grifa C (2020) Damage 27. Gutiérrez-Carrillo ML, Bestué Cardiel I, Molero Melgarejo diagnosis of Ponte Rotto, a Roman bridge along the ancient Appia. E, Marcos Cobaleda M (2020) Pathologic and risk analysis of Int J Conserv Sci 11(1):277–290 the Lojuela Castle (Granada-Spain): methodology and preven- 14. Randazzo L, Collina M, Ricca M, Barbieri L, Bruno F, Arcudi tive conservation for medieval earthen fortifications. Appl Sci A, La Russa MF (2020) Damage indices and photogrammetry for 10(18):6491. https:// doi. org/ 10. 3390/ app10 186491 decay assessment of stone-built cultural heritage: the case study of 28. Thornbush MJ, Viles HA (2007) Photo-based decay mapping of the San Domenico church main entrance Portal (South Calabria, replaced stone blocks on the boundary wall of Worcester college. Italy). Sustainability 12(12):5198. https:// doi. org/ 10. 3390/ su121 Oxford Geol Soc Spec Publ 271:69–75. https:// doi. org/ 10. 1144/ GSL. SP. 2007. 271. 01. 08 15. Puy-Alquiza MJ, Ordaz Zubia VY, Aviles RM, MaDC S-H (2021) 29. Russo M, Carnevali L, Russo V, Savastano D, Taddia Y (2019) Damage detection historical building using mapping method in Modeling and deterioration mapping of façades in historical urban music school of the University of Guanajuato. Mexico Mech Adv context by close-range ultra-lightweight UAVs photogrammetry. Mater Struct 28(10):1049–1060. https:// doi. org/ 10. 1080/ 15376 Int J Archit Herit 13(4):549–568. https://doi. org/ 10. 1080/ 15583 494. 2019. 16290 49 058. 2018. 14400 30 1 3 41 Page 14 of 15 Journal of Building Pathology and Rehabilitation (2021) 6:41 30. Jo YH, Lee CH (2014) Quantitative modeling and mapping of a tool for mapping and evaluation of stone decay. Constr Build blistering zone of the Magoksa Temple stone pagoda (13th cen- Mater 25(4):1603–1611. https:// doi. org/ 10. 1016/j. conbu ildmat. tury, Republic of Korea) by graduated heating thermography. 2010. 10.003 Infrared Phys Technol 65:43–50. https:// doi. org/ 10. 1016/j. infra 48. Adamopoulos E, Rinaudo F (2021) Documenting the state of pres- red. 2014. 02. 011 ervation of historical stone sculptures in three dimensions with 31. Napolitano R, Hess M, Glisic B (2020) Quantifying the differ - digital tools. In: Del Bimbo A, Cucchiara R, Sclaroff S, Farinella ences in documentation and modeling levels for building pathol- GM, Mei T, Bertini M, Escalante HJ, Vezzani R (eds) Pattern rec- ogy and diagnostics. Arch Computat Methods Eng 27(4):1135– ognition. ICPR international workshops and challenges. Springer 1152. https:// doi. org/ 10. 1007/ s11831-019- 09350-y International Publishing, Cham, pp 666–673 32. Maierhofer C, Röllig M, Krankenhagen R (2010) Integration 49. Hatir ME, Barstuğan M, İnce İ (2020) Deep learning-based weath- of active thermography into the assessment of cultural heritage ering type recognition in historical stone monuments. J Cult Herit buildings. J Mod Opt 57(18):1790–1802. https://doi. or g/10. 1080/ 45:193–203. https:// doi. org/ 10. 1016/j. culher. 2020. 04. 008 09500 34100 37034 97 50. Mansuri LE, Patel DA (2021) Artificial intelligence-based auto- 33. Moropoulou A, Labropoulos KC, Delegou ET, Karoglou M, matic visual inspection system for built heritage. SASBE. https:// Bakolas A (2013) Non-destructive techniques as a tool for the doi. org/ 10. 1108/ SASBE- 09- 2020- 0139 protection of built cultural heritage. Constr Build Mater 48:1222– 51. Hatır ME, İnce İ, Korkanç M (2021) Intelligent detection of dete- 1239. https:// doi. org/ 10. 1016/j. conbu ildmat. 2013. 03. 044 rioration in cultural stone heritage. J Build Eng 44:102690. https:// 34. Brooke C (2018) Thermal Imaging for the archaeological investi- doi. org/ 10. 1016/j. jobe. 2021. 102690 gation of historic buildings. Remote Sens 10(9):1401. https:// doi. 52. Pathak R, Saini A, Wadhwa A, Sharma H, Sangwan D (2021) An org/ 10. 3390/ rs100 91401 object detection approach for detecting damages in heritage sites 35. Lerma JL (2001) Multiband versus multispectral supervised clas- using 3-D point clouds and 2-D visual data. J Cult Herit 48:74–82. sification of architectural images. Photogramm Rec 17(97):89– https:// doi. org/ 10. 1016/j. culher. 2021. 01. 002 101. https:// doi. org/ 10. 1111/ 0031- 868X. 00169 53. Meroño JE, Perea AJ, Aguilera MJ, Laguna AM (2015) Recogni- 36. Lerma JL (2005) Automatic plotting of architectural facades with tion of materials and damage on historical buildings using digital multispectral images. J Surv Eng 131(3):73–77. https:// doi. org/ image classification. S Afr J Sci 111(1/2):1– 9. https://doi. or g/10. 10. 1061/ (ASCE) 0733- 9453(2005) 131: 3(73) 17159/ sajs. 2015/ 20140 001 37. Lerma JL, Cabrelles M, Akasheh TS, Haddad NA (2012) Docu- 54. Sánchez-Aparicio LJ, Del Pozo S, Ramos LF, Arce A, Fernandes mentation of weathered architectural heritage with visible, near FM (2018) Heritage site preservation with combined radiometric infrared, thermal and laser scanning data. Int J Herit Digit Era and geometric analysis of TLS data. Autom Constr 85:24–39. 1(2):251–275. https:// doi. org/ 10. 1260/ 2047- 4970.1. 2. 251 https:// doi. org/ 10. 1016/j. autcon. 2017. 09. 023 38. Armesto-González J, Riveiro-Rodríguez B, González-Aguilera D, 55. Grilli E, Remondino F (2019) Classic fi ation of 3D digital heritage. Rivas-Brea MT (2010) Terrestrial laser scanning intensity data Remote Sens 11(7):847. https:// doi. org/ 10. 3390/ rs110 70847 applied to damage detection for historical buildings. J Archaeol 56. Adamopoulos E, Rinaudo F (2021) Combining multiband imag- Sci 37(12):3037–3047. https:// doi. org/10. 1016/j. jas. 2010. 06. 031 ing, photogrammetric techniques, and FOSS GIS for affordable 39. Del Pozo S, Herrero-Pascual J, Felipe-García B, Hernández-López degradation mapping of stone monuments. Buildings 11(7):304. D, Rodríguez-Gonzálvez P, González-Aguilera D (2016) Multi- https:// doi. org/ 10. 3390/ build ings1 10703 04 spectral radiometric analysis of façades to detect pathologies from 57. Sánchez-Aparicio LJ, Del Pozo S, Rodriguez-Gonzalvez P, active and passive remote sensing. Remote Sens 8(1):80. https:// Herrero-Pascual J, Muñoz-Nieto A, Gonzalez-Aguilera D (2016) doi. org/ 10. 3390/ rs801 0080 Practical use of multispectral techniques for the detection of 40. Nocerino E, Dubbini M, Menna F, Remondino F, Gattelli M, pathologies in constructions. In: Riveiro B, Solla M (eds) Non- Covi D (2017) Geometric calibration and radiometric correc- destructive techniques for the evaluation of structures and infra- tion of the MAIA multispectral camera. Int Arch Photogramm structure, 1st edn. CRC Press, London, pp 253–271 Remote Sens Spat Inf Sci. https:// doi. org/ 10. 5194/ isprs- archi 58. Pozo-Antonio JS, Puente I, Pereira MFC, Rocha CSA (2019) ves- XLII-3- W3- 149- 2017 Quantification and mapping of deterioration patterns on granite 41. Verhoeven G (2008) Imaging the invisible using modified digital surfaces by means of mobile LiDAR data. Measurement 140:227– still cameras for straightforward and low-cost archaeological near- 236. https:// doi. org/ 10. 1016/j. measu rement. 2019. 03. 066 infrared photography. J Archaeol Sci 35(12):3087–3100. https:// 59. Kaynig V, Fischer B, Müller E, Buhmann JM (2010) Fully auto- doi. org/ 10. 1016/j. jas. 2008. 06. 012 matic stitching and distortion correction of transmission electron 42. Falco CM (2009) Invited Article: High resolution digital camera microscope images. J Struct Biol 171(2):163–173. https://doi. or g/ for infrared reflectography. Rev Sci Instrum 80(7):071301. https:// 10. 1016/j. jsb. 2010. 04. 012 doi. org/ 10. 1063/1.31744 31 60. Rueden CT, Schindelin J, Hiner MC, DeZonia BE, Walter AE, 43. Webb EK, Robson S, MacDonald L, Garside D, Evans R (2018) Arena ET, Eliceiri KW (2017) Image J2: ImageJ for the next Spectral and 3D cultural heritage documentation using a modi- generation of scientific image data. BMC Bioinform 18(1):529. fied camera. Int Arch Photogramm Remote Sens Spatial Inf Sci. https:// doi. org/ 10. 1186/ s12859- 017- 1934-z https:// doi. org/ 10. 5194/ isprs- archi ves- XLII-2- 1183- 2018 61. Shepard D (1968) A two-dimensional interpolation function 44. Dias IS, Flores-Colen I, Silva A (2021) Critical analysis about for irregularly-spaced data. Proceedings of the 1968 23rd ACM emerging technologies for building’s Façade inspection. Buildings national conference. ACM Press, London, pp 517–524. https:// 11(2):53. https:// doi. org/ 10. 3390/ build ings1 10200 53 doi. org/ 10. 1145/ 800186.810616 45. Marzouk M, Zaher M (2020) Artificial intelligence exploita- 62. HyperCube: Spectral Imagery Analysis Software (2021). https:// tion in facility management using deep learning. Constr Innov www. erdc. usace. army. mil/ Media/ Fact- Sheets/ Fact- Sheet- Artic 20(4):609–624. https:// doi. org/ 10. 1108/ CI- 12- 2019- 0138 le- View/ Artic le/ 610433/ hyper cube. Accessed 21 June 2021 46. Cossu R, Chiappini L (2004) A color image segmentation method 63. Arganda-Carreras I, Kaynig V, Rueden C, Eliceiri KW, Schindelin as used in the study of ancient monument decay. J Cult Herit J, Cardona A, Sebastian Seung H (2017) Trainable Weka Segmen- 5(4):385–391. https:// doi. org/ 10. 1016/j. culher. 2004. 03. 005 tation: a machine learning tool for microscopy pixel classification. 47. Vázquez MA, Galán E, Guerrero MA, Ortiz P (2011) Digital Bioinformatics 33(15):2424–2426. https:// doi. org/ 10. 1093/ bioin image processing of weathered stone caused by efflorescences: forma tics/ btx180 1 3 Journal of Building Pathology and Rehabilitation (2021) 6:41 Page 15 of 15 41 64. Quinlan JR (1993) C4.5: programs for machine learning, 1st edn. 69. Congalton RG, Green K (2019) Assessing the accuracy of Morgan Kaufmann, San Mateo remotely sensed data: principles and practices, 3rd edn. Taylor & 65. Breiman L (2001) Random forests. Mach Learn 45:5–23 Francis Group, Boca Raton 66. Quinlan JR (1996) Learning decision tree classifiers. ACM Com- 70. Das R (2020) Content-based image classification: efficient put Surv 28(1):71–72 machine learning using robust feature extraction techniques, 1st 67. Zhao Y, Zhang Y (2008) Comparison of decision tree methods for edn. Chapman and Hall/CRC, Boca Raton finding active objects. Adv Space Res 41(12):1955– 1959. https:// 71. ICOMOS-ISCS (2008) Illustrated glossary on stone deterioration doi. org/ 10. 1016/j. asr. 2007. 07. 020 patterns, 1st edn. Ateliers 30 Impression, Champigny-sur-Marne 68. Friedman J, Hastie T, Tibshirani R (2000) Additive Logis- tic Regression: a Statistical View of Boosting. Ann Stat Publisher's Note Springer Nature remains neutral with regard to 28(2):337–374 jurisdictional claims in published maps and institutional affiliations. 1 3

Journal

Journal of Building Pathology and Rehabilitation – Springer Journals

Published: Dec 1, 2021

Keywords: Built heritage; Deterioration mapping; Multispectral reflectance imaging; Thermal infrared imaging; Supervised image segmentation; Machine learning

There are no references for this article.