Intelligent Thermal Imaging-Based Diagnostics of Turbojet Engines

Rudolf Andoga; Ladislav Főző; Martin Schrötter; Marek Češkovič; Stanislav Szabo; Róbert Bréda; Michal Schreiner

doi:10.3390/app9112253

Intelligent Thermal Imaging-Based Diagnostics of Turbojet Engines

Andoga, Rudolf;Főző, Ladislav;Schrötter, Martin;Češkovič, Marek;Szabo, Stanislav;Bréda, Róbert;Schreiner, Michal 2019-05-31 00:00:00 applied sciences Article Intelligent Thermal Imaging-Based Diagnostics of Turbojet Engines Rudolf Andoga * , Ladislav Foz ˝ o ˝ , Martin Schrötter, Marek Ceškovic, ˇ Stanislav Szabo, Róbert Bréda and Michal Schreiner Technical University of Košice, Faculty of Aeronautics, Rampová 7, 041 21 Košice, Slovakia; ladislav.fozo@tuke.sk (L.F.); martin.schrotter@tuke.sk (M.S.); marek.ceskovic@tuke.sk (M.C.); stanislav.szabo@tuke.sk (S.S.); robert.breda@tuke.sk (R.B.); michal.schreiner@tuke.sk (M.S.) * Correspondence: rudolf.andoga@tuke.sk; Tel.: +421-55-602-6140 Received: 2 April 2019; Accepted: 27 May 2019; Published: 31 May 2019 Featured Application: The methodology presented in this paper is directly intended for application in the area of turbojet engines. This paper presents the design of an automated system which can create diagnostic information using data from an infrared thermal camera, utilizing methods of computational intelligence and expert systems, which can be considered as a novel approach. The developed system segments infrared images use a self-organizing feature map and an expert system to automatically extract diagnostic information about the technical state of a turbojet engine. The diagnostic system was developed using a small iSTC-21v turbojet engine with a thrust of 500 N in laboratory conditions and can be simply modiﬁed for application to other engine types. The system is not limited to engines and can be considered for application in other technical systems which produce heat, such as generators, transmissions, and electronic circuit boards. Abstract: There are only a few applications of infrared thermal imaging in aviation. In the area of turbojet engines, infrared imaging has been used to detect temperature ﬁeld anomalies in order to identify structural defects in the materials of engine casings or other engine parts. In aviation applications, the evaluation of infrared images is usually performed manually by an expert. This paper deals with the design of an automatic intelligent system which evaluates the technical state and diagnoses a turbojet engine during its operation based on infrared thermal (IRT) images. A hybrid system interconnecting a self-organizing feature map and an expert system is designed for this purpose. A Kohonen neural network (the self-organizing feature map) is successfully applied to segment IRT images of a turbojet engine with high precision, and the expert system is then used to create diagnostic information from the segmented images. This paper represents a proof of concept of this hybrid system using data from a small iSTC-21v turbojet engine operating in laboratory conditions. Keywords: expert systems; infrared thermography; intelligent diagnostics; image segmentation; neural networks; thermal imaging camera; turbojet engines 1. Introduction Infrared thermography (IRT) is one of the many diagnostic methodologies currently applied in various types of industry and research [1]. It can be used to detect so-called invisible defects in a non-destructive way. Compared to traditional areas of IRT applications, such as biomedical applications [2], energetics, buildings, and other constructions [3–6], the application of this methodology in aviation is still at an early stage [1]. Appl. Sci. 2019, 9, 2253; doi:10.3390/app9112253 www.mdpi.com/journal/applsci Appl. Sci. 2019, 9, 2253 2 of 22 The advantages of IRT have been fully demonstrated in pre-ﬂight and post-ﬂight aircraft checks, where damage (e.g., water in honeycomb structures of composite parts), is monitored by thermal imaging cameras [7,8]. In aviation, extensive research and development is being conducted on the IRT diagnostics of corrosion in aluminum materials [9,10], and especially composite structures [11–13]. In aircraft operation, it is possible to use IRT to detect air leaks inside the cabin [14], abnormal strain in individual parts of aircraft casings, fuselage, and wheel attachments [15], damage to electrical wiring and insulation [16,17], etc. In aviation, IRT is also used for rescue, ﬁreﬁghting, and surveillance operations [18,19]. Other applications of IRT diagnostics can be found in the area of unmanned vehicles, whose thermal imaging cameras are used to inspect power lines [20], pipelines [21], buildings [22], constructions [23], vegetation [24], etc. Thermography has also recently found an application in the design of stealth aircraft [25]. For the diagnostics of aviation engines, IRT is mainly used to check their composite covers [26]. Another area where IRT is applied, which is closer to the topic of this paper, is the diagnostics of turbojet engines; infrared images of the diagnosed engines’ gas turbines are compared to standard IRT images of engines without defects, which allows temperature ﬁeld anomalies to be detected and thus allows defects to be identiﬁed [27]. These defects can be damage to the turbine blades, combustion chamber disturbance, disturbance to the homogeneous ﬁeld of fuel burning, etc. [28]. Aircraft engine diagnostics and health monitoring [29–31] can be divided into two basic processes: ground diagnostics and on-board diagnostics with their associated systems. On-board, or online, diagnostics involve the checking of engine components and thermodynamic processes during the operation of the engine [32]. These components include individual engine parts, such as compressors, combustion chambers, and turbines and associated accessories, while thermodynamic process checks include reviewing parameters such as temperature, pressure, rotational turbo-compressor speed, and vibrations. Ground, or oine, diagnostics [33,34] involve tribo-technical analysis, non-destructive checking methods, and checks by thermal cameras as mentioned earlier. This article aims to develop a progressive method for the online evaluation of the technical state of a turbojet engine using infrared thermal imaging. Contrary to traditional methods, in which a comparative analysis of thermal images is performed on some normalized samples, this paper proposes an intelligent adaptive method which can discriminate anomalous areas of thermal images and create resulting diagnostic information automatically without any human input or previous training. Since the output of a thermal imaging camera is a digital image, computer vision techniques and methodologies should be used to automatically process this output. In order to extract meaningful information from a thermal image for a diagnostic system, a pattern recognition/classiﬁcation approach has to be applied. Many methodologies and an array of methods can be used for this purpose [35,36]. These include traditional methodologies, which perform pattern recognition using deterministic image-processing methods such as ﬁltering, feature analysis, and support vector machines. [37]. However, the most successful methodologies for image classiﬁcation come from the area of computational intelligence. These include artiﬁcial neural networks, which have been widely and successfully applied in image processing, pattern recognition, and diagnostic tasks [38,39]. The most complex neural networks, such as convolutional neural networks, are able to classify any pattern in an image with high precision, as well as detect objects [40]. However, these approaches are computationally complex and require large amounts of data and training [41]. In order to create a successful diagnostic method for pattern recognition in IRT images, a simpler methodology should be used, since such images are more homogeneous than traditional images; they have fewer colors, lower resolution, and are well structured, with well-deﬁned edges. The motivation for this study was to develop a methodology which does not require a large amount of computationally intensive training to successfully segment infrared images using intelligent adaptive algorithms. Accordingly, in this paper, the ﬁeld of neural networks with uncontrolled learning is explored. One of the methods that has been successfully used in pattern recognition and image segmentation is self-organizing feature maps (SOFMs), which comply with the demand for simplicity, uncontrolled Appl. Sci. 2019, 9, 2253 3 of 22 learning, and good classiﬁcation results. SOFMs can be transformed and formalized into the form of a Kohonen neural network [42,43]. Kohonen neural networks have been successfully applied in image segmentation tasks, the detection of speciﬁc areas in geographic maps [44–46], and the detection of cracks and corrosion in materials [47]. The results obtained from these applications represent a solid foundation and an inspiration for this approach to be applied in the segmentation of infrared images for the detection of anomalies in the temperature ﬁelds of dierent objects. The results of pattern recognition and image segmentation come in the form of classes present in the image. In order to extract knowledge from this information, the classes have to be interpreted. This can be done by simply labeling the classes; however, in the ﬁeld of technical systems, this may not be sucient. In technical objects, some areas are naturally hot and some are naturally cold, and therefore there needs to be some inference system which interprets the displacement and size of the classes from thermal images, and generates diagnostic information, which in the simplest case can be the determination of whether the diagnosed object is in a normal or an abnormal operational state [48,49]. The traditional approach is to consider a diagnosed system as healthy if it operates within some speciﬁc limits; otherwise, a failure is indicated [48,49]. This is a natural approach, and can be used when dealing with a technical system where parameters which determine the technical state (temperature, vibrations, etc.) can be simply measured, such as a jet engine. However, to extract diagnostic information from a classiﬁed image, a more complex approach needs to be used [49,50]. Diagnostic expert systems are a natural candidate for extracting diagnostic information from complex data, and have been applied in the diagnostics of turbojet engines [49,50]. The aim of the present study is to create a specialized approach for the creation of a rule-base of an expert system which can transform class information in an infrared image into useful diagnostic information, thus creating hybrid architecture. The functionality of this approach has been evaluated and designed speciﬁcally for application to turbojet engines, and the approach can be considered as novel compared to other intelligent approaches in turbojet engine diagnostics, which are applied to data measured by the engines’ sensors and do not deal with visual/infrared image information [51–53]. The aim of this paper is to explore the design of this new approach and prove its functionality by applying thermal vision-based diagnostics in an automated adaptive system in the area of turbojet engine diagnostics. The approach can be expanded and applied to almost any technical system which has been observed by an infrared camera. The hybrid architecture of our approach, which combines a Kohonen neural network and a specialized expert system with a designed rule-base, can be considered as a novel and original contribution of our research. A proof of concept is presented using real data from an iSTC-21v turbojet engine tested and operated in laboratory conditions. 2. Methodology 2.1. Using Infrared Imaging for Diagnostics Every object whose temperature is above absolute zero radiates thermal energy. IRT systems measure this energy to create thermal infrared images. These images can be used to identify heat radiation anomalies of an object, which are usually characterized by increases or decreases in the radiated temperature [8,9,13]. The basic principle of IRT diagnostics is shown in Figure 1, with the following general meaning: Image: an object to be captured by a thermal camera. Classiﬁcation: monitoring the object, scanning it, obtaining infrared data, classifying hot and cold parts as well as possible problematic areas. Diagnostic information: evaluating infrared thermal images, for example, identifying defects or deﬁciencies, deciding about the technical state of the engine. Appl. Sci. 2019, 9, x FOR PEER REVIEW 4 of 23 Appl. Sci. 2019, 9, 2253 4 of 22 Figure 1. Procedure for the detection of temperature anomalies in engine casings by analyzing an Figure 1. Procedure for the detection of temperature anomalies in engine casings by analyzing an infrared image. infrared image. The aforementioned principle is a standard principle when using human expertise to obtain The aforementioned principle is a standard principle when using human expertise to obtain diagnostic information from infrared images. Examples of applications in aviation are pre-ﬂight and diagnostic information from infrared images. Examples of applications in aviation are pre-flight and post-ﬂight engine checks, where the engine’s casing is checked for defects and cracks caused by freezing post-flight engine checks, where the engine’s casing is checked for defects and cracks caused by water or material deﬁciencies. In such cases, the evaluation of thermal images and the detection of freezing water or material deficiencies. In such cases, the evaluation of thermal images and the anomalies is strictly dependent on human expert knowledge [49]. detection of anomalies is strictly dependent on human expert knowledge [49]. When a human expert inspects infrared images of a technical object, he or she basically performs When a human expert inspects infrared images of a technical object, he or she basically two operations. First, he/she discriminates dierent areas of the thermal image and classiﬁes them performs two operations. First, he/she discriminates different areas of the thermal image and into categories according to their temperature. They next use their expert knowledge to inspect classifies them into categories according to their temperature. They next use their expert knowledge hot or cold areas for some anomalies according to their position and temperature. The traditional to inspect hot or cold areas for some anomalies according to their position and temperature. The approach is to superimpose the obtained infrared image over a standardized infrared image of an traditional approach is to superimpose the obtained infrared image over a standardized infrared object without defects and evaluate the dierences between the two images [54]. The whole process image of an object without defects and evaluate the differences between the two images [54]. The can therefore be formalized into two basic parts—image processing and classiﬁcation, and knowledge whole process can therefore be formalized into two basic parts—image processing and classification, acquisition. A variety of methods exist to solve both problems, as mentioned in the introduction, and knowledge acquisition. A variety of methods exist to solve both problems, as mentioned in the and are suitable to identify and evaluate anomalies. As previously stated, one aim of this study was introduction, and are suitable to identify and evaluate anomalies. As previously stated, one aim of to develop an autonomous diagnostic system which is able to learn and locate anomalies without this study was to develop an autonomous diagnostic system which is able to learn and locate any prior knowledge (e.g., normal faultless operation IRT images). The other aim was to make the anomalies without any prior knowledge (e.g., normal faultless operation IRT images). The other aim system autonomous. was to make the system autonomous. The general concept of the intelligent diagnostic system can be expressed by the scheme shown in The general concept of the intelligent diagnostic system can be expressed by the scheme shown Figure 2. The data acquisition part of the approach is usually done using the infrared camera, and in Figure 2. The data acquisition part of the approach is usually done using the infrared camera, and the image pre-processing can be done before the data are fed into a classiﬁer. The eects of the image the image pre-processing can be done before the data are fed into a classifier. The effects of the image pre-processing and ﬁltering are not directly investigated in this study; however, the methodology is pre-processing and filtering are not directly investigated in this study; however, the methodology is designed to be relatively robust in order to use image data of reasonable quality with low noise levels. designed to be relatively robust in order to use image data of reasonable quality with low noise Infrared cameras usually pre-process the image data in order to compensate for air humidity and levels. Infrared cameras usually pre-process the image data in order to compensate for air humidity other environmental eects. The robustness is achieved by the application of an intelligent adaptive and other environmental effects. The robustness is achieved by the application of an intelligent algorithm, and the design therefore consists mainly of the classiﬁer part by ﬁnding a methodology and adaptive algorithm, and the design therefore consists mainly of the classifier part by finding a selecting its structure, input/output relations, and the training algorithm in the case of the application methodology and selecting its structure, input/output relations, and the training algorithm in the of neural networks. Information about classes in the thermal image can be further used to evaluate the case of the application of neural networks. Information about classes in the thermal image can be technical state of the investigated object (in this case, a jet engine), and a diagnostic expert system is a further used to evaluate the technical state of the investigated object (in this case, a jet engine), and a natural candidate for the completion of this task. The basis of the proposed methodology is therefore diagnostic expert system is a natural candidate for the completion of this task. The basis of the centered around: proposed methodology is therefore centered around: Image processing classiﬁcation system: involving a SOFM. • Image processing classification system: involving a SOFM. Knowledge acquisition system: an expert system for the evaluation of the technical state of • Knowledge acquisition system: an expert system for the evaluation of the technical state of an an engine. engine. Appl. Sci. 2019, 9, x FOR PEER REVIEW 5 of 23 The other advantage of the designed system is that it does not use the absolute temperature data obtained from the camera, which are dependent on many environmental and material factors, Appl. Sci. 2019, 9, x FOR PEER REVIEW 5 of 23 but rather only uses image data, which describe relative temperatures and relations as well as the Appl. Sci. 2019, 9, 2253 5 of 22 displacement of the temperature field of the investigated object. The other advantage of the designed system is that it does not use the absolute temperature data obtained from the camera, which are dependent on many environmental and material factors, but rather only uses image data, which describe relative temperatures and relations as well as the displacement of the temperature field of the investigated object. Figure 2. The conceptual design of the diagnostic system. Figure 2. The conceptual design of the diagnostic system. The other advantage of the designed system is that it does not use the absolute temperature 2.2. Self-Organizing Feature Maps in Image Segmentation and Classification data obtained from the camera, which are dependent on many environmental and material factors, SOFMs are characterized by their outstanding properties in clustering (classification), data but rather only uses image Figure 2. data, which The conceptu describe al de rsi elative gn of the d temperatur iagnostices syst and em.r elations as well as the visualization, probability density estimation, etc. Kohonen artificial neural networks are one of the displacement of the temperature ﬁeld of the investigated object. most widely used SOFMs [42]. 2.2. Self-Organizing Feature Maps in Image Segmentation and Classification 2.2. Self-Organizing Feature Maps in Image Segmentation and Classiﬁcation Kohonen artificial neural networks (KNN) map a high-dimensional input space/layer (xi for i = SOFMs are characterized by their outstanding properties in clustering (classification), data 1, 2, ..., n, where n is the number of inputs) onto an output space/layer (Nj = 1, 2, ..., m, where m is the SOFMs are characterized by their outstanding properties in clustering (classiﬁcation), data visualization, probability density estimation, etc. Kohonen artificial neural networks are one of the number of neurons) with fewer or an equal number of dimensions. KNN contains only a single layer visualization, probability density estimation, etc. Kohonen artiﬁcial neural networks are one of the most widely used SOFMs [42]. of neurons—the Kohonen layer. The number of neurons n, which is the network’s parameter, is most widely used SOFMs [42]. Kohonen artificial neural networks (KNN) map a high-dimensional input space/layer (xi for i = optional and usually equals the number of target classes. The inputs to the network are Kohonen artiﬁcial neural networks (KNN) map a high-dimensional input space/layer (x for i = 1, 1, 2, ..., n, where n is the number of inputs) onto an output space/layer (Nj = 1, 2, ..., m, where m is the interconnected with all neurons in the Kohonen layer. This means that every neuron has information 2, ..., n, where n is the number of inputs) onto an output space/layer (N = 1, 2, ..., m, where m is the number of neurons) with fewer or an equal number of dimensions. KNN contain j s only a single layer from all the inputs. Synaptic weights leading to each neuron (designated as a weight vector, wij, of number of neurons) with fewer or an equal number of dimensions. KNN contains only a single layer of of neurons—the Kohonen layer. The number of neurons n, which is the network’s parameter, is the neuron) can be understood as coordinates, which indicate the location of the neuron in the array neurons—the Kohonen layer. The number of neurons n, which is the network’s parameter, is optional optional and usually equals the number of target classes. The inputs to the network are (Figure 3). Neurons are mostly organized in a regular two-dimensional grid/matrix or and usually equals the number of target classes. The inputs to the network are interconnected with interconnected with all neurons in the Kohonen layer. This means that every neuron has information one-dimensional row/vector array. all neurons in the Kohonen layer. This means that every neuron has information from all the inputs. from all the inputs. Synaptic weights leading to each neuron (designated as a weight vector, wij, of Synaptic weights leading to each neuron (designated as a weight vector, w , of the neuron) can be ij the neuron) can be understood as coordinates, which indicate the location of the neuron in the array understood as coordinates, which indicate the location of the neuron in the array (Figure 3). Neurons (Figure 3). Neurons are mostly organized in a regular two-dimensional grid/matrix or are mostly organized in a regular two-dimensional grid/matrix or one-dimensional row/vector array. one-dimensional row/vector array. Figure 3. The structure of a Kohonen network. Figure 3. The structure of a Kohonen network. Figure 3. The structure of a Kohonen network. Appl. Sci. 2019, 9, 2253 6 of 22 Appl. Sci. 2019, 9, x FOR PEER REVIEW 6 of 23 There are lateral links (local connections) between the neurons in the Kohonen layer (dashed line There are lateral links (local connections) between the neurons in the Kohonen layer (dashed in Figure 3), which means that any particular neuron aects other neurons in its neighborhood. line in Figure 3), which means that any particular neuron affects other neurons in its neighborhood. In order to segment an IRT image, a layer that has 1 m neurons (Figure 4) is proposed. Using In order to segment an IRT image, a layer that has 1×𝑚 neurons (Figure 4) is proposed. Using this type of array, it is possible to segment thermal infrared images into m dierent segments, which this type of array, it is possible to segment thermal infrared images into m different segments, which can be labeled as dierent areas of the temperature ﬁeld (cold, hot, etc.). Thus, if a higher number of can be labeled as different areas of the temperature field (cold, hot, etc.). Thus, if a higher number of neurons is used, the classiﬁcation will be more detailed. neurons is used, the classification will be more detailed. Figure 4. A single-row structure of the Kohonen layer. Figure 4. A single-row structure of the Kohonen layer. In the most simpliﬁed way, the self-organization process (Kohonen learning) consists of four main In the most simplified way, the self-organization process (Kohonen learning) consists of four steps: initialization, competition, cooperation, and adaptation [55]: main steps: initialization, competition, cooperation, and adaptation [55]: Initialization: Random values are set for all neuron weights. The input vector from the input • Initialization: Random values are set for all neuron weights. The input vector from the input space is selected. space is selected. Competition: From the inputs, the discriminant function (Equation (1), below) of a particular • Competition: From the inputs, the discriminant function (Equation (1), below) of a particular neuron is computed. A Kohonen unit computes the Euclidean distance between an input neuron is computed. A Kohonen unit computes the Euclidean distance between an input xi and x and its weight vector w . The winning neuron is the one with the smallest value of the i ij its weight vector wij. The winning neuron is the one with the smallest value of the discriminant discriminant function. function. Cooperation: The winning neuron inﬂuences its adjacent neurons (i.e., the weight vectors of • Cooperation: The winning neuron influences its adjacent neurons (i.e., the weight vectors of adjacent neurons are inﬂuenced), however this inﬂuence decreases with increasing distance to adjacent neurons are influenced), however this influence decreases with increasing distance to other neurons. other neurons. • Adaptation: Adaptation: The The excited n excited neur eurons d ons decr ecre ease ase their values their values of of the di the discriminant scriminant ffunction unction iin n rel relation ation to to the i then input put pa pattern ttern through througha adjustment djustment of of the the associated associated weights. weights. The The response response of of the w the winning inning neur neuron to th on to the subsequent e subsequent application application of aof similar a siminput ilar inpu pattern t pattern i is enhanced. s enhanced. The pr The ocess process wi will stop ll ifstop if the maximum number the maximum number of iterations of iterat ision reached. s is reached. m 2 𝑑 = 𝑥 −𝑤 (1) d = x w (1) j i i j i=1 The The sel self-or f-o ganization rganization proc process ess isis graphically graphically d depicted epicted in in Figur Figu ere 5,5wher , where b e blue lue dots dots repr repre esent sent neur neurons, green dots rep ons, green dots representresent input input vectors, vectors, and red an lines d red lines represent c represent connections between onnections b neighboring etween neur neighborin ons. Theg temperatur neurons. The temperature e ﬁeld has ﬁve arfield h eas, which as five are are indicated as, which byare ﬁve indic neur aons. ted bIn y fiv this e ne simplistic urons. In example, this simp the list labeling ic example is performed , the labeby lincoloring g is performed b the neurons y color andithe ng t corr he neurons esponding an training d the corresp data, which onding activates training da the ta neur , whi ons. ch a Incti this vates the neurons. In thi manner, the blue areasafter manner, the bl adaptationue a repr resents ea after ada the coldest ptation rep area in resents the temperatur the coldest e ﬁeld area in the tem and the red perature area the field hottest andar the red ea. area the hottest area. Appl. Appl. Sci. Sci. 2019 2019 , 9 , ,92253 , x FOR PEER REVIEW 77 of of 22 23 Appl. Sci. 2019, 9, x FOR PEER REVIEW 7 of 23 Figure 5. The self-organization process with a single-row structure of the Kohonen layer. Figure Figure 5. 5. The The self-organizatio self-organizationn process with process with a a sing single-r le-row ow str structure of the ucture of the Kohonen Kohonen layer. layer. 2.3. Expert Diagnostic Systems 2.3. Expert Diagnostic Systems 2.3. Expert Diagnostic Systems An expert system consists of a knowledge base, an inference engine (solving or deducing), an An expert system consists of a knowledge base, an inference engine (solving or deducing), an input/output data interface (user interface or links to other systems and tools), a working memory, An expert system consists of a knowledge base, an inference engine (solving or deducing), an input/output data interface (user interface or links to other systems and tools), a working memory, and and an explaining or communication module in the form of a user interface. These elements are input/output data interface (user interface or links to other systems and tools), a working memory, an explaining or communication module in the form of a user interface. These elements are shown in shown in Figure 6. A very important part of every expert system is its inference engine, which uses and an explaining or communication module in the form of a user interface. These elements are Figure 6. A very important part of every expert system is its inference engine, which uses data from data from the knowledge base (knowledge gained from an expert) and specific problem-oriented shown in Figure 6. A very important part of every expert system is its inference engine, which uses the knowledge base (knowledge gained from an expert) and speciﬁc problem-oriented data [56]. data [56]. data from the knowledge base (knowledge gained from an expert) and specific problem-oriented data [56]. Figure 6. The structure of expert systems and their integration (reproduced from [57]). Figure 6. The structure of expert systems and their integration (reproduced from [57]). The inference engine and the knowledge base represent the core of every expert system. The Figure 6. The structure of expert systems and their integration (reproduced from [57]). The inference engine and the knowledge base represent the core of every expert system. The inference engine provides the mathematical means to evaluate the rules in the knowledge base to reach inference engine provides the mathematical means to evaluate the rules in the knowledge base to The inference engine and the knowledge base represent the core of every expert system. The a conclusion about the state of the diagnosed object [58–60]. In applications for turbojet engines, the reach a conclusion about the state of the diagnosed object [58–60]. In applications for turbojet inference engine provides the mathematical means to evaluate the rules in the knowledge base to knowledge base contains rules deﬁning the technical state of the turbojet engine based on the outputs engines, the knowledge base contains rules defining the technical state of the turbojet engine based reach a conclusion about the state of the diagnosed object [58–60]. In applications for turbojet from the segmentation algorithm (i.e., the Kohonen neural network). The design of the diagnostic on the outputs from the segmentation algorithm (i.e., the Kohonen neural network). The design of engines, the knowledge base contains rules defining the technical state of the turbojet engine based system interconnected with a SOFM, which was used in the present study, is shown in Figure 7. the diagnostic system interconnected with a SOFM, which was used in the present study, is shown on the outputs from the segmentation algorithm (i.e., the Kohonen neural network). The design of in Figure 7. the diagnostic system interconnected with a SOFM, which was used in the present study, is shown in Figure 7. Appl. Sci. 2019, 9, 2253 8 of 22 Appl. Sci. 2019, 9, x FOR PEER REVIEW 8 of 23 Appl. Sci. 2019, 9, x FOR PEER REVIEW 8 of 23 Figure 7. The connection between the Kohonen neural network and an expert system. Figure 7. The connection between the Kohonen neural network and an expert system. In this diagnostic system, the knowledge is represented by IF->THEN rules, which can be written In this diagnostic system, the knowledge is represented by IF->THEN rules, which can be Figure 7. The connection between the Kohonen neural network and an expert system. in a simple way as IF E THEN H, as follows: written in a simple way as IF E THEN H, as follows: In this diagnostic system, the knowledge is represented by IF->THEN rules, which can be 𝐸→𝐻 (2) E ! H (2) written in a simple way as IF E THEN H, as follows: where E represents an observation (evidence) and H represents a hypothesis. Rule-based systems differ from classical logic systems in their non-monotonic “thinking” and (2) where E represents an observation (evidence) and𝐸→ H repr 𝐻 esents a hypothesis. their ability to process uncertainties. In rule-based systems, inference is based on the rule of modus Rule-based systems dier from classical logic systems in their non-monotonic “thinking” and where E represents an observation (evidence) and H represents a hypothesis. ponens (direct thinking) [61], which is described as: their ability to process uncertainties. In rule-based systems, inference is based on the rule of modus Rule-based systems differ from classical logic systems in their non-monotonic “thinking” and 𝐸,𝐸 → 𝐻 ponens (direct thinking) [61], which is described as: their ability to process uncertainties. In rule-based systems, inference is based on the rule of(3 modu ) s ponens (direct thinking) [61], which is described as: E, E ! H This means that, if the E assumption is true and the rule implies E→H, then the conclusion H is (3) 𝐸,H𝐸 → 𝐻 true. In the case of an inference network, the conclusions are represented by facts, which correspond (3) to assumptions of other rules. These inference networks are mostly used in domains where the This means that, if the E assumption is true and the rule implies E!H, then the conclusion H is number of possible solutions is limited (e.g., diagnostic problems or classification problems). The This means that, if the E assumption is true and the rule implies E→H, then the conclusion H is true. In the case of an inference network, the conclusions are represented by facts, which correspond benefit of inference networks is that they can be easily implemented. The following model rules true. In the case of an inference network, the conclusions are represented by facts, which correspond to assumptions of other rules. These inference networks are mostly used in domains where the (Equation (4)) [61] can be formalized in an inference network structure as shown in Figure 8 [61]: to assumptions of other rules. These inference networks are mostly used in domains where the number of possible solutions is limited (e.g., diagnostic problems or classiﬁcation problems). The A ∧ B → P, B ∧ D → Q, P → X, Q ∧ R → Y, C → P, C ∧ E → R, Q → X (4) number of possible solutions is limited (e.g., diagnostic problems or classification problems). The beneﬁt of inference networks is that they can be easily implemented. The following model rules benefit of inference networks is that they can be easily implemented. The following model rules (Equation (4)) [61] can be formalized in an inference network structure as shown in Figure 8 [61]: (Equation (4)) [61] can be formalized in an inference network structure as shown in Figure 8 [61]: A^ B! P, B^ D! Q, P! X, Q^ R! Y, C! P, C^ E! R, Q! X (4) A ∧ B → P, B ∧ D → Q, P → X, Q ∧ R → Y, C → P, C ∧ E → R, Q → X (4) Figure 8. An example of an AND/OR inference network (reproduced from [61]). Figure 8. An example of an AND/OR inference network (reproduced from [61]). Figure 8. An example of an AND/OR inference network (reproduced from [61]). Appl. Sci. 2019, 9, 2253 9 of 22 Appl. Sci. 2019, 9, x FOR PEER REVIEW 9 of 23 2.4. Experimental Setup for Design of the Infrared Imaging-Based Diagnostics System 2.4. Experimental Setup for Design of the Infrared Imaging-Based Diagnostics System In order to design an infrared imaging-based diagnostic system for use with turbojet engines and In order to design an infrared imaging-based diagnostic system for use with turbojet engines to construct a working knowledge base, a small iSTC-21v turbojet engine, developed in the Laboratory and to construct a working knowledge base, a small iSTC-21v turbojet engine, developed in the of Intelligent Control Systems of Aircraft Engines at the Faculty of Aeronautics, Technical University of Laboratory of Intelligent Control Systems of Aircraft Engines at the Faculty of Aeronautics, Košice (Figure 9), was used [62]. This is a single-stream, single-shaft turbojet engine with a single-stage Technical University of Košice (Figure 9), was used [62]. This is a single-stream, single-shaft turbojet one-sided radial compressor, an annular combustion chamber, and a single-stage non-cooled gas engine with a single-stage one-sided radial compressor, an annular combustion chamber, and a turbine with a variable exhaust nozzle with full authority intelligent electronic digital control system single-stage non-cooled gas turbine with a variable exhaust nozzle with full authority intelligent developed from the TS-21 engine (Ljulka-Saturn, Rybinsk, Russia). A schematic of the measurement electronic digital control system developed from the TS-21 engine (Ljulka-Saturn, Rybinsk, Russia). and data acquisition system is shown in Figure 10, and the arrangement of the camera is shown in A schematic of the measurement and data acquisition system is shown in Figure 10, and the Figure 11 [62]. arrangement of the camera is shown in Figure 11 [62]. Appl. Sci. Figure 2019, 9. 9, x FO Views R Pof EER the RE iSTC-21v VIEW turbojet engine (left) and its starting process (right) in a laboratory10 of . 23 Figure 9. Views of the iSTC-21v turbojet engine (left) and its starting process (right) in a laboratory. A digital data acquisition system was used to observe the state of the engine [62]. Various sensors and data acquisition (DAQ) devices serve this purpose and send all the data to the control personal computer (PC). This PC contains a control system created in the LabVIEW (National Instruments Corporation, Austin, TX, USA) and Matlab/Simulink (MathWorks, Natick, MA, USA) software which is able to control and visualize the engine’s performance, save and track all measured parameters, and update the control algorithms. For the design of the IRT diagnostics system, the following parameters are utilized: • n (rpm)—speed of the engine’s shaft. • T2 (°C)—air temperature at the output of the radial compressor. • T3 (°C)—gas temperature at the input to the gas turbine. • T4 (°C)—gas temperature at the output of the gas turbine. These parameters represent the inner state and temperature of the engine. The interior temperature was measured by a thermal sensor contained in the engine. The temperature of the engine’s casing was measured by a thermal camera; this temperature can be correlated with the inner temperature although it has different dynamics. Figure 10. Figure 10. Sche Schematic matic of of the me the measur asurement and data a ement and data acquisition cquisition sy system stem u used sed in in this this stu study dy.. To measure the thermal temperature field around the engine, a FLIR Thermovision A40 thermal imaging camera was used (FLIR Systems, Wilsonville, OR, USA), and a stationary IR camera was used for scanning during long measurement processes, where continuous measurement in real time is necessary. The IR detector of the camera is a 320 × 240 focal plane array uncooled microbolometer. Its spectral range is 7.5 to 13 µm, its instantaneous field of view (IFOV) is 1.3 mrad, its noise equivalent temperature difference at 60 Hz is 0.08 °C at 30 °C, its measurement uncertainty is ±2 °C or ± 2%, its temperature sampling frequency is 60 Hz, and its observable temperature range is from – 40 to 1500 °C. The camera was controlled via the PC using the ThermaCAM Researcher software (FLIR Systems, Wilsonville, OR, USA) through the FireWire interface. One of the settings when capturing a video is the choice of color palette, which is used to convert the levels of the infrared spectrum into the visible spectrum. This selection is mainly important when there are small temperature differences between parts of an object or its surroundings. In the case of an object with a wide range of temperatures, such as a turbojet engine, the standard iron-bow palette is a suitable option [63]. As the engine is a dynamic object, the displacement of its thermal field needs to be evaluated continuously. The camera allows uncompressed video to be captured at 25 frames per second in audio video interleave (AVI) format using the selected color palette. The video can then be decomposed into single frames in uncompressed bitmap (BMP) format. The resulting pictures contain IR images of the engine with the assigned RGB pixel values from the selected color palette during its operation. Full color resolution of the camera can be obtained only when taking static images of the engine, which is not an option for the application, but a video with this resolution would definitely improve the resulting IR images. It can be concluded here that parameters of the camera are sufficient for application on an object operating in a wide temperature range, where fine temperature resolution is not that important. Appl. Sci. 2019, 9, 2253 10 of 22 Appl. Sci. 2019, 9, x FOR PEER REVIEW 11 of 23 Figure 11. The arrangement of the FLIR Thermovision A40 thermal imaging camera in the Figure 11. The arrangement of the FLIR Thermovision A40 thermal imaging camera in the laboratory. laboratory. A digital data acquisition system was used to observe the state of the engine [62]. Various sensors The thermal fields were captured and observed during the whole engine operational process, and data acquisition (DAQ) devices serve this purpose and send all the data to the control personal from its start-up to the time of full thrust, as shown by the measured basic parameters displayed in computer (PC). This PC contains a control system created in the LabVIEW (National Instruments Figure 12. The infrared camera was positioned 1.5 m from the engine and was turned on 0.5 h before Corporation, Austin, TX, USA) and Matlab/Simulink (MathWorks, Natick, MA, USA) software which the measurement in order to allow the temperature of its sensor to equalize with that of the is able to control and visualize the engine’s performance, save and track all measured parameters, and environment. Before the measurement, the values of the background air temperature, air humidity, update the control algorithms. and the emissivity of the engine surface were determined and inputted in the camera. The For the design of the IRT diagnostics system, the following parameters are utilized: environment in the laboratory had a relatively stable humidity (around 50%) and temperature n (rpm)—speed of the engine’s shaft. (around 21 °C) during the experiments with no direct sunlight. These parameters can have an impact T ( C)—air temperature at the output of the radial compressor. on the precision of the IRT measurements, but can be quite well-controlled in a laboratory T ( C)—gas temperature at the input to the gas turbine. environment [39]. The measured state parameters (n (rpm), T2 (°C), T3 (°C), T4 (°C)) for a single run of T ( C)—gas temperature at the output of the gas turbine. the iSTC-21v engine in laboratory conditions are shown in Figure 12. These parameters represent the inner state and temperature of the engine. The interior temperature was measured by a thermal sensor contained in the engine. The temperature of the engine’s casing was measured by a thermal camera; this temperature can be correlated with the inner temperature although it has dierent dynamics. To measure the thermal temperature ﬁeld around the engine, a FLIR Thermovision A40 thermal imaging camera was used (FLIR Systems, Wilsonville, OR, USA), and a stationary IR camera was used for scanning during long measurement processes, where continuous measurement in real time is necessary. The IR detector of the camera is a 320 240 focal plane array uncooled microbolometer. Its spectral range is 7.5 to 13 m, its instantaneous ﬁeld of view (IFOV) is 1.3 mrad, its noise equivalent temperature dierence at 60 Hz is 0.08 C at 30 C, its measurement uncertainty is2 C or 2%, its temperature sampling frequency is 60 Hz, and its observable temperature range is from40 to 1500 C. The camera was controlled via the PC using the ThermaCAM Researcher software (FLIR Systems, Wilsonville, OR, USA) through the FireWire interface. One of the settings when capturing a video is the choice of color palette, which is used to convert the levels of the infrared spectrum into the visible spectrum. This selection is mainly important when there are small temperature dierences between parts of an object or its surroundings. In the case of an object with a wide range of temperatures, such as a turbojet engine, the standard iron-bow palette is a suitable option [63]. As the engine is a dynamic object, the displacement of its thermal ﬁeld needs to be evaluated continuously. The camera allows uncompressed video to be captured at 25 frames per second in audio video interleave (AVI) format using the selected color palette. The video can then be decomposed into single frames in uncompressed Figure 12. The measured parameters of the iSTC-21v turbojet engine that are usable for infrared bitmap (BMP) format. The resulting pictures contain IR images of the engine with the assigned RGB thermography-based diagnostics. pixel values from the selected color palette during its operation. Full color resolution of the camera can be obtained only when taking static images of the engine, which is not an option for the application, but a video with this resolution would deﬁnitely improve the resulting IR images. It can be concluded Appl. Sci. 2019, 9, x FOR PEER REVIEW 11 of 23 Appl. Sci. 2019, 9, 2253 11 of 22 Figure 11. The arrangement of the FLIR Thermovision A40 thermal imaging camera in the here that parameters of the camera are sucient for application on an object operating in a wide laboratory. temperature range, where ﬁne temperature resolution is not that important. The thermal ﬁelds were captured and observed during the whole engine operational process, The thermal fields were captured and observed during the whole engine operational process, from its start-up to the time of full thrust, as shown by the measured basic parameters displayed from its start-up to the time of full thrust, as shown by the measured basic parameters displayed in in Figure 12. The infrared camera was positioned 1.5 m from the engine and was turned on 0.5 h Figure 12. The infrared camera was positioned 1.5 m from the engine and was turned on 0.5 h before before the measurement in order to allow the temperature of its sensor to equalize with that of the the measurement in order to allow the temperature of its sensor to equalize with that of the environment. Before the measurement, the values of the background air temperature, air humidity, and environment. Before the measurement, the values of the background air temperature, air humidity, the emissivity of the engine surface were determined and inputted in the camera. The environment in and the emissivity of the engine surface were determined and inputted in the camera. The the laboratory had a relatively stable humidity (around 50%) and temperature (around 21 C) during environment in the laboratory had a relatively stable humidity (around 50%) and temperature the experiments with no direct sunlight. These parameters can have an impact on the precision of the (around 21 °C) during the experiments with no direct sunlight. These parameters can have an impact IRT measurements, but can be quite well-controlled in a laboratory environment [39]. The measured on the precision of the IRT measurements, but can be quite well-controlled in a laboratory state parameters (n (rpm), T ( C), T ( C), T ( C)) for a single run of the iSTC-21v engine in laboratory 2 3 4 environment [39]. The measured state parameters (n (rpm), T2 (°C), T3 (°C), T4 (°C)) for a single run of conditions are shown in Figure 12. the iSTC-21v engine in laboratory conditions are shown in Figure 12. Figure 12. The measured parameters of the iSTC-21v turbojet engine that are usable for infrared Figure 12. The measured parameters of the iSTC-21v turbojet engine that are usable for infrared thermography-based diagnostics. thermography-based diagnostics. 3. Thermal Image Segmentation and Classiﬁcation 3.1. Design and Training of the Kohonen Self-Organizing Feature Map The Kohonen SOFM was used for the segmentation of the IRT images obtained with the thermal camera at a resolution of 320 240 pixels as described in Section 2.4. Each pixel was deﬁned as a ﬁve-dimensional vector P (t) = {R,G,B,x,y}, where R, G, and B represent the values of the color _i spectrum {R,G,B} = <0,255> and x and y represent the coordinates of each pixel where x = <0,320>; y = <0,240>. The engine’s temperature as evaluated and processed by the camera was not taken into account, since it may contain errors created by the camera compensating for environmental conditions and assigning dierent temperatures to a single color, thus confusing the classiﬁer. In order to evaluate the technical state of the engine, the relative temperature-ﬁeld displacement was used as it is holds the most important diagnostic information. The dynamic range of the camera was set in the interval <0,500> C and was ﬁxed. These parameters deﬁne the design of the SOFM. The pilot testing showed that these multi-dimensional vectors should be classiﬁed into ﬁve classes. As proposed in Section 2.2, Appl. Sci. 2019, 9, x FOR PEER REVIEW 12 of 23 3. Thermal Image Segmentation and Classification 3.1. Design and Training of the Kohonen Self-Organizing Feature Map The Kohonen SOFM was used for the segmentation of the IRT images obtained with the thermal camera at a resolution of 320×240 pixels as described in Section 2.4. Each pixel was defined as a five-dimensional vector P_i (t) = {R,G,B,x,y}, where R, G, and B represent the values of the color spectrum {R,G,B} = <0,255> and x and y represent the coordinates of each pixel where x = <0,320>; y = <0,240>. The engine’s temperature as evaluated and processed by the camera was not taken into account, since it may contain errors created by the camera compensating for environmental conditions and assigning different temperatures to a single color, thus confusing the classifier. In order to evaluate the technical state of the engine, the relative temperature-field displacement was used as it is holds the most important diagnostic information. The dynamic range of the camera was set in the interval <0,500> °C and was fixed. These parameters define the design of the SOFM. The pilot testing showed that these multi-dimensional vectors should be classified into five classes. As proposed in Section 2.2, a feature map with a single row is able to acceptably segment the IRT image, with low demands of computational time and complexity, which was one of the design goals. The classes are labeled as Ni and represented by different colors as follows: • N1: the hottest and most critical part of the IRT image, red. • N2: a hot and critical part of the IRT image, yellow. • N3: a hot and non-critical part of the IRT image, green. • N4: a low-temperature and non-critical part, cyan. Appl. Sci. 2019, 9, 2253 12 of 22 • N5: the temperature of the surrounding objects (not a part of the diagnosed object), blue. The structure of the network can be visualized as shown in Figure 13. The training parameters a feature map with a single row is able to acceptably segment the IRT image, with low demands of which were found to be efficient were set as defined in Table 1, and the standard Kohonen network computational time and complexity, which was one of the design goals. learning rule was used [42,43]. The classes are labeled as N and represented by dierent colors as follows: Table 1. Training parameters. N : the hottest and most critical part of the IRT image, red. N : a hot and critical part of the IRT image, yellow. Grid Topology function N : a hot and non-critical part of the IRT image, green. Distance computation Direct N : a low-temperature and non-critical part, cyan. Initial neighborhood size 3 N : the temperature of the surrounding objects (not a part of the diagnosed object), blue. The structure of the network can Init be iavisualized l learning rat aseshown in Figur 0.9 e 13. The training parameters which were found to be ecient were set as deﬁned in Table 1, and the standard Kohonen network Training iterations 500 learning rule was used [42,43]. Figure Figure 13 13. . TheTh str e stru ucturctu e ofrethe of the Kohonen Kohon self-or en se ganizing lf-orgfeatur anizing e map featu forre map infrared image for infrared image segmentation. segmentation. Table 1. Training parameters. A training dataset was prepared to train the network as follows. The engine was run for 120 s Topology function Grid and thermal images were recorded at two frames per second, as the pilot experiments showed that Distance computation Direct Initial neighborhood size 3 this timing is sufficient since the dynamics of the temperature of the engine casing have a time Initial learning rate 0.9 constant of 1.2 s. The training sample therefore contained a set of 240 thermal images. The run of the Training iterations 500 A training dataset was prepared to train the network as follows. The engine was run for 120 s and thermal images were recorded at two frames per second, as the pilot experiments showed that this timing is sucient since the dynamics of the temperature of the engine casing have a time constant of 1.2 s. The training sample therefore contained a set of 240 thermal images. The run of the engine, which was used to obtain the IRT images is shown in Figure 14. To demonstrate the ability of the trained Kohonen SOFM, ﬁve operational points are shown in Figure 14, where the network was tested if it produced sensible results. The ﬁrst operational point, O , corresponds to the lowest evaluated p1 rotational speed of the engine 42,058 rpm, and the last, O , corresponds to the highest rotational speed p5 (i.e., 50,289 rpm). The classiﬁcations of the ﬁve example images are shown in Figure 15a, and the created segment areas are shown in Figure 15b. Following expert examination of the infrared images and heuristic evaluation of the temperature ﬁeld, it can be concluded that the Kohonen network works as expected and is able to sensibly assign the hottest and coldest parts of the object to each of the ﬁve classes/segments. In order to quantify the results, the number of activations of each neuron are summed, giving the total segment areas S , : : : , S corresponding to each of the classes N , : : : , N . This can be formalized 1 5 1 5 as follows: S = x ; x = 0; 1 (5) f g i j j=1 where p is the number of activations of the i-th neuron. Appl. Sci. 2019, 9, x FOR PEER REVIEW 13 of 23 engine, which was used to obtain the IRT images is shown in Figure 14. To demonstrate the ability of the trained Kohonen SOFM, five operational points are shown in Figure 14, where the network was tested if it produced sensible results. The first operational point, Op1, corresponds to the lowest evaluated rotational speed of the engine 42,058 rpm, and the last, Op5, corresponds to the highest Appl. Sci. 2019, 9, 2253 13 of 22 rotational speed (i.e., 50,289 rpm). Appl. Sci. 2019, 9Figure 14. , x FOR PEER The REmeasured parameters during a si VIEW ngle run of the iSTC-21v engine. 14 of 23 Figure 14. The measured parameters during a single run of the iSTC-21v engine. The classifications of the five example images are shown in Figure 15a, and the created segment areas are shown in Figure 15b. Following expert examination of the infrared images and heuristic evaluation of the temperature field, it can be concluded that the Kohonen network works as expected and is able to sensibly assign the hottest and coldest parts of the object to each of the five classes/segments. In order to quantify the results, the number of activations of each neuron are summed, giving the total segment areas S1, …, S5 corresponding to each of the classes N1, …, N5. This can be formalized as follows: 𝑆 = 𝑥 ; x = {0;1} (5) where p is the number of activations of the i-th neuron. In this way, the size of each class and the total area of each temperature segment can be obtained. As can be seen in Figure 16, the total area of the segments in the three hottest classes increased as the engine speed and temperature increased. As it is not possible to define the exact areas which correspond to each class (as in traditional image segmentation where, for example, a river area is clearly defined), a system for the objective evaluation of the trained classifier performance was designed, and is described in the next section. Figure 15. The segmented thermal images produced by the Kohonen map. Figure 15. The segmented thermal images produced by the Kohonen map. In this way, the size of each class and the total area of each temperature segment can be obtained. As can be seen in Figure 16, the total area of the segments in the three hottest classes increased as the engine speed and temperature increased. As it is not possible to deﬁne the exact areas which correspond to each class (as in traditional image segmentation where, for example, a river area is clearly deﬁned), a system for the objective evaluation of the trained classiﬁer performance was designed, and is described in the next section. Figure 16. The size Si of each segmented area for five example points. 3.2. Classification System Testing and Results To validate the ability of the trained Kohonen network to segment the thermal image, a system for its evaluation was designed as follows. Five artificial spheres, each with an area of 150 pixels, were created, each of which fell into only one of the classes. As can be seen in Figure 17, the network was able to correctly classify these artificial anomalies in this particular case. In order to properly evaluate the functionality of the classifier, five testing infrared images were divided into 60 segments. The artificially created spherical temperature field anomaly for each class was moved across all segments, giving a total of 300 testing images for each class and 1500 testing images in total. The division of the image into segments when the artificial anomaly was included is shown in Figure 18. Appl. Sci. 2019, 9, x FOR PEER REVIEW 14 of 23 Appl. Sci. 2019, 9, 2253 14 of 22 Figure 15. The segmented thermal images produced by the Kohonen map. Figure Figure 16. 16. The The size size S Si of of each each segmented area segmented area for for five ﬁve example points. example points. 3.2. Classiﬁcation System Testing and Results 3.2. Classification System Testing and Results To validate the ability of the trained Kohonen network to segment the thermal image, a system for To validate the ability of the trained Kohonen network to segment the thermal image, a system its evaluation was designed as follows. Five artiﬁcial spheres, each with an area of 150 pixels, were for its evaluation was designed as follows. Five artificial spheres, each with an area of 150 pixels, created, each of which fell into only one of the classes. As can be seen in Figure 17, the network was were created, each of which fell into only one of the classes. As can be seen in Figure 17, the network able to correctly classify these artiﬁcial anomalies in this particular case. Appl. Sci. 2019, 9, x FOR PEER REVIEW 15 of 23 was able to correctly classify these artificial anomalies in this particular case. In order to properly evaluate the functionality of the classifier, five testing infrared images were divided into 60 segments. The artificially created spherical temperature field anomaly for each class was moved across all segments, giving a total of 300 testing images for each class and 1500 testing images in total. The division of the image into segments when the artificial anomaly was included is shown in Figure 18. Figure 17. Thermal anomalies correctly classified by the Kohonen network. Figure 17. Thermal anomalies correctly classiﬁed by the Kohonen network. In order to properly evaluate the functionality of the classiﬁer, ﬁve testing infrared images were divided into 60 segments. The artiﬁcially created spherical temperature ﬁeld anomaly for each class was moved across all segments, giving a total of 300 testing images for each class and 1500 testing images in total. The division of the image into segments when the artiﬁcial anomaly was included is shown in Figure 18. The results are summarized in the confusion matrix shown in Table 2; the ﬁrst row represents the real class of the anomaly and the ﬁrst column represents the class output by the network. The contingency table gives a clear picture of the network’s ability to correctly segment the infrared images. The network achieved a very good accuracy for the ﬁrst two classes, and only misclassiﬁed objects for the three hottest classes. The lowest obtained accuracy was for the classiﬁcation of class N , which was the coldest class and mostly represented the surroundings of the engine, which are of a low temperature. This class also contained misclassiﬁed objects belonging to class N , which only Figure 18. Division of the infrared image for the insertion of the artificial anomaly. represented relatively cold parts of the engine or its surroundings. The most important classes for the evaluation of the technical state of a turbojet engine are those which represent its hottest parts, The results are summarized in the confusion matrix shown in Table 2; the first row represents the real class of the anomaly and the first column represents the class output by the network. The contingency table gives a clear picture of the network’s ability to correctly segment the infrared images. Table 2. The confusion matrix of the trained Kohonen self-organizing feature map (SOFM). Actual class N1 N2 N3 N4 N5 N1 286 5 2 0 0 N2 10 284 15 1 0 Predicted Class N3 4 10 272 4 6 N4 0 1 10 275 34 N5 0 0 1 20 260 Accuracy 95.3% 94.6% 90.6% 91.6% 86% Total Accuracy 91.8% (1377/1500) The network achieved a very good accuracy for the first two classes, and only misclassified objects for the three hottest classes. The lowest obtained accuracy was for the classification of class N5, which was the coldest class and mostly represented the surroundings of the engine, which are of a low temperature. This class also contained misclassified objects belonging to class N4, which only represented relatively cold parts of the engine or its surroundings. The most important classes for the evaluation of the technical state of a turbojet engine are those which represent its hottest parts, that is, N1, N2, and N3. The results could probably be improved by using a larger training dataset or different Kohonen layer topologies, however they can nevertheless be evaluated as very successful for the purpose of turbojet engine diagnostics. Appl. Sci. 2019, 9, x FOR PEER REVIEW 15 of 23 Appl. Sci. 2019, 9, 2253 15 of 22 that is, N , N , and N . The results could probably be improved by using a larger training dataset or 1 2 3 dierent Kohonen layer topologies, however they can nevertheless be evaluated as very successful for the purpose of turbojet engine diagnostics. Figure 17. Thermal anomalies correctly classified by the Kohonen network. Figure 18. Figure 18. Divi Division sion of of the the infrared image for infrared image for the the insertion insertion of of the artif the artiﬁcial icial a anomaly nomaly. . Table 2. The confusion matrix of the trained Kohonen self-organizing feature map (SOFM). The results are summarized in the confusion matrix shown in Table 2; the first row represents the real class of the anomaly and the first column represents the class output by the network. The Actual Class contingency table gives a clear picture of the network’s ability to correctly segment the infrared N N N N N 1 2 3 4 5 images. N 286 5 2 0 0 N 10 284 15 1 0 Table 2. The confusion matrix of the trained Kohonen self-organizing feature map (SOFM). N 4 10 272 4 6 Predicted Class N 0 1 10 275 34 Actual class N 0 0 1 20 260 N1 N2 N3 N4 N5 Accuracy 95.3% 94.6% 90.6% 91.6% 86% N1 286 5 2 0 0 Total N2 10 284 15 1 0 91.8% (1377/1500) Accuracy Predicted Class N3 4 10 272 4 6 N4 0 1 10 275 34 4. Expert Diagnostic System Design Based on Infrared Images N5 0 0 1 20 260 Accuracy 95.3% 94.6% 90.6% 91.6% 86% 4.1. Design of the Expert System Total Accuracy 91.8% (1377/1500) As presented in Figure 15, areas of the segments are not absolutely representative of the engine’s technical state, as their changes are quite slow as the engine heats up or cools down and the internal The network achieved a very good accuracy for the first two classes, and only misclassified thermal energy is transferred to its casing, which has its own heating and cooling dynamics. The main objects for the three hottest classes. The lowest obtained accuracy was for the classification of class task of the expert system is to evaluate the areas of the segments and their changes, and to then present N5, which was the coldest class and mostly represented the surroundings of the engine, which are of to the operator some information about the technical state of the engine. a low temperature. This class also contained misclassified objects belonging to class N4, which only To fulﬁll this task, the designed expert system needs as an input the total area of the segments and represented relatively cold parts of the engine or its surroundings. The most important classes for some indication of how they evolve over time. Temporal changes in the areas of the segments can be the evaluation of the technical state of a turbojet engine are those which represent its hottest parts, predicted by the computation of derivations/dierences between two segmented images in a set time that is, N1, N2, and N3. The results could probably be improved by using a larger training dataset or interval Dt. The standard derivative equation can be used for this purpose for the i-th neuron and the different Kohonen layer topologies, however they can nevertheless be evaluated as very successful total area of the i-th segment S : for the purpose of turbojet engine diagnostics. ( ) ( ) S t + Dt S t i i dS (t) = (6) Dt From a diagnostic point of view, it makes sense to investigate only the three hottest states (levels S –S ) and their derivations S –S , as they represent the most critical temperatures for the engine. 1 3 1_d 3_d The time derivation of the surface areas of the segments allows the magnitude of the change in the area of dierent temperature regions to be determined and early structural failures to be detected. The knowledge base of the proposed expert diagnostic system consists of a set of rules that describe normal and atypical engine operation based on the outputs of the Kohonen SOFM. Each rule R (i = 1, 2, ..., 8) i Appl. Sci. 2019, 9, 2253 16 of 22 is designed according to precisely deﬁned intervals of the allowed amount of pixels (area sizes) of individual color levels and intervals of allowed values of their derivations, based on results for the normal state and atypical states of the object. The outputs of the rules R take Boolean logical values R i i Є{0; 1}. The rules R in Table 3 are deﬁned as follows: IF S is the area size <from;to> AND S is the derivative <from;to> => R , where S (i = 1, 2, 3) (7) i i_d i i Table 3. A partial set of rules in the knowledge base of the proposed expert diagnostic system. IF S IS <13,200; 13,300> AND S is <5; 50 > => R 1 1_d 1 IF S IS <3600; 3700> AND S is <10; 30> => R 2 2_d 2 IF S IS <7300; 7420> AND S is <5 ; 45> => R 3 3_d 3 IF S IS <46; 160 > – – => R 3_d 4 IF S IS <85;6> – – => R 3_d 5 IF S IS <31; 140> – – => R 2_d 6 IF S IS <120;11> – – => R 2_d 7 IF S IS <51; 160> – – => R 1_d 8 The rules R are partial conclusions about the technical state of the investigated engine. In order to obtain some knowledge about the engine’s technical state, the hypotheses which are to be connected with these rules have to be deﬁned. In order to keep the rule base simple at ﬁrst, diagnostic signals (hypotheses) for a turbojet engine are proposed in Table 4. Table 4. Technical states of the turbojet engine. D Normal condition D Light overheating D Moderate overheating D Critical overheating D Light structural defect D Critical structural defect These diagnostic signals (D ; i = 0, 1, 2, ..., 5) are activated by combinations of the proposed rules. The rules to obtain the diagnostic signals are deﬁned in Table 5. Table 5. Inference of the diagnostic signals. IF R AND R AND R => D 1 2 3 0 IF R AND R AND R => D 1 2 4 1 IF R AND R AND R => D 1 5 6 2 IF R AND R AND R => D 3 7 8 3 IF R AND R AND R => D 2 5 8 4 IF R AND R AND R => D 5 7 8 5 The rules, hypotheses, and inputs can be formalized in the form of an oriented graph. The inference network is deﬁned in Section 2.3. In the case of a turbojet engine with diagnostics based on the sizes and derivatives of segmented thermal images, the inference network shown in Figure 19 is proposed based on rules in Tables 3 and 4, which represent the knowledge base of the expert system. The inference engine then performs a bottom–up search from data of the segments’ sizes S and their derivatives S through intermediate rules R up to evaluation of the hypotheses D . i_d j k Appl. Sci. 2019, 9, 2253 17 of 22 Appl. Sci. 2019, 9, x FOR PEER REVIEW 18 of 23 Figure 19. The inference network of the designed diagnostic expert system. Figure 19. The inference network of the designed diagnostic expert system. 4.2. Pilot Testing of the Expert System Using Infrared Thermography (IRT) Inputs 4.2. Pilot Testing of the Expert System Using Infrared Thermography (IRT) Inputs The idea behind the design of the presented expert system is to evaluate its possibilities and The idea behind the design of the presented expert system is to evaluate its possibilities and its its ability to correctly present diagnostic signals to the engine’s operator or supervisory control ability to correctly present diagnostic signals to the engine’s operator or supervisory control system system [64,65]. The designed rule base was therefore constructed to be very simple in order to evaluate [64,65]. The designed rule base was therefore constructed to be very simple in order to evaluate the the ﬁve pre-deﬁned temperature states represented by hypotheses D. Pilot tests were conducted to five pre-defined temperature states represented by hypotheses D. Pilot tests were conducted to prove that the system works as intended. For real-world engine operation, it was only possible to test prove that the system works as intended. For real-world engine operation, it was only possible to the states D , D , D , and D . The resulting classiﬁcations for these tested situations are shown in 0 1 4 5 test the states D0, D1, D4, and D5. The resulting classifications for these tested situations are shown in Table 6. States D and D were artiﬁcially induced by quickly adding heating material on the surface 4 5 Table 6. States D4 and D5 were artificially induced by quickly adding heating material on the surface of the engine’s exhaust nozzle, which created anomalies in the temperature ﬁeld of the engine. The of the engine’s exhaust nozzle, which created anomalies in the temperature field of the engine. The condition of light overheating was achieved when the engine was running at a high speed setting, (i.e., condition of light overheating was achieved when the engine was running at a high speed setting, at an operating speed of 50,000 rpm for around 20 s). The rules which were evaluated as true, and the (i.e., at an operating speed of 50,000 rpm for around 20 s). The rules which were evaluated as true, way the diagnostic signal/hypothesis was obtained, are shown in Table 6. These results suggest that and the way the diagnostic signal/hypothesis was obtained, are shown in Table 6. These results the diagnostic expert system is able to determine basic states represented by hypotheses D , D , D , 0 1 4 suggest that the diagnostic expert system is able to determine basic states represented by hypotheses D and that the designed rules work. D0 , D1, D4, D5 and that the designed rules work. Additionally, another test was set up using the data from a single engine run, as shown in Figure 14. Normal and light overheating states were tested during standard operation of the engine, and are shown in Figure 20. It can be seen that the expert system selected the state D (light overheating) even though the speed of the engine and the corresponding temperatures had decreased. This was the correct selection, since the temperature of the casing takes more time to cool down, whereas the inner gas temperature as measured by internal sensors decreases immediately. This set of pilot tests showed that the inference engine works as intended and that the expert system is able to produce some sensible results in the evaluation of the technical state of the engine during normal conditions and to detect artiﬁcially induced structural defects. In order to prove the correctness of the setting of the rules R , as well as the deﬁnition of intervals in these rules, a more systematic approach has to be taken. This will be designed in a follow-up study of the system. Appl. Sci. 2019, 9, 2253 18 of 22 Appl. Sci. 2019, 9, x FOR PEER REVIEW 19 of 23 Appl. Appl. Sci. Sci. 2019 2019, , 9 9,, x FO x FOR P R PEER EER RE REVIEW VIEW 19 of 19 of 23 23 Appl. Appl. Appl. Appl. Appl. Sci. Sci. Sci. Sci. Sci. 2019 2019 2019 2019 2019 , , 9 9 , , , ,,9 x FO 9 9 x FO ,, , x FO x FO x FO R P R P R P R P R P EER EER EER EER EER RE RE RE RE RE VIEW VIEW VIEW VIEW VIEW 19 of 19 of 19 of 19 of 19 of 23 23 23 23 23 Table 6. Table 6. Te Te stin stin g of the ex g of the ex pert pert sy sy stem stem fo fo r the r the evaluation of evaluation of hypotheses hypotheses DD 0, D 0, D 1, D 1, D 4, and 4, and DD 5. 5. Table 6. Table 6. Table 6. Table 6. Table 6. Table 6. Te Te Te Te Te Te stin stin stin stin stin stin g g g of the ex of the ex g g g of the ex of the ex of the ex of the ex pert pert pert pert pert pert sy sy sy sy sy sy stem stem stem stem stem stem fo fo fo fo fo fo r the r the r the r the r the r the evaluation of evaluation of evaluation of evaluation of evaluation of evaluation of hypotheses hypotheses hypotheses hypotheses hypotheses hypotheses D D D D D D 0 00, D , D , D 0 0 0, D , D , D 1 11, D , D , D 1 1 1, D , D , D 4 44,,, and and 4 4 4 and ,,, and and and D D D D D D 5 55. . . 5 5 5. . . Table 6. Testing of the expert system for the evaluation of hypotheses D , D , D , and D . 0 1 4 5 D0—normal condition DD0 0—norm —norma al c l co ondition ndition DDDDD 00—norm —norm 0 0 0—norm —norm —norm a al c a a a l c l c l c l c o ondition o o o ndition ndition ndition ndition R1: S1 = 13,270 AND S1_d = 20 R R RR R 1 1 1: S : S : S 1 1: S : S 1 1 1 = = = 1 1 1 1 1 = = 1 1 3 3 3,270 ,270 ,270 3 3,270 ,270 AND AND AND AND AND S S S1 1 1S S _d _d _d 1 1 _d _d = = = 20 20 20 = = 20 20 RR 1: S 1: S 1 1 == 1 1 33 ,270 ,270 AND AND SS 1_d 1_d = = 20 20 D —normal condition R : S = 13,270 AND S = 20 1 1 1_d R2: S2 = 3628 AND S2_d = 14 R R2 2: S : S2 2 = 36 = 362 28 8 AND AND S S2 2_d _d = 14 = 14 R RR R R 22: S : S 2 2 2: S : S : S 2 2 = 36 2 = 36 2 2 = 36 = 36 = 36 2 28 2 2 2 8 AND 8 8 8 AND AND AND AND S S 2 S S S 2 _d _d 2 2 2_d _d _d = 14 = 14 = 14 = 14 = 14 R : S = 3628 AND S = 14 2 2 2_d R : S = 7374, AND S = 24 3 3 3_d RR 3: S 3: S 3 = 73 3 = 73 74 7, AND 4, AND S3 S _d 3 _d = 24 = 24 R RR R 3 3: S : S 3 3: S : S 3 3 = 73 = 73 3 3 = 73 = 73 7 74 4 7 7, AND , AND 4 4, AND , AND S S3 3S S _d _d 3 3 _d _d = 24 = 24 = 24 = 24 RR 3: S 3: S 3 3 = 73 = 73 77 44 , AND , AND SS 3_d 3_d = 24 = 24 R AND R AND R => D 1 2 3 0 R1 AND R2 AND R3 => D0 R R1 1 AND AND R R2 2 AN AND D R R3 3 => => D D0 0 R RR R R 11 AND AND 1 1 1 AND AND AND R RR R R 22 AN AN 2 2 2 AN AN AN D D D D D R RR R R 33 => => 3 3 3 => => => D D D D D 00 0 0 0 D1—light overheating DDDD 1 1 1—l —l —l 1—l igh igh igh igh t t t o o o t o v v ve e e vrhe rhe rhe erhe at at atat ing ing ing ing DDD 1—l 1 1—l —l igh igh igh t o t t o o vv v ee e rhe rhe rhe at at at ing ing ing RR 1: S 1: S 1 = 1 1 = 1 3,227 3,227 AND AND S1 S _d 1 _d = 31 = 31 R RR R R 1 1: S : S 1 1 1D : S : S : S 1 1 —light = = 1 1 1 1 1 = = = 1 1 1 3 3,227 ,227 3 3 3,227 ,227 ,227 overheating AND AND AND AND AND S S1 1 S S S _d _d 1 1 1_d _d _d = = 31 31 = = = 31 31 31 R1: S1 = 13,227 AND S1_d = 31 R : S = 13,227 AND S = 31 1 1 1_d R2: S2 = 3643 AND S2_d = -5 R R RR 2 2 2: S : S : S 2: S 2 2 2 = 36 = 36 = 36 2 = 36 4 4 43 3 3 4 AND AND AND 3 AND S S S2 2 2S _d _d _d 2 _d = -5 = -5 = -5 = -5 RRR :R 2: S S 2 2: S : S 2 = 2 = 36 2 = 36 = 36 3643 44 4 3 AND 3 3 AND AND AND S SS S 2_d 2 2 = _d _d = -5 = -5 = -5 5 2 2 2_d R : S = 112 4 3_d RR 4: S 4: S 3_d 3 = _d = 11 11 2 2 R RR R R 4 4: S : S 4 4 4: S : S : S 3 3_d _d 3 3 3_d = = _d _d = = = 11 11 11 11 11 2 2 2 2 2 R ANDR R 4: S AND 3_d = 11 R 2 => D 1 2 4 1 R1 AND R2 AND R4 => D1 R R RR 1 1 1 AND AND AND 1 AND R R RR 2 2 2 AN AN AN 2 AN D D D D R R RR 4 4 4 => => => 4 => D D D D 1 1 1 1 RR R 1 AND 1 1 AND AND RR R 2 AN 2 2 AN AN D D D RR R 4 => 4 4 => => D D D 1 1 1 DD 4—ligh 4—ligh t s t s trtu ru ctur ctur al defec al defec t t DDDD 4 4—ligh —ligh 4 4—ligh —ligh t t s s t t s s t tr rt tu u r rctur u u ctur ctur ctur al defec al defec al defec al defec t t t t DD 4—ligh 4—ligh t s t s tr tu ru ctur ctur al defec al defec t t R2: S2 = 3671 AND S2_d = 3 R R2 2: S : S2 2 = 36 = 367 71 1 AND AND S S2 2_d _d = 3 = 3 D R RR R R 22—light : S : S 2 2 2: S : S : S 2 2 = 36 2 = 36 2 2 = 36 = 36 = 36 structural 7 71 7 7 7 1 AND 1 1 1 AND AND AND AND S defect S 2 S S S 2 _d _d 2 2 2_d _d _d = 3 = 3 = 3 = 3 = 3 R : S = 3671 AND S = 3 2 2 2_d RR 5: S 5: S 3_d 3 _d = -67 = -67 RR R:R R 5 5: S : S S 5 5: S : S 3 3_d _d 3 3= _d _d = = -67 -67 = = -67 -67 67 RR 5: S 5: S 3_d 3_d = = -67 -67 5 3_d R : S = 92 8 1_d R8: S1_d = 92 R R8 8: S : S1 1_d _d = 92 = 92 R ANDR R R R R R 88: S : S 8 8 8AND : S : S : S 11 _d _d 1 1 1_d _d _d = 92 = 92 = 92 = 92 = 92 R => D 2 5 8 4 RR 2 AND 2 AND RR 5 AN 5 AN D D RR 8 => 8 => D D 4 4 R RR R R 2 2 AND AND 2 2 2 AND AND AND R RR R R 5 5 AN AN 5 5 5 AN AN AN D D D D D R RR R R 8 8 => => 8 8 8 => => => D D D D D 4 4 4 4 4 R2 AND R5 AND R8 => D4 D5—critical structural defect DDDDD 5 5 5—critic —critic —critic 5 5—critic —critic a a al s l s l s a al s l s t t tructural d ructural d ructural d t tructural d ructural d e e ef f f e ee e e f fct ct ct e ect ct DD 5—critic 5—critic aa l s l s tructural d tructural d ee fe fe ct ct R5: S3_d = -72 D —critical R R5 5: S : S structural 3 3_d _d = = -72 -72 defect R RR R R 55: S : S 5 5 5: S : S : S 33 _d _d 3 3 3_d _d _d = = -72 = = = -72 -72 -72 -72 R : S =72 5 3_d RR 7: S 7: S 2_d 2 _d = -45 = -45 RR R:R R 7 7: S : S S 7 7: S : S 2 2_d _d 2 2= _d _d = = -45 -45 = = -45 -45 45 RR 7: S 7: S 2_d 2_d = = -45 -45 7 2_d R : S = 118 8 1_d R8: S1_d = 118 R R8 8: S : S1 1_d _d = 11 = 118 8 R AND R R R R R R 88: S : S 8 8 8: S : S : S AND 11 _d _d 1 1 1_d _d _d = 11 = 11 = 11 = 11 = 11 R 8 8 => 8 8 8 D 5 7 8 5 RR 5 AND 5 AND RR 7 AN 7 AN D D RR 8 => 8 => D D 5 5 R RR R 5 5 AND AND 5 5 AND AND R RR R 7 7 AN AN 7 7 AN AN D D D D R RR R 8 8 => => 8 8 => => D D D D 5 5 5 5 RR 5 AND 5 AND RR 7 AN 7 AN D D RR 8 => 8 => D D 5 5 Appl. Sci. 2019, 9, x FOR PEER REVIEW 20 of 23 Additionally, another test was set up using the data from a single engine run, as shown in Addit Addit Addit Addit iiion on on ion a a ally lly lly ally ,,, anot anot anot , anot her her her her t t t t e e est st st est was was was was set set set set up up up up usin usin usin usin g t g t g t g t h h he e e he dat dat dat dat a a a afrom from from from a a a a sin sin sin sin g g gle le le gle engin engin engin engin e e e r r r e r u u un, a n, a n, a un, a s s s sshown shown shown shown in in in in Addit Addit Addit ion iion on aa a lly lly lly , anot ,, anot anot her her her t t t ee e stst st was was was set set set up up up usin usin usin g t g t g t hh h e e e dat dat dat aa a from from from a a a sin sin sin gg g le le le engin engin engin ee e r r r uu u n, a n, a n, a ss sshown shown shown in in in Fig Fig uu re re 1 4 1.4 No . No rm rm alal and li and li gh gh t overheat t overheat ing ing st st atat es es were were te te sted d sted d uu rin rin g g stand stand ard ard operation of the operation of the en en gine, gine, Fig Fig Fig Fig Fig u uu re re u ure re re 1 1 4 4 1 1 1..4 4 4 No No ... No No No rm rm rm rm rm al alal al al and li and li and li and li and li gh gh gh gh gh t t overheat t overheat t t overheat overheat overheat ing ing ing ing ing st st st st st at at at at at es es es es es were were were were were te te te te te sted d sted d sted d sted d sted d u uu rin rin u urin rin rin g g g g g stand stand stand stand stand a ard rd a a ard rd rd operation of the operation of the operation of the operation of the operation of the en en en en en gine, gine, gine, gine, gine, Figure 14. Normal and light overheating states were tested during standard operation of the engine, and are shown in Figure 20. It can be seen that the expert system selected the state D1 (light and and are are sho show wn in n in Fig Figu ure re 20. It c 20. It ca an b n bee seen seen that t that th he expert sy e expert system selected stem selected the state the state D D1 1 (l (light ight and and and and and are are are are are sho sho sho sho sho w ww w w n in n in n in n in n in Fig Fig Fig Fig Fig uuu re u urere re re 20. It c 20. It c 20. It c 20. It c 20. It c aan b aa an b n b n b n b ee seen eee seen seen seen seen that t that t that t that t that t hhe expert sy h h he expert sy e expert sy e expert sy e expert sy stem selected stem selected stem selected stem selected stem selected the state the state the state the state the state D D D D D 11 1 1 1(l (l (l ight (l (l ight ight ight ight overheating) even though the speed of the engine and the corresponding temperatures had overheating overheating overheating overheating overheating ))) even thoug even thoug even thoug )) even thoug even thoug h hh the speed the speed the speed h h the speed the speed of the of the of the of the of the eng eng eng eng eng iiine and ne and ne and iine and ne and the the the the the correspondin correspondin correspondin correspondin correspondin g temperatur g temperatur g temperatur g temperatur g temperatur es h es h es h es h es h aaad d daa d d overheating overheating) even thoug ) even thoughh the speed the speed of the of the eng engine and ine and the the correspondin corresponding temperatur g temperatures h es haadd decreased. Th decreased. Th is was the correct selection is was the correct selection , since the te , since the te mpera mpera ture of ture of the ca the ca sisi ng ta ng ta kes more ti kes more ti me to cool me to cool decreased. Th decreased. Th decreased. Th decreased. Th decreased. Th decreased. Th is was the correct selection is was the correct selection is was the correct selection is was the correct selection is was the correct selection is was the correct selection ,,, since the te since the te since the te ,, , since the te since the te since the te mpera mpera mpera mpera mpera mpera t tture of ure of ure of t t ture of ure of ure of the ca the ca the ca the ca the ca the ca si si si si si si ng ta ng ta ng ta ng ta ng ta ng ta kes more ti kes more ti kes more ti kes more ti kes more ti kes more ti me to cool me to cool me to cool me to cool me to cool me to cool down, whereas the inner gas temperature as measured by internal sensors decreases immediately. down, whe down, whe down, whe down, whe r r re e era a a es the s the s the as the inner inner inner inner gas temper gas temper gas temper gas temper ature ature ature ature as me as me as me as me asur asur asur asur ed by ed by ed by ed by intern intern intern intern al sen al sen al sen al sen s s sors de ors de ors de sors de creases imme creases imme creases imme creases imme diate diate diate diate llly. y. y. ly. down, whe down, whe down, whe re r re e aa a s the s the s the inner inner inner gas temper gas temper gas temper ature ature ature as me as me as me asur asur asur ed by ed by ed by intern intern intern al sen al sen al sen ss ors de sors de ors de creases imme creases imme creases imme diate diate diate ly. lly. y. This set of pilot tests showed that the inference en This set of pilot tests showed that the inference en gin gin e works as int e works as int ended and th ended and th at the expert system at the expert system This set of pilot tests showed that the inference en This set of pilot tests showed that the inference en This set of pilot tests showed that the inference en This set of pilot tests showed that the inference en This set of pilot tests showed that the inference en gin gin gin gin gin e e works as int works as int e e e works as int works as int works as int e ended and th nded and th e e ended and th nded and th nded and th at the expert system at the expert system at the expert system at the expert system at the expert system This set of pilot tests showed that the inference engine works as intended and that the expert system is able to produce some sensible results in the evaluation of the technical state of the engine during is is able able t to o pro prod duce uce some s some se ensible nsible res resu ult lts s in t in th he ev e eval aluat uatiion of the tec on of the technical st hnical state o ate of f the en the engine gine durin during g is is is is is able able able able able t t t o t t o pro o o o pro pro pro pro d dd uce d d uce uce uce uce some s some s some s some s some s e ensible e e e nsible nsible nsible nsible res res res res res u uu lt u u ltlt s lt lt s s s s in t in t in t in t in t h he ev h h h e ev e ev e ev e ev al al al al al uat uat uat uat uat iion of the tec on of the tec ii ion of the tec on of the tec on of the tec hnical st hnical st hnical st hnical st hnical st ate o ate o ate o ate o ate o ff the en the en f f f the en the en the en gine gine gine gine gine durin durin durin durin durin g g g g g normal conditions and to detect artificially induced structural defects. In order to prove the normal cond normal cond normal cond normal cond normal cond it it itions ions it it ions ions ions and t and t and t and t and t ooo det det det oo det det ect ect ect ect ect art art art art art iiifffiiiiiccffciiia ia ia ccia ia ll ll lly ind y ind y ind ll lly ind y ind u u uce ce ce u uce ce d structur d structur d structur d structur d structur al al al al al defects. In o defects. In o defects. In o defects. In o defects. In o rrrder to prove the der to prove the der to prove the rrder to prove the der to prove the normal cond normal condititions ions and t and too det detect ect art artifiifciciaialllly ind y induuceced structur d structural al defects. In o defects. In order to prove the rder to prove the correctness of the setti correctness of the setti ng of ng of the rules the rules RR i, ias w , as w ele l a ll a s t s t he defin he defin itit ion o ion o f int f int erva erva ls ls in in thtese rul hese rul es, es, a more a more correctness of the setti correctness of the setti correctness of the setti correctness of the setti correctness of the setti correctness of the setti ng of ng of ng of ng of ng of ng of the rules the rules the rules the rules the rules the rules R R RR R R i ii, , , ii ias w as w , , , as w as w as w as w e e elle e e llll a l a ll a lll a a a s ss t t s t s s t t t h h he defin h e defin h h e defin e defin e defin e defin it it itit ion o it it ion o ion o ion o ion o ion o f ff int int int ff f int int int e e erva rva e e e rva rva rva rva llls s s llls s s in in in in in in ttth h h tt tese rul h ese rul h h ese rul ese rul ese rul ese rul e e es, s, e e e s, s, s, s, a a a more more a a a more more more more systematic approach has to be taken. This will be designed in a follow-up study of the system. syst syst syst syst emat emat emat emat ic ic ic ic approach h approach h approach h approach h a a as t s t s t as t o o o be t be t be t o be t a a aken ken ken aken ... Thi Thi Thi . Thi s s s wi wi wi s wi ll ll ll be de be de be de ll be de si si sisi gned gned gned gned in in in in a fo a fo a fo a fo ll ll llow-up ow-up ow-up llow-up st st stst udy udy udy udy of t of t of t of t h h he s e s e s he s y y yst st st yst em. em. em. em. syst syst syst emat emat emat ic ic ic approach h approach h approach h aa a s t s t s t oo o be t be t be t aa a ken ken ken . Thi .. Thi Thi ss wi s wi wi llll ll be de be de be de si si si gned gned gned in in in a fo a fo a fo llll ll ow-up ow-up ow-up st st st udy udy udy of t of t of t hh h e s e s e s yy y st st st em. em. em. Figure 20. The division of the engine’s operation into technical states D and D . 0 1 Figure 20. The division of the engine’s operation into technical states D0 and D1. 5. Conclusions This article defines a way to use infrared imaging directly in a new area of aviation, namely, in the evaluation of the technical state of turbojet engines. A concept of a hybrid system combining a SOFM, namely a Kohonen neural network, and an expert system, is shown to be a promising approach to solve the problem of thermal image segmentation and to extract knowledge about the technical state of the engine from the obtained segmented IRT images. The Kohonen neural network defined in this article is a computationally very simple algorithm, and was able to achieve a precision of higher than 90% in the segmentation of infrared images obtained by the FLIR A40 camera when introducing an artificial temperature anomaly. By using the total areas of segmented infrared images corresponding to different temperatures, and their time derivatives, the technical state of the engine can be extracted and evaluated. This can be considered as a novel approach, and it has shown promising results as a proof of concept, which can be further developed and investigated, for example, to take into account environmental or high noise conditions or the impact of application of different image filtering techniques. Moreover, accuracy of the classification could also be increased by utilizing a camera with the ability to record a video with 10 bit color resolution instead of the reduced iron-bow palette for applications, where fine temperature resolution is important. On the other hand, the developed methodology with an expert system seems to be robust enough to work well with the reduced color spectrum and the engine is working in a wide temperature range. The rule base of the expert system proposed in this article can be further expanded to contain more hypotheses, and can also be developed to use fuzzy logic to evaluate the rules to improve the precision of the system. It can be considered as a progressive methodology in the application of computational intelligence and infrared thermography to the area of aviation. The methodology can also be applied in other fields of technical practice to evaluate the temperature fields of different technical objects, such as electronic circuit boards, transmissions, rotating machinery, generators, and electric engines operating in laboratory conditions. The system is designed to be objective, and does not need any input or examination of the infrared images by humans. It can also be integrated into a complex situational control algorithm of a turbojet engine in laboratory conditions, where outputs of the expert system can be used as inputs for the situational classifier and also to tune adaptive controllers of the engine [62,64,65]. Appl. Sci. 2019, 9, 2253 19 of 22 5. Conclusions This article deﬁnes a way to use infrared imaging directly in a new area of aviation, namely, in the evaluation of the technical state of turbojet engines. A concept of a hybrid system combining a SOFM, namely a Kohonen neural network, and an expert system, is shown to be a promising approach to solve the problem of thermal image segmentation and to extract knowledge about the technical state of the engine from the obtained segmented IRT images. The Kohonen neural network deﬁned in this article is a computationally very simple algorithm, and was able to achieve a precision of higher than 90% in the segmentation of infrared images obtained by the FLIR A40 camera when introducing an artiﬁcial temperature anomaly. By using the total areas of segmented infrared images corresponding to dierent temperatures, and their time derivatives, the technical state of the engine can be extracted and evaluated. This can be considered as a novel approach, and it has shown promising results as a proof of concept, which can be further developed and investigated, for example, to take into account environmental or high noise conditions or the impact of application of dierent image ﬁltering techniques. Moreover, accuracy of the classiﬁcation could also be increased by utilizing a camera with the ability to record a video with 10 bit color resolution instead of the reduced iron-bow palette for applications, where ﬁne temperature resolution is important. On the other hand, the developed methodology with an expert system seems to be robust enough to work well with the reduced color spectrum and the engine is working in a wide temperature range. The rule base of the expert system proposed in this article can be further expanded to contain more hypotheses, and can also be developed to use fuzzy logic to evaluate the rules to improve the precision of the system. It can be considered as a progressive methodology in the application of computational intelligence and infrared thermography to the area of aviation. The methodology can also be applied in other ﬁelds of technical practice to evaluate the temperature ﬁelds of dierent technical objects, such as electronic circuit boards, transmissions, rotating machinery, generators, and electric engines operating in laboratory conditions. The system is designed to be objective, and does not need any input or examination of the infrared images by humans. It can also be integrated into a complex situational control algorithm of a turbojet engine in laboratory conditions, where outputs of the expert system can be used as inputs for the situational classiﬁer and also to tune adaptive controllers of the engine [62,64,65]. Author Contributions: Conceptualization, R.A., M.S. (Martin Schrötter), and M.C.; data curation, R.A. and L.F.; formal analysis, R.A., L.F., M.S. (Martin Schrötter), and S.S.; funding acquisition, R.A. and S.S.; investigation, R.A., L.F., M.S. (Martin Schrötter), M.C., R.B., and M.S. (Michal Schreiner); methodology, R.A., L.F., and R.B.; project administration, R.A., M.S. (Martin Schrötter), and S.S.; resources, L.F., M.S. (Martin Schrötter), M.C., R.B., and M.S. (Michal Schreiner); supervision, R.A., M.C., and S.S.; validation, R.A., L.F., M.S. (Martin Schrötter), and R.B.; visualization, L.F., M.S. (Martin Schrötter), M.C., and M.S. (Michal Schreiner); writing—original draft, R.A., L.F., ˇ ˇ M.S. (Martin Schrötter), and M.C.; writing—review and editing, R.A., L.F., M.S. (Martin Schrötter), and M.C. Funding: This work was supported by the following projects: Ecient Systems and Propulsion for Small Aircraft (ESPOSA), funded by the European commission in the seventh framework program under grant agreement ACP1-GA-2011-284859-ESPOSA, the Slovak Research and Development Agency (APVV) under grant agreement DO7RP-0023-11 and KEGA 044TUKE-4/2019—a small unmanned airplane—the platform for education in the area of intelligent avionics. Conﬂicts of Interest: The authors declare no conﬂicts of interest. References 1. Osornio-Rios, R.A.; Antonino-Daviu, J.A.; Jesus Romero-Troncoso, R. Recent Industrial Applications of Infrared Thermography: A Review. IEEE Trans. Ind. Inform. 2019, 15, 615–625. [CrossRef] 2. Lahiri, B.B.; Bagavathiappan, S.; Jayakumar, T.; Philip, J. Medical applications of infrared thermography: A review. Infrared Phys. Technol. 2012, 55, 221–235. [CrossRef] 3. Balaras, C.A.; Argiriou, A.A. Infrared thermography for building diagnostics. Energy Build. 2002, 34, 171–183. [CrossRef] 4. Lisowska, A. Thermographic monitoring of the power transformers. Meas. Autom. Monit. 2017, 63, 154–157. Appl. Sci. 2019, 9, 2253 20 of 22 5. López-Pérez, D.; Antonino-Daviu, J. Application of Infrared Thermography to Failure Detection in Industrial Induction Motors: Case Stories. IEEE Trans. Ind. Appl. 2017, 53, 1901–1908. [CrossRef] 6. Yang, R.; He, Y.; Mandelis, A.; Wang, N.; Wu, X.; Huang, S. Induction Infrared Thermography and Thermal-Wave-Radar Analysis for Imaging Inspection and Diagnosis of Blade Composites. IEEE Trans. Ind. Inform. 2018, 14, 5637–5647. [CrossRef] 7. Vavilov, V.; Nesteruk, D.A. Detecting water in aviation honeycomb structures: The quantitative approach. Quant. Infrared Thermogr. J. 2004, 1, 173–184. [CrossRef] 8. Vavilov, V.; Pan, Y.Y.; Nesteruk, D. Infrared thermographic inspection of water ingress in composite honeycomb panels. Appl. Opt. 2016, 55, 120–125. [CrossRef] [PubMed] 9. Vavilov, V.P. Non-contact one-sided evaluation of hidden corrosion in metallic constructions by using transient infrared thermography. Rev. Metal. 2003, 39, 235–242. [CrossRef] 10. Eddazi, A.; Belattar, S. Nondestructive Testing Evaluation of Aircraft Fuselage Corrosion by Infrared Thermography and Finite Element Method. In Proceedings of the 14th International Conference on Computer Graphics, Imaging and Visualization, Marrakesh, Morocco, 23–25 May 2017. 11. Addepalli, S.; Zhao, Y.; Roy, R.; Galhenege, W.; Colle, M.; Yu, J.; Ucur, A. Non-destructive evaluation of localised heat damage occurring in carbon composites using thermography and thermal diusivity measurement. Measurement 2019, 131, 706–713. [CrossRef] 12. Meola, C.; Boccardi, S.; Carlomagno, G.M. Infrared Thermography in the Evaluation of Aerospace Composite Materials: Infrared Thermography to Composites; Woodhead Publishing: Cambridge, UK, 2016. 13. Swiderski, W. IR Thermography Nondestructive Testing Methods of Composite Materials Used in Aerospace Applications. In Proceedings of the 12th International Conference on Quantitative Infrared Thermography, The e-Journal of Nondestructive Testing, Mahabalipuram, India, 6–10 July 2016. 14. Orton, B.R.; Copaul, M.; Velez-Ospina, C.E. Potential use of infrared thermography to detect aircraft pressurization leaks. Insight Non-Destr. Test. Cond. Monit. 1999, 41, 164–166. 15. Wang, Y.; Gao, B.; Lok Woo, W.; Tian, G.; Maldague, X.; Zheng, L.; Guo, Z.; Zhu, Y. Thermal Pattern Contrast Diagnostic of Microcracks With Induction Thermography for Aircraft Braking Components. IEEE Trans. Ind. Inform. 2018, 14, 5563–5574. [CrossRef] 16. White, J.E.; Tucholski, E.J.; Green, R.E. Nondestructive Testing of Aircraft and Spacecraft Wiring. Mater. Eval. 2003, 61, 1315–1320. 17. Tulloch, J.S.; Jennings, M.C. Thermographic Wiring Inspection. U.S. Patent No. US 6,995,565 B1, 7 February 18. Short, M.; Torres, J.; Kreckie, J. Thermal Imaging for Aircraft Rescue and Fire Fighting Applications; DOT/FAA/TC-17/27 Final Report; U.S. Department of Transportation, Federal Aviation Administration: Cambridge, MA, USA, 2017. 19. Štumper, M.; Kraus, J. Thermal Imaging in Aviation. Mag. Aviat. Dev. 2015, 3, 13–16. [CrossRef] 20. Matikainen, L.; Lehtomäki, M.; Ahokas, E.; Hyyppä, J.; Karjalainen, M.; Jaakkola, A.; Kukko, A.; Heinonen, T. Remote sensing methods for power line corridor surveys. ISPRS J. Photogramm. Remote Sens. 2016, 119, 10–31. [CrossRef] 21. Gómez, C.; Green, D.R. Small unmanned airborne systems to support oil and gas pipeline monitoring and mapping. Arab. J. Geosci. 2017, 10, 202. [CrossRef] 22. Kylili, A.; Fokaides, P.A.; Christou, P.; Kalogirou, S.A. Infrared thermography (IRT) applications for building diagnostics: A review. Appl. Energy 2014, 134, 531–549. [CrossRef] 23. Omar, T.; Nehdi, M.L. Remote sensing of concrete bridge decks using unmanned aerial vehicle infrared thermography. Autom. Constr. 2017, 83, 360–371. [CrossRef] 24. Harvey, M.C.; Rowland, J.; Luketina, K.M. Drone with Thermal Infrared Camera Provides high resolution georeferenced imagery of the Waikite Geothermal Area, New Zealand. J. Volcanol. Geotherm. Res. 2016, 325, 61–69. [CrossRef] 25. Baranwal, N.; Mahulikar, S.P. Infrared Signature of Aircraft Engine with Choked Converging Nozzl. J. Thermophys. Heat Transf. 2016, 30, 1–9. [CrossRef] 26. Ciampa, F.; Mahmoodi, P.; Pinto, F.; Meo, M. Recent Advances in Active Infrared Thermography for Non-Destructive Testing of Aerospace Components. Sensors 2018, 18, 609. [CrossRef] Appl. Sci. 2019, 9, 2253 21 of 22 27. MacLeod, J.D.; Steckhan, P.; He, D. Infrared Thermal Imaging System as a Diagnostic Tool for Gas Turbine Engine Faults. In ASME 1994, International Gas Turbine and Aeroengine Congress and Exposition; American Society of Mechanical Engineers (ASME): New York, NY, USA, 1994; Volume 2. 28. Bogdan, M.; Błachnio, J.; Kułaszka, A.; Derlatka, M. Assessing the Condition of Gas Turbine Rotor Blades with the Optoelectronic and Thermographic Methods. Metals 2019, 9, 31. [CrossRef] 29. Yang, X.; Pang, S.; Shen, W.; Lin, X.; Jiang, K.; Wang, Y. Aero Engine Fault Diagnosis Using an Optimized Extreme Learning Machine. Int. J. Aerosp. Eng. 2016, 2016. [CrossRef] 30. Ntantis, E.; Botsaris, P. Diagnostic Methods for an Aircraft Engine Performance. J. Eng. Sci. Technol. Rev. 2015, 8, 64–72. [CrossRef] 31. Rajani, J. Engine Health Condition Monitoring—A Brief Review. Control Data Fusion E J. 2017, 1, 25–38. 32. Armstrong, J.B.; Simon, D.L. Implementation of an Integrated On-Board Aircraft Engine Diagnostic Architecture. In Proceedings of the 47th Joint Propulsion Conference and Exhibit cosponsored by the AIAA, ASME, SAE, and ASEE, San Diego, CA, USA, 31 July 2011–3 August 2011. 33. Rabenoro, T.; Lacaille, J.; Cottrell, M.; Rossi, F. A Methodology for the Diagnostic of Aircraft Engine Based on Indicators Aggregation. In Proceedings of the CDM 2014: Advances in Data Mining, Applications and Theoretical Aspects, St. Petersburg, Russia, 16–20 July 2014; Volume 8557, pp. 144–158. 34. Seabra, J.; Campos, A. EHD Analysis of a Roller/Inner Ring Contact in a Jet Engine Roller Bearing. In Proceedings of the 82nd Meeting of the AGARD Structures and Materials Panel, Sesimbra, Portugal, 6–7 May 1996; Volume 589. 35. Zaitouna, N.M.; Aqel, M.J. Survey on Image Segmentation Techniques. Procedia Comput. Sci. 2015, 65, 797–806. [CrossRef] 36. Karamizadeh, S.; Abdullah, S.M.; Zamani, M.; Kherikhah, A. Pattern Recognition Techniques: Studies on Appropriate Classiﬁcations. ARPN J. Eng. Appl. Sci. 2014, 9, 791–799. 37. Spontón, H.; Cardelino, J. A Review of Classic Edge Detectors. Image Process. Line 2015, 5, 90–123. [CrossRef] 38. Egmont-Petersen, M.; Ridder, D.; Handels, H. Image processing with neural networks—A review. Pattern Recognit. 2002, 35, 2279–2301. [CrossRef] 39. Vardasca, R.; Vaz, L.; Mendes, J. Classiﬁcation and Decision Making of Medical Infrared Thermal Images. In Classiﬁcation in BioApps; Lecture Notes in Computational Vision and Biomechanics; Dey, N., Ashour, A., Borra, S., Eds.; Springer: Cham, Switzerland, 2018; Volume 26. 40. Zhiqiang, W.; Jun, L. A review of object detection based on convolutional neural network. In Proceedings of the 2017 36th Chinese Control Conference (CCC), Dalian, China, 26–28 July 2017. 41. Rawat, W.; Wang, Z. Deep Convolutional Neural Networks for Image Classiﬁcation: A Comprehensive Review. Neural Comput. 2017, 29, 1–98. [CrossRef] 42. Kohonen, T. Self-Organizing Maps, 3rd ed.; Springer: Berlin/Heidelberg, Germany, 2001. 43. Kohonen, T. MATLAB Implementations and Applications of the Self-Organizing Map; Unigraﬁa Oy: Helsinki, Finland, 2014. 44. Zimmermann, R.; Brandmeier, M.; Andreani, L.; Mhopjeni, K.; Gloaguen, R. Remote Sensing Exploration of Nb-Ta-LREE-Enriched Carbonatite (Epembe/Namibia). Remote Sens. 2016, 8, 620. [CrossRef] 45. Kim, S.; Singh, V.P. Spatial Disaggregation of Areal Rainfall Using Two Dierent Artiﬁcial Neural Networks Models. Water 2015, 7, 2707–2727. [CrossRef] 46. Chi, D. Self-Organizing Map-Based Color Image Segmentation with k-Means Clustering and Saliency Map. ISRN Signal Process. 2011, 2011. [CrossRef] 47. Xu, Y.; Ran, J.; Chen, H. Kohonen Neural Network Classiﬁcation for Failure Process of Metallic Organic Coating in Corrosion Environment. Metals 2017, 7, 147. 48. Ganguli, R. Gas Turbine Diagnostics: Signal Processing and Fault Isolation; CRC Press: Boca Raton, FL, USA, 49. Li, Y.G. Performance Analysis Based Gas Turbine Diagnostics: A Review. Proc. Inst. Mech. Eng. Part A J. Power Energy 2002, 216, 363–377. [CrossRef] 50. Mohammadi, E.; Montazeri-Gh, M. Performance Enhancement of Global Optimization-Based Gas Turbine Fault Diagnosis Systems. J. Propuls. Power 2016, 32, 214–224. [CrossRef] 51. Li, Z.; Zhong, S.-S.; Lin, L. Novel Gas Turbine Fault Diagnosis Method Based on Performance Deviation Model. J. Propuls. Power 2017, 33, 730–739. [CrossRef] Appl. Sci. 2019, 9, 2253 22 of 22 52. Meskin, N.; Naderi, E.; Khorasani, K. A Multiple Model-Based Approach for Fault Diagnosis of Jet Engines. IEEE Trans. Control Syst. Technol. 2013, 21, 254–262. [CrossRef] 53. Vanini, Z.N.S.; Khorasani, K.; Meskin, N. Fault detection and isolation of a dual spool gas turbine engine using dynamic neural networks and multiple model approach. Inf. Sci. 2014, 259, 234–251. [CrossRef] 54. Szentkuti, A.; Kavanagh, H.S.; Grazio, S. Infrared thermography and image analysis for biomedical use. Period. Biol. 2011, 113, 385–392. 55. Rojas, R. Neural Networks—A Systematic Introduction; Springer: Berlin, Germany; New York, NY, USA, 1996. 56. Abraham, A. Handbook of Measuring System: Design Rule-Based Expert Systems; John Wiley & Sons: New York, NY, USA, 2005. 57. Buchanan, B.G.; Smith, R.G. Fundamentals of Expert Systems. Annu. Rev. Comput. Sci. 1988, 3, 23–58. [CrossRef] 58. Yang, J.; Ye, C.; Zhang, X. An expert system shell for fault diagnosis. Robotica 2001, 19, 669–674. [CrossRef] 59. Chlebus, E.; Krot, K.; Kuliberda, M. Hybrid Artiﬁcial Intelligent Systems: Rule-Based Expert System Dedicated for Technological Applications; Springer: Berlin/Heidelberg, Germany, 2011; pp. 373–380. 60. Dvor ˇák, J. Expertní Systémy; Vysoké Ucen ˇ í Technické v Brne: 2004. Available online: http://www.uai.fme. vutbr.cz/~{}jdvorak/Opory/ExpertniSystemy.pdf (accessed on 22 February 2019). 61. Sinha, A.P.; Zhao, H. Tuning Expert Systems for Cost-Sensitive Decisions. Adv. Artif. Intell. 2011, 2011. [CrossRef] 62. Andoga, R.; Foz ˝ o, ˝ L.; Judic ˇák, J.; Bréda, R.; Szabo, S.; Rozenberg, R.; Džunda, M. Intelligent Situational Control of Small Turbojet Engines. Int. J. Aerosp. Eng. 2018, 2018. [CrossRef] 63. Cruz-Albarran, I.A.; Benitez-Rangel, J.P.; Osornio-Rios, R.; Dominguez, B.T.; Medina, D.A.R.; Morales-Hernández, L.A. A new approach to obtain a colour palette in thermographic images. Quant. InfraRed Thermogr. J. 2019, 16, 35–54. [CrossRef] 64. Roman, R.-C.; Radac, M.-B.; Precup, R.-E.; Petriu, E.M. Data-driven Model-Free Adaptive Control Tuned by Virtual Reference Feedback Tuning. Acta Polytech. Hung. 2016, 13, 83–96. [CrossRef] 65. Andoga, R.; Adamc ˇík, F.; Hrabovský, J.; Vaispacher, T. A hybrid diagnostic system for a small turbojet engine. NASE More 2016, 63, 86–92. [CrossRef] © 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/). http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Applied Sciences Multidisciplinary Digital Publishing Institute http://www.deepdyve.com/lp/multidisciplinary-digital-publishing-institute/intelligent-thermal-imaging-based-diagnostics-of-turbojet-engines-E8zmnPGwXx

Loading next page...

References (48)

Roque Osornio-Rios, J. Antonino-Daviu, R. Romero-Troncoso (2019)
Recent Industrial Applications of Infrared Thermography: A Review
IEEE Transactions on Industrial Informatics, 15
Haldo Sponton, J. Cardelino (2015)
A Review of Classic Edge Detectors
Image Process. Line, 5
M. Egmont-Petersen, D. Ridder, H. Handels (2002)
Image processing with neural networks - a review
Pattern Recognit., 35
Yi-Guang Li (2002)
Performance-analysis-based gas turbine diagnostics: A review
Proceedings of the Institution of Mechanical Engineers, Part A: Journal of Power and Energy, 216
D. López-Pérez, J. Antonino-Daviu (2017)
Application of Infrared Thermography to Failure Detection in Industrial Induction Motors: Case Stories
IEEE Transactions on Industry Applications, 53
D. Chi (2011)
Self-Organizing Map-Based Color Image Segmentation with k-Means Clustering and Saliency Map
International Scholarly Research Notices, 2011
Atish Sinha, Huimin Zhao (2011)
Tuning Expert Systems for Cost-Sensitive Decisions
Adv. Artif. Intell., 2011
M. Harvey, J. Rowland, K. Luketina (2016)
Drone with thermal infrared camera provides high resolution georeferenced imagery of the Waikite geothermal area, New Zealand
Journal of Volcanology and Geothermal Research, 325
A. Lisowska-Lis (2017)
Thermographic monitoring of the power transformers
Measurement Automation Monitoring
R. Andoga, F. Adamčík, J. Hrabovský, T. Vaispacher (2016)
A hybrid diagnostic system for a small turbojet engine
, 63
V. Vavilov (2003)
Non-contact one-sided evaluation of hidden corrosion in metallic constructions by using transient infrared thermography
Revista De Metalurgia, 39
A. Kylili, P. Fokaides, P. Christou, S. Kalogirou (2014)
Infrared thermography (IRT) applications for building diagnostics: A review
Applied Energy, 134
V. Vavilov, Yangyang Pan, D. Nesteruk (2016)
Infrared thermographic inspection of water ingress in composite honeycomb panels.
Applied optics, 55 34
Yuanming Xu, Junshuang Ran, Hao Chen (2017)
Kohonen Neural Network Classification for Failure Process of Metallic Organic Coating in Corrosion Environment
, 7
E. Mohammadi, M. Montazeri-Gh (2016)
Performance Enhancement of Global Optimization-Based Gas Turbine Fault Diagnosis Systems
Journal of Propulsion and Power, 32
Xinyi Yang, Shan Pang, Wei Shen, Xuesen Lin, Keyi Jiang, W. Yonghua (2016)
Aero Engine Fault Diagnosis Using an Optimized Extreme Learning Machine
International Journal of Aerospace Engineering, 2016
J. White, E. Tucholski, R. Green (2003)
Nondestructive testing of aircraft and spacecraft wiring
Materials evaluation, 61
Zhen Li, S. Zhong, Lin Lin (2017)
Novel Gas Turbine Fault Diagnosis Method Based on Performance Deviation Model
Journal of Propulsion and Power, 33
C. Balaras, A. Argiriou (2002)
Infrared thermography for building diagnostics
Energy and Buildings, 34
T. Omar, M. Nehdi (2017)
Remote sensing of concrete bridge decks using unmanned aerial vehicle infrared thermography
Automation in Construction, 83
Waseem Rawat, Zenghui Wang (2017)
Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review
Neural Computation, 29
Nida Zaitoun, M. Aqel (2015)
Survey on Image Segmentation Techniques
Procedia Computer Science, 65
M. Bogdan, J. Błachnio, A. Kułaszka, M. Derlatka (2019)
Assessing the Condition of Gas Turbine Rotor Blades with the Optoelectronic and Thermographic Methods
Metals
S. Addepalli, Yifan Zhao, R. Roy, Wathsala Galhenege, M. Colle, Jingjing Yu, Aziz Ucur (2019)
Non-destructive evaluation of localised heat damage occurring in carbon composites using thermography and thermal diffusivity measurement
Measurement
R. Zimmermann, M. Brandmeier, L. Andreani, Kombada Mhopjeni, R. Gloaguen (2016)
Remote Sensing Exploration of Nb-Ta-LREE-Enriched Carbonatite (Epembe/Namibia)
Remote. Sens., 8
J. Macleod, P. Steckhan, D. He (1994)
Infrared Thermal Imaging System as a Diagnostic Tool for Gas Turbine Engine Faults
Sasan Karamizadeh, Shahidan Abdullah, M. Zamani, Atabak Kherikhah (2014)
Pattern Recognition Techniques: Studies on Appropriate Classifications
ARPN journal of engineering and applied sciences, 9
M. Štumper, J. Kraus (2015)
Thermal Imaging in Aviation
, 3
R. Vardasca, L. Vaz, J. Mendes (2018)
Classification and Decision Making of Medical Infrared Thermal Images
Nidhi Baranwal, S. Mahulikar (2016)
Infrared Signature of Aircraft Engine with Choked Converging Nozzle
Journal of Thermophysics and Heat Transfer, 30
Efstratios Ntantis, P. Botsaris (2015)
Diagnostic Methods for an Aircraft Engine Performance
Journal of Engineering Science and Technology Review, 8
Ruizhen Yang, Yunze He, A. Mandelis, Nichen Wang, Xuan Wu, Shoudao Huang (2018)
Induction Infrared Thermography and Thermal-Wave-Radar Analysis for Imaging Inspection and Diagnosis of Blade Composites
IEEE Transactions on Industrial Informatics, 14
N. Meskin, E. Naderi, K. Khorasani (2013)
A Multiple Model-Based Approach for Fault Diagnosis of Jet Engines
IEEE Transactions on Control Systems Technology, 21
Sungwon Kim, V. Singh (2015)
Spatial Disaggregation of Areal Rainfall Using Two Different Artificial Neural Networks Models
Water, 7
B. Lahiri, S. Bagavathiappan, T. Jayakumar, J. Philip (2012)
Medical applications of infrared thermography: A review
Infrared Physics & Technology, 55
V. Vavilov, D. Nesteruk (2004)
Detecting water in aviation honeycomb structures: the quantitative approach
Quantitative InfraRed Thermography Journal, 1
L. Matikainen, M. Lehtomäki, E. Ahokas, J. Hyyppä, M. Karjalainen, A. Jaakkola, A. Kukko, Tero Heinonen (2016)
Remote sensing methods for power line corridor surveys
Isprs Journal of Photogrammetry and Remote Sensing, 119
Jie Yang, C. Ye, Xiaoli Zhang (2001)
An Expert System Shell for Fault Diagnosis
Robotica, 19
Cristina Gómez, D. Green (2017)
Small unmanned airborne systems to support oil and gas pipeline monitoring and mapping
Arabian Journal of Geosciences, 10
A. Szentkuti, H. Kavanagh, S. Grazio (2011)
Infrared thermography and image analysis for biomedical use
Periodicum Biologorum, 113
(2017)
Engine Health Condition Monitoring—A Brief Review
Yizhe Wang, B. Gao, W. Woo, G. Tian, X. Maldague, Li Zheng, Zheyou Guo, Yuyu Zhu (2018)
Thermal Pattern Contrast Diagnostic of Microcracks With Induction Thermography for Aircraft Braking Components
IEEE Transactions on Industrial Informatics, 14
F. Ciampa, P. Mahmoodi, F. Pinto, M. Meo (2018)
Recent Advances in Active Infrared Thermography for Non-Destructive Testing of Aerospace Components
Sensors (Basel, Switzerland), 18
I. Cruz-Albarrán, J. Benitez-Rangel, R. Osornio-Ríos, B. Domínguez-Trejo, D. Rodriguez-Medina, L. Morales-Hernández (2018)
A new approach to obtain a colour palette in thermographic images
Quantitative InfraRed Thermography Journal, 16
B. Orton, M. Copaul, C. Velez-Ospina (1999)
The potential use of infrared thermography to detect aircraft pressurisation leaks
Insight, 41
R. Andoga, L. Főző, J. Judičák, R. Bréda, S. Szabo, R. Rozenberg, M. Džunda (2018)
Intelligent Situational Control of Small Turbojet Engines
International Journal of Aerospace Engineering
B. Buchanan, Reid Smith (1988)
Fundamentals of expert systems
, 3
Z. Vanini, K. Khorasani, N. Meskin (2014)
Fault detection and isolation of a dual spool gas turbine engine using dynamic neural networks and multiple model approach
Inf. Sci., 259

Publisher: Multidisciplinary Digital Publishing Institute
Copyright: © 1996-2019 MDPI (Basel, Switzerland) unless otherwise stated
ISSN: 2076-3417
DOI: 10.3390/app9112253
Publisher site: See Article on Publisher Site

Abstract

applied sciences Article Intelligent Thermal Imaging-Based Diagnostics of Turbojet Engines Rudolf Andoga * , Ladislav Foz ˝ o ˝ , Martin Schrötter, Marek Ceškovic, ˇ Stanislav Szabo, Róbert Bréda and Michal Schreiner Technical University of Košice, Faculty of Aeronautics, Rampová 7, 041 21 Košice, Slovakia; ladislav.fozo@tuke.sk (L.F.); martin.schrotter@tuke.sk (M.S.); marek.ceskovic@tuke.sk (M.C.); stanislav.szabo@tuke.sk (S.S.); robert.breda@tuke.sk (R.B.); michal.schreiner@tuke.sk (M.S.) * Correspondence: rudolf.andoga@tuke.sk; Tel.: +421-55-602-6140 Received: 2 April 2019; Accepted: 27 May 2019; Published: 31 May 2019 Featured Application: The methodology presented in this paper is directly intended for application in the area of turbojet engines. This paper presents the design of an automated system which can create diagnostic information using data from an infrared thermal camera, utilizing methods of computational intelligence and expert systems, which can be considered as a novel approach. The developed system segments infrared images use a self-organizing feature map and an expert system to automatically extract diagnostic information about the technical state of a turbojet engine. The diagnostic system was developed using a small iSTC-21v turbojet engine with a thrust of 500 N in laboratory conditions and can be simply modiﬁed for application to other engine types. The system is not limited to engines and can be considered for application in other technical systems which produce heat, such as generators, transmissions, and electronic circuit boards. Abstract: There are only a few applications of infrared thermal imaging in aviation. In the area of turbojet engines, infrared imaging has been used to detect temperature ﬁeld anomalies in order to identify structural defects in the materials of engine casings or other engine parts. In aviation applications, the evaluation of infrared images is usually performed manually by an expert. This paper deals with the design of an automatic intelligent system which evaluates the technical state and diagnoses a turbojet engine during its operation based on infrared thermal (IRT) images. A hybrid system interconnecting a self-organizing feature map and an expert system is designed for this purpose. A Kohonen neural network (the self-organizing feature map) is successfully applied to segment IRT images of a turbojet engine with high precision, and the expert system is then used to create diagnostic information from the segmented images. This paper represents a proof of concept of this hybrid system using data from a small iSTC-21v turbojet engine operating in laboratory conditions. Keywords: expert systems; infrared thermography; intelligent diagnostics; image segmentation; neural networks; thermal imaging camera; turbojet engines 1. Introduction Infrared thermography (IRT) is one of the many diagnostic methodologies currently applied in various types of industry and research [1]. It can be used to detect so-called invisible defects in a non-destructive way. Compared to traditional areas of IRT applications, such as biomedical applications [2], energetics, buildings, and other constructions [3–6], the application of this methodology in aviation is still at an early stage [1]. Appl. Sci. 2019, 9, 2253; doi:10.3390/app9112253 www.mdpi.com/journal/applsci Appl. Sci. 2019, 9, 2253 2 of 22 The advantages of IRT have been fully demonstrated in pre-ﬂight and post-ﬂight aircraft checks, where damage (e.g., water in honeycomb structures of composite parts), is monitored by thermal imaging cameras [7,8]. In aviation, extensive research and development is being conducted on the IRT diagnostics of corrosion in aluminum materials [9,10], and especially composite structures [11–13]. In aircraft operation, it is possible to use IRT to detect air leaks inside the cabin [14], abnormal strain in individual parts of aircraft casings, fuselage, and wheel attachments [15], damage to electrical wiring and insulation [16,17], etc. In aviation, IRT is also used for rescue, ﬁreﬁghting, and surveillance operations [18,19]. Other applications of IRT diagnostics can be found in the area of unmanned vehicles, whose thermal imaging cameras are used to inspect power lines [20], pipelines [21], buildings [22], constructions [23], vegetation [24], etc. Thermography has also recently found an application in the design of stealth aircraft [25]. For the diagnostics of aviation engines, IRT is mainly used to check their composite covers [26]. Another area where IRT is applied, which is closer to the topic of this paper, is the diagnostics of turbojet engines; infrared images of the diagnosed engines’ gas turbines are compared to standard IRT images of engines without defects, which allows temperature ﬁeld anomalies to be detected and thus allows defects to be identiﬁed [27]. These defects can be damage to the turbine blades, combustion chamber disturbance, disturbance to the homogeneous ﬁeld of fuel burning, etc. [28]. Aircraft engine diagnostics and health monitoring [29–31] can be divided into two basic processes: ground diagnostics and on-board diagnostics with their associated systems. On-board, or online, diagnostics involve the checking of engine components and thermodynamic processes during the operation of the engine [32]. These components include individual engine parts, such as compressors, combustion chambers, and turbines and associated accessories, while thermodynamic process checks include reviewing parameters such as temperature, pressure, rotational turbo-compressor speed, and vibrations. Ground, or oine, diagnostics [33,34] involve tribo-technical analysis, non-destructive checking methods, and checks by thermal cameras as mentioned earlier. This article aims to develop a progressive method for the online evaluation of the technical state of a turbojet engine using infrared thermal imaging. Contrary to traditional methods, in which a comparative analysis of thermal images is performed on some normalized samples, this paper proposes an intelligent adaptive method which can discriminate anomalous areas of thermal images and create resulting diagnostic information automatically without any human input or previous training. Since the output of a thermal imaging camera is a digital image, computer vision techniques and methodologies should be used to automatically process this output. In order to extract meaningful information from a thermal image for a diagnostic system, a pattern recognition/classiﬁcation approach has to be applied. Many methodologies and an array of methods can be used for this purpose [35,36]. These include traditional methodologies, which perform pattern recognition using deterministic image-processing methods such as ﬁltering, feature analysis, and support vector machines. [37]. However, the most successful methodologies for image classiﬁcation come from the area of computational intelligence. These include artiﬁcial neural networks, which have been widely and successfully applied in image processing, pattern recognition, and diagnostic tasks [38,39]. The most complex neural networks, such as convolutional neural networks, are able to classify any pattern in an image with high precision, as well as detect objects [40]. However, these approaches are computationally complex and require large amounts of data and training [41]. In order to create a successful diagnostic method for pattern recognition in IRT images, a simpler methodology should be used, since such images are more homogeneous than traditional images; they have fewer colors, lower resolution, and are well structured, with well-deﬁned edges. The motivation for this study was to develop a methodology which does not require a large amount of computationally intensive training to successfully segment infrared images using intelligent adaptive algorithms. Accordingly, in this paper, the ﬁeld of neural networks with uncontrolled learning is explored. One of the methods that has been successfully used in pattern recognition and image segmentation is self-organizing feature maps (SOFMs), which comply with the demand for simplicity, uncontrolled Appl. Sci. 2019, 9, 2253 3 of 22 learning, and good classiﬁcation results. SOFMs can be transformed and formalized into the form of a Kohonen neural network [42,43]. Kohonen neural networks have been successfully applied in image segmentation tasks, the detection of speciﬁc areas in geographic maps [44–46], and the detection of cracks and corrosion in materials [47]. The results obtained from these applications represent a solid foundation and an inspiration for this approach to be applied in the segmentation of infrared images for the detection of anomalies in the temperature ﬁelds of dierent objects. The results of pattern recognition and image segmentation come in the form of classes present in the image. In order to extract knowledge from this information, the classes have to be interpreted. This can be done by simply labeling the classes; however, in the ﬁeld of technical systems, this may not be sucient. In technical objects, some areas are naturally hot and some are naturally cold, and therefore there needs to be some inference system which interprets the displacement and size of the classes from thermal images, and generates diagnostic information, which in the simplest case can be the determination of whether the diagnosed object is in a normal or an abnormal operational state [48,49]. The traditional approach is to consider a diagnosed system as healthy if it operates within some speciﬁc limits; otherwise, a failure is indicated [48,49]. This is a natural approach, and can be used when dealing with a technical system where parameters which determine the technical state (temperature, vibrations, etc.) can be simply measured, such as a jet engine. However, to extract diagnostic information from a classiﬁed image, a more complex approach needs to be used [49,50]. Diagnostic expert systems are a natural candidate for extracting diagnostic information from complex data, and have been applied in the diagnostics of turbojet engines [49,50]. The aim of the present study is to create a specialized approach for the creation of a rule-base of an expert system which can transform class information in an infrared image into useful diagnostic information, thus creating hybrid architecture. The functionality of this approach has been evaluated and designed speciﬁcally for application to turbojet engines, and the approach can be considered as novel compared to other intelligent approaches in turbojet engine diagnostics, which are applied to data measured by the engines’ sensors and do not deal with visual/infrared image information [51–53]. The aim of this paper is to explore the design of this new approach and prove its functionality by applying thermal vision-based diagnostics in an automated adaptive system in the area of turbojet engine diagnostics. The approach can be expanded and applied to almost any technical system which has been observed by an infrared camera. The hybrid architecture of our approach, which combines a Kohonen neural network and a specialized expert system with a designed rule-base, can be considered as a novel and original contribution of our research. A proof of concept is presented using real data from an iSTC-21v turbojet engine tested and operated in laboratory conditions. 2. Methodology 2.1. Using Infrared Imaging for Diagnostics Every object whose temperature is above absolute zero radiates thermal energy. IRT systems measure this energy to create thermal infrared images. These images can be used to identify heat radiation anomalies of an object, which are usually characterized by increases or decreases in the radiated temperature [8,9,13]. The basic principle of IRT diagnostics is shown in Figure 1, with the following general meaning: Image: an object to be captured by a thermal camera. Classiﬁcation: monitoring the object, scanning it, obtaining infrared data, classifying hot and cold parts as well as possible problematic areas. Diagnostic information: evaluating infrared thermal images, for example, identifying defects or deﬁciencies, deciding about the technical state of the engine. Appl. Sci. 2019, 9, x FOR PEER REVIEW 4 of 23 Appl. Sci. 2019, 9, 2253 4 of 22 Figure 1. Procedure for the detection of temperature anomalies in engine casings by analyzing an Figure 1. Procedure for the detection of temperature anomalies in engine casings by analyzing an infrared image. infrared image. The aforementioned principle is a standard principle when using human expertise to obtain The aforementioned principle is a standard principle when using human expertise to obtain diagnostic information from infrared images. Examples of applications in aviation are pre-ﬂight and diagnostic information from infrared images. Examples of applications in aviation are pre-flight and post-ﬂight engine checks, where the engine’s casing is checked for defects and cracks caused by freezing post-flight engine checks, where the engine’s casing is checked for defects and cracks caused by water or material deﬁciencies. In such cases, the evaluation of thermal images and the detection of freezing water or material deficiencies. In such cases, the evaluation of thermal images and the anomalies is strictly dependent on human expert knowledge [49]. detection of anomalies is strictly dependent on human expert knowledge [49]. When a human expert inspects infrared images of a technical object, he or she basically performs When a human expert inspects infrared images of a technical object, he or she basically two operations. First, he/she discriminates dierent areas of the thermal image and classiﬁes them performs two operations. First, he/she discriminates different areas of the thermal image and into categories according to their temperature. They next use their expert knowledge to inspect classifies them into categories according to their temperature. They next use their expert knowledge hot or cold areas for some anomalies according to their position and temperature. The traditional to inspect hot or cold areas for some anomalies according to their position and temperature. The approach is to superimpose the obtained infrared image over a standardized infrared image of an traditional approach is to superimpose the obtained infrared image over a standardized infrared object without defects and evaluate the dierences between the two images [54]. The whole process image of an object without defects and evaluate the differences between the two images [54]. The can therefore be formalized into two basic parts—image processing and classiﬁcation, and knowledge whole process can therefore be formalized into two basic parts—image processing and classification, acquisition. A variety of methods exist to solve both problems, as mentioned in the introduction, and knowledge acquisition. A variety of methods exist to solve both problems, as mentioned in the and are suitable to identify and evaluate anomalies. As previously stated, one aim of this study was introduction, and are suitable to identify and evaluate anomalies. As previously stated, one aim of to develop an autonomous diagnostic system which is able to learn and locate anomalies without this study was to develop an autonomous diagnostic system which is able to learn and locate any prior knowledge (e.g., normal faultless operation IRT images). The other aim was to make the anomalies without any prior knowledge (e.g., normal faultless operation IRT images). The other aim system autonomous. was to make the system autonomous. The general concept of the intelligent diagnostic system can be expressed by the scheme shown in The general concept of the intelligent diagnostic system can be expressed by the scheme shown Figure 2. The data acquisition part of the approach is usually done using the infrared camera, and in Figure 2. The data acquisition part of the approach is usually done using the infrared camera, and the image pre-processing can be done before the data are fed into a classiﬁer. The eects of the image the image pre-processing can be done before the data are fed into a classifier. The effects of the image pre-processing and ﬁltering are not directly investigated in this study; however, the methodology is pre-processing and filtering are not directly investigated in this study; however, the methodology is designed to be relatively robust in order to use image data of reasonable quality with low noise levels. designed to be relatively robust in order to use image data of reasonable quality with low noise Infrared cameras usually pre-process the image data in order to compensate for air humidity and levels. Infrared cameras usually pre-process the image data in order to compensate for air humidity other environmental eects. The robustness is achieved by the application of an intelligent adaptive and other environmental effects. The robustness is achieved by the application of an intelligent algorithm, and the design therefore consists mainly of the classiﬁer part by ﬁnding a methodology and adaptive algorithm, and the design therefore consists mainly of the classifier part by finding a selecting its structure, input/output relations, and the training algorithm in the case of the application methodology and selecting its structure, input/output relations, and the training algorithm in the of neural networks. Information about classes in the thermal image can be further used to evaluate the case of the application of neural networks. Information about classes in the thermal image can be technical state of the investigated object (in this case, a jet engine), and a diagnostic expert system is a further used to evaluate the technical state of the investigated object (in this case, a jet engine), and a natural candidate for the completion of this task. The basis of the proposed methodology is therefore diagnostic expert system is a natural candidate for the completion of this task. The basis of the centered around: proposed methodology is therefore centered around: Image processing classiﬁcation system: involving a SOFM. • Image processing classification system: involving a SOFM. Knowledge acquisition system: an expert system for the evaluation of the technical state of • Knowledge acquisition system: an expert system for the evaluation of the technical state of an an engine. engine. Appl. Sci. 2019, 9, x FOR PEER REVIEW 5 of 23 The other advantage of the designed system is that it does not use the absolute temperature data obtained from the camera, which are dependent on many environmental and material factors, Appl. Sci. 2019, 9, x FOR PEER REVIEW 5 of 23 but rather only uses image data, which describe relative temperatures and relations as well as the Appl. Sci. 2019, 9, 2253 5 of 22 displacement of the temperature field of the investigated object. The other advantage of the designed system is that it does not use the absolute temperature data obtained from the camera, which are dependent on many environmental and material factors, but rather only uses image data, which describe relative temperatures and relations as well as the displacement of the temperature field of the investigated object. Figure 2. The conceptual design of the diagnostic system. Figure 2. The conceptual design of the diagnostic system. The other advantage of the designed system is that it does not use the absolute temperature 2.2. Self-Organizing Feature Maps in Image Segmentation and Classification data obtained from the camera, which are dependent on many environmental and material factors, SOFMs are characterized by their outstanding properties in clustering (classification), data but rather only uses image Figure 2. data, which The conceptu describe al de rsi elative gn of the d temperatur iagnostices syst and em.r elations as well as the visualization, probability density estimation, etc. Kohonen artificial neural networks are one of the displacement of the temperature ﬁeld of the investigated object. most widely used SOFMs [42]. 2.2. Self-Organizing Feature Maps in Image Segmentation and Classification 2.2. Self-Organizing Feature Maps in Image Segmentation and Classiﬁcation Kohonen artificial neural networks (KNN) map a high-dimensional input space/layer (xi for i = SOFMs are characterized by their outstanding properties in clustering (classification), data 1, 2, ..., n, where n is the number of inputs) onto an output space/layer (Nj = 1, 2, ..., m, where m is the SOFMs are characterized by their outstanding properties in clustering (classiﬁcation), data visualization, probability density estimation, etc. Kohonen artificial neural networks are one of the number of neurons) with fewer or an equal number of dimensions. KNN contains only a single layer visualization, probability density estimation, etc. Kohonen artiﬁcial neural networks are one of the most widely used SOFMs [42]. of neurons—the Kohonen layer. The number of neurons n, which is the network’s parameter, is most widely used SOFMs [42]. Kohonen artificial neural networks (KNN) map a high-dimensional input space/layer (xi for i = optional and usually equals the number of target classes. The inputs to the network are Kohonen artiﬁcial neural networks (KNN) map a high-dimensional input space/layer (x for i = 1, 1, 2, ..., n, where n is the number of inputs) onto an output space/layer (Nj = 1, 2, ..., m, where m is the interconnected with all neurons in the Kohonen layer. This means that every neuron has information 2, ..., n, where n is the number of inputs) onto an output space/layer (N = 1, 2, ..., m, where m is the number of neurons) with fewer or an equal number of dimensions. KNN contain j s only a single layer from all the inputs. Synaptic weights leading to each neuron (designated as a weight vector, wij, of number of neurons) with fewer or an equal number of dimensions. KNN contains only a single layer of of neurons—the Kohonen layer. The number of neurons n, which is the network’s parameter, is the neuron) can be understood as coordinates, which indicate the location of the neuron in the array neurons—the Kohonen layer. The number of neurons n, which is the network’s parameter, is optional optional and usually equals the number of target classes. The inputs to the network are (Figure 3). Neurons are mostly organized in a regular two-dimensional grid/matrix or and usually equals the number of target classes. The inputs to the network are interconnected with interconnected with all neurons in the Kohonen layer. This means that every neuron has information one-dimensional row/vector array. all neurons in the Kohonen layer. This means that every neuron has information from all the inputs. from all the inputs. Synaptic weights leading to each neuron (designated as a weight vector, wij, of Synaptic weights leading to each neuron (designated as a weight vector, w , of the neuron) can be ij the neuron) can be understood as coordinates, which indicate the location of the neuron in the array understood as coordinates, which indicate the location of the neuron in the array (Figure 3). Neurons (Figure 3). Neurons are mostly organized in a regular two-dimensional grid/matrix or are mostly organized in a regular two-dimensional grid/matrix or one-dimensional row/vector array. one-dimensional row/vector array. Figure 3. The structure of a Kohonen network. Figure 3. The structure of a Kohonen network. Figure 3. The structure of a Kohonen network. Appl. Sci. 2019, 9, 2253 6 of 22 Appl. Sci. 2019, 9, x FOR PEER REVIEW 6 of 23 There are lateral links (local connections) between the neurons in the Kohonen layer (dashed line There are lateral links (local connections) between the neurons in the Kohonen layer (dashed in Figure 3), which means that any particular neuron aects other neurons in its neighborhood. line in Figure 3), which means that any particular neuron affects other neurons in its neighborhood. In order to segment an IRT image, a layer that has 1 m neurons (Figure 4) is proposed. Using In order to segment an IRT image, a layer that has 1×𝑚 neurons (Figure 4) is proposed. Using this type of array, it is possible to segment thermal infrared images into m dierent segments, which this type of array, it is possible to segment thermal infrared images into m different segments, which can be labeled as dierent areas of the temperature ﬁeld (cold, hot, etc.). Thus, if a higher number of can be labeled as different areas of the temperature field (cold, hot, etc.). Thus, if a higher number of neurons is used, the classiﬁcation will be more detailed. neurons is used, the classification will be more detailed. Figure 4. A single-row structure of the Kohonen layer. Figure 4. A single-row structure of the Kohonen layer. In the most simpliﬁed way, the self-organization process (Kohonen learning) consists of four main In the most simplified way, the self-organization process (Kohonen learning) consists of four steps: initialization, competition, cooperation, and adaptation [55]: main steps: initialization, competition, cooperation, and adaptation [55]: Initialization: Random values are set for all neuron weights. The input vector from the input • Initialization: Random values are set for all neuron weights. The input vector from the input space is selected. space is selected. Competition: From the inputs, the discriminant function (Equation (1), below) of a particular • Competition: From the inputs, the discriminant function (Equation (1), below) of a particular neuron is computed. A Kohonen unit computes the Euclidean distance between an input neuron is computed. A Kohonen unit computes the Euclidean distance between an input xi and x and its weight vector w . The winning neuron is the one with the smallest value of the i ij its weight vector wij. The winning neuron is the one with the smallest value of the discriminant discriminant function. function. Cooperation: The winning neuron inﬂuences its adjacent neurons (i.e., the weight vectors of • Cooperation: The winning neuron influences its adjacent neurons (i.e., the weight vectors of adjacent neurons are inﬂuenced), however this inﬂuence decreases with increasing distance to adjacent neurons are influenced), however this influence decreases with increasing distance to other neurons. other neurons. • Adaptation: Adaptation: The The excited n excited neur eurons d ons decr ecre ease ase their values their values of of the di the discriminant scriminant ffunction unction iin n rel relation ation to to the i then input put pa pattern ttern through througha adjustment djustment of of the the associated associated weights. weights. The The response response of of the w the winning inning neur neuron to th on to the subsequent e subsequent application application of aof similar a siminput ilar inpu pattern t pattern i is enhanced. s enhanced. The pr The ocess process wi will stop ll ifstop if the maximum number the maximum number of iterations of iterat ision reached. s is reached. m 2 𝑑 = 𝑥 −𝑤 (1) d = x w (1) j i i j i=1 The The sel self-or f-o ganization rganization proc process ess isis graphically graphically d depicted epicted in in Figur Figu ere 5,5wher , where b e blue lue dots dots repr repre esent sent neur neurons, green dots rep ons, green dots representresent input input vectors, vectors, and red an lines d red lines represent c represent connections between onnections b neighboring etween neur neighborin ons. Theg temperatur neurons. The temperature e ﬁeld has ﬁve arfield h eas, which as five are are indicated as, which byare ﬁve indic neur aons. ted bIn y fiv this e ne simplistic urons. In example, this simp the list labeling ic example is performed , the labeby lincoloring g is performed b the neurons y color andithe ng t corr he neurons esponding an training d the corresp data, which onding activates training da the ta neur , whi ons. ch a Incti this vates the neurons. In thi manner, the blue areasafter manner, the bl adaptationue a repr resents ea after ada the coldest ptation rep area in resents the temperatur the coldest e ﬁeld area in the tem and the red perature area the field hottest andar the red ea. area the hottest area. Appl. Appl. Sci. Sci. 2019 2019 , 9 , ,92253 , x FOR PEER REVIEW 77 of of 22 23 Appl. Sci. 2019, 9, x FOR PEER REVIEW 7 of 23 Figure 5. The self-organization process with a single-row structure of the Kohonen layer. Figure Figure 5. 5. The The self-organizatio self-organizationn process with process with a a sing single-r le-row ow str structure of the ucture of the Kohonen Kohonen layer. layer. 2.3. Expert Diagnostic Systems 2.3. Expert Diagnostic Systems 2.3. Expert Diagnostic Systems An expert system consists of a knowledge base, an inference engine (solving or deducing), an An expert system consists of a knowledge base, an inference engine (solving or deducing), an input/output data interface (user interface or links to other systems and tools), a working memory, An expert system consists of a knowledge base, an inference engine (solving or deducing), an input/output data interface (user interface or links to other systems and tools), a working memory, and and an explaining or communication module in the form of a user interface. These elements are input/output data interface (user interface or links to other systems and tools), a working memory, an explaining or communication module in the form of a user interface. These elements are shown in shown in Figure 6. A very important part of every expert system is its inference engine, which uses and an explaining or communication module in the form of a user interface. These elements are Figure 6. A very important part of every expert system is its inference engine, which uses data from data from the knowledge base (knowledge gained from an expert) and specific problem-oriented shown in Figure 6. A very important part of every expert system is its inference engine, which uses the knowledge base (knowledge gained from an expert) and speciﬁc problem-oriented data [56]. data [56]. data from the knowledge base (knowledge gained from an expert) and specific problem-oriented data [56]. Figure 6. The structure of expert systems and their integration (reproduced from [57]). Figure 6. The structure of expert systems and their integration (reproduced from [57]). The inference engine and the knowledge base represent the core of every expert system. The Figure 6. The structure of expert systems and their integration (reproduced from [57]). The inference engine and the knowledge base represent the core of every expert system. The inference engine provides the mathematical means to evaluate the rules in the knowledge base to reach inference engine provides the mathematical means to evaluate the rules in the knowledge base to The inference engine and the knowledge base represent the core of every expert system. The a conclusion about the state of the diagnosed object [58–60]. In applications for turbojet engines, the reach a conclusion about the state of the diagnosed object [58–60]. In applications for turbojet inference engine provides the mathematical means to evaluate the rules in the knowledge base to knowledge base contains rules deﬁning the technical state of the turbojet engine based on the outputs engines, the knowledge base contains rules defining the technical state of the turbojet engine based reach a conclusion about the state of the diagnosed object [58–60]. In applications for turbojet from the segmentation algorithm (i.e., the Kohonen neural network). The design of the diagnostic on the outputs from the segmentation algorithm (i.e., the Kohonen neural network). The design of engines, the knowledge base contains rules defining the technical state of the turbojet engine based system interconnected with a SOFM, which was used in the present study, is shown in Figure 7. the diagnostic system interconnected with a SOFM, which was used in the present study, is shown on the outputs from the segmentation algorithm (i.e., the Kohonen neural network). The design of in Figure 7. the diagnostic system interconnected with a SOFM, which was used in the present study, is shown in Figure 7. Appl. Sci. 2019, 9, 2253 8 of 22 Appl. Sci. 2019, 9, x FOR PEER REVIEW 8 of 23 Appl. Sci. 2019, 9, x FOR PEER REVIEW 8 of 23 Figure 7. The connection between the Kohonen neural network and an expert system. Figure 7. The connection between the Kohonen neural network and an expert system. In this diagnostic system, the knowledge is represented by IF->THEN rules, which can be written In this diagnostic system, the knowledge is represented by IF->THEN rules, which can be Figure 7. The connection between the Kohonen neural network and an expert system. in a simple way as IF E THEN H, as follows: written in a simple way as IF E THEN H, as follows: In this diagnostic system, the knowledge is represented by IF->THEN rules, which can be 𝐸→𝐻 (2) E ! H (2) written in a simple way as IF E THEN H, as follows: where E represents an observation (evidence) and H represents a hypothesis. Rule-based systems differ from classical logic systems in their non-monotonic “thinking” and (2) where E represents an observation (evidence) and𝐸→ H repr 𝐻 esents a hypothesis. their ability to process uncertainties. In rule-based systems, inference is based on the rule of modus Rule-based systems dier from classical logic systems in their non-monotonic “thinking” and where E represents an observation (evidence) and H represents a hypothesis. ponens (direct thinking) [61], which is described as: their ability to process uncertainties. In rule-based systems, inference is based on the rule of modus Rule-based systems differ from classical logic systems in their non-monotonic “thinking” and 𝐸,𝐸 → 𝐻 ponens (direct thinking) [61], which is described as: their ability to process uncertainties. In rule-based systems, inference is based on the rule of(3 modu ) s ponens (direct thinking) [61], which is described as: E, E ! H This means that, if the E assumption is true and the rule implies E→H, then the conclusion H is (3) 𝐸,H𝐸 → 𝐻 true. In the case of an inference network, the conclusions are represented by facts, which correspond (3) to assumptions of other rules. These inference networks are mostly used in domains where the This means that, if the E assumption is true and the rule implies E!H, then the conclusion H is number of possible solutions is limited (e.g., diagnostic problems or classification problems). The This means that, if the E assumption is true and the rule implies E→H, then the conclusion H is true. In the case of an inference network, the conclusions are represented by facts, which correspond benefit of inference networks is that they can be easily implemented. The following model rules true. In the case of an inference network, the conclusions are represented by facts, which correspond to assumptions of other rules. These inference networks are mostly used in domains where the (Equation (4)) [61] can be formalized in an inference network structure as shown in Figure 8 [61]: to assumptions of other rules. These inference networks are mostly used in domains where the number of possible solutions is limited (e.g., diagnostic problems or classiﬁcation problems). The A ∧ B → P, B ∧ D → Q, P → X, Q ∧ R → Y, C → P, C ∧ E → R, Q → X (4) number of possible solutions is limited (e.g., diagnostic problems or classification problems). The beneﬁt of inference networks is that they can be easily implemented. The following model rules benefit of inference networks is that they can be easily implemented. The following model rules (Equation (4)) [61] can be formalized in an inference network structure as shown in Figure 8 [61]: (Equation (4)) [61] can be formalized in an inference network structure as shown in Figure 8 [61]: A^ B! P, B^ D! Q, P! X, Q^ R! Y, C! P, C^ E! R, Q! X (4) A ∧ B → P, B ∧ D → Q, P → X, Q ∧ R → Y, C → P, C ∧ E → R, Q → X (4) Figure 8. An example of an AND/OR inference network (reproduced from [61]). Figure 8. An example of an AND/OR inference network (reproduced from [61]). Figure 8. An example of an AND/OR inference network (reproduced from [61]). Appl. Sci. 2019, 9, 2253 9 of 22 Appl. Sci. 2019, 9, x FOR PEER REVIEW 9 of 23 2.4. Experimental Setup for Design of the Infrared Imaging-Based Diagnostics System 2.4. Experimental Setup for Design of the Infrared Imaging-Based Diagnostics System In order to design an infrared imaging-based diagnostic system for use with turbojet engines and In order to design an infrared imaging-based diagnostic system for use with turbojet engines to construct a working knowledge base, a small iSTC-21v turbojet engine, developed in the Laboratory and to construct a working knowledge base, a small iSTC-21v turbojet engine, developed in the of Intelligent Control Systems of Aircraft Engines at the Faculty of Aeronautics, Technical University of Laboratory of Intelligent Control Systems of Aircraft Engines at the Faculty of Aeronautics, Košice (Figure 9), was used [62]. This is a single-stream, single-shaft turbojet engine with a single-stage Technical University of Košice (Figure 9), was used [62]. This is a single-stream, single-shaft turbojet one-sided radial compressor, an annular combustion chamber, and a single-stage non-cooled gas engine with a single-stage one-sided radial compressor, an annular combustion chamber, and a turbine with a variable exhaust nozzle with full authority intelligent electronic digital control system single-stage non-cooled gas turbine with a variable exhaust nozzle with full authority intelligent developed from the TS-21 engine (Ljulka-Saturn, Rybinsk, Russia). A schematic of the measurement electronic digital control system developed from the TS-21 engine (Ljulka-Saturn, Rybinsk, Russia). and data acquisition system is shown in Figure 10, and the arrangement of the camera is shown in A schematic of the measurement and data acquisition system is shown in Figure 10, and the Figure 11 [62]. arrangement of the camera is shown in Figure 11 [62]. Appl. Sci. Figure 2019, 9. 9, x FO Views R Pof EER the RE iSTC-21v VIEW turbojet engine (left) and its starting process (right) in a laboratory10 of . 23 Figure 9. Views of the iSTC-21v turbojet engine (left) and its starting process (right) in a laboratory. A digital data acquisition system was used to observe the state of the engine [62]. Various sensors and data acquisition (DAQ) devices serve this purpose and send all the data to the control personal computer (PC). This PC contains a control system created in the LabVIEW (National Instruments Corporation, Austin, TX, USA) and Matlab/Simulink (MathWorks, Natick, MA, USA) software which is able to control and visualize the engine’s performance, save and track all measured parameters, and update the control algorithms. For the design of the IRT diagnostics system, the following parameters are utilized: • n (rpm)—speed of the engine’s shaft. • T2 (°C)—air temperature at the output of the radial compressor. • T3 (°C)—gas temperature at the input to the gas turbine. • T4 (°C)—gas temperature at the output of the gas turbine. These parameters represent the inner state and temperature of the engine. The interior temperature was measured by a thermal sensor contained in the engine. The temperature of the engine’s casing was measured by a thermal camera; this temperature can be correlated with the inner temperature although it has different dynamics. Figure 10. Figure 10. Sche Schematic matic of of the me the measur asurement and data a ement and data acquisition cquisition sy system stem u used sed in in this this stu study dy.. To measure the thermal temperature field around the engine, a FLIR Thermovision A40 thermal imaging camera was used (FLIR Systems, Wilsonville, OR, USA), and a stationary IR camera was used for scanning during long measurement processes, where continuous measurement in real time is necessary. The IR detector of the camera is a 320 × 240 focal plane array uncooled microbolometer. Its spectral range is 7.5 to 13 µm, its instantaneous field of view (IFOV) is 1.3 mrad, its noise equivalent temperature difference at 60 Hz is 0.08 °C at 30 °C, its measurement uncertainty is ±2 °C or ± 2%, its temperature sampling frequency is 60 Hz, and its observable temperature range is from – 40 to 1500 °C. The camera was controlled via the PC using the ThermaCAM Researcher software (FLIR Systems, Wilsonville, OR, USA) through the FireWire interface. One of the settings when capturing a video is the choice of color palette, which is used to convert the levels of the infrared spectrum into the visible spectrum. This selection is mainly important when there are small temperature differences between parts of an object or its surroundings. In the case of an object with a wide range of temperatures, such as a turbojet engine, the standard iron-bow palette is a suitable option [63]. As the engine is a dynamic object, the displacement of its thermal field needs to be evaluated continuously. The camera allows uncompressed video to be captured at 25 frames per second in audio video interleave (AVI) format using the selected color palette. The video can then be decomposed into single frames in uncompressed bitmap (BMP) format. The resulting pictures contain IR images of the engine with the assigned RGB pixel values from the selected color palette during its operation. Full color resolution of the camera can be obtained only when taking static images of the engine, which is not an option for the application, but a video with this resolution would definitely improve the resulting IR images. It can be concluded here that parameters of the camera are sufficient for application on an object operating in a wide temperature range, where fine temperature resolution is not that important. Appl. Sci. 2019, 9, 2253 10 of 22 Appl. Sci. 2019, 9, x FOR PEER REVIEW 11 of 23 Figure 11. The arrangement of the FLIR Thermovision A40 thermal imaging camera in the Figure 11. The arrangement of the FLIR Thermovision A40 thermal imaging camera in the laboratory. laboratory. A digital data acquisition system was used to observe the state of the engine [62]. Various sensors The thermal fields were captured and observed during the whole engine operational process, and data acquisition (DAQ) devices serve this purpose and send all the data to the control personal from its start-up to the time of full thrust, as shown by the measured basic parameters displayed in computer (PC). This PC contains a control system created in the LabVIEW (National Instruments Figure 12. The infrared camera was positioned 1.5 m from the engine and was turned on 0.5 h before Corporation, Austin, TX, USA) and Matlab/Simulink (MathWorks, Natick, MA, USA) software which the measurement in order to allow the temperature of its sensor to equalize with that of the is able to control and visualize the engine’s performance, save and track all measured parameters, and environment. Before the measurement, the values of the background air temperature, air humidity, update the control algorithms. and the emissivity of the engine surface were determined and inputted in the camera. The For the design of the IRT diagnostics system, the following parameters are utilized: environment in the laboratory had a relatively stable humidity (around 50%) and temperature n (rpm)—speed of the engine’s shaft. (around 21 °C) during the experiments with no direct sunlight. These parameters can have an impact T ( C)—air temperature at the output of the radial compressor. on the precision of the IRT measurements, but can be quite well-controlled in a laboratory T ( C)—gas temperature at the input to the gas turbine. environment [39]. The measured state parameters (n (rpm), T2 (°C), T3 (°C), T4 (°C)) for a single run of T ( C)—gas temperature at the output of the gas turbine. the iSTC-21v engine in laboratory conditions are shown in Figure 12. These parameters represent the inner state and temperature of the engine. The interior temperature was measured by a thermal sensor contained in the engine. The temperature of the engine’s casing was measured by a thermal camera; this temperature can be correlated with the inner temperature although it has dierent dynamics. To measure the thermal temperature ﬁeld around the engine, a FLIR Thermovision A40 thermal imaging camera was used (FLIR Systems, Wilsonville, OR, USA), and a stationary IR camera was used for scanning during long measurement processes, where continuous measurement in real time is necessary. The IR detector of the camera is a 320 240 focal plane array uncooled microbolometer. Its spectral range is 7.5 to 13 m, its instantaneous ﬁeld of view (IFOV) is 1.3 mrad, its noise equivalent temperature dierence at 60 Hz is 0.08 C at 30 C, its measurement uncertainty is2 C or 2%, its temperature sampling frequency is 60 Hz, and its observable temperature range is from40 to 1500 C. The camera was controlled via the PC using the ThermaCAM Researcher software (FLIR Systems, Wilsonville, OR, USA) through the FireWire interface. One of the settings when capturing a video is the choice of color palette, which is used to convert the levels of the infrared spectrum into the visible spectrum. This selection is mainly important when there are small temperature dierences between parts of an object or its surroundings. In the case of an object with a wide range of temperatures, such as a turbojet engine, the standard iron-bow palette is a suitable option [63]. As the engine is a dynamic object, the displacement of its thermal ﬁeld needs to be evaluated continuously. The camera allows uncompressed video to be captured at 25 frames per second in audio video interleave (AVI) format using the selected color palette. The video can then be decomposed into single frames in uncompressed Figure 12. The measured parameters of the iSTC-21v turbojet engine that are usable for infrared bitmap (BMP) format. The resulting pictures contain IR images of the engine with the assigned RGB thermography-based diagnostics. pixel values from the selected color palette during its operation. Full color resolution of the camera can be obtained only when taking static images of the engine, which is not an option for the application, but a video with this resolution would deﬁnitely improve the resulting IR images. It can be concluded Appl. Sci. 2019, 9, x FOR PEER REVIEW 11 of 23 Appl. Sci. 2019, 9, 2253 11 of 22 Figure 11. The arrangement of the FLIR Thermovision A40 thermal imaging camera in the here that parameters of the camera are sucient for application on an object operating in a wide laboratory. temperature range, where ﬁne temperature resolution is not that important. The thermal ﬁelds were captured and observed during the whole engine operational process, The thermal fields were captured and observed during the whole engine operational process, from its start-up to the time of full thrust, as shown by the measured basic parameters displayed from its start-up to the time of full thrust, as shown by the measured basic parameters displayed in in Figure 12. The infrared camera was positioned 1.5 m from the engine and was turned on 0.5 h Figure 12. The infrared camera was positioned 1.5 m from the engine and was turned on 0.5 h before before the measurement in order to allow the temperature of its sensor to equalize with that of the the measurement in order to allow the temperature of its sensor to equalize with that of the environment. Before the measurement, the values of the background air temperature, air humidity, and environment. Before the measurement, the values of the background air temperature, air humidity, the emissivity of the engine surface were determined and inputted in the camera. The environment in and the emissivity of the engine surface were determined and inputted in the camera. The the laboratory had a relatively stable humidity (around 50%) and temperature (around 21 C) during environment in the laboratory had a relatively stable humidity (around 50%) and temperature the experiments with no direct sunlight. These parameters can have an impact on the precision of the (around 21 °C) during the experiments with no direct sunlight. These parameters can have an impact IRT measurements, but can be quite well-controlled in a laboratory environment [39]. The measured on the precision of the IRT measurements, but can be quite well-controlled in a laboratory state parameters (n (rpm), T ( C), T ( C), T ( C)) for a single run of the iSTC-21v engine in laboratory 2 3 4 environment [39]. The measured state parameters (n (rpm), T2 (°C), T3 (°C), T4 (°C)) for a single run of conditions are shown in Figure 12. the iSTC-21v engine in laboratory conditions are shown in Figure 12. Figure 12. The measured parameters of the iSTC-21v turbojet engine that are usable for infrared Figure 12. The measured parameters of the iSTC-21v turbojet engine that are usable for infrared thermography-based diagnostics. thermography-based diagnostics. 3. Thermal Image Segmentation and Classiﬁcation 3.1. Design and Training of the Kohonen Self-Organizing Feature Map The Kohonen SOFM was used for the segmentation of the IRT images obtained with the thermal camera at a resolution of 320 240 pixels as described in Section 2.4. Each pixel was deﬁned as a ﬁve-dimensional vector P (t) = {R,G,B,x,y}, where R, G, and B represent the values of the color _i spectrum {R,G,B} = <0,255> and x and y represent the coordinates of each pixel where x = <0,320>; y = <0,240>. The engine’s temperature as evaluated and processed by the camera was not taken into account, since it may contain errors created by the camera compensating for environmental conditions and assigning dierent temperatures to a single color, thus confusing the classiﬁer. In order to evaluate the technical state of the engine, the relative temperature-ﬁeld displacement was used as it is holds the most important diagnostic information. The dynamic range of the camera was set in the interval <0,500> C and was ﬁxed. These parameters deﬁne the design of the SOFM. The pilot testing showed that these multi-dimensional vectors should be classiﬁed into ﬁve classes. As proposed in Section 2.2, Appl. Sci. 2019, 9, x FOR PEER REVIEW 12 of 23 3. Thermal Image Segmentation and Classification 3.1. Design and Training of the Kohonen Self-Organizing Feature Map The Kohonen SOFM was used for the segmentation of the IRT images obtained with the thermal camera at a resolution of 320×240 pixels as described in Section 2.4. Each pixel was defined as a five-dimensional vector P_i (t) = {R,G,B,x,y}, where R, G, and B represent the values of the color spectrum {R,G,B} = <0,255> and x and y represent the coordinates of each pixel where x = <0,320>; y = <0,240>. The engine’s temperature as evaluated and processed by the camera was not taken into account, since it may contain errors created by the camera compensating for environmental conditions and assigning different temperatures to a single color, thus confusing the classifier. In order to evaluate the technical state of the engine, the relative temperature-field displacement was used as it is holds the most important diagnostic information. The dynamic range of the camera was set in the interval <0,500> °C and was fixed. These parameters define the design of the SOFM. The pilot testing showed that these multi-dimensional vectors should be classified into five classes. As proposed in Section 2.2, a feature map with a single row is able to acceptably segment the IRT image, with low demands of computational time and complexity, which was one of the design goals. The classes are labeled as Ni and represented by different colors as follows: • N1: the hottest and most critical part of the IRT image, red. • N2: a hot and critical part of the IRT image, yellow. • N3: a hot and non-critical part of the IRT image, green. • N4: a low-temperature and non-critical part, cyan. Appl. Sci. 2019, 9, 2253 12 of 22 • N5: the temperature of the surrounding objects (not a part of the diagnosed object), blue. The structure of the network can be visualized as shown in Figure 13. The training parameters a feature map with a single row is able to acceptably segment the IRT image, with low demands of which were found to be efficient were set as defined in Table 1, and the standard Kohonen network computational time and complexity, which was one of the design goals. learning rule was used [42,43]. The classes are labeled as N and represented by dierent colors as follows: Table 1. Training parameters. N : the hottest and most critical part of the IRT image, red. N : a hot and critical part of the IRT image, yellow. Grid Topology function N : a hot and non-critical part of the IRT image, green. Distance computation Direct N : a low-temperature and non-critical part, cyan. Initial neighborhood size 3 N : the temperature of the surrounding objects (not a part of the diagnosed object), blue. The structure of the network can Init be iavisualized l learning rat aseshown in Figur 0.9 e 13. The training parameters which were found to be ecient were set as deﬁned in Table 1, and the standard Kohonen network Training iterations 500 learning rule was used [42,43]. Figure Figure 13 13. . TheTh str e stru ucturctu e ofrethe of the Kohonen Kohon self-or en se ganizing lf-orgfeatur anizing e map featu forre map infrared image for infrared image segmentation. segmentation. Table 1. Training parameters. A training dataset was prepared to train the network as follows. The engine was run for 120 s Topology function Grid and thermal images were recorded at two frames per second, as the pilot experiments showed that Distance computation Direct Initial neighborhood size 3 this timing is sufficient since the dynamics of the temperature of the engine casing have a time Initial learning rate 0.9 constant of 1.2 s. The training sample therefore contained a set of 240 thermal images. The run of the Training iterations 500 A training dataset was prepared to train the network as follows. The engine was run for 120 s and thermal images were recorded at two frames per second, as the pilot experiments showed that this timing is sucient since the dynamics of the temperature of the engine casing have a time constant of 1.2 s. The training sample therefore contained a set of 240 thermal images. The run of the engine, which was used to obtain the IRT images is shown in Figure 14. To demonstrate the ability of the trained Kohonen SOFM, ﬁve operational points are shown in Figure 14, where the network was tested if it produced sensible results. The ﬁrst operational point, O , corresponds to the lowest evaluated p1 rotational speed of the engine 42,058 rpm, and the last, O , corresponds to the highest rotational speed p5 (i.e., 50,289 rpm). The classiﬁcations of the ﬁve example images are shown in Figure 15a, and the created segment areas are shown in Figure 15b. Following expert examination of the infrared images and heuristic evaluation of the temperature ﬁeld, it can be concluded that the Kohonen network works as expected and is able to sensibly assign the hottest and coldest parts of the object to each of the ﬁve classes/segments. In order to quantify the results, the number of activations of each neuron are summed, giving the total segment areas S , : : : , S corresponding to each of the classes N , : : : , N . This can be formalized 1 5 1 5 as follows: S = x ; x = 0; 1 (5) f g i j j=1 where p is the number of activations of the i-th neuron. Appl. Sci. 2019, 9, x FOR PEER REVIEW 13 of 23 engine, which was used to obtain the IRT images is shown in Figure 14. To demonstrate the ability of the trained Kohonen SOFM, five operational points are shown in Figure 14, where the network was tested if it produced sensible results. The first operational point, Op1, corresponds to the lowest evaluated rotational speed of the engine 42,058 rpm, and the last, Op5, corresponds to the highest Appl. Sci. 2019, 9, 2253 13 of 22 rotational speed (i.e., 50,289 rpm). Appl. Sci. 2019, 9Figure 14. , x FOR PEER The REmeasured parameters during a si VIEW ngle run of the iSTC-21v engine. 14 of 23 Figure 14. The measured parameters during a single run of the iSTC-21v engine. The classifications of the five example images are shown in Figure 15a, and the created segment areas are shown in Figure 15b. Following expert examination of the infrared images and heuristic evaluation of the temperature field, it can be concluded that the Kohonen network works as expected and is able to sensibly assign the hottest and coldest parts of the object to each of the five classes/segments. In order to quantify the results, the number of activations of each neuron are summed, giving the total segment areas S1, …, S5 corresponding to each of the classes N1, …, N5. This can be formalized as follows: 𝑆 = 𝑥 ; x = {0;1} (5) where p is the number of activations of the i-th neuron. In this way, the size of each class and the total area of each temperature segment can be obtained. As can be seen in Figure 16, the total area of the segments in the three hottest classes increased as the engine speed and temperature increased. As it is not possible to define the exact areas which correspond to each class (as in traditional image segmentation where, for example, a river area is clearly defined), a system for the objective evaluation of the trained classifier performance was designed, and is described in the next section. Figure 15. The segmented thermal images produced by the Kohonen map. Figure 15. The segmented thermal images produced by the Kohonen map. In this way, the size of each class and the total area of each temperature segment can be obtained. As can be seen in Figure 16, the total area of the segments in the three hottest classes increased as the engine speed and temperature increased. As it is not possible to deﬁne the exact areas which correspond to each class (as in traditional image segmentation where, for example, a river area is clearly deﬁned), a system for the objective evaluation of the trained classiﬁer performance was designed, and is described in the next section. Figure 16. The size Si of each segmented area for five example points. 3.2. Classification System Testing and Results To validate the ability of the trained Kohonen network to segment the thermal image, a system for its evaluation was designed as follows. Five artificial spheres, each with an area of 150 pixels, were created, each of which fell into only one of the classes. As can be seen in Figure 17, the network was able to correctly classify these artificial anomalies in this particular case. In order to properly evaluate the functionality of the classifier, five testing infrared images were divided into 60 segments. The artificially created spherical temperature field anomaly for each class was moved across all segments, giving a total of 300 testing images for each class and 1500 testing images in total. The division of the image into segments when the artificial anomaly was included is shown in Figure 18. Appl. Sci. 2019, 9, x FOR PEER REVIEW 14 of 23 Appl. Sci. 2019, 9, 2253 14 of 22 Figure 15. The segmented thermal images produced by the Kohonen map. Figure Figure 16. 16. The The size size S Si of of each each segmented area segmented area for for five ﬁve example points. example points. 3.2. Classiﬁcation System Testing and Results 3.2. Classification System Testing and Results To validate the ability of the trained Kohonen network to segment the thermal image, a system for To validate the ability of the trained Kohonen network to segment the thermal image, a system its evaluation was designed as follows. Five artiﬁcial spheres, each with an area of 150 pixels, were for its evaluation was designed as follows. Five artificial spheres, each with an area of 150 pixels, created, each of which fell into only one of the classes. As can be seen in Figure 17, the network was were created, each of which fell into only one of the classes. As can be seen in Figure 17, the network able to correctly classify these artiﬁcial anomalies in this particular case. Appl. Sci. 2019, 9, x FOR PEER REVIEW 15 of 23 was able to correctly classify these artificial anomalies in this particular case. In order to properly evaluate the functionality of the classifier, five testing infrared images were divided into 60 segments. The artificially created spherical temperature field anomaly for each class was moved across all segments, giving a total of 300 testing images for each class and 1500 testing images in total. The division of the image into segments when the artificial anomaly was included is shown in Figure 18. Figure 17. Thermal anomalies correctly classified by the Kohonen network. Figure 17. Thermal anomalies correctly classiﬁed by the Kohonen network. In order to properly evaluate the functionality of the classiﬁer, ﬁve testing infrared images were divided into 60 segments. The artiﬁcially created spherical temperature ﬁeld anomaly for each class was moved across all segments, giving a total of 300 testing images for each class and 1500 testing images in total. The division of the image into segments when the artiﬁcial anomaly was included is shown in Figure 18. The results are summarized in the confusion matrix shown in Table 2; the ﬁrst row represents the real class of the anomaly and the ﬁrst column represents the class output by the network. The contingency table gives a clear picture of the network’s ability to correctly segment the infrared images. The network achieved a very good accuracy for the ﬁrst two classes, and only misclassiﬁed objects for the three hottest classes. The lowest obtained accuracy was for the classiﬁcation of class N , which was the coldest class and mostly represented the surroundings of the engine, which are of a low temperature. This class also contained misclassiﬁed objects belonging to class N , which only Figure 18. Division of the infrared image for the insertion of the artificial anomaly. represented relatively cold parts of the engine or its surroundings. The most important classes for the evaluation of the technical state of a turbojet engine are those which represent its hottest parts, The results are summarized in the confusion matrix shown in Table 2; the first row represents the real class of the anomaly and the first column represents the class output by the network. The contingency table gives a clear picture of the network’s ability to correctly segment the infrared images. Table 2. The confusion matrix of the trained Kohonen self-organizing feature map (SOFM). Actual class N1 N2 N3 N4 N5 N1 286 5 2 0 0 N2 10 284 15 1 0 Predicted Class N3 4 10 272 4 6 N4 0 1 10 275 34 N5 0 0 1 20 260 Accuracy 95.3% 94.6% 90.6% 91.6% 86% Total Accuracy 91.8% (1377/1500) The network achieved a very good accuracy for the first two classes, and only misclassified objects for the three hottest classes. The lowest obtained accuracy was for the classification of class N5, which was the coldest class and mostly represented the surroundings of the engine, which are of a low temperature. This class also contained misclassified objects belonging to class N4, which only represented relatively cold parts of the engine or its surroundings. The most important classes for the evaluation of the technical state of a turbojet engine are those which represent its hottest parts, that is, N1, N2, and N3. The results could probably be improved by using a larger training dataset or different Kohonen layer topologies, however they can nevertheless be evaluated as very successful for the purpose of turbojet engine diagnostics. Appl. Sci. 2019, 9, x FOR PEER REVIEW 15 of 23 Appl. Sci. 2019, 9, 2253 15 of 22 that is, N , N , and N . The results could probably be improved by using a larger training dataset or 1 2 3 dierent Kohonen layer topologies, however they can nevertheless be evaluated as very successful for the purpose of turbojet engine diagnostics. Figure 17. Thermal anomalies correctly classified by the Kohonen network. Figure 18. Figure 18. Divi Division sion of of the the infrared image for infrared image for the the insertion insertion of of the artif the artiﬁcial icial a anomaly nomaly. . Table 2. The confusion matrix of the trained Kohonen self-organizing feature map (SOFM). The results are summarized in the confusion matrix shown in Table 2; the first row represents the real class of the anomaly and the first column represents the class output by the network. The Actual Class contingency table gives a clear picture of the network’s ability to correctly segment the infrared N N N N N 1 2 3 4 5 images. N 286 5 2 0 0 N 10 284 15 1 0 Table 2. The confusion matrix of the trained Kohonen self-organizing feature map (SOFM). N 4 10 272 4 6 Predicted Class N 0 1 10 275 34 Actual class N 0 0 1 20 260 N1 N2 N3 N4 N5 Accuracy 95.3% 94.6% 90.6% 91.6% 86% N1 286 5 2 0 0 Total N2 10 284 15 1 0 91.8% (1377/1500) Accuracy Predicted Class N3 4 10 272 4 6 N4 0 1 10 275 34 4. Expert Diagnostic System Design Based on Infrared Images N5 0 0 1 20 260 Accuracy 95.3% 94.6% 90.6% 91.6% 86% 4.1. Design of the Expert System Total Accuracy 91.8% (1377/1500) As presented in Figure 15, areas of the segments are not absolutely representative of the engine’s technical state, as their changes are quite slow as the engine heats up or cools down and the internal The network achieved a very good accuracy for the first two classes, and only misclassified thermal energy is transferred to its casing, which has its own heating and cooling dynamics. The main objects for the three hottest classes. The lowest obtained accuracy was for the classification of class task of the expert system is to evaluate the areas of the segments and their changes, and to then present N5, which was the coldest class and mostly represented the surroundings of the engine, which are of to the operator some information about the technical state of the engine. a low temperature. This class also contained misclassified objects belonging to class N4, which only To fulﬁll this task, the designed expert system needs as an input the total area of the segments and represented relatively cold parts of the engine or its surroundings. The most important classes for some indication of how they evolve over time. Temporal changes in the areas of the segments can be the evaluation of the technical state of a turbojet engine are those which represent its hottest parts, predicted by the computation of derivations/dierences between two segmented images in a set time that is, N1, N2, and N3. The results could probably be improved by using a larger training dataset or interval Dt. The standard derivative equation can be used for this purpose for the i-th neuron and the different Kohonen layer topologies, however they can nevertheless be evaluated as very successful total area of the i-th segment S : for the purpose of turbojet engine diagnostics. ( ) ( ) S t + Dt S t i i dS (t) = (6) Dt From a diagnostic point of view, it makes sense to investigate only the three hottest states (levels S –S ) and their derivations S –S , as they represent the most critical temperatures for the engine. 1 3 1_d 3_d The time derivation of the surface areas of the segments allows the magnitude of the change in the area of dierent temperature regions to be determined and early structural failures to be detected. The knowledge base of the proposed expert diagnostic system consists of a set of rules that describe normal and atypical engine operation based on the outputs of the Kohonen SOFM. Each rule R (i = 1, 2, ..., 8) i Appl. Sci. 2019, 9, 2253 16 of 22 is designed according to precisely deﬁned intervals of the allowed amount of pixels (area sizes) of individual color levels and intervals of allowed values of their derivations, based on results for the normal state and atypical states of the object. The outputs of the rules R take Boolean logical values R i i Є{0; 1}. The rules R in Table 3 are deﬁned as follows: IF S is the area size <from;to> AND S is the derivative <from;to> => R , where S (i = 1, 2, 3) (7) i i_d i i Table 3. A partial set of rules in the knowledge base of the proposed expert diagnostic system. IF S IS <13,200; 13,300> AND S is <5; 50 > => R 1 1_d 1 IF S IS <3600; 3700> AND S is <10; 30> => R 2 2_d 2 IF S IS <7300; 7420> AND S is <5 ; 45> => R 3 3_d 3 IF S IS <46; 160 > – – => R 3_d 4 IF S IS <85;6> – – => R 3_d 5 IF S IS <31; 140> – – => R 2_d 6 IF S IS <120;11> – – => R 2_d 7 IF S IS <51; 160> – – => R 1_d 8 The rules R are partial conclusions about the technical state of the investigated engine. In order to obtain some knowledge about the engine’s technical state, the hypotheses which are to be connected with these rules have to be deﬁned. In order to keep the rule base simple at ﬁrst, diagnostic signals (hypotheses) for a turbojet engine are proposed in Table 4. Table 4. Technical states of the turbojet engine. D Normal condition D Light overheating D Moderate overheating D Critical overheating D Light structural defect D Critical structural defect These diagnostic signals (D ; i = 0, 1, 2, ..., 5) are activated by combinations of the proposed rules. The rules to obtain the diagnostic signals are deﬁned in Table 5. Table 5. Inference of the diagnostic signals. IF R AND R AND R => D 1 2 3 0 IF R AND R AND R => D 1 2 4 1 IF R AND R AND R => D 1 5 6 2 IF R AND R AND R => D 3 7 8 3 IF R AND R AND R => D 2 5 8 4 IF R AND R AND R => D 5 7 8 5 The rules, hypotheses, and inputs can be formalized in the form of an oriented graph. The inference network is deﬁned in Section 2.3. In the case of a turbojet engine with diagnostics based on the sizes and derivatives of segmented thermal images, the inference network shown in Figure 19 is proposed based on rules in Tables 3 and 4, which represent the knowledge base of the expert system. The inference engine then performs a bottom–up search from data of the segments’ sizes S and their derivatives S through intermediate rules R up to evaluation of the hypotheses D . i_d j k Appl. Sci. 2019, 9, 2253 17 of 22 Appl. Sci. 2019, 9, x FOR PEER REVIEW 18 of 23 Figure 19. The inference network of the designed diagnostic expert system. Figure 19. The inference network of the designed diagnostic expert system. 4.2. Pilot Testing of the Expert System Using Infrared Thermography (IRT) Inputs 4.2. Pilot Testing of the Expert System Using Infrared Thermography (IRT) Inputs The idea behind the design of the presented expert system is to evaluate its possibilities and The idea behind the design of the presented expert system is to evaluate its possibilities and its its ability to correctly present diagnostic signals to the engine’s operator or supervisory control ability to correctly present diagnostic signals to the engine’s operator or supervisory control system system [64,65]. The designed rule base was therefore constructed to be very simple in order to evaluate [64,65]. The designed rule base was therefore constructed to be very simple in order to evaluate the the ﬁve pre-deﬁned temperature states represented by hypotheses D. Pilot tests were conducted to five pre-defined temperature states represented by hypotheses D. Pilot tests were conducted to prove that the system works as intended. For real-world engine operation, it was only possible to test prove that the system works as intended. For real-world engine operation, it was only possible to the states D , D , D , and D . The resulting classiﬁcations for these tested situations are shown in 0 1 4 5 test the states D0, D1, D4, and D5. The resulting classifications for these tested situations are shown in Table 6. States D and D were artiﬁcially induced by quickly adding heating material on the surface 4 5 Table 6. States D4 and D5 were artificially induced by quickly adding heating material on the surface of the engine’s exhaust nozzle, which created anomalies in the temperature ﬁeld of the engine. The of the engine’s exhaust nozzle, which created anomalies in the temperature field of the engine. The condition of light overheating was achieved when the engine was running at a high speed setting, (i.e., condition of light overheating was achieved when the engine was running at a high speed setting, at an operating speed of 50,000 rpm for around 20 s). The rules which were evaluated as true, and the (i.e., at an operating speed of 50,000 rpm for around 20 s). The rules which were evaluated as true, way the diagnostic signal/hypothesis was obtained, are shown in Table 6. These results suggest that and the way the diagnostic signal/hypothesis was obtained, are shown in Table 6. These results the diagnostic expert system is able to determine basic states represented by hypotheses D , D , D , 0 1 4 suggest that the diagnostic expert system is able to determine basic states represented by hypotheses D and that the designed rules work. D0 , D1, D4, D5 and that the designed rules work. Additionally, another test was set up using the data from a single engine run, as shown in Figure 14. Normal and light overheating states were tested during standard operation of the engine, and are shown in Figure 20. It can be seen that the expert system selected the state D (light overheating) even though the speed of the engine and the corresponding temperatures had decreased. This was the correct selection, since the temperature of the casing takes more time to cool down, whereas the inner gas temperature as measured by internal sensors decreases immediately. This set of pilot tests showed that the inference engine works as intended and that the expert system is able to produce some sensible results in the evaluation of the technical state of the engine during normal conditions and to detect artiﬁcially induced structural defects. In order to prove the correctness of the setting of the rules R , as well as the deﬁnition of intervals in these rules, a more systematic approach has to be taken. This will be designed in a follow-up study of the system. Appl. Sci. 2019, 9, 2253 18 of 22 Appl. Sci. 2019, 9, x FOR PEER REVIEW 19 of 23 Appl. Appl. Sci. Sci. 2019 2019, , 9 9,, x FO x FOR P R PEER EER RE REVIEW VIEW 19 of 19 of 23 23 Appl. Appl. Appl. Appl. Appl. Sci. Sci. Sci. Sci. Sci. 2019 2019 2019 2019 2019 , , 9 9 , , , ,,9 x FO 9 9 x FO ,, , x FO x FO x FO R P R P R P R P R P EER EER EER EER EER RE RE RE RE RE VIEW VIEW VIEW VIEW VIEW 19 of 19 of 19 of 19 of 19 of 23 23 23 23 23 Table 6. Table 6. Te Te stin stin g of the ex g of the ex pert pert sy sy stem stem fo fo r the r the evaluation of evaluation of hypotheses hypotheses DD 0, D 0, D 1, D 1, D 4, and 4, and DD 5. 5. Table 6. Table 6. Table 6. Table 6. Table 6. Table 6. Te Te Te Te Te Te stin stin stin stin stin stin g g g of the ex of the ex g g g of the ex of the ex of the ex of the ex pert pert pert pert pert pert sy sy sy sy sy sy stem stem stem stem stem stem fo fo fo fo fo fo r the r the r the r the r the r the evaluation of evaluation of evaluation of evaluation of evaluation of evaluation of hypotheses hypotheses hypotheses hypotheses hypotheses hypotheses D D D D D D 0 00, D , D , D 0 0 0, D , D , D 1 11, D , D , D 1 1 1, D , D , D 4 44,,, and and 4 4 4 and ,,, and and and D D D D D D 5 55. . . 5 5 5. . . Table 6. Testing of the expert system for the evaluation of hypotheses D , D , D , and D . 0 1 4 5 D0—normal condition DD0 0—norm —norma al c l co ondition ndition DDDDD 00—norm —norm 0 0 0—norm —norm —norm a al c a a a l c l c l c l c o ondition o o o ndition ndition ndition ndition R1: S1 = 13,270 AND S1_d = 20 R R RR R 1 1 1: S : S : S 1 1: S : S 1 1 1 = = = 1 1 1 1 1 = = 1 1 3 3 3,270 ,270 ,270 3 3,270 ,270 AND AND AND AND AND S S S1 1 1S S _d _d _d 1 1 _d _d = = = 20 20 20 = = 20 20 RR 1: S 1: S 1 1 == 1 1 33 ,270 ,270 AND AND SS 1_d 1_d = = 20 20 D —normal condition R : S = 13,270 AND S = 20 1 1 1_d R2: S2 = 3628 AND S2_d = 14 R R2 2: S : S2 2 = 36 = 362 28 8 AND AND S S2 2_d _d = 14 = 14 R RR R R 22: S : S 2 2 2: S : S : S 2 2 = 36 2 = 36 2 2 = 36 = 36 = 36 2 28 2 2 2 8 AND 8 8 8 AND AND AND AND S S 2 S S S 2 _d _d 2 2 2_d _d _d = 14 = 14 = 14 = 14 = 14 R : S = 3628 AND S = 14 2 2 2_d R : S = 7374, AND S = 24 3 3 3_d RR 3: S 3: S 3 = 73 3 = 73 74 7, AND 4, AND S3 S _d 3 _d = 24 = 24 R RR R 3 3: S : S 3 3: S : S 3 3 = 73 = 73 3 3 = 73 = 73 7 74 4 7 7, AND , AND 4 4, AND , AND S S3 3S S _d _d 3 3 _d _d = 24 = 24 = 24 = 24 RR 3: S 3: S 3 3 = 73 = 73 77 44 , AND , AND SS 3_d 3_d = 24 = 24 R AND R AND R => D 1 2 3 0 R1 AND R2 AND R3 => D0 R R1 1 AND AND R R2 2 AN AND D R R3 3 => => D D0 0 R RR R R 11 AND AND 1 1 1 AND AND AND R RR R R 22 AN AN 2 2 2 AN AN AN D D D D D R RR R R 33 => => 3 3 3 => => => D D D D D 00 0 0 0 D1—light overheating DDDD 1 1 1—l —l —l 1—l igh igh igh igh t t t o o o t o v v ve e e vrhe rhe rhe erhe at at atat ing ing ing ing DDD 1—l 1 1—l —l igh igh igh t o t t o o vv v ee e rhe rhe rhe at at at ing ing ing RR 1: S 1: S 1 = 1 1 = 1 3,227 3,227 AND AND S1 S _d 1 _d = 31 = 31 R RR R R 1 1: S : S 1 1 1D : S : S : S 1 1 —light = = 1 1 1 1 1 = = = 1 1 1 3 3,227 ,227 3 3 3,227 ,227 ,227 overheating AND AND AND AND AND S S1 1 S S S _d _d 1 1 1_d _d _d = = 31 31 = = = 31 31 31 R1: S1 = 13,227 AND S1_d = 31 R : S = 13,227 AND S = 31 1 1 1_d R2: S2 = 3643 AND S2_d = -5 R R RR 2 2 2: S : S : S 2: S 2 2 2 = 36 = 36 = 36 2 = 36 4 4 43 3 3 4 AND AND AND 3 AND S S S2 2 2S _d _d _d 2 _d = -5 = -5 = -5 = -5 RRR :R 2: S S 2 2: S : S 2 = 2 = 36 2 = 36 = 36 3643 44 4 3 AND 3 3 AND AND AND S SS S 2_d 2 2 = _d _d = -5 = -5 = -5 5 2 2 2_d R : S = 112 4 3_d RR 4: S 4: S 3_d 3 = _d = 11 11 2 2 R RR R R 4 4: S : S 4 4 4: S : S : S 3 3_d _d 3 3 3_d = = _d _d = = = 11 11 11 11 11 2 2 2 2 2 R ANDR R 4: S AND 3_d = 11 R 2 => D 1 2 4 1 R1 AND R2 AND R4 => D1 R R RR 1 1 1 AND AND AND 1 AND R R RR 2 2 2 AN AN AN 2 AN D D D D R R RR 4 4 4 => => => 4 => D D D D 1 1 1 1 RR R 1 AND 1 1 AND AND RR R 2 AN 2 2 AN AN D D D RR R 4 => 4 4 => => D D D 1 1 1 DD 4—ligh 4—ligh t s t s trtu ru ctur ctur al defec al defec t t DDDD 4 4—ligh —ligh 4 4—ligh —ligh t t s s t t s s t tr rt tu u r rctur u u ctur ctur ctur al defec al defec al defec al defec t t t t DD 4—ligh 4—ligh t s t s tr tu ru ctur ctur al defec al defec t t R2: S2 = 3671 AND S2_d = 3 R R2 2: S : S2 2 = 36 = 367 71 1 AND AND S S2 2_d _d = 3 = 3 D R RR R R 22—light : S : S 2 2 2: S : S : S 2 2 = 36 2 = 36 2 2 = 36 = 36 = 36 structural 7 71 7 7 7 1 AND 1 1 1 AND AND AND AND S defect S 2 S S S 2 _d _d 2 2 2_d _d _d = 3 = 3 = 3 = 3 = 3 R : S = 3671 AND S = 3 2 2 2_d RR 5: S 5: S 3_d 3 _d = -67 = -67 RR R:R R 5 5: S : S S 5 5: S : S 3 3_d _d 3 3= _d _d = = -67 -67 = = -67 -67 67 RR 5: S 5: S 3_d 3_d = = -67 -67 5 3_d R : S = 92 8 1_d R8: S1_d = 92 R R8 8: S : S1 1_d _d = 92 = 92 R ANDR R R R R R 88: S : S 8 8 8AND : S : S : S 11 _d _d 1 1 1_d _d _d = 92 = 92 = 92 = 92 = 92 R => D 2 5 8 4 RR 2 AND 2 AND RR 5 AN 5 AN D D RR 8 => 8 => D D 4 4 R RR R R 2 2 AND AND 2 2 2 AND AND AND R RR R R 5 5 AN AN 5 5 5 AN AN AN D D D D D R RR R R 8 8 => => 8 8 8 => => => D D D D D 4 4 4 4 4 R2 AND R5 AND R8 => D4 D5—critical structural defect DDDDD 5 5 5—critic —critic —critic 5 5—critic —critic a a al s l s l s a al s l s t t tructural d ructural d ructural d t tructural d ructural d e e ef f f e ee e e f fct ct ct e ect ct DD 5—critic 5—critic aa l s l s tructural d tructural d ee fe fe ct ct R5: S3_d = -72 D —critical R R5 5: S : S structural 3 3_d _d = = -72 -72 defect R RR R R 55: S : S 5 5 5: S : S : S 33 _d _d 3 3 3_d _d _d = = -72 = = = -72 -72 -72 -72 R : S =72 5 3_d RR 7: S 7: S 2_d 2 _d = -45 = -45 RR R:R R 7 7: S : S S 7 7: S : S 2 2_d _d 2 2= _d _d = = -45 -45 = = -45 -45 45 RR 7: S 7: S 2_d 2_d = = -45 -45 7 2_d R : S = 118 8 1_d R8: S1_d = 118 R R8 8: S : S1 1_d _d = 11 = 118 8 R AND R R R R R R 88: S : S 8 8 8: S : S : S AND 11 _d _d 1 1 1_d _d _d = 11 = 11 = 11 = 11 = 11 R 8 8 => 8 8 8 D 5 7 8 5 RR 5 AND 5 AND RR 7 AN 7 AN D D RR 8 => 8 => D D 5 5 R RR R 5 5 AND AND 5 5 AND AND R RR R 7 7 AN AN 7 7 AN AN D D D D R RR R 8 8 => => 8 8 => => D D D D 5 5 5 5 RR 5 AND 5 AND RR 7 AN 7 AN D D RR 8 => 8 => D D 5 5 Appl. Sci. 2019, 9, x FOR PEER REVIEW 20 of 23 Additionally, another test was set up using the data from a single engine run, as shown in Addit Addit Addit Addit iiion on on ion a a ally lly lly ally ,,, anot anot anot , anot her her her her t t t t e e est st st est was was was was set set set set up up up up usin usin usin usin g t g t g t g t h h he e e he dat dat dat dat a a a afrom from from from a a a a sin sin sin sin g g gle le le gle engin engin engin engin e e e r r r e r u u un, a n, a n, a un, a s s s sshown shown shown shown in in in in Addit Addit Addit ion iion on aa a lly lly lly , anot ,, anot anot her her her t t t ee e stst st was was was set set set up up up usin usin usin g t g t g t hh h e e e dat dat dat aa a from from from a a a sin sin sin gg g le le le engin engin engin ee e r r r uu u n, a n, a n, a ss sshown shown shown in in in Fig Fig uu re re 1 4 1.4 No . No rm rm alal and li and li gh gh t overheat t overheat ing ing st st atat es es were were te te sted d sted d uu rin rin g g stand stand ard ard operation of the operation of the en en gine, gine, Fig Fig Fig Fig Fig u uu re re u ure re re 1 1 4 4 1 1 1..4 4 4 No No ... No No No rm rm rm rm rm al alal al al and li and li and li and li and li gh gh gh gh gh t t overheat t overheat t t overheat overheat overheat ing ing ing ing ing st st st st st at at at at at es es es es es were were were were were te te te te te sted d sted d sted d sted d sted d u uu rin rin u urin rin rin g g g g g stand stand stand stand stand a ard rd a a ard rd rd operation of the operation of the operation of the operation of the operation of the en en en en en gine, gine, gine, gine, gine, Figure 14. Normal and light overheating states were tested during standard operation of the engine, and are shown in Figure 20. It can be seen that the expert system selected the state D1 (light and and are are sho show wn in n in Fig Figu ure re 20. It c 20. It ca an b n bee seen seen that t that th he expert sy e expert system selected stem selected the state the state D D1 1 (l (light ight and and and and and are are are are are sho sho sho sho sho w ww w w n in n in n in n in n in Fig Fig Fig Fig Fig uuu re u urere re re 20. It c 20. It c 20. It c 20. It c 20. It c aan b aa an b n b n b n b ee seen eee seen seen seen seen that t that t that t that t that t hhe expert sy h h he expert sy e expert sy e expert sy e expert sy stem selected stem selected stem selected stem selected stem selected the state the state the state the state the state D D D D D 11 1 1 1(l (l (l ight (l (l ight ight ight ight overheating) even though the speed of the engine and the corresponding temperatures had overheating overheating overheating overheating overheating ))) even thoug even thoug even thoug )) even thoug even thoug h hh the speed the speed the speed h h the speed the speed of the of the of the of the of the eng eng eng eng eng iiine and ne and ne and iine and ne and the the the the the correspondin correspondin correspondin correspondin correspondin g temperatur g temperatur g temperatur g temperatur g temperatur es h es h es h es h es h aaad d daa d d overheating overheating) even thoug ) even thoughh the speed the speed of the of the eng engine and ine and the the correspondin corresponding temperatur g temperatures h es haadd decreased. Th decreased. Th is was the correct selection is was the correct selection , since the te , since the te mpera mpera ture of ture of the ca the ca sisi ng ta ng ta kes more ti kes more ti me to cool me to cool decreased. Th decreased. Th decreased. Th decreased. Th decreased. Th decreased. Th is was the correct selection is was the correct selection is was the correct selection is was the correct selection is was the correct selection is was the correct selection ,,, since the te since the te since the te ,, , since the te since the te since the te mpera mpera mpera mpera mpera mpera t tture of ure of ure of t t ture of ure of ure of the ca the ca the ca the ca the ca the ca si si si si si si ng ta ng ta ng ta ng ta ng ta ng ta kes more ti kes more ti kes more ti kes more ti kes more ti kes more ti me to cool me to cool me to cool me to cool me to cool me to cool down, whereas the inner gas temperature as measured by internal sensors decreases immediately. down, whe down, whe down, whe down, whe r r re e era a a es the s the s the as the inner inner inner inner gas temper gas temper gas temper gas temper ature ature ature ature as me as me as me as me asur asur asur asur ed by ed by ed by ed by intern intern intern intern al sen al sen al sen al sen s s sors de ors de ors de sors de creases imme creases imme creases imme creases imme diate diate diate diate llly. y. y. ly. down, whe down, whe down, whe re r re e aa a s the s the s the inner inner inner gas temper gas temper gas temper ature ature ature as me as me as me asur asur asur ed by ed by ed by intern intern intern al sen al sen al sen ss ors de sors de ors de creases imme creases imme creases imme diate diate diate ly. lly. y. This set of pilot tests showed that the inference en This set of pilot tests showed that the inference en gin gin e works as int e works as int ended and th ended and th at the expert system at the expert system This set of pilot tests showed that the inference en This set of pilot tests showed that the inference en This set of pilot tests showed that the inference en This set of pilot tests showed that the inference en This set of pilot tests showed that the inference en gin gin gin gin gin e e works as int works as int e e e works as int works as int works as int e ended and th nded and th e e ended and th nded and th nded and th at the expert system at the expert system at the expert system at the expert system at the expert system This set of pilot tests showed that the inference engine works as intended and that the expert system is able to produce some sensible results in the evaluation of the technical state of the engine during is is able able t to o pro prod duce uce some s some se ensible nsible res resu ult lts s in t in th he ev e eval aluat uatiion of the tec on of the technical st hnical state o ate of f the en the engine gine durin during g is is is is is able able able able able t t t o t t o pro o o o pro pro pro pro d dd uce d d uce uce uce uce some s some s some s some s some s e ensible e e e nsible nsible nsible nsible res res res res res u uu lt u u ltlt s lt lt s s s s in t in t in t in t in t h he ev h h h e ev e ev e ev e ev al al al al al uat uat uat uat uat iion of the tec on of the tec ii ion of the tec on of the tec on of the tec hnical st hnical st hnical st hnical st hnical st ate o ate o ate o ate o ate o ff the en the en f f f the en the en the en gine gine gine gine gine durin durin durin durin durin g g g g g normal conditions and to detect artificially induced structural defects. In order to prove the normal cond normal cond normal cond normal cond normal cond it it itions ions it it ions ions ions and t and t and t and t and t ooo det det det oo det det ect ect ect ect ect art art art art art iiifffiiiiiccffciiia ia ia ccia ia ll ll lly ind y ind y ind ll lly ind y ind u u uce ce ce u uce ce d structur d structur d structur d structur d structur al al al al al defects. In o defects. In o defects. In o defects. In o defects. In o rrrder to prove the der to prove the der to prove the rrder to prove the der to prove the normal cond normal condititions ions and t and too det detect ect art artifiifciciaialllly ind y induuceced structur d structural al defects. In o defects. In order to prove the rder to prove the correctness of the setti correctness of the setti ng of ng of the rules the rules RR i, ias w , as w ele l a ll a s t s t he defin he defin itit ion o ion o f int f int erva erva ls ls in in thtese rul hese rul es, es, a more a more correctness of the setti correctness of the setti correctness of the setti correctness of the setti correctness of the setti correctness of the setti ng of ng of ng of ng of ng of ng of the rules the rules the rules the rules the rules the rules R R RR R R i ii, , , ii ias w as w , , , as w as w as w as w e e elle e e llll a l a ll a lll a a a s ss t t s t s s t t t h h he defin h e defin h h e defin e defin e defin e defin it it itit ion o it it ion o ion o ion o ion o ion o f ff int int int ff f int int int e e erva rva e e e rva rva rva rva llls s s llls s s in in in in in in ttth h h tt tese rul h ese rul h h ese rul ese rul ese rul ese rul e e es, s, e e e s, s, s, s, a a a more more a a a more more more more systematic approach has to be taken. This will be designed in a follow-up study of the system. syst syst syst syst emat emat emat emat ic ic ic ic approach h approach h approach h approach h a a as t s t s t as t o o o be t be t be t o be t a a aken ken ken aken ... Thi Thi Thi . Thi s s s wi wi wi s wi ll ll ll be de be de be de ll be de si si sisi gned gned gned gned in in in in a fo a fo a fo a fo ll ll llow-up ow-up ow-up llow-up st st stst udy udy udy udy of t of t of t of t h h he s e s e s he s y y yst st st yst em. em. em. em. syst syst syst emat emat emat ic ic ic approach h approach h approach h aa a s t s t s t oo o be t be t be t aa a ken ken ken . Thi .. Thi Thi ss wi s wi wi llll ll be de be de be de si si si gned gned gned in in in a fo a fo a fo llll ll ow-up ow-up ow-up st st st udy udy udy of t of t of t hh h e s e s e s yy y st st st em. em. em. Figure 20. The division of the engine’s operation into technical states D and D . 0 1 Figure 20. The division of the engine’s operation into technical states D0 and D1. 5. Conclusions This article defines a way to use infrared imaging directly in a new area of aviation, namely, in the evaluation of the technical state of turbojet engines. A concept of a hybrid system combining a SOFM, namely a Kohonen neural network, and an expert system, is shown to be a promising approach to solve the problem of thermal image segmentation and to extract knowledge about the technical state of the engine from the obtained segmented IRT images. The Kohonen neural network defined in this article is a computationally very simple algorithm, and was able to achieve a precision of higher than 90% in the segmentation of infrared images obtained by the FLIR A40 camera when introducing an artificial temperature anomaly. By using the total areas of segmented infrared images corresponding to different temperatures, and their time derivatives, the technical state of the engine can be extracted and evaluated. This can be considered as a novel approach, and it has shown promising results as a proof of concept, which can be further developed and investigated, for example, to take into account environmental or high noise conditions or the impact of application of different image filtering techniques. Moreover, accuracy of the classification could also be increased by utilizing a camera with the ability to record a video with 10 bit color resolution instead of the reduced iron-bow palette for applications, where fine temperature resolution is important. On the other hand, the developed methodology with an expert system seems to be robust enough to work well with the reduced color spectrum and the engine is working in a wide temperature range. The rule base of the expert system proposed in this article can be further expanded to contain more hypotheses, and can also be developed to use fuzzy logic to evaluate the rules to improve the precision of the system. It can be considered as a progressive methodology in the application of computational intelligence and infrared thermography to the area of aviation. The methodology can also be applied in other fields of technical practice to evaluate the temperature fields of different technical objects, such as electronic circuit boards, transmissions, rotating machinery, generators, and electric engines operating in laboratory conditions. The system is designed to be objective, and does not need any input or examination of the infrared images by humans. It can also be integrated into a complex situational control algorithm of a turbojet engine in laboratory conditions, where outputs of the expert system can be used as inputs for the situational classifier and also to tune adaptive controllers of the engine [62,64,65]. Appl. Sci. 2019, 9, 2253 19 of 22 5. Conclusions This article deﬁnes a way to use infrared imaging directly in a new area of aviation, namely, in the evaluation of the technical state of turbojet engines. A concept of a hybrid system combining a SOFM, namely a Kohonen neural network, and an expert system, is shown to be a promising approach to solve the problem of thermal image segmentation and to extract knowledge about the technical state of the engine from the obtained segmented IRT images. The Kohonen neural network deﬁned in this article is a computationally very simple algorithm, and was able to achieve a precision of higher than 90% in the segmentation of infrared images obtained by the FLIR A40 camera when introducing an artiﬁcial temperature anomaly. By using the total areas of segmented infrared images corresponding to dierent temperatures, and their time derivatives, the technical state of the engine can be extracted and evaluated. This can be considered as a novel approach, and it has shown promising results as a proof of concept, which can be further developed and investigated, for example, to take into account environmental or high noise conditions or the impact of application of dierent image ﬁltering techniques. Moreover, accuracy of the classiﬁcation could also be increased by utilizing a camera with the ability to record a video with 10 bit color resolution instead of the reduced iron-bow palette for applications, where ﬁne temperature resolution is important. On the other hand, the developed methodology with an expert system seems to be robust enough to work well with the reduced color spectrum and the engine is working in a wide temperature range. The rule base of the expert system proposed in this article can be further expanded to contain more hypotheses, and can also be developed to use fuzzy logic to evaluate the rules to improve the precision of the system. It can be considered as a progressive methodology in the application of computational intelligence and infrared thermography to the area of aviation. The methodology can also be applied in other ﬁelds of technical practice to evaluate the temperature ﬁelds of dierent technical objects, such as electronic circuit boards, transmissions, rotating machinery, generators, and electric engines operating in laboratory conditions. The system is designed to be objective, and does not need any input or examination of the infrared images by humans. It can also be integrated into a complex situational control algorithm of a turbojet engine in laboratory conditions, where outputs of the expert system can be used as inputs for the situational classiﬁer and also to tune adaptive controllers of the engine [62,64,65]. Author Contributions: Conceptualization, R.A., M.S. (Martin Schrötter), and M.C.; data curation, R.A. and L.F.; formal analysis, R.A., L.F., M.S. (Martin Schrötter), and S.S.; funding acquisition, R.A. and S.S.; investigation, R.A., L.F., M.S. (Martin Schrötter), M.C., R.B., and M.S. (Michal Schreiner); methodology, R.A., L.F., and R.B.; project administration, R.A., M.S. (Martin Schrötter), and S.S.; resources, L.F., M.S. (Martin Schrötter), M.C., R.B., and M.S. (Michal Schreiner); supervision, R.A., M.C., and S.S.; validation, R.A., L.F., M.S. (Martin Schrötter), and R.B.; visualization, L.F., M.S. (Martin Schrötter), M.C., and M.S. (Michal Schreiner); writing—original draft, R.A., L.F., ˇ ˇ M.S. (Martin Schrötter), and M.C.; writing—review and editing, R.A., L.F., M.S. (Martin Schrötter), and M.C. Funding: This work was supported by the following projects: Ecient Systems and Propulsion for Small Aircraft (ESPOSA), funded by the European commission in the seventh framework program under grant agreement ACP1-GA-2011-284859-ESPOSA, the Slovak Research and Development Agency (APVV) under grant agreement DO7RP-0023-11 and KEGA 044TUKE-4/2019—a small unmanned airplane—the platform for education in the area of intelligent avionics. Conﬂicts of Interest: The authors declare no conﬂicts of interest. References 1. Osornio-Rios, R.A.; Antonino-Daviu, J.A.; Jesus Romero-Troncoso, R. Recent Industrial Applications of Infrared Thermography: A Review. IEEE Trans. Ind. Inform. 2019, 15, 615–625. [CrossRef] 2. Lahiri, B.B.; Bagavathiappan, S.; Jayakumar, T.; Philip, J. Medical applications of infrared thermography: A review. Infrared Phys. Technol. 2012, 55, 221–235. [CrossRef] 3. Balaras, C.A.; Argiriou, A.A. Infrared thermography for building diagnostics. Energy Build. 2002, 34, 171–183. [CrossRef] 4. Lisowska, A. Thermographic monitoring of the power transformers. Meas. Autom. Monit. 2017, 63, 154–157. Appl. Sci. 2019, 9, 2253 20 of 22 5. López-Pérez, D.; Antonino-Daviu, J. Application of Infrared Thermography to Failure Detection in Industrial Induction Motors: Case Stories. IEEE Trans. Ind. Appl. 2017, 53, 1901–1908. [CrossRef] 6. Yang, R.; He, Y.; Mandelis, A.; Wang, N.; Wu, X.; Huang, S. Induction Infrared Thermography and Thermal-Wave-Radar Analysis for Imaging Inspection and Diagnosis of Blade Composites. IEEE Trans. Ind. Inform. 2018, 14, 5637–5647. [CrossRef] 7. Vavilov, V.; Nesteruk, D.A. Detecting water in aviation honeycomb structures: The quantitative approach. Quant. Infrared Thermogr. J. 2004, 1, 173–184. [CrossRef] 8. Vavilov, V.; Pan, Y.Y.; Nesteruk, D. Infrared thermographic inspection of water ingress in composite honeycomb panels. Appl. Opt. 2016, 55, 120–125. [CrossRef] [PubMed] 9. Vavilov, V.P. Non-contact one-sided evaluation of hidden corrosion in metallic constructions by using transient infrared thermography. Rev. Metal. 2003, 39, 235–242. [CrossRef] 10. Eddazi, A.; Belattar, S. Nondestructive Testing Evaluation of Aircraft Fuselage Corrosion by Infrared Thermography and Finite Element Method. In Proceedings of the 14th International Conference on Computer Graphics, Imaging and Visualization, Marrakesh, Morocco, 23–25 May 2017. 11. Addepalli, S.; Zhao, Y.; Roy, R.; Galhenege, W.; Colle, M.; Yu, J.; Ucur, A. Non-destructive evaluation of localised heat damage occurring in carbon composites using thermography and thermal diusivity measurement. Measurement 2019, 131, 706–713. [CrossRef] 12. Meola, C.; Boccardi, S.; Carlomagno, G.M. Infrared Thermography in the Evaluation of Aerospace Composite Materials: Infrared Thermography to Composites; Woodhead Publishing: Cambridge, UK, 2016. 13. Swiderski, W. IR Thermography Nondestructive Testing Methods of Composite Materials Used in Aerospace Applications. In Proceedings of the 12th International Conference on Quantitative Infrared Thermography, The e-Journal of Nondestructive Testing, Mahabalipuram, India, 6–10 July 2016. 14. Orton, B.R.; Copaul, M.; Velez-Ospina, C.E. Potential use of infrared thermography to detect aircraft pressurization leaks. Insight Non-Destr. Test. Cond. Monit. 1999, 41, 164–166. 15. Wang, Y.; Gao, B.; Lok Woo, W.; Tian, G.; Maldague, X.; Zheng, L.; Guo, Z.; Zhu, Y. Thermal Pattern Contrast Diagnostic of Microcracks With Induction Thermography for Aircraft Braking Components. IEEE Trans. Ind. Inform. 2018, 14, 5563–5574. [CrossRef] 16. White, J.E.; Tucholski, E.J.; Green, R.E. Nondestructive Testing of Aircraft and Spacecraft Wiring. Mater. Eval. 2003, 61, 1315–1320. 17. Tulloch, J.S.; Jennings, M.C. Thermographic Wiring Inspection. U.S. Patent No. US 6,995,565 B1, 7 February 18. Short, M.; Torres, J.; Kreckie, J. Thermal Imaging for Aircraft Rescue and Fire Fighting Applications; DOT/FAA/TC-17/27 Final Report; U.S. Department of Transportation, Federal Aviation Administration: Cambridge, MA, USA, 2017. 19. Štumper, M.; Kraus, J. Thermal Imaging in Aviation. Mag. Aviat. Dev. 2015, 3, 13–16. [CrossRef] 20. Matikainen, L.; Lehtomäki, M.; Ahokas, E.; Hyyppä, J.; Karjalainen, M.; Jaakkola, A.; Kukko, A.; Heinonen, T. Remote sensing methods for power line corridor surveys. ISPRS J. Photogramm. Remote Sens. 2016, 119, 10–31. [CrossRef] 21. Gómez, C.; Green, D.R. Small unmanned airborne systems to support oil and gas pipeline monitoring and mapping. Arab. J. Geosci. 2017, 10, 202. [CrossRef] 22. Kylili, A.; Fokaides, P.A.; Christou, P.; Kalogirou, S.A. Infrared thermography (IRT) applications for building diagnostics: A review. Appl. Energy 2014, 134, 531–549. [CrossRef] 23. Omar, T.; Nehdi, M.L. Remote sensing of concrete bridge decks using unmanned aerial vehicle infrared thermography. Autom. Constr. 2017, 83, 360–371. [CrossRef] 24. Harvey, M.C.; Rowland, J.; Luketina, K.M. Drone with Thermal Infrared Camera Provides high resolution georeferenced imagery of the Waikite Geothermal Area, New Zealand. J. Volcanol. Geotherm. Res. 2016, 325, 61–69. [CrossRef] 25. Baranwal, N.; Mahulikar, S.P. Infrared Signature of Aircraft Engine with Choked Converging Nozzl. J. Thermophys. Heat Transf. 2016, 30, 1–9. [CrossRef] 26. Ciampa, F.; Mahmoodi, P.; Pinto, F.; Meo, M. Recent Advances in Active Infrared Thermography for Non-Destructive Testing of Aerospace Components. Sensors 2018, 18, 609. [CrossRef] Appl. Sci. 2019, 9, 2253 21 of 22 27. MacLeod, J.D.; Steckhan, P.; He, D. Infrared Thermal Imaging System as a Diagnostic Tool for Gas Turbine Engine Faults. In ASME 1994, International Gas Turbine and Aeroengine Congress and Exposition; American Society of Mechanical Engineers (ASME): New York, NY, USA, 1994; Volume 2. 28. Bogdan, M.; Błachnio, J.; Kułaszka, A.; Derlatka, M. Assessing the Condition of Gas Turbine Rotor Blades with the Optoelectronic and Thermographic Methods. Metals 2019, 9, 31. [CrossRef] 29. Yang, X.; Pang, S.; Shen, W.; Lin, X.; Jiang, K.; Wang, Y. Aero Engine Fault Diagnosis Using an Optimized Extreme Learning Machine. Int. J. Aerosp. Eng. 2016, 2016. [CrossRef] 30. Ntantis, E.; Botsaris, P. Diagnostic Methods for an Aircraft Engine Performance. J. Eng. Sci. Technol. Rev. 2015, 8, 64–72. [CrossRef] 31. Rajani, J. Engine Health Condition Monitoring—A Brief Review. Control Data Fusion E J. 2017, 1, 25–38. 32. Armstrong, J.B.; Simon, D.L. Implementation of an Integrated On-Board Aircraft Engine Diagnostic Architecture. In Proceedings of the 47th Joint Propulsion Conference and Exhibit cosponsored by the AIAA, ASME, SAE, and ASEE, San Diego, CA, USA, 31 July 2011–3 August 2011. 33. Rabenoro, T.; Lacaille, J.; Cottrell, M.; Rossi, F. A Methodology for the Diagnostic of Aircraft Engine Based on Indicators Aggregation. In Proceedings of the CDM 2014: Advances in Data Mining, Applications and Theoretical Aspects, St. Petersburg, Russia, 16–20 July 2014; Volume 8557, pp. 144–158. 34. Seabra, J.; Campos, A. EHD Analysis of a Roller/Inner Ring Contact in a Jet Engine Roller Bearing. In Proceedings of the 82nd Meeting of the AGARD Structures and Materials Panel, Sesimbra, Portugal, 6–7 May 1996; Volume 589. 35. Zaitouna, N.M.; Aqel, M.J. Survey on Image Segmentation Techniques. Procedia Comput. Sci. 2015, 65, 797–806. [CrossRef] 36. Karamizadeh, S.; Abdullah, S.M.; Zamani, M.; Kherikhah, A. Pattern Recognition Techniques: Studies on Appropriate Classiﬁcations. ARPN J. Eng. Appl. Sci. 2014, 9, 791–799. 37. Spontón, H.; Cardelino, J. A Review of Classic Edge Detectors. Image Process. Line 2015, 5, 90–123. [CrossRef] 38. Egmont-Petersen, M.; Ridder, D.; Handels, H. Image processing with neural networks—A review. Pattern Recognit. 2002, 35, 2279–2301. [CrossRef] 39. Vardasca, R.; Vaz, L.; Mendes, J. Classiﬁcation and Decision Making of Medical Infrared Thermal Images. In Classiﬁcation in BioApps; Lecture Notes in Computational Vision and Biomechanics; Dey, N., Ashour, A., Borra, S., Eds.; Springer: Cham, Switzerland, 2018; Volume 26. 40. Zhiqiang, W.; Jun, L. A review of object detection based on convolutional neural network. In Proceedings of the 2017 36th Chinese Control Conference (CCC), Dalian, China, 26–28 July 2017. 41. Rawat, W.; Wang, Z. Deep Convolutional Neural Networks for Image Classiﬁcation: A Comprehensive Review. Neural Comput. 2017, 29, 1–98. [CrossRef] 42. Kohonen, T. Self-Organizing Maps, 3rd ed.; Springer: Berlin/Heidelberg, Germany, 2001. 43. Kohonen, T. MATLAB Implementations and Applications of the Self-Organizing Map; Unigraﬁa Oy: Helsinki, Finland, 2014. 44. Zimmermann, R.; Brandmeier, M.; Andreani, L.; Mhopjeni, K.; Gloaguen, R. Remote Sensing Exploration of Nb-Ta-LREE-Enriched Carbonatite (Epembe/Namibia). Remote Sens. 2016, 8, 620. [CrossRef] 45. Kim, S.; Singh, V.P. Spatial Disaggregation of Areal Rainfall Using Two Dierent Artiﬁcial Neural Networks Models. Water 2015, 7, 2707–2727. [CrossRef] 46. Chi, D. Self-Organizing Map-Based Color Image Segmentation with k-Means Clustering and Saliency Map. ISRN Signal Process. 2011, 2011. [CrossRef] 47. Xu, Y.; Ran, J.; Chen, H. Kohonen Neural Network Classiﬁcation for Failure Process of Metallic Organic Coating in Corrosion Environment. Metals 2017, 7, 147. 48. Ganguli, R. Gas Turbine Diagnostics: Signal Processing and Fault Isolation; CRC Press: Boca Raton, FL, USA, 49. Li, Y.G. Performance Analysis Based Gas Turbine Diagnostics: A Review. Proc. Inst. Mech. Eng. Part A J. Power Energy 2002, 216, 363–377. [CrossRef] 50. Mohammadi, E.; Montazeri-Gh, M. Performance Enhancement of Global Optimization-Based Gas Turbine Fault Diagnosis Systems. J. Propuls. Power 2016, 32, 214–224. [CrossRef] 51. Li, Z.; Zhong, S.-S.; Lin, L. Novel Gas Turbine Fault Diagnosis Method Based on Performance Deviation Model. J. Propuls. Power 2017, 33, 730–739. [CrossRef] Appl. Sci. 2019, 9, 2253 22 of 22 52. Meskin, N.; Naderi, E.; Khorasani, K. A Multiple Model-Based Approach for Fault Diagnosis of Jet Engines. IEEE Trans. Control Syst. Technol. 2013, 21, 254–262. [CrossRef] 53. Vanini, Z.N.S.; Khorasani, K.; Meskin, N. Fault detection and isolation of a dual spool gas turbine engine using dynamic neural networks and multiple model approach. Inf. Sci. 2014, 259, 234–251. [CrossRef] 54. Szentkuti, A.; Kavanagh, H.S.; Grazio, S. Infrared thermography and image analysis for biomedical use. Period. Biol. 2011, 113, 385–392. 55. Rojas, R. Neural Networks—A Systematic Introduction; Springer: Berlin, Germany; New York, NY, USA, 1996. 56. Abraham, A. Handbook of Measuring System: Design Rule-Based Expert Systems; John Wiley & Sons: New York, NY, USA, 2005. 57. Buchanan, B.G.; Smith, R.G. Fundamentals of Expert Systems. Annu. Rev. Comput. Sci. 1988, 3, 23–58. [CrossRef] 58. Yang, J.; Ye, C.; Zhang, X. An expert system shell for fault diagnosis. Robotica 2001, 19, 669–674. [CrossRef] 59. Chlebus, E.; Krot, K.; Kuliberda, M. Hybrid Artiﬁcial Intelligent Systems: Rule-Based Expert System Dedicated for Technological Applications; Springer: Berlin/Heidelberg, Germany, 2011; pp. 373–380. 60. Dvor ˇák, J. Expertní Systémy; Vysoké Ucen ˇ í Technické v Brne: 2004. Available online: http://www.uai.fme. vutbr.cz/~{}jdvorak/Opory/ExpertniSystemy.pdf (accessed on 22 February 2019). 61. Sinha, A.P.; Zhao, H. Tuning Expert Systems for Cost-Sensitive Decisions. Adv. Artif. Intell. 2011, 2011. [CrossRef] 62. Andoga, R.; Foz ˝ o, ˝ L.; Judic ˇák, J.; Bréda, R.; Szabo, S.; Rozenberg, R.; Džunda, M. Intelligent Situational Control of Small Turbojet Engines. Int. J. Aerosp. Eng. 2018, 2018. [CrossRef] 63. Cruz-Albarran, I.A.; Benitez-Rangel, J.P.; Osornio-Rios, R.; Dominguez, B.T.; Medina, D.A.R.; Morales-Hernández, L.A. A new approach to obtain a colour palette in thermographic images. Quant. InfraRed Thermogr. J. 2019, 16, 35–54. [CrossRef] 64. Roman, R.-C.; Radac, M.-B.; Precup, R.-E.; Petriu, E.M. Data-driven Model-Free Adaptive Control Tuned by Virtual Reference Feedback Tuning. Acta Polytech. Hung. 2016, 13, 83–96. [CrossRef] 65. Andoga, R.; Adamc ˇík, F.; Hrabovský, J.; Vaispacher, T. A hybrid diagnostic system for a small turbojet engine. NASE More 2016, 63, 86–92. [CrossRef] © 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

Journal

Applied Sciences – Multidisciplinary Digital Publishing Institute

Published: May 31, 2019

Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Intelligent Thermal Imaging-Based Diagnostics of Turbojet Engines

Intelligent Thermal Imaging-Based Diagnostics of Turbojet Engines

Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Intelligent Thermal Imaging-Based Diagnostics of Turbojet Engines

Intelligent Thermal Imaging-Based Diagnostics of Turbojet Engines

References (48)

Abstract

Journal

Recommended Articles

There are no references for this article.

Our policy towards the use of cookies