Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Creating a Diagnostic Assistance System for Diseases in Kampo Medicine

Creating a Diagnostic Assistance System for Diseases in Kampo Medicine applied sciences Article Creating a Diagnostic Assistance System for Diseases in Kampo Medicine 1 , 2 3 4 5 6 Reimei Koike *, Keiko Ogawa-Ochiai , Akiko Shirai , Katsumi Hayashi , Junsuke Arimitsu , Hongyang Li and Norimichi Tsumura Division of Creative Engineering, Graduate School of Science and Engineering, Chiba 263-8522, Japan Kampo Clinical Center, Department of General Medicine, Hiroshima University Hospital 1-2-3, Kasumi, Minami-ku, Hiroshima 734-8551, Japan; okeiko22@hiroshima-u.ac.jp Department of Otolaryngology, Head and Neck Surgery, Kanazawa University Hospital, Kanazawa 920-8641, Japan; shiraiaki@med.kanazawa-u.ac.jp Honjo Waseda Clinic, Saitama 367-0030, Japan; kachan18@gmail.com Senrioka Kagayaki Clinic, Osaka 565-0816, Japan; arimitsu@hiroshima-u.ac.jp Department of Japanese Traditional (Kampo) Medicine, Kanazawa University Hospital, Kanazawa 920-8641, Japan; hoyalee328@stu.kanazawa-u.ac.jp Graduate School of Engineering, Chiba University, Chiba 263-8522, Japan; tsumura@faculty.chiba-u.jp * Correspondence: reimei0217@chiba-u.jp Abstract: The aim of this study was to propose a method to assess images of the tongue captured using a polarized light camera for diagnostic use in Kampo medicine. Glossy and non-glossy images of the tongue were captured simultaneously using a polarizing camera and a polarizing plate. Data augmentation was performed by modulating the color and gloss, resulting in an increase in the number of images from 11 to 275. To create a data set, the values for which diseases were evaluated by Kampo doctors for all tongue images were taken as the correct values and combined with the Citation: Koike, R.; Ogawa-Ochiai, features extracted from the tongue images. Using this data set, we constructed a diagnostic support K.; Shirai, A.; Hayashi, K.; Arimitsu, module to evaluate diseases. The resulting mean absolute error of the assessment was 0.44 for qi J.; Li, H.; Tsumura, N. Creating a deficiency, 0.42 for blood deficiency, 0.33 for blood stagnation, 0.36 for yin deficiency, and 0.55 for Diagnostic Assistance System for Diseases in Kampo Medicine. Appl. fluid stagnation, suggesting that the diagnostic assistance module was accurate, and our proposed Sci. 2021, 11, 9716. https://doi.org/ learning and data augmentation methods were effective. 10.3390/app11219716 Keywords: machine learning; Kampo medicine; tongue; diagnostic assistance Academic Editor: Andrew Teoh Beng Jin Received: 27 August 2021 1. Introduction Accepted: 9 October 2021 The general medical care received in hospitals worldwide is based on Western medicine. Published: 20 October 2021 However, in China and other East Asian countries, the field of traditional medicine—called Kampo medicine in Japan—has developed over the past 1500 years. Western medicine and Publisher’s Note: MDPI stays neutral Kampo medicine differ greatly in their diagnostic methods. In Kampo medicine, the pathol- with regard to jurisdictional claims in ogy is assessed using the three elements of “qi,” “blood,” and “fluid” that comprise the qi, published maps and institutional affil- blood, and fluid theory [1]. A healthy state is one in which these three elements circulate iations. normally and in sufficient quantities. Qi represents the energy of the entire body, blood rep- resents the blood and its flow through the body, and fluid represents fluids other than the blood (such as lymph and digestive fluids) and its flow through the body. The conditions in which qi, blood, and fluid are “insufficient” are called “qi deficiency,” “blood deficiency,” Copyright: © 2021 by the authors. and “yin deficiency,” respectively, and those in which qi, blood, and fluid are “stagnated” Licensee MDPI, Basel, Switzerland. are called “qi stagnation,” “blood stagnation,” and “fluid stagnation,” respectively [2]. This article is an open access article Doctors practicing Kampo medicine diagnose diseases via inspection, inquiry, palpa- distributed under the terms and tion, and aural and olfactory examinations [3]. Inspection, especially visual examination, is conditions of the Creative Commons among the most important diagnostic methods in Kampo medicine but is rarely applied Attribution (CC BY) license (https:// in Western medicine. One reason for this is that it is not possible to make a quantitative creativecommons.org/licenses/by/ diagnosis because inspection is subjective and based on the doctor ’s experience. 4.0/). Appl. Sci. 2021, 11, 9716. https://doi.org/10.3390/app11219716 https://www.mdpi.com/journal/applsci Appl. Sci. 2021, 11, 9716 2 of 13 To solve this problem, a quantitative descriptive diagnosis method has been formu- lated using image processing in recent years [4,5]. By using a computer to perform an advisory diagnosis based on images of the face and tongue, it is possible to quantify the diagnosis without prior experience and identify the relevant disease. In addition, Kampo medicine contains many concepts useful for preventive medicine. The qi deficiency, blood deficiency, blood stagnation, yin deficiency, and fluid stagnation that we evaluated indicate imbalances in the body. In Kampo medicine, doctors take appropriate treatments to adjust these imbalances in order to prevent diseases. There are no clear distinction between healthy people and patients; in this study, the closer to 0, the better the condition is, and the closer to 4, the worse the condition is. About 0 indicates a healthy state, 1 and 2 indicate conditions that require follow-up, and 3 and 4 indicate conditions that require treatment., but there are individual differences among doctors. Table 1 lists diseases that can be diagnosed based on an inspection of the face and tongue of the patient using Kampo medicine. To computerize descriptive diagnosis, Matsushita et al. [6] proposed a method to assess diseases of Kampo medicine using the facial images of patients. Blood deficiency, blood stagnation, and yin deficiency can be diagnosed using facial images. However, this facial image assessment might not be sufficient to achieve a Kampo medical diagnosis. Observations of the patient’s tongue increases the number of conditions that can be diagnosed using computer image analysis. By observing the color, fluid content, and shape of the tongue, and the condition of the tongue coating, five of the six conditions—qi stagnation, qi deficiency, blood deficiency, blood stagnation, yin deficiency, and fluid stagnation—can be diagnosed. Table 1. Body parts (tongue and face) used to diagnose each of the six diseases in Kampo medicine. Qi Blood Fluid Face Face Deficiency Tongue Tongue Tongue Face Stagnation Tongue Tongue As the tongue is closely related to the internal organs with exposed mucous mem- branes, it is regarded as an important indicator of many diseases in Kampo medicine. The tongue is very glossy under normal lighting conditions owing to its moist surface, which makes it difficult to obtain accurate information on the color of the tongue with normal imaging methods. However, Kampo doctors consider gloss to be just as important as color information in making a diagnosis. Therefore, to perform a computerized tongue-based diagnosis, it is necessary to use an imaging method that can accurately acquire information on both color and gloss. In the latest study of diagnosis from the tongue, Xu et al. [7] collected tongue images from about 1000 people using a device that can perform gloss removal imaging to diagnose diabetes. Kainuma et al. [8] conducted a study to estimate the Frequency scale for the symptoms of gastroesophageal reflux disease from images of the tongue. However, these recent studies have the problem of gloss removal: in these studies, the gloss of the tongue was completely removed, which is different from the Kampo doctor ’s observation. To solve this problem, Nakaguchi et al. [9] used a combination of the integrating sphere method and directional illumination to capture images of glossy and non-glossy tongues in order to estimate the moisture content of the tongue. However, because the integrating sphere is large and expensive, it is not appropriate for personal use. In the present study, we photographed the tongue using a polarizing camera that simultaneously captures glossy and non-glossy images with more compact and cheaper equipment than an integrating sphere which is conventionally used. In addition, to collect tongue images more efficiently, we proposed a physiologically compensated and effective data augmentation method. We then used these images to assess each patient’s diseases. Appl. Sci. 2021, 11, 9716 3 of 13 A system of quantitative evaluation was constructed by machine learning of the values and features obtained from the tongue images. The accuracy of the evaluation system was verified via comparison with the distribution and variability of the assessments by four Kampo medicine specialists. This research provided a quantitative diagnostic method for Kampo diseases from the tongue using image processing and machine learning techniques. The contributions and novelty of this paper are therefore as follows: We proposed a tongue imaging method using a polarized light camera that is simpler and more effective than the conventional method. In particular, the use of polarized light cameras provided accurate gloss information and showed their effectiveness in diagnosing yin deficiency and fluid stagnation, conditions closely related to fluid content, as shown in Section 2 We proposed a physiologically compensated and effective data augmentation method. We show that the proposed data augmentation method improved the distribution of correct values in Section 4. In addition, the estimation accuracy was improved by the data augmentation. These results indicate the effectiveness of our method. 2. Materials and Methods 2.1. Capturing Glossy and Non-Glossy Images Using Polarized Light Incident light from a source is reflected from the surface of an object. This reflection of the light is classified into two types: surface reflection, where the angles of incidence and reflection of light are the same with respect to the surface of reflection, and internal diffuse reflection caused by the scattering of the incident light inside the object. The human eye and a camera perceive the color and luster of an object by capturing the reflected light. In general, an object appears colored owing to internal diffuse reflection, while luster is caused by surface reflection. Light is an electromagnetic wave that oscillates in a direction perpendicular to its direction of travel, and normal light is in an unpolarized or partially polarized state, which is a mixture of light oscillating in all directions. Polarization, conversely, refers to light that oscillates only in a certain direction. Passing ordinary unpolarized light through an optical filter called a polarizing plate creates polarized light in a direction identical to that of the polarizing plate. This property of polarization has important links to the surface reflection and internal diffusion. Light is reflected while the polarization of the incident light is maintained; when the incident light is reflected off the surface of the object, it is linearly polarized in the direction of the polarizing plate, as in the case of incident light. Conversely, internal diffuse light is unpolarized even if the incident light is linearly polarized when internally reflected, because the incident light is reflected in various directions and the light is scattered. We used this property to obtain glossy and non-glossy images. Figure 1 shows the method used to obtain an image with enhanced gloss and one with reduced gloss using two orthogonal polarizing plates. S indicates the surface-reflected light, D indicates diffuse reflected light, and the subscripts p and s indicate the directions of orthogonal polarization. Since S in Figure 1a maintains the same polarization as the incident light, the relationship between the intensity of S and that of the p-polarization component (S ) of S is shown in Equation (1), where I indicates the intensity of x: p x I = I (1) S S The diffuse component (D) in Figure 1a is the loss of polarization of the incident light. The ratio of intensity of the p-polarization component D to that of the s-polarization component (D ) in D is 1:1, and can be expressed as: I = I + I = 2I = 2I (2) D D D D D p s p s Appl. Sci. 2021, 11, x FOR PEER REVIEW 4 of 13 = =2 =2 (2) Figure 1b shows the setup used to attain a gloss-enhanced image by placing a second polarizing plate between the object and the camera. When light passes through the p-po- larizing plate, the component of s-polarization in the vertical direction cannot pass through the p-polarizing plate. Thus, the internally reflected light (D), which contains , has half the intensity; however, , which has only the p-polarization component, can pass through the p-polarizing plate illumination and the object. In the present study, we used a polarizing camera instead of a second polarizing plate placed in front of the cam- era. The polarizing camera can capture 0°, 45°, 90°, and 145° polarization images simulta- Appl. Sci. 2021, 11, 9716 4 of 13 neously, enabling the capture of a gloss-enhanced image from a 0° polarization image and a non-glossy image from a 90° polarization image simultaneously. (a) (b) (c) Figure 1. Acquisition of surface and internal reflection using polarizing plate. (a) Polarization component of surface re- Figure 1. Acquisition of surface and internal reflection using polarizing plate. (a) Polarization component of surface flection and diffuse reflection; (b) Taking a gloss-enhanced image; (c) Taking a non-glossy image. reflection and diffuse reflection; (b) Taking a gloss-enhanced image; (c) Taking a non-glossy image. 2.2. SFigur eparate ing th 1b shows e Tongthe ue Coat setup ing fr used om t to he attain Tongue a B gloss-enhanced ody image by placing a sec- ond polarizing plate between the object and the camera. When light passes through the The surface of the tongue can be categorized into the coating and the body. The coat- p-polarizing plate, the component of s-polarization in the vertical direction cannot pass ing of the tongue refers to the part coated with food debris, mucosal cells, and bacteria on through the p-polarizing plate. Thus, the internally reflected light (D), which contains tissues called filamentous papillae, and in the gaps between protrusions on the surface of D , has half the intensity; however, S , which has only the p-polarization component, s p the tongue. The body of the tongue refers to the part that is not covered by the coating of can pass through the p-polarizing plate illumination and the object. In the present study, the tongue [10]. The color of the lichen of the tongue can be white, yellow, gray, and black, we used a polarizing camera instead of a second polarizing plate placed in front of the while the body of the tongue can be light-red, red, crimson, and purple. As the lichen and camera. The polarizing camera can capture 0 , 45 , 90 , and 145 polarization images body of the tongue show different color changes, it is necessary to distinguish between simultaneously, enabling the capture of a gloss-enhanced image from a 0 polarization them when evaluating tongue color. image and a non-glossy image from a 90 polarization image simultaneously. Xu et al. [11] focused on the fact that the tongue has a large proportion of red and so the R component of the RGB color space tends to be prominent, whereas the lichen of the 2.2. Separating the Tongue Coating from the Tongue Body tongue does not have a prominent R component. This previous study calculated I for all The surface of the tongue can be categorized into the coating and the body. The pixels in the tongue image using the following equation: coating of the tongue refers to the part coated with food debris, mucosal cells, and bacteria .. on tissues called filamentous papillae, and in the gaps between protrusions on the surface = − . (3) of the tongue. The body of the tongue refers to the part that is not covered by the coating of All pixels were classified as belonging to the body or coating of the tongue according the tongue [10]. The color of the lichen of the tongue can be white, yellow, gray, and black, to Equation (4). Taking as the mean value of I classified as belonging to the body of while the body of the tongue can be light-red, red, crimson, and purple. As the lichen and the tongue, Mean( ) as the mean value of I classified as belonging to the tongue body, body of the tongue show different color changes, it is necessary to distinguish between and Mean( ) as the mean value of I classified as belonging to the tongue coating, them when evaluating tongue color. is determi Xu etnal. ed to ma [11] focused ximize on Eqthe uati fact on (5 that ): the tongue has a large proportion of red and so the R component of the RGB color space tends to be prominent, whereas the lichen of the ( ) = , tongue does not have a prominent R component. This previous study calculated I for(4 all ) ( ) pixels in the tongue image using the following equation: ( ): ( )− ( ) . (5) 1.5G + 0.5B I = R . (3) 2.3. Extracting Feature Values from Tongue Images All pixels were classified as belonging to the body or coating of the tongue according to Equation (4). Taking I as the mean value of I classified as belonging to the body of body the tongue, Mean(I ) as the mean value of I classified as belonging to the tongue body, body and Mean(I ) as the mean value of I classified as belonging to the tongue coating, I is coating m determined to maximize Equation (5): I (i f I  I ) body I = , (4) I (i f I < I ) coating Maximize(I ) : Mean(I ) Mean(I ) . (5) body coating 2.3. Extracting Feature Values from Tongue Images We then extracted feature values from the tongue images for machine learning. The first step is to manually extract the tongue region from whole image. After that, as mentioned above, the two images with and without gloss were used to obtain three Appl. Sci. 2021, 11, x FOR PEER REVIEW 5 of 13 We then extracted feature values from the tongue images for machine learning. The first step is to manually extract the tongue region from whole image. After that, as men- Appl. Sci. 2021, 11, 9716 5 of 13 tioned above, the two images with and without gloss were used to obtain three images— (a) gloss component, (b) tongue coating, and (c) tongue body—as shown in Figure 2. From (a), we obtained the mean, standard deviation, and maximum, minimum, and median images—(a) gloss component, (b) tongue coating, and (c) tongue body—as shown in ∗ ∗ ∗ ∗ values of in the color space of the glossy area and the percentages of the total Figure 2. From (a), we obtained the mean, standard deviation, and maximum, minimum, area of the tongue by gloss. Since the area and brightness of the gloss on the tongue surface and median values of L in the L a b color space of the glossy area and the percentages changes with the amount of fluid, features that indicate these changes are obtained from of the total area of the tongue by gloss. Since the area and brightness of the gloss on the tongue surface changes with the amount of fluid, features that indicate these changes are the gloss image. From (b) and (c), we obtained the mean, standard deviation, and maxi- obtained from the gloss image. From (b) and (c), we∗obtained the ∗ mean, standar ∗ ∗ d ∗ deviation, mum, minimum, and median values of and in the color space of the and maximum, minimum, and median values of a and b in the L a b color space of the tongue coating. Changes in blood status can be seen in the color of the tongue body. In tongue coating. Changes in blood status can be seen in the color of the tongue body. In addition, the area and color of the tongue coating changes with physical condition and addition, the area and color of the tongue coating changes with physical condition and lifestyle. Therefore, we obtained the feature values that represent the color and coating lifestyle. Therefore, we obtained the feature values that represent the color and coating are area. a. As As shown shown in T in Table able 2, a 2total , a to of ta27 l of types 27 tof ypes of feature fe values ature va werlue e obtained. s were obtained. (a) (b) (c) Figure 2. Separation of the tongue into three components: (a) glossy image, (b) tongue coating, and Figure 2. Separation of the tongue into three components: (a) glossy image, (b) tongue coating, and (c) tongue body. (c) tongue body. Table 2. Total (27) features obtained from each separated image. Table 2. Total (27) features obtained from each separated image. From Glossy Image From Tongue Coating From Tongue Body From Glossy Image From Tongue Coating From Tongue Body Mean of gloss L Mean of coating a , b Mean of body a , b ∗ ∗ ∗ ∗ ∗ Mean of gloss Mean of coating , Mean of body , Max of gloss L Max of coating a , b Max of body a , b ∗ ∗ ∗ ∗ ∗ Max of gloss Max of coating , Max of body , Min of gloss L Min of coating a , b Min of body a , b ∗ ∗ ∗ ∗ ∗ Median of gloss L Median of coating a , b Median of body a , b Min of gloss Min of coating , Min of body , Std of gloss L Std of coating a , b Std of body a , b ∗ ∗ ∗ ∗ ∗ Median of gloss Median of coating , Median of body , Gross area Coating area ∗ ∗ ∗ ∗ ∗ Std of gloss Std of coating , Std of body , Gross area 2.4. Set-Up for Capturing Tongue Coating are Images a Figure 3 shows the photographic environment in which the learning data were col- lected. A polarizing camera (Lucid, PHX050S-Q) was used, with a resolution per polarized 2.4. Set-Up for Capturing Tongue Images image of 1224  1024 pixels, bit depth of 16 bits, exposure time of 8000 s, and distance Figure 3 shows the photographic environment in which the learning data were col- between the camera and the subject of 73 cm. The images were taken in a dark room with lected. A polarizing camera (Lucid, PHX050S-Q) was used, with a resolution per polarized an LED light source (Neewer, PT-176S) with a polarizing plate. Eleven healthy subjects image of 1224 × 1024 pixels, bit depth of 16 bits, exposure time of 8000 µs, and distance were photographed. between the camera and the subject of 73 cm. The images were taken in a dark room with an LED light source (Neewer, PT-176S) with a polarizing plate. Eleven healthy subjects were photographed. Appl. Sci. 2021, 11, x FOR PEER REVIEW 6 of 13 Appl. Sci. 2021, 11, 9716 6 of 13 Appl. Sci. 2021, 11, x FOR PEER REVIEW 6 of 13 Figure 3. Experimental setup used to capture the tongue images. 2.5. Data Augmentation by Color and Gloss Modulation As the number of captured tongue images was insufficient for machine learning, we performed a physiologically corrected modulation of each sample to produce more train- ing data. Data augmentation was performed by applying color and gloss modulation to each captured image. The color of the tongue is determined by the amount of hemoglobin in the blood. Therefore Figure 3., m Experimental setup odulating the hemog used to lobin capture the tongue images. components enables the creation of images with Figure 3. Experimental setup used to capture the tongue images. different tongue colors. To extract the components of hemoglobin from non-glossy tongue 2.5. Data Augmentation by Color and Gloss Modulation images, we separated the components of pigmentation using the method proposed by 2.5. Data Augmentation by Color and Gloss Modulation Tsumura As the et al number . [12]. The col of captur or- ed modula tongue ted i images mages of the tongue was insufficient we for re obtain machine ed b learning, y multi- we As the number of captured tongue images was insufficient for machine learning, we plying the hemoglobin performed a physiologically component by corrected modulation the color modu of each lati sample on factor ( to pr oduce ) in log space more training and performed a physiologically corrected modulation of each sample to produce more train- data. Data augmentation was performed by applying color and gloss modulation to each resynthesizing the components of melanin and shading by reverse processing the separa- ing data. Data augmentation was performed by applying color and gloss modulation to captured image. tion of the pigment component. Figure 4a–e shows color-modulated images with each captured image. The color of the tongue is determined by the amount of hemoglobin in the blood. ( , , , , ) = (0.6, 0.8, 1.0, 1.2, 1.4), respectively. Therefor The color of e, modulating the tongue is determined the hemoglobin components by enables the amount of hemoglobin in the blood. the creation of images with As both the glossy and non-glossy images were captured simultaneously, the glossy different tongue colors. To extract the components of hemoglobin from non-glossy tongue component was e Therefore, modulating asily extr the acted by subt hemoglobin ractin compon g the non-glo ents ssy enaimage from the g bles the creation of lossy im- images with images, we separated the components of pigmentation using the method proposed by age. The image of the tongue with modulated gloss was obtained by multiplying the ex- different tongue colors. To extract the components of hemoglobin from non-glossy tongue Tsumura et al. [12]. The color-modulated images of the tongue were obtained by multi- tracted components of gloss with the gloss modulation factor ( ), as well as the color images, we separated the components of pigmentation usi ng the method proposed by plying the hemoglobin component by the color modulation factor (a ) in log space and modulation, and resynthesizing the image with no gloss. Figure 5a–e shows the gloss- Tsumura et al. [12]. The color-modulated images of the tongue were obtained by multi- resynthesizing the components of melanin and shading by reverse processing the sep- modulated images for ( , , , , ) = (0.0, 0.3, 0.6, 1.0, 1.5), respectively. As shown in plying the hemoglobin component by the color modulation factor ( ) in log space and aration of the pigment component. Figure 4a–e shows color-modulated images with Figure 6, by combining these two modulation methods, the number of original images of resynthesizing the components of melanin and shading by reverse processing the separa- (a , a , a , a , a ) = (0.6, 0.8, 1.0, 1.2, 1.4), respectively. 1 2 3 4 5 the tongue was augmented by a factor of 25. tion of the pigment component. Figure 4a–e shows color-modulated images with ( , , , , ) = (0.6, 0.8, 1.0, 1.2, 1.4), respectively. As both the glossy and non-glossy images were captured simultaneously, the glossy component was easily extracted by subtracting the non-glossy image from the glossy im- age. The image of the tongue with modulated gloss was obtained by multiplying the ex- tracted components of gloss with the gloss modulation factor ( ), as well as the color modulation, and resynthesizing the image with no gloss. Figure 5a–e shows the gloss- modulated images for ( , , , , ) = (0.0, 0.3, 0.6, 1.0, 1.5), respectively. As shown in (a) =0.6 (b) =0.8 (c) =1.0 (d) =1.2 (e) =1.4 Figure 6, by combining these two modulation methods, the number of original images of Figure 4. Five levels of color modulation using the modulation coefficient . Figure 4. Five levels of color modulation using the modulation coefficient a . the tongue was augmented by a factor of 25. As both the glossy and non-glossy images were captured simultaneously, the glossy component was easily extracted by subtracting the non-glossy image from the glossy image. The image of the tongue with modulated gloss was obtained by multiplying the extracted components of gloss with the gloss modulation factor (b ), as well as the color modulation, and resynthesizing the image with no gloss. Figure 5a–e shows the gloss-modulated images for (b , b , b , b , b ) = (0.0, 0.3, 0.6, 1.0, 1.5), respectively. As shown in Figure 6, 1 2 3 4 5 by combining these two modulation methods, the number of original images of the tongue was augmented by a factor of 25. (a) =0.6 (b) =0.8 (c) =1.0 (d) =1.2 (e) =1.4 Figure 4. Five levels of color modulation using the modulation coefficient . Appl. Sci. 2021, 11, x FOR PEER REVIEW 7 of 13 Appl. Sci. 2021, 11, 9716 7 of 13 Appl. Sci. 2021, 11, x FOR PEER REVIEW 7 of 13 (a) =0.0 (b) =0.3 (c) =0.6 (d) =1.0 (e) =1.5 (a) =0.0 (b) =0.3 (c) =0.6 (d) =1.0 (e) =1.5 Figure 5. Five levels of gloss modulation using the modulation coefficient . Figure 5. Five levels of gloss modulation using the modulation coefficient . Figure 5. Five levels of gloss modulation using the modulation coefficient b . Figure 6. Data augmentation by color and gloss modulation. 2.6. Subjective Evaluation of Diseases by Kampo Doctors Using Tongue Images Figure 6. Data augmentation by color and gloss modulation. Figure 6. One doctor Data augmentation who specialized by color and gloss mo in Kampo medicine dulation (the evaluator) ev . aluated each tongue 2.6. Subjective image to diagno Evaluation se t ofhDiseases e subjects. T by Kampo he evaluation Doctors Using was pe Tongue rformeImages d in a darkened room, with 80 cm between the display and the evaluator, and a display size of 51 cm. The screen One doctor who specialized in Kampo medicine (the evaluator) evaluated each tongue 2.6. Subjective Evaluation of Diseases by Kampo Doctors Using Tongue Images was displayed in device-independent sRGB color. There was no other lighting in the dark- image to diagnose the subjects. The evaluation was performed in a darkened room, One doctor who specialized in Kampo medicine (the evaluator) evaluated each ened room, and the Kampo doctor evaluated the results using only the display light. The with 80 cm between the display and the evaluator, and a display size of 51 cm. The tongue evaluator imag assessed e to diagno the five condit se the subjects. T ions that can be diag he evaluation nosed from images was performe of the tongue— d in a darkened room, screen was displayed in device-independent sRGB color. There was no other lighting in qi deficiency, blood deficiency, blood stagnation, yin deficiency, and fluid stagnation—as wi the th 80 cm between the di darkened room, and the Kampo splay doctor and the evaluated evaluathe torr , esults and a d using isponly lay size the display of 51 cm. The screen values of 0, 1, 2, 3, and 4, with smaller values indicating greater severity. As the data were light. The evaluator assessed the five conditions that can be diagnosed from images of was displayed in device-independent sRGB color. There was no other lighting in the dark- augmented by modulation, this produced some images of tongues that were unrepre- the tongue—qi deficiency, blood deficiency, blood stagnation, yin deficiency, and fluid ened room, and the Kampo doctor evaluated the results using only the display light. The sentative; tongue images judged as unrepresentative based on the experience of the doctor stagnation—as values of 0, 1, 2, 3, and 4, with smaller values indicating greater severity. As evaluator assessed the five conditions that can be diagnosed from images of the tongue— were excluded. The display showed images of the tongue randomly extracted from a set the data were augmented by modulation, this produced some images of tongues that were qi of unex defici amin ency, ed images. Th blood dee evaluator e ficiency, bloo valua d st ted the agnaimages, tion, yiand the disp n deficiency lay turned d , and fluid stagn ark ation—as unrepresentative; tongue images judged as unrepresentative based on the experience of when the evaluation ended. Four seconds later, the randomly selected images were dis- the doctor were excluded. The display showed images of the tongue randomly extracted values of 0, 1, 2, 3, and 4, with smaller values indicating greater severity. As the data were playe fromd a again. This was rep set of unexamined e images. ated until th The evaluator e entire collec evaluated tion of tongue imag the images, and es ha the d been as- display augmented by modulation, this produced some images of tongues that were unrepre- sessed. To prevent a decline in evaluation capability due to fatigue, the evaluator was turned dark when the evaluation ended. Four seconds later, the randomly selected images sentative; tongue images judged as unrepresentative based on the experience of the doctor allowed to t were displayed ake appropr again. iThis ate breaks. Interruptions was repeated until the in th entir e experimen e collection t were timed of tongueto com- images were excluded. The display showed images of the tongue randomly extracted from a set plete had been the evaluation of all five assessed. To prevent condit a decline ions for the in evaluation immediately pre capability ceding duesamp to fatigue, le, and the the evaluator was allowed to take appropriate breaks. Interruptions in the experiment were of unex display was amin dark edened d images. Th uring the e evaluator e interruptionv . Sinc alua e the experiment was co ted the images, and the disp nducted in lay turned d ark timed to complete the evaluation of all five conditions for the immediately preceding two locations and it was difficult to unify the light, the experiment was conducted in a when the evaluation ended. Four seconds later, the randomly selected images were dis- sample, and the display was darkened during the interruption. Since the experiment was dark room. By using the above method for assessment, the environment was unified played again. This was repeated until the entire collection of tongue images had been as- conducted in two locations and it was difficult to unify the light, the experiment was among all the evaluators, and the assessment was not affected by factors other than the sessed. To prevent a decline in evaluation capability due to fatigue, the evaluator was conducted in a dark room. By using the above method for assessment, the environment tongue image. This makes machine learning of disease diagnosis from tongue images allowed to take appropriate breaks. Interruptions in the experiment were timed to com- was unified among all the evaluators, and the assessment was not affected by factors other work effectively. Evaluations were then conducted by four other Kampo doctors. Figure than the tongue image. This makes machine learning of disease diagnosis from tongue plete 7 shows the the evaluation of all five percentages of correct valu condit es assi ion gned b s for the y each o imm f the four doctor ediately precs eding for each sample, and the images work effectively. Evaluations were then conducted by four other Kampo doctors. display was darkened during the interruption. Since the experiment was conducted in two locations and it was difficult to unify the light, the experiment was conducted in a dark room. By using the above method for assessment, the environment was unified among all the evaluators, and the assessment was not affected by factors other than the tongue image. This makes machine learning of disease diagnosis from tongue images work effectively. Evaluations were then conducted by four other Kampo doctors. Figure 7 shows the percentages of correct values assigned by each of the four doctors for each Appl. Sci. 2021, 11, x FOR PEER REVIEW 8 of 13 Appl. Sci. 2021, 11, 9716 8 of 13 condition when data modulation was not performed, and the percentages when modula- Figure 7 shows the percentages of correct values assigned by each of the four doctors tion was performed. As the data were obtained from generally healthy subjects, the per- for each condition when data modulation was not performed, and the percentages when centage of correct answers was biased before data modulation, but this bias decreased modulation was performed. As the data were obtained from generally healthy subjects, the after modulation for many conditions. The percentage of each correct value was calculated percentage of correct answers was biased before data modulation, but this bias decreased using Equation (6): after modulation for many conditions. The percentage of each correct value was calculated (%) = 100 (6) using Equation (6): We used a dataset of 275 pairs obtained by modulating the images of the tongues of Number o f each evaluation value Percentage o f correct values (%) =  100 (6) 11 subjects for machine learning, and to assess its accuracy. Number o f images (a) (b) (c) Figure 7. Cont. Appl. Sci. 2021, 11, x FOR PEER REVIEW 9 of 13 Appl. Sci. 2021, 11, 9716 9 of 13 (d) (e) Figure 7. Percentage of values evaluated by each of the four Kampo doctors (A, B, C, and D) for each disease after data Figure 7. Percentage of values evaluated by each of the four Kampo doctors (A, B, C, and D) for each disease after modulation (blue bars), and without data modulation (orange bars): (a) Percentage of each correct value for qi deficiency; data modulation (blue bars), and without data modulation (orange bars): (a) Percentage of each correct value for qi (b) Percentage of each correct value for blood deficiency; (c) Percentage of each correct value for blood stagnation; (d) deficiency; (b) Percentage of each correct value for blood deficiency; (c) Percentage of each correct value for blood stagnation; Percentage of each correct value for yin deficiency; (e) Percentage of each correct value for fluid stagnation. (d) Percentage of each correct value for yin deficiency; (e) Percentage of each correct value for fluid stagnation. 2.7. Feature Selection from 27 Features We used a dataset of 275 pairs obtained by modulating the images of the tongues of As a pre-processing step for machine learning, the following Equation (7) was used 11 subjects for machine learning, and to assess its accuracy. to standardize the acquired features: 2.7. Feature Selection from 27 Features (7) As a pre-processing step for machine learning, the following Equation (7) was used to standardize the acquired features: where is the feature to be standardized, is the feature after standardization, is the mean value of the feature to be standardized, and is the standard deviation of X X mean X = (7) stn the feature to be standardized. std Support vector regression (SVR) was used for machine learning [13]. In machine where X is the feature to be standardized, X is the feature after standardization, X is learning, training with ineffective features m stn ay reduce the accuracy of estimation com- mean the mean value of the feature to be standardized, and X is the standard deviation of the pared with training using only the optimal features. This is because gener std alization perfor- feature to be standardized. mance decreases due to over-learning, and the selection of optimal features plays an im- Support vector regression (SVR) was used for machine learning [13]. In machine portant role in improving the diagnostic accuracy of machine learning. We used the step- learning, training with ineffective features may reduce the accuracy of estimation compared forward selection (SFS) [14] method to choose the best combination of features to evaluate with training using only the optimal features. This is because generalization performance each disease. SFS selects features using the following procedure: the computer learns and decreases due to over-learning, and the selection of optimal features plays an important independently evaluates each image using all features, and then chooses the ones with role in improving the diagnostic accuracy of machine learning. We used the step-forward the highest evaluation accuracy; it then adds one of the remaining features to the selected selection (SFS) [14] method to choose the best combination of features to evaluate each feature and learns and evaluates it. This process was repeated for all features. Features disease. SFS selects features using the following procedure: the computer learns and were added until the evaluation accuracy no longer increased. Appl. Sci. 2021, 11, 9716 10 of 13 independently evaluates each image using all features, and then chooses the ones with the highest evaluation accuracy; it then adds one of the remaining features to the selected feature and learns and evaluates it. This process was repeated for all features. Features were added until the evaluation accuracy no longer increased. The SVR parameters used for training were C = 100, = 0.1, and " = 0.1. Features that were determined to be optimal for the evaluation of each disease are shown in Table 3. The average of the four evaluators was used as the correct value for learning. Table 4 shows the mean absolute error (MAE) between the values of the correct answers by the Kampo doctors and the estimated values obtained by SVR before and after feature selection. The evaluation accuracy increased when feature selection was used. Thus, the feature selection was appropriate. The MAE can be calculated from Equation (8): n1 MAE = jy y ˆ j (8) å i i i=0 where n is the number of evaluations made, y is the ith evaluation value given by the doctor, and y ˆ is the ith prediction value. Table 3. Features selected by step-forward selection for each disease. Diseases Qi Deficiency Blood Deficiency Blood Stagnation Yin Deficiency Fluid Stagnation Min of body b Mean of body b Std of gloss L Median of body b Min of gloss L Median of body b Mean of body b Mean of gloss L Max of gloss L Max of gloss L Max of gloss L Min of gloss L Median of gloss L Table 4. Mean absolute error between the correct value assigned by the Kampo doctors and the estimated value by SVR before and after feature selection for each disease. Diseases Qi Deficiency Blood Deficiency Blood Stagnation Yin Deficiency Fluid Stagnation Using all features 0.51 0.52 0.49 0.50 0.60 Using selected features 0.44 0.42 0.33 0.36 0.55 3. Results Table 5 shows the MAE of the correct and estimated values before and after data modulation when the evaluation value of each Kampo doctor was taken as the correct value. Each disease was diagnosed using features selected through the SFS method and the parameters were optimized by grid searching. Data modulation improved the accuracy of the diagnosis of most diseases. Table 5. Mean absolute error between the correct value (using the value assigned by each Kampo doctor) and the value estimated by support vector regression for each disease using data with no modulation (no modu.) and modulation (modu.). Diseases Qi Blood Blood Yin Fluid Deficiency Deficiency Stagnation Deficiency Stagnation no modu. 0.94 0.75 0.91 1.00 0.62 Doctor.A modu. 0.64 0.71 0.62 0.55 0.77 no modu. 0.99 0.90 0.80 0.85 0.84 Doctor.B modu. 0.70 0.73 0.55 0.57 0.75 Appl. Sci. 2021, 11, 9716 11 of 13 Table 5. Cont. Diseases Qi Blood Blood Yin Fluid Deficiency Deficiency Stagnation Deficiency Stagnation no modu. 1.01 0.93 0.82 0.94 0.73 Doctor.C modu. 0.84 0.99 0.64 0.81 0.90 no modu. 0.95 0.76 0.62 0.91 0.65 Doctor.D modu. 0.83 0.71 0.60 0.86 0.56 no modu. 0.59 0.61 0.53 0.58 0.68 Average of all doctors modu. 0.44 0.42 0.33 0.36 0.55 4. Discussion The correct answers were assigned a value from 0 to 4, with adjacent values having close meanings. Therefore, this method yielded good diagnostic performance when the MAE was less than 1. Table 5 shows that the highest accuracy was achieved by using the average assessment of the four Kampo doctors as the correct value. As shown in Table 6, RMSE also showed high accuracy. This can be attributed to the fact that taking the average suppressed the degree of error and lessened the subjectivity. Also shown in Table 6 is the accuracy of the evaluation using RMSE (root mean squared error), with the average value of the evaluation of the four Kampo doctors as the correct value. As in Table 5, it was shown that the accuracy was improved by modulation and high accuracy was obtained. The distribution of the ratings of the Kampo doctors shown in Figure 7 indicates some evaluation error between doctors. The proposed method was accurate because the error in its results was smaller than 1, and the neighboring assessment values had very close meanings. The features chosen by feature selection were closely related to the aspects of evaluation attended to by the Kampo doctors. The most important features were the tongue gloss, color, and shape in the diagnosis of qi deficiency; color and shape for blood deficiency; color for blood stagnation; gloss for yin deficiency; and shape for fluid stagnation. This shows that the features that were considered important by the Kampo doctors had been selected by the proposed method. However, we did not use the features of form (for example, size and tooth marks). Therefore, it was not possible to determine the dominant features of fluid stagnation, which is highly influenced by form. This was also evident in the final diagnostic results. The diagnostic accuracies for the detection of qi deficiency and fluid stagnation, where shape is important, were worse than those for the other diseases. These results suggest that the addition of new features related to shape may improve the diagnostic accuracy. Table 6. Root mean squared error between the correct value (using average of values assigned by each Kampo doctor) and the value estimated by support vector regression for each disease using data with no modulation (no modu.) and modulation (modu.). Diseases Qi Blood Blood Yin Fluid Deficiency Deficiency Stagnation Deficiency Stagnation no modu. 0.71 0.73 0.62 0.67 0.77 Average of all doctors modu. 0.53 0.50 0.41 0.43 0.56 In this study, tongue images were taken from healthy subjects. However, we were able to efficiently generate tongue data that could be diagnosed as score 4 (sick) by data augmentation. The doctors who evaluated the tongue images commented that the evalua- tion images were realistic, but some of the tongue colors were not realistic. In the future, we would like to improve this point in cooperation with doctors. Appl. Sci. 2021, 11, 9716 12 of 13 The data set created in this study contains only a small number of subjects and only Japanese subjects. In the future, we would like to collect data from more races to create a meaningful dataset that can be released to the public. The limitations of this study are therefore: Some modulated data for data augmentation were not naturally observed tongue data. A polarization camera is required instead of a regular camera. 5. Conclusions We developed an evaluation system that considers the gloss and color components of images of the tongue taken by a polarizing camera to diagnose diseases according to the principles of Kampo medicine. The number of samples was increased from 11 original images to 275 images by modulating the color and gloss. A total of 27 gloss- and color-related features were obtained and combined with subjective evaluation by a Kampo doctor to generate the training data. The MAE of SVR was 0.44 for qi deficiency, 0.42 for blood deficiency, 0.33 for blood stagnation, 0.36 for yin deficiency, and 0.55 for fluid stagnation, when the parameters were optimized by grid searching. The evaluation errors for all diseases were less than 1, indicating reasonable evaluation accuracy. As in Matsushita et al. [6], after each of the four Kampo doctors made a diagnosis, machine learning was performed using each evaluation as the ground truth. In this study, it was found that individual differences were generated among doctors in their evaluation of tongue images. The highest accuracy was obtained by taking the average of the evaluation values of the doctors as the ground truth. This is because errors and blur caused by one doctor ’s diagnosis and knowledge were reduced by taking the average. By using machine learning to reproduce the average of multiple doctors’ evaluations, it is hoped that quantitative diagnosis that eliminates the subjective blurring of individual doctors will become possible. The results of feature selection were consistent with the evaluation values assigned by the Kampo doctors, indicating their correctness. As the diagnosis of fluid stagnation requires numerous features, no useful set of features was available for this condition. Adding the shape of the tongue as a feature may improve the accuracy of the diagnosis. The advantage of this study is that the equipment is simple and easy to reproduce, and the number of data can be easily collected through data augmentation. The results of this study indicate that the polarization camera can be used to diagnose the condition of individual patients. By collecting more data and improving the accuracy of the system, it can be expected to develop into a system that allows doctors to monitor the condition of patients and provide appropriate care without having to diagnose all patients. As for the issues, the later assessments may have been more accurate as the Kampo doctors became familiar with the evaluation of tongue images. Future research should include more stable subjective evaluation methods, such as multiple practice sessions for the Kampo doctors. In future work, more tongue images should be collected and re-evaluated because some of the pathological conditions were biased, as described above. In addition, by adding the features of tongue shape, the accuracy of the proposed method can be improved for the diseases with low diagnostic accuracy in the present study. Another option is to incorporate continuous learning, as implemented by Pianykh et al. [15]. By building a system that can continuously learn from the opinions of more physicians and tongue data with new cases, we can make more objective diagnoses with better generalization performance. Author Contributions: Conceptualization, R.K. and K.O.-O.; formal analysis, R.K.; investigation, K.O.-O., A.S., K.H., J.A. and H.L.; methodology, R.K., K.O.-O. and H.L.; software, R.K.; supervision, N.T.; validation, A.S., K.H. and J.A.; writing—original draft, R.K.; writing—review and editing, K.O.-O., A.S., K.H., J.A. and N.T. All authors have read and agreed to the published version of the manuscript. Appl. Sci. 2021, 11, 9716 13 of 13 Funding: This research received no external funding. Institutional Review Board Statement: This study was approved by the Ethical Board of Kanazawa University, School of Medicine (2018-154). All participants agree to the publication of the results obtained from the study. All results are decoded. Individual results cannot be tracked. Informed Consent Statement: Informed consent was obtained from all subjects involved in the study. Data Availability Statement: The data presented in this study are available on request from the corresponding author. The data are not publicly available due to we have not received permission from the subjects to do so. Acknowledgments: We thank Sakura Matsumoto and Mako Iwahashi for providing the facilities and collecting tongue images. We also thank Kelly Zammit, BVSc, for editing a draft of this manuscript. Conflicts of Interest: The authors declare no conflict of interest. References 1. Sato, Y. Introduction to Kampo: Japanese Traditional Medicine; The Japan Society for Oriental Medicine: Tokyo, Japan, 2005. 2. Mitani, K. The lecture for the diagnosis of Kampo medicine: Boshin, especially for the diagnosis of tongue. Kampo Med. 2007, 58, 673–685. [CrossRef] 3. Satoh, H. An introduction to Kampo medicine. Folia Pharmacol. Jpn. 2008, 132, 260–264. [CrossRef] [PubMed] 4. Yamamoto, S.; Ishilawa, Y.; Nakaguchi, T.; Ogawa-Ochiai, K.; Tsumura, N.; Kasahara, Y.; Namiki, T.; Miyake, Y. Temporal Changes in Tongue Color as Criterion for Tongue Diagnosis in Kampo Medicine. Forsch. Komplementmed. 2012, 19, 80–85. [CrossRef] [PubMed] 5. Wang, Y.; Zhou, Y.; Yang, J.; Xu, Q. An image analysis system for tongue diagnosis in traditional Chinese medicine. Comput. Inf. Sci. 2005, 3314, 1181–1186. 6. Matsushita, F.; Kiyomitsu, K.; Ogawa, K.; Tsumura, N. Evaluation of Kampo disease states using facial images. Artif. Life Robot. 2019, 24, 44–51. [CrossRef] 7. Zhang, J.; Xu, J.; Hu, X.; Chen, Q.; Tu, L.; Huang, J.; Cui, J. Diagnostic method of diabetes based on support vector machine and tongue images. BioMed Res. Int. 2017, 2017, 7961494. [CrossRef] [PubMed] 8. Kainuma, M.; Furusyo, N.; Urita, Y.; Ikezaki, H.; Ura, K.; Nakaguchi, T.; Namiki, T. Objective tongue color and gastroesophageal reflux disease: Cross-sectional study. Tradit. Kampo Med. 2019, 6, 19–25. [CrossRef] 9. Nakaguchi, T.; Ishilawa, Y.; Oji, T.; Yamamoto, S.; Tsumura, N.; Ueda, K.; Nagamine, K.; Namiki, T.; Miyake, Y. Proposal for a new noncontact method for measuring tongue moisture to assist in tongue diagnosis and development of the tongue image analyzing system, which can separately record the gloss components of the tongue. BioMed Res. Int. 2015, 2015, 249609. [CrossRef] [PubMed] 10. Danser, M.M.; Mantilla Gomez, S.; Van der Weijden, G.A. Tongue coating and tongue brushing: A literature review. Int. J. Dent. Hyg. 2003, 1, 151–158. [CrossRef] [PubMed] 11. Xu, J.; Tu, L.; Zhang, Z.; Zhou, C. The region partition of quality and coating for tongue image based on color image segmentation method. In Proceedings of the 2008 IEEE International Symposium on IT in Medicine and Education, Xiamen, China, 12–14 December 2008; pp. 817–821. 12. Tsumura, N.; Ojima, N.; Sato, K.; Shiraishi, M.; Shimizu, H. Image-based skin color and quality analysis/synthesis by extracting hemoglobin and melanin information in the skin. ACM Trans. Graph. 2003, 22, 770–779. [CrossRef] 13. Burges, C.J. A Tutorial on support vector machines for pattern recognition. Data Min. Knowl. Discov. 1998, 2, 121–167. [CrossRef] 14. Leslie, K.P. An Introduction to Variable and Feature Selection. J. Mach. Learn. Res. 1998, 3, 1157–1182. 15. Pianykh, O.S.; Langs, G.; Dewey, M.; Enzmann, D.R.; Herold, C.J.; Schoenberg, S.O.; Brink, J.A. A tutorial on support vector machines for pattern recognition. Radiology 2020, 297, 6–14. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Applied Sciences Multidisciplinary Digital Publishing Institute

Creating a Diagnostic Assistance System for Diseases in Kampo Medicine

Loading next page...
 
/lp/multidisciplinary-digital-publishing-institute/creating-a-diagnostic-assistance-system-for-diseases-in-kampo-medicine-omcJOX96bR
Publisher
Multidisciplinary Digital Publishing Institute
Copyright
© 1996-2021 MDPI (Basel, Switzerland) unless otherwise stated Disclaimer The statements, opinions and data contained in the journals are solely those of the individual authors and contributors and not of the publisher and the editor(s). MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. Terms and Conditions Privacy Policy
ISSN
2076-3417
DOI
10.3390/app11219716
Publisher site
See Article on Publisher Site

Abstract

applied sciences Article Creating a Diagnostic Assistance System for Diseases in Kampo Medicine 1 , 2 3 4 5 6 Reimei Koike *, Keiko Ogawa-Ochiai , Akiko Shirai , Katsumi Hayashi , Junsuke Arimitsu , Hongyang Li and Norimichi Tsumura Division of Creative Engineering, Graduate School of Science and Engineering, Chiba 263-8522, Japan Kampo Clinical Center, Department of General Medicine, Hiroshima University Hospital 1-2-3, Kasumi, Minami-ku, Hiroshima 734-8551, Japan; okeiko22@hiroshima-u.ac.jp Department of Otolaryngology, Head and Neck Surgery, Kanazawa University Hospital, Kanazawa 920-8641, Japan; shiraiaki@med.kanazawa-u.ac.jp Honjo Waseda Clinic, Saitama 367-0030, Japan; kachan18@gmail.com Senrioka Kagayaki Clinic, Osaka 565-0816, Japan; arimitsu@hiroshima-u.ac.jp Department of Japanese Traditional (Kampo) Medicine, Kanazawa University Hospital, Kanazawa 920-8641, Japan; hoyalee328@stu.kanazawa-u.ac.jp Graduate School of Engineering, Chiba University, Chiba 263-8522, Japan; tsumura@faculty.chiba-u.jp * Correspondence: reimei0217@chiba-u.jp Abstract: The aim of this study was to propose a method to assess images of the tongue captured using a polarized light camera for diagnostic use in Kampo medicine. Glossy and non-glossy images of the tongue were captured simultaneously using a polarizing camera and a polarizing plate. Data augmentation was performed by modulating the color and gloss, resulting in an increase in the number of images from 11 to 275. To create a data set, the values for which diseases were evaluated by Kampo doctors for all tongue images were taken as the correct values and combined with the Citation: Koike, R.; Ogawa-Ochiai, features extracted from the tongue images. Using this data set, we constructed a diagnostic support K.; Shirai, A.; Hayashi, K.; Arimitsu, module to evaluate diseases. The resulting mean absolute error of the assessment was 0.44 for qi J.; Li, H.; Tsumura, N. Creating a deficiency, 0.42 for blood deficiency, 0.33 for blood stagnation, 0.36 for yin deficiency, and 0.55 for Diagnostic Assistance System for Diseases in Kampo Medicine. Appl. fluid stagnation, suggesting that the diagnostic assistance module was accurate, and our proposed Sci. 2021, 11, 9716. https://doi.org/ learning and data augmentation methods were effective. 10.3390/app11219716 Keywords: machine learning; Kampo medicine; tongue; diagnostic assistance Academic Editor: Andrew Teoh Beng Jin Received: 27 August 2021 1. Introduction Accepted: 9 October 2021 The general medical care received in hospitals worldwide is based on Western medicine. Published: 20 October 2021 However, in China and other East Asian countries, the field of traditional medicine—called Kampo medicine in Japan—has developed over the past 1500 years. Western medicine and Publisher’s Note: MDPI stays neutral Kampo medicine differ greatly in their diagnostic methods. In Kampo medicine, the pathol- with regard to jurisdictional claims in ogy is assessed using the three elements of “qi,” “blood,” and “fluid” that comprise the qi, published maps and institutional affil- blood, and fluid theory [1]. A healthy state is one in which these three elements circulate iations. normally and in sufficient quantities. Qi represents the energy of the entire body, blood rep- resents the blood and its flow through the body, and fluid represents fluids other than the blood (such as lymph and digestive fluids) and its flow through the body. The conditions in which qi, blood, and fluid are “insufficient” are called “qi deficiency,” “blood deficiency,” Copyright: © 2021 by the authors. and “yin deficiency,” respectively, and those in which qi, blood, and fluid are “stagnated” Licensee MDPI, Basel, Switzerland. are called “qi stagnation,” “blood stagnation,” and “fluid stagnation,” respectively [2]. This article is an open access article Doctors practicing Kampo medicine diagnose diseases via inspection, inquiry, palpa- distributed under the terms and tion, and aural and olfactory examinations [3]. Inspection, especially visual examination, is conditions of the Creative Commons among the most important diagnostic methods in Kampo medicine but is rarely applied Attribution (CC BY) license (https:// in Western medicine. One reason for this is that it is not possible to make a quantitative creativecommons.org/licenses/by/ diagnosis because inspection is subjective and based on the doctor ’s experience. 4.0/). Appl. Sci. 2021, 11, 9716. https://doi.org/10.3390/app11219716 https://www.mdpi.com/journal/applsci Appl. Sci. 2021, 11, 9716 2 of 13 To solve this problem, a quantitative descriptive diagnosis method has been formu- lated using image processing in recent years [4,5]. By using a computer to perform an advisory diagnosis based on images of the face and tongue, it is possible to quantify the diagnosis without prior experience and identify the relevant disease. In addition, Kampo medicine contains many concepts useful for preventive medicine. The qi deficiency, blood deficiency, blood stagnation, yin deficiency, and fluid stagnation that we evaluated indicate imbalances in the body. In Kampo medicine, doctors take appropriate treatments to adjust these imbalances in order to prevent diseases. There are no clear distinction between healthy people and patients; in this study, the closer to 0, the better the condition is, and the closer to 4, the worse the condition is. About 0 indicates a healthy state, 1 and 2 indicate conditions that require follow-up, and 3 and 4 indicate conditions that require treatment., but there are individual differences among doctors. Table 1 lists diseases that can be diagnosed based on an inspection of the face and tongue of the patient using Kampo medicine. To computerize descriptive diagnosis, Matsushita et al. [6] proposed a method to assess diseases of Kampo medicine using the facial images of patients. Blood deficiency, blood stagnation, and yin deficiency can be diagnosed using facial images. However, this facial image assessment might not be sufficient to achieve a Kampo medical diagnosis. Observations of the patient’s tongue increases the number of conditions that can be diagnosed using computer image analysis. By observing the color, fluid content, and shape of the tongue, and the condition of the tongue coating, five of the six conditions—qi stagnation, qi deficiency, blood deficiency, blood stagnation, yin deficiency, and fluid stagnation—can be diagnosed. Table 1. Body parts (tongue and face) used to diagnose each of the six diseases in Kampo medicine. Qi Blood Fluid Face Face Deficiency Tongue Tongue Tongue Face Stagnation Tongue Tongue As the tongue is closely related to the internal organs with exposed mucous mem- branes, it is regarded as an important indicator of many diseases in Kampo medicine. The tongue is very glossy under normal lighting conditions owing to its moist surface, which makes it difficult to obtain accurate information on the color of the tongue with normal imaging methods. However, Kampo doctors consider gloss to be just as important as color information in making a diagnosis. Therefore, to perform a computerized tongue-based diagnosis, it is necessary to use an imaging method that can accurately acquire information on both color and gloss. In the latest study of diagnosis from the tongue, Xu et al. [7] collected tongue images from about 1000 people using a device that can perform gloss removal imaging to diagnose diabetes. Kainuma et al. [8] conducted a study to estimate the Frequency scale for the symptoms of gastroesophageal reflux disease from images of the tongue. However, these recent studies have the problem of gloss removal: in these studies, the gloss of the tongue was completely removed, which is different from the Kampo doctor ’s observation. To solve this problem, Nakaguchi et al. [9] used a combination of the integrating sphere method and directional illumination to capture images of glossy and non-glossy tongues in order to estimate the moisture content of the tongue. However, because the integrating sphere is large and expensive, it is not appropriate for personal use. In the present study, we photographed the tongue using a polarizing camera that simultaneously captures glossy and non-glossy images with more compact and cheaper equipment than an integrating sphere which is conventionally used. In addition, to collect tongue images more efficiently, we proposed a physiologically compensated and effective data augmentation method. We then used these images to assess each patient’s diseases. Appl. Sci. 2021, 11, 9716 3 of 13 A system of quantitative evaluation was constructed by machine learning of the values and features obtained from the tongue images. The accuracy of the evaluation system was verified via comparison with the distribution and variability of the assessments by four Kampo medicine specialists. This research provided a quantitative diagnostic method for Kampo diseases from the tongue using image processing and machine learning techniques. The contributions and novelty of this paper are therefore as follows: We proposed a tongue imaging method using a polarized light camera that is simpler and more effective than the conventional method. In particular, the use of polarized light cameras provided accurate gloss information and showed their effectiveness in diagnosing yin deficiency and fluid stagnation, conditions closely related to fluid content, as shown in Section 2 We proposed a physiologically compensated and effective data augmentation method. We show that the proposed data augmentation method improved the distribution of correct values in Section 4. In addition, the estimation accuracy was improved by the data augmentation. These results indicate the effectiveness of our method. 2. Materials and Methods 2.1. Capturing Glossy and Non-Glossy Images Using Polarized Light Incident light from a source is reflected from the surface of an object. This reflection of the light is classified into two types: surface reflection, where the angles of incidence and reflection of light are the same with respect to the surface of reflection, and internal diffuse reflection caused by the scattering of the incident light inside the object. The human eye and a camera perceive the color and luster of an object by capturing the reflected light. In general, an object appears colored owing to internal diffuse reflection, while luster is caused by surface reflection. Light is an electromagnetic wave that oscillates in a direction perpendicular to its direction of travel, and normal light is in an unpolarized or partially polarized state, which is a mixture of light oscillating in all directions. Polarization, conversely, refers to light that oscillates only in a certain direction. Passing ordinary unpolarized light through an optical filter called a polarizing plate creates polarized light in a direction identical to that of the polarizing plate. This property of polarization has important links to the surface reflection and internal diffusion. Light is reflected while the polarization of the incident light is maintained; when the incident light is reflected off the surface of the object, it is linearly polarized in the direction of the polarizing plate, as in the case of incident light. Conversely, internal diffuse light is unpolarized even if the incident light is linearly polarized when internally reflected, because the incident light is reflected in various directions and the light is scattered. We used this property to obtain glossy and non-glossy images. Figure 1 shows the method used to obtain an image with enhanced gloss and one with reduced gloss using two orthogonal polarizing plates. S indicates the surface-reflected light, D indicates diffuse reflected light, and the subscripts p and s indicate the directions of orthogonal polarization. Since S in Figure 1a maintains the same polarization as the incident light, the relationship between the intensity of S and that of the p-polarization component (S ) of S is shown in Equation (1), where I indicates the intensity of x: p x I = I (1) S S The diffuse component (D) in Figure 1a is the loss of polarization of the incident light. The ratio of intensity of the p-polarization component D to that of the s-polarization component (D ) in D is 1:1, and can be expressed as: I = I + I = 2I = 2I (2) D D D D D p s p s Appl. Sci. 2021, 11, x FOR PEER REVIEW 4 of 13 = =2 =2 (2) Figure 1b shows the setup used to attain a gloss-enhanced image by placing a second polarizing plate between the object and the camera. When light passes through the p-po- larizing plate, the component of s-polarization in the vertical direction cannot pass through the p-polarizing plate. Thus, the internally reflected light (D), which contains , has half the intensity; however, , which has only the p-polarization component, can pass through the p-polarizing plate illumination and the object. In the present study, we used a polarizing camera instead of a second polarizing plate placed in front of the cam- era. The polarizing camera can capture 0°, 45°, 90°, and 145° polarization images simulta- Appl. Sci. 2021, 11, 9716 4 of 13 neously, enabling the capture of a gloss-enhanced image from a 0° polarization image and a non-glossy image from a 90° polarization image simultaneously. (a) (b) (c) Figure 1. Acquisition of surface and internal reflection using polarizing plate. (a) Polarization component of surface re- Figure 1. Acquisition of surface and internal reflection using polarizing plate. (a) Polarization component of surface flection and diffuse reflection; (b) Taking a gloss-enhanced image; (c) Taking a non-glossy image. reflection and diffuse reflection; (b) Taking a gloss-enhanced image; (c) Taking a non-glossy image. 2.2. SFigur eparate ing th 1b shows e Tongthe ue Coat setup ing fr used om t to he attain Tongue a B gloss-enhanced ody image by placing a sec- ond polarizing plate between the object and the camera. When light passes through the The surface of the tongue can be categorized into the coating and the body. The coat- p-polarizing plate, the component of s-polarization in the vertical direction cannot pass ing of the tongue refers to the part coated with food debris, mucosal cells, and bacteria on through the p-polarizing plate. Thus, the internally reflected light (D), which contains tissues called filamentous papillae, and in the gaps between protrusions on the surface of D , has half the intensity; however, S , which has only the p-polarization component, s p the tongue. The body of the tongue refers to the part that is not covered by the coating of can pass through the p-polarizing plate illumination and the object. In the present study, the tongue [10]. The color of the lichen of the tongue can be white, yellow, gray, and black, we used a polarizing camera instead of a second polarizing plate placed in front of the while the body of the tongue can be light-red, red, crimson, and purple. As the lichen and camera. The polarizing camera can capture 0 , 45 , 90 , and 145 polarization images body of the tongue show different color changes, it is necessary to distinguish between simultaneously, enabling the capture of a gloss-enhanced image from a 0 polarization them when evaluating tongue color. image and a non-glossy image from a 90 polarization image simultaneously. Xu et al. [11] focused on the fact that the tongue has a large proportion of red and so the R component of the RGB color space tends to be prominent, whereas the lichen of the 2.2. Separating the Tongue Coating from the Tongue Body tongue does not have a prominent R component. This previous study calculated I for all The surface of the tongue can be categorized into the coating and the body. The pixels in the tongue image using the following equation: coating of the tongue refers to the part coated with food debris, mucosal cells, and bacteria .. on tissues called filamentous papillae, and in the gaps between protrusions on the surface = − . (3) of the tongue. The body of the tongue refers to the part that is not covered by the coating of All pixels were classified as belonging to the body or coating of the tongue according the tongue [10]. The color of the lichen of the tongue can be white, yellow, gray, and black, to Equation (4). Taking as the mean value of I classified as belonging to the body of while the body of the tongue can be light-red, red, crimson, and purple. As the lichen and the tongue, Mean( ) as the mean value of I classified as belonging to the tongue body, body of the tongue show different color changes, it is necessary to distinguish between and Mean( ) as the mean value of I classified as belonging to the tongue coating, them when evaluating tongue color. is determi Xu etnal. ed to ma [11] focused ximize on Eqthe uati fact on (5 that ): the tongue has a large proportion of red and so the R component of the RGB color space tends to be prominent, whereas the lichen of the ( ) = , tongue does not have a prominent R component. This previous study calculated I for(4 all ) ( ) pixels in the tongue image using the following equation: ( ): ( )− ( ) . (5) 1.5G + 0.5B I = R . (3) 2.3. Extracting Feature Values from Tongue Images All pixels were classified as belonging to the body or coating of the tongue according to Equation (4). Taking I as the mean value of I classified as belonging to the body of body the tongue, Mean(I ) as the mean value of I classified as belonging to the tongue body, body and Mean(I ) as the mean value of I classified as belonging to the tongue coating, I is coating m determined to maximize Equation (5): I (i f I  I ) body I = , (4) I (i f I < I ) coating Maximize(I ) : Mean(I ) Mean(I ) . (5) body coating 2.3. Extracting Feature Values from Tongue Images We then extracted feature values from the tongue images for machine learning. The first step is to manually extract the tongue region from whole image. After that, as mentioned above, the two images with and without gloss were used to obtain three Appl. Sci. 2021, 11, x FOR PEER REVIEW 5 of 13 We then extracted feature values from the tongue images for machine learning. The first step is to manually extract the tongue region from whole image. After that, as men- Appl. Sci. 2021, 11, 9716 5 of 13 tioned above, the two images with and without gloss were used to obtain three images— (a) gloss component, (b) tongue coating, and (c) tongue body—as shown in Figure 2. From (a), we obtained the mean, standard deviation, and maximum, minimum, and median images—(a) gloss component, (b) tongue coating, and (c) tongue body—as shown in ∗ ∗ ∗ ∗ values of in the color space of the glossy area and the percentages of the total Figure 2. From (a), we obtained the mean, standard deviation, and maximum, minimum, area of the tongue by gloss. Since the area and brightness of the gloss on the tongue surface and median values of L in the L a b color space of the glossy area and the percentages changes with the amount of fluid, features that indicate these changes are obtained from of the total area of the tongue by gloss. Since the area and brightness of the gloss on the tongue surface changes with the amount of fluid, features that indicate these changes are the gloss image. From (b) and (c), we obtained the mean, standard deviation, and maxi- obtained from the gloss image. From (b) and (c), we∗obtained the ∗ mean, standar ∗ ∗ d ∗ deviation, mum, minimum, and median values of and in the color space of the and maximum, minimum, and median values of a and b in the L a b color space of the tongue coating. Changes in blood status can be seen in the color of the tongue body. In tongue coating. Changes in blood status can be seen in the color of the tongue body. In addition, the area and color of the tongue coating changes with physical condition and addition, the area and color of the tongue coating changes with physical condition and lifestyle. Therefore, we obtained the feature values that represent the color and coating lifestyle. Therefore, we obtained the feature values that represent the color and coating are area. a. As As shown shown in T in Table able 2, a 2total , a to of ta27 l of types 27 tof ypes of feature fe values ature va werlue e obtained. s were obtained. (a) (b) (c) Figure 2. Separation of the tongue into three components: (a) glossy image, (b) tongue coating, and Figure 2. Separation of the tongue into three components: (a) glossy image, (b) tongue coating, and (c) tongue body. (c) tongue body. Table 2. Total (27) features obtained from each separated image. Table 2. Total (27) features obtained from each separated image. From Glossy Image From Tongue Coating From Tongue Body From Glossy Image From Tongue Coating From Tongue Body Mean of gloss L Mean of coating a , b Mean of body a , b ∗ ∗ ∗ ∗ ∗ Mean of gloss Mean of coating , Mean of body , Max of gloss L Max of coating a , b Max of body a , b ∗ ∗ ∗ ∗ ∗ Max of gloss Max of coating , Max of body , Min of gloss L Min of coating a , b Min of body a , b ∗ ∗ ∗ ∗ ∗ Median of gloss L Median of coating a , b Median of body a , b Min of gloss Min of coating , Min of body , Std of gloss L Std of coating a , b Std of body a , b ∗ ∗ ∗ ∗ ∗ Median of gloss Median of coating , Median of body , Gross area Coating area ∗ ∗ ∗ ∗ ∗ Std of gloss Std of coating , Std of body , Gross area 2.4. Set-Up for Capturing Tongue Coating are Images a Figure 3 shows the photographic environment in which the learning data were col- lected. A polarizing camera (Lucid, PHX050S-Q) was used, with a resolution per polarized 2.4. Set-Up for Capturing Tongue Images image of 1224  1024 pixels, bit depth of 16 bits, exposure time of 8000 s, and distance Figure 3 shows the photographic environment in which the learning data were col- between the camera and the subject of 73 cm. The images were taken in a dark room with lected. A polarizing camera (Lucid, PHX050S-Q) was used, with a resolution per polarized an LED light source (Neewer, PT-176S) with a polarizing plate. Eleven healthy subjects image of 1224 × 1024 pixels, bit depth of 16 bits, exposure time of 8000 µs, and distance were photographed. between the camera and the subject of 73 cm. The images were taken in a dark room with an LED light source (Neewer, PT-176S) with a polarizing plate. Eleven healthy subjects were photographed. Appl. Sci. 2021, 11, x FOR PEER REVIEW 6 of 13 Appl. Sci. 2021, 11, 9716 6 of 13 Appl. Sci. 2021, 11, x FOR PEER REVIEW 6 of 13 Figure 3. Experimental setup used to capture the tongue images. 2.5. Data Augmentation by Color and Gloss Modulation As the number of captured tongue images was insufficient for machine learning, we performed a physiologically corrected modulation of each sample to produce more train- ing data. Data augmentation was performed by applying color and gloss modulation to each captured image. The color of the tongue is determined by the amount of hemoglobin in the blood. Therefore Figure 3., m Experimental setup odulating the hemog used to lobin capture the tongue images. components enables the creation of images with Figure 3. Experimental setup used to capture the tongue images. different tongue colors. To extract the components of hemoglobin from non-glossy tongue 2.5. Data Augmentation by Color and Gloss Modulation images, we separated the components of pigmentation using the method proposed by 2.5. Data Augmentation by Color and Gloss Modulation Tsumura As the et al number . [12]. The col of captur or- ed modula tongue ted i images mages of the tongue was insufficient we for re obtain machine ed b learning, y multi- we As the number of captured tongue images was insufficient for machine learning, we plying the hemoglobin performed a physiologically component by corrected modulation the color modu of each lati sample on factor ( to pr oduce ) in log space more training and performed a physiologically corrected modulation of each sample to produce more train- data. Data augmentation was performed by applying color and gloss modulation to each resynthesizing the components of melanin and shading by reverse processing the separa- ing data. Data augmentation was performed by applying color and gloss modulation to captured image. tion of the pigment component. Figure 4a–e shows color-modulated images with each captured image. The color of the tongue is determined by the amount of hemoglobin in the blood. ( , , , , ) = (0.6, 0.8, 1.0, 1.2, 1.4), respectively. Therefor The color of e, modulating the tongue is determined the hemoglobin components by enables the amount of hemoglobin in the blood. the creation of images with As both the glossy and non-glossy images were captured simultaneously, the glossy different tongue colors. To extract the components of hemoglobin from non-glossy tongue component was e Therefore, modulating asily extr the acted by subt hemoglobin ractin compon g the non-glo ents ssy enaimage from the g bles the creation of lossy im- images with images, we separated the components of pigmentation using the method proposed by age. The image of the tongue with modulated gloss was obtained by multiplying the ex- different tongue colors. To extract the components of hemoglobin from non-glossy tongue Tsumura et al. [12]. The color-modulated images of the tongue were obtained by multi- tracted components of gloss with the gloss modulation factor ( ), as well as the color images, we separated the components of pigmentation usi ng the method proposed by plying the hemoglobin component by the color modulation factor (a ) in log space and modulation, and resynthesizing the image with no gloss. Figure 5a–e shows the gloss- Tsumura et al. [12]. The color-modulated images of the tongue were obtained by multi- resynthesizing the components of melanin and shading by reverse processing the sep- modulated images for ( , , , , ) = (0.0, 0.3, 0.6, 1.0, 1.5), respectively. As shown in plying the hemoglobin component by the color modulation factor ( ) in log space and aration of the pigment component. Figure 4a–e shows color-modulated images with Figure 6, by combining these two modulation methods, the number of original images of resynthesizing the components of melanin and shading by reverse processing the separa- (a , a , a , a , a ) = (0.6, 0.8, 1.0, 1.2, 1.4), respectively. 1 2 3 4 5 the tongue was augmented by a factor of 25. tion of the pigment component. Figure 4a–e shows color-modulated images with ( , , , , ) = (0.6, 0.8, 1.0, 1.2, 1.4), respectively. As both the glossy and non-glossy images were captured simultaneously, the glossy component was easily extracted by subtracting the non-glossy image from the glossy im- age. The image of the tongue with modulated gloss was obtained by multiplying the ex- tracted components of gloss with the gloss modulation factor ( ), as well as the color modulation, and resynthesizing the image with no gloss. Figure 5a–e shows the gloss- modulated images for ( , , , , ) = (0.0, 0.3, 0.6, 1.0, 1.5), respectively. As shown in (a) =0.6 (b) =0.8 (c) =1.0 (d) =1.2 (e) =1.4 Figure 6, by combining these two modulation methods, the number of original images of Figure 4. Five levels of color modulation using the modulation coefficient . Figure 4. Five levels of color modulation using the modulation coefficient a . the tongue was augmented by a factor of 25. As both the glossy and non-glossy images were captured simultaneously, the glossy component was easily extracted by subtracting the non-glossy image from the glossy image. The image of the tongue with modulated gloss was obtained by multiplying the extracted components of gloss with the gloss modulation factor (b ), as well as the color modulation, and resynthesizing the image with no gloss. Figure 5a–e shows the gloss-modulated images for (b , b , b , b , b ) = (0.0, 0.3, 0.6, 1.0, 1.5), respectively. As shown in Figure 6, 1 2 3 4 5 by combining these two modulation methods, the number of original images of the tongue was augmented by a factor of 25. (a) =0.6 (b) =0.8 (c) =1.0 (d) =1.2 (e) =1.4 Figure 4. Five levels of color modulation using the modulation coefficient . Appl. Sci. 2021, 11, x FOR PEER REVIEW 7 of 13 Appl. Sci. 2021, 11, 9716 7 of 13 Appl. Sci. 2021, 11, x FOR PEER REVIEW 7 of 13 (a) =0.0 (b) =0.3 (c) =0.6 (d) =1.0 (e) =1.5 (a) =0.0 (b) =0.3 (c) =0.6 (d) =1.0 (e) =1.5 Figure 5. Five levels of gloss modulation using the modulation coefficient . Figure 5. Five levels of gloss modulation using the modulation coefficient . Figure 5. Five levels of gloss modulation using the modulation coefficient b . Figure 6. Data augmentation by color and gloss modulation. 2.6. Subjective Evaluation of Diseases by Kampo Doctors Using Tongue Images Figure 6. Data augmentation by color and gloss modulation. Figure 6. One doctor Data augmentation who specialized by color and gloss mo in Kampo medicine dulation (the evaluator) ev . aluated each tongue 2.6. Subjective image to diagno Evaluation se t ofhDiseases e subjects. T by Kampo he evaluation Doctors Using was pe Tongue rformeImages d in a darkened room, with 80 cm between the display and the evaluator, and a display size of 51 cm. The screen One doctor who specialized in Kampo medicine (the evaluator) evaluated each tongue 2.6. Subjective Evaluation of Diseases by Kampo Doctors Using Tongue Images was displayed in device-independent sRGB color. There was no other lighting in the dark- image to diagnose the subjects. The evaluation was performed in a darkened room, One doctor who specialized in Kampo medicine (the evaluator) evaluated each ened room, and the Kampo doctor evaluated the results using only the display light. The with 80 cm between the display and the evaluator, and a display size of 51 cm. The tongue evaluator imag assessed e to diagno the five condit se the subjects. T ions that can be diag he evaluation nosed from images was performe of the tongue— d in a darkened room, screen was displayed in device-independent sRGB color. There was no other lighting in qi deficiency, blood deficiency, blood stagnation, yin deficiency, and fluid stagnation—as wi the th 80 cm between the di darkened room, and the Kampo splay doctor and the evaluated evaluathe torr , esults and a d using isponly lay size the display of 51 cm. The screen values of 0, 1, 2, 3, and 4, with smaller values indicating greater severity. As the data were light. The evaluator assessed the five conditions that can be diagnosed from images of was displayed in device-independent sRGB color. There was no other lighting in the dark- augmented by modulation, this produced some images of tongues that were unrepre- the tongue—qi deficiency, blood deficiency, blood stagnation, yin deficiency, and fluid ened room, and the Kampo doctor evaluated the results using only the display light. The sentative; tongue images judged as unrepresentative based on the experience of the doctor stagnation—as values of 0, 1, 2, 3, and 4, with smaller values indicating greater severity. As evaluator assessed the five conditions that can be diagnosed from images of the tongue— were excluded. The display showed images of the tongue randomly extracted from a set the data were augmented by modulation, this produced some images of tongues that were qi of unex defici amin ency, ed images. Th blood dee evaluator e ficiency, bloo valua d st ted the agnaimages, tion, yiand the disp n deficiency lay turned d , and fluid stagn ark ation—as unrepresentative; tongue images judged as unrepresentative based on the experience of when the evaluation ended. Four seconds later, the randomly selected images were dis- the doctor were excluded. The display showed images of the tongue randomly extracted values of 0, 1, 2, 3, and 4, with smaller values indicating greater severity. As the data were playe fromd a again. This was rep set of unexamined e images. ated until th The evaluator e entire collec evaluated tion of tongue imag the images, and es ha the d been as- display augmented by modulation, this produced some images of tongues that were unrepre- sessed. To prevent a decline in evaluation capability due to fatigue, the evaluator was turned dark when the evaluation ended. Four seconds later, the randomly selected images sentative; tongue images judged as unrepresentative based on the experience of the doctor allowed to t were displayed ake appropr again. iThis ate breaks. Interruptions was repeated until the in th entir e experimen e collection t were timed of tongueto com- images were excluded. The display showed images of the tongue randomly extracted from a set plete had been the evaluation of all five assessed. To prevent condit a decline ions for the in evaluation immediately pre capability ceding duesamp to fatigue, le, and the the evaluator was allowed to take appropriate breaks. Interruptions in the experiment were of unex display was amin dark edened d images. Th uring the e evaluator e interruptionv . Sinc alua e the experiment was co ted the images, and the disp nducted in lay turned d ark timed to complete the evaluation of all five conditions for the immediately preceding two locations and it was difficult to unify the light, the experiment was conducted in a when the evaluation ended. Four seconds later, the randomly selected images were dis- sample, and the display was darkened during the interruption. Since the experiment was dark room. By using the above method for assessment, the environment was unified played again. This was repeated until the entire collection of tongue images had been as- conducted in two locations and it was difficult to unify the light, the experiment was among all the evaluators, and the assessment was not affected by factors other than the sessed. To prevent a decline in evaluation capability due to fatigue, the evaluator was conducted in a dark room. By using the above method for assessment, the environment tongue image. This makes machine learning of disease diagnosis from tongue images allowed to take appropriate breaks. Interruptions in the experiment were timed to com- was unified among all the evaluators, and the assessment was not affected by factors other work effectively. Evaluations were then conducted by four other Kampo doctors. Figure than the tongue image. This makes machine learning of disease diagnosis from tongue plete 7 shows the the evaluation of all five percentages of correct valu condit es assi ion gned b s for the y each o imm f the four doctor ediately precs eding for each sample, and the images work effectively. Evaluations were then conducted by four other Kampo doctors. display was darkened during the interruption. Since the experiment was conducted in two locations and it was difficult to unify the light, the experiment was conducted in a dark room. By using the above method for assessment, the environment was unified among all the evaluators, and the assessment was not affected by factors other than the tongue image. This makes machine learning of disease diagnosis from tongue images work effectively. Evaluations were then conducted by four other Kampo doctors. Figure 7 shows the percentages of correct values assigned by each of the four doctors for each Appl. Sci. 2021, 11, x FOR PEER REVIEW 8 of 13 Appl. Sci. 2021, 11, 9716 8 of 13 condition when data modulation was not performed, and the percentages when modula- Figure 7 shows the percentages of correct values assigned by each of the four doctors tion was performed. As the data were obtained from generally healthy subjects, the per- for each condition when data modulation was not performed, and the percentages when centage of correct answers was biased before data modulation, but this bias decreased modulation was performed. As the data were obtained from generally healthy subjects, the after modulation for many conditions. The percentage of each correct value was calculated percentage of correct answers was biased before data modulation, but this bias decreased using Equation (6): after modulation for many conditions. The percentage of each correct value was calculated (%) = 100 (6) using Equation (6): We used a dataset of 275 pairs obtained by modulating the images of the tongues of Number o f each evaluation value Percentage o f correct values (%) =  100 (6) 11 subjects for machine learning, and to assess its accuracy. Number o f images (a) (b) (c) Figure 7. Cont. Appl. Sci. 2021, 11, x FOR PEER REVIEW 9 of 13 Appl. Sci. 2021, 11, 9716 9 of 13 (d) (e) Figure 7. Percentage of values evaluated by each of the four Kampo doctors (A, B, C, and D) for each disease after data Figure 7. Percentage of values evaluated by each of the four Kampo doctors (A, B, C, and D) for each disease after modulation (blue bars), and without data modulation (orange bars): (a) Percentage of each correct value for qi deficiency; data modulation (blue bars), and without data modulation (orange bars): (a) Percentage of each correct value for qi (b) Percentage of each correct value for blood deficiency; (c) Percentage of each correct value for blood stagnation; (d) deficiency; (b) Percentage of each correct value for blood deficiency; (c) Percentage of each correct value for blood stagnation; Percentage of each correct value for yin deficiency; (e) Percentage of each correct value for fluid stagnation. (d) Percentage of each correct value for yin deficiency; (e) Percentage of each correct value for fluid stagnation. 2.7. Feature Selection from 27 Features We used a dataset of 275 pairs obtained by modulating the images of the tongues of As a pre-processing step for machine learning, the following Equation (7) was used 11 subjects for machine learning, and to assess its accuracy. to standardize the acquired features: 2.7. Feature Selection from 27 Features (7) As a pre-processing step for machine learning, the following Equation (7) was used to standardize the acquired features: where is the feature to be standardized, is the feature after standardization, is the mean value of the feature to be standardized, and is the standard deviation of X X mean X = (7) stn the feature to be standardized. std Support vector regression (SVR) was used for machine learning [13]. In machine where X is the feature to be standardized, X is the feature after standardization, X is learning, training with ineffective features m stn ay reduce the accuracy of estimation com- mean the mean value of the feature to be standardized, and X is the standard deviation of the pared with training using only the optimal features. This is because gener std alization perfor- feature to be standardized. mance decreases due to over-learning, and the selection of optimal features plays an im- Support vector regression (SVR) was used for machine learning [13]. In machine portant role in improving the diagnostic accuracy of machine learning. We used the step- learning, training with ineffective features may reduce the accuracy of estimation compared forward selection (SFS) [14] method to choose the best combination of features to evaluate with training using only the optimal features. This is because generalization performance each disease. SFS selects features using the following procedure: the computer learns and decreases due to over-learning, and the selection of optimal features plays an important independently evaluates each image using all features, and then chooses the ones with role in improving the diagnostic accuracy of machine learning. We used the step-forward the highest evaluation accuracy; it then adds one of the remaining features to the selected selection (SFS) [14] method to choose the best combination of features to evaluate each feature and learns and evaluates it. This process was repeated for all features. Features disease. SFS selects features using the following procedure: the computer learns and were added until the evaluation accuracy no longer increased. Appl. Sci. 2021, 11, 9716 10 of 13 independently evaluates each image using all features, and then chooses the ones with the highest evaluation accuracy; it then adds one of the remaining features to the selected feature and learns and evaluates it. This process was repeated for all features. Features were added until the evaluation accuracy no longer increased. The SVR parameters used for training were C = 100, = 0.1, and " = 0.1. Features that were determined to be optimal for the evaluation of each disease are shown in Table 3. The average of the four evaluators was used as the correct value for learning. Table 4 shows the mean absolute error (MAE) between the values of the correct answers by the Kampo doctors and the estimated values obtained by SVR before and after feature selection. The evaluation accuracy increased when feature selection was used. Thus, the feature selection was appropriate. The MAE can be calculated from Equation (8): n1 MAE = jy y ˆ j (8) å i i i=0 where n is the number of evaluations made, y is the ith evaluation value given by the doctor, and y ˆ is the ith prediction value. Table 3. Features selected by step-forward selection for each disease. Diseases Qi Deficiency Blood Deficiency Blood Stagnation Yin Deficiency Fluid Stagnation Min of body b Mean of body b Std of gloss L Median of body b Min of gloss L Median of body b Mean of body b Mean of gloss L Max of gloss L Max of gloss L Max of gloss L Min of gloss L Median of gloss L Table 4. Mean absolute error between the correct value assigned by the Kampo doctors and the estimated value by SVR before and after feature selection for each disease. Diseases Qi Deficiency Blood Deficiency Blood Stagnation Yin Deficiency Fluid Stagnation Using all features 0.51 0.52 0.49 0.50 0.60 Using selected features 0.44 0.42 0.33 0.36 0.55 3. Results Table 5 shows the MAE of the correct and estimated values before and after data modulation when the evaluation value of each Kampo doctor was taken as the correct value. Each disease was diagnosed using features selected through the SFS method and the parameters were optimized by grid searching. Data modulation improved the accuracy of the diagnosis of most diseases. Table 5. Mean absolute error between the correct value (using the value assigned by each Kampo doctor) and the value estimated by support vector regression for each disease using data with no modulation (no modu.) and modulation (modu.). Diseases Qi Blood Blood Yin Fluid Deficiency Deficiency Stagnation Deficiency Stagnation no modu. 0.94 0.75 0.91 1.00 0.62 Doctor.A modu. 0.64 0.71 0.62 0.55 0.77 no modu. 0.99 0.90 0.80 0.85 0.84 Doctor.B modu. 0.70 0.73 0.55 0.57 0.75 Appl. Sci. 2021, 11, 9716 11 of 13 Table 5. Cont. Diseases Qi Blood Blood Yin Fluid Deficiency Deficiency Stagnation Deficiency Stagnation no modu. 1.01 0.93 0.82 0.94 0.73 Doctor.C modu. 0.84 0.99 0.64 0.81 0.90 no modu. 0.95 0.76 0.62 0.91 0.65 Doctor.D modu. 0.83 0.71 0.60 0.86 0.56 no modu. 0.59 0.61 0.53 0.58 0.68 Average of all doctors modu. 0.44 0.42 0.33 0.36 0.55 4. Discussion The correct answers were assigned a value from 0 to 4, with adjacent values having close meanings. Therefore, this method yielded good diagnostic performance when the MAE was less than 1. Table 5 shows that the highest accuracy was achieved by using the average assessment of the four Kampo doctors as the correct value. As shown in Table 6, RMSE also showed high accuracy. This can be attributed to the fact that taking the average suppressed the degree of error and lessened the subjectivity. Also shown in Table 6 is the accuracy of the evaluation using RMSE (root mean squared error), with the average value of the evaluation of the four Kampo doctors as the correct value. As in Table 5, it was shown that the accuracy was improved by modulation and high accuracy was obtained. The distribution of the ratings of the Kampo doctors shown in Figure 7 indicates some evaluation error between doctors. The proposed method was accurate because the error in its results was smaller than 1, and the neighboring assessment values had very close meanings. The features chosen by feature selection were closely related to the aspects of evaluation attended to by the Kampo doctors. The most important features were the tongue gloss, color, and shape in the diagnosis of qi deficiency; color and shape for blood deficiency; color for blood stagnation; gloss for yin deficiency; and shape for fluid stagnation. This shows that the features that were considered important by the Kampo doctors had been selected by the proposed method. However, we did not use the features of form (for example, size and tooth marks). Therefore, it was not possible to determine the dominant features of fluid stagnation, which is highly influenced by form. This was also evident in the final diagnostic results. The diagnostic accuracies for the detection of qi deficiency and fluid stagnation, where shape is important, were worse than those for the other diseases. These results suggest that the addition of new features related to shape may improve the diagnostic accuracy. Table 6. Root mean squared error between the correct value (using average of values assigned by each Kampo doctor) and the value estimated by support vector regression for each disease using data with no modulation (no modu.) and modulation (modu.). Diseases Qi Blood Blood Yin Fluid Deficiency Deficiency Stagnation Deficiency Stagnation no modu. 0.71 0.73 0.62 0.67 0.77 Average of all doctors modu. 0.53 0.50 0.41 0.43 0.56 In this study, tongue images were taken from healthy subjects. However, we were able to efficiently generate tongue data that could be diagnosed as score 4 (sick) by data augmentation. The doctors who evaluated the tongue images commented that the evalua- tion images were realistic, but some of the tongue colors were not realistic. In the future, we would like to improve this point in cooperation with doctors. Appl. Sci. 2021, 11, 9716 12 of 13 The data set created in this study contains only a small number of subjects and only Japanese subjects. In the future, we would like to collect data from more races to create a meaningful dataset that can be released to the public. The limitations of this study are therefore: Some modulated data for data augmentation were not naturally observed tongue data. A polarization camera is required instead of a regular camera. 5. Conclusions We developed an evaluation system that considers the gloss and color components of images of the tongue taken by a polarizing camera to diagnose diseases according to the principles of Kampo medicine. The number of samples was increased from 11 original images to 275 images by modulating the color and gloss. A total of 27 gloss- and color-related features were obtained and combined with subjective evaluation by a Kampo doctor to generate the training data. The MAE of SVR was 0.44 for qi deficiency, 0.42 for blood deficiency, 0.33 for blood stagnation, 0.36 for yin deficiency, and 0.55 for fluid stagnation, when the parameters were optimized by grid searching. The evaluation errors for all diseases were less than 1, indicating reasonable evaluation accuracy. As in Matsushita et al. [6], after each of the four Kampo doctors made a diagnosis, machine learning was performed using each evaluation as the ground truth. In this study, it was found that individual differences were generated among doctors in their evaluation of tongue images. The highest accuracy was obtained by taking the average of the evaluation values of the doctors as the ground truth. This is because errors and blur caused by one doctor ’s diagnosis and knowledge were reduced by taking the average. By using machine learning to reproduce the average of multiple doctors’ evaluations, it is hoped that quantitative diagnosis that eliminates the subjective blurring of individual doctors will become possible. The results of feature selection were consistent with the evaluation values assigned by the Kampo doctors, indicating their correctness. As the diagnosis of fluid stagnation requires numerous features, no useful set of features was available for this condition. Adding the shape of the tongue as a feature may improve the accuracy of the diagnosis. The advantage of this study is that the equipment is simple and easy to reproduce, and the number of data can be easily collected through data augmentation. The results of this study indicate that the polarization camera can be used to diagnose the condition of individual patients. By collecting more data and improving the accuracy of the system, it can be expected to develop into a system that allows doctors to monitor the condition of patients and provide appropriate care without having to diagnose all patients. As for the issues, the later assessments may have been more accurate as the Kampo doctors became familiar with the evaluation of tongue images. Future research should include more stable subjective evaluation methods, such as multiple practice sessions for the Kampo doctors. In future work, more tongue images should be collected and re-evaluated because some of the pathological conditions were biased, as described above. In addition, by adding the features of tongue shape, the accuracy of the proposed method can be improved for the diseases with low diagnostic accuracy in the present study. Another option is to incorporate continuous learning, as implemented by Pianykh et al. [15]. By building a system that can continuously learn from the opinions of more physicians and tongue data with new cases, we can make more objective diagnoses with better generalization performance. Author Contributions: Conceptualization, R.K. and K.O.-O.; formal analysis, R.K.; investigation, K.O.-O., A.S., K.H., J.A. and H.L.; methodology, R.K., K.O.-O. and H.L.; software, R.K.; supervision, N.T.; validation, A.S., K.H. and J.A.; writing—original draft, R.K.; writing—review and editing, K.O.-O., A.S., K.H., J.A. and N.T. All authors have read and agreed to the published version of the manuscript. Appl. Sci. 2021, 11, 9716 13 of 13 Funding: This research received no external funding. Institutional Review Board Statement: This study was approved by the Ethical Board of Kanazawa University, School of Medicine (2018-154). All participants agree to the publication of the results obtained from the study. All results are decoded. Individual results cannot be tracked. Informed Consent Statement: Informed consent was obtained from all subjects involved in the study. Data Availability Statement: The data presented in this study are available on request from the corresponding author. The data are not publicly available due to we have not received permission from the subjects to do so. Acknowledgments: We thank Sakura Matsumoto and Mako Iwahashi for providing the facilities and collecting tongue images. We also thank Kelly Zammit, BVSc, for editing a draft of this manuscript. Conflicts of Interest: The authors declare no conflict of interest. References 1. Sato, Y. Introduction to Kampo: Japanese Traditional Medicine; The Japan Society for Oriental Medicine: Tokyo, Japan, 2005. 2. Mitani, K. The lecture for the diagnosis of Kampo medicine: Boshin, especially for the diagnosis of tongue. Kampo Med. 2007, 58, 673–685. [CrossRef] 3. Satoh, H. An introduction to Kampo medicine. Folia Pharmacol. Jpn. 2008, 132, 260–264. [CrossRef] [PubMed] 4. Yamamoto, S.; Ishilawa, Y.; Nakaguchi, T.; Ogawa-Ochiai, K.; Tsumura, N.; Kasahara, Y.; Namiki, T.; Miyake, Y. Temporal Changes in Tongue Color as Criterion for Tongue Diagnosis in Kampo Medicine. Forsch. Komplementmed. 2012, 19, 80–85. [CrossRef] [PubMed] 5. Wang, Y.; Zhou, Y.; Yang, J.; Xu, Q. An image analysis system for tongue diagnosis in traditional Chinese medicine. Comput. Inf. Sci. 2005, 3314, 1181–1186. 6. Matsushita, F.; Kiyomitsu, K.; Ogawa, K.; Tsumura, N. Evaluation of Kampo disease states using facial images. Artif. Life Robot. 2019, 24, 44–51. [CrossRef] 7. Zhang, J.; Xu, J.; Hu, X.; Chen, Q.; Tu, L.; Huang, J.; Cui, J. Diagnostic method of diabetes based on support vector machine and tongue images. BioMed Res. Int. 2017, 2017, 7961494. [CrossRef] [PubMed] 8. Kainuma, M.; Furusyo, N.; Urita, Y.; Ikezaki, H.; Ura, K.; Nakaguchi, T.; Namiki, T. Objective tongue color and gastroesophageal reflux disease: Cross-sectional study. Tradit. Kampo Med. 2019, 6, 19–25. [CrossRef] 9. Nakaguchi, T.; Ishilawa, Y.; Oji, T.; Yamamoto, S.; Tsumura, N.; Ueda, K.; Nagamine, K.; Namiki, T.; Miyake, Y. Proposal for a new noncontact method for measuring tongue moisture to assist in tongue diagnosis and development of the tongue image analyzing system, which can separately record the gloss components of the tongue. BioMed Res. Int. 2015, 2015, 249609. [CrossRef] [PubMed] 10. Danser, M.M.; Mantilla Gomez, S.; Van der Weijden, G.A. Tongue coating and tongue brushing: A literature review. Int. J. Dent. Hyg. 2003, 1, 151–158. [CrossRef] [PubMed] 11. Xu, J.; Tu, L.; Zhang, Z.; Zhou, C. The region partition of quality and coating for tongue image based on color image segmentation method. In Proceedings of the 2008 IEEE International Symposium on IT in Medicine and Education, Xiamen, China, 12–14 December 2008; pp. 817–821. 12. Tsumura, N.; Ojima, N.; Sato, K.; Shiraishi, M.; Shimizu, H. Image-based skin color and quality analysis/synthesis by extracting hemoglobin and melanin information in the skin. ACM Trans. Graph. 2003, 22, 770–779. [CrossRef] 13. Burges, C.J. A Tutorial on support vector machines for pattern recognition. Data Min. Knowl. Discov. 1998, 2, 121–167. [CrossRef] 14. Leslie, K.P. An Introduction to Variable and Feature Selection. J. Mach. Learn. Res. 1998, 3, 1157–1182. 15. Pianykh, O.S.; Langs, G.; Dewey, M.; Enzmann, D.R.; Herold, C.J.; Schoenberg, S.O.; Brink, J.A. A tutorial on support vector machines for pattern recognition. Radiology 2020, 297, 6–14.

Journal

Applied SciencesMultidisciplinary Digital Publishing Institute

Published: Oct 20, 2021

Keywords: machine learning; Kampo medicine; tongue; diagnostic assistance

There are no references for this article.