Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Tooth Segmentation of 3D Scan Data Using Generative Adversarial Networks

Tooth Segmentation of 3D Scan Data Using Generative Adversarial Networks applied sciences Article Tooth Segmentation of 3D Scan Data Using Generative Adversarial Networks 1 1 1 1 , 2 , Taeksoo Kim , Youngmok Cho , Doojun Kim , Minho Chang * and Yoon-Ji Kim * Department of Mechanical Engineering, Korea University, Seoul 02841, Korea; pqowep@korea.ac.kr (T.K.); dudahr2002@korea.ac.kr (Y.C.); rlaenwns95@naver.com (D.K.) Department of Orthodontics, Korea University Anam Hospital, Seoul 02841, Korea * Correspondence: mhchang@korea.ac.kr (M.C.); yn0331@korea.ac.kr (Y.-J.K.); Tel.: +82-2-3290-3379 (M.C.) Received: 27 November 2019; Accepted: 8 January 2020; Published: 9 January 2020 Abstract: The use of intraoral scanners in the field of dentistry is increasing. In orthodontics, the process of tooth segmentation and rearrangement provides the orthodontist with insights into the possibilities and limitations of treatment. Although, full-arch scan data, acquired using intraoral scanners, have high dimensional accuracy, they have some limitations. Intraoral scanners use a stereo-vision system, which has diculties scanning narrow interdental spaces. These areas, with a lack of accurate scan data, are called areas of occlusion. Owing to such occlusions, intraoral scanners often fail to acquire data, making the tooth segmentation process challenging. To solve the above problem, this study proposes a method of reconstructing occluded areas using a generative adversarial network (GAN). First, areas of occlusion are eliminated, and the scanned data are sectioned along the horizontal plane. Next, images are trained using the GAN. Finally, the reconstructed two-dimensional (2D) images are stacked to a three-dimensional (3D) image and merged with the data where the occlusion areas have been removed. Using this method, we obtained an average improvement of 0.004 mm in the tooth segmentation, as verified by the experimental results. Keywords: tooth segmentation; generative adversarial networks; intraoral scanners; reconstruction; image completion; dental scan data; occlusion areas 1. Introduction Intraoral scanners are widely used in the diagnosis and fabrication of orthodontic appliances. A digital impression technique using intraoral scanners has been replacing a conventional tooth model setup applied manually using plaster casts. However, intraoral scanners have a problem of occlusions due to the stereo-vision system applied. This lowers the accuracy of the tooth’s anatomic form, and therefore, a segmentation of each tooth is required, which requires a lot of time and e ort by the operator. This study aims to solve the occlusion problem by preprocessing the scan data obtained by intraoral scanners through the use of a generative adversarial network (GAN). 1.1. Background 1.1.1. Digital Orthodontics An orthodontic diagnosis is a crucial step, whereby, the patient is examined based on various modalities, such as radiographs, intraoral photos, and tooth impressions. The information gathered from the patient is analyzed, and the final diagnosis and treatment plans, such as extraction versus non-extraction, and an anchorage preparation are made. As the visual treatment objective, patient’s dental plaster model is used to simulate the orthodontic treatment by dissecting each tooth from the plaster model and rearranging them into the desired position [1]. Although, this process provides Appl. Sci. 2020, 10, 490; doi:10.3390/app10020490 www.mdpi.com/journal/applsci Appl. Sci. 2020, 10, 490 2 of 18 Appl. Sci. 2020, 10, 490 2 of 17 Appl. Sci. 2020, 10, 490 2 of 17 orthodon the orthodontist tist insiinsight ght into t into he the poss possibilities ibilities and lim and limitations itations ofof tre tra eatment, tment, cre cra eating ting a di a diagnostic agnostic set setup up orthodontist insight into the possibilities and limitations of treatment, creating a diagnostic setup model from a dental plaster model is time-consuming and labor intensive. model model fr from om a a dental plaste dental plaster r model is model is time-consuming time-consuming a and nd la labor bor intensive. intensive. Since Since the the a advent dvent of of in intraoral traoral sc scanners, anners, iinstead nstead of of us using ing the the den dental tal p plaster laster m model, odel, in intraoral traoral sc scan an Since the advent of intraoral scanners, instead of using the dental plaster model, intraoral scan data can be manipulated to achieve a digital tooth setup. After acquiring the full arch scan data of the data data can can b be e m manipulated anipulated to to achieve achieve a a di digital gital tooth tooth se setup. tup. Af After ter ac acquiring quiring the the full full ar arch ch sc scan an d data ata of of the the p patient, atient, the the t teeth eeth ar are e segmented segmented and and repositioned repositioned to t the o the desir des ed ire position d position using usthr ing th ee-dimensional ree-dimension (3D) al patient, the teeth are segmented and repositioned to the desired position using three-dimensional (3D) CAD software. This is more efficient because it eliminates the process of making a plaster model, CAD (3D) C softwar AD sofe. tw This are. T ishmor is ise m eo  re cient efficien because t because it eliminates it eliminates the pr the p ocess rocess of m of making aking a plaster a plaster model, model, and and it has been reported to be as effective and accurate as a conventional method [2,3]. it and has it ha been s bre eported en reporte to be d to asbe e as ective effecand tive and accurate accuas rat ae a conventional s a convention method al metho [2,3d ]. [2,3]. 1.1.2. Tooth Segmentation 1.1.2. Tooth Segmentation 1.1.2. Tooth Segmentation An intraoral scan data of a full dental arch leads to a single scanned object that contains all teeth An intraoral scan data of a full dental arch leads to a single scanned object that contains all teeth An intraoral scan data of a full dental arch leads to a single scanned object that contains all teeth as a single unit. Although, each tooth has its own boundary, and teeth are not connected to each as a single unit. Although, each tooth has its own boundary, and teeth are not connected to each as a single unit. Although, each tooth has its own boundary, and teeth are not connected to each other, this problem is caused by an occlusion of the intraoral scanner. Intraoral scanners usually use other, this problem is caused by an occlusion of the intraoral scanner. Intraoral scanners usually use other, this problem is caused by an occlusion of the intraoral scanner. Intraoral scanners usually use a stereo-vision system, and the interdental areas of the teeth are not captured. This space is called a stereo-vision system, and the interdental areas of the teeth are not captured. This space is called an a stereo-vision system, and the interdental areas of the teeth are not captured. This space is called an an occlusion area and leads to a low accuracy (Figure 1) [4]. Therefore, after the operator manually occlusion area and leads to a low accuracy (Figure 1) [4]. Therefore, after the operator manually occlusion area and leads to a low accuracy (Figure 1) [4]. Therefore, after the operator manually defines the tooth boundary and the long axis of the tooth, the plane to be used for tooth separation is defines the tooth boundary and the long axis of the tooth, the plane to be used for tooth separation is defines the tooth boundary and the long axis of the tooth, the plane to be used for tooth separation is also designated manually to separate the teeth. also designated manually to separate the teeth. also designated manually to separate the teeth. Figure 1. Occlusion areas of dental scan data. Figure 1. Occlusion areas of dental scan data. Figure 1. Occlusion areas of dental scan data. 1.2. Related Studies 1.2. Related Studies 1.2. Related Studies Several studies have been conducted to improve the tooth segmentation. Ongoing studies Several studies have been conducted to improve the tooth segmentation. Ongoing studies can Several studies have been conducted to improve the tooth segmentation. Ongoing studies can can be divided into two categories. One is a boundary-based segmentation and the other is a be divided into two categories. One is a boundary-based segmentation and the other is a region-based be divided into two categories. One is a boundary-based segmentation and the other is a region-based region-based segmentation. Region-based methods are mainly conducted using K-means clustering [5], segmentation. Region-based methods are mainly conducted using K-means clustering [5], a segmentation. Region-based methods are mainly conducted using K-means clustering [5], a a watershed [6], an active contour [7], and Mask-MCNet [8]. Boundary-based methods include a graph watershed [6], an active contour [7], and Mask-MCNet [8]. Boundary-based methods include a graph watershed [6], an active contour [7], and Mask-MCNet [8]. Boundary-based methods include a graph cut [9], random walks [10], and a snake [11]. Among them, the most popular method is to set the cut [9], random walks [10], and a snake [11]. Among them, the most popular method is to set the cut [9], random walks [10], and a snake [11]. Among them, the most popular method is to set the initial position of the boundary between teeth and gums using a snake. After the initial position is set, initial position of the boundary between teeth and gums using a snake. After the initial position is initial position of the boundary between teeth and gums using a snake. After the initial position is the operator finishes by modifying the initial position (Figure 2). However, because of the occlusion set, the operator finishes by modifying the initial position (Figure 2). However, because of the set, the operator finishes by modifying the initial position (Figure 2). However, because of the problem, each of these methods is limited in accuracy. occlusion problem, each of these methods is limited in accuracy. occlusion problem, each of these methods is limited in accuracy. (a) (b) (a) (b) Figure 2. Figure 2. Segmentation using Segmentation using a snake: ( a snake: (a a)) Ini Initial; tial; and ( and (b b) m ) modified odified po positions sitions u using sing snak snake. e. Figure 2. Segmentation using a snake: (a) Initial; and (b) modified positions using snake. Appl. Sci. 2020, 10, 490 3 of 17 Appl. Sci. 2020, 10, 490 3 of 18 1.3. Motivation and Contributions of This Paper 1.3. Motivation and Contributions of This Paper 1.3.1. Generative Adversarial Network (GAN) 1.3.1. Generative Adversarial Network (GAN) Research on artificial intelligence has recently been actively conducted. A generative adversarial network (GAN) is a type of artificial intelligence that method that learns a generative model and a Research on artificial intelligence has recently been actively conducted. A generative adversarial discriminative model simultaneously. The system configuration of a GAN is described as follows network (GAN) is a type of artificial intelligence that method that learns a generative model and a (Figure 3). The generative model learns the distribution of data, and the discriminative model discriminative model simultaneously. The system configuration of a GAN is described as follows evaluate the data generated by the generative model. Next, the generative model then generates data (Figure 3). The generative model learns the distribution of data, and the discriminative model evaluate such that an error when evaluating the learning model and an error when evaluating the generated the data generated by the generative model. Next, the generative model then generates data such that model are similar. The generative model is briefly trained to induce a discrimination of the an error when evaluating the learning model and an error when evaluating the generated model are discriminative model. similar. The generative model is briefly trained to induce a discrimination of the discriminative model. Figure 3. Overview of generative adversarial network (GAN). Figure 3. Overview of generative adversarial network (GAN). 1.3.2. Image Completion 1.3.2. Image Completion Image completion is a technique used to remove and complete parts of an image that we want Image completion is a technique used to remove and complete parts of an image that we want to restore. In this way, we can remove an unwanted part of an image and regenerate the occluded to restore. In this way, we can remove an unwanted part of an image and regenerate the occluded regions. Previously, there have been many approaches using patch-based methods [12–15]. However, regions. Previously, there have been many approaches using patch-based methods [12–15]. However, the results were unsatisfactory owing to the complexity of the image pattern in various of objects, and the results were unsatisfactory owing to the complexity of the image pattern in various of objects, it was dicult to increase the accuracy. and it was difficult to increase the accuracy. Methods using artificial intelligence have recently been studied. A context encoder is one such Methods using artificial intelligence have recently been studied. A context encoder is one such method and employs a convolutional neural network (CNN) [16,17]. Globally and locally consistent method and employs a convolutional neural network (CNN) [16,17]. Globally and locally consistent image completion (GLCIC) and Edge Connect also use artificial intelligence [18,19]. Whereas a context image completion (GLCIC) and Edge Connect also use artificial intelligence [18,19]. Whereas a encoder is based on a CNN, a GLCIC and Edge Connect are based on a GAN. Among all methods context encoder is based on a CNN, a GLCIC and Edge Connect are based on a GAN. Among all described above, Edge Connect achieves the best accuracy [19]. Therefore, in this study, we employed methods described above, Edge Connect achieves the best accuracy [19]. Therefore, in this study, we an Edge Connect as the image completion method. employed an Edge Connect as the image completion method. 1.3.3. Goals 1.3.3. Goals Various studies have been made to solve the diculties of segmentation, and automation has resulted Variin ous stud a reduction ies ha in veworking been made time. toHowever solve the , both difficu methods lties of s using egmen the tatooth tion, and boundary automand ation tooth has resu region lted do in a not resolve duction in the occlusion working tim problem. e. However, In this bst ot udy h m ,ewe thods aim us to ing th solve e too theth occlusion boundarpr y an oblem d tooby th region do not solve the occlusion problem. In this study, we aim to solve the occlusion problem by Appl. Sci. 2020, 10, 490 4 of 18 Appl. Sci. 2020, 10, 490 4 of 17 pre-processing dental scan data using the image completion method. Additionally, using this approach, pre-processing dental scan data using the image completion method. Additionally, using this we expect to improve the tooth segmentation accuracy. approach, we expect to improve the tooth segmentation accuracy. 2. Proposed Method 2. Proposed Method 2.1. Overview 2.1. Overview The system used to solve the occlusion problem of dental scan data when applying a GAN can The system used to solve the occlusion problem of dental scan data when applying a GAN can be introduced through three steps. The first step is to remove the occlusion areas of the dental scan be introduced through three steps. The first step is to remove the occlusion areas of the dental scan data. The next step is to complete the images on each plane. The final step is to stack the images from data. The next step is to complete the images on each plane. The final step is to stack the images from the previous step and reconstruct the dental scan data. The figure below depicts an overview of the the previous step and reconstruct the dental scan data. The figure below depicts an overview of the proposed method (Figure 4). proposed method (Figure 4). Figure 4. Overview of the proposed method. Figure 4. Overview of the proposed method. 2.2. Reconstruction of Dental Scan Data 2.2. Reconstruction of Dental Scan Data The reconstruction process is as follows. First, the occlusion area is specified by visual inspection The reconstruction process is as follows. First, the occlusion area is specified by visual inspection of the interdental area whereby the intraoral scan data are not captured accurately. Here, the removed of the interdental area whereby the intraoral scan data are not captured accurately. Here, the removed occlusion areas will be used as masks and the location must be saved. The occlusion area was deleted, occlusion areas will be used as masks and the location must be saved. The occlusion area was deleted, and this removed area must be reconstructed. Figure 5 depicts images from before and after removing and this removed area must be reconstructed. Figure 5 depicts images from before and after removing thethe occ occlusion lusion area. area. Appl. Sci. 2020, 10, 490 5 of 18 Appl. Appl. Sci. Sci. 2020 2020,, 10 10,, 490 490 5 of 5 of 17 17 Appl. Sci. 2020, 10, 490 5 of 17 (a) (b) (a) (b) (a) (b) Figure 5. Example of removing the occlusion area: (a) before and (b) after removal. Figure 5. Example of removing the occlusion area: (a) before and (b) after removal. Figure 5. Figure 5. Example of Example of removing the occlusion removing the occlusion area: ( area: (a a) befor ) before e and ( and (b b) after ) after removal. removal. Next, dental scan data are sectioned at regular intervals using the horizontal plane direction. Next, dental scan data are sectioned at regular intervals using the horizontal plane direction. Next, dental scan data are sectioned at regular intervals using the horizontal plane direction. Next, dental scan data are sectioned at regular intervals using the horizontal plane direction. The more images we apply during the reconstruction at narrow intervals, the higher the accuracy The more images we apply during the reconstruction at narrow intervals, the higher the accuracy The more images we apply during the reconstruction at narrow intervals, the higher the accuracy The more images we apply during the reconstruction at narrow intervals, the higher the accuracy that will be acquired, but the longer the computational time required. We used 0.1 mm intervals and that will be acquired, but the longer the computational time required. We used 0.1 mm intervals and that will be acquired, but the longer the computational time required. We used 0.1 mm intervals and that will be acquired, but the longer the computational time required. We used 0.1 mm intervals and achieved an image completion using 100 images per patient. The figures below show several images achieved an image completion using 100 images per patient. The figures below show several images achieved an image completion using 100 images per patient. The figures below show several images achieved an image completion using 100 images per patient. The figures below show several images acquired using horizontal plane sectioning (Figure 6). Image completion is then conducted using a acquired using horizontal plane sectioning (Figure 6). Image completion is then conducted using a acquired using horizontal plane sectioning (Figure 6). Image completion is then conducted using a acquired using horizontal plane sectioning (Figure 6). Image completion is then conducted using a GAN (Figure 7). GAN (Figure 7). GAN (Figure 7). GAN (Figure 7). Figure 6. Examples of horizontal plane sectioning. Figure 6. Examples of horizontal plane sectioning. Figure 6. Examples of horizontal plane sectioning. Figure 6. Examples of horizontal plane sectioning. Figure 7. Examples of image completion. Figure 7. Examples of image completion. Figure 7. Examples of image completion. Figure 7. Examples of image completion. Appl. Sci. 2020, 10, 490 6 of 18 Appl. Sci. 2020, 10, 490 6 of 17 Appl. Sci. 2020, 10, 490 6 of 17 Finally, the completed images are stacked using the previously saved location and height interval. Finally, the completed images are stacked using the previously saved location and height Finally, the completed images are stacked using the previously saved location and height We use only the restored area, as indicated by the circle in Figure 7. When the location of the interval. We use only the restored area, as indicated by the circle in Figure 7. When the location of the interval. We use only the restored area, as indicated by the circle in Figure 7. When the location of the reconstructed part is unknown, a registration should be applied using an algorithm, such as the reconstructed part is unknown, a registration should be applied using an algorithm, such as the reconstructed part is unknown, a registration should be applied using an algorithm, such as the iterative closest point (ICP) [20], which can cause additional errors. After the images are stacked iterative closest point (ICP) [20], which can cause additional errors. After the images are stacked to iterative closest point (ICP) [20], which can cause additional errors. After the images are stacked to to achieve a 3D image, the image is merged with the data where the occlusion areas have been achieve a 3D image, the image is merged with the data where the occlusion areas have been removed. achieve a 3D image, the image is merged with the data where the occlusion areas have been removed. removed. The process is completed through remeshing. The figures below illustrate the merging The process is completed through remeshing. The figures below illustrate the merging process The process is completed through remeshing. The figures below illustrate the merging process process (Figure 8). (Figure 8). (Figure 8). (a) (b) (c) (a) (b) (c) Figure 8. Example after reconstruction: (a) Dental scan data with occlusion area removed, (b) Figure 8. Example after reconstruction: (a) Dental scan data with occlusion area removed, Figure 8. Example after reconstruction: (a) Dental scan data with occlusion area removed, (b) reconstructed data by stacking images, and (c) dental scan data by merging (a,b). (b) reconstructed data by stacking images, and (c) dental scan data by merging (a,b). reconstructed data by stacking images, and (c) dental scan data by merging (a,b). 2.3. Training GAN 2.3. Training GAN 2.3. Training GAN Edge Connect, which adds the contour reconstruction described above, requires two learning Edge Connect, which adds the contour reconstruction described above, requires two learning Edge Connect, which adds the contour reconstruction described above, requires two learning processes. One process is contour restoration learning and the other is image completion learning. processes. One process is contour restoration learning and the other is image completion learning. processes. One process is contour restoration learning and the other is image completion learning. Both processes use a GAN. The neural networks of the generative model and the discriminative model Both processes use a GAN. The neural networks of the generative model and the discriminative Both processes use a GAN. The neural networks of the generative model and the discriminative used in this study are similar to those of Edge Connect. In detail, a generative model follows the model used in this study are similar to those of Edge Connect. In detail, a generative model follows model used in this study are similar to those of Edge Connect. In detail, a generative model follows super-resolution architecture [21,22], and the discriminative model follows a 70  70 patch GAN the super-resolution architecture [21,22], and the discriminative model follows a 70 × 70 patch GAN the super-resolution architecture [21,22], and the discriminative model follows a 70 × 70 patch GAN architecture [23,24]. architecture [23,24]. architecture [23,24]. 2.3.1. Data Preparation 2.3.1. Data Preparation 2.3.1. Data Preparation This study was approved by the Institutional Review Board of Korea University Anam Hospital This study was approved by the Institutional Review Board of Korea University Anam Hospital This study was approved by the Institutional Review Board of Korea University Anam Hospital (2019AN0470). Intraoral scan data of ten patients, who had taken a digital impression as part of an (2019AN0470). Intraoral scan data of ten patients, who had taken a digital impression as part of an (2019AN0470). Intraoral scan data of ten patients, who had taken a digital impression as part of an orthodontic diagnosis, were exported and segmented by a trained dental technician. The scans were orthodontic diagnosis, were exported and segmented by a trained dental technician. The scans were orthodontic diagnosis, were exported and segmented by a trained dental technician. The scans were acquired by one orthodontist by using the Trios 3 intraoral scanner (3Shape, Denmark). The figure acquired by one orthodontist by using the Trios 3 intraoral scanner (3Shape, Denmark). The figure acquired by one orthodontist by using the Trios 3 intraoral scanner (3Shape, Denmark). The figure below depicts the dental scan data after the segmentation process and one of the horizontal plane below depicts the dental scan data after the segmentation process and one of the horizontal plane below depicts the dental scan data after the segmentation process and one of the horizontal plane images (Figure 9). images (Figure 9). images (Figure 9). (a) (b) (a) (b) Figure 9. Data after tooth segmentation: (a) 3D and (b) horizontal views of the data. Figure 9. Data after tooth segmentation: (a) 3D and (b) horizontal views of the data. Figure 9. Data after tooth segmentation: (a) 3D and (b) horizontal views of the data. Appl. Sci. 2020, 10, 490 7 of 18 Appl. Sci. 2020, 10, 490 7 of 17 Appl. Sci. 2020, 10, 490 7 of 17 The area to be used for learning is cropped into a 256 (px)  256 (px) square in the horizontally The area to be used for learning is cropped into a 256 (px) × 256 (px) square in the horizontally The area to be used for learning is cropped into a 256 (px) × 256 (px) square in the horizontally sectioned images, as shown in (b) of Figure 9. In this study, we used 10,000 cropped images for training sectioned images, as shown in (b) of Figure 9. In this study, we used 10,000 cropped images for sectioned images, as shown in (b) of Figure 9. In this study, we used 10,000 cropped images for the GAN. The ratio of the training, validation, and test sets was 28:1:1. In general, approximately training the GAN. The ratio of the training, validation, and test sets was 28:1:1. In general, training the GAN. The ratio of the training, validation, and test sets was 28:1:1. In general, 100,000 images are used for training complex images, such as CelebA or Paris Street View. However, in approximately 100,000 images are used for training complex images, such as CelebA or Paris Street approximately 100,000 images are used for training complex images, such as CelebA or Paris Street the case of tooth data, the images are not complicated and can be trained using only 10,000 images. The View. However, in the case of tooth data, the images are not complicated and can be trained using View. However, in the case of tooth data, the images are not complicated and can be trained using training data was organized such that the tooth images were contained. Figure 10 shows the process of only 10,000 images. The training data was organized such that the tooth images were contained. only 10,000 images. The training data was organized such that the tooth images were contained. image Figure cr 10 sho opping. ws the process of image cropping. Figure 10 shows the process of image cropping. Figure 10. Cropping images from segmented data. Figure 10. Cropping images from segmented data. Figure 10. Cropping images from segmented data. Five types of data are prepared: Cropped images, edge images, masks, masked images, and Five types of data are prepared: Cropped images, edge images, masks, masked images, and Five types of data are prepared: Cropped images, edge images, masks, masked images, and masked edge images. The cropped images can already be seen as edge images, but using edge images masked edge images. The cropped images can already be seen as edge images, but using edge images masked edge images. The cropped images can already be seen as edge images, but using edge images of the cropped images is important in terms of accuracy. We can apply the Edge Connect method of the cropped images is important in terms of accuracy. We can apply the Edge Connect method of the cropped images is important in terms of accuracy. We can apply the Edge Connect method when we consider only the cropped images as a picture, and not the edge images. For the edge images, when we consider only the cropped images as a picture, and not the edge images. For the edge when we consider only the cropped images as a picture, and not the edge images. For the edge we used the Canny edge detector with a threshold of 0.5 [25]. We determined the masks based on images, we used the Canny edge detector with a threshold of 0.5 [25]. We determined the masks images, we used the Canny edge detector with a threshold of 0.5 [25]. We determined the masks the dental spaces, where occlusions mainly occur. Any mask shape can be used, and the mask only based on the dental spaces, where occlusions mainly occur. Any mask shape can be used, and the based on the dental spaces, where occlusions mainly occur. Any mask shape can be used, and the needs to cover the areas where an occlusion is expected to occur. Herein, we used a rectangular shape. mask only needs to cover the areas where an occlusion is expected to occur. Herein, we used a mask only needs to cover the areas where an occlusion is expected to occur. Herein, we used a Figure 11 illustrates a few examples of the five types of data. rectangular shape. Figure 11 illustrates a few examples of the five types of data. rectangular shape. Figure 11 illustrates a few examples of the five types of data. (a) (b) (c) (d) (e) (a) (b) (c) (d) (e) Figure 11. Example of five types of training data: (a) cropped images from Figure 10, (b) edge images Figure 11. Example of five types of training data: (a) cropped images from Figure 10, (b) edge images Figure 11. Example of five types of training data: (a) cropped images from Figure 10, (b) edge images from (a), (c) masks, (d) masked images of (a), and (e) masked edges of (a). from (a), (c) masks, (d) masked images of (a), and (e) masked edges of (a). from (a), (c) masks, (d) masked images of (a), and (e) masked edges of (a). 2.3.2. Training Steps 2.3.2. Training Steps 2.3.2. Training Steps The first step is to pre-train the edge model. Among the data in Figure 11, we used masks The first step is to pre-train the edge model. Among the data in Figure 11, we used masks (Figure The first step is to pre-train the edge model. Among the data in Figure 11, we used masks (Figure (Figure 11c), masked images (Figure 11d), and masked edges (Figure 11e) as input images. Output 11c), masked images (Figure 11d), and masked edges (Figure 11e) as input images. Output images 11c), masked images (Figure 11d), and masked edges (Figure 11e) as input images. Output images are full edge images, and the target images of the output are depicted in Figure 11b. The goal of this are full edge images, and the target images of the output are depicted in Figure 11b. The goal of this Appl. Sci. 2020, 10, 490 8 of 18 Appl. Sci. 2020, 10, 490 8 of 17 images are full edge images, and the target images of the output are depicted in Figure 11b. The goal Appl. Sci. 2020, 10, 490 8 of 17 of this process is to pre-train only the edge model. Figure 12 depicts an image generated using the process is to pre-train only the edge model. Figure 12 depicts an image generated using the pre- process is to pre-train only the edge model. Figure 12 depicts an image generated using the pre- pre-trained edge model. As depicted in the figure, it is dicult to make an accurate image by training trained edge model. As depicted in the figure, it is difficult to make an accurate image by training the trained edge model. As depicted in the figure, it is difficult to make an accurate image by training the the edge alone. edge alone. edge alone. Figure 12. Image generated using the pre-trained edge model. Figure 12. Image generated using the pre-trained edge model. Figure 12. Image generated using the pre-trained edge model. The second step is to pre-train the image completion model. During this process, we use the edge The second step is to pre-train the image completion model. During this process, we use the The second step is to pre-train the image completion model. During this process, we use the images, masks, and masked images depicted in Figure 11b–d as input images, respectively. The output edge images, masks, and masked images depicted in Figure 11b–d as input images, respectively. The edge images, masks, and masked images depicted in Figure 11b–d as input images, respectively. The images are full images and the target images of the output are indicated in Figure 11a. The goal of this output images are full images and the target images of the output are indicated in Figure 11a. The output images are full images and the target images of the output are indicated in Figure 11a. The process is to pre-train only the image completion model. This is a separate learning process rather than goal of this process is to pre-train only the image completion model. This is a separate learning goal of this process is to pre-train only the image completion model. This is a separate learning an additional training of the model trained in the first step. The following is an image created by the process rather than an additional training of the model trained in the first step. The following is an process rather than an additional training of the model trained in the first step. The following is an pre-trained image completion model (Figure 13). The high mask ratio on the test image reduces the image created by the pre-trained image completion model (Figure 13). The high mask ratio on the image created by the pre-trained image completion model (Figure 13). The high mask ratio on the accuracy of the image completion. test image reduces the accuracy of the image completion. test image reduces the accuracy of the image completion. Figure 13. Image generated using the pre-trained image completion model. Figure 13. Image generated using the pre-trained image completion model. Figure 13. Image generated using the pre-trained image completion model. The third step is to train images by using the two pre-trained models, the edge model and the The third step is to train images by using the two pre-trained models, the edge model and the The third step is to train images by using the two pre-trained models, the edge model and the image completion model. First, we used the pre-trained edge model to make edge images. Then we image completion model. First, we used the pre-trained edge model to make edge images. Then we image completion model. First, we used the pre-trained edge model to make edge images. Then we used the generated edge images, the masks from Figure 11c, and masked images from Figure 11d as used the generated edge images, the masks from Figure 11c, and masked images from Figure 11d as used the generated edge images, the masks from Figure 11c, and masked images from Figure 11d as the input. The output images are restored images, and the target images are depicted in Figure 11a. We the input. The output images are restored images, and the target images are depicted in Figure 11a. the input. The output images are restored images, and the target images are depicted in Figure 11a. used the weight file of the pre-trained image completion model to fine-tune the edge connect model. We used the weight file of the pre-trained image completion model to fine-tune the edge connect We used the weight file of the pre-trained image completion model to fine-tune the edge connect Figure 14 is an image generated after the last step of training. We can now see the reason why the model. Figure 14 is an image generated after the last step of training. We can now see the reason why model. Figure 14 is an image generated after the last step of training. We can now see the reason why edge images are made. Figure 14 shows that restoring images by simultaneously using both, the edge the edge images are made. Figure 14 shows that restoring images by simultaneously using both, the the edge images are made. Figure 14 shows that restoring images by simultaneously using both, the generative model and the image completion model delivers better results. edge generative model and the image completion model delivers better results. edge generative model and the image completion model delivers better results. Appl. Sci. 2020, 10, 490 9 of 17 Appl. Sci. 2020, 10, 490 9 of 18 Appl. Sci. 2020, 10, 490 9 of 17 Figure 14. Image generated after the last training step. Figure 14. Image generated after the last training step. Figure 14. Image generated after the last training step. 3. Results and Discussion 3. Results and Discussion 3. Results and Discussion 3.1. Result of Image Completion 3.1. Result of Image Completion 3.1. Result of Image Completion We applied the trained GAN using the proposed method to the actual tooth image completion. We applied the trained GAN using the proposed method to the actual tooth image completion. We applied the trained GAN using the proposed method to the actual tooth image completion. Overall, it took 4 days for training: Steps 1 and step 2 required 1 day each, and step 3 took 2 days. We Overall, it took 4 days for training: Steps 1 and step 2 required 1 day each, and step 3 took 2 days. We Overall, it took 4 days for training: Steps 1 and step 2 required 1 day each, and step 3 took 2 days. We used a personal computer with an i7-6700 (3.4 GHz) processor, 24 GB of RAM, and an RTX 2080 Ti used a personal computer with an i7-6700 (3.4 GHz) processor, 24 GB of RAM, and an RTX 2080 Ti used a personal computer with an i7-6700 (3.4 GHz) processor, 24 GB of RAM, and an RTX 2080 Ti GPU. The figures below show some examples of the results (Figure 15). GPU. The figures below show some examples of the results (Figure 15). GPU. The figures below show some examples of the results (Figure 15). Figure 15. Cont. Appl. Sci. 2020, 10, 490 10 of 18 Appl. Sci. 2020, 10, 490 10 of 17 Figure 15. Some examples of generated images. Figure 15. Some examples of generated images. We used the structural similarity index (SSIM) [26] and peak signal-to-noise ratio (PSNR) [27] to We used the structural similarity index (SSIM) [26] and peak signal-to-noise ratio (PSNR) [27] to quantitatively measure the training result of a GAN. The SSIM is a measure of the di erence between quantitatively measure the training result of a GAN. The SSIM is a measure of the difference between the ground truth image and the image being compared. The value closer to 1 indicates better results. the ground truth image and the image being compared. The value closer to 1 indicates better results. The PSNR is a measure of the sharpness and uses the di erence in noise between the ground truth The PSNR is a measure of the sharpness and uses the difference in noise between the ground truth image and the restored image. The table below presents the accuracy measured by the SSIM (Table 1), image and the restored image. The table below presents the accuracy measured by the SSIM (Table which we classified according to the mask ratios to assess the e ect on the image completion. The 1), which we classified according to the mask ratios to assess the effect on the image completion. The closer the value is to 1, the better the result. closer the value is to 1, the better the result. Table 1. Evaluation using structural similarity index (SSIM). Table 1. Evaluation using structural similarity index (SSIM). Mask Incisor Canine Premolar Molar Mask. Incisor Canine Premolar Molar 0–10% 0.921 0.918 0.911 0.923 0–10% 0.921 0.918 0.911 0.923 10–20% 0. 10–20% 915 0.915 0.0.911911 0.9060.906 0.913 0.913 20–30% 0.885 0.883 0.879 0.894 20–30% 0.885 0.883 0.879 0.894 30–40% 0.819 0.822 0.839 0.859 30–40% 0.819 0.822 0.839 0.859 For a more intuitive explanation of the SSIM, images with values of 0.75–0.95 among those used in the experiments are shown at 0.05 intervals (Figure 16). The left side shows the ground truth images, and the right side shows the restored images. From a value of 0.85 and above, we can see the edge lined up, and a value above 0.9 is similar to that of the ground truth images. Appl. Sci. 2020, 10, 490 11 of 17 For a more intuitive explanation of the SSIM, images with values of 0.75–0.95 among those used in the experiments are shown at 0.05 intervals (Figure 16). The left side shows the ground truth Appl. imagSci. es,2020 and , 10 the ri , 490 ght side shows the restored images. From a value of 0.85 and above, we can see the 11 of 18 edge lined up, and a value above 0.9 is similar to that of the ground truth images. Figure 16. Images of various structural similarity index (SSIMs). Figure 16. Images of various structural similarity index (SSIMs). The following is the result measured based on the PSNR (Table 2). Higher values indicate better The following is the result measured based on the PSNR (Table 2). Higher values indicate better results. The images used in the experiment are 16.42 at minimum and 28.27 at maximum. results. The images used in the experiment are 16.42 at minimum and 28.27 at maximum. As with the SSIM, among the images used in the experiment, images with a PSNR of 16–28 are shown at three intervals (Figure 17). The edges on the left are the ground truth images and the images on the right are the generated images. When the PSNR is less than 20, we can see that the edge is broken or missing. A small amount of noise occurs from above 22, although this is similar to the ground truth images. Appl. Sci. 2020, 10, 490 12 of 17 Table 2. Evaluation based on PSNR. Mask Incisor Canine Premolar Molar 0–10% 26.68 24.42 24.71 26.27 Appl. Sci.1 20200–2,010% , 2 490 5.82 24.19 23.69 25.07 12 of 18 20–30% 22.72 22.57 21.62 23.33 30–40% 19.49 19.78 19.71 21.93 Table 2. Evaluation based on PSNR. As with the SSIM, among the images used in the experiment, images with a PSNR of 16–28 are Mask Incisor Canine Premolar Molar shown at three intervals (Figure 17). The edges on the left are the ground truth images and the images 0–10% 26.68 24.42 24.71 26.27 on the right are the generated images. When the PSNR is less than 20, we can see that the edge is 10–20% 25.82 24.19 23.69 25.07 20–30% 22.72 22.57 21.62 23.33 broken or missing. A small amount of noise occurs from above 22, although this is similar to the 30–40% 19.49 19.78 19.71 21.93 ground truth images. Figure 17. Images of various PSNRs. Figure 17. Images of various PSNRs. Appl. Sci. 2020, 10, 490 13 of 18 Appl. Sci. 2020, 10, 490 13 of 17 Appl. Sci. 2020, 10, 490 13 of 17 The SSIM and PSNR differed significantly according to the mask sizes. Figures 18 and 19 show The SSIM and PSNR di ered significantly according to the mask sizes. Figures 18 and 19 show The SSIM and PSNR differed significantly according to the mask sizes. Figures 18 and 19 show the changes according to the mask sizes. We can see this visually in the figures below (Figure 20). The the changes according to the mask sizes. We can see this visually in the figures below (Figure 20). The the changes according to the mask sizes. We can see this visually in the figures below (Figure 20). The smaller the mask size is the better. However, there is a minimum mask size of an image because the smaller the mask size is the better. However, there is a minimum mask size of an image because the smaller the mask size is the better. However, there is a minimum mask size of an image because the mask must cover the occlusion area. Therefore, the use of large and high-resolution images lead to mask must cover the occlusion area. Therefore, the use of large and high-resolution images lead to mask must cover the occlusion area. Therefore, the use of large and high-resolution images lead to better results. However, computer performance was limited, and we used cropped images. better results. However, computer performance was limited, and we used cropped images. better results. However, computer performance was limited, and we used cropped images. Figure 18. Effect of mask sizes on the SSIM. Figure 18. E ect of mask sizes on the SSIM. Figure 18. Effect of mask sizes on the SSIM. Figure 19. E ect of mask sizes on the PSNR. Figure 19. Effect of mask sizes on the PSNR. Figure 19. Effect of mask sizes on the PSNR. Appl. Sci. 2020, 10, 490 14 of 18 Appl. Sci. 2020, 10, 490 14 of 17 Figure 20. Results of di erent mask sizes. Figure 20. Results of different mask sizes. In this study, we used images with an SSIM of over 0.9 and a PSNR of over 22 for 3D reconstruction. In this study, we used images with an SSIM of over 0.9 and a PSNR of over 22 for 3D Overall, when the mask size is less than 10%, 95% of the images obtained are available for 3D reconstruction. Overall, when the mask size is less than 10%, 95% of the images obtained are available reconstruction. It took 0.1 s for an image completion, and we used 100 slices of both the maxilla for 3D reconstruction. It took 0.1 s for an image completion, and we used 100 slices of both the maxilla and mandible. We conducted the image completion five times per slice. Therefore, the entire image and mandible. We conducted the image completion five times per slice. Therefore, the entire image completion process took approximately 100 s. completion process took approximately 100 s. 3.2. Results of Tooth Segmentation 3.2. Results of Tooth Segmentation 3.2.1. Tooth Model 3.2.1. Tooth Model A tooth pin model of a malocclusion patient was fabricated for the testing of the segmentation algorithm (Figure 21). This model allows each tooth to be removed and replaced to its original position A tooth pin model of a malocclusion patient was fabricated for the testing of the segmentation from the tooth model base. Therefore, it is suitable for use in our experiments because each tooth can algorithm (Figure 21). This model allows each tooth to be removed and replaced to its original be removed and scanned individually with no occlusions. The scan data of each tooth were used as position from the tooth model base. Therefore, it is suitable for use in our experiments because each the ground truth. tooth can be removed and scanned individually with no occlusions. The scan data of each tooth were used as the ground truth. 3.2.2. Accuracy Measurement Process Two data are needed for an accurate measurement. One is the entire scan data and the other is separated scan data. The former has occlusion areas, whereas, the latter does not. Therefore, we used the separated scan data as the ground truth. Next, we conducted segmentation on the entire scan data using the conventional and proposed methods (Figure 22). Then, we measured the accuracy of the separated scan data as the ground truth and compared it with the tooth data segmented by each method. Appl. Sci. 2020, 10, 490 15 of 18 Appl. Sci. 2020, 10, 490 15 of 17 Appl. Sci. 2020, 10, 490 15 of 17 Figure 21. Tooth pin model. Figure 21. Tooth pin model. 3.2.2. Accuracy Measurement Process Two data are needed for an accurate measurement. One is the entire scan data and the other is separated scan data. The former has occlusion areas, whereas, the latter does not. Therefore, we used the separated scan data as the ground truth. Next, we conducted segmentation on the entire scan data using the conventional and proposed methods (Figure 22). Then, we measured the accuracy of the separated scan data as the ground truth and compared it with the tooth data segmented by Figure 21. Tooth pin model. each method. (a) (b) Figure 22. (a) Data obtained through segmentation obtained using the proposed and (b) conventional methods. 3.2.3. Measurement Accuracy Results Jung et al. [28] proposed an effective registration technique, which we used for the registration of the three data types. After the registration, we used Geomagic software (2012, Ping Fu, Herbert (a) (b) Edelsbrunner, Morrisville, NC, USA) to compare the distances between the data. We set the ground Figure 22. (a) Data obtained through segmentation obtained using the proposed and (b) conventional Figure 22. (a) Data obtained through segmentation obtained using the proposed and truth data as a mesh and the data segmented by each method as point clouds. The data segmented (b) conventional methods. methods. by each method were then sampled for a constant number of points. We classified the results into four tooth types, namely, incisor, canine, premolar, and molar, and measured the average mean 3.2.3. Measurement Accuracy Results 3.2.3. Measurement Accuracy Results distance between the point cloud and the mesh. The table below presents the results (Table 3). The Jung et al. [28] proposed an e ective registration technique, which we used for the registration conventional method indicates the manual segmentation using the snake. The proposed method Jung et al. [28] proposed an effective registration technique, which we used for the registration of the three data types. After the registration, we used Geomagic software (2012, Ping Fu, Herbert indicates segmentation by using GAN. of the three data types. After the registration, we used Geomagic software (2012, Ping Fu, Herbert Edelsbrunner, Morrisville, NC, USA) to compare the distances between the data. We set the ground Edelsbrunner, Morrisville, NC, USA) to compare the distances between the data. We set the ground truth data as a mesh and the data segmented by each method as point clouds. The data segmented Table 3. Average mean distance between the point cloud and mesh. truth data as a mesh and the data segmented by each method as point clouds. The data segmented by each method were then sampled for a constant number of points. We classified the results into by each method were then sampled for a constant number of points. We classified the results into four tooth types, Method namely , incisor, canine, premolar Mean d , and istmolar ance (m , and m) measured the average p mean four tooth types, namely, incisor, canine, premolar, and molar, and measured the average mean Conventional 0.031 ± 0.008 0.033 distance between the point cloud and the mesh. The table below presents the results (Table 3). The Proposed 0.027 ± 0.007 conventional method indicates the manual segmentation using the snake. The proposed method p, p-value for independent t-test. indicates segmentation by using GAN. Table 3. Average mean distance between the point cloud and mesh. Method Mean distance (mm) p Conventional 0.031 ± 0.008 0.033 Proposed 0.027 ± 0.007 p, p-value for independent t-test. Appl. Sci. 2020, 10, 490 16 of 18 distance between the point cloud and the mesh. The table below presents the results (Table 3). The conventional method indicates the manual segmentation using the snake. The proposed method indicates segmentation by using GAN. Table 3. Average mean distance between the point cloud and mesh. Method Mean Distance (mm) p Conventional 0.031  0.008 0.033 Proposed 0.027  0.007 p, p-value for independent t-test. The mean distance between the point cloud and the mesh for each method is displayed in Table 3. The above results show that the proposed method is approximately 0.004 mm more accurate than the conventional method, and this di erence was significantly di erent (p = 0.033). 4. Conclusions In this study, we proposed an automated method to reconstruct the missing data in the interdental area of teeth, scanned using an intraoral scanner by applying a GAN. A limitation is that the masks were manually applied to the occlusion area, therefore, some of the normal scan data may have been removed. A further study carried out to automatically detect the occlusion area would define the masks with higher precision and increase the accuracy of segmentation. Nonetheless, based on our results on the dimensional accuracy of the segmented teeth, it can be inferred that the segmentation method for full arch intraoral scan data is as accurate as a manual segmentation method, in addition to being e ective and time saving. This automatic segmentation tool can be used to facilitate the digital setup process in orthodontic treatment. Author Contributions: Conceptualization, T.K.; data curation, D.K.; formal analysis, T.K.; investigation, M.C.; Methodology, T.K. and Y.C.; project administration, M.C.; resources, Y.-J.K.; supervision, Y.-J.K. and M.C.; validation, T.K.; visualization, T.K.; writing—original draft, T.K.; writing—review and editing, Y.-J.K. All authors have read and agreed to the published version of the manuscript. Funding: This research received no external funding. Acknowledgments: We are grateful to the Department of Orthodontics, Korea University Anam Hospital for providing the patients’ scan data and the tooth pin model. Conflicts of Interest: The authors declare no conflicts of interest. References 1. Kesling, H. The diagnostic setup with consideration of the third dimension. Am. J. Orthod. Dentofac. Orthop. 1956, 42, 740–748. [CrossRef] 2. Barreto, M.S.; Faber, J.; Vogel, C.J.; Araujo, T.M. Reliability of digital orthodontic setups. Angle Orthod. 2016, 86, 255–259. [CrossRef] [PubMed] 3. Im, J.; Cha, J.Y.; Lee, K.J.; Yu, H.S.; Hwang, C.J. Comparison of virtual and manual tooth setups with digital and plaster models in extraction cases. Am. J. Orthod. Dentofac. Orthop. 2014, 145, 434–442. [CrossRef] [PubMed] 4. Chen, B.T.; Lou, W.S.; Chen, C.C.; Lin, H.C. A 3D scanning system based on low-occlusion approach. In Proceedings of the Second International Conference on 3-D Digital Imaging and Modeling (Cat. No. PR00062), Ottawa, ON, Canada, 8 October 1999; IEEE: Piscataway, NJ, USA, 1999; pp. 506–515. 5. Shlafman, S.; Tal, A.; Katz, S. Metamorphosis of polyhedral surfaces using decomposition. In Computer Graphics Forum; Blackwell Publishing, Inc.: Oxford, UK, 2002; pp. 219–228. 6. Mangan, A.P.; Whitaker, R.T. Partitioning 3D surface meshes using watershed segmentation. IEEE Trans. Vis. Comput. Graph. 1999, 5, 308–321. [CrossRef] Appl. Sci. 2020, 10, 490 17 of 18 7. Lee, Y.; Lee, S.; Shamir, A.; Cohen-Or, D.; Seidel, H.P. Mesh scissoring with minima rule and part salience. Comput. Aided Geom. Des. 2005, 22, 444–465. [CrossRef] 8. Zanjani, F.G.; Moin, D.A.; Claessen, F.; Cherici, T.; Parinussa, S.; Pourtaherian, A.; Zinger, S. Mask-MCNet: Instance segmentation in 3D point cloud of intra-oral scans. In International Conference on Medical Image Computing and Computer-Assisted Intervention; Springer: Cham, Switzerland, 2019; pp. 128–136. 9. Podolak, J.; Shilane, P.; Golovinskiy, A.; Rusinkiewicz, S.; Funkhouser, T. A planar-reflective symmetry transform for 3D shapes. ACM Trans. Graph. (TOG) 2006, 25, 549–559. [CrossRef] 10. Lai, Y.K.; Hu, S.M.; Martin, R.R.; Rosin, P.L. Fast mesh segmentation using random walks. In Proceedings of the 2008 ACM Symposium on Solid and Physical Modeling, New York, NY, USA, 2–4 June 2008; ACM: New York, NY, USA, 2008; pp. 183–191. 11. Kronfeld, T.; Brunner, D.; Brunnett, G. Snake-based segmentation of teeth from virtual dental casts. Comput. Aided Des. Appl. 2010, 7, 221–233. [CrossRef] 12. Barnes, C.; Shechtman, E.; Goldman, D.B.; Finkelstein, A. The generalized patchmatch correspondence algorithm. In European Conference on Computer Vision; Springer: Berlin/Heidelberg, Germany, 2010; pp. 29–43. 13. Darabi, S.; Shechtman, E.; Barnes, C.; Goldman, D.B.; Sen, P. Image melding: Combining inconsistent images using patch-based synthesis. ACM Trans. Graph. 2012, 31, 82-1–82-10. [CrossRef] 14. Huang, J.B.; Kang, S.B.; Ahuja, N.; Kopf, J. Image completion using planar structure guidance. ACM Trans. Graph. (TOG) 2014, 33, 129. [CrossRef] 15. Simakov, D.; Caspi, Y.; Shechtman, E.; Irani, M. Summarizing visual data using bidirectional similarity. In Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition, Anchorage, AK, USA, 23–28 June 2008; IEEE: Piscataway, NJ, USA, 2008; pp. 1–8. 16. Köhler, R.; Schuler, C.; Schölkopf, B.; Harmeling, S. Mask-specific inpainting with deep neural networks. In German Conference on Pattern Recognition; Springer: Cham, Switzerland, 2014; pp. 523–534. 17. Ren, J.S.; Xu, L.; Yan, Q.; Sun, W. Shepard convolutional neural networks. In Advances in Neural Information Processing Systems; The MIT Press: Cambridge, MA, USA, 2015; pp. 901–909. 18. Iizuka, S.; Simo-serra, E.; Ishikawa, H. Globally and locally consistent image completion. ACM Trans Graph. (ToG) 2017, 36, 107. [CrossRef] 19. Nazeri, K.; Ng, E.; Joseph, T.; Qureshi, F.; Ebrahimi, M. Edgeconnect: Generative image inpainting with adversarial edge learning. arXiv 2019, arXiv:1901.00212. 20. Besl, P.J.; Mckay, N.D. Method for registration of 3-D shapes. In Sensor Fusion IV: Control Paradigms and Data Structures; International Society for Optics and Photonics: Bellingham, WA, USA, 1992; pp. 586–606. 21. Sajjadi, M.S.; Scholkopf, B.; Hirsch, M. Enhancenet: Single image super-resolution through automated texture synthesis. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 4491–4500. 22. Waleed Gondal, M.; Scholkopf, B.; Hirsch, M. The unreasonable e ectiveness of texture transfer for single image super-resolution. In Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 8–14 September 2018. 23. Isola, P.; Zhu, J.Y.; Zhou, T.; Efros, A.A. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 1125–1134. 24. Zhu, J.Y.; Park, T.; Isola, P.; Efros, A.A. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 2223–2232. 25. Canny, J. A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell. 1986, 6, 679–698. [CrossRef] 26. Wang, Z.; Bovik, A.C.; Sheikh, H.R.; Simoncelli, E.P. Image quality assessment: From error visibility to structural similarity. IEEE Trans. Image Process. 2004, 13, 600–612. [CrossRef] [PubMed] Appl. Sci. 2020, 10, 490 18 of 18 27. De Boer, J.F.; Cense, B.; Park, B.H.; Pierce, M.C.; Tearney, G.J.; Bouma, B.E. Improved signal-to-noise ratio in spectral-domain compared with time-domain optical coherence tomography. Opt. Lett. 2003, 28, 2067–2069. [CrossRef] [PubMed] 28. Jung, S.; Song, S.; Chang, M.; Park, S. Range image registration based on 2D synthetic images. Comput. Aided Des. 2018, 94, 16–27. [CrossRef] © 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/). http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Applied Sciences Multidisciplinary Digital Publishing Institute

Tooth Segmentation of 3D Scan Data Using Generative Adversarial Networks

Loading next page...
 
/lp/multidisciplinary-digital-publishing-institute/tooth-segmentation-of-3d-scan-data-using-generative-adversarial-eKQIDAlalc
Publisher
Multidisciplinary Digital Publishing Institute
Copyright
© 1996-2020 MDPI (Basel, Switzerland) unless otherwise stated Disclaimer The statements, opinions and data contained in the journals are solely those of the individual authors and contributors and not of the publisher and the editor(s). Terms and Conditions Privacy Policy
ISSN
2076-3417
DOI
10.3390/app10020490
Publisher site
See Article on Publisher Site

Abstract

applied sciences Article Tooth Segmentation of 3D Scan Data Using Generative Adversarial Networks 1 1 1 1 , 2 , Taeksoo Kim , Youngmok Cho , Doojun Kim , Minho Chang * and Yoon-Ji Kim * Department of Mechanical Engineering, Korea University, Seoul 02841, Korea; pqowep@korea.ac.kr (T.K.); dudahr2002@korea.ac.kr (Y.C.); rlaenwns95@naver.com (D.K.) Department of Orthodontics, Korea University Anam Hospital, Seoul 02841, Korea * Correspondence: mhchang@korea.ac.kr (M.C.); yn0331@korea.ac.kr (Y.-J.K.); Tel.: +82-2-3290-3379 (M.C.) Received: 27 November 2019; Accepted: 8 January 2020; Published: 9 January 2020 Abstract: The use of intraoral scanners in the field of dentistry is increasing. In orthodontics, the process of tooth segmentation and rearrangement provides the orthodontist with insights into the possibilities and limitations of treatment. Although, full-arch scan data, acquired using intraoral scanners, have high dimensional accuracy, they have some limitations. Intraoral scanners use a stereo-vision system, which has diculties scanning narrow interdental spaces. These areas, with a lack of accurate scan data, are called areas of occlusion. Owing to such occlusions, intraoral scanners often fail to acquire data, making the tooth segmentation process challenging. To solve the above problem, this study proposes a method of reconstructing occluded areas using a generative adversarial network (GAN). First, areas of occlusion are eliminated, and the scanned data are sectioned along the horizontal plane. Next, images are trained using the GAN. Finally, the reconstructed two-dimensional (2D) images are stacked to a three-dimensional (3D) image and merged with the data where the occlusion areas have been removed. Using this method, we obtained an average improvement of 0.004 mm in the tooth segmentation, as verified by the experimental results. Keywords: tooth segmentation; generative adversarial networks; intraoral scanners; reconstruction; image completion; dental scan data; occlusion areas 1. Introduction Intraoral scanners are widely used in the diagnosis and fabrication of orthodontic appliances. A digital impression technique using intraoral scanners has been replacing a conventional tooth model setup applied manually using plaster casts. However, intraoral scanners have a problem of occlusions due to the stereo-vision system applied. This lowers the accuracy of the tooth’s anatomic form, and therefore, a segmentation of each tooth is required, which requires a lot of time and e ort by the operator. This study aims to solve the occlusion problem by preprocessing the scan data obtained by intraoral scanners through the use of a generative adversarial network (GAN). 1.1. Background 1.1.1. Digital Orthodontics An orthodontic diagnosis is a crucial step, whereby, the patient is examined based on various modalities, such as radiographs, intraoral photos, and tooth impressions. The information gathered from the patient is analyzed, and the final diagnosis and treatment plans, such as extraction versus non-extraction, and an anchorage preparation are made. As the visual treatment objective, patient’s dental plaster model is used to simulate the orthodontic treatment by dissecting each tooth from the plaster model and rearranging them into the desired position [1]. Although, this process provides Appl. Sci. 2020, 10, 490; doi:10.3390/app10020490 www.mdpi.com/journal/applsci Appl. Sci. 2020, 10, 490 2 of 18 Appl. Sci. 2020, 10, 490 2 of 17 Appl. Sci. 2020, 10, 490 2 of 17 orthodon the orthodontist tist insiinsight ght into t into he the poss possibilities ibilities and lim and limitations itations ofof tre tra eatment, tment, cre cra eating ting a di a diagnostic agnostic set setup up orthodontist insight into the possibilities and limitations of treatment, creating a diagnostic setup model from a dental plaster model is time-consuming and labor intensive. model model fr from om a a dental plaste dental plaster r model is model is time-consuming time-consuming a and nd la labor bor intensive. intensive. Since Since the the a advent dvent of of in intraoral traoral sc scanners, anners, iinstead nstead of of us using ing the the den dental tal p plaster laster m model, odel, in intraoral traoral sc scan an Since the advent of intraoral scanners, instead of using the dental plaster model, intraoral scan data can be manipulated to achieve a digital tooth setup. After acquiring the full arch scan data of the data data can can b be e m manipulated anipulated to to achieve achieve a a di digital gital tooth tooth se setup. tup. Af After ter ac acquiring quiring the the full full ar arch ch sc scan an d data ata of of the the p patient, atient, the the t teeth eeth ar are e segmented segmented and and repositioned repositioned to t the o the desir des ed ire position d position using usthr ing th ee-dimensional ree-dimension (3D) al patient, the teeth are segmented and repositioned to the desired position using three-dimensional (3D) CAD software. This is more efficient because it eliminates the process of making a plaster model, CAD (3D) C softwar AD sofe. tw This are. T ishmor is ise m eo  re cient efficien because t because it eliminates it eliminates the pr the p ocess rocess of m of making aking a plaster a plaster model, model, and and it has been reported to be as effective and accurate as a conventional method [2,3]. it and has it ha been s bre eported en reporte to be d to asbe e as ective effecand tive and accurate accuas rat ae a conventional s a convention method al metho [2,3d ]. [2,3]. 1.1.2. Tooth Segmentation 1.1.2. Tooth Segmentation 1.1.2. Tooth Segmentation An intraoral scan data of a full dental arch leads to a single scanned object that contains all teeth An intraoral scan data of a full dental arch leads to a single scanned object that contains all teeth An intraoral scan data of a full dental arch leads to a single scanned object that contains all teeth as a single unit. Although, each tooth has its own boundary, and teeth are not connected to each as a single unit. Although, each tooth has its own boundary, and teeth are not connected to each as a single unit. Although, each tooth has its own boundary, and teeth are not connected to each other, this problem is caused by an occlusion of the intraoral scanner. Intraoral scanners usually use other, this problem is caused by an occlusion of the intraoral scanner. Intraoral scanners usually use other, this problem is caused by an occlusion of the intraoral scanner. Intraoral scanners usually use a stereo-vision system, and the interdental areas of the teeth are not captured. This space is called a stereo-vision system, and the interdental areas of the teeth are not captured. This space is called an a stereo-vision system, and the interdental areas of the teeth are not captured. This space is called an an occlusion area and leads to a low accuracy (Figure 1) [4]. Therefore, after the operator manually occlusion area and leads to a low accuracy (Figure 1) [4]. Therefore, after the operator manually occlusion area and leads to a low accuracy (Figure 1) [4]. Therefore, after the operator manually defines the tooth boundary and the long axis of the tooth, the plane to be used for tooth separation is defines the tooth boundary and the long axis of the tooth, the plane to be used for tooth separation is defines the tooth boundary and the long axis of the tooth, the plane to be used for tooth separation is also designated manually to separate the teeth. also designated manually to separate the teeth. also designated manually to separate the teeth. Figure 1. Occlusion areas of dental scan data. Figure 1. Occlusion areas of dental scan data. Figure 1. Occlusion areas of dental scan data. 1.2. Related Studies 1.2. Related Studies 1.2. Related Studies Several studies have been conducted to improve the tooth segmentation. Ongoing studies Several studies have been conducted to improve the tooth segmentation. Ongoing studies can Several studies have been conducted to improve the tooth segmentation. Ongoing studies can can be divided into two categories. One is a boundary-based segmentation and the other is a be divided into two categories. One is a boundary-based segmentation and the other is a region-based be divided into two categories. One is a boundary-based segmentation and the other is a region-based region-based segmentation. Region-based methods are mainly conducted using K-means clustering [5], segmentation. Region-based methods are mainly conducted using K-means clustering [5], a segmentation. Region-based methods are mainly conducted using K-means clustering [5], a a watershed [6], an active contour [7], and Mask-MCNet [8]. Boundary-based methods include a graph watershed [6], an active contour [7], and Mask-MCNet [8]. Boundary-based methods include a graph watershed [6], an active contour [7], and Mask-MCNet [8]. Boundary-based methods include a graph cut [9], random walks [10], and a snake [11]. Among them, the most popular method is to set the cut [9], random walks [10], and a snake [11]. Among them, the most popular method is to set the cut [9], random walks [10], and a snake [11]. Among them, the most popular method is to set the initial position of the boundary between teeth and gums using a snake. After the initial position is set, initial position of the boundary between teeth and gums using a snake. After the initial position is initial position of the boundary between teeth and gums using a snake. After the initial position is the operator finishes by modifying the initial position (Figure 2). However, because of the occlusion set, the operator finishes by modifying the initial position (Figure 2). However, because of the set, the operator finishes by modifying the initial position (Figure 2). However, because of the problem, each of these methods is limited in accuracy. occlusion problem, each of these methods is limited in accuracy. occlusion problem, each of these methods is limited in accuracy. (a) (b) (a) (b) Figure 2. Figure 2. Segmentation using Segmentation using a snake: ( a snake: (a a)) Ini Initial; tial; and ( and (b b) m ) modified odified po positions sitions u using sing snak snake. e. Figure 2. Segmentation using a snake: (a) Initial; and (b) modified positions using snake. Appl. Sci. 2020, 10, 490 3 of 17 Appl. Sci. 2020, 10, 490 3 of 18 1.3. Motivation and Contributions of This Paper 1.3. Motivation and Contributions of This Paper 1.3.1. Generative Adversarial Network (GAN) 1.3.1. Generative Adversarial Network (GAN) Research on artificial intelligence has recently been actively conducted. A generative adversarial network (GAN) is a type of artificial intelligence that method that learns a generative model and a Research on artificial intelligence has recently been actively conducted. A generative adversarial discriminative model simultaneously. The system configuration of a GAN is described as follows network (GAN) is a type of artificial intelligence that method that learns a generative model and a (Figure 3). The generative model learns the distribution of data, and the discriminative model discriminative model simultaneously. The system configuration of a GAN is described as follows evaluate the data generated by the generative model. Next, the generative model then generates data (Figure 3). The generative model learns the distribution of data, and the discriminative model evaluate such that an error when evaluating the learning model and an error when evaluating the generated the data generated by the generative model. Next, the generative model then generates data such that model are similar. The generative model is briefly trained to induce a discrimination of the an error when evaluating the learning model and an error when evaluating the generated model are discriminative model. similar. The generative model is briefly trained to induce a discrimination of the discriminative model. Figure 3. Overview of generative adversarial network (GAN). Figure 3. Overview of generative adversarial network (GAN). 1.3.2. Image Completion 1.3.2. Image Completion Image completion is a technique used to remove and complete parts of an image that we want Image completion is a technique used to remove and complete parts of an image that we want to restore. In this way, we can remove an unwanted part of an image and regenerate the occluded to restore. In this way, we can remove an unwanted part of an image and regenerate the occluded regions. Previously, there have been many approaches using patch-based methods [12–15]. However, regions. Previously, there have been many approaches using patch-based methods [12–15]. However, the results were unsatisfactory owing to the complexity of the image pattern in various of objects, and the results were unsatisfactory owing to the complexity of the image pattern in various of objects, it was dicult to increase the accuracy. and it was difficult to increase the accuracy. Methods using artificial intelligence have recently been studied. A context encoder is one such Methods using artificial intelligence have recently been studied. A context encoder is one such method and employs a convolutional neural network (CNN) [16,17]. Globally and locally consistent method and employs a convolutional neural network (CNN) [16,17]. Globally and locally consistent image completion (GLCIC) and Edge Connect also use artificial intelligence [18,19]. Whereas a context image completion (GLCIC) and Edge Connect also use artificial intelligence [18,19]. Whereas a encoder is based on a CNN, a GLCIC and Edge Connect are based on a GAN. Among all methods context encoder is based on a CNN, a GLCIC and Edge Connect are based on a GAN. Among all described above, Edge Connect achieves the best accuracy [19]. Therefore, in this study, we employed methods described above, Edge Connect achieves the best accuracy [19]. Therefore, in this study, we an Edge Connect as the image completion method. employed an Edge Connect as the image completion method. 1.3.3. Goals 1.3.3. Goals Various studies have been made to solve the diculties of segmentation, and automation has resulted Variin ous stud a reduction ies ha in veworking been made time. toHowever solve the , both difficu methods lties of s using egmen the tatooth tion, and boundary automand ation tooth has resu region lted do in a not resolve duction in the occlusion working tim problem. e. However, In this bst ot udy h m ,ewe thods aim us to ing th solve e too theth occlusion boundarpr y an oblem d tooby th region do not solve the occlusion problem. In this study, we aim to solve the occlusion problem by Appl. Sci. 2020, 10, 490 4 of 18 Appl. Sci. 2020, 10, 490 4 of 17 pre-processing dental scan data using the image completion method. Additionally, using this approach, pre-processing dental scan data using the image completion method. Additionally, using this we expect to improve the tooth segmentation accuracy. approach, we expect to improve the tooth segmentation accuracy. 2. Proposed Method 2. Proposed Method 2.1. Overview 2.1. Overview The system used to solve the occlusion problem of dental scan data when applying a GAN can The system used to solve the occlusion problem of dental scan data when applying a GAN can be introduced through three steps. The first step is to remove the occlusion areas of the dental scan be introduced through three steps. The first step is to remove the occlusion areas of the dental scan data. The next step is to complete the images on each plane. The final step is to stack the images from data. The next step is to complete the images on each plane. The final step is to stack the images from the previous step and reconstruct the dental scan data. The figure below depicts an overview of the the previous step and reconstruct the dental scan data. The figure below depicts an overview of the proposed method (Figure 4). proposed method (Figure 4). Figure 4. Overview of the proposed method. Figure 4. Overview of the proposed method. 2.2. Reconstruction of Dental Scan Data 2.2. Reconstruction of Dental Scan Data The reconstruction process is as follows. First, the occlusion area is specified by visual inspection The reconstruction process is as follows. First, the occlusion area is specified by visual inspection of the interdental area whereby the intraoral scan data are not captured accurately. Here, the removed of the interdental area whereby the intraoral scan data are not captured accurately. Here, the removed occlusion areas will be used as masks and the location must be saved. The occlusion area was deleted, occlusion areas will be used as masks and the location must be saved. The occlusion area was deleted, and this removed area must be reconstructed. Figure 5 depicts images from before and after removing and this removed area must be reconstructed. Figure 5 depicts images from before and after removing thethe occ occlusion lusion area. area. Appl. Sci. 2020, 10, 490 5 of 18 Appl. Appl. Sci. Sci. 2020 2020,, 10 10,, 490 490 5 of 5 of 17 17 Appl. Sci. 2020, 10, 490 5 of 17 (a) (b) (a) (b) (a) (b) Figure 5. Example of removing the occlusion area: (a) before and (b) after removal. Figure 5. Example of removing the occlusion area: (a) before and (b) after removal. Figure 5. Figure 5. Example of Example of removing the occlusion removing the occlusion area: ( area: (a a) befor ) before e and ( and (b b) after ) after removal. removal. Next, dental scan data are sectioned at regular intervals using the horizontal plane direction. Next, dental scan data are sectioned at regular intervals using the horizontal plane direction. Next, dental scan data are sectioned at regular intervals using the horizontal plane direction. Next, dental scan data are sectioned at regular intervals using the horizontal plane direction. The more images we apply during the reconstruction at narrow intervals, the higher the accuracy The more images we apply during the reconstruction at narrow intervals, the higher the accuracy The more images we apply during the reconstruction at narrow intervals, the higher the accuracy The more images we apply during the reconstruction at narrow intervals, the higher the accuracy that will be acquired, but the longer the computational time required. We used 0.1 mm intervals and that will be acquired, but the longer the computational time required. We used 0.1 mm intervals and that will be acquired, but the longer the computational time required. We used 0.1 mm intervals and that will be acquired, but the longer the computational time required. We used 0.1 mm intervals and achieved an image completion using 100 images per patient. The figures below show several images achieved an image completion using 100 images per patient. The figures below show several images achieved an image completion using 100 images per patient. The figures below show several images achieved an image completion using 100 images per patient. The figures below show several images acquired using horizontal plane sectioning (Figure 6). Image completion is then conducted using a acquired using horizontal plane sectioning (Figure 6). Image completion is then conducted using a acquired using horizontal plane sectioning (Figure 6). Image completion is then conducted using a acquired using horizontal plane sectioning (Figure 6). Image completion is then conducted using a GAN (Figure 7). GAN (Figure 7). GAN (Figure 7). GAN (Figure 7). Figure 6. Examples of horizontal plane sectioning. Figure 6. Examples of horizontal plane sectioning. Figure 6. Examples of horizontal plane sectioning. Figure 6. Examples of horizontal plane sectioning. Figure 7. Examples of image completion. Figure 7. Examples of image completion. Figure 7. Examples of image completion. Figure 7. Examples of image completion. Appl. Sci. 2020, 10, 490 6 of 18 Appl. Sci. 2020, 10, 490 6 of 17 Appl. Sci. 2020, 10, 490 6 of 17 Finally, the completed images are stacked using the previously saved location and height interval. Finally, the completed images are stacked using the previously saved location and height Finally, the completed images are stacked using the previously saved location and height We use only the restored area, as indicated by the circle in Figure 7. When the location of the interval. We use only the restored area, as indicated by the circle in Figure 7. When the location of the interval. We use only the restored area, as indicated by the circle in Figure 7. When the location of the reconstructed part is unknown, a registration should be applied using an algorithm, such as the reconstructed part is unknown, a registration should be applied using an algorithm, such as the reconstructed part is unknown, a registration should be applied using an algorithm, such as the iterative closest point (ICP) [20], which can cause additional errors. After the images are stacked iterative closest point (ICP) [20], which can cause additional errors. After the images are stacked to iterative closest point (ICP) [20], which can cause additional errors. After the images are stacked to to achieve a 3D image, the image is merged with the data where the occlusion areas have been achieve a 3D image, the image is merged with the data where the occlusion areas have been removed. achieve a 3D image, the image is merged with the data where the occlusion areas have been removed. removed. The process is completed through remeshing. The figures below illustrate the merging The process is completed through remeshing. The figures below illustrate the merging process The process is completed through remeshing. The figures below illustrate the merging process process (Figure 8). (Figure 8). (Figure 8). (a) (b) (c) (a) (b) (c) Figure 8. Example after reconstruction: (a) Dental scan data with occlusion area removed, (b) Figure 8. Example after reconstruction: (a) Dental scan data with occlusion area removed, Figure 8. Example after reconstruction: (a) Dental scan data with occlusion area removed, (b) reconstructed data by stacking images, and (c) dental scan data by merging (a,b). (b) reconstructed data by stacking images, and (c) dental scan data by merging (a,b). reconstructed data by stacking images, and (c) dental scan data by merging (a,b). 2.3. Training GAN 2.3. Training GAN 2.3. Training GAN Edge Connect, which adds the contour reconstruction described above, requires two learning Edge Connect, which adds the contour reconstruction described above, requires two learning Edge Connect, which adds the contour reconstruction described above, requires two learning processes. One process is contour restoration learning and the other is image completion learning. processes. One process is contour restoration learning and the other is image completion learning. processes. One process is contour restoration learning and the other is image completion learning. Both processes use a GAN. The neural networks of the generative model and the discriminative model Both processes use a GAN. The neural networks of the generative model and the discriminative Both processes use a GAN. The neural networks of the generative model and the discriminative used in this study are similar to those of Edge Connect. In detail, a generative model follows the model used in this study are similar to those of Edge Connect. In detail, a generative model follows model used in this study are similar to those of Edge Connect. In detail, a generative model follows super-resolution architecture [21,22], and the discriminative model follows a 70  70 patch GAN the super-resolution architecture [21,22], and the discriminative model follows a 70 × 70 patch GAN the super-resolution architecture [21,22], and the discriminative model follows a 70 × 70 patch GAN architecture [23,24]. architecture [23,24]. architecture [23,24]. 2.3.1. Data Preparation 2.3.1. Data Preparation 2.3.1. Data Preparation This study was approved by the Institutional Review Board of Korea University Anam Hospital This study was approved by the Institutional Review Board of Korea University Anam Hospital This study was approved by the Institutional Review Board of Korea University Anam Hospital (2019AN0470). Intraoral scan data of ten patients, who had taken a digital impression as part of an (2019AN0470). Intraoral scan data of ten patients, who had taken a digital impression as part of an (2019AN0470). Intraoral scan data of ten patients, who had taken a digital impression as part of an orthodontic diagnosis, were exported and segmented by a trained dental technician. The scans were orthodontic diagnosis, were exported and segmented by a trained dental technician. The scans were orthodontic diagnosis, were exported and segmented by a trained dental technician. The scans were acquired by one orthodontist by using the Trios 3 intraoral scanner (3Shape, Denmark). The figure acquired by one orthodontist by using the Trios 3 intraoral scanner (3Shape, Denmark). The figure acquired by one orthodontist by using the Trios 3 intraoral scanner (3Shape, Denmark). The figure below depicts the dental scan data after the segmentation process and one of the horizontal plane below depicts the dental scan data after the segmentation process and one of the horizontal plane below depicts the dental scan data after the segmentation process and one of the horizontal plane images (Figure 9). images (Figure 9). images (Figure 9). (a) (b) (a) (b) Figure 9. Data after tooth segmentation: (a) 3D and (b) horizontal views of the data. Figure 9. Data after tooth segmentation: (a) 3D and (b) horizontal views of the data. Figure 9. Data after tooth segmentation: (a) 3D and (b) horizontal views of the data. Appl. Sci. 2020, 10, 490 7 of 18 Appl. Sci. 2020, 10, 490 7 of 17 Appl. Sci. 2020, 10, 490 7 of 17 The area to be used for learning is cropped into a 256 (px)  256 (px) square in the horizontally The area to be used for learning is cropped into a 256 (px) × 256 (px) square in the horizontally The area to be used for learning is cropped into a 256 (px) × 256 (px) square in the horizontally sectioned images, as shown in (b) of Figure 9. In this study, we used 10,000 cropped images for training sectioned images, as shown in (b) of Figure 9. In this study, we used 10,000 cropped images for sectioned images, as shown in (b) of Figure 9. In this study, we used 10,000 cropped images for the GAN. The ratio of the training, validation, and test sets was 28:1:1. In general, approximately training the GAN. The ratio of the training, validation, and test sets was 28:1:1. In general, training the GAN. The ratio of the training, validation, and test sets was 28:1:1. In general, 100,000 images are used for training complex images, such as CelebA or Paris Street View. However, in approximately 100,000 images are used for training complex images, such as CelebA or Paris Street approximately 100,000 images are used for training complex images, such as CelebA or Paris Street the case of tooth data, the images are not complicated and can be trained using only 10,000 images. The View. However, in the case of tooth data, the images are not complicated and can be trained using View. However, in the case of tooth data, the images are not complicated and can be trained using training data was organized such that the tooth images were contained. Figure 10 shows the process of only 10,000 images. The training data was organized such that the tooth images were contained. only 10,000 images. The training data was organized such that the tooth images were contained. image Figure cr 10 sho opping. ws the process of image cropping. Figure 10 shows the process of image cropping. Figure 10. Cropping images from segmented data. Figure 10. Cropping images from segmented data. Figure 10. Cropping images from segmented data. Five types of data are prepared: Cropped images, edge images, masks, masked images, and Five types of data are prepared: Cropped images, edge images, masks, masked images, and Five types of data are prepared: Cropped images, edge images, masks, masked images, and masked edge images. The cropped images can already be seen as edge images, but using edge images masked edge images. The cropped images can already be seen as edge images, but using edge images masked edge images. The cropped images can already be seen as edge images, but using edge images of the cropped images is important in terms of accuracy. We can apply the Edge Connect method of the cropped images is important in terms of accuracy. We can apply the Edge Connect method of the cropped images is important in terms of accuracy. We can apply the Edge Connect method when we consider only the cropped images as a picture, and not the edge images. For the edge images, when we consider only the cropped images as a picture, and not the edge images. For the edge when we consider only the cropped images as a picture, and not the edge images. For the edge we used the Canny edge detector with a threshold of 0.5 [25]. We determined the masks based on images, we used the Canny edge detector with a threshold of 0.5 [25]. We determined the masks images, we used the Canny edge detector with a threshold of 0.5 [25]. We determined the masks the dental spaces, where occlusions mainly occur. Any mask shape can be used, and the mask only based on the dental spaces, where occlusions mainly occur. Any mask shape can be used, and the based on the dental spaces, where occlusions mainly occur. Any mask shape can be used, and the needs to cover the areas where an occlusion is expected to occur. Herein, we used a rectangular shape. mask only needs to cover the areas where an occlusion is expected to occur. Herein, we used a mask only needs to cover the areas where an occlusion is expected to occur. Herein, we used a Figure 11 illustrates a few examples of the five types of data. rectangular shape. Figure 11 illustrates a few examples of the five types of data. rectangular shape. Figure 11 illustrates a few examples of the five types of data. (a) (b) (c) (d) (e) (a) (b) (c) (d) (e) Figure 11. Example of five types of training data: (a) cropped images from Figure 10, (b) edge images Figure 11. Example of five types of training data: (a) cropped images from Figure 10, (b) edge images Figure 11. Example of five types of training data: (a) cropped images from Figure 10, (b) edge images from (a), (c) masks, (d) masked images of (a), and (e) masked edges of (a). from (a), (c) masks, (d) masked images of (a), and (e) masked edges of (a). from (a), (c) masks, (d) masked images of (a), and (e) masked edges of (a). 2.3.2. Training Steps 2.3.2. Training Steps 2.3.2. Training Steps The first step is to pre-train the edge model. Among the data in Figure 11, we used masks The first step is to pre-train the edge model. Among the data in Figure 11, we used masks (Figure The first step is to pre-train the edge model. Among the data in Figure 11, we used masks (Figure (Figure 11c), masked images (Figure 11d), and masked edges (Figure 11e) as input images. Output 11c), masked images (Figure 11d), and masked edges (Figure 11e) as input images. Output images 11c), masked images (Figure 11d), and masked edges (Figure 11e) as input images. Output images are full edge images, and the target images of the output are depicted in Figure 11b. The goal of this are full edge images, and the target images of the output are depicted in Figure 11b. The goal of this Appl. Sci. 2020, 10, 490 8 of 18 Appl. Sci. 2020, 10, 490 8 of 17 images are full edge images, and the target images of the output are depicted in Figure 11b. The goal Appl. Sci. 2020, 10, 490 8 of 17 of this process is to pre-train only the edge model. Figure 12 depicts an image generated using the process is to pre-train only the edge model. Figure 12 depicts an image generated using the pre- process is to pre-train only the edge model. Figure 12 depicts an image generated using the pre- pre-trained edge model. As depicted in the figure, it is dicult to make an accurate image by training trained edge model. As depicted in the figure, it is difficult to make an accurate image by training the trained edge model. As depicted in the figure, it is difficult to make an accurate image by training the the edge alone. edge alone. edge alone. Figure 12. Image generated using the pre-trained edge model. Figure 12. Image generated using the pre-trained edge model. Figure 12. Image generated using the pre-trained edge model. The second step is to pre-train the image completion model. During this process, we use the edge The second step is to pre-train the image completion model. During this process, we use the The second step is to pre-train the image completion model. During this process, we use the images, masks, and masked images depicted in Figure 11b–d as input images, respectively. The output edge images, masks, and masked images depicted in Figure 11b–d as input images, respectively. The edge images, masks, and masked images depicted in Figure 11b–d as input images, respectively. The images are full images and the target images of the output are indicated in Figure 11a. The goal of this output images are full images and the target images of the output are indicated in Figure 11a. The output images are full images and the target images of the output are indicated in Figure 11a. The process is to pre-train only the image completion model. This is a separate learning process rather than goal of this process is to pre-train only the image completion model. This is a separate learning goal of this process is to pre-train only the image completion model. This is a separate learning an additional training of the model trained in the first step. The following is an image created by the process rather than an additional training of the model trained in the first step. The following is an process rather than an additional training of the model trained in the first step. The following is an pre-trained image completion model (Figure 13). The high mask ratio on the test image reduces the image created by the pre-trained image completion model (Figure 13). The high mask ratio on the image created by the pre-trained image completion model (Figure 13). The high mask ratio on the accuracy of the image completion. test image reduces the accuracy of the image completion. test image reduces the accuracy of the image completion. Figure 13. Image generated using the pre-trained image completion model. Figure 13. Image generated using the pre-trained image completion model. Figure 13. Image generated using the pre-trained image completion model. The third step is to train images by using the two pre-trained models, the edge model and the The third step is to train images by using the two pre-trained models, the edge model and the The third step is to train images by using the two pre-trained models, the edge model and the image completion model. First, we used the pre-trained edge model to make edge images. Then we image completion model. First, we used the pre-trained edge model to make edge images. Then we image completion model. First, we used the pre-trained edge model to make edge images. Then we used the generated edge images, the masks from Figure 11c, and masked images from Figure 11d as used the generated edge images, the masks from Figure 11c, and masked images from Figure 11d as used the generated edge images, the masks from Figure 11c, and masked images from Figure 11d as the input. The output images are restored images, and the target images are depicted in Figure 11a. We the input. The output images are restored images, and the target images are depicted in Figure 11a. the input. The output images are restored images, and the target images are depicted in Figure 11a. used the weight file of the pre-trained image completion model to fine-tune the edge connect model. We used the weight file of the pre-trained image completion model to fine-tune the edge connect We used the weight file of the pre-trained image completion model to fine-tune the edge connect Figure 14 is an image generated after the last step of training. We can now see the reason why the model. Figure 14 is an image generated after the last step of training. We can now see the reason why model. Figure 14 is an image generated after the last step of training. We can now see the reason why edge images are made. Figure 14 shows that restoring images by simultaneously using both, the edge the edge images are made. Figure 14 shows that restoring images by simultaneously using both, the the edge images are made. Figure 14 shows that restoring images by simultaneously using both, the generative model and the image completion model delivers better results. edge generative model and the image completion model delivers better results. edge generative model and the image completion model delivers better results. Appl. Sci. 2020, 10, 490 9 of 17 Appl. Sci. 2020, 10, 490 9 of 18 Appl. Sci. 2020, 10, 490 9 of 17 Figure 14. Image generated after the last training step. Figure 14. Image generated after the last training step. Figure 14. Image generated after the last training step. 3. Results and Discussion 3. Results and Discussion 3. Results and Discussion 3.1. Result of Image Completion 3.1. Result of Image Completion 3.1. Result of Image Completion We applied the trained GAN using the proposed method to the actual tooth image completion. We applied the trained GAN using the proposed method to the actual tooth image completion. We applied the trained GAN using the proposed method to the actual tooth image completion. Overall, it took 4 days for training: Steps 1 and step 2 required 1 day each, and step 3 took 2 days. We Overall, it took 4 days for training: Steps 1 and step 2 required 1 day each, and step 3 took 2 days. We Overall, it took 4 days for training: Steps 1 and step 2 required 1 day each, and step 3 took 2 days. We used a personal computer with an i7-6700 (3.4 GHz) processor, 24 GB of RAM, and an RTX 2080 Ti used a personal computer with an i7-6700 (3.4 GHz) processor, 24 GB of RAM, and an RTX 2080 Ti used a personal computer with an i7-6700 (3.4 GHz) processor, 24 GB of RAM, and an RTX 2080 Ti GPU. The figures below show some examples of the results (Figure 15). GPU. The figures below show some examples of the results (Figure 15). GPU. The figures below show some examples of the results (Figure 15). Figure 15. Cont. Appl. Sci. 2020, 10, 490 10 of 18 Appl. Sci. 2020, 10, 490 10 of 17 Figure 15. Some examples of generated images. Figure 15. Some examples of generated images. We used the structural similarity index (SSIM) [26] and peak signal-to-noise ratio (PSNR) [27] to We used the structural similarity index (SSIM) [26] and peak signal-to-noise ratio (PSNR) [27] to quantitatively measure the training result of a GAN. The SSIM is a measure of the di erence between quantitatively measure the training result of a GAN. The SSIM is a measure of the difference between the ground truth image and the image being compared. The value closer to 1 indicates better results. the ground truth image and the image being compared. The value closer to 1 indicates better results. The PSNR is a measure of the sharpness and uses the di erence in noise between the ground truth The PSNR is a measure of the sharpness and uses the difference in noise between the ground truth image and the restored image. The table below presents the accuracy measured by the SSIM (Table 1), image and the restored image. The table below presents the accuracy measured by the SSIM (Table which we classified according to the mask ratios to assess the e ect on the image completion. The 1), which we classified according to the mask ratios to assess the effect on the image completion. The closer the value is to 1, the better the result. closer the value is to 1, the better the result. Table 1. Evaluation using structural similarity index (SSIM). Table 1. Evaluation using structural similarity index (SSIM). Mask Incisor Canine Premolar Molar Mask. Incisor Canine Premolar Molar 0–10% 0.921 0.918 0.911 0.923 0–10% 0.921 0.918 0.911 0.923 10–20% 0. 10–20% 915 0.915 0.0.911911 0.9060.906 0.913 0.913 20–30% 0.885 0.883 0.879 0.894 20–30% 0.885 0.883 0.879 0.894 30–40% 0.819 0.822 0.839 0.859 30–40% 0.819 0.822 0.839 0.859 For a more intuitive explanation of the SSIM, images with values of 0.75–0.95 among those used in the experiments are shown at 0.05 intervals (Figure 16). The left side shows the ground truth images, and the right side shows the restored images. From a value of 0.85 and above, we can see the edge lined up, and a value above 0.9 is similar to that of the ground truth images. Appl. Sci. 2020, 10, 490 11 of 17 For a more intuitive explanation of the SSIM, images with values of 0.75–0.95 among those used in the experiments are shown at 0.05 intervals (Figure 16). The left side shows the ground truth Appl. imagSci. es,2020 and , 10 the ri , 490 ght side shows the restored images. From a value of 0.85 and above, we can see the 11 of 18 edge lined up, and a value above 0.9 is similar to that of the ground truth images. Figure 16. Images of various structural similarity index (SSIMs). Figure 16. Images of various structural similarity index (SSIMs). The following is the result measured based on the PSNR (Table 2). Higher values indicate better The following is the result measured based on the PSNR (Table 2). Higher values indicate better results. The images used in the experiment are 16.42 at minimum and 28.27 at maximum. results. The images used in the experiment are 16.42 at minimum and 28.27 at maximum. As with the SSIM, among the images used in the experiment, images with a PSNR of 16–28 are shown at three intervals (Figure 17). The edges on the left are the ground truth images and the images on the right are the generated images. When the PSNR is less than 20, we can see that the edge is broken or missing. A small amount of noise occurs from above 22, although this is similar to the ground truth images. Appl. Sci. 2020, 10, 490 12 of 17 Table 2. Evaluation based on PSNR. Mask Incisor Canine Premolar Molar 0–10% 26.68 24.42 24.71 26.27 Appl. Sci.1 20200–2,010% , 2 490 5.82 24.19 23.69 25.07 12 of 18 20–30% 22.72 22.57 21.62 23.33 30–40% 19.49 19.78 19.71 21.93 Table 2. Evaluation based on PSNR. As with the SSIM, among the images used in the experiment, images with a PSNR of 16–28 are Mask Incisor Canine Premolar Molar shown at three intervals (Figure 17). The edges on the left are the ground truth images and the images 0–10% 26.68 24.42 24.71 26.27 on the right are the generated images. When the PSNR is less than 20, we can see that the edge is 10–20% 25.82 24.19 23.69 25.07 20–30% 22.72 22.57 21.62 23.33 broken or missing. A small amount of noise occurs from above 22, although this is similar to the 30–40% 19.49 19.78 19.71 21.93 ground truth images. Figure 17. Images of various PSNRs. Figure 17. Images of various PSNRs. Appl. Sci. 2020, 10, 490 13 of 18 Appl. Sci. 2020, 10, 490 13 of 17 Appl. Sci. 2020, 10, 490 13 of 17 The SSIM and PSNR differed significantly according to the mask sizes. Figures 18 and 19 show The SSIM and PSNR di ered significantly according to the mask sizes. Figures 18 and 19 show The SSIM and PSNR differed significantly according to the mask sizes. Figures 18 and 19 show the changes according to the mask sizes. We can see this visually in the figures below (Figure 20). The the changes according to the mask sizes. We can see this visually in the figures below (Figure 20). The the changes according to the mask sizes. We can see this visually in the figures below (Figure 20). The smaller the mask size is the better. However, there is a minimum mask size of an image because the smaller the mask size is the better. However, there is a minimum mask size of an image because the smaller the mask size is the better. However, there is a minimum mask size of an image because the mask must cover the occlusion area. Therefore, the use of large and high-resolution images lead to mask must cover the occlusion area. Therefore, the use of large and high-resolution images lead to mask must cover the occlusion area. Therefore, the use of large and high-resolution images lead to better results. However, computer performance was limited, and we used cropped images. better results. However, computer performance was limited, and we used cropped images. better results. However, computer performance was limited, and we used cropped images. Figure 18. Effect of mask sizes on the SSIM. Figure 18. E ect of mask sizes on the SSIM. Figure 18. Effect of mask sizes on the SSIM. Figure 19. E ect of mask sizes on the PSNR. Figure 19. Effect of mask sizes on the PSNR. Figure 19. Effect of mask sizes on the PSNR. Appl. Sci. 2020, 10, 490 14 of 18 Appl. Sci. 2020, 10, 490 14 of 17 Figure 20. Results of di erent mask sizes. Figure 20. Results of different mask sizes. In this study, we used images with an SSIM of over 0.9 and a PSNR of over 22 for 3D reconstruction. In this study, we used images with an SSIM of over 0.9 and a PSNR of over 22 for 3D Overall, when the mask size is less than 10%, 95% of the images obtained are available for 3D reconstruction. Overall, when the mask size is less than 10%, 95% of the images obtained are available reconstruction. It took 0.1 s for an image completion, and we used 100 slices of both the maxilla for 3D reconstruction. It took 0.1 s for an image completion, and we used 100 slices of both the maxilla and mandible. We conducted the image completion five times per slice. Therefore, the entire image and mandible. We conducted the image completion five times per slice. Therefore, the entire image completion process took approximately 100 s. completion process took approximately 100 s. 3.2. Results of Tooth Segmentation 3.2. Results of Tooth Segmentation 3.2.1. Tooth Model 3.2.1. Tooth Model A tooth pin model of a malocclusion patient was fabricated for the testing of the segmentation algorithm (Figure 21). This model allows each tooth to be removed and replaced to its original position A tooth pin model of a malocclusion patient was fabricated for the testing of the segmentation from the tooth model base. Therefore, it is suitable for use in our experiments because each tooth can algorithm (Figure 21). This model allows each tooth to be removed and replaced to its original be removed and scanned individually with no occlusions. The scan data of each tooth were used as position from the tooth model base. Therefore, it is suitable for use in our experiments because each the ground truth. tooth can be removed and scanned individually with no occlusions. The scan data of each tooth were used as the ground truth. 3.2.2. Accuracy Measurement Process Two data are needed for an accurate measurement. One is the entire scan data and the other is separated scan data. The former has occlusion areas, whereas, the latter does not. Therefore, we used the separated scan data as the ground truth. Next, we conducted segmentation on the entire scan data using the conventional and proposed methods (Figure 22). Then, we measured the accuracy of the separated scan data as the ground truth and compared it with the tooth data segmented by each method. Appl. Sci. 2020, 10, 490 15 of 18 Appl. Sci. 2020, 10, 490 15 of 17 Appl. Sci. 2020, 10, 490 15 of 17 Figure 21. Tooth pin model. Figure 21. Tooth pin model. 3.2.2. Accuracy Measurement Process Two data are needed for an accurate measurement. One is the entire scan data and the other is separated scan data. The former has occlusion areas, whereas, the latter does not. Therefore, we used the separated scan data as the ground truth. Next, we conducted segmentation on the entire scan data using the conventional and proposed methods (Figure 22). Then, we measured the accuracy of the separated scan data as the ground truth and compared it with the tooth data segmented by Figure 21. Tooth pin model. each method. (a) (b) Figure 22. (a) Data obtained through segmentation obtained using the proposed and (b) conventional methods. 3.2.3. Measurement Accuracy Results Jung et al. [28] proposed an effective registration technique, which we used for the registration of the three data types. After the registration, we used Geomagic software (2012, Ping Fu, Herbert (a) (b) Edelsbrunner, Morrisville, NC, USA) to compare the distances between the data. We set the ground Figure 22. (a) Data obtained through segmentation obtained using the proposed and (b) conventional Figure 22. (a) Data obtained through segmentation obtained using the proposed and truth data as a mesh and the data segmented by each method as point clouds. The data segmented (b) conventional methods. methods. by each method were then sampled for a constant number of points. We classified the results into four tooth types, namely, incisor, canine, premolar, and molar, and measured the average mean 3.2.3. Measurement Accuracy Results 3.2.3. Measurement Accuracy Results distance between the point cloud and the mesh. The table below presents the results (Table 3). The Jung et al. [28] proposed an e ective registration technique, which we used for the registration conventional method indicates the manual segmentation using the snake. The proposed method Jung et al. [28] proposed an effective registration technique, which we used for the registration of the three data types. After the registration, we used Geomagic software (2012, Ping Fu, Herbert indicates segmentation by using GAN. of the three data types. After the registration, we used Geomagic software (2012, Ping Fu, Herbert Edelsbrunner, Morrisville, NC, USA) to compare the distances between the data. We set the ground Edelsbrunner, Morrisville, NC, USA) to compare the distances between the data. We set the ground truth data as a mesh and the data segmented by each method as point clouds. The data segmented Table 3. Average mean distance between the point cloud and mesh. truth data as a mesh and the data segmented by each method as point clouds. The data segmented by each method were then sampled for a constant number of points. We classified the results into by each method were then sampled for a constant number of points. We classified the results into four tooth types, Method namely , incisor, canine, premolar Mean d , and istmolar ance (m , and m) measured the average p mean four tooth types, namely, incisor, canine, premolar, and molar, and measured the average mean Conventional 0.031 ± 0.008 0.033 distance between the point cloud and the mesh. The table below presents the results (Table 3). The Proposed 0.027 ± 0.007 conventional method indicates the manual segmentation using the snake. The proposed method p, p-value for independent t-test. indicates segmentation by using GAN. Table 3. Average mean distance between the point cloud and mesh. Method Mean distance (mm) p Conventional 0.031 ± 0.008 0.033 Proposed 0.027 ± 0.007 p, p-value for independent t-test. Appl. Sci. 2020, 10, 490 16 of 18 distance between the point cloud and the mesh. The table below presents the results (Table 3). The conventional method indicates the manual segmentation using the snake. The proposed method indicates segmentation by using GAN. Table 3. Average mean distance between the point cloud and mesh. Method Mean Distance (mm) p Conventional 0.031  0.008 0.033 Proposed 0.027  0.007 p, p-value for independent t-test. The mean distance between the point cloud and the mesh for each method is displayed in Table 3. The above results show that the proposed method is approximately 0.004 mm more accurate than the conventional method, and this di erence was significantly di erent (p = 0.033). 4. Conclusions In this study, we proposed an automated method to reconstruct the missing data in the interdental area of teeth, scanned using an intraoral scanner by applying a GAN. A limitation is that the masks were manually applied to the occlusion area, therefore, some of the normal scan data may have been removed. A further study carried out to automatically detect the occlusion area would define the masks with higher precision and increase the accuracy of segmentation. Nonetheless, based on our results on the dimensional accuracy of the segmented teeth, it can be inferred that the segmentation method for full arch intraoral scan data is as accurate as a manual segmentation method, in addition to being e ective and time saving. This automatic segmentation tool can be used to facilitate the digital setup process in orthodontic treatment. Author Contributions: Conceptualization, T.K.; data curation, D.K.; formal analysis, T.K.; investigation, M.C.; Methodology, T.K. and Y.C.; project administration, M.C.; resources, Y.-J.K.; supervision, Y.-J.K. and M.C.; validation, T.K.; visualization, T.K.; writing—original draft, T.K.; writing—review and editing, Y.-J.K. All authors have read and agreed to the published version of the manuscript. Funding: This research received no external funding. Acknowledgments: We are grateful to the Department of Orthodontics, Korea University Anam Hospital for providing the patients’ scan data and the tooth pin model. Conflicts of Interest: The authors declare no conflicts of interest. References 1. Kesling, H. The diagnostic setup with consideration of the third dimension. Am. J. Orthod. Dentofac. Orthop. 1956, 42, 740–748. [CrossRef] 2. Barreto, M.S.; Faber, J.; Vogel, C.J.; Araujo, T.M. Reliability of digital orthodontic setups. Angle Orthod. 2016, 86, 255–259. [CrossRef] [PubMed] 3. Im, J.; Cha, J.Y.; Lee, K.J.; Yu, H.S.; Hwang, C.J. Comparison of virtual and manual tooth setups with digital and plaster models in extraction cases. Am. J. Orthod. Dentofac. Orthop. 2014, 145, 434–442. [CrossRef] [PubMed] 4. Chen, B.T.; Lou, W.S.; Chen, C.C.; Lin, H.C. A 3D scanning system based on low-occlusion approach. In Proceedings of the Second International Conference on 3-D Digital Imaging and Modeling (Cat. No. PR00062), Ottawa, ON, Canada, 8 October 1999; IEEE: Piscataway, NJ, USA, 1999; pp. 506–515. 5. Shlafman, S.; Tal, A.; Katz, S. Metamorphosis of polyhedral surfaces using decomposition. In Computer Graphics Forum; Blackwell Publishing, Inc.: Oxford, UK, 2002; pp. 219–228. 6. Mangan, A.P.; Whitaker, R.T. Partitioning 3D surface meshes using watershed segmentation. IEEE Trans. Vis. Comput. Graph. 1999, 5, 308–321. [CrossRef] Appl. Sci. 2020, 10, 490 17 of 18 7. Lee, Y.; Lee, S.; Shamir, A.; Cohen-Or, D.; Seidel, H.P. Mesh scissoring with minima rule and part salience. Comput. Aided Geom. Des. 2005, 22, 444–465. [CrossRef] 8. Zanjani, F.G.; Moin, D.A.; Claessen, F.; Cherici, T.; Parinussa, S.; Pourtaherian, A.; Zinger, S. Mask-MCNet: Instance segmentation in 3D point cloud of intra-oral scans. In International Conference on Medical Image Computing and Computer-Assisted Intervention; Springer: Cham, Switzerland, 2019; pp. 128–136. 9. Podolak, J.; Shilane, P.; Golovinskiy, A.; Rusinkiewicz, S.; Funkhouser, T. A planar-reflective symmetry transform for 3D shapes. ACM Trans. Graph. (TOG) 2006, 25, 549–559. [CrossRef] 10. Lai, Y.K.; Hu, S.M.; Martin, R.R.; Rosin, P.L. Fast mesh segmentation using random walks. In Proceedings of the 2008 ACM Symposium on Solid and Physical Modeling, New York, NY, USA, 2–4 June 2008; ACM: New York, NY, USA, 2008; pp. 183–191. 11. Kronfeld, T.; Brunner, D.; Brunnett, G. Snake-based segmentation of teeth from virtual dental casts. Comput. Aided Des. Appl. 2010, 7, 221–233. [CrossRef] 12. Barnes, C.; Shechtman, E.; Goldman, D.B.; Finkelstein, A. The generalized patchmatch correspondence algorithm. In European Conference on Computer Vision; Springer: Berlin/Heidelberg, Germany, 2010; pp. 29–43. 13. Darabi, S.; Shechtman, E.; Barnes, C.; Goldman, D.B.; Sen, P. Image melding: Combining inconsistent images using patch-based synthesis. ACM Trans. Graph. 2012, 31, 82-1–82-10. [CrossRef] 14. Huang, J.B.; Kang, S.B.; Ahuja, N.; Kopf, J. Image completion using planar structure guidance. ACM Trans. Graph. (TOG) 2014, 33, 129. [CrossRef] 15. Simakov, D.; Caspi, Y.; Shechtman, E.; Irani, M. Summarizing visual data using bidirectional similarity. In Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition, Anchorage, AK, USA, 23–28 June 2008; IEEE: Piscataway, NJ, USA, 2008; pp. 1–8. 16. Köhler, R.; Schuler, C.; Schölkopf, B.; Harmeling, S. Mask-specific inpainting with deep neural networks. In German Conference on Pattern Recognition; Springer: Cham, Switzerland, 2014; pp. 523–534. 17. Ren, J.S.; Xu, L.; Yan, Q.; Sun, W. Shepard convolutional neural networks. In Advances in Neural Information Processing Systems; The MIT Press: Cambridge, MA, USA, 2015; pp. 901–909. 18. Iizuka, S.; Simo-serra, E.; Ishikawa, H. Globally and locally consistent image completion. ACM Trans Graph. (ToG) 2017, 36, 107. [CrossRef] 19. Nazeri, K.; Ng, E.; Joseph, T.; Qureshi, F.; Ebrahimi, M. Edgeconnect: Generative image inpainting with adversarial edge learning. arXiv 2019, arXiv:1901.00212. 20. Besl, P.J.; Mckay, N.D. Method for registration of 3-D shapes. In Sensor Fusion IV: Control Paradigms and Data Structures; International Society for Optics and Photonics: Bellingham, WA, USA, 1992; pp. 586–606. 21. Sajjadi, M.S.; Scholkopf, B.; Hirsch, M. Enhancenet: Single image super-resolution through automated texture synthesis. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 4491–4500. 22. Waleed Gondal, M.; Scholkopf, B.; Hirsch, M. The unreasonable e ectiveness of texture transfer for single image super-resolution. In Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 8–14 September 2018. 23. Isola, P.; Zhu, J.Y.; Zhou, T.; Efros, A.A. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 1125–1134. 24. Zhu, J.Y.; Park, T.; Isola, P.; Efros, A.A. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 2223–2232. 25. Canny, J. A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell. 1986, 6, 679–698. [CrossRef] 26. Wang, Z.; Bovik, A.C.; Sheikh, H.R.; Simoncelli, E.P. Image quality assessment: From error visibility to structural similarity. IEEE Trans. Image Process. 2004, 13, 600–612. [CrossRef] [PubMed] Appl. Sci. 2020, 10, 490 18 of 18 27. De Boer, J.F.; Cense, B.; Park, B.H.; Pierce, M.C.; Tearney, G.J.; Bouma, B.E. Improved signal-to-noise ratio in spectral-domain compared with time-domain optical coherence tomography. Opt. Lett. 2003, 28, 2067–2069. [CrossRef] [PubMed] 28. Jung, S.; Song, S.; Chang, M.; Park, S. Range image registration based on 2D synthetic images. Comput. Aided Des. 2018, 94, 16–27. [CrossRef] © 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

Journal

Applied SciencesMultidisciplinary Digital Publishing Institute

Published: Jan 9, 2020

There are no references for this article.