Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

4-mm-diameter three-dimensional imaging endoscope with steerable camera for minimally invasive surgery (3-D-MARVEL)

4-mm-diameter three-dimensional imaging endoscope with steerable camera for minimally invasive... High-resolution three-dimensional (3-D) imaging (stereo imaging) by endoscopes in minimally invasive surgery, especially in space-constrained applications such as brain surgery, is one of the most desired capabilities. Such capability exists at larger than 4-mm overall diameters. We report the development of a stereo imaging endoscope of 4-mm maximum diameter, called Multiangle, Rear-Viewing Endoscopic Tool (MARVEL) that uses a single-lens system with complementary multibandpass filter (CMBF) technology to achieve 3-D imaging. In addition, the system is endowed with the capability to pan from side-to-side over an angle of Æ25 deg, which is another unique aspect of MARVEL for such a class of endoscopes. The design and construction of a single-lens, CMBF aperture camera with integrated illumination to generate 3-D images, and the actuation mechanism built into it is summarized. © 2017 Society of Photo-Optical Instrumentation Engineers (SPIE) [DOI: 10.1117/1.NPh.4.1 .011008] Keywords: endoscopy; stereo imaging; minimally invasive surgery; three-dimensional imaging; three-dimensional camera. Paper 16040SSR received Jun. 15, 2016; accepted for publication Sep. 14, 2016; published online Oct. 7, 2016. Introduction Enhanced understanding of the spatial relationship of an object around us occurs when there are more depth cues.1 The cues are classified into binocular cues requiring input from both eyes and monocular cues requiring input from just one eye. Stereopsis belongs to the class of binocular cues. The human eyes, typically, are separated by a mean distance of 65 mm causing the visual axes of the eyes to intersect and provide two different viewpoints of an object.2 This puts the object at the center of each retina, where our vision is the most accurate. Other objects out of the plane intersecting the object of interest are image-shifted from the center of the retina causing a horizontal disparity as observed in the retinas. This disparity is perceived as depth.3,4 The goal of the stereo imaging is to present horizontal disparity in addition to pictorial cues to a human observer, who then can perceive an impression of depth from the disparity. Stereo imaging involves two devices working in tandem: a stereo imaging device and a stereo display. Typically, the device is composed of two duplicate cameras to mimic the arrangement of human binocular vision. The device captures left and right perspective images of an object and then feeds them to a stereo display. Then, the display feeds the images to corresponding human eyes. The idea of stereopsis in relation to the resolution of the image produced is pictorially shown in Fig. 1 along with related quantitative relations. The motivation for the development reported here is minimally invasive brain and skull base surgeries [minimally invasive neurosurgery (MIN)], which have higher space restrictions than other types of minimally invasive surgeries. The sensitivity of operating in the head and neck regions resulting from the close proximity of critical anatomy presents challenges and highlights the importance of having depth perception. While high-resolution monocular endoscopes of 4- and 2-mm diameter are routinely used, three-dimensional (3-D) imaging capability within 4-mm diameter has been a challenge. Several different techniques have been studied and some developed to realize 3-D imaging endoscopes that use dual apertures that are manipulated using different methods,5­12 but none to our knowledge meets the dimensional and resolution requirements under consideration here, that is to be within an overall diameter of 4 mm and producing an image resolution better than 1000 × 1000. The past techniques are all larger in overall diameter and some produce lower image resolution. The most well-developed technology uses a dual pinhole in the lens system, whose simultaneous images are split by a lenticular lens array placed in front of the image sensor.13,14 This technology has been used in minimally invasive surgeries under clinical settings.15­18 Three-Dimensional-Multiangle, RearViewing Endoscopic Tool Three main components of Multiangle, Rear-Viewing Endoscopic Tool (MARVEL) are the stereo camera (SC), bending section with actuation mechanism, and an illumination system. Development of these three components is described in the following sections. *Address all correspondence to: Harish M. Manohara, E-mail: Harish. Manohara@jpl.nasa.gov Present address: NuVasive, Inc., 101 Enterprise, Suite 100, Aliso Viejo, California 92656, United States. Present address: Terra Bella-Google, 1600 Amphitheatre Parkway, Mountain View, California 94043, United States. Stereo Camera with Illumination An SC requires two viewpoints like other stereo imaging devices except that the separation of the two viewpoints is restricted 2329-423X/2017/$25.00 © 2017 SPIE 011008-1 Fig. 1 Sketch depicting the principle of stereopsis and related relations to depth resolution. by the diameter of the endoscope. For MIN application identified here, the diameter of the endoscope is set to be 4 mm. This ensures easy entry through the surgical opening that is 20 mm in diameter while sparing sufficient room to allow insertion of the remaining surgical paraphernalia such as the irrigation system and diagnostic and surgical tools. The expected image is of high-definition (HD) quality rendering at least 1000 horizontal lines in the display. The two preferred characteristics of SC, small diameter and HD image generation, are in conflict because the small diameter lowers the image definition. Image definition is determined by the optical resolution of the lens and the detector array, which are in turn dictated by the available cross-sectional area. More specifically, the optical resolution of the lens is proportional to the diameter of the aperture, which is restricted by the overall diameter of the endoscope. In addition, the resolution of the detector array is limited by the pixel density, which is also restricted by the SC diameter. To yield HD images, both the optics as well as the image sensor has to yield HD-grade resolutions. Another important challenge is to achieve sufficient stereopsis to produce 3-D images of acceptable depth perception. The overall diameter of 4 mm does not allow implementation of the traditional two-camera approach because of the insufficient baseline distance between the two cameras when housed inside a 4-mm-diameter barrel, whose wall thickness is at least 0.5 mm with integrated illumination. This leaves barely 3-mm diameter to place two lenses side-by-side. Such arrangement not only decreases baseline distance between the lenses to provide sufficient base line distance, but also reduces the image quality. 2.1.1 Dual-aperture scheme stereo imaging and single-lens, dual-aperture stereo imaging is shown schematically in Fig. 2. Two-camera, traditional system is simpler to implement. However, stereoscopes based on the dual-aperture technique have several advantages over dual-lens stereoscopes. First, the left and right perspective images of the dual aperture have identical features. This is because the use of the single lens makes channels' focal lengths, apertures, magnifications, image planes, and distortions the same, whereas the channels of a dual-lens system are often different from each other. Studies show that observers can experience headache, fatigue, and strain from watching an unmatched stereo image pair.19 The dual-aperture system potentially can reduce these stresses. Second, the dual aperture yields a natural singular image. The dual-aperture system converges the two optical axes at the focal point thus producing perfectly overlapped images from the two channels. A dual-lens system, on the contrary, requires additional mechanisms to achieve similar results.20 Such mechanisms are extremely challenging to accommodate in spaceconstrained applications, such as an endoscope. Finally, the dual-aperture system can yield better image resolution than that possible by a dual-lens system because the former does not split the focal plane for imaging as the latter does. The image resolution is directly related to the depth resolution.21,22 Another important disadvantage of dual-lens systems in spaceconstrained applications is the maximum achievable aperture dimensions. A dual lens can maximize the size of the apertures by shaping them into circles, whereas, a single-lens, dual-aperture system can do this by shaping apertures into half-moons. For example, given a 3-mm-diameter SC, the circular aperture in a dual lens can have a maximum diameter of 1.5- or 1.77-mm2 aperture area. The same for the half-moon case of a single-lens, dual-aperture system will be 3.53 mm2 , almost twice the area of the osculating circles. In this work, we have selected single-lens, dual-aperture approach for stereo image generation. The view points are distinguished by employing complementary multibandpass filters (CMBFs). (a) In cases where there are space constraints, such as in an endoscope, an attractive alternative to traditional dual-lens stereo imaging is a single-lens, dual-aperture system. This technique places two apertures at the inlet of a single-lens system. The apertures are offset from the optical axis by a designed separation. This technique enables a single-lens system to capture two perspective images just like in a dual-lens system, and the dual apertures help project the two perspective images separately on the same detector array. Stereo images can be generated from these projected images by time multiplexing the projection process from each of the apertures so that the image pairs are distinguished. The difference between dual lens or two-camera (b) Fig. 2 Comparison between size of the lens and the corresponding image of dual-aperture scheme and dual lens scheme, given the same overall diameter cross section area: (a) dual lens scheme and (b) dual-aperture scheme. 011008-2 Complementary multibandpass filters Here, the stereo imaging technique developed uses CMBFs that are coated over dual apertures. The operating principle can be understood from Figs. 3 and 4. Each CMBF can sample red, green, and blue (RGB) bands to yield a color image. The two filters are designed so that they are complementary in their transmission characteristics to specific bands of RGB wavelengths. This is achieved through interference thin-film technology, which can create coatings whose transmission bands look like "top hats" such that transmission bands of one filter is complementary to that of the second. Figure 5 shows filter transmission band characteristics of the two filters used in this work. As a result, a light source matching one filter does not go through the other filter, and thus the cross talk can be made minimal to none. Additionally, since matching light bands switch viewpoints, the imaging time can be as short as microseconds. To implement this technique, it is necessary to devise an illumination system that can cycle different RGB bands that match passbands of the stereo imaging filters. In this work, a dedicated illumination system was developed in which, a commercial digital mirror array was adapted to steer white light through wavelength selecting CMBFs at its output (of same transmission characteristics as in Fig. 5). That is, the white light is split into wavelength bands that exactly matches that of the CMBF passbands of the dual-aperture plate in the SC. Through sequential cycling of the illumination over all wavelengths the two CMBFs Fig. 4 Dual aperture created by a pair of CMBFs: (a) schematic drawing showing the CMBF pair opening to SIBs and (b) ray traces of the CMBF lens system simulated by an optical simulation software package. (a) (b) Fig. 5 CMBF transmission bands and SIBs. Fig. 3 Spectral schematic shows the operating principle of CMBFbased stereo imaging technique. (a) Shows transmission characteristic of the CMBF filters and (b) shows time-multiplexed SIBs synchronized to the transmission bands of the CMBF filters. are sensitive to, one can generate two sets of color images at the focal plane that are time multiplexed and combined together in a stereo display to produce stereo images. The CMBF technique developed in this work overcomes most of the short comings of stereo imaging techniques, and we found this to be the best approach from simplicity of operation, implementation in space-constrained applications, and ability to produce real-time stereo images with true color capability. Current multiple bandpass interference filters can produce waveforms that are very close to the sharp on and cutoff characteristics of the ideal waveforms. This property allows us to spectrally "interleave" the passbands of each filter with negligible cross talk (spectral overlap) between the complementary filters' passbands. Using a monochromatic camera, the technique generates a total of six time-multiplexed, multispectral images, with each viewpoint generating three spectral images (RGB images) for color rendition. Typically, a 3-D imaging system arranged with two cameras has a depth resolution, dZc , defined by the following equation: 23 011008-3 Fig. 6 (a) Ray traces of the light passing through the lens elements. The blue and green rays represent the light path through one or the other apertures, providing left and right viewpoints. (b) The location of a dual-aperture plate. Notice that the light incident on the plate is near orthogonal. EQ-TARGET;temp:intralink-;e001;63;548 dZc ¼ Z2 T ; Mb (1) determines the resolution of the FPA, which can be calculated as EQ-TARGET;temp:intralink-;e002;326;526 where z is the working distance, T is the angular resolution (stereo acuity), M is the magnification, and b is the baseline (a distance between the two viewpoints of a 3-D camera). This equation applies to any stereo imaging system providing two viewpoints for depth interpretation. Therefore, the equation also applies to the stereo technique presented here. The baseline, b, is achieved by the interpupillary distance created by the bipartite filter. It is well known that an object viewed from two different angles produces two perspective images, left and right. When these two perspective images are seen exclusively by the corresponding human eyes (i.e., left perspective image seen only by the left eye and the right perspective image seen only by the right eye), the depth perception is reproduced. The baseline of the bipartite filter gives the two angular views of the same object, and the CMBF filters incorporated in the bipartite filter provide mutually exclusive viewpoints when the corresponding spectral illumination band (SIB) illuminates the object. Hence, it can be argued that the condition for stereo imaging shown in Eq. (1) is satisfied in the technique presented here. 2.1.3 Lens system 1000 mmð3 m pixel size × 2Þ ¼ 167 line pairsmm: (2) The lens was modeled using Zemax software package. The model involved multiple lens elements to produce the following outcome: (i) achieve pupil diameter as large as possible because it determines the upper bound separation of the two viewpoints, which determines the degree of stereopsis effectiveness. (ii) Achieve a field of view of 80 deg, the largest possible in this design because of the inherent limitation from the ring light material, which cannot disperse light greater than 80 deg cone. The ring light itself is made up of optical fibers having numerical aperture of 0.87. When the tip of the fiber is flat, the light will be dispersed from the tip with an angle no more than 80 deg. This means, the object outside this angle will be dark and cannot be imaged by the lens. (iii) While the optical resolution is capable of reaching HD quality, in this custom design, it was optimized to match that of the commercial-offthe-shelf (COTS) image sensor, the Misumi's focal plane array (FPA; MO-B1003). The Misumi image sensor was adopted as a part of the imaging system. This FPA has the pixel size of 3-m spanning over a 1.2 × 1.2-mm area. The pixel width The overall resolution of the imaging system is limited by the smallest resolving element in the system, which in this case turned out to be the COTS image sensor. (iv) Accommodate inclusion of a dual-aperture plate in the lens train [see Fig. 6(b)] and ensure entry of light orthogonal to the pupil to the dual aperture. The latter is because the apertures are coated with interference filters that require incident angle of light rays to be no more than 15 deg to the orthogonal. Otherwise, the transmission bands can become distorted. Figure 7 shows schematically different bipartite filter apertures tested: (a) an osculating dual aperture, (b) a dual aperture of 0.6-mm diameter, separated by 1.2 mm, and (c) a dual aperture with half-moon apertures separated by 0.32 mm. The best performance in terms of achieving higher image brightness and higher stereopsis was seen by half-moon design. This was selected over the other two to realize MARVEL SC. The SCs were fabricated and integrated with ring lights. The form-factor was comparable to the traditional endoscopes as shown comparatively in Fig. 8. The lenses were commercially fabricated and assembled inside a steel lens barrel. The lenses themselves were of 2.98-mm diameter. The COTS image sensor was attached to the lens system at its focal plan using standard epoxy. The lens barrel is then slid into an optical fiber-based ring light of 0.5-mm thickness and laser welded in place completing the CMBF-based SC of 4-mm diameter. The resolution of the SC was tested using standard Air Force (AF) Target method. Figure 9 shows the image of an AF Target Fig. 7 Aperture diameter and separation of bipartite filter options tested: (a) 0.61.2-mm, (b) osculating, and (c) half-moons. 011008-4 Fig. 8 Picture of the custom 3-D endoscope (left) next to a commercial 2-D endoscope to show the dimensional comparability. Notice that in the left endoscope half moon CMBF filters can be seen. Fig. 9 Image of AF target taken by lens 4, the half-moons. captured using the SC developed here that uses half-moon apertures. We can see that the SC can resolve Group 4-Element 1 in the AF target. This group corresponds to 14.3 lpmm. The correct interpretation of resolution requires the resolution number to be multiplied by the lens system magnification. So, the final lpmm is calculated to be EQ-TARGET;temp:intralink-;e003;63;246 14.3 lpmm × 10.118ðmagnification of the lensÞ ¼ 121 lpmm: (3) This agrees well with the lens design model's resolution and approaches the targeted HD resolution from the optics. The development of CMBF-based SC along with stereopsis study and optics designs is presented in literature elsewhere.24­27 Bending Section with Actuation Mechanism The capability described in this section further makes MARVEL a unique endoscope. To provide a panning capability to the SC, it is connected to a bending section followed by a hollow steel shaft. The bending section is actuated through a cable-pulley system using a servo motor mounted inside a control housing at the proximal end. The hollow shaft extension serves as the housing for the optical fibers, data, and power cables of the image sensor as well as the steel cable used for translating the actuation force from the motor to the bending section. The development started by modeling the MARVEL. The model included assembly plans for components and the motorization mechanism for sweeping. An effort was put into making the MARVEL compact and standalone by adopting miniature components. Both precision machining and rapid prototyping were used to construct different parts of MARVEL as appropriate. The ultimate goal was to enable the camera's viewpoint to sweep over an arc of Æ60 deg. The design plan included: (1) the placement of the camera at the end of a stem like other commercial endoscopes while keeping the diameter 4 mm, (2) the implementation of a servo motor to steer the viewpoint of the camera, and (3) the integration of a translation stage to move the FPA for focusing. To minimize the cost of the overall system, mostly COTS parts were adopted for this task. A typical 4-mm-diameter flexible endoscope bending section was procured and modified to provide the turning radius needed for MARVEL (see Fig. 10). These bending sections are built small and rigid, and created to be used reliably. The bending section is rotated by pushing and pulling the steel cables attached at diametrically opposite points on the inside of the rim. Typically, a bending section is composed of many links of the same kind, each making a Æ 10- deg turn. By joining multiple links, a larger angle of turning can be achieved. For example, a 180- deg turn is shown in Fig. 10 that uses 15 links. Similar process could be followed to achieve Æ60 deg turning for the MARVEL bending section by joining 6 links, which also would reduce the overall bending radius. The latter helps in the space-constrained application that MARVEL is being developed for. To motorize the bending section actuation, a COTS servomotor (HiTEC HS-7940TH), a COTS servo controller (HiTEC HPP-21Plus), and a COTS miniature battery (Hobbico LiFe 6.6V 200mAh 1C Rx LiFeSource U Conn Receiver Battery HCAM6402) were implemented. To achieve bending, a dual lever was connected to the shaft of the servo and the cables from the bending section were fastened to each end of the levers. This caused the servo's rotation to pull on the cables in effect resulting in steering motion of the bending section. The bending section (Fig. 10) was attached to the end of a 15-cm metal shaft. Then, the SC was connected to the distal end of the bending section. The shaft was hollow so that the pigtail cable from the FPA and the optical fiber bundle from the ring light pass through the bending section and through the shaft to the electronics and the light source, respectively. The overall integration strategy is shown in the drawing of 3-D-MARVEL in Fig. 11. A limited provision was made to move the Misumi's FPA with respect to the lens assembly for minor focusing adjustments using a single-axis translation stage. The stage was used to push or pull on the FPA precisely over a small range of Æ0.25 mm. Since the stage cannot be connected directly to the FPA, the actuation force for focusing adjustment is communicated through the stiff pigtail data/power cable connected to the FPA. This method is useful for minor and occasional adjustments only. A fully assembled 3-D-MARVEL prototype is shown in Fig. 12. The components, including the platform and the associated blocks, were rapid-prototyped. The prototype was mainly made of plastic. The cover was made of see-through plastic so that the mechanical motion can be watched from the outside. 011008-5 Fig. 10 (a) Typical bending section with steering cables. Pulling the left-hand side of the cable rotates the bending section to left. (b) Close-up image of links used in bending sections employed in MARVEL (Example showed here uses 15 links to achieve 180 deg bend). Fig. 11 Mechanical drawing of the 3-D-MARVEL layout. Fig. 12 Photograph of the 3-D-MARVEL covered with a see-through plastic showing both top and bottom. Results and Conclusions Using the prototype developed above, a successful demonstration of 3-D-MARVEL was done. Figure 13 captures the bending motion resulting from the servomotor. Several 3-D images of test subjects involving circuit board components, insides of a walnut, and folded pictures with high contrast colors were imaged to show different aspects of the stereo image such as the resolution, image definition and brightness, and stereopsis while panning the camera. The demonstration clearly proved the concept of 3-D-MARVEL and is paving the way for further development. The prototype demonstration also helped to identify the areas needing improvement. They are described here. The bending section could not be panned over the entire Æ60 deg arc. The maximum achievable arc was Æ 25 deg. Fig. 13 Photographs showing the SC panning action of the 3-DMARVEL. 011008-6 Fig. 14 (a) Photo of the disjointed welding. (b) Photo of the illumination power lost after the joint welded for the second time due to loss of optical fibers. The main reason for this was the stiff pigtail of the COTS imager, whose thick insulation and high stiffness, especially at the point of attachment to the image sensor, offered sufficient resistance to prevent a larger bending angle. We believe that this problem can be overcome with a custom data/power cable pigtail that can be made more flexible allowing the planned full range of the bending arc. The second problem was ensuring a strong bond between the steel cables used for actuation with the bending section. This is the most sensitive part of 3-D-MARVEL fabrication. The welding process involved here is highly delicate and repeated attempts to reweld can damage the optical fibers in addition to weakening the welded joint itself. Figure 14 shows weakened weld and loss of optical fibers resulting from repeated welding process. Acknowledgments We thank Dr. Pantazis Mouroulis for all the advice on optics design, Dr. John Choi [ex-Jet Propulsion Laboratory (JPL)], and summer students Allen Ream, Eric Fritz, and Andrew Strongrich for their valuable contributions to this development. This work was carried out with funding from the Skull Base Institute of Los Angeles, California. This research was carried out at the JPL, California Institute of Technology, under a contract with the National Aeronautics and Space Administration. Government sponsorship is acknowledged. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Neurophotonics SPIE

4-mm-diameter three-dimensional imaging endoscope with steerable camera for minimally invasive surgery (3-D-MARVEL)

Download PDF
8 pages

Loading next page...
 
/lp/spie/4-mm-diameter-three-dimensional-imaging-endoscope-with-steerable-rXXsSf5EV0

References

References for this paper are not available at this time. We will be adding them shortly, thank you for your patience.

Publisher
SPIE
Copyright
© 2017 Society of Photo-Optical Instrumentation Engineers (SPIE)
Subject
Special Section on Brain Mapping and Therapeutics; Paper
ISSN
2329-423X
eISSN
2329-4248
DOI
10.1117/1.NPh.4.1.011008
Publisher site
See Article on Publisher Site

Abstract

High-resolution three-dimensional (3-D) imaging (stereo imaging) by endoscopes in minimally invasive surgery, especially in space-constrained applications such as brain surgery, is one of the most desired capabilities. Such capability exists at larger than 4-mm overall diameters. We report the development of a stereo imaging endoscope of 4-mm maximum diameter, called Multiangle, Rear-Viewing Endoscopic Tool (MARVEL) that uses a single-lens system with complementary multibandpass filter (CMBF) technology to achieve 3-D imaging. In addition, the system is endowed with the capability to pan from side-to-side over an angle of Æ25 deg, which is another unique aspect of MARVEL for such a class of endoscopes. The design and construction of a single-lens, CMBF aperture camera with integrated illumination to generate 3-D images, and the actuation mechanism built into it is summarized. © 2017 Society of Photo-Optical Instrumentation Engineers (SPIE) [DOI: 10.1117/1.NPh.4.1 .011008] Keywords: endoscopy; stereo imaging; minimally invasive surgery; three-dimensional imaging; three-dimensional camera. Paper 16040SSR received Jun. 15, 2016; accepted for publication Sep. 14, 2016; published online Oct. 7, 2016. Introduction Enhanced understanding of the spatial relationship of an object around us occurs when there are more depth cues.1 The cues are classified into binocular cues requiring input from both eyes and monocular cues requiring input from just one eye. Stereopsis belongs to the class of binocular cues. The human eyes, typically, are separated by a mean distance of 65 mm causing the visual axes of the eyes to intersect and provide two different viewpoints of an object.2 This puts the object at the center of each retina, where our vision is the most accurate. Other objects out of the plane intersecting the object of interest are image-shifted from the center of the retina causing a horizontal disparity as observed in the retinas. This disparity is perceived as depth.3,4 The goal of the stereo imaging is to present horizontal disparity in addition to pictorial cues to a human observer, who then can perceive an impression of depth from the disparity. Stereo imaging involves two devices working in tandem: a stereo imaging device and a stereo display. Typically, the device is composed of two duplicate cameras to mimic the arrangement of human binocular vision. The device captures left and right perspective images of an object and then feeds them to a stereo display. Then, the display feeds the images to corresponding human eyes. The idea of stereopsis in relation to the resolution of the image produced is pictorially shown in Fig. 1 along with related quantitative relations. The motivation for the development reported here is minimally invasive brain and skull base surgeries [minimally invasive neurosurgery (MIN)], which have higher space restrictions than other types of minimally invasive surgeries. The sensitivity of operating in the head and neck regions resulting from the close proximity of critical anatomy presents challenges and highlights the importance of having depth perception. While high-resolution monocular endoscopes of 4- and 2-mm diameter are routinely used, three-dimensional (3-D) imaging capability within 4-mm diameter has been a challenge. Several different techniques have been studied and some developed to realize 3-D imaging endoscopes that use dual apertures that are manipulated using different methods,5­12 but none to our knowledge meets the dimensional and resolution requirements under consideration here, that is to be within an overall diameter of 4 mm and producing an image resolution better than 1000 × 1000. The past techniques are all larger in overall diameter and some produce lower image resolution. The most well-developed technology uses a dual pinhole in the lens system, whose simultaneous images are split by a lenticular lens array placed in front of the image sensor.13,14 This technology has been used in minimally invasive surgeries under clinical settings.15­18 Three-Dimensional-Multiangle, RearViewing Endoscopic Tool Three main components of Multiangle, Rear-Viewing Endoscopic Tool (MARVEL) are the stereo camera (SC), bending section with actuation mechanism, and an illumination system. Development of these three components is described in the following sections. *Address all correspondence to: Harish M. Manohara, E-mail: Harish. Manohara@jpl.nasa.gov Present address: NuVasive, Inc., 101 Enterprise, Suite 100, Aliso Viejo, California 92656, United States. Present address: Terra Bella-Google, 1600 Amphitheatre Parkway, Mountain View, California 94043, United States. Stereo Camera with Illumination An SC requires two viewpoints like other stereo imaging devices except that the separation of the two viewpoints is restricted 2329-423X/2017/$25.00 © 2017 SPIE 011008-1 Fig. 1 Sketch depicting the principle of stereopsis and related relations to depth resolution. by the diameter of the endoscope. For MIN application identified here, the diameter of the endoscope is set to be 4 mm. This ensures easy entry through the surgical opening that is 20 mm in diameter while sparing sufficient room to allow insertion of the remaining surgical paraphernalia such as the irrigation system and diagnostic and surgical tools. The expected image is of high-definition (HD) quality rendering at least 1000 horizontal lines in the display. The two preferred characteristics of SC, small diameter and HD image generation, are in conflict because the small diameter lowers the image definition. Image definition is determined by the optical resolution of the lens and the detector array, which are in turn dictated by the available cross-sectional area. More specifically, the optical resolution of the lens is proportional to the diameter of the aperture, which is restricted by the overall diameter of the endoscope. In addition, the resolution of the detector array is limited by the pixel density, which is also restricted by the SC diameter. To yield HD images, both the optics as well as the image sensor has to yield HD-grade resolutions. Another important challenge is to achieve sufficient stereopsis to produce 3-D images of acceptable depth perception. The overall diameter of 4 mm does not allow implementation of the traditional two-camera approach because of the insufficient baseline distance between the two cameras when housed inside a 4-mm-diameter barrel, whose wall thickness is at least 0.5 mm with integrated illumination. This leaves barely 3-mm diameter to place two lenses side-by-side. Such arrangement not only decreases baseline distance between the lenses to provide sufficient base line distance, but also reduces the image quality. 2.1.1 Dual-aperture scheme stereo imaging and single-lens, dual-aperture stereo imaging is shown schematically in Fig. 2. Two-camera, traditional system is simpler to implement. However, stereoscopes based on the dual-aperture technique have several advantages over dual-lens stereoscopes. First, the left and right perspective images of the dual aperture have identical features. This is because the use of the single lens makes channels' focal lengths, apertures, magnifications, image planes, and distortions the same, whereas the channels of a dual-lens system are often different from each other. Studies show that observers can experience headache, fatigue, and strain from watching an unmatched stereo image pair.19 The dual-aperture system potentially can reduce these stresses. Second, the dual aperture yields a natural singular image. The dual-aperture system converges the two optical axes at the focal point thus producing perfectly overlapped images from the two channels. A dual-lens system, on the contrary, requires additional mechanisms to achieve similar results.20 Such mechanisms are extremely challenging to accommodate in spaceconstrained applications, such as an endoscope. Finally, the dual-aperture system can yield better image resolution than that possible by a dual-lens system because the former does not split the focal plane for imaging as the latter does. The image resolution is directly related to the depth resolution.21,22 Another important disadvantage of dual-lens systems in spaceconstrained applications is the maximum achievable aperture dimensions. A dual lens can maximize the size of the apertures by shaping them into circles, whereas, a single-lens, dual-aperture system can do this by shaping apertures into half-moons. For example, given a 3-mm-diameter SC, the circular aperture in a dual lens can have a maximum diameter of 1.5- or 1.77-mm2 aperture area. The same for the half-moon case of a single-lens, dual-aperture system will be 3.53 mm2 , almost twice the area of the osculating circles. In this work, we have selected single-lens, dual-aperture approach for stereo image generation. The view points are distinguished by employing complementary multibandpass filters (CMBFs). (a) In cases where there are space constraints, such as in an endoscope, an attractive alternative to traditional dual-lens stereo imaging is a single-lens, dual-aperture system. This technique places two apertures at the inlet of a single-lens system. The apertures are offset from the optical axis by a designed separation. This technique enables a single-lens system to capture two perspective images just like in a dual-lens system, and the dual apertures help project the two perspective images separately on the same detector array. Stereo images can be generated from these projected images by time multiplexing the projection process from each of the apertures so that the image pairs are distinguished. The difference between dual lens or two-camera (b) Fig. 2 Comparison between size of the lens and the corresponding image of dual-aperture scheme and dual lens scheme, given the same overall diameter cross section area: (a) dual lens scheme and (b) dual-aperture scheme. 011008-2 Complementary multibandpass filters Here, the stereo imaging technique developed uses CMBFs that are coated over dual apertures. The operating principle can be understood from Figs. 3 and 4. Each CMBF can sample red, green, and blue (RGB) bands to yield a color image. The two filters are designed so that they are complementary in their transmission characteristics to specific bands of RGB wavelengths. This is achieved through interference thin-film technology, which can create coatings whose transmission bands look like "top hats" such that transmission bands of one filter is complementary to that of the second. Figure 5 shows filter transmission band characteristics of the two filters used in this work. As a result, a light source matching one filter does not go through the other filter, and thus the cross talk can be made minimal to none. Additionally, since matching light bands switch viewpoints, the imaging time can be as short as microseconds. To implement this technique, it is necessary to devise an illumination system that can cycle different RGB bands that match passbands of the stereo imaging filters. In this work, a dedicated illumination system was developed in which, a commercial digital mirror array was adapted to steer white light through wavelength selecting CMBFs at its output (of same transmission characteristics as in Fig. 5). That is, the white light is split into wavelength bands that exactly matches that of the CMBF passbands of the dual-aperture plate in the SC. Through sequential cycling of the illumination over all wavelengths the two CMBFs Fig. 4 Dual aperture created by a pair of CMBFs: (a) schematic drawing showing the CMBF pair opening to SIBs and (b) ray traces of the CMBF lens system simulated by an optical simulation software package. (a) (b) Fig. 5 CMBF transmission bands and SIBs. Fig. 3 Spectral schematic shows the operating principle of CMBFbased stereo imaging technique. (a) Shows transmission characteristic of the CMBF filters and (b) shows time-multiplexed SIBs synchronized to the transmission bands of the CMBF filters. are sensitive to, one can generate two sets of color images at the focal plane that are time multiplexed and combined together in a stereo display to produce stereo images. The CMBF technique developed in this work overcomes most of the short comings of stereo imaging techniques, and we found this to be the best approach from simplicity of operation, implementation in space-constrained applications, and ability to produce real-time stereo images with true color capability. Current multiple bandpass interference filters can produce waveforms that are very close to the sharp on and cutoff characteristics of the ideal waveforms. This property allows us to spectrally "interleave" the passbands of each filter with negligible cross talk (spectral overlap) between the complementary filters' passbands. Using a monochromatic camera, the technique generates a total of six time-multiplexed, multispectral images, with each viewpoint generating three spectral images (RGB images) for color rendition. Typically, a 3-D imaging system arranged with two cameras has a depth resolution, dZc , defined by the following equation: 23 011008-3 Fig. 6 (a) Ray traces of the light passing through the lens elements. The blue and green rays represent the light path through one or the other apertures, providing left and right viewpoints. (b) The location of a dual-aperture plate. Notice that the light incident on the plate is near orthogonal. EQ-TARGET;temp:intralink-;e001;63;548 dZc ¼ Z2 T ; Mb (1) determines the resolution of the FPA, which can be calculated as EQ-TARGET;temp:intralink-;e002;326;526 where z is the working distance, T is the angular resolution (stereo acuity), M is the magnification, and b is the baseline (a distance between the two viewpoints of a 3-D camera). This equation applies to any stereo imaging system providing two viewpoints for depth interpretation. Therefore, the equation also applies to the stereo technique presented here. The baseline, b, is achieved by the interpupillary distance created by the bipartite filter. It is well known that an object viewed from two different angles produces two perspective images, left and right. When these two perspective images are seen exclusively by the corresponding human eyes (i.e., left perspective image seen only by the left eye and the right perspective image seen only by the right eye), the depth perception is reproduced. The baseline of the bipartite filter gives the two angular views of the same object, and the CMBF filters incorporated in the bipartite filter provide mutually exclusive viewpoints when the corresponding spectral illumination band (SIB) illuminates the object. Hence, it can be argued that the condition for stereo imaging shown in Eq. (1) is satisfied in the technique presented here. 2.1.3 Lens system 1000 mmð3 m pixel size × 2Þ ¼ 167 line pairsmm: (2) The lens was modeled using Zemax software package. The model involved multiple lens elements to produce the following outcome: (i) achieve pupil diameter as large as possible because it determines the upper bound separation of the two viewpoints, which determines the degree of stereopsis effectiveness. (ii) Achieve a field of view of 80 deg, the largest possible in this design because of the inherent limitation from the ring light material, which cannot disperse light greater than 80 deg cone. The ring light itself is made up of optical fibers having numerical aperture of 0.87. When the tip of the fiber is flat, the light will be dispersed from the tip with an angle no more than 80 deg. This means, the object outside this angle will be dark and cannot be imaged by the lens. (iii) While the optical resolution is capable of reaching HD quality, in this custom design, it was optimized to match that of the commercial-offthe-shelf (COTS) image sensor, the Misumi's focal plane array (FPA; MO-B1003). The Misumi image sensor was adopted as a part of the imaging system. This FPA has the pixel size of 3-m spanning over a 1.2 × 1.2-mm area. The pixel width The overall resolution of the imaging system is limited by the smallest resolving element in the system, which in this case turned out to be the COTS image sensor. (iv) Accommodate inclusion of a dual-aperture plate in the lens train [see Fig. 6(b)] and ensure entry of light orthogonal to the pupil to the dual aperture. The latter is because the apertures are coated with interference filters that require incident angle of light rays to be no more than 15 deg to the orthogonal. Otherwise, the transmission bands can become distorted. Figure 7 shows schematically different bipartite filter apertures tested: (a) an osculating dual aperture, (b) a dual aperture of 0.6-mm diameter, separated by 1.2 mm, and (c) a dual aperture with half-moon apertures separated by 0.32 mm. The best performance in terms of achieving higher image brightness and higher stereopsis was seen by half-moon design. This was selected over the other two to realize MARVEL SC. The SCs were fabricated and integrated with ring lights. The form-factor was comparable to the traditional endoscopes as shown comparatively in Fig. 8. The lenses were commercially fabricated and assembled inside a steel lens barrel. The lenses themselves were of 2.98-mm diameter. The COTS image sensor was attached to the lens system at its focal plan using standard epoxy. The lens barrel is then slid into an optical fiber-based ring light of 0.5-mm thickness and laser welded in place completing the CMBF-based SC of 4-mm diameter. The resolution of the SC was tested using standard Air Force (AF) Target method. Figure 9 shows the image of an AF Target Fig. 7 Aperture diameter and separation of bipartite filter options tested: (a) 0.61.2-mm, (b) osculating, and (c) half-moons. 011008-4 Fig. 8 Picture of the custom 3-D endoscope (left) next to a commercial 2-D endoscope to show the dimensional comparability. Notice that in the left endoscope half moon CMBF filters can be seen. Fig. 9 Image of AF target taken by lens 4, the half-moons. captured using the SC developed here that uses half-moon apertures. We can see that the SC can resolve Group 4-Element 1 in the AF target. This group corresponds to 14.3 lpmm. The correct interpretation of resolution requires the resolution number to be multiplied by the lens system magnification. So, the final lpmm is calculated to be EQ-TARGET;temp:intralink-;e003;63;246 14.3 lpmm × 10.118ðmagnification of the lensÞ ¼ 121 lpmm: (3) This agrees well with the lens design model's resolution and approaches the targeted HD resolution from the optics. The development of CMBF-based SC along with stereopsis study and optics designs is presented in literature elsewhere.24­27 Bending Section with Actuation Mechanism The capability described in this section further makes MARVEL a unique endoscope. To provide a panning capability to the SC, it is connected to a bending section followed by a hollow steel shaft. The bending section is actuated through a cable-pulley system using a servo motor mounted inside a control housing at the proximal end. The hollow shaft extension serves as the housing for the optical fibers, data, and power cables of the image sensor as well as the steel cable used for translating the actuation force from the motor to the bending section. The development started by modeling the MARVEL. The model included assembly plans for components and the motorization mechanism for sweeping. An effort was put into making the MARVEL compact and standalone by adopting miniature components. Both precision machining and rapid prototyping were used to construct different parts of MARVEL as appropriate. The ultimate goal was to enable the camera's viewpoint to sweep over an arc of Æ60 deg. The design plan included: (1) the placement of the camera at the end of a stem like other commercial endoscopes while keeping the diameter 4 mm, (2) the implementation of a servo motor to steer the viewpoint of the camera, and (3) the integration of a translation stage to move the FPA for focusing. To minimize the cost of the overall system, mostly COTS parts were adopted for this task. A typical 4-mm-diameter flexible endoscope bending section was procured and modified to provide the turning radius needed for MARVEL (see Fig. 10). These bending sections are built small and rigid, and created to be used reliably. The bending section is rotated by pushing and pulling the steel cables attached at diametrically opposite points on the inside of the rim. Typically, a bending section is composed of many links of the same kind, each making a Æ 10- deg turn. By joining multiple links, a larger angle of turning can be achieved. For example, a 180- deg turn is shown in Fig. 10 that uses 15 links. Similar process could be followed to achieve Æ60 deg turning for the MARVEL bending section by joining 6 links, which also would reduce the overall bending radius. The latter helps in the space-constrained application that MARVEL is being developed for. To motorize the bending section actuation, a COTS servomotor (HiTEC HS-7940TH), a COTS servo controller (HiTEC HPP-21Plus), and a COTS miniature battery (Hobbico LiFe 6.6V 200mAh 1C Rx LiFeSource U Conn Receiver Battery HCAM6402) were implemented. To achieve bending, a dual lever was connected to the shaft of the servo and the cables from the bending section were fastened to each end of the levers. This caused the servo's rotation to pull on the cables in effect resulting in steering motion of the bending section. The bending section (Fig. 10) was attached to the end of a 15-cm metal shaft. Then, the SC was connected to the distal end of the bending section. The shaft was hollow so that the pigtail cable from the FPA and the optical fiber bundle from the ring light pass through the bending section and through the shaft to the electronics and the light source, respectively. The overall integration strategy is shown in the drawing of 3-D-MARVEL in Fig. 11. A limited provision was made to move the Misumi's FPA with respect to the lens assembly for minor focusing adjustments using a single-axis translation stage. The stage was used to push or pull on the FPA precisely over a small range of Æ0.25 mm. Since the stage cannot be connected directly to the FPA, the actuation force for focusing adjustment is communicated through the stiff pigtail data/power cable connected to the FPA. This method is useful for minor and occasional adjustments only. A fully assembled 3-D-MARVEL prototype is shown in Fig. 12. The components, including the platform and the associated blocks, were rapid-prototyped. The prototype was mainly made of plastic. The cover was made of see-through plastic so that the mechanical motion can be watched from the outside. 011008-5 Fig. 10 (a) Typical bending section with steering cables. Pulling the left-hand side of the cable rotates the bending section to left. (b) Close-up image of links used in bending sections employed in MARVEL (Example showed here uses 15 links to achieve 180 deg bend). Fig. 11 Mechanical drawing of the 3-D-MARVEL layout. Fig. 12 Photograph of the 3-D-MARVEL covered with a see-through plastic showing both top and bottom. Results and Conclusions Using the prototype developed above, a successful demonstration of 3-D-MARVEL was done. Figure 13 captures the bending motion resulting from the servomotor. Several 3-D images of test subjects involving circuit board components, insides of a walnut, and folded pictures with high contrast colors were imaged to show different aspects of the stereo image such as the resolution, image definition and brightness, and stereopsis while panning the camera. The demonstration clearly proved the concept of 3-D-MARVEL and is paving the way for further development. The prototype demonstration also helped to identify the areas needing improvement. They are described here. The bending section could not be panned over the entire Æ60 deg arc. The maximum achievable arc was Æ 25 deg. Fig. 13 Photographs showing the SC panning action of the 3-DMARVEL. 011008-6 Fig. 14 (a) Photo of the disjointed welding. (b) Photo of the illumination power lost after the joint welded for the second time due to loss of optical fibers. The main reason for this was the stiff pigtail of the COTS imager, whose thick insulation and high stiffness, especially at the point of attachment to the image sensor, offered sufficient resistance to prevent a larger bending angle. We believe that this problem can be overcome with a custom data/power cable pigtail that can be made more flexible allowing the planned full range of the bending arc. The second problem was ensuring a strong bond between the steel cables used for actuation with the bending section. This is the most sensitive part of 3-D-MARVEL fabrication. The welding process involved here is highly delicate and repeated attempts to reweld can damage the optical fibers in addition to weakening the welded joint itself. Figure 14 shows weakened weld and loss of optical fibers resulting from repeated welding process. Acknowledgments We thank Dr. Pantazis Mouroulis for all the advice on optics design, Dr. John Choi [ex-Jet Propulsion Laboratory (JPL)], and summer students Allen Ream, Eric Fritz, and Andrew Strongrich for their valuable contributions to this development. This work was carried out with funding from the Skull Base Institute of Los Angeles, California. This research was carried out at the JPL, California Institute of Technology, under a contract with the National Aeronautics and Space Administration. Government sponsorship is acknowledged.

Journal

Neurophotonics – SPIE

Published: Jan 1, 2017

There are no references for this article.