Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Single-Shot Light-Field Microscopy: An Emerging Tool for 3D Biomedical Imaging

Single-Shot Light-Field Microscopy: An Emerging Tool for 3D Biomedical Imaging 3D microscopy is a useful tool to visualize the detailed structures and mechanisms of biomedical specimens. In particular, biophysical phenomena such as neural activity require fast 3D volumetric imaging because fluorescence signals degrade quickly. A light-field microscope (LFM) has recently attracted attention as a high-speed volumetric imaging technique by recording 3D information in a single-snapshot. This review highlighted recent progress in LFM techniques for 3D biomedical applications. In detail, various image reconstruction algorithms according to LFM configurations are explained, and several biomedical applications such as neuron activity localization, live-cell imaging, locomotion analysis, and single-molecule visualization are introduced. We also discuss deep learning-based LFMs to enhance image resolution and reduce reconstruc- tion artifacts. Keywords Light-field microscope (LFM) · 3D biomedical imaging · Neuron activity · Deep learning-enhanced LFM 1 Introduction increased shot noise and reduced signal when the frequency of illumination is close to the cutoff frequency of a detector 3D volumetric microscopy technology is evolving to ana- [10–12]. Light-sheet microscopy is a technique of collecting lyze cell behaviors and functions for biomedical applica- scattered light with an objective lens by projecting a light tions. In particular, 3D biological information such as neural sheet onto a sample using an illuminating objective after activity in vivo is a crucial parameter for brain or biologi- aligning the imaging lens and the illuminating lens vertically cal research [1, 2]. Techniques for 3D microscopic imaging [6, 13, 14]. The light-sheet approach has also limitations in can be classified into point-scanning, structured illumina- that the FOV is limited by the depth of field (DOF) and the tion, and light-sheet imaging, which have pros and cons, image quality can be reduced by scattering. 3D tomographic respectively (Fig. 1a–c) [3–6]. First, scanning microscopy microscopy is also an emerging method to investigate bio- captures only light emitted from a focal plane by removing logical cells without fluorescent labeling [15– 17]. The tomo- out-focused fluorescence through a pinhole, and obtains 3D graphic approach uses the reconstruction of the 3D refractive images through the raster scanning of excitation and detec- index (RI) distribution by capturing holograms with various tion [4, 7, 8]. However, the overall processing time of scan- incident angles of illuminations. Tomographic microscopy ning microscopy is relatively slow due to the scanning of can not only acquire the morphology, protein concentration, an entire field-of-view (FOV), and the poor axial resolution and dry mass of a target but also evade photo-bleaching and is an unresolved limitation [9]. The structured illumination photo-toxicity. However, the frame rate of 3D imaging is microscopy (SIM) uses patterned illumination to improve relatively low due to mechanical scanning of illuminations, spatial resolution, whereas the illumination has limitations in and the tomographic system has a limitation in observing electrical signals such as neural activity [18]. Light-field (LF) imaging is a strategy that captures ple- * Kisoo Kim noptic functions representing the intensity and directions of kisookim@kopti.re.kr light rays in a 3D space [19, 20]. In particular, a handheld LF camera has emerged for recording 3D imaging without Intelligent Optical Module Research Center, Korea Photonics Technology Institute (KOPTI), 9, Cheomdan an external light source or array cameras [21–23]. The LF Venture-ro 108beon-gil, Buk-gu, Gwangju 61007, camera usually uses microlens arrays (MLAs) placed in the Republic of Korea 1 3 Vol.:(0123456789) 398 BioChip Journal (2022) 16:397–408 Fig. 1 3D volumetric microscopy technology for biomedical applica- single-shot light-field microscopic imaging for biomedical appli- tions. a A confocal laser scanning microscope using scanning mirrors cations. The biomedical applications include the imaging of neu- to scan a laser across a sample. b Structured illumination microscopy rons, live-cell, locomotion, and single-molecule. Reprinted from ref (SIM) which illuminates a sample with patterned light to improve [42] with permission by Nature Publishing Group, copyright 2021, spatial resolution. c Light-sheet microscopy which observes a speci- Reprinted from ref [40] with permission by The Optical Society, men illuminated perpendicular to an observation lens. d Light-field copyright 2019, Reprinted from ref [43] with permission from Public microscopy with microlens arrays (MLA) to obtain volumetric infor- Library of Science (PLoS), copyright 2018, and Reprinted from ref mation in a single snapshot. e Schematic illustrations and figures of [39] with permission by The Optical Society, copyright 2020 intermediate image plane of optical configuration to col - tomographic microscopy that require mechanical scanning lect spatial and directional information about light [24–26]. or multiple frames, the LFM captures spatial and angular Such a simple optical configuration allows 3D-depth estima - information of an object without delay by collecting par- tion, sub-aperture imaging, and depth refocusing after single tial images through each channel of the MLA in a single exposure capturing [27–30]. Also, the optical configuration frame (Fig. 1d). The 3D volumetric imaging of LFM realizes of the LF camera extends the depth of field by reconstructing a faster frame rate compared to the other 3D microscopy sub-aperture images [31, 32]. because the LFM can acquire 3D imaging through a single- A light-field microscope (LFM) was first designed by snapshot [34–36]. The LFM is available for various biomedi- Levoy et al. and the 3D information of objects was acquired cal applications such as the imaging of neuronal activity [37, by placing a microlens array (MLA) on an image plane [33]. 38], single-molecule [39], and live-cell [40] owing to the Unlike conventional 3D microscopies such as confocal or fast 3D imaging of LFM. However, the limitations of LFM 1 3 BioChip Journal (2022) 16:397–408 399 Table1 A summary of light-field microscopy according to applications Applications LFM configuration Imaging volume Imaging resolution Illumination method Volume References imaging speed (Hz) Neuron imaging LFM with relay lens 700 × 700 × 200 µm Lateral: ~ 1.4 Widefield illumina- 50 [53] µm tion Axial: 2.6 µm LFM with relay lens ~ 200 × 200 × 200 μm Lateral: ~ 2 µm Widefield illumina- 100 [74] Axial: 3 µm tion LFM with two-photon 900 × 900 × 260 mm Lateral:3.5 µm Widefield illumina- 30 [85] scanning micros- Axial: 9 µm tion copy + Laser scanning Fourier LFM 500 × 350 × 400 µm Lateral: ~ 3.5 µm Widefield illumina- 50 [76] Axial: 7.4 µm tion Confocal LFM ø800 × 200 µm Lateral: ~ 2.1 µm Laser scanning illu- 70 [55] Axial: 2.5 µm mination Live-cell imaging Conventional LFM 100 × 100 × 5 µm Lateral: 0.6 µm Widefield illumina- 10 [40] Axial: 0.3 µm tion Fourier LFM 3 × 4 × 1.2 µm Lateral: 0.3 µm Widefield illumina- 10 [83] Axial: 0.5 µm tion Mirror-enhanced 90 × 70 × 70 µm Lateral: 0.4 µm Widefield illumina- 2 [80] scanning LFM Axial: 1.5 µm tion Locomotion analysis LFM with relay lens Over Axial: 11.1 µm Widefield illumina- 20 [43] 676 × 365 × 110 µm tion Conventional LFM 300 × 300 × 50 µm Lateral: 2.6 µm Widefield illumina- 100 [84] Axial: 5 µm tion Single-molecule Conventional LFM 15 × 15 × 6 µm Lateral, Axial: 20 nm Widefield illumina- 1.25 [39] imaging tion are spatial resolution and a low signal-to-noise ratio (SNR) is required to observe biological structures or signals. due to the superimposition of spatial information through Recent studies are being progressed to solve the resolution the MLA [41]. In addition, the issues of photo-bleaching issues through novel LFM configurations and deep learning and photo-toxicity still remain because the labeling process algorithms. Fig. 2 Schematic illustrations of LFM configurations. a A schematic High-resolution LFM configuration with the MLA positioned behind of experimental setup for single-shot 3D microscopic imaging. b the NIP. e The configuration of Fourier LFM arranged in order of Conventional LFM configuration comprising an objective lens, tube tube lens, Fourier lens, and the MLA. f The configuration of LFM lens, MLA, and an image sensor. c Galilean-mode LFM configuration with relay lens. g A confocal LFM configuration with a mask for the with the MLA positioned in front of a native image plane (NIP). d elimination of background noise 1 3 400 BioChip Journal (2022) 16:397–408 In this review, we will introduce microscopes using light- The conventional LFM comprises the MLA positioned on field technology that can acquire 3D volumetric information the native image plane of the microscope system [45, 48], in a single-shot (Fig. 1e). This review aims to mainly discuss like the plenoptic 1.0 system. Each MLA perceives the in- image acquisition methods and various biomedical appli- focus image of an objective lens and divides the images as cations via the LFM. This article starts by introducing the micro-images. A Galilean-scheme LFM is also similar to descriptions of an LFM principle, optical configurations, and the Galilean LF scheme with the MLA in front of the native image processing methods. The subsequent sections cover plane [49]. The Galilean-scheme LFM can reduce the overall biomedical applications such as the imaging of neurons, thickness of microscope system by diminishing the optical live-cell, worm locomotion, and single-molecule through path length, but needs the MLA with a relatively long focal light-field microscopic imaging (Table  1). Also, the review length. The LFM that the MLA positioned behind the native includes recent studies of deep learning-based light-field image plane is called a Keplerian-scheme LFM [40]. The microscopic imaging to enhance image quality. Finally, the Keplerian-scheme LFM with a relatively long optical path outlook and summary conclude the review. has a high-spatial resolution at the cost of angular resolution degradation, whereas the Keplerian LFM has the disadvan- tage of a relatively large optical configuration. A Fourier 2 LFM Principle LFM uses the MLA placed on a Fourier (pupil) plane to acquire uniform resolution between each MLA and increase 2.1 LF Imaging spatial resolution and depth of field [50– 52]. In addition, the Fourier LFM configuration can increase an image vol- A conventional wide-field imaging method captures an ume and perform efficient computational costs by reduc- image from an image plane formed by a single objective ing the overlap of images formed by the MLA. However, lens, which has a limitation in acquiring the depth informa- the Fourier system still has a trade-off between spatial and tion of an object. On the other hand, light-field imaging can angular sampling issues. Various optical parameters such record 3D information in a 2D image sensor by dividing as magnification and focal length can be controlled by addi- spatial and directional data through an objective lens and tionally integrating relay lenses of various magnifications. MLA [44–46]. The initial type of LF imaging device con- Additional optical components such as the relay lens can sists of an objective lens, MLA, and an image sensor, and efficiently improve the optical performance of LFM when the MLA is placed at the image plane of the objective lens applying a commercial MLA with limitations in parameter [20]. This concept is called a plenoptic 1.0 system, and the selection [53, 54]. The confocal LFM technique utilizes a MLA separates the converging ray to acquire the spatial and mask to selectively and efficiently detect signals in a focus angular information of light. A plenoptic 2.0 system trans- volume [55]. The confocal LFM has the advantage of a high formed the position of MLA to acquire a high-resolution signal-to-noise ratio (SNR) and low reconstruction artifacts LF image, and the system can be divided by a Galilean LF through the removal of background noise. scheme and a Keplerian LF scheme [47]. The Galilean LF scheme means that the MLA is placed in front of an image 2.3 LF Image Processing plane position, and the Keplerian LF scheme uses the MLA placed behind an image plane. The LF system has a trade-off Each microlens captures micro-images containing spatial relationship between spatial and angular resolution because and angular information by dividing image volume and the system acquires spatial and angular information with recording the partial images in a 2D image sensor. The a single image sensor. The LF configurations of Galilean micro-images captured by the LF imaging are converted and Keplerian have the advantage of high-spatial resolution into 3D images through computational reconstruction. The compared with the plenoptic 1.0 system due to the trade-off pixel signal information recorded in the image sensor can be relationship. The Keplerian LF scheme captures a real image converted into sub-aperture images as obtained from differ - by recording the image plane in front of the MLA, whereas ent angles because the pixel information represents spatial the Galilean LF scheme collects an inverted virtual image and angular information. Depth information of an object can of image plane formed behind the MLA. be analyzed through stereo-matching, which uses visual dis- parity of each sub-aperture image [28, 56]. Focal stacking 2.2 LFM Configurations or slope estimation of sub-aperture images also allows 3D depth imaging [57, 58]. More efficient and effective algo- The basic configuration of LFM consists of an objective rithms are evolving in LFM applications compared with lens, a tube lens, MLA, and an image sensor. The meth- conventional LF imaging to observe the location of neurons ods of image acquisition and reconstruction can depend and demix fast signals of neurons [53, 59, 60]. However, on the location of MLA and additional lenses (Fig.  2). conventional LFM algorithms still have issues due to ringing 1 3 BioChip Journal (2022) 16:397–408 401 effects, block-wise artifacts, and depth cross talk. The draw - 3 Applications of LFM backs are generated by the process of storing 3D informa- tion in a 2D image sensor plane, and various studies have 3.1 Neuron Imaging been introduced to solve the artifacts by using novel LFM algorithms. A method for reducing artifacts was developed One of the critical techniques for understanding neuronal through depth-dependent sampling patterns [61]. Different activations in neuroscience is the visualization of neural filter shapes and sizes were registered depending on depths, signals. The analysis of neural activity requires high- and artifact-free 3D reconstruction was efficiently performed speed and high-resolution 3D imaging in entire brain through additional anti-aliasing filters. A novel deconvolu- areas because of weak and rapid neural signals [63–65]. tion algorithm using a phase-space deconvolution has also Although neural imaging has been developed in various been reported to solve optical aberrations and background methods such as functional magnetic resonance imag- noises during image reconstruction [62]. The phase-space ing (fMRI) and positron emission tomography (PET) to approach offers a more uniform 3D point-spread-function visualize neuron activity, optical neuron activity imag- (PSF) than the beam propagation model as well as high ing provides higher spatial resolution than other methods image contrast and fast convergence speed imaging. [40]. A calcium indicator is a representative factor that can observe neural activity using fluorescence changes [66, 67]. The indicator indirectly monitors nerve membrane potential but has the advantage of relatively simple obser- vation due to slow transients and high-signal strength. Calcium imaging is also a useful method for measuring thousands of neuron activity at the same time. Optical Fig. 3 Neural imaging of LFM. Imaging of a Zebrafish brain and decomposition technique, and the signal graph of neuronal activities. b Zebrafish blood flow in  vivo through a conventional LFM and a Reprinted from ref [76] with permission by The Optical Society, cop- dictionary LFM. The dictionary LFM offers high-contrast 3D imag- yright 2020. d 3D volumetric imaging of awake mouse brains through ing compared with the conventional LFM. Reprinted from ref [75] the quantitative LFM approach. Reprinted by ref [42] with permis- with permission by Nature Publishing Group, copyright 2021. c The sion from Nature Publishing Group, copyright 2021 reconstructed LFM images of Drosophila brain through the sparse 1 3 402 BioChip Journal (2022) 16:397–408 imaging of neuronal activity is often observed through Therefore, approximately 50  Hz of LFM imaging is 2+ signals from genetically encoded organic fluorescence required to observe Ca signals in C. elegans in the freely dyes for high contrast imaging [68, 69]. Neuron com- behaving state. munication by electrical stimulation lasts about 1 ms and A compressive LFM was introduced for the high-res- repeats at about hundreds of Hz [35]. Therefore, observa- olution and high-speed 3D imaging of the zebrafish brain tion devices with high-spatial and temporal resolution are [74]. Conventional LFMs had a problem with image degra- required for precise analysis, and the 3D imaging of neural dation due to scattering in deep brain tissue. To solve this imaging offers a broad understanding of neural activity problem, the compressive LFM utilized high-accuracy 3D in  vivo. Various animals such as C. elegans, zebrafish, neuron localization by applying a wave-optical multi-slice and mice are usually used for neural activity imaging, and model. The position and fluorescence data of neurons were each animal has its own characteristics [70]. C. elegans quickly collected by skipping 3D image reconstruction steps. and zebrafish are representative objects as animal models As a result, 3D neural structures of the zebrafish brain were in the field of neuroscience due to their transparency and obtained at a sampling rate of 100 Hz. A dictionary LFM small size [71, 72]. In addition, the objects are relatively technique was also introduced for observing the brain and easy to identify because the density of neurons is sparse blood vessels of zebrafish (Fig.  3a and b) [75]. The system compared to a mammalian brain. For example, the brain reduces image noise and artifacts, which are chronic issues of zebrafish contains approximately 100,000 neurons, and in conventional LFMs due to low laser power, by using the the overall brain size is approximately 700 × 500 × 250 μm dictionary information trained from general biological sam- [40, 73]. The neural activity of objects in a freely behav- ples. The dictionary LFM also demonstrates high contrast ing state requires a faster frame rate compared to a static and artifacts-free Zebrafish calcium imaging by reducing state because the object movements cause motion artifacts. Fig. 4 Live-cell imaging of LFM. a The reconstructed 3D images ing resolution compared with the conventional scanning LFM. Repro- of membrane vehicles in COS-7 cells. The living COS-7 cells were duced from ref [80] with permission by Nature Publishing Group, labeled with a mEmerald-Golgi-7 vector. Reprinted from ref [40] copyright 2021. c The reconstructed z-stack images of GFP and with permission by The Optical Society, copyright 2019. b The 3D MitoTracker-stained COS-7 cells. Mitochondria and peroxisomes are fluorescent imaging of cell membrane and mitochondria by using a stained by the MitoTracker and GFP, respectively. Reprinted from ref. mirror enhanced scanning LFM. Magnified images (Right) clearly [83] with permission by The Optical Society, copyright 2019 show that the scanning LFM with the slanted mirror improves imag- 1 3 BioChip Journal (2022) 16:397–408 403 ambiguity in blood cell counting and providing clarity of LFM offers clear images with up to a 300 μm depth in the nerve observation. awakened mouse brain (Fig. 3d). Conventional LFMs had a Drosophila brain is opaque and has a dense neural struc- limitation that an object is required to be fixed in a position ture, thus the observation of neural activity in the Drosophila near an objective lens due to the bulky size of LFM sys- is more challenging compared to C. elegans or zebrafish tems, except for tiny models such as C. elegans. The method due to light scattering. The removal brain cuticle layer is of observing neural activity through head fixation has also also required to perform the calcium imaging of the opaque limitations in acquiring information according to move- brain. A technique that sparse decomposition LFM com- ments; thus, a head-mounted mini-scope is usually used bined with light-sheet microscopy was reported for the high- to observe the neural activity of freely moving animals. A resolution and wide-volume imaging of the Drosophila brain head-mounted miniaturized LFM was developed to observe (Fig. 3c) [76]. Two microscopic images were acquired by the brains of freely behaving mice [77]. The LFM comprises shifting in the light-sheet mode and the wide-field mode, miniaturized components such as grin objective lens, MLA, and the images were merged for clear high-resolution imag- ball lens, and tube lens, resulting in achieving the weight 2+ ing. In addition, the sparse decomposition LFM acquired the of 4 g. The volumetric Ca imaging was demonstrated by image of neuronal cell bodies at a depth of 300 µm through fixing the miniaturized LFM on the head of freely behaving a GCaMP6 injection that efficiently improves neural activity mice. The compact LFM system provides volumetric imag- signals. The intensity of neuronal activities was expressed as ing in the hippocampus area with a 16-Hz frame rate and an signal traces by dividing spatial information. imaging area of 700 × 600 × 360 µm. The visualization of neural activity in a mammalian model such as a mouse is a challenging technique due to 3.2 Live‑Cell Imaging huge scattering. Recently, a quantitative LFM using the incoherent multiple-scattering method was introduced for The observation of structures and mechanisms inside liv- the imaging of mouse brain [42]. The quantitative LFM ing cells is a crucial technique in biomedical fields such considered various factors such as system aberrations and as pharmacology and diagnosis [78, 79]. High-resolution non-uniform resolution along axial planes to improve image 3D imaging is required to visualize the anatomy and func- resolution and contrast. The improved fluorescence signal of tions of cellular components such as mitochondria or cell calcium was also acquired by labeling GCaMP6s and using membranes. An LFM through the compressed sampling of a high NA water-immersion objective lens. The quantitative spatial and angular data was introduced for high-resolution Fig. 5 Locomotion analysis and single-molecule imaging of LFM. a imaging of single-molecule thought the SMLFM. The Scale bar is 3D postures and movement of C. elegans through the LFM with the 2 µm. The characteristics such as photon number, fit error, and axial computational depth algorithm. The LFM system offers quantitative location are expressed by the density plots. Reprinted from ref [39] position analysis of a freely swimming worm. Reproduced from ref with permission by The Optical Society, copyright 2019 [43] with permission by PLOS, copyright 2018. b Super-resolved 3D 1 3 404 BioChip Journal (2022) 16:397–408 live-cell imaging (Fig. 4a) [40]. The dense sampling was computational depth imaging-based LFM (Fig. 5a) [43]. The optimized through MLA and an image sensor, and the image posture and movement speed of C. elegans were analyzed by was reconstructed by using wave-optics with an inverse- comparing different phenotypes of cuticle collagen mutants. problem deconvolution framework. A spatial resolution of The experimental results show that the movement speed in the LFM achieves 300–700 nm, and the reconstruction time the two mutants is similar, but that of non-planar deviation of 3D volume is less than milliseconds. The LFM with the and curving rate are clearly distinguished. A study analyz- compressed sampling provides not only the mitochondria ing the locomotion of C. elegans through an FM embedded imaging of mouse embryo fibroblasts and HeLa cells but with a deep learning algorithm was also reported for fast and also the membrane imaging of COS-7 with 3D volume ren- artifact-free 3D imaging [84]. The system classified behav - dering. An LFM using mirror-enhanced scanning was devel- iors as irregular crawling, forward, or backward as well as oped for high-speed and high-resolution 3D cellular imaging analyzed movement speeds and curvatures. The approach (Fig. 4b) [80]. The conventional LFM had a limitation in the also observed calcium signal patterns changes according to low axial resolution due to a missing-cone issue, i.e., limited the motion behaviors. spatial frequency components caused by the projection angle of an objective lens [81, 82]. A scanning LFM with a tilted 3.4 Single‑Molecule Imaging mirror was devised to simultaneously capture a target image and a reflected image by a mirror. A sample was placed on Single-molecule localization microscopy (SMLM) is an the tilted mirror, and the captured target and mirror images imaging technique that efficiently detects molecules in bio- were reconstructed through a phase-space deconvolution logical structures with a high-spatial resolution. The device algorithm. The approach provides a clear image of the cell allows the observation of subcellular compartments such edge in an X–Z direction and shows the lateral resolution as neuronal synapses, lysosomes, and nuclear proteins that of 0.4 μm and the axial resolution of 1.5 μm. The 3D volu- perform significant roles in cellular functions. The conven- metric images of mitochondria and membranes within NRK tional 3D SMLM had some issues in that an axial resolu- cells were acquired through the scanning LFM, which has tion is reduced due to low photon throughput and extended a 2-Hz volume rate and an FOV of 90 × 70 × 70 µm. The PSFs. An LFM integrated with an SMLM was developed conventional 3D microscopic systems had difficulty observ - to improve the axial resolution (Fig. 5b) [39]. The system ing Dictyostelium discoideum due to light sensitivity and included the configuration of Fourier light-field microscopy, quick movements of the object. The scanning LFM using and utilized algorithms and software optimized for the con- the tilted mirror also achieves the 3D imaging of contractile ventional 2D SMLM. The analyzed results through intervals vacuoles and membranes in the Dictyostelium discoideum. between beads show that the near-isotropic precision of the A 3D cellular imaging method was also introduced through system achieves 20 nm over a DOF of 6 µm. The LFM com- a Fourier LFM with a customized MLA by optimizing aper- bined with the SMLM also demonstrates sufficient resolu- ture division and a wave optics framework (Fig. 4c) [83]. tion to observe DNA origami nanorulers and the microvilli The aperture division and wave optics framework improve of Jurkat T cells. image quality for subcellular imaging, and hybrid PSFs combining experimental and numerical considerations reduce computational artifacts. Imaging performance was 4 Deep Learning Enhanced LFM analyzed through fluorescent beads before in vitro cellular imaging, and the full width half maximum (FWHM) in 3 µm Deep learning is a powerful technique that utilizes artifi - observation ranges shows 0.3–0.7 µm in a lateral dimension cial neural networks for automatically performing feature and 0.5–1.5 µm in an axial dimension. The Fourier LFM detections and classifying data. Recently, this approach has achieves imaging of immune-stained mitochondria and GFP- been implemented in various biomedical applications such stained peroxisomes in COS-7 cells. as microscopy [86–89], MRI [90], and ultrasound imag- ing [91]. In particular, microscopic imaging can efficiently 3.3 Locomotion Analysis enhance image quality through deep learning algorithms because the gap in image performances is obvious accord- C. elegans is a useful model not only for neural activity ing to system configurations. For example, high-resolution imaging but also for analyzing genetic mutations and exter- microscopic imaging was performed through a compact nal stimuli through behavioral changes. Conventional 2D portable microscope system implemented with a deep imaging had limitations in observing the postures and move- learning algorithm [92, 93]. This method improves image ments of the worm on agar gel; thus, fast 3D microscopic resolution and corrects color aberrations through the deep imaging in a wide range is required for accurate analysis. The learning algorithm trained by high-resolution images from 3D motion imaging of C. elegans was introduced through a 1 3 BioChip Journal (2022) 16:397–408 405 Fig. 6 Deep learning-based LFM structure overview. Various biologi- field images and high-resolution images acquired by conventional 3D cal models such as a worm and a zebrafish are captured through the microscopes. Image artifacts reduction and image resolution improve- LFM system. Each 2D light-field channel image is reconstructed into ment can be achieved by diminishing the difference between infer - 3D depth images through a pre-trained light-field network. The light- ence data and ground truths field network is usually trained through iteratively matching light- a benchtop microscope and low-resolution images from a raw light-field images because ground truths rely on images portable microscope. The deep neural network was itera- acquired by conventional microscope systems. tively trained to reduce the gap in image quality between A framework-based hybrid LFM technique was developed the conventional microscope and the portable microscope. to overcome dependence on conventional systems and perform This deep learning-based microscopic imaging is being the fast and high-fidelity reconstruction [ 94]. A new algorithm, extended to LFM applications to solve conventional LFM called HyLFM-Net, offers high-resolution images by simul - issues (Fig. 6). taneously performing verification and learning light-field The LFM has limitations in low and non-uniform reso- images. This method also combined selective illumination lution as well as reconstruction artifacts and requires high microscopy (SIM) setup with an LFM setup for continuously throughput computational processing to recover complex scanning high-speed volume and contributing to image learn- pixel data. An LFM using a view-channel-depth (VCD) neu- ing. The LFM and SIM images were simultaneously acquired ral network was introduced to overcome these limitations through a 30/70 beam-splitter, and each acquired image was [84]. Synthetic light-field images were produced by match- registered in the same reference volume. A convolutional ing 3D high-resolution images already acquired through neural networks (CNN) architecture was designed for image confocal microscopy and input data. The VCD network learning, and the LFM image was reconstructed by rearrang- converted 2D light-field raw data into 3D depth informa- ing individual pixels. The reconstructed 2D convolution layer tion, and the reliability of the VCD network was improved was converted into a 3D image using axial network filters, by iteratively comparing transformed 3D depth images with and the 3D image was further processed through 3D residual ground-truth images. Iterations of signal inference contin- blocks to improve fidelity. The hybrid LFM technique not only ued until the resolution became uniform across the imag- achieves a field of view of 350 × 300 × 150 µm and a volume ing depth. The LFM with the VCD network demonstrates rate of 40–100 Hz but also shows the high-resolution imag- the calcium imaging of C. elegans moving in a microfluidic ing of medaka fish heart. Also, the performance improvement channel, and represents an acquisition rate of 100 Hz and a was demonstrated through the multi-scale-structural similar- processing speed of 13 volumes/s. Various factors such as ity index measure (MS-SSIM) and peak signal-to-noise ratio the number of output channels, filter sizes, and stride were (PSNR) of images. modified to efficiently observe the heart of zebrafish. As a A deep learning LFM technique using a learned iterative result, the direction of blood flow movement was success- shrinkage and thresholding algorithm (LISTA) was also intro- fully predicted, and cardiac dynamics were investigated at duced to reduce reconstruction artifacts and simplify a system an acquisition rate of 200 Hz across a 250 × 250 × 150 µm model [95]. This method can efficiently perform a forward chamber. However, this approach has a limitation in the veri- model calibration with labeled data by compressing the num- fication of deep-learning algorithm network for interpreting ber of light-field views. In addition, an unsupervised technique with Wasserstein generative adversarial networks (WGANs) 1 3 406 BioChip Journal (2022) 16:397–408 was developed to perform image reconstruction in a no-labeled microfluidic systems such as organ-on-a chips can help dataset. The background noise reduction by the unsupervised to understand the functions of the body. Especially, the technique was demonstrated by the imaging of genetically LFM can efficiently acquire the in vitro calcium imaging encoded mouse brain slices. of 3D neural environments with the advantage of fast volu- metric imaging. Studies for the miniaturization of LFM are also expected to be developed for expanding various 5 Conclusions and Outlooks applications such as endoscopy, and point-of-care testing devices. The LFM will also help to efficiently acquire vari - In this review, the principles of LFM, image processing ous biological information in diverse animal models with methods, and biomedical applications for exploring living fast volumetric imaging. organisms have been presented. The LFM is evolving into Acknowledgements This research was financially supported by various LFM configurations through the arrangement of a grant of the National Research Foundation of Korea (NRF) (No. optical components such as an objective lens, MLA, and a 2021R1F1A1048603), the Ministry of SMEs and Startups (No. relay lens. In addition, various image reconstruction algo- S3103859), and the Ministry of Trade, Industry and Energy(No. rithms have been reported to increase image resolution and 20020866). reduce artifacts. The LFM has been demonstrated through Funding Korea Technology and Information Promotion Agency for various biomedical applications such as neuron activity SMEs, No. S3103859, Kisoo Kim, National Research Foundation of visualization, live-cell monitoring, locomotion analysis, Korea (NRF), No. 2021R1F1A1048603, Kisoo Kim, Ministry of Trade, and single-molecule imaging. Various LFM approaches Industry and Energy, No. 20020866, Kisoo Kim. were introduced to achieve optimal performances in each application. Also, the deep learning-based LFM success- Declarations fully provides images with improved spatial resolution and without artifacts. Despite the current progress of Conflict of interest The authors declare no competing financial inter - ests. LFM, continuous advanced studies are required to real- ize superior performance compared to other 3D micro- Open Access This article is licensed under a Creative Commons Attri- scope imaging techniques. Improved image resolution bution 4.0 International License, which permits use, sharing, adapta- and deep penetration performances are required in LFM tion, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, imaging. The resolution of LFM is inevitably low because provide a link to the Creative Commons licence, and indicate if changes the MLA divides spatial information, which reduces reso- were made. The images or other third party material in this article are lution compared to other super-resolution microscopes. included in the article's Creative Commons licence, unless indicated Sub-cellular imaging requires a high-resolution perfor- otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not mance for a deeper understanding of mechanisms in vivo. permitted by statutory regulation or exceeds the permitted use, you will In addition, the penetration depth of LFM has a restric- need to obtain permission directly from the copyright holder. To view a tion due to the scattering of tissue, and image resolution copy of this licence, visit http://cr eativ ecommons. or g/licen ses/ b y/4.0/ . is degraded according to the depth. Overcoming these challenges requires new approaches that diversify optical arrangement, illumination, or image processing algorithms to improve image resolution and penetration depth com- References parable to that of advanced microscopes. Also, improve- 1. Vogt, N.: Monitoring 3D neural activity at large scale. Nat. Meth- ments in image-processing speed are also required. One of ods 13, 195–195 (2016) the LFM advantages is a fast 3D volume image acquisition 2. Marquet, P., Depeursinge, C., Magistretti, P.J.: Review of quantita- speed compared to other microscopes, but the image pro- tive phase-digital holographic microscopy: promising novel imag- cessing time occupies the most time of volumetric imag- ing technique to resolve neuronal network activity and identify cellular biomarkers of psychiatric disorders. Neurophotonics 1, ing. The physical acquisition time of 3D information is 020901 (2014) relatively fast compared to scanning methods because the 3. Schermelleh, L., et al.: Super-resolution microscopy demystified. LFM acquired 3D information about an object through a Nat. Cell Biol. 21, 72–84 (2019) single-shot. However, computational processes with time- 4. Jonkman, J., Brown, C.M., Wright, G.D., Anderson, K.I., North, A.J.: Tutorial: guidance for quantitative confocal microscopy. Nat. consuming are required for relocating the mixed informa- Protoc. 15, 1585–1611 (2020) tion. Advanced techniques for real-time 3D volumetric 5. Wu, Y., Shroff, H.: Faster, sharper, and deeper: structured illu- imaging with deep learning algorithms may continue to mination microscopy for biological imaging. Nat. Methods 15, reduce the time. The combination of LFM with a micro- 1011–1019 (2018) 6. Santi, P.A.: Light sheet fluorescence microscopy: a review. J. His- fluidic chip has the advantage of fixing a target model tochem. Cytochem. 59, 129–138 (2011) within the observation range. 3D light-field imaging of 1 3 BioChip Journal (2022) 16:397–408 407 7. Pawley, J.: Handbook of biological confocal microscopy, vol. 236. 31. Bishop, T.E., Favaro, P.: The light field camera: extended depth Springer, US (2006) of field, aliasing, and super resolution. IEEE Trans. Pattern 8. Yoo, H.-K., et al.: Confocal scanning microscopy: a high-resolu- Anal. Mach. Intell. 34, 972–986 (2011) tion nondestructive surface profiler. Int. J. Precis. Eng. Manuf. 7 , 32. Kim, H.M., et al.: Vari-focal light field camera for extended 3–7 (2006) depth of field. Micromachines 12, 1453 (2021) 9. Fischer, R.S., Wu, Y., Kanchanawong, P., Shroff, H., Waterman, 33. Levoy, M., Ng, R., Adams, A., Footer, M., Horowitz, M.: Light C.M.: Microscopy in 3D: a biologist’s toolbox. Trends Cell Biol. field microscopy. ACM Trans. Graph. 25(4), 924–934 (2006) 21, 682–691 (2011) 34. Bimber, O., Schedl, D.C.: Light-field microscopy: a review. J. 10. Fu, H.L., et al.: Optimization of a wide field structured illumina- Neurol. Neuromed. 4, 1–6 (2019) tion microscope for non-destructive assessment and quantic fi ation 35. Song, P., Verinaz-Jadan, H., Howe, C.L., Foust, A.J., Dragotti, of nuclear features in tumor margins of a primary mouse model of P.L.: Light-field microscopy for the optical imaging of neu- sarcoma. PLoS ONE 8, e68868 (2013) ronal activity: when model-based methods meet data-driven 11. Schermelleh, L., et al.: Subdiffraction multicolor imaging of the approaches. IEEE Signal Process. Mag. 39, 58–72 (2022) nuclear periphery with 3D structured illumination microscopy. 36. Wang, D., Roy, S., Rudzite, A.M., Field, G.D., Gong, Y.: High- Science 320, 1332–1336 (2008) resolution light-field microscopy with patterned illumination. 12. Gustafsson, M.G.: Surpassing the lateral resolution limit by a fac- Biomed. Opt. Express 12, 3887–3901 (2021) tor of two using structured illumination microscopy. J. Microsc. 37. Wang, D., et al.: Hybrid light-sheet and light-field microscope 198, 82–87 (2000) for high resolution and large volume neuroimaging. Biomed. 13. Hillman, E.M., Voleti, V., Li, W., Yu, H.: Light-sheet microscopy Opt. Express 10, 6595–6610 (2019) in neuroscience. Annu. Rev. Neurosci. 42, 295–313 (2019) 38. Wang, D., Zhu, Z., Xu, Z., Zhang, D.: Neuroimaging with light 14. Olarte, O.E., Andilla, J., Gualda, E.J., Loza-Alvarez, P.: Light- field microscopy: a mini review of imaging systems. Eur. Phys. sheet microscopy: a tutorial. Adv. Opt. Photonics 10, 111–179 J. Spec. Top. 231, 749–761 (2022) (2018) 39. Sims, R.R., et al.: Single molecule light field microscopy. Optica 15. Kim, K., Kim, K.S., Park, H., Ye, J.C., Park, Y.: Real-time visu- 7, 1065–1072 (2020) alization of 3-D dynamic microscopic objects using optical dif- 40. Li, H., et  al.: Fast, volumetric live-cell imaging using high- fraction tomography. Opt. Express 21, 32269–32278 (2013) resolution light-field microscopy. Biomed. Opt. Express 10, 16. Lee, M., Kim, K., Oh, J., Park, Y.: Isotropically resolved label- 29–49 (2019) free tomographic imaging based on tomographic moulds for 41. Broxton, M., et al.: Wave optics theory and 3-D deconvolution optical trapping. Light: Sci. Appl. 10, 1–9 (2021) for the light field microscope. Opt. Express 21, 25418–25439 17. Kleiber, A., Kraus, D., Henkel, T., Fritzsche, W.: Tomographic (2013) imaging flow cytometry. Lab Chip 21, 3655–3666 (2021) 42. Zhang, Y., et al.: Computational optical sectioning with an inco- 18. Lee, A.J., Hugonnet, H., Park, W., Park, Y.: Three-dimensional herent multiscale scattering model for light-field microscopy. label-free imaging and quantification of migrating cells during Nat. Commun. 12, 1–11 (2021) wound healing. Biomed. Opt. Express 11, 6812–6824 (2020) 43. Shaw, M., et al.: Three-dimensional behavioural phenotyping of 19. Wilburn, B., et al.: High performance imaging using large cam- freely moving C. elegans using quantitative light field micros- era arrays. ACM Trans. Graph. 24(3), 765–776 (2005) copy. Plos one 13, e0200108 (2018) 20. Ng, R., et al.: Light Field Photography with a Hand-held Ple- 44. Lin, Z., Shum, H.-Y.: A geometric analysis of light field render - noptic Camera. Stanford Tech Report, pp. 1–11 (2005) ing. Int. J. Comput. Vis. 58, 121–138 (2004) 21. Lin, R.J., et al.: Achromatic metalens array for full-color light- 45. Levoy, M., Zhang, Z., McDowall, I.: Recording and controlling field imaging. Nat. Nanotechnol. 14, 227–231 (2019) the 4D light field in a microscope using microlens arrays. J. 22. Shehzad, K., Xu, Y.: Graphene light-field camera. Nat. Photon- Microsc. 235, 144–162 (2009) ics 14, 134–136 (2020) 46. Ng, R.: Digital light field photography. Stanford University 23. Fan, Q., et al.: Trilobite-inspired neural nanophotonic light-field (2006) camera with extreme depth-of-field. Nat. Commun. 13, 1–10 47. Lumsdaine, A. and T. Georgiev. The focused plenoptic camera. In (2022) 2009 IEEE International Conference on Computational Photogra- 24. Bae, S.I., Kim, K., Jang, K.W., Kim, H.K., Jeong, K.H.: High phy (ICCP). IEEE (2009) contrast ultrathin light-field camera using inverted microlens 48. Kim, J., Jung, J.-H., Jeong, Y., Hong, K., Lee, B.: Real-time inte- arrays with metal–insulator–metal optical absorber. Adv. Opt. gral imaging system for light field microscopy. Opt. Express 22, Mater. 9, 2001657 (2021) 10210–10220 (2014) 25. Kim, K., Jang, K.W., Ryu, J.K., Jeong, K.H.: Biologically 49. Chen, Y., et al.: Design of a high-resolution light field miniscope inspired ultrathin arrayed camera for high-contrast and high- for volumetric imaging in scattering tissue. Biomed. Opt. Express resolution imaging. Light Sci. Appl. 9, 28 (2020) 11, 1662–1678 (2020) 26. Kim, K., et al.: Ultrathin arrayed camera for high-contrast near- 50. Guo, C., Liu, W., Hua, X., Li, H., Jia, S.: Fourier light-field infrared imaging. Opt. Express 29, 1333–1339 (2021) microscopy. Opt. Express 27, 25573–25594 (2019) 27. Martínez-Corral, M., Javidi, B.: Fundamentals of 3D imaging 51. Sánchez-Ortiga, E., G. Scrofani, M. Martinez-Corral, and G. and displays: a tutorial on integral imaging, light-field, and ple- Saavedra. Fourier-domain lightfield microscopy: a new paradigm noptic systems. Adv. Opt. Photonics 10, 512–566 (2018) in 3D microscopy. In Biomedical Imaging and Sensing Confer- 28. Jeon, H.-G., et al.: Accurate depth map estimation from a lenslet ence 2020. International Society for Optics and Photonics (2020) light field camera. in Proceedings of the IEEE conference on 52. Liu, F.L., Kuo, G., Antipa, N., Yanny, K., Waller, L.: Fourier dif- computer vision and pattern recognition (2015) fuser scope: single-shot 3D Fourier light field microscopy with a 29. Hahne, C., Aggoun, A., Velisavljevic, V., Fiebig, S., Pesch, diffuser. Opt. Express 28, 28969–28986 (2020) M.: Refocusing distance of a standard plenoptic camera. Opt. 53. Prevedel, R., et al.: Simultaneous whole-animal 3D imaging of Express 24, 21521–21540 (2016) neuronal activity using light-field microscopy. Nat. Methods 11, 30. Kim, H.M., Kim, M.S., Lee, G.J., Jang, H.J., Song, Y.M.: Min- 727–730 (2014) iaturized 3D depth sensing-based smartphone light field camera. Sensors 20, 2129 (2020) 1 3 408 BioChip Journal (2022) 16:397–408 54. Quicke, P., et al.: Subcellular resolution three-dimensional light- 76. Yoon, Y.-G., et al.: Sparse decomposition light-field microscopy field imaging with genetically encoded voltage indicators. Neu- for high speed imaging of neuronal activity. Optica 7, 1457–1468 rophotonics 7, 035006 (2020) (2020) 55. Zhang, Z., et al.: Imaging volumetric dynamics at high speed in 77. Skocek, O., et al.: High-speed volumetric imaging of neuronal mouse and zebrafish brain with confocal light field microscopy. activity in freely moving rodents. Nat. Methods 15, 429–432 Nat. Biotechnol. 39, 74–83 (2021) (2018) 56. Rogge, S., Schiopu, I., Munteanu, A.: Depth estimation for light- 78. Yun, H., Kim, K., Lee, W.G.: Cell manipulation in microfluidics. field images using stereo matching and convolutional neural net- Biofabrication 5, 022001 (2013) works. Sensors 20, 6188 (2020) 79. Kim, K., Lee, W.G.: Electroporation for nanomedicine: a review. 57. Zhang, C., J. Bastian, C. Shen, A. Van Den Hengel, and T. Shen. J. Mater. Chem. B 5, 2726–2738 (2017) Extended depth-of-field via focus stacking and graph cuts. In 2013 80. Xiong, B., et al.: Mirror-enhanced scanning light-field microscopy IEEE International Conference on Image Processing. IEEE (2013) for long-term high-speed 3D imaging with isotropic resolution. 58. Zhang, Y., et al.: Light-field depth estimation via epipolar plane Light: Sci. Appl. 10, 1–11 (2021) image analysis and locally linear embedding. IEEE Trans. Circuits 81. Gustafsson, M.G., et al.: Three-dimensional resolution doubling Syst. Video Technol. 27, 739–747 (2016) in wide-field fluorescence microscopy by structured illumination. 59. Perez, C.C., et al.: Calcium neuroimaging in behaving zebrafish Biophys. J. 94, 4957–4970 (2008) larvae using a turnkey light field camera. J. Biomed. Opt. 20, 82. Campagnola, P.J.: High-speed 3D mapping of nonlinear struc- 096009 (2015) tures. Nat. Photonics 14, 531–532 (2020) 60. Taylor, M.A., Nöbauer, T., Pernia-Andrade, A., Schlumm, F., 83. Hua, X., Liu, W., Jia, S.: High-resolution Fourier light-field Vaziri, A.: Brain-wide 3D light-field imaging of neuronal activ - microscopy for volumetric multi-color live-cell imaging. Optica ity with speckle-enhanced resolution. Optica 5, 345–353 (2018) 8, 614–620 (2021) 61. Stefanoiu, A., Page, J., Symvoulidis, P., Westmeyer, G.G., Lasser, 84. Wang, Z., et al.: Real-time volumetric reconstruction of biologi- T.: Artifact-free deconvolution in light field microscopy. Opt. cal dynamics with light-field microscopy and deep learning. Nat. Express 27, 31644–31666 (2019) Methods 18, 551–556 (2021) 62. Lu, Z., et al.: Phase-space deconvolution for light field micros- 85. Nöbauer, T., et al.: Video rate volumetric Ca2+ imaging across copy. Opt. Express 27, 18131–18145 (2019) cortex using seeded iterative demixing (SID) microscopy. Nat. 63. Lin, A., et al.: Imaging whole-brain activity to understand behav- Methods 14, 811–818 (2017) ior. Nat. Rev. Phys. 4(5), 292–305 (2022) 86. Rivenson, Y., et al.: Deep learning microscopy. Optica 4, 1437– 64. Yang, W., Yuste, R.: In vivo imaging of neural activity. Nat. Meth- 1443 (2017) ods 14, 349–359 (2017) 87. Rivenson, Y., Wu, Y., Ozcan, A.: Deep learning in holography and 65. Ji, N., Freeman, J., Smith, S.L.: Technologies for imaging neural coherent imaging. Light: Sci. Appl. 8, 1–8 (2019) activity in large volumes. Nat. Neurosci. 19, 1154–1164 (2016) 88. de Haan, K., et al.: Automated screening of sickle cells using a 66. Akerboom, J., et al.: Optimization of a GCaMP calcium indicator smartphone-based microscope and deep learning. NPJ Digit. Med for neural activity imaging. J. Neurosci. 32, 13819–13840 (2012) 3, 1–9 (2020) 67. Tian, L., et al.: Imaging neural activity in worms, flies and mice 89. Chen, X., et al.: Deep-learning on-chip light-sheet microscopy with improved GCaMP calcium indicators. Nat. Methods 6, 875– enabling video-rate volumetric imaging of dynamic biological 881 (2009) specimens. Lab Chip 21, 3420–3428 (2021) 68. Hires, S.A., Tian, L., Looger, L.L.: Reporting neural activity with 90. Lundervold, A.S., Lundervold, A.: An overview of deep learning genetically encoded calcium indicators. Brain Cell Biol. 36, 69–86 in medical imaging focusing on MRI. Z. Med. Phys. 29, 102–127 (2008) (2019) 69. Lin, M.Z., Schnitzer, M.J.: Genetically encoded indicators of neu- 91. Liu, S., et al.: Deep learning in medical ultrasound analysis: a ronal activity. Nat. Neurosci. 19, 1142–1153 (2016) review. Engineering 5, 261–275 (2019) 70. Bansal, P., Abraham, A., Garg, J., Jung, E.E.: Neuroscience 92. Rivenson, Y., et  al.: Deep learning enhanced mobile-phone research using small animals on a chip: from nematodes to microscopy. ACS Photonics 5, 2354–2364 (2018) zebrafish larvae. BioChip J. 15, 42–51 (2021) 93. Huang, X., et al.: Smartphone-based analytical biosensors. Ana- 71. Andalman, A.S., et al.: Neuronal dynamics regulating brain and lyst 143, 5339–5351 (2018) behavioral state transitions. Cell 177, 970-985.e20 (2019) 94. Wagner, N., et al.: Deep learning-enhanced light-field imaging 72. Chronis, N., Zimmer, M., Bargmann, C.I.: Microfluidics for with continuous validation. Nat. Methods 18, 557–563 (2021) in vivo imaging of neuronal and behavioral activity in Caeno- 95. Verinaz-Jadan, H., et al.: Deep Learning for Light Field Micros- rhabditis elegans. Nat. Methods 4, 727–731 (2007) copy Using Physics-Based Models. In 2021 IEEE 18th Interna- 73. Sumbre, G., De Polavieja, G.G.: The world according to zebrafish: tional Symposium on Biomedical Imaging (ISBI). IEEE (2021) how neural circuits generate behavior. Front. Neural Circuits 8, 91 (2014) Publisher's Note Springer Nature remains neutral with regard to 74. Pégard, N.C., et al.: Compressive light-field microscopy for 3D jurisdictional claims in published maps and institutional affiliations. neural activity recording. Optica 3, 517–524 (2016) 75. Zhang, Y., et al.: DiLFM: an artifact-suppressed and noise-robust light-field microscopy through dictionary learning. Light: Sci. Appl. 10, 1–12 (2021) 1 3 http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png BioChip Journal Springer Journals

Single-Shot Light-Field Microscopy: An Emerging Tool for 3D Biomedical Imaging

Kim, Kisoo
BioChip Journal , Volume 16 (4) – Dec 1, 2022
Download PDF
12 pages

Loading next page...
 
/lp/springer-journals/single-shot-light-field-microscopy-an-emerging-tool-for-3d-biomedical-0uzFcKO0li

References (108)

Publisher
Springer Journals
Copyright
Copyright © The Author(s) 2022
ISSN
1976-0280
eISSN
2092-7843
DOI
10.1007/s13206-022-00077-w
Publisher site
See Article on Publisher Site

Abstract

3D microscopy is a useful tool to visualize the detailed structures and mechanisms of biomedical specimens. In particular, biophysical phenomena such as neural activity require fast 3D volumetric imaging because fluorescence signals degrade quickly. A light-field microscope (LFM) has recently attracted attention as a high-speed volumetric imaging technique by recording 3D information in a single-snapshot. This review highlighted recent progress in LFM techniques for 3D biomedical applications. In detail, various image reconstruction algorithms according to LFM configurations are explained, and several biomedical applications such as neuron activity localization, live-cell imaging, locomotion analysis, and single-molecule visualization are introduced. We also discuss deep learning-based LFMs to enhance image resolution and reduce reconstruc- tion artifacts. Keywords Light-field microscope (LFM) · 3D biomedical imaging · Neuron activity · Deep learning-enhanced LFM 1 Introduction increased shot noise and reduced signal when the frequency of illumination is close to the cutoff frequency of a detector 3D volumetric microscopy technology is evolving to ana- [10–12]. Light-sheet microscopy is a technique of collecting lyze cell behaviors and functions for biomedical applica- scattered light with an objective lens by projecting a light tions. In particular, 3D biological information such as neural sheet onto a sample using an illuminating objective after activity in vivo is a crucial parameter for brain or biologi- aligning the imaging lens and the illuminating lens vertically cal research [1, 2]. Techniques for 3D microscopic imaging [6, 13, 14]. The light-sheet approach has also limitations in can be classified into point-scanning, structured illumina- that the FOV is limited by the depth of field (DOF) and the tion, and light-sheet imaging, which have pros and cons, image quality can be reduced by scattering. 3D tomographic respectively (Fig. 1a–c) [3–6]. First, scanning microscopy microscopy is also an emerging method to investigate bio- captures only light emitted from a focal plane by removing logical cells without fluorescent labeling [15– 17]. The tomo- out-focused fluorescence through a pinhole, and obtains 3D graphic approach uses the reconstruction of the 3D refractive images through the raster scanning of excitation and detec- index (RI) distribution by capturing holograms with various tion [4, 7, 8]. However, the overall processing time of scan- incident angles of illuminations. Tomographic microscopy ning microscopy is relatively slow due to the scanning of can not only acquire the morphology, protein concentration, an entire field-of-view (FOV), and the poor axial resolution and dry mass of a target but also evade photo-bleaching and is an unresolved limitation [9]. The structured illumination photo-toxicity. However, the frame rate of 3D imaging is microscopy (SIM) uses patterned illumination to improve relatively low due to mechanical scanning of illuminations, spatial resolution, whereas the illumination has limitations in and the tomographic system has a limitation in observing electrical signals such as neural activity [18]. Light-field (LF) imaging is a strategy that captures ple- * Kisoo Kim noptic functions representing the intensity and directions of kisookim@kopti.re.kr light rays in a 3D space [19, 20]. In particular, a handheld LF camera has emerged for recording 3D imaging without Intelligent Optical Module Research Center, Korea Photonics Technology Institute (KOPTI), 9, Cheomdan an external light source or array cameras [21–23]. The LF Venture-ro 108beon-gil, Buk-gu, Gwangju 61007, camera usually uses microlens arrays (MLAs) placed in the Republic of Korea 1 3 Vol.:(0123456789) 398 BioChip Journal (2022) 16:397–408 Fig. 1 3D volumetric microscopy technology for biomedical applica- single-shot light-field microscopic imaging for biomedical appli- tions. a A confocal laser scanning microscope using scanning mirrors cations. The biomedical applications include the imaging of neu- to scan a laser across a sample. b Structured illumination microscopy rons, live-cell, locomotion, and single-molecule. Reprinted from ref (SIM) which illuminates a sample with patterned light to improve [42] with permission by Nature Publishing Group, copyright 2021, spatial resolution. c Light-sheet microscopy which observes a speci- Reprinted from ref [40] with permission by The Optical Society, men illuminated perpendicular to an observation lens. d Light-field copyright 2019, Reprinted from ref [43] with permission from Public microscopy with microlens arrays (MLA) to obtain volumetric infor- Library of Science (PLoS), copyright 2018, and Reprinted from ref mation in a single snapshot. e Schematic illustrations and figures of [39] with permission by The Optical Society, copyright 2020 intermediate image plane of optical configuration to col - tomographic microscopy that require mechanical scanning lect spatial and directional information about light [24–26]. or multiple frames, the LFM captures spatial and angular Such a simple optical configuration allows 3D-depth estima - information of an object without delay by collecting par- tion, sub-aperture imaging, and depth refocusing after single tial images through each channel of the MLA in a single exposure capturing [27–30]. Also, the optical configuration frame (Fig. 1d). The 3D volumetric imaging of LFM realizes of the LF camera extends the depth of field by reconstructing a faster frame rate compared to the other 3D microscopy sub-aperture images [31, 32]. because the LFM can acquire 3D imaging through a single- A light-field microscope (LFM) was first designed by snapshot [34–36]. The LFM is available for various biomedi- Levoy et al. and the 3D information of objects was acquired cal applications such as the imaging of neuronal activity [37, by placing a microlens array (MLA) on an image plane [33]. 38], single-molecule [39], and live-cell [40] owing to the Unlike conventional 3D microscopies such as confocal or fast 3D imaging of LFM. However, the limitations of LFM 1 3 BioChip Journal (2022) 16:397–408 399 Table1 A summary of light-field microscopy according to applications Applications LFM configuration Imaging volume Imaging resolution Illumination method Volume References imaging speed (Hz) Neuron imaging LFM with relay lens 700 × 700 × 200 µm Lateral: ~ 1.4 Widefield illumina- 50 [53] µm tion Axial: 2.6 µm LFM with relay lens ~ 200 × 200 × 200 μm Lateral: ~ 2 µm Widefield illumina- 100 [74] Axial: 3 µm tion LFM with two-photon 900 × 900 × 260 mm Lateral:3.5 µm Widefield illumina- 30 [85] scanning micros- Axial: 9 µm tion copy + Laser scanning Fourier LFM 500 × 350 × 400 µm Lateral: ~ 3.5 µm Widefield illumina- 50 [76] Axial: 7.4 µm tion Confocal LFM ø800 × 200 µm Lateral: ~ 2.1 µm Laser scanning illu- 70 [55] Axial: 2.5 µm mination Live-cell imaging Conventional LFM 100 × 100 × 5 µm Lateral: 0.6 µm Widefield illumina- 10 [40] Axial: 0.3 µm tion Fourier LFM 3 × 4 × 1.2 µm Lateral: 0.3 µm Widefield illumina- 10 [83] Axial: 0.5 µm tion Mirror-enhanced 90 × 70 × 70 µm Lateral: 0.4 µm Widefield illumina- 2 [80] scanning LFM Axial: 1.5 µm tion Locomotion analysis LFM with relay lens Over Axial: 11.1 µm Widefield illumina- 20 [43] 676 × 365 × 110 µm tion Conventional LFM 300 × 300 × 50 µm Lateral: 2.6 µm Widefield illumina- 100 [84] Axial: 5 µm tion Single-molecule Conventional LFM 15 × 15 × 6 µm Lateral, Axial: 20 nm Widefield illumina- 1.25 [39] imaging tion are spatial resolution and a low signal-to-noise ratio (SNR) is required to observe biological structures or signals. due to the superimposition of spatial information through Recent studies are being progressed to solve the resolution the MLA [41]. In addition, the issues of photo-bleaching issues through novel LFM configurations and deep learning and photo-toxicity still remain because the labeling process algorithms. Fig. 2 Schematic illustrations of LFM configurations. a A schematic High-resolution LFM configuration with the MLA positioned behind of experimental setup for single-shot 3D microscopic imaging. b the NIP. e The configuration of Fourier LFM arranged in order of Conventional LFM configuration comprising an objective lens, tube tube lens, Fourier lens, and the MLA. f The configuration of LFM lens, MLA, and an image sensor. c Galilean-mode LFM configuration with relay lens. g A confocal LFM configuration with a mask for the with the MLA positioned in front of a native image plane (NIP). d elimination of background noise 1 3 400 BioChip Journal (2022) 16:397–408 In this review, we will introduce microscopes using light- The conventional LFM comprises the MLA positioned on field technology that can acquire 3D volumetric information the native image plane of the microscope system [45, 48], in a single-shot (Fig. 1e). This review aims to mainly discuss like the plenoptic 1.0 system. Each MLA perceives the in- image acquisition methods and various biomedical appli- focus image of an objective lens and divides the images as cations via the LFM. This article starts by introducing the micro-images. A Galilean-scheme LFM is also similar to descriptions of an LFM principle, optical configurations, and the Galilean LF scheme with the MLA in front of the native image processing methods. The subsequent sections cover plane [49]. The Galilean-scheme LFM can reduce the overall biomedical applications such as the imaging of neurons, thickness of microscope system by diminishing the optical live-cell, worm locomotion, and single-molecule through path length, but needs the MLA with a relatively long focal light-field microscopic imaging (Table  1). Also, the review length. The LFM that the MLA positioned behind the native includes recent studies of deep learning-based light-field image plane is called a Keplerian-scheme LFM [40]. The microscopic imaging to enhance image quality. Finally, the Keplerian-scheme LFM with a relatively long optical path outlook and summary conclude the review. has a high-spatial resolution at the cost of angular resolution degradation, whereas the Keplerian LFM has the disadvan- tage of a relatively large optical configuration. A Fourier 2 LFM Principle LFM uses the MLA placed on a Fourier (pupil) plane to acquire uniform resolution between each MLA and increase 2.1 LF Imaging spatial resolution and depth of field [50– 52]. In addition, the Fourier LFM configuration can increase an image vol- A conventional wide-field imaging method captures an ume and perform efficient computational costs by reduc- image from an image plane formed by a single objective ing the overlap of images formed by the MLA. However, lens, which has a limitation in acquiring the depth informa- the Fourier system still has a trade-off between spatial and tion of an object. On the other hand, light-field imaging can angular sampling issues. Various optical parameters such record 3D information in a 2D image sensor by dividing as magnification and focal length can be controlled by addi- spatial and directional data through an objective lens and tionally integrating relay lenses of various magnifications. MLA [44–46]. The initial type of LF imaging device con- Additional optical components such as the relay lens can sists of an objective lens, MLA, and an image sensor, and efficiently improve the optical performance of LFM when the MLA is placed at the image plane of the objective lens applying a commercial MLA with limitations in parameter [20]. This concept is called a plenoptic 1.0 system, and the selection [53, 54]. The confocal LFM technique utilizes a MLA separates the converging ray to acquire the spatial and mask to selectively and efficiently detect signals in a focus angular information of light. A plenoptic 2.0 system trans- volume [55]. The confocal LFM has the advantage of a high formed the position of MLA to acquire a high-resolution signal-to-noise ratio (SNR) and low reconstruction artifacts LF image, and the system can be divided by a Galilean LF through the removal of background noise. scheme and a Keplerian LF scheme [47]. The Galilean LF scheme means that the MLA is placed in front of an image 2.3 LF Image Processing plane position, and the Keplerian LF scheme uses the MLA placed behind an image plane. The LF system has a trade-off Each microlens captures micro-images containing spatial relationship between spatial and angular resolution because and angular information by dividing image volume and the system acquires spatial and angular information with recording the partial images in a 2D image sensor. The a single image sensor. The LF configurations of Galilean micro-images captured by the LF imaging are converted and Keplerian have the advantage of high-spatial resolution into 3D images through computational reconstruction. The compared with the plenoptic 1.0 system due to the trade-off pixel signal information recorded in the image sensor can be relationship. The Keplerian LF scheme captures a real image converted into sub-aperture images as obtained from differ - by recording the image plane in front of the MLA, whereas ent angles because the pixel information represents spatial the Galilean LF scheme collects an inverted virtual image and angular information. Depth information of an object can of image plane formed behind the MLA. be analyzed through stereo-matching, which uses visual dis- parity of each sub-aperture image [28, 56]. Focal stacking 2.2 LFM Configurations or slope estimation of sub-aperture images also allows 3D depth imaging [57, 58]. More efficient and effective algo- The basic configuration of LFM consists of an objective rithms are evolving in LFM applications compared with lens, a tube lens, MLA, and an image sensor. The meth- conventional LF imaging to observe the location of neurons ods of image acquisition and reconstruction can depend and demix fast signals of neurons [53, 59, 60]. However, on the location of MLA and additional lenses (Fig.  2). conventional LFM algorithms still have issues due to ringing 1 3 BioChip Journal (2022) 16:397–408 401 effects, block-wise artifacts, and depth cross talk. The draw - 3 Applications of LFM backs are generated by the process of storing 3D informa- tion in a 2D image sensor plane, and various studies have 3.1 Neuron Imaging been introduced to solve the artifacts by using novel LFM algorithms. A method for reducing artifacts was developed One of the critical techniques for understanding neuronal through depth-dependent sampling patterns [61]. Different activations in neuroscience is the visualization of neural filter shapes and sizes were registered depending on depths, signals. The analysis of neural activity requires high- and artifact-free 3D reconstruction was efficiently performed speed and high-resolution 3D imaging in entire brain through additional anti-aliasing filters. A novel deconvolu- areas because of weak and rapid neural signals [63–65]. tion algorithm using a phase-space deconvolution has also Although neural imaging has been developed in various been reported to solve optical aberrations and background methods such as functional magnetic resonance imag- noises during image reconstruction [62]. The phase-space ing (fMRI) and positron emission tomography (PET) to approach offers a more uniform 3D point-spread-function visualize neuron activity, optical neuron activity imag- (PSF) than the beam propagation model as well as high ing provides higher spatial resolution than other methods image contrast and fast convergence speed imaging. [40]. A calcium indicator is a representative factor that can observe neural activity using fluorescence changes [66, 67]. The indicator indirectly monitors nerve membrane potential but has the advantage of relatively simple obser- vation due to slow transients and high-signal strength. Calcium imaging is also a useful method for measuring thousands of neuron activity at the same time. Optical Fig. 3 Neural imaging of LFM. Imaging of a Zebrafish brain and decomposition technique, and the signal graph of neuronal activities. b Zebrafish blood flow in  vivo through a conventional LFM and a Reprinted from ref [76] with permission by The Optical Society, cop- dictionary LFM. The dictionary LFM offers high-contrast 3D imag- yright 2020. d 3D volumetric imaging of awake mouse brains through ing compared with the conventional LFM. Reprinted from ref [75] the quantitative LFM approach. Reprinted by ref [42] with permis- with permission by Nature Publishing Group, copyright 2021. c The sion from Nature Publishing Group, copyright 2021 reconstructed LFM images of Drosophila brain through the sparse 1 3 402 BioChip Journal (2022) 16:397–408 imaging of neuronal activity is often observed through Therefore, approximately 50  Hz of LFM imaging is 2+ signals from genetically encoded organic fluorescence required to observe Ca signals in C. elegans in the freely dyes for high contrast imaging [68, 69]. Neuron com- behaving state. munication by electrical stimulation lasts about 1 ms and A compressive LFM was introduced for the high-res- repeats at about hundreds of Hz [35]. Therefore, observa- olution and high-speed 3D imaging of the zebrafish brain tion devices with high-spatial and temporal resolution are [74]. Conventional LFMs had a problem with image degra- required for precise analysis, and the 3D imaging of neural dation due to scattering in deep brain tissue. To solve this imaging offers a broad understanding of neural activity problem, the compressive LFM utilized high-accuracy 3D in  vivo. Various animals such as C. elegans, zebrafish, neuron localization by applying a wave-optical multi-slice and mice are usually used for neural activity imaging, and model. The position and fluorescence data of neurons were each animal has its own characteristics [70]. C. elegans quickly collected by skipping 3D image reconstruction steps. and zebrafish are representative objects as animal models As a result, 3D neural structures of the zebrafish brain were in the field of neuroscience due to their transparency and obtained at a sampling rate of 100 Hz. A dictionary LFM small size [71, 72]. In addition, the objects are relatively technique was also introduced for observing the brain and easy to identify because the density of neurons is sparse blood vessels of zebrafish (Fig.  3a and b) [75]. The system compared to a mammalian brain. For example, the brain reduces image noise and artifacts, which are chronic issues of zebrafish contains approximately 100,000 neurons, and in conventional LFMs due to low laser power, by using the the overall brain size is approximately 700 × 500 × 250 μm dictionary information trained from general biological sam- [40, 73]. The neural activity of objects in a freely behav- ples. The dictionary LFM also demonstrates high contrast ing state requires a faster frame rate compared to a static and artifacts-free Zebrafish calcium imaging by reducing state because the object movements cause motion artifacts. Fig. 4 Live-cell imaging of LFM. a The reconstructed 3D images ing resolution compared with the conventional scanning LFM. Repro- of membrane vehicles in COS-7 cells. The living COS-7 cells were duced from ref [80] with permission by Nature Publishing Group, labeled with a mEmerald-Golgi-7 vector. Reprinted from ref [40] copyright 2021. c The reconstructed z-stack images of GFP and with permission by The Optical Society, copyright 2019. b The 3D MitoTracker-stained COS-7 cells. Mitochondria and peroxisomes are fluorescent imaging of cell membrane and mitochondria by using a stained by the MitoTracker and GFP, respectively. Reprinted from ref. mirror enhanced scanning LFM. Magnified images (Right) clearly [83] with permission by The Optical Society, copyright 2019 show that the scanning LFM with the slanted mirror improves imag- 1 3 BioChip Journal (2022) 16:397–408 403 ambiguity in blood cell counting and providing clarity of LFM offers clear images with up to a 300 μm depth in the nerve observation. awakened mouse brain (Fig. 3d). Conventional LFMs had a Drosophila brain is opaque and has a dense neural struc- limitation that an object is required to be fixed in a position ture, thus the observation of neural activity in the Drosophila near an objective lens due to the bulky size of LFM sys- is more challenging compared to C. elegans or zebrafish tems, except for tiny models such as C. elegans. The method due to light scattering. The removal brain cuticle layer is of observing neural activity through head fixation has also also required to perform the calcium imaging of the opaque limitations in acquiring information according to move- brain. A technique that sparse decomposition LFM com- ments; thus, a head-mounted mini-scope is usually used bined with light-sheet microscopy was reported for the high- to observe the neural activity of freely moving animals. A resolution and wide-volume imaging of the Drosophila brain head-mounted miniaturized LFM was developed to observe (Fig. 3c) [76]. Two microscopic images were acquired by the brains of freely behaving mice [77]. The LFM comprises shifting in the light-sheet mode and the wide-field mode, miniaturized components such as grin objective lens, MLA, and the images were merged for clear high-resolution imag- ball lens, and tube lens, resulting in achieving the weight 2+ ing. In addition, the sparse decomposition LFM acquired the of 4 g. The volumetric Ca imaging was demonstrated by image of neuronal cell bodies at a depth of 300 µm through fixing the miniaturized LFM on the head of freely behaving a GCaMP6 injection that efficiently improves neural activity mice. The compact LFM system provides volumetric imag- signals. The intensity of neuronal activities was expressed as ing in the hippocampus area with a 16-Hz frame rate and an signal traces by dividing spatial information. imaging area of 700 × 600 × 360 µm. The visualization of neural activity in a mammalian model such as a mouse is a challenging technique due to 3.2 Live‑Cell Imaging huge scattering. Recently, a quantitative LFM using the incoherent multiple-scattering method was introduced for The observation of structures and mechanisms inside liv- the imaging of mouse brain [42]. The quantitative LFM ing cells is a crucial technique in biomedical fields such considered various factors such as system aberrations and as pharmacology and diagnosis [78, 79]. High-resolution non-uniform resolution along axial planes to improve image 3D imaging is required to visualize the anatomy and func- resolution and contrast. The improved fluorescence signal of tions of cellular components such as mitochondria or cell calcium was also acquired by labeling GCaMP6s and using membranes. An LFM through the compressed sampling of a high NA water-immersion objective lens. The quantitative spatial and angular data was introduced for high-resolution Fig. 5 Locomotion analysis and single-molecule imaging of LFM. a imaging of single-molecule thought the SMLFM. The Scale bar is 3D postures and movement of C. elegans through the LFM with the 2 µm. The characteristics such as photon number, fit error, and axial computational depth algorithm. The LFM system offers quantitative location are expressed by the density plots. Reprinted from ref [39] position analysis of a freely swimming worm. Reproduced from ref with permission by The Optical Society, copyright 2019 [43] with permission by PLOS, copyright 2018. b Super-resolved 3D 1 3 404 BioChip Journal (2022) 16:397–408 live-cell imaging (Fig. 4a) [40]. The dense sampling was computational depth imaging-based LFM (Fig. 5a) [43]. The optimized through MLA and an image sensor, and the image posture and movement speed of C. elegans were analyzed by was reconstructed by using wave-optics with an inverse- comparing different phenotypes of cuticle collagen mutants. problem deconvolution framework. A spatial resolution of The experimental results show that the movement speed in the LFM achieves 300–700 nm, and the reconstruction time the two mutants is similar, but that of non-planar deviation of 3D volume is less than milliseconds. The LFM with the and curving rate are clearly distinguished. A study analyz- compressed sampling provides not only the mitochondria ing the locomotion of C. elegans through an FM embedded imaging of mouse embryo fibroblasts and HeLa cells but with a deep learning algorithm was also reported for fast and also the membrane imaging of COS-7 with 3D volume ren- artifact-free 3D imaging [84]. The system classified behav - dering. An LFM using mirror-enhanced scanning was devel- iors as irregular crawling, forward, or backward as well as oped for high-speed and high-resolution 3D cellular imaging analyzed movement speeds and curvatures. The approach (Fig. 4b) [80]. The conventional LFM had a limitation in the also observed calcium signal patterns changes according to low axial resolution due to a missing-cone issue, i.e., limited the motion behaviors. spatial frequency components caused by the projection angle of an objective lens [81, 82]. A scanning LFM with a tilted 3.4 Single‑Molecule Imaging mirror was devised to simultaneously capture a target image and a reflected image by a mirror. A sample was placed on Single-molecule localization microscopy (SMLM) is an the tilted mirror, and the captured target and mirror images imaging technique that efficiently detects molecules in bio- were reconstructed through a phase-space deconvolution logical structures with a high-spatial resolution. The device algorithm. The approach provides a clear image of the cell allows the observation of subcellular compartments such edge in an X–Z direction and shows the lateral resolution as neuronal synapses, lysosomes, and nuclear proteins that of 0.4 μm and the axial resolution of 1.5 μm. The 3D volu- perform significant roles in cellular functions. The conven- metric images of mitochondria and membranes within NRK tional 3D SMLM had some issues in that an axial resolu- cells were acquired through the scanning LFM, which has tion is reduced due to low photon throughput and extended a 2-Hz volume rate and an FOV of 90 × 70 × 70 µm. The PSFs. An LFM integrated with an SMLM was developed conventional 3D microscopic systems had difficulty observ - to improve the axial resolution (Fig. 5b) [39]. The system ing Dictyostelium discoideum due to light sensitivity and included the configuration of Fourier light-field microscopy, quick movements of the object. The scanning LFM using and utilized algorithms and software optimized for the con- the tilted mirror also achieves the 3D imaging of contractile ventional 2D SMLM. The analyzed results through intervals vacuoles and membranes in the Dictyostelium discoideum. between beads show that the near-isotropic precision of the A 3D cellular imaging method was also introduced through system achieves 20 nm over a DOF of 6 µm. The LFM com- a Fourier LFM with a customized MLA by optimizing aper- bined with the SMLM also demonstrates sufficient resolu- ture division and a wave optics framework (Fig. 4c) [83]. tion to observe DNA origami nanorulers and the microvilli The aperture division and wave optics framework improve of Jurkat T cells. image quality for subcellular imaging, and hybrid PSFs combining experimental and numerical considerations reduce computational artifacts. Imaging performance was 4 Deep Learning Enhanced LFM analyzed through fluorescent beads before in vitro cellular imaging, and the full width half maximum (FWHM) in 3 µm Deep learning is a powerful technique that utilizes artifi - observation ranges shows 0.3–0.7 µm in a lateral dimension cial neural networks for automatically performing feature and 0.5–1.5 µm in an axial dimension. The Fourier LFM detections and classifying data. Recently, this approach has achieves imaging of immune-stained mitochondria and GFP- been implemented in various biomedical applications such stained peroxisomes in COS-7 cells. as microscopy [86–89], MRI [90], and ultrasound imag- ing [91]. In particular, microscopic imaging can efficiently 3.3 Locomotion Analysis enhance image quality through deep learning algorithms because the gap in image performances is obvious accord- C. elegans is a useful model not only for neural activity ing to system configurations. For example, high-resolution imaging but also for analyzing genetic mutations and exter- microscopic imaging was performed through a compact nal stimuli through behavioral changes. Conventional 2D portable microscope system implemented with a deep imaging had limitations in observing the postures and move- learning algorithm [92, 93]. This method improves image ments of the worm on agar gel; thus, fast 3D microscopic resolution and corrects color aberrations through the deep imaging in a wide range is required for accurate analysis. The learning algorithm trained by high-resolution images from 3D motion imaging of C. elegans was introduced through a 1 3 BioChip Journal (2022) 16:397–408 405 Fig. 6 Deep learning-based LFM structure overview. Various biologi- field images and high-resolution images acquired by conventional 3D cal models such as a worm and a zebrafish are captured through the microscopes. Image artifacts reduction and image resolution improve- LFM system. Each 2D light-field channel image is reconstructed into ment can be achieved by diminishing the difference between infer - 3D depth images through a pre-trained light-field network. The light- ence data and ground truths field network is usually trained through iteratively matching light- a benchtop microscope and low-resolution images from a raw light-field images because ground truths rely on images portable microscope. The deep neural network was itera- acquired by conventional microscope systems. tively trained to reduce the gap in image quality between A framework-based hybrid LFM technique was developed the conventional microscope and the portable microscope. to overcome dependence on conventional systems and perform This deep learning-based microscopic imaging is being the fast and high-fidelity reconstruction [ 94]. A new algorithm, extended to LFM applications to solve conventional LFM called HyLFM-Net, offers high-resolution images by simul - issues (Fig. 6). taneously performing verification and learning light-field The LFM has limitations in low and non-uniform reso- images. This method also combined selective illumination lution as well as reconstruction artifacts and requires high microscopy (SIM) setup with an LFM setup for continuously throughput computational processing to recover complex scanning high-speed volume and contributing to image learn- pixel data. An LFM using a view-channel-depth (VCD) neu- ing. The LFM and SIM images were simultaneously acquired ral network was introduced to overcome these limitations through a 30/70 beam-splitter, and each acquired image was [84]. Synthetic light-field images were produced by match- registered in the same reference volume. A convolutional ing 3D high-resolution images already acquired through neural networks (CNN) architecture was designed for image confocal microscopy and input data. The VCD network learning, and the LFM image was reconstructed by rearrang- converted 2D light-field raw data into 3D depth informa- ing individual pixels. The reconstructed 2D convolution layer tion, and the reliability of the VCD network was improved was converted into a 3D image using axial network filters, by iteratively comparing transformed 3D depth images with and the 3D image was further processed through 3D residual ground-truth images. Iterations of signal inference contin- blocks to improve fidelity. The hybrid LFM technique not only ued until the resolution became uniform across the imag- achieves a field of view of 350 × 300 × 150 µm and a volume ing depth. The LFM with the VCD network demonstrates rate of 40–100 Hz but also shows the high-resolution imag- the calcium imaging of C. elegans moving in a microfluidic ing of medaka fish heart. Also, the performance improvement channel, and represents an acquisition rate of 100 Hz and a was demonstrated through the multi-scale-structural similar- processing speed of 13 volumes/s. Various factors such as ity index measure (MS-SSIM) and peak signal-to-noise ratio the number of output channels, filter sizes, and stride were (PSNR) of images. modified to efficiently observe the heart of zebrafish. As a A deep learning LFM technique using a learned iterative result, the direction of blood flow movement was success- shrinkage and thresholding algorithm (LISTA) was also intro- fully predicted, and cardiac dynamics were investigated at duced to reduce reconstruction artifacts and simplify a system an acquisition rate of 200 Hz across a 250 × 250 × 150 µm model [95]. This method can efficiently perform a forward chamber. However, this approach has a limitation in the veri- model calibration with labeled data by compressing the num- fication of deep-learning algorithm network for interpreting ber of light-field views. In addition, an unsupervised technique with Wasserstein generative adversarial networks (WGANs) 1 3 406 BioChip Journal (2022) 16:397–408 was developed to perform image reconstruction in a no-labeled microfluidic systems such as organ-on-a chips can help dataset. The background noise reduction by the unsupervised to understand the functions of the body. Especially, the technique was demonstrated by the imaging of genetically LFM can efficiently acquire the in vitro calcium imaging encoded mouse brain slices. of 3D neural environments with the advantage of fast volu- metric imaging. Studies for the miniaturization of LFM are also expected to be developed for expanding various 5 Conclusions and Outlooks applications such as endoscopy, and point-of-care testing devices. The LFM will also help to efficiently acquire vari - In this review, the principles of LFM, image processing ous biological information in diverse animal models with methods, and biomedical applications for exploring living fast volumetric imaging. organisms have been presented. The LFM is evolving into Acknowledgements This research was financially supported by various LFM configurations through the arrangement of a grant of the National Research Foundation of Korea (NRF) (No. optical components such as an objective lens, MLA, and a 2021R1F1A1048603), the Ministry of SMEs and Startups (No. relay lens. In addition, various image reconstruction algo- S3103859), and the Ministry of Trade, Industry and Energy(No. rithms have been reported to increase image resolution and 20020866). reduce artifacts. The LFM has been demonstrated through Funding Korea Technology and Information Promotion Agency for various biomedical applications such as neuron activity SMEs, No. S3103859, Kisoo Kim, National Research Foundation of visualization, live-cell monitoring, locomotion analysis, Korea (NRF), No. 2021R1F1A1048603, Kisoo Kim, Ministry of Trade, and single-molecule imaging. Various LFM approaches Industry and Energy, No. 20020866, Kisoo Kim. were introduced to achieve optimal performances in each application. Also, the deep learning-based LFM success- Declarations fully provides images with improved spatial resolution and without artifacts. Despite the current progress of Conflict of interest The authors declare no competing financial inter - ests. LFM, continuous advanced studies are required to real- ize superior performance compared to other 3D micro- Open Access This article is licensed under a Creative Commons Attri- scope imaging techniques. Improved image resolution bution 4.0 International License, which permits use, sharing, adapta- and deep penetration performances are required in LFM tion, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, imaging. The resolution of LFM is inevitably low because provide a link to the Creative Commons licence, and indicate if changes the MLA divides spatial information, which reduces reso- were made. The images or other third party material in this article are lution compared to other super-resolution microscopes. included in the article's Creative Commons licence, unless indicated Sub-cellular imaging requires a high-resolution perfor- otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not mance for a deeper understanding of mechanisms in vivo. permitted by statutory regulation or exceeds the permitted use, you will In addition, the penetration depth of LFM has a restric- need to obtain permission directly from the copyright holder. To view a tion due to the scattering of tissue, and image resolution copy of this licence, visit http://cr eativ ecommons. or g/licen ses/ b y/4.0/ . is degraded according to the depth. Overcoming these challenges requires new approaches that diversify optical arrangement, illumination, or image processing algorithms to improve image resolution and penetration depth com- References parable to that of advanced microscopes. Also, improve- 1. Vogt, N.: Monitoring 3D neural activity at large scale. Nat. Meth- ments in image-processing speed are also required. One of ods 13, 195–195 (2016) the LFM advantages is a fast 3D volume image acquisition 2. Marquet, P., Depeursinge, C., Magistretti, P.J.: Review of quantita- speed compared to other microscopes, but the image pro- tive phase-digital holographic microscopy: promising novel imag- cessing time occupies the most time of volumetric imag- ing technique to resolve neuronal network activity and identify cellular biomarkers of psychiatric disorders. Neurophotonics 1, ing. The physical acquisition time of 3D information is 020901 (2014) relatively fast compared to scanning methods because the 3. Schermelleh, L., et al.: Super-resolution microscopy demystified. LFM acquired 3D information about an object through a Nat. Cell Biol. 21, 72–84 (2019) single-shot. However, computational processes with time- 4. Jonkman, J., Brown, C.M., Wright, G.D., Anderson, K.I., North, A.J.: Tutorial: guidance for quantitative confocal microscopy. Nat. consuming are required for relocating the mixed informa- Protoc. 15, 1585–1611 (2020) tion. Advanced techniques for real-time 3D volumetric 5. Wu, Y., Shroff, H.: Faster, sharper, and deeper: structured illu- imaging with deep learning algorithms may continue to mination microscopy for biological imaging. Nat. Methods 15, reduce the time. The combination of LFM with a micro- 1011–1019 (2018) 6. Santi, P.A.: Light sheet fluorescence microscopy: a review. J. His- fluidic chip has the advantage of fixing a target model tochem. Cytochem. 59, 129–138 (2011) within the observation range. 3D light-field imaging of 1 3 BioChip Journal (2022) 16:397–408 407 7. Pawley, J.: Handbook of biological confocal microscopy, vol. 236. 31. Bishop, T.E., Favaro, P.: The light field camera: extended depth Springer, US (2006) of field, aliasing, and super resolution. IEEE Trans. Pattern 8. Yoo, H.-K., et al.: Confocal scanning microscopy: a high-resolu- Anal. Mach. Intell. 34, 972–986 (2011) tion nondestructive surface profiler. Int. J. Precis. Eng. Manuf. 7 , 32. Kim, H.M., et al.: Vari-focal light field camera for extended 3–7 (2006) depth of field. Micromachines 12, 1453 (2021) 9. Fischer, R.S., Wu, Y., Kanchanawong, P., Shroff, H., Waterman, 33. Levoy, M., Ng, R., Adams, A., Footer, M., Horowitz, M.: Light C.M.: Microscopy in 3D: a biologist’s toolbox. Trends Cell Biol. field microscopy. ACM Trans. Graph. 25(4), 924–934 (2006) 21, 682–691 (2011) 34. Bimber, O., Schedl, D.C.: Light-field microscopy: a review. J. 10. Fu, H.L., et al.: Optimization of a wide field structured illumina- Neurol. Neuromed. 4, 1–6 (2019) tion microscope for non-destructive assessment and quantic fi ation 35. Song, P., Verinaz-Jadan, H., Howe, C.L., Foust, A.J., Dragotti, of nuclear features in tumor margins of a primary mouse model of P.L.: Light-field microscopy for the optical imaging of neu- sarcoma. PLoS ONE 8, e68868 (2013) ronal activity: when model-based methods meet data-driven 11. Schermelleh, L., et al.: Subdiffraction multicolor imaging of the approaches. IEEE Signal Process. Mag. 39, 58–72 (2022) nuclear periphery with 3D structured illumination microscopy. 36. Wang, D., Roy, S., Rudzite, A.M., Field, G.D., Gong, Y.: High- Science 320, 1332–1336 (2008) resolution light-field microscopy with patterned illumination. 12. Gustafsson, M.G.: Surpassing the lateral resolution limit by a fac- Biomed. Opt. Express 12, 3887–3901 (2021) tor of two using structured illumination microscopy. J. Microsc. 37. Wang, D., et al.: Hybrid light-sheet and light-field microscope 198, 82–87 (2000) for high resolution and large volume neuroimaging. Biomed. 13. Hillman, E.M., Voleti, V., Li, W., Yu, H.: Light-sheet microscopy Opt. Express 10, 6595–6610 (2019) in neuroscience. Annu. Rev. Neurosci. 42, 295–313 (2019) 38. Wang, D., Zhu, Z., Xu, Z., Zhang, D.: Neuroimaging with light 14. Olarte, O.E., Andilla, J., Gualda, E.J., Loza-Alvarez, P.: Light- field microscopy: a mini review of imaging systems. Eur. Phys. sheet microscopy: a tutorial. Adv. Opt. Photonics 10, 111–179 J. Spec. Top. 231, 749–761 (2022) (2018) 39. Sims, R.R., et al.: Single molecule light field microscopy. Optica 15. Kim, K., Kim, K.S., Park, H., Ye, J.C., Park, Y.: Real-time visu- 7, 1065–1072 (2020) alization of 3-D dynamic microscopic objects using optical dif- 40. Li, H., et  al.: Fast, volumetric live-cell imaging using high- fraction tomography. Opt. Express 21, 32269–32278 (2013) resolution light-field microscopy. Biomed. Opt. Express 10, 16. Lee, M., Kim, K., Oh, J., Park, Y.: Isotropically resolved label- 29–49 (2019) free tomographic imaging based on tomographic moulds for 41. Broxton, M., et al.: Wave optics theory and 3-D deconvolution optical trapping. Light: Sci. Appl. 10, 1–9 (2021) for the light field microscope. Opt. Express 21, 25418–25439 17. Kleiber, A., Kraus, D., Henkel, T., Fritzsche, W.: Tomographic (2013) imaging flow cytometry. Lab Chip 21, 3655–3666 (2021) 42. Zhang, Y., et al.: Computational optical sectioning with an inco- 18. Lee, A.J., Hugonnet, H., Park, W., Park, Y.: Three-dimensional herent multiscale scattering model for light-field microscopy. label-free imaging and quantification of migrating cells during Nat. Commun. 12, 1–11 (2021) wound healing. Biomed. Opt. Express 11, 6812–6824 (2020) 43. Shaw, M., et al.: Three-dimensional behavioural phenotyping of 19. Wilburn, B., et al.: High performance imaging using large cam- freely moving C. elegans using quantitative light field micros- era arrays. ACM Trans. Graph. 24(3), 765–776 (2005) copy. Plos one 13, e0200108 (2018) 20. Ng, R., et al.: Light Field Photography with a Hand-held Ple- 44. Lin, Z., Shum, H.-Y.: A geometric analysis of light field render - noptic Camera. Stanford Tech Report, pp. 1–11 (2005) ing. Int. J. Comput. Vis. 58, 121–138 (2004) 21. Lin, R.J., et al.: Achromatic metalens array for full-color light- 45. Levoy, M., Zhang, Z., McDowall, I.: Recording and controlling field imaging. Nat. Nanotechnol. 14, 227–231 (2019) the 4D light field in a microscope using microlens arrays. J. 22. Shehzad, K., Xu, Y.: Graphene light-field camera. Nat. Photon- Microsc. 235, 144–162 (2009) ics 14, 134–136 (2020) 46. Ng, R.: Digital light field photography. Stanford University 23. Fan, Q., et al.: Trilobite-inspired neural nanophotonic light-field (2006) camera with extreme depth-of-field. Nat. Commun. 13, 1–10 47. Lumsdaine, A. and T. Georgiev. The focused plenoptic camera. In (2022) 2009 IEEE International Conference on Computational Photogra- 24. Bae, S.I., Kim, K., Jang, K.W., Kim, H.K., Jeong, K.H.: High phy (ICCP). IEEE (2009) contrast ultrathin light-field camera using inverted microlens 48. Kim, J., Jung, J.-H., Jeong, Y., Hong, K., Lee, B.: Real-time inte- arrays with metal–insulator–metal optical absorber. Adv. Opt. gral imaging system for light field microscopy. Opt. Express 22, Mater. 9, 2001657 (2021) 10210–10220 (2014) 25. Kim, K., Jang, K.W., Ryu, J.K., Jeong, K.H.: Biologically 49. Chen, Y., et al.: Design of a high-resolution light field miniscope inspired ultrathin arrayed camera for high-contrast and high- for volumetric imaging in scattering tissue. Biomed. Opt. Express resolution imaging. Light Sci. Appl. 9, 28 (2020) 11, 1662–1678 (2020) 26. Kim, K., et al.: Ultrathin arrayed camera for high-contrast near- 50. Guo, C., Liu, W., Hua, X., Li, H., Jia, S.: Fourier light-field infrared imaging. Opt. Express 29, 1333–1339 (2021) microscopy. Opt. Express 27, 25573–25594 (2019) 27. Martínez-Corral, M., Javidi, B.: Fundamentals of 3D imaging 51. Sánchez-Ortiga, E., G. Scrofani, M. Martinez-Corral, and G. and displays: a tutorial on integral imaging, light-field, and ple- Saavedra. Fourier-domain lightfield microscopy: a new paradigm noptic systems. Adv. Opt. Photonics 10, 512–566 (2018) in 3D microscopy. In Biomedical Imaging and Sensing Confer- 28. Jeon, H.-G., et al.: Accurate depth map estimation from a lenslet ence 2020. International Society for Optics and Photonics (2020) light field camera. in Proceedings of the IEEE conference on 52. Liu, F.L., Kuo, G., Antipa, N., Yanny, K., Waller, L.: Fourier dif- computer vision and pattern recognition (2015) fuser scope: single-shot 3D Fourier light field microscopy with a 29. Hahne, C., Aggoun, A., Velisavljevic, V., Fiebig, S., Pesch, diffuser. Opt. Express 28, 28969–28986 (2020) M.: Refocusing distance of a standard plenoptic camera. Opt. 53. Prevedel, R., et al.: Simultaneous whole-animal 3D imaging of Express 24, 21521–21540 (2016) neuronal activity using light-field microscopy. Nat. Methods 11, 30. Kim, H.M., Kim, M.S., Lee, G.J., Jang, H.J., Song, Y.M.: Min- 727–730 (2014) iaturized 3D depth sensing-based smartphone light field camera. Sensors 20, 2129 (2020) 1 3 408 BioChip Journal (2022) 16:397–408 54. Quicke, P., et al.: Subcellular resolution three-dimensional light- 76. Yoon, Y.-G., et al.: Sparse decomposition light-field microscopy field imaging with genetically encoded voltage indicators. Neu- for high speed imaging of neuronal activity. Optica 7, 1457–1468 rophotonics 7, 035006 (2020) (2020) 55. Zhang, Z., et al.: Imaging volumetric dynamics at high speed in 77. Skocek, O., et al.: High-speed volumetric imaging of neuronal mouse and zebrafish brain with confocal light field microscopy. activity in freely moving rodents. Nat. Methods 15, 429–432 Nat. Biotechnol. 39, 74–83 (2021) (2018) 56. Rogge, S., Schiopu, I., Munteanu, A.: Depth estimation for light- 78. Yun, H., Kim, K., Lee, W.G.: Cell manipulation in microfluidics. field images using stereo matching and convolutional neural net- Biofabrication 5, 022001 (2013) works. Sensors 20, 6188 (2020) 79. Kim, K., Lee, W.G.: Electroporation for nanomedicine: a review. 57. Zhang, C., J. Bastian, C. Shen, A. Van Den Hengel, and T. Shen. J. Mater. Chem. B 5, 2726–2738 (2017) Extended depth-of-field via focus stacking and graph cuts. In 2013 80. Xiong, B., et al.: Mirror-enhanced scanning light-field microscopy IEEE International Conference on Image Processing. IEEE (2013) for long-term high-speed 3D imaging with isotropic resolution. 58. Zhang, Y., et al.: Light-field depth estimation via epipolar plane Light: Sci. Appl. 10, 1–11 (2021) image analysis and locally linear embedding. IEEE Trans. Circuits 81. Gustafsson, M.G., et al.: Three-dimensional resolution doubling Syst. Video Technol. 27, 739–747 (2016) in wide-field fluorescence microscopy by structured illumination. 59. Perez, C.C., et al.: Calcium neuroimaging in behaving zebrafish Biophys. J. 94, 4957–4970 (2008) larvae using a turnkey light field camera. J. Biomed. Opt. 20, 82. Campagnola, P.J.: High-speed 3D mapping of nonlinear struc- 096009 (2015) tures. Nat. Photonics 14, 531–532 (2020) 60. Taylor, M.A., Nöbauer, T., Pernia-Andrade, A., Schlumm, F., 83. Hua, X., Liu, W., Jia, S.: High-resolution Fourier light-field Vaziri, A.: Brain-wide 3D light-field imaging of neuronal activ - microscopy for volumetric multi-color live-cell imaging. Optica ity with speckle-enhanced resolution. Optica 5, 345–353 (2018) 8, 614–620 (2021) 61. Stefanoiu, A., Page, J., Symvoulidis, P., Westmeyer, G.G., Lasser, 84. Wang, Z., et al.: Real-time volumetric reconstruction of biologi- T.: Artifact-free deconvolution in light field microscopy. Opt. cal dynamics with light-field microscopy and deep learning. Nat. Express 27, 31644–31666 (2019) Methods 18, 551–556 (2021) 62. Lu, Z., et al.: Phase-space deconvolution for light field micros- 85. Nöbauer, T., et al.: Video rate volumetric Ca2+ imaging across copy. Opt. Express 27, 18131–18145 (2019) cortex using seeded iterative demixing (SID) microscopy. Nat. 63. Lin, A., et al.: Imaging whole-brain activity to understand behav- Methods 14, 811–818 (2017) ior. Nat. Rev. Phys. 4(5), 292–305 (2022) 86. Rivenson, Y., et al.: Deep learning microscopy. Optica 4, 1437– 64. Yang, W., Yuste, R.: In vivo imaging of neural activity. Nat. Meth- 1443 (2017) ods 14, 349–359 (2017) 87. Rivenson, Y., Wu, Y., Ozcan, A.: Deep learning in holography and 65. Ji, N., Freeman, J., Smith, S.L.: Technologies for imaging neural coherent imaging. Light: Sci. Appl. 8, 1–8 (2019) activity in large volumes. Nat. Neurosci. 19, 1154–1164 (2016) 88. de Haan, K., et al.: Automated screening of sickle cells using a 66. Akerboom, J., et al.: Optimization of a GCaMP calcium indicator smartphone-based microscope and deep learning. NPJ Digit. Med for neural activity imaging. J. Neurosci. 32, 13819–13840 (2012) 3, 1–9 (2020) 67. Tian, L., et al.: Imaging neural activity in worms, flies and mice 89. Chen, X., et al.: Deep-learning on-chip light-sheet microscopy with improved GCaMP calcium indicators. Nat. Methods 6, 875– enabling video-rate volumetric imaging of dynamic biological 881 (2009) specimens. Lab Chip 21, 3420–3428 (2021) 68. Hires, S.A., Tian, L., Looger, L.L.: Reporting neural activity with 90. Lundervold, A.S., Lundervold, A.: An overview of deep learning genetically encoded calcium indicators. Brain Cell Biol. 36, 69–86 in medical imaging focusing on MRI. Z. Med. Phys. 29, 102–127 (2008) (2019) 69. Lin, M.Z., Schnitzer, M.J.: Genetically encoded indicators of neu- 91. Liu, S., et al.: Deep learning in medical ultrasound analysis: a ronal activity. Nat. Neurosci. 19, 1142–1153 (2016) review. Engineering 5, 261–275 (2019) 70. Bansal, P., Abraham, A., Garg, J., Jung, E.E.: Neuroscience 92. Rivenson, Y., et  al.: Deep learning enhanced mobile-phone research using small animals on a chip: from nematodes to microscopy. ACS Photonics 5, 2354–2364 (2018) zebrafish larvae. BioChip J. 15, 42–51 (2021) 93. Huang, X., et al.: Smartphone-based analytical biosensors. Ana- 71. Andalman, A.S., et al.: Neuronal dynamics regulating brain and lyst 143, 5339–5351 (2018) behavioral state transitions. Cell 177, 970-985.e20 (2019) 94. Wagner, N., et al.: Deep learning-enhanced light-field imaging 72. Chronis, N., Zimmer, M., Bargmann, C.I.: Microfluidics for with continuous validation. Nat. Methods 18, 557–563 (2021) in vivo imaging of neuronal and behavioral activity in Caeno- 95. Verinaz-Jadan, H., et al.: Deep Learning for Light Field Micros- rhabditis elegans. Nat. Methods 4, 727–731 (2007) copy Using Physics-Based Models. In 2021 IEEE 18th Interna- 73. Sumbre, G., De Polavieja, G.G.: The world according to zebrafish: tional Symposium on Biomedical Imaging (ISBI). IEEE (2021) how neural circuits generate behavior. Front. Neural Circuits 8, 91 (2014) Publisher's Note Springer Nature remains neutral with regard to 74. Pégard, N.C., et al.: Compressive light-field microscopy for 3D jurisdictional claims in published maps and institutional affiliations. neural activity recording. Optica 3, 517–524 (2016) 75. Zhang, Y., et al.: DiLFM: an artifact-suppressed and noise-robust light-field microscopy through dictionary learning. Light: Sci. Appl. 10, 1–12 (2021) 1 3

Journal

BioChip Journal – Springer Journals

Published: Dec 1, 2022

Keywords: Light-field microscope (LFM); 3D biomedical imaging; Neuron activity; Deep learning-enhanced LFM

There are no references for this article.