All-Optical Selective Target Highlighting for Augmented Reality Microscopy
All-Optical Selective Target Highlighting for Augmented Reality Microscopy
Maha Sahloul1,2, Kaan Akşit3, and M. Fatih Toy1,2
1 Istanbul Medipol University
2 Biophotonics and Computational Imaging Laboratory (Phi-X), Research Institute for Health Sciences and Technologies (SABITA)
3 University College London
Resources
📄 Manuscript 📑 Supplementary Material 🖼️ Poster
Video
Abstract
Augmented Reality (AR) microscopy can enhance microscopic observation by highlighting specific targets, but computational processing can limit its real-time use. We introduce an all-optical solution based on precomputed SLM phase masks that selectively highlight objects according to their shape or size. The approach maintains highlighting as objects move within the field of view, without requiring continuous image processing or tracking.
Methods
Optical Setup
We use a 655 nm laser, collimated by L1, to illuminate the sample mounted on a Zeiss Axio Observer 7 microscope. The beam passes through a linear polarizer aligned with the SLM phase-modulation axis and illuminates the sample through a 10×/0.45 NA objective. The transmitted image is relayed through a 4f optical system to an intermediate image plane, followed by a beam splitter and a 300 mm focal-length lens that positions the SLM at the Fourier plane. We suppress the undesired zeroth-order reflection and select the first diffraction order by adding a blazed grating phase to the SLM phase pattern and applying a 1.16° physical SLM tilt. This configuration shifts the desired first diffraction order onto the camera ROI while displacing the zeroth order outside the detection region.
Figure 1. a) Schematic presents our optical setup including L1: Collimation lens, P1: Polarizer, MO: Microscope objective, L2: Tube lens, BS: Beam splitter, L3: Fourier lens, SLM: Spatial light modulator, ϕ(x, y): Adaptive phase, ψ(x, y): Blazed grating phase, M(x, y): SLM phase. b) Photograph of the experimental setup.
Forward Model
The optical system is numerically modeled as a 4f system, where the input field \(U_{in}(x,y)\) propagates from the intermediate image plane to the SLM plane using the Angular Spectrum Method (ASM):
\[ U_{p1}(x,y;z)=\mathcal{F}^{-1} \left\{ \mathcal{F}\left(U_{in}(x,y,0)\right) H(f_x,f_y;z) \right\} \]
\[ H(f_x,f_y;z)= \exp\left[ ikz\sqrt{1-(\lambda f_x)^2-(\lambda f_y)^2} \right]. \]
The propagated field is modulated by the lens and then reaches the SLM, where the optimized phase mask is applied:
\[ M(x,y)=\exp\left[i\left(\phi(x,y)+\psi(x,y)\right)\right]. \]
where \(\phi(x,y)\) is the adaptive phase and \(\psi(x,y)\) is the blazed grating phase. The resulting field is propagated to the camera plane to obtain the output intensity \(I(x,y)\). The phase mask is optimized by minimizing a combined Mean Squared Error (MSE) and Total Variation (TV) loss:
\[ \mathcal{L}=\mathcal{L}_{MSE}+\mathcal{L}_{TV}. \]
Optimization Methods
We evaluate three optimization strategies that progressively increase the diversity of input data, from a single scene to individual objects and numerically shifted image datasets. All methods use the same forward model to optimize the phase mask, allowing us to examine the effects of input diversity on target highlighting, robustness to new scenes, and computational cost.
Method 1: Single-scene optimization. The phase mask is optimized using a single scene containing multiple objects, with the target distribution highlighting only the desired object.
Method 2: Individual object presentation. We present the objects individually during optimization, allowing the phase mask to encode target-specific patterns. This approach reduces dependence on the optimized scene and allows the target to be highlighted more independently of surrounding objects.
Method 3: Numerically shifted image dataset. The input scene is numerically shifted to generate diverse spatial configurations without requiring additional image acquisition. The target is shifted accordingly to maintain the desired augmentation across different object positions.
Results
Scenario 1: Object highlighting
In the offline stage (I), the phase mask is optimized using a 30 µm diameter circle as the target, generating the precomputed SLM phase pattern (SLM-P). In the real-time augmentation stage (II), the precomputed SLM-P is applied to novel scenes to selectively highlight the target objects. Across 45 novel scenes, the experimental and simulated outputs achieved a mean CC of 0.75 ± 0.03 and PSNR of 27.60 ± 0.61 dB.
Scenario 2: Size-based selective object highlighting
We optimize phase mask to highlight 40 µm-diameter circular objects. Method 1 performs effectively on the optimization scene and scenes with high similarity to it, but its performance decreases as the input scene differs from the optimized scene. Method 2 and Method 3 provide more consistent target highlighting across novel scenes. Method 2 achieve the highest CC of 0.68 and PSNR of 24.07 dB, followed by Method 3 with a CC of 0.65 and PSNR of 23.76 dB, while Method 1 achieve a CC of 0.60 and PSNR of 23.77 dB.
Scenario 3: Morphology-based selective object highlighting
This scenario evaluates morphology-based target augmentation by selectively highlighting a square object among circle, square, and triangle objects of ≈ 40 µm. Quantitatively, Method 1 achieves a CC of 0.59 and PSNR of 25.01 dB, while Method 2 achieves the highest CC of 0.71 and PSNR of 25.84 dB. Method 3 achieves a CC of 0.70 and PSNR of 25.57 dB. These results indicate that Method 2 and Method 3 provide more consistent target highlighting across different morphological scenes.
| Method | Size-Based | Morphology-Based | ||
|---|---|---|---|---|
| CC ↑ | PSNR ↑ | CC ↑ | PSNR ↑ | |
| Method 1 | 0.60 | 23.77 | 0.59 | 25.01 |
| Method 2 | 0.68 | 24.07 | 0.71 | 25.84 |
| Method 3 | 0.65 | 23.76 | 0.70 | 25.57 |