All-Optical Selective Target Highlighting for Augmented Reality Microscopy

Fri, 18 Sep 2026 00:00:00 +0000

All-Optical Selective Target Highlighting for Augmented Reality Microscopy

Maha Sahloul^1,2, Kaan Akşit³, and M. Fatih Toy^1,2

¹ Istanbul Medipol University
² Biophotonics and Computational Imaging Laboratory (Phi-X), Research Institute for Health Sciences and Technologies (SABITA)
³ University College London

Resources

📄 Manuscript 📑 Supplementary Material 🖼️ Poster

Video

Abstract

Augmented Reality (AR) microscopy can enhance microscopic observation by highlighting specific targets, but computational processing can limit its real-time use. We introduce an all-optical solution based on precomputed SLM phase masks that selectively highlight objects according to their shape or size. The approach maintains highlighting as objects move within the field of view, without requiring continuous image processing or tracking.

Methods

Optical Setup

We use a 655 nm laser, collimated by L1, to illuminate the sample mounted on a Zeiss Axio Observer 7 microscope. The beam passes through a linear polarizer aligned with the SLM phase-modulation axis and illuminates the sample through a 10×/0.45 NA objective. The transmitted image is relayed through a 4f optical system to an intermediate image plane, followed by a beam splitter and a 300 mm focal-length lens that positions the SLM at the Fourier plane. We suppress the undesired zeroth-order reflection and select the first diffraction order by adding a blazed grating phase to the SLM phase pattern and applying a 1.16° physical SLM tilt. This configuration shifts the desired first diffraction order onto the camera ROI while displacing the zeroth order outside the detection region.

Figure 1. a) Schematic presents our optical setup including L1: Collimation lens, P1: Polarizer, MO: Microscope objective, L2: Tube lens, BS: Beam splitter, L3: Fourier lens, SLM: Spatial light modulator, ϕ(x, y): Adaptive phase, ψ(x, y): Blazed grating phase, M(x, y): SLM phase. b) Photograph of the experimental setup.

Forward Model

The optical system is numerically modeled as a 4f system, where the input field \(U_{in}(x,y)\) propagates from the intermediate image plane to the SLM plane using the Angular Spectrum Method (ASM):

\[ U_{p1}(x,y;z)=\mathcal{F}^{-1} \left\{ \mathcal{F}\left(U_{in}(x,y,0)\right) H(f_x,f_y;z) \right\} \]

\[ H(f_x,f_y;z)= \exp\left[ ikz\sqrt{1-(\lambda f_x)^2-(\lambda f_y)^2} \right]. \]

The propagated field is modulated by the lens and then reaches the SLM, where the optimized phase mask is applied:

\[ M(x,y)=\exp\left[i\left(\phi(x,y)+\psi(x,y)\right)\right]. \]

where \(\phi(x,y)\) is the adaptive phase and \(\psi(x,y)\) is the blazed grating phase. The resulting field is propagated to the camera plane to obtain the output intensity \(I(x,y)\). The phase mask is optimized by minimizing a combined Mean Squared Error (MSE) and Total Variation (TV) loss:

\[ \mathcal{L}=\mathcal{L}_{MSE}+\mathcal{L}_{TV}. \]

Optimization Methods

We evaluate three optimization strategies that progressively increase the diversity of input data, from a single scene to individual objects and numerically shifted image datasets. All methods use the same forward model to optimize the phase mask, allowing us to examine the effects of input diversity on target highlighting, robustness to new scenes, and computational cost.

Method 1: Single-scene optimization. The phase mask is optimized using a single scene containing multiple objects, with the target distribution highlighting only the desired object.

Method 2: Individual object presentation. We present the objects individually during optimization, allowing the phase mask to encode target-specific patterns. This approach reduces dependence on the optimized scene and allows the target to be highlighted more independently of surrounding objects.

Method 3: Numerically shifted image dataset. The input scene is numerically shifted to generate diverse spatial configurations without requiring additional image acquisition. The target is shifted accordingly to maintain the desired augmentation across different object positions.

Results

Scenario 1: Object highlighting

In the offline stage (I), the phase mask is optimized using a 30 µm diameter circle as the target, generating the precomputed SLM phase pattern (SLM-P). In the real-time augmentation stage (II), the precomputed SLM-P is applied to novel scenes to selectively highlight the target objects. Across 45 novel scenes, the experimental and simulated outputs achieved a mean CC of 0.75 ± 0.03 and PSNR of 27.60 ± 0.61 dB.

Scenario 2: Size-based selective object highlighting

We optimize phase mask to highlight 40 µm-diameter circular objects. Method 1 performs effectively on the optimization scene and scenes with high similarity to it, but its performance decreases as the input scene differs from the optimized scene. Method 2 and Method 3 provide more consistent target highlighting across novel scenes. Method 2 achieve the highest CC of 0.68 and PSNR of 24.07 dB, followed by Method 3 with a CC of 0.65 and PSNR of 23.76 dB, while Method 1 achieve a CC of 0.60 and PSNR of 23.77 dB.

Scenario 3: Morphology-based selective object highlighting

This scenario evaluates morphology-based target augmentation by selectively highlighting a square object among circle, square, and triangle objects of ≈ 40 µm. Quantitatively, Method 1 achieves a CC of 0.59 and PSNR of 25.01 dB, while Method 2 achieves the highest CC of 0.71 and PSNR of 25.84 dB. Method 3 achieves a CC of 0.70 and PSNR of 25.57 dB. These results indicate that Method 2 and Method 3 provide more consistent target highlighting across different morphological scenes.

Method	Size-Based		Morphology-Based
Method	CC ↑	PSNR ↑	CC ↑	PSNR ↑
Method 1	0.60	23.77	0.59	25.01
Method 2	0.68	24.07	0.71	25.84
Method 3	0.65	23.76	0.70	25.57

Acknowledgment

This work is supported by TÜBİTAK under project number 123N774. M. F. Toy acknowledges support from the Turkish Academy of Sciences - Outstanding Young Scientist Award Program (TUBA - GEBIP). Open access publication of this work was supported by the institutional agreement between UCL and Optica.

Augmented Reality Microscopy | Phi - X

All-Optical Selective Target Highlighting for Augmented Reality Microscopy

All-Optical Selective Target Highlighting for Augmented Reality Microscopy

Resources

Video

Abstract

Methods

Optical Setup

Forward Model

Optimization Methods

Results

Scenario 1: Object highlighting

Scenario 2: Size-based selective object highlighting

Scenario 3: Morphology-based selective object highlighting

Acknowledgment