Mesh-represented and learning-empowered hologram synthesis for full 3D holographic displays

Xiangyu Meng1†, Wenbin Zhou1†, Yifan (Evan) Peng1*
1The University of Hong Kong
Nature Communications, 2026
†Equal contribution
*Corresponding author
Paper Supplementary Code - Coming soon...
teaser

Overview of proposed mesh-represented 3D holography. a Mesh-based CGH pipeline incorporating the POS-WRP and PAGD. The 3D scenes are rendered into complex-valued wave fields using POS-WRP, which are then encoded into phase-only holograms using PAGD before being displayed and captured on our holographic display prototype. b Visualization comparison of wave field propagation models with exemplary 3D content representations. c Proposed view-dependent holography scheme. The filtered PAGD is employed to synthesize view-dependent holograms from rendered multi-view complex-valued wave fields.

Abstract

Holographic displays are a transforming technology for immersive virtual and augmented reality systems. Exploring accurate yet efficient computer-generated holography (CGH) algorithms for three-dimensional (3D) content is a valuable research field. Recent advancements in layer-based CGH may exhibit limited capacity to convey comprehensive 3D information in accurately representing tilted angular spectrum and realizing realistic defocus blur. Alternative approaches based on point clouds and light fields may demand significant computational resources for preparing adequate target data for optimization. In addition, most existing CGH algorithms rely on heuristics to encode complex amplitudes into phase-only holograms for display, which can be highly ill-posed. Here we investigate an innovative CGH framework that overcomes these challenges using a unique combination of mesh-based representation, tilt-angle tailored wave propagation modeling, and complex-valued optimization, alongside a learning-empowered display calibration scheme using camera feedback.The resulting expanded hologram encoding capabilities enable the delivery of natural 3D depth cues, including smooth defocus blur and view-dependent effects. Experimental results conducted on our holographic near-eye display prototype demonstrate unprecedented full 3D visual quality, representing a significant advancement in creating immersive visualization experiences.

Results

Simulation Results

Scene:
Ground truth
Reconstruction
Simulation result
Ground truth

Experimental Results

Scene:
Ground truth
Reconstruction
Simulation result
Ground truth

Quantitative Evaluation of Quality and Efficiency

Results presented in the following table reveal that leveraging a true 3D dataset further enhances the model's capacity to handle tilted wave fields, leading to superior reconstruction quality for 3D meshes.

Quantitative assessment of holographic reconstructions with varying CGH algorithms.
PSNR ↑ SSIM ↑
Mesh + DPAC 13.54 0.328
Mesh + ASM-PAGD 15.06 0.321
RGBD + Learned Amp-only SGD 16.00 0.294
Mesh + Learned PAGD (2D*) 16.21 0.306
Mesh + Learned PAGD (3D*) 17.09 0.341

* 2D phase: The forward model employed was trained with 2D phase-only holograms;
* 3D phase: The forward model employed was trained with 3D holograms generated by ASM-PAGD.

The following figure illustrates that phase information can help reduce the number of required supervised planes while maintaining equivalent imaging quality.

Figure 3
Image quality and algorithm efficiency study with amplitude vs. complex loss. In iterative CGH algorithms, incorporating phase regularization in the loss function reduces the demand for multi‑plane supervision. Enforcing complex target supervision on 1‑plane yields an improvement of 7 dB in PSNR over 2‑plane amplitude‑only supervision while slightly faster, and is 2.39× faster than 5‑plane amplitude‑only supervision at the same image quality.

Mesh Holography Pipeline

Rendering Wave Field of Mesh-represented Scenes

We introduce a novel rendering pipeline that transforms 3D mesh scenes into holographic wave fields by combining texture baking with a perspective-to-orthogonal transformation, preserving depth cues, lighting, and geometric detail during propagation.The projection-corrected meshes and textures generate smoother phase distributions and reduce artifacts and speckles.

Figure 2
Rendering pipeline of the perspective-to-orthogonal smooth wave field. a The original target scene in a perspective camera projection space. b Mesh and texture post-twisted by POS-WRP in SLM's orthogonal projection space. c The twisted mesh and texture in (b) deliver the same shading and perspective depth cues as the original target scene in (a). d Rendered wave field by POS-WRP with both amplitude and phase. e Textures before (left) and after (right) perspective-correct mapping. The twisted texture (right) produces the correct shading effects as shown in (c). f The depth mapping of mesh coordinates between perspective scene space and the orthogonal SLM space with/without eyepiece. g Rendering results without perspective-correct mapping.

Phase-only Hologram Synthesis

We propose PAGD, a phase-aware optimization method that converts POS-WRP-rendered complex wave fields into phase-only holograms. By supervising both amplitude and phase and using a camera-calibrated learned propagation model, our method better matches real optical hardware and preserves continuous 3D depth cues, producing sharper, higher-contrast holographic reconstructions with fewer artifacts and more natural defocus blur.

Figure 5
Propagation model and dataset for phase-aware gradient descent. a The phase-only hologram displayed on the SLM is processed by a learned forward model, propagating as a complex-valued wave field to three target planes. The phase-aware loss function is calculated between the reconstructed wave fields and the target wave fields derived by ASM at each target plane. This loss function is utilized in back-propagation via auto-gradient for optimizing phase-only holograms. b Depiction of a 3D scene within the dataset featuring randomized planes. c Corresponding wave field comprising amplitude and phase, depth map, and the phase-only hologram generated by PAGD.

View-dependent Holography with PAGD

Our framework naturally supports view-dependent holography by encoding multi-view complex wave fields with blazed gratings and optimizing them through PAGD. With a camera-calibrated forward model and Fourier-plane pupil filtering, the system reconstructs viewpoint-specific images with correct occlusion changes, detailed textures, and continuous depth cues.

Figure 5
View-dependent holographic display Setup and results. a Optical configuration schematic of the view‑dependent display, where blue and green beams represent left and right views converging at different positions on the Fourier plane of the eyepiece, while an iris mimics the pupil to filter out images from unwanted viewpoints. b Diagram of benchtop holographic display setup, where a 35 mm camera lens controlled by Arduino captures multi‑plane images at predefined focal planes. c Experimental results of view-dependent holographic display. Proposed mesh‑based CGH algorithm generates true 3D holograms with accurate parallax and defocus effects. Results are captured by translating the pupil-like iris horizontally and adjusting the camera focus across depth planes, revealing clear shifts and occlusion between foreground and background objects in the left and right views. Defocus blur is shown by comparing defocused (white box) and focused (orange box) regions in zoomed‑in views.

BibTeX


    @Article{MeshHolography2026,
    author={Meng, Xiangyu
    and Zhou, Wenbin
    and Peng, Yifan},
    title={Mesh-represented and learning-empowered hologram synthesis for full 3D holographic displays},
    journal={Nature Communications},
    year={2026},
    month={Jun},
    day={24},
    issn={2041-1723},
    doi={10.1038/s41467-026-74557-0},
    url={https://doi.org/10.1038/s41467-026-74557-0}
    }