Towards Human in the Loop Analysis of Complex Point Clouds: Advanced Visualizations, Quantifications, and Communication Features in Virtual Reality

Thomas Blanc (1, 2) , Hippolyte Verdier (3) , Louise Regnier (1, 2) , Guillaume Planchon (3) , Corentin Guérinot (3, 4) , Mohamed El Beheiry (3) , Jean-Baptiste Masson (3) , Bassam Hajj (1)

1 PCC - Laboratoire Physico-Chimie Curie [Institut Curie]
2 SU - Sorbonne Université
3 Décision et processus Bayesiens - Decision and Bayesian Computation
4 Collège Doctoral

Multiple fields in biological and medical research produce large amounts of point cloud data with high dimensionality and complexity. In addition, a large set of experiments generate point clouds, including segmented medical data or single-molecule localization microscopy. In the latter, individual molecules are observed within their natural cellular environment. Analyzing this type of experimental data is a complex task and presents unique challenges, where providing extra physical dimensions for visualization and analysis could be beneficial. Furthermore, whether highly noisy data comes from single-molecule recordings or segmented medical data, the necessity to guide analysis with user intervention creates both an ergonomic challenge to facilitate this interaction and a computational challenge to provide fluid interactions as information is being processed. Several applications, including our software DIVA for image stack and our platform Genuage for point clouds, have leveraged Virtual Reality (VR) to visualize and interact with data in 3D. While the visualization aspects can be made compatible with different types of data, quantifications, on the other hand, are far from being standard. In addition, complex analysis can require significant computational resources, making the real-time VR experience uncomfortable. Moreover, visualization software is mainly designed to represent a set of data points but lacks flexibility in manipulating and analyzing the data. This paper introduces new libraries to enhance the interaction and human-in-theloop analysis of point cloud data in virtual reality and integrate them into the open-source platform Genuage. We first detail a new toolbox of communication tools that enhance user experience and improve flexibility. Then, we introduce a mapping toolbox allowing the representation of physical properties in space overlaid on a 3D mesh while maintaining a point cloud dedicated shader. We introduce later a new and programmable video capture tool in VR and desktop modes for intuitive data dissemination. Finally, we highlight the protocols that allow simultaneous analysis and fluid manipulation of data with a high refresh

Mots clés: virtual reality; point clouds; inference; microscopy; human in the loop; 3D maps

Domaines: Intelligence artificielle [cs.AI]; Sciences du Vivant [q-bio]

Thomas Blanc, Hippolyte Verdier, Louise Regnier, Guillaume Planchon, Corentin Guérinot, et al.. Towards Human in the Loop Analysis of Complex Point Clouds: Advanced Visualizations, Quantifications, and Communication Features in Virtual Reality. Frontiers in Bioinformatics, 2022, 1, ⟨10.3389/fbinf.2021.775379⟩. ⟨hal-03795768⟩