Advancing 3D scene generation through GCWA with dynamic upsampling using two-stage diffusion model

This paper studies a novel 3D scene generation framework with enhancement of spatial semantic consistency. In the considered system, diffusion models aim to synthesize high-fidelity 3D outdoor scenes for autonomous driving applications under the challenge of maintaining spatial semantic coherence. Existing methods struggle to balance contextual semantic accuracy and computational efficiency during feature distributions learning and upsampling processes. Unlike existing approaches, we propose a two-stage discrete diffusion framework based on 3D scene triplane representation with two novel components to address these interdependent issues. Specifically, we exploit a Geometry-aligned Cross-plane Windows Attention (GCWA) based on the triplane representation of 3D scenes to enhance global consistency through intra-plane self-attention and cross-plane cross-attention. This manipulation not only enhances the global semantic consistency of the 3D scene but also reduces the excessive computational complexity. Then we design a multi-scale dynamic upsampling module DySample3D that adaptively integrates the semantic context into the resolution enhancement process, which is to improve consistency in boundary areas of 3D scenes and reduce semantic artifacts. Comprehensive evaluations on CarlaSC and SemanticKITTI datasets demonstrate our framework’s superior performance in both unconditional and conditional generation tasks, achieving significant improvements over the latest existing benchmarks without substantial increases in computational complexity. Furthermore, our model can be seamlessly adapted to scene inpainting, outpainting tasks and achieve the conversion of semantic scenes to realistic RGB images. Code has been released on https://github.com/HITysx/3D-scene-generation.