Generative XR

Overview

Generative XR is a research project exploring how generative AI models can be integrated into extended reality pipelines to enable dynamic, adaptive, and content-rich XR experiences. The project focuses on treating generation not as an offline asset creation step, but as a live component of the XR system.

The goal is to understand how AI-generated content can meaningfully interact with spatial context, user intent, and real-time constraints.

Research Motivation

XR systems traditionally rely on pre-authored assets and deterministic logic. While this ensures predictability, it limits variability, personalization, and scalability.

Generative XR investigates questions such as:

• How can generative models respond to spatial and contextual input?
• What role can AI play in real-time scene composition?
• How does generation affect user perception of agency and immersion?

The project aims to bridge generative AI workflows with interactive 3D environments.

System Architecture

The prototype is structured around a modular architecture:

1. Prompt & Context Layer User input, spatial data, and environmental cues are combined into structured prompts.

2. Generative Engine Diffusion-based models are used to generate textures, imagery, and visual elements on demand.

3. XR Runtime Integration Generated assets are streamed into Unity and adapted to the spatial context.

4. Feedback Loop User interaction influences subsequent generations, enabling iterative refinement.

Python is used for model orchestration, while Unity manages rendering and interaction.

Experiments

Initial experiments focused on:

• Generating environment textures based on spatial constraints
• AI-assisted scene theming and mood adaptation
• Prompt-driven visual variation inside immersive environments

Evaluation emphasized latency, coherence, and perceived responsiveness rather than raw visual fidelity.

Challenges & Findings

Key challenges identified include:

• Managing generation latency within immersive contexts
• Maintaining visual consistency across generated assets
• Balancing creative variability with spatial constraints

Early findings suggest that constrained generation, guided by spatial data, significantly improves user acceptance and immersion.

Future Directions

Planned developments include:

• Real-time multimodal generation (text, image, 3D cues)
• Integration with agent-based AI workflows
• Multi-user shared generative spaces
• Exploration of generative XR for design and prototyping tools

Generative XR remains an active research effort aimed at expanding the expressive potential of XR systems.