Missing depth and judder when streaming Unreal Engine to Apple Vision Pro

Restore passthrough occlusion and remove head-motion judder in Unreal Engine 5.5 – 5.7 projects streamed to Apple Vision Pro.

Summary

When an Unreal Engine 5.5, 5.6, or 5.7 project streams to Apple Vision Pro through Innoactive Spatial Runtime and NVIDIA CloudXR, the headset may display virtual content without depth (no passthrough occlusion) and with a judder that worsens as you move closer to objects. The cause is internal to Unreal Engine on those versions and is fixed either by enabling Substrate on the project or by upgrading to Unreal Engine 5.8.

Applies to

  • Unreal Engine 5.5, 5.6, and 5.7
  • Streaming to Apple Vision Pro via Innoactive Spatial Runtime (NVIDIA CloudXR)

Earlier and later UE versions are not affected — see Option B below for 5.8.

Symptoms

  • Virtual content shows no passthrough occlusion — real-world surfaces never appear in front of or behind virtual geometry; everything floats at the same depth.
  • The image shakes / judders as the viewer moves their head, and the effect gets stronger the closer you are to an object. Standing back from a large scene is tolerable; reaching toward a virtual surface visibly destabilises the image.

Cause

On Unreal Engine 5.5 – 5.7, the engine has the OpenXR depth composition layer enabled, but it only submits depth to the runtime when the scene's depth buffer matches the OpenXR depth swapchain in size and format. In a standard deferred project the formats do not match, so the engine silently drops the depth layer. With no depth, the headset cannot do depth-correct reprojection and cannot composite virtual content against the real world — which produces both the missing occlusion and the close-range judder you observe.

For advanced readers: this is the XR_KHR_composition_layer_depth path; the format mismatch is in the GBuffer-vs-swapchain depth layout used by the deferred renderer.

Resolution

Pick one of the two options below. Option A is a project-level configuration change you can apply on your current engine version. Option B is an engine upgrade.

Option A — Enable Substrate (Unreal Engine 5.5 – 5.7)

Substrate changes the GBuffer and depth layout so the scene depth matches the OpenXR depth swapchain, and the engine then submits the depth layer correctly.

Enable it in one of the following two ways:

From the editor

  1. Open Project Settings → Rendering.
  2. Enable Substrate.
  3. Restart the editor when prompted.

By editing project config

Add these lines under [/Script/Engine.RendererSettings] in Config/DefaultEngine.ini:

r.Substrate=True
r.Substrate.ProjectGBufferFormat=0

The two settings together are what makes depth submission work — r.Substrate=True activates the new path, and r.Substrate.ProjectGBufferFormat=0 selects the GBuffer format that matches the OpenXR depth swapchain.

Before you toggle Substrate, please note:

  • Switching Substrate on triggers a full shader recompile of the project. Expect a long initial build.
  • Substrate converts and re-evaluates existing materials. Some materials may render differently after the switch.
  • Test on a copy of the project first, and review materials that matter to your scene before applying the change to your production project.

Option B — Upgrade to Unreal Engine 5.8

Unreal Engine 5.8 reworked the depth and motion-vector reprojection path. It submits the depth layer correctly without requiring Substrate, so a project on 5.8 has working depth and stable reprojection out of the box. If you are starting a new project or are comfortable with an engine upgrade, this is the simpler long-term path.

How to verify the fix

After applying Option A or Option B and relaunching the streaming session on Apple Vision Pro:

  • Depth and occlusion are present. Real-world surfaces correctly occlude virtual content (and the other way around, depending on how your scene is composed).
  • The image is stable during head motion, including when the viewer moves close to objects. Reaching toward a virtual surface no longer produces the worsening shake.

If you still see the symptoms after enabling Substrate, double-check that both r.Substrate=True and r.Substrate.ProjectGBufferFormat=0 are present and that the project has actually rebuilt shaders against the new setting.

What we ruled out

So you do not chase the wrong thread, the following are not the cause of this issue and do not need to be changed:

  • Firewall, network, or router configuration
  • Forward vs. deferred shading
  • Anti-aliasing method
  • Screen percentage / pixel density
  • The xr.OpenXRAllowDepthLayer console variable — this is already True by default on the affected engine versions, and the depth layer is being enabled. The issue is that the engine then does not submit it because of the depth-buffer format mismatch described above.