“Extended Mode” - Before Direct Mode

Regardless of actual hardware display arrangement, HMD displays are usually connected with a single HDMI or Displayport connector to a GPU and present themselves as a conventional monitor to the operating systems.

In the early days there was no special handling of these HMD displays and users had to use the operating system’s monitor setup utility and “extend” their desktop to this display.

The desktop would then be displayed on the HMD display, however the desktop in this “extended mode” is not usable for several reasons:

The left eye sees the left half of the desktop, the right eye sees the right half.
Some HMD displays are portrait mode - either the application would need to rotate its rendering, or the monitor would need to be configured as a rotated monitor.
Only software that is aware of the specific lens distortion of the currently connected HMD will be able to correct for it.
Due to the circular or oval lenses the edges and especially the corners are not well visible.

VR software¹ would render a lens distortion-corrected stereo view in a fullscreen window.

Further drawback of “extended mode” include

Depending on the operating system/desktop software the VR software can have a hard time to ensure it actually displays on the correct monitor.
Multi Monitor setups with mixed refresh rates may fail to apply vsync correctly.
Other windows can be displayed on top of VR software, the mouse pointer is not restricted to the non-VR monitors, …

Direct Mode Overview

Around 2015 Oculus worked with Microsoft and GPU driver vendors to improve this situation. The result is called “direct mode”.

Direct mode allows VR software to work on the “display” level of the operating system, and skipping the “windowing system” level.

This means:

There is no need to extend the desktop to the HMD monitor. Starting VR software will use the graphics driver to directly access the display.
VR software will always display on the correct display.
VR software can always properly sync the frame rate to the display.
Desktop windows can not be accidentally moved to the HMD display.

To avoid accidentally extending the desktop to an HMD display, the display will typically be hidden from monitor configuration utilities. This is technically not required but because the HMD displays are usually not well usable without VR software anyway this has become the default.

Because the HMD displays still fundamentally work as monitors and are only hidden via operating system/drivers, software that is not aware of the VR HMDs may still accidentally use the display. For example UEFI/BIOS and bootloaders may accidentally display on the HMD and not on a monitor.

As a countermeasure some HMDs physically power off the display by default and require VR software to enabled the display with an USB command (e.g. Oculus Rift CV1). With such an USB command the display will have to be physically powered on, before it is ready to be used for either extended or direct mode.

PSVR headsets do not physically power off the display by default, but they default to a “cinema mode” that interprets the incoming HDMI signal as a 2D surface in a rotationally tracked VR space. In a similar fashion, it has to be switched to a VR mode with an USB command before it can be “properly” used for extended or direct mode.

Direct Mode in Monado (Linux)

Regardless of driver, HMD displays are only hidden from X.Org, not wayland.

Nvidia

The closed source nvidia driver was the first linux driver to hide HMD displays from X.Org. The driver contains an allowlist (“whitelist”) of HMD displays which it hides completely from X.Org, meaning the display will not appear at all in the output of xrandr. Only the X.Org log file /var/log/Xorg.0.log or ~/.local/share/xorg/Xorg.0.log may indicate the presence of a HMD. The nvidia specific Option "AllowHMD" "yes" xorg.conf setting can disable this behavior.

With the nvidia driver, the vulkan extension VK_KHR_display is used to enumerate connected displays as VkDisplayKHR. Unlike randr it will contain the hidden HMD displays.

Unfortunately there may not be a direct indication which display is a HMD display. VR software may check the EDID for flags indicating a HMD but not all HMDs have these flags. At this time, Monado maintains its own hardcoded allowlist (“whitelist”) which can be dynamically extended with an environment variable: XRT_COMPOSITOR_FORCE_NVIDIA_DISPLAY="Display Name" where the desired “Display Name” should be mentioned as one of the present displays in the output of a previous run.

The nvidia driver does not require a display to be in its allowlist and to be hidden for using it with direct mode. As a result, any connected monitor/display can be used in direct mode. However when trying to use a display that a desktop is currently extended to for direct mode, the direct mode initialization will fail.

Intel/AMD

Direct mode on the open source graphics stack was developed by Valve and Keith Packard.

A list of EDID identifiers for HMD displays is maintained in the linux kernel and applies a non-desktop quirk to them. The X.Org graphics stack reads this list and applies a non-desktop randr property to connected displays that match.

Running xrandr --prop will print non-desktop: 1 for those HMD displays. Unlike the nvidia driver non-desktop displays are not completely hidden, they are merely marked as disconnected in randr. The available modes (resolutions) are still shown in the output of xrandr, monitor configuration tools ignore them only because the display is marked as disconnected.

The non-desktop property can be set and unset dynamically for any connected monitor/display at runtime: xrandr --output HDMI-A-0 --set non-desktop 1 or xrandr --output HDMI-A-0 --set non-desktop 0. When setting the non-desktop property on a display that is currently used for an extended desktop, it should be treated as if that display had been disconnected and the desktop should be automatically “unextended”.

Monado first enumerates connected displays with randr to identify a display that has the non-desktop randr property set and then figures out the corresponding VkDisplayKHR.

Common to all Linux drivers

After a VkDisplayKHR to use for direct mode has been found with a driver specific method, the display is acquired for direct mode with VK_EXT_acquire_xlib_display. Acquiring a display puts it into direct mode.

VK_KHR_display_swapchain is then used to present to the display.

Lastly, when the VR software quits, the display is released with VK_EXT_direct_mode_display.

Wayland

On vanilla wayland compositors HMD displays are not hidden, they appear in monitor configuration tools as conventional monitors.

It is also not possible to acquire a display for direct mode.

Patches for a WIP mesa and a wayland protocol implementation can be found on Drew DeVault’s blog.

Vulkan on KMS/DRM

The VK_EXT_acquire_xlib_display/VK_EXT_direct_mode_display Vulkan extensions are only required to get exclusive control over a display for direct mode when using a windowing system like X.Org.

For standalone XR systems or systems that don’t support these extensions, it is possible to use VK_KHR_display and VK_KHR_display_swapchain directly on KMS/DRM without running X.Org altogether.

Monado v21.0.0 introduces such a VkDisplayKHR based backend. Its internal structure is very similar to the nvidia backend.

This backend requires the Vulkan driver to gain exclusive (“drm master”) control over the display hardware. This means that an X server or wayland compositor must not be control the “active” vt, it must be a text vt (an X server running “in the background” can still be running in the background). The active vt can be switched with Ctrl+Alt+F1 to F12 or sudo chvt 1 to 12.

This backend is only enabled when the environment variable XRT_COMPOSITOR_FORCE_VK_DISPLAY=0 is used. Its value is the nth display connector. Using a too big number like XRT_COMPOSITOR_FORCE_VK_DISPLAY=100 will print the available display connectors and the names of the connected displays.

Note that X servers may not handle well switching to them while the monado compositor is running. Switching to another text vt should work fine.

Many OpenXR applications will require to be started with an X server - however X servers that are “running in the background” may not accept connections for applications that use the GPU. See Tegra for an example of how to get it working on the proprietary nvidia driver.

An OpenXR application that does not require an X server is hello_xr with the Vulkan backend: hello_xr -G Vulkan.

This implementation has been tested on amd (radv) and the proprietary nvidia driver.

Extended Mode in Monado (Linux)

For the various mentioned reasons direct mode is preferred over extended mode. Nevertheless Monado supports running its compositor in extended mode, i.e. creating a X.Org window.

The compositor will fall back to extended mode if no viable direct mode display is found with the driver specific methods described above. If a viable display is found but direct mode fails to initialize, Monado will issue an error message and quit.

With the environment variable XRT_COMPOSITOR_FORCE_XCB=1 Monado will not attempt to use direct mode and will directly create a window.

The compositor window is not shown in fullscreen by default. The window manager may support putting the window into fullscreen. Alternatively with the environment variable XRT_COMPOSITOR_XCB_FULLSCREEN=1 Monado will open the compositor window in fullscreen. Monado does not ensure displaying the compositor on the correct display.

SteamVR-Monado

When using Monado’s SteamVR plugin, the SteamVR compositor and its direct mode implementation is used. SteamVR’s exact implementation of direct mode is not public but it largely follows the same methods described here.

Footnotes

In the early days each VR application included a part of a vendor SDK that was responsible for managing an extended mode window or direct mode for just this application. Nowadays the “VR software” responsible for managing direct mode is usually a long running VR runtime (Oculus Home, SteamVR, WMR Portal, Monado) that VR applications connect to. ↩

Monado