Monado - GSoC Ideas 2022
Contact and possible mentors
For all projects possible mentors are Jakob Bornecrantz (en/sv), Mateo de Mayo (en/es), Moses Turner (en), Christoph Haag (en/de) & Ryan Pavlik (en/es).
Project Ideas
Head Tracked Fish Tank Display
The core OpenXR 1.0 specification supports stereo VR headsets and monoscopic handheld displays like smartphones or tablets which are supposed to be used as a “magic window” into a VR world or for AR purposes; for this purpose the device’s orientation and position is tracked to provide users the ability to move the “window” view by moving the display device. A further use of monoscopic displays is the “fish tank” configuration with a fixed display like a TV and instead the head position of the user is tracked, to render the content behind the magic window from the right perspective. (Example: https://www.youtube.com/watch?v=Jd3-eiid-Uw). For this project, the student will add support in Monado for tracking a face, figure out the relation of the face/eyes to the a monitor and calculate fov values. The focus of this is not making creating production ready code that in 100% of the cases, but the integration of the code into Monado. The small test application hello_xr will need changes to add better support for mono-scopic fish tank views, like improving the scene setup. Depending on progress, the student can modify one or all of Godot, Blender and Unreal to support mono-scopic fish tank mode.
Deliverables:
- Integrated face-tracking code in Monado that transforms the data and pipes it up to the application.
Requirements:
- Camera: Any webcame or laptop camera will suffice.
- Advanced C
- basic linear algebra
- Some computer vision experience would be helpful. OpenCV provides some required functionality.
Difficulty: medium
Project Size : Large (~350 hours)
Frame Timing Tools
In VR, stutters, judders and jitters in rendering are a much bigger deal than on desktop monitors. Instead of being just annoying, they can cause disorientation, and even motion sickness. Most importantly the VR runtime must push frames to the VR hardware at the native hardware refresh rate. To that end it must keep track of the frames submitted by client applications, reproject old frames if the client application missed an interval, and push the frames to the VR hardware while sharing GPU hardware resources with the client application. Such a system benefits greatly from good visual frame timing tools. A good frame timing tool would gather timing information from both the frames submitted by the client application and the frames submitted by the compositor to the hardware and relate them in a visual way. The visualization could be either done with an existing visualization tool like Perfetto, but it could also be a custom built UI for Monado.
Deliverables
- A system gathering frame timing information from client applications and the runtime compositor and relating them to each other
Requirements:
- Basic C
- Experience with graphics programming (Vulkan) would be helpful
Difficulty: medium
Project Size : Large (~350 hours)
Portable and convenient config UI
Monado has a lot of configurable options and currently, there is no convenient visual means to update them. This project consists in implementing a config UI in Monado and could be broken down into three project phases. First, choose a UI technology that satisfies the below UI requirements. Second, Integrate that UI technology into Monado codebase. Third, implement the required UI menus and elements using the integrated UI codebase.
UI toolkit requirements:
- Must be open-source
- Must be cross-platform (GNU/Linux, Android, Windows)
- Must be capable of rendering rich (and nice) contents.
- Must be lightweight
- Must be accessible from both host console and from within immersive environment.
Deliverables:
- A set of MRs in Monado codebase effectively implementing config UI.
Requirements:
- Scripting knowledge… python, javascript, html, …
- Basic knowledge in human/machine interface
Difficulty: easy
Project Size : Small (~150 hours)
Immersive Dashboard
Monado is an OpenXR runtime used to run immersive OpenXR applications. When the Monado server is first launched, the user wearing any AR or VR headset ends up in a very simple space without any means for directly seeing runtime options or launching applications. Apps need to be executed from another means, such as the Host’s console and so, this project Idea consists in implementing a full-blown Immersive System Dashboard within Monado. This visually appealing dashboard should not only let the user interact with applications’ lifecycle (start, pause, stop), but also allows for seeing debug variables in realtime (frame statistics, timing, etc…). The dashboard as an application in itself should be built with lovr (https://lovr.org/), a framework coded in lua language that allows developers to create very nice immersive objects and scenes very easily and efficiently.
Deliverables:
- A standalone application (The Immersive Dashboard) implementing a nice-looking, conveninent system dashboard.
- A series of MRs into Monado codebase to integrate the Dashboard UI element with the right hooks (for example : For seeing dashboard app as privileged).
Requirements:
- Basic C knowledge (for code additions to Monado)
- Scripting knowledge… python, javascript, html.
- Basic knowledge in human/machine interface
Difficulty: medium
Project Size : Small (~150 hours)
Your own idea
We greatly welcome a student to come up with their own project idea, we could not possibly cover ever single idea that is possible for Monado on our own. Please join our chats to discuss them.