Playstation Move and PSVR 6DoF tracking

Playstation Move and PSVR tracking is a work in progress and the implementation described here will be improved/changed over time.

The camera setup has to be done once per camera and does not have to be repeated for PSVR and Playstation Move controller setup.

The PSVR is fully set up after calibrating the camera. The Playstation Move controller setup requires an additional bluetooth pairing step.

Requirements

Camera Setup

PS4 camera: firmware

By default the ps4 camera does not work as a camera, a firmware file has to be loaded onto the device first.

Download the firmware upload script, and a firmware.bin file to the same directory. The firmware.bin file used here with md5sum 99ca1e97ea2bb9edaf4c6f511292b4e2 comes with uvc support and has been tested with Monado.

wget https://raw.githubusercontent.com/ps4eye/ps4eye/master/python/ps4eye_init.py
wget -O firmware.bin 'https://github.com/psxdev/luke_firmwares/blob/master/101_85C8E0_64036.bin?raw=true'

The firmware.bin file is uploaded with the command

sudo ./ps4eye_init.py

This should print PS4 camera firmware uploaded and device reset.

The camera should now work with standard UVC tools like guvcview. In Video Controls, choose the first entry called USB Camera-OV580: USB Camera-OV.

If the camera does not show up, or guvcview can not produce images, try a different USB port/host controller and especially without a USB hub. The ps4 camera is known to not work on all USB controllers and hubs.

The script has to be run every time the camera is plugged in or the system is rebooted.

To automate loading the firmware when the camera is plugged in, create an udev rule file /etc/udev/rules.d/91-ps4.rules with the content

ACTION=="add",SUBSYSTEMS=="usb",ATTRS{idProduct}=="0580",ATTRS{idVendor}=="05a9",RUN+="/usr/local/opt/ps4eye/ps4eye_init.py"

Adjust the path /usr/local/opt/ps4eye/ps4eye_init.py to where you stored the script and the firmware.

Calibrating the camera with monado

Currently Monado only supports interactive calibration with a checkerboard pattern. In the future a calibration file will be distributed with monado and calibration will be simplified, perhaps even unnecessary.

The monado-gui utility is used to calibrate the camera before using it with Monado. This is necessary to compensate for the usual camera/lens distortion. monado-gui is usually installed in /user/bin. In a monado build tree the gui utility is located at ./src/xrt/targets/gui/monado-gui

  1. Get a checkerboard grid. It can be printed on a sheet of paper or displayed on a monitor. For example you can use a checkerboard generator:
    1. Go to https://calib.io/pages/camera-calibration-pattern-generator
    2. Set Target Type to Checkerboard. The calibration looks for black and white squares with no symbols.
    3. Set Rows to 7 and Columns to 9. Other configurations can work too, but 7x9 is a good default.
    4. Show the resulting image/pdf on screen or print it.
    5. Measure the size of a single square.
  2. In the monado-gui menu choose Calibrate
  3. Choose the USB Camera-OV580 camera (the PS4 camera)
  4. Choose the second resolution 1748x408 (the full resolution takes quite some CPU power)
  5. Confirm the 7 rows, 9 columns setting matches your checkerboard
  6. set Checker Size to the previously measured size of one square in meters and click Done.
  7. Move the camera around, until it captures enough images.
    • tilt the camera and view the checkerboard from different angles
    • capture the checkerboard in as much of the camera’s field of view as possible
  8. After the calibration finishes, do not forget to click save before exiting.

Two configuration files should be stored. ~/.config/monado/config_v0.json contains information about the calibration and ~/.config/monado/PS4.calibration contains the actual calibration data.

After these steps a PSVR HMD should automatically begin to track when Monado is started.

Controller setup

USB/HID Permissions

Even when connected over bluetooth, your user will require the permissions to access the psmove’s hid-over-bluetooth device.

If you use the “xr-hardware” udev rules, you don’t need to do anything, it includes rules to set permissions for the psmove controllers.

If you want to install only rules specific to the psmove instead, you can follow the psmove documentation.

Bluetooth pairing the controllers

Bluetooth pairing the controllers requires a USB connection. However IMU data can only be read through bluetooth, once the controllers are paired, reading IMU data through USB is not supported.

The pairing procedure is detailed at https://github.com/thp/psmoveapi/blob/master/docs/pairing.rst.

In short:

git clone --recursive https://github.com/thp/psmoveapi.git
cd psmoveapi
cmake .
make
sudo ./psmove pair

Pairing doesn’t always work reliably, it may be necessary to restart the bluez daemon sudo systemctl restart bluetooth and repeat pairing until it works. When powering on the controller with its power button, it should automatically connect to the PC, and bluetooth tools should show the controller (“Motion Controller”) as connected. Trying to initiate a connection from the PC usually does not work.

Controller Usage

The psmove poses and buttons are fully exposed through a monado specific interaction profile /interaction_profiles/mnd/ball_on_stick_controller (see bindings in the code), and limited bindings are available through the khronos simple controller interaction profile.

When a PS4 camera with uvc enabled firmware is connected, a calibration file is present, and a paired and connected psmove controller is connected, monado should automatically enable positional tracking for the psmove controllers.

As an example, hello_xr -g Vulkan should have the psmove controller enabled as a positionally tracked “khronos simple controller”, including the trigger and haptic feedback.

To debug issues, monado implements a debug UI in a separate window which can be enabled with an environment variable: XRT_DEBUG_GUI=1 monado-service and then hello_xr -g Vulkan. If monado can find and enable the camera, there will be windows HSV Filter and PSMV Tracker #1 and PSMV Tracker #2 available in the debug UI, showing the raw camera image as well as the color filtered camera images.