Ptcg-AR: Augmented Reality for Real-Time Multi-View Streaming of the Pokémon Trading Card Game

Abstract

In this Master’s thesis, we aimed to achieve similar visual results as the solution bring by the youtuber SuperZouloux without hardware constraints. To achieve our goal, we developed a novel pipeline purely based on optical sensors, i.e., RGB cameras, to locate and identify the cards on any gaming board. Specifically, we designed a real-time pipeline that ingests a multi-view camera stream, automatically detects and recognizes the cards on the board from a zenithal camera viewpoint, registers the cameras to localize the cards in each video stream, projects 2D or 3D models of the monsters on top of their corresponding cards, and allows to live stream on typical platforms such as YouTube or Twitch through and integration within the common OBS streaming software. Moreover, we designed our system to be as generic as possible. This characteristics allows to integrate seamlessly any trading card game such as Yu-Gi-Oh!™, Magic the Gathering™, or Pokémon™. This generic feature also allow to more easily integrate new features in the future. Lastly, we ensured that the user interface is easy to use. This includes that the setup is easy to install, the application intuitive to use through a codeless interface and easily connectable to other software susceptible to be used along with it. We tested and evaluated our pipeline with the Pokemon™ trading card game that contains 18.000 different cards and 1025 Pokemon creatures in total. To train our detection and identification modules, we created a synthetic dataset by scrapping an online database and generating realistic card positions on different board types and camera noise. Our system achieves 0,9 of mean average precision for the detection part and 0,5 of accuracy in the identification part. Overall, our system achieves a video output of 20 frames per second with a card identification refresh rate of around 6 frames per second. We also qualitatively evaluated the performance of our pipeline and show that it achieves satisfying results during a full game. We believe that this work will democratize the use of augmented reality for all trading card game players and streamers.