I love playing Beat Saber, a VR rhythm game in which you slice blocks in time with music. Turning a song into a Beat Saber map requires serious time, dedication and talent. I’ll attempt to bypass the requirements by trying to use artificial intelligence to make the Beat Saber maps.
In order to create such a model, I first need to understand the format of beat maps so I can read them for data and also to ultimately create my own. I then need to analyse them so I can gain a better understanding of the feature space and obtain some intuition about what sort of model I can use. Then, I’ll need to craft the model, train it, figure out how to evaluate the results, and iterate until something usable hopefully comes out. Finally, I’d like to make the model available for other people to use.
This first post will begin the exploration part of the project by having a look at the data.
The cookiecutter data science project template is a great starting point for the project repository. I performed the analysis presented in this post using a Jupyter notebook which can be found here.
What does the data look like
While having never created a map, playing the game gave me an intuition for what to expect. The first step was to investigate an existing song. Looking into a song directory we see the following structure:
. ├── Expert.dat ├── Hard.dat ├── Normal.dat ├── Toto\ -\ Africa.egg ├── cover.jpg └── info.dat
The folder contains a jpg cover photo, an egg audio file which contains the music track, and multiple dat files. The info file represents the meta-information about the song in JSON format: the name and artist, but also direct links to the cover image, audio file, and each difficulty map, also stored as JSON dat files.
The maps are also JSON files. Notably, they contain lists such as “_events”, “_obstacles”, and “_notes”. I did not know what events are, so I simply removed them from the list and played the song. A dark scene with no lights confirmed that “_events” handles the lighting effects. For simplicity, I’ll study the notes, since they represent the core of the gameplay. Each note is composed of the following parameters:
{
"_time": 28.5625,
"_lineIndex": 2,
"_lineLayer": 0,
"_type": 1,
"_cutDirection": 0
},
I used BeatMapper to create a pattern of notes to help me understand what each parameter means.
"_time": time (seconds) of the note from the beginning of the song "_lineIndex": the position on the x-axis of the note, from 0-3 "_lineLayer": the position on the y-axis of the note, from 0-2 "_type": the hand of the note, 0 = left, 1 = right "_cutDirection": the direction in which to cut the note. 0-7 are the 7 directions and 8 is the dot-block
Song analysis
To get some statistics on the data, I used the Jupyter notebook linked in the introduction. Since the notebook is commented, I’ll only quickly present my findings. For raw data, I linked my existing library of 500 songs.
The game is designed to handle any lineIndex and lineLayer value. Even so, most Expert+ songs use the base 3×4 grid. I use this to justify simplifying the dataset and therefore the feature space.
Additionally, I used bit-shifting to create a unique 16-bit identifier for each possible simultaneous note pattern. I limited each pattern to at most one block per hand.
This analysis shows that an output of 16 neurons can represent a time step worth of note data. In addition, while the number of theoretically possible combinations is astronomical in size, almost 99% of all occurring configurations are represented by the top 1000 configurations.
Conclusion
Investigating data plays an essential role in building a machine learning model. While not a complete analysis yet, I hope it can already be a source of inspiration and illustrate the importance of getting a good idea about the data you’re working with. Thank you for reading the first post about my project to use machine learning to make beat saber maps.