Category Archives: Uncategorized

The Zen Balls Interactive Music System

Zen Balls Interactive Music System – A postmortem

The music implementation for the music in Zen Balls was by far the most complicated part of the project. I was unable to use any middleware, but I was also unwilling to just have completely looping music. This left me with the only other option of coding whatever interactive music system I could manage to come up with. This had its ups and downs, both of which will be discussed here, but ultimately I was able to create something that mostly works. Nevertheless, despite being able to ship the game with an interactive music system that largely functions the way I intended, I will conclude with why I hope to never write my own music system this way again. What follows is a four part postmortem of the music system I made for Zen Balls. It starts with a description of what it actually does, continues with a short “what went right” paragraph and a much longer “what went wrong” section, and finishes with some concluding thoughts based on my overall experience.

What I built:

First of all, I needed to think of what the game needed because of what it was fundamentally. I initially thought about the music implementation in terms of what Wwise calls the “vertical” and “horizontal” approaches.

The vertical approach was tempting at first, but it didn’t really fit the feel of the game. Since there are two collectibles in each level, I briefly entertained the possibility of certain tracks in the music fading in once you had collected them, but ultimately I decided that since the levels can be completed rather quickly, a vertical approach would be distracting and not ideal.

In the context of the game’s design, the horizontal approach seemed best. There are ten levels per chapter, but each chapter uses only one piece of music. I decided to let the current level being played dictate how far along in the chapter’s composition the music would play. Regardless of what level you navigate to from the main menu, the music will play from the beginning, and will transition to the first loopable section after the intro finishes. If the first loopable section finishes playing and the player is still on the first couple levels of the chapter, the music jumps back to the beginning of the first looping section. If the player is further along in the chapter, the music continues forward through the composition. This was the core functionality I needed to build.

One of the first things this approach meant for implementation was that I had to create loops manually, rather than using Game Maker’s built in ability to loop played audio clips. This is because, at the moment the game starts playing an audio clip, it is uncertain whether or not it will loop or transition to the next clip when it finishes. In code, I check this by setting a variable “playback_time” to 0 when the music starts, and increment it by delta_time every frame. Each frame, I also check if playback_time has exceeded the length of the clip. If so, I check which level the player is playing within the chapter; if they are far enough along, the next music section plays, and if not, the same one that just finished is played again. Here’s an example of what it looks like in GML (from Chapter 1):

(This case is nested in “switch global.song_section”)

postmortem code 01


case 2: // music section 2 - first section, flute lead
        if (playback_time >= ((audio_sound_length(music_current) * 1000000 ) - latency)){
            if global.current_level >= 2{ // transition if level 3 or higher
                playback_time -= music_length;
                audio_play_sound(snd_ch1_pt2_post_ogg,10,0);
                music_current = audio_play_sound(snd_ch1_pt3_ogg,10,0);
                music_length = (audio_sound_length(music_current) * 1000000 );
                global.song_section = 3;
                cymbal_transition = 1;
            }
            else{ // just loop (no transition)
                playback_time -= music_length;
                audio_play_sound(snd_ch1_pt2_post_ogg,10,0);
                music_current = audio_play_sound(snd_ch1_pt2_ogg,10,0);
                music_length = (audio_sound_length(music_current) * 1000000 );
            }
        }
        break;

There are a few other things here I haven’t mentioned yet. First of all, there’s a line that plays an additional audio file to the one that is being either looped or transitioned to (line 37). In this case, regardless of whether or not the song is looping the current section or transitioning to the next, the clip “snd_ch1_pt2_post_ogg” is being played at the same time as the next music section. This is the “post” audio from the music section 2 loop, which is essentially the fade out or tail after the moment the loop resets or transitions (Wwise calls this “post-exit”). And if you’re wondering about “pre-entry” as well, I’m about to talk about that line that says “cymbal_transition = 1;”

If you listen to the soundtrack, at about 1:47 into the music for Chapter 1 you’ll hear the cymbal transition refered to in that line of code. It starts before the end of the measure that transitions to the next section of the song, so this had to be handled differently in code. If the player is far enough along in the chapter to transition to the next song section, the cymbal transition plays, and shortly thereafter, the music transitions to the appropriate next section. Otherwise, the cymbal transition doesn’t play and the section loops again. Here it is in GML, inside the next case of the switch statement from above (the cymbal transition starts at line 69):

postmortem code 02


    case 3: // music section 3 - first section, cello lead
        if (playback_time >= ((audio_sound_length(music_current) * 1000000 ) - latency)){
            if global.current_level >= 4{
                playback_time -= music_length;
                audio_play_sound(snd_ch1_pt3_post_ogg,10,0);
                music_current = audio_play_sound(snd_ch1_pt4_ogg,10,0);
                music_length = (audio_sound_length(music_current) * 1000000 );
                global.song_section = 4;
            }
            else{ // just loop (no transition)
                playback_time -= music_length;
                audio_play_sound(snd_ch1_pt3_post_ogg,10,0);
                music_current = audio_play_sound(snd_ch1_pt3_ogg,10,0);
                music_length = (audio_sound_length(music_current) * 1000000 );
                cymbal_transition = 1;
            }
        }
        // cymbal transition
        if ((audio_sound_get_track_position(music_current) >= 33.96) && (cymbal_transition == 1)){
            if global.current_level >= 4{
                audio_play_sound(snd_ch1_pt3_trans,10,0);
                cymbal_transition = 0;
            }
            else{
                cymbal_transition = 0;
            }
        }
        break;

The audio clip “snd_ch1_pt3_trans” is the soloed cymbal transition. The value “33.96” is the time offset (in seconds) of when the cymbal transition starts relative to the start time of the current music clip. (The music clip being played, snd_ch1_pt3_ogg, is longer than 33.96 seconds.) Technically it’s possible to press the “continue” button at the end of the level at just the right time and hear the music transition to the next section without the cymbal transition, but it’s really hard to get it to happen, so I don’t think it’s a big deal.

So, at this point, this is essentially all of the core functionality of the music system for Zen Balls. It decides at the end of a music clip whether to loop or transition to the next clip, it includes the current clip’s “post-exit” audio regardless of looping or transitioning, and it also has the ability for a “pre-entry” to start shortly before its corresponding transition. This is the bare minimum of the horizontal approach, and it doesn’t include any of the functionality of the vertical approach. So now, I’m going to talk about what went right, what went wrong, and what I will do differently in the future.

What went right:

It works! (For the most part.) The music is genuinely interactive; the features I wanted to build are functioning as well as I can expect them to; and I made something on my own that I never thought would be good enough to ship, despite it not being as good as I wish it was. The system admittedly has a lot of bugs, but ultimately they are either far less common on newer devices, something I could never fix, or “edge cases” insofar as they only show up when exiting and reopening the app. I’ll talk about bugs in the “what went wrong” section, which will inevitably be much longer than the “what went right” section, but I guess the “what went right” section really just boils down to the fact that the “what I built” section actually works at all. At the end of the day, at least I was able to create something that I’m happy to show other people, even if I feel like a race car that barely limped over the finish line in last place with a missing wheel and a smoking engine.

What went wrong:

So many bugs! While I am genuinely satisfied with the overall state of the final game and its music system, it is not without its (many) problems. For example, one of the worst bugs I have seen but only superficially diagnosed has to do with exiting the app to the home screen and opening it again. Often, multiple instances of the current song section will play simultaneously, offset from each other by a short amount of time. It’s super irritating, and basically requires the user to quit the game and restart it in order to get it back to normal.

The other big bug I’m aware of is unfortunately probably common for most users, and something I think is literally out of my control: there are often clicks and pops between different music clips. Ultimately I think this is simply because GML isn’t “low level” enough as a code language (I barely know what that means, but bear with me).

My music scripts run in the step event of my audio manger game object. This means that the code I wrote is executed every frame the game is running. To the best of my knowledge, this is the most frequently that code can be executed in GML. Audio, however, includes an amount of information that must be processed much more often than every frame of the game in order for it to sound the way it is supposed to. Out of my own naivete, Zen Balls has a target frame rate of 60 frames per second (30 is much more common, and is probably what I should have used for performance reasons), which means that in a best case scenario, my music scripts are being executed every 1/60th of a second. For the sake of comparison, an audio clip recorded at CD quality (44100 Hz) has discrete samples every 1/44100th of a second. The game isn’t using WAV files at runtime (it uses ogg), but this comparison is still a good example of why a script that runs every frame will always be insufficient for consistently producing transitions or loops that are perceptibly seamless. There is almost always a guaranteed difference between the playback point of the audio clip and the moment the next frame is executed.

This is really the reason I don’t ever want to write a music system this way again. Middleware like Wwise and FMOD are built to process audio on that scale in order to make all playback completely seamless. As far as I know, that simply can’t be done in GML alone. And regardless of whether or not it’s possible, building a system from scratch is ton of work just to get functionality already provided by other software. It also forces unnecessary compromises of creativity insofar as I am far more likely to not do something creative with the composition or playback if I need to build additional functionality in order to implement it.

In the future:

I will almost certainly be using middleware for my next project. I really don’t ever want to have to do this again, but I had to in order to finish this game. I will likely also start using Unity for personal projects as well, partly because of middleware integration, but also out of an interest in making 3D projects. In a lot of circumstances, I enjoy composing within the constraints of limitations, but if the limitation is my ability to program something I’m vastly underqualified to program, it’s a lot more frustrating.

Thanks for reading. If you have any questions for me about anything I mentioned (or not) here, feel free to contact me through my website.