Rhythm Quest Devlog 4 — Music/Game Synchronization

Background

The art of synchronizing music to gameplay is a bit of an arcane one, with numerous compounding factors and difficulties interfering with what seems like a simple concept. As much as you’d like to just “play the music and start the game at the same time”, audio buffers, DSP timing, input latency, display lag, timing drift, and other things need to be considered in order to have a robust approach. The exact nature of these problems and their corresponding solutions are also engine-specific most of the time.

A Naive Approach

An initial stab at creating music/gameplay synchronization might be to do something like this:

void Update() {
player.transform.position.x = audioSource.time * SOME_SPEED_CONSTANT;
}

Using PlayScheduled

Unity exposes an AudioSettings.dspTime value which will return the current time from the audio system’s point of view. From here on out I’ll refer to this as “DSP Time”, as opposed to “Game Time” which is the frame-based timing that you’d get from Unity’s Time.unscaledTime value. If you ever get confused, remember that for the most part, doubles are DSP time and floats are game time.

void Start() {
_audioDspStartTime = AudioSettings.dspTime + kStartScheduleBuffer;
_musicSource.PlayScheduled(_audioDspStartTime);
}

Mapping from DSP Time to Game Time

Unfortunately, there’s no “exact” way to map from a DSP timestamp to a game time value, since the two systems update at different intervals. However, we can get pretty close by using a linear regression. The Twitter thread at https://twitter.com/FreyaHolmer/status/845954862403780609 contains a graph illustration of what this looks like, if that helps.

void Update() {
float currentGameTime = Time.realtimeSinceStartup;
double currentDspTime = AudioSettings.dspTime;
// Update our linear regression model by adding another data point.
UpdateLinearRegression(currentGameTime, currentDspTime);
}
public double SmoothedDSPTime() {
return Time.unscaledTimeAsDouble * _coeff1 + _coeff2;
}
public double SmoothedDSPTime() {
double result = Time.unscaledTimeAsDouble * _coeff1 + _coeff2;
if (result > _currentSmoothedDSPTime) {
_currentSmoothedDSPTime = result;
}
return _currentSmoothedDSPTime;
}

Putting it together

We now have an AudioDspTimeKeeper.SmoothedDSPTime() function that will give us the (smoothed) current audio DSP time for the current frame. We can now use this as our timekeeping function, in conjunction with our _audioDspStartTime that we set when we first scheduled the backing music track:

double GetCurrentTimeInSong() {
return AudioDspTimeKeeper.SmoothedDSPTime() - _audioDspStartTime;
}
void Update() {
player.transform.position.x = GetCurrentTimeInSong() * SOME_SPEED_CONSTANT;
}

Latency compensation

Adding latency compensation into the mix is actually really easy! We can add it here:

double GetCurrentTimeInSong() {
return AudioDspTimeKeeper.SmoothedDSPTime() - _audioDspStartTime - _latencyAdjustment;
}

Resynchronization Fallback

In theory and in practice, everything above works just great. …as long as nothing goes wrong.

  • Audio buffer underruns if the audio buffer size is too low to mix everything in time
  • Some sort of CPU spike which causes the application to lag
  • The application could be forcibly paused — for example, this happens when dragging/resizing applications around the desktop in Windows.
  • The application could be minimized, or even backgrounded on a mobile device
  • etc…
void CheckForDrift() {
double timeFromDSP = AudioDspTimeKeeper.SmoothedDSPTime() - _audioDspStartTime;
double timeFromAudioSource = _musicSource.time;
double drift = timeFromDSP - timeFromAudioSource; if (Mathf.Abs(drift) > kTimingDriftMargin) {
Debug.LogWarningFormat("Music drift of {0} detected, resyncing!", musicDrift);
_audioDspStartTime += musicDrift;
}
}

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store