How to Generate Music With High Time Accuracy

The main difficulties with Unity and generated music is the precision and accuracy in time of each notes. Human are able to distinguish delay between sound until 20 ms, below we hear only one sound. It’s obvious that accuracy in music is important !

Look at this post for more information about Midi Timing.

To get a good accuracy, the method to trigger notes is of course, crucial !

Which methods are available ?

From the worse to the best! if you are in a hurry, read only the 4th 😉

The worse is to use the classical Update Unity method. Unity can’t guarantee that the call is stable. So the tempo of your music will note be stable !
The second method less worse is using the coroutine API. WaitForSeconds could be used to wait between each beat, but it doesn’t actually wait for the specified time instead it seems to check every frame whether or not the specified time has elapsed or not. That means there is always a margin of error roughly equal to the time between frames. On low frame rates this seems to be very evident.
See here order of execution for Unity event functions.
The third method almost acceptable is the use a C# thread. This method was used in the previous MPTK demo. But, it wasn’t so perfect.
Its’ the reason why we added a new method to the MPTK PRO API : OnAudioFrameStart. This callback will be run before processing each audio frame. Thank to the audio engine thread for doing the job.

OnAudioFrameStart our hero!

OnAudioFrameStart is an event defined in the low class of MPTK Pro, so it is available for these two MPTK classes : MidiStreamPlayer and MidiFilePlayer. Classically you will use MidiStreamPlayer prefab to generate notes from your script.

With the MPTK API you can associate a callback function to this event. Of course, you are responsible to build this callback function. This callback receive in parameter the synthesizer time in millisecond. You can use it to decide what action/note to done. See the example below to understand how to transform this absolute time in beat time.

This event is triggered at very precise time depending of the synthesizer rate and the buffer size. See prefabs inspector to change these values.

It’s easy to calculate the latency :

latency_ms = (BufferSize / SynthRate) * 1000

Synth Rate in Hertz	Buffer Size in byte	Delay between each audio frame in millisecond
96000	512	5,33
96000	1024	10,67
48000	256	5,33
48000	512	10,67
48000	1024	21,33
48000	2048	42,67
24000	1024	42,67

Calculation grid of latency vs synth setting (frequency and buffer size)

Important Notes (to be read !) :

It’s not possible to call Unity API function inside the callback functions (PlaysHits in the example). It’s because the callback is running in a different thread that Unity, so outside the Unity lifecycle. One exception for the Debug.Log, that could be useful but read the next notes.
It’s obvious that processing in the callback must be as quick as possible otherwise the sound will become disastrous. Try to keep outside the callback all costly processing and communicate to the callback with buffer, List<>, Queue<> …
The responsible of the Latency is not only the Audio engine! Its’ also the hardware and the UI …. and, sorry, your code!
On Android with Oboe, the latency with the audio engine is excellent : 5.33 ms 🙂
On iOS, we advice you to select the best latency from Unity menu

Warning: with Mac M1 and M2 architecture, in some case the DSP buffer length is not a multiple of 64: no sound will be produced or error will be displayed. Try to change the ‘DSP Buffer Size’ to get a buffer size of 512 or 1024.

Code example

I encourage you to look at the demo EuclideanRhythm and the source code TestEuclideanRhythme.cs to see the full example (MPTK Pro only). Below some extract :

Define the callback function to be called at each audio frame :

void Start()
{
   BtPlay.onClick.AddListener(() =>
   {
      IsPlaying = !IsPlaying;
      Play();
   });
}

/// <summary>
/// Play or stop playing.
/// Set the PlayHits function to process midi generated music at each audio frame
/// </summary>
public void Play()
{
    lastSynthTime = 0f;
    timeMidiFromStartPlay = 0d;
    timeSinceLastBeat = 999999d; // start with a first beat
    
    if (IsPlaying)
       // Associate the callback function (PlayHits) to the event
       midiStream.OnAudioFrameStart += PlayHits;
    else
       // Remove association, the callback will not more called
        midiStream.OnAudioFrameStart -= PlayHits;
}

Code language: C# (cs)

The PlayHits method

The PlayHits method is where you can generate midi note (it’s your code, you can named it as you want). It’s a callback function, normally you never call this method from your code (but it’s possible), the call is done back from the system 😉

How you generate these notes are out of the competency of MPTK: Markov chain, Star position (yes, that exist!), Procedural, … See here, the subject is huge ! I’m sure you will be creative !

/// <summary>
/// This callback function will be called at each audio frame.
/// The frequency depends on the buffer size and the synth rate (see inspector of the MidiStreamPlayer prefab)
/// Recommended values: Freq=48000 Buffer Size=1024 --> call every 11 ms with a high accuracy.
/// You can't call Unity API in this function (only Debug.Log) but the most part of MPTK API are available.
/// For example : MPTK_PlayDirectEvent or MPTK_PlayEvent to play music note from MPTKEvent (see PlayEuclideanRhythme)
/// </summary>
/// <param name="synthTimeMS"></param>
private void PlayHits(double synthTimeMS)
{
    if (lastSynthTime <= 0d)
    {
        // First call, init the last time
        lastSynthTime = synthTimeMS;
    }

    // Calculate time in millisecond since the last call
    double deltaTime = synthTimeMS - lastSynthTime;
    lastSynthTime = synthTimeMS;
    timeMidiFromStartPlay += deltaTime;

    // Calculate time since last beat played
    timeSinceLastBeat += deltaTime;

    /// SldTempo in BPM.
    ///  60 BPM means 60 beats in each minute, or 1 beat per second.
    /// 120 BPM would be twice as fast: 120 beats in each minute or 2 per second.
    /// Calculate the delay between two quarter notes in millisecond
    CurrentTempo = (60d / SldTempo.Value) * 1000d;

    // Is it time to play a hit ?
    if (IsPlaying && timeSinceLastBeat > CurrentTempo)
    {
        //Debug.Log($"{synthMidiMS:F0} {midiStream.StatDeltaAudioFilterReadMS:F2} {deltaTime:F2} {timeSinceLastBeat:F2}");
        timeSinceLastBeat = 0d;

        // Create a random note. 
        MPTKEvent sequencerEvent = new MPTKEvent()
        {
            Channel = 0,
            Duration = 1000, // 1 second
            // use of System.Random to generate the note 
            // because Unity API can't be used outside the Unity Thread.
            Value = Rnd.Next(50,62), 
            Velocity = 100,
        };

        // Send the note to the MPTK synthesizer.
        midiStream.MPTK_PlayDirectEvent(sequencerEvent);
    }
}

Code language: PHP (php)

Have Fun !