How to Generate Music With High Time Accuracy

The main difficulties with Unity and generated music is the precision and accuracy in time of each notes. Human are able to distinguish delay between sound until 20 ms, below we hear only one sound. It’s obvious that accuracy in music is important !

Look at this post for more information about Midi Timing.

To get a good accuracy, the method to trigger notes is of course, crucial !

Which methods are available ?

From the worse to the best! if you are in a hurry, read only the 4th 😉

1. The least optimal method is using the classical Update() Unity method. Unity cannot guarantee stable call timing, resulting in an unstable music tempo.
2. Prefer FixedUpdate() to ensure a stable execution:
   • Consistent Timing: FixedUpdate() is called at regular, fixed intervals (defined by Time.fixedDeltaTime), ensuring consistent timing for your operations.
   • Frame Rate Independence: It runs independently of the frame rate, making it more reliable for tasks that need stable execution intervals.
   • Synchronization with Physics: Ideal for physics-related calculations, which also run in FixedUpdate().
   • But, be aawre of these consideration:
     • Performance Impact: Frequent updates (e.g., every few milliseconds) can impact performance, so it’s crucial to ensure that the operations inside FixedUpdate() are efficient.
     • Granularity: The interval of FixedUpdate() is set by Time.fixedDeltaTime. Ensure this interval aligns with your desired tempo. If the interval is too long, it may not suit very fast tempos.
     • Interleaving: Since FixedUpdate() can run multiple times per frame or not at all in some frames (depending on frame rate), ensure that your tempo calculations account for these variations.
3. A slightly better approach is using the coroutine API.
   WaitForSeconds can delay between beats, but it doesn’t wait exactly for the specified time; it checks each frame if the time has elapsed, introducing a margin of error equal to the frame duration. This is especially noticeable at low frame rates. See here order of execution for Unity event functions.
4. An almost acceptable method is using a C# thread, which was employed in previous MPTK demos. But you will not able to call Unity API, only Debug.Log and of course all the MPTK API 😉
5. This is why we introduced a new method in the MPTK PRO API: OnAudioFrameStart. This callback executes before processing each audio frame, thanks to the audio engine thread, ensuring precise timing. With the same limitation: you will not able to call Unity API, only Debug.Log and of course all the MPTK API 😉

OnAudioFrameStart our hero!

OnAudioFrameStart is an event defined in the lower-level MPTK Pro class, available for both MidiStreamPlayer and MidiFilePlayer. Typically, you’ll use the MidiStreamPlayer prefab to generate notes from your script.

Using the MPTK API, you can associate a callback function with this event. You are responsible for creating this callback function. The callback receives the synthesizer time in milliseconds as a parameter, which you can use to determine what action or note to play. Refer to the example below to learn how to convert this absolute time to beat time.

This event is triggered at very precise intervals, depending on the synthesizer rate and buffer size.” See prefabs inspector to change these values.

It’s easy to calculate the audio latency :

latency_ms = (BufferSize / SynthRate) * 1000

Synth Rate in Hertz	Buffer Size in byte	Delay between each audio frame in millisecond
96000	512	5,33
96000	1024	10,67
48000	256	5,33
48000	512	10,67
48000	1024	21,33
48000	2048	42,67
24000	1024	42,67

Important Notes (to be read !)

• It is not possible to call Unity API functions within callback functions (such as PlaysHits in this example) because callbacks run on a different thread than Unity, outside of the Unity lifecycle. An exception is Debug.Log, which can be useful, but please see the notes below.
• Processing within the callback must be as quick as possible; otherwise, the sound quality will suffer. Avoid costly processing inside the callback. Instead, use buffers, lists, or queues to communicate with the callback.
• Latency is not solely the responsibility of the audio engine; hardware, the UI, and, unfortunately, your code also play roles.
• On Android with Oboe, the audio engine latency is excellent: 5.33 ms. 🙂
• For iOS, we recommend selecting the best latency settings from the Unity menu.

Warning: with Mac M1 and M2 architecture, in some case the DSP buffer length is not a multiple of 64: no sound will be produced or error will be displayed. Try to change the ‘DSP Buffer Size’ to get a buffer size of 512 or 1024.

Code example

I encourage you to look at the demo EuclideanRhythm and the source code TestEuclideanRhythme.cs to see the full example (MPTK Pro only). Below some extract :

Define the callback function to be called at each audio frame :

void Start()
{
   BtPlay.onClick.AddListener(() =>
   {
      IsPlaying = !IsPlaying;
      Play();
   });
}

/// <summary>
/// Play or stop playing.
/// Set the PlayHits function to process midi generated music at each audio frame
/// </summary>
public void Play()
{
    lastSynthTime = 0f;
    timeMidiFromStartPlay = 0d;
    timeSinceLastBeat = 999999d; // start with a first beat
    
    if (IsPlaying)
       // Associate the callback function (PlayHits) to the event
       midiStream.OnAudioFrameStart += PlayHits;
    else
       // Remove association, the callback will not more called
        midiStream.OnAudioFrameStart -= PlayHits;
}

The PlayHits method

The PlayHits method is where you can generate midi note (it’s your code, you can named it as you want). It’s a callback function, normally you never call this method from your code (but it’s possible), the call is done back from the system 😉

How you generate these notes are out of the competency of MPTK: Markov chain, Star position (yes, that exist!), Procedural, … See here, the subject is huge ! I’m sure you will be creative !

/// <summary>
/// This callback function will be called at each audio frame.
/// The frequency depends on the buffer size and the synth rate (see inspector of the MidiStreamPlayer prefab)
/// Recommended values: Freq=48000 Buffer Size=1024 --> call every 11 ms with a high accuracy.
/// You can't call Unity API in this function (only Debug.Log) but the most part of MPTK API are available.
/// For example : MPTK_PlayDirectEvent or MPTK_PlayEvent to play music note from MPTKEvent (see PlayEuclideanRhythme)
/// </summary>
/// <param name="synthTimeMS"></param>
private void PlayHits(double synthTimeMS)
{
    if (lastSynthTime <= 0d)
    {
        // First call, init the last time
        lastSynthTime = synthTimeMS;
    }

    // Calculate time in millisecond since the last call
    double deltaTime = synthTimeMS - lastSynthTime;
    lastSynthTime = synthTimeMS;
    timeMidiFromStartPlay += deltaTime;

    // Calculate time since last beat played
    timeSinceLastBeat += deltaTime;

    /// SldTempo in BPM.
    ///  60 BPM means 60 beats in each minute, or 1 beat per second.
    /// 120 BPM would be twice as fast: 120 beats in each minute or 2 per second.
    /// Calculate the delay between two quarter notes in millisecond
    CurrentTempo = (60d / SldTempo.Value) * 1000d;

    // Is it time to play a hit ?
    if (IsPlaying && timeSinceLastBeat > CurrentTempo)
    {
        //Debug.Log($"{synthMidiMS:F0} {midiStream.StatDeltaAudioFilterReadMS:F2} {deltaTime:F2} {timeSinceLastBeat:F2}");
        timeSinceLastBeat = 0d;

        // Create a random note. 
        MPTKEvent sequencerEvent = new MPTKEvent()
        {
            Channel = 0,
            Duration = 1000, // 1 second
            // use of System.Random to generate the note 
            // because Unity API can't be used outside the Unity Thread.
            Value = Rnd.Next(50,62), 
            Velocity = 100,
        };

        // Send the note to the MPTK synthesizer.
        midiStream.MPTK_PlayDirectEvent(sequencerEvent);
    }
}

Have Fun !