On my way to finishing my PhD today I started with editing speech for my track 'Futures' – with that completed all editing should be done and I can move on to mixing. In the afternoon I worked a bit on the thesis commentary, finding a couple of dangling references (one regarding the digital radio switchover, the other about neural networks for music). But it has been the evening that has provided the greatest satisfaction, and to explain why I’m going to explain Futures EP a bit more.

Futures EP is what I call an 'open outcome record', a sonic medium that resembles a traditional recorded release in every way but for its ability to play tracks back differently on each repeat. This is done by having a partial backing track for each tune, and overlaying the remaining parts on top at run-time. This requires it to play back from a device that can accommodate this process, and I have chosen iOS as the platform here.

Having experimented with the various audio APIs in iOS I found that the higher level ones – AVAudioPlayer and Audio Queue Services – didn't provide the precise timing synchronisation that I needed (the only way of scheduling audio was through sloppy NSTimers); it was necessary to use a lower level Audio Units implementation for this reason, and I have managed to get this working pretty well.

The problem with this approach is that my audio files, which are stored in the app in compressed formats – MP3, AAC or CAF – are converted into a 32-bit sample type when loaded into the audio buffers. All of a sudden the 10.2MB backing track for the first tune, 'Tapes', is expanded to nearly 80MB in memory, and very soon the device's memory is filled (it is vital to test for this on a device as the simulator doesn’t replicate this limit sufficiently).

I considered two solutions: either load all my audio in chunks into many small buffers and rotate which one plays (this is like how Audio Queue Services works with 'enqueuing'), or find a way of loading the compressed files into memory and convert them on the fly. Either solution would be complicated to implement, especially the first. So I went online to figure out how to proceed.

On my travels I came across Cocoa with Love's History of iOS Media APIs and noticed, under the iOS 4 section, a framework I hadn't seen before: AVComposition, part of the AVFoundation framework. This API allows media assets to be organised in time using AVCompositionTrack objects and played back through an AVPlayer instance (I posit that this framework exists thanks to the iOS versions of iMovie and GarageBand).

So this evening I set up a test app to try and get audio playing and synchronised using this new approach. And it worked! I tried it in the simulator and on the iPad, and in both the overlaid part was nicely synched with the backing, and with no need for AUGraphs and render callbacks, and with none of the memory issues.

As a comparison, my Audio Units app easily filled the 128MB memory limit of my iPad with just 'Tapes' (it crashes when I try to load the backing track of 'Futures' as well :-/). The AVPlayer test version I put together this evening (albeit without the fancy UI of the AU one) actively used only 0.75MB of memory when running. Now that's a solution. So this weekend I'm going to be rewriting the audio engine for Futures EP to make use of this new approach. :-)