Adding redundancy to your Automated Playout

One day it’s going to happen, your primary Automation system will fail and hopefully your plan B (you do have a plan B don’t you?) will kick in and take over your station output, providing some branded output until you can fix whatever fault caused everything to fall over.

At Radio Royal we have a lot of pre-recorded content for when we are not live. Whilst we have relied on an Elan Audio REP-01 twinned with a Sonifex silence detector, that only gives us some basic song-song-jingle style backup. Now whilst that’s better than nothing, we also have some redundancies in our systems: we have primary and backup servers for our audio and SQL databases and have tools to keep them synchronised.

Audio is synchronised between the servers using Windows DFS (“Distributed File System”). For the Myriad SQL Databases, the primary server is set up as a “Publisher”, whilst the backup server is set up as a “Subscriber”. Any changes made on 1 server are written over to the other server in effectively real-time.

We already had the infrastructure to enable an entirely separate Automation systems running on the primary and backup servers. Our primary automation already ran on a virtualised server with Axia livewire used for the audio, although of course this could be any AES67 based AoIP solution which saves the need for hardware audio cards.

Given we have 2 Citrix XenServer boxes, we set up a backup ‘virtual’ Myriad playout instance on the second server. That instance of Myriad is set to use a child “Station” of the primary on air station all within the same Myriad Database. Every hour it will pull down the running order from its parent station which is the primary Station for the main automation instance. By doing it this way any voice tracks are also pulled over from the primary systems. As we are using SQL merge publication it’s not possible to use the main “station” in the database as it would be simultaneously in use with the primary Myriad playout instance and they would conflict with each other. This way, each Playout get’s it’s own copy of the log to work with.

Next task is to swap over to the backup instance if the primary playout fails. We use Axia Pathfinder at Radio Royal for all our studio and transmission management. Pathfinder can monitor any Livewire audio source for silence or clipping. We have some stack events in Pathfinder monitor the audio levels on the primary audio feeds, should that be silent for 15 seconds then it will switch the Automation feed to the backup playout as well as send out an email alert. Once audio returns it is swapped back to the primary system automatically.

And of course, if it all goes pear shaped and somehow both systems go down, then the Elan Audio and Sonifex boxes live to fight another day!