Lip Sync and Delay Equalization in TesiraLUX
Synchronization of audio with the presentation of video is a critical part of any AV installation. If the two are not properly aligned, content shown on your high-profile AV installation might look like a bad kung fu film! This article will highlight the basic behavior of TesiraLUX with respect to maintaining lip sync across the entire AV system.
Lip sync
Lip-sync is the timing offset between the video and audio signals. There are often differences in processing time between the audio components and video components. It is relatively “cheap” to delay audio and relatively “expensive” to delay video due to the buffer size required for each. When we speak of “lip-sync issues” we are typically talking about the misalignment of audio and video.
The speed of light is much faster than the speed of sound. From our experience of the natural world, we know that we'll always see a distant flash of lightning before we hear the sound of thunder. In other words, our brains are used to seeing things happen slightly before hearing them. However, hearing something slightly before seeing it is unnatural and much more jarring to the senses.
Audio and video are processed separately by the brain. Studies published in ITU-R publication BT.1359 have shown that audio can lead video by up to 45 msec or lag video by up to 125 msec before it is noticeable to most people. The values are asymmetrical because the brain doesn't like audio arriving before the video (eg - the thunder before the lightning).
As a functional specification, TesiraLUX calls for the presentation of audio relative to video to be between 15 milliseconds advanced (leading) and 25 milliseconds delayed (lagging) relative to the video (often stated as a range of +15 ms to -25 ms of video-to-audio delay.) This means that the audio will not shift relative to video beyond the +15ms to -25ms range within the TesiraLUX environment.
In audio systems, timing is typically measured in milliseconds (ms or msec). In video systems, timing is often measured in frames per second. The frame rate determines the duration of a single video frame. In TesiraLUX, a frame can be as short as 17 msec (1 frame at 60 Hz frame rate) and it can be as long as 67 msec (1 frame at 15 Hz frame rate).
For reference, a few common frame-to-millisecond equivalent values are:
- 1 frame at 60 Hz frame rate = 1000 msec / 60 fps = 16.6667 msec per frame
- 1 frame at 50 Hz frame rate = 1000 msec / 50 fps = 20 msec per frame
- 1 frame at 30 Hz frame rate = 1000 msec / 30 fps = 33.3333 msec per frame
- 1 frame at 15 Hz frame rate = 1000 msec / 15 fps = 66.6667 msec per frame
Note the use of the term Hertz, meaning "cycles per second".
In video Hertz is used to describe frame rate or frames per second. A 60 Hz frame rate equates to 60 frames per second. (Frame rate is not the same thing as the refresh rate of a monitor, although both are described in Hertz.)
In audio Hertz is used to describe frequency or vibrations per second, for example, 1,000 Hz equals 1,000 vibrations per second.
System latency and delay equalization
System latency is not simply the shortest time it takes to process or render audio through the DSP components. Additional delay may need to be added to allow devices to maintain synchronization of various audio and video signals. Tesira software handles these calculations automatically. There are user-definable constraints which can be applied to guide the software in making the correct decisions for your application.
DSP processing incurs latency which may vary by the complexity of the signal paths through the processors. Tesira’s Delay Equalization works to maintain the synchronization of related input signals – those which would be interpreted by the audience as needing to be correlated – such as video with accompanying audio or a set of microphones in a room. Delay Equalization also synchronizes multiple outputs which will require synchronous playout – whether video with audio, multiple video outputs in a common space, multiple audio outputs to a common space, or any combination of these.
Delay Equalization is an automatic function of the Tesira compilation engine, it places delay elements in the file to align related I/O blocks. Specific delay equalization values are not editable by the user.
Latency for system elements that are external to the Tesira system (e.g., camera latency, external digital media networks, etc.) cannot be factored into Tesira system latency and will need to be manually calculated.
Implicit AVB streams are included in the latency calculations, since Tesira knows everything it needs to know to calculate the latencies involved in the network transmission.
Tesira partition connectors
Tesira layout files allow the use of one or more partitions. Tesira audio and video each use different types of partitions with unique behaviors: audio partitions allow DSP blocks to process the audio, video partitions manage AV streams between devices. Changes in audio or video routes between partitions may necessitate re-calculating system Delay Eq either in whole or in part.
Within audio-only system files, AVB streams are fixed, the route times are predetermined, and Delay Eq times are unchanging once the configuration is loaded.
Within TesiraLUX systems video source resolution, compression rates, and frame rates can be altered during run time and streams can be re-routed on demand – these changes will create new AVB streams and / or require new Delay Eq times. Related audio signals always need to remain in sync with the video component. Tesira calculates and automatically applies necessary changes dynamically as video is modified or re-routed.
The dynamic delay equalization is implemented seamlessly through the use of auxiliary audio delay on the AV input and AV output devices. Delay of up to 64ms (2 frames at 30Hz, 4 frames at 60 Hz) is dynamically applied at the appropriate locations when sources are changed. This dynamic delay is managed by the system and is not user definable.
Delay Equalization Groups
Delay Equalization Group tags allow users to place input and output blocks into groups to manage their time alignment behavior.
When adding an input or output block (audio or video), Tesira software will apply a "Default" Delay Equalization Group tag. A set of blocks within the same Group will have guaranteed time alignment of their input or output presentation.
Only input and output blocks have Delay Eq Group tags as system entry and egress are where the time alignment requirement is defined.
- For video blocks the tag “Default Video” will be applied.
- For analog audio blocks the tag “Default” or “Default AEC” will be applied.
- For audio Network I/O and USB blocks the tag “None” will be applied.
Assigning an input block to the same Group as an output block is allowed, but internally they will be treated as different Groups. Input signals are only synchronized with other input signals, and output signals are only synchronized with other output signals.
User-defined Delay Eq Groups
Users can define additional Delay Eq Groups in the System > Delay Equalization Groups… menu. (Type a new name in the “Enter Group Name:” field then click Add.) User-defined Delay Eq Group tags are applied in the block’s Properties > DSP Properties tab under Delay Eq Group.
Blocks placed in the same user-defined Group will always be synchronized, regardless of Delay Eq Mode (described below) or whether they interact with one another (via connecting signal paths in the DSP layout).
User-defined Delay Eq Groups are a very powerful tool for customizing the behavior of your TesiraLUX system.
Delay Equalization Modes
Delay Equalization Modes are used to manage behavior of Delay Eq Groups within every Tesira file document. The settings are accessed via the Tools > Options > Document Settings... menu, under the General Settings sub-page.
Mode choices are Global, Per Partition, or Smart. These modes define how Delay Equalization Groups in different partitions interact with one another and thus how the inputs and outputs are managed for the system as a whole.
These modes have no effect on the synchronization of signals that are in the same partition; it only affects signals that are in different partitions. Additionally, Delay Equalization Modes only affect the behavior of I/O blocks that are assigned to the “Default” Delay Equalization Groups (Default, Default AEC, Default Video).
I/O blocks that are assigned to the same user-defined Delay Equalization Group will always be synchronized, regardless of the currently selected Mode. I/O blocks that are assigned to the “None” group will not be synchronized in any Mode.
Global
When the document Delay Equalization Mode is set to “Global” then Tesira will effectively treat all partitions as one common shared space. It ensures that all I/O blocks in the default Delay Equalization Group are synchronized, even if they are in different partitions.
Per Partition
When the document’s Delay Equalization Mode is set to “Per Partition” then Tesira only applies delay equalization within individual partitions. The "Per Partition" Delay Equalization Mode ensures that signals in different partitions are not synchronized.
In "Per Partition" Mode I/O blocks in different partitions will not be synchronized with one another, even if they are all assigned to the same Default group. In other words, this mode only allows delay equalization to operate within single partitions. Note that I/O blocks that are in custom (non-default) groups will still be synchronized, even if they are in different partitions.
Smart
When the document Delay Equalization mode is set to “Smart”, I/O blocks that are in the Default group but in different partitions will be synchronized only if the signals associated with those I/O blocks are connected to a common mixer/router block at some point.
The Smart Delay Equalization Mode is the middle ground between the Global and Per Partition modes. If the signals from two I/O blocks don’t interact with one another, they will not be synchronized. Note that this mode only affects blocks in the Default group; blocks in the same user-defined group will always be synchronized, regardless of whether they interact with one another.
The Smart Delay Equalization Mode is the default document setting.
DSP Use
After compilation, Tesira reports the DSP use for a file in the DSP Resources window. This is a percentage of available DSP used by the layout you have created. The Scale percentage indicates a 0 to 100% utilization value of the DSP used based on the available DSP Resource in the device. The available DSP reported is influenced by the DSP Use slider found in Tools > Options > Document Settings > General Settings.
This slider manually ramps between 2 ends of a scale marked Conservative and Aggressive. Conservative indicates that the compiler should maintain a larger reserve of resources for delay equalization. Aggressive indicates that the compiler should maintain a smaller reserve of resources for delay equalization.
When changing the DSP Use settings, the actual processing resources of the device are unchanged, however, the reported percentage of available DSP used for the the layout is rescaled - reserving more or less DSP resources for Delay Equalization. In other words, this slider changes the balance of how much DSP power is reserved for delay equalization vs. how much is reserved for the DSP blocks in the layout. For a compiled file this will result in a change in the reported available DSP Usage by Device in the DSP Resources window. When DSP Use is set to the Conservative end of the scale Tesira maintains a larger DSP reserve for Delay Eq by reporting 100% DSP usage earlier. In systems which require less Delay Eq you can free up slightly more DSP resources for DSP processing objects by moving the slider to the Aggressive end of the scale.
Tesira's compilation engine first builds the structure of the file, connecting all of the signal paths end-to-end. It then goes through the layout and attempts to fit the Delay Equalization group and mode requirements to the compiled file. It is possible to run out of DSP resources at this time. The software will warn you that insufficient resources exist and that you may want to try changing the DSP Use settings.
In some particularly complex files the DSP required for audio processing components may still be well under the 100% usage but the software will warn that DSP resources are depleted and that users may be able to free up resources for Delay Equalization using the DSP Use slider. There are occasionally cases where files will not compile due to the demands of Delay Equalization - in these instances it is usually possible to redefine the I/O block Group settings to come up with a workable configuration.