Streaming Video and Audio with Low Delay

Data sent over the Internet will traverse a large number of smaller links. The combination of these links forms a path from the sender to the receiver. Sometimes data is lost when it traverses this path. For video and audio streams such losses can lead to playback glitches if not corrected.

A common solution to this problem is to use a reliable protocol like TCP. With TCP losses are corrected by retransmitting lost packets. The worst-case delay of such approaches will always be high or even unbounded. Hence, in applications where low delay is paramount, approaches based on retransmissions are problematic.

In this post we will go through the basic operations of the industry standard SMPTE 2022-1-2007, which instead corrects packet loss using proactive redundancy. We will look at the protection it offers against packet losses and show how this performance can be significantly improved simply by swapping the underlying algorithm with another - improving the protection against random losses by 100% or more without increasing the delay or overhead.

SMPTE 2022-1-2007

The SMPTE 2022-1-2007 standard entitle Forward Error Correction for Real-Time Video/Audio Transport Over IP Networks is a forward error correction (FEC) scheme that is used in the broadcast industry today. It uses an erasure correcting code (ECC) that is based on combining previously sent packets using the XOR operation and it can operate in two modes:

1. The single FEC mode provides weaker protection witha lower overhead.

2. The dual FEC mode provides stronger protection with higher overhead.

The SMPTE standard mandates that the transmitted packets are arranged in a matrix with L columns and D rows.

In the following, four different FEC modes will be presented. This includes the two modes specified in the SMPTE standard and two extensions which uses a Random Linear Network Code (RLNC). Each mode will be shown using an example configuration with L=5 columns and D=4 rows. For each mode, you can click on a packet to see which FEC packet(s) provide protection, or select an FEC packet to see which packets are protected by it.

Single FEC mode

In the single FEC mode, the D packets in each column are combined to create an FEC packet, for a total of L FEC packets.

The standard mandates that these column FEC packets are sent between L and L⋅D packets after the last protected packet. This means that the minimal worst-case delay is equal to L⋅D. This approach can cope with a single burst loss of up to any L consecutive packets. The scheme protects against any single random loss, since each protected packet is included in a single FEC packet. The number of FEC packets is L.

Dual FEC mode

The dual FEC mode, extends the single FEC mode by also generating FEC packets from each of the D rows, by combining the L packets in each row.

Jointly the two FEC streams can cope with burst losses of up to L + 1 consecutive packets, because a burst of L + 1 packets will always cover at least L columns and 1 row. In combination, the row and column FEC packets protect against any 2 random packet losses, because each protected packet is a part of 2 FEC packets. The number of FEC packets is L + D.

RLNC FEC

When each packet is only protected by few FEC packets, the protection against random losses will suffer as a consequence. Fortunately, with a modern code like RLNC each FEC packet can be a combination of all D⋅L packets. For this to work, RLNC uses a larger finite field, instead of SMPTE’s XOR operation.

If L FEC packets are generated from this set, they can recover any L consecutive burst losses. Additionally, any L random losses can be repaired, since all protected packets are included in all L FEC packets. The number of FEC packets is L.

Single FEC mode plus RLNC FEC

The single/dual FEC mode of 2022 supports receivers with different capabilities, by having two separate FEC streams such that receivers can either support only column FEC packets, called Level A devices, or both column and row FEC packets, called Level B devices. Similarly, an RLNC FEC stream could be added in addition to the column FEC stream which would retain compatibility with existing A and B devices and provide better protection for devices supporting the RLNC stream.

This approach is effective against L+D burst losses, since the column FEC packets protect against any burst of length L and the D RLNC FEC packets protect against any D losses. Any D+1 random losses will at least cover two different columns, therefore any D+2 random losses can be recovered: D by RLNC FEC packets and 2 by column FEC packets. The number of FEC packets is L+D.

Comparison of FEC modes

Based on the derived properties we have:

	Burst Loss Correction	Random Loss Correction	Overhead
2022 single mode	$L$	$1$	$L$
2022 dual mode	$L+1$	$2$	$L + D$
RLNC	$L$	$L$	$L$
2022 single mode + RLNC	$L+D$	$D+2$	$L + D$

For the example with L=5 columns and D=4 rows this becomes:

	Burst Loss Correction	Random Loss Correction	Overhead
2022 single mode	$5$	$1$	$5$
2022 dual mode	$6$	$2$	$9$
RLNC	$5$	$5$	$5$
2022 single mode + RLNC	$9$	$6$	$9$

As seen in the table above when using an RLNC code we gain:

• a stronger random loss correction,

• a stronger burst loss correction, and

• no additional packet overhead.

In the above comparison, the parameters are chosen such that the delay and packet overhead are identical to that of the 2022 scheme, with the gain used to improve burst and random loss protection. Alternatively, the gain could be used to improve the delay and/or packet overhead at the same loss protection.

Better codes for better quality

The SMPTE 2022-1-2007 defines a relatively sparse code using the XOR operation to provide protection of packets. This scheme is relatively simple, but unfortunately offers fairly weak protection, especially against random losses. A modern FEC code such as RLNC can offer much stronger protection at the same cost in packet overhead and delay.

When using FEC, a useful feature is the ability to react to changing channel conditions which will lead to varying packet losses. With 2022-1-2007 the amount of protection is fixed based on D and L, and one cannot simply discard some of the FEC packets at the sender, since this would strongly influence the burst loss protection. With RLNC any number of FEC packets can be generated and this number can be adjusted at any time, hence any level of protection and overhead can be used.

When arranging the packets to be protected in a matrix, as in 2022-1-2007, the number of packets that are protected (and consequently the delay) is proportional to the product of the number of columns and rows. Consequently, the design flexibility is reduced, especially when the delay requirement is strict, since here the dimensions of the matrix will be small. In the case of the RLNC code, there is no need to arrange the packets in a matrix, instead it suffices to define over how many packets coding is performed, providing much greater design flexibility. Similarly, for the single FEC mode + RLNC it would be possible to choose L to be small and instead utilize the better performance of the RLNC FEC stream.

With the RLNC code, FEC packets can be generated at any time, and not just when a row or column has been completed. For example, FEC packets can be generated when there is spare capacity available and the sending rate can be kept constant if so desired.

Evaluate your own gain

If you are interested in improving your products’ protection against losses, you can make your own comparison using the interactive illustration below and contact us if you have any questions.

Rows: 4

Cols: 5

FEC: 2022 single mode 2022 dual mode RLNC 2022 single mode + RLNC