Hello everyone, Balder here. As mentioned by Carina I will talk about streaming protocols and their usage in streaming media. As the list can get quite extensive I will only focus on protocols in common use that handle both video and audio data.

Types of protocols

To keep it simple, a protocol is the method used to send/receive media data. You can divide the protocol in two types: - Stream based protocol - File based protocol

Stream based protocol

Stream based protocols are true streaming protocols. It has two-way communication between the server and the client and maintain an open connection. This allows for faster, more efficient delivery and lower latency. These properties make it the best option, if available. On the downside you need a player that supports the protocol.

File based protocol

File based protocols use media containers to move data from one point to another. The two main methods within file based protocols are progressive delivery and segmented delivery. Progressive delivery has shorter start times and latencies, while segmented delivery has longer start times and latencies.

Progressive delivery

With progressive delivery a single file is either simulated or present and transmitted in one large chunk to the viewer. The advantage of this is that when downloaded it becomes a regular file, the disadvantage is that trick play (seeking, reverse playback, fast forward, etc.) is only possible if the player supports it for the container format being used.

Segmented delivery

With segmented delivery a single multiple smaller files are transmitted to the viewer in a smaller chunk each. The advantage of this is that trick play is possible even without direct support for the container and it is more suited to infinite duration streams, for example live streams.

Short summary per protocol

Streaming protocols

Flash: RTMP

Real-Time Messaging Protocol, also known as RTMP, is used to deliver Flash video to viewers. One of the true streaming protocols. Since Flash is on the decline it’s less used as delivery method to viewers and more as a streaming ingest for streaming platforms.

RTSP

Real Time Streaming Protocol, also known as RTSP. The oldest streaming protocol, in its prime it was the most used protocol for media delivery. Nowadays it sees most of its use in IoT devices such as cameras and has uses for LAN broadcasts. Internally it uses RTP for delivery. Because of its excellent latency and compatibility with practically all codecs it is still a popular choice.

WebRTC

Web Real-Time Communications, also known as WebRTC. This is a set of browser APIs that make them compatible with secure RTP (SRTP). As such it has all the properties of RTSP with the added bonus of browser compatibility. Because it is still relatively new it hasn’t seen much use yet, but there is a good chance that this will be the most prevalent protocol in the near future.

Progressive file delivery

Flash: Progressive FLV

Flash Video format, also known as FLV. It used to be the most common delivery format for web-based streaming, back when Flash was the only way to get a playing video in a browser. With the standardization of HTML5 and the decline of Flash installations in browsers it is seeing decreasing use.

Progressive MP4

MPEG-4, also more commonly known as MP4. Most modern devices and browsers support MP4 files, which makes it an excellent choice as protocol. The particularities of the format make it relatively high overhead and complicated for live streams, but the excellent compatibility still makes it a popular choice.

MPEG-Transport Stream (MPEG-TS)

Also known as MPEG-TS. It is the standard used for Digital Video Broadcast (DVB), the old-and-proven TV standard. Because it was made for broadcast it is extremely robust and can handle even high levels of packet loss. The downside to this is almost 20% of overhead.

Ogg

Ogg was one of the first open source and patent unencumbered container formats. Due to the open nature and free usage it has seen some popularity, but mostly as an addition to existing solutions because the compatibility is not wide enough to warrant it being used as the only delivery method.

Matroska

Also known as MKV. It is a later open source container format that enjoys widespread adoption, mainly because of its excellent compatibility with almost all codecs and subtitle formats in existence. Because of the wide compatibility in codecs it is very unpredictable whether it will play in-browser however.

WebM

WebM is a subset of Matroska, simplified for use on the web. Made to solve the browser compatibility issues of Matroska, it plays in almost all modern browsers. The downside is that the codecs are very restricted.

Segmented file delivery

HTTP Smooth Streaming (HSS)

Also known as HSS. Created by Microsoft as adaptive streaming solution for web browsers. The only downside is that it has no support outside of Windows systems, and is even dropped in their latest browser Edge in favor for HLS.

HTTP Live Streaming (HLS)

Also known as HLS. This uses segmented TS files internally and is the streaming protocol developed for iOS devices. Due to the many iOS devices it sees widespread use. The biggest downsides are the high overhead and high latency.

Flash: HTTP Dynamic Streaming (HDS)

HTTP Dynamic Streaming, also known as HDS. This uses segmented F4S (FLV-based) files internally and is the last Flash protocol. It was created as a response to HLS, but never saw widespread use.

Dynamic Adaptive Streaming over HTTP (MPEG-DASH)

More commonly known as MPEG-DASH. It was meant to unify the splintered segmented streaming ecosystem under DASH, instead it standardized all of the existing protocols under a single name. Their current focus is on reducing complexity and latency.

Which protocol should you pick?

If available and an option you should always pick a stream based protocol as these will give you the best results. Sadly, most devices/browsers do not support most stream based protocols. Every device has its own preferred delivery format, which complicates matters.

Some protocols look like they’re supported by every device, so why not just stick to those? Well, every protocol also comes with their own advantages and disadvantages. Below is a table that attempts to clear up the differences:

Protocol	Type	Platforms	Trick play	Latency	Overhead	Video codecs*	Audio codecs*	Status
RTMP	Streaming	Flash	Yes	Low	Low	H264	AAC, MP3	Legacy
RTSP	Streaming	Android, native players	Yes	Low	Low	Practically all	Practically all	Legacy
WebRTC	Streaming	Browsers	Yes	Low	Low	H264, VP8, VP9	Opus, AAC, MP3	Active development
FLV	Progressive file	Flash	No	Low	Medium	H264	AAC, MP3	Legacy
MP4	Progressive file	Browsers, native players	No	Low	High	H264, HEVC, VP8, VP9	AAC, MP3	Maintained
MPEG-TS	Progressive file	TV, native players	No	Low	Very high	H264, HEVC	AAC, MP3, Opus	Maintained
Ogg	Progressive file	Browsers, native players	No	Low	Medium	Theora	Opus, Vorbis	Maintained
Matroska	Progressive file	Native players	No	Low	Low	Practically all	Practically all	Maintained
WebM	Progressive file	Browsers, native players	No	Low	Low	VP8, VP9	Opus, Vorbis	Active development
HSS	Segmented file	Scripted players, Silverlight	Yes	High	High	H264, HEVC	AAC, MP3	Legacy
HLS	Segmented file	iOS, Safari, Android, scripted players	Yes	Very high	Very high	H264, HEVC	AAC, MP3	Active development
HDS	Segmented file	Flash	Yes	High	Medium	H264	AAC, MP3	Legacy
DASH	Segmented file	Scripted players	Yes	High	Varies	Varies	Varies	Active development

*Only codecs still in common use today are listed.

In the end you can pick your preferred protocol based on the features and support, but to truly reach everyone you will most likely have to settle for a combination of protocols.

Relating all of the above to MistServer, you can see that we are still missing some of the protocols mentioned above. We are adding support for those in the near future. For the next blog post Erik will write about AV1, the upcoming (open) video codec.