[Blog] Fantastic protocols and where to stream them
Hello everyone, Balder here. As mentioned by Carina I will talk about streaming protocols and their usage in streaming media. As the list can get quite extensive I will only focus on protocols in common use that handle both video and audio data.
Types of protocols
To keep it simple, a protocol is the method used to send/receive media data. You can divide the protocol in two types: - Stream based protocol - File based protocol
Stream based protocol
Stream based protocols are true streaming protocols. It has two-way communication between the server and the client and maintain an open connection. This allows for faster, more efficient delivery and lower latency. These properties make it the best option, if available. On the downside you need a player that supports the protocol.
File based protocol
File based protocols use media containers to move data from one point to another. The two main methods within file based protocols are progressive delivery and segmented delivery. Progressive delivery has shorter start times and latencies, while segmented delivery has longer start times and latencies.
With progressive delivery a single file is either simulated or present and transmitted in one large chunk to the viewer. The advantage of this is that when downloaded it becomes a regular file, the disadvantage is that trick play (seeking, reverse playback, fast forward, etc.) is only possible if the player supports it for the container format being used.
With segmented delivery a single multiple smaller files are transmitted to the viewer in a smaller chunk each. The advantage of this is that trick play is possible even without direct support for the container and it is more suited to infinite duration streams, for example live streams.
Short summary per protocol
Real-Time Messaging Protocol, also known as RTMP, is used to deliver Flash video to viewers. One of the true streaming protocols. Since Flash is on the decline it’s less used as delivery method to viewers and more as a streaming ingest for streaming platforms.
Real Time Streaming Protocol, also known as RTSP. The oldest streaming protocol, in its prime it was the most used protocol for media delivery. Nowadays it sees most of its use in IoT devices such as cameras and has uses for LAN broadcasts. Internally it uses RTP for delivery. Because of its excellent latency and compatibility with practically all codecs it is still a popular choice.
Web Real-Time Communications, also known as WebRTC. This is a set of browser APIs that make them compatible with secure RTP (SRTP). As such it has all the properties of RTSP with the added bonus of browser compatibility. Because it is still relatively new it hasn’t seen much use yet, but there is a good chance that this will be the most prevalent protocol in the near future.
Progressive file delivery
Flash: Progressive FLV
Flash Video format, also known as FLV. It used to be the most common delivery format for web-based streaming, back when Flash was the only way to get a playing video in a browser. With the standardization of HTML5 and the decline of Flash installations in browsers it is seeing decreasing use.
MPEG-4, also more commonly known as MP4. Most modern devices and browsers support MP4 files, which makes it an excellent choice as protocol. The particularities of the format make it relatively high overhead and complicated for live streams, but the excellent compatibility still makes it a popular choice.
MPEG-Transport Stream (MPEG-TS)
Also known as MPEG-TS. It is the standard used for Digital Video Broadcast (DVB), the old-and-proven TV standard. Because it was made for broadcast it is extremely robust and can handle even high levels of packet loss. The downside to this is almost 20% of overhead.
Ogg was one of the first open source and patent unencumbered container formats. Due to the open nature and free usage it has seen some popularity, but mostly as an addition to existing solutions because the compatibility is not wide enough to warrant it being used as the only delivery method.
Also known as MKV. It is a later open source container format that enjoys widespread adoption, mainly because of its excellent compatibility with almost all codecs and subtitle formats in existence. Because of the wide compatibility in codecs it is very unpredictable whether it will play in-browser however.
WebM is a subset of Matroska, simplified for use on the web. Made to solve the browser compatibility issues of Matroska, it plays in almost all modern browsers. The downside is that the codecs are very restricted.
Segmented file delivery
HTTP Smooth Streaming (HSS)
Also known as HSS. Created by Microsoft as adaptive streaming solution for web browsers. The only downside is that it has no support outside of Windows systems, and is even dropped in their latest browser Edge in favor for HLS.
HTTP Live Streaming (HLS)
Also known as HLS. This uses segmented TS files internally and is the streaming protocol developed for iOS devices. Due to the many iOS devices it sees widespread use. The biggest downsides are the high overhead and high latency.
Flash: HTTP Dynamic Streaming (HDS)
HTTP Dynamic Streaming, also known as HDS. This uses segmented F4S (FLV-based) files internally and is the last Flash protocol. It was created as a response to HLS, but never saw widespread use.
Dynamic Adaptive Streaming over HTTP (MPEG-DASH)
More commonly known as MPEG-DASH. It was meant to unify the splintered segmented streaming ecosystem under DASH, instead it standardized all of the existing protocols under a single name. Their current focus is on reducing complexity and latency.
Which protocol should you pick?
If available and an option you should always pick a stream based protocol as these will give you the best results. Sadly, most devices/browsers do not support most stream based protocols. Every device has its own preferred delivery format, which complicates matters.
Some protocols look like they’re supported by every device, so why not just stick to those? Well, every protocol also comes with their own advantages and disadvantages. Below is a table that attempts to clear up the differences:
|Protocol||Type||Platforms||Trick play||Latency||Overhead||Video codecs*||Audio codecs*||Status|
|RTSP||Streaming||Android, native players||Yes||Low||Low||Practically all||Practically all||Legacy|
|WebRTC||Streaming||Browsers||Yes||Low||Low||H264, VP8, VP9||Opus, AAC, MP3||Active development|
|FLV||Progressive file||Flash||No||Low||Medium||H264||AAC, MP3||Legacy|
|MP4||Progressive file||Browsers, native players||No||Low||High||H264, HEVC, VP8, VP9||AAC, MP3||Maintained|
|MPEG-TS||Progressive file||TV, native players||No||Low||Very high||H264, HEVC||AAC, MP3, Opus||Maintained|
|Ogg||Progressive file||Browsers, native players||No||Low||Medium||Theora||Opus, Vorbis||Maintained|
|Matroska||Progressive file||Native players||No||Low||Low||Practically all||Practically all||Maintained|
|WebM||Progressive file||Browsers, native players||No||Low||Low||VP8, VP9||Opus, Vorbis||Active development|
|HSS||Segmented file||Scripted players, Silverlight||Yes||High||High||H264, HEVC||AAC, MP3||Legacy|
|HLS||Segmented file||iOS, Safari, Android, scripted players||Yes||Very high||Very high||H264, HEVC||AAC, MP3||Active development|
|HDS||Segmented file||Flash||Yes||High||Medium||H264||AAC, MP3||Legacy|
|DASH||Segmented file||Scripted players||Yes||High||Varies||Varies||Varies||Active development|
*Only codecs still in common use today are listed.
In the end you can pick your preferred protocol based on the features and support, but to truly reach everyone you will most likely have to settle for a combination of protocols.
Relating all of the above to MistServer, you can see that we are still missing some of the protocols mentioned above. We are adding support for those in the near future. For the next blog post Erik will write about AV1, the upcoming (open) video codec.