[Blog] Fantastic protocols and where to stream them
Hello everyone, Balder here. As mentioned by Carina I will talk about streaming protocols and their usage in streaming media. As the list can get quite extensive I will only focus on protocols in common use that handle both video and audio data.
Types of protocols
To keep it simple, a protocol is the method used to send/receive media data. You can divide the protocol in two types: - Stream based protocol - File based protocol
Stream based protocol
Stream based protocols are true streaming protocols. It has two-way communication between the server and the client and maintain an open connection. This allows for faster, more efficient delivery and lower latency. These properties make it the best option, if available. On the downside you need a player that supports the protocol.
File based protocol
File based protocols use media containers to move data from one point to another. The two main methods within file based protocols are progressive delivery and segmented delivery. Progressive delivery has shorter start times and latencies, while segmented delivery has longer start times and latencies.
With progressive delivery a single file is either simulated or present and transmitted in one large chunk to the viewer. The advantage of this is that when downloaded it becomes a regular file, the disadvantage is that trick play (seeking, reverse playback, fast forward, etc.) is only possible if the player supports it for the container format being used.
With segmented delivery a single multiple smaller files are transmitted to the viewer in a smaller chunk each. The advantage of this is that trick play is possible even without direct support for the container and it is more suited to infinite duration streams, for example live streams.
Short summary per protocol
Real-Time Messaging Protocol, also known as RTMP, is used to deliver Flash video to viewers. One of the true streaming protocols. Since Flash is on the decline it’s less used as delivery method to viewers and more as a streaming ingest for streaming platforms.
Real Time Streaming Protocol, also known as RTSP. The oldest streaming protocol, in its prime it was the most used protocol for media delivery. Nowadays it sees most of its use in IoT devices such as cameras and has uses for LAN broadcasts. Internally it uses RTP for delivery. Because of its excellent latency and compatibility with practically all codecs it is still a popular choice.
Web Real-Time Communications, also known as WebRTC. This is a set of browser APIs that make them compatible with secure RTP (SRTP). As such it has all the properties of RTSP with the added bonus of browser compatibility. Because it is still relatively new it hasn’t seen much use yet, but there is a good chance that this will be the most prevalent protocol in the near future.
Progressive file delivery
Flash: Progressive FLV
Flash Video format, also known as FLV. It used to be the most common delivery format for web-based streaming, back when Flash was the only way to get a playing video in a browser. With the standardization of HTML5 and the decline of Flash installations in browsers it is seeing decreasing use.
MPEG-4, also more commonly known as MP4. Most modern devices and browsers support MP4 files, which makes it an excellent choice as protocol. The particularities of the format make it relatively high overhead and complicated for live streams, but the excellent compatibility still makes it a popular choice.
MPEG-Transport Stream (MPEG-TS)
Also known as MPEG-TS. It is the standard used for Digital Video Broadcast (DVB), the old-and-proven TV standard. Because it was made for broadcast it is extremely robust and can handle even high levels of packet loss. The downside to this is almost 20% of overhead.
Ogg was one of the first open source and patent unencumbered container formats. Due to the open nature and free usage it has seen some popularity, but mostly as an addition to existing solutions because the compatibility is not wide enough to warrant it being used as the only delivery method.
Also known as MKV. It is a later open source container format that enjoys widespread adoption, mainly because of its excellent compatibility with almost all codecs and subtitle formats in existence. Because of the wide compatibility in codecs it is very unpredictable whether it will play in-browser however.
WebM is a subset of Matroska, simplified for use on the web. Made to solve the browser compatibility issues of Matroska, it plays in almost all modern browsers. The downside is that the codecs are very restricted.
Segmented file delivery
HTTP Smooth Streaming (HSS)
Also known as HSS. Created by Microsoft as adaptive streaming solution for web browsers. The only downside is that it has no support outside of Windows systems, and is even dropped in their latest browser Edge in favor for HLS.
HTTP Live Streaming (HLS)
Also known as HLS. This uses segmented TS files internally and is the streaming protocol developed for iOS devices. Due to the many iOS devices it sees widespread use. The biggest downsides are the high overhead and high latency.
Flash: HTTP Dynamic Streaming (HDS)
HTTP Dynamic Streaming, also known as HDS. This uses segmented F4S (FLV-based) files internally and is the last Flash protocol. It was created as a response to HLS, but never saw widespread use.
Dynamic Adaptive Streaming over HTTP (MPEG-DASH)
More commonly known as MPEG-DASH. It was meant to unify the splintered segmented streaming ecosystem under DASH, instead it standardized all of the existing protocols under a single name. Their current focus is on reducing complexity and latency.
Which protocol should you pick?
If available and an option you should always pick a stream based protocol as these will give you the best results. Sadly, most devices/browsers do not support most stream based protocols. Every device has its own preferred delivery format, which complicates matters.
Some protocols look like they’re supported by every device, so why not just stick to those? Well, every protocol also comes with their own advantages and disadvantages. Below is a table that attempts to clear up the differences:
|Protocol||Type||Platforms||Trick play||Latency||Overhead||Video codecs*||Audio codecs*||Status|
|RTSP||Streaming||Android, native players||Yes||Low||Low||Practically all||Practically all||Legacy|
|WebRTC||Streaming||Browsers||Yes||Low||Low||H264, VP8, VP9||Opus, AAC, MP3||Active development|
|FLV||Progressive file||Flash||No||Low||Medium||H264||AAC, MP3||Legacy|
|MP4||Progressive file||Browsers, native players||No||Low||High||H264, HEVC, VP8, VP9||AAC, MP3||Maintained|
|MPEG-TS||Progressive file||TV, native players||No||Low||Very high||H264, HEVC||AAC, MP3, Opus||Maintained|
|Ogg||Progressive file||Browsers, native players||No||Low||Medium||Theora||Opus, Vorbis||Maintained|
|Matroska||Progressive file||Native players||No||Low||Low||Practically all||Practically all||Maintained|
|WebM||Progressive file||Browsers, native players||No||Low||Low||VP8, VP9||Opus, Vorbis||Active development|
|HSS||Segmented file||Scripted players, Silverlight||Yes||High||High||H264, HEVC||AAC, MP3||Legacy|
|HLS||Segmented file||iOS, Safari, Android, scripted players||Yes||Very high||Very high||H264, HEVC||AAC, MP3||Active development|
|HDS||Segmented file||Flash||Yes||High||Medium||H264||AAC, MP3||Legacy|
|DASH||Segmented file||Scripted players||Yes||High||Varies||Varies||Varies||Active development|
*Only codecs still in common use today are listed.
In the end you can pick your preferred protocol based on the features and support, but to truly reach everyone you will most likely have to settle for a combination of protocols.
Relating all of the above to MistServer, you can see that we are still missing some of the protocols mentioned above. We are adding support for those in the near future. For the next blog post Erik will write about AV1, the upcoming (open) video codec.
[Blog] Connecting to our API with PHP
Hello there! Carina here.
For most users, the MI is all they'll need to configure MistServer to suit their needs. However, for some, it's not suitable. For example: suppose you'd want a group of users to be able to configure only a stream that belongs to them. MistServer can facilitate multiple users, but it doesn't support a permission system of the sort directly. Instead, PHP could be used to translate user inputs to API calls.
In this blog post, I'll explain what such a PHP implementation might look like. All the PHP code used for this example will be provided. I won't get too technical in the blog post itself, because I reckon that those who want all the details can read the PHP code, and those who can't, needn't be bothered with the exact reasons for everything. Additional information about how to use MistServer's API can be found in the documentation.
Communicating with MistServer
The first step is to set up a way to send HTTP POST requests to MistServer, with a JSON payload. I'll call this
I've used CURL to handle the HTTP request. MistServer will respond with a JSON encoded reply, which is translated to an associative array.
If MistServer is hosted on the same machine that will be running the php, authorization isn't required (new in 2.11). Otherwise, MistServer will reply telling us to authenticate using a special string. If this is the case, the password will be encrypted together with the authentication string, and sent back to MistServer along with the username. Once MistServer reports that the status is OK, the authentication part is stripped from MistServer's reply, and returned.
By default, we'll be using minimal mode, which means we'll be using less bandwidth, but mostly that MistServer's response will be less bulky and thus more readable.
Actually requesting things!
I've purposely left out showing any errors in
mistserver.php, so that you guys can just copy the file and use it in your projects. We should probably still tell users when something happens, though! I've made a few examples of usage here, that does actually output something readable in a separate file,
MistServer never replies with an error key directly in its main object, so I've used that to report CURL or API errors.
I've created a new function,
getData(), that adds error printing around the main communication function. It returns
false if there was an error and the data array if there wasn't.
We're all set! Let's actually start requesting some information. How about we check which protocols are enabled?
getData() and asks MistServer to respond with the config object. We check if the communication was successful, and if the config key exists. Then, we loop over the configured protocols and print the connector name.
Array( "config" => true )
Array( "config" => Array( "protocols" => Array( 0 => Array( "connector" => "HTTP", "online" => 1 ), [..] ), [..] ) )
Another example. Let's read the logs, and if there are any errors in it, print the most recent one. Note that MistServer only provides the last 100 log entries through the API, so there might not be any.
getLastErrorLog() also calls
getData(), but this time requests the log array. If we get it back, we reverse the array to get the newest entries first, and then start looping over them, until we find an error. If we do, we print it.
Array( "log" => true )
Array( "log" => Array( 0 => 1494493673, 1 => "CONF", 2 => "Controller started" ), [..] )
As you can see, using MistServer's API is actually quite straightforward. It's time we up the bar (slightly, don't worry) and try changing some of MistServer's configuration.
How about adding a stream? For this we'll use the
addstream command. This command is available in the Open Source version from 2.11; it was already available in Pro versions.
Two parameters need to be included: the stream name and source. There can be more options, depending on what kind of source it is. Note that if
addstream is used, and a stream with that name already exists, it will be overwritten.
addStream($name,$options) function calls
getData() with these values, checks if the stream exists in the reply, and returns it.
Array( "addstream" => Array( "example" => Array( "source" => "/path/to/file.flv" ) ) )
Array( "streams" => Array( "example" => Array( "name" => "example", "source" => "/path/to/file.flv" ), "incomplete list" => 1 ) )
Alright, great. Now, let's remove the stream again with the
deletestream command. This command is available in the Open Source version from 2.11 as well.
deleteStream($name) function calls
getData(), and checks if the stream has indeed been removed.
Array( "deletestream" => Array( 0 => "example" ) )
Array( "streams" => Array( "incomplete list" => 1 ) )
And there you have it. The API truly isn't that complicated. So get on with it and integrate MistServer into your project!
Next time, you can look forward to Balder, who will be talking about the different streaming protocols.
[Release] Stable release 2.11 now available!
- Pro feature:Access log added Access log information of viewers connecting to MistServer is now logged.
- Pro feature: New UDP-based API added
- Pro feature: Session tagging You can now freely add tags to sessions and execute actions on sessions with a specific tag.
- Pro feature: HLS file and URL (pull) input You can now use .m3u8 file structures or HTTP urls as input for MistServer.
- Pro feature: .wav output support is added
- Pro feature: PCM A-law codec support for RTSP and WAV
- Pro feature: Opus audio codec support for RTSP
- Feature: Opus audio codec support for Ogg
- Feature: Password no longer required when logging into the interface using localhost.
- Pro Improvement: Prometheus settings can now be changed during runtime
- Pro Improvement: Updater no longer blocks API access while running, updates can now be performed as rolling update without disconnecting users
- Pro Improvement: RTMP push output now compatible with Facebook and Youtube
- Improvement: Console output is now colour-coded
- Improvement: Local API access no longer requires authorization
- Improvement: Overhaul on all analysers, now all standardized in usage.
- Improvement: API changed to always return minimized-style output
- Improvement: Backported many previous Pro-only API calls to OS edition see manual for details
- Bugfix: ".html" access to streams now works correctly when used behind a proxy
[Blog] Deep-dive: the triggers system
Hello streaming media enthusiasts! We're back from the NAB show and have slept off our jetlag. But that's enough about that - a post about our triggers system was promised, and here it is.
We wanted to add a method to influence the behavior of MistServer on a low level, without being overly complicated. To do so, we came up with the triggers system. This system is meant to allow you to intercept certain events happening inside the server, and change the outcome depending on some decision logic. The "thing" with the decision logic is called the trigger handler, as it effectively "handles" the trigger event. Further requirements to the trigger system were no ties to any specific programming language, and the ability to accept handlers both remote and local. After all, at MistServer we are all about openness and making sure integrations are as friction-less as possible.
The types of events you can intercept were purposefully made very wide: all the way from high-level events such as server boot and server shutdown to low-level events such as connections being opened or closed, and everything in between.
Particularly popular is the
USER_NEW trigger, which allows you to accept or deny views on a per-session level, and as such as very suitable for access control systems.
After a lot of deliberation and tests, we finally settled on a standard input / output system as the lowest common denominator that all scripting and programming languages would support. As input, a trigger handler will receive a newline-separated list of parameters that contain information about the event being triggered. As output, the system expect either a simple boolean true/false or a replacement value to be used (where applicable, e.g. when rewriting URLs).
Two implementations were made: a simple executable style, and an HTTP style. In the executable style, the trigger type is sent as the only argument to an executable which is ran, the trigger input is piped into its standard input, and the result is read from standard output. In the HTTP style, the trigger type is sent as an HTTP header, the URL is requested, and the trigger input is sent as POST body while the result is read from the response body. These implementations allowed us to keep nearly identical internal handling mechanics, ensuring the behavior would be consistent regardless of trigger handler type used.
Further influencing behavior
While the triggers system allows for a lot of decision logic and possible use cases to be implemented, there are some things that are beyond the reach of a simple system like this. For example, you may want to have a system that runs a check if some connection is allowed to continue periodically, or when certain events happen in other parts of your business logic (e.g. payment received (or not), stream content/subject changing over time, etc).
For this purpose, after we released our triggers system, we've slowly been expanding it with API calls that supplement it. For example, the
invalidate_sessions call will cause the
USER_NEW trigger to be re-run for all already-accepted connections, allowing you to "change your mind" on these decisions, effectively halting playback at any desired point.
In the near future, we will also be releasing something we've been working on for a while now: our stream and session tagging systems. These will allow you to add "tags" to streams and sessions on the fly, and run certain triggers only when particular tags are present (or missing), plus the list of tags will be passed as one of the parameters to all applicable trigger handlers. This will add flexibility to the system for even more possibilities. Also coming soon is a new localhost-only UDP interface for the API, allowing you to simply blast JSON-format API calls to port 4242 to localhost over UDP, and they will be executed. This is a very low-cost entry point for the API, as UDP sockets can be created and destroyed on a whim and do practically no error checking.
We'd love to hear from our users what things they would like to influence and use the triggers system and API calls for. What cool things are you planning (or wanting) to do? Let us know! Reach out to us using the contact form, or simply e-mail us at email@example.com.
That was it for this post! The next blog post will be Carina, showing how to effectively use our API from PHP. Until next time!
[Blog] Stream Latency
Hi readers, as promised by Balder, this blog post will be about latency.
When streaming live footage, latency is the amount of time that passes between what happens on camera, and the time that it is shown on the stream where it is watched. And while there are cases where artificial latency is induced into the stream to allow for error correction and selecting the right camera to display at the correct time, in general you want your latency to be as small as possible. Apart from this artificial latency, I will cover some major causes of latency encountered when handling live streaming, and the available options for reduction of latency in these steps.
The three main categories where latency is introduced are the following:
The encoding step is the first in the process when we follow our live footage from the camera towards the viewer. Due to the wide availability of playback capabilities, H.264 is the most common used codec to encode video for consumer-grade streams, and I will therefore mostly focus on this codec.
While encoders are becoming faster at a rapid pace, the basic settings for most of them are geared towards optimization for VoD assets. To reduce size on disk, and through this reduce the bandwidth needed to stream over a network, most encoders will generate an in-memory buffer of several packets before sending out any. The codec allows for referencing frames both before and after the current for its data, which allows for better compression, as when the internal buffer is large enough, the encoder can pick which frames to reference in order to obtain the smallest set of relative differences to obtain it. Turning off the option for these so-called bi-predictive frames, or B-frames as they are commonly called, decreases latency in exchange for a somewhat higher bandwidth requirement.
The next bottleneck that can be handled in the encoding step is the keyframe interval. When using a codec based on references between frames, sending a 'complete' set of data on a regular interval helps with decreasing the bandwidth necessary, and is therefore employed widely when switching between different camera's on live streams. It is easily overlooked however, that these keyframe intervals also affect the latency on a large scale, as new viewers can not start viewing the stream unless they have received such a full frame — they have no data to base the different references on before this keyframe. This either causes new viewers to have to wait for the stream to be viewable, or, more often, causes new viewers to be delayed by a couple of seconds, merely because this was the latest available keyframe at the time they start viewing.
The protocol used both to the server hosting the stream and from the server to the viewers has a large amount of control over the latency in the entire process. With many vendors switching towards segment based protocols in order to allow for using widely available caching techniques, the requirement to buffer an entire segment before being able to send it to the viewer is introduced. In order to evade bandwidth overhead, these segments are usually multiple seconds in length, but even when considering smaller segment sizes, the different buffering regulations for these protocols and the players capable of displaying them causes an indeterminate factor of latency in the entire process.
While the most effective method of decreasing the latency introduced here is to avoid the use of these protocols where possible, on some platforms using segmented protocols is the only option available. In these cases, setting the correct segment size along with tweaking the keyframe interval is the best method to reduce the latency as much as possible. This segment size is configurable through the API in MistServer; even mid-stream if required.
Any processing done on the machine serving the streams introduces latency as well, though often to increase the functionality of your stream. A transmuxing system, for example, processes the incoming streams into the various different protocols needed to support all viewers, and to this purpose must maintain an internal buffer of some size in order to facilitate this. Within MistServer, this buffer is configurable through the API.
On top of this, for various protocols, MistServer employs some tricks to keep the stream as live as possible. To do this we monitor the current state of each viewer, and skip ahead in the live stream when they are falling behind. This ensures that your viewers observe as little latency as possible, regardless of their available bandwidth.
In the near future, the next release of MistServer will contain a rework of the internal communication system, removing the need to wait between data becoming available on the server itself, and the data being available for transmuxing to the outputs, reducing the total server latency introduced even further.
Our next post will be by Jaron, providing a deep technical understanding of our trigger system and the underlying processes behind it.