News

30 Nov 2017

[Release] Stable release 2.13 now available!

Hello everyone! Stable release 2.13 of MistServer is now available! The full change log is available here and downloads are here. Our Pro-edition customers with active licenses will receive a new build notification in their e-mail automatically. Here are some highlights: (Pro-only) Subtitle support:...

Hello everyone! Stable release 2.13 of MistServer is now available! The full change log is available here and downloads are here. Our Pro-edition customers with active licenses will receive a new build notification in their e-mail automatically.

Here are some highlights:

  • (Pro-only) Subtitle support: sideloaded srt input, HLS output, DASH output, MP4 input & output
  • (Pro-only) Significantly improved recording and scheduled pushes system
  • Most core systems now support proxy servers, HTTP Basic/Digest authentication and HTTPS
  • (Pro-only) Triggers and HLS pull input likewise support all of the above now
  • Fixes for Facebook and YouTube support
  • TS-based inputs now support timestamp rollover properly and repeatedly
  • Fixed CVE-2017-16884, discovered by hyp3rlinx / apparitionsec
  • Many other small fixes/improvements/etc. See changelog for full list!
read moreless
20 Nov 2017

[Blog] Scheduled Playout

Hello everyone, Erik here with a quick post about a new feature we're currently working on: simulating a live stream based on one or multiple VoD assets scheduled to play out continuously. This 'simulated live' type of stream is usable...

Hello everyone, Erik here with a quick post about a new feature we're currently working on: simulating a live stream based on one or multiple VoD assets scheduled to play out continuously.

This 'simulated live' type of stream is usable in various scenarios, ranging from a small broadcaster or hobby user who wants to set up a continuous stream consisting of a limited amount of pre-recorded footage, to a large station wanting to maintain full control of what programs to broadcast at which time.

We decided that this would be a nice feature to include in our software, and after our — nearly finished — 2.13 release, we will be working at full speed to incorporate our new style of internal metadata storage in memory, giving way to allowing multiple inputs to contribute to a single stream. This feature will not only allow the development of our scheduled playout system, but will also create the possibilities from merging multiple separate source files into a single multi-bitrate stream, or having multiple different versions of encryption active at the same time.

Playlist creation

The first scenario we will support is single file playout loopback. This will allow for a single file to be looped continuously, while for the viewers it will look like a single live stream. Though not useful for many cases, it will provide an easy setup if you don't change your programming, keeping the stream active at all times.

With some more advanced configuration, we will be supporting m3u files. This file format is what most people know from the HLS format, while at the same time providing a straightforward way to handle a simple list of filenames without providing extra information. Any extended headers available in the file will be ignored, and both .m3u and .m3u8 extensions will be supported.

Using this playlist setup will not only allow you to schedule multiple files in a loop, it will also allow you to edit the playlist while it is running. For every entry in the playlist, after playing out the entry, the playlist will be reloaded before determining what the next file in the playlist is. This allows you to amend your playlist as it runs, replacing one program with another, or even continuously updating your playlist file so it will never loop, but acting as an actual live broadcast channel.

That's it for this small update, if you have any requests or questions about this upcoming feature, please let us know.

Our next post will be by Jaron, with an explanation about how MistServer works internally.

read moreless
2 Nov 2017

[Blog] DTSC: MistServer's internal media format

Hey everyone, this is Jaron again, and it's time for another blog post! This time I'm going to dedicate some time to our internal media format, DTSC. DTSC stands for "DDVTech Stream Container", and is the format MistServer internally converts all inputs...

Hey everyone, this is Jaron again, and it's time for another blog post! This time I'm going to dedicate some time to our internal media format, DTSC.

DTSC stands for "DDVTech Stream Container", and is the format MistServer internally converts all inputs into. The outputs then read this format and convert it to whatever they are supposed to output to the end-user. Doing things this way, allows us to generically write outputs and inputs without needing to know which of each will be available in advance. It's one of the biggest reasons why MistServer is so modular, and why we have been able to add inputs and outputs so regularly.

DTSC itself is a container/transport format taking inspiration from the FLV, JSON, MKV and MP4 formats. It tries to take the good parts of those formats without making anything overly complicated. It's packet-based, with a (repeatable) header that contains initialization and seeking information. All packets are size-prepended, and the data format is based on a simplified form of binary JSON. These properties allow DTSC to be both used as a storage format on disk and as a transport format over networks.

We're planning to release the DTSC specification as well as a sample implementation as public domain in the near future, because we see possibilities in replacing RTMP with DTSC for live ingest purposes in the long term. (On that note: if you are interested in contributing to or discussing the possibilities of the DTSC specification, please contact us!)

Besides the internal use (which only usually exists in RAM and is never written to disk at any point), DTSC is used by MistServer in two more places: our header files (the .dtsh files you may have noticed appearing alongside your VoD assets) and by the DTSC Pull input.

When MistServer's various inputs read a VoD file from storage, they generate a DTSC header and store it beside the original file as a .dtsh file. On future accesses to the file, this header is used to speed up loading from the file. It can safely be deleted (and will regenerate the next time the file is accessed) and will auto-delete if the format changes because of MistServer updates (for example, the upcoming 2.13 update will add a new optional field, and thus force a regenerate of all headers so the new field will show up). This file helps us provide a consistent speed across all media storage formats, and provides an index for files that normally do not have an index, such as TS files.

The DTSC Pull input allows you to pull live streams from other MistServer instances, using DTSC as the transport. This means it is a live replication and distribution format that is compatible with an unlimited number of tracks/qualities and works for all codecs. Unfortunately, MistServer is the only software (at time of writing) that has implemented DTSC as a streaming input/output format, so you can only take advantage of DTSC distribution between MistServer instances (for example, for load balancing live streams). There are plans to also make DTSC usable for VoD distribution in the near future.

Hopefully this article helped shed some light on MistServer's internal processes regarding file formats and replication. Next time, my colleague Erik will write about our upcoming scheduled play-out system!

read moreless
27 Oct 2017

[Blog] Library playback with the STREAM_SOURCE trigger

Hello readers! Today I will talk about how MistServer can be set up to work with a large video library. The issue When you want MistServer to serve a VOD file, you'd normally add a stream, and configure the path to the...

Hello readers! Today I will talk about how MistServer can be set up to work with a large video library.

The issue

When you want MistServer to serve a VOD file, you'd normally add a stream, and configure the path to the file as the stream source.
If you have a folder of several video files, you'd add a folder stream, with its source pointing to the path of the folder. The individual files are then accessible using the main streamname, a +, and then the filename, myfolderstream+video.mp4 for example.
However, these implementations have limits. Having too many configured streams will considerably slow down MistServer. Folder streams are inconvenient for larger libraries, look unprofessional, and are not able to access subfolders.
Once your video library grows beyond a certain point, it's wise to consider a different configuration. You can set up one or more streams with the settings you'd like to use, and then use the STREAM_SOURCE trigger to rewrite the stream source to which file you'd like to stream.

In this blog post I'll discuss an example implementation of this method, using PHP.

Resources:

Understanding the STREAM_SOURCE trigger

I'd like to explain the inner workings of this method by setting up a stream, that, when requested, plays a random video from the library.
The first step is to configure the stream in MistServer, through the Management Interface.
Let's create a new stream, and call it random. The source doesn't really matter, as we'll be overwriting it, but let's set it to one of our video files. That way, if the trigger doesn't work for some reason, that video will be played.

Now, let's go ahead and configure the trigger.
It should trigger on STREAM_SOURCE. Applies to should be set to the stream we just created, random. The handler url should point to where our trigger script will be hosted. We want MistServer to use the page output, so it should be set to blocking. Let's set the default response to the same fallback video we configured as the stream source.

Alright, to the PHP! We'll need to echo the path to the video we want the stream to play.

First, confirm the page is being requested by MistServer's STREAM_SOURCE trigger:

if ($_SERVER["HTTP_X_TRIGGER"] != "STREAM_SOURCE") {
  http_response_code(405);
  error_log("Unsupported trigger type.");
  echo("Unsupported trigger type.");
  return;
}

Next, we want to retrieve the stream name that MistServer reports to the PHP script. This is sent on the first line of the POST body.

//retrieve the post body
$post_body = file_get_contents("php://input");
//convert to an array
$post_body = explode("\n",$post_body);
$stream = $post_body[0];

If the stream name equals random, we'll select a random video id from the library, and return the path. Make sure the path is the only thing that is sent.

if ($stream == "random") {    
  //select a random video from the library array
  $library = get_library();
  $random_video_id = array_rand($library);

  //return the path
  echo $library[$random_video_id];
  return;
}

To simulate a video library, I've set up a little function that indexes video files from a folder, just for this demo. This should be replaced with the library database system when implemented.

That's all. When the stream random is requested, a random video will be shown.
There's a little caveat here, though. If the stream random is already active, a new request will receive the same video. We can prevent this by adding a unique wildcard token to the stream name; that way the stream won't be active yet. In the trigger script, edit the stream name condition:

if (($stream == "random") || (substr($stream,0,7)) == "random+") {

And, on the video page:

$streamname = "random+".rand();

Embed the stream on a page, and every page load a random video will be selected.

Back to the library scenario

Now that we understand how the trigger should be used, let's get back to a more practical use case. There isn't that much of a difference: we want to pass the desired video id to the trigger script, and return the appropriate path. We can simply add the video id to the stream as the wildcard token.
I've also created a new stream, library, and applied the trigger to it.

if ((substr($stream,0,8)) == "library+") {
  $wildcard = substr($stream,8); //strip the "library+" part

  $library = get_library();

  echo $library[$wildcard];
  return;
}

With, on the video page:

$streamname = "library+".intval($_GET["video_id"]);

The URL to our video page could be something user friendly like /Movies/45/Big Buck Bunny Goes To Town/. Simply configure your HTTP server to rewrite the url so that the video id can be extracted.

You can consider using multiple streams (movies, shows, live) for different categories. That way, you can have different stream settings in MistServer to suit your needs.

I hope these examples can help you if you are looking to set up a video library using MistServer. As always, if you have any questions or want to tell us about your awesome project, feel free to contact us.
Next time, Jaron will be back with another behind the scenes blog post.
In the meantime: happy streaming!

read moreless
23 Oct 2017

[Blog] Recording live streams with MistServer

Hey everyone, lately we've had some questions regarding recording and how to set it up. I thought it would be a good subject to talk about as while it's quite easy to use, there are always a few things to...

Hey everyone, lately we've had some questions regarding recording and how to set it up. I thought it would be a good subject to talk about as while it's quite easy to use, there are always a few things to keep in mind.

The basics of recording in MistServer

All recording in MistServer is done through the push panel within your MistServer interface. This is because we see recording as pushing the stream to a local file. To set one up we'll only have to push towards a path/file.ext, I’ll talk more about that below. At the time of writing this blog post MistServer can only record in FLV, TS, MP3 and WAV files, but we will be adding additional protocols in the future, our aim is to have a recording possibility for any protocol that makes sense to record in.

There's two flavors of pushing, push and automatic push.

Normal pushes

Normal pushes are one time events. A push, or recording in this case will be started of your selected stream and after it's done it will be removed and disappear leaving just your recording. These pushes are unaffected by the push settings at the top of the menu as they’re single non-restarting recordings. It's the easiest method to start a recording with if you just want a single recording and be done with it.

Automatic pushes

Automatic pushes are set to monitor a stream and start recording the moment that stream becomes active. This could be a live stream or the wildcard streams of a live stream, any recording set up using this method will be recorded under the settings used. It's the method to choose if you want to setup something that automatically records live streams.

Push settings: stream name, target and target options

Stream name

Stream name is the stream you'll be recording with this push command. There’s 3 options here: streamname, streamname+ and streamname+wildcard. Streamname will just use the stream with the matching stream name, streamname+ will use all wildcard streams of the chosen stream and streamname+wildcard will just use the chosen wildcard stream.

Target

Target is the push method. To record you must choose a path/file.flv or path/file.ts, for example /media/recordings/recording01.ts.

Target options

We have two types of target options, variables and parameters. Variables are replaced by their corresponding variable upon push/recording while parameters change the way the recording is handled. Variables are used by $variable while parameters should be used as ?parameter, however, using multiple parameters follows a different syntax: ?parameter01&parameter02&parameter03&etc. Variables are available in every MistServer that has pushing functionality, while parameters will be available from MistServer version 2.13 and up.

Variables

VariableBehaviour
$streamreplaced by the `streamname+wildcard` in the file name
$dayreplaced by the day number in the file name
$monthreplaced by the month number in the file name
$yearreplaced by the year number in the file name
$hourreplaced by the hour (00-23) at recording start in the file name
$minutereplaced by the minute (00-59) at recording start in the file name
$secondsreplaced by the seconds (00-59) at recording start in the file name
$datetimereplaced by $year.$month.$day.$hour.$minute.$seconds at recording start in the file name

Parameters

ParameterBehaviour
?recstart=time_in_MSStarts the recording at the closest frame when the given time in milliseconds is reached.
?recstop=time_in_MSStops the recording at the closest frame after the given time in milliseconds is reached
?recstartunix=time_in_SECONDSStarts the recording at the closest frame when the given UNIX time in seconds is reached
?recstopunix=time_in_SECONDSStops the recording at the closest frame when the given UNIX time in seconds is reached
?passthrough=1Activates passthrough: All inputs will be used in the push/recording.

Things to keep in mind, common mistakes

Here I will list a few things that are handy to keep in mind or usually go wrong the first time when setting up recordings.

Variables are necessary to not automatically overwrite automatic recordings

Without using variables you will overwrite your automatic recordings the second your stream source becomes active. Adding a date and timestamp will make sure your file gets an unique enough name to avoid this. This is especially important if your source tends to be unstable and restarts when recording.

When using recstart/stop parameters, the timestamp used is not necessary the same timestamp used by players

Players usually start their playback starting at 0 seconds no matter what the stream data says, but depending on the source a recording can contain information starting from a timestamp higher than 0ms. A stream could for example claim to start at timestamp 120000 if you started recording after the stream was active for 2 minutes already. This will mean you will have to account for this timestamp as well.

When using recstart/stopunix the unix time between machines doesn’t have to be the same

The unix time used is the unix time of the machine running MistServer, this time is changeable depending on the machine settings so make sure you do not assume that this time is the same over every machine.

Make sure you have write access to the folder you are writing

It sounds obvious, but MistServer will not be able to record anything if it is not allowed to write files. Make sure the given path is a path that can be used.

If you start a single record of a live stream before it is available you have about 10 seconds before it gets auto removed

Any streams that are not available within 10 seconds of starting the recording will be assumed not working/active and removed. Automatic recordings should be used if you do not plan to set the source live right before or after setting the recording.

Make sure the protocol you want to record as is active and available

If MistServer cannot access the protocol you want to record in (HTTPTS for TS and FLV for FLV) it will not be able to record the file at all.

A few examples

Lastly I will leave a few examples of how you could set up recordings.

Automatic recording of wildcard streams with their name+month.day.hour.minute added

Stream: live+

Target: /media/recording/$stream$month$day$hour$minute.ts

Automatic recording of wildcard streams and automatically grabbing any multibitrate tracks

Stream: live+

Target: /media/recording/$stream.ts?passthrough=1

Recording of a stream as myrecording.ts starting at 1 minute for 2 minutes

Stream:live

Target: /media/recording/myrecording.ts?recstart=60000&recstop=180000

Automatic recording of the wildcard stream live+specific_stream as a multibitrate ts file named mymultibitratefile.ts

Stream:live+specific_stream

Target: /media/recording/mymultibitratefile.ts?passthrough=1

That should cover you for most uses when recording. The next blog will be by Carina covering how to set up MistServer when you've got to work with a huge library

read moreless
Latest 2017 2016 2015 2014 2013 2012