In this post we will cover installing VictoriaMetrics and Grafana and setting them up. We will also cover upgrading your Prometheus setup to Victoriametrics in case you are looking to update.

Why do I want Victoriametrics and Grafana for analytics?

MistServer has a tremendous amount of data available and in use to optimize streaming workflows with the sole intention of running in the moment. MistServer itself does not store this data anywhere as it would quickly flood your server storage. Data collection applications such as VictoriaMetrics are made to specifically counter this problem and are capable of scraping (collecting) the data and storing it with superior compression. Grafana in turn is an application made to make data like this what we call "human friendly". Providing graphs, bars and other visual representations of the data easy to understand.

Best practises for setting up your analytics server

As you might have guessed using VictoriaMetrics and Grafana will require some resources therefore we recommend running it on a different device than you are running MistServer on. This is for a few reasons, but the most important being that you would want your analytics collection to keep going if your MistServer instance goes dark for some reason or has run into trouble.

As such we would recommend setting up a server whose sole focus is to get the analytics from your MistServer instances. It can be any kind of server, just make sure it has access to all your MistServer instances.

Requirements

• One or more running MistServer instance(s)
• A server with connection to all MistServer instances you want to capture analytics from

The steps to follow

• First we will choose an operating system
• Then we will install VictoriaMetrics & Grafana
• Afterwards we will set up Grafana
• Lastly we will show some default MistServer dashboards for Grafana as a template

01 OS Selection

While VictoriaMetrics is able to run in nearly any OS and has docker images available as well our personal preference goes to running it on a dedicated Linux server. Linux allows us to throw a very minimal OS on the server and fully dedicated it to the task of data collection. Other OS tend to add unnecessary features and complicate usage.

If you do want to use a different OS please feel free to do so, with the exception of the actual installation process the rest of this guide should still help you set everything up.

02 Installing VictoriaMetrics and Grafana

VictoriaMetrics

VictoriaMetrics is wide-spread since its release in 2018. By now it's almost certainly in the default package managers for your Linux distro. If for some reason it is not we would recommend manually installing it using their available downloads.

Setting up VictoriaMetrics

Now before we start, obviously we cannot cover every setting available within VictoriaMetrics. We'll give something to work with. We would always recommend reading up on the applications you're using and determining what settings you want to use yourself as well.

We would recommend some minor tweaks to the boot arguments. The default retention period is 1 month. We will want to edit it to something longer, let's say 120 months. Adding the following to the arguments list will do the trick: -retentionPeriod 120

Now depending on how VictoriaMetrics is installed the arguments could be directly in your service script or link to an EnvironmentFile.

Another thing to consider is to change the path where the data is stored. While not strictly necessary we usually change it towards: -storageDataPath /var/lib/victoriametrics/

Make sure that the user VictoriaMetrics runs as exists and has access to this folder.

Updating your Prometheus setup to VictoriaMetrics

If you're updating from Prometheus to VictoriaMetrics you can follow the set up above, but also need to do two more things. - Load in the existing prometheus.yml settings - Import the existing prometheus data

Loading in the existing prometheus.yml settings

The flag -promscrape.config handles this, however your existing prometheus.yml is most likely not compatible with VictoriaMetrics. Therefore we recommend the following: - Copy your prometheus.yml to /etc/victoria.yml - Edit the new victoria.yml to be compatible with VictoriaMetrics - set the promscrape.config to victoria.yml

You can copy your current prometheus.yml with the following command: cp /path/to/prometheus.yml /etc/victoria.yml

If you do not know where your prometheus.yml is run the following: ps aux|grep prometheus

Within the output you should see a config file argument like: --config.file /etc/prometheus.yml

That is where your prometheus.yml is located.

We will now edit this file, grab your preferred text editor and we'll start making it compatible. As bare minimal you only need the scrape config. So unless you absolutely want to keep a setting I would delete everything but the scrape config.

A minimal victoria.yml would look like this: ``` scrapeconfigs: # The job name is added as a label job=<job_name> to any timeseries scraped from this config. - jobname: "mist" scrapeinterval: 10s scrapetimeout: 10s metricspath: '/PROMETHEUSPASSPHRASE' static_configs: - targets: ['SERVER01:4242', 'SERVER02:4242']

# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.

```

Now go back to your VictoriaMetrics service script or EnvironmentFile and add the following line to the arguments:

-promscrape.config /etc/victoria.yml

Importing the existing prometheus data

We can now start VictoriaMetrics. If you've skipped the original set up for VictoriaMetrics keep in mind that you want to at least add -retentionPeriod 120 to the argument list. This makes the retention period 120 months instead of the default of 1 month.

Let's boot up VictoriaMetrics with the correct configurations

systemctl daemon-reload systemctl restart victoriametrics

Take a coffee break of 5-10 minutes so VictoriaMetrics has had time to collect data while Prometheus is still active (to avoid data loss).

Go to your Grafana setup and change the Datasource to go to port 8428 instead of your Prometheus port (9090 by default).

VERIFY THIS IS WORKING Open one of your Dashboards, set the window to 5 minutes to cover the "new" collection time and see if you're getting the stats you expect. If you're seeing your dashboards work continue, if you're not getting anything start debugging why. I would start with systemctl status victoriametrics and continue from there.

We will now need vmctl if your VictoriaMetrics installation came without you can find vmctl at the VictoriaMetrics Download page under the vmutils packages.

Check where the Prometheus storage path is: ps aux|grep prometheus Look for the argument: --storage.tsdb.path /path/to/prometheus/ Write this down, you will need it in a bit.

Close down prometheus: systemctl stop prometheus systemctl disable prometheus

and start the import: vmctl prometheus --prom-snapshot /path/to/prometheus/

Take some time off as now you need to wait for the import to finish. Once it's done open your Dashboard again and you should see all your old data back. Congratulations, you've upgraded succesfully.

Grafana

Grafana is well distributed within the Linux community. You should be able to simply install it as a service using the default installation process for your chosen OS. If for some reason it is not available check for the Grafana website for installation instructions.

03: Setting up Grafana

Through your installation method Grafana should be active and available as a service. If not you can start it simply by booting the executable, but I would look into getting it ran as a service.

Once active Grafana will have an interface available at http://HOST:3000 by default. Open this in a browser and get started on setting up Grafana.

Adding a data source

The next step is to add a data source. As we're running Grafana and Prometheus in the same location, this is quite easy. All we need to set is the Name, Type and URL all other settings will be fine by default.

• Name can be anything you'd want.
• Type has to be set to: Prometheus (yes, this is correct.)
• URL will be the location of the VictoriaMetrics interface: http://localhost:8428

Add those and you're ready for the next step.

4. Adding the dashboard

We've got a few Dashboards available immediately which should give the most basic things you'd want. You can add a dashboard by following these steps:

Click on the grafana icon in the top left corner → hover Dashboards → Select Import

You should see the following

Fill in the Grafana.com Dashboard number with our preset dashboards (for example our MistServer Vitals: 1096)

If recognised you will see the following

Just add that and you should have your first basic dashboard. Our other dashboards can be added in the same manner. More information about what each dashboard is for can be found below.

MistServer provided dashboards

All of the dashboards can be found here on Grafana Labs as well.

MistServer Vitals: 1096

This is our most basic overview which includes pretty much all of the statistics you should want to see anyway. It covers how your server is doing resource and bandwidth wise.

You switch between given MistServers at the top of given panels by clicking and selecting the server you want to inspect.

MistServer Stream Details: 4526

This shows generic details per active stream. Streams and Servers are selected at the top of the panel. You'll be able to see the amount of viewers, total bandwidth use and amount of log messages generated by the stream.

MistServer All Streams Details: 4529

This shows the same details as the MistServer Stream Details Dashboard, but for all streams at the same time. This can be quite a lot of data, and will become unusable if you have a lot of streams. If you have a low amount of streams per server this gives an easy to use overview however.