đź…­

How to monitor system bandwidth usage statistics with Darkstat

Darkstat is a handy little program that will monitor and track your system’s bandwidth utilization and help you keep an eye on what is eating up your bandwidth quota. Here is how you set it up in Fedora Linux.

You can install Darkstat on your router, server, or even just your personal computer.

Darkstat will generate statistics and graphs tracking average network utilization in total per hour, day, and month (screenshot below), and statistics per destination host and per port per destination host. (It took me two years to discover the last set of statistics. Click on an IP address in the hosts table to see it.)

Getting Darkstat up and running

Start by installing the darkstat package:

dnf install darkstat

You must manually configure your network interfaces in /etc/sysconfig/darkstat configuration file before you can start the service. Adjust the OPTIONS variable and list the network interfaces you wish to track with the -i argument. (You can see your active network interfaces by running the command ip a.) You may only want to monitor your mobile network interface or all external interfaces.

Once you’ve configured a network interface, you can already start Darkstat by issuing the following command:

systemctl enable --now darkstat.service

Darkstat should be up and running and you can see all the network traffic ticking by your system being counted up by visiting http://[::1]:667/ (localhost on port 667) in your preferred web browser.

The top of the web interface gives you two options: graphs, which shows your traffic totals, and hosts which shows a per-destination host table. It can take some time for hostnames to resolve, depending on the status of your local or network DNS cache. Learn how to configure a local DNS cache.

Darkstat graphs

Restricting the service

By default, Darkstat runs as the nobody user. The installer adds a dedicated darkstat system user, but this isn’t used in the default configuration. In this section we’ll switch to using the Darkstat system user to keep other services from reading it’s data (we’ll get back to this in the next section) and move the service to a more secure chroot environment for additional security.

Start by creating the directory we’ll be storing data in, and change the owner to darkstat using the following commands:

mkdir /var/lib/darkstat/
chown darkstat:nobody /var/lib/darkstat/
chmod 700 /var/lib/darkstat/

Next, you’ll need to add some additional arguments to the OPTIONS variable in /etc/sysconfig/darkstat to let it know where to store its data and which user to run as:

OPTIONS="-i p10p1 \
         --user darkstat \
         --chroot /var/lib/darkstat/"

The -i argument is the interfaces you configured in the initial setup step in the previous setting.

Restart the Darkstat service to apply the changes using the following command:

systemctl restart darkstat.service

Configuring persistent records

You may have noticed that Darkstat forgot all the data it had tracked so far when you restarted the service. By default, Darkstat won’t save your usage data to disk and will start tracking anew on restart. Darkstat can store up to 31 days of data in memory or on disk.

If you don’t restart your system often, this may be what you want. However, you may wish to enable persistent storage so that your records will survive past a reboot.

OPTIONS="-i p10p1 \
         --user darkstat \
         --chroot /var/lib/darkstat/ \
         --export /netuse.db \
         --import /netuse.db"

The other arguments in the above example were configured in the previous sections.

If you’re interested in tracking more than the 300 destination host maximum, you can increase the number of hosts to follow by adding --hosts-max 5000 (top hosts to track in memory) and --hosts-keep 4500 (top hosts to save to disk) to the OPTIONS variable. This will also enable Darkstat to better keep track of destination hosts that use a “trickle amount” of data over longer periods of time.

5000 addresses will use about 9–12 MB of memory and disk space, and won’t have any noticeable delay on your network performance. You can go above this if you believe there’s any value in tracking even more hosts, but let the data run for a few days before decreasing or increasing the suggested numbers.

Restart the Darkstat service again to apply the changes using the following command:

systemctl restart darkstat.service

Darkstat was reset one last time for this restart. Try restarting it again and you’ll see that the restart no longer forgets your previously recorded data.

Surviving sleep and downed network links

Darkstat will exit and not be restarted when a network interface goes down or the system goes to sleep. You can wok around these issues by intruding persistent storage, as described above, combined with the below fixes to the systemd unit file.

Run the following command to create a local system unit file overwrite file to modify the default behavior:

systemctl edit darkstat.service

Copy the below systemd unit file and include them in the overwrite file that opened when you ran the above command:

[Unit]
After=syslog.target network-online.target

[Service]
Restart=always

These changes does is cause the service to be restarted if it stops and to wait for the network to not only become available but online before starting Darkstat. Combined with the persistent storage configured above, you won’t notice it when Darkstat restarts.

To apply the changes, you need to reload all systemd unit files, and then restart Darkstat one last time:

systemctl daemon-reload
systemctl restart darkstat.service

These improvements have been suggested to the Fedora project, but haven’t yet been accepted by the package maintainer for Darkstat. These steps should be unnecessary in the future.

If you’re having problems or want to run Darkstat on a device with frequently changing network configurations (like a laptop), then I recommend you also add --wait 5 to the OPTIONS variable. This just delays the startup of Darkstat to allow for more time to get the network up and running.