Howto Raspberry Pi: Monitor your Raspberry Pi with Observium!

The Goal: 

With Observium associated with Unix agent check_mk the goal will be to monitor any available indicator (CPU, Mem, Traffic interface...) and most of all, specific Raspberry Pi main indicators dynamically allocated when running Overclocked with Turbo mode:

• CPU Frequency
• CORE Frequency
• CORE Voltage
• BCM2835 Soc Temperature

Corresponding "vgencmd" commands:

# CPU Frequency
vcgencmd measure_clock arm

# CORE Frequency
vcgencmd measure_clock core

# CORE Voltage
vcgencmd measure_volts core

# SoC Temp
vcgencmd measure_temp

There are also other indicators you may want to monitor, even i don't feed it myself useful.
The present article will take care of these 4 indicators.

Take a look here: http://www.elinux.org/RPI_vcgencmd_usage

Global list of indicators available through "vgencmd":

vcgencmd measure_clock arm
vcgencmd measure_clock core
vcgencmd measure_clock h264
vcgencmd measure_clock isp
vcgencmd measure_clock v3d
vcgencmd measure_clock uart
vcgencmd measure_clock pwm
vcgencmd measure_clock emmc
vcgencmd measure_clock pixel
vcgencmd measure_clock vec
vcgencmd measure_clock hdmi
vcgencmd measure_clock dpi
vcgencmd measure_volts core
vcgencmd measure_volts sdram_c
vcgencmd measure_volts sdram_i
vcgencmd measure_volts sdram_p

Installing Observium is out of the scope of this article, Observium installations documentations and well known and easy to read, see above.

Main sources:

• Observium Website
• Ubuntu installation

I recommend to install Observium and Mysql into a central server which will request our Rpi to generate graphs and so on.

We will use an additional agent called "check_mk" to request the Rpi, system load generated by snmp and Unix agent are very limited which is very great, the Rpi is a small power device and you don't want monitoring to generate high system load!

One time you have Observium up and running, follow this guide to integrate any Raspberry Pi you want to monitor :-)

Summary of steps: 

Step 1: Install and configure snmpd
Step 2: Install check_mk agent (Unix Agent)
Step 3: Add the custom Raspberry agent script
Step 4: Observium custom application configuration
Step 5: Configure your Rpi in Observium, the easy part!

Memorandum

Step 1: Install and configure snmpd

First thing, we will begin by installing the snmpd daemon, to do so:

$ sudo apt-get install snmpd snmp-mibs-downloader

Let's configure some little things:

Edit "/etc/default/snmpd" and:

• set: export MIBS=UCD-SNMP-MIB

• Replace the line "SNMPDOPTS=" with the following values to prevent snmpd to log each connection (default behavior):

SNMPDOPTS='-LS 0-4 d -Lf /dev/null -u snmp -g snmp -I -smux -p /var/run/snmpd.pid'

Edit "/etc/snmp/snmpd.conf" and:

• Comment with "#" the default line "agentaddress udp:127.0.0.1:161" which only allows connections from the localhost itself

• Comment out the line "agentaddress udp:161,udp6:[::1]:161" to allow remote connections

• Comment out the line "rocommunity secret <LANSUBNET>" (adapt <LANSUBNET> to the CIDR value of your LAN subnet, example: 192.168.0/24" 

Note: "secret" will the name of the snmp community, only accessible through your local network) 

• Configure "sysLocation" and "sysContact"

• Look for the section "EXTENDING THE AGENT" and add the following line:

extend .1.3.6.1.4.1.2021.7890.1 distro /usr/bin/distro

• Install the "distro" script coming from observium (to recognize the remote OS)

$ sudo wget http://www.observium.org/svn/observer/trunk/scripts/distro -O /usr/bin/distro
$ sudo chmod 755 /usr/bin/distro

Finally restart snmpd daemon:

$ sudo service snmpd restart

Step 2: Install check_mk agent (Unix agent)

We will used the great Unix agent "check_mk" called Unix agent by Observium.

If you want more information about this very cool tool, check its main Web site:
http://mathias-kettner.de/checkmk_monitoring_system.html

Install Xinetd requirement:

$ sudo apt-get install xinetd

Download and install check_mk:

$ wget http://mathias-kettner.com/download/check-mk-agent_1.2.0p3-2_all.deb
$ sudo dpkg -i check-mk-agent_1.2.0p3-2_all.deb

Verify that the package installation generated the xinetd configuration file called "

/etc/xinetd.d/check_mk".

If not (it seems this part fails under Rpi), create the file with the following content:

# +------------------------------------------------------------------+
# |             ____ _               _        __  __ _  __           |
# |            / ___| |__   ___  ___| | __   |  \/  | |/ /           |
# |           | |   | '_ \ / _ \/ __| |/ /   | |\/| | ' /            |
# |           | |___| | | |  __/ (__|   <    | |  | | . \            |
# |            \____|_| |_|\___|\___|_|\_\___|_|  |_|_|\_\           |
# |                                                                  |
# | Copyright Mathias Kettner 2012             mk@mathias-kettner.de |
# +------------------------------------------------------------------+
#
# This file is part of Check_MK.
# The official homepage is at http://mathias-kettner.de/check_mk.
#
# check_mk is free software;  you can redistribute it and/or modify it
# under the  terms of the  GNU General Public License  as published by
# the Free Software Foundation in version 2.  check_mk is  distributed
# in the hope that it will be useful, but WITHOUT ANY WARRANTY;  with-
# out even the implied warranty of  MERCHANTABILITY  or  FITNESS FOR A
# PARTICULAR PURPOSE. See the  GNU General Public License for more de-
# ails.  You should have  received  a copy of the  GNU  General Public
# License along with GNU Make; see the file  COPYING.  If  not,  write
# to the Free Software Foundation, Inc., 51 Franklin St,  Fifth Floor,
# Boston, MA 02110-1301 USA.

service check_mk
{
 type           = UNLISTED
 port           = 6556
 socket_type    = stream
 protocol       = tcp
 wait           = no
 user           = root
 server         = /usr/bin/check_mk_agent

 # If you use fully redundant monitoring and poll the client
 # from more then one monitoring servers in parallel you might
 # want to use the agent cache wrapper:
 #server         = /usr/bin/check_mk_caching_agent

 # configure the IP address(es) of your Nagios server here:
 #only_from      = 127.0.0.1 10.0.20.1 10.0.20.2

 # Don't be too verbose. Don't log every check. This might be
 # commented out for debugging. If this option is commented out
 # the default options will be used for this service.
 log_on_success =

 disable        = no
}

Restart xinetd:

$ sudo service xinetd restart

Finally, ensure your Observium machine willbe authorized to access to the Rpi check_mk service running on port TCP/6556.

Step 3: Add the custom Raspberry agent script

Create a new file "/usr/lib/check_mk_agent/local/raspberry":

#!/bin/bash
#set -x
echo "<<<app-raspberry>>>"
# CPU Frequency
expr `vcgencmd measure_clock arm|cut -f 2 -d "="` / 1000000
# CORE Frequency
expr `vcgencmd measure_clock core|cut -f 2 -d "="` / 1000000
# CORE Voltage
vcgencmd measure_volts core|cut -f 2 -d "="|cut -f 1 -d "V"
# SoC Temp
vcgencmd measure_temp|cut -f 2 -d "="| cut -f 1 -d "'"

Add execution right:

$ sudo chmod a+rx /usr/lib/check_mk_agent/local/raspberry

This script will be called by Observium at each poller time.

Step 4: Observium custom application configuration

Ok now a bigger part, we need to configure Observium to add our custom application has any other.
By this way, we could run this with as many Rpi as you want ;-)

To do so, we need to create and/or modify different configuration files.

Go into your Observium root directory, usually "/opt/observium"

1. "./includes/polling/unix-agent.inc.php" (modify)

Look for the section containing:

      if ($section == "apache") { $sa = "app"; $sb = "apache"; }

And add new one just under :

      if ($section == "raspberry") { $sa = "app"; $sb = "raspberry"; }

2. "./includes/polling/applications/raspberry.inc.php" (create)

Create with following content:

<?php

if (!empty($agent_data['app']['raspberry']))
{
  $raspberry = $agent_data['app']['raspberry'];
}

$raspberry_rrd  = $config['rrd_dir'] . "/" . $device['hostname'] . "/app-raspberry-".$app['app_id'].".rrd";

echo(" raspberry statistics\n");

list($cpufreq, $corefreq, $corevoltage, $soctemp) = explode("\n", $raspberry);
if (!is_file($raspberry_rrd))
{
  rrdtool_create ($raspberry_rrd, "--step 300 \
        DS:cpufreq:GAUGE:600:0:125000000000 \
        DS:corefreq:GAUGE:600:0:125000000000 \
        DS:corevoltage:GAUGE:600:0:125000000000 \
        DS:soctemp:GAUGE:600:0:125000000000 ".$config['rrd_rra']);
}

print "cpufreq: $cpufreq corefreq: $corefreq corevoltage: $corevoltage soctemp: $soctemp";
rrdtool_update($raspberry_rrd, "N:$cpufreq:$corefreq:$corevoltage:$soctemp");

// Unset the variables we set here

unset($raspberry);
unset($raspberry_rrd);
unset($cpufreq);
unset($corefreq);
unset($corevoltage);
unset($soctemp);

?>

3. "./html/includes/graphs/application/raspberry_soctemp.inc.php" (create)

Create with following content:

<?php

$scale_min = 0;

include("includes/graphs/common.inc.php");

$raspberry_rrd   = $config['rrd_dir'] . "/" . $device['hostname'] . "/app-raspberry-".$app['app_id'].".rrd";

if (is_file($raspberry_rrd))
{
  $rrd_filename = $raspberry_rrd;
}

$ds = "soctemp";

$colour_area = "F0E68C";
$colour_line = "FF4500";

$colour_area_max = "FFEE99";

$graph_max = 1;

$unit_text = "°C";

include("includes/graphs/generic_simplex.inc.php");

?>

4. "./html/includes/graphs/application/raspberry_corevoltage.inc.php" (create)

Create with following content:

<?php

$scale_min = 0;

include("includes/graphs/common.inc.php");

$raspberry_rrd   = $config['rrd_dir'] . "/" . $device['hostname'] . "/app-raspberry-".$app['app_id'].".rrd";

if (is_file($raspberry_rrd))
{
  $rrd_filename = $raspberry_rrd;
}

$ds = "corevoltage";

$colour_area = "CDEB8B";
$colour_line = "006600";

$colour_area_max = "FFEE99";

$graph_max = 1;

$unit_text = "Volts";

include("includes/graphs/generic_simplex.inc.php");

?>

5. "./html/includes/graphs/application/raspberry_corefreq.inc.php" (create)

Create with following content:

<?php

$scale_min = 0;

include("includes/graphs/common.inc.php");

$raspberry_rrd   = $config['rrd_dir'] . "/" . $device['hostname'] . "/app-raspberry-".$app['app_id'].".rrd";

if (is_file($raspberry_rrd))
{
  $rrd_filename = $raspberry_rrd;
}

$ds = "corefreq";

$colour_area = "B0C4DE";
$colour_line = "191970";

$colour_area_max = "FFEE99";

$graph_max = 1;

$unit_text = "Mhz";

include("includes/graphs/generic_simplex.inc.php");

?>

6. "./html/includes/graphs/application/raspberry_cpufreq.inc.php" (create)

Create with following content:

<?php

$scale_min = 0;

include("includes/graphs/common.inc.php");

$raspberry_rrd   = $config['rrd_dir'] . "/" . $device['hostname'] . "/app-raspberry-".$app['app_id'].".rrd";

if (is_file($raspberry_rrd))
{
  $rrd_filename = $raspberry_rrd;
}

$ds = "cpufreq";

$colour_area = "B0C4DE";
$colour_line = "191970";

$colour_area_max = "FFEE99";

$graph_max = 1;

$unit_text = "Mhz";

include("includes/graphs/generic_simplex.inc.php");

?>

7. "./html/pages/device/apps/raspberry.inc.php" (create)

Create with following content:

<?php

global $config;

$graphs = array('raspberry_cpufreq' => 'CPU Frequency',
                'raspberry_corefreq' => 'CORE Frequency',
                'raspberry_corevoltage' => 'CORE Voltage',
                'raspberry_soctemp' => 'BCM2835 SoC Temperature',

);

foreach ($graphs as $key => $text)
{

  $graph_array['to']     = $config['time']['now'];
  $graph_array['id']     = $app['app_id'];
  $graph_array['type']   = "application_".$key;

  echo('<h3>'.$text.'</h3>');

  echo("<tr bgcolor='$row_colour'><td colspan=5>");

  include("includes/print-graphrow.inc.php");

  echo("</td></tr>");
}

?>

8. "./html/pages/apps.inc.php" (modify)

Look for the section containing:

$graphs['apache']     = array('bits', 'hits', 'scoreboard', 'cpu');

And add new one just under :

$graphs['raspberry']  = array('cpufreq', 'corefreq', 'corevoltage', 'soctemp');

Ok, we're done!

Step 5: Configure your Rpi in Observium, the easy part!

Now the easiest, add your Rpi into Observium, go to the menu <Devices>, <Add device>.

In our case:

• Hostname: Enter the hostname or IP of your Rpi
• snmp Community: secret

Let all the rest by default.

The Rpi shall be detected with sucess, and the Debian logo appears:

Now enter the device and go to device settings:

Go to "Applications" and activate the box corresponding to our Raspberry application:

Then, Go to "Modules" and Activate the Unix agent (disabled by default):

Great, you're done will all configuration parts, wait for a view poller execution (by default Observium proposes a cron task every 5 minutes)

You can run manually the poller under the host running Observium:

$ sudo /opt/observium/poller.php -h all

And if you want to run it into debug mode to get more details:

$ sudo /opt/observium/poller.php -h all -d

In my experience, you have to wait for 10-15 minutes before getting data being graphed.

Some screenshots with application data:

CPU Frequency:

CORE Frequency:

CORE Voltage:

BCM2835 Soc Temperature:

Great :-)