
Dear Diary, today I ventured into one of the darkest realms of the sysadmin profession: I started playing with SNMP. My goal was very simple and quite clearly stated: Make the output of "SHOW GLOBAL STATUS" available to a SNMP client. One would think that this is a reasonable and easily fulfilled wish.
Little did I know of the madness and despair that linger in the depths which are guarded by the dread named ASN.1 and where the noxious fumes of the organisation no longer called CCITT can still strongly be smelled.
But let us begin this story at the beginning - with a clean install of Suse Linux 10.0 and my trusty apt4rpm and me. It was my thought that the perl support of net-snmp might me handy to get me where I wanted to me: perl is easily the more convenient language for prototyping that C or C++ and getting to the data source using perl-DBI would be pretty easy. So "apt install net-snmp net-snmp-devel perl-SNMP" it was. So with my spirits high and a song on my iPod I ventured out into the documentation...
And quickly discovered that when you try to reproduce the example from "perldoc NetSNMP::agent" you get a crash.
CODE:
/usr/bin/perl: symbol lookup error:
/usr/lib/libnetsnmpagent.so.5: undefined symbol: hosts_ctl
This, dear diary, looks quite familiar. It seems that Suse in their desire to make Unix a more secure operating system have been linking libnetsnmpagent.so.5 against libwrap, but forgot to somehow make libnetsnmpagent to actually load libwrap. Annoying, but once you have identified the symbol and its source it is easily fixed. The first patch, and I haven't even started.
CODE:
linux:/etc/init.d # diff -u snmpd.orig snmpd
--- snmpd.orig 2006-10-27 21:22:01.000000000 +0200
+++ snmpd 2006-10-27 21:22:05.000000000 +0200
@@ -18,6 +18,9 @@
AGENTDIR=/usr/lib/net-snmp/agents
SNMPDCONF=/etc/snmpd.conf
+LD_PRELOAD=libwrap.so.0.7.6
+export LD_PRELOAD
+
test -x $SNMPD || exit 5
# Shell functions sourced from /etc/rc.status:
Now SNMP tools are enterprisey. They won't even give you the current uptime without a MIB. So for "SHOW GLOBAL STATUS" we need a MIB file as well. That is why I like perl so much. It gets things done. In this case, it is making me a MIB file out of a MySQL connection.
CODE:
kris@linux:~/.snmp/mibs> cat makemysqlmib.pl
#!/usr/bin/perl -w --
use strict;
use DBI;
my $dsn = "DBI:mysql:host=127.0.0.1;port=3340";
my $user = "root";
my $pass = "";
my @global_status = ();
# Connect database
my $dbh = DBI->connect($dsn, $user, $pass) or die("connect");
my $cmd = "show global status";
my $sth = $dbh->prepare($cmd) or die("prepare");
$sth->execute();
my $i = 1;
while (my $ref = $sth->fetchrow_hashref()) {
my $name = $ref->{'Variable_name'};
$name =~ s!_(.)!\U$1\E!g;
$name = "mysql$name";
$global_status[$i++] = $name;
}
$dbh->disconnect();
print qq(MYSQL-MIB DEFINITIONS ::= BEGIN
IMPORTS
MODULE-IDENTITY, OBJECT-TYPE
FROM SNMPv2-SMI
DisplayString
FROM SNMPv2-TC
netSnmpPlaypen
FROM NET-SNMP-MIB;
mysqlMIB MODULE-IDENTITY
LAST-UPDATED "200610270000Z"
ORGANIZATION "Kristian Koehntopp"
CONTACT-INFO
"Some string"
DESCRIPTION
"A sample table"
REVISION "200610270000Z"
DESCRIPTION
"This is a sample table for testing"
::= { netSnmpPlaypen 1767 }
mysqlMIBObjects OBJECT IDENTIFIER ::= { mysqlMIB 1 }
mysqlStatus OBJECT IDENTIFIER ::= { mysqlMIBObjects 1 }
);
for (my $i=1; $i < @global_status; $i++) {
printf qq(%s OBJECT-TYPE
SYNTAX DisplayString
MAX-ACCESS read-only
STATUS current
DESCRIPTION "MySQL SHOW GLOBAL STATUS LIKE %s"
::= { mysqlStatus %d }
),
$global_status[$i],
$global_status[$i],
$i;
}
print qq(
END
);
kris@linux:~/.snmp/mibs> perl makemysqlmib.pl > M
kris@linux:~/.snmp/mibs> smistrip M
MYSQL-MIB: 1773 lines.
kris@linux:~/.snmp/mibs> smilint -m /usr/share/snmp/mibs/NET-SNMP-MIB.txt ./MYSQL-MIB
No complaints! Everything is going well! Let's check!
CODE:
kris@linux:~/.snmp/mibs> mv MYSQL-MIB MYSQL-MIB.txt
kris@linux:~/.snmp/mibs> cd ..
kris@linux:~/.snmp> echo "mibs +MYSQL-MIB" > snmp.conf
kris@linux:~/.snmp> snmptranslate -On -Tp -IR mysqlMIB| less
+--mysqlMIB(1767)
|
+--mysqlMIBObjects(1)
|
+--mysqlStatus(1)
|
+-- -R-- String mysqlAbortedClients(1)
| Textual Convention: DisplayString
| Size: 0..255
+-- -R-- String mysqlAbortedConnects(2)
| Textual Convention: DisplayString
| Size: 0..255
...
Wow! I am so cool, I could be Domas! Now let us go root and write something to the specification called Agent X! SuSE, still inclined on world domination, keeps close tabs on it's agents. If it detects one on your system, it automatically creates a headquarter and becomes the agentx master. We can check:
CODE:
linux:/etc/init.d # cat snmpd
...
AGENTDIR=/usr/lib/net-snmp/agents
...
# check whether to enable agentx support and get list of installed
# agents
get_agents()
{
agents=
agentargs=''
for agent in $AGENTDIR/*; do
test -x $agent || continue
agents="$agents $agent"
agentargs="-x /var/run/agentx/master"
done
}
...
linux:/etc/init.d # cd /usr/lib/net-snmp/agents
linux:/usr/lib/net-snmp/agents # touch mysql_agent.pl
linux:/usr/lib/net-snmp/agents # chmod a+x mysql_agent.pl
linux:/usr/lib/net-snmp/agents # rcsnmpd start
Starting snmpd done
Starting mysql_agent.pl
startproc: cannot execute /usr/lib/net-snmp/agents/mysql_agent.pl: Exec format error
failed
So the remaining task is just to write a SNMP agent to the MIBs specification. Let's import all the toys and play.
CODE:
#!/usr/bin/perl --
use NetSNMP::OID (':all');
use NetSNMP::agent(':all');
use NetSNMP::ASN(':all');
use DBI;
my $dsn = "DBI:mysql:host=127.0.0.1;port=3340";
my $user = "root";
my $pass = "";
# set to 1 to get extra debugging information
$debugging = 0;
$subagent = 0;
$refresh_interval = 30;
$global_status = ();
$global_last_refresh = 0;
Now we do the MySQL dance and ask nicely for some status information. But not too often - we do not want to be too intrusive on the server.
CODE:
###
### Called automatically now and then
### Refreshes the $global_status and $global_variables
### caches.
###
sub refresh_status {
my $now = time();
# Check if we have been called quicker than once every $refresh_interval
if (($now - $global_last_refresh) < $refresh_interval) {
# if yes, do not do anything
print STDERR "Not refreshing\n" if ($debugging);
return;
}
# Connect database
my $dbh = DBI->connect($dsn, $user, $pass);
my $cmd = "show global status";
# Get status info
my $sth = $dbh->prepare($cmd);
$sth->execute();
# 'status_data' becomes mysqlStatusData
my $oldname = "";
while (my $ref = $sth->fetchrow_hashref()) {
my $name = $ref->{'Variable_name'};
$name =~ s!_(.)!\U\1\E!g;
$name = "mysql$name";
$global_status{$name}{'value'} = $ref->{'Value'};
$global_status{$oldname}{'next'} = $name if ($oldname ne "");
$oldname = $name;
}
# No next for the last
$global_status{$name}{'next'} = '';
# Fixes
$global_status{'mysqlMIB'}{'value'} = "0";
$global_status{'mysqlMIB'}{'next'} = "mysqlMIBOBjects";
$global_status{'mysqlMIBObjects'}{'value'} = "0";
$global_status{'mysqlMIBObjects'}{'next'} = "mysqlStatus";
$global_status{'mysqlStatus'}{'value'} = "0";
$global_status{'mysqlStatus'}{'next'} = "mysqlAbortedClients";
$dbh->disconnect();
$global_last_refresh = $now;
print STDERR "Refreshed at $now ", time()-$now, "\n" if ($debugging);
return;
}
All that is pretty standard fare and contains no surprises for the seasoned MySQL+Perl developer.
Now we need to write the agent. The agent will register itself for a subtree of Object Identifiers - we have already seen our tree in the snmptranslate above - so we will register for the mysqlMIB tree and snmpd will be kind enought to ask agentX for all the secret information about our mysql.
CODE:
my $regOID = new NetSNMP::OID("mysqlMIB");
if (!$agent) {
$agent = new NetSNMP::agent('Name' => 'mysql_snmp', # reads mysqlsnmp.conf
'AgentX' => 1); # make us a subagent
$subagent = 1;
print STDERR "started us as a subagent ($agent)\n" if ($debugging);
}
$agent->register("mysql_snmp", $regOID, \&mysql_snmp_handler);
if ($subagent) {
# We need to perform a loop here waiting for snmp requests. We
# also check for new STATUS data.
$SIG{'INT'} = \&shut_it_down;
$SIG{'QUIT'} = \&shut_it_down;
$running = 1;
while($running) {
do refresh_status();
$agent->agent_check_and_process(1); # 1 = block
}
$agent->shutdown();
}
sub shut_it_down {
$running = 0;
print STDERR "shutting down\n" if ($debugging);
}
Now, isn't that simple?
We create one of these nicely prepackaged NetSNMP::agent objects and register the callback function mysql_snmp_handler for our mysqlMIB OID. Then we go into a loop and whenever the agent feels like it it will cycle through the loop and fetch new status data or process a request. Should a signal arrive during that time, we will catch it in our signal handler and $running will become false. That will shut down the agent cleanly.
Now the dirty business.
The handler:
We get called.
We get a bunch of parameters.
We detect the currently en vogue OID.
We check what we should do to it.
We do.
We are finished.
Easy?
CODE:
sub mysql_snmp_handler {
my ($handler, $registration_info, $request_info, $requests ) = @_;
my ($request);
# print STDERR "refs: ",join(", ", ref($handler), ref($registration_info),
# ref($request_info), ref($requests)),"\n" if ($debugging);
for ($request = $requests; $request; $request = $request->next()) {
# Process request for $oid (e.g. mysqlUptime)
my $oid = $request->getOID();
my $mode = $request_info->getMode();
my $value;
my $next;
print STDERR "- asking for oid $oid (mode $mode)\n" if ($debugging);
if ($mode == MODE_GET) {
$value = $global_status{$oid}{'value'};
$request->setValue(ASN_OCTET_STR, "$value");
print STDERR " - GET handling ($oid = $value)\n" if ($debugging);
}
if ($mode == MODE_GETNEXT) {
$next = $global_status{$oid}{'next'};
if ($next ne "") {
$value = $global_status{$next}{'value'};
$request->setOID($next);
$request->setValue(ASN_OCTET_STR, "$value");
} else {
$request->setOID($oid);
}
print STDERR " - GETNEXT handling ($oid: $next = $value)\n" if ($debugging
);
}
}
print STDERR "- finished processing\n" if ($debugging);
}
Now SNMP is a pretty crappy protocol. It has one operation to get a value for a OID, called GET. When we get a GET, we deliver the value. It is that easy.
But there also is GETNEXT. When we get a GETNEXT, we find the next value in sequence and deliver the OID of that value and the actual value associated with it. This is used to walk the SNMP OID tree without knowledge of the actual tree as if you had no copy of the MIB - the primary reason for the strange value/next construction in our @global_status array.
To debug, we start the snmpd, kill the perl, and restart the agent manually with $debugging = 1. Then we ask nicely for data.
CODE:
linux:~ # rcsnmpd start
Starting snmpd done
Starting mysql_agent.pl done
linux:~ # pkill -f perl
linux:~ # export LD_PRELOAD=libwrap.so
linux:~ # /usr/lib/net-snmp/agents/mysql_agent.pl
started us as a subagent (NetSNMP::agent=HASH(0x8300804))
Refreshed at 1161979732 0
Not refreshing
Connection from <UNKNOWN>
- asking for oid mysqlUptime
- variable mysqlUptime = 3009 (next )
- GET handling
- finished processing
Not refreshing
The debug output is the result of running the following command elsewhere:
CODE:
kris@linux:~/.snmp> snmpget -v 2c -c public localhost mysqlUptime
MYSQL-MIB::mysqlUptime = STRING: 3009
We can also get all status variables step by step using the dreaded GETNEXT:
CODE:
kris@linux:~/.snmp> snmpwalk -v 2c -c public localhost mysqlMIB| less
MYSQL-MIB::mysqlMIBObjects = STRING: "0"
MYSQL-MIB::mysqlStatus = STRING: "0"
MYSQL-MIB::mysqlAbortedClients = STRING: 0
MYSQL-MIB::mysqlAbortedConnects = STRING: 0
MYSQL-MIB::mysqlBinlogCacheDiskUse = STRING: 0
...
MYSQL-MIB::mysqlThreadsCreated = STRING: 1
MYSQL-MIB::mysqlThreadsRunning = STRING: 26
MYSQL-MIB::mysqlUptime = STRING: 1
Now, dear diary, the way I am telling you this everything sounds nice and clear. But that is only because I left out so many things I have seen on my way to get here, things that have no names and others whose name shall not be mentioned.
The one thing to remember is this: If somebody is telling you about SNMP tables and the NetSNMP table API - don't go there. It is
The King in Yellow, like gazing into the Abyss. Especially if you try to pair it with perl.
Now let me close this entry, dear diary, and try to clean my mind of the things that I have seen and which keep me awake at night. I can hear this nights rain drumming at the window from the outside, and other, darker darker things move in it... A peated whisky might be the right thing tonight to let me sleep and not have dreams. Maybe this entry in my diary will keep others straight on the path and away from Those Things That No Man Should Ever Know.
Heute habe ich Das Böse gesehen. Und überlebt. Lest meinen Tagebucheintrag während ich dem Trunke verfalle um das Vergessen zu suchen.
Tracked: Oct 27, 20:30