Enemy Action

All other graphs are more or less normal, indeed. But around 13:20 there is a short full stop for all processing, and then said surge.

Now, we do have log_processlist.pl. So if there is a reason for the locking, I will have logs in /var/log/mysql_pl/Wed/13_2[01]*. Those files are indeed larger, but only a little so: Up to over 400 connections from a regular 250.

And: The logs are clean. Nothing irregular, whatsoever. So either our monitoring has big holes, or this is not a server stall and pileup, but has an external cause.

This is a master. Masters have a datadir on NetApp. I am asking the original client to talk to the Filer people. Maybe they have been playing with their toys at that time. While the original client is walking away, I am checking more logs - it is not that we have a shortage of them.

And indeed:

Oct 31 13:20:10 master kernel: bnx2 0000:03:00.0: eth0: NIC Copper Link is Down
Oct 31 13:20:20 master kernel: bnx2 0000:03:00.0: eth0: NIC Copper Link is Up, 1000 Mbps full duplex, receive & transmit flow control ON

So the Filer people are off the hook. We have instead caught some friendly fire from the data center team. Maybe somebody has been assuming that all boxes are being created equal, because they are looking equal. Let's see if that is the case and educate them about box labelling.

Sometimes the database is not the cause.