MySQL-dump

On each machine, in addition to the most basic Suse 10 Pro installation, the packages km_drbd and drbd have been installed. On left, we are partitioning the second hard disk completely for use with Linux using fdisk, creating /dev/sdb1 for use with DRBD.



We now have to define a /etc/drbd.conf to configure that disk into DRBD. A basic drbd.conf has to define a resource. A ressource is something that contains a disk partition on the left node, a matching partition on the right node, a network connection between them and definitions for error handlung and synchronisation.

The config file format is straight forward and uses named blocks in curly brackets and semicolon-terminated statements - if you know named, you'll feel right at home.



global options are set in a section aptly named global, and are currently limited to "minor-count" (the number of DRBD resources you'll be able to define), "dialog-refresh" (how quickly the startup dialog will redraw itself) and "disable-ip-verification" (disable some startup sanity checking that verifies that we are on the right machine).

All other config happens inside a resource section. That section needs to have a name, and the name can be anything, and can be quoted. We happen to use something boring like r0. Any ressource needs to have a protocol defined, and requires two "on" sections which define the two hosts we are going to use. It can also have startup, syncher, net and disk sections.

The protocol in DRBD is something like the "innodb_flush_log_at_trx_commit" in MySQL: It determines when the node comitting a disk write considers that write to be a success, and has essential influence on the speed and resilency of your DRBDs. It can be defined as A, B or C:

• In protcol C, which is the recommended setting, a write is considered completed when it has reached stable storage on the local and the remote node.
• In protocol B, we relax the constraints a little and consider the write completed when it
  has reached the local disk and the remote buffer cache. This should be faster than C, but for some reason currently is not, so you should not be using it.
• In protocol A we consider a write completed when it has reached the local disk and the local TCP send buffer. This may be okay for you, but for most people it is not.

The beef is in the "on" sections, which also need names, specifically the hostname of the system carrying the device you want to mirror to. Inside your "on" section you'll define a device, a disk, an address and a meta-disk. This is fairly straightforward: The device is the /dev/drbdn which we will be using later on to work with.

The disk is the underlying real storage that will carry all our data, the address is an ip:udp-port pair used to talk to the local DRBD instance for this device (a different UDP port is needed for each DRBD disk) and the meta-disk is either internal or some dedicated metadata storage device. For simplicity of our example, we are using internal for now (DRBD will then use 128M of /dev/sdb1 for its internal purposes. Yes, that is a lot!).

Look at the on-sections in the example above: On left and right we will be using /dev/drbd0, and have it write to /dev/sdb1. We will communicate using UDP port 7788 on both ip numbers, 128 and 129.

How we handle local disk errors, we specify using the on-io-error handler of the unnamed disk section. "detach" means that on error we simply forget the local disk and operate in diskless mode: We read and write data from and to the disk of the remote node across the network. Other options are "pass_on" (the primary reports the error, the secondary ignores it) and "panic" (the nodes leaves the cluster with a kernel panic).

Both nodes are connected using a net section. Inside the net section, which is unnamed, we define the buffers and timeouts used by DRBD: sndbuf-size, timeout, connect-int, ping-int, max-buffers, max-epoch-size, ko-count, on-disconnect.

The sndbuf is being specified in KB, and determines how much buffer the local DRBD will reserve for communication with the remote node. It should be no smaller than 32K and no larger than 1M. The optimum size is dependent on your bandwidth delay product for the connection to the remote node.

If the partner node does not reply in timeout tenths of a second, this counts as a K.O. After ko-count of these, the partner is considered dead and dropped out of the cluster. The primary then goes into standalone mode. Also, the connection to the partner node is dropped on timeout, and will be restablished immediately. If that fails, every connect-int seconds a new try is being made. If on the other hand the connection between the two nodes is idle for more than ping-int seconds, a DRBD-internal ping is sent to the remote to check if it is still present.

How the node handles a disconnect can be specified using the on-disconnect handler: Valid choices are stand_alone (go from primary to standalone mode), reconnect (try to reconnect as described above) or freeze_io (try to reconnect, but halt all I/O as in a NFS hard mount, until the reconnect is successful).

DRBD uses 4K buffers to buffer writes to disk, and it does use at most max-buffers many of these (minimum 32, coming up to at least 128K). If you see many writes, this number needs to be set to some larger values.

max-epoch-size needs to be at least 10, and determines how many data blocks may be seen at most between two write barriers.

Across the network connection, the syncer does its work to keep both disks neat and tidy. It will use at most rate K/sec of bandwidth to do so, and the default is quite low (250 K/sec). For synchronisation, the disks are cut up slices and for each slice an al-extent is being used to indicate if and where it has been changed. A larger number of al-extents makes resynchronisation slower, but requires fewer metadata writes.
The number used here should be a prime, because it is used internally in hashes that benefit from prime number sized structures.

If you have multiple devices, all of these that are in the same group are resynchronized in parallel. If two DRBD devices reside on different physical disks, you can put them into the same group so that they are resynchronized in parallel without competing for seeks on the same disk. If two DRBD devices are partition on the same physical device, put them into different groups to avoid disk head threshing.

Inside the startup section, which is also is unnamed, we define two wait-for-connection timeouts: On startup, DRBD will try to find its partner node on the network. DRBD will remember if it was "degraded" the last time it went down or not - "degraded" here means that the partner node already was down and we are missing a mirror half. If we have been degraded prior to the node restart, we wait for 120 seconds for the second node to come up and continue the boot otherwise. If we have not been degraded, we require the second node to be present for our own
boot to complete (or require manual intervention).

Using this config file on left and right (copy it over using scp!), we can start DRBD on left and right. According to our config, left will hang until the start on right has completed ("wfc-timeout" is set to 0).



The system is now not in sync, and DRBD does not know which disk
is the leading disk in our little cluster. So both disks are in
secondary mode, and cannot be written to.



To change that, we need to make one disk (the one on left) a
primary, and then watch the system synchronize.



When we try to switch the left copy of r0 to primary, this does not work - the local replica is marked inconsistent and the system cannot decide if it can act as a proper primary. We have to insist that the left copy of r0 is the one we want to become the primary using the "-- --do-what-I-say" sledgehammer.

The system will then start to sync both disks, and by monitoring /proc/drbd we can follow the progress of this operation.

Our syncher is limited to 10M per second due to configuration, and so we will a synchronisation rate of approximately 10M/sec in /proc/drbd - the sync will take almost 1.5 minutes to complete.

We do not have to wait: Even with the sync running we are free to operate on the primary copy as we like. We like to have a file system on /dev/drbd0 and then mount this. Here is how:



We get 860M useable, plus 32M reiserfs journal, plus 128M drbd overhead. This really is not useful for small partitions.

Meanwhile, on right:



Onto this system we now install MySQL.



To fail over from left to right, a number of things need to be done:

• MySQL needs to be stopped.
• The disk needs to be unmounted.
• The disk needs to be put in secondary on left.
• The disk needs to be put in primary on right.
• The disk needs to be mounted on right.
• MySQL needs to be started.

Actually, for the change to be completely transparent to the applications using MySQL, MySQL needs to be running on a virtual IP (e.g. 10.99.99.130), which also needs transfer from left to right). This can be automated, and even forced if left crashes, but not with DRBD. An additional package is needed, the Linux heartbeat package, which offers this additional functionality - we will be covering this in a later article.

Here is the manual switchover, on left:

And the inverse, on right:

In a real MySQL failover scenario, we do not know why the failover took place in the first place, and if the server data on the DRBD disk is useable. Thus, MySQL would most likely need to run a InnoDB recover and should also be run with MyISAM table autorepair. This will slow down failover time, a lot if you happen to require a very large InnoDB log.