Operating System

Today we discovered a particularly subtle way of fucking up a server restart. After a routine configuration change and RPM upgrade, a colleague tried to restart an important master. That failed. The message:
The colleague tried multiple times, and finally resorted to manually typing a into a screen console, which he detached.

That took care of production for now and left us with an investigation. Why is the init script trying to start the MySQL manager?

It is not, and never tried to. What happen?

Even when it is being repeated once more it is not true:

Stripping binaries using the ‘strip’ utility can also significantly reduce the memory footprint of the application

claims John Coggeshall.

While it is true that a file is smaller on disk after a strip, a quick run of "size" on a binary will show you that the actual binary part of the file is unchanged. Let's have a quick look at /proc/pid/maps to understand what happens.

Oprofile is a profiling tool that requires no instrumentation and can profile an entire system. Binaries with symbol tables (-g compiled binaries) and an uncompressed Kernel with symbol table are helpful.

Oprofile is capable of monitoring and analyzing a running -g compiled mysqld and can tell you which functions are using the most CPU time.

Sample output:

This is a quick tour of DRBD and how it compares to local RAID and to MySQL replication. DRBD is short for "distributed raw block device", so what it does is essentially RAID-1 over a network cable. You will be able to have two copies of a block device on two different physical machines, one of them the primary, active node and the other one a secondary, passive node.

The DRBD tour in this blog post has been created on two vmware instances with a Suse 10.0 Professional installation on each which I am using to show the most essential features of DRBD. Each vmware has a bit of memory, a network card, a boot disk with a text only Suse 10 installation and a second simulated 1 GB SCSI disk besides the boot disk to demonstrate stuff. The two instances are connected on a simulated local vmnet instance and share the 10.99.99.x/24 network, they are called left
(10.99.99.128) and right (10.99.99.129).

This is a quick tour of LVM and a demonstration how it is superior to static partitions. Basically, LVM provides you with a way to create dynamic partitions - you will be able to grow and shrink partitions on demand, move them between disks and snapshot them for backup, all while the filesystem and database on top of it are active and busy.

The LVM tour in this blog post has been created on a vmware instance with a Suse 10.0 Professional installation which I am using to show a combination of RAID and LVM configuration examples. The vmware has a bit of memory, a network card, a boot disk with a text only Suse 10 installation and 8 small simulated SCSI disks besides the boot disk to demonstrate stuff.

Here is the configuration for the basic system.

If you run the following program as a non-root user, you will see exactly how much memory a single process can allocate on your Linux system. The program is simple, it gets memory in 1MB chunks until it is shot down by the Operating System. In order to prevent overcommitment, the memory allocated is actually written to with memset().

On a Debian 2.4 smp kernel with 4 GB of RAM, it stops at 2933 MB.

(via Torsten Sievers)