Today we discovered a particularly subtle way of fucking up a server restart. After a routine configuration change and RPM upgrade, a colleague tried to restart an important master. That failed. The message:
CODE:
root@master ~]# /etc/init.d/mysql start
Starting MySQLCouldn't find MySQL manager (//bin/mysqlmanag[FAILED]erver (//bin/mysqld_safe)
The colleague tried multiple times, and finally resorted to manually typing a
CODE:
nohup mysqld_safe ...
into a screen console, which he detached.
That took care of production for now and left us with an investigation. Why is the init script trying to start the MySQL manager?
It is not, and never tried to. What happen?
Continue reading "Fighting the mysqld init script"
Even when it is being repeated
once more it is not true:
Stripping binaries using the ‘strip’ utility can also significantly reduce the memory footprint of the application
claims John Coggeshall.
While it is true that a file is smaller on disk after a strip, a quick run of "size" on a binary will show you that the actual binary part of the file is unchanged. Let's have a quick look at /proc/
pid/maps to understand what happens.
Continue reading "Notes on VM"
Oprofile is a profiling tool that requires no instrumentation and can profile an entire system. Binaries with symbol tables (-g compiled binaries) and an uncompressed Kernel with symbol table are helpful.
Oprofile is capable of monitoring and analyzing a running -g compiled mysqld and can tell you which functions are using the most CPU time.
Sample output:
CODE:
CPU: CPU with timer interrupt, speed 0 MHz (estimated)
Profiling through timer interrupt
samples % image name symbol name
197404 22.4460 libc-2.3.4.so memcpy
165764 18.8484 no-vmlinux (no symbols)
76051 8.6475 mysqld.debug _mi_rec_unpack
30059 3.4179 mysqld.debug Query_cache::insert_into_
free_memory_sorted_list
(Query_cache_block*, Query_cache_block**)
17468 1.9862 mysqld.debug get_hash_link
16767 1.9065 mysqld.debug ha_key_cmp
14106 1.6039 mysqld.debug MYSQLparse(void*)
Continue reading "Using oprofile"
This is a quick tour of DRBD and how it compares to local RAID and to MySQL replication. DRBD is short for "distributed raw block device", so what it does is essentially RAID-1 over a network cable. You will be able to have two copies of a block device on two different physical machines, one of them the primary, active node and the other one a secondary, passive node.
The DRBD tour in this blog post has been created on two vmware instances with a Suse 10.0 Professional installation on each which I am using to show the most essential features of DRBD. Each vmware has a bit of memory, a network card, a boot disk with a text only Suse 10 installation and a second simulated 1 GB SCSI disk besides the boot disk to demonstrate stuff. The two instances are connected on a simulated local vmnet instance and share the 10.99.99.x/24 network, they are called left
(10.99.99.128) and right (10.99.99.129).
Continue reading "A quick tour of DRBD"
This is a quick tour of LVM and a demonstration how it is superior to static partitions. Basically, LVM provides you with a way to create dynamic partitions - you will be able to grow and shrink partitions on demand, move them between disks and snapshot them for backup, all while the filesystem and database on top of it are active and busy.
The LVM tour in this blog post has been created on a vmware instance with a Suse 10.0 Professional installation which I am using to show a combination of RAID and LVM configuration examples. The vmware has a bit of memory, a network card, a boot disk with a text only Suse 10 installation and 8 small simulated SCSI disks besides the boot disk to demonstrate stuff.
Here is the configuration for the basic system.
Continue reading "A quick tour of LVM"
If you run the following program as a non-root user, you will see exactly how much memory a single process can allocate on your Linux system. The program is simple, it gets memory in 1MB chunks until it is shot down by the Operating System. In order to prevent overcommitment, the memory allocated is actually written to with memset().
On a Debian 2.4 smp kernel with 4 GB of RAM, it stops at 2933 MB.
(via Torsten Sievers)
Continue reading "How much memory can a process use?"
