Using oprofile
Oprofile is a profiling tool that requires no instrumentation and can profile an entire system. Binaries with symbol tables (-g compiled binaries) and an uncompressed Kernel with symbol table are helpful.
Oprofile is capable of monitoring and analyzing a running -g compiled mysqld and can tell you which functions are using the most CPU time.
Sample output:
Oprofile is capable of monitoring and analyzing a running -g compiled mysqld and can tell you which functions are using the most CPU time.
Sample output:
CODE:
CPU: CPU with timer interrupt, speed 0 MHz (estimated)
Profiling through timer interrupt
samples % image name symbol name
197404 22.4460 libc-2.3.4.so memcpy
165764 18.8484 no-vmlinux (no symbols)
76051 8.6475 mysqld.debug _mi_rec_unpack
30059 3.4179 mysqld.debug Query_cache::insert_into_
free_memory_sorted_list
(Query_cache_block*, Query_cache_block**)
17468 1.9862 mysqld.debug get_hash_link
16767 1.9065 mysqld.debug ha_key_cmp
14106 1.6039 mysqld.debug MYSQLparse(void*)
Profiling through timer interrupt
samples % image name symbol name
197404 22.4460 libc-2.3.4.so memcpy
165764 18.8484 no-vmlinux (no symbols)
76051 8.6475 mysqld.debug _mi_rec_unpack
30059 3.4179 mysqld.debug Query_cache::insert_into_
free_memory_sorted_list
(Query_cache_block*, Query_cache_block**)
17468 1.9862 mysqld.debug get_hash_link
16767 1.9065 mysqld.debug ha_key_cmp
14106 1.6039 mysqld.debug MYSQLparse(void*)
A short cheatsheet:
- yum install oprofile (or whatever in your system installs a package)
- opcontrol --init
Load kernel module oprofile - opcontrol --setup --separate=lib,kernel,thread --no-vmlinux
Select how to collect data, unfortunately CentOS here does not provide a vmlinux kernel image. If you know how to reliably convert a vmlinuz image into a vmlinux image that is being accepted by oprofile, I'd like to hear that. - opcontrol --start-daemon
Preload the kernel collection daemon - ps axuwww| grep opro
Double check that the daemon is running - opcontrol --start
Start data collection - opcontrol --dump
Force data to disk - opreport --demangle=smart --symbols --long-filenames --merge tgid $(which mysqld) | less
Get a report as above (-g binary req'd) - opcontrol --stop
Stop profiling - opcontrol --deinit
Stop daemon - opcontrol --reset
Delete collected data, free disk space
Trackbacks
bonglonglong on : OProfile, Kernel Images and InnoDB (oh my!)
Kristian Köhntopp has a wonderful article about using oprofile to track down problems in running programs. I thought I’d add a few thoughts.
If you need to get a vmlinux kernel on redhat, apparently you just need to install kernel-debuginfo, whi...
Comments
Display comments as Linear | Threaded
John Levon on :
(don't use strip on the binary).