Serving Images from a File System

MySQL-dump

March 14

Serving Images from a File System

In Serving Images from a Database, Sheeri responded to me:
I appreciate your ideas, but you haven't read what I've written: "the sheer number of files is enough to kill any os."

I'm serious about that. we're talking 36 million images -- that's 36 million files (no thumbnails, they're all reduced size) plus directories to store them in. We have cacheing tuned very well, and it's still kills the OS.


Here is what I tried to simulate that:

I took my new Strato MR2. It is currently running tor, INN and MySQL. There are two HDS722516VLSA80 160 GB SATA HDDs. These disks are rated at 8.8ms avg seek time, so there are 113 seeks in a second with this disk.

I am using one otherwise idle disk to rebuild Sheeri's test setup:

CODE:
kris@h743107:~/x> df -Th .
Filesystem    Type    Size  Used Avail Use% Mounted on
/dev/sdb2 reiserfs    153G   55G   98G  36% /home
kris@h743107:~/x> free -m
             total       used       free     shared    buffers     cached
Mem:          1010        996         13          0        224        346
-/+ buffers/cache:        425        584
Swap:         3074          3       3071
kris@h743107:~/x> cat /proc/cpuinfo
processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 15
model           : 39
model name      : AMD Opteron(tm) Processor 146
stepping        : 1
cpu MHz         : 1995.305
cache size      : 1024 KB
...


There is a bit of code I am using to generate 32 Megafiles, each 1KB in size.

CODE:
kris@h743107:~/x> cat testscript.pl
#! /usr/bin/perl -w

sub create_dir($) {
        my $name = shift;
        mkdir $name;
}

sub create_file($) {
        my $name = shift;

        open FILE, ">$name" or die "cannot create $name";
        print FILE "x"x1023, "\n";
        close FILE

}

for (my $m=0; 2>$m; $m++) {
        $dir  = sprintf("%02x", $m);
        create_dir($dir);

        for (my $i=0; 256>$i; $i++) {
                $dir  = sprintf("%02x/%02x", $m, $i);
                create_dir($dir);

                for (my $j=0; 256>$j; $j++) {
                        $dir  = sprintf("%02x/%02x/%02x", $m, $i, $j);
                        create_dir($dir);

                        for (my $k=0; 256>$k; $k++) {
                                $name = sprintf("$dir/%02x.jpg", $k);
                                print "$name\r";
                                create_file($name);
                        }
                }
        }
}


When running this, I am getting approximately 320,000 files created per minute. This is sustained through the entire creation process, and does not slow down at all.

CODE:
Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sdb          0.00 2378.00  0.00 97.00    0.00 19744.00     0.00  9872.00   203.55     7.58  100.66   4.87  47.20


As you can see, the disk is not yet maxed out, so there probably is room for tuning (I did not tune anything).

We are seeing a service time of 4.8ms, so probably every second request involves seeks, and we see 90 w/s - following that math, that would be 45 "w/s with seeks" out of 113 "w/s with seeks" possible.

A very trivial accessscript can look like this:

CODE:
kris@h743107:~/x> cat accessscript.pl
#! /usr/bin/perl -w

my $lim = 32*1024*1024;
my $lim = 0.5*1024*1024;

my $newtime = 0, $oldtime = 0;
my $counter = 0;
my $x;

while (1) {
        my $a = int(rand $lim);
        my $m = ($a>> 24);
        my $i = ($a>> 16) & 255;
        my $j = ($a>> 8) & 255;
        my $k =  $a& 255;
        $name = sprintf "%02x/%02x/%02x/%02x.jpg", $m, $i, $j, $k;

        $counter++;
        $newtime = time;
        if ($newtime != $oldtime) {
                print "counter = $counter\n";
                $counter = 0;
                $oldtime = $newtime;
        }

        open FILE, "<$name" or next;
        read FILE , $x, 1024;
        close FILE;
}


When you start this, it is very slow at first, but the buffer and dentry caches will heat up eventually. I am running all the following tests with remounting the filesystem with "noatime".

Setting $lim to 1E4 (10,000 files of working set) runs from the cache completely:

CODE:
counter = 45305
counter = 45307
counter = 44972
counter = 45316
counter = 45310
counter = 45307

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sdb          0,00   0,00  0,00  0,00    0,00    0,00     0,00     0,00     0,00     0,00    0,00   0,00   0,00


Setting $lim to 2E5 (200,000 files in working set) yields

CODE:
counter = 156
counter = 155
counter = 187
counter = 154
counter = 181

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sdb          0,00   0,00 103,00  0,00  824,00    0,00   412,00     0,00     8,00     1,00    9,67   9,67  99,60


at first. You have to let it run quite a while for the system to heat up. After 10 minutes of curning, the numbers will read

CODE:
counter = 1748
counter = 1800
counter = 1987
counter = 1802
counter = 1882

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sdb          0,00   0,00 77,00  0,00  608,00    0,00   304,00     0,00     7,90     0,85   11,17  11,01  84,80


and from there they will eventually increase more rapidly to

CODE:
counter = 4562
counter = 4112
counter = 5572
counter = 5544
counter = 5116


Observing /proc/sys/fs/dentry-state over time shows the system reconfiguring itself:

CODE:
61796   58525   45      0       0       0
65698   62366   45      0       0       0
...
91724   87924   45      0       0       0
91843   88042   45      0       0       0
...
107483  102187  45      0       0       0
106838  101579  45      0       0       0


Larger tests take even more time. I will eventually edit this post and add to it.
Posted by Kristian Köhntopp in Scaleout and Tuning at 12:40 | Comments (4) | Trackback (1)

Trackbacks
Trackback specific URI for this entry

Statification
In Semi-Dynamic Data, Sheeri writes about Semi-Dynamic Data and content pregeneration. In her article, she suggests that for rarely changing data it is often adviseable to precompute the result pages and store them as static content. Sheeri is right: Noth
Weblog: MySQL-dump
Tracked: Aug 17, 21:06

Comments
Display comments as (Linear | Threaded)

I'm glad someone's actually trying! And indeed, our problem is when the cache is empty, and performance of the site suffers greatly. As we expand more and more, that 36 million number will only grow larger -- we project by the end of 2007 that number will be 330 million. Which means the cache won't always be hit (sure we can add memory)....etc.

We're sacrificing small bits of performance, this is true. It's not something I'm denying. But we're sacrificing a bit of performance always, to maintain consistent performance. ie, a 5% performance hit to make sure that those days when the filesystem cache dies that it doesn't slow down the entire site.

Not that anyone asked. Seeing folks scream "BUT YOU'LL TAKE A PERFORMANCE HIT!" is amusing, but unhelpful. In the long run, it's better. For us. In this narrow case.

I'd like to see a comparison between retrieval time from disk versus retrieval time from db. overhead, yes, but it's low enough that we can live with it!
#1 Sheeri (Homepage) on 2006-03-14 16:42 (Reply)
QUOTE:
I'd like to see a comparison between retrieval time from disk versus retrieval time from db. overhead, yes, but it's low enough that we can live with it!


I'm afraid my statistics went in the recycle bin once I satisfied myself of the results, but I recall seeing between 0 and a 10% hit for using the database to serve smallish images.

I took five measurements, threw out the outliers, and averaged the middle three. Still, some trials showed database performance hits below the "noise level" -- the variations between individual measurements were greater than the average difference between file system and database.

I decided that, in my particular case, the many advantages to storing the images in the database far outweighed the relatively small speed impact. So many other things have bigger impact. (Having an image much bigger than the display size, for example -- I pre-scale images to match display size when I store them in the database in three sizes.)

The funny thing about "conventional wisdom" is that it generally respects only other people's conventions. You can "prove" anything you want on paper!
#1.1 Jan Steinman (Homepage) on 2006-03-14 18:28 (Reply)
i agree, and dont forget the update case. Deleting files and replacing them with bigger severly will result in fragmentation. (However growing files are much much worse than a few blocks variation in images)

One advantage of db backed up files (besides searching for attributes, which can also be solved with a hybrid solution) is backup/replication.

Some filesystems may also degenerate much faster, especially if the main memory of the system not only can be used for dcache.

Me thinks a fair compariion would compare a database server against an NFS server or cluster filesystem. I guess nobody would want to scale-up in an web-server scenario.

Gruss
Bernd
#1.1.1 eckes (Homepage) on 2006-03-14 22:50 (Reply)
For the sake of backup and clustering, I still don't get why people scream when you save images to a database.

Point taken that it's maybe not as fast (and so on), then serving directly from a filesystem (using a small httpd), yet, the features (clustering, backup) easily outrule the loss of performance you encounter.

Benchmarks are one thing, real-life experiences another. The TOC on buying another node outrules tweaking servers for hours and hours, and hours because you gain too little. I am not saying that it is sometimes not useful to do this, or one should not write effecient code. Not at all!

But this is along the lines of writing hard to maintain code/applications for the sake of microseconds here and there.
#2 tfk on 2006-03-15 19:34 (Reply)

Add Comment

E-Mail addresses will not be displayed and will only be used for E-Mail notifications

To prevent automated Bots from commentspamming, please enter the string you see in the image below in the appropriate input box. Your comment will only be submitted if the strings match. Please ensure that your browser supports and accepts cookies, or your comment cannot be verified correctly.
CAPTCHA

BBCode format allowed
 
 

Quicksearch

Comments

delbertino about phpvikinger.org: Things that have no name
Sun, 11.05.2008 06:34
In reply to "stuff with no nam es":very informative and succi nct. I am retired and need to learn to build a website [...]


ashok about What is the difference between MySQL and Postgres?
Wed, 30.04.2008 14:08
what is the difference between MySql and PostgreSql?


fortunecookienotes about Fortune Cookie
Wed, 09.04.2008 21:46
What a random fortune, who kno ws what it means. My favorite random fortune cookie note rea ds "you will make a good [...]


Harrison Fisk about Configuring InnoDB - An InnoDB tutorial
Thu, 27.03.2008 19:17
Hi Kristian, I noticed a fe w things that are wrong with y our above article. Overall it is good however, so don [...]


Erik about Rubyisms
Wed, 19.03.2008 15:01
Sorry, I should mention that t he create transaction simply i nvolves doing: content = Bl ogContent.new(params[:co [...]


Erik about Rubyisms
Wed, 19.03.2008 14:51
I like to separate text and bl ob columns into their own mode ls and tables, storing and man aging the record content [...]


James deville about Rubyisms
Wed, 27.02.2008 04:47
IMHO: Rails is being designed and primarily used by people w ith a dedicated DB. So the ide a of constraining in the [...]


James deville about Rubyisms
Wed, 27.02.2008 04:33
Rails only uses transactions i f you ask it to. If you just d o a People.find(:all), it does a select * from people, [...]


Categories


All categories

Authors


All authors

Syndicate This Blog

Blog Administration

Open login screen

Disclaimer

All examples, code-snippets, advice, trivia, etc. are supplied in the hope that they are useful, but no warranty of any kind is given. Use of trademarks is not intended as a challenge to them.

Impressum

Copyright 2005-2007 bei den jeweiligen Autorinnen der Beiträge. Die Infrastruktur von MySQL-dump wird betrieben von: Azundris, Immanuelkirchstraße 18, 710405 Berlin, Germany