Serving Images from a File System

Tuesday, March 14. 2006

Serving Images from a File System

In Serving Images from a Database, Sheeri responded to me:

I appreciate your ideas, but you haven't read what I've written: "the sheer number of files is enough to kill any os."

I'm serious about that. we're talking 36 million images -- that's 36 million files (no thumbnails, they're all reduced size) plus directories to store them in. We have cacheing tuned very well, and it's still kills the OS.

Here is what I tried to simulate that:

I took my new Strato MR2. It is currently running tor, INN and MySQL. There are two HDS722516VLSA80 160 GB SATA HDDs. These disks are rated at 8.8ms avg seek time, so there are 113 seeks in a second with this disk.

I am using one otherwise idle disk to rebuild Sheeri's test setup:

CODE:

kris@h743107:~/x> df -Th .
Filesystem    Type    Size  Used Avail Use% Mounted on
/dev/sdb2 reiserfs    153G   55G   98G  36% /home
kris@h743107:~/x> free -m
             total       used       free     shared    buffers     cached
Mem:          1010        996         13          0        224        346
-/+ buffers/cache:        425        584
Swap:         3074          3       3071
kris@h743107:~/x> cat /proc/cpuinfo
processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 15
model           : 39
model name      : AMD Opteron(tm) Processor 146
stepping        : 1
cpu MHz         : 1995.305
cache size      : 1024 KB
...

There is a bit of code I am using to generate 32 Megafiles, each 1KB in size.

CODE:

kris@h743107:~/x> cat testscript.pl
#! /usr/bin/perl -w

sub create_dir($) {
        my $name = shift;
        mkdir $name;
}

sub create_file($) {
        my $name = shift;

        open FILE, ">$name" or die "cannot create $name";
        print FILE "x"x1023, "\n";
        close FILE

}

for (my $m=0; 2>$m; $m++) {
        $dir  = sprintf("%02x", $m);
        create_dir($dir);

        for (my $i=0; 256>$i; $i++) {
                $dir  = sprintf("%02x/%02x", $m, $i);
                create_dir($dir);

                for (my $j=0; 256>$j; $j++) {
                        $dir  = sprintf("%02x/%02x/%02x", $m, $i, $j);
                        create_dir($dir);

                        for (my $k=0; 256>$k; $k++) {
                                $name = sprintf("$dir/%02x.jpg", $k);
                                print "$name\r";
                                create_file($name);
                        }
                }
        }
}

When running this, I am getting approximately 320,000 files created per minute. This is sustained through the entire creation process, and does not slow down at all.

CODE:

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sdb 0.00 2378.00 0.00 97.00 0.00 19744.00 0.00 9872.00 203.55 7.58 100.66 4.87 47.20

As you can see, the disk is not yet maxed out, so there probably is room for tuning (I did not tune anything).

We are seeing a service time of 4.8ms, so probably every second request involves seeks, and we see 90 w/s - following that math, that would be 45 "w/s with seeks" out of 113 "w/s with seeks" possible.

A very trivial accessscript can look like this:

CODE:

kris@h743107:~/x> cat accessscript.pl
#! /usr/bin/perl -w

my $lim = 32*1024*1024;
my $lim = 0.5*1024*1024;

my $newtime = 0, $oldtime = 0;
my $counter = 0;
my $x;

while (1) {
        my $a = int(rand $lim);
        my $m = ($a>> 24);
        my $i = ($a>> 16) & 255;
        my $j = ($a>> 8) & 255;
        my $k =  $a& 255;
        $name = sprintf "%02x/%02x/%02x/%02x.jpg", $m, $i, $j, $k;

        $counter++;
        $newtime = time;
        if ($newtime != $oldtime) {
                print "counter = $counter\n";
                $counter = 0;
                $oldtime = $newtime;
        }

        open FILE, "<$name" or next;
        read FILE , $x, 1024;
        close FILE;
}

When you start this, it is very slow at first, but the buffer and dentry caches will heat up eventually. I am running all the following tests with remounting the filesystem with "noatime".

Setting $lim to 1E4 (10,000 files of working set) runs from the cache completely:

CODE:

counter = 45305
counter = 45307
counter = 44972
counter = 45316
counter = 45310
counter = 45307

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sdb 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00

Setting $lim to 2E5 (200,000 files in working set) yields

CODE:

counter = 156
counter = 155
counter = 187
counter = 154
counter = 181

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sdb 0,00 0,00 103,00 0,00 824,00 0,00 412,00 0,00 8,00 1,00 9,67 9,67 99,60

at first. You have to let it run quite a while for the system to heat up. After 10 minutes of curning, the numbers will read

CODE:

counter = 1748
counter = 1800
counter = 1987
counter = 1802
counter = 1882

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sdb 0,00 0,00 77,00 0,00 608,00 0,00 304,00 0,00 7,90 0,85 11,17 11,01 84,80

and from there they will eventually increase more rapidly to

CODE:

counter = 4562
counter = 4112
counter = 5572
counter = 5544
counter = 5116

Observing /proc/sys/fs/dentry-state over time shows the system reconfiguring itself:

CODE:

61796   58525   45      0       0       0
65698   62366   45      0       0       0
...
91724   87924   45      0       0       0
91843   88042   45      0       0       0
...
107483  102187  45      0       0       0
106838  101579  45      0       0       0

Larger tests take even more time. I will eventually edit this post and add to it.

Posted by Kristian Köhntopp in Scaleout and Tuning at 11:40 | Comments (4) | Trackback (1)

Trackbacks

Trackback specific URI for this entry

Statification
In Semi-Dynamic Data, Sheeri writes about Semi-Dynamic Data and content pregeneration. In her article, she suggests that for rarely changing data it is often adviseable to precompute the result pages and store them as static content. Sheeri is right: Noth

Weblog: MySQL-dump
Tracked: Aug 17, 19:06

Comments

Display comments as (Linear | Threaded)

I'm glad someone's actually trying! And indeed, our problem is when the cache is empty, and performance of the site suffers greatly. As we expand more and more, that 36 million number will only grow larger -- we project by the end of 2007 that number will be 330 million. Which means the cache won't always be hit (sure we can add memory)....etc.

We're sacrificing small bits of performance, this is true. It's not something I'm denying. But we're sacrificing a bit of performance always, to maintain consistent performance. ie, a 5% performance hit to make sure that those days when the filesystem cache dies that it doesn't slow down the entire site.

Not that anyone asked. Seeing folks scream "BUT YOU'LL TAKE A PERFORMANCE HIT!" is amusing, but unhelpful. In the long run, it's better. For us. In this narrow case.

I'd like to see a comparison between retrieval time from disk versus retrieval time from db. overhead, yes, but it's low enough that we can live with it!

#1 Sheeri (Homepage) on 2006-03-14 15:42 (Reply)

QUOTE:

I'd like to see a comparison between retrieval time from disk versus retrieval time from db. overhead, yes, but it's low enough that we can live with it!

I'm afraid my statistics went in the recycle bin once I satisfied myself of the results, but I recall seeing between 0 and a 10% hit for using the database to serve smallish images.

I took five measurements, threw out the outliers, and averaged the middle three. Still, some trials showed database performance hits below the "noise level" -- the variations between individual measurements were greater than the average difference between file system and database.

I decided that, in my particular case, the many advantages to storing the images in the database far outweighed the relatively small speed impact. So many other things have bigger impact. (Having an image much bigger than the display size, for example -- I pre-scale images to match display size when I store them in the database in three sizes.)

The funny thing about "conventional wisdom" is that it generally respects only other people's conventions. You can "prove" anything you want on paper!

#1.1 Jan Steinman (Homepage) on 2006-03-14 17:28 (Reply)

i agree, and dont forget the update case. Deleting files and replacing them with bigger severly will result in fragmentation. (However growing files are much much worse than a few blocks variation in images)

One advantage of db backed up files (besides searching for attributes, which can also be solved with a hybrid solution) is backup/replication.

Some filesystems may also degenerate much faster, especially if the main memory of the system not only can be used for dcache.

Me thinks a fair compariion would compare a database server against an NFS server or cluster filesystem. I guess nobody would want to scale-up in an web-server scenario.

Gruss
Bernd

#1.1.1 eckes (Homepage) on 2006-03-14 21:50 (Reply)

For the sake of backup and clustering, I still don't get why people scream when you save images to a database.

Point taken that it's maybe not as fast (and so on), then serving directly from a filesystem (using a small httpd), yet, the features (clustering, backup) easily outrule the loss of performance you encounter.

Benchmarks are one thing, real-life experiences another. The TOC on buying another node outrules tweaking servers for hours and hours, and hours because you gain too little. I am not saying that it is sometimes not useful to do this, or one should not write effecient code. Not at all!

But this is along the lines of writing hard to maintain code/applications for the sake of microseconds here and there.

#2 tfk on 2006-03-15 18:34 (Reply)

Add Comment

Name
Email
Homepage
In reply to
Comment	E-Mail addresses will not be displayed and will only be used for E-Mail notifications. To prevent automated Bots from commentspamming, please enter the string you see in the image below in the appropriate input box. Your comment will only be submitted if the strings match. Please ensure that your browser supports and accepts cookies, or your comment cannot be verified correctly. Enter the string from the spam-prevention image above: BBCode format allowed
	Remember Information?

Quicksearch

Comments

ddd about Waiting for a full mysqldump

Sun, 14.07.2013 11:45

ddd

Kristian Köhntopp about Enemy Action

Wed, 31.10.2012 13:46

My friendly Offtopic Channel in IRC has horror stories about this, it is Halloween, after all. "Well, I do have [...]

Shibabrata about How large can a MySQL database become?

Mon, 15.10.2012 09:41

What if we use SSDs. Can we then assume memory limits are not as significant because SSDs can handle random access [...]

Kristian Köhntopp about MySQL 5.6-RC: GTID vs. MyISAM

Fri, 05.10.2012 07:46

In fact, I am pretty certain that this also breaks mysql_upgrade or has the potential to do so.

Kristian Köhntopp about MySQL 5.6-RC: GTID vs. MyISAM

Fri, 05.10.2012 06:44

It is quite a bit worse than that: Right now, the default RPM as distributed by Oracle will fail to install if it [...]

Davi Arnaut about MySQL 5.6-RC: GTID vs. MyISAM

Thu, 04.10.2012 18:52

> MyISAM in mysql.* has to go before GTID can be sucessfully deployed. I think you meant to say before it can b [...]

Davi Arnaut about MySQL 5.6-RC: GTID vs. MyISAM

Thu, 04.10.2012 18:36

Sorry, it wasn't clear given that none of this is actually mentioned.

danny about MySQL 5.6-RC: GTID vs. MyISAM

Thu, 04.10.2012 13:47

You can turn it on with FLUSH PRIVILEGES once the tables exist.

Disclaimer

All examples, code-snippets, advice, trivia, etc. are supplied in the hope that they are useful, but no warranty of any kind is given. Use of trademarks is not intended as a challenge to them.

Serving Images from a File System

MySQL-dump

Tuesday, March 14. 2006

Serving Images from a File System

Quicksearch

Comments

Categories

Authors

Syndicate This Blog

Blog Administration

Disclaimer

Impressum