Saving a lot of photos

[eluser]Ignacio[/eluser]
Wow, this is really new for me. Looks hard. I'm kinda dizzy about this threat, I'm keep reading about this.
I want to do step by step this, like a guide.

1) Like Edemilson Lima said, let's start with an empty folder (00000)
2) With a little script we can assign a folder for our photo
3) Assign a server using load balancing
4) After upload create thumbnail, resize, etc...
5) Like darksky0 said, then we are going to save the photos details in a db with:
id
file name
size
date
author
...
folder name
server location
6) With other script we can get the url of the photo and show it

As you see, I didn't understand pretty much about the load balancing thing


I asked to Cal Henderson (flickr's programmer) about this threat, he told me to check mogileFS (www.danga.com/mogilefs/)

Thanks

[eluser]Sean Murphy[/eluser]
Cal Henderson is freaking awesome. I saw a presentation he gave at FOWA Miami. It was the best one of the whole conference. Just throwing that out there...

[eluser]Daniel Eriksson[/eluser]
Most file systems perform best with a limited number of files/folders in each folder. Here's some (pseudo) code from a non-CI app I built a while ago. It sits in the file upload function and given a $fileid of 123456789 it will spit out a filename of /www/files/123/456/123456789 (and create the dirs if they are missing).

Code:

$base = '/www/files/';
$filename = sprintf('*09d', $fileid);
$dir1 = substr($filename, 0, 3);
$dir2 = substr($filename, 3, 3);
$fullname = $base . $dir1 . '/' . $dir2 . '/' . $filename;
if (!is_dir($base . $dir1))
  mkdir($base . $dir1, 0777);
if (!is_dir($base . $dir1 . '/' . $dir2))
  mkdir($base . $dir1 . '/' . $dir2, 0777);

This is suitable for storing up to 999999999 files (slightly less than 2^30). Just add another directory level if you expect more files.

I don't allow direct access to the files. Instead I serve them up using PHP-generated headers and readfile(). The MIME type for the header is stored in the database (removing the need to have any suffixes on the raw files).

Something like this:

Code:

$base = '/www/files/';
$filename = sprintf('*09d', $file->fields['fileid']);
$dir1 = substr($filename, 0, 3);
$dir2 = substr($filename, 3, 3);
$fullname = $base . $dir1 . '/' . $dir2 . '/' . $filename;
header('Content-Disposition: inline; filename="' . $file->fields['filename'] . '"');
header('Content-type: ' . $file->fields['content_type']);
readfile($fullname);

* = % (the code tags prevented me from entering % it seems)

/Daniel

[eluser]Ignacio[/eluser]
Thats awesome Daniel E, thanks for sharing. Now we just need to review the "load balancing" thing assign a server for each folders/photo. And write a nice adaptation for CI, ofcourse. I'm on it btw.

[eluser]Sean Murphy[/eluser]
This is the sort of thing I had in mind when I referred to dumb load balancing: http://www.spiteful.com/2008/03/17/progr...t-hashing/

MogileFS is nice too though. It obviously does more intelligent load balancing and handles all the directory structure/file naming stuff, along with resource location storage in DB. Only thing is, for the smaller side of things (2 media servers) I think using S3 might be a more cost effective solution to HA.

[eluser]Daniel Eriksson[/eluser]
[quote author="Ignacio" date="1207699835"]Thats awesome Daniel E, thanks for sharing. Now we just need to review the "load balancing" thing assign a server for each folders/photo. And write a nice adaptation for CI, ofcourse. I'm on it btw.[/quote]

Beware that my code above is not complete. Among other things you need to distinguish between content disposition types "inline" and "attachment", and sending the size is also a good thing.

/Daniel

[eluser]Ignacio[/eluser]
Thats ok, you just shared a good idea, I wasn't talking about the code it self. Thanks for sharing again.

[eluser]Ignacio[/eluser]
I'm stuck in the folder structure thing yet, I want to do something really really super scalable. For 1 image to 100.000.000.000 images. I like Daniel's idea, but I want to see more ideas.

Thanks.

[eluser]Sean Murphy[/eluser]
BTW, I just come across OpenAFS. Looks like an alternative to MogileFS you might want to look into.

[eluser]Ignacio[/eluser]
Thanks, I'll try it.