Saving a lot of photos

[eluser]Sean Murphy[/eluser]
I'm just crafting one possible solution (of dozens) off the top of my head. So please, take it with a little salt, and the understanding that this is just an example and would obviously need some tweaking and more detail.

In this configuration we'll store all versions (sizes) of the same file on the same server.

Some key points to consider in setting something like this up are:
-hot/warm/cold storage
Media server local discs will be hot. AWS (S3) will be used for hot or warm storage.
-failover
Should a media server go down, files will be served from S3.
-load balancing
Load balancing will focus primarily on storage capacity and will be handled when storing new files.
-performance
Media servers should run a thin web server such as lighttpd or non-threaded Apache. It may also be wise to use a CDN.

In the CI config (on the app servers) you'll put an array containing a list of media servers. This array is used to determine where NEWLY uploaded files are stored. You might write a function for selecting a server that does a little dumb load balancing based on media server capacity, etc.

When a new file is uploaded you'll select a media server from those available, do any image resizing or editing needed, and transfer the files to the media server. You'll also transfer the same files to S3. You might do something like Edemilson Lima mentioned for your directory structure, and make sure to store in the DB which media server the file lives on.

It would probably be wise to have a function for building asset urls for use in your application views.

When a request comes in for something like http://media1.example.com/123456789/123456789.jpg it will go to the load balancer and, if the media1 server is up, the file will be served directly from there. If media1 is down, the LB redirects the request to an application server, which will then serve the file from S3.

I hope that helps at least some. There's a lot I glazed over. Let me know if you want more detail on something.