Caching and content

[eluser]innocast[/eluser]
OK, the next problem I ran into was that I'm caching a specific type of content for a very long time. Let's assume it's user profiles and we've got 100.000 users. Having 100.000 files in the same directory = VERY BAD. The load would go sky high due to the I/O search for the correct file (takes several seconds just to find the file).

A smart way to handle this is to store them in sub directories!

Users ID = 1234567890

[path_to_cache]/profiles/1/2/3/4/5/1234567890.cache

This would speed up the search enough.

Anyway, MP_Cache doesn't support this buy default. You can just have 1 subdirectory (if the parent is created). Or you can have the page loaded 5 times (1 directory created on each load) to make the directories exist.

OR you can use this hotfix:

Change this:

Code:

// create non existing dirs, asumes PHP5
if ( ! @mkdir($test_path)) return false;

Into this:

Code:

// create non existing dirs, asumes PHP5
if ( ! @mkdir($test_path,0777,TRUE)) return false;

The third argument (TRUE) is for recursive creation of the directories. And voila!

[eluser]innocast[/eluser]
Also note, if you're using PHP SAFEMODE this can cause some errors since the directories is owned by apache/httpd or something similar so the script isn't allowed to write files in the directories.

[eluser]innocast[/eluser]
Function for "catting" strings

Code:

function cats($str,$limit=5)
{
    $return = '';
    for( $i=0;$i<$limit;$i++ )
    {
        if ( isset($str[$i]) ) $return .=  $str[$i] . '/';
    }
    return $return;
}

Use:

Code:

cats('1234567890') // Returns 1/2/3/4/5/
cats('1234567890',2) // Returns 1/2/
cats('abcde',3) // Returns a/b/c/

Don't forget that a/b/c/ is one path and a/B/c is another path!

[eluser]Jondolar[/eluser]
I use a pattern of 12/34/ instead of 1/2/3/4. If your files are built from a hash then each filename letter position has 16 possibilities so 16*16 = (thinking...) 256 so you get 256 directories max per level instead of 16. The reason I do this is because I believe it is slightly faster to have your directories wide rather than deep (up to a point). I believe it is faster for the OS to find a directory out of 256 than to find a directory out of 16 then another directory out of 16. It's only when a directory gets 1,000s of subdirectories does it lose it's efficiency.

[eluser]innocast[/eluser]
[quote author="Jondolar" date="1244324541"]I use a pattern of 12/34/ instead of 1/2/3/4. If your files are built from a hash then each filename letter position has 16 possibilities so 16*16 = (thinking...) 256 so you get 256 directories max per level instead of 16. The reason I do this is because I believe it is slightly faster to have your directories wide rather than deep (up to a point). I believe it is faster for the OS to find a directory out of 256 than to find a directory out of 16 then another directory out of 16. It's only when a directory gets 1,000s of subdirectories does it lose it's efficiency.[/quote]

Ah, smart thinking! Will try that out!

[eluser]Xeoncross[/eluser]
If you real need performance then you might try memcached. While I don't use Kohana, it has 5 different caching files that you can steal for this. (File,memcache,APC, etc..).

I'm hoping that someday CodeIgniter will implement my caching libs changes into the core cache so that partial caching isn't so hard to do and we don't have to go off to other projects to find the parts we need.

[eluser]Jelmer[/eluser]
@innocast

Haven't had the time to look into your points completely, but some stuff looks like it works correctly.

I did indeed mess up the variable setting for the default expiration (both name and forgot to add time() to the value). Just fixed that in my version on the website as well.

But the output you described for the line below is correct:

Code:

$this->contents = unserialize(fread($fp, filesize($filepath)));

It's put in an array to prevent problems with caching anything other than an array. If you don't do it this way and you'd try to add the options like creation time, expiration time & dependencies to an object, string or integer you'll have a problem. Your problem wasn't caused by that line, but you might have taken a shortcut that solved the problem for you.

The fact that your contents is within an array is solved a couple of lines below by this line:

Code:

// Cleanup the MP_Cache variables from the contents
$this->contents = $this->contents['__mp_cache_contents'];

Also about the subdirectories: That should work without any problem by just putting the directory name in there: instead of 'name' call it 'dir/dir2/name'.

Which is done by the lines below (lines 224 through 237 of the original within the write() function):

Code:

// check if filename contains dirs
$subdirs = explode('/', $this->filename);
if (count($subdirs) > 1)
{
    array_pop($subdirs);
    $test_path = $this->path.implode('/', $subdirs);
    
    // check if specified subdir exists
    if ( ! @file_exists($test_path))
    {
        // create non existing dirs, asumes PHP5
        if ( ! @mkdir($test_path)) return false;
    }
}

[eluser]innocast[/eluser]
@Jelmer: Ahh, of couse it need it as an array. I'm just using arrays to store in cache, so no wonder it works. I stand corrected

Regarding the recursive directory creation, it can have been a PHP5 + SAFE_MODE related bug. I had to remove safe mode since the ownership didn't match, and that was after i changed the code =)

Anyway, I'm very please with the power of your code. Saved me a lot of "static" fetches from the database.

I'm also on my way to create a script that uses php-function filectime to loop through every directory and remove files older than maximum cache time. This script will be runned with a cronjob. Noticed that I've got some content that's rarely accessed, so I'm just stuck with a whole bunch of static data filling up the disk =)

[eluser]Phil Sturgeon[/eluser]
It's not finalised but here is my fork of Jelmers library. It has MANY improvements such as adding a library and model method which allows libraries and models to be called in cache mode, fixed the default expirey issue (expirey/expires variables named differently) and it now adds time() to the expire time so you dont have to do that yourself.

Cache library

Will package it properly when I find the time.