+- CodeIgniter Forums (https://forum.codeigniter.com)
+-- Forum: Archived Discussions (https://forum.codeigniter.com/forumdisplay.php?fid=20)
+--- Forum: Archived Development & Programming (https://forum.codeigniter.com/forumdisplay.php?fid=23)
+--- Thread: class output cache: aren't there too many files in a single directory? (/showthread.php?tid=42207)
class output cache: aren't there too many files in a single directory? - El Forum - 05-30-2011
[eluser]Bad Penguin[/eluser]
I've enabled the output's cache on a website with like 69.000 pages
all the cache's files are created in a single directory
isn't this a performance problem when there are too many files in a single directory?
does it make any sense to extend the output class to use a 2-level subdirectories
system for those cached views?
class output cache: aren't there too many files in a single directory? - El Forum - 05-30-2011
[eluser]atno[/eluser]
[quote author="Bad Penguin" date="1306783853"]I've enabled the output's cache on a website with like 69.000 pages
all the cache's files are created in a single directory
isn't this a performance problem when there are too many files in a single directory?
does it make any sense to extend the output class to use a 2-level subdirectories
system for those cached views?[/quote]
on unix systems with ext3 each directory can have up to 32k files/folders, yes you better split your cache files in multiple dirs within your cache folder
class output cache: aren't there too many files in a single directory? - El Forum - 05-30-2011
[eluser]WanWizard[/eluser]
Depends a bit on the filesystem used, and the speed of the storage system.
On Windows, you definately want to limit the number of files in a directory to a few thousand. On Linux (ext3), thousands of files is not a problem. You can increase the speed on ext3 by formatting the volume with the 'dir_index' parameter, which implements a btree algorithm for directory indexes.
How big are these cache files? Maybe you can get a better speed gain if you cache in memory (a simple memdisk would be enough, other solutions require a extended cache library).
class output cache: aren't there too many files in a single directory? - El Forum - 05-30-2011
[eluser]WanWizard[/eluser]
[quote author="atno" date="1306789707"]on unix systems with ext3 each directory can have up to 32k files/folders, yes you better split your cache files in multiple dirs within your cache folder[/quote]
Sub directories, yes, this due to the limit of links per inode. ext3 doesn't have a file limit.
I've had a directory with over a million files once, delay wasn't really noticable (with dir_index enabled). Most annoying was that most commandline tools stop working (forget doing an 'ls' in such a directory).
class output cache: aren't there too many files in a single directory? - El Forum - 05-30-2011
[eluser]Bad Penguin[/eluser]
[quote author="atno" date="1306789707"]yes you better split your cache files in multiple dirs within your cache folder[/quote]
They are not my files but the codeigniter's default cache system from the Output class
