Ideas for an improved Query Caching solution.

[eluser]TheLoops[/eluser]
Wiki: Wiki Article
Source: Google Code
Known Issues: Google Code



- - - - - - - - - - - - - - - -

Hi there,

I'm wondering about what's the purpose of having DB caches connected to a controller rather than the actual model that run the query.

Controller-based caching works fine with small and decentralized pages where all controllers are pretty much independent.
But as soon as you have a model that's shared by a handful of controllers, you end up with a big mess.

Just take a model for generating the data for a tag cloud that's displayed on every page.
You would end up with dozens of duplicates and handling those caches would suck as hell.

Why doesn't support some alternative way to bind caches to the model that's actually running the db query?


I for one wouldn't even bind caches to models. I'd instead bind them to the actual database-tables that they deals with!
(this would be little more difficult to implement for queries dealing with multiple tables, like joins, though)
Not only would this make it much easier to keep your cache files updated, it would also be much more precise.

…or am I just missing the point of caching with models being shared by multiple controllers?

I searched the forum and couldn't find a single discussion about this issue.
Well, I actually found one topic: Query cacher - Model instead of Controller/action based
But it provides no code though and has remained unanswered so far.

Thanks in advance,
Loops

[eluser]Derek Allard[/eluser]
We know that controller caching isn't particularly suited to many people's uses. The caching system was implemented before CI used models as it currently does. We're actively looking at ways to improve it.

I'd like to actively engage you for your help. Do you have ideas, code? Have you implemented another solution, and if so, is that solution compatible with PHP4 and 5? Has it been tested in multiple server environments?

The fastest way to get that implemented into CI core, is to save us time

I see this is your third post, welcome to CodeIgniter Loops.

[eluser]TheLoops[/eluser]
Well, the best approach would probably be to let the user choose their preferred DB caching mechanism from a list of options like:

# # # # #

1. Controller-based for controllers-centric and fairly simple projects with mostly independent controllers.
(already available in CI 1.6.x)
No further code would be necessary.

Cache files would be stored like this:
application / cache_database / controller_based / controller + function / md5hash

# # # # #

2. Model-based for model-centric and semi-complex projects with controllers sharing the same model.
(not yet available in CI 1.6.x)

Cache files would be stored like this:
application / cache_database / model_based / model / md5hash

Likely implementation issues with this option:
It would require some method for detecting the name of the model which is running the DB query.

The code for this mechanism would be quite similar to the controller-based approach.

# # # # #

3. DBTable-based for all of us who want the best possible performance and most advanced caching management.
(not yet available in CI 1.6.x)

Cache files would be stored like this:
application / cache_database / table_based / + table1 + table2 + table3 + md5hash

Likely implementation issues with this option:
It would require some reliable parsing methods for detecting the name of the tables used in the DB query.

# # # # #

4. Key=>Value-based for a lightweight but useful caching option.
(not yet available in CI 1.6.x)

Cache files would be stored like this:
application / cache_database / key_based / key

# # # # #

One might even consider allowing geeky users to use all available methods simultaneously.

[eluser]Seppo[/eluser]
I just want a simple Cache system with key/value, like Memcache interface, but with multiple drivers (like storing in to files, database, memcache itself, etc)

[eluser]TheLoops[/eluser]
The implementation of model-based caching should be possible by slightly modifying the existing controller-based mechanism.

# # # # #

DBtable-based caching would need some new functions and stuff

Same for Key=>Val-based caching.

[Edit: I'm currently working on this. Looking good so far, stay tuned ]

[eluser]Derek Allard[/eluser]
Hi Loops. I see you've posted something on the wiki and contacted me privately. Thanks. I think it'd be better to keep this discussion public, and to solicit community input. Could you explain a bit what you've done, and how it is used. Docs would be ideal. Additionally, we need to be sure that PHP 4 works, and that no backwards compatibility is broken.

Thanks for your work so far, I hope the community is able to offer great feedback.

[eluser]TheLoops[/eluser]
[quote author="Derek Allard" date="1209597167"]Hi Loops. I see you've posted something on the wiki and contacted me privately. Thanks. I think it'd be better to keep this discussion public, and to solicit community input.[/quote]Right, I just wasn't sure if it wasn't to early. Never mind.
[quote author="Derek Allard" date="1209597167"]Could you explain a bit what you've done, and how it is used. Docs would be ideal. Additionally, we need to be sure that PHP 4 works, and that no backwards compatibility is broken.[/quote]Well, basically I have built three additional options for cache file storing.
So with my code CodeIgniter now supports four different ways to store/manage cache files.

You should be able to find any necessary documentation on this Google Code Project as well as the actual source code.

As far as am aware of I did not use any PHP5-only functions etc. So it should be pretty much backwards compatible.

[quote author="Derek Allard" date="1209597167"]Thanks for your work so far, I hope the community is able to offer great feedback.[/quote]So do I Gimme your feedback, m8s!

If you have any further questions - feel free to ask!

[eluser]Firestorm ZERO[/eluser]
I voted for the Model-based caching as I think that would be best for my site. Also I think it would be the easiest to implement at the moment. Making it totally customizable for the developer and putting all options would be best option of course .

I don't like the current controller-based cache that is currently in CI. As I currently pull data from various controllers. So whenever I update to the database, I just do a complete wipe of the cache as there are too many places to delete the specific caches (and hard to keep to track of it right now). But to me, that's like using napalm to clear the cache :/

[eluser]BorisK[/eluser]
I found KhCache to work just great. It's a simple key-value cache system that works well with any type of data. It uses JSON to serialize the value, so you can store DB results, views, variables, function results, etc.

I use it to cache Ajax views of dictionary searches at Metak.com. I run a cron job to delete any views which were not accessed in the last 4 days. The most popular dictionary searches are practically cached forever, while less popular words and typos are deleted to save the disk space for other apps.

I plan to work on a mod of Khcache which would allow the web server, Apache, to check if the cached view exists, and if so to return the result without spawning a PHP process. Once that is done, I expect performance to double and memory usage to half.

[eluser]TheLoops[/eluser]
Right, there do exist several 3rd party libraries for caching.

So one might think…

…no, please not another 3rd party caching library. >

But this project is not meant to be used as a hack or whatever but to improve CodeIgniter's native caching.
It is rather an attempt to come up with an improved caching mechanism to be included natively with CodeIgniter a.s.a.p. ;-)

So everyone's input, be it code or just feedback is much appreciated.