Welcome Guest, Not a member yet? Register   Sign In
Paginate Processor Load
#1

(This post was last modified: 05-29-2021, 05:15 AM by Gary.)

Having recently moved a development from a Windows-based to a Linux-based Apache (MPM) platform, I find that the pagination function appears to cause a particularly heavy load on the processor - and this is with a tiny test data-set.

Although I appreciate that some of this is likely tied into the particular implementation of Apache, I do recall being particularly concerned as to how much data potentially needed to be processed with each page-flip when I first started using CI's paginate() and making a mental note to keep my eye on it.  I hadn't expected it to be quite so soon...

Has anyone else noticed this... or perhaps experienced it crippling a production site under load?

Comments and thoughts would also be appreciated, thanks.
Reply
#2

CodeIgniter's caching may help in this matter.
What did you Try? What did you Get? What did you Expect?

Joined CodeIgniter Community 2009.  ( Skype: insitfx )
Reply
#3

How many rows do you have? I have made a test app to try the paginate function and with over 500 rows it is super quick. Did you forgot to add a primary key to your table? Maybe MySQL always do a full table scan instead of using the index on the id column.
CodeIgniter 4 tutorials (EN/FR) - https://includebeer.com
/*** NO support in private message - Use the forum! ***/
Reply
#4

(This post was last modified: 05-30-2021, 03:40 PM by Gary.)

(05-29-2021, 05:17 AM)InsiteFX Wrote: CodeIgniter's caching may help in this matter.

Thanks for the suggestion InsiteFX.  So far I've mostly ignored CI caching, having left it on a list of things to do once I had something up and running... I'll have another look over the caching section of the user manual to see if there's anything that I can currently push into my code.

(05-29-2021, 09:01 AM)includebeer Wrote: How many rows do you have? I have made a test app to try the paginate function and with over 500 rows it is super quick. Did you forgot to add a primary key to your table? Maybe MySQL always do a full table scan instead of using the index on the id column.

Thanks includebeer, your test and observation made me put the whole offending function (query set-up, and execution, pagination and prep for the View) in a for() loop... perhaps something I should have tried from the offset (clearly programming under the influence isn't always helpful)... and found that it had to loop many thousands of times before the delay became observable on the client (and in spite of this, the reported processor loading didn't seem to change significantly, even thought the process was taking fractionally longer)... which seems points toward something pushing the processor into a wait-state (which typically counts at 100% usage during the wait portions of the time 'slices') during (or overlapping with) the pagination function, such that the > 5000x more processing still doesn't make an appreciable difference to the overall work-load (I have double checked MariaDB has the query cache disabled (which it's supposed to be by default from 10.1.7)).

My test data-set currently only has about 30 records... but I had made certain to ensure that (when it grows) any query (this particular one included) that may have even the slightest chance of being time-consuming and impact on performance had a covering index to minimise the damage... which was a large part of my disbelief and dismay at the notable poor performance and heavy processor load during the function call.  In my initial investigation, I had commented out anything that might have even slightly impeded the displaying (in the View), and done some eliminations in the function invoking the pagination (which appeared to make the problem go away, and lead me to mistakenly finger pagination as the guilty party).

The weird thing is the View and response prep routines are universal throughout the whole site, as is the pagination functionality) and I'm not seeing any of this processor loading where there's no significant pagination effort required.

The other observation is that with the server taking longer to respond, the client (currently run on the same development machine) is behaving in a way that I can only describe as more smoothly with it's page update (the client side shows an indication that the server is busy)... so perhaps there's some interaction or wait-state bug with my JavaScript code that's now showing up on the Linux machine for the first time.  Or perhaps I'm just grasping at straws.

There is a fair amount of JavaScript flapping around on the client side... though, here again, this is the same routines that service the rest of the site... but as I can't think of anything else that's simple to try, I'll move the client off the development machine and check if there's any difference in the processor loading...
Reply
#5

Sorry for the late reply. I didn't see your post. The forum doesn't always send email notifications when there's new replies...
Anyway, maybe your performance problem has something to do with session locking. If the front-end makes multiple ajax queries at the same time, you can have performance issue where one query is waiting for the session lock to be released. Calling session_write_close() as early as you can, as soon as you no longer need the session data can help with this. Especially while you wait for the database query to return, you probably no longer need session data at this point.

Quote:Only MySQL and PostgreSQL databases are officially supported, due to lack of advisory locking mechanisms on other platforms. Using sessions without locks can cause all sorts of problems, especially with heavy usage of AJAX, and we will not support such cases. Use session_write_close() after you’ve done processing session data if you’re having performance issues.

Source: http://codeigniter.com/user_guide/librar...sions.html
CodeIgniter 4 tutorials (EN/FR) - https://includebeer.com
/*** NO support in private message - Use the forum! ***/
Reply
#6

(06-04-2021, 02:16 PM)includebeer Wrote: Sorry for the late reply. I didn't see your post. The forum doesn't always send email notifications when there's new replies...
Anyway, maybe your performance problem has something to do with session locking. If the front-end makes multiple ajax queries at the same time, you can have performance issue where one query is waiting for the session lock to be released. Calling session_write_close() as early as you can, as soon as you no longer need the session data can help with this. Especially while you wait for the database query to return, you probably no longer need session data at this point.

Quote:Only MySQL and PostgreSQL databases are officially supported, due to lack of advisory locking mechanisms on other platforms. Using sessions without locks can cause all sorts of problems, especially with heavy usage of AJAX, and we will not support such cases. Use session_write_close() after you’ve done processing session data if you’re having performance issues.

Source: http://codeigniter.com/user_guide/librar...sions.html

That's a good point, thanks includebeer... I do remember reading this a long time ago and making a mental note to keep my eye on it at the time (since this little project is pretty much all Ajax, besides the initial background page load)... of course, in spite of my good intentions, it still managed to quietly slip back into the Ether.

If I recall correctly... and don't quote me on this.. but, when I experimented with session_write_close() at the time, it caused a problem by pretty much completely destroying ALL the user's session data (and signing them out).  I'm using MySQL, so made the mental note, removed all of the session_write_close() references, and left it at that.  I'll put it on a PostIt, so it doesn't drop off the radar again before I get to the point of doing some stress-testing on the site.

What I did find, regarding the processor loading- after a bit of additional experimentation- was that if I accessed the page from a client on another machine (other than the development machine, which had both the server and client on it during my initial tests), it didn't seem to cause a problem... even the 5000x loop, although observable, the increase in load was reasonable, and easily within palatable bounds.  So I ended up attributed the problem to an interaction with, or anomaly in the client-side JavaScript in a very vague, but still all-encompassing, hand-waving conclusion that justified having another glass of whine and moving on (albeit that this phenomenon still hasn't been observed to affect the combined server/client on any other page, even when all the routines are the same ones being invoked for the problem pagination pages (?)).
Reply
#7

(06-19-2021, 12:42 PM)Gary Wrote: If I recall correctly... and don't quote me on this.. but, when I experimented with session_write_close() at the time, it caused a problem by pretty much completely destroying ALL the user's session data (and signing them out).  I'm using MySQL, so made the mental note, removed all of the session_write_close() references, and left it at that.

Hmmm... that's weird. I never had this problem where the session would be destroyed by session_write_close(). Maybe it was something else. I really don't know!
CodeIgniter 4 tutorials (EN/FR) - https://includebeer.com
/*** NO support in private message - Use the forum! ***/
Reply
#8

(This post was last modified: 06-29-2021, 03:25 PM by Gary.)

Because of it being a sideline issue at the time, and my poor recollection of exactly what was done, I may well be getting confused with session_close()... though, I do recall using and experimenting with the recommendations in the manual that you referenced... it may well have been other obtusities in related parts of my code falling over at the time.

Going on your recommendation, I've stuck in in session_write_close() in several places where I've purposefully blocked the code from exiting via the (after) Filters (using exit; to terminate)... just in case... and it didn't cause a problem.  Following that, in a bout of paranoia, I also stuck one of these session_write_close() at the end of the last (after) Filter... to ensure it will be called on every interaction.

Thanks includebeer.

As an aside, I see in my code there is now only a handful of exit; or exit('text_reason' || $message); in the system files (like system/Debug/Toolbar.php) that now terminate without a session_write_close()... possibly not important in many places (?), like the Toolbar that only exists in a non-production environment.
Reply
#9

This is interesting. In CI3 there's a call to register_shutdown_function('session_write_close'); in the session library's constructor that is no longer there in CI4. It doesn't need to be called explicitly, but only if you want to release the session lock early. So I'm pretty sure it's ok to call exit() without closing the session. It will be closed anyway.
Source: https://www.php.net/manual/en/function.s...-close.php
CodeIgniter 4 tutorials (EN/FR) - https://includebeer.com
/*** NO support in private message - Use the forum! ***/
Reply
#10

(06-29-2021, 05:12 PM)includebeer Wrote: This is interesting. In CI3 there's a call to register_shutdown_function('session_write_close'); in the session library's constructor that is no longer there in CI4. It doesn't need to be called explicitly, but only if you want to release the session lock early. So I'm pretty sure it's ok to call exit() without closing the session. It will be closed anyway.
Source: https://www.php.net/manual/en/function.s...-close.php

Yes, that is certainly useful to know... and is also good for putting one's alcohol-induced paranoia to rest... Thanks includebeer.

Some of my concern / interest / paranoia (relating to anything session) stems from a completely different problem that I'm currently battling to find a cure for (whereby the session disappears at what seems to be completely random times... sometimes a few seconds after a user signs in and other times a half-hour later, and it appears to be totally disconnected to what the user is, or isn't doing)... and it's only cropped up recently, in code that, as best I can remember, has not significantly changed from what's been working for many months.  I have another thread open on the (general) topic, but am still busy experimenting with the last suggestions made by InsiteFX and MGatner... though, from the testing I did this evening, it's starting to look like the sessions are getting lost because the session cookie is somehow being removed / emptied?!?  At any rate, that's a different problem, best left for another day and a full bottle of wine.
Reply




Theme © iAndrew 2016 - Forum software by © MyBB