Welcome Guest, Not a member yet? Register   Sign In
Performance Problem [solved]
#1

(This post was last modified: 08-17-2017, 12:27 PM by bitwiz.)

I'm building a codeigniter project and I'm experiencing a strange performance problem. This issue is encountered on every single page of the site.

Environment:
Dell Poweredge R720XD, Dual E5-2660 CPUs, 100GB RAM, 2x Boot SSDs running CentOS 7 as a VM host, 10x 1.2TB SAS in RAID6 for VM storage, ISPConfig 3 installed on a CentOS 7 VM as a guest. We followed the CentOS 7 perfect server guide and our company has been running the ISPConfig hosting control panel for about 10 years, so we're very familiar with the server configuration.

The website is hosted on a guest VM with a 4 other sites - so not a lot of activity or load here. There are a couple other VMs on this server, but they are for our company of 3 people and there's practically zero activity across all VMs. All our VMs respond quickly as expected. We plan to move more hosting accounts onto this server at some point, but right now it's virtually empty. We have a 25mbps x 25mbps fiber connection.

I also recently upgraded the site to the latest CI version, so it's current.

CI Symptoms:
1. Initial page load is extremely fast. A tiny fraction of a second.
2. If I click a link in the site, the next page loads slower, approx 15-30 seconds, then suddenly loads instantly.
3. If I then click again, the site hangs for upwards of 3 minutes, then after sitting and waiting the page suddenly loads instantly.

Likewise, below will also produce the same effect:

1. Load a page - instant.
2. Refresh the page, slower, around 15-30 seconds.
3. Refresh the page again, hangs for up to 3 minutes, then suddenly loads instantly.

I've added debug to the code and it confirms the software times are huge, but there's no activity during the hanging. The loading indicator in the browsers just sit and spin. Yes I've tested it cross-browser, it's not the browser.

I've switched caching between the DB and Files. No effect.

I've scoured through the code and minimized database calls, there's no database activity during these random hangs, there's no DB activity at all.

Running the "top" command on the server shows no activity during one of these tests.

We are using MariaDB as the MySQL engine. No errors in the server logs or hosting logs. I own and run the server.

What could be happening? It seems like some strange compatibility problem between CI and Apache or PHP. This web server also hosts a couple other sites: Wordpress and Prestashop, neither of which show these symptoms.

I have bandwidth monitoring on our internet connection, no bandwidth activity during these tests.

I have PHP 5.6 is installed on the server.

I think I would see 100% CPU activity on one processor thread if my code was getting stuck in a loop somewhere.

If I wait for a while (not sure how long - roughly 5-10 minutes), a page load is nearly instant again, but no visitor will wait 5-10 minutes between clicks!

I'm at a loss how to diagnose and/or fix this issue. Obviously visitors aren't going to wait 5-10 minutes between clicks to get a fast response. CI gurus, HELP! What do I do? Where can I look? What tools would you use to figure out what's going wrong? Have you seen this before in your own code development? Any thoughts what else could be the culprit or where to look?

Thanks for any assistance!
Reply
#2

Any chance you have one or more ad servers or third-party tracking scripts on the site? I've seen those cause pretty severe slowdowns because they're so busy.
Reply
#3

(08-16-2017, 07:50 PM)kilishan Wrote: Any chance you have one or more ad servers or third-party tracking scripts on the site? I've seen those cause pretty severe slowdowns because they're so busy.

Thanks for the reply! Good idea, but no, no ad services.

I also forgot to mention when I enabled the CI debug, the 6 or so DB queries were all listed as 0.001 sec.
Reply
#4

do you use ajax and session?

https://www.codeigniter.com/user_guide/l...oncurrency
Reply
#5

My best guess would be the same as Kilishan, perhaps some api call is used in your code to an external source which does not perform very well. Try to identify it and possibly cache the result to limit the rate at which the calls are made to lets say once an hour or so.

Does fetching the pages with a commandline tool like wget or curl result in the same delay? That way you can make sure that apache/php is indeed still busy (or more likely just waiting) instead of some weird javascript related client side issue.

You could try to use apache's mod_status to get an inpression of how many connections apache is currently processing.

Have you looked at tools iotop to see if there are weird activities in disk I/O?
Reply
#6

(This post was last modified: 08-17-2017, 06:54 AM by bitwiz. Edit Reason: Clarification )

(08-16-2017, 09:46 PM)Paradinight Wrote: do you use ajax and session?

https://www.codeigniter.com/user_guide/l...oncurrency

I'm using sessions. Not using ajax.

I'm wondering about session_write_close() though. Would I put that at the end of all code execution for a particular page request? Would that go in the controller right before calling a particular view or at the end of the code for a view? Is there anything that's called after the view is loaded in CI?

That link says: "Use session_write_close() after you’ve done processing session data if you’re having performance issues."

So I'm thinking since my controllers handle reading/writing session data before passing variables to the view, I can call this function right before loading the view.

Looking at the PHP page on session_write_close(), I guess I'm not familiar with session locking. I think I understand the purpose, similar to DB transactions where you wouldn't want the same record(s) modified by two different scripts at the same time. I'll try this.

Based on the PHP page, I should call session_write_close() as soon as possible in case any CI projects become busy sites. What impact would this have on a very busy site (assuming this is my problem)? Does the session immediately restore close to the beginning of code execution? I'm wondering how I can limit the window where the session locking occurs.
Reply
#7

(This post was last modified: 08-17-2017, 07:37 AM by bitwiz.)

(08-16-2017, 09:46 PM)Paradinight Wrote: do you use ajax and session?

https://www.codeigniter.com/user_guide/l...oncurrency

So I was thinking about this. I understand if you had a lot of simultaneous AJAX, it would be advantageous to call session_write_close() as soon as possible for maximum speed.

I have a small, low traffic site right now and it does not currently utilize AJAX.

PHP usually calls session_write_close() automatically after code execution (and I was used to that happening), I read on the forum someone asked if it was okay to place it in __destruct() and the response was no and I didn't understand their reasoning, sorry I don't remember where I read that. If I want to simulate PHP's call on script completion where would I place session_write_close() or is there a way to turn that automation back on?

Is that this variable? $config['sess_regenerate_destroy'] = FALSE;
Reply
#8

(This post was last modified: 08-17-2017, 08:51 AM by bitwiz.)

(08-17-2017, 12:15 AM)Diederik Wrote: My best guess would be the same as Kilishan, perhaps some api call is used in your code to an external source which does not perform very well. Try to identify it and possibly cache the result to limit the rate at which the calls are made to lets say once an hour or so.

Does fetching the pages with a commandline tool like wget or curl result in the same delay? That way you can make sure that apache/php is indeed still busy (or more likely just waiting) instead of some weird javascript related client side issue.

You could try to use apache's mod_status to get an inpression of how many connections apache is currently processing.

Have you looked at tools iotop to see if there are weird activities in disk I/O?

iotop shows no activity. I'm not familiar with mod_status, I'll read about it.
Reply
#9

(08-17-2017, 07:22 AM)bitwiz Wrote: PHP usually calls session_write_close() automatically after code execution (and I was used to that happening), I read on the forum someone asked if it was okay to place it in __destruct() and the response was no and I didn't understand their reasoning, sorry I don't remember where I read that.

__destruct() will rarely be called when you need it, and is kind of unpredictable at times.

(08-17-2017, 07:22 AM)bitwiz Wrote: If I want to simulate PHP's call on script completion where would I place session_write_close() or is there a way to turn that automation back on?

Is that this variable? $config['sess_regenerate_destroy'] = FALSE;

No, sess_regenerate_destroy has nothing to do with it and it's already automated - PHP will close the session itself. Point of calling session_write_close() yourself is that you know it will be beneficial to do it at a certain execution point, there's no way to automate that.
Reply
#10

(This post was last modified: 08-17-2017, 09:02 AM by bitwiz.)

(08-16-2017, 09:46 PM)Paradinight Wrote: do you use ajax and session?

https://www.codeigniter.com/user_guide/l...oncurrency

I added 
Code:
if (session_status() == PHP_SESSION_ACTIVE) session_write_close();
before loading the views. No change in performance, it's still happening.

Also tried without the if test:
Code:
session_write_close();
Still the same issue.
Reply




Theme © iAndrew 2016 - Forum software by © MyBB