I'm building a codeigniter project and I'm experiencing a strange performance problem. This issue is encountered on every single page of the site.
Environment:
Dell Poweredge R720XD, Dual E5-2660 CPUs, 100GB RAM, 2x Boot SSDs running CentOS 7 as a VM host, 10x 1.2TB SAS in RAID6 for VM storage, ISPConfig 3 installed on a CentOS 7 VM as a guest. We followed the CentOS 7 perfect server guide and our company has been running the ISPConfig hosting control panel for about 10 years, so we're very familiar with the server configuration.
The website is hosted on a guest VM with a 4 other sites - so not a lot of activity or load here. There are a couple other VMs on this server, but they are for our company of 3 people and there's practically zero activity across all VMs. All our VMs respond quickly as expected. We plan to move more hosting accounts onto this server at some point, but right now it's virtually empty. We have a 25mbps x 25mbps fiber connection.
I also recently upgraded the site to the latest CI version, so it's current.
CI Symptoms:
1. Initial page load is extremely fast. A tiny fraction of a second.
2. If I click a link in the site, the next page loads slower, approx 15-30 seconds, then suddenly loads instantly.
3. If I then click again, the site hangs for upwards of 3 minutes, then after sitting and waiting the page suddenly loads instantly.
Likewise, below will also produce the same effect:
1. Load a page - instant.
2. Refresh the page, slower, around 15-30 seconds.
3. Refresh the page again, hangs for up to 3 minutes, then suddenly loads instantly.
I've added debug to the code and it confirms the software times are huge, but there's no activity during the hanging. The loading indicator in the browsers just sit and spin. Yes I've tested it cross-browser, it's not the browser.
I've switched caching between the DB and Files. No effect.
I've scoured through the code and minimized database calls, there's no database activity during these random hangs, there's no DB activity at all.
Running the "top" command on the server shows no activity during one of these tests.
We are using MariaDB as the MySQL engine. No errors in the server logs or hosting logs. I own and run the server.
What could be happening? It seems like some strange compatibility problem between CI and Apache or PHP. This web server also hosts a couple other sites: Wordpress and Prestashop, neither of which show these symptoms.
I have bandwidth monitoring on our internet connection, no bandwidth activity during these tests.
I have PHP 5.6 is installed on the server.
I think I would see 100% CPU activity on one processor thread if my code was getting stuck in a loop somewhere.
If I wait for a while (not sure how long - roughly 5-10 minutes), a page load is nearly instant again, but no visitor will wait 5-10 minutes between clicks!
I'm at a loss how to diagnose and/or fix this issue. Obviously visitors aren't going to wait 5-10 minutes between clicks to get a fast response. CI gurus, HELP! What do I do? Where can I look? What tools would you use to figure out what's going wrong? Have you seen this before in your own code development? Any thoughts what else could be the culprit or where to look?
Thanks for any assistance!
