Website stops responding a few times a day, no apparent reason.

[eluser]alitech[/eluser]
Hi guys

This is my second time on the forum and once again I am facing issues that are unexplained or need some insight from you codeignitor gurus.

I will start from the top. I am not a developer so really I am just representing my developer who is apparently at his wits end with this stuff.

My website can be viewed at http://bit.ly/15XD6k (please refrain from using the real URL in your replies, a humble request)

My website grinds to a halt a few times a day and needs an MySQL restart to kick it back to life. I have had the website live for over a year now but cannot get rid of this issue. I have had profiling turned on and reduced the amount of queries to the database to a bare minimum. It was initially over 1000 for every page, its now down to less than 20 per page.

This has helped for faster loading (i mean really fast) of pages but has not helped in the frequent outtages that occur on a daily basis.

I can see from my apache status in cpanel the amount of requests active. I will post that when I see a spike, for now its stable. As all live requests are shown as "W" in cpanel, when the site is down, the simultanous occurences of "W" go up to about 160. Thats when the whole website comes to a halt. I have done whois lookups on the IP adddresses and it seems that these are all caused by SERPS. I have a dedicated server by the way.

Also, there is no pattern of the type of pages called otherwise we would be able to check the culprits and toothcomb the coding of these pages to find what elements are causing the delay. It seems as if it is caused by ALL pages on the site and triggers overload by SERPS. My suspicion is on a single element on the pages that may be causing this, but so far, we have not been able to trace any issues at all with the coding.

Perhaps its to do with the database? Whats the best way of optimising a database?

Any help in this matter is much appreciated. If you have any questions or want to better understand the problem, please let me know. I need to get this fixed asap. I am unable to grow because of this.

BTW, I also have a couple of instances of wordpress and phpbb runing on the same server, but the traffic is negligible so I dont suspect that.

HELP!

[eluser]Twisted1919[/eluser]
I had a bigger problem that yours, but mine was caused by the session library and had same behavior as you describe.
I used to use the CI default session library and save the sesssions data to database and sometimes just killed the website without reason(the project had somewhere at 50 K unique per day when i had this problem) . I replaced the session class with a custom one and didn't had issues .

If you save your session data to database, then this might be one of the reasons.

Over 1000 queries, really ? how come ? but the real question is, what did you do to reduce from 100 queries to 20 and still keep the features for the website ?

[eluser]smilie[/eluser]
Well, there could be a hundreds of reasons for that behavior.

Try to narrow down possible causes, such as:

- Dedi server is out of RAM, therefore it addresses the swap memory, which on it's turn 'kills' the hard disk;

- MySQL server which get's swamped with requests (bogus or legit one's);

- Apache which has too little (or too much) children / processes that Apache may summon upon;

- DDOS?

You stated that you see a hell of a lot "W" status in Apache. This means it is 'sending reply'. However, if MySQL 'died' it will stay in this state for ever. Can you collect some more MySQL stats when 'crash' occurs?

Also - change all logging (apache, mysql, server itself) to a near maximum as that holds the key to the solution. Do not overdo it - as too many logs can on their turn kill hard disk (too many write requests to HDD).

Cheers,
Smilie

[eluser]alitech[/eluser]
This is what cpanel shows right now

_.WW_WW_W_WWWW.L._WWWWWWW..WLW.._.W.W.W..W..WW....W...WWW....WWW
WW..W...WW..W.........W.WW.._W.....W....WWWW..W...........W.WW..
...W....W..W.W.WW_W.W...........................................
................................................................


And the whole site is just stuck on something. This seems to be caused by some rogue process perhaps that we cannot locate.

Yes, there may be 100 possible reasons for this, and we are trying to narrow down. RAM is not an issue I am sure, as I have 8GB ram on the box. The site is not that popular just yet so I am looking at no more than 10 simultaneous users. Thsi changes obviously if a SERP hits the site and initiates multiple requests. The daily traffic is no more than 1000 visits.

The 1000+ requests were reduced to 20 bby putting a huge amount of research and time into only retaining the needed requests. 99% of requests were not needed. The previous cowboy developer made it that way... loser. My new developer fixed all of this and toothcombed every single dynamic page.

[eluser]alitech[/eluser]
I suspected DDOS, but this is ongoing for over a year so I hardly think that is the case. Saving the session data in the SQL is an interesting point. I am not sure that is or is not happening as I am not technical but I can ask the dev to look into this.

Any other possibilities?

[eluser]tonanbarbarian[/eluser]
if you have phpmyadmin installed on the server try to go to the Processes section and see what MySQL queries are being run
If they are all the same then you may have a poorly designed query causing a bottleneck

also you could consider limiting what parts of the site the search engines index
it is possible that there is a recursive link somewhere on the site and the search engine is getting stuck
or if you have a calendar on the site the search engine is finding all of the links ofr years in the past and future and indexing the content

you have multiple categories of content that can have multiple pages in each
if the search engine is trying to index these all it is not surprising that it is making so many connections
you may have to put some meta tags or robots.txt files in place to limit how deep into the pages the search engines are allowed to crawl

if it really worth while to allow the search engines to index more than the first 2 or 3 pages in any catagory? and if you allow different sorting options on each page you might want to limit that as well

even just limiting one of these, i.e. allow the search engine to index all pages in all categories but do not allow it to change the sorting, would vastly reduce the load when a search engine comes to the site.

[eluser]smilie[/eluser]
Again, set more debugging level for MySQL and for Apache.
Also, set up (if it is not already done) mysql-slow-query-log on. This may also tell you what part of your site is causing slow down.

mytop is also excellent shell tool.

And ofcourse always great iptraf (apt-get install iptraf) which has excellent GUI for monitoring real time traffic.

Keep us posted.

Regards,
Smilie

[eluser]alitech[/eluser]
Hi all and thanks for your replies. It has given me a lot of insight into whats really happening. Upon investigation, I seem to have found an inconsistancy in disk space. I wanted to know whats taking up the most disk space on the host, low and behold, its SQL server replication logs according to my hosting provider.

Thanks for the lead on the session logging stuff. I now need to find out if this is a coding related issue or server related. How best can I deal with this? I just need to point my developer in the right direction here so he can fix the issue once and for all. Could you please let me know how to go about fixing this? What is SQL replication and how can I fix this?

please have a look at the response from my hosting provider when queried about whats using the most disk space files/logs/sql/backup etc.


Hello Ali,

I have checked your server disk space usage and I don't think the backup is taking that much disk space:

------
root@ignito [/backup/cpbackup]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vzfs 846G 58G 742G 8% /
/dev/simfs 846G 58G 742G 8% /tmp
/dev/simfs 846G 58G 742G 8% /var/tmp
none 1.0G 4.0K 1.0G 1% /dev
------

Out of the 58G in use, the backup directory at /backup/cpbackups is only taking around 6G.

-------
root@ignito [~]# du -sch /backup/cpbackup/
5.4G /backup/cpbackup/
5.4G total
-------

You are taking compressed backups of your accounts daily into the /backup/cpbackups directory on your server and you can list the backup files under daily, weekly and monthly directories inside the above backup location.

Your MySQL directory at /var/lib/mysql consumes 19G out of which 18G is used by MySQL replication logs.

-----
root@ignito [~]# du -sch /var/lib/mysql/
19G /var/lib/mysql/
19G total
-----

Are you really using MySQL replication on your server?



Thank you,

Andrew

[eluser]alitech[/eluser]
I'd also appreciate some insight into what replication logs are and why we need them.

If we delete these logs, does it cause any harm to the website? I am guessing that they are in their own DB tables.

How easy is it to disable logging, is it something you do to the coding of the site or at server level?

I am expecting a relief on the downtime when this is sorted out. Counting the minutes for getting this all sorted out. over a year sinice this problem all came to light.

[eluser]alitech[/eluser]
Would appreciate some replies here. Many thanks.