Title: lendacademy web server problems
Post by: Fred93 on October 09, 2016, 11:00:00 PM
Post by: Fred93 on October 09, 2016, 11:00:00 PM
Frequently over the past week, I get a 404 not found error message when attempting to view a page on the lendacademy forum. Clicking refresh fetches the page (usually), but sometimes I have to wait a few minutes, after which a refresh will work.
Here's one of the messages...
<html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
Here's one of the messages...
<html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
Title: lendacademy web server problems
Post by: Rob L on October 09, 2016, 11:00:00 PM
Post by: Rob L on October 09, 2016, 11:00:00 PM
Ditto
Title: lendacademy web server problems
Post by: Lovinglifestyle on October 09, 2016, 11:00:00 PM
Post by: Lovinglifestyle on October 09, 2016, 11:00:00 PM
Same here.
Title: lendacademy web server problems
Post by: TravelingPennies on October 16, 2016, 11:00:00 PM
Post by: TravelingPennies on October 16, 2016, 11:00:00 PM
Getting worse I think. The outages each last about 1 minute. I see a few every day, but I'm not always using the web site, so I suspect there are dozens of them every day.
Title: lendacademy web server problems
Post by: Zach on October 16, 2016, 11:00:00 PM
Post by: Zach on October 16, 2016, 11:00:00 PM
Thanks everyone for pointing this out. I will look into this further.
Title: lendacademy web server problems
Post by: TravelingPennies on October 16, 2016, 11:00:00 PM
Post by: TravelingPennies on October 16, 2016, 11:00:00 PM
Yeah, there is something going on. Maybe a DoS attack 
But seriously, there are problems with the server I've not seen before.
Becoming a real PITA.
But seriously, there are problems with the server I've not seen before.
Becoming a real PITA.
Title: lendacademy web server problems
Post by: TravelingPennies on October 17, 2016, 11:00:00 PM
Post by: TravelingPennies on October 17, 2016, 11:00:00 PM
Some additional observations...
1. During one of these outages, I can ping the lendacademy web server.
2. I have only noticed the outages while reading the forum, but during one of the outages I tried to access the main lendacademy web page. It also timed out. In other words, during the outage, all web pages are down, not just the forum.
3. The symptom is always a long delay (must be a timeout of some kind) followed by a 404 message from your server. Because we get the 404 message, the browser is not timing out. Something internal to the web server is timing out.
4. The 404 message we get is not your customized 404 message, but appears to be the default nginx 404 message. I tried intentionally accessing a nonexistant page, something like www.lendacademy/forum/nonexistentpage.htm and that fails with a nice 404 message containing a lendacademy logo etc.
My guess is that your web server becomes unable to access the files that contain the web site content. Perhaps a file system is going offline, or communication with a back-end server is being lost.
1. During one of these outages, I can ping the lendacademy web server.
2. I have only noticed the outages while reading the forum, but during one of the outages I tried to access the main lendacademy web page. It also timed out. In other words, during the outage, all web pages are down, not just the forum.
3. The symptom is always a long delay (must be a timeout of some kind) followed by a 404 message from your server. Because we get the 404 message, the browser is not timing out. Something internal to the web server is timing out.
4. The 404 message we get is not your customized 404 message, but appears to be the default nginx 404 message. I tried intentionally accessing a nonexistant page, something like www.lendacademy/forum/nonexistentpage.htm and that fails with a nice 404 message containing a lendacademy logo etc.
My guess is that your web server becomes unable to access the files that contain the web site content. Perhaps a file system is going offline, or communication with a back-end server is being lost.
Title: lendacademy web server problems
Post by: RyanL on October 17, 2016, 11:00:00 PM
Post by: RyanL on October 17, 2016, 11:00:00 PM
Hi all, we've made some changes on the hosting side. If you continue to notice issues please let us know. Thanks!
Title: lendacademy web server problems
Post by: rawraw on October 18, 2016, 11:00:00 PM
Post by: rawraw on October 18, 2016, 11:00:00 PM
It just happened to me.
Title: lendacademy web server problems
Post by: TravelingPennies on October 18, 2016, 11:00:00 PM
Post by: TravelingPennies on October 18, 2016, 11:00:00 PM
Ditto, 2 min. or so ago. I copied my post before I lost it and started the process over. Fine now.
Title: lendacademy web server problems
Post by: TravelingPennies on October 18, 2016, 11:00:00 PM
Post by: TravelingPennies on October 18, 2016, 11:00:00 PM
Just happened to me again (twice in a row); the 404, nginx problem.
Title: lendacademy web server problems
Post by: TravelingPennies on October 18, 2016, 11:00:00 PM
Post by: TravelingPennies on October 18, 2016, 11:00:00 PM
Thanks for the heads up. We are continuing to work on this.
Title: lendacademy web server problems
Post by: TravelingPennies on October 19, 2016, 11:00:00 PM
Post by: TravelingPennies on October 19, 2016, 11:00:00 PM
I observed one of these outages Thur morning 10/20 12:38AM PST (may have started earlier) to about 12:41:30AM.
This is the first time I've written down the times. I watched it for over 3.5 minutes!
This is the first time I've written down the times. I watched it for over 3.5 minutes!
Title: lendacademy web server problems
Post by: TravelingPennies on October 19, 2016, 11:00:00 PM
Post by: TravelingPennies on October 19, 2016, 11:00:00 PM
Title: lendacademy web server problems
Post by: TravelingPennies on October 19, 2016, 11:00:00 PM
Post by: TravelingPennies on October 19, 2016, 11:00:00 PM
Title: lendacademy web server problems
Post by: Fred93 on October 19, 2016, 11:00:00 PM
Post by: Fred93 on October 19, 2016, 11:00:00 PM
Tonite 6/20 6:52PM to 6:54PM forum outage.
I checked main page www.lendacademy.com in the middle of the outage, and it was ok.
I checked main page www.lendacademy.com in the middle of the outage, and it was ok.
Title: lendacademy web server problems
Post by: Peter on October 20, 2016, 11:00:00 PM
Post by: Peter on October 20, 2016, 11:00:00 PM
We are trying to get to the bottom of this problem. The challenge is that it is intermittent. We keep trying things and will continue to do so. I am also getting these 404 messages and finding the site much slower than normal.
Title: lendacademy web server problems
Post by: TravelingPennies on October 21, 2016, 11:00:00 PM
Post by: TravelingPennies on October 21, 2016, 11:00:00 PM
Sat 10/22 first noticed at 7:10PM PST. Resolved 7:11PM PST. Both forum and lendacademy main web page down, and both produced the nginx 404 message.
Title: lendacademy web server problems
Post by: Rob L on October 22, 2016, 11:00:00 PM
Post by: Rob L on October 22, 2016, 11:00:00 PM
Sunday AM 10/23/16 and getting multiple occurrences. My initial attempt to reach the LA main page from another web site failed. Also once for a "Preview" post request and another for a "Reply".
Using HTTP Fox I captured details of the GET that failed the initial attempt to reach the LA main page (from fivethirtyeight.com).
RyanL, If you think it might be useful then PM me your email address and I'll reply with a text file attachment of what I captured.
Using HTTP Fox I captured details of the GET that failed the initial attempt to reach the LA main page (from fivethirtyeight.com).
RyanL, If you think it might be useful then PM me your email address and I'll reply with a text file attachment of what I captured.
Title: lendacademy web server problems
Post by: RyanL on October 22, 2016, 11:00:00 PM
Post by: RyanL on October 22, 2016, 11:00:00 PM
Title: lendacademy web server problems
Post by: TravelingPennies on October 22, 2016, 11:00:00 PM
Post by: TravelingPennies on October 22, 2016, 11:00:00 PM
Ryan, check your inbox (or trash) and you should find my email.
Title: lendacademy web server problems
Post by: TravelingPennies on October 23, 2016, 11:00:00 PM
Post by: TravelingPennies on October 23, 2016, 11:00:00 PM
Outage this morning 10/24/16. First observed 10:15 AM PST. Ended 11:21 AM PST.
This was a long one!
With known times of some outages, it seems like you should be able to find log entries of some events associated with the outages.
This was a long one!
With known times of some outages, it seems like you should be able to find log entries of some events associated with the outages.
Title: lendacademy web server problems
Post by: TravelingPennies on October 23, 2016, 11:00:00 PM
Post by: TravelingPennies on October 23, 2016, 11:00:00 PM
Another outage noticed at today at 1:06PM PDT. Resolved at 1:08PM PDT.
Title: lendacademy web server problems
Post by: dr.everett on October 23, 2016, 11:00:00 PM
Post by: dr.everett on October 23, 2016, 11:00:00 PM
2:01PM PDT 10/24 trying to access http://www.lendacademy.com/forum/index.php
Got a 404 and then about 30 seconds later the site finally loaded.
Got a 404 and then about 30 seconds later the site finally loaded.
Title: lendacademy web server problems
Post by: TravelingPennies on October 24, 2016, 11:00:00 PM
Post by: TravelingPennies on October 24, 2016, 11:00:00 PM
I've seen about 5 of these outages today alone. If I'm seeing this many, and I'm not even at my computer for hours at a time, there must be dozens of these events every day.
Title: lendacademy web server problems
Post by: TravelingPennies on October 25, 2016, 11:00:00 PM
Post by: TravelingPennies on October 25, 2016, 11:00:00 PM
Title: lendacademy web server problems
Post by: TravelingPennies on October 25, 2016, 11:00:00 PM
Post by: TravelingPennies on October 25, 2016, 11:00:00 PM
Title: lendacademy web server problems
Post by: TravelingPennies on October 25, 2016, 11:00:00 PM
Post by: TravelingPennies on October 25, 2016, 11:00:00 PM
Hate to state the obvious, but maybe helpful? Probably the area where you're tweaking now:
WordPress creates a rewrite array that is responsible for managing permalinks, or "pretty URLs."
Improperly coded plugins, custom post types, and taxonomy alterations can break or corrupt the permalink array, thus causing 404s.
http://websynthesis.com/fixing-wordpress-404-errors/
BTW: I got a 404 error just now.
WordPress creates a rewrite array that is responsible for managing permalinks, or "pretty URLs."
Improperly coded plugins, custom post types, and taxonomy alterations can break or corrupt the permalink array, thus causing 404s.
http://websynthesis.com/fixing-wordpress-404-errors/
BTW: I got a 404 error just now.
Title: lendacademy web server problems
Post by: TravelingPennies on October 26, 2016, 11:00:00 PM
Post by: TravelingPennies on October 26, 2016, 11:00:00 PM
It seems like we have successfully cleared the errors that we were seeing. I personally haven't had any issues in the last day, but if you notice another outage please note the time so I can continue to dig into the error logs.
Title: lendacademy web server problems
Post by: TravelingPennies on October 26, 2016, 11:00:00 PM
Post by: TravelingPennies on October 26, 2016, 11:00:00 PM
Sorry to be the bearer of bad news, but just got another 404.
Between 9:57 and 9:58 AM ET.
Between 9:57 and 9:58 AM ET.
Title: lendacademy web server problems
Post by: Fred93 on October 26, 2016, 11:00:00 PM
Post by: Fred93 on October 26, 2016, 11:00:00 PM
I've seen no problems in last day or so.
Title: lendacademy web server problems
Post by: Lovinglifestyle on October 26, 2016, 11:00:00 PM
Post by: Lovinglifestyle on October 26, 2016, 11:00:00 PM
5:54 pm to about 6:00 pm, mountain time (Utah). 404 NOT FOUND, nginx.
Title: lendacademy web server problems
Post by: Rob L on October 26, 2016, 11:00:00 PM
Post by: Rob L on October 26, 2016, 11:00:00 PM
Just got another a few minutes ago. Seems they occur more frequently when I'm using my notebook on my wireless LAN rather than when using a tower hardwired to my router. I was on and off today no problems when using a tower, but had problems this AM and just now this PM while using my notebook. Go figure ...
Title: lendacademy web server problems
Post by: TravelingPennies on October 27, 2016, 11:00:00 PM
Post by: TravelingPennies on October 27, 2016, 11:00:00 PM
I just had another nginx 404 not found at 9:36PM tonite.
Title: lendacademy web server problems
Post by: TravelingPennies on October 27, 2016, 11:00:00 PM
Post by: TravelingPennies on October 27, 2016, 11:00:00 PM
Another (on my notebook) about 5 minutes ago.
Title: lendacademy web server problems
Post by: Peter on October 27, 2016, 11:00:00 PM
Post by: Peter on October 27, 2016, 11:00:00 PM
I'm only affected by this whenever I've written some long, drawn-out screed and try to post it - only to be met with having it not submit.
Is the current plan "do nothing, blame Russia"? This sucks. Please get someone competent to investigate it thoroughly and indict her...
Is the current plan "do nothing, blame Russia"? This sucks. Please get someone competent to investigate it thoroughly and indict her...
Title: lendacademy web server problems
Post by: rawraw on October 27, 2016, 11:00:00 PM
Post by: rawraw on October 27, 2016, 11:00:00 PM
Happening intermittently to me as well still.
Title: lendacademy web server problems
Post by: TravelingPennies on October 28, 2016, 11:00:00 PM
Post by: TravelingPennies on October 28, 2016, 11:00:00 PM
Title: lendacademy web server problems
Post by: TravelingPennies on October 28, 2016, 11:00:00 PM
Post by: TravelingPennies on October 28, 2016, 11:00:00 PM
Starting to become optimistic ... it's been a while
Title: lendacademy web server problems
Post by: TravelingPennies on October 29, 2016, 11:00:00 PM
Post by: TravelingPennies on October 29, 2016, 11:00:00 PM
Three more this morning (Sun 10/30/16) around 10:30 am.
On my notebook again! Haven't seen any on my tower in quite a number of days, but will post if I do.
Have been running Linux/Firefox on notebook, but it's a dual boot. Gonna switch to Win 7 & IE and see if errs on notebook continue.
Just grasping for straws and trying to be helpful.
On my notebook again! Haven't seen any on my tower in quite a number of days, but will post if I do.
Have been running Linux/Firefox on notebook, but it's a dual boot. Gonna switch to Win 7 & IE and see if errs on notebook continue.
Just grasping for straws and trying to be helpful.
Title: lendacademy web server problems
Post by: RyanL on October 30, 2016, 11:00:00 PM
Post by: RyanL on October 30, 2016, 11:00:00 PM
Hi all, just an update from our end. Our hosting provider isn't being very helpful resolving this issue so we are going to move to another hosting provider that is optimized for our current forum software. We'll be working on this over the next week and will post any updates as well as any planned downtime here. Thanks for your patience.
Title: lendacademy web server problems
Post by: TravelingPennies on October 30, 2016, 11:00:00 PM
Post by: TravelingPennies on October 30, 2016, 11:00:00 PM
Title: lendacademy web server problems
Post by: Fred on October 31, 2016, 11:00:00 PM
Post by: Fred on October 31, 2016, 11:00:00 PM
Title: lendacademy web server problems
Post by: TravelingPennies on October 31, 2016, 11:00:00 PM
Post by: TravelingPennies on October 31, 2016, 11:00:00 PM
Title: lendacademy web server problems
Post by: jheizer on October 31, 2016, 11:00:00 PM
Post by: jheizer on October 31, 2016, 11:00:00 PM
lol it was just down again. This is just funny. Expanding on Fred is it a VPS or just shared hosting?
Title: lendacademy web server problems
Post by: anabio on October 31, 2016, 11:00:00 PM
Post by: anabio on October 31, 2016, 11:00:00 PM
Title: lendacademy web server problems
Post by: Fred93 on November 03, 2016, 11:00:00 PM
Post by: Fred93 on November 03, 2016, 11:00:00 PM
These events are not random. They are HOURLY, plus or minus a few minutes. In a few hours, after I've captured a few more events, I will post a chart showing the regularity.
Is there something that runs once per hour?
Is there something that runs once per hour?
Title: lendacademy web server problems
Post by: Peter on November 03, 2016, 11:00:00 PM
Post by: Peter on November 03, 2016, 11:00:00 PM
Title: lendacademy web server problems
Post by: TravelingPennies on November 03, 2016, 11:00:00 PM
Post by: TravelingPennies on November 03, 2016, 11:00:00 PM
I wrote a program to periodically fetch the lendacademy main forum page, once per minute, and record how long this took, and whether an error was returned by the server. The following chart shows the time-to-fetch the page. The points at 60 seconds, and some around 30 seconds are the ones that report the 404 errors. You can see that the events are not random, but are roughly periodic, with a period slightly under one hour. If these events were random, you would sometimes see a pair close together, but that never happens.
Title: lendacademy web server problems
Post by: TravelingPennies on November 03, 2016, 11:00:00 PM
Post by: TravelingPennies on November 03, 2016, 11:00:00 PM
Woo hoo. Has now run for a few more hours, and the periodic nature of the problem could not be more clear. I've attached an updated chart where you see 10 evenly spaced events, ie 9 periods complete in 8 hours. Although each event has an uncertainty of about 5 minutes, the cycle is 55 minutes long.
Very few malfunctions would cause something like this.
If you have a cron job or other periodic thing doing something at near this interval, look there.
If you have a back end database server, I would say it is likely restarting every 55 minutes, hence is unavailable to the web server for a couple of minutes. What could make a back end server restart periodically? Some software with a memory leak could do it. After about 55 minutes of operation, memory is used up, and the database server restarts.
Can you examine logs to see if the database server, or some similar component, is restarting? Can you monitor memory on the backend server to see if free memory is steadily decreasing during the 55 minute interval between events?
I think it is unlikely that this is a hardware problem. I can imagine a hardware problem that occurs after X minutes of operation, such as thermal problem for example, but it is difficult to imagine that it gets fixed by a quick reset of something. Once something is too hot would need time to cool.
Very few malfunctions would cause something like this.
If you have a cron job or other periodic thing doing something at near this interval, look there.
If you have a back end database server, I would say it is likely restarting every 55 minutes, hence is unavailable to the web server for a couple of minutes. What could make a back end server restart periodically? Some software with a memory leak could do it. After about 55 minutes of operation, memory is used up, and the database server restarts.
Can you examine logs to see if the database server, or some similar component, is restarting? Can you monitor memory on the backend server to see if free memory is steadily decreasing during the 55 minute interval between events?
I think it is unlikely that this is a hardware problem. I can imagine a hardware problem that occurs after X minutes of operation, such as thermal problem for example, but it is difficult to imagine that it gets fixed by a quick reset of something. Once something is too hot would need time to cool.
Title: lendacademy web server problems
Post by: TravelingPennies on November 03, 2016, 11:00:00 PM
Post by: TravelingPennies on November 03, 2016, 11:00:00 PM
Ok, Fred... it's been 10 years... would you please, for the love of all things Holy, go take some kind of minister-without-portfolio consulting gig at one of the platforms where you're tasked with just walking around the joint and finding inoptimal/broken shit - and then fixing it?
I'd so invest.
I'd so invest.
Title: lendacademy web server problems
Post by: RyanL on November 03, 2016, 11:00:00 PM
Post by: RyanL on November 03, 2016, 11:00:00 PM
Fred, appreciate your analysis - unfortunately I only have access to one set of logs in which there is a consistent apc_store error. The host has said they've looked and have found no issues with APC on the server. They keep coming back to: "Most, if not all of the current processes for the day are queried URLs and as we explained, this bypasses caching." So they are blaming caching on the issue (keep in mind we have made no changes and visits to the site have not substantially increased). At the end of the day, they are not optimized for SMF forums and they don't provide adequate support either.
11/3/2016 11:20:35 AM - www: PHP Warning: apc_store(): GC cache entry 'f5a7014e25789cff0106c3999bf7c344-SMF-modSettingssmf' was on gc-list for 3602 seconds in /var/www/lendacademy.com/forum/Sources/Load.php on line 2687
11/3/2016 11:20:35 AM - www: PHP Warning: apc_store(): GC cache entry 'f5a7014e25789cff0106c3999bf7c344-SMF-modSettingssmf' was on gc-list for 3602 seconds in /var/www/lendacademy.com/forum/Sources/Load.php on line 2687
Title: lendacademy web server problems
Post by: TravelingPennies on November 03, 2016, 11:00:00 PM
Post by: TravelingPennies on November 03, 2016, 11:00:00 PM
Title: lendacademy web server problems
Post by: wiseclerk_com on November 04, 2016, 11:00:00 PM
Post by: wiseclerk_com on November 04, 2016, 11:00:00 PM
extremly slow response in the past 5 minutes
Title: lendacademy web server problems
Post by: TravelingPennies on November 09, 2016, 11:00:00 PM
Post by: TravelingPennies on November 09, 2016, 11:00:00 PM
Hi all, please note that the forum will be moving to our new host on Friday November 11th at 8 pm PST. At this time the forum will be unavailable for *hopefully* just 7 hours.
-Ryan
-Ryan
Title: lendacademy web server problems
Post by: Rob L on November 09, 2016, 11:00:00 PM
Post by: Rob L on November 09, 2016, 11:00:00 PM
Good move! We'll look forward to business as usual thereafter. It has bee tedious.
Title: lendacademy web server problems
Post by: TravelingPennies on November 09, 2016, 11:00:00 PM
Post by: TravelingPennies on November 09, 2016, 11:00:00 PM
Title: lendacademy web server problems
Post by: TravelingPennies on November 10, 2016, 11:00:00 PM
Post by: TravelingPennies on November 10, 2016, 11:00:00 PM
Title: lendacademy web server problems
Post by: TravelingPennies on November 10, 2016, 11:00:00 PM
Post by: TravelingPennies on November 10, 2016, 11:00:00 PM
Title: lendacademy web server problems
Post by: rawraw on November 10, 2016, 11:00:00 PM
Post by: rawraw on November 10, 2016, 11:00:00 PM
Fred93 cracks me up. This thread was awesome
Title: lendacademy web server problems
Post by: .Ryan. on November 10, 2016, 11:00:00 PM
Post by: .Ryan. on November 10, 2016, 11:00:00 PM
The ttl should actually act as the expiration timer by resolvers (when they are configured to honor it), and that appears to be set to 10 minutes. The expiry field is used by the slave servers.
Moot point, I guess since the IP isn't changing.
Moot point, I guess since the IP isn't changing.
Title: lendacademy web server problems
Post by: Fred93 on November 10, 2016, 11:00:00 PM
Post by: Fred93 on November 10, 2016, 11:00:00 PM
Welcome back! Am I the first non-employee to login?
Title: lendacademy web server problems
Post by: RyanL on November 10, 2016, 11:00:00 PM
Post by: RyanL on November 10, 2016, 11:00:00 PM
Title: lendacademy web server problems
Post by: Debt Free on November 10, 2016, 11:00:00 PM
Post by: Debt Free on November 10, 2016, 11:00:00 PM
Caught me by surprise, but then noticed the redirect on the prior login page.
Title: lendacademy web server problems
Post by: TravelingPennies on November 11, 2016, 11:00:00 PM
Post by: TravelingPennies on November 11, 2016, 11:00:00 PM
Thanks for your patience as we worked through this issue. The URL redirects from /forum to the new subdomain are now working. If you notice any issues please let us know.
Title: lendacademy web server problems
Post by: rawraw on November 11, 2016, 11:00:00 PM
Post by: rawraw on November 11, 2016, 11:00:00 PM
Tapatalk isn't working for me just FYI
Title: lendacademy web server problems
Post by: Peter on November 12, 2016, 11:00:00 PM
Post by: Peter on November 12, 2016, 11:00:00 PM
Haven't run into any errors - thx for the fix.
Title: lendacademy web server problems
Post by: TravelingPennies on November 13, 2016, 11:00:00 PM
Post by: TravelingPennies on November 13, 2016, 11:00:00 PM
Title: lendacademy web server problems
Post by: TravelingPennies on November 13, 2016, 11:00:00 PM
Post by: TravelingPennies on November 13, 2016, 11:00:00 PM
Title: lendacademy web server problems
Post by: TravelingPennies on November 13, 2016, 11:00:00 PM
Post by: TravelingPennies on November 13, 2016, 11:00:00 PM
It is now working
Sent from my SAMSUNG-SM-G935A using Tapatalk
Sent from my SAMSUNG-SM-G935A using Tapatalk