QUESTION Server uptime looks like a reboot every two weeks

[balefireeyed]

PIAF Installed Version = 2.0.6.4 under *HARDWARE*
x FreePBX Version = 2.10.1.16
x Running Asterisk Version = 1.8.28.0
x Asterisk Source Version = 1.8.28.0
Operating System = CentOS release 6.4 (Final)
x Kernel Version = 2.6.32-358.23.2.el6.i686 - 32 Bit

~ $ uptime
10:46:18 up 3 days, 8:20, 1 user, load average: 0.45, 0.37, 0.36
So, I've checked crontab -e as well as /etc/crontab, and there's nothing that would reboot the server, but from what I can tell it's happening at 3:00 am every other Friday. This has me completely stumped. I'm monitoring the server with Zabbix, and the only correlation that I can see from Zabbix is MySQL data. I've uploaded a screenshot of the mysql data. Has anyone else run into anything like this or point me in the right direction?

 

[synack]

Anything in /var/log/cron that might point you in the right direction?

 

[balefireeyed]

Looks like it is more random than I had thought. Not the uptime shows The only thing I see run before the last shutdown Monday night. The last thing I see run before the reboot is ipchecker (travelin man). I don't think that's related.

 

[hecatae]

does the cleaner turn up once a fortnight and unplug the machine?

 

[tbrummell]

does the cleaner turn up once a fortnight and unplug the machine?

Funny you mention this. I called out on an emergency "phone system down" call once, on a holiday weekend, going to a major outdoor concert venue where they had no phones. after battling my way to the concert site and getting past security, I found someone had unplugged the phone system to plug in a microwave. Man did they feel stupid. They gave me 2 weekend passes to the event.

 

[Cynjut]

I had a customer that had a problem like this - the system worked fine for about 10 days, then would just reboot. After a couple of months like that, it started rebooting every week, then every couple of days, then every day. I looked at it for a few months before I decided to start over with a complete software refresh - I reformatted the hard drives and, as I usually do with a naked computer, booted from my NetBSD memtest boot disk. It found a bad spot on the 4th DIMM, but it was intermittently bad - it passed the first and fourth passes, but failed the second and third. I replaced the DIMM and the system didn't reboot spontaneously again - it's been running continuously for over a year and a half.

 

[atsak]

Yep you've got a hardware problem, power problem, something like that. Could be a capacitor on the mobo that's on its way also, or a power supply going a bit under or over voltage from time to time.

 

[balefireeyed]

Cynjut atsak Spot on. The server crashed and hung at kernel dump error
kernel_thread_helper+0x7/0x10


I'm going to see if I have memory around to try, otherwise it's looking like it's the mobo?

 

[atsak]

Memory more likely for that error.