FOOD FOR THOUGHT Network Time Sync Not Working

[Mark Thompson]

Hello,

I've just noticed my server time was slow. I thought that IncrediblePBX synced time with NTP by default. NTP is not showing as installed Can I install and configure, or will this cause other problems with the IncrediblePBX bundle?

Incredible PBX 12.7.1 for Ubuntu 14

Asterisk: UP Apache: UP MySQL: UP
SendMail: UP IPtables: UP SSH: UP
LAN eth0: UP Fail2Ban: UP Webmin: UP

RAM:145764 "Ubuntu 14.04.1 LTS" Disk: 63G

Asterisk 12.8.0-rc2 FreePBX 12.0.46

 

[wardmundy]

Absolutely. And thanks for the heads up.

 

[Mark Thompson]

Thanks for IncrediblePBX

 

[wardmundy]

Thanks for IncrediblePBX

Mark, This should get your server squared away:

apt-get update
apt-get install ntpdate -y
ntpdate -s time.nist.gov
sed -i 's|exit 0|/usr/sbin/ntpdate -s time.nist.gov\nexit 0|' /etc/rc.local

Incredible PBX installer also has been patched.

 

[Mark Thompson]

I think I opened a can of worms here. It looks like my /boot partition is full. I'm not sure how to correct this.

root@incrediblepbx:~#
root@incrediblepbx:~# apt-get update

You might want to run 'apt-get -f install' to correct these:
The following packages have unmet dependencies:
linux-image-extra-3.13.0-55-generic : Depends: linux-image-3.13.0-55-generic but it is not going to be installed
linux-image-generic : Depends: linux-image-3.13.0-55-generic but it is not going to be installed
E: Unmet dependencies. Try 'apt-get -f install' with no packages (or specify a solution).

Unpacking linux-image-3.13.0-55-generic (3.13.0-55.94) ...
dpkg: error processing archive /var/cache/apt/archives/linux-image-3.13.0-55-generic_3.13.0-55.94_i386.deb (--unpack):
cannot copy extracted data for './boot/vmlinuz-3.13.0-55-generic' to '/boot/vmlinuz-3.13.0-55-generic.dpkg-new': failed to write (No space left on device)
No apport report written because the error message indicates a disk full error
dpkg-deb: error: subprocess paste was killed by signal (Broken pipe)
Examining /etc/kernel/postrm.d .
run-parts: executing /etc/kernel/postrm.d/initramfs-tools 3.13.0-55-generic /boot/vmlinuz-3.13.0-55-generic
run-parts: executing /etc/kernel/postrm.d/zz-update-grub 3.13.0-55-generic /boot/vmlinuz-3.13.0-55-generic

No apport report written because the error message indicates a disk full error
dpkg-deb: error: subprocess paste was killed by signal (Broken pipe)
Examining /etc/kernel/postrm.d .
run-parts: executing /etc/kernel/postrm.d/initramfs-tools 3.13.0-96-generic /boot/vmlinuz-3.13.0-96-generic
run-parts: executing /etc/kernel/postrm.d/zz-update-grub 3.13.0-96-generic /boot/vmlinuz-3.13.0-96-generic

Preparing to unpack .../linux-image-3.13.0-55-generic_3.13.0-55.94_i386.deb ...
Done.
Unpacking linux-image-3.13.0-55-generic (3.13.0-55.94) ...
dpkg: error processing archive /var/cache/apt/archives/linux-image-3.13.0-55-generic_3.13.0-55.94_i386.deb (--unpack):
cannot copy extracted data for './boot/vmlinuz-3.13.0-55-generic' to '/boot/vmlinuz-3.13.0-55-generic.dpkg-new': failed to write (No space left on device)
No apport report written because the error message indicates a disk full error
dpkg-deb: error: subprocess paste was killed by signal (Broken pipe)
Examining /etc/kernel/postrm.d .
run-parts: executing /etc/kernel/postrm.d/initramfs-tools 3.13.0-55-generic /boot/vmlinuz-3.13.0-55-generic
run-parts: executing /etc/kernel/postrm.d/zz-update-grub 3.13.0-55-generic /boot/vmlinuz-3.13.0-55-generic
^[
Errors were encountered while processing:
/var/cache/apt/archives/linux-image-3.13.0-96-generic_3.13.0-96.143_i386.deb
/var/cache/apt/archives/linux-image-3.13.0-55-generic_3.13.0-55.94_i386.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)
root@incrediblepbx:~#
root@incrediblepbx:~#
root@incrediblepbx:~# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/incrediblepbx--vg-root 75455608 6810040 64789596 10% /
none 4 0 4 0% /sys/fs/cgroup
udev 497564 4 497560 1% /dev
tmpfs 101524 632 100892 1% /run
none 5120 0 5120 0% /run/lock
none 507616 0 507616 0% /run/shm
none 102400 0 102400 0% /run/user
/dev/sda1 240972 238969 0 100% /boot
root@incrediblepbx:~#

 

[Mark Thompson]

Tried to purge old kernels. No luck.

root@incrediblepbx:~# uname -r
3.13.0-54-generic
root@incrediblepbx:~# dpkg -l linux-{image,headers}-"[0-9]*" | awk '/^ii/{ print $2}' | grep -v -e `uname -r | cut -f1,2 -d"-"` | grep -e '[0-9]'
linux-headers-3.13.0-44
linux-headers-3.13.0-44-generic
linux-headers-3.13.0-46
linux-headers-3.13.0-46-generic
linux-headers-3.13.0-48
linux-headers-3.13.0-48-generic
linux-headers-3.13.0-49
linux-headers-3.13.0-49-generic
linux-headers-3.13.0-51
linux-headers-3.13.0-51-generic
linux-headers-3.13.0-52
linux-headers-3.13.0-52-generic
linux-headers-3.13.0-53
linux-headers-3.13.0-53-generic
linux-image-3.13.0-44-generic
linux-image-3.13.0-46-generic
linux-image-3.13.0-48-generic
linux-image-3.13.0-49-generic
linux-image-3.13.0-51-generic
linux-image-3.13.0-52-generic
linux-image-3.13.0-53-generic
root@incrediblepbx:~#
root@incrediblepbx:~#
root@incrediblepbx:~# dpkg -l linux-{image,headers}-"[0-9]*" | awk '/^ii/{ print $2}' | grep -v -e `uname -r | cut -f1,2 -d"-"` | grep -e '[0-9]' | xargs sudo apt-get -y purge
sudo: unable to resolve host incrediblepbx
Reading package lists... Done
Building dependency tree
Reading state information... Done
You might want to run 'apt-get -f install' to correct these:
The following packages have unmet dependencies:
linux-image-extra-3.13.0-44-generic : Depends: linux-image-3.13.0-44-generic but it is not going to be installed
linux-image-extra-3.13.0-46-generic : Depends: linux-image-3.13.0-46-generic but it is not going to be installed
linux-image-extra-3.13.0-48-generic : Depends: linux-image-3.13.0-48-generic but it is not going to be installed
linux-image-extra-3.13.0-49-generic : Depends: linux-image-3.13.0-49-generic but it is not going to be installed
linux-image-extra-3.13.0-51-generic : Depends: linux-image-3.13.0-51-generic but it is not going to be installed
linux-image-extra-3.13.0-52-generic : Depends: linux-image-3.13.0-52-generic but it is not going to be installed
linux-image-extra-3.13.0-53-generic : Depends: linux-image-3.13.0-53-generic but it is not going to be installed
linux-image-extra-3.13.0-55-generic : Depends: linux-image-3.13.0-55-generic but it is not going to be installed
linux-image-extra-3.13.0-96-generic : Depends: linux-image-3.13.0-96-generic but it is not going to be installed
linux-image-generic : Depends: linux-image-3.13.0-96-generic but it is not going to be installed
E: Unmet dependencies. Try 'apt-get -f install' with no packages (or specify a solution).
root@incrediblepbx:~# apt-get -f install
Reading package lists... Done
Building dependency tree
Reading state information... Done
Correcting dependencies... Done
The following packages were automatically installed and are no longer required:
linux-headers-3.13.0-46 linux-headers-3.13.0-46-generic
linux-headers-3.13.0-48 linux-headers-3.13.0-48-generic
linux-headers-3.13.0-49 linux-headers-3.13.0-49-generic
linux-headers-3.13.0-51 linux-headers-3.13.0-51-generic
linux-headers-3.13.0-52 linux-headers-3.13.0-52-generic
linux-headers-3.13.0-53 linux-headers-3.13.0-53-generic
linux-headers-3.13.0-55 linux-headers-3.13.0-55-generic
linux-image-3.13.0-46-generic linux-image-3.13.0-48-generic
linux-image-3.13.0-49-generic linux-image-3.13.0-51-generic
linux-image-3.13.0-52-generic linux-image-3.13.0-53-generic
linux-image-3.13.0-55-generic linux-image-extra-3.13.0-46-generic
linux-image-extra-3.13.0-48-generic linux-image-extra-3.13.0-49-generic
linux-image-extra-3.13.0-51-generic linux-image-extra-3.13.0-52-generic
linux-image-extra-3.13.0-53-generic linux-image-extra-3.13.0-55-generic
Use 'apt-get autoremove' to remove them.
The following extra packages will be installed:
linux-image-3.13.0-55-generic linux-image-3.13.0-96-generic
Suggested packages:
fdutils linux-doc-3.13.0 linux-source-3.13.0 linux-tools
The following NEW packages will be installed:
linux-image-3.13.0-55-generic linux-image-3.13.0-96-generic
0 upgraded, 2 newly installed, 0 to remove and 284 not upgraded.
13 not fully installed or removed.
Need to get 0 B/29.5 MB of archives.
After this operation, 66.0 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
(Reading database ... 370669 files and directories currently installed.)
Preparing to unpack .../linux-image-3.13.0-96-generic_3.13.0-96.143_i386.deb ...
Done.
Unpacking linux-image-3.13.0-96-generic (3.13.0-96.143) ...
dpkg: error processing archive /var/cache/apt/archives/linux-image-3.13.0-96-generic_3.13.0-96.143_i386.deb (--unpack):
cannot copy extracted data for './boot/System.map-3.13.0-96-generic' to '/boot/System.map-3.13.0-96-generic.dpkg-new': failed to write (No space left on device)
No apport report written because the error message indicates a disk full error
dpkg-deb: error: subprocess paste was killed by signal (Broken pipe)
Examining /etc/kernel/postrm.d .
run-parts: executing /etc/kernel/postrm.d/initramfs-tools 3.13.0-96-generic /boot/vmlinuz-3.13.0-96-generic
run-parts: executing /etc/kernel/postrm.d/zz-update-grub 3.13.0-96-generic /boot/vmlinuz-3.13.0-96-generic

 

[Mark Thompson]

The best I can do I think is to manually delete older kernels to create the 66MB needed to run 'apt-get -f install'. I'm hoping this doesn't my still operational system.

 

[wardmundy]

If your drive is full, you're not going to have an "operational system" for very long. Make a backup... SOON!

 

[Mark Thompson]

The drive looks okay. I'm using 11G on an 80G drive. The /boot partition is chuck full though.

root@incrediblepbx:~# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/incrediblepbx--vg-root 75455608 6812296 64787340 10% /
none 4 0 4 0% /sys/fs/cgroup
udev 497564 4 497560 1% /dev
tmpfs 101524 628 100896 1% /run
none 5120 0 5120 0% /run/lock
none 507616 0 507616 0% /run/shm
none 102400 0 102400 0% /run/user
/dev/sda1 240972 238969 0 100% /boot

I'm hoping manually deleting old kernel files won't cause additional problems.

 

[chris_c_]

@Mark Thompson
Do this command:

root@incrediblepbx:~# apt-get autoremove

 

[Mark Thompson]

the OS suggested that yesterday. Just tried again, but no luck ...

Need to get 0 B/14.8 MB of archives.
After this operation, 1,492 MB disk space will be freed.
Do you want to continue? [Y/n] y
(Reading database ... 370669 files and directories currently installed.)
Removing linux-headers-3.13.0-55-generic (3.13.0-55.92) ...
Removing linux-headers-3.13.0-55 (3.13.0-55.92) ...
Removing linux-image-extra-3.13.0-55-generic (3.13.0-55.92) ...
depmod: FATAL: could not load /boot/System.map-3.13.0-55-generic: No such file or directory
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 3.13.0-55-generic /boot/vmlinuz-3.13.0-55-generic
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 3.13.0-55-generic /boot/vmlinuz-3.13.0-55-generic
update-initramfs: Generating /boot/initrd.img-3.13.0-55-generic
grep: /boot/config-3.13.0-55-generic: No such file or directory
WARNING: missing /lib/modules/3.13.0-55-generic
Device driver support needs thus be built-in linux image!
depmod: ERROR: could not open directory /lib/modules/3.13.0-55-generic: No such file or directory
depmod: FATAL: could not search modules: No such file or directory
depmod: WARNING: could not open /tmp/mkinitramfs_8Y1kJ4/lib/modules/3.13.0-55-generic/modules.order: No such file or directory
depmod: WARNING: could not open /tmp/mkinitramfs_8Y1kJ4/lib/modules/3.13.0-55-generic/modules.builtin: No such file or directory

gzip: stdout: No space left on device
E: mkinitramfs failure cpio 141 gzip 1
update-initramfs: failed for /boot/initrd.img-3.13.0-55-generic with 1.
run-parts: /etc/kernel/postinst.d/initramfs-tools exited with return code 1
dpkg: error processing package linux-image-extra-3.13.0-55-generic (--remove):
subprocess installed post-removal script returned error exit status 1
Errors were encountered while processing:
linux-image-extra-3.13.0-55-generic
E: Sub-process /usr/bin/dpkg returned an error code (1)


I think I need to do this ...

ou might want to run 'apt-get -f install' to correct these.
The following packages have unmet dependencies:
linux-image-extra-3.13.0-55-generic : Depends: linux-image-3.13.0-55-generic but it is not installed
linux-image-extra-3.13.0-96-generic : Depends: linux-image-3.13.0-96-generic but it is not installed
linux-image-generic : Depends: linux-image-3.13.0-96-generic but it is not installed

... but need the space in /boot to do it.

 

[Mark Thompson]

I manually deleted the 3 oldest kernel files from //boot and 'apt-get -f install' which as able to run this time. /boot size is down to 57%. 2 additional kernels were installed ...

The following extra packages will be installed:
linux-image-3.13.0-55-generic linux-image-3.13.0-96-generic

This mysql error keeps coming up ...

Setting up mysql-server-5.5 (5.5.43-0ubuntu0.14.04.1) ...

start: Job is already running: mysql
invoke-rc.d: initscript mysql, action "start" failed.
dpkg: error processing package mysql-server-5.5 (--configure):
subprocess installed post-installation script returned error exit status 1
dpkg: dependency problems prevent configuration of mysql-server:
mysql-server depends on mysql-server-5.5; however:
Package mysql-server-5.5 is not configured yet.

dpkg: error processing package mysql-server (--configure):
dependency problems - leaving unconfigured
Errors were encountered while processing:
mysql-server-5.5
mysql-server

Any ideas how to fix this?

When logging in IncediblePBX status screen shows all systems UP, including MySQL

Tx

 

[Mark Thompson]

Should I uninstall and reinstall mysql?

This link suggests 'apt-get dist-upgrade' will this break my IncredblePBX install? http://tutewall.com/package-mysql-server-5-5-is-not-configured-yet/

Also, after trying to install ntpdate per Ward's instructions in post #4 above I get ...

root@incrediblepbx:~# ntpdate
21 Sep 04:42:24 ntpdate[13683]: no servers can be used, exiting

 

[wardmundy]

@Mark Thompson This has morphed into all sorts of other problems. Major upgrades (dist-upgrade) have a way of breaking major things. If MySQL is failing or has failed, you probably should consider building a new server. Reinstalling probably will damage your existing setup. As for ntpdate, the correct command is...

ntpdate -s time.nist.gov

 

[chris_c_]

Should I uninstall and reinstall mysql?

Remove and reinstall:

service mysql stop
apt-get remove mysql-server mysql-client mysql-common
apt-get autoremove
apt-get autoclean
apt-get update
apt-get install mysql-server

 

[wardmundy]

If mysql-server is damaged, your server has serious hardware and/or software problems. Do what you like but repairing and then depending upon a system experiencing these sorts of issues is just asking for future headaches.

 

[chris_c_]

@wardmundy The /boot partition got full during apt-get update of the kernel and mysql etc. A reinstall of mysql and clearing out old kernels at 40MB+ each in /boot probably fixes things just fine IMHO. Also, resize /boot bigger, 500MB is good.

 

[chris_c_]

@Mark Thompson ntpd, which syncs your clock periodically and runs as a service, is more suitable to run on your server, than ntpdate which runs only once and exits. The default settings in /etc/ntp.conf which come with ntpd are fine.

 

[Mark Thompson]

Thanks for the replies. I'm taking a 'belt and suspenders' approach here. I have an 8 month old Clonezilla backup of the drive; there have been no PBX configuration changes since then. I've installed IncrediblePBX 13-12 Ubuntu 14 in Vbox on my laptop for testing.

NOTE:
/etc/ntp.conf does not exist in either my production Incredible PBX 12.7.1-12.0.46 Ubuntu 14 or in my Vbox Incredible PBX 13-12.

~# ntpd
The program 'ntpd' can be found in the following packages:
* ntp
* openntpd
Try: apt-get install <selected package>​

Can I install one of these packages without breaking IncrediblePBX?


Here is my plan when I get onsite:
1) Reboot

If system comes up okay, configure ntpd time sync
if system comes up with MySQl errors, step 2
If system comes up non-operatonal , step 3​

2) Reinstall MySQL

If system comes up okay, configure ntpd time sync
if system comes up with MySQl errors, step 3
If system comes up non-operatonal , step 3​

3) Install Clonzilla PBX12 backup

If system comes up okay, remove old kernels from /boot
Configure ntp time sync
if system comes up with MySQl errors, step 4
If system comes up non-operatonal , step 4​

4) Install preconfigured InPBX 13-12 image from Vbox (experimental)

If system comes up okay, configure ntpd time sync
If system comes up non-operatonal , step 5​

5) Install InPBX 13-12 from scratch

If system comes up okay, configure ntpd time sync​

Thanks,

mark

 

[chris_c_]

@Mark Thompson
Sure you can install it with

~# apt-get install ntp