FOOD FOR THOUGHT Locally backing up VPS

wardmundy

Nerd Uno
Joined
Oct 12, 2007
Messages
19,170
Reaction score
5,199
And now you have a workable answer, use Mondoarchive :)

Speaking of a voice from the past, Mondo was one of our early favorites... over 8 years ago. The problem then was getting a successful restore. The only thing worse than no backup is a backup that won't restore. We finally gave up. So... hopefully, they've overcome that hurdle.

For everyone's benefit, perhaps someone could walk us through the settings that actually work to backup a running VPS on an OpenVZ container. I don't think it's possible so please prove me wrong... again.
 
Last edited:

wardmundy

Nerd Uno
Joined
Oct 12, 2007
Messages
19,170
Reaction score
5,199
Here's what should work (on CentOS platforms only) just using tar. The idea is to backup everything except /dev, /proc, and /tmp. On hosted VPS platforms, I think you would also want to remove /sys and /boot since your new VPS will already have one that is needed on the new platform.

Before you begin the backup, you need to gracefully shut down asterisk, mysql, and apache.

After the backup is complete, you need to remove /var/lib/dhclient, /etc/udev/rules.d, and /etc/sysconfig/network-scripts directories as well as /etc/fstab from the backup, or you cannot restore it to a new server that will boot and have network connectivity. Stated another way, the existing disk and network setup in your new container needs to be preserved when you restore your backup.

We also recommend creating your replacement VPS in as close to identical design as the previous VPS, just as if you were starting over, i.e. if you're using Incredible PBX 13-13 with CentOS 6, build the new system the same way NOT with CentOS 7 or Ubuntu or Asterisk 15.

Be sure to restart Apache, then MySQL, and then Asterisk in that order. And copy /tmp/backup.tar.gz to a safe place far away from your VPS.

Our 2.5GB backup on Vultr took about 17 minutes to build using the script below. Then we copied it with scp to a VPS of another provider in a different city in about 15 minutes.

Code:
amportal stop
service mysqld stop
service httpd stop
service sendmail stop
cd /
tar -cf /tmp/backup.tar /bin /etc /home /lib /lib64 /media /mnt /opt /root /sbin /usr /var
service sendmail start
service httpd start
service mysqld start
amportal start
tar --delete -f /tmp/backup.tar etc/udev/rules.d
tar --delete -f /tmp/backup.tar etc/sysconfig/network-scripts
tar --delete -f /tmp/backup.tar var/lib/dhclient
tar --delete -f /tmp/backup.tar etc/fstab
tar --delete -f /tmp/backup.tar etc/resolv.conf
tar --delete -f /tmp/backup.tar etc/hosts
tar --delete -f /tmp/backup.tar etc/hostname
gzip /tmp/backup.tar
 
Last edited:

krzykat

Telecom Strategist
Joined
Aug 2, 2008
Messages
3,145
Reaction score
1,235
krykat, i understand that and have another server on vultr and will be doing this. My issue is, what if something major happens to Vultr and my VPS goes down and i can't even login to my account.... this is not as impossible as it sounds(sunrocket, CloudAtCost), I always play worse side scenario, it helps me sleep at night(along with a little tequila)...

I would like you to meet my two brothers - "Murphy's Law" and "S.N.A.F.U" - they are always crashing my parties, unannounced !!!!!!

I played with the live linux and mondo rescue, this is perfect for what i am trying to accomplish!!!!

Thanks for ALL the help gang!!!
Brian

I'm not disagreeing with you. I agree 100%. I was only pointing out to @dicko that creating a new VM based on the backup of another VM is possible and that creating a copy from a snapshot is not the only method.
 

krzykat

Telecom Strategist
Joined
Aug 2, 2008
Messages
3,145
Reaction score
1,235
Ward, while that will make a pretty solid backup (thinking incrediblebackup here). Is not the better method to have an alternate live site that is originally populated from a Clonezilla (many sites won't allow you to do that though) and then use rsync to keep them current to each other. I mean Vultr is very cool by allowing you to use your own ISO - load up a Clonezilla and do a restore from whereever you want. But I don't think DO would allow the same, and I'm sure rentpbx wouldn't either. I almost think you'd need your own emergency backup that you maintain, either a prox server or Xen. I have even thought recently of doing a failover to the backups that would be a server that could have all DID's on it with the emergency flow (a cellphone or voicemail for example) - that I'd have in the routing of the DID carrier.
 

wardmundy

Nerd Uno
Joined
Oct 12, 2007
Messages
19,170
Reaction score
5,199
@krzykat I'm going to try bringing up a VirtualBox machine using the Vultr backup just to see what happens. Also removing /boot from backup since that would kill other platforms such as VirtualBox. Would be nice to have a generic backup solution that was not provider-dependent.
 
Last edited:

wardmundy

Nerd Uno
Joined
Oct 12, 2007
Messages
19,170
Reaction score
5,199
OK. I think we've got it. I built a generic Incredible PBX 13-13 Lean machine on VirtualBox, copied the Incredible PBX 13-13 Whole Enchilada backup image including Incredible Fax to /tmp, and then executed the following to restore the Vultr backup.

TIP: If you're just testing this, don't forget to shut down Asterisk on the main server before you restore and bring up the backup server. You don't want your trunks trying to register from two different servers!

Code:
amportal stop
service mysqld stop
service httpd stop
service sendmail stop
chattr +i /etc/resolv.conf
cd /
tar zxvf /tmp/backup.tar.gz
# shut off the primary server or at least Asterisk on the primary server now
reboot

Keep in mind that Vultr uses static IP addresses and our VirtualBox machine had a dynamic IP address so the fact that networking survived the restore is pretty amazing. Your root password and other passwords obviously change to whatever they were on your original server. Be sure IPtables doesn't lock you out of the new machine you're building. In our case, it was on a private LAN that already is automatically whitelisted with Travelin' Man 3.

Everything came up once /etc/fstab was either preserved or removed from the original backup. The only thing that didn't work out of the gate was a V1VoIP trunk because they use IP address mapping rather than SIP credentials. You'd have a similar issue with Anveo Direct that requires your IP address to be whitelisted. Google Voice worked fine with OAuth 2 credentials on the original server. Once the IP address was changed on the V1VoIP side to point to the VirtualBox machine, everything worked just as it did on Vultr. Here's the screenshot when the V1VoIP trunk was still down...

DUK396RX4AAM6Zv.jpg


Finally, just to repeat, there is NO WAY this scenario would work with Mondo for about a half dozen different reasons. But feel free to try it and let us know how it goes.
 
Last edited:

jerrm

Guru
Joined
Sep 23, 2015
Messages
838
Reaction score
405
TIP: If you're just testing this, don't forget to shut down Asterisk on the main server before you restore and bring up the backup server. You don't want your trunks trying to register from two different servers!
Our equivalent wraps "systemctl disable/enable asterisk" commands around the tar command, so the restored version never comes up automatically.

There is always a tradeoff. For scheduled runs, there is the risk something like a reboot during the tar command would cause the host to come back up with asterisk disabled, but I think the convenience of avoiding conflicts when working with the backup image is worth it.
 

wardmundy

Nerd Uno
Joined
Oct 12, 2007
Messages
19,170
Reaction score
5,199
Ward, while that will make a pretty solid backup (thinking incrediblebackup here). Is not the better method to have an alternate live site that is originally populated from a Clonezilla (many sites won't allow you to do that though) and then use rsync to keep them current to each other. I mean Vultr is very cool by allowing you to use your own ISO - load up a Clonezilla and do a restore from whereever you want. But I don't think DO would allow the same, and I'm sure rentpbx wouldn't either. I almost think you'd need your own emergency backup that you maintain, either a prox server or Xen. I have even thought recently of doing a failover to the backups that would be a server that could have all DID's on it with the emergency flow (a cellphone or voicemail for example) - that I'd have in the routing of the DID carrier.

FreePBX in particular stores data in MySQL tables, config files, spool files, and even the Asterisk SQLite database. It would be difficult to replicate all of that with rsync.
 

dicko

Still learning but earning
Joined
Oct 30, 2015
Messages
1,607
Reaction score
826
If neither Mondo nor clonezilla appeal , then another kinda "warm spare " method that would likely work on most anything including containesrsa, might be based on rsnapshot, send your snapshots to the backup server and on that machine, and locally rsync the various directories locally back over the local system "BUT ONLY PROVIDED" it is not currently the master, don't be tempted to backup /var/lib/mysql /* directly, but wrap such data in mysql with mysqldump or better mysqldbexport/mysqldbimport (from the mysql-utilities package)

/var/spool/asterisk/*
vvar//lib/asterisk/*
/var/www/*
the various mysql databases

cover most of it and rsnapshot on;ly does the "differences" so is really quite effective, the crontab(s) for rsnapshot can be tweaked to limit the activity to "busy hours" and periods that make sense, perhaps voicemail and recordings and the asteriskdb and perhaps avantfax every 10 minutes or less, other sources of change less likely to piss off your clients less often, it's really quite feasilble
 
Last edited:

wardmundy

Nerd Uno
Joined
Oct 12, 2007
Messages
19,170
Reaction score
5,199
Under the Incredible PBX covers, we've taken a lot of the drudgery out of using MySQL with FreePBX GPL modules by disabling InnoDB and converting all of the databases to MyISAM. What this means is you can shut down MySQL and directly copy the files in /var/lib/mysql. When the files are restored to another server from a backup, they will actually work without the need for wrappers and other MySQL girations. :aureola:
 

dicko

Still learning but earning
Joined
Oct 30, 2015
Messages
1,607
Reaction score
826
Just to confurm, my last rsnapshot based post is a 'set and forget' thingy, it works in tbe background without needing to stop and start mysql or any other service , hence the mysqldump/mysqldbexport wrapper need :)
 

Jose Pinto

Member
Joined
Oct 26, 2017
Messages
148
Reaction score
20
The backup I undesrtand and I can do it but I have 2 problems.

1- After finishes the backup I do not know how to redirect the copy from the cloud server to my personal notebook.- copy /tmp/backup.tar.gz to a safe place?

2- Also where I can learn cron job to automate backups and copies?
Thank you very much for your time attention and help
 
Last edited:

Brianmac

Active Member
Joined
Oct 7, 2014
Messages
594
Reaction score
177
:offtopic:
Hi @wardmundy, I already saw the new post in Nerd Vittles Blog about the Incredible Backup.
Thank you very much for your work and for your kindness in share with with us.
Regards
From Ward's Nerd Vittles post:http://nerdvittles.com/?p=24610
Just make sure it’s in a location 1,000+ miles away from the primary server which should provide ample protection from North Korea’s Rocket Man
Thanks Ward, spitting up coffee from laughing is NOT the way I like to start my day!!!!:drool5:
 
Joined
Oct 26, 2013
Messages
69
Reaction score
23
For those of you who donated early on to Woothosting, before it was discovered that performance is not acceptable, perhaps this would be a good use of that VPS, a target for your backup.
 

wardmundy

Nerd Uno
Joined
Oct 12, 2007
Messages
19,170
Reaction score
5,199
VMware Update: Please note the following addition to this week's Nerd Vittles article. VMware doesn't play nice with full image restores from other platforms. Either stick with VirtualBox for your backup platform or read and heed the following:

DUZVYg1X0AEcoFu.jpg
 

wardmundy

Nerd Uno
Joined
Oct 12, 2007
Messages
19,170
Reaction score
5,199
Just took a full backup image from Vultr in Miami and restored to a 64-bit CentOS 6 minimal install with no PBX software at HiFormance in Atlanta. These VPSs are very different architectures.

Only wrinkle was /etc/resolv.conf was missing DNS servers because of the incorrect search command which was still looking at the Miami server. Issued the following and rebooted:
Code:
echo "nameserver 8.8.8.8" >> /etc/resolv.conf
chattr +i /etc/resolv.conf
reboot

We've added /etc/resolv.conf to the removal list in the backup script.

Everything came up except PortKnocker because of the venet port names on HiFormance. Edited /etc/sysconfig/knockd to add:
Code:
OPTIONS="-i venet0:0"
Then protected the file and restarted: chattr +i /etc/sysconfig/knockd; service knockd restart

All is well. And the next restore from Vultr backup should work perfectly because these files won't be overwritten.

HiFormance cost is $13/year with link. Cheap insurance!
 
Last edited:

wardmundy

Nerd Uno
Joined
Oct 12, 2007
Messages
19,170
Reaction score
5,199
For those of you who donated early on to Woothosting, before it was discovered that performance is not acceptable, perhaps this would be a good use of that VPS, a target for your backup.

Sounded like a great idea so I dusted off four of our WootHosting VPSs to try things out. All 4 of them were dead as a doornail. I want to amend my comment that WootHosting is "almost as bad" as CloudAtCost. I think they actually are worse because they are charging for their oversubscribed "service."

If we ever find one that works reliably, it should be fine as a backup site since they use the same SolusVM platform as HiFormance which we successfully tested above. Of course, that assumes you never actually have to use it.
 

Jose Pinto

Member
Joined
Oct 26, 2017
Messages
148
Reaction score
20
Hi @wardmundy I still need help wit copy and cron.
But I need to tell you that Woothosting is worst then CloudATCost, as I have a server in CloudAtCost since december and until now no problem but I can't use the Woothosting all week I get problem with it so I did not use it. But I will try to use it just to test the Backup, will be only like a storage.
 

Brianmac

Active Member
Joined
Oct 7, 2014
Messages
594
Reaction score
177
Ward, I ran the backup side of this and got the following error message, is this something to worry about?
Code:
tar: etc/hostname: Not found in archive
tar: Exiting with failure status due to previous errors

Thank you for all your effort!!!!
brian
 

Members online

Forum statistics

Threads
25,782
Messages
167,513
Members
19,203
Latest member
frapu
Get 3CX - Absolutely Free!

Link up your team and customers Phone System Live Chat Video Conferencing

Hosted or Self-managed. Up to 10 users free forever. No credit card. Try risk free.

3CX
A 3CX Account with that email already exists. You will be redirected to the Customer Portal to sign in or reset your password if you've forgotten it.
Top