FOOD FOR THOUGHT RentPBX - Ashley Down

[Hyksos]

I agree that kmcdaniel is wrong about backups.
But because his argument is flawed, I believe everyone else is too quick to blame him for his supposedly unwarranted frustration.
They did say that an error was made and something that was advertised and paid as RAIDed, simply was not.
Sh*t happen, that's true. This is low cost and entry level, that's true and so there is no proper SLA to achieve that price point, that's true...
I get all that, but I can still see how someone could feel cheated in some way.

If I rent a VPS or a dedicated box for a certain price and made my choice because it's advertised as RAIDed and I believe that this will prevent most downtime that would be caused by HDD failure, I too, would be pissed to learn I was cheated out of what I was told, and paid for.
It's like misrepresentation by human error.

So, not trying to bash RentPBX, I have no reason to do so and based on multiple user feedback, they do a good job for a killer price.
But still, their ashley customers were told and paid for a RAID which was not there in reality and so there IS something there to be frustrated about.
If you sell a PBX to a customer and tell him its configured with RAID and he figures out during an outage that the cause of the outage is single disk failure and that you never actually configured the RAID correctly, would you agree your customer would be right to be pissed and would you assume he's likely to ask for compensation?

So I could be wrong, and I could be making fallacious assumptions myself and I'm certainly not assuming SLA where there is none, but to me it does seem like although the OP took the wrong arguments for his frustrations, he is in his rights to feel somewhat disappointed to have suffered downtime due a single disk failure on a supposedly RAID setup. Argued this way, I feel like even RentPBX would agree, they made a mistake and what they said was configured, was not, of course their ashley customers have a right to be pissed about THAT, no?

*edit
Pissed might be too strong a word, 15 pops a month, no SLA. Errors can be made and you have no contract to get compensation, disappointed might be a better fit.

 

[briankelly63]

Just as the forum crash was a lesson so is this. It demands that we take a close look at expectations, contracts, SLA, backups, redundancy, availability requirements etc etc. I just hope the people and firms that have been affected can recover quickly.

 

[howardsl2]

Even with hardware RAID, it is very important to frequently check on all hard drives' health status within the array. I've heard an incident where a company had two drives in RAID 1 configuration failing 8 months apart and they weren't aware of it until after the second drive failed. Needless to say that they lost tons of data.

 

[Hyksos]

Even with hardware RAID, it is very important to frequently check on all hard drives' health status within the array. I've heard an incident where a company had two drives in RAID 1 configuration failing 8 months apart and they weren't aware of it until after the second drive failed. Needless to say that they lost tons of data.

Why tons of data loss? They should just have had to restore a recent backup.

 

[howardsl2]

Why tons of data loss? They should just have had to restore a recent backup.

Not sure if they managed to recover from backups later. That's all I heard at that time though (a couple years ago I think).

 

[TwigsUSAN]

I agree that kmcdaniel is wrong about backups.
But because his argument is flawed, I believe everyone else is too quick to blame him for his supposedly unwarranted frustration.
They did say that an error was made and something that was advertised and paid as RAIDed, simply was not.
Sh*t happen, that's true. This is low cost and entry level, that's true and so there is no proper SLA to achieve that price point, that's true...
I get all that, but I can still see how someone could feel cheated in some way.

If I rent a VPS or a dedicated box for a certain price and made my choice because it's advertised as RAIDed and I believe that this will prevent most downtime that would be caused by HDD failure, I too, would be pissed to learn I was cheated out of what I was told, and paid for.
It's like misrepresentation by human error.

So, not trying to bash RentPBX, I have no reason to do so and based on multiple user feedback, they do a good job for a killer price.
But still, their ashley customers were told and paid for a RAID which was not there in reality and so there IS something there to be frustrated about.
If you sell a PBX to a customer and tell him its configured with RAID and he figures out during an outage that the cause of the outage is single disk failure and that you never actually configured the RAID correctly, would you agree your customer would be right to be pissed and would you assume he's likely to ask for compensation?

So I could be wrong, and I could be making fallacious assumptions myself and I'm certainly not assuming SLA where there is none, but to me it does seem like although the OP took the wrong arguments for his frustrations, he is in his rights to feel somewhat disappointed to have suffered downtime due a single disk failure on a supposedly RAID setup. Argued this way, I feel like even RentPBX would agree, they made a mistake and what they said was configured, was not, of course their ashley customers have a right to be pissed about THAT, no?

*edit
Pissed might be too strong a word, 15 pops a month, no SLA. Errors can be made and you have no contract to get compensation, disappointed might be a better fit.

You are right. I wasn't disagreeing about the downtime frustration, it was purely the thought process that RAID was a backup. I run into this all the time and just shake my head.

 

[kmcdaniel]

So since it is clearly wrong to expect that an entire server would be backed up on their end. And now since they have been successful in getting the recovering the once thought to be lost data, can someone shed some light or point me in the right direction to now make sure that I have a complete backup of my vps PiaF green machine? I want to insure that all incredible settings, system recordings, freepbx configurations, and aastra scripts are backed up in case of another mishap. Thanks in advance and lesson learned!

 

[Hyksos]

Rent another pbx and play with FreePBX backup and manual backup of everything you need to setup another box until you can reload another clone of your working PBX in a timeframe that you find tolerable.
You backup and extract stuff from the working one and practice reloading a new one with what you backed up and extracted from the working one.

Once you successfully reloaded a new PBX with everything and it works like you want, you now know what you need to frequently backup from the working setup to be ready to setup a new one if something was to happen.

But because you chose to be setup that way there are limitations which you can only go around with knowledge of the system.
For example if down the line you want to rebuild but rentpbx doesn't offer the same base installation anymore, you'll have to reload on a different base installation then what you were backing up from and that might throw some curve balls at you. When RentPBX install a new system for you there is no guarantee it's going to be the same base install as your working one. And if the disaster at hand is RentPBX closing shop then you'll need to restore your backup on a different base install and still be able to make it work.

Short version: using a VPS provider like RentPBX has upsides and downsides. Downside being you can't really do entire system image and reload them wherever you want, you didn't rent an hypervisor, you rented a VPS. You use FreePBX backups and manual backup of stuff that might not be included in FreePBX backups and you practice reloading all that in a new base install. And you hope that when the day comes, your simulated scenario still applies and that you can still request a new PBX base install that is identical to what you practiced reloading on.

Other people with other scenarios might have different or more options when it comes to continuity and disaster recovery. If I have an entire system image of a system, I don't care that I couldn't download that ISO and reinstall the same base install and that even if I could, it might not install just like it did back then.
I simply restore the full image and I get a bit for bit clone of the system. Or some further their Linux and FreePBX/asterisk knowledge to a point where if they routinely backup their configs, they can reload that in a different base install and make it work if they hit a snag.

I prefer doing an install once, then using system images, snapshots and backups to make sure I can reload that if need be, as is, not on a new base install. You keep upgrading that system to keep it current if you want to and you basically NEVER do a full reinstall from a fresh installed system. You roll and roll and roll with always the same system through in place upgrades. Secured by tested and simulated full system backups and snapshots. If one day you need or want to reinstall from scratch, you won't be in a hurry, you'll rebuild the new system while the old one is working and transition to the new one only when it's tested functional, once you're in the clear you decommission the old one and start doing the same full system images and perpetual in place upgrades.

 

[dad311]

Well hats off for RentPBX retrieving the data off the damaged drive and restoring the system. THANK YOU!!!!! HOWEVER, this should have never happen.

Im I correct to assume RentPBX has no full server backups in case of a total failure like this one?????

 

[TwigsUSAN]

Well hats off for RentPBX retrieving the data off the damaged drive and restoring the system. THANK YOU!!!!! HOWEVER, this should have never happen.

Im I correct to assume RentPBX has no full server backups in case of a total failure like this one?????

Well, this is a section of what they wrote.

Since our service is an unmanaged service and also due to the sensitivity of data in your PBX (for example: trunk info, voicemails and call detail records), we do not keep copies/backups of your data.

With that statement, you could read it both was. I read it as they don't backup the whole server.

 

[kmcdaniel]

Well hats off for RentPBX retrieving the data off the damaged drive and restoring the system. THANK YOU!!!!! HOWEVER, this should have never happen.

I'm I correct to assume RentPBX has no full server backups in case of a total failure like this one?????

This exactly what I was trying to get at too. I agree Dad311, it should have never happened. However I am ecstatic that they were able to recover the data! It is the full server backup that I guess was my mistake to assume they had a process in place for. I definitely learned my lesson, so thank you to RentPBX for teaching that to me and also a big thanks for recovering the data.

 

[kmcdaniel]

Rent another pbx and play with FreePBX backup and manual backup of everything you need to setup another box until you can reload another clone of your working PBX in a timeframe that you find tolerable.
You backup and extract stuff from the working one and practice reloading a new one with what you backed up and extracted from the working one.

Once you successfully reloaded a new PBX with everything and it works like you want, you now know what you need to frequently backup from the working setup to be ready to setup a new one if something was to happen.

But because you chose to be setup that way there are limitations which you can only go around with knowledge of the system.
For example if down the line you want to rebuild but rentpbx doesn't offer the same base installation anymore, you'll have to reload on a different base installation then what you were backing up from and that might throw some curve balls at you. When RentPBX install a new system for you there is no guarantee it's going to be the same base install as your working one. And if the disaster at hand is RentPBX closing shop then you'll need to restore your backup on a different base install and still be able to make it work.

Short version: using a VPS provider like RentPBX has upsides and downsides. Downside being you can't really do entire system image and reload them wherever you want, you didn't rent an hypervisor, you rented a VPS. You use FreePBX backups and manual backup of stuff that might not be included in FreePBX backups and you practice reloading all that in a new base install. And you hope that when the day comes, your simulated scenario still applies and that you can still request a new PBX base install that is identical to what you practiced reloading on.

Other people with other scenarios might have different or more options when it comes to continuity and disaster recovery. If I have an entire system image of a system, I don't care that I couldn't download that ISO and reinstall the same base install and that even if I could, it might not install just like it did back then.
I simply restore the full image and I get a bit for bit clone of the system. Or some further their Linux and FreePBX/asterisk knowledge to a point where if they routinely backup their configs, they can reload that in a different base install and make it work if they hit a snag.

I prefer doing an install once, then using system images, snapshots and backups to make sure I can reload that if need be, as is, not on a new base install. You keep upgrading that system to keep it current if you want to and you basically NEVER do a full reinstall from a fresh installed system. You roll and roll and roll with always the same system through in place upgrades. Secured by tested and simulated full system backups and snapshots. If one day you need or want to reinstall from scratch, you won't be in a hurry, you'll rebuild the new system while the old one is working and transition to the new one only when it's tested functional, once you're in the clear you decommission the old one and start doing the same full system images and perpetual in place upgrades.

So exactly what is the benefit to running a production machine on vps? Besides saving money on electricity. I use proxmox for my other onsite production client servers and keep full image snapshots pushed to crashplan, so I understand that. I guess I must rethink this.

 

[Hyksos]

So exactly what is the benefit to running a production machine on vps? Besides saving money on electricity.

If we exclude benefits of other hosted solutions and think of those specific to VPS, I'd say price. That's why VPS exist. They do the job well depending on what you're wanting to do and they're a nice value cost wise compared to dedicated servers.
Other advantages like not buying hardware, having big pipes, etc. Would also apply to other hosted solutions like dedicated servers. Proxmox can be hosted too, I use that among other things, so I get the benefit of hosted and the benefit of an hypervisor under my control.

But that doesn't exist for 15 pops a month... you need just a bit more budget. So it's not something one can compare to Rent PBX. It's 2 different things for different needs.
I'd use both, depending on the need, budget and technology requirement and preference.

I insist, RentPBX is a very good product, VPS-PBX. It does have tons of advantages and you can organize around it to minimize downtime and ensure DR and business continuity. You just have to use other tools since you can't click "Backup" in the hypervisor. Which is not a surprise since a cost/benefit analysis always show that less money buys you less flexibility/features then more money if you're comparing good products. And it's not like a lot of things are impossible with Linux, one could rsync his RentPBX I guess, it's just less clean then hypervisor VM backup management.

 

[kmcdaniel]

If we exclude benefits of other hosted solutions and think of those specific to VPS, I'd say price. That's why VPS exist. They do the job well depending on what you're wanting to do and they're a nice value cost wise compared to dedicated servers.
Other advantages like not buying hardware, having big pipes, etc. Would also apply to other hosted solutions like dedicated servers. Proxmox can be hosted too, I use that among other things, so I get the benefit of hosted and the benefit of an hypervisor under my control.

But that doesn't exist for 15 pops a month... you need just a bit more budget. So it's not something one can compare to Rent PBX. It's 2 different things for different needs.
I'd use both, depending on the need, budget and technology requirement and preference.

I insist, RentPBX is a very good product, VPS-PBX. It does have tons of advantages and you can organize around it to minimize downtime and ensure DR and business continuity. You just have to use other tools since you can't click "Backup" in the hypervisor. Which is not a surprise since a cost/benefit analysis always show that less money buys you less flexibility/features then more money if you're comparing good products. And it's not like a lot of things are impossible with Linux, one could rsync his RentPBX I guess, it's just less clean then hypervisor VM backup management.

What is your recommendation for hosted solutions, either pbx with snapshot capability or hosted promox servers? Thanks!

 

[Hyksos]

What is your recommendation for hosted solutions, either pbx with snapshot capability or hosted promox servers? Thanks!

Not sure I get what you mean by PBX with snapshot capability.
As far as recommendations, quite impossible for me to make them since I don't have the required information to form an opinion.
If you need to buy a 50$/month Xeon with 32G RAM and 2TB of storage to replace a 15$/month VPS, just to get access to the hypervisor for easier backup management.... it's quite the price increase just because you suffered a one time event that is very highly unlikely to rehappen.

It might make more sense to implement backup with FreePBX and tar/rsync/whatever and send it out to a remote location.

I was merely pointing out the fact that full system image are quite hard to obtain from a running VPS. And that because of that the backup method is a bit more complex/unpredictable and thus should be investigated, implemented and tested before an actual event occurs. What if this entire RentPBX server gets fried, corrupts his entire FS, or goes down because of something like Sandy or whatever... really.
How would you restore the service, where, with what and within what timeframe. Then simulate that x time a year to make sure it's still working on the newer version that you might have to restore on.
For some the first thing they will realize is that they now need to change the PBX IP in all their endpoints... Very bad design choice, better realize that on a faked disaster then a real one.

Also realize that some just decide to document their config well enough, backup their recordings, voicemail and CDR and simply reconfig from scratch with their documentation if something goes down. If you try that and it works after 3 hours of work, you have the cheapest and most efficient continuity/DR plan ever. If 3 hours of downtime is acceptable. For some that is a complete nightmare, because their config is too big, but usually if the config is big, there is money to implement proper continuity/DR plan to avoid manual work.

Some also have cold spare at AWS EC2. That is in part where the cloud shines. Almost zero fees, sits there all configured and ready to go.
Once a year you boot that puppy and update the PBX FQDN and do a planned DR test with the people using the system or on a Saturday or whatever.
All done in advance, dirt cheap, easily testable, but again, even that might be overkill or not the best compared to in house DR plan on your clustered hypervisor.

Short version: depends...

 

[rentpbx]

Here is our latest status update:
All affected PBXs have been restored and all data from the bad hard disk is recovered. Those affected PBXs have all been moved to a new node (a replacement server) that has a proper mirrored RAID configuration at the same location. The affected customers have the same IPs at the new node.

Our actions in response to this incident:
·Following the event, we worked overnight with our data center to recover the failed hard drive
·During the downtime, we created a temporary PBX for all affected customers for free
·We give one-month credit to those affected PBXs.
·We set up a replacement server with the proper mirrored RAID configuration at the same location.

Just a little bit about our service:
RentPBX is geared toward high uptime. To ensure high uptime, we do have high redundancy in place. For example we hosted our customers’ PBX in a data center that has multiple carriers connected to it and our hardware is designed with redundancy including drive, power supply and CPU.

In some cases such as natural disaster situation, even if you have all these redundancies in place, there is a possibility that your PBX becomes unavailable. If uptime is critical to your PBX, we propose a solution by implementing automatic failover to a secondary PBX at different region using currently available resources. For a rough demo of this idea, please follow this post.

Regardless of any redundancy strategies, it is always prudent as a user to have local or offsite backup as last resort if any redundancy measure in place failed.

Best Regards
RentPBX Team