Is asterisk right for me? Disaster recovery, Virtual machine and other questions

[nicknomo]

I am currently exploring the idea of using an asterisk based system for a corporate PBX. I am a little concerned about moving from an appliance based system to a computer based system.

My company has 21 incoming lines, and about 60 extensions for interoffice communication. The incoming lines are typically 50% idle, and more so on the extensions.


While I am purely an IT person (not really a "phone" person) and I am familiar with Linux, I am feeling a little uncomfortable with running this system without some sort of backup and recovery plan. From the looks of it, you can back up the configuration, but you need to pretty much have the same installation source (same version of PBXIAF). If 4-5 years down the line, the computer dies, I'll likely need an updated kernel and that would probably mean a whole new installation of CentOS, and probably require me to update PBXIAF.

The configuration I'm looking to use is going to take some fiddling to get working. If I have to start from scratch when my computer dies, there will be way too much down time.

I saw mondo being used as a backup/recovery solution, but this is only useful for hard disk failure (not a whole computer change scneario).

The only way I could think to get around this problem would be to use a Virtual Machine, like VirtualBox (much better than VMWare as far as resources go). This is ideal for disaster recovery, as you can easily put your backup of the virtual machine on just about any modern media and be up and running again in ten minutes.

There are of course some other problems:
1) Integration with a PTSN - like a T1, or POTS lines.
2) Performance concerns

As far as #1, I was thinking about implementing some sort of VoIP appliance based gateway. I would think this would solve the issue even if it is more costly. I would like to know how much more difficult this would be and what is a good piece of hardware for handling a single Voice T1. In other words, I'd connect to the PTSN through a PRI on some appliance, and then route connections to my asterisk based VMWare install.

As far as performance goes, Virtualbox is very lean. I know VMware has some choppy voicemail, but I know virtualbox is capable of achieving near native performance. On a fast Quad core box, do you think it could handle the typical loads of my company?

I know this isn't the preferred way of doing things. I do see some advantages to doing it this way. It will be easily deployable in multiple locations, easily backed up, easily cloned, option of having snapshots... and the only real point of failure would be the VoIP gateway.

I'm welcome to other suggestions, or suggestions on how to best implement this.

 

[wardmundy]

Just a few comments, and I'm sure some others will chime in. The key for me (and it sounds like for you) is a reliable backup solution. Mondo is virtually perfect so long as you don't switch drive types. For example, if you have a system with SATA drives and switch to another type of machine that also has a primary SATA drive, the restores work great.

FreePBX gives you another layer of backup protection. It catches most of your setup and data. So long as you have the same version of FreePBX on the old machine and the new one, you can build a new machine with the latest version of PBX in a Flash and restore most of your stuff just with a FreePBX backup.

If you're really paranoid, take a look at Acronis which is a commercial product that works very, very well.

Backups are getting better and better by the month. Mondo is updated almost weekly, and there is a major rewrite of the FreePBX backup module underway as well. Two years ago in the trixbox and Asterisk@Home days, there were no backup solutions. So you're starting at the right time.

Finally, I would ALWAYS purchase a spare duplicate hard disk of the same type as your main drive... AND test it. Then put it away (off site!) for a rainy day.

 

[nicknomo]

Thanks for the reply..

From my experience though, something like Mondo may have issues. It would be fine if I had a disk failure, or swapped a part or two... and probably would work fine if I bought a similar replacement computer. I'm thinking about a full death situation where in the future I need to buy hardware t hat currently doesn't exist.

I've previously tried to load older versions of redhat and/or Debian on newer computers and have had less than stellar luck. I've gotten missing hardware, kernel panics, etc etc. The solution is often to load updated versions of the OS, with newer kernels with newer and better drivers.

We've had our current PBX for 15 years. I'm thinking 5 years from now, when PC architecture has changed some more, the 2.6 kernel and CentOS 5.1/5.2 may not like changing hardware. I could probably apply patches or kernel updates, but there is no guarentee this would even be possible or that it wouldn't "break" the version of asterisk I have on there.

I'm certainly not a Linux "expert" by any means, so I could be wrong here.. Its just that some times Linux distros don't like current, newly released hardware... and we are talking about hardware that doesn't exist until 5 years into the future. There is a good chance I'd have to take a fresh install, and probably have to reconfigure just about everything.

That is why I love VM's though. The hardware is virtual (and never changing). You don't have to worry about this sort of thing. Its the only way I see this working out for another 10-15 years (like the other PBX). Its either that or keep 5 Dell T100's in storage .

 

[jroper]

Hi

I've put some comments inline.

I am currently exploring the idea of using an asterisk based system for a corporate PBX. I am a little concerned about moving from an appliance based system to a computer based system.

At least 2 legacy PBX systems I know are running on a cut down windows NT4. which would you prefer to run, NT4 or Linux. Of course you would not know this as we don't really now anything at all about the bigger manufacturers products. You can be relatively certain though that under the PBX shaped box, its probably broadly the same stuff, e.g. processor, memory and a storage medium.

My company has 21 incoming lines, and about 60 extensions for interoffice communication. The incoming lines are typically 50% idle, and more so on the extensions.

No problem. The limiting factor in Asterisk is the number of concurrent calls, not the number of extensions.

While I am purely an IT person (not really a "phone" person) and I am familiar with Linux, I am feeling a little uncomfortable with running this system without some sort of backup and recovery plan. From the looks of it, you can back up the configuration, but you need to pretty much have the same installation source (same version of PBXIAF). If 4-5 years down the line, the computer dies, I'll likely need an updated kernel and that would probably mean a whole new installation of CentOS, and probably require me to update PBXIAF.

You are in the best position to install this stuff, it's 80% networking and about 20% telephony, you already know 80% of the job. Phones and PBX are just another network device that IT and networking people understand. We can ping them, traceroute etc. As an IT professional you are in familiar territory.

An installation of PBX in a Flash is relatively quick and simple, restoring FreePBX is pretty painless. You should be able to move from hardware to hardware without too many grey hairs

The configuration I'm looking to use is going to take some fiddling to get working. If I have to start from scratch when my computer dies, there will be way too much down time.

Pick the right machine ad the computer is no less likely to die than any other piece of electronic equipment. These machines are very understressed, writing to the hard drive is relatively low especially compared to say a file server. Most machines are broken by fat fingers. Leave it alone and it will be fine.

I saw mondo being used as a backup/recovery solution, but this is only useful for hard disk failure (not a whole computer change scneario).

FreePBX backup, and copies of the config files should get you up and running.

The only way I could think to get around this problem would be to use a Virtual Machine, like VirtualBox (much better than VMWare as far as resources go). This is ideal for disaster recovery, as you can easily put your backup of the virtual machine on just about any modern media and be up and running again in ten minutes.

I'd advise against this, run with good quality dedicated hardware, and consider a warm standby. You can trial it and see how you get on. But start thinking more like a PBX installer rather than an IT pro. I am sure many people can show screen shots of PBX systems with great uptime - the best I can offer is one of my highly used A2Billing installs used for Europe with an uptime of 42 weeks.

There are of course some other problems:
1) Integration with a PTSN - like a T1, or POTS lines.
2) Performance concerns

Performance concerns are a worry, connection to the T1 is easy enough with a pstn gateway if you go that way

As far as #1, I was thinking about implementing some sort of VoIP appliance based gateway. I would think this would solve the issue even if it is more costly. I would like to know how much more difficult this would be and what is a good piece of hardware for handling a single Voice T1. In other words, I'd connect to the PTSN through a PRI on some appliance, and then route connections to my asterisk based VMWare install.

As suggested earlier, go with a good box with a T1 card installed in it. It's best to keep phones separate from the rest of the infrastructure. Bosses get really tetchy if the phone does not work because you have damaged NIC in a PC on your network somewhere.

As far as performance goes, Virtualbox is very lean. I know VMware has some choppy voicemail, but I know virtualbox is capable of achieving near native performance. On a fast Quad core box, do you think it could handle the typical loads of my company?

I know this isn't the preferred way of doing things. I do see some advantages to doing it this way. It will be easily deployable in multiple locations, easily backed up, easily cloned, option of having snapshots... and the only real point of failure would be the VoIP gateway.

I'm welcome to other suggestions, or suggestions on how to best implement this.

I'd suggest chosing one of the guys of the forum who knows this stuff, and getting in some consultancy to hold your hand during the install.

My next project is looking at High Availability PBX systems, based on this http://support.red-fone.com/downloads/elastix/Elastix_HA_Cluster.pdf so if any one knows anything about this stuff, then I could really use some help over the next few days.

Joe

 

[kenn10]

I am a little concerned about moving from an appliance based system to a computer based system.

Interestingly enough, most of the Avaya Communications Manager media servers are running on Red Hat Linux. The hardware it runs on is fairly common, though customized, PC's.

Regardless of the hardware and OS the big boys run on, the big thing to consider is whether Asterisk (or any open source system) can provide you with the specific features and reliability you need. Secondly, you need to consider who will be responsible for hardware, software, telephone set malfunctions, troubleshooting analog and digital telephone company lines, and properly configuring the network for VOIP and VLAN's to prioritize and transport your service. Unless you use a value-added reseller of Asterisk based systems, all those duties will fall on your shop.

The most common misunderstanding of an IT shop is that "voice is just another application on the network." In its simplest form, that Cisco sales pitch is correct, but many IT shops are grossly unprepared to be thrust into a situation where telephone users expect no downtime and the voice solution is now running on their data network.

There are many solutions for a Linux cluster or high availability systems. Just understand that it is highly unlikely that a VOIP system will ever reach the "5 Nines" or 99.999% of uptime that hybrid and proprietary systems can offer. Even the "big boy" systems don't perform well if the network upon which the are running is poorly designed or maintained.

You just need to be thoughtful and deliberate as you think about diving into telephony alone. Good luck!

 

[dobbs]

I'm happy to provide my 2 cents. See my inline comments:

I am currently exploring the idea of using an asterisk based system for a corporate PBX. I am a little concerned about moving from an appliance based system to a computer based system.

If you go with top-notch hardware, you won't have to worry about this distinction.

My company has 21 incoming lines, and about 60 extensions for interoffice communication. The incoming lines are typically 50% idle, and more so on the extensions.

You are describing a fairly typical system profile. Are these pure analog lines? Do you have a data circuit as well? More importantly, do you have a current contract term for your current circuit(s)?

While I am purely an IT person (not really a "phone" person) and I am familiar with Linux, I am feeling a little uncomfortable with running this system without some sort of backup and recovery plan. From the looks of it, you can back up the configuration, but you need to pretty much have the same installation source (same version of PBXIAF). If 4-5 years down the line, the computer dies, I'll likely need an updated kernel and that would probably mean a whole new installation of CentOS, and probably require me to update PBXIAF.

With PIAF, upgrading is not an issue. I just did a complete kernel upgrade and source upgrade, then upgraded FreePBX on a production system. the whole process took about a half hour and everything is running perfectly - not that it wasn't before.

The configuration I'm looking to use is going to take some fiddling to get working. If I have to start from scratch when my computer dies, there will be way too much down time.

I'm not clear on this reasoning. Are you thinking of using your current desktop or server running as the PBX at the same time? Even a brand new computer will fail - someday. So, you want something with redundancies...RAID 1, I prefer 3Ware controlers, but Dell's new controllers work fine too. Redundant power supplies are great, or at least have a spare at the ready. I'm a believer in keeping the server lean and as cheap as possible while not skimping on the processor or RAM. The Dell T300 is what I've been using lately. Redundant power, hot swap drives, and a Dell Service Warrenty for whatever else may go wrong.

I saw mondo being used as a backup/recovery solution, but this is only useful for hard disk failure (not a whole computer change scneario).

With RAID1, you dont' have to worry about a drive failure. Of course, the Raid controller can always go bad.

The only way I could think to get around this problem would be to use a Virtual Machine, like VirtualBox (much better than VMWare as far as resources go). This is ideal for disaster recovery, as you can easily put your backup of the virtual machine on just about any modern media and be up and running again in ten minutes.

Why would you want to do something like this? Why not just have a redundant server either on standby or running if you're uber paranoid.

There are of course some other problems:
1) Integration with a PTSN - like a T1, or POTS lines.
2) Performance concerns

As far as #1, I was thinking about implementing some sort of VoIP appliance based gateway. I would think this would solve the issue even if it is more costly. I would like to know how much more difficult this would be and what is a good piece of hardware for handling a single Voice T1. In other words, I'd connect to the PTSN through a PRI on some appliance, and then route connections to my asterisk based VMWare install.

I would highly recommend sticking with a dedicated server and server-based PRI card, either from Sangoma or Rhino. Get the echo cancellation just to be safe. You may not actually need it depending on your circuit, but if you end up with echo your life will be hell as the entire phone system (and trusting you) may become a perceived mistake.

As far as performance goes, Virtualbox is very lean. I know VMware has some choppy voicemail, but I know virtualbox is capable of achieving near native performance. On a fast Quad core box, do you think it could handle the typical loads of my company?

Probably, but there's no good point to doing this. You are introducing complexity and reducing the performance of your system.

I know this isn't the preferred way of doing things. I do see some advantages to doing it this way. It will be easily deployable in multiple locations, easily backed up, easily cloned, option of having snapshots... and the only real point of failure would be the VoIP gateway.

You will end up with potential points of failure regardless - probably more with the setup you've described. Why add a virtual OS layer to the mix? Routers, NICs, Power Supplies, T1 interfaces, Provider Smart Cards, and the list goes on. Keep your server as simple and hardened as possible. If you use the same hardware at each location, you can still backup FreePBX and use that to clone your configuration to other locations, sans the OS and drivers which would be fairly painless anyway.

I'm welcome to other suggestions, or suggestions on how to best implement this.

My remaining question is the most important of all. What is in this for you? I am a huge fan of telephony and Asterisk and would not trade my system for anything else. I also highly recommend the types of systems I've outlined above to my own clients, mainly because I know that they will be delighted regardless of what goes wrong either during the installation, implementation, or down the road because I can make sure of that. I'm a consultant - that's what I do and get paid to do.

BUT what is in this for you personaly? Of course, it will be a great learning experience. It may look good on your resume. However, you don't want to have to circulate that resume before the ink is dry. Once the installation begins, whatever plan that's in place will most likely be challenged. It may be the T1 that is either delayed or incorretly configured. Or it could be a bad card, bad router, or some other wrinkle rolling out the handsets or any number or conbination of other things. This is not meant to discourage you, only to make sure you have some good incentive to make taking this leap justifiable.

 

[nicknomo]

Ok, a lot of replies.. that is good, but its a lot to reply to. I'm going to try and condense my replies and hit some important points:

1) Linux vs Windows - I do actually think Linux is more stable than Windows. Linux is more likely to work if you transfer a drive image from machine to machine (Windows will almost always have problems without prep work or tinkering). I would indeed prefer Linux to be running on a PC based PBX.

2) Appliance vs PC - Someone mentioned some old PBX's run off of NT, some already use unix/linux, etc. I think you might have missed my point though. These devices don't really change. I can buy another one and overnight it if I want. Pop, in it goes, and I'm running again. Might have to reload the configuration file, or even flash firmware to it... but that is it. No tinkering at all, for the life of the system. If something failed, I don't have to worry about replacing it. I can STILL get parts for my Merlin phone system off of Ebay.

I would like to make an asterisk implementation where problems are easily rectified.

3) Changing Hardware on a Linux kernel - Maybe someone could clarify for me exactly my limitations are here. I know storage drivers, and file system drivers need to be in the Linux kernel... What other requirements come in to play. I have had a lot of bad luck with this. Is there something I am missing regarding moving a drive (or image) to new hardware? I posed this question in a linux IRC and pretty much got flamed to death for even suggesting the idea. The line of thinking there was new hardware (with a 5 year old kernel) needs a distro upgrade, or at the very least a kernel upgrade.

4) Reinstalling PBXIAF and reloading the configuration - I think the problem with this suggestion leads to the above issue. The installation ISO I have now comes with Centos 5.x . What if I need 6.0 to work with my hardware (assuming a new kernel is needed)? I'd probably need a new version to reload it. I'd be stuck with using a FreePBX backup rather than a drive image.. which leads to

5) FreePBX backups : I imagine newer versions of PBXIAF will come with newer versions of FreePBX. Isn't it likely that my FreePBX backup will not work with the new version of FreePBX or similarly the newer version of asterisk? I am currently assuming yes

6) Reliability of hardware: I build most of my own servers in my company (for quality control purposes). I've had a very low failure rate, and if I built a machine for asterisk I'd hope it would last. Still, I've had $300 machines last for a long time, and $3500 machines constantly break. Our corporate server (IBM iSeries) uses very high end expensive parts, but isn't immune to problems. I know enough to keep spare disks, use RAID, and backup my data... but I've had motherboards go on me before.

I think I'm paranoid because the motherboard on my last Exchange server died. The Supermicro board I used was out of production, and I needed to build a new machine. It took me 5 days to get a new machine up and running and transfer the data over. In the mean time, a VMWare based Exchange server held us over.

7) Benefits of Virtual Machines- Some have suggested the VM is an added point of failure, complexity and hurts performance. I agree to some extent. Still, they are handy. As I mentioned I used it to replace a downed exchange server while I built a new physical server. To be fair, I did switch because the performance was really bad for a server servicing 150 users holding 40 GB of email. So, I understand the opposition.

Still, it has to be the best insurance for disaster recovery I know of... I use it a lot for things like AD servers, downed file servers, etc. It can be run almost anywhere, and isn't attached to any one machine.

Maybe there are Hybrid fail-over solutions, like what I did with my exchange server?

8) Buying a spare - I have given this serious thought. If I buy 3 identical machines, that should be more than enough to solve my problem. The cost would be well justified. Still though, its not ideal. What I'm currently trying to do is for a single location. We have 13 locations. It would be really nice to make an easily portable, duplicatable solution (and deply it whenever I want).

9) Why I am interested in asterisk distributions- Well, I like the customization and the capabilities. You can do some pretty cool stuff with VoIP. It seems like Asterisk is the best choice for VoIP (Cisco seems too expensive as well).


I hope all of this doesn't come off combative. Its just the problems I see, and I'm trying to gauge the best way to get around them.. or if maybe I should go another route. I appreciate the candid advice so far.

 

[nicknomo]

Ok, now that I think I replied to everyone, here are some specific questions I have:

1) How do you ensure uptime?: Aside from the basic tenets of server management (have common spare parts, redundancy in the server, quality components, backups, etc etc)... how do I strive for that five 9's uptime? In other words, what do I do if that machine flat out dies and I don't have a spare 5 years from now. What can I do to get something else up and running on a future system, 5 years from now, without a fresh install and manual re-configuration? One choice is to have a spare clone of the machine. What else? Is there another option without resorting to VMs?

I know some of you would opt to just reprogram a new asterisk install... but lets assume the configuration is probably going to take me a while, and I'm only doing it once. What is the best way for me to bypass this perceived issue of mine?


2) Issues with VM's?: I hear there are some conferencing issues with virtual machines, amongst other things. Can anyone expand on what exactly the core issues are? I know the performance up to par, but am I to understand there are some other problems?

3) Any opinions on RedFone Fonebridge's?: These supposedly integrate into asterisk servers throught TDMoE. They remove the T1 hardware from the asterisk server. It lets you move to a blade server, a VM, etc etc.

I was thinking going this route whether I bought a physical server or not. They aren't that expensive, and would allow me to use other high availability solutions.

Jroper's post intrigued me. His PDF listed this as a HA option. I would think it would be perfect if the failover device was a VM. It would be SOMETHING to hold the fort down while a new dedicated machine was fixed, or the downed machine was repaired.

That is the type of idea I could run with. Anyone do anything along these lines? Any other High availability options? Anything planned in the future in either the PBIAX or Asterisk community?

4) Should I stick with a classic digital phone system?: Considering the need for reliability over features, should I stick with a regular digital phone system? We have a pure gigabit network, so I don't anticipate congestion issues, and all calls would go out over a PRI... I can vouch for the local data... but are there any other issues I should consider regarding user/caller satisfaction? Are there other problems I should anticipate?

 

[jroper]

Hi

Just a couple of points to clarify.

FreePBX for a few versions now has been upgradeable, with no issues. It also works across all releases of asterisk (1, 1.2, 1.4 and 1.6). So even if you have to do an OS reload, the chances are that restoring your FreePBX backup will bring your PBX back to the same state it was before.

Secondly, it's probably fair to say that the commercial manufacturers have exactly the same issues and problems we do, because its all made out of the same stuff. The difference being, you never hear about it.

Issues with VM's are in timing issues - you need a timing source, either a card, or it uses USB and latterly the processor clock. These may not exist in a vm session, so no timing source = no conferencing. Not all VMs have this problem.

In my prototype HA system, I used two vmware sessions. there is no reason why one of the machines should not be an vm session. The solution does require a lot of network traffic, so some sort of separate NIC would be good.

In anycase consider putting the phone ona sperate network segment. You don't want problems on your computer network causing problems on the phone network. The phone quality will be the first indication that something is wrong, ad will therefore get the blame.

Those of us who have done a few installs, given a recent freePBX backup and a suitable machine, could probably have a replacement box built working and implemented in under an hour.

In terms of the feature set, the speed and pace of development of this stuff is amazing. Provided that you get the base install correct, the network well sorted, I would not envisage any real problems.

Finally I would re-iterate Kenn10's comment

You just need to be thoughtful and deliberate as you think about diving into telephony alone. Good luck!

Which you clearly are.

Joe

 

[jwells]

A few thoughts to add. I think all the issues you have raised are valid points to consider it just comes down to what you are comfortable with in the end you own it, you built it, you support it.

Quote "We've had our current PBX for 15 years" but does it have the "features" that you need or want now ? I think that is one trade off. The biggest one is that you are doing it yourself picking whatever hardware you are comfortable with. All that throws out most of the agrument about the ability to buy same spares 8 years from now etc.

I have seen many computer based PBX systems in offices for the last 8-10 years when I first noticed them being installed. Altigen comes to mind. They are fairly common PC hardware with proprietary cards for fxo fxs ports. How about the Nortel BCM's they are rack servers running a version of the same.

That all said I have seen many Asterisk Servers up for the last 4 years with no downtime. If the server dies you really can get another up if you have spare hardware in about an hour at least to get calls to a receptionist while you get the rest working and configured. Of course since you take this on yourself you have to have a backup plan as you are working out that includes a spare server. Hardware is cheap compared to what you pay for the "MainStream Proprietary PBX" The Redphone HA is always an option.

I think once you jump in and install it a few times and get more comfortable with the install etc you will have more confidence in your decisions.

Myself I "image" the final setup to a spare drive in the system since Hard Disk failure or corruption is the most likely issue 5 years from now. The image is ready and waiting. Also doing regular backups off the PBX.

Just get good quality hardware which you seem to have a handle on already and have some fun with it


Jim

 

[gaijin]

We use VMWARE ESXi for all builds...

In the mean time, a VMWare based Exchange server held us over.

7) Benefits of Virtual Machines- Some have suggested the VM is an added point of failure, complexity and hurts performance. I agree to some extent. Still, they are handy. As I mentioned I used it to replace a downed exchange server while I built a new physical server. To be fair, I did switch because the performance was really bad for a server servicing 150 users holding 40 GB of email. So, I understand the opposition.

Still, it has to be the best insurance for disaster recovery I know of... I use it a lot for things like AD servers, downed file servers, etc. It can be run almost anywhere, and isn't attached to any one machine.

Maybe there are Hybrid fail-over solutions, like what I did with my exchange server?

We now exclusively use VMWare ESXi (don't use server) for all of our commercial PBX builds 10-100 users. We scale the hardware appropriately, but we have yet to have any issues.
Hell if you can run on an atom CPU, surely you can run on a VM XEON Quad core...

Just make sure you use the VM kernel for centos and you shouldn't have any problems.

The redundancy scenarios available are beyond comprehension.

 

[gaijin]

2) Issues with VM's?: I hear there are some conferencing issues with virtual machines, amongst other things. Can anyone expand on what exactly the core issues are? I know the performance up to par, but am I to understand there are some other problems?

Yes there are timing issues with conferencing and VMWare, however they only apply to meetme conferences, not phone based 3 way conferences (most common). Lets be honest, how often will you even use meetme, given your current system likely doesn't even have that capability?

However, as long you have ZTdummy running you will have no problems. (just run genzapconf)...

 

[oldciscodude]

Reading through the posts, all great ideas. But one area seems to have been missed.
Taking a page from my old Cisco CallManager days, in each of your phone configs, (especially Cisco) you have a place for a second or backup or emergency "proxy". This is the fall back proxy that the phone reaches out to when the first proxy is unavailable. This "should" let your first server fail and your second and third servers will pickup the phones up seamlessly.
My thinking is, and I just started setting up my own last night, is to have a backup proxy. A second PIAF on the same network with the same settings as the first. Then point the "backup proxy" of all phones to that box.
Not sure if its gonna work, but I will give it a try. I will have to install then disable the trunk lines and some tweaks will be required I am sure, but this is the direction I am heading right now. I will let you know what I find. BTW, both of my PIAF servers are VM's and installed on different servers.
Good luck in your qwest. I am sure you will find your way.

==oldciscodude

 

[OTA]

I'm curious about your apprehension.

Practically 99% of the companies out there have NO backup phone system nor plans in place to handle such a failure. I'm willing to bet your company doesn't currently have one either.

Most of them (falsely) believe they'll get ahold of their local telephone equipment dealer and they'll just magically come out, swap a part, and they'll be back up & running again in an hour. What are the chances that the local dealer has the part in stock?

If your Merlin were to die right now, how long would it take for you to obtain parts and get it back up and running? How long would it take for you to reload/reprogram the system configuration? Scary thought, isn't it?

The key to a reliable system is the hardware. If you're looking for triple or five 9's, skip consumer-grade hardware. I know the $99-$300 PCs are a fav. for many PiaF installs, but communications are too critical for most of my clients to risk it.

Look for enterprise-grade equipment w/redundancy. Enterprise servers fit the bill perfectly. Even old ones. These usually will come with dual power supplies, hardware RAID, redundant cooling, and plenty of monitoring/remote control options. And they're dirt cheap on the used market -- cheap enough to buy multiple units. I've been running various computers over the years and have standardized on Dell Poweredge 2650s. $200 or so on the used market for an absolute beast of a server for this sort of thing. I still haven't been able to hit 1% CPU usage on them with a typical Asterisk setup. I'm slowly moving all of my systems over to PiaF from various flavors/distros of Asterisk.

My systems are expected to be up & running full-throttle during the middle of Cat 4 hurricanes. In our most recent Cat 4 hurricane, my Asterisk systems stayed up while the the local phone company's POTS & PRI service crashed & burned city-wide. PRI service was out for up to 10 days in some areas (without physical damage), POTS was out for 2-3 weeks in some areas.

Since I use SIP trunks from an out-of-state provider, I was completely unaffected by Embarq/Sprint's screwups. Also, because my trunks aren't located in Florida, I never had to deal with "all circuits are busy" when the local telco runs out of trunks when everyone's frantically calling each other before/during/after the storm. As long as we had power & internet, it was business as usual. Business & enterprise-grade internet connections are surprisingly resilient. At a few locations we lost power, water, POTS lines, cable TV, but internet somehow stayed up the whole time. Even in some residential areas where cable TV was dead, cable internet still worked. As long as the lines weren't under water, DSL seemed to be stable as well.

Regarding backups, I don't use Mondo. Just the FreePBX backup module and SFTP backups of certain directories when I make changes to the system. As long as you have the .conf files, it's quite a portable system, hardware-agnostic. I think every single one of my Asterisk systems and ALL of my PiaF systems have been transplanted from one PC to another via config files.

My workbench systems are P3 and P4 computers, single IDE HDDs. These are where all of my new installs are built/configured. My production systems are all currently Dell Poweredge systems, dual Xeons, PERC RAID5 on Ultra320 SCSI drives. I've even built a few systems on my Macbook using a VM then transferred the conf files and critical directories over to a production system. Probably takes me 5-10 mins overall. If you're going to a warm spare, the switchover can happen in seconds.

I have 1 warm spare PiaF system in the rack. Default blank config but kept up-to-date with modules & such. If one of my clients' PBXs were to die, I'd just upload the correct config to it, change my DNS server settings and they're up & running in under 10 minutes. And the best part -- I don't even have to set foot on-site. If I was really desperate, I could pull the whole thing off using MidSSH on my Blackberry. Wouldn't be fun, but it is possible. Show me ANY commercial PBX from the big boys that can do that.

I'll echo Joe's comment: I also don't mess with production servers. Once they're running, I make backups of everything and I leave them alone. If I want to experiment I'll transfer the config back over to a workbench or VM system and experiment on there.

I've played around with VMs and they can work, but I'd be hesitant to use it with PRIs or other interfaces that require hardware.

Just curious, why stay with PRIs? SIP trunking's far less expensive and far more flexible, not to mention they don't require special hardware. In my case, SIP trunking means entire data centers can get wiped off the map and warm spares in another state or anywhere in the world can take over. It's just a DNS server entry change for me. Depending on your provider, you don't pay for SIP channels you're not actively using. One of my clients went from $900/mo down to $180/mo by switching from a T1 to SIP.

I'll also echo Kenn's comments as well. If this is your first Asterisk system, you might want to consider a consultant to get things up & running and for guru meditation down the road. I just spent the past two weeks of town cleaning up the mess left when a local ISP decided they were going to sell telephone service and Hosted PBX service. They didn't understand why people were so upset when things didn't work 100% of the time. They also didn't seem to understand that dropped/missed calls and lost voicemails can cost a company dearly. Typical ISP mentality. "Just hit reload" when a website doesn't load the first time. Doesn't quite work for voice. Dropped packets here DO matter. Also didn't help that they didn't understand the equipment they were using.

You said you have 13 locations... ever consider making just a couple of PBXs and running the other locations as remote extensions? On most of my systems about 25-50% of the extensions are off-site, many in different states across the country. I even keep my home phone #/lines running on my most critical client's system, some 200 miles away. No PBX at home, just a pile of Cisco phones and an ATA for the cordless phones.

Here's what I'd do: Pick up a few used phones off eBay and give it a go. I prefer the Cisco 79x0 phones (Sorry Ward, the rubber buttons, non-adjustable base, and weak speakerphone on the Aastras keep them out of my offices). Get a cheap SIP trunk to play with. Learn the system, break the system, learn how to put the wheels back on once they've fallen off. Then roll it out. Since you have 13 locations, maybe try finding the least critical one, or the smallest one and work from there. Don't have time to learn it that deeply? Hire a consultant to get it going and learn it as you go.

 

[jroper]

Hi

Take heed of this advice, there is little I can disagree with here, it's all very good advice, and puts across what I was trying to say in a much more structured fashion.

The very very minor points to disagree with are that there is a steep learning curve with anything that has Cisco written on it, but once mastered, apparently very good, and Server hardware can be noisy, so it may not be suitable for a location where it cannot be hidden away behind closed doors.

Joe

 

[OTA]

Joe,
Definitely valid points. I did forget to mention how loud the Dell Poweredge rackmount servers are -- and they are.
Very valid point. I wouldn't want to have one in my office cranking away behind me. In my installs they've been relegated to server rooms when possible, or file/copy rooms in offices which don't have dedicated server rooms. That said, they're nowhere near as bad as the equivalent HP rack servers. The Dells just have a loud air-noise sound to them while the HP fans have a high pitch whine that'll drive me out of the room.

I think the Cisco phones get a bad rap for the flat-out lack of documentation Cisco's provided and Cisco's short-sided lack of support for anything other than CallManager. That said, I still think the 7940/7960 are still simple enough that most new users can modify the cnf files included w/PiaF to get them up and running. Also can't beat the price on them. I've been snagging these for $50-60 off eBay for a 7960. There's nothing else out there in this price range (even new) that has the same quality to it.

Virtually all the other Cisco phones are a totally different story and I'd advise anyone who hasn't messed around with Cisco & Asterisk for awhile to avoid them. I like them, but what a PITA to get going. And anything that runs IOS is definitely not for beginners.

I have a love-hate with Polycom phones. Ugly, but the sound quality is tremendous. They're simple enough to configure from the keypad/web interface for basic things, but a steep learning curve to get them working to their fullest via remote provisioning. Not something I'd recommend for a beginner.

Ultimately... I've not used a phone system that I like as much as Asterisk and I haven't found a distro. better than PBX in a Flash. I don't know of another system out there which is as portable, will run on such a wide variety of hardware, and can be made as disaster-proof as Asterisk. I even had a hard drive die on a single-drive system and it kept on running the best that it could.