PIONEERS 5-Minute HA for Incredible PBX

Discussion in 'Developers' Corner' started by lbergey, Jul 7, 2017.

  1. lbergey

    lbergey Member

    Joined:
    Nov 25, 2011
    Messages:
    83
    Likes Received:
    17
    I have just completed a 5 minute 2 question HA. Right now I only have tested it on Ubuntu 14, but I think it will work with little or no tweaks on a Raspberry Pi 3.

    Requirement:
    • You need two PBXs and they MUST be running the same OS & Version. The backup can be a fresh install.
    • You MUST run this as root and you need to allow root ssh access (you may need to update /etc/ssh/sshd_config)
    • This setup DOES NOT work with a cloud setup, as the IP address can't failover.
    What it does:
    • It installs keepalived and lsyncd and a couple other utilities.
      • keepalived - is the ip failover, the install configures both servers for the failover
      • lsyncd - is the mirror for voice mail and other config items, this is set up in a master - backup configuration. BUT it switches when the master switches, that keeps your voicemail in sync.
    • does a full backup of the existing machine and copies it to the backup server
    • sets up mysql as a MASTER - MASTER mirror
    • stops asterisk on the backup machine, that way you don't have a sip registration to a backup server.
    • starts asterisk up and does a reload (Apply Settings) when it becomes the Master.
    Please run this in a test environment first! I have tested with voice mail, adding extensions, and failed over & back.

    To install:

    Code:
    wget https://app.qmod.com/ha-pbx.sh
    chmod 777 ha-pbx.sh
    ./ha-pbx.sh
    Everything is done from the original computer!

    You can watch me run the install here:
     
    #1 lbergey, Jul 7, 2017
    Last edited: Jul 19, 2017
    lionheart, krzykat, Jake and 2 others like this.
  2. lbergey

    lbergey Member

    Joined:
    Nov 25, 2011
    Messages:
    83
    Likes Received:
    17
    I just added a Video Link so you can watch the full process!
     
    wardmundy likes this.
  3. lbergey

    lbergey Member

    Joined:
    Nov 25, 2011
    Messages:
    83
    Likes Received:
    17
    Anybody who tries this, please leave feedback with success or failure.

    Also, what OS would users like to see this ported to
     
  4. wardmundy

    wardmundy Nerd Uno

    Joined:
    Oct 12, 2007
    Messages:
    14,247
    Likes Received:
    2,414
    @lbergey Could we get around the cloud server limitation by bringing up both machines as NeoRouter VPN clients which would create 10.0.0.x addresses for both servers? That would be huge because the cloud servers could then be in different locations.
     
    Twilight Sparkle likes this.
  5. lbergey

    lbergey Member

    Joined:
    Nov 25, 2011
    Messages:
    83
    Likes Received:
    17
    I have been thinking about that for a while, you can't have a floating IP but I was thinking about a dynamic DNS entry. It would require a third witness server to verify a server is down.
     
  6. wardmundy

    wardmundy Nerd Uno

    Joined:
    Oct 12, 2007
    Messages:
    14,247
    Likes Received:
    2,414
    With NeoRouter, all of the client IP addresses are static. What would need to run of the third server? Is it a full-blown Asterisk server or just a script that could be run periodically from any Linux box?
     
  7. lbergey

    lbergey Member

    Joined:
    Nov 25, 2011
    Messages:
    83
    Likes Received:
    17
    The issue is what do the phones register to. In a normal HA they would register to the floating IP. In a cloud server you can't float an IP but you need to register to something that can float.

    I have two cloud severs running Neorouter monitoring all of my PBXs as well as scripts running on the PBXs monitoring themselves. Most weeks I have times where one of the monitors cannot see the other.

    In the above example, without a witness server the second would come online and you would have fighting for the DNS.

    By only having the NeoRouter as the link, any brief outage of the VPN would cause a failover and both would then be fighting for the DNS and you would have both servers registering for the SIP trunks as well. It could get ugly in a hurry.

    With a witness server (small Linux box, and it can be a third asterisk server) who ever has two votes becomes the master and controls the DNS. It actually doesn't need to run the script unless the Backup can't ping the Master PBX. In that case the Backup would ask the witness if the Master is down and if the witness agrees that the Master is down, it would then take over the DNS. The only way either one can take over the DNS is if the witness agrees.



    It really is the only safe way to go.
     
    #7 lbergey, Jul 8, 2017
    Last edited: Jul 8, 2017
  8. tbrummell

    tbrummell Guru

    Joined:
    Jan 8, 2011
    Messages:
    504
    Likes Received:
    31
    That sir, is awesome! HA for FreePBX, that doesn't cost an arm and a leg. Nice!
     
    wardmundy likes this.
  9. tbrummell

    tbrummell Guru

    Joined:
    Jan 8, 2011
    Messages:
    504
    Likes Received:
    31
    CentOS would be the next logical OS since you have it working on Ubuntu already.
     
    krzykat likes this.
  10. kyle95wm

    kyle95wm Phone Genius Owner

    Joined:
    Apr 16, 2016
    Messages:
    426
    Likes Received:
    81
    Would this work on something like DigitalOcean with Floating IPs?
     
    wardmundy likes this.
  11. lbergey

    lbergey Member

    Joined:
    Nov 25, 2011
    Messages:
    83
    Likes Received:
    17
    I was just reading up on how DigitalOceans floating IP works, and it should work with a little modifications. Things to remember, as of now it would require a Ubuntu install, and that the floating IP requires that both servers be in the same Data center.

    I don't have a DigitalOceans account, but I would be willing to work with someone to make it work.
     
  12. krzykat

    krzykat Guru

    Joined:
    Aug 2, 2008
    Messages:
    1,314
    Likes Received:
    314
    DNSmadeEasy has a failover for DNS that I think would work pretty nicely. Then all you need to do is have your phone register to sip.domain.com and DNSMadeEasy would do the appropriate name failover for you. They also support dynamic DNS.
     
  13. lbergey

    lbergey Member

    Joined:
    Nov 25, 2011
    Messages:
    83
    Likes Received:
    17
    Yes, I like DNSmadeeasy and I use them for some of my other fail over scripts. You need to remember to set your TTL low and your SIP registration on your phones must be low as well (that applies for either IP or DNS based)
     
  14. krzykat

    krzykat Guru

    Joined:
    Aug 2, 2008
    Messages:
    1,314
    Likes Received:
    314
    Correct, I always set my phones for registration of 1 minute to avoid other usually NAT related issues and potential phone firmware problems. This means that using your failover in conjunction with DNSmadeEasy that you'd have HA of max downtime of 60 seconds. A very powerful offering IMHO. I plan to utilize Vultr for primary, and then my own hosted server (prox right now - but not married to it) for the secondary.
     
  15. lbergey

    lbergey Member

    Joined:
    Nov 25, 2011
    Messages:
    83
    Likes Received:
    17
    I am traveling with spotty internet, so I can't help too much, but I have DNSMADEEASY scripts at home.

    Also I did notice that I am missing /var/lib/asterisk in /etc/keepalived/master.sh as a third lsyncd. I should be able to fix that later this week. But you can just copy one of the other lsyncd lines.

    Let me know if you need any help or have any questions.
     
  16. krzykat

    krzykat Guru

    Joined:
    Aug 2, 2008
    Messages:
    1,314
    Likes Received:
    314
    I'll gladly wait for you to return later this week and will be glad to test it out.
     
    wardmundy likes this.
  17. kyle95wm

    kyle95wm Phone Genius Owner

    Joined:
    Apr 16, 2016
    Messages:
    426
    Likes Received:
    81
    I have one and I'm willing to test. You should probably get one though for your own testing.
     
  18. lbergey

    lbergey Member

    Joined:
    Nov 25, 2011
    Messages:
    83
    Likes Received:
    17
    FYI, I updated the install script that should now work for Centos installs as well. I am working on the cloud solution, the initial solution will require access to a witness server. The witness server will ONLY be used when the backup cannot find the master. At that point the backup asks the witness to verify if the master is up or down. If the witness verifies that the master is down, it will update the dynamic DNS and take over the control.
     
  19. kyle95wm

    kyle95wm Phone Genius Owner

    Joined:
    Apr 16, 2016
    Messages:
    426
    Likes Received:
    81
    Sooooo what you're saying is you can use something like noip or something like that as sort of a "floating IP" type solution.
     
  20. lbergey

    lbergey Member

    Joined:
    Nov 25, 2011
    Messages:
    83
    Likes Received:
    17
    Yes, I already have update scripts for DnsMadeEasy & DuckDNS.org I can add NoIP later.

    You would need to register your phones to that FQDN instead of the IP. The FQDN will float between the servers as needed.

    While DigitalOceans floating IP is great, your backup server must be in the same data center. If you have issues with connecting to that data center both of your servers will be down. If you use a DynamicDNS and a FullyQualifiedDomainName you can have a backup in another part of the country or even world and the backup should work.
     
    krzykat likes this.

Share This Page