TIPS Have I been hacked - Fail2Ban issue?

[awair]

Following on from my earlier report with Fail2Ban issues, I am having increasing difficulties with my remote PIAF:
http://pbxinaflash.com/community/index.php?threads/fail2ban-not-running.16322/

As previously mentioned, I have two identical (hardware & OS Install) PBXes. They were set-up together and tested before one was deployed, and the other shelved for future, remote use. This was some time between Jan-Mar 2014.

The (now) deployed remote is PIAF 3.6.5, Green, Asterisk 11.7.0 & FreePBX 2.11.0.42. Both PBXes sit behind hardware firewalls, with no ports exposed.

The only difference in setup was that I may have enabled GV on the remote machine.

After deployment, the remote machine was patched for the various security issues that arose during the last year, resulting in the reported irregular behaviour of Fail2Ban. After this was apparently resolved, additional problems have surfaced.

Current symptoms:
I received a report from Google that it had blocked a 'suspicious login' on my GV account. The reported IP was actually from my own PBX's IP address, and had been up & running for two weeks.

With Fail2Ban running, CPU usage hovers around 100%, with increasing memory use by both Fail2Ban & Asterisk (700 MB real plus 300 MB virtual): the second (local) machine hovers around 200-300 MB of 1GB.

With Fail2Ban disabled, CPU reduces to 10% or less.

I have disabled GV, (and run the patch), as there may have been an additional conflict.

Since disabling Anonymous SIP/Guest, I have had no strange calls logged, and no unusual billing activity.

I have strong/unique passwords, and the system is accessed via VPN for admin.

Do I have a security issue, or is this a random unfortunate set of glitches caused by applying 'too many updates', or similar?

The remote system replaced an ageing Trixbox, which did not have all the security features of PIAF. After 6-7 years the hardware was due for renewal before it failed. The local & remote systems are connected via IAX over VPN.

Where else should I look? Is there a way to 'cleanse' the system, or initiate a remote re-install that zaps everything? I will not have physical access for some time. The system is headless, so no intervention is possible by helping hands.

Many thanks,

 

[rjm]

I had this before. I just started over. You can try a Digital Ocean instance for like $5 a month: https://www.digitalocean.com/?refcode=56965679b60d and if you use this link I get some credit on my account. Which would be cool. I think they have a deal going for $10 in free usage so that will give you two months free. Then you can check out Ward's amazing new tutorial http://nerdvittles.com/?p=11766 on how to install the new Gotcha Free Incredible PBX, the next toy on my "try it" list. Sometimes things get klugy and you just have to start over.

 

[hecatae]

question,

why have you not updated asterisk?

Your version of asterisk is currently vulnerable to the majority of the following from here: http://www.asterisk.org/downloads/security-advisories

AST-2014-019: Remote Crash Vulnerability in WebSocket Server
Dec 10, 2014
http://downloads.asterisk.org/pub/security/AST-2014-019.pdf
 
AST-2014-018: AMI permission escalation through DB dialplan function
Nov 20, 2014
http://downloads.asterisk.org/pub/security/AST-2014-018.pdf
 
AST-2014-017: Permission escalation through ConfBridge actions/dialplan functions
Nov 20, 2014
http://downloads.asterisk.org/pub/security/AST-2014-017.pdf
 
AST-2014-016: Remote crash vulnerability in PJSIP channel driver
Nov 20, 2014
http://downloads.asterisk.org/pub/security/AST-2014-016.pdf
 
AST-2014-015: Remote crash vulnerability in PJSIP channel driver
Nov 20, 2014
http://downloads.asterisk.org/pub/security/AST-2014-015.pdf
 
AST-2014-014: High call load may result in hung channels in ConfBridge
Nov 20, 2014
http://downloads.asterisk.org/pub/security/AST-2014-014.pdf
 
AST-2014-013: PJSIP ACLs are not loaded on startup
Nov 20, 2014
http://downloads.asterisk.org/pub/security/AST-2014-013.pdf
 
AST-2014-012: Mixed IP address families in access control lists may permit unwanted traffic.
Nov 20, 2014
http://downloads.asterisk.org/pub/security/AST-2014-012.pdf
 
AST-2014-011: Asterisk Susceptibility to POODLE Vulnerability
Oct 20, 2014
http://downloads.asterisk.org/pub/security/AST-2014-011.pdf
 
AST-2014-010: Remote Crash when Handling Out of Call Message in Certain Dialplan Configurations
Sep 18, 2014
http://downloads.asterisk.org/pub/security/AST-2014-010.pdf
 
AST-2014-009: Remote Crash Based on Malformed SIP Subscription Requests in PJSIP Channel Driver
Sep 18, 2014
http://downloads.asterisk.org/pub/security/AST-2014-009.pdf
 
AST-2014-008: Denial of Service in PJSIP Channel Driver Subscriptions
Jun 12, 2014
http://downloads.asterisk.org/pub/security/AST-2014-008.pdf
 
AST-2014-007: Exhaustion of Allowed Concurrent HTTP Connections
Jun 12, 2014
http://downloads.asterisk.org/pub/security/AST-2014-007.pdf
 
AST-2014-006: Asterisk Manager User Unauthorized Shell Access
Jun 12, 2014
http://downloads.asterisk.org/pub/security/AST-2014-006.pdf
 
AST-2014-005: Remote Crash in PJSIP Channel Driver's Publish/Subscribe Framework
Jun 12, 2014
http://downloads.asterisk.org/pub/security/AST-2014-005.pdf
 
AST-2014-004: Remote Crash Vulnerability in PJSIP Channel Driver Subscription Handling
Mar 10, 2014
http://downloads.asterisk.org/pub/security/AST-2014-004.pdf
 
AST-2014-003: Remote Crash Vulnerability in PJSIP channel driver
Mar 10, 2014
http://downloads.asterisk.org/pub/security/AST-2014-003.pdf
 
AST-2014-002: Denial of Service Through File Descriptor Exhaustion with chan_sip Session-Timers
Mar 10, 2014
http://downloads.asterisk.org/pub/security/AST-2014-002.pdf
 
AST-2014-001: Stack Overflow in HTTP Processing of Cookie Headers
Mar 10, 2014
http://downloads.asterisk.org/pub/security/AST-2014-001.pdf

 

[awair]

Now I am confused...

I know that PIAF sits on top of FreePBX, on top of Asterisk, on top of Linux. Incredible PBX does it's own automatic updates (so needs no intervention?), but thought PIAF 'only' needed update-fixes?
http://pbxinaflash.com/community/index.php?threads/updating-asterisk.5146/

I'm sure I'm wrong, but the documentation has now got to the level where there is no clear path to follow. It used to be that if you wanted a reliable PBX, you would follow ward's post on NV for setting up Trixbox or PIAF, and maintain regularly. It now seems that the answer is to install a newer later version. This just isn't feasible on production machines, especially in remote locations.

wardmundy - your documentation, both on NV and the forums is amazing, but it is now so prolific that unless you're a full-time admin, it is likely that it is impossible to 'connect the dots' of which articles to follow, and in which order.

Going back to basics, after installing a new 'PIAF' and running update-fixes, what else needs to be done on a regular basis (excluding ShellShock/Ghostbug type vulnerabilities)?

After recently updating modules on FreePBX and ending up with an unstable system, I am reluctant to mess with a working system unless there are security issues.

hecatae - thanks for the heads up.
rjm - thanks for the suggestion, but due to circumstances, I need to keep my physical host at moment.

Another reluctance, probably historical now, was that advice tended to be 'don't update' every module because of intricacies of the (earlier?) PIAF model, that a general (Linux) update might break the PBX due to unforeseen interdependencies.

That problem seems to be eliminated with Incredible for Asterisk-Gui, but those of us running legacy installations need time to migrate.

 

[wardmundy]

It's definitely a transition period for many of us. Ride the horse you're on until the dust settles. update-fixes still works fine to keep PIAF installations current so long as you have a firewall in place. Personally, I'd also upgrade Asterisk from time to time and there's a script on the forum to do that for you if you're on the Asterisk 11 platform.

 

[awair]

Thanks Ward, so I don't have to worry about the comment above:

question,

why have you not updated asterisk?

Or did I miss a step from my routine maintenance?

 

[wardmundy]

Asterisk 11 upgrade script here.

 

[awair]

Thanks again Ward:

I have now identified a 4th separate issue with this system.

Assuming the underlying Linux installation is sound, do you have a script to remove PIAF completely, so I can install again remotely?

I need to retain SSH capability, log-in & delete all unnecessary folders, update the OS and then reinstall (a flavour of) PIAF.

I do not have physical access to the machine, nor KVM access.

 

[wardmundy]

Sorry. Can't help on that one.

 

[awair]

Is that because no-one's ever wanted to uninstall PIAF ;-)

No problem, I have a cloned drive that I can send over.

 

[hecatae]

Thanks again Ward:

I have now identified a 4th separate issue with this system.

Assuming the underlying Linux installation is sound, do you have a script to remove PIAF completely, so I can install again remotely?

I need to retain SSH capability, log-in & delete all unnecessary folders, update the OS and then reinstall (a flavour of) PIAF.

I do not have physical access to the machine, nor KVM access.

how to install centos remotely using pxe:

http://www.sentris.net/billing/knowledgebase/25/How-to-re-install-Centos-yourself-remotely.html

worth a read, I use this when I have access to another machine to complete the reinstall using vnc.

 

[awair]

do you have a script to remove PIAF completely, so I can install again remotely?

wardmundy: This seems to be what I'm after:

cd /root
wget http://incrediblepbx.com/incrediblepbx11.gz
gunzip incrediblepbx11.gz
chmod +x incrediblepbx11
./incrediblepbx11

It produces the following warning:

WARNING: This install will erase ALL existing FreePBX configurations!

I couldn't (readily) find the command for a 'simple' PIAF install - does that script have the same warning?

Supplementary question: what else gets zapped? (e.g.: Does this also replace Asterisk or only FreePBX; any changes to iptables; what about additional installed items [OpenVPN, Webmin etc]?).

I'm playing on a new RentPBX VM, which failed to install 3.6.5/Green (due to lost connectivity). So until their office is open for a reset, I thought I'd check out other options.

 

[hecatae]

awair http://pbxinaflash.com/community/index.php?resources/piaf3-install.38/

 

[awair]

awair http://pbxinaflash.com/community/index.php?resources/piaf3-install.38/

Thank you - that was the post I was looking for! Hadn't seen it today, but knew it was there somewhere...

WARNING: This install will erase ALL existing configurations on your server!