My Server Has Been Hacked

[entrtnm8]

My PBX in a Flash has been hacked (posting here at the request of Ward). Over 2200 calls from 1/21/09 to 1/22/09, and I thought I was safe. I am not computer guru, just a regular guy that likes tech. I don’t make a living with PBX systems and I don’t visit the forums regularly unless I am trying to figure out how to fix something, so please let me know what I did wrong. I set up my small home office PBX (3 extensions) and have regularly run update-scripts, update-fixes, (never ran update-source) and updated the FreePBX modules as I was notified. All was good until I received a call from a person in the 337 area code saying that they saw my number on their caller ID. I didn’t call, so I checked my call records and WOW (lots of calls to the 337 area code)! Shutdown the PBX server and trying to figure out what happened?
What I Have:
Orgasmatron I with the Walmart special (installed following all www.nerdvittles.com instructions)
2 VOIP providers (Viatalk and Vitelity)
Linksys WRT 150N router running DDWRT (SPI FIREWALL)
Block Anonymous WAN Requests (ping)
Filter Multicast
Filter IDENT (Port 113)
Dyndns
Ports forwarded to my PBX IP address (192.168.x.xxx
4569 UDP
5004 - 5037 UDP
5038 – 5082 UDP
10,000 – 20,000

My 3 extensions don’t have passwords that match the extension numbers and I did run passwd-master when I first set up the system. I think my “hole” was that I left extension 501 (cellphone) untouced from the original ISO Orgasmatron base install. I had plans to incorporate into my dialplan, however, I never did.
I have now downloaded Orgasmatron II and understand that it is running a newer version of Asterisk and Cent OS and plan on a fresh install. Is this version more secure?

It would really be helpful if someone could explain how this hack occurred? Since all of the outgoing calls were originated from the extension 501 (cellphone) that had the default password, exactly what happened? And more importantly what else can I do to help prevent further attacks? (yes I will not be leaving extension 501 unchanged on my fresh install)

Thanks for you help

 

[jbh]

What rotten luck

It didn't occur to me before, but if the passwords for the pre-configured extensions on Orgasmatron/VPNinaflash are the same for everyone then this would presumably give a way in to an evil-doer. A warning to us all. Thanks for the warning.

Hopefully making sure the passwords are changed and then doing Ward's fail2ban update should let you sleep easy. (With the caveat that I'm a newbie so don't have a huge amount of experience myself)

 

[wardmundy]

The Answer Is...

Q: "What could be done differently?"

And the answer is...

CHANGE YOUR EXTENSION PASSWORDS TO SOMETHING SECURE!

READ THE DOCUMENTATION WHICH SAYS CHANGE YOUR EXTENSION PASSWORDS TO SOMETHING SECURE.

READ THE RSS FEED ON THE MAIN WEB PAGE OF YOUR BUILD WHICH SAYS CHANGE YOUR EXTENSION PASSWORDS TO SOMETHING SECURE.


Yes. I'm SHOUTING! Come on guys. If someone has your extension number and extension password and the SIP port is open to receive calls to your server, anyone can make free calls using your server. Read Nerd Vittles regularly or visit the forum regularly or RTFM... any one of them... and this will never happen to you.

 

[entrtnm8]

Ward, the initial build of Orgasmatron uses PBX in a Flash release 1.2 and it doesn't have an RSS feed. All of the update-fixes and update-scripts didn't add RSS feeds to my build. Therefore I didn't get the feeds about changing passwords of extensions.

I do realize that passwords are an important part of security (as I said in my initial post I am not a computer guru, nor am I a complete idiot) I have re-read my copy of the original installation instructions and I did follow all of your instructions for changing passwords (for root, maint, wwwadmin, meetme, and webmin) I later used Joe's script passwd-master.

I was unaware that someone could enter the PBX and bypass all of those passwords and dial out by guessing (?) and extension and extension password.

Perhaps those update-scripts and update-fixes could have patched me to PBX 1.3 and I would have had the an RSS feed to warn me?

I am not looking to blame anyone (other then myself) for not fully grasping the risks of a internet based PBX. I am looking for an understanding as to how someone was able to enter my PBX (bypassing all passwords and connect to an extension and dial out).

As the extensions passwords are limited to numbers only, how secure can they really be against a brute force attack?

No, I am not YELLING. I am listening very carefully so that I can learn, and perhaps help others in the future.

Thanks

 

[jbh]

fail2ban will protect you against a brute force attack - because after 5 (I think) tries, the IP address the attacker is, er, attacking you from will be banned for 30 minutes.

 

[sanitycheck]

I was unaware that someone could enter the PBX and bypass all of those passwords and dial out by guessing (?) and extension and extension password.

If you don't have remote SIP extensions (extensions crossing the Internet), you might not have to forward any ports at all. I, too, use a dd-wrt-equipped router (v2.4-sp1) and don't need to forward any ports.

This is true even though I use Voicepulse trunks for outbound calling (only). Some routers do need some port forwarding to get 2-way audio to work, but mine does not.

Perhaps those update-scripts and update-fixes could have patched me to PBX 1.3 and I would have had the an RSS feed to warn me?

Forum references to the RSS feed I thought were to the main Nerdvittles feed, which I monitor. If my Orgasmatron II system has an RSS feed, I don't know how to connect to it. Time for me to start looking for it. A generic hot-topic or security RSS feed for all builds might be good.

As the extensions passwords are limited to numbers only, how secure can they really be against a brute force attack?

The extensions are numbers only, but the passwords (called secrets under FreePBX extension section ) do not have that limitation.

Longer extension passwords/secrets made of random characters + fail2ban can go a long way to prevent someone outside from cracking your system. Again, this assumes you do actually need to have SIP open to the Internet in the first place.

Fail2ban has given me confidence to try non-VPN remote extensions in the near future.

As you mentioned, the stock easy-to-guess extension password for 501 being left in place was your downfall. The passwords you made on the other extensions, even if they were all numeric (but not for long, I hope), were apparently good enough to keep the drive-by hackers out.

 

[wardmundy]

Sorry for yelling, but this is very frustrating at our end. The Kennonsoft UI update has been available on the forum for download for over 3 months. It includes our RSS Feed. We've got the best documentation in the business and the best forum in the industry, but if you never look...

How do we fix this? We could implement a push technology to fix things like this, but we have been hesitant to do so because of the Fonality scenario that included not only a phone home component but also the ability to adjust the script running on users' computers to do anything they chose to do... without any notice to those using the systems. While I have no first hand knowledge, we have been told that this is basically the design of their commercial systems and they merely carried it over to tb.

Some options that come to mind...

1. We could let folks opt into a push system that would keep your system secure and up to date.

2. We could let folks opt out of push system that would keep your system secure and up to date and continue with a pull system plus the RSS Feed.

3. We could drop support for older releases which become more and more difficult to secure as time goes by.

4. We could require everyone to provide a valid email address to use our software. That would give us one more way to contact users about potential problems.

Understand that looking for security vulnerabilities means that the probing would be examining your passwords if you want us to assist with your responsibility to secure your system.

We've had messages posted on the forums and in numerous Nerd Vittles articles for over 3 months warning about the dangers of having insecure passwords on extensions. It's also been at the top of the RSS Feed in the new UI for at least that long. Yet we still see a new post about this almost every week. We're all ears for suggestions. And I apologize for losing it.

P.S. Please read the Nerd Vittles article tomorrow on Asterisk security.

 

[marv]

A few random options

People don't always have time to read the forums or be proactive in looking for security alerts. Maybe there is some really nice smart guy that could setup a mailing list that could send out alerts that are critical or informative? Then people can opt-in or out for patch alerts and such?

Another option would develop a monitoring application which a person could set alarm thresholds for inbound/outbound calls processed?

I would be more creative, but I smell food cooking in there...

 

[sanitycheck]

The Kennonsoft UI update has been available on the forum for download for over 3 months. It includes the RSS Feed. We've got the best documentation in the business and the best forum in the industry, but if you never look...

...It's also been at the top of the RSS Feed in the new UI for at least that long.

As Entrtnm8 and Marv point out, many users who genuinely care about security do not regularly visit the forums or the websites.

The forums are a wealth of information, but I would guess a majority of PIAF users visit the forum only to look for a solution to a problem. Reviewing the forum entires is also no small task, and I believe that is another reason a lot of important information is missed by many users.

An RSS feed can be a solution to that problem, but the Kennonsoft UI Update highlights a shortcoming of the information system.

To get that RSS feed update, you have to install the Kennonsoft Update. To know about the Kennonsoft Update, you would have had to been reviewing the forum fairly regularly over the last few months (even then you might miss it or dismiss it). Once the Kennonsoft Update is installed, to view the RSS feed, you have to visit your PBX's webpage regularly. [WM: Huh? Our RSS Feed works with any feed reader including most browsers: http://pbxinaflash.com/rssfeed.xml]

I'm guessing a large number of PIAF users visit their PBX webpage as infrequently as they visit the forums, even if they have the Kennonsoft update installed. If the system is working, the occasional update-scripts and update-fixes is probably the best you can expect.

I fall into that category. I genuinely care about security, but I don't often have the time to sort through all the forum entries for the key bits I might need. PIAF is high on my priority list of projects to monitor, but I have many other projects to monitor, too.

A few ideas (some mentioned by other posters earlier):

RSS FEED: A normal RSS feed limited to security (e.g fail2ban updates, fail2ban update-fixes installer bug) and program announcements (e.g. Kennonsoft Update) would be great. Short posts for a quick read with links to the details (like the associated forum post) would be best. I monitor the Nerdvittles feed in my list of feeds in Firefox already (lots of reading usually, but infrequent enough so I can keep up). I'm much more likely to catch updates to a regular RSS feed in Firefox than having to log into my PBX's main page.

MAILING LIST: A mailing list with the same info as the feed above might also be a good idea. A mailing list is more old-school, but I'd be the first on the subscriber list if you made it available. Hit me with an RSS feed and a mailing list, and I'm bound to pay attention. I've been on the GNU Motion project mailing list for years; I've found it to be an efficient way for me to keep up to date with that great project. Motion's mailing list is for announcements and user troubleshooting because they don't have a forum.

HELP-PBX COMMAND LINE NEWS READER: How about a help-pbx menu option (e.g. show-news) to pull the same information suggested for the RSS feed and mailing list above? Maybe it would be a command line RSS reader hard-linked to the RSS feed suggested above. Or, maybe it just downloads and displays a regularly updated text file with abbreviated versions of what would appear in the RSS feed or mailing list. An option like this would be useful for those people who work most often at the command prompt. I'd think security- conscious people (like me) SSH into their PBX to install updates much more frequently than accessing the PBX's main page or FreePBX to install updates. If you're already there to update-scripts and update-fixes, you might as well read the news.

PIAF is a great project. I look forward to monitoring it more closely, even if that means taking more time looking through the forum. Over the last few days I've been surprised and disappointed to learn how much I was missing.

 

[wardmundy]

Thanks for the suggestions. Keep them coming.

Here's our bottom line...

If you don't have time for Asterisk security (see today's article), either hire someone to manage your PBX or switch to Vonage and shut down your PBX.

You're not doing yourself or anyone else in the world a favor by providing an unsecured platform that can be used to create all sorts of mischief when handed over to some of the really bad creeps roaming the planet. You don't leave your wallet on a public lunch counter. So don't do essentially the same thing with your PBX.

 

[jroper]

Hi

To clear up one point, Tom is the Guru behind passwd-master.

The point about all of this, is that your PBX is simply another network application, and should be treated as such. The main issue is that being hacked is not just an inconvenience, it can be expensive.

The tools that are provided with the PBX are just the the basics or a starting point for security - there is shedloads more work to do to tighten it all down.

Looking at the firewall rules in Webmin shows what ports are open on the PBX - and you need to close them down one by one, either using the linux firewall, or your external firewall.

Lets look at each one in turn: -

1. SSH - ports 22 and 9022 - do you need access from the outside world - can you use an existing VPN solution to get access to the network, if not, you can move the entry points to another port - but obscurity is not security, so get a copy of puttygen from here - http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html and disallow username password access via webmin, and only enter using your public / private key. Therefore no SSH passwords to hack.

2 80 and 9080 - do you need web access to the server from outside your building - did you know with Putty, you can use SSH tunnelling, and pass the traffic through that. so they can be closed off.

3. 443 -thats SSL, it's only in there for my A2Billing apps so I can use Paypal - remove it, unless you are using https.

4. 9001 - that's webmin - again you can use an SSH tunnel - and you can switch the service off - service webmin stop. Close the port.

5. 4569 - the IAX port - I open it sometimes for test calls, but close it afterwards - you do not need this port open unless some external IAX device or asterisk server is going to access the system. If the other end is on a fixed IP, you can use the deny=0.0.0.0/0.0.0.0 which denies everything from the outside, and permit 123.123.123.123/255.255.255.255 to only allow from a particlular IP address where 123... is your IP address. this is done in the trunk or extension definition. You might as well close down the firewall as well to all places but your other IAX device. If your server is sending traffic out to a IAX carrier, like ours, then you DO NOT NEED TO OPEN ANY ports. - The act of registration keeps the NAT device and firewall open to that IP for inbound calls.

6. 5000-5082 and 10,000 to 20,000 - the SIP ports - you are ging to have to open these ports if you are using a SIP carrier - so use permit and deny on all your sip extensions and trunks, use fail2ban, use secure passwords, and in the external firewall, only allow traffic from your carrier. If you need external extensions, consider using a VPN or Hamachi tunnel - or putting another PBX at the other end and do IAX trunking. Think carefully about allowig remote extensions.

7. 4445 - FOP - do you need access to FOP from the outside?

8. 123 - Timeserver - do you need access to this from the outside?

9. 69 - t*f*t*p - do you need access from the outside?



These are all the entry points there are.

Do an audit on these, and you should be good to go, and relatively safe.


Joe

 

[sanitycheck]

[WM: Huh? Our RSS Feed works with any feed reader including most browsers: http://pbxinaflash.com/rssfeed.xml]

Thank you for that link!! Where is it published on your website!?

I've monitored the Nerdvittles.com feed for at least 6 months, probably longer. The PIAF site lists a feed http://pbxinaflash.net/rssfeed.xml (lower right corner), but that one has only one entry and does not seem to get updated ever. Is the wrong address listed?

Please consider posting that feed link prominently on the main PIAF page. You may want to describe what the feed will contain, highlighting the point that hot security topics will be featured.

Please consider including feed entries about program additions or updates such as the Kennonsoft UI update (maybe I'm late to the party, and that entry got pushed out already).

If you don't have time for Asterisk security (see today's article), either hire someone to manage your PBX or switch to Vonage and shut down your PBX.

I agree completely, but having an efficient means of getting the word out to users will go a long way to help people help themselves.

As you pointed out, Nerdvittles and PIAF is one of the best-documented free projects out there. I would respectfully argue, however, that the large volume of information can work against the end user if you simply want to keep up-to-date. It also helps when the information comes looking for you (unlike the forum).

That's where a hot-topic RSS feed and mailing list can help. If an end-user can't be bothered to monitor an RSS feed or messages from a mailing list, as you said, PIAF is not for them.

 

[awair]

Could you confirm that the vulnerability relates only to the Orgasmatron builds [ie with extensions ready to use]?

I have just set up a PIAF server, which will be linked to 2nd remote site. However my plan is to configure this manually.

If this assumption is correct, a possible fix would be for an additonal stage in the scripts to force users to change extension passwords during setup [or when they run password-master].

As a newbie to PIAF, I find the volume and quality of the information extremely impressive but daunting. Despite trying to keep up with sources of security information, this is the first time I realised the possibility of extension 'masquerading'.

I have been running a remote TB [with the assistance of NV documentation] for two years without a security problem, but I can't claim to have read or considered everything.

I do appreciate that using public free wi-fi can expose the system to password sniffing, but a vpn connection is not always possible or practical. On this note, has anyone been able to succesfully setup a Nokia E65 or E51 to use a VPN access point to connect to any kind of service? [I believe it is meant to be possible, and this could close my main vulnerability].

Many thanks for PIAF and the great information.

 

[jeremywillden]

One more suggestion that might help the Orgasmatron builds (and anyone setting up their system):

On all extensions, set NAT to NEVER on the extension configuration page in FreePBX. This will (in my experience) prevent that extension from being used from outside the local network, with or without the correct password.

The default when creating a new extension is NAT=yes, making you less secure. Only use that setting if you want to allow a remote extension (with or without a VPN).

 

[flatpack]

this is really very helpfull, couldnt get PBXINF for remote extension but now i think i m getting there,