FOOD FOR THOUGHT Questions concerning Incredible PBX 13-12.2 upgrade

Discussion in 'Help' started by restamp, Feb 28, 2017.

  1. restamp

    restamp Member

    Joined:
    Apr 24, 2016
    Messages:
    97
    Likes Received:
    52
    I'm currently running an old 13-12.3 version of PIAF (green) (Asterisk 13.8) on an external VPS (CentOS 6). This version has worked fine for me, but I am in the process of trying to add encryption on all my pjsip extensions, and am having no luck with it. However, if I load the latest version of IncrediblePBX13-12 on a local Virtualbox, encryption (TLS/SDES) works fine. (To be honest, I modified the Virtualbox install script slightly to pull the latest pjproject code (2.6).) In any event, I am looking at upgrading my external VPS workhorse, but before I take that plunge I have a couple questions:

    + The current incrediblePBX13-12R.sh grabs Asterisk 13.13 and pjproject 2.5.5, so upgrading from it is not a very good test of whether an upgrade would have a similarly high probability of being successful on my more ancient existing system. Anyone out there have any empirical evidence that points one way or the other?

    + The upgrade script seems to step us back down to pjproject 2.3. Is there a reason for this? I've heard that 2.5.5 is the first pjsip that works well with encryption. (Oddly, when you query asterisk "rasterisk -x 'pjsip show version'", it invariably indicates it is running against pjproject 2.3 regardless of what resides in /usr/src, so maybe I am missing something important here.)

    + We are told backup, backup, backup before attempting such an upgrade, but what sort of backup is appropriate? I doubt "incrediblebackup" would be enough, since it doesn't appear to save /usr/src. Do I need to backup the entire VPS including CentOS or is there a simpler way?

    + Bottom line: I'd like to update my pjsip code and probably bring Asterisk up to its latest release as well. If I simply run the "upgrade-asterisk13-to-current-rhel" script on my field VM -- most likely after substituting pjproject 2.6 for 2.3 in the script itself -- what is the probability of success?

    + One more question: Is there any CLI (or other) command that can be used to show definitively that the RTP media is being encrypted? I'm presuming that when I set up Asterisk to squawk sdes on an extension and it breaks the voice path until I turn on SRTP in my local ATA boxes that that means encryption is indeed working.

    I can't stress enough how much I've enjoyed using PIAF over the past year and, as I learned again today, the instructions for loading it are crystal clear. Thanks again to all the principals who make it possible.
     
    progs_00 and wardmundy like this.
  2. restamp

    restamp Member

    Joined:
    Apr 24, 2016
    Messages:
    97
    Likes Received:
    52
    An update on the above: I pulled the upgrade script and modified it to grab pjproject2.6 instead of 2.3. I also added "--disable-libyuv" to the configure line since libyuv id broken in 2.6 (and since we don't need it in this context anyway). I then ran it on my external Asterisk box and it installed and upgraded everything without a hitch. I'm now running Asterisk 13.14 on my server with the latest pjproject code. As far as I can tell, nothing broke. One annoyance is that I now get the warning: "No ethernet interface found for seeding global EID. You will have to set it manually." every time I bring up the CLI (I'm running on an OpenVZ box which really does have no MAC addresses listed in the "ifconfig"), but the only way I can find to "manually set it" is to modify and recompile Asterisk.

    Another (serendipitous) find: I did heed the comments to delete my GV trunks before upgrading, but after not having fiddled with them for a year, I forgot that there is a special tab for handling Motif. So, I dutifully deleted the trunks, but not the Motif data. After upgrading, I was surprised to find my GV lines still connecting to the Google mothership just like they'd always done. I went to Motif, clicked "Submit" without changing anything and voila!, my deleted trunks got reincarnated automagically.

    So, kudos to the writers of the upgrade script -- you done good!

    Unfortunately, the upgrade didn't really resolve the main problem for which I attempted it: My Asterisk server resides on the greater Internet, but my extensions all reside behind NAT walls. I can get SIP TLS to work fine in this environment, but the ability to get SRTP (SDES or other -- I really don't care) to function with my ATA boxes still eludes me. If anyone has accomplished this and can give me some pointers I'd greatly appreciate it. (I can get SRTP to work fine when everything is on the same LAN, just not when the extensions are behind a NAT.)

    In any event, that's my current status.
     
    progs_00 likes this.
  3. progs_00

    progs_00 Member

    Joined:
    Jan 6, 2014
    Messages:
    93
    Likes Received:
    24
    Hi restamp

    A very interesting post you've made. I am pretty much in the same situation as you and I've given up trying but I was unaware of the fact that PBXIAF didn't update to latest pjsip. I'm on IncrediblePBX 13.13.1 on latest Asterisk 13.14.0 but pjsip is still on v2.3. I will try to update also just to see if encryption starts working and I will post back my findings.
    I'm just wondering on if there is any specific reason for PBXIAF to not update pjsip. Guess @wardmundy can help us on that

    Bye

    Edit: Just finished updating to pjsip 2.6 in my test box whitout any issues. Will start testing encryption
     
    #3 progs_00, Mar 3, 2017
    Last edited: Mar 3, 2017
  4. restamp

    restamp Member

    Joined:
    Apr 24, 2016
    Messages:
    97
    Likes Received:
    52
    Progs_00, just to clarify, I have been able to get encryption (TLS for SIP coupled with SDES for RTP) to work on the Asterisk 13.13.1 / PJSIP 2.6 software load when everything is on the same LAN. (If you want me to document what I did, I'd be happy to do so.) What continues to elude me is the ability to get an Asterisk server with a public IP address to reliably encrypt the media channel when talking to PJSIP extensions which are located behind a NAT firewall. (TLS in this scenario continues to work fine, though, and I've been using it to encrypt my SIP data for a couple months now.) I've read several places that you should really be on at least pjproject release 2.5.5 if you want to play with encryption. My overall assessment at this point, though, is that it's just not production quality yet. On my VirtualBox test box, I've managed to cause Asterisk to segfault and core on several occasions. I'm by no means an Asterisk guru, and the default load is not compiled for debugging, but gdb definitely indicates the fault occurred in the pjsip cryptocode arena. Glad to hear someone else is interested in encryption and Asterisk. If you have some success, let us know!
     
  5. ostridge

    ostridge Guru

    Joined:
    Jan 22, 2015
    Messages:
    372
    Likes Received:
    66
    I'll vote for that

    I'm guessing here that the extensions referred to are provisioned with some sort of certificate?
    For example on my Sipura (cisco) PAP2t ATA I have for line 1 Subscriber Information
    Code:
    Display Name: extn#
    User ID: extn#
    Password: absolutelysecret
    Use Auth ID: Yes/No
    Auth ID: extn#
    
    Mini Certificate:  #  ( fwiw mine=508 chrs long  # i.e.= &h1fb +1 =&h1fc)
    SRTP Private Key: *************
    
    In Regional Settings
    Secure One Call Act Code: *18[/code]So to secure a call via srtp dial “*18” followed by the phone number. If you hear three unique tones after dialing, your connection is secure.

    I got (long ago) some info from voxilla.com however their cert generator does nothing and a linux binary mini-certificate generator called gen-mc.c-v0.98.tar.gz had to be downloaded from ??? emule or something.
     
    #5 ostridge, Mar 4, 2017
    Last edited: Mar 4, 2017
  6. restamp

    restamp Member

    Joined:
    Apr 24, 2016
    Messages:
    97
    Likes Received:
    52
    OK, I'll try to get to the documentation later on today, but to address your other points, you shouldn't don't need to provision any certs on your ATA to make the encryption work, at least if you can tell it not too bother with validating the certs. What provisioning does is to allow the extension ATA or phone to determine, by being able to trace the cert back to a trusted Certifcate Authority, that it is indeed interacting with a known server. Think of the TLS channel as providing essentially the same security for SIP that HTTPS does for web services: If you are visiting a known IP and only want encryption, you don't need a CA to validate the cert. And for now, I'm not worried about things like Man-In-The-Middle. I just want a proof of concept that I can make encryption work.

    I also have an old PAP2T from back in the days when it was branded Linksys. I didn't think that ATA supported encryption, but I'll have to take a second look. In any event, it sounds like what you are talking about is end-to-end encryption, which is basically saying that if you have two supporting devices, you can push a button (or TT in a sequence) and have the two devices negotiate and encrypt the media channel between them. This would be a nice thing to have -- theoretically even an Asterisk admin could not listen in on the conversation transiting the server -- but what I'm trying to obtain is simply the encryption of one link between the server and the client ATA.

    More later.
     
  7. restamp

    restamp Member

    Joined:
    Apr 24, 2016
    Messages:
    97
    Likes Received:
    52
    OK. Here is a brief set of notes on how I set up TLS encryption on 13-12.3 (CentOS 6):
    • Before doing anything else, backup, backup, backup!
    • Generate the Public Key Certificates for Asterisk: Many writeups suggest using the ast_tls_cert shell command to generate the necessary certs (for instance, see https://wiki.asterisk.org/wiki/display/AST/Secure+Calling+Tutorial), but I've found that FreePBX will do it from the web browser: Go to Admin -> Certificate Management -> Certificate Authority Settings and ignore the warnings; delete the existing CA and create a new one. (Make sure you remember the password you chose.) Then generate a new certificate. I called mine asterisk.
    • There is no FreePBX template for TLS, so we'll have to create a TLS transport by directly modifying an /etc/asterisk .conf file. Edit the file /etc/asterisk/pjsip.transports_custom.conf and add:
    [0.0.0.0-tls]
    type=transport
    protocol=tls
    bind=0.0.0.0:5061
    cert_file=/etc/asterisk/keys/asterisk.crt
    priv_key_file=/etc/asterisk/keys/asterisk.key
    method=tlsv1​
    • While you're at it, you might as well go to Settings -> Asterisk SIP Settings -> Chan PJSIP and activate the TCP protocol as well (requires an Asterisk restart to "take").
    • Finally, on your phone or ATA, select TLS for the SIP transport. On the Grandstream HT802, you can do this by FXS_PORTx -> SIP Transport -> TLS (and click Apply). On the OBi200 it's ITSP Profile X -> ProxyServerPort -> 5061 and ProxyServerTransport -> TLS (save and reboot). Make sure that all CA validation requirements are either turned off on the ATA box or phone, or that you have provided the device with the proper CA cert you generated above.
    • To verify TLS is active, go to Reports -> Asterisk Info -> Chan PJSip Info. You should see ";transport=TLS" appended to the contact info on the encrypted SIP channels.
    That's basically what I did to get TLS encryption working. At this point, we have encryption working on the SIP channels but not the media channels, which means outsiders can't tell who we're calling, but they can still listen in on the voice portion of our calls. Later I'll describe what I did to get the RTP media stream to encrypt. Good luck.
     
    #7 restamp, Mar 4, 2017
    Last edited: Mar 4, 2017
    ostridge likes this.
  8. restamp

    restamp Member

    Joined:
    Apr 24, 2016
    Messages:
    97
    Likes Received:
    52
    OK, so we now have TLS encryption working and presumably no one can observe our SIP (signaling) traffic. But our media (voice) traffic, which is handled by RTP, is still unencrypted. How do we encrypt that? First, FreePBX only seems to offer DTLS as an encryption option. DTLS is one of several RTP encryption schemes. It is complicated and powerful, and there is a high likelihood it will eventually win out, but it is problematic today for several reasons: First, it really is complex, with options for end-to-end encryption superseding link encryption. It is not particularly well implemented in the phones and ATAs in use today, and the FreePBX implementation we have at our disposal is one done rather early in the game. In short, I never could get it to work with any of my extensions. Instead, I elected to play with SDES, which is far more widely supported in today's FXO market. Our FreePBX unfortunately does not have support for SDES, so we once again are forced to implement it by directly editing the Asterisk config files.

    When you create a FreePBX pjsip extension, it is unencrypted by default. Therefore, you have to tell Asterisk to use encryption. You cannot just alter the same /etc/asterisk files that FreePBX maintains, as your changes will get overwritten every time one does an update. The secret is to add encryption via one of the custom files, and here's the secret sauce:

    Edit /etc/asterisk/pjsip.registration_custom.conf and insert the following two lines for each extension that you want to encrypt:

    [701](+)
    media_encryption=sdes​

    Here 701 is the extension. Replicate these lines, changing 701 to other extensions as appropriate. The net result will be, for each line you add above, to add the media_encryption=sdes option to the specifications for each pjsip endpoint that FreePBX creates. (Obviously, you must tell Asterisk to "reload" after making any changes to the above file.)

    Next you have to enable sdes on the extension device. For OBis: Voice Service -> SPx Service -> X_SRTP -> Use SRTP Only. For Grandstreams: FXS Portx -> SRTP Mode -> Enabled and forced.

    There are a number of potential concerns with RTP and SRTP behind a firewall: I've read in several places that Asterisk does not support the Crypto Life Time field and actually will reject keys that include it. A perusal of the code itself reveals that this is not true for the versions of Asterisk we are using here: Asterisk 13.13 (and up) handle reasonable Crypto Life Time values quite well. (Crypto Life Time specifies the number of UDP packets that can be encrypted with a given key before it must be changed.)

    Additionaly, various how-to websites list other pjsip options which they claim should be set or unset when employing SRTP encryption. The following is a list of the one's I would deem appropriate:

    direct_media=no ;...................can media flow directly between endpoints?
    force_rport=yes ;.....................force A* to send SIP packets to last port they were rcvd from
    ice_support=no ;......................turn on unspecified A* support for dealing with NATs
    rewrite_contact=yes ;..............rewrite Contact Header with latest source addr and port
    rtp_symmetric=yes ;................send RTP packets to same source & port last rcvd from
    media_encryption=sdes ;.........force use of sdes encryption (the reason we are here)​

    FWIW, most of the above options are FreePBX defaults.

    Finally, I have read in more than one place that SRTP can really only be made to work across a NAT firewall with the aid of a TURN or STUN server. It's difficult to understand why an external server would be necessary for SRTP when it is not needed for unencrypted RTP.

    Sometimes it is difficult to ascertain which of these assertions are based on solid evidence and which are simply antiquated advice or old-wives-tales, so Ive learned to treat all such pronouncements with a grain of salt.

    I believe the pjsip.registration_custom.conf additions are really all I had to do to get encryption to work if the PBX and phone is on the same LAN, but the local LAN is probably not the area where the need for encryption is most pressing. As stated above, I've never gotten encryption to work reliably across a NAT. Occasionally, a (presumably encrypted) call will go through immediately after changing some parameter. This leads me to believe that the NAT is timing out some channel which is critical to the SRTP setup. Most of the time, I either get SIT 488 errors (Obis) or a busy signal (Grandstream). However, non-encrypted calls work fine with this same setup. If anyone can figure out the failure mode here, I'd be delighted to learn what I've been missing.

    Some final caveats: SDES sends it's session key in cleartext over the SIP channel, so unless you are also encrypting this signaling channel (TLS) there is little advantage to using sdes as anyone can simply pick up the key and use it to decrypt the packets that follow. Also, I've observed several Asterisk cores since I began testing SRTP. These trace back to various encryption subroutines and obviously reduce the stability of Asterisk. You may well want to weigh the advantages of media encryption against the backdrop of using a less-than-mature implementation before making these changes to a production server.

    If anyone tries any of this, let us know what you see.
     
    #8 restamp, Mar 4, 2017
    Last edited: Mar 6, 2017
    ostridge likes this.
  9. restamp

    restamp Member

    Joined:
    Apr 24, 2016
    Messages:
    97
    Likes Received:
    52
    To bring this thread to closure, I was never able to find a combination of pjsip options that would reliably allow media encryption through a NATted firewall. Perhaps it's possible to make it work with a STUN or TURN server, but I didn't want to have to deal with one of those. However, if you have several endpoints behind a common firewall which require secure media, the following scenario might be worth exploring. I actually set this up locally as a proof of concept:
    1. Configure a stripped down asterisk server on a small machine behind the firewall that can be left up 24/7. (I used a PogoPlug E02 for this purpose because that's what I happened to have handy, but something like a Raspberry Pi should work equally well.) Just remember: you don't need a full PIAF system here; you want something you can hone down fairly severely: All you really need are the capabilities for doing PJsip, IAX2, and bridging the two.
    2. Home all your behind-the-NAT endpoints to the above mini-server as opposed to your main Asterisk workhorse.
    3. Configure a pair of encrypted virtual IAX2 trunks (one in each direction) between the two machines. (Unlike PJsip, encrypted IAX2 trunks are designed to work fine through a firewall, and do.) These trunks will provide your access to the outside world via the main system.
    At this point, all media traffic between the two Asterisk servers is encrypted, and since new server and endpoints are on the same LAN, traffic between them may be encrypted as well if desired.

    Here's the instructions I used to load a stripped-down Asterisk+FreePBX on my PogoPlug EO2: <<< http://wiki.freepbx.org/display/FOP/Installing+FreePBX+13+on+Debian+8.1 >>>. (And, note: You will most likely not need to load the DAHDI package here, so don't waste your time trying to include it.)

    Here's the instructions I used for creating the pair of encrypted IAX2 trunks between the two machines: <<< http://wiki.freepbx.org/pages/viewpage.action?pageId=4161588 >>>.

    Both of the above articles are worth perusing in their own right in order to better understand how the various parts of a FreePBX Asterisk system are built, how they come together, and how IAX2 trunks work.

    I did have to open up one hole in my firewall, but it is a minor one: I allowed my external PIAF Asterisk server (and only that server) to send IAX2 packets to the mini-Asterisk box via port 4569 (the IAX2 port). You should also verify that iptables won't block these packets.

    One downside -- and there is always a downside, isn't there? -- is that for some reason IAX2 doesn't encrypt the outgoing dialed numbers, so there is still some leakage of information from the encrypted IAX2 trunks.

    Another option might be to avoid NATs entirely by routing the media traffic over IPv6, but I have not explored that angle as I currently have no IPv6 access here.

    If you have any specific questions about any of my posts in this thread, feel free to ask. Now, if I could only seamlessly upgrade my main PIAF server to FreePBX 13.
     
    ostridge likes this.
  10. restamp

    restamp Member

    Joined:
    Apr 24, 2016
    Messages:
    97
    Likes Received:
    52
    You all must be getting tired of the ceaseless addendums to this thread, but this is something I believe should be published to the community for current and future reference. The other night, I came across the following two articles which describe how to build the pjproject code for Asterisk:

    https://blogs.asterisk.org/2017/03/01/pjproject-2-6/
    https://wiki.asterisk.org/wiki/display/AST/Building+and+Installing+pjproject

    Both are worth reading in their entirety if you are planning on upgrading your Asterisk, but some highlights: (1) Since pjproject 2.5, the Asterisk team has been making mods to the PJ software before compiling it. If you just pull down the official software without making these mods, either the make will fail or the Asterisk produced will be unstable. (2) In order to make the process of building the PJ modules almost trivial, Asterisk has written a "third-party app" to pull in the correct PJ code, modify it, compile it, and statically link it into Asterisk as part of the Asterisk build procedure. All you have to do is add "--with-pjproject-bundled" to the list of Asterisk configure options. (3) A nice side effect is that, if you upgrade this way, you don't have to touch the dynamic libs from the earlier build, making it easy to back back down to the original Asterisk if for some reason you need to do so.

    I modified Ward's upgrade script to utilize this method for building pjproject and now have a stable PIAF 13.12.3 incorporating Asterisk 13.14.0 and pjsip 2.5.5.

    In the previous post, I mentioned that pjsip 2.5.5 was really the first version of the pjproject code that worked and was stable with srtp. To test this, I built a 13.12.3 server, upgraded it as shown above, threw a spare NATted router into the mix between the server and the endpoints, and was able to attach two encrypted endpoints which could call and converse with each other reliably across the firewall. (An added bonus: I forgot to specify "direct_media=no" and because of this, after bridging these calls, Asterisk would reissue an INVITE to the two endpoints to establish a direct media session between the two of them, and the latency, which was noticeable immediately after call setup, would simply fade away. Really impressive to observe!)

    The downside is that I still cannot get Asterisk to encrypt the media stream between my home ATAs and my Asterisk server located on a VPS in Washington DC, and indeed, I can even coerce Asterisk into segfaulting when attempting to do so. I'm beginning to think there is something else going on here.

    I've modified Ward's Asterisk upgrade script to utilize the above method for building the PJ code. You can find the copy I used here:

    http://cboh.org/piaf/a*13-upgrade-with-pj-bundled-rhel

    I must warn you that this script has not seen rigorous testing, but if you have any questions, I'll try to answer them.

    A friend once observed that if you think you understand something, it is only because you haven't studied it closely enough. I believe this is true of Asterisk in spades.
     
    #10 restamp, Mar 16, 2017
    Last edited: Mar 17, 2017
    ostridge and wardmundy like this.
  11. KUMARULLAL

    KUMARULLAL Guru

    Joined:
    Feb 20, 2008
    Messages:
    208
    Likes Received:
    16
    Hi restamp. Pretty impressive work so far. The holy grail is to have end to end encrytion working through Natted firewall.
    Here is a video demonstrating decrypting sdes based srtp traffic.
    Somewhere I read zrtp is a new standard for sip based encryption.
    Were you able to encrypt SRTP on Natted firewall?
     
    ostridge likes this.
  12. restamp

    restamp Member

    Joined:
    Apr 24, 2016
    Messages:
    97
    Likes Received:
    52
    Sorry for the delay in seeing your post. True end-to-end encryption does not involve Asterisk at all, or only peripherally: Protocols like ZRTP are negotiated between the two endpoints and simple use the media channel for transporting their bits; it's just that the bits don't mean anything anymore to the listener in the middle. What I was doing is encrypting the pipe between Asterisk and the instrument (or in this case, and ATA box) sitting on someone's desk. The former buys you security if you want to hold a private conversation over an insecure circuit. The latter protects the final links from the prying ears of an ISP or its other users, but it does so without requiring any action or knowledge on the user's part.

    What I did was implement the latter: I was finally able to reliably establish a connection between two endpoints where each phone's SIP and RTP channels were encrypted to the Asterisk server. (The data was not encrypted inside the server.) The endpoints were standard desksets connected to either Grandstream or OBiHAI ATAs. I was able to do this with the server on the WAN side of a NATted firewall and the ATAs on the other. Unfortunately, I was never able to get this to work with my Incredible PBX 13-12.3 system. I believe this is because its base FreePBX code is too old to support encryption, for whatever reason success eluded me.

    I hope this addresses your questions.
     
    #12 restamp, Sep 19, 2017
    Last edited: Sep 19, 2017