R.I.P. Wable: Fun While It Lasted

w1ve

Guru
Joined
Nov 15, 2007
Messages
819
Reaction score
218
Another Wable issue today -- lost SIP connectivity, everything was crazy... Started tracing it to DNS (this was NY POP)
Got my answer from Wable support:

"Hello,

Earlier today in NYC we were receiving a few gigs of ntp reflection attack, we put new policies in place to drop the bad traffic. In doing so it appears that some other udp traffic was dropped, affecting DNS lookups (making them slow). The polocies have been rewritten and now all seems to be working as expected with the NTP flood being dropped and DNS working correctly.

Thank you."

Yup, like SIP UDP traffi!!!
 

wardmundy

Nerd Uno
Joined
Oct 12, 2007
Messages
19,168
Reaction score
5,199
You couldn't reimage your servers from snapshots either. It was a royal mess. And like all good sysadmins, they blamed it on a defective script at our end to begin with so we wiped out several days work trying to load an older snapshot. Grrrrrrr.
 

krzykat

Telecom Strategist
Joined
Aug 2, 2008
Messages
3,145
Reaction score
1,235
Not only that, I had live clients on this. What I did was I kicked up a new instance in Dallas based on the backup of the site in question in NY. I'm now up in Dallas and I'm using DNSmadeEasy for auto triggering redundancy on that side. I've asked them before, and think its worth asking again if we can have mirrored servers between the Dallas and NY site in case anything like this happens again.
 

krzykat

Telecom Strategist
Joined
Aug 2, 2008
Messages
3,145
Reaction score
1,235
I think they should have found this a bit sooner, when did you report to them?


  • Subject: Odd Happenings [i109,b3]

    Posted On: February 10, 2016, 9:45 am

    My server @ xx.xx.xx.xx not replying to asterisk items anymore. Not sure whats up - didnt do any changes.

    Getting ready to just restore from backup. But making a snapshot beforehand taking forever making me think theres a problem
    on your side ???
  • Wable WableStaff
    Posted On: February 10, 2016, 10:30 am

    Nothing changed on our end.
 

krzykat

Telecom Strategist
Joined
Aug 2, 2008
Messages
3,145
Reaction score
1,235
FYI - if you tried to do any work on your server in NY, you still may be waiting. I don't have anything showing on the actions tab. I questioned them about this, and received: "The server is currently queued for a snapshot, there is a backlog due to ddos earlier." Must be some backlog, as I put in this request probably over 7 hours ago. Usually this process takes only a couple of minutes.
 

JLMcCloud

New Member
Joined
Aug 15, 2015
Messages
4
Reaction score
1
I couldn't complete a snapshot, and entered a ticket at 11:34p on 2/8/16 (Subject: Snapshot not completing / appears locked [i57,b3]).

Now, to be honest, what happened at the end of this astonished me, and I'd like to inquire if a few words from Mr. Mundy might be warranted, to be directed at the right individual(s) at Wable. (I was ready to respond just a few minutes after w1ve posted the response he got from Wable, but, frankly, I was afraid of what I've read from others that have had unpleasant experiences with Wable... namely, if you complain, they'll just "turn you off". Yikes! But bullies need to be called out, I figured.)

I went back-and-forth for hours, with me reporting what's going on, and Wable saying they'd look into it. All comm was courteous (in fact, I was being particularly undemanding, I thought). But finally, after 35 hours of mostly no phone service for my farm (and 22 hours since the last communication from them), I asked,

"Guys... whats going on?... my phone system keeps dying for days now!

And here was the response,

"Simply, bundle 3 is not a priority to us at $8/mo. We'll check on this when convenient on our side. Bundle 5 and higher receive priority support."

Wow! So, after cooling off a bit, I responded with,

"XXX [name omitted], if youre going to sell a service, you should stand behind it. It shouldnt matter what price is set... YOU set it.

I went with your service because it was recommended by Ward Mundy. This is really an unacceptable answer to a customer, and Im wondering what he, and/or the FreePBX community, will have to say about your response not to mention your superiors at Incero, should they be made aware of it.

So, please,, fix my server!"

Then, at 4:01p today, I received the same response that w1ve received.

Well, I've been around the block, am pretty thick skinned and I certainly won't lose any sleep over this. But I ran a pretty good-sized PC diagnostic hardware, software, firmware company (Landmark Research Int'l. Corp), and had years of providing tech support to very frustrated, and sometimes desperate, customers (think IRQ and DMA conflicts on the PCs in the late '80s, early '90s. Good gawd, you could literally pull you hare out on that stuff!) I think this kind of behavior can cast a totally unjustified, but nonetheless ugly shadow on the really cool work you've done, and are doing, with PIAF, Incredible PBX, etc., etc., etc. I'm simply blown away at having a full-blown phone system for just a few dollars a month.

Anyway, I thought you should know. Thanks for listening,

Jenni McCloud
 

wardmundy

Nerd Uno
Joined
Oct 12, 2007
Messages
19,168
Reaction score
5,199
Makes Digital Ocean look better and better. It's too bad because Wable has had a good service with terrific performance. But you separate the men from the boys quickly when problems pop up and you see the type of response you get. Can't help believing that, if "bundle 3" VMs were dying, the same thing was happening to those paying the big bucks.
 

krzykat

Telecom Strategist
Joined
Aug 2, 2008
Messages
3,145
Reaction score
1,235
Jenni, I would suggest creating a new machine in Dallas based on a snapshot of your server in NY. That's what I did, and got things back up. I've just upgraded to Bundle 5, so let's see how the responses I get are now.
 

JLMcCloud

New Member
Joined
Aug 15, 2015
Messages
4
Reaction score
1
I still can't do a Snapshot... and BTW, NY is having DNS trouble again.
[2016-02-15 12:29:46] ERROR[17277] netsock2.c: getaddrinfo("gvgw.simonics.com", "(null)", ...): Temporary failure in name resolution
 

JLMcCloud

New Member
Joined
Aug 15, 2015
Messages
4
Reaction score
1
I do have an older Snapshot, but I'm just waiting to get a current one. Things were fine for days with NY until this morning (with the exception of I can't do a Snapshot). Arrgghh!
 

krzykat

Telecom Strategist
Joined
Aug 2, 2008
Messages
3,145
Reaction score
1,235
I've had a snapshot pending for 3 days. I thought it was stuck in neverland queue, so I did a snapshot of another server I have in NY, and its not done yet either. Dallas seems to be working fine. FYI - when I get my snapshots done again, I will be having duplicate servers in both NY and Dallas, and then I will use DNSmadeEasy with their failover so in the occasion of an outage, it should fail over automatically. Of course - you need to make sure your upstream inbound carrier will flow automatically upon failure as well.
 

w1ve

Guru
Joined
Nov 15, 2007
Messages
819
Reaction score
218
I had a snapshot of a NY server that I ported to Dallas, and it is working perfectly, so far. A second was ported, and it did not work in Dallas -- So we rebuilt the server from scratch and reloaded from a backup, and so far all is good. I do not have any paying customers on Wable -- was about to pull the trigger -- but this changes EVERYTHING. I have also had pilot PBXs on Digital Ocean for some time, and they have been rock solid! So the difference between $2 and $5/month servers is huge... actually, it's the difference between a vendor who cares and one that does not. Also, can't say enough about RentPBX. It's where all my production, client PBXs are, and, since RentPBX's inception, service and uptime have been flawless. Much more expensive, but you get what you pay for, I guess.
 

krzykat

Telecom Strategist
Joined
Aug 2, 2008
Messages
3,145
Reaction score
1,235
Yeah, whenever I've had a trouble server, I place it on RentPBX and no issues. If I put Wable into production (which I plan to do), it will be at their Dallas site, and then I'll have a backup server on another system for failover.
 

krzykat

Telecom Strategist
Joined
Aug 2, 2008
Messages
3,145
Reaction score
1,235
FYI - Wable NY appears to be back up in full capacity. Snapshots have completed and I even created another one to make sure - and it completed in only a couple of minutes.

Message I received from them tonight: "Our snapshot queue backed up a lot this past week in NYC due to some network attacks, this has been rectified, and the queue is quickly processing now. "
 

kmcdaniel

Member
Joined
Jan 23, 2008
Messages
416
Reaction score
19
Just a heads up that DNS resolution can be a challenge with Wable especially on the latest CentOS platforms. Here's how we finally fixed it.

In /etc/sysconfig/network-scripts/ifcfg-venet0, add PEERDNS=no and chattr +i /etc/sysconfig/network-scripts/ifcfg-venet0

In /etc/resolv.conf, add nameserver 8.8.8.8 AND nameserver 8.8.4.4 AND chattr +i /etc/resolv.conf

Also, remove postfix since SendMail is included: rpm -e postfix

Then reboot and be sure you can ping incrediblepbx.com.


So I have done the above and rebooted. After every reboot, I am unable tot access the pbx until I do a service network restart. What am I missing?
 

krzykat

Telecom Strategist
Joined
Aug 2, 2008
Messages
3,145
Reaction score
1,235
I Know that on VPS you're responsible for its implementation, but Wable support really is pretty good. You might want to send them a ticket and explain the situation and see if they aren't willing to assist. Might not, but worth a try.
 

kmcdaniel

Member
Joined
Jan 23, 2008
Messages
416
Reaction score
19
I Know that on VPS you're responsible for its implementation, but Wable support really is pretty good. You might want to send them a ticket and explain the situation and see if they aren't willing to assist. Might not, but worth a try.
I guess a quick fix is put service network restart in rc.local. Anyone have clear step by step for wable installs outlined?
 

wardmundy

Nerd Uno
Joined
Oct 12, 2007
Messages
19,168
Reaction score
5,199
I guess a quick fix is put service network restart in rc.local. Anyone have clear step by step for wable installs outlined?

You're going to need to add sleep 10 && /usr/local/sbin/iptables-restart AFTER the network is restarted, or you won't have a functioning firewall. Be sure to check it with iptables -nL to be sure.
 

kmcdaniel

Member
Joined
Jan 23, 2008
Messages
416
Reaction score
19
You're going to need to add sleep 10 && /usr/local/sbin/iptables-restart AFTER the network is restarted, or you won't have a functioning firewall. Be sure to check it with iptables -nL to be sure.
Thanks Ward!
 

Members online

No members online now.

Forum statistics

Threads
25,782
Messages
167,509
Members
19,202
Latest member
pbxnewguy
Get 3CX - Absolutely Free!

Link up your team and customers Phone System Live Chat Video Conferencing

Hosted or Self-managed. Up to 10 users free forever. No credit card. Try risk free.

3CX
A 3CX Account with that email already exists. You will be redirected to the Customer Portal to sign in or reset your password if you've forgotten it.
Top