XO Communications Outage - Vitelity

mruge

Guru
Joined
Jan 28, 2008
Messages
121
Reaction score
0
It appears that there is an outage with XO Communications and this is affecting their network of DIDs. Here is the latest that is being reported by Vitelity:

Thursday May 20th 2010, 6:19pm MDT - XO communications is currently experiencing difficulty causing a global DID outage for some users. We are aware of the problem and are working with them to restore service as soon as possible. We have been informed by XO that technicians are on sight and should have this resolved shortly. We apologize for any inconvenience this may cause and thank you for your patience.
 

Fortel

Guru
Joined
Oct 19, 2007
Messages
122
Reaction score
4
Vitelity DIDs / XO Communications down

Right...

It's been a bit over 4 hours now... all of our Vitelity DIDs are failing with "Your call cannot be completed..." or some such...

If VoIP is a bit slow to catch on, these are the kinds of problems that don't help.

Ugg!
 

NorCal Internet

New Member
Joined
Jan 9, 2010
Messages
13
Reaction score
0
Same here... Luckily, I guess, my number is down too, so none of my customer's can call and complain...
 

The Deacon

Guru
Joined
Jan 29, 2008
Messages
296
Reaction score
14
I just called XO and spoke with a rep named Heidi. She reports that XO is not having an outage, nor is she aware of any outage affecting any DIDs. She also said that I'm the second or third call that she has received today.

FWIW: I noticed the same "...XO is experiencing an outage..." message posted on the voip.ms site as well.

The weird thing is that the failover numbers for Vitelity aren't failing over; it's telling people that the number is no longer in service. :shocked:
 

cjkeeme

Guru
Joined
Jun 18, 2008
Messages
203
Reaction score
0
This is the longest downtime I have ever seen. What is the deal?

FWIW: I noticed the same "...XO is experiencing an outage..." message posted on the voip.ms site as well.
This is because Vitelity provides voip.ms with their DID's.

EDIT: Still down...I have purchased new inbound DIDs from another provider and am forwarding all calls to these new numbers. At least Vitelity's failover and forwarding system seem to be working... This is a very expensive solution. I just can't see how an outage can last this long. I hate to speculate, but could this be a contract dispute?
 

Fortel

Guru
Joined
Oct 19, 2007
Messages
122
Reaction score
4
Vitelity's failover / forwarding isn't working for most of our DIDs...

(Oh, and I called XO earlier...tech was not aware of any issues affecting any DIDs.) My guess is some service that XO provides to Vitelity is affecting Vitelity DIDs, not XO DIDs...

Big problem here...
 

phinphan

Active Member
Joined
Oct 19, 2007
Messages
641
Reaction score
130
Same problem here with Vitelity. Calls to all my numbers get the not in service message and I cannot forward the calls. This is a big no-no to have a 10 hour and counting outage.
 

cjkeeme

Guru
Joined
Jun 18, 2008
Messages
203
Reaction score
0
Vitelity's failover / forwarding isn't working for most of our DIDs...

I have encountered a few instances where forwarding isn't working, but failover is working in those cases. It's worth a shot.

Does anyone find it impossible for a Tier 1 provider like XO Communications to have a 12 hours outage. I have never heard of anything like this.
 

cjkeeme

Guru
Joined
Jun 18, 2008
Messages
203
Reaction score
0
All is back up again. That was not fun...need sleep now.
 

SpaethCo

Member
Joined
Jan 10, 2009
Messages
32
Reaction score
0
Does anyone find it impossible for a Tier 1 provider like XO Communications to have a 12 hours outage. I have never heard of anything like this.
Sorry to say this, but then you haven't been paying attention.

http://www.datacenterknowledge.com/archives/category/downtime/

There is no shortage of stories related to outages at major hosting and carrier hotel facilities.

Downtime sucks, but you can only design so much redundancy into your system and account for so many failures. Sometimes you just can't plan for events like an explosion in your electrical room taking out 3 walls and causing the Fire Department to force a complete electrical blackout of the building for 5+ days.

Major outages are rare, but not impossible.
 

wardmundy

Nerd Uno
Joined
Oct 12, 2007
Messages
19,168
Reaction score
5,199
One nice advantage of adding the Google Voice DID layer is that it makes it incredibly easy to switch inbound calls to another DID when there is a failure like this... at least until the Google Voice DIDs croak. :wink5:
 

phinphan

Active Member
Joined
Oct 19, 2007
Messages
641
Reaction score
130
If only Google Voice would let you get more than 1 call per account that would be perfect but that would not make much sense for them.
 

NorCal Internet

New Member
Joined
Jan 9, 2010
Messages
13
Reaction score
0
It would seem as though they are partially back...

I'm still seeing issues when calls are made from AT&T wireless at least, and possibly all of AT&T, getting "Disconnected" messages.

Calls from Verizon, Surewest and others seem to be working to the same numbers...
 
Joined
May 16, 2008
Messages
87
Reaction score
0
If this really hosed you, you should ask for a root cause analysis (RCA) or request for outage (RFO) document from them. Something like this would be that the mgmt or executive level and would have some type of document to explain the issue. They generally have a public one. If they do not have one, or they will not give it to you, that says something. That they do not take availability really seriously.

If they blame a partner, they would have already asked for this from them and should provide regardless.

It is pretty standard.
 

NorCal Internet

New Member
Joined
Jan 9, 2010
Messages
13
Reaction score
0
The issue with calls completing from AT&T wireless seems to be resolved now... So, as far as I can tell, everything is working again...
 

parker

New Member
Joined
Apr 30, 2009
Messages
75
Reaction score
0
XO DID outage

Scuttlebutt is that the outage was XO, and affected other ITSP's, not just Vitelity.
 

The Deacon

Guru
Joined
Jan 29, 2008
Messages
296
Reaction score
14
I just received an email from Vitelity regarding the outage. It appears to be a hardware issue on the XO side, but from my vantage point, Vitelity had a part to play in this by dropping the ball regarding notification, failover and updates to its customers.

So here's to having been on both sides of this issue and knowing what a lesson this can be. :cheers2:

Here's a copy of the email:

Dear Customer,

On May 20, 2010, 4:03pm MDT, our DID origination service with XO Communications suffered an outage. The cause of the outage was an equipment failure at XO’s central office. While XO is investigating the root cause of the equipment failure, we are working with XO to take steps to prevent this kind of failure in the future. This includes:

* Investigate opportunities to add redundancy to single points of failure in the delivery path of XO origination DIDs.

* Develop escalation procedures to meet SLAs and resolve service impacts quickly.

* Develop better notification and updates for impacting events.

We apologize for this service impact - and we are using this outage with XO as an opportunity to increase redundancy for all of our services XO provides, as well as build a better relationship with their support staff to ensure timely response and resolution to impacting events.

A full root cause analysis will be distributed as XO completes their investigation and provides us with the information in the coming two weeks.

Sincerely,

Andy Goodwin
Chief Technology Officer
Vitelity Communications
 

sfr33man2

New Member
Joined
Jul 23, 2008
Messages
8
Reaction score
0
Ongoing and increaing Vitelity issues

We have been a customer of Vitelity for years now. We actually signed up when it was EXGN (who was acquired or merged with Vitelity)

For the most part, our Vitelity service has been good and has gotten better over the past few years with outages happening less frequently, but in 2010 it seems that we are having nearly constant issues with DIDs and clients complaining to us that people come in or call a cell and say that they cannot get through.

Vitelity, understandably, wants alot of info to investigate. Unfortunately because the issues are intermittent and the calls never hit their switch, I cant pull CDRs to give them or be very descriptive. Clients dont track every call, nor would I expect them to. Their service should just work and issues should be an exception. I know that before we open a ticket we investigate the heck out of our softswitches, borders, and internet connectivity to ensure its not an issue with us.

The problem is that if one person makes the effort to get hold of my client through alternate means and inform them of an issue, its probably the tip of an iceberg. People calling a business to shop around or get a quote aren't going to keep trying to call my clients back, they are going to go down the list in the phonebook or online and someone else is going to get the business my client could have gotten if their phone service worked properly.

As resellers, our clients dot know who Vitelity is nor should they. Our clients pay us for phone service and when there is an issue, the issue is ours. Not someone elses.

I have had a couple MAJOR issues when porting from XO to Vitelity where XO does not unbuild the number from the PRI/Tandem platform it was on when they move to SONUS for the port to Vitelity. That makes it so that they get incoming calls through us from most callers, but anyone else with XO who tries to call that client gets a disconnect message as XO tries to route that call on-net, but theres nothing at the other end as that equipment has been removed and replaced with our equipment. We usually dont find out about this until a week later, at which point its a MAJOR issue and the client is extremely upset. In every case that this has happened, Vitelity didn't get the results to fix it. I had to personally make phone calls to their NOC and sit on conference calls and even go to an XO regional office to get it fixed. This is difficult as I am not XOs customer. Vitelity is. Understanding that they are big and cant treat every client as their only client as I have the luxury of doing is hard at times. And I have broken my back fixing that kind of issue, nearly getting sued twice over it. Had I let Vitelity handle those to completion, I would likely be out of business, or had a judgment against me.

Anyhow, is anyone else experiencing these issues with intermittent DID issues increasing as of late? I really do like Vitelity, but for the sake of protecting my brand I am considering porting new DIDs to other carriers who use level 3 or 360 networks. If this continues and I continue to get canned answers of "we aren't experiencing any issues" and the like, I may have no choice but to start transitioning to these other carriers where I have not had any issue whatsoever with incoming DIDs.
 

Members online

Forum statistics

Threads
25,778
Messages
167,504
Members
19,198
Latest member
serhii
Get 3CX - Absolutely Free!

Link up your team and customers Phone System Live Chat Video Conferencing

Hosted or Self-managed. Up to 10 users free forever. No credit card. Try risk free.

3CX
A 3CX Account with that email already exists. You will be redirected to the Customer Portal to sign in or reset your password if you've forgotten it.
Top