[1st-mile-nm] CenturyLink is DOWN, can't even create a ticket

[John Brown]

For their DWDM / Optical Transport network to break in this fashion
limits the possible root causes.

Of course when the DWDM Optical network craters, then there is an
issue with the OSI Layer 3 IP Network side
and their own internal networks.     I don't expect this to get
resolved for at least another 12 to 24 hours :(
The SLA credits that will be written are going to be HUGE.   Maybe
they need the expense for this fiscal quarter :|


Updates from one of our carrier partners

QUOTE
We are still chasing the partner and the update remains the same that
the circuits are affected due to Nation wide outage.

We would like to tell you that Tickets are being escalated and are
promptly chased with our partner for updates , but unfortunately still
the issue is yet to be isolated by our partner and the outage is
affecting multiple customers.

More updates to follow soon.

END QUOTE

On Thu, Dec 27, 2018 at 2:39 PM Michael Harris <mharris at visgence.com> wrote:
>
> The outage started around 3am Eastern, which feels a lot like a maintenance window. My wild speculation is that some bad configs got pushed, en-masse, and now someone has to throw a console cable onto a thousand routers.
>
> -Michael
>
> On Thu, Dec 27, 2018 at 2:35 PM David Cortez <cortezdl at mac.com> wrote:
>>
>> What is the cause of the outage?
>>
>> On Dec 27, 2018, at 2:33 PM, Michael Harris <mharris at visgence.com> wrote:
>>
>>> So techs can't even create a ticket or find out what the status of things are...
>>
>>
>> Yes, I saw the message on Control Center this morning and thought "hmm, this isn't good..." KOB made some inquiries and has the only ETA (24-48 hrs) I have seen reported so far:
>>
>> https://www.kob.com/new-mexico-news/verizon-century-link-outages-affect-new-mexicans/5191901/?cat=500
>>
>> I saw some tweets in the last few minutes about pockets of customers starting to come up in the midwest.
>>
>> For what it's worth, our links seem to be up, but can't ping gateways on CL side... Verizon LTE seems like it is showing some strain here in Las Cruces, as well...
>>
>> -Michael
>>
>> On Thu, Dec 27, 2018 at 11:57 AM John Brown <john at citylinkfiber.com> wrote:
>>>
>>> So part of the problem is that CenturyLink's own internal ticketing
>>> system is down.
>>> So techs can't even create a ticket or find out what the status of things are...
>>> _______________________________________________
>>> 1st-mile-nm mailing list
>>> 1st-mile-nm at mailman.dcn.org
>>> http://www2.dcn.org/mailman/listinfo/1st-mile-nm
>>
>>
>>
>> --
>> Michael Harris
>> --
>> President, Visgence Inc.
>> www.visgence.com
>>
>> _______________________________________________
>> 1st-mile-nm mailing list
>> 1st-mile-nm at mailman.dcn.org
>> http://www2.dcn.org/mailman/listinfo/1st-mile-nm
>
>
>
> --
> Michael Harris
> --
> President, Visgence Inc.
> www.visgence.com