Silvey, David via Outages | 5 Feb 04:16 2016

[outages] Xo Infinera 1600G span failure since about 5:15 pm Eastern




  • 2/4/16 6:01 PM

    Alert Detail: Infinera 1600G span failure between Smyrna, GA and Washington, DC  
    Alert Update: Infinera 1600G span failure between 4000 Highlands Pkwy SE, Smyrna, GA and 4301 Connecticut Ave NW, Washington, DC. Transport Operations is investigating. 



The information contained in this E-mail message is privileged, confidential, and may be protected from disclosure; please be aware that any other use, printing, copying, disclosure or dissemination of this communication may be subject to legal restriction or sanction. If you think that you have received this E-mail message in error, please reply to the sender.

This E-mail message and any attachments have been scanned for viruses and are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened. However, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by Kelley Drye & Warren LLP for any loss or damage arising in any way from its use.


<div>
<br><br><div><br></div>
<div>
<ul class="list-group"><li class="list-group-item ng-scope" ng-class-odd="odd" ng-repeat="activity in vm.ticket.details.activities">
<p class="h4 ng-binding">
<span>2/4/16 6:01 PM</span></p>
<p><span ng-bind-html="vm.cleanHtml(activity.detail)" class="ng-binding">Alert Detail: Infinera 1600G span failure between Smyrna,
 GA and Washington, DC&nbsp;&nbsp;<br>
Alert Update: Infinera 1600G span failure between 4000 Highlands Pkwy SE, Smyrna, GA and 4301 Connecticut Ave NW, Washington, DC. Transport Operations is investigating.&nbsp;</span></p>
</li></ul>
<br>
</div>

    <br>The information contained in this E-mail message is privileged, confidential, and may be protected from disclosure;
please be aware that any other use, printing, copying, disclosure or dissemination of this communication may be 
subject to legal restriction or sanction. If you think that you have received this E-mail message in 
error, please reply to the sender.    <br><br>
This E-mail message and any attachments have been scanned for viruses and are believed to be free of any virus or 
other defect that might affect any computer system into which it is received and opened. However, it is the 
responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by Kelley 
Drye &amp; Warren LLP for any loss or damage arising in any way from its use.    <br><br><br>
</div>
Mark Felder via Outages | 4 Feb 01:00 2016

[outages] FYI: TMobile USA NAT64 is broken

Hi,

If you have an iPhone and have iOS 9.3 beta2 you receive carrier update
23.5.4 which makes your LTE IPv6-only. There are major issues with
TMobile's existing NAT64 implementation. I suspect it is limited to UDP
traffic, but I am limited from what I can test with my iPhone. Most TCP
traffic seems to work, but it's impossible to do Facetime (audio or
video) calls, etc. The push messages (TCP) are received by both ends,
but the session never successfully connects and it times out.

Alternatively I've noticed it's impossible to connect to any Plex
servers (TCP, port 32400).

All of this is resolved if you use a VPN, which is frustrating. I hope
this is fixed before 9.3 rolls out to everyone.

--

-- 
  Mark Felder
  feld@...
Hyoun Kim via Outages | 3 Feb 17:03 2016

[outages] Level 3 Fiber Cut in Cincinnati, OH

My circuit from Nashville, TN to Chambersburg, PA went down around 8:40 AM CST.  When I opened a ticket w/ Level 3, they mentioned there is a major event that occurred exactly at 8:40 AM on their end as well.

The automated email system reported: 

02/03/2016 15:24:42 GMT - The Transport NOC has reported that suspected fiber cut in Cincinnati, OH is impacting unprotected Transport services.  Field Services have been engaged to investigate further. Please be advised that updates for this event will be relayed hourly unless otherwise noted.

<div><div dir="ltr">My circuit from Nashville, TN to Chambersburg, PA went down around 8:40 AM CST.&nbsp; When I opened a ticket w/ Level 3, they mentioned there is a major event that occurred exactly at 8:40 AM on their end as well.<div><br></div>
<div>The automated email system reported:&nbsp;</div>
<div><p class="">02/03/2016 15:24:42 GMT - The Transport NOC has reported
that suspected fiber cut in Cincinnati, OH is impacting unprotected Transport
services.&nbsp; Field Services have been
engaged to investigate further. Please be advised that updates for this event
will be relayed hourly unless otherwise noted.</p></div>
</div></div>
Tailor Mitchell via Outages | 29 Jan 16:19 2016

[outages] Intermedia/Comcast Outage

New Event: 1078 Connectivity issues to Intermedia services via Comcast
Started at: 01/29/2016 08:30 AM ET
Updated at: 01/29/2016 10:04 AM ET
Summary of Issue:
Some customers may experience connectivity issues when connecting to Intermedia servers due to an incident on Comcast ISP side.
Business Impact:
Some users on Comcast ISP may observe Outlook, OWA, Cloud Server and HPBX connectivity issues.
Technical Details
Customers whose trace route connections are passing through Comcast ISP may experience Outlook, OWA, Cloud Server and HPBX connectivity issues due to an outage on Comcast ISP side. Please contact your ISP provider for further troubleshooting.

 

 

<div>
<div class="WordSection1">
<p class="MsoNormal"><span>New Event:
</span>1078 Connectivity issues to Intermedia services via Comcast <br><span>Started at: </span>01/29/2016 08:30 AM ET
<br><span>Updated at: </span>01/29/2016 10:04 AM ET
<br><span>Summary of Issue: </span><br>
Some customers may experience connectivity issues when connecting to Intermedia servers due to an incident on Comcast ISP side.
<br><span>Business Impact: </span><br>
Some users on Comcast ISP may observe Outlook, OWA, Cloud Server and HPBX connectivity issues.
<br><span>Technical Details</span>
<br>
Customers whose trace route connections are passing through Comcast ISP may experience Outlook, OWA, Cloud Server and HPBX connectivity issues due to an outage on Comcast ISP side. Please contact your ISP provider for further troubleshooting.<span><p></p></span></p>
<p class="MsoNormal"><span><p>&nbsp;</p></span></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
</div>
</div>

[outages] Central Logic Web Hosting Outage

Any central logic customers seeing an outage? To us they appear down hard, and don't appear to be running off of a backup connection.

--
Nathan J. Lichtenstein
 

Rochester Institute of Technology
Networking and Systems Administration, BS

Cisco Certified Entry Network Technician
Verification No. 421913973673ASCH
<div><div dir="ltr">Any central logic customers seeing an outage? To us they appear down hard, and don't appear to be running off of a backup connection.<br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr">
<div>
<span>Nathan J. Lichtenstein<br>&nbsp;</span><br><span>Rochester Institute of Technology</span><br>Networking and Systems Administration, BS</div>
<div><br></div>
<div>Cisco Certified Entry Network Technician</div>
<div>
<span>Verification No.&nbsp;</span>421913973673ASCH</div>
</div></div></div></div>
</div></div>
Thom Cleary via Outages | 28 Jan 22:06 2016

[outages] Level 3 Kansas City issue updates

Here's the latest:

01/28/2016 20:37:04 GMT -
*** CASCADED EXTERNAL NOTES 28-Jan-2016 20:36:37 GMT From CASE: 10303209 - Event
The Fiber Splice crew has arrived at the cut site.  The fiber will need to be excavated approximately 20 feet back from the man hole to create enough slack for repairs.  There remains no ETR to provide at this time.

01/28/2016 19:50:56 GMT - 01/28/2016 19:47:51 GMT - Field Services have identified the fiber cut in Leavenworth, KS and a fiber splice crew is en route.  Fiber prep work has begun but there is no ETR available at this time.

01/28/2016 18:58:19 GMT - The Transport NOC has reported a possible fiber cut between Kansas City, MO and Lancaster, KS is impacting unprotected transport services.  Field Services have been engaged to investigate with no ETR to provide at this time. Please be advised that updates for this event will be relayed on an hourly basis unless otherwise noted

<div>
<p dir="ltr">Here's the latest:</p>
<p dir="ltr">01/28/2016 20:37:04 GMT -<br>
*** CASCADED EXTERNAL NOTES 28-Jan-2016 20:36:37 GMT From CASE:&nbsp;10303209&nbsp;- Event<br>
The Fiber Splice crew has arrived at the cut site.&nbsp; The fiber will need to be excavated approximately 20 feet back from the man hole to create enough slack for repairs.&nbsp; There remains no ETR to provide at this time.<br></p>
<p dir="ltr">01/28/2016 19:50:56 GMT - 01/28/2016 19:47:51 GMT - Field Services have identified the fiber cut in Leavenworth, KS and a fiber splice crew is en route.&nbsp; Fiber prep work has begun but there is no ETR available at this time.</p>
<p dir="ltr">01/28/2016 18:58:19 GMT - The Transport NOC has reported a possible fiber cut between Kansas City, MO and Lancaster, KS is impacting unprotected transport services.&nbsp; Field Services have been engaged to investigate with no ETR to provide at this time. Please be advised that updates for this event will be relayed on an hourly basis unless otherwise noted</p>
</div>
Thom Cleary via Outages | 28 Jan 21:46 2016

[outages] Bad Level 3 /TW Telecom issues out of Chicago

We're seeing some terrible packet loss/issues in Chicago.  I just called Level 3, they know of issues in New York, Ohio, and a fiber cut in KC (Missouri)

Yay.
<div><div dir="ltr">We're seeing some terrible packet loss/issues in Chicago.&nbsp; I just called Level 3, they know of issues in New York, Ohio, and a fiber cut in KC (Missouri)<br><br>Yay.</div></div>
Wolf, Geoffrey via Outages | 28 Jan 21:01 2016

[outages] Level 3 <--> Verizon loss in Washington, DC

Seeing large amounts of packet loss between Verizon and Level 3 in Washington, DC. Anyone else?

 

Snippet of an MTR:

 

5. 0.ae1.BR2.IAD8.ALTER.NET 0.0% 35 13.8 20.1 11.7 86.8 16.3
6. ae17.edge1.washingtondc12.level3.net 82.4% 35 19424 19161 18951 19424 193.9
7. ae-11-3601.ebr4.Washington12.Level3.net 94.1% 35 13.6 13.6 13.6 13.7 0.1
8. ae-101-101.ebr2.Washington12.Level3.net 0.0% 35 16.3 18.7 15.3 83.9 12.3
9. ae-58-58.ebr3.NewYork1.Level3.net 97.1% 35 15.7 15.7 15.7 15.7 0.0
10. ae-1-3502.ear2.NewYork1.Level3.net 82.4% 35 22651 23886 22651 24288 637.3
11. TWTC-level3-10G.NewYork1.Level3.net 17.1% 35 15.5 19.2 15.3 79.7 13.1
12. cmh1-ar1-xe-2-0-0-0.us.twtelecom.net 91.3% 24 520.0 519.9 519.7 520.0 0.2


The information contained in this message is confidential and may be privileged and/or protected under law. If you receive this message in error, please notify us immediately by forwarding a copy to compliance-n7suLNCBRUtN8Ch2cx6nig@public.gmane.org and delete the original message and any attachments.
<div>
<div class="WordSection1">
<p class="MsoNormal"><span>Seeing large amounts of packet loss between Verizon and Level 3 in Washington, DC. Anyone else?<p></p></span></p>
<p class="MsoNormal"><span><p>&nbsp;</p></span></p>
<p class="MsoNormal"><span>Snippet of an MTR:<p></p></span></p>
<p class="MsoNormal"><span><p>&nbsp;</p></span></p>
<p class="MsoNormal">5. 0.ae1.BR2.IAD8.ALTER.NET 0.0% 35 13.8 20.1 11.7 86.8 16.3<br>6. ae17.edge1.washingtondc12.level3.net 82.4% 35 19424 19161 18951 19424 193.9<br>7. ae-11-3601.ebr4.Washington12.Level3.net 94.1% 35 13.6 13.6 13.6 13.7 0.1<br>8. ae-101-101.ebr2.Washington12.Level3.net 0.0% 35 16.3 18.7 15.3 83.9 12.3<br>9. ae-58-58.ebr3.NewYork1.Level3.net 97.1% 35 15.7 15.7 15.7 15.7 0.0<br>10. ae-1-3502.ear2.NewYork1.Level3.net 82.4% 35 22651 23886 22651 24288 637.3<br>11. TWTC-level3-10G.NewYork1.Level3.net 17.1% 35 15.5 19.2 15.3 79.7 13.1<br>12. cmh1-ar1-xe-2-0-0-0.us.twtelecom.net 91.3% 24 520.0 519.9 519.7 520.0 0.2<span><p></p></span></p>
</div>
<br>
The information contained in this message is confidential and may be privileged and/or protected under law. If you receive this message in error, please notify us immediately by forwarding a copy to compliance@... and delete the original message and any attachments.

<br>
</div>
dl via Outages | 27 Jan 23:02 2016

[outages] Anyone else see Apple store purchase failures starting around 12:20?

We detected a drop in successful transactions to the Apple Store/iTunes Store starting around 12:22 today, which seems to have fully recovered by 13:30. Just wondering if anyone else saw anything similar.

Ping response times and basic http connections to the service seemed to remain functional and within averages, but the volume of transaction failures increased significantly during this time frame. Just looking to know if anyone saw anything similar, since so far there have been no reports on their system status page.

Thanks for your time,

- Steph 
<div><div dir="ltr">We detected a drop in successful transactions to the Apple Store/iTunes Store starting around 12:22 today, which seems to have fully recovered by 13:30. Just wondering if anyone else saw anything similar.<div><br></div>
<div>Ping response times and basic http connections to the service seemed to remain functional and within averages, but the volume of transaction failures increased significantly during this time frame. Just looking to know if anyone saw anything similar, since so far there have been no reports on their system status page.</div>
<div><br></div>
<div>Thanks for your time,</div>
<div><br></div>
<div>- Steph&nbsp;</div>
</div></div>
Brian Chaffins via Outages | 27 Jan 22:04 2016

[outages] Windstream Web Blocking

This is the second day this week that WindStream’s Chicago/Dallas Cloud services are down. They are
randomly blocking websites at this time with a Fortinet Error.

Brian Chaffins
_______________________________________________
Outages mailing list
Outages <at> outages.org
https://puck.nether.net/mailman/listinfo/outages
Benjamin Monjay via Outages | 28 Jan 01:26 2016

[outages] Sprint DS3 Outage

Our Long Distance in our service center in Hillsboro is out. Sprint is saying a DS3 is down affecting all of Oregon. Anyone else having issues? Not seeing anything at all on the web.

p.s: I have a Sprint cell phone and cannot receive inbound calls at this time as well.

Ben
<div><div dir="ltr">
<div>
<div>Our Long Distance in our service center in Hillsboro is out. Sprint is saying a DS3 is down affecting all of Oregon. Anyone else having issues? Not seeing anything at all on the web. <br><br>
</div>p.s: I have a Sprint cell phone and cannot receive inbound calls at this time as well.<br><br>
</div>Ben<br>
</div></div>

Gmane