Ca By via Outages | 4 Mar 22:17 2015

[outages] Level3 not reaching AWS / Netflix in Southwest with IPv6

Folks,

I posted a ticket with Level3 on this issue, wondering if others are seeing the same thing

Using the Level3 looking glass in Chicago i can ping the www.netflix.com at 2620:108:700f::36f5:7e0f -- which is hosted on AWS


But i cannot ping the same IPv6 address from the Level3 Phoenix looking glass site, which lines up with the issues i see in the southwest on Level3 IPv6


Hopefully someone at Level3 can resolve this issue soon.




<div><div dir="ltr">Folks,<div><br></div>
<div>I posted a ticket with Level3 on this issue, wondering if others are seeing the same thing</div>
<div><br></div>
<div>Using the Level3 looking glass in Chicago i can ping the <a href="http://www.netflix.com">www.netflix.com</a> at&nbsp;<span>2620:108:700f::36f5:7e0f -- which is hosted on AWS</span>
</div>
<div><span><br></span></div>
<div>See image&nbsp;<a href="https://imgur.com/cROsraJ">https://imgur.com/cROsraJ</a>
</div>
<div><br></div>
<div>But i cannot ping the same IPv6 address from the Level3 Phoenix looking glass site, which lines up with the issues i see in the southwest on Level3 IPv6</div>
<div><br></div>
<div>
<a href="https://i.imgur.com/0AhUqnO.png">https://i.imgur.com/0AhUqnO.png</a><br>
</div>
<div><br></div>
<div>Hopefully someone at Level3 can resolve this issue soon.</div>
<div><br></div>
<div><br></div>
<div><br></div>
<div><br></div>
</div></div>
Dantzig, Brian via Outages | 4 Mar 19:51 2015

[outages] Ethernet access in chicago north suburbs.

Anyone else seeing problems with at&t Ethernet access in Chicago far north suburbs. We are seeing problems with multiple carrier connection across multiple Cos. There are delivered on at&t ehternet. Specifically, we see total loss of traffic for 26 seconds to over 1 minute. This is occurring at 30-60 minute intervals.

Brian

<div>
<div class="WordSection1">
<p class="MsoNormal">Anyone else seeing problems with at&amp;t Ethernet access in Chicago far north suburbs. We are seeing problems with multiple carrier connection across multiple Cos. There are delivered on at&amp;t ehternet. Specifically, we see total loss of traffic
 for 26 seconds to over 1 minute. This is occurring at 30-60 minute intervals.<p></p></p>
<p class="MsoNormal">Brian<p></p></p>
</div>
</div>
Peter Beckman via Outages | 26 Feb 21:00 2015

[outages] SoftLayer Latency Jump 4x last 30 minutes

Anyone else seeing increased latency to SoftLayer destinations?

Beginning at 2:33:25pm EST and continuing for the last 26 minutes, Round
Trip Time jumped from a fairly constant 36-44ms to 142-225ms. This may not
seem like much, but it impacts Voice Quality for VoIP calls.

HOST: www.nova.org                Loss%   Snt   Last   Avg  Best  Wrst StDev
   1. xl0.tic.nova.org              0.0%    15    0.2   0.2   0.1   0.2   0.0
   2. wsip-70-168-111-17.dc.dc.cox  0.0%    15    1.4  16.4   0.9 204.2  52.4
   3. ip68-100-3-109.dc.dc.cox.net  0.0%    15    0.9   1.1   0.8   2.3   0.5
   4. ashbbprj01-ae2.rd.as.cox.net  0.0%    15    1.6   2.3   1.5   8.7   1.8
   5. xe-1-0-2.bbr02.eq01.wdc02.ne  0.0%    15    3.9   2.1   1.7   3.9   0.6
   6. ae1.bbr02.eq01.chi01.network  0.0%    15  225.8 225.4 223.3 226.1   0.8
   7. ae7.bbr01.eq01.chi01.network  6.7%    15  226.4 226.1 225.3 226.7   0.4
   8. ae20.bbr01.eq01.dal03.networ  0.0%    15  142.2 143.0 141.8 151.9   2.5
   9. ae0.dar02.sr01.dal01.network  0.0%    15  142.1 142.0 140.6 142.5   0.5
  10. po2.fcr03.sr04.dal01.network  6.7%    15  142.3 142.8 140.8 151.4   2.5
  11. 67.228.7.104-static.reverse. 13.3%    15  142.5 142.3 141.5 142.5   0.3

---------------------------------------------------------------------------
Peter Beckman                                                  Internet Guy
beckman@...                                 http://www.angryox.com/
---------------------------------------------------------------------------
John Pinkerton via Outages | 26 Feb 18:21 2015

[outages] Windstream Phone Outage - Southeast (TN)

I'm currently experiencing a total phone outage with Windstream for the 4th day in less than one weeks time and the 3rd time since Monday. Poor weather has caused significant issues in Georgia according to their support staff, with snow and ice in TN being blamed for prior day outages.

Typical time to repair has been about 1hr over the past week.
<div><div dir="ltr">
<div>I'm currently experiencing a total phone outage with Windstream for the 4th day in less than one weeks time and the 3rd time since Monday. Poor weather has caused significant issues in Georgia according to their support staff, with snow and ice in TN being blamed for prior day outages.<br><br>
</div>Typical time to repair has been about 1hr over the past week.<br clear="all"><div><div>
<br>-- <br><div class="gmail_signature"><div dir="ltr">
<div>
<div>John Pinkerton | IT Manager<br><a href="http://goog_487791471" target="_blank"><br></a>
</div>
<a href="http://www.trewaudio.com" target="_blank">www.trewaudio.com</a><br>
</div>
<a href="mailto:john.pinkerton@..." target="_blank">john.pinkerton@...</a><span><br>615-256-3542<br></span><br><a href="http://www.trewaudio.com" target="_blank"></a>
</div></div>
</div></div>
</div></div>
Roland Dobbins via Outages | 26 Feb 16:30 2015

[outages] Vandalism of CenturyLink fibre in Arizona causes large outage footprint.

		
<https://uk.news.yahoo.com/police-probing-outage-cut-internet-phones-arizona-092936953.html>

-----------------------------------
Roland Dobbins <rdobbins@...>

L.T. Easterly via Outages | 26 Feb 01:00 2015

[outages] Prescott, AZ Fiber Cut (Centurylink)

http://dcourier.com/main.asp?SectionID=1&SubSectionID=1&ArticleID=142238


--
__________________________________________________________________________________________
QTH KMLB/Eau Gallie, Florida::ICOM R75::Alinco DJ-X11::RFSpace SDR-IQ::Pixel Loop RF-1B:: Uniden HP2
SDR Captures/loggings posted to: https://corq.co 
when audio quality allows.
__________________________________________________________________________________________
<div><div dir="ltr">
<a href="http://dcourier.com/main.asp?SectionID=1&amp;SubSectionID=1&amp;ArticleID=142238">http://dcourier.com/main.asp?SectionID=1&amp;SubSectionID=1&amp;ArticleID=142238</a><div>
<br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div>
<div dir="ltr">
<div><span>__________________________________________________________________________________________</span></div>
<span>QTH KMLB/Eau Gallie, Florida::</span><span><span>ICOM R75::Alinco DJ-X11::RFSpace SDR-IQ::Pixel Loop RF-1B:: Uniden HP2</span><br><span>SDR Captures/loggings posted to:&nbsp;</span><a href="https://corq.co" target="_blank">https://corq.co</a>&nbsp;</span><span>when audio quality allows.</span>
</div>
<div dir="ltr">
<span>__________________________________________________________________________________________</span><br>
</div>
</div></div></div>
</div>
</div></div>
Chuck Anderson via Outages | 20 Feb 18:18 2015

[outages] NameCheap down, was it hacked?

It looks like NameCheap is down.  It may have been hacked, because I
got a Security Notification from them about a domain I have.

https://www.namecheap.com/

Website is offline

No cached version of this page is available.
Error 522 Ray ID: 1bbc4b52c02609be
Connection timed out

http://downforeveryoneorjustme.com/www.namecheap.com

 It's not just you! http://www.namecheap.com looks down from here.

Check another site?

The e-mail says:

Received: from mta.messagebus.com (mba1056.namecheap.messagebus.com [108.175.18.57])
From: "Namecheap.com - Security Notification" <support@...>
Subject: Security Notification

----------------------------------------------------------------------
Namecheap.com
Activity Notification
----------------------------------------------------------------------
Dear <name>

There was some activity in your namecheap account. Information on what
type of change occurred is available below.

Notification For : HOSTS
Date : 2/20/2015 11:36:09 AM 
IP Address : 192.168.1.10
Username : <username>
Domain (if relevant) : <domain>.org

Old Details
-----------

Host Information
------------------

New Details
-----------

Host Information
------------------

  Name: www
  Type: CNAME
  Address:parkingpage.namecheap.com.
  Name:  <at> 
  Type: URL
  Address: http://www.<domain>.org/?from= <at> 

Additional Information
----------------------
N/A

Thanks,
Namecheap.com
http://www.namecheap.com

>host 108.175.18.57
57.18.175.108.in-addr.arpa domain name pointer mba1056.namecheap.messagebus.com.
>whois 108.175.18.57
[Querying whois.arin.net]
[Redirected to whois.messagebus.com:4321]
[Querying whois.messagebus.com]
[whois.messagebus.com]
%rwhois V-1.5:003fff:00 rwhois.messagebus.com (by Network Solutions, Inc. V-1.5.10-pre6)
network:Class-Name:network
network:ID:MESSAGEBUS-BLOCK1.108.175.16.0/20
network:Auth-Area:108.175.16.0/20
network:Network-Name:MESSAGEBUS-BLOCK1.108.175.106.0
network:IP-Network:108.175.16.0/20
network:IP-Network-Block:108.175.16.0-108.175.24.0
network:Organization;I:Message Bus
network:Tech-Contact;I:hostmaster@...
network:Admin-Contact;I:hostmaster@...
network:Created:20120706
network:Updated:20130313
network:Updated-By:hostmaster@...

%referral rwhois://whois.messagebus.com:4321/auth-area=108.175.16.0/20
Frank Bulk via Outages | 20 Feb 04:44 2015

[outages] Cogent home page issues?

According to the status page (http://status.cogentco.com) there's nothing
going on, but from AS53347 our monitoring system hasn't been able to
reliably pull www.cogentco.com over IPv4 and IPv6 (since 8:15 pm Central)
nor ipv6.cogentco.com (same address as the AAAA for www.cogentco.com).  From
a jump box that's on NTT's network I was able to load www.cogentco.com over
IPv4, but I can't load it over IPv6 (get a "No data received") or it takes a
very long time.

Frank

From AS53347:

root <at> nagios:/etc# tcptraceroute www.cogentco.com
Selected device eth0.3, address 96.31.0.5, port 55934 for outgoing packets
Tracing the path to www.cogentco.com (38.100.128.10) on TCP port 80 (www),
30 hops max
 1  router-core-inside.mtcnet.net (96.31.0.254)  0.256 ms  0.200 ms  0.200
ms
 2  sxct.sxcy.mtcnet.net (167.142.156.197)  0.188 ms  0.124 ms  0.120 ms
 3  premier.sxcy-mlx.fbnt.netins.net (173.215.60.5)  1.672 ms  1.667 ms
1.597 ms
 4  ins-db4-et-0-7-0-0.desm.netins.net (167.142.67.198)  7.933 ms  7.753 ms
7.659 ms
 5  ins-dc2-et-9-4.desm.netins.net (167.142.67.197)  7.899 ms  7.934 ms
7.958 ms
 6  te0-0-1-1.rcr11.dsm01.atlas.cogentco.com (38.104.184.25)  7.889 ms
7.965 ms  7.897 ms
 7  te0-2-0-0.ccr41.ord01.atlas.cogentco.com (154.54.46.237)  15.095 ms
15.216 ms  15.077 ms
 8  be2351.ccr21.cle04.atlas.cogentco.com (154.54.44.86)  23.406 ms  23.469
ms  23.863 ms
 9  be2485.mpd21.dca01.atlas.cogentco.com (154.54.30.18)  35.619 ms  36.489
ms  35.699 ms
10  * * *
11  * * *
12  * * cogentco.sg (38.100.128.10) [open] 47.213 ms
root <at> nagios:/etc# tcptraceroute6 www.cogentco.com
traceroute to cogentco.com (2001:550:1::cc01) from 2607:fe28:0:1000::5, port
80, from port 40783, 30 hops max, 60 bytes packets
 1  router-core.mtcnet.net (2607:fe28:0:1000::1)  0.230 ms  0.216 ms  0.204
ms
 2  sxct.sxcy.mtcnet.net (2607:fe28:11:1002::197)  0.201 ms  0.130 ms  0.127
ms
 3  v6-premier.sxcy-mlx.fbnt.netins.net (2001:5f8:7f0a:2::1)  1.604 ms
1.587 ms  1.598 ms
 4  v6-ins-db4-te-0-6-0-4-219.desm.netins.net (2001:5f8:1:1::1)  8.139 ms
7.777 ms  8.205 ms
 5  v6-ins-dc2-lo0.desm.netins.net (2001:5f8::5)  7.992 ms  7.959 ms  7.970
ms
 6  2001:550:2:24::9:1 (2001:550:2:24::9:1)  8.053 ms  7.954 ms  8.100 ms
 7  te0-2-0-0.ccr41.ord01.atlas.cogentco.com (2001:550:0:1000::9a36:2eed)
15.226 ms  15.371 ms  15.153 ms
 8  be2351.ccr21.cle04.atlas.cogentco.com (2001:550:0:1000::9a36:2c56)
22.562 ms  22.777 ms  22.460 ms
 9  be2485.mpd21.dca01.atlas.cogentco.com (2001:550:0:1000::9a36:1e12)
35.259 ms  35.377 ms  35.435 ms
10  te2-8.ccr01.iad03.atlas.cogentco.com (2001:550:0:1000::9a36:29fe)
35.191 ms  35.144 ms  35.335 ms
11  te2-1.mag01.iad03.atlas.cogentco.com (2001:550:0:1000::9a36:563e)
35.129 ms  35.171 ms  35.129 ms
12  * cogentco.com (2001:550:1::cc01)  47.093 ms [open]  *
root <at> nagios:/etc#

Jeremy Chadwick via Outages | 20 Feb 00:07 2015

[outages] Comcast LA issue?

Something weird is going on with Comcast in Los Angeles: small packets
appear to be getting discarded with a loss rate of ~6-10% while larger
packets (I've tried 1000 and 1200) seem to make it through without any
loss.  Things like SSH and IMAP+SSL are impacted consistently
(particularly during key exchange), while HTTP does not appear to be, so
the issue appears to be affecting both TCP and ICMP (have not tested
UDP).

First sign of this issue, according to my logs, is at 11:51 PT (UTC-8),
and has gradually gotten worse over the past few hours.

Looks like the issue is with the forward path (Source->Destination
listed below), as the return path only shows loss at the final
destination.  (I currently cannot provide a return-path mtr because the
only way I can access my box is through VNC, and I don't like doing
things like screenshots of a VNC window, so you'll just have to trust
me)

Nothing anomalous shows up when using L3's looking glass to ping things,
and the only Comcast LG I know of is in NYC.

I most likely won't see Email responses from anyone (directly or to the
list) until this problem is addressed, since my mail is hosted on shown
box and cannot currently be fetched due to the problem.  Frustrating.

Source IP: 69.181.142.213
Destination IP: 174.136.108.146

ICMP 56 bytes:

                                                                      Packets               Pings
 Host                                                               Loss%   Snt   Rcv  Last   Avg  Best  Wrst
 1. gw.home.lan (192.168.1.1)                                        0.0%   185   185   0.2   0.2   0.2   0.3
 2. 69.181.136.1                                                     0.0%   185   185   8.9   9.9   7.0  21.1
 3. te-0-2-0-5-sur03.santaclara.ca.sfba.comcast.net (68.86.249.69)   0.0%   185   185   8.7   9.9   7.9  17.0
 4. te-0-0-0-13-sur03.sanrafael.ca.sfba.comcast.net (162.151.78.134  0.0%   185   185  10.7  12.9   9.7  20.7
 5. he-1-5-0-0-cr01.sanjose.ca.ibone.comcast.net (68.86.90.93)      98.4%   184     3 49496 49006 47906 49615
 6. he-2-9-0-0-cr01.losangeles.ca.ibone.comcast.net (68.86.86.102)   5.4%   184   174  11.9  14.1  10.9  23.5
 7. 50.242.148.34                                                    6.5%   184   172  11.7  13.4  10.4  69.9
 8. ae-3-80.edge6.LosAngeles1.Level3.net (4.69.144.142)             66.7%   184    61  44.9  20.6  18.0  44.9
 9. ae-3-80.edge6.LosAngeles1.Level3.net (4.69.144.142)             59.0%   184    75  18.7  19.8  18.0  26.0
10. ae-27-27.car2.LosAngeles1.Level3.net (4.69.201.18)               2.2%   184   180  20.8  63.1  18.2 293.3
11. GIGLINX-INC.car2.LosAngeles1.Level3.net (4.30.195.86)            6.0%   184   173  19.2  22.2  19.1  39.3
12. mambo.koitsu.org (174.136.108.146)                               6.5%   184   172  18.8  20.6  18.5  36.9

Note: Loss shown at above hops #5, #8, and #9 are normal according to
historic data (i.e. ICMP prioritisation is in effect) -- however loss at
hop #6 is new.

ICMP 1200 bytes:

                                                                      Packets               Pings
 Host                                                               Loss%   Snt   Rcv  Last   Avg  Best  Wrst
 1. gw.home.lan (192.168.1.1)                                        0.0%    43    43   0.3   0.3   0.2   0.4
 2. 69.181.136.1                                                     0.0%    43    43   9.7  10.4   8.6  14.3
 3. te-0-2-0-5-sur03.santaclara.ca.sfba.comcast.net (68.86.249.69)   0.0%    42    42  10.0  10.0   8.7  16.0
 4. te-0-0-0-13-sur03.sanrafael.ca.sfba.comcast.net (162.151.78.134  0.0%    42    42  11.7  13.3  10.3  18.9
 5. ???
 6. he-2-9-0-0-cr01.losangeles.ca.ibone.comcast.net (68.86.86.102)   0.0%    42    42  14.4  15.9  12.4  25.1
 7. 50.242.148.34                                                    0.0%    42    42  12.7  13.6  11.2  23.1
 8. ae-3-80.edge6.LosAngeles1.Level3.net (4.69.144.142)              9.8%    42    37  20.0  20.7  18.8  25.0
 9. ae-3-80.edge6.LosAngeles1.Level3.net (4.69.144.142)              0.0%    42    42  19.1  21.1  18.9  29.9
10. ae-27-27.car2.LosAngeles1.Level3.net (4.69.201.18)               0.0%    42    42  64.3 105.7  20.9 412.5
11. GIGLINX-INC.car2.LosAngeles1.Level3.net (4.30.195.86)            0.0%    42    42  20.6  22.5  19.9  31.2
12. mambo.koitsu.org (174.136.108.146)                               0.0%    42    42  20.7  22.0  20.1  26.4

--

-- 
| Jeremy Chadwick                                   jdc@... |
| UNIX Systems Administrator                http://jdc.koitsu.org/ |
| Making life hard for others since 1977.             PGP 4BD6C0CB |

Pete E via Outages | 18 Feb 18:07 2015

[outages] Verizon BGP issues?

Anyone else out there experience a pretty vast routing issue with Verizon, starting around 10:50 eastern? We first saw VPN tunnels and SIP Gateways drop off then (caused by) BGP instability. 
<div><div dir="ltr">Anyone else out there experience a pretty vast routing issue with Verizon, starting around 10:50 eastern? We first saw VPN tunnels and SIP Gateways drop off then (caused by) BGP instability.&nbsp;</div></div>
Eric Dugas via Outages | 13 Feb 22:31 2015

[outages] NYIIX / 60 Hudson / TelX blip?

Any one located in the NYC area seen any events from 15:20:00 and 15:45:00 at NYIIX and/or 60 Hudson and/or TelX?


TelX confirmed there was no power outage reported. Will contact NYIIX.

Thanks

 

Eric Dugas

AS40191

<div>
<div class="WordSection1">
<p class="MsoNormal">Any one located in the NYC area seen any events from 15:20:00 and 15:45:00 at NYIIX and/or 60 Hudson and/or TelX?<p></p></p>
<p class="MsoNormal"><br>
TelX confirmed there was no power outage reported. Will contact NYIIX.<br><br>
Thanks<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal"><span>Eric Dugas<p></p></span></p>
<p class="MsoNormal"><span>AS40191<p></p></span></p>
</div>
</div>

Gmane