Tore Anderson via Outages | 4 May 08:28 2016

[outages] Level3 AS3356 major issues

http://sqa.ring.nlnog.net/event/313

Seems to have started at around 05:55 UTC, still ongoing. 40-50k of the
routes on my AS3356 feed that have a BGP age that matches this time.

One way to reproduce is to go to http://lookinglass.level3.net and do a
traceroute or ping to 87.238.33.1 or 193.212.1.10 from San Jose. Both
addresses should have respond to ping and belong to two different AS3356
downstreams; so the only common factor is Level 3. They respond fine
from other PoPs, at least EU ones.

Tore
frnkblk--- via Outages | 3 May 04:32 2016

[outages] 360 Networks/Zayo fiber cut?

Anyone aware of a possible 360 Networks/Zayo fiber cut that happened around
6:15 pm U.S. Central?  It's affecting the ability of our CEA tandem's LD
partner from setting up long distance calls over that path.

Frank

Gary Staff via Outages | 30 Apr 08:04 2016

[outages] Fw: new message

Hello!

 

You have a new message, please read http://omginfographics.com/behave.php

 

Gary Staff

<div><div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US">Hello!<p></p></span></p>
<p class="MsoNormal"><span lang="EN-US"><p>&nbsp;</p></span></p>
<p class="MsoNormal"><span lang="EN-US">You have a new message, please read <a href="http://omginfographics.com/behave.php?1g">http://omginfographics.com/behave.php</a><p></p></span></p>
<p class="MsoNormal"><span lang="EN-US"><p>&nbsp;</p></span></p>
<p class="MsoNormal"><span lang="EN-US">Gary Staff<p></p></span></p>
</div></div>
Joe Klein via Outages | 30 Apr 03:47 2016

[outages] Level 3 and/or Nitel Outage in Miami, FL Area

Received an email from our carrier that they're experiencing an outage in the Miami, FL market - 

Hello Nitel customer,

We are currently experiencing a major outage in Level 3's network affecting multiple customers in the
Miami market. Nitel engineers are working with the carrier to restore your service as soon as possible.

Our circuit at the airport went down at 1725 site local time and is still down (2146).
Robert Glover via Outages | 27 Apr 19:40 2016

[outages] Verizon issues?

Is anyone seeing weirdness to/from Verizon?

Keynote sees a problem:

http://internetpulse.keynote.com/

Peter Beckman via Outages | 26 Apr 21:59 2016

[outages] HE.net Chicago router link issue?

A few minutes ago I was notified that one of my servers in Denver, CO was
down (ip: 72.18.145.210).

Anyone seeing a similar loss on he.net in Chicago?

nog2.angryox.com (0.0.0.0)                                 Tue Apr 26 15:54:54 2016
                                            Packets               Pings
  Host                                    Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. xl0.tic.nova.org                      0.0%    36    0.1   0.1   0.1   1.0   0.0
  2. wsip-70-168-111-17.dc.dc.cox.net      2.8%    36    1.8  13.3   1.1 183.2  40.1
  3. ip68-100-3-109.dc.dc.cox.net          0.0%    36    1.1   3.0   1.1  25.6   4.9
  4. ashbbprj01-ae5.0.rd.as.cox.net        0.0%    36    1.7   3.6   1.7  20.7   4.3
  5. 68.105.30.114                         2.8%    36    7.8  15.1   1.8 120.4  21.0
  6. 10ge3-6.core2.chi1.he.net             2.8%    36   33.2  35.2  28.1  51.4   6.9
  7. ??

But from Verizon it is up:

max.angryox.com (0.0.0.0)                                  Tue Apr 26 15:56:08 2016
                                            Packets               Pings
  Host                                    Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. tomato.angryox.com                    0.0%   158    0.3   0.4   0.3   2.4   0.2
  2. lo0-100.WASHDC-VFTTP-312.verizon-gni  0.0%   158    3.7   7.0   3.0 153.9  13.1
  3. T1-3-0-4.WASHDC-LCR-21.verizon-gni.n  0.0%   158    7.9   8.1   3.1 156.9  12.0
  4. ???
  5. ???
  6. 0.ae3.GW11.IAD8.ALTER.NET             0.0%   157    6.5   8.5   5.4 118.2  12.1
  7. teliasonera-gw.customer.alter.net     0.0%   157    7.2   9.0   5.5 164.1  15.7
  8. ash-bb3-link.telia.net                0.0%   157    8.0   8.9   5.5 148.9  14.4
  9. sjo-b21-link.telia.net                0.0%   157   71.1  72.6  68.8 150.8   7.2
10. den-b1-link.telia.net                 0.0%   157   99.3  99.6  98.0 122.4   2.4
11. handynet-ic-145041-den-b1.c.telia.ne  0.0%   157  130.4 100.1  98.0 130.4   3.3
12. ae2-14.core1.denver2.handynetworks.c  0.0%   157   97.1  97.1  95.5 113.8   1.8
13. ae0-10.dist2.denver2.handynetworks.c  0.0%   157   99.5 103.9  95.6 828.6  58.2
14. vlan954.sw2-g4.denver2.handynetworks  0.0%   157   98.5 104.7  98.1 761.2  52.8
15. vlan11.sw1-h3.softsyshosting.com      0.0%   157   99.8 103.8  97.1 430.8  33.8
16. 72.18.145.210                         0.0%   157   97.3  97.1  95.5 128.1   2.7

---------------------------------------------------------------------------
Peter Beckman                                                  Internet Guy
beckman@...                                 http://www.angryox.com/
---------------------------------------------------------------------------
Frank Bulk via Outages | 22 Apr 19:20 2016

www.amazon.com is down for us

Anyone else see the same?

root <at> nagios:/# wget -4 www.amazon.com
--2016-04-22 12:19:00--  http://www.amazon.com/
Resolving www.amazon.com... 54.239.26.128
Connecting to www.amazon.com|54.239.26.128|:80... failed: Connection timed
out.
Retrying.

--2016-04-22 12:19:22--  (try: 2)  http://www.amazon.com/
Connecting to www.amazon.com|54.239.26.128|:80... failed: Connection timed
out.
Retrying.

--2016-04-22 12:19:45--  (try: 3)  http://www.amazon.com/
Connecting to www.amazon.com|54.239.26.128|:80... ^C
root <at> nagios:/#

root <at> nagios:/etc/cron.weekly# tcptraceroute www.amazon.com
Selected device eth0.3, address 96.31.0.5, port 41361 for outgoing packets
Tracing the path to www.amazon.com (54.239.26.128) on TCP port 80 (www), 30
hops max
 1  router-core-inside.mtcnet.net (96.31.0.254)  0.258 ms  0.366 ms  0.356
ms
 2  sxct.sxcy.mtcnet.net (167.142.156.197)  0.274 ms  0.173 ms  0.150 ms
 3  premier.sxcy-mlx.fbnt.netins.net (173.215.60.5)  2.131 ms  1.453 ms
1.448 ms
 4  ins-db4-te-0-6-0-4.219.desm.netins.net (167.142.60.157)  7.797 ms  7.594
ms  7.592 ms
 5  ins-dc1-po200.desm.netins.net (167.142.67.193)  9.304 ms  9.177 ms
9.219 ms
 6  ins-kc2-po20.kmrr.netins.net (167.142.67.30)  9.407 ms  10.021 ms  9.228
ms
 7  v504.core1.oma1.he.net (184.105.18.169)  39.532 ms  32.855 ms  41.492 ms
 8  10ge11-15.core2.chi1.he.net (184.105.81.229)  20.037 ms  20.076 ms
21.987 ms
 9  100ge5-2.core1.chi1.he.net (184.105.81.98)  20.034 ms  20.106 ms  19.997
ms
10  100ge5-2.core1.nyc4.he.net (184.105.223.162)  40.097 ms  47.921 ms
38.699 ms
11  100ge11-1.core1.par2.he.net (184.105.81.78)  121.681 ms  113.899 ms *
12  10ge3-2.core1.zrh1.he.net (184.105.222.50)  140.341 ms  131.058 ms
139.553 ms
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
^C
root <at> nagios:/etc/cron.weekly#

-----Original Message-----
From: nagios@...
[mailto:nagios@...] 
Sent: Friday, April 22, 2016 12:18 PM
Subject: PROBLEM: Host amazon is DOWN!

***** Nagios *****

Host: This is a PROBLEM notice that Amazon is DOWN

Date/Time: Fri Apr 22 12:18:06 CDT 2016

CRITICAL - Socket timeout after 10 seconds

John Kinsella via Outages | 22 Apr 19:19 2016

[outages] AWS issues

Currently able to connect to multiple services hosted by AWS (Slack, iTunes, and my own EC2 instances).
http://status.aws.amazon.com/ claims situation normal, so far, although I can’t get into my AWS dashboard.

Might be wider than AWS, but I seem to have connectivity out to other major services...

                                                 Packets               Pings
 Host                                          Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 10.0.1.1                                    0.0%    29    1.0  10.7   0.9 123.2  26.2
 2. 10.0.1.49                                   0.0%    29    1.7  12.1   1.5 101.1  26.7
 3. 173.247.205.38                              0.0%    29    3.7  24.3   3.5 120.4  36.8
 4. x.196.247.173.web-pass.com                  0.0%    29    3.3  20.3   3.2 127.4  36.9
 5. x.196.247.173.web-pass.com                  0.0%    29    3.9  21.0   3.6 106.7  31.3
 6. v505.core1.sfo1.he.net                      0.0%    29    3.9  19.7   3.8 123.5  30.3
 7. 10ge11-2.core1.sjc2.he.net                  0.0%    29   14.8  27.7   4.9 165.2  39.3
 8. 100ge1-2.core1.nyc4.he.net                  6.9%    29   74.4 112.9  66.5 285.1  71.3
 9. 100ge11-1.core1.par2.he.net                21.4%    29  296.4 171.1 137.0 296.4  57.3
10. 10ge3-2.core1.zrh1.he.net                   0.0%    28  175.8 173.3 150.4 280.4  34.0
11. ???
12. ???
13. ???
14. ???
15. te2-2.er01.zrh01.ip-max.net                92.6%    28  391.1 318.5 246.0 391.1 102.6
16. xe0-0-3.cr02.gva253.ip-max.net             96.3%    28  331.7 331.7 331.7 331.7   0.0
17. xe0-0-1.cr02.gva252.ip-max.net             96.2%    27  321.6 321.6 321.6 321.6   0.0
18. ???

_______________________________________________
Outages mailing list
Outages <at> outages.org
https://puck.nether.net/mailman/listinfo/outages
Jeff Agle via Outages | 22 Apr 16:02 2016

[outages] Level 3 issues?

Anyone else seeing peering issues / loss in Level3 this morning?  The below is to/from Eastern Canada.  Keep
getting route changes in the path as
well.

Hop 	Sent 	PL%   	Min    	Max    	Avg 	Host	Name/IP
1 	1535   	0  	0.58  	22.72   	0.87 	192.168.224.1	[192.168.224.1]
2 	1535   	0  	0.83 	284.45  	19.79 	142.166.210.153	[142.166.210.153]
3  	765   	0  	1.31 	202.85  	17.73 	142.166.211.113	[142.166.211.113]
4  	775   	0 	18.50 	289.86  	34.47 	207.231.227.53	[207.231.227.53]
5  	771   	0 	16.26 	278.12  	34.58 	ae52.edge1.toronto2.level3.net	[4.31.208.9]
6 	1532  	90 	84.42 	289.61  	94.59 	ae-1-60.ear1.sanjose1.level3.net	[4.69.152.23]

<---------
7 	1532  	90 	84.40 	304.22  	93.41 	ae-1-60.ear1.sanjose1.level3.net	[4.69.152.23]
   <--------
8 	1535   	0 	84.62 	416.69 	100.82 	one-communi.ear1.sanjose1.level3.net	[4.15.122.98]
9 	1535   	0 	83.31	 410.75  	98.50 	user-uinjs2k.biz.mindspring.com	[165.121.240.84]
10 	1535   	0 	84.63 	406.80  	97.94 	66.32.80.116	[66.32.80.116]
11	1535   	0 	84.63 	367.45 	101.48 	199.180.64.8	[199.180.64.8]
							
Hop	Sent	PL%	Min	Max	Avg	Host	Name / IP
1	299	0	1	3.99	1.75	108.59.240.61	[108.59.240.61]
2	299	0	10.44	31.52	13.29	165.121.238.180		[165.121.238.180]
3	299	0	10.32	29.26	12.99	209.178.150.68   		[209.178.150.68]
4	297	11	10	2437.3	74.79	5-1-34.ear2.newyork1.level3.net	[4.35.90.65]        <----------
5	299	0	10.46	14.35	10.99	4.71.230.102	[4.71.230.102]
6	299	0	26	30.17	26.82	be5.cr01.stjh.nb.aliant.net	[207.231.227.113]
7	299	0	25.76	40.32	30.25	loop0.1kw.ba06.stjh.nb.aliant.net	[142.166.182.14]

~jeff

Grant Ridder via Outages | 21 Apr 23:20 2016

[outages] us-east-1 network partition?

Looks like 1 zone in us-east-1 got network partitioned for ~5 min

Start: 21:01:10,698 UTC
Recover: 21:06:38,946 UTC

VPC / Classic both affected

-Grant
<div><div dir="ltr">Looks like 1 zone in us-east-1 got network partitioned for ~5 min<div><br></div>
<div>
<div>Start: 21:01:10,698 UTC</div>
<div>Recover: 21:06:38,946 UTC</div>
</div>
<div><br></div>
<div>VPC / Classic both affected</div>
<div><br></div>
<div>-Grant</div>
</div></div>
chris via Outages | 21 Apr 20:30 2016

[outages] Verizon Issues?

Anyone aware of any known outages/issues for FIOS? Have a handful of customers in northern NJ with business FIOS with 50%+  packet loss on all verizon hops. We have checked these customers and theres no congestion on their networks.

We're aware of the Verizon strike in progress and wondering if maybe some repair or maintenance is not being done because of the strike?

If anyone from VZ has ability to dig deeper or if anyone else is seeing similar issues would be good to hear.

thanks
chris
<div><div dir="ltr">Anyone aware of any known outages/issues for FIOS? Have a handful of customers in northern NJ with business FIOS with 50%+ &nbsp;packet loss on all verizon hops. We have checked these customers and theres no congestion on their networks.<div><br></div>
<div>We're aware of the Verizon strike in progress and wondering if maybe some repair or maintenance is not being done because of the strike?</div>
<div><br></div>
<div>If anyone from VZ has ability to dig deeper or if anyone else is seeing similar issues would be good to hear.</div>
<div><br></div>
<div>thanks</div>
<div>chris</div>
</div></div>

Gmane