Outages (planned & unplanned) Reporting

headers

Ben Bartsch | 25 May 17:33

[outages] ATT fiber cut near Lake Charles, LA

we have confirmed there is an 'issue' in the area with ATT

headers

doug schmidt | 24 May 07:17

[outages] Vzw voicemail from cell phone

Can not access vm, fast busy signal, Nj area.

headers

Chris Stone | 22 May 00:25

[outages] Cogent <---> L3 latency

Had some customers not some slow web sites and looking appears to be the result of latency between Cogent and Level3. The http://www.internetpulse.net/ site is also reflecting high latency at the Cogent/Level3 connections. Seeing up and down from here in Denver:

[root <at> sys ~]# mtr --report --report-cycles=25 --report-wide 8-25-240-45.netecin.net
HOST: sys.axint.net                             Loss%   Snt   Last   Avg Best Wrst StDev
1. edge.axint.net                              0.0%    25    0.6   0.7   0.3   3.1   0.6
2. fa0-2.na01.b009854-0.den01.atlas.cogentco.com 0.0%    25    0.9   1.1   0.7   3.1   0.5
3. vl3509.mag02.den01.atlas.cogentco.com          0.0%    25    1.6   5.6   0.7 58.4 15.8
4. te7-4.mpd02.den01.atlas.cogentco.com           0.0%    25 135.1 11.0   0.6 135.1 34.4
5. te0-4-0-2.mpd22.mci01.atlas.cogentco.com       0.0%    25   12.8 12.9 12.6 14.8   0.5
6. te0-3-0-6.ccr22.dfw01.atlas.cogentco.com       0.0%    25   23.5 23.6 23.2 26.1   0.6
7. te0-0-0-0.ccr21.dfw01.atlas.cogentco.com       0.0%    25   22.9 23.3 22.7 27.7   1.0
8. te7-3.mpd01.dfw03.atlas.cogentco.com           8.0%    25 193.8 31.6 23.2 193.8 35.5
9. te-3-2.car3.Dallas1.Level3.net                 4.0%    25 144.8 65.8 46.6 228.5 44.5
10. vlan70.csw2.Dallas1.Level3.net                32.0%    25 147.5 151.4 147.5 159.6   4.4
11. ae-72-72.ebr2.Dallas1.Level3.net              24.0%    25 147.5 147.7 147.3 148.3   0.3
12. ae-2-2.ebr1.Denver1.Level3.net                36.0%    25 149.5 150.3 146.2 159.3   4.2
13. ae-12-51.car2.Denver1.Level3.net               0.0%    25   46.0 79.9 45.9 237.3 58.0
14. EASTERN-COL.car2.Denver1.Level3.net            0.0%    25   47.4 47.4 47.1 48.2   0.3
15. 8-25-240-45.netecin.net                       32.0%    25 158.4 154.4 152.1 158.4   1.9

[root <at> sys ~]# mtr --report --report-cycles=25 --report-wide 8-25-240-45.netecin.net
HOST: sys.axint.net                             Loss%   Snt   Last   Avg Best Wrst StDev
1. edge.axint.net                              0.0%    25    0.6   0.6   0.4   1.5   0.2
2. fa0-2.na01.b009854-0.den01.atlas.cogentco.com 0.0%    25    0.9   1.0   0.7   1.4   0.2
3. vl3509.mag02.den01.atlas.cogentco.com          0.0%    25   67.0   6.0   0.6 67.0 17.6
4. te7-4.mpd02.den01.atlas.cogentco.com           0.0%    25 139.0 32.7   0.8 173.3 55.2
5. te0-4-0-2.mpd22.mci01.atlas.cogentco.com       0.0%    25   12.8 13.0 12.4 15.5   0.6
6. te0-3-0-6.ccr22.dfw01.atlas.cogentco.com       0.0%    25   23.4 23.8 23.2 25.1   0.5
7. te0-0-0-0.ccr21.dfw01.atlas.cogentco.com       0.0%    25   22.9 23.0 22.6 24.5   0.4
8. te7-3.mpd01.dfw03.atlas.cogentco.com           8.0%    25   28.7 84.7 23.2 368.1 94.9
9. te-3-2.car3.Dallas1.Level3.net                 0.0%    25   23.4 28.3 23.2 116.9 19.1
10. vlan70.csw2.Dallas1.Level3.net                 0.0%    25   29.8 26.7 23.2 34.9   4.3
11. ae-72-72.ebr2.Dallas1.Level3.net               0.0%    25   23.6 23.9 23.5 25.9   0.5
12. ae-2-2.ebr1.Denver1.Level3.net                 0.0%    25   38.0 41.4 37.4 50.1   4.5
13. ae-12-51.car2.Denver1.Level3.net               0.0%    25   37.8 68.3 37.5 229.7 58.8
14. EASTERN-COL.car2.Denver1.Level3.net            0.0%    25   38.4 38.9 38.4 40.3   0.5
15. 8-25-240-45.netecin.net                        0.0%    25   49.1 47.4 44.3 51.9   2.1

Anyone hear of anything? (waiting to hear from Cogent)

Chris

headers

Jason Hellenthal | 19 May 22:30

[outages] TcpDump.org ? Cogent ? Something else ?


Anyone else seeing this ?

traceroute to bpf.tcpdump.org (132.213.238.6), 64 hops max, 40 byte packets
[...]
 3  [AS65534] 10.179.128.1 (10.179.128.1)  19.811 ms  17.924 ms  19.907 ms
 4  [AS20115] dtr01hlldmi-gbe-1-14.hlld.mi.charter.com (96.34.33.162)  19.865 ms  17.277 ms  19.886 ms
 5  [AS20115] crr02aldlmi-tge-0-2-0-2.aldl.mi.charter.com (96.34.32.76)  20.099 ms  17.531 ms  19.782 ms
 6  [AS20115] bbr01aldlmi-tge-0-0-0-5.aldl.mi.charter.com (96.34.1.162)  19.832 ms  17.328 ms  19.993 ms
 7  [AS20115] bbr01chcgil-tge-0-0-0-6.chcg.il.charter.com (96.34.0.57)  29.771 ms  27.124 ms  29.923 ms
 8  [AS20115] bbr02chcgil-tge-0-3-0-1.chcg.il.charter.com (96.34.1.185)  19.973 ms  27.418 ms  29.916 ms
 9  [AS174] te0-4-0-4.ccr21.ord03.atlas.cogentco.com (38.104.102.233)  19.841 ms  17.317 ms  19.748 ms
10  [AS174] te0-2-0-1.mpd21.ord01.atlas.cogentco.com (154.54.2.93)  22.040 ms
    [AS174] te0-5-0-7.mpd22.ord01.atlas.cogentco.com (154.54.44.181)  27.478 ms
    [AS174] te0-5-0-7.mpd21.ord01.atlas.cogentco.com (154.54.44.173)  16.951 ms
11  [AS174] te0-1-0-3.mpd22.yyz02.atlas.cogentco.com (154.54.6.181)  51.184 ms
    [AS174] te0-1-0-3.mpd21.yyz02.atlas.cogentco.com (154.54.2.162)  47.444 ms
    [AS174] te0-3-0-3.mpd22.yyz02.atlas.cogentco.com (154.54.42.14)  47.155 ms
12  [AS174] te0-3-0-5.ccr21.ymq02.atlas.cogentco.com (154.54.42.226)  47.212 ms
    [AS174] te0-3-0-3.ccr21.ymq02.atlas.cogentco.com (154.54.7.214)  47.067 ms
    [AS174] te0-5-0-2.ccr21.ymq02.atlas.cogentco.com (154.54.44.117)  46.930 ms
13  [AS174] 38.104.154.62 (38.104.154.62)  57.172 ms  47.932 ms  39.896 ms
14  [AS852] dmtrl-rq.risq.net (192.77.63.50)  39.825 ms  47.695 ms  49.698 ms
15  [AS852] uq-gw.risq.net (132.202.61.142)  50.180 ms  47.221 ms  59.678 ms
16  [AS852] uq-membre.risq.net (206.167.128.90)  49.880 ms  47.619 ms  49.141 ms
17  * * *
18  [AS376] 132.213.254.254 (132.213.254.254)  3790.347 ms !H  3907.398 ms !H  4000.063 ms !H

--

-- 

 - (2^(N-1))

headers

Roberts, Brent | 15 May 00:41

[outages] Outages in Hawaii?

TW/HawaiianTel is reporting an outage in Hawaii as an FYI. Multiple Last mile breaks. Existing Breaks specifically on Oahu at the moment.

This email and any attached files may contain confidential and/or privileged material and is intended solely for the use of the person to whom it is addressed. Any review, retransmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender immediately and delete it and all attachments from your computer. Progressive Solutions is not liable for any errors or omissions in the content or transmission of this email.

headers

Troy Bourque | 14 May 23:23

[outages] Cogent issues in New Orleans

We’re getting a few spikes here in there in our internet bound traffic through the cogent feed. One of their routers is spiking in latency intermittently and slowing our upstream speeds down. Anyone seeing anything similar events?

Troy,

Troy Bourque, MBA

CCNA, CQS-CWLSS

Troy-k/8DdyzwaCEAvxtiuMwx3w@public.gmane.org

headers

Chad Sutton | 14 May 21:07

[outages] Cogent issues in NY?

Hello,

Is anyone experiencing issues with web browsing on Cogent? For about an hour now, we’re seeing intermittent slowness and page loads stalling. All other Internet connectivity seem to be fine. DNS appears to be working properly (and we’re seeing the same issue if we use 4.2.2.2 or other public DNS.)

We called Cogent support and they said there are no reported issues, but they are going to open a ticket and get back to us.

headers

Wallace Keith | 10 May 20:14

[outages] Tuxedo NY cut

My service through Tuxedo NY just normalized around 1:30 eastern

-Keith

From: outages-bounces-w8q5NolJmqtAfugRpC6u6w@public.gmane.org [mailto:outages-bounces-w8q5NolJmqtAfugRpC6u6w@public.gmane.org] On Behalf Of Jean-François Lévesque
Sent: Wednesday, May 09, 2012 6:09 PM
To: Jonathan Towne
Cc: outages-w8q5NolJmqtAfugRpC6u6w@public.gmane.org
Subject: Re: [outages] FW: Manhole fire, Pittsburgh PA

Fiber cut in Tuxedo, NY. Affecting Level3 and others.

Master ticket: 5584082

No ETR.

JF

On Wed, May 9, 2012 at 5:54 PM, Jonathan Towne <jtowne-wwmPNlwsX4w@public.gmane.org> wrote:

Has anyone heard anything else about this? I've been having issues on Level3
in upstate NY all day.

-- Jonathan Towne

On Wed, May 09, 2012 at 12:52:19PM -0500, Frank Bulk scribbled:
# FYI
#
# -----Original Message-----
# From: Justin M. Streiner [mailto:streiner <at> cluebyfour.org]

# Sent: Wednesday, May 09, 2012 11:37 AM

# To: nanog-eEdOSA2Fa3PYtjvyW6yDsg@public.gmane.org
# Subject: Manhole fire, Pittsburgh PA
#
# There was a manhole/electrical vault fire this morning in Pittsburgh on

# Fifth Ave in the Oakland area. Numerous telcos/dark fiber providers
# affected (Level3, Verizon, DQE Communications, Fibertech Networks,
# possibly others). Initial outage accurred at about 03:15 EDT today.
# Current restoration estimate is 15:00-16:00 EDT today.

#
# jms
#
#
#
# _______________________________________________
# Outages mailing list
# Outages-w8q5NolJmqtAfugRpC6u6w@public.gmane.org
# https://puck.nether.net/mailman/listinfo/outages
#

_______________________________________________
Outages mailing list
Outages-w8q5NolJmqtAfugRpC6u6w@public.gmane.org
https://puck.nether.net/mailman/listinfo/outages

headers

Jeremy Chadwick | 10 May 01:48

Fwd: Re: [outages] FW: Manhole fire, Pittsburgh PA

Anyone here speak English?  Or manage the list?  Or own pantaloons?
You get the idea.

-- 
| Jeremy Chadwick                                   jdc@... |
| UNIX Systems Administrator                http://jdc.koitsu.org/ |
| Mountain View, CA, US                                            |
| Making life hard for others since 1977.             PGP 4BD6C0CB |

----- Forwarded message from "Carol E. Sarver"
<sarver.carol@...> -----

> From: "Carol E. Sarver" <sarver.carol@...>
> To: Jeremy Chadwick <outages@...>
> Date: Wed, 9 May 2012 19:40:36 -0400
> Reply-To: "Carol E. Sarver" <cesarver@...>
> Subject: Re: [outages] FW: Manhole fire, Pittsburgh PA
> X-Mailer: Microsoft Outlook Express 6.00.2900.5931
> 
> I don't I'm suosed to be on this list.
> ----- Original Message ----- From: "Jeremy Chadwick"
> <outages@...>
> To: "Chris Wright" <cjwright@...>
> Cc: <outages@...>
> Sent: Wednesday, May 09, 2012 6:19 PM
> Subject: Re: [outages] FW: Manhole fire, Pittsburgh PA
> 
> 
> >It's affecting, bare minimum, Abovenet.  Given the relationship between
> >Abovenet and L3 (for physical transport), I'm not surprised they're both
> >affected.
> >
> >We were told initially it was "a fiber cut in Albany, NY", then
> >"a fiber event near New Platz, NY", and now "a fiber outage in Suffern,
> >NY".  So based on this, it's probably a fiber cut in Bolivia.
> >
> >-- 
> >| Jeremy Chadwick                                   jdc@... |
> >| UNIX Systems Administrator                http://jdc.koitsu.org/ |
> >| Mountain View, CA, US                                            |
> >| Making life hard for others since 1977.             PGP 4BD6C0CB |
> >
> >On Wed, May 09, 2012 at 06:08:59PM -0400, Chris Wright wrote:
> >>There is a fiber cut in Suffern, NY.  I'm not sure which carriers it is
> >>affecting.
> >>
> >>-Chris
> >>
> >>
> >>On Wed, May 9, 2012 at 5:54 PM, Jonathan Towne <jtowne@...> wrote:
> >>
> >>> Has anyone heard anything else about this?  I've been having issues on
> >>> Level3
> >>> in upstate NY all day.
> >>>
> >>> -- Jonathan Towne
> >>>
> >>> On Wed, May 09, 2012 at 12:52:19PM -0500, Frank Bulk scribbled:
> >>> # FYI
> >>> #
> >>> # -----Original Message-----
> >>> # From: Justin M. Streiner [mailto:streiner@...]
> >>> # Sent: Wednesday, May 09, 2012 11:37 AM
> >>> # To: nanog@...
> >>> # Subject: Manhole fire, Pittsburgh PA
> >>> #
> >>> # There was a manhole/electrical vault fire this morning in
> >>Pittsburgh > on
> >>> # Fifth Ave in the Oakland area.  Numerous telcos/dark fiber providers
> >>> # affected (Level3, Verizon, DQE Communications, Fibertech Networks,
> >>> # possibly others).  Initial outage accurred at about 03:15 EDT today.
> >>> # Current restoration estimate is 15:00-16:00 EDT today.
> >>> #
> >>> # jms
> >>> #
> >>> #
> >>> #
> >>> # _______________________________________________
> >>> # Outages mailing list
> >>> # Outages@...
> >>> # https://puck.nether.net/mailman/listinfo/outages
> >>> #
> >>> _______________________________________________
> >>> Outages mailing list
> >>> Outages@...
> >>> https://puck.nether.net/mailman/listinfo/outages
> >>>
> >
> >>_______________________________________________
> >>Outages mailing list
> >>Outages@...
> >>https://puck.nether.net/mailman/listinfo/outages
> >
> >_______________________________________________
> >Outages mailing list
> >Outages@...
> >https://puck.nether.net/mailman/listinfo/outages
> >

----- End forwarded message -----

headers

Network Operations | 8 May 04:49

[outages] myhosting.com down

I have a VPS that lost connectivity around 10:07pm EDT.  I cannot access their main web site or their
management site either.  I am guessing they lost something on their core network.

11  ix-21-0.tcore2.aeq-ashburn.as6453.net (216.6.87.65)  55.261 ms  55.238 ms  54.771 ms
12  if-3-2.tcore2.njy-newark.as6453.net (216.6.87.10)  79.472 ms  79.164 ms  79.678 ms
13  if-2-2.tcore1.njy-newark.as6453.net (66.198.70.1)  79.585 ms  79.830 ms  77.996 ms
14  if-3-0-0.mcore3.ttt-scarborough.as6453.net (216.6.98.1)  3714.893 ms
    if-12-0-0.mcore3.ttt-scarborough.as6453.net (216.6.98.33)  78.635 ms  78.258 ms
15  if-1-0-0.core4.tnk-toronto.as6453.net (63.243.172.1)  78.768 ms  78.559 ms  78.556 ms
16  * * *

Jim Lohiser
Imagine Net, Inc.

headers

david raistrick | 3 May 21:46

[outages] weather.com issues - contacts?


Folks,

Anyone have any ops contacts over @ weather.com ?  We use a service they
provide us (complete with custom hostnames) and...well...it's timing out a
lot.

As usual with these sorts of things, it's a years-gone-by setup done by
people long gone....

[note that it's definitely application side, anyway, not network. ;) but
still breaking for us. ]

--
david raistrick        http://www.netmeister.org/news/learn2quote.html
drais@...             http://www.expita.com/nomime.html

40	May 2012
94	April 2012
42	March 2012
88	February 2012
87	January 2012
79	December 2011
150	November 2011
144	October 2011
143	September 2011
134	August 2011
88	July 2011
92	June 2011
91	May 2011
51	April 2011
126	March 2011
59	February 2011
48	January 2011
109	December 2010
122	November 2010
44	October 2010
131	September 2010
69	August 2010
73	July 2010
49	June 2010
95	May 2010
66	April 2010
86	March 2010
42	February 2010
24	January 2010
93	December 2009
71	November 2009
88	October 2009
70	September 2009
117	August 2009
33	July 2009
62	June 2009
75	May 2009
69	April 2009
17	March 2009
19	February 2009
26	January 2009
61	December 2008
45	November 2008
20	October 2008
73	September 2008
13	August 2008
77	July 2008
260	June 2008
40	May 2008
11	March 2008

Mon	Tue	Wed	Thu	Fri	Sat	Sun

	1	2	3	4	5	6
7	8	9	10	11	12	13
14	15	16	17	18	19	20
21	22	23	24	25	26	27
28	29	30	31