Geoffrey Mina | 17 Apr 18:42 2014

[outages] Level(3) Issues this morning

Is anyone aware of a widespread Level(3) outage this morning shortly before 10AM Mountain?  Our traces were dying at the edge of both twtelecom and Comcast.

 

 

<div>
<div class="WordSection1">
<p class="MsoNormal">Is anyone aware of a widespread Level(3) outage this morning shortly before 10AM Mountain?&nbsp; Our traces were dying at the edge of both twtelecom and Comcast.<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
</div>
</div>
Grant Ridder | 17 Apr 04:31 2014
Picon

[outages] bluehost.com hosting

Hi Everyone,

I have a couple sites hosted on bluehost.com and a couple are down right now but not all of them.  BlueHosts online chat system is down and there is an automated message on their help line saying they are having network issues.

-Grant
<div><div dir="ltr">Hi Everyone,<div><br></div>
<div>I have a couple sites hosted on <a href="http://bluehost.com">bluehost.com</a> and a couple are down right now but not all of them. &nbsp;BlueHosts online chat system is down and there is an automated message on their help line saying they are having network issues.</div>
<div><br></div>
<div>-Grant</div>
</div></div>
John Kinsella | 16 Apr 21:35 2014

[outages] Level3/LT fiber break in NYC

Just got on update on our Lightower ticket for the outage since 11:20am eastern:

Lightower has identified damage near Park Avenue and 18th (2 blocks south of 300 Blvd). A Backhoe onsite has dug up Level 3 owned conduit which contains Lightower fiber. Level 3 has technicians and civil contractors headed to the location for repair. Lightower is investigating all opportunities to repair this cable ahead of the Level 3 conduit repair. Lightower provided fiber contractors are mobilizing to the location to assist with this investigation. No ETTR At this time.

<div>Just got on update on our Lightower ticket for the outage since 11:20am eastern:<div><br></div>
<div>
<span>Lightower has identified damage near Park Avenue and 18th (2 blocks south of 300 Blvd). A Backhoe onsite has dug up Level 3 owned conduit which contains Lightower fiber. Level 3 has technicians and civil contractors headed to the location for repair. Lightower is investigating all opportunities to repair this cable ahead of the Level 3 conduit repair. Lightower provided fiber contractors are mobilizing to the location to assist with this
  investigation. No ETTR At this time.</span><br><br>
</div>
</div>
John Kinsella | 16 Apr 17:48 2014

[outages] Lightower issues in NJ

Looks like lightower’s having issues in NJ, sitting in a hold queue to see what’s up.

From where I am - 

 1. 10.0.1.49                                                                  0.0%    14    1.4   5.8   1.1  20.9   5.7
 2. 10.0.24.73                                                                 0.0%    14    2.9   4.1   1.2   9.4   2.4
 3. 25.66.208.web-pass.com                                                     0.0%    14    3.5   7.2   1.4  17.5   5.0
 4. 2-1.760.core.fre.web-pass.com                                              0.0%    14    7.5   7.3   2.7  18.7   4.5
 5. ge5-8.core1.fmt1.he.net                                                    0.0%    14    9.6  13.0   3.0  44.9  11.8
 6. 10ge11-1.core1.sjc2.he.net                                                 0.0%    14    3.3  10.4   3.3  34.2   9.5
 7. 10ge14-2.core1.nyc4.he.net                                                 0.0%    14   73.3  80.1  73.1  93.8   6.7
 8. 10ge4-1.core1.nyc5.he.net                                                  0.0%    14   78.6  81.5  73.5  89.9   5.8
 9. ???

Ray Van Dolson | 15 Apr 22:04 2014

[outages] USDA Outage (DNS?)

Seems that usda.gov's primary DNS servers are not responding.

From glue (b.gov-servers.net):

ns1.usda.gov.           86400   IN      A       199.141.126.202
ns2.usda.gov.           86400   IN      A       199.141.126.206

This is obviously impacting many things as caches expire.

Ray
Elijah Savage | 11 Apr 17:00 2014

Re: [outages] Any Verizon BGP AS 701 Issues

Exact same thing here. I have not reintroduced the circuit on our side yet but hear things are starting to restore.

----- Original Message -----
From: "David Hiers" <David.Hiers@...>
To: "Elijah Savage" <esavage@...>, outages@...
Sent: Friday, April 11, 2014 10:53:04 AM
Subject: RE: Any Verizon BGP AS 701 Issues

BGP stayed up, but much traffic inside VZN died.

Coming back up for us now.

David

-----Original Message-----
From: Outages [mailto:outages-bounces@...] On Behalf Of
Elijah Savage
Sent: Friday, April 11, 2014 07:48
To: outages@...
Subject: [outages] Any Verizon BGP AS 701 Issues

We have lost all routing to them and had to remove their peer from our connectivity?
_______________________________________________
Outages mailing list
Outages@...
https://puck.nether.net/mailman/listinfo/outages

This message and any attachments are intended only for the use of the addressee and may contain information
that is privileged and confidential. If the reader of the message is not the intended recipient or an
authorized representative of the intended recipient, you are hereby notified that any dissemination of
this communication is strictly prohibited. If you have received this communication in error, please
notify us immediately by e-mail and delete the message and any attachments from your system.
Elijah Savage | 11 Apr 16:47 2014

[outages] Any Verizon BGP AS 701 Issues

We have lost all routing to them and had to remove their peer from our connectivity?
Michael Peterman | 9 Apr 01:34 2014

[outages] Charter Outage in Madison/Whitewater

Anyone else seeing down cable modems in the Whitewater and Fitchburg area. Started to happen around 30 minutes ago.
 
 
Michael Peterman
 
 
 
<div>
<span>
<div>Anyone else seeing down cable modems in the Whitewater and Fitchburg area. Started to happen around 30 minutes ago. </div>
<div>&nbsp;</div>
<div>&nbsp;</div>
<div>Michael Peterman</div>
<div>&nbsp;</div>
<div>&nbsp;</div>
<div>&nbsp;</div>
</span>
</div>
Mitchell Kuch | 7 Apr 19:54 2014

[outages] AT&T's U-verse

We are seeing service degradation with AT&T's U-verse in the Detroit
and Ann Arbor metros.

It doesn't look like it is just us:
http://mashable.com/2014/04/07/att-u-verse-outage/
Neil Ticktin | 4 Apr 22:17 2014

[outages] Crazy amts of spoofing?

Anyone seeing crazy amounts of spoofing that are going out to what looks like address book entries?

In other words, not from your client, not from your server, but spoofing an email address that's yours, and going to recipients that look like your address book (e.g., grouped by last name and to people you know).

I don't want to point fingers, and I have no evidence of this in any way, but it almost looks like a social network site, that may have access to address book entries, got hit -- and someone is spoofing big time.

The other option would be a Mac virus hitting address book entries.

Anyone seeing anything this?

Neil
<div><div dir="ltr">Anyone seeing crazy amounts of spoofing that are going out to what looks like address book entries?<div><br></div>
<div>In other words, not from your client, not from your server, but spoofing an email address that's yours, and going to recipients that look like your address book (e.g., grouped by last name and to people you know).</div>
<div><br></div>
<div>I don't want to point fingers, and I have no evidence of this in any way, but it almost looks like a social network site, that may have access to address book entries, got hit -- and someone is spoofing big time.</div>
<div><br></div>
<div>The other option would be a Mac virus hitting address book entries.</div>
<div><br></div>
<div>Anyone seeing anything this?</div>
<div><br></div>
<div>Neil</div>
</div></div>
Bill Fehring | 4 Apr 06:44 2014

[outages] Level(3) Anycast DNS down in SF Bay Area?

So I've tried from a few different networks in the SF Bay Area and Level(3)'s anycast DNS servers seem to be having trouble resolving anything. 

4.2.2.1 through 6, and 209.244.0.3 and 4

% dig <at> 4.2.2.2 google.com

; <<>> DiG 9.5.1-P3 <<>> <at> 4.2.2.2 google.com
; (1 server found)
;; global options:  printcmd
;; connection timed out; no servers could be reached

They do still seem to have root hints though.

% dig <at> 4.2.2.2

; <<>> DiG 9.5.1-P3 <<>> <at> 4.2.2.2
; (1 server found)
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 61287
;; flags: qr rd ra; QUERY: 1, ANSWER: 13, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;. IN NS

;; ANSWER SECTION:
. 18209 IN NS d.root-servers.net.
. 18209 IN NS i.root-servers.net.
. 18209 IN NS l.root-servers.net.
. 18209 IN NS h.root-servers.net.
. 18209 IN NS b.root-servers.net.
. 18209 IN NS c.root-servers.net.
. 18209 IN NS k.root-servers.net.
. 18209 IN NS m.root-servers.net.
. 18209 IN NS e.root-servers.net.
. 18209 IN NS g.root-servers.net.
. 18209 IN NS f.root-servers.net.
. 18209 IN NS a.root-servers.net.
. 18209 IN NS j.root-servers.net.

;; Query time: 1 msec
;; SERVER: 4.2.2.2#53(4.2.2.2)
;; WHEN: Thu Apr  3 21:38:51 2014
;; MSG SIZE  rcvd: 228

--Bill Fehring
<div><div dir="ltr">
<div>
<div dir="ltr">So I've tried from a few different networks in the SF Bay Area and Level(3)'s anycast DNS servers seem to be having trouble resolving anything.&nbsp;</div>
<div dir="ltr"><br></div>
<div>

4.2.2.1 through 6, and 209.244.0.3 and 4</div>
</div>
<div dir="ltr"><br></div>
<div dir="ltr">
<div dir="ltr">% dig  <at> <a href="http://4.2.2.2">4.2.2.2</a> <a href="http://google.com">google.com</a>
</div>
<div dir="ltr"><br></div>

<div dir="ltr">; &lt;&lt;&gt;&gt; DiG 9.5.1-P3 &lt;&lt;&gt;&gt;  <at> <a href="http://4.2.2.2">4.2.2.2</a> <a href="http://google.com">google.com</a>
</div>
<div dir="ltr">; (1 server found)</div>
<div dir="ltr">;; global options: &nbsp;printcmd</div>

<div dir="ltr">;; connection timed out; no servers could be reached</div>
</div>
<div dir="ltr"><br></div>
<div>They do still seem to have root hints though.</div>
<div><br></div>
<div>
<div>% dig  <at> <a href="http://4.2.2.2">4.2.2.2</a>
</div>

<div><br></div>
<div>; &lt;&lt;&gt;&gt; DiG 9.5.1-P3 &lt;&lt;&gt;&gt;  <at> <a href="http://4.2.2.2">4.2.2.2</a>
</div>
<div>; (1 server found)</div>
<div>;; global options: &nbsp;printcmd</div>
<div>;; Got answer:</div>
<div>;; -&gt;&gt;HEADER&lt;&lt;- opcode: QUERY, status: NOERROR, id: 61287</div>

<div>;; flags: qr rd ra; QUERY: 1, ANSWER: 13, AUTHORITY: 0, ADDITIONAL: 0</div>
<div><br></div>
<div>;; QUESTION SECTION:</div>
<div>;.<span class="">				</span>IN<span class="">	</span>NS</div>

<div><br></div>
<div>;; ANSWER SECTION:</div>
<div>.<span class="">			</span>18209<span class="">	</span>IN<span class="">	</span>NS<span class="">	</span><a href="http://d.root-servers.net">d.root-servers.net</a>.</div>

<div>.<span class="">			</span>18209<span class="">	</span>IN<span class="">	</span>NS<span class="">	</span><a href="http://i.root-servers.net">i.root-servers.net</a>.</div>

<div>.<span class="">			</span>18209<span class="">	</span>IN<span class="">	</span>NS<span class="">	</span><a href="http://l.root-servers.net">l.root-servers.net</a>.</div>

<div>.<span class="">			</span>18209<span class="">	</span>IN<span class="">	</span>NS<span class="">	</span><a href="http://h.root-servers.net">h.root-servers.net</a>.</div>

<div>.<span class="">			</span>18209<span class="">	</span>IN<span class="">	</span>NS<span class="">	</span><a href="http://b.root-servers.net">b.root-servers.net</a>.</div>

<div>.<span class="">			</span>18209<span class="">	</span>IN<span class="">	</span>NS<span class="">	</span><a href="http://c.root-servers.net">c.root-servers.net</a>.</div>

<div>.<span class="">			</span>18209<span class="">	</span>IN<span class="">	</span>NS<span class="">	</span><a href="http://k.root-servers.net">k.root-servers.net</a>.</div>

<div>.<span class="">			</span>18209<span class="">	</span>IN<span class="">	</span>NS<span class="">	</span><a href="http://m.root-servers.net">m.root-servers.net</a>.</div>

<div>.<span class="">			</span>18209<span class="">	</span>IN<span class="">	</span>NS<span class="">	</span><a href="http://e.root-servers.net">e.root-servers.net</a>.</div>

<div>.<span class="">			</span>18209<span class="">	</span>IN<span class="">	</span>NS<span class="">	</span><a href="http://g.root-servers.net">g.root-servers.net</a>.</div>

<div>.<span class="">			</span>18209<span class="">	</span>IN<span class="">	</span>NS<span class="">	</span><a href="http://f.root-servers.net">f.root-servers.net</a>.</div>

<div>.<span class="">			</span>18209<span class="">	</span>IN<span class="">	</span>NS<span class="">	</span><a href="http://a.root-servers.net">a.root-servers.net</a>.</div>

<div>.<span class="">			</span>18209<span class="">	</span>IN<span class="">	</span>NS<span class="">	</span><a href="http://j.root-servers.net">j.root-servers.net</a>.</div>

<div><br></div>
<div>;; Query time: 1 msec</div>
<div>;; SERVER: 4.2.2.2#53(4.2.2.2)</div>
<div>;; WHEN: Thu Apr &nbsp;3 21:38:51 2014</div>
<div>;; MSG SIZE &nbsp;rcvd: 228</div>
</div>
<div dir="ltr"><br></div>
<div>--Bill Fehring</div>

</div></div>

Gmane