Michael Loftis via Outages | 20 Oct 21:57 2014

[outages] Comcast - Seattle (maybe other areas)

Having serious issues getting outside of Comcast's network starting
around 1250 Pacific time from Seattle area.

--

-- 

"Genius might be described as a supreme capacity for getting its possessors
into trouble of all kinds."
-- Samuel Butler
James Cornman via Outages | 17 Oct 22:07 2014

Re: [outages] Equinix Chicago exchange

I tracked it down to an offending device belonging to 123.net. Still haven't received any peep out of Equinix support.


Filtered it..

mac access-list extended equinix-bad-neighbors
 deny   host 8478.ac4e.90be any
 permit any any
!
interface GigabitEthernet0/50

  mac access-group equinix-bad-neighbors in


On Fri, Oct 17, 2014 at 4:06 PM, Nicholas Schmidt <nschmidt-Qg2TNkI2BF61Z/+hSey0Gg@public.gmane.org> wrote:
Seeing similar. 

Nicholas Schmidt
MediaMath

From: James Cornman via Outages <outages <at> outages.org>
Reply-To: James Cornman <james <at> atlanticmetro.net>
Date: Friday, October 17, 2014 at 3:50 PM
To: "outages-w8q5NolJmqtAfugRpC6u6w@public.gmane.org" <outages-w8q5NolJmqtAfugRpC6u6w@public.gmane.org>
Subject: [outages] Equinix Chicago exchange

Hey all,

Anyone seeing a large influx of broadcast traffic since a little after 3pm EDT on the Equinix exchange in Chicago?

Opening a ticket with Equinix, but just curious if any others have seen anything.



--

James Cornman

Chief Technology Officer
jcornman-q63CSc06zXuqyuYpCWRUfF6hYfS7NtTn@public.gmane.org
212.792.9950 - ext 101

Atlantic Metro Communications, Inc.

4 Century Drive, Parsippany NJ  07054


Data Centers • Managed Hosting • Internet Access • Cloud Computing
Follow  <at> atlanticmetro on Twitter to learn more about daily specials.
www.atlanticmetro.net




--

James Cornman

Chief Technology Officer
jcornman-q63CSc06zXuqyuYpCWRUfF6hYfS7NtTn@public.gmane.org
212.792.9950 - ext 101

Atlantic Metro Communications, Inc.

4 Century Drive, Parsippany NJ  07054


Data Centers • Managed Hosting • Internet Access • Cloud Computing
Follow  <at> atlanticmetro on Twitter to learn more about daily specials.
www.atlanticmetro.net

<div>
<div dir="ltr">I tracked it down to an offending device belonging to <a href="http://123.net">123.net</a>. Still haven't received any peep out of Equinix support.<div><br></div>
<div><br></div>
<div>Filtered it..</div>
<div><br></div>
<div>
<div>mac access-list extended equinix-bad-neighbors</div>
<div>&nbsp;deny &nbsp; host 8478.ac4e.90be any</div>
<div>&nbsp;permit any any</div>
<div>!</div>
<div>interface GigabitEthernet0/50&#8232;</div>
<div>&nbsp; mac access-group equinix-bad-neighbors in</div>
</div>
<div><br></div>
</div>
<div class="gmail_extra">
<br><div class="gmail_quote">On Fri, Oct 17, 2014 at 4:06 PM, Nicholas Schmidt <span dir="ltr">&lt;<a href="mailto:nschmidt@..." target="_blank">nschmidt@...</a>&gt;</span> wrote:<br><blockquote class="gmail_quote">

<div>
<div>Seeing similar.&nbsp;</div>
<div><br></div>
<div>Nicholas Schmidt</div>
<div>MediaMath</div>
<div><br></div>
<span>
<div>
<span>From: </span>James Cornman via Outages &lt;<a href="mailto:outages@..." target="_blank">outages <at> outages.org</a>&gt;<br><span>Reply-To: </span>James Cornman &lt;<a href="mailto:james@..." target="_blank">james <at> atlanticmetro.net</a>&gt;<br><span>Date: </span>Friday, October 17, 2014 at 3:50 PM<br><span>To: </span>"<a href="mailto:outages <at> outages.org" target="_blank">outages@...</a>" &lt;<a href="mailto:outages@..." target="_blank">outages@...</a>&gt;<br><span>Subject: </span>[outages] Equinix Chicago exchange<br>
</div>
<div><div class="h5">
<div><br></div>
<div>
<div>
<div dir="ltr">Hey all,
<div><br></div>
<div>Anyone seeing a large influx of broadcast traffic since a little after 3pm EDT on the Equinix exchange in Chicago?</div>
<div><br></div>
<div>Opening a ticket with Equinix, but just curious if any others have seen anything.</div>
<div><br></div>
<div>
<br clear="all"><div><br></div>
-- <br><span>
<p><span>James Cornman</span></p>
<p><span>Chief Technology Officer<br></span><span><a href="mailto:jcornman <at> atlanticmetro.net" target="_blank"><span>jcornman@...</span></a><br><a href="tel:212.792.9950" value="+12127929950" target="_blank">212.792.9950</a> - ext 101<br><span><br>
Atlantic Metro Communications, Inc.</span></span></p>
<p><span>4 Century Drive, Parsippany NJ&nbsp; 07054</span></p>
<p><span><br></span><span>Data Centers &bull; Managed Hosting &bull; Internet Access &bull; Cloud Computing</span><span><br></span><span>Follow&nbsp;</span><span><a href="http://www.twitter.com/atlanticmetro" target="_blank"><span> <at> atlanticmetro</span></a></span><span>&nbsp;on
 Twitter to learn more about daily specials.</span><span><br><a href="http://www.atlanticmetro.net/" target="_blank"><span>www.atlanticmetro.net</span></a></span></p>
</span>
</div>
</div>
</div>
</div>
</div></div></span>
</div>

</blockquote>
</div>
<br><br clear="all"><div><br></div>-- <br><span><p><span>James Cornman</span></p>
<p><span>Chief Technology Officer<br></span><span><a href="mailto:jcornman@..." target="_blank"><span>jcornman@...</span></a><br>212.792.9950 - ext 101<br><span><br>Atlantic Metro Communications, Inc.</span></span></p>
<p><span>4 Century Drive, Parsippany NJ&nbsp; 07054</span></p>
<p><span><br></span><span>Data Centers &bull; Managed Hosting &bull; Internet Access &bull; Cloud Computing</span><span><br></span><span>Follow&nbsp;</span><span><a href="http://www.twitter.com/atlanticmetro" target="_blank"><span> <at> atlanticmetro</span></a></span><span>&nbsp;on Twitter to learn more about daily specials.</span><span><br><a href="http://www.atlanticmetro.net/" target="_blank"><span>www.atlanticmetro.net</span></a></span></p></span>
</div>
</div>
James Cornman via Outages | 17 Oct 21:50 2014

[outages] Equinix Chicago exchange

Hey all,

Anyone seeing a large influx of broadcast traffic since a little after 3pm EDT on the Equinix exchange in Chicago?

Opening a ticket with Equinix, but just curious if any others have seen anything.



--

James Cornman

Chief Technology Officer
jcornman-q63CSc06zXuqyuYpCWRUfF6hYfS7NtTn@public.gmane.org
212.792.9950 - ext 101

Atlantic Metro Communications, Inc.

4 Century Drive, Parsippany NJ  07054


Data Centers • Managed Hosting • Internet Access • Cloud Computing
Follow  <at> atlanticmetro on Twitter to learn more about daily specials.
www.atlanticmetro.net

<div><div dir="ltr">Hey all,<div><br></div>
<div>Anyone seeing a large influx of broadcast traffic since a little after 3pm EDT on the Equinix exchange in Chicago?</div>
<div><br></div>
<div>Opening a ticket with Equinix, but just curious if any others have seen anything.</div>
<div><br></div>
<div>
<br clear="all"><div><br></div>-- <br><span><p><span>James Cornman</span></p>
<p><span>Chief Technology Officer<br></span><span><a href="mailto:jcornman <at> atlanticmetro.net" target="_blank"><span>jcornman@...</span></a><br>212.792.9950 - ext 101<br><span><br>Atlantic Metro Communications, Inc.</span></span></p>
<p><span>4 Century Drive, Parsippany NJ&nbsp; 07054</span></p>
<p><span><br></span><span>Data Centers &bull; Managed Hosting &bull; Internet Access &bull; Cloud Computing</span><span><br></span><span>Follow&nbsp;</span><span><a href="http://www.twitter.com/atlanticmetro" target="_blank"><span> <at> atlanticmetro</span></a></span><span>&nbsp;on Twitter to learn more about daily specials.</span><span><br><a href="http://www.atlanticmetro.net/" target="_blank"><span>www.atlanticmetro.net</span></a></span></p></span>
</div>
</div></div>
Morgan Miskell via Outages | 17 Oct 15:51 2014

[outages] Windstream Long Distance

Windstream Long Distance appears to be down for us....might be a decent
size issue for them since I've been on hold to report the issue for 15
minutes which I interpret to mean that their call center is getting
slammed!
--

-- 
Morgan A. Miskell
CaroNet Data Centers
704-643-8330 x206
----------------------------------------------------------------------------
The information contained in this e-mail is confidential and is intended
only for the named recipient(s). If you are not the intended recipient
you must not copy, distribute, or take any action or reliance on it. If
you have received this e-mail in error, please notify the sender. Any
unauthorized disclosure of the information contained in this e-mail is
strictly prohibited.
----------------------------------------------------------------------------

Rodrick Brown via Outages | 16 Oct 06:05 2014

[outages] delays on the interweb?

I'm seeing delays and outages hitting random sites anyone else noticing the same? 

vimeo
etc.. ? 
<div><div dir="ltr">I'm seeing delays and outages hitting random sites anyone else noticing the same?&nbsp;<div><br></div>
<div><a href="http://news.ycombinator.com">news.ycombinator.com</a></div>
<div>vimeo</div>
<div>etc.. ?&nbsp;</div>
</div></div>
Vipul Desai via Outages | 16 Oct 05:53 2014

[outages] Century Link/Qwest Outage


Does anyone else experience routing issue for websites hosted on CDN from CenturyLink's edge router?

jfk-brdr-04.inet.qwest.net [67.14.5.98] 
 63.146.26.2

I can't open multiple websites if routing goes through their JFK edge router. It's been happening over 24 hours and no resolution from Century Link/Qwest.
Same sites will open if route goes through dcx2-edge-01.inet.qwest.net [67.14.28.66] or nyc-edge-05.inet.qwest.net [205.171.134.10] or chp-brdr-03.inet.qwest.net [67.14.8.194] edge router.

Thank you,
Vipul Desai

 
<div><div>
<div class=""><br class=""></div>
<div class="">Does anyone else experience routing issue for websites hosted on CDN from CenturyLink's edge router?</div>
<div class=""><br class=""></div>
<div class="">
<span class="">jfk-brdr-04.inet.qwest.net [67.14.5.98]</span>&nbsp;</div>
<div class="">
<span class="">&nbsp;63.146.26.2</span><br class="">
</div>
<div class=""><br class=""></div>
<div class="">I can't open multiple websites if routing goes through their JFK edge router. It's been happening over 
 24 hours and no resolution from Century Link/Qwest.</div>
<div class="">Same sites will open if route goes through&nbsp;dcx2-edge-01.inet.qwest.net [67.14.28.66] or&nbsp;<span class="">nyc-edge-05.inet.qwest.net [205.171.134.10] or&nbsp;chp-brdr-03.inet.qwest.net [67.14.8.194] edge router.</span>
</div>
<div class=""><br class=""></div>
<div class="">Thank you,</div>
<div class="">Vipul Desai
 </div>
<div class=""><br class=""></div>
<div class="">&nbsp;</div>
</div></div>
Sajal Kayan via Outages | 13 Oct 13:25 2014

[outages] Google Public DNS is down in south-east asia

Google Public DNS just stopped working for me (and many others locally).

xxx <at> xxx:~# dig google.com <at> 8.8.8.8

; <<>> DiG 9.8.4-rpz2+rl005.12-P1 <<>> google.com <at> 8.8.8.8
;; global options: +cmd
;; connection timed out; no servers could be reached
xxx <at> xxx:~# 

Traces from few locations.

True ADSL (Thailand)

xxx <at> xxx:~# mtr --report 8.8.8.8
HOST: xxx                         Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 192.168.1.1                0.0%    10   30.5  17.4   0.8  57.5  22.5
  2.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
  3.|-- 10.169.212.9               0.0%    10   23.3  24.0  23.3  25.0   0.6
  4.|-- 10.100.50.67               0.0%    10   24.0  24.3  23.7  25.1   0.5
  5.|-- 58.97.121.81               0.0%    10   25.4  25.1  24.5  25.6   0.4
  6.|-- 61.91.213.177              0.0%    10   24.6  25.0  24.6  25.6   0.3
  7.|-- 61.91.213.36               0.0%    10   26.1  25.9  25.4  26.4   0.3
  8.|-- 61.91.213.130              0.0%    10   25.5  26.0  25.1  27.3   0.6
  9.|-- 113.21.242.53              0.0%    10   27.0  26.5  25.1  28.2   1.0
 10.|-- 72.14.210.155              0.0%    10   49.1  52.2  48.1  80.4   9.9
 11.|-- 209.85.242.244             0.0%    10   54.6  53.6  50.0  60.9   4.1
 12.|-- 72.14.239.201              0.0%    10   85.6  85.7  80.2 100.8   6.5
 13.|-- 209.85.250.11              0.0%    10  107.2 107.7 106.7 109.7   0.9
 14.|-- 209.85.243.21              0.0%    10  109.6 110.4 109.5 111.3   0.6
 15.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
xxx <at> xxx:~# 


Digital Ocean (Singapore)

xxx <at> xxx:~# mtr --report 8.8.8.8
Start: Mon Oct 13 07:18:28 2014
HOST: xxx                  Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- a.b.c.d            0.0%    10    0.6   1.6   0.5   6.0   2.1
  2.|-- 103.253.144.241            0.0%    10    0.4   0.5   0.4   0.6   0.0
  3.|-- 103.16.102.64              0.0%    10    1.8   2.0   1.7   3.4   0.3
  4.|-- 66.249.95.122              0.0%    10    2.1   2.1   2.0   2.3   0.0
  5.|-- 209.85.241.149             0.0%    10   54.5  56.1  54.3  61.8   2.5
  6.|-- 72.14.237.179              0.0%    10   58.0  58.1  57.9  59.2   0.0
  7.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
xxx <at> xxx:~# 


EC2 (Singapore)

xxx <at> xxx:~$ mtr --report 8.8.8.8
HOST: xxx            Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- ip-10-a-b-2.ap-southea  0.0%    10    0.6   0.6   0.3   1.5   0.4
  2.|-- ip-10-1-16-0.ap-southeast  0.0%    10    6.1   1.6   0.5   6.1   1.8
  3.|-- 100.64.12.131              0.0%    10   17.8   3.8   1.0  17.8   5.7
  4.|-- ec2-175-41-128-232.ap-sou  0.0%    10    0.8   0.6   0.5   0.9   0.1
  5.|-- 203.83.223.230             0.0%    10    0.6   1.0   0.4   2.3   0.7
  6.|-- 203.83.223.14              0.0%    10    1.5   1.6   1.4   2.4   0.3
  7.|-- 203.83.223.57              0.0%    10    1.7   1.8   1.5   2.4   0.2
  8.|-- 198.32.141.140             0.0%    10    1.5   1.7   1.4   3.2   0.5
  9.|-- 66.249.95.122              0.0%    10    2.9   5.4   2.2  14.7   4.8
 10.|-- 209.85.241.149             0.0%    10   54.2  57.7  54.0  76.5   7.6
 11.|-- 209.85.243.23              0.0%    10   57.9  60.1  57.7  79.4   6.8
 12.|-- 72.14.238.8                0.0%    10   58.9  59.3  58.8  62.5   1.1
 13.|-- 72.14.237.179              0.0%    10   63.0  65.3  62.8  79.2   5.4
 14.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
xxx <at> xxx:~$ 
<div><div dir="ltr">
<div>Google Public DNS just stopped working for me (and many others locally).</div>
<div><br></div>
<div>
<div>xxx <at> xxx:~# dig <a href="http://google.com">google.com</a>  <at> <a href="http://8.8.8.8">8.8.8.8</a>
</div>
<div><br></div>
<div>; &lt;&lt;&gt;&gt; DiG 9.8.4-rpz2+rl005.12-P1 &lt;&lt;&gt;&gt; <a href="http://google.com">google.com</a>  <at> <a href="http://8.8.8.8">8.8.8.8</a>
</div>
<div>;; global options: +cmd</div>
<div>;; connection timed out; no servers could be reached</div>
<div>xxx <at> xxx:~#&nbsp;</div>
</div>
<div><br></div>
<div>Traces from few locations.</div>
<div><br></div>
<div>True ADSL (Thailand)</div>
<div><br></div>
<div>xxx <at> xxx:~# mtr --report 8.8.8.8</div>
<div>HOST: xxx &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Loss% &nbsp; Snt &nbsp; Last &nbsp; Avg &nbsp;Best &nbsp;Wrst StDev</div>
<div>&nbsp; 1.|-- 192.168.1.1 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.0% &nbsp; &nbsp;10 &nbsp; 30.5 &nbsp;17.4 &nbsp; 0.8 &nbsp;57.5 &nbsp;22.5</div>
<div>&nbsp; 2.|-- ??? &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 100.0 &nbsp; &nbsp;10 &nbsp; &nbsp;0.0 &nbsp; 0.0 &nbsp; 0.0 &nbsp; 0.0 &nbsp; 0.0</div>
<div>&nbsp; 3.|-- 10.169.212.9 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 0.0% &nbsp; &nbsp;10 &nbsp; 23.3 &nbsp;24.0 &nbsp;23.3 &nbsp;25.0 &nbsp; 0.6</div>
<div>&nbsp; 4.|-- 10.100.50.67 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 0.0% &nbsp; &nbsp;10 &nbsp; 24.0 &nbsp;24.3 &nbsp;23.7 &nbsp;25.1 &nbsp; 0.5</div>
<div>&nbsp; 5.|-- 58.97.121.81 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 0.0% &nbsp; &nbsp;10 &nbsp; 25.4 &nbsp;25.1 &nbsp;24.5 &nbsp;25.6 &nbsp; 0.4</div>
<div>&nbsp; 6.|-- 61.91.213.177 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.0% &nbsp; &nbsp;10 &nbsp; 24.6 &nbsp;25.0 &nbsp;24.6 &nbsp;25.6 &nbsp; 0.3</div>
<div>&nbsp; 7.|-- 61.91.213.36 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 0.0% &nbsp; &nbsp;10 &nbsp; 26.1 &nbsp;25.9 &nbsp;25.4 &nbsp;26.4 &nbsp; 0.3</div>
<div>&nbsp; 8.|-- 61.91.213.130 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.0% &nbsp; &nbsp;10 &nbsp; 25.5 &nbsp;26.0 &nbsp;25.1 &nbsp;27.3 &nbsp; 0.6</div>
<div>&nbsp; 9.|-- 113.21.242.53 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.0% &nbsp; &nbsp;10 &nbsp; 27.0 &nbsp;26.5 &nbsp;25.1 &nbsp;28.2 &nbsp; 1.0</div>
<div>&nbsp;10.|-- 72.14.210.155 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.0% &nbsp; &nbsp;10 &nbsp; 49.1 &nbsp;52.2 &nbsp;48.1 &nbsp;80.4 &nbsp; 9.9</div>
<div>&nbsp;11.|-- 209.85.242.244 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 0.0% &nbsp; &nbsp;10 &nbsp; 54.6 &nbsp;53.6 &nbsp;50.0 &nbsp;60.9 &nbsp; 4.1</div>
<div>&nbsp;12.|-- 72.14.239.201 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.0% &nbsp; &nbsp;10 &nbsp; 85.6 &nbsp;85.7 &nbsp;80.2 100.8 &nbsp; 6.5</div>
<div>&nbsp;13.|-- 209.85.250.11 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.0% &nbsp; &nbsp;10 &nbsp;107.2 107.7 106.7 109.7 &nbsp; 0.9</div>
<div>&nbsp;14.|-- 209.85.243.21 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.0% &nbsp; &nbsp;10 &nbsp;109.6 110.4 109.5 111.3 &nbsp; 0.6</div>
<div>&nbsp;15.|-- ??? &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 100.0 &nbsp; &nbsp;10 &nbsp; &nbsp;0.0 &nbsp; 0.0 &nbsp; 0.0 &nbsp; 0.0 &nbsp; 0.0</div>
<div>xxx <at> xxx:~#&nbsp;</div>
<div><br></div>
<div><br></div>
<div>Digital Ocean (Singapore)</div>
<div><br></div>
<div>xxx <at> xxx:~# mtr --report 8.8.8.8</div>
<div>Start: Mon Oct 13 07:18:28 2014</div>
<div>HOST: xxx &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;Loss% &nbsp; Snt &nbsp; Last &nbsp; Avg &nbsp;Best &nbsp;Wrst StDev</div>
<div>&nbsp; 1.|-- a.b.c.d &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.0% &nbsp; &nbsp;10 &nbsp; &nbsp;0.6 &nbsp; 1.6 &nbsp; 0.5 &nbsp; 6.0 &nbsp; 2.1</div>
<div>&nbsp; 2.|-- 103.253.144.241 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.0% &nbsp; &nbsp;10 &nbsp; &nbsp;0.4 &nbsp; 0.5 &nbsp; 0.4 &nbsp; 0.6 &nbsp; 0.0</div>
<div>&nbsp; 3.|-- 103.16.102.64 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.0% &nbsp; &nbsp;10 &nbsp; &nbsp;1.8 &nbsp; 2.0 &nbsp; 1.7 &nbsp; 3.4 &nbsp; 0.3</div>
<div>&nbsp; 4.|-- 66.249.95.122 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.0% &nbsp; &nbsp;10 &nbsp; &nbsp;2.1 &nbsp; 2.1 &nbsp; 2.0 &nbsp; 2.3 &nbsp; 0.0</div>
<div>&nbsp; 5.|-- 209.85.241.149 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 0.0% &nbsp; &nbsp;10 &nbsp; 54.5 &nbsp;56.1 &nbsp;54.3 &nbsp;61.8 &nbsp; 2.5</div>
<div>&nbsp; 6.|-- 72.14.237.179 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.0% &nbsp; &nbsp;10 &nbsp; 58.0 &nbsp;58.1 &nbsp;57.9 &nbsp;59.2 &nbsp; 0.0</div>
<div>&nbsp; 7.|-- ??? &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 100.0 &nbsp; &nbsp;10 &nbsp; &nbsp;0.0 &nbsp; 0.0 &nbsp; 0.0 &nbsp; 0.0 &nbsp; 0.0</div>
<div>xxx <at> xxx:~#&nbsp;</div>
<div><br></div>
<div><br></div>
<div>EC2 (Singapore)</div>
<div><br></div>
<div>xxx <at> xxx:~$ mtr --report 8.8.8.8</div>
<div>HOST: xxx &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;Loss% &nbsp; Snt &nbsp; Last &nbsp; Avg &nbsp;Best &nbsp;Wrst StDev</div>
<div>&nbsp; 1.|-- ip-10-a-b-2.ap-southea &nbsp;0.0% &nbsp; &nbsp;10 &nbsp; &nbsp;0.6 &nbsp; 0.6 &nbsp; 0.3 &nbsp; 1.5 &nbsp; 0.4</div>
<div>&nbsp; 2.|-- ip-10-1-16-0.ap-southeast &nbsp;0.0% &nbsp; &nbsp;10 &nbsp; &nbsp;6.1 &nbsp; 1.6 &nbsp; 0.5 &nbsp; 6.1 &nbsp; 1.8</div>
<div>&nbsp; 3.|-- 100.64.12.131 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.0% &nbsp; &nbsp;10 &nbsp; 17.8 &nbsp; 3.8 &nbsp; 1.0 &nbsp;17.8 &nbsp; 5.7</div>
<div>&nbsp; 4.|-- ec2-175-41-128-232.ap-sou &nbsp;0.0% &nbsp; &nbsp;10 &nbsp; &nbsp;0.8 &nbsp; 0.6 &nbsp; 0.5 &nbsp; 0.9 &nbsp; 0.1</div>
<div>&nbsp; 5.|-- 203.83.223.230 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 0.0% &nbsp; &nbsp;10 &nbsp; &nbsp;0.6 &nbsp; 1.0 &nbsp; 0.4 &nbsp; 2.3 &nbsp; 0.7</div>
<div>&nbsp; 6.|-- 203.83.223.14 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.0% &nbsp; &nbsp;10 &nbsp; &nbsp;1.5 &nbsp; 1.6 &nbsp; 1.4 &nbsp; 2.4 &nbsp; 0.3</div>
<div>&nbsp; 7.|-- 203.83.223.57 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.0% &nbsp; &nbsp;10 &nbsp; &nbsp;1.7 &nbsp; 1.8 &nbsp; 1.5 &nbsp; 2.4 &nbsp; 0.2</div>
<div>&nbsp; 8.|-- 198.32.141.140 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 0.0% &nbsp; &nbsp;10 &nbsp; &nbsp;1.5 &nbsp; 1.7 &nbsp; 1.4 &nbsp; 3.2 &nbsp; 0.5</div>
<div>&nbsp; 9.|-- 66.249.95.122 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.0% &nbsp; &nbsp;10 &nbsp; &nbsp;2.9 &nbsp; 5.4 &nbsp; 2.2 &nbsp;14.7 &nbsp; 4.8</div>
<div>&nbsp;10.|-- 209.85.241.149 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 0.0% &nbsp; &nbsp;10 &nbsp; 54.2 &nbsp;57.7 &nbsp;54.0 &nbsp;76.5 &nbsp; 7.6</div>
<div>&nbsp;11.|-- 209.85.243.23 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.0% &nbsp; &nbsp;10 &nbsp; 57.9 &nbsp;60.1 &nbsp;57.7 &nbsp;79.4 &nbsp; 6.8</div>
<div>&nbsp;12.|-- 72.14.238.8 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.0% &nbsp; &nbsp;10 &nbsp; 58.9 &nbsp;59.3 &nbsp;58.8 &nbsp;62.5 &nbsp; 1.1</div>
<div>&nbsp;13.|-- 72.14.237.179 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.0% &nbsp; &nbsp;10 &nbsp; 63.0 &nbsp;65.3 &nbsp;62.8 &nbsp;79.2 &nbsp; 5.4</div>
<div>&nbsp;14.|-- ??? &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 100.0 &nbsp; &nbsp;10 &nbsp; &nbsp;0.0 &nbsp; 0.0 &nbsp; 0.0 &nbsp; 0.0 &nbsp; 0.0</div>
<div>xxx <at> xxx:~$&nbsp;</div>
</div></div>
Chris Woodfield via Outages | 10 Oct 19:12 2014

[outages] Akamai problems on east coast/Verizon?

I've seen two sites now fronted by Akamai - travelocity.com and retailmenot.com - produce "DNS Failure"
errors from the edge, suggesting Akamai edge servers unable to reach back-end caches or origin servers.
Anyone else noticing this?

IPs I've seen for the edge sites in question:

[~/]$ host www.travelocity.com
www.travelocity.com is an alias for www.travelocity.com.edgekey.net.
www.travelocity.com.edgekey.net is an alias for e8329.b.akamaiedge.net.
e8329.b.akamaiedge.net has address 23.203.11.138

[~/]$ host www.retailmenot.com
www.retailmenot.com is an alias for wildcard.retailmenot.com.edgekey.net.
wildcard.retailmenot.com.edgekey.net is an alias for e3849.b.akamaiedge.net.
e3849.b.akamaiedge.net has address 172.231.217.199

I'm on a Verizon Jetpack right now in eastern Maryland, if that helps...

Thanks,

-C
Blair Trosper via Outages | 10 Oct 01:07 2014

[outages] GApps admin = rogered

Just a heads up to our friends at Google Apps.

Despite the status page saying all is peachy:

...the administration page for any Google Apps for domains is totally rogered.  It's either an endless redirect loop or a deluge of errors.

I'd call for premium support, but I can't even see that.  

Again, a friendly heads up and nudge that perhaps the status page should at least be updated to reflect the fact that it's non-operational.
<div><div dir="ltr">Just a heads up to our friends at Google Apps.<div><br></div>
<div>Despite the status page saying all is peachy:</div>
<div>
<a href="http://www.google.com/appsstatus#hl=en&amp;v=status">http://www.google.com/appsstatus#hl=en&amp;v=status</a><br>
</div>
<div><br></div>
<div>...the administration page for any Google Apps for domains is totally rogered.&nbsp; It's either an endless redirect loop or a deluge of errors.</div>
<div><br></div>
<div>I'd call for premium support, but I can't even see that. &nbsp;</div>
<div><br></div>
<div>Again, a friendly heads up and nudge that perhaps the status page should at least be updated to reflect the fact that it's non-operational.</div>
</div></div>

Re: [outages] Belkin [heartbeat.belkin.com]

On 07 Oct 2014, at 19:57, Frank Bulk via Outages <outages@...> wrote:

> Why do that if Belkin is up and running again?

This thing seems to go down and up like a yoyo.
(I’m seeing it one hop behind 66.179.229.150 — works this very moment.)

Can’t get a reliable OS fingerprint (unless it’s an Airport Extreme…).
For extra fun, try nmapping it around the outages.
Sometimes it is like:

Interesting ports on 67.20.176.130:
Not shown: 1689 closed ports
PORT    STATE    SERVICE
22/tcp  open     ssh
111/tcp open     rpcbind
137/tcp filtered netbios-ns
138/tcp filtered netbios-dgm
139/tcp filtered netbios-ssn
199/tcp open     smux
445/tcp filtered microsoft-ds
937/tcp open     unknown

What?

Grüße, Carsten

Josh Luthman via Outages | 7 Oct 20:14 2014

Re: [outages] Belkin [heartbeat.belkin.com]

Doesn't appear to be all running, at least their official posting doesn't say so  - 




Josh Luthman
Office: 937-552-2340
Direct: 937-552-2343
1100 Wayne St
Suite 1337
Troy, OH 45373

On Tue, Oct 7, 2014 at 1:57 PM, Frank Bulk via Outages <outages-w8q5NolJmqtAfugRpC6u6w@public.gmane.org> wrote:

Why do that if Belkin is up and running again? Or are you suggesting that Belkin doesn’t have it all worked out?

 

Frank

 

From: Outages [mailto:outages-bounces <at> outages.org] On Behalf Of John Neiberger via Outages
Sent: Tuesday, October 07, 2014 12:42 PM
To: outages-w8q5NolJmqtAfugRpC6u6w@public.gmane.org


Subject: Re: [outages] Belkin [heartbeat.belkin.com]

 

Has anyone else tried the loopback/heartbeat workaround with success? I've seen a few reports so far that this has temporarily solved the problem.

 

John

 

On Tue, Oct 7, 2014 at 10:28 AM, Frank Bulk <frnkblk-YCpy1HD0M2cAvxtiuMwx3w@public.gmane.org> wrote:

I can ping heartbeat.belkin.com and the DSL Reports link and other connected links are stating it was back up roughly 30 to 45 minutes ago, so I think we’re out of the woods.

 

Frank

 

From: Outages [mailto:outages-bounces-w8q5NolJmqtAfugRpC6u6w@public.gmane.org] On Behalf Of John Neiberger via Outages
Sent: Tuesday, October 07, 2014 11:04 AM
To: Josh Luthman
Cc: outages-w8q5NolJmqtAfugRpC6u6w@public.gmane.org; John Barbieri
Subject: Re: [outages] Belkin [heartbeat.belkin.com]

 

I think it was initially suspected to be a firmware upgrade, but I doubt that's the case now. It's interesting to me that there are two seemingly unrelated fixes:

 

1. Put the IP for heartbeat.belkin.com on a loopback 

2. Manually add working DNS entries to affected PCs

 

Reports indicate that either fix works, but why? They seem unrelated to one another unless the failing heartbeat messages mess up DNS handling on the router.

 

John

 

On Tue, Oct 7, 2014 at 9:48 AM, Josh Luthman via Outages <outages-w8q5NolJmqtAfugRpC6u6w@public.gmane.org> wrote:

Where?  I keep reading "bad firmware" problems but everything suggests there is no automatic upgrade and I think we can safely assume resi users didn't upgrade their routers over night.


 


Josh Luthman
Office: 937-552-2340
Direct: 937-552-2343
1100 Wayne St
Suite 1337
Troy, OH 45373

 

On Tue, Oct 7, 2014 at 11:35 AM, John Barbieri via Outages <outages-w8q5NolJmqtAfugRpC6u6w@public.gmane.org> wrote:

Funny,

Just saw something about Belkin routers and a botched firmware upgrade.

I wonder if this is what they meant.

That being said, it is working fine from Zito Media in Clanton, AL.

--John


On 10/07/2014 10:17 AM, Paul Miller via Outages wrote:

On one of my netblocks my subscribers can nolonger ping Belkin.  I can ping
heartbeat.belkin.com from my webservers and from subscriber units on a
different netblock.

Apparently Belkin routers ping the home base every couple seconds to determine
the correct color of the Internet light on the router (orange or blue *sigh*).
Worse, when it can't ping the Belkin site the router seems to pack up and
redirect my subscribers to their router configuration page.

I found this pretty easy to work around.  I simply DNATed the pings intended
for heartbeat.belkin.com to one of my local nameservers and that seems to get
them back online.  It's a silly workaround, but if anyone else is seeing all
the Belkin routers on their network pack up, this could be why.

-Paul

_______________________________________________
Outages mailing list
Outages-w8q5NolJmqtAfugRpC6u6w@public.gmane.org
https://puck.nether.net/mailman/listinfo/outages


_______________________________________________
Outages mailing list
Outages-w8q5NolJmqtAfugRpC6u6w@public.gmane.org
https://puck.nether.net/mailman/listinfo/outages

 


_______________________________________________
Outages mailing list
Outages-w8q5NolJmqtAfugRpC6u6w@public.gmane.org
https://puck.nether.net/mailman/listinfo/outages

 

 


_______________________________________________
Outages mailing list
Outages-w8q5NolJmqtAfugRpC6u6w@public.gmane.org
https://puck.nether.net/mailman/listinfo/outages


<div>
<div dir="ltr">Doesn't appear to be all running, at least their official posting doesn't say so &nbsp;-&nbsp;<div><br></div>
<div>
<a href="https://belkininternationalinc.statuspage.io/">https://belkininternationalinc.statuspage.io/</a><br>
</div>
</div>
<div class="gmail_extra">
<br clear="all"><div><div dir="ltr">
<div><br></div>
<br>Josh Luthman<br>Office: 937-552-2340<br>Direct: 937-552-2343<br>1100 Wayne St<br>Suite 1337<br>Troy, OH 45373</div></div>
<br><div class="gmail_quote">On Tue, Oct 7, 2014 at 1:57 PM, Frank Bulk via Outages <span dir="ltr">&lt;<a href="mailto:outages@..." target="_blank">outages@...</a>&gt;</span> wrote:<br><blockquote class="gmail_quote">
<div lang="EN-US" link="blue" vlink="purple"><div>
<p class="MsoNormal"><span>Why do that if Belkin is up and running again? Or are you suggesting that Belkin doesn&rsquo;t have it all worked out?</span></p>
<p class="MsoNormal"><span>&nbsp;</span></p>
<p class="MsoNormal"><span>Frank</span></p>
<p class="MsoNormal"><span>&nbsp;</span></p>
<p class="MsoNormal"><span>From:</span><span> Outages [mailto:<a href="mailto:outages-bounces@..." target="_blank">outages-bounces <at> outages.org</a>] On Behalf Of John Neiberger via Outages<br>Sent: Tuesday, October 07, 2014 12:42 PM<br>To: <a href="mailto:outages <at> outages.org" target="_blank">outages@...</a></span></p>
<div><div class="h5">
<br>Subject: Re: [outages] Belkin [<a href="http://heartbeat.belkin.com" target="_blank">heartbeat.belkin.com</a>]</div></div>
<p></p>
<div><div class="h5">
<p class="MsoNormal">&nbsp;</p>
<div>
<div><p class="MsoNormal"><span>Has anyone else tried the loopback/heartbeat workaround with success? I've seen a few reports so far that this has temporarily solved the problem.</span></p></div>
<div><p class="MsoNormal"><span>&nbsp;</span></p></div>
<div><p class="MsoNormal"><span>John</span></p></div>
</div>
<div>
<p class="MsoNormal">&nbsp;</p>
<div>
<p class="MsoNormal">On Tue, Oct 7, 2014 at 10:28 AM, Frank Bulk &lt;<a href="mailto:frnkblk@..." target="_blank">frnkblk@...</a>&gt; wrote:</p>
<blockquote><div><div>
<p class="MsoNormal"><span>I can ping <a href="http://heartbeat.belkin.com" target="_blank">heartbeat.belkin.com</a> and the DSL Reports link and other connected links are stating it was back up roughly 30 to 45 minutes ago, so I think we&rsquo;re out of the woods.</span></p>
<p class="MsoNormal"><span>&nbsp;</span></p>
<p class="MsoNormal"><span>Frank</span></p>
<p class="MsoNormal"><span>&nbsp;</span></p>
<p class="MsoNormal"><span>From:</span><span> Outages [mailto:<a href="mailto:outages-bounces@..." target="_blank">outages-bounces@...</a>] On Behalf Of John Neiberger via Outages<br>Sent: Tuesday, October 07, 2014 11:04 AM<br>To: Josh Luthman<br>Cc: <a href="mailto:outages@..." target="_blank">outages@...</a>; John Barbieri<br>Subject: Re: [outages] Belkin [<a href="http://heartbeat.belkin.com" target="_blank">heartbeat.belkin.com</a>]</span></p>
<p class="MsoNormal">&nbsp;</p>
<div>
<div><p class="MsoNormal"><span>I think it was initially suspected to be a firmware upgrade, but I doubt that's the case now. It's interesting to me that there are two seemingly unrelated fixes:</span></p></div>
<div><div>
<div><p class="MsoNormal"><span>&nbsp;</span></p></div>
<div><p class="MsoNormal"><span>1. Put the IP for <a href="http://heartbeat.belkin.com" target="_blank">heartbeat.belkin.com</a> on a loopback&nbsp;</span></p></div>
<div><p class="MsoNormal"><span>2. Manually add working DNS entries to affected PCs</span></p></div>
<div><p class="MsoNormal"><span>&nbsp;</span></p></div>
<div><p class="MsoNormal"><span>Reports indicate that either fix works, but why? They seem unrelated to one another unless the failing heartbeat messages mess up DNS handling on the router.</span></p></div>
<div><p class="MsoNormal"><span>&nbsp;</span></p></div>
<div><p class="MsoNormal"><span>John</span></p></div>
</div></div>
</div>
<div><div><div>
<p class="MsoNormal">&nbsp;</p>
<div>
<p class="MsoNormal">On Tue, Oct 7, 2014 at 9:48 AM, Josh Luthman via Outages &lt;<a href="mailto:outages@..." target="_blank">outages@...</a>&gt; wrote:</p>
<blockquote>
<div><p class="MsoNormal">Where?&nbsp; I keep reading "bad firmware" problems but everything suggests there is no automatic upgrade and I think we can safely assume resi users didn't upgrade their routers over night.</p></div>
<div>
<p class="MsoNormal"><br clear="all"></p>
<div><div>
<div><p class="MsoNormal">&nbsp;</p></div>
<p class="MsoNormal"><br>Josh Luthman<br>Office: <a href="tel:937-552-2340" target="_blank">937-552-2340</a><br>Direct: <a href="tel:937-552-2343" target="_blank">937-552-2343</a><br>1100 Wayne St<br>Suite 1337<br>Troy, OH 45373</p>
</div></div>
<p class="MsoNormal">&nbsp;</p>
<div><div>
<div>
<p class="MsoNormal">On Tue, Oct 7, 2014 at 11:35 AM, John Barbieri via Outages &lt;<a href="mailto:outages@..." target="_blank">outages@...</a>&gt; wrote:</p>
<blockquote>
<p class="MsoNormal">Funny,<br><br>Just saw something about Belkin routers and a botched firmware upgrade.<br><br>I wonder if this is what they meant.<br><br>That being said, it is working fine from Zito Media in Clanton, AL.<br><br>--John</p>
<div><div>
<p class="MsoNormal"><br>On 10/07/2014 10:17 AM, Paul Miller via Outages wrote:</p>
<blockquote><p class="MsoNormal">On one of my netblocks my subscribers can nolonger ping Belkin.&nbsp; I can ping<br><a href="http://heartbeat.belkin.com" target="_blank">heartbeat.belkin.com</a> from my webservers and from subscriber units on a<br>different netblock.<br><br>Apparently Belkin routers ping the home base every couple seconds to determine<br>the correct color of the Internet light on the router (orange or blue *sigh*).<br>Worse, when it can't ping the Belkin site the router seems to pack up and<br>redirect my subscribers to their router configuration page.<br><br>I found this pretty easy to work around.&nbsp; I simply DNATed the pings intended<br>for <a href="http://heartbeat.belkin.com" target="_blank">heartbeat.belkin.com</a> to one of my local nameservers and that seems to get<br>them back online.&nbsp; It's a silly workaround, but if anyone else is seeing all<br>the Belkin routers on their network pack up, this could be why.<br><br>-Paul<br><br>_______________________________________________<br>Outages mailing list<br><a href="mailto:Outages <at> outages.org" target="_blank">Outages@...</a><br><a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a></p></blockquote>
<p class="MsoNormal"><br>_______________________________________________<br>Outages mailing list<br><a href="mailto:Outages@..." target="_blank">Outages@...</a><br><a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a></p>
</div></div>
</blockquote>
</div>
<p class="MsoNormal">&nbsp;</p>
</div></div>
</div>
<p class="MsoNormal"><br>_______________________________________________<br>Outages mailing list<br><a href="mailto:Outages@..." target="_blank">Outages@...</a><br><a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a></p>
</blockquote>
</div>
<p class="MsoNormal">&nbsp;</p>
</div></div></div>
</div></div></blockquote>
</div>
<p class="MsoNormal">&nbsp;</p>
</div>
</div></div>
</div></div>
<br>_______________________________________________<br>
Outages mailing list<br><a href="mailto:Outages@...">Outages@...</a><br><a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br><br>
</blockquote>
</div>
<br>
</div>
</div>

Gmane