Kurt Duncan via Outages | 23 Oct 02:37 2014

[outages] FW: Broadvox Down (RESOLVED)

Problem was resolved <15 minutes

 

 

Kurt Duncan

 

From: Kurt Duncan
Sent: Wednesday, October 22, 2014 5:21 PM
To: 'Outages-w8q5NolJmqtAfugRpC6u6w@public.gmane.org'
Subject: Broadvox Down

 

Broadvox  VOIP circuits for inbound DID numbers are 100% down at this time at least on the west coast.  No calls being processed. 

 

 

Kurt Duncan

 

<div>
<div class="WordSection1">
<p class="MsoNormal"><span>Problem was resolved &lt;15 minutes<p></p></span></p>
<p class="MsoNormal"><span><p>&nbsp;</p></span></p>
<div>
<p class="MsoNormal"><span><p>&nbsp;</p></span></p>
<p class="MsoNormal"><span>Kurt Duncan<p></p></span></p>
</div>
<p class="MsoNormal"><span><p>&nbsp;</p></span></p>
<div>
<div>
<p class="MsoNormal"><span>From:</span><span> Kurt Duncan
<br>Sent: Wednesday, October 22, 2014 5:21 PM<br>To: 'Outages@...'<br>Subject: Broadvox Down<p></p></span></p>
</div>
</div>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal">Broadvox&nbsp; VOIP circuits for inbound DID numbers are 100% down at this time at least on the west coast.&nbsp; No calls being processed.&nbsp;
<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal">Kurt Duncan<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
</div>
</div>
Kurt Duncan via Outages | 23 Oct 02:21 2014

[outages] Broadvox Down

Broadvox  VOIP circuits for inbound DID numbers are 100% down at this time at least on the west coast.  No calls being processed. 

 

 

Kurt Duncan

 

<div>
<div class="WordSection1">
<p class="MsoNormal">Broadvox&nbsp; VOIP circuits for inbound DID numbers are 100% down at this time at least on the west coast.&nbsp; No calls being processed.&nbsp;
<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
<p class="MsoNormal">Kurt Duncan<p></p></p>
<p class="MsoNormal"><p>&nbsp;</p></p>
</div>
</div>
Everton Marques via Outages | 22 Oct 18:19 2014

[outages] Comcast packet loss

High packet loss from Highwinds towards github.com.
It seems the issue is rooted within Comcast.

Time is 16:16:44.371 Wed Oct 22 2014 GMT-0

ping -t 192.30.252.129

Pinging 192.30.252.129 with 32 bytes of data:
Reply from 192.30.252.129: bytes=32 time=474ms TTL=44
Request timed out.
Reply from 192.30.252.129: bytes=32 time=480ms TTL=44
Reply from 192.30.252.129: bytes=32 time=483ms TTL=44
Reply from 192.30.252.129: bytes=32 time=488ms TTL=44
Request timed out.
Reply from 192.30.252.129: bytes=32 time=485ms TTL=44
Reply from 192.30.252.129: bytes=32 time=485ms TTL=44
Reply from 192.30.252.129: bytes=32 time=481ms TTL=44
Reply from 192.30.252.129: bytes=32 time=484ms TTL=44
Reply from 192.30.252.129: bytes=32 time=478ms TTL=44

Ping statistics for 192.30.252.129:
    Packets: Sent = 11, Received = 9, Lost = 2 (18% loss),
Approximate round trip times in milli-seconds:
    Minimum = 474ms, Maximum = 488ms, Average = 482ms
Control-C

#traceroute 192.30.252.129

Type escape sequence to abort.
Tracing the route to 192.30.252.129

  1 200.147.29.137 [AS 7162] 0 msec 0 msec 0 msec
  2 200.147.29.146 [AS 7162] 0 msec 0 msec 0 msec
  3  *  *  *
  4 173.167.57.1 [AS 7922] 120 msec 120 msec 120 msec
  5 68.86.82.249 [AS 7922] 120 msec 124 msec
    68.86.82.253 [AS 7922] 124 msec
  6  *  *  *
  7  *  *  *
  8 68.86.83.86 [AS 7922] 472 msec 472 msec 472 msec
  9  *  *  *
 10  *  *  *
  
Everton
<div><div dir="ltr">
<div>High packet loss from Highwinds towards <a href="http://github.com">github.com</a>.</div>
<div>It seems the issue is rooted within Comcast.</div>
<div><br></div>
<div>Time is 16:16:44.371 Wed Oct 22 2014 GMT-0<br>
</div>
<div><br></div>
<div>ping -t 192.30.252.129</div>
<div><br></div>
<div>Pinging 192.30.252.129 with 32 bytes of data:</div>
<div>Reply from <a href="http://192.30.252.129">192.30.252.129</a>: bytes=32 time=474ms TTL=44</div>
<div>Request timed out.</div>
<div>Reply from <a href="http://192.30.252.129">192.30.252.129</a>: bytes=32 time=480ms TTL=44</div>
<div>Reply from <a href="http://192.30.252.129">192.30.252.129</a>: bytes=32 time=483ms TTL=44</div>
<div>Reply from <a href="http://192.30.252.129">192.30.252.129</a>: bytes=32 time=488ms TTL=44</div>
<div>Request timed out.</div>
<div>Reply from <a href="http://192.30.252.129">192.30.252.129</a>: bytes=32 time=485ms TTL=44</div>
<div>Reply from <a href="http://192.30.252.129">192.30.252.129</a>: bytes=32 time=485ms TTL=44</div>
<div>Reply from <a href="http://192.30.252.129">192.30.252.129</a>: bytes=32 time=481ms TTL=44</div>
<div>Reply from <a href="http://192.30.252.129">192.30.252.129</a>: bytes=32 time=484ms TTL=44</div>
<div>Reply from <a href="http://192.30.252.129">192.30.252.129</a>: bytes=32 time=478ms TTL=44</div>
<div><br></div>
<div>Ping statistics for <a href="http://192.30.252.129">192.30.252.129</a>:</div>
<div>&nbsp; &nbsp; Packets: Sent = 11, Received = 9, Lost = 2 (18% loss),</div>
<div>Approximate round trip times in milli-seconds:</div>
<div>&nbsp; &nbsp; Minimum = 474ms, Maximum = 488ms, Average = 482ms</div>
<div>Control-C</div>
<div><br></div>
<div>#traceroute 192.30.252.129</div>
<div><br></div>
<div>Type escape sequence to abort.</div>
<div>Tracing the route to 192.30.252.129</div>
<div><br></div>
<div>&nbsp; 1 200.147.29.137 [AS 7162] 0 msec 0 msec 0 msec</div>
<div>&nbsp; 2 200.147.29.146 [AS 7162] 0 msec 0 msec 0 msec</div>
<div>&nbsp; 3 &nbsp;* &nbsp;* &nbsp;*</div>
<div>&nbsp; 4 173.167.57.1 [AS 7922] 120 msec 120 msec 120 msec</div>
<div>&nbsp; 5 68.86.82.249 [AS 7922] 120 msec 124 msec</div>
<div>&nbsp; &nbsp; 68.86.82.253 [AS 7922] 124 msec</div>
<div>&nbsp; 6 &nbsp;* &nbsp;* &nbsp;*</div>
<div>&nbsp; 7 &nbsp;* &nbsp;* &nbsp;*</div>
<div>&nbsp; 8 68.86.83.86 [AS 7922] 472 msec 472 msec 472 msec</div>
<div>&nbsp; 9 &nbsp;* &nbsp;* &nbsp;*</div>
<div>&nbsp;10 &nbsp;* &nbsp;* &nbsp;*</div>
<div>&nbsp;&nbsp;<br>
</div>
<div>Everton</div>
</div></div>
Frank Bulk via Outages | 22 Oct 00:01 2014

[outages] Email to Microsoft Office 365 users

Our last queue check reported DNS failures looking up <domain
name>.mail.protection.outlook.com for 11 such domains.

Just picking one, staples.com: 

root <at> nagios:/tmp# dig MX staples.com +short
10 staples-com.mail.protection.outlook.com.
root <at> nagios:/tmp# dig  staples-com.mail.protection.outlook.com

; <<>> DiG 9.7.3 <<>> staples-com.mail.protection.outlook.com
;; global options: +cmd
;; connection timed out; no servers could be reached
root <at> nagios:/tmp#
root <at> nagios:/tmp# host MX staples.com
;; connection timed out; no servers could be reached
root <at> nagios:/tmp#

Anyone else seeing this?  A colleague says that O365 is having issues and
will be forwarding me some details.

Regards,

Frank Bulk

Michael Loftis via Outages | 20 Oct 21:57 2014

[outages] Comcast - Seattle (maybe other areas)

Having serious issues getting outside of Comcast's network starting
around 1250 Pacific time from Seattle area.

--

-- 

"Genius might be described as a supreme capacity for getting its possessors
into trouble of all kinds."
-- Samuel Butler
James Cornman via Outages | 17 Oct 22:07 2014

Re: [outages] Equinix Chicago exchange

I tracked it down to an offending device belonging to 123.net. Still haven't received any peep out of Equinix support.


Filtered it..

mac access-list extended equinix-bad-neighbors
 deny   host 8478.ac4e.90be any
 permit any any
!
interface GigabitEthernet0/50

  mac access-group equinix-bad-neighbors in


On Fri, Oct 17, 2014 at 4:06 PM, Nicholas Schmidt <nschmidt-Qg2TNkI2BF61Z/+hSey0Gg@public.gmane.org> wrote:
Seeing similar. 

Nicholas Schmidt
MediaMath

From: James Cornman via Outages <outages <at> outages.org>
Reply-To: James Cornman <james <at> atlanticmetro.net>
Date: Friday, October 17, 2014 at 3:50 PM
To: "outages-w8q5NolJmqtAfugRpC6u6w@public.gmane.org" <outages-w8q5NolJmqtAfugRpC6u6w@public.gmane.org>
Subject: [outages] Equinix Chicago exchange

Hey all,

Anyone seeing a large influx of broadcast traffic since a little after 3pm EDT on the Equinix exchange in Chicago?

Opening a ticket with Equinix, but just curious if any others have seen anything.



--

James Cornman

Chief Technology Officer
jcornman-q63CSc06zXuqyuYpCWRUfF6hYfS7NtTn@public.gmane.org
212.792.9950 - ext 101

Atlantic Metro Communications, Inc.

4 Century Drive, Parsippany NJ  07054


Data Centers • Managed Hosting • Internet Access • Cloud Computing
Follow  <at> atlanticmetro on Twitter to learn more about daily specials.
www.atlanticmetro.net




--

James Cornman

Chief Technology Officer
jcornman-q63CSc06zXuqyuYpCWRUfF6hYfS7NtTn@public.gmane.org
212.792.9950 - ext 101

Atlantic Metro Communications, Inc.

4 Century Drive, Parsippany NJ  07054


Data Centers • Managed Hosting • Internet Access • Cloud Computing
Follow  <at> atlanticmetro on Twitter to learn more about daily specials.
www.atlanticmetro.net

<div>
<div dir="ltr">I tracked it down to an offending device belonging to <a href="http://123.net">123.net</a>. Still haven't received any peep out of Equinix support.<div><br></div>
<div><br></div>
<div>Filtered it..</div>
<div><br></div>
<div>
<div>mac access-list extended equinix-bad-neighbors</div>
<div>&nbsp;deny &nbsp; host 8478.ac4e.90be any</div>
<div>&nbsp;permit any any</div>
<div>!</div>
<div>interface GigabitEthernet0/50&#8232;</div>
<div>&nbsp; mac access-group equinix-bad-neighbors in</div>
</div>
<div><br></div>
</div>
<div class="gmail_extra">
<br><div class="gmail_quote">On Fri, Oct 17, 2014 at 4:06 PM, Nicholas Schmidt <span dir="ltr">&lt;<a href="mailto:nschmidt@..." target="_blank">nschmidt@...</a>&gt;</span> wrote:<br><blockquote class="gmail_quote">

<div>
<div>Seeing similar.&nbsp;</div>
<div><br></div>
<div>Nicholas Schmidt</div>
<div>MediaMath</div>
<div><br></div>
<span>
<div>
<span>From: </span>James Cornman via Outages &lt;<a href="mailto:outages@..." target="_blank">outages <at> outages.org</a>&gt;<br><span>Reply-To: </span>James Cornman &lt;<a href="mailto:james@..." target="_blank">james <at> atlanticmetro.net</a>&gt;<br><span>Date: </span>Friday, October 17, 2014 at 3:50 PM<br><span>To: </span>"<a href="mailto:outages <at> outages.org" target="_blank">outages@...</a>" &lt;<a href="mailto:outages@..." target="_blank">outages@...</a>&gt;<br><span>Subject: </span>[outages] Equinix Chicago exchange<br>
</div>
<div><div class="h5">
<div><br></div>
<div>
<div>
<div dir="ltr">Hey all,
<div><br></div>
<div>Anyone seeing a large influx of broadcast traffic since a little after 3pm EDT on the Equinix exchange in Chicago?</div>
<div><br></div>
<div>Opening a ticket with Equinix, but just curious if any others have seen anything.</div>
<div><br></div>
<div>
<br clear="all"><div><br></div>
-- <br><span>
<p><span>James Cornman</span></p>
<p><span>Chief Technology Officer<br></span><span><a href="mailto:jcornman <at> atlanticmetro.net" target="_blank"><span>jcornman@...</span></a><br><a href="tel:212.792.9950" value="+12127929950" target="_blank">212.792.9950</a> - ext 101<br><span><br>
Atlantic Metro Communications, Inc.</span></span></p>
<p><span>4 Century Drive, Parsippany NJ&nbsp; 07054</span></p>
<p><span><br></span><span>Data Centers &bull; Managed Hosting &bull; Internet Access &bull; Cloud Computing</span><span><br></span><span>Follow&nbsp;</span><span><a href="http://www.twitter.com/atlanticmetro" target="_blank"><span> <at> atlanticmetro</span></a></span><span>&nbsp;on
 Twitter to learn more about daily specials.</span><span><br><a href="http://www.atlanticmetro.net/" target="_blank"><span>www.atlanticmetro.net</span></a></span></p>
</span>
</div>
</div>
</div>
</div>
</div></div></span>
</div>

</blockquote>
</div>
<br><br clear="all"><div><br></div>-- <br><span><p><span>James Cornman</span></p>
<p><span>Chief Technology Officer<br></span><span><a href="mailto:jcornman@..." target="_blank"><span>jcornman@...</span></a><br>212.792.9950 - ext 101<br><span><br>Atlantic Metro Communications, Inc.</span></span></p>
<p><span>4 Century Drive, Parsippany NJ&nbsp; 07054</span></p>
<p><span><br></span><span>Data Centers &bull; Managed Hosting &bull; Internet Access &bull; Cloud Computing</span><span><br></span><span>Follow&nbsp;</span><span><a href="http://www.twitter.com/atlanticmetro" target="_blank"><span> <at> atlanticmetro</span></a></span><span>&nbsp;on Twitter to learn more about daily specials.</span><span><br><a href="http://www.atlanticmetro.net/" target="_blank"><span>www.atlanticmetro.net</span></a></span></p></span>
</div>
</div>
James Cornman via Outages | 17 Oct 21:50 2014

[outages] Equinix Chicago exchange

Hey all,

Anyone seeing a large influx of broadcast traffic since a little after 3pm EDT on the Equinix exchange in Chicago?

Opening a ticket with Equinix, but just curious if any others have seen anything.



--

James Cornman

Chief Technology Officer
jcornman-q63CSc06zXuqyuYpCWRUfF6hYfS7NtTn@public.gmane.org
212.792.9950 - ext 101

Atlantic Metro Communications, Inc.

4 Century Drive, Parsippany NJ  07054


Data Centers • Managed Hosting • Internet Access • Cloud Computing
Follow  <at> atlanticmetro on Twitter to learn more about daily specials.
www.atlanticmetro.net

<div><div dir="ltr">Hey all,<div><br></div>
<div>Anyone seeing a large influx of broadcast traffic since a little after 3pm EDT on the Equinix exchange in Chicago?</div>
<div><br></div>
<div>Opening a ticket with Equinix, but just curious if any others have seen anything.</div>
<div><br></div>
<div>
<br clear="all"><div><br></div>-- <br><span><p><span>James Cornman</span></p>
<p><span>Chief Technology Officer<br></span><span><a href="mailto:jcornman <at> atlanticmetro.net" target="_blank"><span>jcornman@...</span></a><br>212.792.9950 - ext 101<br><span><br>Atlantic Metro Communications, Inc.</span></span></p>
<p><span>4 Century Drive, Parsippany NJ&nbsp; 07054</span></p>
<p><span><br></span><span>Data Centers &bull; Managed Hosting &bull; Internet Access &bull; Cloud Computing</span><span><br></span><span>Follow&nbsp;</span><span><a href="http://www.twitter.com/atlanticmetro" target="_blank"><span> <at> atlanticmetro</span></a></span><span>&nbsp;on Twitter to learn more about daily specials.</span><span><br><a href="http://www.atlanticmetro.net/" target="_blank"><span>www.atlanticmetro.net</span></a></span></p></span>
</div>
</div></div>
Morgan Miskell via Outages | 17 Oct 15:51 2014

[outages] Windstream Long Distance

Windstream Long Distance appears to be down for us....might be a decent
size issue for them since I've been on hold to report the issue for 15
minutes which I interpret to mean that their call center is getting
slammed!
--

-- 
Morgan A. Miskell
CaroNet Data Centers
704-643-8330 x206
----------------------------------------------------------------------------
The information contained in this e-mail is confidential and is intended
only for the named recipient(s). If you are not the intended recipient
you must not copy, distribute, or take any action or reliance on it. If
you have received this e-mail in error, please notify the sender. Any
unauthorized disclosure of the information contained in this e-mail is
strictly prohibited.
----------------------------------------------------------------------------

Rodrick Brown via Outages | 16 Oct 06:05 2014

[outages] delays on the interweb?

I'm seeing delays and outages hitting random sites anyone else noticing the same? 

vimeo
etc.. ? 
<div><div dir="ltr">I'm seeing delays and outages hitting random sites anyone else noticing the same?&nbsp;<div><br></div>
<div><a href="http://news.ycombinator.com">news.ycombinator.com</a></div>
<div>vimeo</div>
<div>etc.. ?&nbsp;</div>
</div></div>
Vipul Desai via Outages | 16 Oct 05:53 2014

[outages] Century Link/Qwest Outage


Does anyone else experience routing issue for websites hosted on CDN from CenturyLink's edge router?

jfk-brdr-04.inet.qwest.net [67.14.5.98] 
 63.146.26.2

I can't open multiple websites if routing goes through their JFK edge router. It's been happening over 24 hours and no resolution from Century Link/Qwest.
Same sites will open if route goes through dcx2-edge-01.inet.qwest.net [67.14.28.66] or nyc-edge-05.inet.qwest.net [205.171.134.10] or chp-brdr-03.inet.qwest.net [67.14.8.194] edge router.

Thank you,
Vipul Desai

 
<div><div>
<div class=""><br class=""></div>
<div class="">Does anyone else experience routing issue for websites hosted on CDN from CenturyLink's edge router?</div>
<div class=""><br class=""></div>
<div class="">
<span class="">jfk-brdr-04.inet.qwest.net [67.14.5.98]</span>&nbsp;</div>
<div class="">
<span class="">&nbsp;63.146.26.2</span><br class="">
</div>
<div class=""><br class=""></div>
<div class="">I can't open multiple websites if routing goes through their JFK edge router. It's been happening over 
 24 hours and no resolution from Century Link/Qwest.</div>
<div class="">Same sites will open if route goes through&nbsp;dcx2-edge-01.inet.qwest.net [67.14.28.66] or&nbsp;<span class="">nyc-edge-05.inet.qwest.net [205.171.134.10] or&nbsp;chp-brdr-03.inet.qwest.net [67.14.8.194] edge router.</span>
</div>
<div class=""><br class=""></div>
<div class="">Thank you,</div>
<div class="">Vipul Desai
 </div>
<div class=""><br class=""></div>
<div class="">&nbsp;</div>
</div></div>
Sajal Kayan via Outages | 13 Oct 13:25 2014

[outages] Google Public DNS is down in south-east asia

Google Public DNS just stopped working for me (and many others locally).

xxx <at> xxx:~# dig google.com <at> 8.8.8.8

; <<>> DiG 9.8.4-rpz2+rl005.12-P1 <<>> google.com <at> 8.8.8.8
;; global options: +cmd
;; connection timed out; no servers could be reached
xxx <at> xxx:~# 

Traces from few locations.

True ADSL (Thailand)

xxx <at> xxx:~# mtr --report 8.8.8.8
HOST: xxx                         Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 192.168.1.1                0.0%    10   30.5  17.4   0.8  57.5  22.5
  2.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
  3.|-- 10.169.212.9               0.0%    10   23.3  24.0  23.3  25.0   0.6
  4.|-- 10.100.50.67               0.0%    10   24.0  24.3  23.7  25.1   0.5
  5.|-- 58.97.121.81               0.0%    10   25.4  25.1  24.5  25.6   0.4
  6.|-- 61.91.213.177              0.0%    10   24.6  25.0  24.6  25.6   0.3
  7.|-- 61.91.213.36               0.0%    10   26.1  25.9  25.4  26.4   0.3
  8.|-- 61.91.213.130              0.0%    10   25.5  26.0  25.1  27.3   0.6
  9.|-- 113.21.242.53              0.0%    10   27.0  26.5  25.1  28.2   1.0
 10.|-- 72.14.210.155              0.0%    10   49.1  52.2  48.1  80.4   9.9
 11.|-- 209.85.242.244             0.0%    10   54.6  53.6  50.0  60.9   4.1
 12.|-- 72.14.239.201              0.0%    10   85.6  85.7  80.2 100.8   6.5
 13.|-- 209.85.250.11              0.0%    10  107.2 107.7 106.7 109.7   0.9
 14.|-- 209.85.243.21              0.0%    10  109.6 110.4 109.5 111.3   0.6
 15.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
xxx <at> xxx:~# 


Digital Ocean (Singapore)

xxx <at> xxx:~# mtr --report 8.8.8.8
Start: Mon Oct 13 07:18:28 2014
HOST: xxx                  Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- a.b.c.d            0.0%    10    0.6   1.6   0.5   6.0   2.1
  2.|-- 103.253.144.241            0.0%    10    0.4   0.5   0.4   0.6   0.0
  3.|-- 103.16.102.64              0.0%    10    1.8   2.0   1.7   3.4   0.3
  4.|-- 66.249.95.122              0.0%    10    2.1   2.1   2.0   2.3   0.0
  5.|-- 209.85.241.149             0.0%    10   54.5  56.1  54.3  61.8   2.5
  6.|-- 72.14.237.179              0.0%    10   58.0  58.1  57.9  59.2   0.0
  7.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
xxx <at> xxx:~# 


EC2 (Singapore)

xxx <at> xxx:~$ mtr --report 8.8.8.8
HOST: xxx            Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- ip-10-a-b-2.ap-southea  0.0%    10    0.6   0.6   0.3   1.5   0.4
  2.|-- ip-10-1-16-0.ap-southeast  0.0%    10    6.1   1.6   0.5   6.1   1.8
  3.|-- 100.64.12.131              0.0%    10   17.8   3.8   1.0  17.8   5.7
  4.|-- ec2-175-41-128-232.ap-sou  0.0%    10    0.8   0.6   0.5   0.9   0.1
  5.|-- 203.83.223.230             0.0%    10    0.6   1.0   0.4   2.3   0.7
  6.|-- 203.83.223.14              0.0%    10    1.5   1.6   1.4   2.4   0.3
  7.|-- 203.83.223.57              0.0%    10    1.7   1.8   1.5   2.4   0.2
  8.|-- 198.32.141.140             0.0%    10    1.5   1.7   1.4   3.2   0.5
  9.|-- 66.249.95.122              0.0%    10    2.9   5.4   2.2  14.7   4.8
 10.|-- 209.85.241.149             0.0%    10   54.2  57.7  54.0  76.5   7.6
 11.|-- 209.85.243.23              0.0%    10   57.9  60.1  57.7  79.4   6.8
 12.|-- 72.14.238.8                0.0%    10   58.9  59.3  58.8  62.5   1.1
 13.|-- 72.14.237.179              0.0%    10   63.0  65.3  62.8  79.2   5.4
 14.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
xxx <at> xxx:~$ 
<div><div dir="ltr">
<div>Google Public DNS just stopped working for me (and many others locally).</div>
<div><br></div>
<div>
<div>xxx <at> xxx:~# dig <a href="http://google.com">google.com</a>  <at> <a href="http://8.8.8.8">8.8.8.8</a>
</div>
<div><br></div>
<div>; &lt;&lt;&gt;&gt; DiG 9.8.4-rpz2+rl005.12-P1 &lt;&lt;&gt;&gt; <a href="http://google.com">google.com</a>  <at> <a href="http://8.8.8.8">8.8.8.8</a>
</div>
<div>;; global options: +cmd</div>
<div>;; connection timed out; no servers could be reached</div>
<div>xxx <at> xxx:~#&nbsp;</div>
</div>
<div><br></div>
<div>Traces from few locations.</div>
<div><br></div>
<div>True ADSL (Thailand)</div>
<div><br></div>
<div>xxx <at> xxx:~# mtr --report 8.8.8.8</div>
<div>HOST: xxx &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Loss% &nbsp; Snt &nbsp; Last &nbsp; Avg &nbsp;Best &nbsp;Wrst StDev</div>
<div>&nbsp; 1.|-- 192.168.1.1 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.0% &nbsp; &nbsp;10 &nbsp; 30.5 &nbsp;17.4 &nbsp; 0.8 &nbsp;57.5 &nbsp;22.5</div>
<div>&nbsp; 2.|-- ??? &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 100.0 &nbsp; &nbsp;10 &nbsp; &nbsp;0.0 &nbsp; 0.0 &nbsp; 0.0 &nbsp; 0.0 &nbsp; 0.0</div>
<div>&nbsp; 3.|-- 10.169.212.9 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 0.0% &nbsp; &nbsp;10 &nbsp; 23.3 &nbsp;24.0 &nbsp;23.3 &nbsp;25.0 &nbsp; 0.6</div>
<div>&nbsp; 4.|-- 10.100.50.67 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 0.0% &nbsp; &nbsp;10 &nbsp; 24.0 &nbsp;24.3 &nbsp;23.7 &nbsp;25.1 &nbsp; 0.5</div>
<div>&nbsp; 5.|-- 58.97.121.81 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 0.0% &nbsp; &nbsp;10 &nbsp; 25.4 &nbsp;25.1 &nbsp;24.5 &nbsp;25.6 &nbsp; 0.4</div>
<div>&nbsp; 6.|-- 61.91.213.177 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.0% &nbsp; &nbsp;10 &nbsp; 24.6 &nbsp;25.0 &nbsp;24.6 &nbsp;25.6 &nbsp; 0.3</div>
<div>&nbsp; 7.|-- 61.91.213.36 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 0.0% &nbsp; &nbsp;10 &nbsp; 26.1 &nbsp;25.9 &nbsp;25.4 &nbsp;26.4 &nbsp; 0.3</div>
<div>&nbsp; 8.|-- 61.91.213.130 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.0% &nbsp; &nbsp;10 &nbsp; 25.5 &nbsp;26.0 &nbsp;25.1 &nbsp;27.3 &nbsp; 0.6</div>
<div>&nbsp; 9.|-- 113.21.242.53 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.0% &nbsp; &nbsp;10 &nbsp; 27.0 &nbsp;26.5 &nbsp;25.1 &nbsp;28.2 &nbsp; 1.0</div>
<div>&nbsp;10.|-- 72.14.210.155 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.0% &nbsp; &nbsp;10 &nbsp; 49.1 &nbsp;52.2 &nbsp;48.1 &nbsp;80.4 &nbsp; 9.9</div>
<div>&nbsp;11.|-- 209.85.242.244 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 0.0% &nbsp; &nbsp;10 &nbsp; 54.6 &nbsp;53.6 &nbsp;50.0 &nbsp;60.9 &nbsp; 4.1</div>
<div>&nbsp;12.|-- 72.14.239.201 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.0% &nbsp; &nbsp;10 &nbsp; 85.6 &nbsp;85.7 &nbsp;80.2 100.8 &nbsp; 6.5</div>
<div>&nbsp;13.|-- 209.85.250.11 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.0% &nbsp; &nbsp;10 &nbsp;107.2 107.7 106.7 109.7 &nbsp; 0.9</div>
<div>&nbsp;14.|-- 209.85.243.21 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.0% &nbsp; &nbsp;10 &nbsp;109.6 110.4 109.5 111.3 &nbsp; 0.6</div>
<div>&nbsp;15.|-- ??? &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 100.0 &nbsp; &nbsp;10 &nbsp; &nbsp;0.0 &nbsp; 0.0 &nbsp; 0.0 &nbsp; 0.0 &nbsp; 0.0</div>
<div>xxx <at> xxx:~#&nbsp;</div>
<div><br></div>
<div><br></div>
<div>Digital Ocean (Singapore)</div>
<div><br></div>
<div>xxx <at> xxx:~# mtr --report 8.8.8.8</div>
<div>Start: Mon Oct 13 07:18:28 2014</div>
<div>HOST: xxx &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;Loss% &nbsp; Snt &nbsp; Last &nbsp; Avg &nbsp;Best &nbsp;Wrst StDev</div>
<div>&nbsp; 1.|-- a.b.c.d &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.0% &nbsp; &nbsp;10 &nbsp; &nbsp;0.6 &nbsp; 1.6 &nbsp; 0.5 &nbsp; 6.0 &nbsp; 2.1</div>
<div>&nbsp; 2.|-- 103.253.144.241 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.0% &nbsp; &nbsp;10 &nbsp; &nbsp;0.4 &nbsp; 0.5 &nbsp; 0.4 &nbsp; 0.6 &nbsp; 0.0</div>
<div>&nbsp; 3.|-- 103.16.102.64 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.0% &nbsp; &nbsp;10 &nbsp; &nbsp;1.8 &nbsp; 2.0 &nbsp; 1.7 &nbsp; 3.4 &nbsp; 0.3</div>
<div>&nbsp; 4.|-- 66.249.95.122 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.0% &nbsp; &nbsp;10 &nbsp; &nbsp;2.1 &nbsp; 2.1 &nbsp; 2.0 &nbsp; 2.3 &nbsp; 0.0</div>
<div>&nbsp; 5.|-- 209.85.241.149 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 0.0% &nbsp; &nbsp;10 &nbsp; 54.5 &nbsp;56.1 &nbsp;54.3 &nbsp;61.8 &nbsp; 2.5</div>
<div>&nbsp; 6.|-- 72.14.237.179 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.0% &nbsp; &nbsp;10 &nbsp; 58.0 &nbsp;58.1 &nbsp;57.9 &nbsp;59.2 &nbsp; 0.0</div>
<div>&nbsp; 7.|-- ??? &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 100.0 &nbsp; &nbsp;10 &nbsp; &nbsp;0.0 &nbsp; 0.0 &nbsp; 0.0 &nbsp; 0.0 &nbsp; 0.0</div>
<div>xxx <at> xxx:~#&nbsp;</div>
<div><br></div>
<div><br></div>
<div>EC2 (Singapore)</div>
<div><br></div>
<div>xxx <at> xxx:~$ mtr --report 8.8.8.8</div>
<div>HOST: xxx &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;Loss% &nbsp; Snt &nbsp; Last &nbsp; Avg &nbsp;Best &nbsp;Wrst StDev</div>
<div>&nbsp; 1.|-- ip-10-a-b-2.ap-southea &nbsp;0.0% &nbsp; &nbsp;10 &nbsp; &nbsp;0.6 &nbsp; 0.6 &nbsp; 0.3 &nbsp; 1.5 &nbsp; 0.4</div>
<div>&nbsp; 2.|-- ip-10-1-16-0.ap-southeast &nbsp;0.0% &nbsp; &nbsp;10 &nbsp; &nbsp;6.1 &nbsp; 1.6 &nbsp; 0.5 &nbsp; 6.1 &nbsp; 1.8</div>
<div>&nbsp; 3.|-- 100.64.12.131 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.0% &nbsp; &nbsp;10 &nbsp; 17.8 &nbsp; 3.8 &nbsp; 1.0 &nbsp;17.8 &nbsp; 5.7</div>
<div>&nbsp; 4.|-- ec2-175-41-128-232.ap-sou &nbsp;0.0% &nbsp; &nbsp;10 &nbsp; &nbsp;0.8 &nbsp; 0.6 &nbsp; 0.5 &nbsp; 0.9 &nbsp; 0.1</div>
<div>&nbsp; 5.|-- 203.83.223.230 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 0.0% &nbsp; &nbsp;10 &nbsp; &nbsp;0.6 &nbsp; 1.0 &nbsp; 0.4 &nbsp; 2.3 &nbsp; 0.7</div>
<div>&nbsp; 6.|-- 203.83.223.14 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.0% &nbsp; &nbsp;10 &nbsp; &nbsp;1.5 &nbsp; 1.6 &nbsp; 1.4 &nbsp; 2.4 &nbsp; 0.3</div>
<div>&nbsp; 7.|-- 203.83.223.57 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.0% &nbsp; &nbsp;10 &nbsp; &nbsp;1.7 &nbsp; 1.8 &nbsp; 1.5 &nbsp; 2.4 &nbsp; 0.2</div>
<div>&nbsp; 8.|-- 198.32.141.140 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 0.0% &nbsp; &nbsp;10 &nbsp; &nbsp;1.5 &nbsp; 1.7 &nbsp; 1.4 &nbsp; 3.2 &nbsp; 0.5</div>
<div>&nbsp; 9.|-- 66.249.95.122 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.0% &nbsp; &nbsp;10 &nbsp; &nbsp;2.9 &nbsp; 5.4 &nbsp; 2.2 &nbsp;14.7 &nbsp; 4.8</div>
<div>&nbsp;10.|-- 209.85.241.149 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 0.0% &nbsp; &nbsp;10 &nbsp; 54.2 &nbsp;57.7 &nbsp;54.0 &nbsp;76.5 &nbsp; 7.6</div>
<div>&nbsp;11.|-- 209.85.243.23 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.0% &nbsp; &nbsp;10 &nbsp; 57.9 &nbsp;60.1 &nbsp;57.7 &nbsp;79.4 &nbsp; 6.8</div>
<div>&nbsp;12.|-- 72.14.238.8 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.0% &nbsp; &nbsp;10 &nbsp; 58.9 &nbsp;59.3 &nbsp;58.8 &nbsp;62.5 &nbsp; 1.1</div>
<div>&nbsp;13.|-- 72.14.237.179 &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0.0% &nbsp; &nbsp;10 &nbsp; 63.0 &nbsp;65.3 &nbsp;62.8 &nbsp;79.2 &nbsp; 5.4</div>
<div>&nbsp;14.|-- ??? &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 100.0 &nbsp; &nbsp;10 &nbsp; &nbsp;0.0 &nbsp; 0.0 &nbsp; 0.0 &nbsp; 0.0 &nbsp; 0.0</div>
<div>xxx <at> xxx:~$&nbsp;</div>
</div></div>

Gmane