Frank Bulk via Outages | 17 Dec 03:50 2014

[outages] ipv6.google.com

Anyone else see ipv6.google.com [2607:f8b0:4009:808::2000] down from 7:21 to
7:25 pm (U.S. Central) this evening?

Frank

root <at> nagios:/# tcptraceroute6 2607:f8b0:4009:808::2000
traceroute to 2607:f8b0:4009:808::2000 (2607:f8b0:4009:808::2000) from
2607:fe28:0:1000::5, port 80, from port 38504, 30 hops max, 60 bytes packets
 1  router-core.mtcnet.net (2607:fe28:0:1000::1)  0.847 ms  0.257 ms  0.203
ms
 2  sxct.spnc.mtcnet.net (2607:fe28:11:1002::194)  0.196 ms  0.135 ms  0.155
ms
 3  v6-premier.movl-mlx.fbnt.netins.net (2001:5f8:7f0a:1::1)  4.951 ms
11.178 ms  4.900 ms
 4  v6-ins-kb1-te-12-2-3031.kmrr.netins.net (2001:5f8:2:2::1)  8.831 ms
8.756 ms  8.851 ms
 5  v6-ins-kc1-et-9-3.kmrr.netins.net (2001:5f8::28:1)  8.862 ms  9.098 ms
8.912 ms
 6  2001:438:fffe::e99 (2001:438:fffe::e99)  14.566 ms  13.961 ms  13.968 ms
 7  ae5.cr1.ord2.us.zip.zayo.com (2001:438:ffff::407d:1e59)  23.338 ms
22.321 ms  22.347 ms
 8  ae4.er1.ord7.us.zip.zayo.com (2001:438:ffff::407d:1c32)  23.726 ms
22.467 ms  22.442 ms
 9  2001:4860:1:1:0:193d:0:1d (2001:4860:1:1:0:193d:0:1d)  22.449 ms  22.453
ms  22.383 ms
10  2001:4860::1:0:3f7 (2001:4860::1:0:3f7)  22.770 ms  26.090 ms  22.576 ms
11  2001:4860:0:1::1247 (2001:4860:0:1::1247)  23.265 ms  23.246 ms  23.279
ms
12  2607:f8b0:4009:808::2000 (2607:f8b0:4009:808::2000)  22.808 ms [open]
22.719 ms  22.708 ms
(Continue reading)

[outages] Cloudflare - sporadic ipv

Anyone else seeing anything ?  Their status page is all green at the moment.

I’m getting failures on certain Cloudflare IPv6 address.  Traceroutes look okay - i.e. I’m not super
concerned that they don’t complete, they make it to GTT from where I’m having the issues. 
www.cloudflare.com is an example, as is (some of the records for) 0.t.4cdn.org.  Traceroutes and curl
loops below.

root <at> phosphor:/home/bapplegate# traceroute -I -6 0.t.4cdn.org
traceroute to 0.t.4cdn.org (2400:cb00:2048:1::8d65:721a), 30 hops max, 80 byte packets
 1  2607:ff70:11::1 (2607:ff70:11::1)  0.485 ms  0.529 ms  0.608 ms
 2  pc44.core0-5.cncndc.net (2607:ff70:2::2d)  0.276 ms  0.308 ms  0.361 ms
 3  tenG-0-2-0.transit0-5.cncndc.net (2607:ff70:2::be)  0.644 ms  0.641 ms  0.638 ms
 4  xe-5-0-0.bar1.Cincinnati1.Level3.net (2001:1900:2100::2b5)  0.360 ms  0.363 ms  0.363 ms
 5  vl-4066.car2.Chicago1.Level3.net (2001:1900:4:1::27d)  24.373 ms  24.454 ms  24.549 ms
 6  vl-4062.edge2.Chicago2.Level3.net (2001:1900:4:1::2a)  23.884 ms  23.800 ms  23.811 ms
 7  2001:1900:4:1::be (2001:1900:4:1::be)  24.321 ms  24.276 ms  24.277 ms
 8  vl-4061.car1.NewYork2.Level3.net (2001:1900:4:1::106)  29.840 ms  29.811 ms  29.875 ms
 9  vl-4041.edge1.NewYork1.Level3.net (2001:1900:4:1::101)  29.135 ms  29.250 ms  29.229 ms
10  * * *
11  NTT-level3-40G.NewYork1.Level3.net (2001:1900:4:3::2b6)  29.440 ms  29.353 ms  29.380 ms
12  ae-3.r05.nycmny01.us.bb.gin.ntt.net (2001:418:0:2000::2bd)  29.957 ms  30.181 ms  30.231 ms
13  2001:590:2:1b::1 (2001:590:2:1b::1)  44.675 ms  39.849 ms  39.800 ms
14  ae8-60g.cr1.nyc3.us.as4436.gtt.net (2001:590::451f:5f5c)  29.564 ms  29.540 ms  29.526 ms
15  ip6.gtt.net (2001:668:0:3::8000:30a1)  29.472 ms  29.585 ms  29.915 ms
16  xe-3-0-0.nyc41.ip6.gtt.net (2001:668:0:2::1:2672)  104.521 ms  37.085 ms  29.862 ms
17  lo0.cr1.ewr1.us.as4436.gtt.net (2001:590::451f:5ff8)  30.635 ms  30.138 ms  30.032 ms
18  * * *
19  * * *
20  * * *
21  * * *
(Continue reading)

Mike Lyon via Outages | 15 Dec 19:35 2014

[outages] Anyone else seeing problems with 8x8 today?

Have a bunch of phones that cant register. Their tech support simply said there are large problems on the internet today and alot of customers are having problems.

No details though. Some details would be nice...

Anyone else having issues?

-Mike

<div>
<p dir="ltr">Have a bunch of phones that cant register. Their tech support simply said there are large problems on the internet today and alot of customers are having problems.</p>
<p dir="ltr">No details though. Some details would be nice...</p>
<p dir="ltr">Anyone else having issues?</p>
<p dir="ltr">-Mike</p>
</div>
Eric Tykwinski via Outages | 15 Dec 15:42 2014

[outages] Verizon FiOS issues in Philadelphia/Delaware

We've received numerous calls from customers about issues, so just wanted to
let people know.
It seems like it's limited to FiOS Voice and Internet services...

Sincerely,

Eric Tykwinski
TrueNet, Inc.
P: 610-429-8300
F: 610-429-3222

Ca By via Outages | 13 Dec 07:59 2014

[outages] Level3 Looping Traffic in LAX

Seems hop 7 and 8 are the same box.... This traceroute works fine from AT&T and Sprint, they deliver it correctly to Korea instead of black-holing it in LAX

traceroute to 1.201.0.39 (1.201.0.39), 30 hops max, 60 byte packets
 x
 3  te-0-4-0-21-sur03.seattle.wa.seattle.comcast.net (68.86.113.61)  25.810 ms  25.771 ms  25.733 ms
 4  be-20-ar01.seattle.wa.seattle.comcast.net (69.139.164.129)  25.670 ms  25.630 ms  25.577 ms
 5  he-1-3-0-0-10-cr01.seattle.wa.ibone.comcast.net (68.86.93.165)  25.508 ms  25.473 ms  25.405 ms
 6  4.68.63.65 (4.68.63.65)  22.755 ms  13.858 ms  16.425 ms
 7  * * ae-3-80.edge5.LosAngeles1.Level3.net (4.69.144.139)  61.317 ms
 8  * ae-3-80.edge5.LosAngeles1.Level3.net (4.69.144.139)  58.978 ms  58.956 ms
 9  * * *
10  * * *
11  * * *
12  * * *

<div><div dir="ltr">
<div>Seems hop 7 and 8 are the same box.... This traceroute works fine from AT&amp;T and Sprint, they deliver it correctly to Korea instead of black-holing it in LAX</div>
<div><br></div>
<div>traceroute to 1.201.0.39 (1.201.0.39), 30 hops max, 60 byte packets</div>
<div>&nbsp;x</div>
<div>&nbsp;3 &nbsp;<a href="http://te-0-4-0-21-sur03.seattle.wa.seattle.comcast.net">te-0-4-0-21-sur03.seattle.wa.seattle.comcast.net</a> (68.86.113.61) &nbsp;25.810 ms &nbsp;25.771 ms &nbsp;25.733 ms</div>
<div>&nbsp;4 &nbsp;<a href="http://be-20-ar01.seattle.wa.seattle.comcast.net">be-20-ar01.seattle.wa.seattle.comcast.net</a> (69.139.164.129) &nbsp;25.670 ms &nbsp;25.630 ms &nbsp;25.577 ms</div>
<div>&nbsp;5 &nbsp;<a href="http://he-1-3-0-0-10-cr01.seattle.wa.ibone.comcast.net">he-1-3-0-0-10-cr01.seattle.wa.ibone.comcast.net</a> (68.86.93.165) &nbsp;25.508 ms &nbsp;25.473 ms &nbsp;25.405 ms</div>
<div>&nbsp;6 &nbsp;4.68.63.65 (4.68.63.65) &nbsp;22.755 ms &nbsp;13.858 ms &nbsp;16.425 ms</div>
<div>&nbsp;7 &nbsp;* * <a href="http://ae-3-80.edge5.LosAngeles1.Level3.net">ae-3-80.edge5.LosAngeles1.Level3.net</a> (4.69.144.139) &nbsp;61.317 ms</div>
<div>&nbsp;8 &nbsp;* <a href="http://ae-3-80.edge5.LosAngeles1.Level3.net">ae-3-80.edge5.LosAngeles1.Level3.net</a> (4.69.144.139) &nbsp;58.978 ms &nbsp;58.956 ms</div>
<div>&nbsp;9 &nbsp;* * *</div>
<div>10 &nbsp;* * *</div>
<div>11 &nbsp;* * *</div>
<div>12 &nbsp;* * *</div>
<div><br></div>
</div></div>
Robert Glover via Outages | 12 Dec 09:45 2014

[outages] Cogent - Latency/Packet Loss out of SFO

Good morning,

Anyone else seeing gnarly (50%+) packet loss through Cogent in the Bay 
Area??  Looks like it's been coming & going for the last hour.

Looks like in the last 5 minutes things have stabilized.

-Robert
Frank Bulk | 11 Dec 00:02 2014
Picon

1&1 outage Tuesday afternoon/evening

It wasn't posted at the time, but access to 1&1's website and some of its
DNS servers were down Tuesday from ~3 pm to ~6 pm (U.S. Central).  I can’t
pin the time down any tighter because I don’t monitor the 1&1 website or DNS
servers.

There's still a generic system status on their page
(http://status-1and1.com/#Incident), but no indication that it's over, even
though it appears most customers are in good shape.

1and1 posted (https://twitter.com/1and1help/status/542433725649137664) that
it was a DDoS attack.  Their last twitter message
(https://twitter.com/1and1help/status/542760687203020802) says the issue is
nearly resolved.

Frank

[outages] Experiencing major packet loss and Latency out of Latisys Chicago

Hi Outages.org List,

 

I’m seeing very intermittent packet loss and latency spikes inbound and outbound from Latisys Chicago. Is anyone else experiencing issues? Our host who collocates their equipment there has told us that Latisys is having a network event. Any more information on this?

 

IP: 192.184.114.232

 

TCPING from Verizon-NJ to Latisys Chicago: http://pastebin.com/CYvxP1qi

TRACERT from Verizon-NJ to Latisys Chicago: http://pastebin.com/bskiJk4r

 

TCPING From EarthlinkBusiness-NYC to Latisys Chicago: http://pastebin.com/KyNLVR0n

TRACERT From EarthlinkBusiness-NYC to Latisys Chicago: http://pastebin.com/m94nXyd0

 

PING from Latisys to GoogleDNS: http://pastebin.com/p8w3HC7m

Traceroute from Latisys to GoogleDNS: http://pastebin.com/TXsJjypf

 

According to the tracert’s the issue seems to be within Latisys.

<!-- /* Font Definitions */ <at> font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4; mso-font-charset:1; mso-generic-font-family:roman; mso-font-format:other; mso-font-pitch:variable; mso-font-signature:0 0 0 0 0 0;} <at> font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4; mso-font-charset:0; mso-generic-font-family:swiss; mso-font-pitch:variable; mso-font-signature:-536870145 1073786111 1 0 415 0;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {mso-style-unhide:no; mso-style-qformat:yes; mso-style-parent:""; margin:0in; margin-bottom:.0001pt; mso-pagination:widow-orphan; font-size:11.0pt; font-family:"Calibri",sans-serif; mso-fareast-font-family:Calibri; mso-fareast-theme-font:minor-latin; mso-bidi-font-family:"Times New Roman";} a:link, span.MsoHyperlink {mso-style-noshow:yes; mso-style-priority:99; color:#0563C1; text-decoration:underline; text-underline:single;} a:visited, span.MsoHyperlinkFollowed {mso-style-noshow:yes; mso-style-priority:99; color:#954F72; mso-themecolor:followedhyperlink; text-decoration:underline; text-underline:single;} .MsoChpDefault {mso-style-type:export-only; mso-default-props:yes; font-size:10.0pt; mso-ansi-font-size:10.0pt; mso-bidi-font-size:10.0pt;} <at> page WordSection1 {size:8.5in 11.0in; margin:1.0in 1.0in 1.0in 1.0in; mso-header-margin:.5in; mso-footer-margin:.5in; mso-paper-source:0;} div.WordSection1 {page:WordSection1;} --> -- -- Alex Vanino LinkedIn: http://lnkd.in/v63TbM
<div>
    <p class="MsoNormal">Hi Outages.org List,<p></p></p>
    <p class="MsoNormal"><p>&nbsp;</p></p>
    <p class="MsoNormal">I&rsquo;m seeing very intermittent packet loss and
      latency spikes
      inbound and outbound from Latisys Chicago. Is anyone else
      experiencing issues?
      Our host who collocates their equipment there has told us that
      Latisys is
      having a network event. Any more information on this?<p></p></p>
    <p class="MsoNormal"><p>&nbsp;</p></p>
    <p class="MsoNormal">IP: 192.184.114.232<p></p></p>
    <p class="MsoNormal"><p>&nbsp;</p></p>
    <p class="MsoNormal">TCPING from Verizon-NJ to Latisys Chicago: <a href="http://pastebin.com/CYvxP1qi">http://pastebin.com/CYvxP1qi</a>
      <p></p></p>
    <p class="MsoNormal">TRACERT from Verizon-NJ to Latisys Chicago: <a href="http://pastebin.com/bskiJk4r">http://pastebin.com/bskiJk4r</a>
      <p></p></p>
    <p class="MsoNormal"><p>&nbsp;</p></p>
    <p class="MsoNormal">TCPING From EarthlinkBusiness-NYC to Latisys
      Chicago: <a href="http://pastebin.com/KyNLVR0n">http://pastebin.com/KyNLVR0n</a>
      <p></p></p>
    <p class="MsoNormal">TRACERT From EarthlinkBusiness-NYC to Latisys
      Chicago: <a href="http://pastebin.com/m94nXyd0">http://pastebin.com/m94nXyd0</a>
      <p></p></p>
    <p class="MsoNormal"><p>&nbsp;</p></p>
    <p class="MsoNormal">PING from Latisys to GoogleDNS: <a href="http://pastebin.com/p8w3HC7m">http://pastebin.com/p8w3HC7m</a>
      <p></p></p>
    <p class="MsoNormal">Traceroute from Latisys to GoogleDNS: <a href="http://pastebin.com/TXsJjypf">http://pastebin.com/TXsJjypf</a>
      <p></p></p>
    <p class="MsoNormal"><p>&nbsp;</p></p>
    <p class="MsoNormal">According to the tracert&rsquo;s the issue seems to
      be within
      Latisys.<p></p></p>

&lt;!--
 /* Font Definitions */
  <at> font-face
	{font-family:"Cambria Math";
	panose-1:2 4 5 3 5 4 6 3 2 4;
	mso-font-charset:1;
	mso-generic-font-family:roman;
	mso-font-format:other;
	mso-font-pitch:variable;
	mso-font-signature:0 0 0 0 0 0;}
 <at> font-face
	{font-family:Calibri;
	panose-1:2 15 5 2 2 2 4 3 2 4;
	mso-font-charset:0;
	mso-generic-font-family:swiss;
	mso-font-pitch:variable;
	mso-font-signature:-536870145 1073786111 1 0 415 0;}
 /* Style Definitions */
 p.MsoNormal, li.MsoNormal, div.MsoNormal
	{mso-style-unhide:no;
	mso-style-qformat:yes;
	mso-style-parent:"";
	margin:0in;
	margin-bottom:.0001pt;
	mso-pagination:widow-orphan;
	font-size:11.0pt;
	font-family:"Calibri",sans-serif;
	mso-fareast-font-family:Calibri;
	mso-fareast-theme-font:minor-latin;
	mso-bidi-font-family:"Times New Roman";}
a:link, span.MsoHyperlink
	{mso-style-noshow:yes;
	mso-style-priority:99;
	color:#0563C1;
	text-decoration:underline;
	text-underline:single;}
a:visited, span.MsoHyperlinkFollowed
	{mso-style-noshow:yes;
	mso-style-priority:99;
	color:#954F72;
	mso-themecolor:followedhyperlink;
	text-decoration:underline;
	text-underline:single;}
.MsoChpDefault
	{mso-style-type:export-only;
	mso-default-props:yes;
	font-size:10.0pt;
	mso-ansi-font-size:10.0pt;
	mso-bidi-font-size:10.0pt;}
 <at> page WordSection1
	{size:8.5in 11.0in;
	margin:1.0in 1.0in 1.0in 1.0in;
	mso-header-margin:.5in;
	mso-footer-margin:.5in;
	mso-paper-source:0;}
div.WordSection1
	{page:WordSection1;}
--&gt;
--

-- 

--
Alex Vanino
LinkedIn: <a class="moz-txt-link-freetext" href="http://lnkd.in/v63TbM">http://lnkd.in/v63TbM</a>
  </div>
Anthony Hook via Outages | 10 Dec 17:30 2014

[outages] Charter non-PRI phones - Central/Northern Wisconsin

Non-PRI phones by Charter Communications are down in Central and Northern Wisconsin, included the ability to call their support/ticketing entirely.

Update: as I type this, it looks like things have been resolved.

Further updates as events warrant.


--
- Anthony Hook
<div><div dir="ltr">Non-PRI phones by Charter Communications are down in Central and Northern Wisconsin, included the ability to call their support/ticketing entirely.<div><br></div>
<div>Update: as I type this, it looks like things have been resolved.</div>
<div><br></div>
<div>Further updates as events warrant.</div>
<div><br></div>
<div><br></div>
<div>-- <br><div class="gmail_signature">- Anthony Hook</div>
</div>
</div></div>
Frank Bulk | 10 Dec 10:33 2014
Picon

Hacker collective targets PlayStation Network, causes service outage

http://www.scmagazine.com/hacker-collective-targets-playstation-network-caus
es-service-outage/article/387319/

Something a little more light hearted (well, at least for some folk --
others would take this outage more seriously than a fb outage).

Frank

Jim Popovitch via Outages | 8 Dec 16:58 2014

[outages] NameCheap DNS failure

"We regret to let you know that we are experiencing a DDoS attack
against our default DNS system v2."

http://status.namecheap.com/archives/17727

Gmane