Jump to content

Nightly Backup


Recommended Posts

Spa, your connection is not that bad but gets a hit in Australia already at the node of "ii Net ltd."

 

The other traces a very clear:

Whenever some1 is not using a route passing Kansas, everything is fine (everyone from Europe, Stanig too etc).

 

All the problems in the logs above are clearly naming "Level 3" in Kansas having serious packet drops at the time players are experiencing problems.

It looks like Level-3 is shuffling tons of data at midnight, overloading their switches/structures at this time.

Link to comment
Share on other sites

  • Replies 128
  • Created
  • Last Reply

Top Posters In This Topic

Out of curiosity, I did a google on "level 3 kansas" and came up with this site...

 

http://www.datacentermapping.com/Level-3-Communications/Kansas-City-Data-Center/1278

 

Which lists Level 3 as a Data Center located in Kansas City Kansas. It also provides their contact info if anyone wants to call them toll free and complain. It says they are one of the only Tier 1 providers in the world, which may explain why so much data, but not why they are not set up to do it well.

Edited by Bloody Riz
Link to comment
Share on other sites

@Bloody Riz

 

Actually, the ball is on oneandone's court, as sunrise.net-7's ISP provider.  They can always route traffic around the Kansas node during the vulnerable timeframe.

Edited by VincentTH
Link to comment
Share on other sites

Level3 hosts much of the worlds financial traffic too id guess its on 1and1 myself. The server is not in philly, thats our registered whois address Stanig. It is in 1and1s Kansas Datacenter.
Link to comment
Share on other sites

Level3 hosts much of the worlds financial traffic too id guess its on 1and1 myself. The server is not in philly, thats our registered whois address Stanig. It is in 1and1s Kansas Datacenter.

 

Kyp, is there no way then to have the EnB server host put you on some kind of exclusion from allowing routes through this bottle neck?  I just went through today and MTR'd a lot of the other games I play and none of them hit any hops with this kind of issue.  Not even close, actually.  This is only happening to me for the EnB server.  I'd imagine the same results would apply to everyone else being affected.

Edited by DigitalHytop
Link to comment
Share on other sites

Kyp, is there no way then to have the EnB server host put you on some kind of exclusion from allowing routes through this bottle neck?  I just went through today and MTR'd a lot of the other games I play and none of them hit any hops with this kind of issue.  Not even close, actually.  This is only happening to me for the EnB server.  I'd imagine the same results would apply to everyone else being affected.

 

Back to Germany I would say would fix the problem :P

 

May add a few seconds to gate but lag would not be noticable considering it is not FPS depending on pure milliseconds... 

Edited by SiSL
Link to comment
Share on other sites

Level3 hosts much of the worlds financial traffic too id guess its on 1and1 myself. The server is not in philly, thats our registered whois address Stanig. It is in 1and1s Kansas Datacenter.

I agree with your assessment, Kyp.  My tracert does not go through Level3 and I get similar issues as the others during the same time.  What appears to be heavy packet loss is in the 1and1 network (from me to Net-7).

 

This does not mean that there is not more than one location with similar issues.  There could always be issues in both the 1and1 and Leve3 networks, or one could be causing an issue for the other.

 

 

-----

 

Some things that people need to keep in mind.  A tracert can only show the (potential) data path in one direction: from you (Location A) to Net-7 (Location B).  In order to see the potential path from B to A, you have to do a tracert from Location B.  I say potential path because the tracert uses a different packet type than the UDP packets the game uses.  Different packet types could be routed over a different path, and have different priorities.  Also, when a router/switch gets overloaded, it is more likely to drop a tracert packet than a UDP packet (lower priority packets will get dropped first).  So while a packet loss is not 100% guarantee that packets the gaming packets are being lost (and needing to be resent), it can be indicative of a potential location for an issue to occur.

 

Also, 1and1 can only direct the route for the packets from them to you, and only then to the point where it leaves their network.  They have no control over the route packets take to reach them.

Link to comment
Share on other sites

I ran Winmtr at 945pm pacific time heres the results:

 

{i was experience many issues at the time,  severe lag, when i got in my ship was invisible in all camera views except internal.. i logged and ran this test.}

 

|------------------------------------------------------------------------------------------|
|                                      WinMTR statistics                                   |
|                       Host              -   %  | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
|                           router.belkin -    0 |   28 |   28 |    0 |    0 |    1 |    0 |
|adsl-75-31-175-254.dsl.frs2ca.sbcglobal.net -    0 |   23 |   23 |  355 |  856 | 1309 |  633 |
|             dist2-vlan50.frsn02.pbi.net -    0 |   26 |   26 |  304 |  824 | 1282 |  304 |
|          bb1-p12-0.irvnca.sbcglobal.net -    0 |   23 |   23 |  365 |  871 | 1307 |  612 |
|                            12.122.200.9 -    0 |   25 |   25 |  430 |  845 | 1271 |  437 |
|                            208.51.134.1 -    0 |   25 |   25 |  346 |  853 | 1570 |  346 |
|                          64.209.105.234 -    0 |   25 |   25 |  380 |  893 | 1549 |  380 |
|     ae-10.bb-c.slr.lxa.us.oneandone.net -    0 |   25 |   25 |  397 |  889 | 1548 |  424 |
|  ae-10.gw-distp-a.slr.lxa.oneandone.net -    5 |   22 |   21 |  421 |  831 | 1488 |  421 |
| ae-1.gw-prtr-r5-a.slr.lxa.oneandone.net -    5 |   22 |   21 |  438 |  818 | 1341 |  441 |
|         u16148424.onlinehome-server.com -    0 |   25 |   25 |  400 |  886 | 1445 |  400 |
|________________________________________________|______|______|______|______|______|______|
   WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider
Link to comment
Share on other sites

I say potential path because the tracert uses a different packet type than the UDP packets the game uses.

 

This is only valid for Windows (using ICMP for traceroutes) while Linux is using UDP packets.

 

 



 

Link to comment
Share on other sites

Rock solid connection again, but at sunrise a 5 percent packet loss, 11pm - 1120pm central.

 

I am not observing any lag or connection issue in game however, but as stated, confirming packet loss at this time.

 

[attachment=2392:stanigday2.htm]

Link to comment
Share on other sites

I have a 15 percent loss on 1500 hits...not sure what this node is

207.88.14.194.ptr.us.xo.net 15 963 828 23 27 267 24


this was at around 11:45pm 2/15/2013

 

 

 

ok ...did a global trace via a map function it is ...........drum roll please...... Wichita Kansas

 

(we have found the problem it is Kansas.........)

 

anyway complete report below


WinMTR statistics
Host % Sent Recv Best Avrg Wrst Last
192.168.0.1 0 1501 1501 0 0 227 0
10.148.120.1 0 1501 1501 5 8 113 8
dtr01mnktmn-tge-0-0-1-2.mnkt.mn.charter.com 0 1501 1501 5 9 109 6
dtr01owtnmn-tge-0-1-1-0.owtn.mn.charter.com 1 1497 1496 6 9 100 8
crr01owtnmn-tge-0-0-0-10.owtn.mn.charter.com 0 1501 1501 6 10 109 10
crr01stcdmn-tge-0-3-0-3.stcd.mn.charter.com 0 1501 1501 13 22 118 25
bbr01stcdmn-bue-2.stcd.mn.charter.com 1 1497 1496 14 22 128 15
ip65-46-47-77.z47-46-65.customer.algx.net 0 1501 1501 14 19 119 16
vb1710.rar3.chicago-il.us.xo.net 1 1497 1496 24 31 254 28
207.88.14.194.ptr.us.xo.net 15 963 828 23 27 267 24
206.111.2.10.ptr.us.xo.net 1 1493 1491 24 30 225 25
te-2-4.bb-d.ws.mkc.us.oneandone.net 0 1501 1501 35 39 143 40
ae-105.bb-c.ws.mkc.us.oneandone.net 1 1497 1496 35 39 256 38
ae-11.bb-c.ms.mkc.us.oneandone.net 0 1501 1501 35 39 137 38
ae-10.bb-c.slr.lxa.us.oneandone.net 1 1497 1496 35 39 259 38
ae-11.gw-distp-a.slr.lxa.oneandone.net 1 1497 1496 35 44 343 210
ae-1.gw-prtr-r4-1a.slr.lxa.oneandone.net 1 1497 1496 35 43 377 38
u15438806.onlinehome-server.com 5 1267 1208 35 43 312 39


anyway

Searing

Link to comment
Share on other sites

I have a 15 percent loss on 1500 hits...not sure what this node is

207.88.14.194.ptr.us.xo.net 15 963 828 23 27 267 24


 

 

 

 

ok ...did a global trace via a map function it is ...........drum roll please...... Wichita Kansas

 

(we have found the problem it is Kansas.........)

 

anyway complete report below



Searing

 

Well us rednecks in kansas make do with an old 5hp gas genset and the network is a bunch of cans and string. :P

Link to comment
Share on other sites

So how well do we make our case heard at oneandone?  I guess all we need to know is whether this problem is solvable by the ISP, or whether there is a workaround for it (besides playing other games between 9-10PM PST every night :) )

Edited by VincentTH
Link to comment
Share on other sites

1and1 is of the opinion that those of you that are unfortunately being routed across 'xo.net' is the problem. it is registered as an alternate of http://www.xo.com/

 

That based upon all of the data provided (which was over 1000 packets) seemed to indicate that was where the bottleneck was. Unfortunately, I don't know how to assist here, since we aren't running any jobs and only some folks are experiencing the issue. I do not know a way you can exclude that from being used for your own routing.

Link to comment
Share on other sites

They have a blog, post comments along with others complaining about them there?

 

http://blog.xo.com/

 

Public image about a company like this is sort of important.  Facebook and LinkedIn would be other places to post comments about their company.

Edited by Willbonney
Link to comment
Share on other sites

Here's the story. All times are 2013-02-21 PST:

 

  • 20:37 - logged in, started traceroutes and captures 
  • 20:57 - Captured packet stats for traceroute, 0 loss in 1327 packets
  • 21:06 - 2% packet loss in 9 minutes
  • .. (see below)
  • 21:57 - event appears over
  • 22:21 - 22:43 0% packet loss 

The problem isn't the packets being lost, its the large chunks in which they were being lost. Usually packet loss is seen at a constant rate during a traceroute on a congested pipe for various reasons - ICMP responses are the lowest priority for a router/device to respond to, and if they are too busy they will ignore your ping request. This lack of response is usually either a 100% loss rate (icmp filters) or the router is busy and starts ignoring your pings - but this is usually at a very consistent rate, every 5 to 10 packets you'd lose a couple or so. If you are actually dropping packets and it isn't the router ignoring them. its correctable by the application with TCP and its retransmit powers, or in the UDP case the protocol is written to allow for lossy packets and still function normal (I know Tienbau mentioned tonight a rework of the client which should improve the UDP functionality, and thats awesome, since packet loss will happen occasionally, and lossy connections could explain the gate to logins, long lasting tractor beams, funky warps etc.).

 

However: when running my tests tonight, I'm seeing very large chunks of packets lost at a time instead of slow losses over time (think garden hose: the hose gets kinked and everything stops rather than just the leak you see at the faucet when its not screwed on properly). I can see my request being sent out in the UDP stream, and the server taking a good amount of time to respond if it received my request at all during these chunks of loss. During these periods, all ICMP requests to the last hop stall out, indicating either super over saturated link or CPU spike on that networking device so severe it can't respond (of course I'm speculating here from my experiences, you get the idea).

 

As you can see in the traceroutes I'm only 12 hops away on a very solid and stable connection (I feel no server communication issues 99% of the time which I find fantastic. Gate to login only happens for me once in a blue moon, for example).

 

TCPDump of the traffic between my machine and the server is attached (excluding ICMP traffic used to generate the attached traceroutes). 

 

The last few IPs in the traceroute in the oneandone network block are

 

9 - 74.208.6.106

10 - 74.208.1.117

11 - 74.208.1.176

12 - 74.208.192.215 (of course) 

 

One other interesting note - I see a chunk of interesting packets between 21:32:33 and 21:37:50 that are all from my machine out, using the FF protocol (Protocols in frame: eth:ip:udp:ff). It reads like its pinging home, but this is one part of the stack I'm not familiar with (yet). 

 

I run in packet optimization and locked port mode in the normal net7 proxy launcher.

Please let me know if you need any more information or my specific IP address if the provider so feels inclined to run return path traceroutes during the events.

 

 

Thanks,

 

 

- Censored

Link to comment
Share on other sites

I gave a call today out to a service tech company that I have done work for before that handles a lot of data centers out in Kansas, they know of this site for XO Communications, and has had many trouble calls out to it.  They advised just trying to go around them, as he doesn't expect their issues to be resolved any time soon, they've been having issues with quite a few hackers using this node as a go between, and so have some sort of increased security scan that goes on from 11pm - 12am local time (Central US Time).

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...

×
×
  • Create New...