Zackman [LDEV] Posted February 14, 2013 Report Share Posted February 14, 2013 Spa, your connection is not that bad but gets a hit in Australia already at the node of "ii Net ltd." The other traces a very clear: Whenever some1 is not using a route passing Kansas, everything is fine (everyone from Europe, Stanig too etc). All the problems in the logs above are clearly naming "Level 3" in Kansas having serious packet drops at the time players are experiencing problems. It looks like Level-3 is shuffling tons of data at midnight, overloading their switches/structures at this time. Quote Link to comment Share on other sites More sharing options...
Riz Posted February 14, 2013 Report Share Posted February 14, 2013 (edited) Out of curiosity, I did a google on "level 3 kansas" and came up with this site... http://www.datacentermapping.com/Level-3-Communications/Kansas-City-Data-Center/1278 Which lists Level 3 as a Data Center located in Kansas City Kansas. It also provides their contact info if anyone wants to call them toll free and complain. It says they are one of the only Tier 1 providers in the world, which may explain why so much data, but not why they are not set up to do it well. Edited February 14, 2013 by Bloody Riz Quote Link to comment Share on other sites More sharing options...
VincentTH Posted February 14, 2013 Report Share Posted February 14, 2013 (edited) @Bloody Riz Actually, the ball is on oneandone's court, as sunrise.net-7's ISP provider. They can always route traffic around the Kansas node during the vulnerable timeframe. Edited February 14, 2013 by VincentTH Quote Link to comment Share on other sites More sharing options...
Kyp [LDEV] Posted February 14, 2013 Report Share Posted February 14, 2013 Level3 hosts much of the worlds financial traffic too id guess its on 1and1 myself. The server is not in philly, thats our registered whois address Stanig. It is in 1and1s Kansas Datacenter. Quote Link to comment Share on other sites More sharing options...
DigitalHytop Posted February 14, 2013 Author Report Share Posted February 14, 2013 (edited) Level3 hosts much of the worlds financial traffic too id guess its on 1and1 myself. The server is not in philly, thats our registered whois address Stanig. It is in 1and1s Kansas Datacenter. Kyp, is there no way then to have the EnB server host put you on some kind of exclusion from allowing routes through this bottle neck? I just went through today and MTR'd a lot of the other games I play and none of them hit any hops with this kind of issue. Not even close, actually. This is only happening to me for the EnB server. I'd imagine the same results would apply to everyone else being affected. Edited February 14, 2013 by DigitalHytop Quote Link to comment Share on other sites More sharing options...
SiSL Posted February 14, 2013 Report Share Posted February 14, 2013 (edited) Kyp, is there no way then to have the EnB server host put you on some kind of exclusion from allowing routes through this bottle neck? I just went through today and MTR'd a lot of the other games I play and none of them hit any hops with this kind of issue. Not even close, actually. This is only happening to me for the EnB server. I'd imagine the same results would apply to everyone else being affected. Back to Germany I would say would fix the problem :P May add a few seconds to gate but lag would not be noticable considering it is not FPS depending on pure milliseconds... Edited February 14, 2013 by SiSL Quote Link to comment Share on other sites More sharing options...
PirateLad Posted February 15, 2013 Report Share Posted February 15, 2013 Level3 hosts much of the worlds financial traffic too id guess its on 1and1 myself. The server is not in philly, thats our registered whois address Stanig. It is in 1and1s Kansas Datacenter. I agree with your assessment, Kyp. My tracert does not go through Level3 and I get similar issues as the others during the same time. What appears to be heavy packet loss is in the 1and1 network (from me to Net-7). This does not mean that there is not more than one location with similar issues. There could always be issues in both the 1and1 and Leve3 networks, or one could be causing an issue for the other. ----- Some things that people need to keep in mind. A tracert can only show the (potential) data path in one direction: from you (Location A) to Net-7 (Location B). In order to see the potential path from B to A, you have to do a tracert from Location B. I say potential path because the tracert uses a different packet type than the UDP packets the game uses. Different packet types could be routed over a different path, and have different priorities. Also, when a router/switch gets overloaded, it is more likely to drop a tracert packet than a UDP packet (lower priority packets will get dropped first). So while a packet loss is not 100% guarantee that packets the gaming packets are being lost (and needing to be resent), it can be indicative of a potential location for an issue to occur. Also, 1and1 can only direct the route for the packets from them to you, and only then to the point where it leaves their network. They have no control over the route packets take to reach them. Quote Link to comment Share on other sites More sharing options...
Woodstock HGM [HGM] Posted February 15, 2013 Report Share Posted February 15, 2013 I ran Winmtr at 945pm pacific time heres the results: {i was experience many issues at the time, severe lag, when i got in my ship was invisible in all camera views except internal.. i logged and ran this test.} |------------------------------------------------------------------------------------------| | WinMTR statistics | | Host - % | Sent | Recv | Best | Avrg | Wrst | Last | |------------------------------------------------|------|------|------|------|------|------| | router.belkin - 0 | 28 | 28 | 0 | 0 | 1 | 0 | |adsl-75-31-175-254.dsl.frs2ca.sbcglobal.net - 0 | 23 | 23 | 355 | 856 | 1309 | 633 | | dist2-vlan50.frsn02.pbi.net - 0 | 26 | 26 | 304 | 824 | 1282 | 304 | | bb1-p12-0.irvnca.sbcglobal.net - 0 | 23 | 23 | 365 | 871 | 1307 | 612 | | 12.122.200.9 - 0 | 25 | 25 | 430 | 845 | 1271 | 437 | | 208.51.134.1 - 0 | 25 | 25 | 346 | 853 | 1570 | 346 | | 64.209.105.234 - 0 | 25 | 25 | 380 | 893 | 1549 | 380 | | ae-10.bb-c.slr.lxa.us.oneandone.net - 0 | 25 | 25 | 397 | 889 | 1548 | 424 | | ae-10.gw-distp-a.slr.lxa.oneandone.net - 5 | 22 | 21 | 421 | 831 | 1488 | 421 | | ae-1.gw-prtr-r5-a.slr.lxa.oneandone.net - 5 | 22 | 21 | 438 | 818 | 1341 | 441 | | u16148424.onlinehome-server.com - 0 | 25 | 25 | 400 | 886 | 1445 | 400 | |________________________________________________|______|______|______|______|______|______| WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider Quote Link to comment Share on other sites More sharing options...
Zackman [LDEV] Posted February 15, 2013 Report Share Posted February 15, 2013 I say potential path because the tracert uses a different packet type than the UDP packets the game uses. This is only valid for Windows (using ICMP for traceroutes) while Linux is using UDP packets. Quote Link to comment Share on other sites More sharing options...
SiSL Posted February 16, 2013 Report Share Posted February 16, 2013 Today finally I could get in middle of lagzone time :) 19% loss at ae-5-5.car1.KansasCity1.Level3.net Even it is better than analyze fail chance, gating is something we do more than analyze... Quote Link to comment Share on other sites More sharing options...
Stanig[IS] Posted February 16, 2013 Report Share Posted February 16, 2013 Rock solid connection again, but at sunrise a 5 percent packet loss, 11pm - 1120pm central. I am not observing any lag or connection issue in game however, but as stated, confirming packet loss at this time. [attachment=2392:stanigday2.htm] Quote Link to comment Share on other sites More sharing options...
Woodstock HGM [HGM] Posted February 16, 2013 Report Share Posted February 16, 2013 ran WinMTR at 9pm pacific time winthe server issues happen. heres the results: Quote Link to comment Share on other sites More sharing options...
Searing Posted February 16, 2013 Report Share Posted February 16, 2013 I have a 15 percent loss on 1500 hits...not sure what this node is 207.88.14.194.ptr.us.xo.net 15 963 828 23 27 267 24 this was at around 11:45pm 2/15/2013 ok ...did a global trace via a map function it is ...........drum roll please...... Wichita Kansas (we have found the problem it is Kansas.........) anyway complete report below WinMTR statistics Host % Sent Recv Best Avrg Wrst Last 192.168.0.1 0 1501 1501 0 0 227 0 10.148.120.1 0 1501 1501 5 8 113 8 dtr01mnktmn-tge-0-0-1-2.mnkt.mn.charter.com 0 1501 1501 5 9 109 6 dtr01owtnmn-tge-0-1-1-0.owtn.mn.charter.com 1 1497 1496 6 9 100 8 crr01owtnmn-tge-0-0-0-10.owtn.mn.charter.com 0 1501 1501 6 10 109 10 crr01stcdmn-tge-0-3-0-3.stcd.mn.charter.com 0 1501 1501 13 22 118 25 bbr01stcdmn-bue-2.stcd.mn.charter.com 1 1497 1496 14 22 128 15 ip65-46-47-77.z47-46-65.customer.algx.net 0 1501 1501 14 19 119 16 vb1710.rar3.chicago-il.us.xo.net 1 1497 1496 24 31 254 28 207.88.14.194.ptr.us.xo.net 15 963 828 23 27 267 24 206.111.2.10.ptr.us.xo.net 1 1493 1491 24 30 225 25 te-2-4.bb-d.ws.mkc.us.oneandone.net 0 1501 1501 35 39 143 40 ae-105.bb-c.ws.mkc.us.oneandone.net 1 1497 1496 35 39 256 38 ae-11.bb-c.ms.mkc.us.oneandone.net 0 1501 1501 35 39 137 38 ae-10.bb-c.slr.lxa.us.oneandone.net 1 1497 1496 35 39 259 38 ae-11.gw-distp-a.slr.lxa.oneandone.net 1 1497 1496 35 44 343 210 ae-1.gw-prtr-r4-1a.slr.lxa.oneandone.net 1 1497 1496 35 43 377 38 u15438806.onlinehome-server.com 5 1267 1208 35 43 312 39 anyway Searing Quote Link to comment Share on other sites More sharing options...
Kyp [LDEV] Posted February 16, 2013 Report Share Posted February 16, 2013 Given our history with your computers Searing, I consider anything less than 20% packet loss to be 'Normal'. ;) Quote Link to comment Share on other sites More sharing options...
Klyde Posted February 16, 2013 Report Share Posted February 16, 2013 I have a 15 percent loss on 1500 hits...not sure what this node is 207.88.14.194.ptr.us.xo.net 15 963 828 23 27 267 24 ok ...did a global trace via a map function it is ...........drum roll please...... Wichita Kansas (we have found the problem it is Kansas.........) anyway complete report below Searing Well us rednecks in kansas make do with an old 5hp gas genset and the network is a bunch of cans and string. :P Quote Link to comment Share on other sites More sharing options...
VincentTH Posted February 18, 2013 Report Share Posted February 18, 2013 (edited) So how well do we make our case heard at oneandone? I guess all we need to know is whether this problem is solvable by the ISP, or whether there is a workaround for it (besides playing other games between 9-10PM PST every night :) ) Edited February 18, 2013 by VincentTH Quote Link to comment Share on other sites More sharing options...
Kyp [LDEV] Posted February 19, 2013 Report Share Posted February 19, 2013 Will submit everything that's been supplied and see what they have to say. Quote Link to comment Share on other sites More sharing options...
Rezwalker Posted February 20, 2013 Report Share Posted February 20, 2013 One more set I did the other day and forgot to post [attachment=2402:10 45-11 00 PM EST.htm] [attachment=2403:12.15-12.30 AM EST.htm] Quote Link to comment Share on other sites More sharing options...
Kyp [LDEV] Posted February 22, 2013 Report Share Posted February 22, 2013 1and1 is of the opinion that those of you that are unfortunately being routed across 'xo.net' is the problem. it is registered as an alternate of http://www.xo.com/ That based upon all of the data provided (which was over 1000 packets) seemed to indicate that was where the bottleneck was. Unfortunately, I don't know how to assist here, since we aren't running any jobs and only some folks are experiencing the issue. I do not know a way you can exclude that from being used for your own routing. Quote Link to comment Share on other sites More sharing options...
Willbonney Posted February 22, 2013 Report Share Posted February 22, 2013 (edited) They have a blog, post comments along with others complaining about them there? http://blog.xo.com/ Public image about a company like this is sort of important. Facebook and LinkedIn would be other places to post comments about their company. Edited February 22, 2013 by Willbonney Quote Link to comment Share on other sites More sharing options...
kokushibyou Posted February 22, 2013 Report Share Posted February 22, 2013 Here's the story. All times are 2013-02-21 PST: 20:37 - logged in, started traceroutes and captures 20:57 - Captured packet stats for traceroute, 0 loss in 1327 packets 21:06 - 2% packet loss in 9 minutes .. (see below) 21:57 - event appears over 22:21 - 22:43 0% packet loss The problem isn't the packets being lost, its the large chunks in which they were being lost. Usually packet loss is seen at a constant rate during a traceroute on a congested pipe for various reasons - ICMP responses are the lowest priority for a router/device to respond to, and if they are too busy they will ignore your ping request. This lack of response is usually either a 100% loss rate (icmp filters) or the router is busy and starts ignoring your pings - but this is usually at a very consistent rate, every 5 to 10 packets you'd lose a couple or so. If you are actually dropping packets and it isn't the router ignoring them. its correctable by the application with TCP and its retransmit powers, or in the UDP case the protocol is written to allow for lossy packets and still function normal (I know Tienbau mentioned tonight a rework of the client which should improve the UDP functionality, and thats awesome, since packet loss will happen occasionally, and lossy connections could explain the gate to logins, long lasting tractor beams, funky warps etc.). However: when running my tests tonight, I'm seeing very large chunks of packets lost at a time instead of slow losses over time (think garden hose: the hose gets kinked and everything stops rather than just the leak you see at the faucet when its not screwed on properly). I can see my request being sent out in the UDP stream, and the server taking a good amount of time to respond if it received my request at all during these chunks of loss. During these periods, all ICMP requests to the last hop stall out, indicating either super over saturated link or CPU spike on that networking device so severe it can't respond (of course I'm speculating here from my experiences, you get the idea). As you can see in the traceroutes I'm only 12 hops away on a very solid and stable connection (I feel no server communication issues 99% of the time which I find fantastic. Gate to login only happens for me once in a blue moon, for example). TCPDump of the traffic between my machine and the server is attached (excluding ICMP traffic used to generate the attached traceroutes). The last few IPs in the traceroute in the oneandone network block are 9 - 74.208.6.106 10 - 74.208.1.117 11 - 74.208.1.176 12 - 74.208.192.215 (of course) One other interesting note - I see a chunk of interesting packets between 21:32:33 and 21:37:50 that are all from my machine out, using the FF protocol (Protocols in frame: eth:ip:udp:ff). It reads like its pinging home, but this is one part of the stack I'm not familiar with (yet). I run in packet optimization and locked port mode in the normal net7 proxy launcher. Please let me know if you need any more information or my specific IP address if the provider so feels inclined to run return path traceroutes during the events. Thanks, - Censored Quote Link to comment Share on other sites More sharing options...
kokushibyou Posted February 22, 2013 Report Share Posted February 22, 2013 Also please feel free to forward any and all of this data on to the hosting provider. Its very clearly the last hop and it should be apparent in this data. Quote Link to comment Share on other sites More sharing options...
Kyp [LDEV] Posted February 22, 2013 Report Share Posted February 22, 2013 I have forwarded your information, thanks Censored. Quote Link to comment Share on other sites More sharing options...
Willbonney Posted February 23, 2013 Report Share Posted February 23, 2013 I gave a call today out to a service tech company that I have done work for before that handles a lot of data centers out in Kansas, they know of this site for XO Communications, and has had many trouble calls out to it. They advised just trying to go around them, as he doesn't expect their issues to be resolved any time soon, they've been having issues with quite a few hackers using this node as a go between, and so have some sort of increased security scan that goes on from 11pm - 12am local time (Central US Time). Quote Link to comment Share on other sites More sharing options...
caine6621 Posted March 2, 2013 Report Share Posted March 2, 2013 I was having sever lag issues for two days. Game was unplayable. At around packet 800 of the first test the lag stopped so I ran a second test as well Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.