Here's the story. All times are 2013-02-21 PST:
20:37 - logged in, started traceroutes and captures
20:57 - Captured packet stats for traceroute, 0 loss in 1327 packets
21:06 - 2% packet loss in 9 minutes
.. (see below)
21:57 - event appears over
22:21 - 22:43 0% packet loss
The problem isn't the packets being lost, its the large chunks in which they were being lost. Usually packet loss is seen at a constant rate during a traceroute on a congested pipe for various reasons - ICMP responses are the lowest priority for a router/device to respond to, and if they are too busy they will ignore your ping request. This lack of response is usually either a 100% loss rate (icmp filters) or the router is busy and starts ignoring your pings - but this is usually at a very consistent rate, every 5 to 10 packets you'd lose a couple or so. If you are actually dropping packets and it isn't the router ignoring them. its correctable by the application with TCP and its retransmit powers, or in the UDP case the protocol is written to allow for lossy packets and still function normal (I know Tienbau mentioned tonight a rework of the client which should improve the UDP functionality, and thats awesome, since packet loss will happen occasionally, and lossy connections could explain the gate to logins, long lasting tractor beams, funky warps etc.).
However: when running my tests tonight, I'm seeing very large chunks of packets lost at a time instead of slow losses over time (think garden hose: the hose gets kinked and everything stops rather than just the leak you see at the faucet when its not screwed on properly). I can see my request being sent out in the UDP stream, and the server taking a good amount of time to respond if it received my request at all during these chunks of loss. During these periods, all ICMP requests to the last hop stall out, indicating either super over saturated link or CPU spike on that networking device so severe it can't respond (of course I'm speculating here from my experiences, you get the idea).
As you can see in the traceroutes I'm only 12 hops away on a very solid and stable connection (I feel no server communication issues 99% of the time which I find fantastic. Gate to login only happens for me once in a blue moon, for example).
TCPDump of the traffic between my machine and the server is attached (excluding ICMP traffic used to generate the attached traceroutes).
The last few IPs in the traceroute in the oneandone network block are
9 - 74.208.6.106
10 - 74.208.1.117
11 - 74.208.1.176
12 - 74.208.192.215 (of course)
One other interesting note - I see a chunk of interesting packets between 21:32:33 and 21:37:50 that are all from my machine out, using the FF protocol (Protocols in frame: eth:ip:udp:ff). It reads like its pinging home, but this is one part of the stack I'm not familiar with (yet).
I run in packet optimization and locked port mode in the normal net7 proxy launcher.
Please let me know if you need any more information or my specific IP address if the provider so feels inclined to run return path traceroutes during the events.
Thanks,
- Censored