Jump to content

Nightly Backup


Recommended Posts

  • Replies 128
  • Created
  • Last Reply

Top Posters In This Topic

I've made another change, let me know how it goes.

Last night seemed somewhat better but I still had several stuck at load screens and a lot of lag.  Spent a lot of time in combat and not so much roaming around, so I'm not sure if this was the reason for less issues or not.  But I was noticing the lag during combat as well (while targeting next mob to kill, taking several seconds to actually engage the thrusters to autofollow.  Hitting fire button and also taking several seconds to engage).  Also, on a couple occasions I was dropped to desktop attempting to select jobs at the terminal in Aragoth Station, prior to grinding combat.  All of this started at the usual 12pm EST and lasted on through to 1:30am.

Link to comment
Share on other sites

Since the fix Kyp put in, and especially Build 371, gating has improved a lot.  THere are still occasional hangs in the Blue Galactic Map Screen (BGMS), but those happen rarely outside of the 9PM-10PM PST window, when the ISP does its nightly backup/maintenance (I guess).  I think we could improve that with a longer timeout or more retries from the server, because the client is obviously waiting for a packet from the server which never comes.

 

Last night, I did a single simple trade run from Pluto to Earth, and it took 4 BGMS's to reach Earth stations (2 dock./undocks and 4 gates).  All happened starting at 9PM PST, and ended around 10PM PST.  I was fighting in Rag, so when the blackout time was about to start (8:30-45PM PST), I thought, let's find something less dangerous to do instead of risking XP debt !!!!!  :-)

 

[EDIT] It is definitely not a DNS problem, since the Net-7 portal web pages load fine.

Edited by VincentTH
Link to comment
Share on other sites

I have disabled or moved everything that was running around this time of night, I'm running out of options. I'm thinking we might have to try to duplicate it on a weekend so I can trace what's going on at the game server and see if anythings getting dropped.

Link to comment
Share on other sites

I noticed around that time whilst gating to desktop and freezing on loading screens, when I was warping between gates, all of the mobs that I clicked on were at 99% and "bleeding"

Just another observation

Link to comment
Share on other sites

  • 2 weeks later...
Have to resurrect this post. This problem is still happening. Although it seemed to lighten up some, it is still going on. Tonight is worse than ever. I've already had to end task two times exiting Arogoth Station after taking job runs. Again, I'm on east coast and have a guild member on the west coast who is also experiencing serious lag while doing regular combat.

Kyp, as a side note, I saw your post about disabling the ipv6. I did this and it hasn't done anything to alleviate the issue. Something is still interfering with the server between 12m and 1 to 1:30am ET. Also, lots of serious lag moments during other times for the past week.
Link to comment
Share on other sites

Then it must be somewhere in the network of the hoster since i have a script running on a server here in Europe doing traces to the Net-7 servers, checking ping/response times in that time window.

The logs from tonight don't show anything strange.

 

What i can think of tho:

Server is hosted in a segment at the ISP where 40-50 other machines are also located.

 

90% of all people/customers tend to make backups (scheduled jobs) all at midnight (funny but true - that why i have mine spooled to 4am).

 

Anyway, if a dozen machines in that segment start to transfer gigabytes(!) of backupdata to a central backup server the hoster offers, the switches in that network segment start to get hit a lot, so everything starts to lag. This is normally no problem at all, but in a realtime game environment, a few 100ms start to become really recognizable.

 

BUT...then i would see a ping-lag in my logs as well, which i dont - so i think of something else which seems more likely:

Maybe it is not even the ISP network farm but some other hoster jamming the backbone by transfering gigabytes of data nightly. Eg. these new "Windmill power generators" have a software that is producing tons of data per hour and they get transfered nightly. At least i know this here from Europe because a friend of mine works for a hoster and told me once that needed to put a customer (windmill farm) on a 10 GIG fibre line...

 

This would explain why MY(!) route to the server is fine while some other users in the U.S. are maybe using a route touching a backbone that might be jammed for you at midnight.

 

Just a guess but seeing the logs indicates such scenario - esp. due to the fact knowing we are not running backups at that time.

Finally....there isn't really much (nun) we can do about it because it's outside of our "world".

Link to comment
Share on other sites

A pity, the time I get the problems is gmt +9.5 at 4pm till 6pm

 

I cannot gate or use terminals for the 2 hours or suffer the gate to desktop or terminal freeze, so I usually jump on my JE, and go mining in a sector for that time

Link to comment
Share on other sites

Zack,

 

I'm in Europe if you can say Europe as you can see <-- 

 

However, I feel it at same time with US people feels as well...

 

During this hours, not only lag (it is not constant, but like peeks all sudden and gone), but I think master server handoffs greatly increase during gating... That's how I noticed the hours above... What I feel like packet losses rather than ping times... After 7:00am Germany time, it all ends and goes back to normal... But during 6am and 7am basically best th

 

Three nights ago, during this hour, me & a few others (what I got from chat) locked on analyze, it never responded back about analyze or fail (Lost a good Enforcer's Retort)

 

Two nights ago, my character is so lagged that I ended up "out of map"

 

Yes, I was literally out of Glenn map, closest nav was 250k (west of Sol gate I was), I could not warp anywhere, can talk, chat even. A few JD's tried to pull me from that hole. may be they read this and confirm anomaly...

 

 

Best thing during this hours, is to stay in the zone rather than moving anywhere... If you are in a single zone, lag comes and goes momenteraily, but if you travel or gate a lot master handoffs increase...

 

You might just like to check logins/logouts during these hours.

Link to comment
Share on other sites

Yep, my log is exactly at these hours.

I start the log from 05am my time (which is 11pm server time) and have it running for 3 hours (8am my time, 02am server time).

 

I was just pointing out that it is obv(!) not originating from the Net-7 servers or the server farm the machine is hosted at since the log would show lag for me as well then - so it must be related to certain lines/backbones going TO our server (which im not using on my route).

Link to comment
Share on other sites

Yep, my log is exactly at these hours.

I start the log from 05am my time (which is 11pm server time) and have it running for 3 hours (8am my time, 02am server time).

 

I was just pointing out that it is obv(!) not originating from the Net-7 servers or the server farm the machine is hosted at since the log would show lag for me as well then - so it must be related to certain lines/backbones going TO our server (which im not using on my route).

Yes, it looks like connection rather than server itself.

 

Somewhere in between it stucks, most likely after Germany nodes but which one :)

 

  3     8 ms     7 ms     7 ms  212.156.123.69.static.turktelekom.com.tr [212.156.123.69] 
  4     7 ms     7 ms     7 ms  gyrttpe-2-3-gyrttpe-3-1.turktelekom.com.tr [212.156.118.30] 
  5    48 ms    49 ms    49 ms  fr-col-3-gyrttpe-2-3.turktelekom.com.tr [212.156.101.85] 
  6    56 ms    55 ms    56 ms  xe-8-2-2.edge5.Frankfurt1.Level3.net [212.162.4.157] 
  7   186 ms   186 ms   186 ms  vlan60.csw1.Frankfurt1.Level3.net [4.69.154.62] 
  8   189 ms   188 ms   188 ms  ae-61-61.ebr1.Frankfurt1.Level3.net [4.69.140.1] 
  9   186 ms   187 ms   187 ms  ae-47-47.ebr2.Paris1.Level3.net [4.69.143.142] 
 10   188 ms   188 ms   188 ms  ae-43-43.ebr2.Washington1.Level3.net [4.69.137.58] 
 11   195 ms   200 ms   198 ms  ae-72-72.csw2.Washington1.Level3.net [4.69.134.150] 
 12   187 ms   188 ms   190 ms  ae-71-71.ebr1.Washington1.Level3.net [4.69.134.133] 
 13   189 ms   189 ms   188 ms  ae-2-2.ebr3.Atlanta2.Level3.net [4.69.132.85] 
 14   189 ms   190 ms   188 ms  ae-7-7.ebr3.Dallas1.Level3.net [4.69.134.21] 
 15   190 ms   199 ms   200 ms  ae-73-73.csw2.Dallas1.Level3.net [4.69.151.145] 
 16   193 ms   188 ms   188 ms  ae-72-72.ebr2.Dallas1.Level3.net [4.69.151.142] 
 17   196 ms   189 ms   204 ms  ae-5-5.car1.KansasCity1.Level3.net [4.69.135.229] 
 18   236 ms   237 ms   237 ms  11-INTERNET.car1.KansasCity1.Level3.net [4.53.32.10] 
 19   235 ms   235 ms   236 ms  ae-11.bb-c.ms.mkc.us.oneandone.net [74.208.1.82] 
 20   250 ms   250 ms   250 ms  ae-10.bb-c.slr.lxa.us.oneandone.net [74.208.6.106] 
 21   179 ms   179 ms   179 ms  ae-10.gw-distp-a.slr.lxa.oneandone.net [74.208.1.101] 
 22   179 ms   179 ms   179 ms  ae-1.gw-prtr-r4-1a.slr.lxa.oneandone.net [74.208.1.176] 
 23   247 ms   252 ms   247 ms  sunrise [74.208.192.215] 
Link to comment
Share on other sites

One thing we can do is to increase the retry count in both the proxy on the client side and the server side.  It appears now, that the server sends the master handoff only once with no retries (or too short/too few a retry). 

 

(Note that the Blue Galactic Map hang during gating also happens outside of the 9PM-10PM PST window too, but less frequently.)

 

@Zackman: IMHO the ping test is not a good test.  During the 9-10P PST window, if I gate 4 toons at the same time on 2 comps, 3 would go through, one would get the blue screen.  So it is not the ping latency that causes that, more like routers fragment the data packets, and some of them got dropped.  A more accurate ping test would be using the -f -l flag during that window to see how often the fragmentation is, compared to regular time outside that window).

 

My 2 cents

Edited by VincentTH
Link to comment
Share on other sites

  • 2 weeks later...

Kyp, any further find for a resolve on this problem?  Last night was another really, really bad night.  This time around though, when I'd hang on gating, I'd force quit the game and log back in. When I came back in, I only had 1/8th of my reactor and hadn't done anything prior to hang other than hit the enter gate button.

 

Also, prior to the midnight lagfest, while selecting 150 jobs at Prasad terminal, I was dropped completely out of game to desktop.  This happened about 5 or 6 times in the course of about an hour.  Odd thing is,  I had an alt logged in at Jove's Fury and it showed that I immediately logged out on each occasion.  Usually during a hang event, it won't show me logging out until I've actually logged back in after end tasking.  I had been grinding jobs for the past week and never had this issue.  I had just started doing the 150 jobs at Prasad, so I'm not sure if it has something to do with this station specifically or not, but figured I'd mention it.

Link to comment
Share on other sites

Not sure, nothing shows in our logs around these times and no jobs appear to be running. Despite it being a coincidence with it happening around the time i was previously running a plesk backup im not sure what to say. Can you provide an MTR trace to us when youre experiencing the issue?
Link to comment
Share on other sites

The MTR i had running the last night shows no packet drops - so the complete route including the game server was ok for the route from germany.

If there is something happening, it might be some network parts / backbones within the USA affecting certain (but not all) routes to the sunrise server.

Link to comment
Share on other sites

I still get ctd at around 3:30 pm till 6pm (gmt +9.5) gating or in terminals . Very common to get terminal freeze and/or loading screen freeze. 

worse place for me is Earth Station, in job terminal, drops to desktop very regulary followed secondly by loading screen freeze

 

when in a terminal,I can tell when I'm about to be booted when clicking on a explore job it freezes and takes about 1 to 2 minutes to display where the job is going, or ctd instantly

 

9 times out of 10 when in loading screen I know its frozen as the red circle around destination is not present, but now and then the red circle is present but I'm frozen,

weird thing tho when first logging on, after selecting avatar, sometimes the loading screen has no red circle but I load into sector/station after a longer than normal delay,

about 1 minute before I am in sector/docking bay

Link to comment
Share on other sites

I pulled an MTR from Melbourne, AU to sunrise via a service called 'websitepulse' and have forwarded it to the host to analyze their routes. Hopefully they will have good news for us.

Link to comment
Share on other sites

They're asking for someone from the location experiencing trouble to use this tool:

 

http://winmtr.net/

 

and record: 

 

"Our Network team requires a minimum of 1000
packets sent (which should only take a few minutes with WinMTR), as this gives
us a good snapshot of network behavior."

 

Preferably when you are noticing the issues and also when you are not.

Link to comment
Share on other sites

They're asking for someone from the location experiencing trouble to use this tool:
 
http://winmtr.net/
 
and record: 
 
"Our Network team requires a minimum of 1000
packets sent (which should only take a few minutes with WinMTR), as this gives
us a good snapshot of network behavior."
 
Preferably when you are noticing the issues and also when you are not.

Kyp, I couldn't figure out how to attach the file so I'm just pasting it below.  I ran this during normal game time between 9 and 9:30pm EST.  I'll try to do another run around midnight tonight. Not sure what the onlinehome-server.com is, I configured the test to run on sunrise.net-7.org.
 
 
 
|                                      WinMTR statistics                                   |
 
|                                            Host              -   %  | Sent | Recv | Best | Avrg | Wrst | Last |
 
|         u15438806.onlinehome-server.com -    1 | 1382 | 1377 |   62 |  133 | 1157 |  926 |
 
|___________________________________________|______|______|______|______|______|______|
 
   WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider Edited by DigitalHytop
Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...

×
×
  • Create New...