Server Crashes

[Akroma666]

Hey Staff, was just curious why the server keeps crashing. It seems VERY frequent and is SUPER annoying because of the rollbacks. Care to enlighten us why? Perhaps we can lower the reset time so at least we have a warning its coming?

Crash:
17 September 201618:22 GMT / 6:22 pm GMT
11:22 PDT / 11:22 am PDT
( GMT -7 )

[AlwaysSummer Day]

[Maecius]

Hey Staff, was just curious why the server keeps crashing. It seems VERY frequent and is SUPER annoying because of the rollbacks. Care to enlighten us why? Perhaps we can lower the reset time so at least we have a warning its coming?

Crash:
17 September 201618:22 GMT / 6:22 pm GMT
11:22 PDT / 11:22 am PDT
( GMT -7 )

Happy to do so!

The answer is we don't know why!

We think it's probably related to module bloat. The module's simply gotten too big, with too many custom (and probably at points conflicting) scripts and features, for the game's engine and nwnx4 to handle it all.

We're trying to crashproof content (mostly scripts) with every update to try and lower or eliminate the crash rate, but the real answer might be to start taking content back out. Which we're trying to avoid unless absolutely necessary for obvious reasons (potentially breaks a lot of characters, nobody likes losing content, etc.).

It's not related to the actual server machine or host. So we can't just throw money at the problem and watch it go away unfortunately.

[Akroma666]

Can we turn the reset timer down to maybe 2-3 hours then? So we dont lose content?

[Theodore01]

Can we turn the reset timer down to maybe 2-3 hours then? So we dont lose content?

And we gain loot, loot, loot


Is it possible to turn off the ranged cover system
or the custom spells/combat feats which produce an extra avatar icon
and see if that helps (performancewise) ?

[Steve]

I'm just going to put this out there: I experience the Server crashing MOSTLY when DMs are online.

So just to be sure, that DMs are not possessing NPCs, especially for/in combat. And, many VFX can cause the Server to tank.

An additional experience is seeing some Player log on, then the Server hangs for up to 1 minute, mostly continuing, but sometimes crashing. How likely is that some Players are using Overides that are NOT approved?

Otherwise, let's just dial the Content down to Fighter, Mage, Cleric, Rogue. That should speed things up!

[Akroma666]

Can we turn the reset timer down to maybe 2-3 hours then? So we dont lose content?

And we gain loot, loot, loot

Well.. if the server is going to crash every 3-4 hours.. what difference does it make?

[Eclypticon]

To the best of my knowledge, overrides should not effect the server through normal client server login. That issue was patched a long time ago. All the common tricks that were problematic seem to be covered now. The override rule is from the old days before the patch and security touch ups. There were some minor issues after the initial patch and I think they were addressed.

[Aspect of Sorrow]

We'll be running an NWNX profiler in time to help investigating this.

[Blame The Rogue]

to split the server in two, do we need a new physical server?

or can we just split bg into two mods, and run each, in a separate instance, on the same physical server?

i'd much rather start the multi-server/instance approach, than remove content

that being said, while i find the ranged cover combat system interesting, such things are not necessary, and if we do end up having to trim, we need to keep only necessary scripts

rot runs the multi-server approach, server broken down into smaller mods. but i am unclear how many physical servers they have

[Aspect of Sorrow]

to split the server in two, do we need a new physical server?

No.

or can we just split bg into two mods, and run each, in a separate instance, on the same physical server?

Doable with the same module and a bit of AT work but that doesn't solve the underlying issue given players don't evenly distribute their locales. I can toss in logic and a UI toggler that allows players to move across "sessions" like Guild Wars did, but I'd rather focus on the actual problem than putting a bandaid on a symptom of the problem.

i'd much rather start the multi-server/instance approach, than remove content

that being said, while i find the ranged cover combat system interesting, such things are not necessary, and if we do end up having to trim, we need to keep only necessary scripts

A couple of things, Skywing's nwscript accelerator helps shorten execution time, and the NWNX profiler can help determine which systems are contributors to the issue, but the module needs some more fine tuning before we can take advantage of either which is underway (Thanks to Rasael Hill giant throws no longer cause the object invalid issue that crashes the MSIL plugin).

rot runs the multi-server approach, server broken down into smaller mods. but i am unclear how many physical servers they have

I think a lot boils down to the underlying game engine, and there are some things that we can do to completely reduce the impact when refactoring objects throughout, which I am undertaking now by introducing an nwnx framework for us to replace the more expensive nwvm logic with and testing on a NWN1 PW I run. I'd rather see a single optimized module running than maintain several broken ones.

As Endelyon has pointed out, we can run tests all we like with a separate server, but we can't expect things to operate smoothly without the playerbase's involvement. To that degree, perhaps we can schedule some time during the week where the server restarts to enable the Profiler logging, and some other debugging tests, then upon the next crash reverts back to the standard mode we have and ships up to development what the findings were. Perhaps on a Wednesday, instead of our busier weekend timeframes.

[Steve]

I just want to note, and say, how great it is that people who have some excellent technical skills are giving their time and the expertise to BGTSCC.

Really, thank you!

[Steve]

Also...I suspect it is Valefort, logging in and out, that is crashing the Server!!!

[Valefort]

Let me sort my mules ! And i lost a bunch of things to that thing, the server hates me !

[Steve]

Let me sort my mules ! And i lost a bunch of things to that thing, the server hates me !

Screenshots, buddy. Take screenshots!!