Server down?

[electric mayhem]

*pours some wake up juice into the server*

[Endelyon]

Looks like the process hung up. It should be back on.

[amber91]

Hidden: show

I'm not seeing any indication of DDOS attacks in the past 24 hours. There may have been an issue that snuck in during our last server update (probably script related).

We'll be doing another server update soon-ish, and hopefully that will help restore stability somewhat! I know the devs are all looking at their own scripts from the last update to make sure everything's looking good.

Another option, if that doesn't work, may be cutting more areas/content. We're really playing at the edge of what this game engine can handle, with our amount of area maps and custom content.

For what it's worth - I have been playing here since 2009 (with a couple shorter and longer absences) and it never before was as smooth as it was after the large area removal.

The current rather laggy situation started for me at the exact moment when the last update (including the re-appearance of several areas) was published.
Since then I more often than not have experienced lag in every fight (unrelated to the number of players online/DM events happening or not), in many npc talks, transitions etc.

I wanted to add to what she said, in light of Maecius' post that they identified the Storage system as a potential culprit.

Consider the fact that sometimes there is still bouts of severe server-wide network lag during off-hours like very early EST, when there's under 20-30 players on.

Such phases of lag may be induced or triggered by something other than player-scale-related-usage such as the Storage system; because if that was the culprit, one would think there would be less lag with fewer players (assuming less use of storage, etc.); the issue seems not so much related to player scale or player-usage of a system, and more to do with something else...

If history serves, the more likely is something along the lines of what VillageGreenWitch was alluding to, and more likely a script or erratic value gone rogue.

Also consider how the server has performed with the Storage system for many months before without these intermittent but severe lag/crash issues we are seeing now. The pattern seems to suggest some other culprit, perhaps in relation to, but not seemingly solely due to the Storage system.

Of course, I throw speculation around easily, not knowing how the server actually works

Maybe sprinkle some fairy dust over it for the fixin'?

[Endelyon]

The lag isn't caused by "lots of people accessing the storage." It's caused by certain individual people using the storage. If one of these certain individual people are using the storage, it's going to lag, period, the same amount, whether there's 30 people online or 100.

Incidentally, no areas have been added back in since they were removed, so I'm not entirely sure what VGW means about the last update in general. We aren't getting no-memory crashes anymore.

[chad878262]

has anything been identified about why these certain players accessing storage lags the server while others don't? Anything to do with how many items a character has in storage, or how many items a player has in storage across all characters?

[Endelyon]

has anything been identified about why these certain players accessing storage lags the server while others don't? Anything to do with how many items a character has in storage, or how many items a player has in storage across all characters?

Nope. Using logs I'm able to identify which players it is but not why as of yet. Valefort and I have done some work trying to reproduce the issue and Aspect of Sorrow is looking into a lead on some dropped SQL connection but so far we haven't been able to identify the cause.

[amber91]

The lag isn't caused by "lots of people accessing the storage." It's caused by certain individual people using the storage. If one of these certain individual people are using the storage, it's going to lag, period, the same amount, whether there's 30 people online or 100.

Ah, that's good root-cause analysis!

[Let me know if my voodoo-blind blackbox suggestions are unhelpful or not welcome, but I'm hoping the server gets more stable ]

You may have already considered this or looked into it, but I thought I saw a post a month or so ago about some old player who used to help test the Persistent Storage system, and he had a special test widget; however, he said using that resulted in his storage getting bugged, etc. (OP: viewtopic.php?f=10&t=59295&p=746839&hil ... ge#p746839)

Can't say that's when the issue started...but I wonder:
1) Are you seeing the issue more prevalent with older characters or with new characters?
2) Could older iterations of the Persistent Storage system having any relation to older characters have anything to do with it?
3) Could it have something to do with these old "tester" widgets floating around causing Storage to bug out?
4) This thread too may be related?: viewtopic.php?f=25&t=55958&p=704431&hil ... ge#p704431
5) How could the person(s) causing the lag/issues know they are triggering the issue? any troubleshooting indicators we should look out for? (a PSA, like the one about cleaning up loot bags, may go a long way to mitigate the recurrence; maybe like "don't click Storage icons too fast" or anything like that if that helps any?)
6) Why not straight up tell the "identified individuals", "hey for some reason you are causing everyone to lag and the server to crash and burn, what are you doing?" or "Could you try doing [this fix] instead?", then add "Comply or permastrike/ban" lol. j/k. Up to you :p

Just my two cents worth of blind-voodoo speculation

[Endelyon]

To answer your questions:

1. I haven't personally noticed a trend either way.

2. There's only one iteration of it.

3. No one I'm aware of should have a KEMO storage widget (these were only ever for internal testing), though if they do and deign to use it, it simply accesses the same storage system that the NPCs do via tag-based scripting instead of a conversation. This won't cause any inherent problems, and the issue is communication between nwn2server and the SQL server as opposed to how the UI elements are being accessed.

4. No. While this is a known bug, it has always happened with these UI elements since their introduction and more likely is simply a minor bug within nwn2server itself and how it handles opening these UI elements on characters. What we're seeing with the persistent storage is a problem with nwnx's communication with the MySQL server (slow reads or outright dropped connections.)

5. If you really want, occasionally type something in the chat like (( test )) while you're using the storage. If there is a long delay before the message shows up while you are depositing that suddenly goes away when you stop depositing, then you're probably causing lag.

6. Simply put because they aren't doing anything wrong. It's a problem with the system internally. We've confirmed now that it's not something they're intentionally causing, and Valefort and I identified this morning that even normal, conventional, every-day use of the system with no abuse or spam-clicking can cause lag spikes with one of Valefort's own characters.

We're working on a couple of leads. We'll get it fixed one way or another (or pull it out once we feel like we've exhausted every possibility).

[Face]

Its kaput

[Wandering_Woodsman]

After reading the last few posts, I will admit I was in the process of using the storage when it went kerblooey...

[Aspect of Sorrow]

Maintenance downtime : 4 minutes from this post.
Duration : 1/2 hour.
Services affected : NWN2, Player Tracker

Edit : Services restored.

[Endelyon]

Now it's a DDOS attack.

[Calodan]

Now it's a DDOS attack.

Guess he thought we missed him! Or her. Could be a girl too. They are smarter anyway.....

[Endelyon]

We'll re-enable the protection when Aspect of Sorrow wakes up. Everyone will have to use the link for a while to whitelist themselves until he gets bored again.

[chad878262]

just went to log in and crashy-crashy... 17.23 GMT, 30 June.