Bolstering the Capacity of NEStalgia Servers

Wouldn’t it be great if you could play NEStalgia on much larger servers? If population surges didn’t mean lots of new servers and inevitable server merges?

This week’s update probably isn’t a super fun read, but it heralds a major change that should significantly improve the NEStalgia experience. Hopefully when the Key of the Exiles expansion goes live, instead of worrying about which official server you and your friends are playing on, you’ll instead be able to focus on saving up gold to purchase some sweet ship upgrades.

Before I get started, I’ll just quickly note that NEStalgia is currently available at a huge discount as part of the Steam Summer Sale. Check it out!

The Indie MMO Problem
I apparently missed the memo about not developing an MMO style game unless you have access to MMO style resources.

The core of NEStalgia’s server system is the common “island server” design where each named server is basically a separate instance of the game running off of its own savefile database. That’s why your characters tied to the Malice server can’t be used on Midgard unless someone manually transfers your savefile. It’s also why each server has separate guild registries and market place listings. This would all be fine and good if the game didn’t have so many players; when I was originally developing NEStalgia I didn’t expect that the game would find such a large audience. It’s a good problem to have, but a problem nonetheless.

Right now a single NEStalgia server can accommodate 50 or so players online at once before lag starts to become an issue; usually each server caps at about 70 players online. This active player limit isn’t a glaring deficit in and of itself, and is actually pretty high compared to most indie games. The difference is that most other multiplayer indie games aren’t MORPGs, and thus they aren’t as reliant on maintaining balanced population equilibrium between servers, nor do they have to be 24/7 persistent worlds. For example, you usually play games like Terraria on small private servers with friends. In games like DayZ you play on larger public servers, but because each play session of the game is generally self contained, switching to a different server and starting over isn’t all that painful.

How We’ve Been Coping

Whenever NEStalgia has received a large influx of new or returning players, we’ve opened up new official servers to accommodate the population load, and then merged those servers into each other as the populations recedes. It’s hard to time these merges so that every single server maintains a satisfactory active population level, and it can also lead to players being isolated from their friends on a more popular server.

All of those factors are why server management has easily been the biggest source of frustration both for players and the dev team. As a player it’s not fun to see your server go into decline after a population surge, and as a developer it’s a huge time suck trying to stay on top of server expansions, merges and transfers. On several occasions we’ve had to open 16+ official servers at once just to accommodate demand… but even the smaller population surges tend to tangle us up in server and savefile management nightmares that bring actual NEStalgia development to a grinding halt.

From here I could  probably write a few more paragraphs about the challenges involved in finding a solution that is a good fit for NEStalgia. However, I’m fairly certain that most of you don’t care about the drawbacks of client side saving or headaches with online shared savefile databases, so let’s skip right to what I’ve actually come up with…

Server Cluster Diagram

The New Solution

The v1.70.0 update should keep NEStalgia’s core “island server” design intact while still allowing us to accommodate many more players on what is technically a single server. Instead of having every single instance of a server store its own savefile database, official servers will now operate in clusters of server instances that share savefiles. Each of the servers in these server clusters will operate under the same server name, but can have additional instances opened to accommodate demand. Ex: There might be a Midgard Cluster, with open server instances Midgard (1), Midgard (2), Midgard (3), and so on.

Like the current named servers, each server instance is its own separate world. You won’t see or directly interact with players logged in to other server instances, though you can come and go as you please between instances. For example, if the first server instance the Midgard Cluster – Midgard (1) – fills up, you can simply log off and join the second instance Midgard (2) to pick up right where you left off. The same things goes if your friends are playing on Midgard (3) and you’re currently logged into Midgard (1) – you just need to join Midgard (3) in order to play with them.

The different server instances in each cluster share the same guild registry and marketplace database. That means that if  you make a change to your guild cape or recruit a new member on Midgard (1), the change will automatically be registered on Midgard (2) and Midgard (3). The marketplace communicates between server instances as well – every instance of the server sees the same market listings, and purchasing an item on one instance will instantly take it off the market on all other server instances.

Don’t be concerned if all this talk of “clusters” and “instances” has your head spinning. All that you need to know is that you’re going to be able to play the game the same way as before, minus concerns about over/underpopulated servers. In practice all of this should be fairly intuitive even to first time players.

Making the Transition

If all goes well, I hope to get v1.70.0 and the server clusters up and running in the next week or so. The biggest concern is dealing with players who still have savefiles on multiple servers, mostly in regards to merging files and/or dealing with excess characters that can’t be merged. There’s no perfect solution to that issue and it’s probably going to eat up a ton of time, so bear with me as I get it sorted out.

Between all four of the current named servers we’re talking about 30,000 savefiles – and that’s after previously setting aside 50,000+ “legacy” savefiles back in December. Because of the sheer number of savefiles we’ll probably end up with two different server clusters, but I can’t say for sure yet. If that’s the case then we’ll only need one server instance per cluster at the moment, so you probably won’t need to even think about instancing for awhile.

I’m sure that some of you have questions, so feel free to ask away. Thanks for reading!







12 responses to “Bolstering the Capacity of NEStalgia Servers”

  1. jozeppi Avatar

    Question not relevant to server capacity, but perhaps a future post topic: Crafting is to be greatly expanded in KotE, in what way will it be expanded, and what sorts of things will be craftable?

    1. Silk Avatar

      Thanks for the topic idea, that will actually make for a good blog post because there is a lot to talk about on the subject. I’ll make note of it 🙂

  2. geekydad Avatar

    Will we be able to chat (gsay especially) between instances?

    1. Silk Avatar

      Good question. I’d like to enable cross-instance guild chat, though I’m holding off until I can analyze what kind of an impact it will have on server speed. I’ll pull the trigger on it as soon as I’m confident that it won’t slow down important things like marketplace transactions.

      1. omeganecros Avatar

        at the very least we should be able to see online guild members across the cluster and which instance they are on so we can attempt to be organized as a guild.

  3. cjsutt1 Avatar

    Thanks Silk, I’m sure we all really appreciate these weekly updates and the things to come!

  4. Prince Mononoke Avatar
    Prince Mononoke

    I just have one question–with the change in server style, will that mean other, related things will change? In particular I speak of bank space storage. If a game has an unlimited inventory or an item storage, you can be sure I’m going full pack rat mode–especially if it’s an RPG of any kind. I’ve got a little under 2k hours in this game, but I’ve stopped playing for a while.

    Now, this is not because of the lack of new content, but because quite frankly, if I can imagine myself using it, I put it in my bank, and as a result, it’s just become to big of a hassle to manage my inventory when I have about 6 characters as dedicated mules and an almost full bank to ever transfer anything over from any of my characters. I don’t like dealing with that, but I also enjoy just having each and every one of the items I’m holding–thus, stagnation. If the server change allows, can we please, please, PLEASE get more bank storage? (more pet storage would be great too)

  5. jeff Avatar

    On the technical side, can the system easily support future modifications on the instance system?
    As now, it would seem like it works on a sort of “first come, first serve” basis, filling up a server once its full. With the current population this is likely not to cause issues. Still, there is the possibility that a good amount of people will come back playing once the expansion comes out.
    When that happens, and multiple instances are opened, people on a guild may find themselves fragmented upon multiple full instances. Sure, they could all log out and relog to another instance (i suppose you can directly choose what instance to log in once multiple instances are opened, right?) but that could end up impractical.
    Thus, in the future, one such improvement that could be done is the use of some parameters to decide whether to put the incoming player in a new instance or in an already existing one, as long as the system permits to do so (or at least without heavy overhaulings on the whole cluster/instance code).
    For example, the system could try to put incoming players along friends and guilds by reserving some space in instances and by using some weighted formula to decide whether to put a player in an existing instance or in another, new instance.
    While it isn’t something to be concerned right now – as you said, we’ll probably have a single instance per cluster – the technical possibility of doing something like that could end up useful in the future.

  6. deadfool Avatar

    I really hope the Capacity Bolstering will go out well.

    Regarding my save files, you can just delete the ones that are not on the Midgard server.

    I am aware that the following players have large savefiles on many of the servers, and you should probably contact with them pirvately regarding on which savefile to keep:
    Ker_Blam, HeWhoHasNoLife, Blackwrensniper, Nasca and Lord Rith.

  7. blaxun Avatar

    Bought 6 copies. Hope this will support you 🙂
    And, I would really LOVE to see some option ingame to fast-forward battles or an option to increase animation speed and text speed. Battles are really taking a LOT longer then they should. As a Warlock, using LEECH and then POISON on enemies will play a lot of animations and text…its really makin it hard to enjoy battles 🙁

    1. Mahn Avatar

      The text speed and battle speed is there to preventing cheating by players using bots and yes it has been tried before.

Leave a Reply