Multiplayer Game Programming: Architecting Networked Games (2016)

Chapter 13. Cloud Hosting Dedicated Servers

The changing cloudscape means even small studios can afford to host their own dedicated servers. No longer must the fate of a game rely on players with fast net connections hosting fairly administered servers. This chapter explores the pros, cons, and methods necessary to get your game’s servers running in the cloud.

To Host or Not To Host

In the early days of online gaming, hosting your own dedicated servers required the Herculean task of acquiring and maintaining large amounts of computer hardware, networking infrastructure, and IT staff. Any hardware ramp-up was a gamble at that. If you overestimated the number of players at launch, you’d end up with racks and racks of machines lying fallow. Worse, if you underestimated, your paying players would be unable to connect due to processing and bandwidth constraints. While you struggled to obtain more last minute equipment, your players would give up, write bad reviews, and warn their friends not to play your game.

Those days of terror are over. Thanks to the abundance of on-demand processing power available from giant cloud host providers like Amazon, Microsoft, and Google, gaming companies are able to spin up and down servers on a whim. Third-party services like Heroku and MongoLabs make deployment even easier by providing server and database management services as needed.

With the huge barrier to entry gone, the proposition of hosting dedicated servers is one that every developer should consider, no matter how small the studio. Despite the lack of upfront server cost, there are still some potential drawbacks to consider:

Complexity. Running a dedicated fleet of servers is more complex than allowing players to host their own. Even though cloud hosts provide the infrastructure and some of the management software, you still need to write custom process and virtual machine management code, as described later in this chapter. Also you have to interface with one or more cloud host providers, which means adapting to changing APIs.

Cost. Even though the cloud decreases upfront and long-term cost significantly, it’s still not free. Increased player interest may cover the increased cost, but that’s not always the case.

Reliance on a third party. Hosting your game on Amazon or Microsoft’s servers means the entire fate of your game rests on Amazon or Microsoft’s shoulders. Although hosting companies offer service-level agreements that guarantee minimum uptime, these do little to console paying players when every server suddenly goes down at once.

Unexpected hardware changes. Hosting providers usually guarantee to provide hardware that meets certain minimum specifications. This does not prevent them from changing hardware without warning, as long as it is above the minimum specification. If they suddenly introduce a bizarre hardware configuration which you have not tested, it may cause issues.

Loss of player ownership. In the early days of multiplayer gaming, administering your own game server was a matter of pride. It was a way for players to be an important part of the game community, and it created alpha players that spread the gospel of whatever game they were hosting. Even today the culture still lives on in the myriad custom Minecraft servers hosted across the land. The intangible benefits of player ownership are lost when the responsibility of running servers moves to the cloud.

Although these downsides can be significant, the benefits often outweigh them:

Reliable, scalable, high-bandwidth servers. Upstream bandwidth comes at a premium, and there’s no guarantee that the right players will be hosting the right servers when your other players want to play. With cloud hosting and a good server management program, you can spin up whatever server is necessary, wherever and whenever you need it.

Cheat prevention. If you run all the servers, you can make sure they’re running unmodified, legitimate versions of the games. This means all players get a uniform experience not subject to the whims of player administrators. This enables not only reliable rankings and leaderboards, but also persistent player progress based on gameplay, as found in Call of Duty, for example.

Reasonable copy protection. Players have a lot of hate for intrusive copy protection and digital rights management (DRM). However, DRM can be a necessity for some types of games, especially those that rely on microtransactions for revenue, like League of Legends. Restricting your game to run on company hosted, dedicated servers provides a de facto, nonintrusive form of DRM. You never have to release server executables to players, which makes it much harder for them to run cracked servers that illegally unlock content. It also allows you to check login credentials for every player, ensuring that they really should be playing your game.

As a multiplayer engineer, the choice of whether to host dedicated servers may be above your pay grade. However, given the value of full stack engineers in the work force, it is important to understand all the implications of the decision so you can weigh in with an informed opinion based on the specifics of the game your team is making.

Tools of the Trade

When working in a new environment, it is most efficient to work with tools tailored for that environment. Backend server development is a rapidly evolving field, with a rapidly evolving set of tools. There are many languages, platforms, and protocols designed to make life easier for the backend developer. At the time of this writing, there is a definite trend for services to use REST APIs, JSON data, and Node.JS. These are flexible and widely accepted tools for server development, and the examples in this chapter make use of them. You can choose different tools for your cloud server hosting development and the basic concepts will remain the same.

REST

REST stands for representational state transfer. A REST interface is one that supports the idea that all requests to a server should be self-contained and not rely on previous or future requests for interpretation. HTTP, the protocol that drives the web, is a perfect example of this, and thus typical REST APIs are built heavily around the use of HTTP requests to store, fetch, and modify server-side data. Requests are sent using the common HTTP methods GET and POST, and also the less common PUT, DELETE, and PATCH. Although various authors have proposed standards on exactly how these HTTP requests need to be structured to qualify as a REST interface, many engineers end up creating interfaces that are REST-flavored to best suit the needs of the users, but do not adhere strictly to any set of REST requirements. Generally, REST interfaces should use the HTTP methods in a fairly consistent manor: GET requests fetch data, POST requests create new pieces of data, PUT requests store data in a specific place, DELETE requests remove data, and PATCH requests edit data directly.

One major advantage of REST interfaces is that they are mostly plain text. Thus, they are human readable, discoverable, and debuggable. In addition, they employ HTTP, which itself uses TCP for transport and thus they are reliable. The self-contained nature of the REST request expands request debuggability, cementing REST as the chosen API style for the backbone of today’s cloud services. More details on REST style interfaces and proposed REST standards can be found in the resources listed in this chapter’s “Additional Readings” section.

JSON

In the late 1990s and early 2000s, XML was heralded as the universal data exchange format that would change the world. It started to change the world, but it had way too many angle brackets, equal signs, and closing element tags to last forever. These days JSON is the new darling for universal data exchange. Standing for JavaScript object notation, JSON is actually a subset of the JavaScript language. An object serialized to JSON is exactly the JavaScript that would be needed to recreate that object. It is text based, maintaining all the human readability of XML, but with fewer formatting and tag closing requirements. This makes it even more pleasant to read and debug. Additionally, because it is valid JavaScript, you can paste it directly into a JavaScript program to debug it.

JavaScript works well as a data format for REST queries. By specifying a Content-Type of application/json in the HTTP header, you can pass data to a POST, PATCH, or PUT request in JSON format, or return data from a GET request. It supports all the basic JavaScript datatypes, such as bools, strings, numbers, arrays, and objects.

Node.JS

Built on Google’s V8 JavaScript engine, Node JS is an open-source engine for building backend services in JavaScript. The idea behind the language choice was that it would facilitate development of AJAX style websites that also used JavaScript on the frontend. By using the same language on both client and server, developers can write functions and easily switch or share them between layers as necessary. The idea caught on and a very rich community has grown up around Node. Part of its success is due to the vast number of open-source packages available for Node, easily installable through the Node package manager (npm). Almost all popular services with REST APIs have node package wrappers, making it trivial to interface with the vast array of cloud service providers.

Node itself provides a single-threaded, event-driven JavaScript environment. An event loop runs on the main thread, much like in a video game, dispatching event handlers for any incoming events. These event handlers can in turn make long running requests to the file system, or to external services like databases or REST servers, that execute as asynchronous jobs on non-JavaScript threads. While the jobs execute, the main thread returns to the processing of incoming events. When an asynchronous job completes, it sends an event to the main thread, so the event loop can call an appropriate callback and execute an appropriate JavaScript handler. In this way, Node provides an environment that prevents the pain of race conditions while still allowing for non-blocking asynchronous behavior. As such it is a prime candidate for building services to handle incoming REST requests.

Node ships with a simple built-in HTTP server, but the task of decoding incoming HTTP requests, headers and parameters, and routing them to the appropriate JavaScript functions is usually handled by one of several open-source Node packages dedicated to the purpose. Express JS is one such very popular package and the one used by the examples in this chapter. More information on Express JS and Node JS can be found in the resources listed in the “Additional Readings” section.

Overview and Terminology

From the player’s perspective, the cloud server spin-up process should be transparent. When a player wants to join a game, the player’s client requests info on a match from the matchmaking service endpoint. The endpoint looks for one, and if it can’t find one, it should somehow trigger a new server to spin up. It then returns the IP address and port of the new server instance to the client. The client connects there automatically and the player joins the game.

Note that it can be tempting to combine the processes of matchmaking and dedicated server deployment into one giant blob of functionality. It saves on some redundant code and data, and can even aid in performance a little. However, it can be more useful to keep them separate for the single fact that you may want to plug one or more third-party matchmaking solutions into your dedicated server system. Just because your studio hosts its own dedicated servers, does not mean it can’t take advantage of third-party matchmaking solutions like Steam, Xbox Live, or PlayStation Network. In fact, depending on the platform for which you’re developing, it may be required. For this reason, it is sensible to keep the server deployment module clearly isolated from your matchmaking module.

When your deployment system finishes spinning up a new server, it should simply register itself with the matchmaking system just as a player hosted game server would. After that, the matchmaking system can take over matching players to server instances and your cloud deployment system can focus on what it does best—spinning up and down game instances as necessary.

Server Game Instance

Before going on, it is worthwhile to disambiguate some of the overloaded meanings of the word “server” when used in various contexts. Sometimes “server” refers to an instance of the class in code that simulates the one true version of the game world and replicates it to clients. Other times, it refers to the process listening for incoming connections, hosting that class instance. Still other times, it refers to the physical piece of hardware running that process, as in “check out all the servers I can fit on this rack.”

To avoid confusion, this chapter uses the term server game instance or just game instance to represent the entity that simulates the game world and replicates information to clients. The concept is an abstraction that represents a single reality shared by a group of players playing together. If your game supports 16-player battles, then a server game instance is a running 16-player battle. In League of Legends it is typically a 5 versus 5 game in the “Summoner’s Rift” level. In matchmaking terms, it is a single match.

Game Server Process

A game instance does not exist in a void. It lives inside a game server process, which updates it, manages its clients, interacts with the operating system, and does everything else a process typically does. It is the embodiment of your game, as far as the operating system is concerned. In all previous chapters, the concepts of game server process and game instance were not separated because there was a one-to-one mapping between them. Each game server process was responsible for maintaining only one game instance. However, in the world of dedicated server hosting, that can change.

In properly abstracted code, a single process can manage multiple game instances. As long as the process updates each instance, binds a unique port for each instance, and does not share mutable data between the instances, multiple game worlds can coexist peacefully in the same process.

Multiple instances per process can be an efficient way to host multiple games, because it allows sharing of large immutable resources like collision geometry, navigation meshes, and animation data. When multiple game instances run in their own processes, they each need copies of this data, which can cause unnecessary memory pressure. Games employing multiple instances per process also benefit from finer control of scheduling: By iterating through each instance each update, they can assure a roughly regular update pattern across instances. With multiple processes on the same host, this is not necessarily the case, as the operating system scheduler decides which process is updated when. This is not always a problem, but finer-grained control can be useful at times.

The significant advantages of the multi-instance approach may seem compelling, but the disadvantages of the tactic are just as significant. If a single instance crashes it can bring down the entire process, with all of its contained game instances. This can be particularly nasty if an individual instance corrupts a shared, supposedly immutable resource. Alternatively, when each game instance runs in a dedicated process, a corrupted or crashing game instance can only bring down itself. In addition, single game instance processes are easier to maintain and test. Engineers developing server code commonly only need a single game instance at a time to test and debug code. If the process supports multiple instances and engineers aren’t running them, it leaves a large code path without regular development coverage. A good QA team with a solid test plan can partially accommodate for this, but there is no substitution for engineers having full coverage of production code paths during development. For these reasons, it is most common for game server processes to contain a single game instance.

Game Server Machine

Just as a game instance needs to live in a game server process, a game server process needs to live on a game server machine, and just as a single process can host multiple instances, a single machine can host multiple processes. The choice of how many processes to run per machine should depend on the performance requirements of your specific game. For maximum performance, you can run a single process per machine. This ensures the machine’s full resources, including CPU, GPU, and RAM, are dedicated to your game process. However, it can be quite wasteful. Each machine needs an operating system and a typical OS is an immense consumer of resources.

Running a unique OS just for a single game process, especially one that contains only a single game instance, can be too expensive an endeavor. Luckily operating systems are designed to support multiple processes with features like protected memory to keep them from interfering with each other’s immutable assets. On a modern operating system, it is extremely unlikely that a crashing process can bring down another process on the same game server machine. Therefore, to be cost-efficient, it is typical to run multiple game server processes per server machine—often as many as the performance requirements will allow. Tweaking and tuning server performance and RAM use can pay off many times over if it allows more game processes to be hosted on the same server machine.

Hardware

In the cloud, a game server machine does not necessarily equate to a physical piece of hardware. Instead, machine images represent virtual machines (VMs) which are spun up and down at will, sometimes residing alone on a physical machine, or other times sharing resources with multiple other virtual machines on a physical machine of 16 cores or more. Depending on your cloud hosting provider, and your budget, you may not get to choose how your virtual machines are hosted. At lower price points, they must often share hardware, and are put to sleep when not used for a set amount of time. This can result in erratic performance. At higher price points, you can often specify the exact physical hardware configurations you desire.