Server based game augmentations. The transition to cloud. Really possible?

[Scott_Arm]

There's a first-person mmo using cloudgine that's supposed to offer a 400 player Battle Royale mode. Their server setup looks like it uses a distributed work system, rather than the typical instancing setup you'd get with mmos. They showed multiple clients connecting to a network where the work can be distributed anywhere throughout the network.

I'll see if I can't find the name.

Edit:
My mistake, it's SpatialOS they're using, not gloudgine.

https://improbable.io/games

 

[iroboto]

So is the water indeed calculated "on the Cloud"?
I am pretty sure aging can be implemented numerous ways. Even without the cloud, locally.

It must be server side; to be fair to the players. The height of waves for all players must be in sync in all directions. Where you hit a boat with cannon balls do matter, as if you hit below the water line they take on water, if they are above the waterline, they don't take on water unless they plunge their boat in. The game is vast with no loading or streaming involved with high view distances, i'm fairly positive we're seeing a lot of server side work here in general.

As for aging, I suppose it can be done locally. There are definite ways to just regenerate your character by taking the time stamps between logins.

Is it any more than was already happening? This is using the game server aspect to do local computations, which is what online game servers do anyway. It's no different to having an online shooter with destructible physics and having the server pass the results on, for example. So is something finally happening, or has it been happening in online games for that past 5 years to various degrees?

The latter it would appear. We're typically looking at scale. Battlefield series do incorporate these technologies at a basic level, but the zones are small in comparison, and the wave simulation is not nearly as complex and the server is calculating waves for the world (likely intelligently to save computation).

The waves can be calculated locally see (anything look familiar the entire art style i mean, they are definitely not throwing away all that work lol)

I think KSR is local, though I could be wrong. Most MS 1P games, in fact all that I can remember, have been dedicated servers.

 

[Alucardx23]

Is it any more than was already happening? This is using the game server aspect to do local computations, which is what online game servers do anyway. It's no different to having an online shooter with destructible physics and having the server pass the results on, for example. So is something finally happening, or has it been happening in online games for that past 5 years to various degrees?

It's probably time to forget the XBox Cloud story and just look at online gaming and server augmentations independently of that.

The one game I know doing obvious server-side game stuff beyond the basics of gameplay are Battlefront. Found this vid which seems to show a good recap of the tech in these games, though haven't watched it properly.

Appreciating that server-side game computation isn't unique, we can look at what is unique, so in the case of Crackdown, what methods are they using to pack the physics data, and for SoT, what data is passed for the waves and how is that resolved? For that one, you'd really want to see two player's streams at the same location to compare waves. That's also important to understand what aspects of the rendering at local and what are cloud based - I presume they're just getting height-field/water-state updates and the tessellation and shading is local.

I agree with the fact that we have seen examples of offloading computation to the cloud. As it usually happens things gradually change. On this case I like that it is something very integral to the game. I believe that with time this will also happen with single player games. I am anxious to see what an unleashed developer can do if they completely rely on the power of the cloud.

 

[Shifty Geezer]

I believe that with time this will also happen with single player games.

Big economic discussion about that. Why augment solo player games with the cloud, at added cost? What happens to the game when it's no longer cost effective to support the cloud enhancements and they get turned off? Wouldn't it be more economical to use such cloud-based resources for remote gaming if you're already shifting large amounts of workload to the cloud?

The idea of augmenting single consoles that way was a PR pipe-dream from MS, not founded particularly realistically in economics. Similar to the concept of Cell in everything - another PR pipe-dream. We've not seen that in any guise, and even MS have shifted to just providing server-based features to multiplayer games running on a server (that can be turned off one day without leaving players with broken single-player campaigns). Until we have a financial model change, such as cheap thin clients and an always on requirement and games as rentals more than purchases, maybe, large-scale server-side gaming will be online multiplayer or game streaming.

 

[iroboto]

Big economic discussion about that. Why augment solo player games with the cloud, at added cost? What happens to the game when it's no longer cost effective to support the cloud enhancements and they get turned off? Wouldn't it be more economical to use such cloud-based resources for remote gaming if you're already shifting large amounts of workload to the cloud?

The idea of augmenting single consoles that way was a PR pipe-dream from MS, not founded particularly realistically in economics. Similar to the concept of Cell in everything - another PR pipe-dream. We've not seen that in any guise, and even MS have shifted to just providing server-based features to multiplayer games running on a server (that can be turned off one day without leaving players with broken single-player campaigns). Until we have a financial model change, such as cheap thin clients and an always on requirement and games as rentals more than purchases, maybe, large-scale server-side gaming will be online multiplayer or game streaming.

you mean like portions of rendering done server side and shooting it down to the console? Cause that's not gonna happen anytime soon I don't think CPU support is possible though, AI etc don't need to be local necessarily. But lack of connectivity would make this an issue for offline consoles.

 

[Alucardx23]

Big economic discussion about that. Why augment solo player games with the cloud, at added cost? What happens to the game when it's no longer cost effective to support the cloud enhancements and they get turned off? Wouldn't it be more economical to use such cloud-based resources for remote gaming if you're already shifting large amounts of workload to the cloud?

The idea of augmenting single consoles that way was a PR pipe-dream from MS, not founded particularly realistically in economics. Similar to the concept of Cell in everything - another PR pipe-dream. We've not seen that in any guise, and even MS have shifted to just providing server-based features to multiplayer games running on a server (that can be turned off one day without leaving players with broken single-player campaigns). Until we have a financial model change, such as cheap thin clients and an always on requirement and games as rentals more than purchases, maybe, large-scale server-side gaming will be online multiplayer or game streaming.

I can see this working if the single player game can only be played if you also pay the online service (PSN Plus, Xbox Live). "Why augment solo player games with the cloud, at added cost?" For the same reason you augment the water in Sea of Thieves or simulate the environment destruction in Battlefield; to make a more compelling experience that differentiates itself from the competition. "What happens to the game when it's no longer cost effective to support the cloud enhancements and they get turned off?" I think I can answer this one in a couple of ways. The way that I understand things work from the perspective of server profitability, is that what you want is to keep the server available processing power occupied at a 100% all the time. Servers like Azure or AWS are not limited to a single task type, so the way that I envision it, is that if only one person in the world is playing a single player game that requires cloud processing, what is needed is the available processing % in the cloud to run the simulation. The key is to keep the simulation cost within the profit range for what the customer is paying monthly (Xbox Live, PSN Plus). If no one is running the simulation then the goal of the server is to keep the server occupied with something else. In the worst case scenario support for the game will just end like any other online RPG. My goal here is to imagine what experience I might have that is impossible to get just with local processing, what I care about is to have my mind blown away, if the servers get shut down a couple of years later after I have that incredible experience, then I'm OK with that. Something better will provably be available by that time.

PS: English is not my main language, so if something is not clear on my side, pls let me know.

 

[Tkumpathenurple]

Maybe a compromise would be some sort of server based enhancements?

Similar to what iroboto said in reference to AI - things like larger crowds, which behave in a more diverse, interesting manner when you're online, but still function well enough offline. But that brings up the difficulty of defining an acceptable difference and incorporating that into game design.

Crackdown 3 can't base single player gameplay mechanics around full scale environmental destruction, so they can just design the single player game around a more limited scale of environmental destruction. Simple enough. But would other differences of scale and complexity be prohibitive?

Would it be worthwhile, in terms of balancing game design, to implement smart online AI and dumb offline AI, or would just one be preferable, feasible, or economical?

 

[Silent_Buddha]

Big economic discussion about that. Why augment solo player games with the cloud, at added cost? What happens to the game when it's no longer cost effective to support the cloud enhancements and they get turned off? Wouldn't it be more economical to use such cloud-based resources for remote gaming if you're already shifting large amounts of workload to the cloud?

The idea of augmenting single consoles that way was a PR pipe-dream from MS, not founded particularly realistically in economics. Similar to the concept of Cell in everything - another PR pipe-dream. We've not seen that in any guise, and even MS have shifted to just providing server-based features to multiplayer games running on a server (that can be turned off one day without leaving players with broken single-player campaigns). Until we have a financial model change, such as cheap thin clients and an always on requirement and games as rentals more than purchases, maybe, large-scale server-side gaming will be online multiplayer or game streaming.

In a connected future that all the major publishers envision and are working towards (player engagement over time = more money if you can get them to invest in the game) games are all connected all the time. Not quite MMO...maybe SMO (small multiplayer online)? Things like Monster Hunter World, Destiny, Diablo 3 (PC at least), Sea of Thieves, etc. then why not take advantage of spare cloud capacity?

Cloud based servers are arguably more easily scaled for multiplayer game launches assuming the cloud provider has a large and robust cloud implementation. That beats the hell out of trying to guess exactly how many dedicated servers you'll need to handle the influx of new players at launch and then what to do with all that spare server capacity after the first month (many players will drop a game within the first month, of course there are always exceptions).

So, if going with a cloud based server hosting system for multiplayer, and you are contracting cloud services for cheaper prices, that'll usually tend to be long term contracts with some sort of guarantee for X amount of cloud server capacity and Y amount of excess capacity to handle unexpected events.

Now, assuming all of the above comes to pass (which it may not), then why not use as much as you can within the contract a publisher has negotiated? Sure something like Sea of Thieves is going to generate a relatively large amount of cloud server activity at launch but after that things will normalize. Meanwhile it gets free advertising as this amazing looking pirate game.

Virtually ever stream I've watched from a popular streamer has the streamer constantly going on about how absolutely amazing the game looks, and most of that is due to the water simulation. Advertising that is basically paid for by the additional cloud server load. Of course, they still have to follow through with a good game.

In other words, when there starts to be graphics parity between a lot of AAA games, what can you do to "WOW" the players and make your game stand out? Sure eventually, if everyone is doing it then you'll have relative parity again, but for now, you can take advantage of it to make your game stand out visually. And in the case of Sea of Thieves, have it also affect gameplay to some degree. Just the visual effect on how a ship behaves on the waves becomes an integral part of gameplay if everyone is seeing the ships react the same way on the same waves (visibility, kill and hit confirmation, maneuvering, etc.). I've been surprised at how many times streamers have been caught by surprise when fighting multiple ships where they think they've sunk a ship (it's in the bottom of the wave swells, so a quick glance makes you think they are sinking) start to focus on another ship and then get attacked and potentially sunk by the ship they thought they'd taken out.

Regards,
SB

 

[iroboto]

totally forgot the economics part of the discussion. As per Shifty's comment, the added costs is a burden to studios, I agree with that.
There are two ways to mitigate those costs if studios desire.

1) virtual cloud: it only needs to spin up as many instances as required to support the player base, thus reducing the overall server costs. This is a bit different from just dedicated server models, and usually this is where I make a distinction between what cloud and dedicated server models are, namely the ability to spin up and down resources on demand.

2) Subscription based model: gamers are paying for 'Game Pass' already where money from the subscribers can be used to keep these online services alive ie. traditional MMO model.

 

[Alucardx23]

totally forgot the economics part of the discussion. As per Shifty's comment, the added costs is a burden to studios, I agree with that.
There are two ways to mitigate those costs if studios desire.

1) virtual cloud: it only needs to spin up as many instances as required to support the player base, thus reducing the overall server costs. This is a bit different from just dedicated server models, and usually this is where I make a distinction between what cloud and dedicated server models are, namely the ability to spin up and down resources on demand.

2) Subscription based model: gamers are paying for 'Game Pass' already where money from the subscribers can be used to keep these online services alive ie. traditional MMO model.

I need help to understand why the added cost is not a factor for multiplayer games like Sea of Thieves and the water calculation it has, but it is for a single player game with the same water simulation.

 

[iroboto]

I need help to understand why the added cost is not a factor for multiplayer games like Sea of Thieves and the water calculation it has, but it is for a single player game with the same water simulation.

Same Cost.
The only difference is that with a MP game dedicated servers are something we factor into the running costs of the service. As long as their is a population servers will always be up to Support it.

SP games generally aren’t “service” based, so the cost of cloud compute being tied to the game (say years later) is an issue. The game won’t be playable unless the servers are on and you only stand to lose more money as people jump onto the title late into the life cycle (reduced pricing) but server costs still present meaning less margin or possibly no margin.

 

[Shifty Geezer]

In a connected future that all the major publishers envision and are working towards (player engagement over time = more money if you can get them to invest in the game) games are all connected all the time. Not quite MMO...maybe SMO (small multiplayer online)? Things like Monster Hunter World, Destiny, Diablo 3 (PC at least), Sea of Thieves, etc. then why not take advantage of spare cloud capacity?

I'm talking specifically about single player games. HZD for example. Where's the business sense in implementing procedural snow server-side using several teraflops of server processing power for a single person playing at home?

For MS, they had a potential platform differentiator and Azure servers sitting idle, so it made sense that they could use some of that idle time for cloud augments. But that's not happening, leaving a noteworthy cost to publishers to offer single-player augmentations. Potentially as a differentiator, it might make sense for some games, but I feel that's hard to justify in the board room.

Virtually ever stream I've watched from a popular streamer has the streamer constantly going on about how absolutely amazing the game looks, and most of that is due to the water simulation.

I think we need to learn more about the water simulation first to know what is and isn't possible locally. The rendering is client side, so the physics is coming from the server, and how much of that is due to wanting to keep it uniform across players versus how much is because the console isn't able to simulate it locally? Looking at what Uncharted 3 managed on PS3, I'd expect something like SoT water on a next-gen console. I guess we all did, because no-one looked at it and said, "there's no way that's running on an Xbox One!"

In other words, when there starts to be graphics parity between a lot of AAA games, what can you do to "WOW" the players and make your game stand out? Sure eventually, if everyone is doing it then you'll have relative parity again, but for now, you can take advantage of it to make your game stand out visually. And in the case of Sea of Thieves, have it also affect gameplay to some degree. Just the visual effect on how a ship behaves on the waves becomes an integral part of gameplay if everyone is seeing the ships react the same way on the same waves (visibility, kill and hit confirmation, maneuvering, etc.). I've been surprised at how many times streamers have been caught by surprise when fighting multiple ships where they think they've sunk a ship (it's in the bottom of the wave swells, so a quick glance makes you think they are sinking) start to focus on another ship and then get attacked and potentially sunk by the ship they thought they'd taken out.

As consoles get more powerful locally, that stuff happens locally. Again, Uncharted 3 had huge storm waves and disappearing vessels. That affect on gameplay you talk about doesn't require the cloud. Cloud needs a role that requires many teraflops, making it stand head-and-shoulders above what a console can do, if it's to have a presence in single player games.

I need help to understand why the added cost is not a factor for multiplayer games like Sea of Thieves and the water calculation it has, but it is for a single player game with the same water simulation.

Economies. Let's say SoT requires $500 of GPU to run the sea simulation for 100 concurrent users. Now let's say all those players are playing solo and you aren't sharing the same world model across them all. You now need $500 of server GPU to run each instance of the simulation for each player, or 100x the cost. Or, in Battlefield destruction. 32 people in a map and you blow a building up, that data is computed once and shared across 32 players. If they're all playing solo and each blow a building up, you need to run 32 instances of the simulations.

When you go multiplayer, you spread the cost per player for any data they all use. The more players you have, the wider the cost is spread.

 

[Shifty Geezer]

1) virtual cloud: it only needs to spin up as many instances as required to support the player base, thus reducing the overall server costs. This is a bit different from just dedicated server models, and usually this is where I make a distinction between what cloud and dedicated server models are, namely the ability to spin up and down resources on demand.

I agree with that, but I don't make the distinction when talking about 'game servers' as to whether they are dedicate to a game or spun up or down from a pool. There's still the cost to run them.

2) Subscription based model: gamers are paying for 'Game Pass' already where money from the subscribers can be used to keep these online services alive ie. traditional MMO model.

Except MS and Sony want that money for their profits. Why would they give it all up to run servers to make solo games prettier? That defeats the purpose of having these consoles and raking in the profits! You'd need a more expensive subscription, considerably more I think. Also, where does PC sit on that? Are cloud augments left out for PC users not subscribed to one of the platforms, thus making them purely cosmetic? Or is someone going to start a subscription service for PC gamers?

 

[Silent_Buddha]

As consoles get more powerful locally, that stuff happens locally. Again, Uncharted 3 had huge storm waves and disappearing vessels. That affect on gameplay you talk about doesn't require the cloud. Cloud needs a role that requires many teraflops, making it stand head-and-shoulders above what a console can do, if it's to have a presence in single player games.

The point here is that everyone see's the same thing happen. If the waves were modeled on each individual machine then not everyone would see the same thing. If you attempted to have all machines sync with each other that increases the latency relative to the server solution.

In Sea of Thieves, this is important as waves + horizon effect affects both the ability to see the enemy as well as where cannons will impact the enemy. If everyone wasn't seeing the same thing then modeling where cannonballs hit would vary from player to player. Same with seeing and reacting to the other ship.

The ship to ship combat as it exists in Sea of Thieves right now would be different if each machine rendered and physically modeled it's own waves. If nothing else it provides consistency. Nothing annoys me more in a game with other players than for me to call out something cool that I just saw, only their machine didn't render what I saw and rendered something else instead. One example I can think of immediately is pets (minions) in FFXIV. Minions are all rendered and animated client side. The only serverside communication WRT to minions is that one exists and is attached to player X. So when they do something cute, it is highly unlikely that it'll also be happening on another players screen.

Now-a-days enemy locations are generally synched as that's important. IMO, having environmental effects that visually affects what is going on be synced is the next big step in immersion. Especially as that means it can no be an integral part of gameplay. Having the ocean be sync'd among players is significantly more intensive than just syncing up a building falling (especially if the building destruction features a small set of canned animations as it is with virtually all games except Crackdown 3) as it allows player and ship positions to be synced and remain logical visually with the motion of the ocean.

As to your question earlier about single player games, it becomes a question of why not?

Let's use the ocean simulation from SOT as an example. Lets say a game has both a single player and multiplayer component. Why couldn't the ocean data be simulated for all players and shared among all players whether in a single player instance or multiplayer instance? The single player instance can just use the simulation data that has already been generated for multiplayer.

Unlike something like building destruction, why couldn't the server simulated ocean be shared among a great number of player whether single player or multiplayer? The ocean (including weather) can be the same in all worlds and instances, the only thing that changes is where players are. Unlike building destruction which is going to change depending on instance.

So in that sense, some forms of server side physics and modeling would be more conducive to cost effective use in single player games than others (world state versus object state).

It'd be interesting to see a cost breakdown. Looking at something like PUBG, everything is server side and everything must be tracked for all players across the entire map as well as any changes to world items due to player interaction. And that's just a low cost game without monthly fees. I'm not sure it's using that much less server timer per player than SOT. PUBG has to track persistent changes per match. SOT doesn't. It tracks things per player, but any changes to an island are basically reset after a short time. Quests will instance and populate an island with relevant items [treasure chests] which when found can trigger enemies to spawn, for example. Once that leg of the quest is done, the island is basically as it was before for anyone else to spawn a quest on it.

Regards,
SB

 

[iroboto]

Except MS and Sony want that money for their profits. Why would they give it all up to run servers to make solo games prettier? That defeats the purpose of having these consoles and raking in the profits!

I think the answer in this case would likely lay with the studio. I’m not sure how gamepass games are paid back to the developer. I assume the the number of play hours from game pass players are likely some metric. More playing = more paying.

I assume in this model as long as people are playing the game the revenue from the studio collected some of it will have to go towards paying server costs. I don’t think this Cost will be incurred by the platform.

As to “why” a studio would go this route. It’s entirely possible that with servers you don’t need to pay to patch. Looking into a Cost dynamics here at play, you could build an evolving and robust AI by making tweaks bi-weekly or monthly. You can continually add more enemies to the screen and have more going on because the CPU doesn’t need to perform the AI code for it.

The AI could be Tougher or more lively since you can update the AI code on the server side continually without the Cost issues of patching.

You can add new features to the world as well. There are some advantages with a living single player world. It also makes the generation last longer if the only major deficiencies are CPU based.

 

[AzBat]

PUBG has to track persistent changes per match. SOT doesn't. It tracks things per player, but any changes to an island are basically reset after a short time. Quests will instance and populate an island with relevant items [treasure chests] which when found can trigger enemies to spawn, for example. Once that leg of the quest is done, the island is basically as it was before for anyone else to spawn a quest on it.

SOT does track changes per server. You leave a chest somewhere on an island & you can go back to it hours later, as long as you're on the same server & nobody else picked it up. Gold treasure quests only instance the chest for that team. It stays hidden until it is dug up. Another team can't dig it up beforehand. But they can get it once the team who received the quest starts digging it up. Any emergent items are available to & seen by all players. You're right about enemy skeletons spawning once a chest is dug up. Though they can also spawn up randomly too.

Tommy McClain

 

[Shifty Geezer]

The point here is that everyone see's the same thing happen. If the waves were modeled on each individual machine then not everyone would see the same thing. If you attempted to have all machines sync with each other that increases the latency relative to the server solution.

Yes, it's necessary for multiplayer. It isn't for solo play.

As to your question earlier about single player games, it becomes a question of why not?

Let's use the ocean simulation from SOT as an example. Lets say a game has both a single player and multiplayer component. Why couldn't the ocean data be simulated for all players and shared among all players whether in a single player instance or multiplayer instance? The single player instance can just use the simulation data that has already been generated for multiplayer.

Great as long as what you're simulating doesn't impact the game at all and is just cosmetic. The wave simulation would have hulls in the water deforming it. In the solo game, you'd see invisible ships using the same data. In a crowd simulation, you could see all the same entities, and then an invisible man running through the crowd causing reactions, from the model simulated for another player.

Unlike something like building destruction, why couldn't the server simulated ocean be shared among a great number of player whether single player or multiplayer? The ocean (including weather) can be the same in all worlds and instances, the only thing that changes is where players are. Unlike building destruction which is going to change depending on instance.

There are probably some cases where it could make sense, but we need an argument that it's all round a good thing to fullfil the 'dream' and give Alucardx23 what he's wanting see in games.

 

[Alucardx23]

There are probably some cases where it could make sense, but we need an argument that it's all round a good thing to fullfil the 'dream' and give Alucardx23 what he's wanting see in games.

Sharing the water simulation with all players in a single player game would be a good solution. The same can be said for volumetric clouds or crowd simulation, but you're right I need more than that to have my mind blown. Having a real conversation with an AI/cloud powered character that improves with time, based on the interaction data would be around there. I would pee myself if I see something like this, specially in a VR game. Now think about how much you have to pay to use Siri, Google assistant, Alexa or Cortana. .

I want to see this not just with you talking with an AI character, but with an AI character having a real conversation with another AI character, based in the context of what is happening in the game.

 

[AlNom]

Sounds like a nightmare for QA.

 

[Nesh]

Sharing the water simulation with all players in a single player game would be a good solution. The same can be said for volumetric clouds or crowd simulation, but you're right I need more than that to have my mind blown. Having a real conversation with an AI/cloud powered character that improves with time, based on the interaction data would be around there. I would pee myself if I see something like this, specially in a VR game. Now think about how much you have to pay to use Siri, Google assistant, Alexa or Cortana. .

I want to see this not just with you talking with an AI character, but with an AI character having a real conversation with another AI character, based in the context of what is happening in the game.

Well one thing seems certain The cloud still doesnt improve visuals in any way as originally promised. It is more of something that keeps everyone in sync with whats happening on screen.
And I suspect that whatever cloud features/implementations are in store for XBOX it will become a defacto feature for all platforms.