Server based game augmentations. The transition to cloud. Really possible?

[Billy Idol]

I'll go one step further. Enormous GPU compute is only possible with parallel workloads, meaning large datasets, yet the bandwidth limits of the internet massively cap the rate data can be sent to/from the servers. 1.2 TF burns through 1.2 trillion 4 byte single floats a second, 4.8 terabytes/s. If you could upload 1 MB/s to the servers (which is an enormous 8 Mbps upload BW), 1.2 TFs could process each byte with 4.8 million operations. There's no way you're going to want to process the byte that many times to arrive at a final value! If we compare that to MS's claimed 200 GB/s in Xbox One, that's enough for the GPU to process each byte with 24 operations. There's no way to utilise massive processing power for sustained periods unless all the data is local to the server, which means running them more like game servers than distributed computing nodes.

Maybe that's how MS will provide their supposed performance? Each console has access to 4 TF/s, but only needs a microsecond timeslice of that power. That way a 4 TF/s total power server could deliver 4 TF/s to thousands of consoles, but only in tiny bursts. In essence comparing a sprinter to a marathon runner using a choice but unrealistic number to sound faster.

That is an interesting post Shifty. I remember that we once did a project on distributed High Performance Computing, where we connected two machines across Europe via an internet protocol...we really had to work hard to hide the latency introduced by the slow transfer rate.

It was super hard to find a task with low enough amount data that needed enough operations to make it worth...at the end we found some configurations that were reasonable, but what helped us was that in large scale HPC you often have timings of several seconds to get a critical result due to the massive amount of operations needed with total run times in the range of hours.

So in conclusion maybe high fidelity simulation tasks can be done in the cloud. Just an example: fluid simulation. Character jumps into some water and generates fluid reactions. Resistance 2 did this by solving a wave type equation with FFTs accelerated on the SPUs with a source term to simulate the player. But you could see how low res the simulation is, as single triangles in the waves were visible as only a 32x32 simulation grid was used (iirc).

So basically to get a high fidelity version, you need just the position of the player and maybe velocity to calculate the source...with this low amount of data a wave simulation, or with enough power a fluid simulation kicks in on the cloud to compute the result, transfer back 2D displacement data which is then rendered again local on the box?!?

 

[patsu]

They'd need to have a good physics engine to handle stuff within the sphere of influence at that fidelity, sure. But that region can be relatively small depending on the game's design. And I'm envisioning exclusive games that REQUIRE the cloud for simplicity here. So no worries about stuff that has to be ported. Wanna start big (opposite of Scott's approach to building a model for conceptualizing things).

My thesis is that at this point in time realistic movement, physics, and animation is a significantly more visually convincing payoff than any amount of GPU power short of path tracing, at least for any relatively next gen rendering engine.

Any ideas how to consider the data flow though? That is likely the bottleneck I'd imagine. How much data flow is actually needed for local assets and local rendering to playback these scripts back? Remember though, player's can wait for a bit if the data takes a bit to all get through. Still, some guesstimates would be a nice start. Any devs out there care to take a stab at answering?

Even for first party, they have to plan for downtime -- unless they are making an online only game. ^_^

Today, when the network is down, we may play alone, on LAN or vs bots. If the network is down, this thing won't even render properly if you use it for core rendering experience.

Judging from their responses, they haven't really thought through the edge cases.

EDIT:
We have to remember something may go down during heavy holiday playtime.

 

[Tap In]

300K servers later this year. I thought they are already here or close enough, which is why I think it makes no sense and should be virtual.

not sure I follow. Are you saying they just decided to build a facility or add these servers this month rather than perhaps a year ago?

 

[patsu]

not sure I follow. Are you saying they just decided to build a facility or add these servers this month rather than perhaps a year ago?

When they announced 300K servers, I thought they are already here (or close to 300K).

EDIT:
Did a quick check. Google has 1 million servers around 2011, growing at 100K+ a quarter at that time.
http://www.pandia.com/sew/481-gartner.html

 

[Tap In]

When they announced 300K servers, I thought they are already here (or close to 300K).

well if you look at that article they have 300000 in Chicago alone (built in 2009 and is a gen 3) and at that time were on pace at 20,000/mo continued growth WW in other facilities and here we are 4 years later since Chicago.

they way the talking heads sounded they are being deployed physically and for Live specifically and would be available by launch for full service. Each game gets servers for matchmaking as well from that group as live/OS now has a custom matchmaking system built into it.

 

[patsu]

Yes, but they will need to deploy the 300K servers for Live all over the world to ensure good latency.

 

[Tap In]

Yes, but they need to deploy the 300K servers for Live all over the world to ensure good latency.

yep and it appears they have been doing that for a while now to have ready by launch


judging by that video they have more than a million already deployed perhaps many more

 

[Shifty Geezer]

So basically to get a high fidelity version, you need just the position of the player and maybe velocity to calculate the source...with this low amount of data a wave simulation, or with enough power a fluid simulation kicks in on the cloud to compute the result, transfer back 2D displacement data which is then rendered again local on the box?!?

That might work. You'd need to upload the game state for objects in the water, but 100 ms wouldn't be bad for water response while wading through it, and interpolating on laggy data wouldn't be too distracting either. the 2D data could be heavily compressed and still work.

Another simulation could be wave simulation perhaps. Maybe batches of states could be rendered and cached, and the console tween them, applying local distortions?

Still, there's going to be some issue with immediate response. The moment you shoot at the water, you need an instant splash. That'll have to be rendered locally, and that'll need some alternative solution to the awesome cloud-based simulation. Or maybe the same physics could be applied just to a very small area?

 

[patsu]

yep and it appears they have been doing that for a while now to have ready by launch

Then Microsoft can already use the deployed ones as standby servers for existing services for efficiency and environment sake. That's how EC2 works too. Otherwise, it's just a waste of space, electricity, connectivity, etc.

 

[Tap In]

Then Microsoft can already use the deployed ones as standby servers for existing services for efficiency and environment sake. That's how EC2 works too. Otherwise, it's just a waste of space, electricity, connectivity, etc.

cool.

I think You would enjoy that video then.

but yes MS has the hardware for this.

 

[patsu]

That's what I meant partly by counting existing servers. Conversely, the existing servers may already have spare standby boxes.

When Google shutdown unwanted services, they can reclaim (usable) old servers too.

 

[Tap In]

yes and if they are up to 20k+ per month / 300k/year growth already, I'm guessing this has been in the plan for a while now as well since they could account just 1 year aggregate of growth for the new Live and cloud computing

 

[Lalaland]

Why would they do that when they plan on making every X1 act as a thin client for full cloud gaming in a few yrs anyhow? They'd have to build those server farms and invest just as much in the tech then. Might as well do it and save money on hardware costs in the interim while getting a foothold on that kind of tech approach early.... is far more well thought out than many ppl are giving them credit for. This isn't just marketing speak.

I don't think the X1 (I think I prefer XBOne but I am childish like that ) is a good thin client candidate. It's too hot and heavy to deliver the '$50 bucks and a sub = awesum grafix' that remote gaming will one day deliver. I'd envision an Xbox version of the OnLive thin client as you can make those for half nothing. In terms of investment in the back end to support that investing later is always better as the cost curve on compute density, power consumption and capital cost only goes down over time.

I've no doubt they've more work done on this than we're we're being told but that's what is so frustrating to me. Surely someone in Redmond has coded up a tech demo or can be wheeled out to talk sensibly about this stuff. The current guff from MS has the faint whiff of desperation that I'm used to hearing from grasping SaS vendors who can't quite explain why I should invest except 'but the cloud....'

The remote physics stuff sounds interesting but that fundamentally seems like a flavour of in engine cut scene if it only happens beyond the borders of my 'sphere of influence'

 

[sebbbi]

The main idea about the cloud computing is improved utilization of the computing resources. This is true because resources are shared between all users. Home computer appliances (tablets, game consoles, PCs, laptops) are offline for most of the time. When a device is not used, it's compute cycles are totally wasted. In a cloud server farm, the computers are running all the time, 24/7 at full utilization.

Lets assume for a moment that an average console owner plays on average 10 hours per week (this includes also casual players and those who bought the console mainly for BluRay/netflix/TV purposes). That is 10 hours / (7*24 hours) = 0.06. 6% of the total time. This means that on average 16.8 consoles could share the same cloud computing unit.

Lets assume our cloud servers are built using 8 core Sandy Bridge E server CPUs. Sandy has around 1.5x IPC compared to Jaguar (according to various recent Jaguar benchmarks), and has around 2x clocks. In total a single Sandy Bridge E is around 6x CPU processing power compared to a 4 core Jaguar (or approx 3x compared to a 8 core Jaguar). A single 1U rack server could be equipped with four of these CPUs. An industry standard rack is 42U tall. So it has a total of 168 of these Sandy Bridge E processors. A single processor can serve 16.8 consoles (average 6% online time per user). Thus a single rack can serve 2822 sold consoles (if the target is 3x CPU performance).

Result: For each 2822 consoles sold, the console manufacturer needs to add a single new server rack to one of their existing server farms. That's not impossible or insane by any means, especially if they are selling the extra cloud processing time to research facilities, universities and big companies (finance, etc). That would balance the cloud utilization (around the clock).

Latency is course another matter. Cloud is not good for tasks that require single frame latency. However games have lots of tasks that are latency tolerant. It's true that the could has too high latency to directly render graphics, but it could be used to bake lighting to static scenery (lightmaps & probes). This is especially good for dynamic game worlds (and for user created content). Partially baked lighting reduces the lighting cost a lot. Thus the GPU doesn't need to process as many dynamic lights per frame. Cloud will be a big thing in the next 10 years. Xbox One is a pioneer in this area in regards to gaming. Is it too early? Only time will tell. The average speed of internet connections has been improving a lot every year. We have to assume that these consoles will be around for many many years, and lots of things will happen during that time.

 

[Arwin]

True, but peaks for internet tend to be around the same time, e.g. between 21:30 and 22:00, and gaming is likely similar. Now not all games are going to use these features and not everyone lives in the same timezone, so that helps some, but then servers are going to have to be in the same timezone for a large part in order to serve with low enough latencies (at least for a while).

Main problem I'm still seeing is that almost all successful applications have been for either things that are properly shared across multiple platforms/interfaces, or where few need the power of many. But say that you have a Forza 5 session with a lot of trading going on, or like GT5 where you could have your B-class driver race and earn money and experience in online races that did not require you to be in front of your PS3, then having a private always on session that you can use for hosting all sorts of permanent activities, including perhaps a racing series, could definitely be interesting.

And I bet there are areas where actual game worlds can benefit from this technology. But I do wonder whether that will be cost effective, both in terms of development complexity (taking into account consumers from all over the world, vastly differing internet performance and so on) and the question whether the money spent on computing power is really that efficient anyway.

Still I definitely see potential for open world games having a much more complex economy going on with a lot more npcs - but the question will then be if that is necessary versus just having that complexity being fed by other players and their games to begin with.

Maybe a more important puzzle will be to solve the vastly differing load demands that online servers can have for online games, and having a fixed amount of server-side resources available from Microsoft per machine can help mitigate issues here, but then there is the question how many first party titles there are going to be that use this and how many multi-platform titles if they need to support the same type of functionality on two other platforms as well.

In a way it feels like a reverse idea of distributed computing, and somehow I can't put my finger on it, but it seems ... counterintuitive? And why only CPU? Is that really the most important resource? I think this may actually be used more and sooner for the available additional memory, perhaps. Maybe like a very complex save-game system, where building up all the different things that change in a world like Skyrim will never increase the burden on the game engine?

 

[astrograd]

Even for first party, they have to plan for downtime -- unless they are making an online only game. ^_^

And? I am personally 100% in favor of online-only SP games if they are leveraging the cloud. I agree with what Marc Whitten said. If your game is so amazing that it has to leverage 32 cores and loads of memory in the cloud, then gamers will be thrilled to experience that kinda thing.

Judging from their responses, they haven't really thought through the edge cases.

Don't really think that's fair to say at all. They just said that they allow devs to make their title online-only (which we've known for a while now) and that they'd be thrilled if devs made games so amazing that could happen. I don't think that suggests anything about edge cases.

EDIT:
We have to remember something may go down during heavy holiday playtime.

They are going to be operating under the assumption that their servers never go down. That sounds risky, but it's really not given the backup tech they have in their professional-level server farms. This isn't like how XBL is today with 80mil ppl capable of accessing digital content and almost 50mil of them capable of playing games head to head online. That's done on 15k servers today and is still pretty damn robust. They bumped that up to twenty times that number just for launch and you can bet they will continue expanding for a while in 2014 and 2015. 300k should be WAY more than enough. Especially for launch when shipments are still a bit low. I can't imagine they would even attempt this if they weren't 100% confident in the infrastructure's complete integrity.

Btw, did you watch that video posted by Scott earlier? My impression of what MS said about this 300k figure was that they are moving the 15k servers that were previously used just for XBL over to 300k servers. Just XBL running on those servers. At launch. May be wrong though. I also got the impression they hadn't deployed all of that yet, but in the video it suggests they can do huge scale deployments extremely rapidly thanks to how modular their setup has become. So if ya haven't seen the vid, check it out. Might answer some questions/concerns you had, especially environmentally.

 

[Betanumerical]

And? I am personally 100% in favor of online-only SP games if they are leveraging the cloud. I agree with what Marc Whitten said. If your game is so amazing that it has to leverage 32 cores and loads of memory in the cloud, then gamers will be thrilled to experience that kinda thing.



Don't really think that's fair to say at all. They just said that they allow devs to make their title online-only (which we've known for a while now) and that they'd be thrilled if devs made games so amazing that could happen. I don't think that suggests anything about edge cases.



They are going to be operating under the assumption that their servers never go down. That sounds risky, but it's really not given the backup tech they have in their professional-level server farms. This isn't like how XBL is today with 80mil ppl capable of accessing digital content and almost 50mil of them capable of playing games head to head online. That's done on 15k servers today and is still pretty damn robust. They bumped that up to twenty times that number just for launch and you can bet they will continue expanding for a while in 2014 and 2015. 300k should be WAY more than enough. Especially for launch when shipments are still a bit low. I can't imagine they would even attempt this if they weren't 100% confident in the infrustructure's complete integrity.

Its 24 cores, they haven't mentioned memory yet either, only HDD storage I believe and levering a lot of cores doesnt really mean much. All it means is your using a lot of power, doesnt mean your using it for anything worthwhile.

 

[Scott_Arm]

Even for first party, they have to plan for downtime -- unless they are making an online only game. ^_^

Today, when the network is down, we may play alone, on LAN or vs bots. If the network is down, this thing won't even render properly if you use it for core rendering experience.

Judging from their responses, they haven't really thought through the edge cases.

EDIT:
We have to remember something may go down during heavy holiday playtime.

They have thought about the edge case you're mentioning. In the arstechnica interview they mentioned having local versions of the effects that can run if the cloud is not available. Depending on what is being done, I imagine some effects might be cloud only, or the game itself may be an online only title.

 

[astrograd]

I don't think the X1 (I think I prefer XBOne but I am childish like that ) is a good thin client candidate. It's too hot and heavy to deliver the '$50 bucks and a sub = awesum grafix' that remote gaming will one day deliver. I'd envision an Xbox version of the OnLive thin client as you can make those for half nothing. In terms of investment in the back end to support that investing later is always better as the cost curve on compute density, power consumption and capital cost only goes down over time.

I've no doubt they've more work done on this than we're we're being told but that's what is so frustrating to me. Surely someone in Redmond has coded up a tech demo or can be wheeled out to talk sensibly about this stuff. The current guff from MS has the faint whiff of desperation that I'm used to hearing from grasping SaS vendors who can't quite explain why I should invest except 'but the cloud....'

The remote physics stuff sounds interesting but that fundamentally seems like a flavour of in engine cut scene if it only happens beyond the borders of my 'sphere of influence'

After thinking on it, I maybe shouldn't have been so narrow in my phrasing there. You don't really need to limit yourself to just stuff outside some given radius. You really need to just limit your cloud processing to elements that the player cannot interact with, which means in a game like GTA, for instance, there is a huge city full of buildings with npc's in them potentially (think: next gen city-scale AI simulations)...but the player isn't going to be able to actually see, let alone interact, with any of them unless he is in the same building or even room as them (even if spatially the building is right next to him).

So even in my conceptualized 'sphere of influence' you could have cloud operations running continuously updating animations or AI routines etc, so long as the player can't interact with it immediately. I bet if you took a modern game world and actually analyzed it you'd find that in most games the VAST majority of "stuff" even within this sphere of influence is non-interactive.

As for tech demos, I agree 100%. All indications suggest MS has been planning this for a while as pit stop on the way to full server rendering/cloud computing later on. I want to see them flesh out the claims with some data/graphs. I then want to see tech demos and some behind the scenes info on how those tech demos were put together. And then I wanna see actual gameplay demonstrations showing off the results, especially side by side with a purely local solution if at all possible.

I think MS should make 30min shorts where they focus on each major aspect of the new console and its selling points leading into E3 so we arr fully up to speed on the platform as a whole, their ambitions, and their specific claims. Save the show for major announcements and stage demos showing proving these claims are legit.

 

[Jedi2016]

So even in my conceptualized 'sphere of influence' you could have cloud operations running continuously updating animations or AI routines etc, so long as the player can't interact with it immediately. I bet if you took a modern game world and actually analyzed it you'd find that in most games the VAST majority of "stuff" even within this sphere of influence is non-interactive.

It would seem to me that a lot of that non-interactive stuff isn't very computationally intensive anyway, for the very reason that it's not interactive. I wonder if the cycles required to determine what should and shouldn't be offloaded, then sending that info out to the cloud, receiving it back, then incorporating it into the world is about the same as what it would take to simply compute it locally. Cloud for the sake of cloud.

Other non-interactive stuff, like the bridge destruction on the last page, could be simply pre-computed and stored locally. If that scene, or scenes like it, are just cutscenes, even if done in-engine in realtime, then all of that could be computed ahead of time, and the physics and deformation could just be "played back" off of the disc or HDD. In a scene like that, CPU overhead would be surprisingly small.

Personally, I think the whole "cloud" thing is just marketing talk, and I don't see them really using it for anything that gamers would even notice except for obvious stuff like multiplayer or MMO-type games.