*Game Development Issues*

[JPT]

I mean, I can see why the 360 is the lead sku so often. It came out first. It's simpler to work with and faster to produce results, but if your goal is to create a multi-platform title for simultaneous release there are decisions that can be made up front to save you grief down the road.

But you must factor in that the projects that are being released now, might have been started before the PS3 dev kits etc was available for 3rd party devs.
Which means there is no other choice than go from 360 -> PS3 for these rounds of games, but it has not been mentioned for future projects, except Midway seem to have given out some statement about next projects will have PS3 as lead dev platform.

I think Jokers comment about that they are halfway to the point where they are looking at work for the next Xbox generation of machine, is very interesting. Ie if MS does come with a new platform in 3-4 years then Sony will/might have a problem, since they seem to be betting on the PS3 for the next 6-7 years. And then it looks like ease of use or maybe less advanced technology will win over more advanced technology, something in my head just said VHS vs Betamax now.

Barbarians comments are insightful, but as a manager in a small SI in the telecom industry, I see his problems more as a new business opportunity challenge than anything else. Even if the statements like "Studios are 1 flop from going under" etc are true, the time to do the foundation to release for major platforms simultaneously must be worth the risk?
If the time to do the things Nao says, especially if its on the timescale he indicates, then building those tools, it got to be a must win solution to do it. Even if it takes timeout from your current project, it looks to me to be what will give you extra time on your next 5-10 projects and increase your target market segment from 360+PC to 360 + PC + PS3, well if having a quality product on each platform that is.
It really does seem like a no brainer to me, heck even spinning it of and license it as some extra tools for other developers to also use, especially if it is missing from the tool chest Sony supplies, could help on the bottom line of a studio.

 

[Shifty Geezer]

Great post, though it doesn't really excuse Booth for the demonstrably false claims he has made. The telling part is that sure, 360 code runs like shit on the PS3, but the solution is as simple as starting on the PS3 and porting to the 360.

That's missing some devs point though, such as Barbarian above. It's not just getting PS3 up to speed, which is one problem addressed by the PS3 being the primary platform, but also the difficulties in developing for Cell and having to get Cell to help out RSX. The fundamental systems are inherently more complex.

Barbarians comments are insightful, but as a manager in a small SI in the telecom industry...If the time to do the things Nao says, especially if its on the timescale he indicates, then building those tools, it got to be a must win solution to do it.

How many managers, specifically of software developments, are happy to let their teams take time out from making products they can sell to make products that'll help them make products they can sell? From my experiences, and traditional stereotypes and parodies, your typical manager or at least SEO giving orders from above, sets their targets on immediate product releases, and 'clever' programming, preparing things for a comfortable future, never happens. It's probably the case some studios could have benefited immensely taking a year out to play with the systems, develop tools and toolchains, explore methods etc., before doing any actual game development. But then what would pay the bills? Long term, they'd possibly make savings to justify the expense if they could afford all that preparation. However, driven by having to get products out doors, and with one/two easy-peasy systems and one untrained pig that they don't feel, or aren't allowed due to orders from above, to train, that pig will be very frustrating!

 

[iceberg187]

I think PS3 being hard to code for is good thing in the long run because that means there's going to be progress, later generation PS3 games are bound to be more impressive.

Kind of like how early 2d psone games went on to impressive 3d and how ps2 games went on from early scratchy Tekken to later jaw dropping titles like Transformers. Or I could talk a while about how nes (mario bros to super mario 3) and snes games (early titles to donkey kong country) progressed. I find that kind of progress appealing, me.

I didn't particularly like how the original Xbox was almost immediately spawning games that were pretty much as impressive as they were going to get.

I may be wrong in suggesting this, but even though the industry is seemingly lead with a 360 sku, few of these 3rd party games are really pushing the console like a first party would since they are developing for two consoles at once and developing too much for one console would leave people scrambling like a chicken with their heads cut off. Back to your Xbox game comment, very few games are developed with the Xbox as an exclusive title. The ones that were, like Ninja Gaiden, Panzer Dragoon Orta, Jet Set Radio Future, were far and few between. Even though the console is relatively easy to develop for, look at the progress Bizarre made from PGR3 to PGR4, there are links to pics in the the PGR4 thread. It is astounding what they have done from 3 to 4. In any event, Microsoft still has Rare and Rare can push visuals. They are better than all other first parties people they have. From what I been told by various people and Blimblim on GAF, their R1 engine has made tremendous progress on the 360. So, basically wait and see what Rare does before you say something is as impressive as launch date.

 

[Betanumerical]

I'll give you IW and id...

Haven't all reports of UT3 been good?, I realise that previous iterations / builds / optimised versions of UE3 engine have been poopoo on the PS3 but it seems to be different with UT3 for some reason.

 

[wco81]

If PS3 sold better and PS3 games did well, you wouldn't hear this bitching.

Right now, the developers don't see the economic incentive to do the extra work for PS3.

A couple of years from now, even if PS3 is still a distant 3rd, there should be enough units out there that multiplatform publishers just can't ignore or write it off. After all, that is how Xbox continued to get support til the end.

But it seems some developers just don't think the extra work is worth it right now, just about a year after launch.

 

[Shifty Geezer]

A couple of years from now, even if PS3 is still a distant 3rd, there should be enough units out there that multiplatform publishers just can't ignore or write it off. After all, that is how Xbox continued to get support til the end.

They're ignoring ~33% of the HD gaming market now then? If they were willing to develop for XB with more like 20% of the market, making difficult ports from PS2's architecture to XB's, why ignore PS3? AFAICS most titles aren't ignoring PS3 for just that reason. Even though it's a PITA, a full ROI for a title creation requires a PS3 version. The incentive to do a decent job will come presumably from restructuring their development to better work with PS3 rather than to continue beating their heads against the wall trying to get it to work with difficult systems and XB360 ports. Install base shouldn't be a driving factor, any different to now anyhow.

 

[Crossbar]

All the name calling aside, these hurdles have been faced by a lot of good developers. Nevermind engine developers like Epic and id who have a substantial stake in getting their engines to run well on the PS3. How would people feel if GTAIV was delayed because the PS3 version was lagging behind the 360 version, something that has happened to a fair number of studios? Rockstar may be full of lazy hacks, but their games are extremely popular and critically praised.

Another rumour said that Rockstar wanted the game to depend on the harddrive in the console to make the game they wanted, but Microsoft was not willing to change the rules for such a high-profile title before the holidays.

I don´t know if rumours from msnbc are more credible than others, but as FUD is part of the game I am not putting to much trust into any rumours.

The comments from Barbarian also give testimony of that poor project management is part of the problem for some multiplatform titles.

If they are not prepared to let their best software architects to design a common basic framework at the beginning of the project then you are really asking for it.
In the same sense it couldn´t be that hard to make a framework for SPU handling that contained a few "ifdefs" that let you run the code on the PPU while debugging, letting you have stack supervision etc..

 

[wco81]

Not saying ignoring it. Publishers will support it because that is their business model and PS3 may yet get more PS2 gamers on board.

But it sounds like some developers are questioning whether it's worth their pain. They're bearing the brunt of the decisions by publishers.

 

[Arwin]

Trust me I do wish I had the time to write all these nice things.

Is nAo right though? Would it take three weeks? Because if so, than is it just not really a case of bad priorities?

And it's great that your team had few nice chaps to sit down and design that system, and lucky for you you had 3+ years exclusive platform development.
The important thing about X360 and I'll repeat that again is that "It Just Works". From the get go I get decent performance, CPU and GPU. Only and I really mean, ONLY if I need to optimize something for the needs of my game I then can spend time refactoring the system to take advantage of multiple cores. And that takes very little time because of the nature of unified memory address space. [edit - and by that I mean that the multiple cores all see the same memory, can read and write to it within the same address space, the only "problem" being you need to protect(syncronize) any shared resources]
On the contrary, on PS3, you need top engineers devoted just to get basic stuff off the ground

'just'? Isn't that understating the importance of the basic stuff?

, and you NEED to use SPUs to get decent performance out of RSX.

That latter comment surprises me at least a little bit.

And let me go back to the topic of SPU programming since I forgot to mention few aspects of it. The SPU has local address space so any pointers in your data have to be translated, DMAed etc, which is PITA. Structures need to be refactored and often "flattened" which often defeats clever memory sharing/optimizations. The memory is limited so you have to partition the problem. The code might not even fit initially, and you might not be able to run a debug version of the code. Until recently one couldn't even do breakpoints inside SPU jobs, and now you have to find the SPU code in memory before it gets uploaded and set a break point there, very similar to VU programming back in the day. Forget about asserts working, and printf might or might not be reliable. A "HALT" on the SPU could mean anything, especially if it's inside a middleware SPU job. It could be their bug, or yours, someone stomping the code or data, or it could be stack overflow. You get basically zero tolerance for error. That is not what I call "It Just Works".

Many of these problems do sound like code that is translated to SPUs, not designed for SPUs / streaming. But that's understandable. And from that perspective, I can see how optimising a game for the PS3 is a pain. And as you say these coding problems are not new, and being able to debug your code in runtime is useful. Though can't you do at least part of that work in the emulator on a PC?

Also, does your company talk at all with other companies developing for the PS3? Any sharing of experience at all? I would imagine that companies like Insomniac or Naughty Dog have a strategy for dealing with stuff like this, that you could benefit from. Personally I would develop these SPU programs in isolation, test their input and output individually, and well what nAo said. But that's just silly sideline second guessing on my part. Instead, it seems that perhaps your company isn't receiving the best of Sony's support for political reasons, which is sad.

I mean come on, Microsoft of all companies are known to be sloppy, and their APIs are not perfect and they don't get everything right from the get go. Sony with all their open source/Linux fundamentals, how can they screw it up so bad that they have worse offering than MS?

How much of the stuff you are using is from IBM? IBM handles a lot of Cell stuff and documentation, right? I can right now (and have a long time ago) download an excellent programmers manual for Cell, that even someone like me, a programmer with mostly business and .NET experience (though a decent and long IT background and some low-level and PSP experimentation), can understand pretty well. I can right now install Linux and get the latest Cell SDK (3.0), and start working on SPU code.

That said, it's obvious that for the most part, for me personally, I'd probably get code up and running with the 360's SDK much more easily as well probably. Heck if stepping through code is possible, I'm sure a company could hire me right now and make some decent use of me within days. So that right there is a big plus, no doubt.

But I've always felt that competition in the console space always eventually comes down to proper low level code design, and then it's best to bite the bullet as soon as you possibly can. Maybe in today's transitional market that's not as true as it has been. But if what we've seen in console history has any bearing on the future, it will be again very soon.

If what you say is true though, then Microsoft has, what I think, succeeded in this particular area, which is giving developers a head-start on 360. But if that helps them postpone the in my opinion inevitable, then it may come to bite them again later on when PS3 development has been forced to be more efficient, which is basically what John Carmack said about the differences between 360 and PS3 already two years ago. I would not dare to say which of these will come to have a heavier bearing on gaining market space ... eventually, exclusives matter a lot, and good exclusives can come from both these areas.

 

[Arwin]

All the name calling aside, these hurdles have been faced by a lot of good developers. Nevermind engine developers like Epic and id who have a substantial stake in getting their engines to run well on the PS3. How would people feel if GTAIV was delayed because the PS3 version was lagging behind the 360 version,

Sorry but even this link to a Microsoft owned network doesn't mention that the PS3 was the reason for this delay. If I remember correctly, the guys in charge got to see the 360 version, and on the 'merit' of that version decided that the game wasn't ready for release. You're not going to convince me that they signed off another year of development on this game just so they could optimise the PS3 version while the 360's version was all ready and waiting. If it was only a matter of setting the whole team at work on the PS3 version so that it could get the same decent framerate that the 360 version had(n't), then that wouldn't have stopped them from hitting a Christmas release.

 

[Shifty Geezer]

Sorry but even this link to a Microsoft owned network doesn't mention that the PS3 was the reason for this delay.

It does (when you use Microsoft's browser, because the story doesn't display properly in Firefox. I wonder why?...) but only as a 'slanderous' allegation. They claim it's a well-known secret that Sony held it back. Maybe, maybe not, but this article certainly isn't a suitable source for basing opinions. A better link would be to some of those rumour sources.

 

[one]

Forget about asserts working, and printf might or might not be reliable. A "HALT" on the SPU could mean anything, especially if it's inside a middleware SPU job. It could be their bug, or yours, someone stomping the code or data, or it could be stack overflow. You get basically zero tolerance for error. That is not what I call "It Just Works".

Does the SDK lack a SPU debugger or a memory viewer/watcher for LS? I'm curious how a debugger on a processor with a limited scratch pad memory works, does anyone know the details?

If PS3 sold better and PS3 games did well, you wouldn't hear this bitching.

I'm pretty sure it happens regardless of sales just like PS2

 

[wco81]

Why would RockStar/Take Two delay both versions if the PS3 version was behind?

Why would they do Sony that kind of favor and take the hit to their own stock price, especially when MS paid big for the exclusive content?

 

[Arwin]

It does (when you use Microsoft's browser, because the story doesn't display properly in Firefox. I wonder why?...) but only as a 'slanderous' allegation. They claim it's a well-known secret that Sony held it back. Maybe, maybe not, but this article certainly isn't a suitable source for basing opinions. A better link would be to some of those rumour sources.

Thanks for pointing that out. Interestingly, it also doesn't show up properly in Opera ... No further comment.

 

[AlphaWolf]

Sorry but even this link to a Microsoft owned network doesn't mention that the PS3 was the reason for this delay. If I remember correctly, the guys in charge got to see the 360 version, and on the 'merit' of that version decided that the game wasn't ready for release. You're not going to convince me that they signed off another year of development on this game just so they could optimise the PS3 version while the 360's version was all ready and waiting. If it was only a matter of setting the whole team at work on the PS3 version so that it could get the same decent framerate that the 360 version had(n't), then that wouldn't have stopped them from hitting a Christmas release.

I love how you throw your own fud in there as an official source, but dismiss other rumours, thats cute.

The official word is basically nothing, they've not said (specifically) why the game was delayed.

 

[Shifty Geezer]

I love how you throw your own fud in there as an official source...

Arwin didn't add his as an official source. He's offering a different rumour he 'heard'. So far no-one's actually linked to sources of any side, so all are equally worthless basis for any arguments, like all Bloke-down-the-pub 'facts'.

 

[Arwin]

I love how you throw your own fud in there as an official source, but dismiss other rumours, thats cute.

The official word is basically nothing, they've not said (specifically) why the game was delayed.

Exactly.

By the way, the rumor I threw in came from the 1up show podcast. Noteworthy that those guys did in fact see the 360 build shortly before the decision was made, and they confirmed the rough framerates in that build. I think they also talked to the guys at Rockstar, but to be sure I get the details right I'd have to listen again (which is possible, you can listen the old episodes at their website)

 

[Mintmaster]

On the contrary, on PS3, you need top engineers devoted just to get basic stuff off the ground, and you NEED to use SPUs to get decent performance out of RSX.
And let me go back to the topic of SPU programming since I forgot to mention few aspects of it. The SPU has local address space so any pointers in your data have to be translated, DMAed etc, which is PITA. Structures need to be refactored and often "flattened" which often defeats clever memory sharing/optimizations. The memory is limited so you have to partition the problem. The code might not even fit initially, and you might not be able to run a debug version of the code. Until recently one couldn't even do breakpoints inside SPU jobs, and now you have to find the SPU code in memory before it gets uploaded and set a break point there, very similar to VU programming back in the day. Forget about asserts working, and printf might or might not be reliable. A "HALT" on the SPU could mean anything, especially if it's inside a middleware SPU job. It could be their bug, or yours, someone stomping the code or data, or it could be stack overflow. You get basically zero tolerance for error. That is not what I call "It Just Works".
I mean come on, Microsoft of all companies are known to be sloppy, and their APIs are not perfect and they don't get everything right from the get go. Sony with all their open source/Linux fundamentals, how can they screw it up so bad that they have worse offering than MS?

Thanks for illustrating a lot of the issues that devs face working with a PS3. This is the sort of thing that we can't really figure out with real development experience on the PS3 platform. A lot of things you mentioned there are things coders haven't had to think about for decades.

 

[Acert93]

Another rumour said that Rockstar wanted the game to depend on the harddrive in the console to make the game they wanted, but Microsoft was not willing to change the rules for such a high-profile title before the holidays.

I don´t know if rumours from msnbc are more credible than others, but as FUD is part of the game I am not putting to much trust into any rumours.

MSNBC is a profession news site; they are always careful to give the warning, "We are an MSFT Affilliate" but that doesn't prevent them from praising the competition--or running Op Eds saying Halo 3 is overhyped or reporting on the RRoD.

When a news site reports the same rumor I am hearing I am going to believe it over a FUD site dedicated to fighting the, "Evil empire of Microsoft's PR department" that claims the issue is due to the 360.

What we do know is there was some technical issues. Take Two had stated, "Certain elements of development proved to be more time-intensive than expected, especially given the commitment for a simultaneous release on two very different platforms" and has further noted that the decision to delay was, "almost strictly technological challenges". Take Two declined to get into specifics because they wouldn't be helpful. So we know that:

1. Parts of development (not all) were more time-intensive than anticipated

2. Notably in the context of a simultaneous release--i.e. one was taking longer than the other

3. The cause of the delay was primarily related to said technological issues

Shorthand: One of the platforms is taking a lot more time and effort to get into a shippable shape.

As Michael Pachter floated the idea immediately after the announcement, "We think it is likely that the Rockstar team had difficulty in building an exceptionally complicated game for the PS3, and failed to recognize how far away from completion the game truly was until recently". GTAIV wouldn't be the first title that used the Xbox 360 as its lead platform and had difficulties getting the PS3 version up to par.

On the flip side, the "rumor" that the result is due to the Xbox 360 is weak all around. First is the sources. Second, Rockstar has had mature 360 development kits for a decently long period of time and the press has played smoothly running demos of GTAIV on the Xbox 360. The Blu-ray angle doesn't hold much water because Rockstar knew the 360 had a DVD drive back in early 2005 at the latest. With the early release and larger western audiance (and MS moneyhats, including $50M for DLC) it seems nearly moronic that they would run into space issues months before release. Especially when the solution, using lower quality assets and audio, is trivial and has been widely deployed already on ports going the other direction. The angle that Rockstar was trying to require the HDD also falls flat because they would have needed MS approval to begin with. MS has rules about what game types can require a HDD. Rockstar clearly has been developing with the 360 in view and has received a bit of compensation for "Day and Date" and exclusive DLC, so the idea to build a game that doesn't work with the platform is nonsensical.

Sorry but even this link to a Microsoft owned network doesn't mention that the PS3 was the reason for this delay.

Yes it does!

Although analysts had predicted that "GTA 4," “Halo 3” and “Madden 08” would account for a third of all game sales this holiday, Take-Two Interactive, the game’s publisher, acknowledged in August that the title just wasn’t ready for prime time. Or more specifically, the PlayStation 3 version of the game wasn’t ready for prime time. Take-Two won’t comment on that part, but it’s about as big a secret as Joan Rivers’ plastic surgery.

The fact that game developers are having a difficult time with PlayStation 3 development is also no secret — the tech inside the high-performance machine is very technical, very complicated stuff.

Take-Two won’t confirm that it held the Xbox 360 and PS3 versions of “GTA 3” due to an agreement with Sony. But shipping one version of “Grand Theft Auto 4” several months before another gives a decided advantage to whoever’s first out of the gate.

(MSNBC is a joint Microsoft - NBC Universal venture.)

If I remember correctly, the guys in charge got to see the 360 version, and on the 'merit' of that version decided that the game wasn't ready for release.

And your source for this is?

Take-Two said it was technical issues preventing a simultaneous release, so where are the guys in charge saying it was the 360 version's fault?

You're not going to convince me that they signed off another year of development on this game just so they could optimise the PS3 version while the 360's version was all ready and waiting. If it was only a matter of setting the whole team at work on the PS3 version so that it could get the same decent framerate that the 360 version had(n't), then that wouldn't have stopped them from hitting a Christmas release.

How many B3D developers have stated in no uncertain terms that throwing more people at a problem isn't a magic bullet? That "solution" ignores two things (besides being bad management!): First is that the delay was announced in August and the release date was about 80 days away, meaning they had less than 2 months of actual development left--which would have included the traditional crunch on BOTH versions. So if the 360 version needed those two months of crunch to get a polished release, shifting those resources over to the PS3 version would delay that process as well. People and Time are finite resources, as I said above.

Second, the "throw more people" at the problem hasn't helped a fair number of titles that have had issues on the PS3, many of which after delays did ship the 360 version and the delayed PS3 version still had issues.

Of coure no one is going to convince you of your silly scenario!

Name one huge multiplatform game where the last 2 months of development the entire team shifted resources from one platform to the other because they magically had the other version completely finished 2 months ahead of time.

The scenario laid out is that Take-Two's commitment to a simultaneous release wasn't possible due to technical issues due to unanticipated time demands due to technical issues. I don't see how your suggestion fits with their position at all.

If PS3 sold better and PS3 games did well, you wouldn't hear this bitching.

Green makes [almost] everyone happy So why isn't PS3 software selling better?

Delays and price are factors, as are multi-console ownership (wrt software sales, where many multiplatform games are better on the competition), but at some point there will be a circular relationship. Whether it is technical difficulties or "laziness" if PS3 continue to be perceived as the less experience it will impact sales, and low sales will result in less resources focused on addressing these issues.

The PS3 gets at most 5-6 years of primetime market exposure (far less before marketshares are solidified) so these issues need to be resolved quickly. 10 years and 100M console sales on the back of strong software releases depends on extremely strong software sales and deep market penetration.

Potential isn't going to drive the PS3 there. Only software can.

A couple of years from now, even if PS3 is still a distant 3rd, there should be enough units out there that multiplatform publishers just can't ignore or write it off. After all, that is how Xbox continued to get support til the end.

I don't think people are expecting the PS3 to lose across the board support. That said the Xbox did a couple things to ensure support: As it failed in Japan, the Xbox had a high concentration of units in the West, notably NA. Substantial sales of west-please titles was attainable. Further, the Xbox was quite powerful and was easy to develop for so a port was far easier than a 360=>PS3 port.

Now shift focus to the GameCube. It sold about the same amount of units but they were spread across 3 territories (more variety in software adoption patterns), had a much lower attach rate for 3rd party software, and the GCN lacked the decided power/resources the Xbox had. And the GCN did lose a lot of publisher support, saw cancelled and delayed titles, and inferior ports.

But it seems some developers just don't think the extra work is worth it right now, just about a year after launch.

From what I read and am told developers ARE putting extra work into the PS3 versions. They are putting "extra work" into the PS3 version to get them out the door at the same time and at similar quality levels.

Does anyone have any real sources saying, "Yeah, we skimped on the PS3 version and focused on the 360 version" ?

 

[betan]

I mean come on, Microsoft of all companies are known to be sloppy, and their APIs are not perfect and they don't get everything right from the get go. Sony with all their open source/Linux fundamentals, how can they screw it up so bad that they have worse offering than MS?

Sony is probably doing this on purpose.They could easily with little effort ,match the 10 more years experience put in the MS toolchain integration : .net + visual studio + dirextx

Don't forget the developer friendly CELL architecture. After all it's trivial to write tools for CELL compared to complex PCish multicores that MS is familiar with.