*Game Development Issues*

Status
Not open for further replies.
Sure, totally fair on both points. I'm actually looking at this less on technical accuracy and more as a general problem with the industry. Yeah he did make some errors. But, theres a general problem brewing in the industry that he possibly inadvertently points out, where difficult beat my skull against the wall consoles with limited support are creating issues.

Maybe this is just in the USA, but I'm noticing many trends here. The obvious are previously mentioned things like skyrocketing budgets. Less obvious are constantly missed deadlines and their consequences. Not noticed by many is whats happening internally at many studios. I'm seeing more and more juniors, less and less veterans. I'm seeing people who've been coding games for 10 years burn out and exit the industry totally. I'm hearing grumblings of people tired of having to sleep at work or not see their spouse for weeks on end just to meet a milestone. I'm seeing people devote their lives to ship a game on both platforms by crunching 12-16 hours days for weeks on end, only to all get laid off right after the project ships.

To a certain extent, this has always happened in the industry. But it now seems to be getting more frequent. People exiting the business is becoming as frequent as people entering it. Expectations are becoming unrealistic. Quality of life is severely down. In some ways his blog to me was a way to say that overly complex platforms are not welcome anymore, when it's clear that things can be done in a simpler manner with a similar result. 360 was a step in the right direction to me not because it was done by Microsoft, but because it helped the industry from a quality of life perspective. For perhaps the first time ever, the developper got great tools and support from day one. PS3 to me is trying to pull things back in the other direction where the developer is basically cattle and expected to sacrifice himself to the mighty mother ship. That's just antiquated to me now.

I recall an event that happened in the last crunch where we were trying solve an issue on PS3 and one of the coders I was with told me he had missed his daughters first words because he was crunching. Sounds silly and mellow dramatic, but that kind of thing just gets one thinking. At the time, which was actually just before my now ill fated charalatan post, it got me thinking why in the hell should we be struggling like this to get basics working when this was all so easy on 360? Why do we tolerate this? Well, from the amount of people exiting the industry, it would appear that less and less are.

He may not have intended any of this with his blog, but thats partly how I read it. Help us out, don't treat us like a cog in the machine. Give us a balanced machine, and support us. I like the direction 360 took the industry. I do not like the PS3 pulling us back. It doesn't need to be this way. Worse yet, I can't help but wonder how long it can continue on this way. As the people playing games have aged, so have the people creating them. People want to see their kids and families and will be less and less tolerant of sacrificing them to meet milestone 3b.

Well all the things you descrped it seems will continue to exist and get worse with or no PS3.

I ve been reading a few articles on your continuous difficulties as game developers and I dont feel like this is going to change or remain stable. In all the gaming industry no matter what the platform is, developers are and will continue to sacrifice from their job security and social life. A survey showed a couple of years ago that divorces sky rocketed for game developers because they had no time to spend with their spouse or family. They were also I think failed efforts to create unions

The 360 didnt make things easier for you as I see it, it just makes your lives less hard compared to the PS3. And it seems as for now that the extra problems you are finding with the PS3 are a result of a transitional period trying to learn to work on something newly indroduced from a company (Sony) that lacks in development tools due to their lack of experience in the field compared to MS. You see this from your field of expertise and you are right, but this is a similar phenomenon observed in many industries in general as they change. Such changes are inevitable, so a change in one area will for some time have costs in other area especially for those that are old in the industry. Economists have been studying these phenomenons for years. Many will be bitter and disappointed about it for a period of time, but they are temporary. You are both lucky and unlucky that you belong in this field. Many industries seem interconnected in game development, which makes things even more complicated for you and everyone in each area, and as a result goals, interests, targets will sometimes collide.
 
Yeah it's MLB2K. I don't mind saying that now since it's my last year with the team anyways, I'll be moving on after we ship.

Sales +1.

I appreciate very much you sharing your experiences on B3D, and knowing a little bit about one of the devs behind the game makes me much more interested in picking it up, even though I never got into baseball video games before.

Same thing weighed in on my decision to pick up Heavenly Sword, nAo, Dean.
 
Now that right there, is very ugly. It reminds me of back in the day when EA and Microsoft had this big row over Live support and how the way it was set-up then would give Microsoft heaps of interesting inside information on the use of online play in EA games that they could then go on and use in their own competing sports games line. In the end, EA and Microsoft settled with Microsoft giving up on their own sports games line, and allowing EA to run everything but the actual user authentication on their own server, if I remember correctly.

I'd say that as a platform holder, you should avoid conflicts of interest like that. The good thing about having your own sports-line is making sure that the competition works harder for your platform - witness the NBA games that both went for 1080p @ 60fps in their first year. But your ultimate goal should still be to have better sports games on your platform, not promote your own sportsgame at the cost of another, and certainly not by abusing your position as a platform holder.

Was any kind of lawsuit considered, or was the company too worried about damaging their relationship with the platform holder and that you'd not have been able to achieve results until it was too late anyway? Certainly right now I don't think Sony would get away with that kind of thing anymore, being in a much more vulnerable position in the market. Guess that also explaines why they are less forthcoming in other areas ... :S

It also just goes to show that you always need at least two competiting platforms, preferably three. So long live Nintendo, Microsoft and Sony! :)

The question really is interesting but it seems more like competition than unfair practices.

I mean Baseball is like a super huge sport in Japan ever since it was introduced by US marines or Navy men back in the ancient days and Sony, Sega and many other Japanese game publishers make or have made in the past many a baseball game over there.

Also isn't the 3d engine in Sony's Baseball game developed in Japan for their game and later localized or converted, not really reprogramed as its just changing the roster and baseball stadiums for the US release?

I personally am not a fan of sports at all and I think that the MLB2k series is pretty good according to some of my friends tell me, I am sure the other game has its nice points but I think its great that there is a choice as it keeps dev teams on its toes.

I was curious though why was a patch needed?
 
So for me it was an easy decision, make less than 50k with decent benefits seeing the family occasionally, dealing with the stress of the team struggling to get the PS3 sku out which caused the other two skus to be launched in a non-opportune month, to making over 50k, 3 weeks vacation, 6 holidays, 2 weeks of sick pay, and working from 0830 to 1700 Monday through Friday, the tragedy is that I'm almost at a loss with what to do with all this free time!

I'm not quite convinced on the role of the PS3 in this scenario however?
 
In the 'Junior Programmer' debate, isn't the idea of modern development to separate code into low and high level code, where you have your 'I know what I'm doing' experts creating the engines at the low level and the junior bulk programmers using the high-level tools produced?

It's harder to separate runtime graphics work into junior/senior tasks since pretty much everyone needs to know exactly what they are doing if we're gonna hit our optimization goals. I think typically if there will be juniors doing graphics work, it will likely be tool side stuff.


Arwin said:
Was any kind of lawsuit considered,

The issue escalated pretty fast internally, but I got shut out once the suits stepped in so I don't know how far it went.


aselto said:
The last game looked really good (at least on 360), but it was 30fps IIRC. Did you sacrifice in terms of geometry/effects/shinyness a lot to get to 60fps this time around?

Thanks! Nope, nothing sacrificed. Just tons and tons of optimizing :) PS3 version this time around is visually almost identical to the 360 version as well.
 
We've hit 60fps (2xmsaa at the moment) on our own on PS3. I guess that makes us only mildly lazy incompetent devs ;) Sony has already screwed us in the past by not letting us patch our baseball game on PS3 during the critical baseball season months while they went ahead and conveniently patched theirs. So no, we're not expecting any help from Sony any time soon.

:LOL: C-O-N-G-R-A-T-U-L-A-T-I-O-N-S ( <--- Imagine this in blinking neon lights).

To me, Booth's "PS3 misconception" article is as accurate/generalized as the "lazy developers" comment.

As for Sony help... I think with software and network getting more important even for embedded devices, I wouldn't be surprise in the future, their support unit is beefed up (may have already, I have no clue).

EDIT:
Finally, just want to let you know that I might get the baseball game too.

I suppose you can add "I sold 2 games, one to jobabbey and one to patsu" on your resume.
 
Last edited by a moderator:
But you don't need to! It's not about writing assembly code or doing crazy optimizations.
What is so complicated about writing simple C and C++ code?

You missed my point, so maybe I should explain better. It's not about the actual SPU code that is difficult. I'd even say writing SPU code is a pleasure, even if I have to stay away from fancy templates and classes and just stick to good old C with intrinsics. It's still fun and the SPU has a very nice orthogonal instruction set with unified register file that, well, makes things a breeze. The whole issue of SPU programming is how you get your data to and from your SPU code. Setting that up is by no means trivial and it's very error prone. Basically Sony's libraries (ie SPURS etc) are VERY low level and not suitable for just anyone to start messing around with DMAs and syncronization.
As someone else asked, why is this not abstracted, why is there no low level/high level separation? Well, good question, but currently there are NO high level libraries for SPU of any kind. And when I say high level, I mean something where I can code up some little "jobs" with carefully abstracted input/output semantics (think shaders) and I can run these abstractions even on X360 and I wouldn't even know the difference.
This high level abstraction doesn't exist right now, and someone has to write it and that's going to take some time. Until then I'd not let any junior guy touch SPUs, thank you very much.
 
I think PS3 being hard to code for is good thing in the long run because that means there's going to be progress, later generation PS3 games are bound to be more impressive.

Kind of like how early 2d psone games went on to impressive 3d and how ps2 games went on from early scratchy Tekken to later jaw dropping titles like Transformers. Or I could talk a while about how nes (mario bros to super mario 3) and snes games (early titles to donkey kong country) progressed. I find that kind of progress appealing, me.

I didn't particularly like how the original Xbox was almost immediately spawning games that were pretty much as impressive as they were going to get.
 
Thanks! Nope, nothing sacrificed. Just tons and tons of optimizing :) PS3 version this time around is visually almost identical to the 360 version as well.
Wow, that's very impressive, great effort from you team! Much room left from future improvements or do you think you pretty much hit the ceiling already?
 
The whole issue of SPU programming is how you get your data to and from your SPU code. Setting that up is by no means trivial and it's very error prone.
What about spending a couple of days writing and debugging a stupidly simple DMA class to abstract DMA transfers? Transfering datas via DMA is not really rocket science, no one (except the usual suspects I suppose..) directly poke MFC registers anyway.
Basically Sony's libraries (ie SPURS etc) are VERY low level and not suitable for just anyone to start messing around with DMAs and syncronization.
How do you solve the syncronization probem on 360? Does the SDK write good multithreaded code for you?
As someone else asked, why is this not abstracted, why is there no low level/high level separation?
Well, good question, but currently there are NO high level libraries for SPU of any kind.
Unless they removed stuff from the SDK you might want to double check this..
I mean something where I can code up some little "jobs" with carefully abstracted input/output semantics (think shaders) and I can run these abstractions even on X360 and I wouldn't even know the difference.
In the company I work for a couple of smart chaps spent a couple of weeks writing and debugging a multiplaform (win32,360,ps3) multiproc enviroment/abstraction layer, which is now used even by not experience programmers.
Do you want Sony to write one for you that runs on 360 as well?
Writing a simple abstraction layer that let exposes some minimum features set (kick a job, basic sync primitves) is not really that hard, I wish all tasks were as simple as this one.
What's complicated is to make your junior programmer do any real GOOD work on it, and you more or less face the same hurdles on any platform, it's not like coding on 360 on win32 automatically solves all your parallel programming issues, does it? :)
This high level abstraction doesn't exist right now, and someone has to write it and that's going to take some time. Until then I'd not let any junior guy touch SPUs, thank you very much.
Maybe you'd better start programming it then ;)
 
I think PS3 being hard to code for is good thing in the long run because that means there's going to be progress, later generation PS3 games are bound to be more impressive.

Kind of like how early 2d psone games went on to impressive 3d and how ps2 games went on from early scratchy Tekken to later jaw dropping titles like Transformers. Or I could talk a while about how nes (mario bros to super mario 3) and snes games (early titles to donkey kong country) progressed. I find that kind of progress appealing, me.

I didn't particularly like how the original Xbox was almost immediately spawning games that were pretty much as impressive as they were going to get.

I remember a similar discussion on here not that long ago and while i respect your opinion, i find it hard to understand. It's a bit like saying you'd rather read a novel that starts out crap and gets better and better until it's a masterpiece at the end. Sure, the improvement is more visible, but wouldn't you rather read a masterpiece from start to finish? To me, when i get to the end of a console generation, i'd rather look back on my collection and see all the games that still stand up to a certain level of scrutiny than look back at the earliest games and marvel at how poor they were compared to the latest ones. Anyway, it's all preference i suppose.
 
I remember a similar discussion on here not that long ago and while i respect your opinion, i find it hard to understand. It's a bit like saying you'd rather read a novel that starts out crap and gets better and better until it's a masterpiece at the end. Sure, the improvement is more visible, but wouldn't you rather read a masterpiece from start to finish? To me, when i get to the end of a console generation, i'd rather look back on my collection and see all the games that still stand up to a certain level of scrutiny than look back at the earliest games and marvel at how poor they were compared to the latest ones. Anyway, it's all preference i suppose.

I think it's just "wow, this novel is crap, but maybe it will turn into a masterpiece later" to tell the truth. I think everyone, on some level, is disappointed with much of the ps3 library but holding out in hopes it will get better. Don't get me wrong, I love my ps3 and many of the games, but I think this is the origin of this type of thinking.
 
It's more like holding out for more developers to understand and design for Cell. The PS3 SDK seems to throw a few roadblocks of its own (Not feature rich enough, or may be even buggy in some areas), but should already include everything developers need to get started.
 
Maybe you'd better start programming it then ;)

Trust me I do wish I had the time to write all these nice things. And it's great that your team had few nice chaps to sit down and design that system, and lucky for you you had 3+ years exclusive platform development.
The important thing about X360 and I'll repeat that again is that "It Just Works". From the get go I get decent performance, CPU and GPU. Only and I really mean, ONLY if I need to optimize something for the needs of my game I then can spend time refactoring the system to take advantage of multiple cores. And that takes very little time because of the nature of unified memory address space. [edit - and by that I mean that the multiple cores all see the same memory, can read and write to it within the same address space, the only "problem" being you need to protect(syncronize) any shared resources]
On the contrary, on PS3, you need top engineers devoted just to get basic stuff off the ground, and you NEED to use SPUs to get decent performance out of RSX.
And let me go back to the topic of SPU programming since I forgot to mention few aspects of it. The SPU has local address space so any pointers in your data have to be translated, DMAed etc, which is PITA. Structures need to be refactored and often "flattened" which often defeats clever memory sharing/optimizations. The memory is limited so you have to partition the problem. The code might not even fit initially, and you might not be able to run a debug version of the code. Until recently one couldn't even do breakpoints inside SPU jobs, and now you have to find the SPU code in memory before it gets uploaded and set a break point there, very similar to VU programming back in the day. Forget about asserts working, and printf might or might not be reliable. A "HALT" on the SPU could mean anything, especially if it's inside a middleware SPU job. It could be their bug, or yours, someone stomping the code or data, or it could be stack overflow. You get basically zero tolerance for error. That is not what I call "It Just Works".
I mean come on, Microsoft of all companies are known to be sloppy, and their APIs are not perfect and they don't get everything right from the get go. Sony with all their open source/Linux fundamentals, how can they screw it up so bad that they have worse offering than MS?
 
Last edited by a moderator:
The disparity in opinion appears to be polarized with Sony-exclusive developers on one side and multiplatform developers on the other. True, there are exceptions, but the early concerns about the skill required to effectively manage Cell and the split memory configuration in a "next generation title" within reasonable budgets do appear to be manifest in some studios.

All the name calling aside, these hurdles have been faced by a lot of good developers. Nevermind engine developers like Epic and id who have a substantial stake in getting their engines to run well on the PS3. How would people feel if GTAIV was delayed because the PS3 version was lagging behind the 360 version, something that has happened to a fair number of studios? Rockstar may be full of lazy hacks, but their games are extremely popular and critically praised. But if a studio like Rockstar is having problems I think it brings to point a very valid longterm question: At what point do the games become so large and complex and the systems so powerful that the focus of system design needs to shift away from peak performance and towards a design that is easy for developers of all walks to "effortlessly" tap. Are we barking up the wrong tree with the persuit of peak power over the power to create and execute design goals? I think the next consoles really need to address this question... but I digress.

It is great that a number of studios had the know how, people, time and resources to hit the ground running with the PS3. But it does appear it takes extra work to get the PS3 version to perform as well as the 360 version (as Carmack said long ago) and that the SPEs do limit who can, and cannot, be writing code (as Newell said long ago).

Time and People are finite resources.

I can appreciate that the PS3 is easier to work with than the PS2. But the PS3 isn't competing with the PS2. Further, to remain competitive the PS3 is having to play on the same field for developers who didn't bother with the PS2. A lot of this seems relative. If the Xbox 360 didn't give similar results with less effort there would be no baseline to complain about. This is compounded by the fact the 360 was released first (and was the platform most started cutting their teeth on for next gen software, so everything is relative to this) and, much more notably, that the 360's install base and attach rate are very high in the NA/Europe while Sony has floundered some in their first year relative to the current market penetration of the competition.

Developers were "spoiled" with the 360 and the PS3, while easier than the PS2, some have been quite frustrated when trying to migrate their projects over because it isn't easier than the 360. Developers on the PS3/360 aren't PS2-complex games (for the most part!), so comparisons ring hallow to most multiplatform developers because the PS2 is no longer the baseline. If the PS3 had the market position that PS2 had this would be different. You can get away with a lot when you are the market leader and hold a firm grip on the industry.

In some ways this is all unfortunate because, lost in such, is the fact is Sony did address some issues with the PS3 design. They went with an over the counter GPU, leveraged a great existing API in an OpenGL varient, and aquired some excellent tools in Cg. Cell is a step in the right direction and the basic concept appears to be the future of CPUs (at least in some quarters) and Sony has improved their tools and support. Their work with the PSN is also notable and good for the industry as it offers a true alternative to the Live model of business.

But games are bigger, more complex, and have more hands on them--and more at stake. More time, more money. An average title can sink a development studios; a couple bad "big titles" could do longterm harm to a publisher. If Sony were first to market and had captured sizable early marketshare I think some of these issues would have been minimized to a degree. As has been noted, going PS3=>360 is a lot easier than vice versa. The flexiblity and memory situation on the 360, part of what makes it friendly to developers, allows for some designs and practices that don't perform as well on the PS3.

Of course from the Sony camp we hear, "You are doing it wrong, THAT is your problem" whereas in the Microsoft/Multiplatform camp we hear, "This works fine on the PC and 360, so YOUR design is the problem."

And the majority of the industry could very well be wrong... but the reality is these are the people making games. If the industry was levels of degree from, "We will adapt to you" all the way to "You will adapt to us" it seems in broad strokes MS and Sony are straddling opposite sides of the fence.

The situation will get better, over time, as more studios adopt a PS3-lead-SKU approach and more 'good practices' prevail, experience grows, and libraries & tools become available. But the PS3 is on the clock. It needs software sales momentum to encourage this shift and it has limited time for such. At some point the competition will hit critical mass and the new consoles will be gaining steam. With a lot of the more notable titles taking 3 years (and more) to develop a title begun today may appear in the last holiday before the new consoles arrive.

It is nice to say everything is there for developers to "get started" but from a business perspective if the competition gives you a head start and a faster development window (more time either means a more polished project or making money by moving on to something new) you are going to hear grumbling. If your titles get delayed (and this IS happening) because the PS3, for whatever reason, is taking more time to get up to the same performance as the other 2 platforms, you are going to hear grumbling. We can blame developers.

But in the same breath we can ask why didn't Sony do more to account for these same developers when designing the machine. Obviously Sony did (and balanced such with their corporate vision and the technology they had available) but it shows how the constant blaming of developers really doesn't address the realities some of these companies are facing. Ok, they got their data structures wrong and seem unhappy about investing the time to re-structure their game to get equal performance on both systems. But at the end of the day this is what they have been doing for years and it works on the market leader. So as much as developers need to change some of their practices, Sony is in the inevitable position to helping developers find solutions NOW before their titles slip. And that there doesn't breed much goodwill among some of these smaller, tighter budgeted multiplatform studios. They would much prefer to spend the time making the product better instead of just getting it to work.
 
IMO lets put this in prospective lets say there was no x360 what would happen then. It just seems that everyone got spoiled with what Microsoft offered and did not want to change(not a programmer just my experience with being a long time gamer), no one want to design a ps3 specific game. It seems like they want ps3 to EMULATE everything the x360 does w/o any performance hit.
 

I completely agree with what you say, but there is something I have never really understood about some of the multi platform developers. It seems that they have no intent at all to optimize their game for the PS3 (This could be a time constraint?), and instead just throw code from the 360 version of their game at it and sell it at is. Wouldn't it be cheaper over all to create one optimised engine that works on both platforms (EPIC, IW and ID have done so) and just modify the engine from their on?, instead of just making it 'just work'.

I presume the problem with this is that developers don't have unlimited time and money?.
 
The disparity in opinion appears to be polarized with Sony-exclusive developers on one side and multiplatform developers on the other.

--Clip!

Great post, though it doesn't really excuse Booth for the demonstrably false claims he has made. The telling part is that sure, 360 code runs like shit on the PS3, but the solution is as simple as starting on the PS3 and porting to the 360. Moving from LS to hardware cache is easier than the other way. Moving from NUMA to a single memory pool is easier than the other way (and moving from the smaller available memory to the larger is the same). Moving from vertex limited GPU to one that isn't is easier than the other way round. I guess if you start with Blu-ray going down to DVD9 might be a challenge, but not on the same scale as needing to start over with portions of your rendering engine. All that seems pretty obvious to me. Multi-platform developers really ought to be conscientious of these considerations much earlier in their development cycle. That way they can avoid coming to the end and running into roadblocks.

I mean, I can see why the 360 is the lead sku so often. It came out first. It's simpler to work with and faster to produce results, but if your goal is to create a multi-platform title for simultaneous release there are decisions that can be made up front to save you grief down the road.
 
I think you answered your own question. The 360 wasn't only out first, but dev kits were more mature and readily available long before Sony deployed widely available dev kits, let alone mature ones. So a title begun in late 2004 or early 2005 may have spent over a year being developed with the 360 in view, so the option of co-development and addressing these issues wasn't quite transparent. Fast forwarding to today, if you were working on a west-oriented title that had a target date of Fall 2007 this issue is further complicated by the events of the last year (namely market penetration and software attach rate) that the prospect of shifting lead SKU status would cause some consternation, and after an extended crunch and product delays that could result in some bitterness toward such prospects.
 
Sony is probably doing this on purpose.They could easily with little effort ,match the 10 more years experience put in the MS toolchain integration : .net + visual studio + dirextx

...But they just want to piss off those who don't know anything but that.
 
Status
Not open for further replies.
Back
Top