*Game Development Issues*

[Arwin]

He may not have intended any of this with his blog, but thats partly how I read it. Help us out, don't treat us like a cog in the machine. Give us a balanced machine, and support us. I like the direction 360 took the industry. I do not like the PS3 pulling us back. It doesn't need to be this way. Worse yet, I can't help but wonder how long it can continue on this way. As the people playing games have aged, so have the people creating them. People want to see their kids and families and will be less and less tolerant of sacrificing them to meet milestone 3b.

I found one job opening for you at a company you might like:

We are based in beautiful downtown Burbank, CA where the sun shines just about every day of the year. We attribute our continued success and stability to a philosophy based on five key principles, covering everyone from the CEO to our QA Testers: quality over quantity; innovation; collaboration; efficiency; and independence. Today, the company has grown to more than 150 hard-working, creative individuals, but we still hold onto the feel of a small indie developer, and the kitchen to prove it.

And when it comes to treating our employees right- it’s no mistake that Insomniac Games has been voted in the “Top 10” Best Small Companies to Work for THREE YEARS IN A ROW!!!! We are the only game company that has been honored in this way! Our employees gave their feedback, and their input on the company is what took us the distance! We value our employees, as we are only as strong as the people who work with us! So how about becoming a part of this amazing, collaborative team? Sound like a plan? Check us out at www.insomniacgames.com, we look forward to hearing from you.

You can guess where I'm getting at. If Insomniac can be in the top 10 list of best medium (or small?) companies to work at in the US, would you say that is because they exclusively work on the PS3, or because they have a superior business strategy? While they're not denying that they don't go into crunch mode themselves once in a while, even during those times they make life at their company very comfortable for their employees.

Seriously though, the games industry, to me, sounds like one where a lot of bad business practices rule and with so many people really eager to work there, a lot of executives take advantage of this and can get away with very bad management. And once you've got bad management, and you've got your people working at 120% (or 150-200% in some cases) on a structural basis, then you're not going to be able to take the steps back you need and invest that extra 30% to make sure you create a more efficient environment. It's a classic management error.

Now I'm not going to say that this means the 360 doesn't have better tools than the PS3, and I agree that Microsoft brings something important to the table right there. But I am going to say that if you work in a properly structured manner, that's going to matter much less, and in the console world, where hardware is supposed to last longer, and therefore invited to be used more efficiently, this eventually should result in better stuff. I think it's good that both platforms exist, and that we will see the benefit of the former first, and the latter later. Though for exclusive developers like Insomniac and Naughty Dog, as well as clever people like the Infinity Ward teams, those benefits are imho already becoming very apparent.

 

[nAo]

I think when people are saying PS3 is hard to code for, it's when compared to 360. When things go so smoothly on that platform, they can spend lots of time making the game more complex and go after their vision without worrying about making code machine friendly.

Do you believe you can write code on 360 not thinking about the underlying architecture and get away with it? You can't, it will simply run like crap.

If there was no 360, then there wouldn't be anything to complain about since PS3 would be the only game in town. Perhaps part of the problem is that Sony oversold the powers of PS3, so people just assumed that if 360 could run it then PS3 could blow through it.

Let say you're right and Sony oversold the power of PS3 (like MS hasn't done that..), the ppl complaining about PS3 lack of power wouldn't know that to do with it anyway.
What they are really complaining about is: my code doesn't run as fast as on PS3 as it does on 360, I don't understand why, and I don't give a damn.
Now this is perfectly fine, no problems with that..but the guy who ranted on his blog was complaining about technical issues he knows nothing about.
If this was a thread about Developer X writing that he's losing money developing on PS3 because as a games platform it requires additional effort that is not needed on 360 then fine, I wouldn't have wasted any time writing an answer on his blog.
Wait..I wasted my time anyway

Another reason for complaints could be that porting is just seen as a pain in the ass. For PS2->XBox, MS minimized that pain, so people didn't complain much. Now that MS has taken over as the lead platform, 360->PS3 seems a lot harder.

Yep, it's a lot harder, cause many messed up in the beginning not designing their code to run decently on PS3, which would have made their code running smoothly (or decently at least) on 360 as well..

 

[Barbarian]

Well that's kinda moot if Sony follow the path a lot of us are expecting them to take and use a Cell variant in PS4. Then everything that's been a struggle this gen will be a doddle next gen. Worst case PS4 has the same memory architecture etc. as PS3 which everyone'll be used to, and better case things are improved while the algorithms and design philosophies will carry over exactly, with developers entering the cycle running full pelt.

Going by precedent it's very likely Sony will NOT make PS4 similar in any way with previous generations of hardware. Even more likely is that they will ditch all their tools and start from scratch. This has happened before, PS1 -> PS2, PS2 -> PSP, PS2 -> PS3. Nothing in common. Maybe some distant abstract ideas like, synergistic processors that are PITA to program and debug for. Compare that with PC->Xbox, Xbox->Xbox360, DirectX, Win32 OS/API etc, decades of experience and knowledge.

More on topic though, I think people, when discussing multi core designs, fail to realize that the reason X360 is easier to program for is the shared memory address space and hardware cache capabilities.
That gets the job done fast and easy. With some performance tuning and cache prefetching you can squeeze more performance if needed.
On PS3 on the other hand, there is no easy version. It's all hard to the metal from day 1. A single mistake can set you back days and weeks. The API being error prone, and the debugger having trouble identifying what the problem of a crash is, doesn't help either.

 

[Barbarian]

Do you believe you can write code on 360 not thinking about the underlying architecture and get away with it? You can't, it will simply run like crap.

Actually most of the junior programmers on my team probably do not know the underlying architecture and the X360 still runs like a champion. Even if it craps out, with MS tools it will take 10 seconds to determine what's going on.

But let me ask you the opposite question, do you honestly believe I should let junior programmers code their stuff to run on SPUs? I certainly wouldn't want to be on that project and to have to debug the horrible crashes that will happen with 100% certainty.

See that's the problem that is so nicely emphasized in Booth's article. Not everyone is a super programmer with 10+ years experience. And not everyone needs to know the detailed cycle count of their target processor. And not everyone can tell the difference between bandwidth and fillrate and shader cycles and raster ops. The game industry is getting more and more junior people, straight from college, and they are expected to deliver code. At the end of the day if you're a sound guy for example you just want to hear your sound play and you definitely don't want to spend your weekend debugging SPU sound code.

 

[nAo]

But let me ask you the opposite question, do you honestly believe I should let junior programmers code their stuff to run on SPUs?

Why not? it's not like they need to write assembly code. What is so scary about writing SPU code?

I certainly wouldn't want to be on that project and to have to debug the horrible crashes that will happen with 100% certainty.

What horrible crashes?

See that's the problem that is so nicely emphasized in Booth's article. Not everyone is a super programmer with 10+ years experience. And not everyone needs to know the detailed cycle count of their target processor.

But you don't need to! It's not about writing assembly code or doing crazy optimizations.
What is so complicated about writing simple C and C++ code?

And not everyone can tell the difference between bandwidth and fillrate and shader cycles and raster ops.

That's why maybe some ppl should keep these technical aspects out of their remarks.
I don't write on my blog rants about the current state of neurosurgery

 

[Arwin]

Actually most of the junior programmers on my team probably do not know the underlying architecture and the X360 still runs like a champion. Even if it craps out, with MS tools it will take 10 seconds to determine what's going on.

But let me ask you the opposite question, do you honestly believe I should let junior programmers code their stuff to run on SPUs? I certainly wouldn't want to be on that project and to have to debug the horrible crashes that will happen with 100% certainty.

So what's your junior/senior ratio? And how well versed are your senior coders in SPU coding?

See that's the problem that is so nicely emphasized in Booth's article. Not everyone is a super programmer with 10+ years experience. And not everyone needs to know the detailed cycle count of their target processor. And not everyone can tell the difference between bandwidth and fillrate and shader cycles and raster ops.

So either you need to make sure that the important bits that involve those four are written by someone who does? Certainly those four sound like rather basic things that you should, in fact, be able to tell the difference between if your code touches that directly even remotely. Or is investing one day of training into a junior programmer already too much?

The game industry is getting more and more junior people, straight from college, and they are expected to deliver code. At the end of the day if you're a sound guy for example you just want to hear your sound play and you definitely don't want to spend your weekend debugging SPU sound code.

So are there no sound libraries whatsoever offered by Sony? Or are they just that buggy? I'm really and honestly curious! I can imagine it being tricky, but surely you can still code for SPUs in C and have better performance than by not using the SPU at all?

Would you be willing to share with us what your team looks like, in terms of basic structure, junior senior bandwidth, on the job training requirements, experience and technology background? Does your project use Unreal Engine or other middleware, or are you guys coding everything from scratch? Maybe transferring existing engines from other platforms?

I do think that the PS4 will in fact be different this time. This particular transition was big, but imho with fast streaming performance and fighting heat issues, Cell is the way to go and here to stay. Sure, perhaps in the future you can get away with more abstraction layers, but looking at where supercomputers are headed and GPUs have been heading for some time, the future seems clear. I have no doubt that this time Sony will consider tools even more than previously. Certainly I should think that already with Cell being documented and SDK supported by IBM should make a considerable difference in that regard?

If you think I'm out of line with these questions then please excuse me. I'm a coder, but not in this industry, but it still fascinates me to no end.

 

[betan]

What they are really complaining about is: my code doesn't run as fast as on PS3 as it does on 360, I don't understand why, and I don't give a damn.

Seriously, that summarizes the impression I get from "them" perfectly.

 

[patsu]

It's fine for Booth to rant. But I don't think he is worried/concerned about the well being and welfare of game development folks. He simply attempted to blog about "PS3 misconception and spin" (His blog title) by introducing his own nuances -- for whatever personal reasons. I am with scificube here (although more mellow) and don't want to credit Booth for what he has not done... or rather what he has accidentally started.

The other article rekator linked to (post #3 I think) is more true to the current discussion theme. I hope Sony responds fast enough to your needs. joker454, it might be a fruitful (marketing) exercise to arrange for a technical interview with the boards when your game is launched. It sounds like you guys have gone through sh*t to bring us the game. I am not a sports video game fan, but let's hear more about your product when it's ready.

 

[SedentaryJourney]

And not everyone can tell the difference between bandwidth and fillrate and shader cycles and raster ops. The game industry is getting more and more junior people, straight from college, and they are expected to deliver code.

And these are people who write shaders for a living?

Ironically the title of the jbooth post("PS3 misconceptions and spin") seems quite apt at referring to the content of the post. It also seems irresponsible on the part of knowledgable persons to lend credence to his arguments simply because ideologically they happen to agree that the Xbox360 is better for coders. I don't see the PS3 suffering from the great deficiency jbooth does, but I suppose to the chagrin of customers who were expecting a visual leap over the 360, the PS3 has yet to deliver. Having come out a year later, it's reasonable to expect you'd see the same performance delta between PS3 and 360 as you did between Xbox and PS2.

 

[scificube]

Booth did not outline that Sony should provide better tools, libraries or direct support in any of his remarks. Booth certainly did not make any mention of doing so and being rejected any assistance.

If Booth wished to motivate Sony to treat him or in general developers better then at the very least he would have mentioned these things in his comments.

It's fine to complain but you have to leave room for someone to actually help you.

I fail to see how Booth is at all indicative of junior programmers given he's supposed to have years or experience and further isn't a programmer in the first place. I am certain and glad that is not the case.

 

[joker454]

I hope Sony responds fast enough to your needs. joker454, it might be a fruitful (marketing) exercise to arrange for a technical interview with the boards when your game is launched. It sounds like you guys have gone through sh*t to bring us the game.

We've hit 60fps (2xmsaa at the moment) on our own on PS3. I guess that makes us only mildly lazy incompetent devs Sony has already screwed us in the past by not letting us patch our baseball game on PS3 during the critical baseball season months while they went ahead and conveniently patched theirs. So no, we're not expecting any help from Sony any time soon.

 

[Akumajou]

Sure, totally fair on both points. I'm actually looking at this less on technical accuracy and more as a general problem with the industry. Yeah he did make some errors. But, theres a general problem brewing in the industry that he possibly inadvertently points out, where difficult beat my skull against the wall consoles with limited support are creating issues.

Maybe this is just in the USA, but I'm noticing many trends here. The obvious are previously mentioned things like skyrocketing budgets. Less obvious are constantly missed deadlines and their consequences. Not noticed by many is whats happening internally at many studios. I'm seeing more and more juniors, less and less veterans. I'm seeing people who've been coding games for 10 years burn out and exit the industry totally. I'm hearing grumblings of people tired of having to sleep at work or not see their spouse for weeks on end just to meet a milestone. I'm seeing people devote their lives to ship a game on both platforms by crunching 12-16 hours days for weeks on end, only to all get laid off right after the project ships.

To a certain extent, this has always happened in the industry. But it now seems to be getting more frequent. People exiting the business is becoming as frequent as people entering it. Expectations are becoming unrealistic. Quality of life is severely down. In some ways his blog to me was a way to say that overly complex platforms are not welcome anymore, when it's clear that things can be done in a simpler manner with a similar result. 360 was a step in the right direction to me not because it was done by Microsoft, but because it helped the industry from a quality of life perspective. For perhaps the first time ever, the developper got great tools and support from day one. PS3 to me is trying to pull things back in the other direction where the developer is basically cattle and expected to sacrifice himself to the mighty mother ship. That's just antiquated to me now.

I recall an event that happened in the last crunch where we were trying solve an issue on PS3 and one of the coders I was with told me he had missed his daughters first words because he was crunching. Sounds silly and mellow dramatic, but that kind of thing just gets one thinking. At the time, which was actually just before my now ill fated charalatan post, it got me thinking why in the hell should we be struggling like this to get basics working when this was all so easy on 360? Why do we tolerate this? Well, from the amount of people exiting the industry, it would appear that less and less are.

He may not have intended any of this with his blog, but thats partly how I read it. Help us out, don't treat us like a cog in the machine. Give us a balanced machine, and support us. I like the direction 360 took the industry. I do not like the PS3 pulling us back. It doesn't need to be this way. Worse yet, I can't help but wonder how long it can continue on this way. As the people playing games have aged, so have the people creating them. People want to see their kids and families and will be less and less tolerant of sacrificing them to meet milestone 3b.

Wow I used to hear alot of stories like that from former EA employees a couple of years ago.

I would like to say though that as far as jobs in the USA treating its workforce like cattle (mandatory overtime, lack of job security, working 12 to 16 hour shifts, etc) that trend has become more and more popular since around 2000/2001 timeframe AFAIR in many different technical field jobs, at least for now US game devs don't have to worry about their jobs being outsourced to Canada or India or any other country where the jobs can be outsourced to.

I would like to point out though that as far as decreasing profits from games being made even though some may dispute it, console modding and game piracy has become extremely rampant to the point where I have over heard conversations in a videogame store of one guy telling the other, "don't buy it, just rent it and rip it"

Basically with PS1, PS2, Dreamcast, XBox generations mod chips and downloading or just ripping retail games was and still is a very popular practice. I guess in the best interest of a game dev they would hope people are not renting their title or making copies.

 

[wco81]

So we're talking about MLB2K? Or the only other baseball game would be The Bigs.

Sounds like you guys have made great strides in improving performance.

 

[joker454]

So we're talking about MLB2K? Or the only other baseball game would be The Bigs.

Sounds like you guys have made great strides in improving performance.

Yeah it's MLB2K. I don't mind saying that now since it's my last year with the team anyways, I'll be moving on after we ship. This years PS3 version is miles better than last years as far as performance goes. Oh just fyi, work conditions are pretty good where I work ever since they started paying overtime. That's forced them to schedule better. It's the other shops that I keep hearing about that make me nervous since we're the exception, not the rule alas ;( Some of the horror stories I hear are mind boggling.

at least for now US game devs don't have to worry about their jobs being outsourced to Canada or India or any other country where the jobs can be outsourced to.

Our crappy dollar is helping with that. Europe is way too expensive to outsource to, and Canada is becoming too expensive as well as their dollar keeps rising.

 

[one]

Easy to program is cool in itself, but doesn't it make the job more outsourcing-friendly and a harder competition ground?

 

[-tkf-]

Sony has already screwed us in the past by not letting us patch our baseball game on PS3 during the critical baseball season months while they went ahead and conveniently patched theirs. So no, we're not expecting any help from Sony any time soon.

Stuff like that just sucks and you guys should make that public so that Sony has to answer to gamespot, ign, eurogamer etc etc and of course to their own close allies (ninja!).

 

[_phil_]

Stuff like that just sucks and you guys should make that public so that Sony has to answer to gamespot, ign, eurogamer etc etc and of course to their own close allies (ninja!).

If you whant that kind of stuff to go mainstream ,it'll be very dirty....even for those with a kiddy public image...

 

[Shifty Geezer]

In the 'Junior Programmer' debate, isn't the idea of modern development to separate code into low and high level code, where you have your 'I know what I'm doing' experts creating the engines at the low level and the junior bulk programmers using the high-level tools produced? From the sounds of it, this separation, at least in some places, isn't very effective and all developers of all levels are needing to get their hands dirty.

 

[aselto]

Yeah it's MLB2K.

The last game looked really good (at least on 360), but it was 30fps IIRC. Did you sacrifice in terms of geometry/effects/shinyness a lot to get to 60fps this time around? I was fairly impressed with how EA managed to step from 30fps with their NHL series, I wonder how you'll do with the transition.

 

[Arwin]

We've hit 60fps (2xmsaa at the moment) on our own on PS3. I guess that makes us only mildly lazy incompetent devs Sony has already screwed us in the past by not letting us patch our baseball game on PS3 during the critical baseball season months while they went ahead and conveniently patched theirs. So no, we're not expecting any help from Sony any time soon.

Now that right there, is very ugly. It reminds me of back in the day when EA and Microsoft had this big row over Live support and how the way it was set-up then would give Microsoft heaps of interesting inside information on the use of online play in EA games that they could then go on and use in their own competing sports games line. In the end, EA and Microsoft settled with Microsoft giving up on their own sports games line, and allowing EA to run everything but the actual user authentication on their own server, if I remember correctly.

I'd say that as a platform holder, you should avoid conflicts of interest like that. The good thing about having your own sports-line is making sure that the competition works harder for your platform - witness the NBA games that both went for 1080p @ 60fps in their first year. But your ultimate goal should still be to have better sports games on your platform, not promote your own sportsgame at the cost of another, and certainly not by abusing your position as a platform holder.

Was any kind of lawsuit considered, or was the company too worried about damaging their relationship with the platform holder and that you'd not have been able to achieve results until it was too late anyway? Certainly right now I don't think Sony would get away with that kind of thing anymore, being in a much more vulnerable position in the market. Guess that also explaines why they are less forthcoming in other areas ... :S

It also just goes to show that you always need at least two competiting platforms, preferably three. So long live Nintendo, Microsoft and Sony!