Carmack again on PS3-360

[archangelmorph]

That's his entire point, in his opinion multi-core programming will not be 'cracked' for a long time. Many people have been working on it, for many years, and it's still coming along slowly, he sees no major breakthroughs for quite some time.

You mis-understanding my points.. Multi-core programming has already been "cracked" in a sense as we can already see the performance benefits of using it.. My point is not that there's some all elusive "supreme algorithm for utilising multi-core architectures at 100% efficiency".. That games programmers are working towards finding.. We all know that there isn't one!

My point is that for someone whose making the transition from serial process code to multi-threaded code, to multi-threaded code for specialised parallel archetectures, the software algorithms and data structures for building such architectures are well-known by the industry as a whole and have been for some time (i.e. super-computers, multi-servers etc..) it's a fixed-period, one-time investment:

- Understand multi-threading theory
- Design a functional game engine based on this (now known) theory
- Build the game engine
- Build a game
- Reap rewards + spend the rest of your time creating new games based on the current engine, investigating how to make your engine more efficient or design a new, more efficient engine for the future..

Looking at the above, the only process which differs from previous game engine development is the first one, and after the development of the first game, you'll never have to do it again..

Just his opinion, but I'd like to know what experience you have that you can sit there and contradict his statements.

Contradict? I'm merely challenging them..!
Or is he so holy and revered that i'm not allowed to do that NEmore..?

If i'm wrong then show me.. Otherwise my point is that I really don't see his arguements as cause for complaint, which he clearly likes to do on this subject..

 

[ERP]

Perhaps you could point me at some research thats going in the right direction.
It's not like this is a new problem.

 

[archangelmorph]

Yeah, people have researched ways to program shared-memory multiprocessors since the 60's, and it's still a fundamentally hard problem, especially if you really want to squeeze maximum performance, which of course game programmers want to do.

All he's saying is that ideally the clock speeds should continue to increase exponentially, but since that is not possible in the real world, he has to adapt, and just like anyone else, he doesn't like it. Every programmer in the world would prefer a single infinitely fast CPU vs an infinite number of slow CPU:s.

But how about an infinite number of fast CPUs?

At the end of the day one of the major advantages with multi-core is that hardware can truely make software scalable (to a degree but the gains can still be tremendous)..

If a developer has an application written so that it litterally dispatches threads to any available core for processing and then reassembles threads dynamically, running the s/w on a 3 core CPU would run great.. But running it on a 50 core CPu would run incredibly better with little to no modification needed..

Yeah you could say the same if you ran a serial app on a 2GHz CPU and then on a 20GHz one true..

But since the state of the art at the moment dictates that were not going to be getting 20GHz chips any time soon, the quicker we slug it out and get used to building scalable, multi-core s/w, the easier it will be later when our software will run fine no matter the hardware configuration (it could even run across multiple computers through some next-gen network interface..)

Yeah this doesn't apply to closed-hardware consoles since the PS3 will never get anymore SPEs..

But you'd have to be silly not to expect the PS4 to have a Cell with around 100-1000 of them..

At least by this point you'd be better equipped after working with multi-core for so long rather than the alternative..

 

[Crossbar]

This is the fundamental misconception, it's certainly a learning process, but there is no such thing as "cracking it" it's a fundamentally hard problem.

Tools and experience can make it easier, but it will always be significantly more difficult.

Multiprocessor architectures are inevitible, using them effectively is a hard problem, and it's one that compounds all of the other hard problems in game development.

Couldn´t you argue that the EE of the PS2 is conceptually a multi-core design similar to the Cell?

If you do that you could argue it represents a well known problem with several proven solutions for developers coming from the PS2 world, some of that experience can probably be transfered to Cell Edit: and Xenon.

Coming from the PC world it may involve some labour pains before you get into it. Maybe this is what Cormack is experiencing right now?

 

[assen]

Oh sorry!

I didn't kno I needed to be a member of the elite of the industries software engineering specialists with a vast library of games/technical masterpieces under my belt before I could comment on a couple of JC's statements which relate to an entirely seperate topic altogether..?

You would be well advised to avoid statements similar to "How dare that whiner complain about multicore programming, people have been doing it since the 1960s". Example of much better sounding statements: "For our game, I paralellized algorithm XXX as described in paper YYY and saw ZZZ percent improvement". His opinion is more valuabe than yours, not because he is some kind of rocket-fuel-breathing, Ferrari-driving, terrain-megatexturing demigod who is always right, but because he actually tried multicore programming, and you most likely only practice multicore trolling on forums. Please, prove me wrong and make me humbly apologize in public

 

[archangelmorph]

You would be well advised to avoid statements similar to "How dare that whiner complain about multicore programming, people have been doing it since the 1960s". Example of much better sounding statements: "For our game, I paralellized algorithm XXX as described in paper YYY and saw ZZZ percent improvement". His opinion is more valuabe than yours, not because he is some kind of rocket-fuel-breathing, Ferrari-driving, terrain-megatexturing demigod who is always right, but because he actually tried multicore programming, and you most likely only practice multicore trolling on forums. Please, prove me wrong and make me humbly apologize in public

Well I hope you do because I've been programming on PC for most of my life and have spent a good degree of time working with multi-threaded coding both when I studied Software Engineering in uni (practical and theory), in my spare time doing hobbyist code and also just started at my new job in the games industry..

It's not the God-aweful-demon-monster-&-root-of-a-coders-nightmares that alot of people around here make it out to be and many non-games industry coders have been working with multi-threading for years building OSs and the like (and i'm pretty sure they'd agree with me that multi-core is the future, AND it's not worth crying over)..

I never went so far as to imply via my posts anything along the lines of:

"How dare that whiner complain about multicore programming, people have been doing it since the 1960s"

And if that's what you seriously believe then i'd suggest re-reading my threads much more carefully and then come back and tell me I don't have cause to open my mouth..

Seriously why must people be so childish about the fact that I'm only challenging (intellectually) an arguement made by someone who clearly isn't so far experienced in this particular field, regardless of his past accomplishments?

If you don't like what i'm saying then why don't you argue against my statements rather than try to undermine my credibility..?

 

[ERP]

OK I'll try a different track on this to try and explain how hard it really is. Carmack has it easy because he drives game design from technology, it's a luxury a lot of us do not have. He also has a small very experienced team of programmers.

I've written a lot of multithreaded software over the years and I even enjoy it (I rarely have any real challenge to solve), but that isn't really the point.

On my current project there are 80 engineers, they've spent several years generating a million plus lines of code. We spend on average 20% of a milestone stabalising the code base because of the high rate of change (both to requirements and refactoring of older un shippable implementations) and the long test cycle. We will spend between 3 and 6 months in our Beta cycle. And this is on a primarilly single threaded codebase.

I don't think most experienced game programmers could even agree on a definition of "game engine". The bulk of any games code is in the gameplay code anyway and I don't see how you build any core that can isolate the gameplay programmers, and the level authors from the parallelism of the system (and I have given it some significant thought). We have enough problems with level designers inserting half second pauses to solve race conditions in primarilly sequential systems.

To me the larger issue is the additional complexity in getting something shipped.

 

[archangelmorph]

I don't think most experienced game programmers could even agree on a definition of "game engine". The bulk of any games code is in the gameplay code anyway and I don't see how you build any core that can isolate the gameplay programmers, and the level authors from the parallelism of the system (and I have given it some significant thought).

That depends on how you look at it..
Sure from the common conventions of how we define the role of a "gameplay" programmer and the kind of code the individual is expected to develop it would be a very difficult task indeed..

But a change in hardware architecture requires a change in software mindset, and with alot of the physically driven gameplay experiences we are seeing more of today, the possibilities are there for an engine which truely separates software concerns on to hardware processing nodes..

The alternative is to abstract the gameplay programming as far away from the hardware as is required to allow code to be developed as though it were a serial app.. OSs have been doing this for yrs and i'm sure the majority of next-gen games we are seeing at the moment probably utilise the same principles..

In the end both are engine design paradigms which could both be valid (i.e. function on the multi-core hardware to some degree)..

The real work would be optimising either for the hardware, a non-trivial task made even harder if you take the first route (i.e. build the s/w to separate game core components (e.g. physics, AI, Sound etc..) entirely and synchronise communication between them efficiently..)

Either way it makes the job of the Technical director difficult when it comes to designing the system.. But once the core system is down.. You just optimise and improve, the same way it's been in the past..

 

[infinity4]

why should we trust someone who cant even make a game properly??? quake 4 was a turd on 360

 

[ERP]

At some level parrallelism impacts the code you can write, you can't make gameplay code look serial unless it actually is.

Even if you use a scripting language to isolate from the underlying hardware, the parallelism has to creep in when say you can't immediately cast a ray, you have to request a ray cast and respond when the result is computed.

The same occurs when two entities are dependant.

These aren't terribly hard problems to solve in isolation, but what you've introduced are syncronisation primitives and with them all of the bugs inherent in parallel programming.

In the short term with the exception of the interface to physics and animation this largely doesn't matter, since most games gameplay code is just not that CPU hungry outside of physics and animation.

 

[Inane_Dork]

Maybe. But didn't we all think the same back when we made our first recursive routines, or when we first started proper modular programming, using classes and headers, and so on?

Multithreaded programming is not a new field at all. And it's not new to Carmack, either.

Maybe, but the idea of 'cracking it' isn't so far fetched. You just need to change your mindset. As some wise man said some time ago, data-centric thinking is key here.

I'll go out on a limb here and say that most programmers never "get" multithreading well enough to design or debug it. Those who do understand it still make lots of errors. It's a fundamentally difficult thing to make correctly, and there's no best way to design for multithreading, either.

You constantly imply that it's easy, but it's not.

why should we trust someone who cant even make a game properly??? quake 4 was a turd on 360

He didn't make that game.

 

[archangelmorph]

I'll go out on a limb here and say that most programmers never "get" multithreading well enough to design or debug it. Those who do understand it still make lots of errors. It's a fundamentally difficult thing to make correctly, and there's no best way to design for multithreading, either.

Programming is fundamentally difficult but we manage that well enough..

Saying multi-threading is hard and it will never get any easier is rather silly to be honwest because if you approach the solving of the problem like you would any other you'll eventually find "a" solution..

Granted it may not be the best but with refinement and time it will get better and better and will eventually become the optimal solution for the given problem..

Now since the term "multi-threading" is so general that there probably isn't one "optimal" solution but many depending on what it is your trying to multi-thread, I'd say it makes things easier since the hard part is only getting "a" solution to "work" in the first place.. (which obviously isn't "that" hard because every commecial application which utilises multi-threading hasn't failed so far..)

 

[Sethamin]

Programming for multi-core systems is more than just the usual hardware advance. If you look at the history of CPUs, most advancements from the 70s on have all been contained at the hardware or compiler level - e.g. OOO execution, control and data pre-fetching, superscalar processors, dynamic branch prediction, loop unrolling, predicates, etc. There were a few places that it would intrude into the code level - e.g. SIMD instructions - but even that could sometimes be optimized by the compiler. Although it helps for a developer to understand these things, strictly speaking they didn't have to. The most optimizations you really had to concern yourself with were making sure your code avoided cache misses and page faults whenever possible, since memory latencies were so high.

But that era is over. We've reached the limits of extracting thread-level parallelism and bumping clock speeds up. No longer can we just modify the microarchitecture, ISA, or compiler to give you "automatic" speed improvements as past advances had. It's a fundamental paradigm shift where the code itself has to be dramatically re-architected to take advantage of it. This is a big deal, it's not just another hardware advance to be dealt with. I think you're underestimating what a radical shift this is in the software industry (not just the games industry).

 

[almighty]

Ive never really liked carmack, mainly because of his constant complaining. I know he's a respected develoepr and im sure many of the member's in this very forum have a high respect for his skill's but i think he just STFU and get on with it. Sure PS3 might be a pain in the arse and have tool's that are not up to 360's standard but sitting and moaning about them is'nt going to make them any better.

^ IMO

 

[popper]

Ive never really liked carmack, mainly because of his constant complaining. I know he's a respected develoepr and im sure many of the member's in this very forum have a high respect for his skill's but i think he just STFU and get on with it. Sure PS3 might be a pain in the arse and have tool's that are not up to 360's standard but sitting and moaning about them is'nt going to make them any better.

^ IMO

it sounds like youre trying to say that devs here and elswere would be far better served if they worked together to produce patches and improvements to PPC GCC etc to produce FAR BETTER PPC/Altivec and SPU code at a generic level, and DO IT PDQ.

sounds like a plan to me, as it would be good to get faster PPC linux code instead of always being behind the x86 code generation.

it seems looking at the gcc threads that theres currently only a very few PPC devs that are trying to push through much needed improvements and they should be thanked for that effort by every single person that benefits/will benefit from that effort...
go and help them and help yourself in the process, so what did you patch today....

 

[Gubbi]

it sounds like youre trying to say that devs here and elswere would be far better served if they worked together to produce patches and improvements to PPC GCC etc to produce FAR BETTER PPC/Altivec and SPU code at a generic level, and DO IT PDQ.

sounds like a plan to me, as it would be good to get faster PPC linux code instead of always being behind the x86 code generation.

I agree. It's common knowledge that game industry programmers are a lazy overpaid bunch. They have all the time in the world, no such things as crunch time or deadlines or anything ("when does the game ship?", "WHEN IT'S DONE!!! ").

So they should just take a break from their 6-hour-a-day foosball activities and spend some time on open source community efforts, so that their equally lazy competitors might benefit.

Cheers

 

[popper]

I agree. It's common knowledge that game industry programmers are a lazy overpaid bunch. They have all the time in the world, no such things as crunch time or deadlines or anything ("when does the game ship?", "WHEN IT'S DONE!!! ").

So they should just take a break from their 6-hour-a-day foosball activities and spend all their time on open source community efforts, so that their equally lazy competitors might benefit.

Cheers

ROFL, you mean they only get to do 6 hours, thats bad..., call the union, they should be getting 12 hours at least and subsidised pay for community coding for FUN.

 

[Cheezdoodles]

why should we trust someone who cant even make a game properly??? quake 4 was a turd on 360

why should we read what someone people post, when they cant even figure out who made a game?

Quake 4 was ported by Raven Software.

 

[Inane_Dork]

Programming is fundamentally difficult but we manage that well enough..

Let me revise my statement so that you see why what you said is irrelevant. Multithreading is fundamentally hard for programmers. Programming is not hard for programmers.

Saying multi-threading is hard and it will never get any easier is rather silly to be honwest because if you approach the solving of the problem like you would any other you'll eventually find "a" solution..

Well, I don't recall saying or implying it'll never get easier. I do recall implying that it won't become as well understood a problem as, say, recursion.

Secondly, I don't know that if you approach multithreading like any other problem, you will eventually find a solution. But I'll grant it. That doesn't make multithreading easier, though. Some things are difficult no matter how many times they've been done.

Granted it may not be the best but with refinement and time it will get better and better and will eventually become the optimal solution for the given problem..

That optimism is not really warranted. You could just as easily be refining an inherently poor (but workable) solution as an inherently good one. You would likely not even be aware of the good solution. Without having tried your best at literally every option, there's no empirical reason to claim an optimal solution.

 

[-tkf-]

The thing is that if JC actually was forced to develop for the PS3 and write his engines exclusively for the Cell and RSX it would very likely endup being very special

But i think that it´s just not possible anymore for one man to write everything himself and games have just gotten to big and complex. So tools are what makes a difference for him at this point. A shame really