Importance of assembler knowledge for console devs (and its education) *spawn

[Gubbi]

I suppose I'd have to ask, what's your definition of "software development in general?" I'm certain it is useful for engineers at Apple working on iOS devices, at Microsoft working on Office, at Google working on Search, at Oracle working on their DB engine, etc., etc.

General software development, as in the 99% of code written, where the output asm is never looked at.

I bet developers at Apple developing the systen software stack scrutinize the code/asm. But all the application programmers that build on that ? Hell no.

Same with Office. How much of Office is performance sensitive ? And how much is GUI, auxiliary tools, wizards, function etc.

And same again with Oracle. Yes, they have a core DBMS which is super optimized, but in combination with that they have a *massive* Java eco system.

Do Java developers look at asm? Do C#, PHP or Flex developers? Do most C++ developers?

No.

If you need to develop optimized core software, yes understanding asm is a useful skill. But for most it is unnecessary.

Cheers

 

[jonabbey]

The trade off for programming at a higher level is that you can write more functional units for the same amount of effort. Having a good (and common-sensical) knowledge of data structures and algorithms is the important thing at those levels.

 

[Fafalada]

Because I thought assembly programmers for in-order and OOO-cores are equally hard to find and they probably are.

That's true - but the main point he was making is that codebase already 'optimized' for in-order will still benefit from OOOe. My post was a bit unclear, but basically I was getting at that most large-codebases are actually full of general purpose code that is far from optimized (for anything) increasing that benefit.

If your profiling graphs are flat, good for you, you've probably picked the low hanging fruits and optimised the critical loops, sounds pretty normal to me.

Yup - but when things scale (again, with large teams and codebases) you can run out of critical-sections to optimize and still run a very slow codebase overall.

To me understanding the implication of assembly level code and cpu architecture goes hand in hand.

They should, but surprisingly you can learn to write asm without understanding many computing architecture principles. What asm programming can do is give you a more practical understanding of those fundamentals.

If you need to develop optimized core software, yes understanding asm is a useful skill. But for most it is unnecessary.

Other posts have pointed this out already - but I find it more important for debugging - there's precious few people I'd even trust to write asm-level-optimizations, but I expect all non-juniors to be able to do some basic reading of crash-dumps in optimized builds.

 

[ERP]

The notion that any optimization problem in-games is solved by throwing assembler and mad-scientists at it is frankly speaking more then a little silly - even if we DID have armies of those type of guys out there.

Any sufficiently large codebase* will inevitably lead to mostly unoptimized code in production critical paths (usually coupled with largely flat profiling graphs because most of it will be equally slow).


*grown through "standard" production deadlines.

This is indeed my experience.
When asked to optimize a 10 year old codebase written by 50+ programmers the first thing you do it run the profiler and cry.

It's almost always death by a 1000 cuts, the old optimize the hot spots logic doesn't apply because the only place anybody gave any consideration to performance was the hot spots, and it's now the other 99% of the code that's killing you.

OOO processors with decent cache behavior are just so much more efficient running the sort of code that is actually written, their additional transistor budget is easily justified in overall application performance.

 

[Rodéric]

It's almost always death by a 1000 cuts, the old optimize the hot spots logic doesn't apply because the only place anybody gave any consideration to performance was the hot spots, and it's now the other 99% of the code that's killing you.

so true...
I've been fighting this battle a few times never winning, it's so much easier to write the first thing that come to mind, most people don't want to be bothered with good algo/memory access patterns

I can count on a single hand the people I worked with who cared, and of course the others will just say "premature optimisation..." without the least clue about the context of that sentence

 

[sebbbi]

Game technology programmers generally do not need to write that much assembly code. However reading compiler generated assembly code (when doing performance oriented debugging) is a very crucial skill, especially when optimizing for in-order architectures (and when using intrinsics). Compiler intrinsics have pretty much replaced the requirement for directly writing assembly, as the compiler usually does a better job in scheduling instructions and allocating registers. Intrinsics offer programmer a slightly higher level view to the low level CPU instructions than directly writing assembly code. There's still 1:1 mapping to CPU instructions, but the extra effort of managing registers and scheduling (L1 access times, pipeline latency, etc) are left for the compiler (that does it's job very well in most cases).

I agree that compilers, libraries and middlewares should implement the low level code behind the scenes, and make things easier for the general programmers. However, someone needs to program those low level libraries, middlewares and compilers. And that's why we need programmers with low level programming skills. Game development companies tend to need more low level programmers, because we are coding much closer to the metal, and we spend much more time in optimizing our existing code. Console hardware stays the same for 5-10 years, but the expectations of customers (players) increase all the time. Every last bit of performance needs to be squished out of the hardware, and every dirty trick needs to be used, no matter how low we must go

 

[patsu]

Other posts have pointed this out already - but I find it more important for debugging - there's precious few people I'd even trust to write asm-level-optimizations, but I expect all non-juniors to be able to do some basic reading of crash-dumps in optimized builds.

Game technology programmers generally do not need to write that much assembly code. However reading compiler generated assembly code (when doing performance oriented debugging) is a very crucial skill

[Nod violently in agreement, but not just for game developers]

It's also helpful for certain class of platform specific bugs (problems on some architectures but not on others, when you're doing [low level] cross platform development).

 

[Dominik D]

It's not uncommon to look at the disassembly when debugging optimized builds, otherwise it can be very difficult to understand what's actually going on. If you can't at least read and understand small amounts of compiler emitted asm, then your usefulness as an engineer is frankly limited.

That's why you should learn to read assembly first and foremost. Learning to write has little to do with learning to read. I've seen a lot of code that was illegible, even by its creator(s). Don't write if you don't know how to read, that should be the golden rule.

so true...
I've been fighting this battle a few times never winning, it's so much easier to write the first thing that come to mind, most people don't want to be bothered with good algo/memory access patterns

I can count on a single hand the people I worked with who cared, and of course the others will just say "premature optimisation..." without the least clue about the context of that sentence

There was this comic strip I can't find where dude's given a lecture on internal combustion engines and then asked to drive a truck (or lorry for you British chaps ;P). That's basically how education system works today. All around the world. I've been writing code since I was seven or eight and I'm still just OK at it. Employers however expect to hire grads and see something magical being created. I was lucky to work for a large corp for 3 years with some amazing bunch of people who taught me pretty much everything I know today. There are very few places however where new folks are being taught how to write and at the end of the day you get a lot of poor code that kinda works but wasn't well thought-through. It's the game to be blamed, not the player though. It's the environment we're in that's at fault.

 

[hoho]

There was this comic strip I can't find where dude's given a lecture on internal combustion engines and then asked to drive a truck (or lorry for you British chaps ;P). That's basically how education system works today. All around the world.

http://abstrusegoose.com/418

Though I agree about the state of programming education. It's pretty much non-existent. When I was in uni from 2003-2006 I took everything even remotely related to programming, some stuff even from the masters program. With the exception of the most advanced course there was in the whole uni that talked about parallel programming everything else was basically just writing "Hello World" in three different languages with various useless bells and whistles. Yes, there was a course on algorithms and data structures as well but stack and quick sort were the height of that and not even once did they mention anything about how efficient CPUs are at certain kinds of computations.

 

[KKRT]

they now use to teach programming in schools/universities instead of good old C/ASM are not making things better.

Sebbbi what books, lectures, online sites etc materials would recommend with that 'knowledge'? Its hard for me to find good C book/learning materials for low level C, even harder basic knowledge how CPU works exactly and how to code efficiently for it, even for old architectures.
Same for GPUs if You have any links/infos

 

[MJP]

so true...
I've been fighting this battle a few times never winning, it's so much easier to write the first thing that come to mind, most people don't want to be bothered with good algo/memory access patterns

I can count on a single hand the people I worked with who cared, and of course the others will just say "premature optimisation..." without the least clue about the context of that sentence

Well it's easy to get up on a pedestal about these things, but to be fair most game developers are frequently put in positions where it's not really realistic or practical to write optimal code for every feature they implement. Obviously some people just don't even come close to understanding performance and are never going to write something in a truely optimal manner, but even good programmers will write some bad code if the feature is needed ASAP. Personally I always try my best to implement something in a way that's at least decent with regards to performance, but a lot of times I will just write something that works so that I can begin the important step of iterating with the artists to make sure that the feature meets their needs. Or we need something for a milestone, and I get it working by any means necessary. That's just the way it is sometimes, and no matter how much I understand CPU or GPU architecture I'm still not going to write everything in a way that's completely performance-optimal. Honestly I would say that better scheduling/planning combined with better communication between developers and publishers would go a longer way towards better performance than everyone learning how to be a hardcore assembly programmer.

That said, I do agree that an overall apathy towards performance can be really frustrating to deal with. I care very deeply about performance, and it's always weird to me when other people don't feel the same way.

 

[Rodéric]

There's no alternative to good algorithms and memory access patterns.

I once worked with people who kept saying "premature optimisation...", when I was pushing for good code (architecture & algorithms). We spent a month and a half optimising all hot-spots, because according to them it's all you need for high performance, once it was done the game ran at about 30fps... (33-35ms/frame)

I also saw some code taking a pointer as parameter, there was a high level interface, but every function (about 6 calling each other) were checking for null. null wasn't even a legal value on the top level. A reference would have fixed all that, instead of polluting the branching table, had null been allowed, the public function checking for it would have been enough.


Back on topic, reading ASM is a good thing, being familiar with the ISA of the CPU you are using is too.
You're unlikely to write ASM, instrinsics are just way betterin all respects. (easier to read and write)

 

[sebbbi]

I once worked with people who kept saying "premature optimisation...", when I was pushing for good code (architecture & algorithms). We spent a month and a half optimising all hot-spots, because according to them it's all you need for high performance, once it was done the game ran at about 30fps... (33-35ms/frame)

Yeah. Designing good interfaces and good high level code structure should never be considered premature optimization. In an ideal world, the interfaces should be ready before the real coding even starts. Code refactoring should be done immediately after the team notices that the high level structure and interfaces have performance or scalability problems (or will likely have performance or scalability problems with final feature set or data set). Refactoring of interfaces and higher level program structure should never be threated as premature optimization. The later you change your interfaces, the more code you have to change.

If the code has good structure, good set of algorithms and good interfaces, all designed for solid performance in mind, the hot-spot optimizations will be pretty much the only optimizations needed later on. But in the real world you never have perfect technical design (and even game play requirements tend to change during projects), so aggressive core structure/interface/algorithm refactoring during the project is always required in order to hit your performance targets. I completely rewrote our shader and state management system after Trials HD project, because the old system was designed for simple indoor scenes, and our draw call count in the new game (free roaming outdoor scenarios) was going to be several times higher. Of course we used time and resources trying to optimize the old code first, but as the old system couldn't even render unpolished levels (with partial background decorations) at 60 fps, I redesigned the whole system before too much time was spend to optimize the old system. Also in Trials HD project (our first Xbox 360 game) I refactored our lighting system two times before I was happy with the result (light index deferred rendering + stencil shadows --> deferred rendering + geometry lights + shadow mapping -> tiled deferred rendering + ESM shadow mapping). It's important that the need for big changes like these are identified as early as possible, and all the required changes are done as soon as possible. Writing new methods right away in low level assembly/intrinsics is of course premature and can make code maintenance harder, but higher level structure/interface refactoring should never be considered premature.