Sweeney presentation on programming

[DudeMiester]

Well I suppose you would have to either have a condition sepecification capable of overlapping or provide an explicit case for when those two optimisations overlap. I can only assume that existing compilers have some mechanism for applying optimisations that can handle overlaps. Figure out a way to expose that. Maybe something like parsing the code into a data structure that the plugins can analyse and transform. I'm pretty sure you could borrow techniques from dynamic languages to do this.

Also, my talk about MVC and all that wasn't my main point. I was trying to stress that DSLs are just abstractions. I mean take GLSL and Sh (from the university of waterloo). Clearly, C++ is general enough to allow you to integrate GLSL directly into C++ code. It seems apparent to me that GLSL is only there for intuitiveness and ease of use, but not for any functional reason. I think that you will find the same applies to all DSLs, and of course it should, since they are to C++ like C++ is to ASM.

It just seems to me that people get so wrapped up in these DSLs and other things, labelling them as some panacea, when in fact they are nothing more then tools to be employed when and where it is appropriate. I mean, if you compare DSLs to ASM, they much in common. ASM code varies from platform to platform and DSL code varies from context to context. Neither have the uniformity and generality of languages like C++, and that's why none of them will replace it. The only thing I think you will see is the addition of much better contextual/meta/compile-time data/functions (e.g. better type info) to C++ with the additional optimisations to match, and indeed this is the case.

 

[Jawed]

http://channel9.msdn.com/Showpost.aspx?postid=143582

The Concurrency and Coordination Runtime (CCR) is a lightweight port-based concurrency library for C# 2.0 developed by George Chrysanthakopoulos in the Advanced Strategies group at Microsoft. Here, we have a deep discussion about CCR with George, a Software Architect, and Satnam Singh, Architect.

Jawed

 

[Xmas]

The biggest problem with the many domain specific languages model that has been suggested is the problem of communication within teams and with outsiders. Effectively, for every particular programmer and project you encounter you would have to learn a new language, because everyone inherintly thinks differently and thinking varies per project. This would be competely impractical in the real world, and I imagine it would lead to far more spagetti code, errors and other problems then we have today.

Consider DSLs being libraries. For every library you use, you need to learn the interface. The difference is that with a DSL you write the code "using that library" in the DSL, and a good DSL should be more expressive and easier to understand, which makes up for any new syntax you have to learn.

Trouble begins when you start to overuse DSLs and create one for every small problem. But that's the same with too specific libraries, really.

 

[Frank]

So, does anyone have a good model of building objects cache/local store friendly with current procedural languages? And of course in a way that prevent locking conflicts?

I think the best solution would be a custom database engine that uses transactions, but you would probably want a different (read: more direct) interface than SQL. While it does add an extra layer and quite some overhead, you can use one or more of the other cores to run it, so it takes it out of the main thread and off the main core. And it gives you the flexibility to fiddle with the way the data is stored during development for the optimal result, without breaking stuf.

So, you add overhead, but you're able to speed it up by adding more hardware that is unused for now. I think it's actually faster, while eliminating the problems at the same time.

 

[ERP]

So, does anyone have a good model of building objects cache/local store friendly with current procedural languages? And of course in a way that prevent locking conflicts?

I think the best solution would be a custom database engine that uses transactions, but you would probably want a different (read: more direct) interface than SQL. While it does add an extra layer and quite some overhead, you can use one or more of the other cores to run it, so it takes it out of the main thread and off the main core. And it gives you the flexibility to fiddle with the way the data is stored during development for the optimal result, without breaking stuf.

So, you add overhead, but you're able to speed it up by adding more hardware that is unused for now. I think it's actually faster, while eliminating the problems at the same time.

I have a model that I think will work for the basic levels of parallelism were talking about to day and the granularity is controlled by the design.

But there is nothing automatic about it.

You simply stop thinking of objects as blocks of memory, you consider them to be collections of functionality and group the data by function rather than by object, you then update at a functionality level in batches and have those updates explicitly call out dependencies.

We actually started using the system for reasons unrelated to parallelism, primarilly authoring and cache coherency issues and we have the option on a functional block level of grouping the data by object or by functional block. It adds a level of overhead to accessing data across functional blocks, but it's only about the same as a virtual function call.

 

[Frank]

I have a model that I think will work for the basic levels of parallelism were talking about to day and the granularity is controlled by the design.

But there is nothing automatic about it.

You simply stop thinking of objects as blocks of memory, you consider them to be collections of functionality and group the data by function rather than by object, you then update at a functionality level in batches and have those updates explicitly call out dependencies.

We actually started using the system for reasons unrelated to parallelism, primarilly authoring and cache coherency issues and we have the option on a functional block level of grouping the data by object or by functional block. It adds a level of overhead to accessing data across functional blocks, but it's only about the same as a virtual function call.

Yes, I was thinking about that as well. Store the same properties in global arrays, as you expect lookups to be interested in specific properties over lots of objects. But as you said, you still have to fiddle with the granularity when you do that, and it is still a problem because the properties are accessed through the individual member functions who access a single property at a time, they don't use blocks. While that might work for caches, it won't work for local storage.

So you need to make classes that handle all the objects of that kind, instead of having the individual objects do it themselves. And if you do that, why not go the whole way and use a database engine instead?


Edit: isn't your model very alike the one I proposed?

 

[DemoCoder]

See Google Map/Reduce. This model, while not completely a Google invention, scales to extremely large number of processors (in Google's case, we're talking about clusters containing up to 100,000 CPUs)

 

[Fafalada]

Old fart input - we used to do the same in FORTRAN - declare one huge COMMON block, and then handle memory from there.

Well in Forth there was little choice if you wanted fancier stuff. But for what Pana was talking abouit it's more a case of default allocators being exceptionally shitty on some platforms.
But more then that, lack of usefull debug info hurts - like Tim mentioned in his PPT, memory related bugs are dime a dozen in this work, it pays a lot to have any helping facilities to manage that.

It just seems to me that people get so wrapped up in these DSLs and other things, labelling them as some panacea, when in fact they are nothing more then tools to be employed when and where it is appropriate. I mean, if you compare DSLs to ASM, they much in common. ASM code varies from platform to platform and DSL code varies from context to context. Neither have the uniformity and generality of languages like C++, and that's why none of them will replace it.

No reason for them to be a replacement though. Personally what I like about DSLs is that they are effectively extending compiler's domain knowledge, which opens room for much more interesting optimization then what you have by default in normal procedural language.
It's not unlike the idea behind configurable libraries in C++ through metacode, generate optimized solutions for your problems, rather then trying to make a general optimizer.

 

[ShootMyMonkey]

You simply stop thinking of objects as blocks of memory, you consider them to be collections of functionality and group the data by function rather than by object, you then update at a functionality level in batches and have those updates explicitly call out dependencies.

Sounds interesting, though something about it seems vaguely troubling when getting into debugging. It kind of addresses some cross-cutting concerns in that you can tie multiple data types to the same functionality, but I doubt the compiled code would look quite as agreeable. Sounds almost like a further step down the line of aspect-oriented constructs.

I'm kind of in the same boat as DudeMiester in that I do think procedural/imperative languages are the only feasible choices if you want something general enough. But then, I largely think so for hardly unbiased reasons as someone who was basically raised on assembler. My thing isn't so much a matter of not liking abstractions as much as what is potentially lost by abstracting things away altogether. It's easy to say that we can extend a language until it is this sort of monolithic pile that supposedly pleases everybody. But there's no point in ever really doing that, because you'll just make a gigantic highly context-sensitive mess.

I would rather just have something that beats down the path that something like C/C++ offers where various abstractions are available to you, but so is the opposite extreme. Not to say that C++ is perfect or that the perfect language will be a child of C, but the general direction is in that area.

 

[DemoCoder]

I was a child of assembly (learned to program assembly on Commodore VIC-20), and C/C++, but I learned through 1.5 decades of experience that I was spending about 80% of my time tracking down the same kinds of bugs over and over, in both my code, and other, and I was performing the same kinds of optimizations over and over, and frankly, I didn't need to be doing either.

The resason why many programmers in the elder years eventually start fantasizing about creating their own language is from learning this lesson, getting sick of it, and hoping there's a better way. The fact is, a surprising large amount of code can be boiled down to a few functional paradigms (map, filter, fold, etc) and performance doesn't necessarily suffer because you write MAP instead of FOR with a loop iteration variable.

Higher level languages frees programmers from the drudgery of chasing down common errors committed in assembly and C/C++ letting them spend more of their time thinking at the level of problem solving, instead of at the level of tracking down segfaults. And the idea that LISP/Scheme, ML, and Java style languages can't perform well is a myth. Sure, they might perform slower than raw C code, but many people would be willing to sacrifice that performance for enhanced productivity, and deal with mechanisms later to overcome the performance gap (reverting some performance critical sections to native C or assembly)

If you look at the programming languages shootout benchmark, you'll see that OCaml and SBCL (Common LISP) perform quite acceptably.

What Sweeney is trying to imagine is a modification to C/C++ garbage collection, new types and functions for which non-mutability and loop bounds can be verified statically. I think he's trying to think of a way to have an OCAML like imperative language with a functional subset, but one that uses C-style syntax instead of funky ML syntax.

 

[ShootMyMonkey]

Well, for me, ASM was never a matter of performance except when dealing with compilers that were known to produce crap under certain conditions or when compilers simply can't produce the code at all (e.g. vectorization). As I said, my main concern with higher level languages is not the lossess due to the extra baggage they might carry for things like garbage collection, but losses in what you the programmer can do.

I can look at something like Haskell, and there are so many things you can do with it that are just beautifully simple. But I would rather not have to lose the ability to work on certain levels explicitly in order to gain that simplicity. Garbage collection is nice and all, but I don't want to have to lose control of all memory management in order to get it. Fundamentally, that's not so different from what Sweeney is talking about, just that the source of motivation might have been different.

I guess in my case, I probably also reinvented the wheel a thousand times these past 20 years. But more than anything else, what I got out of it is a complete distrust for tools to actually behave like they're supposed to. Much as I'd like to say I could dream of a language or a compiler that will fix everything, I can only truly say that it's little short of a delusion that such a thing can ever actually exist.

 

[AlgebraicRing]

I guess in my case, I probably also reinvented the wheel a thousand times these past 20 years. But more than anything else, what I got out of it is a complete distrust for tools to actually behave like they're supposed to. Much as I'd like to say I could dream of a language or a compiler that will fix everything, I can only truly say that it's little short of a delusion that such a thing can ever actually exist.

Just out of curiosity, what language do you program in most and what is your development environment? What tools do you use to develop with?

 

[DudeMiester]

Thinking about this, I see the merit of lazy-evaluation based languages, but within limits. The whole point of abstraction is to let the compiler/library do the work for you, leaving you provide only the highest level description. However, this must be balanced by the ability to dig deeper when needed for performance or missing functionality. I have the same opnion of GUIs. I don't want to configure anything or fiddle with settings, except when I do want to. In other words, provide me the highest level representation to start, but never forget the "advanced" button. In this regard, I think these more vauge/abstract languages are useful, but only to an extent.

It's also worth noting that in order to have that "advanced" button, you need provide lower-level underpinnings. The easiest way is to built off of what's already there, so I think any higher level language will necessarily be built off of C++, just as it is a superset of C.

Although, I think eventually the compiler will be reduced to nothing but a VM that executes code like a script. This compile-time "script" in turn generates an executable by pulling from libraries that provide functionality, optimisation and machine code translation. You would pull elements from the functional libraries, compose and manipulate them to create new elements, run that through your optimisation library and finally translate the results into machine code using the library for your desired platform. These elements could be anything, such as a runtime variable, a compile time variable or a function, and through their context and total self-description the compiler can fully understand them. Indeed, in this system these elements are everthing, and the act of their logical composition, the act of programming, which obviously is the way it should be.

Of course, to do this, contextual/type/characteristic functionality and reflection (maybe via a DOM sort of thing) would have to be greatly improved from the standards of C++. For example, an optimisation library would need to be able to take a block of code, study the characteristics of the elements involved and alter the code/elements appropriatly. I'm sure this can be done, because current compilers already explicitly represent that kind of understanding, and since it can be explicitly represented, it can be expressed in a progamming language. If you can have enough context to optimise, then surely you have enough for your other language needs.

I would also like to be able to explicitly invoke this compiler and execute the results from within the program. This would allow, for example, JIT compilation to be directly coded by using smart function pointers. You could also use (or abuse) the compiler as a general scripting engine/VM itself, and combined with reflection, use it as a dynamic language. Basically, you get all the advantages of higher level languages, but are still able to code down to the bare metal if that's what you really want.

Maybe there's a language that can do this already, I'm not sure, but even if there was I seriously doubt it can produce machine code or do anything significantly low-level. Still, I think languages are moving towards this model anyways, even if it's a damn slow pace.

 

[AlgebraicRing]

Thinking about this, I see the merit of lazy-evaluation based languages, but within limits. The whole point of abstraction is to let the compiler/library do the work for you, leaving you provide only the highest level description. However, this must be balanced by the ability to dig deeper when needed for performance or missing functionality. I have the same opnion of GUIs. I don't want to configure anything or fiddle with settings, except when I do want to. In other words, provide me the highest level representation to start, but never forget the "advanced" button. In this regard, I think these more vauge/abstract languages are useful, but only to an extent.

Assuming your high-level abstraction layer is complete, any optimisation for speed is something you could and should do below the abstraction layer. What I would want to do is write an extension to the compiler which would allow me to write the necessary optimisations for the abstraction layer. If your abstraction layer is complete, I can see no reason to ever break the abstraction. If you are breaking your abstraction by peeking/tweaking below the hood, your abstraction layer is incomplete. A faster implementation is an orthogonal concern and should be written in a separate scope. Ideally I would want to elegently say: "Here's the definition of my abstraction layer" and "Here's my recommended implementation of the abstraction" Hell, maybe a good IDE would even compile your abstraction layere into executable code which would then be editable if you wanted to do laser precision optimisations. Even in this case, however, you would never need to break your abstraction layer; that always remains pure.

I'm liking what I'm reading from http://www.onboard.jetbrains.com/is1/articles/04/10/lop/ Sergey Dmitriev and I'm anxious to sink my teeth into MPS. Just need a couple more weeks before I have time to experiment around with it.


Maybe I'm wrong. If you can think of a practical use-case where breaking the abstraction layer is very very very helpful if not outright necessary, then please share! I'd love to re-evaluate my disposition.

 

[Xmas]

Hell, maybe a good IDE would even compile your abstraction layere into executable code which would then be editable if you wanted to do laser precision optimisations. Even in this case, however, you would never need to break your abstraction layer; that always remains pure.

But you would lose the ability to edit your abstraction without overwriting the lower level changes. If you want to avoid this, you need to properly integrate a "gate" to lower levels into the higher level language, very much like inline assembly.

I'm liking what I'm reading from http://www.onboard.jetbrains.com/is1/articles/04/10/lop/ Sergey Dmitriev and I'm anxious to sink my teeth into MPS. Just need a couple more weeks before I have time to experiment around with it.

I like the concept, too, and it promises to integrate well with ideas I had going in other directions. But in its current state MPS is still very raw.

Btw, why is this thread in "Console Technology"?

 

[DemoCoder]

The problem with inline assembly is that it can fubar up dataflow and register allocation analysis in the compiler, and therefore, slowdown code elsewhere.

I found that using compiler instrinics when possible ends up with much superior output from VS.NET.

Thus, I think when one has a "gate" to escape a domain, that the domain has adequate knowledge of what's being done outside the gate.

 

[Frank]

While high level functional languages are nice, they miss the immediate problem we face at the moment: having to learn to think differently.

Sure, everyone would want a compiler/library that would just recompile their current programs into multiple concurrent threads, automatically. But that isn't going to work. And while a functional language enables you to put the work with the designers, that won't work either, as long as nobody has come up with a good general method to do it.

I think that is mostly wishful thinking at the moment. And as it has to run on a processor that uses a procedural model in the first place, we will have to learn to think the right way and do it ourselves in our procedural language of choice. After that works and people understand how to do it, we can abstract it.

So I still think designing a fixed result set paradigm with concurrent-friendly data structures is the first thing that has to be done. And when we have learned to think the right way, we can start using libraries/languages to automate it for us.

 

[Frank]

The fact is, a surprising large amount of code can be boiled down to a few functional paradigms (map, filter, fold, etc) and performance doesn't necessarily suffer because you write MAP instead of FOR with a loop iteration variable.

Yes, but exactly that (map, filter, fold) is what database engines excel in. And that would force you into a data-centric model, which is half the battle. The other half is starting the analysis at the output side. Trace back to the logic from there, by writing procedures that see how that data has to be transformed according to the program logic, instead of executing the program logic and seeing what data you need to manipulate. Store that "business logic" where it belongs. After all, that's exactly what large business apps have been doing for some time now, to counter the same problems. Although for quite different reasons.

What Sweeney is trying to imagine is a modification to C/C++ garbage collection, new types and functions for which non-mutability and loop bounds can be verified statically. I think he's trying to think of a way to have an OCAML like imperative language with a functional subset, but one that uses C-style syntax instead of funky ML syntax.

Yes, I agree. But that's still whishing for things to "patch" C++, without having to change the way you think: leave that to the language constructs. It isn't going to happen. If you want a better language, just pick one, don't try to graft bits and pieces to C++. It won't work. It will only add yet another layer and syntax to the mess.

C# and Java are very good and interesting implementations of something alike the above, build for multi-threading from the onset. And while you lose the possibility to decide for yourself how you want the code to be generated, that is as it should be. Because that is just the part where C/C++ is completely broken. You cannot have it both ways. If you want to do it yourself, don't expect the language/compiler to hold your hand.

And for having some kind of language tree, where you can pick a specific level and syntax for each piece of your code, wouldn't that completely invalidate the main goal: making it harder to fuck up, and improving the development efficiency?

 

[DemoCoder]

Functional languages are "thinking differently" GPU HLSL shaders are very close to functional in nature, and HLSL could be compiled easily to run on a cluster of CPUs if desired. There are languages out there modeled on Pi-Calculus, which explicitly model your code in concurrent asynchronous terms, but I suspect this is not what you're after, because you seem to want C++ with tweaks and a library approach, and this is exactly what won't work.

The discussion is about writing code that will run in dozens, maybe hundreds of hardware threads eventually, not on a bog-standard serial CPU, so the discussion is about the right abstraction to achieve high performance.

Relying on the programmer to launch threads, create mutexes, locks, semaphores, critical sections, condition variables, monitors, and other concurrent primitives, and manage them manually results in VERY VERY nasty bugs. Introducing a library of concurrency oriented datastructures doesn't solve the performance problem, it just solves some of the bugs people have in reinventing things like cyclic barriers, countdown latches, read/write locks, concurrent queues, etc.

But concurrent datastructures are not "fire and forget", Java tried this first with "synchronized" monitor locks on collections like Vector, but the result was poor performance. Then they added sync-free datastructures and allowed you to manually manage serialized access, the result was bugs. Then they added concurrent-lock-free datastructures like ConcurrentHashMap, but using concurrent (lock free) algorithms requires care by the programmer.

You think debugging memory leaks and null pointers is a pain? Race conditions and deadlocks are notoriously more difficult, not just because they are hard to replicate, but because they are hard to *think* about all the various interactions occuring at the same time.

We need programming languages to deal with highly data-parallel problems, and I think ERP nailed it when he said that data needs to be grouped by function. Stateless or localized functions (e.g. pure functional) are concurrent by nature.

A domain specific language will force you to think differently, because you will have to rewrite/recast your algorithms to fit within the confines of the DSL, the same way GPGPU researchers today must recast traditional serial algorithms into HLSL code.

I don't think anyone is arguing for a compiler which can auto-parallelize-threadize serial imperative code. But I think what you're missing is that functional languages are not serial imperative. They don't impose order of evaluation constraints. That's why Monads in Haskell and Strict was introduced in Clean, to give programmers to the ability to specify non-lazy serial execution (in addition to mutating state and interacting with the environment)

I'm not arguing we all throw away C++ and go write 100% in Haskell. What I'm advocating is that algorithms in your code which you KNOW to be candidates that you want to parallelize, but recast in a domain specific parallel mini-language.

 

[darkblu]

The problem with inline assembly is that it can fubar up dataflow and register allocation analysis in the compiler, and therefore, slowdown code elsewhere.

yes, under some compilers. just do not judge about the state of the industry by the state of MS products.

I found that using compiler instrinics when possible ends up with much superior output from VS.NET.

same with vc as its optimiser still drops the ball each time it encounters an asm statement. nothing has changed in this regard in redmond for the past 10 years.

Thus, I think when one has a "gate" to escape a domain, that the domain has adequate knowledge of what's being done outside the gate.

wrt inline assembly that's how it's been in gcc for ever. you can embed inline assembly there and actually communicate to the backend how that affects its context. that's not exactly rocket science.