A GCN CU has 256kB vector registers (+8kB scalar ones). Each of the four SIMD units in the CU has its own 64kB of registers.
Don't argue with Nick about that. It was mentioned several times already.
The ISA for a UPU
- Thread starter Nick
- Start date
@Nick:
I won't go in any extended discussion with you anymore, I will just answer questions or correct some stuff, you obviously got wrong.
AMD, intel and nVidia are all working on reducing this overhead. In certain situations one can go around this (or at least hide the "setup cost" behind the execution of another task) with the latest generation GPUs. For example, newer GPUs are basically able to create tasks for themselves (dynamically spawning new threads) for which one gets around this round trip latency to the CPU core.
That's what Novum explicitly mentioned in his post at the very beginning of the thread: He wished for a low latency communication to the throughput cores. And that is being worked on.
I won't go in any extended discussion with you anymore, I will just answer questions or correct some stuff, you obviously got wrong.
The bolded part of your first quote answers your second. Currntly, if you want to do something on a GPU, there some significant driver overhead, buffers gets copied in RAM and sent over PCIe, the results have to come back. That's the overhead that currently kills medium sized tasks. If everything is on the same die, one has the same address space and equal access to everything , one could start a task on a CU array just by a special call providing a pointer to descriptor structure (detailing the arguments, and a pointer to the code to b executed), one could get almost completely rid of it. The penalty for starting a task would go from hundred microseconds or so to nanoseconds.
AMD, intel and nVidia are all working on reducing this overhead. In certain situations one can go around this (or at least hide the "setup cost" behind the execution of another task) with the latest generation GPUs. For example, newer GPUs are basically able to create tasks for themselves (dynamically spawning new threads) for which one gets around this round trip latency to the CPU core.
That's what Novum explicitly mentioned in his post at the very beginning of the thread: He wished for a low latency communication to the throughput cores. And that is being worked on.
That's all cleared up already. It is supposed to mean 8 to 12 32bit registers (what G80 offers at full ocupancy). No need to dig that up from 2 pages ago.
Thanks that you agree that Haswell will be limited to 2 loads and 1 store per cycle for code exclusively consisting of memory accesses (that's what I wrote).
How far up can one usually move a load? Isn't that often limited in average code?
That's just wrong. It's not how the schedulers work in current GPUs. If there is any warp/wavefront with an independent instruction left on a CU/SM, it will get scheduled. And the issue of a texture access won't block other threads from issue (not even the same). At least for GCN (and I would doubt it for Kepler, too) the issue of a vector memory access doesn't take any longer than an arithmetic instruction (the completion of the access does). The only possibility to block the execution is when you've hit a limit of the memory hierarchy (throughput, maximum number of outstanding accesses [which is likely very high as GCN could have up to 16 outstanding vector memory reads and 8 writes per wavefront, scalar accesses come on top of it] or something like that).
Last edited by a moderator:
Indeed we should keep our eyes open for useful instructions of this kind. All I'm saying is that for a unified architecture it probably doesn't make sense to go beyond what the latest GPUs do.
When I was defending my theory that Haswell would have two FMA units (which proved out right), I read a research paper on fused instructions that went beyond FMA. I believe that the conclusion was that unless all you're doing is processing huge matrices or long polynomials, it's not worth it over having two FMA units. FMA itself is a significant improvement over just MUL and ADD though. I can't seem to find the paper any more, but the discussion is somewhere on realworldtech if you're interested.
NVIDIA's SFU design reveals that the complexity of a division (reciprocal really) on the GPU is around the order of a few arithmetic operations. And I believe several architectures match the latency of their FMA units to the SFU latency. However, they have a 6:1 ratio in the number of FMA units and SFU units (which is also used for interpolation). So their biggest concern isn't latency it's the bottleneck that could occur from having too high a number of operations that require the SFU units. Anyway, back on the topic of a unified architecture, there surely is a lot of room for improvement for the performance of a full precision division. But I don't think the perspective division in graphics has a desperate need for that. There's a 2:1 ratio for FMA and RCP and one Newton iteration suffices, which is cheap if you have FMA anyway. Most importantly, only one perspective division is needed per pixel. The increase in average shader length has already made it insignificant a while ago. You can use a low-throughput high-latency division with no appreciable effect on performance.
Well if it's a thing of the past, i.e. GPUs are moving away from it, then I don't think the CPU should be moving toward the old stuff since it would become a burden if it's practically never used.These are pretty common in DSPs.. I don't know where they apply in modern high end GPUs but I know you don't have to dig back terribly far to find more odd multiplication width pairs in GPUs.. they might still lurk in fixed function stuff. But I don't have applications outside of things like emulating archaic platforms like Nintendo DS
That said I've started to realize that there are most definitely mixed width multiplication instructions that could be of great help for texture filtering. So thanks for the suggestion to look for these.
I'm terribly sorry but I laughed out loud when I read this. You can't categorize Larrabee's hardware one way or the other. Aside from the dedicated texture units it is highly generic and can be used any way you like. Whether or not Intel's default driver was intended to be TBDR-based or not, or some form of hybrid or configurable renderer, that's a whole other question.
Again I wouldn't say anything is a "must". There are cases where forms of tiling help a lot and other cases where not tiling avoids wasting cycles and power. And that's the beauty of a unified architecture; you can do things in much smarter ways tailored to the situation. The hardware doesn't impose one approach on you. GPUs have already given up fixed-function vertex and pixel processing for something much better, despite the cost, and now we're getting closer to the point where the rendering processes themselves will cease to be fixed-function. For better.
I hope this clarifies why I think it would potentially derail the discussion to respond to why TBDR is not well suited for tessellated geometry. You can find some answers/opinions here: Early Z IMR vs TBDR - tessellation and everything else.
Note that GPUs dropped support for palletized textures, despite being a perfectly useful way to compress textures if you have custom pallets per texture. It's really the latter restriction that killed this feature, because it can't be done efficiently in dedicated hardware when you're constantly sampling from different textures. That is, it can't be done more efficiently than a generic gather operation from memory.
So gather and PDEP/PEXT are all you can hope for. AVX2's gather is limited to 32-bit indices, but in a couple more silicon shrinks there might be timing room to lower that down without affecting latency. For 4-bit to 16-bit lookup an in-register permute (vpermd) should be of great help already.
Going to have to see then if that's "just FMA" or if it includes any of the other stuff I've referred to. But since you're talking about replacing current embedded GPUs it doesn't make sense to only look at what discrete GPU families are doing, meaning you should be looking at Gen and IMG's current GPUs at the very least (or are you drawing the line at some power budget?)
Put it this way: if you have support for 8, 16, and 32-bit multiplication as optimized SIMD cases like AVX2 does then you WILL benefit from mixed width multiplication. Not as much, but it's a certainty that it will come up. Whether or not it's worth it depends on the granularity that the uarch differentiates performance of different multiplication widths. If you can do a 16x32 multiply twice faster than a 32x32 one then it's worth it.
GPUs moved away from higher performance operation of anything < 32-bit; CPUs haven't (as Intel proves) but throughput ones like Xeon Phi have. Meanwhile mobile GPUs (and Gen) have offered lower width operations, because if you don't need the precision it's an obvious perf/W improvement. So where exactly does your stance on this lie?
Yes, I was thinking filtering right after I posted it
Don't be such a snob. Of course I was talking about Michael Abrash's and co's LRB renderer, not what the "hardware" did. Don't see how you couldn't have gone straight to that.
The big thing that sticks out to me while is that for software rasterizers, if you end up with a bounding box that says if the triangle is in or out (and how much in, linear barycentric coordinates) and want to go straight to rendering that you then have to step through this block and determine which of the (< 50%) of pixels are visible. Then do it again after depth test. Or do it only after depth test, but then you're doing depth test on a bunch of fragments you know you won't use.
The point is, I can't think of a good way to dispatch a bunch of shaders on this without stepping through the tile and looking if there's going to be a fragment at each location or not. You can do it hierarchically but it's still going to be a fair bit of overhead. If you do a TBDR then you first fill up the tile with IDs of all fragments contributing to it, the overhead of discarding unused fragments IDs is cheap (blend it against the tile) and you have no gaps in it.
The way I see it, if you aren't doing TBDR that sort function could be even more useful..
That thread hardly makes a strong support of your case..
I'm not seeing your case here. Compressed textures still use packed small bit width lookups, and they go to larger widths. This is useful all the time outside this case, especially if you're using a lookup vector that doesn't have more than 16 indexes to begin with. Yes you can convert your indexes to wider widths with pdep beforehand but why is it not an advantage to have an instruction which does 4-bit directly? Especially w/o knowing the cost of pdep/pext, I'm led to believe the area is not trivial, while picking out fixed bits for input to the shuffle unit would be..
OpenGL guy
Veteran
Not even close. A simple float add is 9 clocks on GTX 680. However, a single thread can multi-issue, meaning there's only a stall if there's not enough other work to do.
I don't know about Nvidia, but on GCN, we don't swap threads for ALU latency, because there is none to speak of: Each SIMD in a CU runs separate wavefronts. Most instructions run in 4 clocks and that's the smallest execution granularity. If you hit a longer running instruction, say dmad, then there's no point in swapping threads because the ALUs are already busy.Nick said:
Haswell's FMA latency is 5 cycles. So how can you say the GPU's ALU latency isn't worse when Fermi takes 18 cycles? And that's without taking clocks into account. Also, even if the effective latencies were the same, the CPU would still achieve much higher average ILP per thread thanks to out-of-order execution. So it's of a different relevance.
What do you mean by zero latency if the ALU instruction is in the instruction cache? It always takes multiple cycles to have the result available for a dependent instruction.
No, the instruction cache has nothing to do with the latency I'm talking about. I'm talking about the ALU latency plus the bypass latency.
No I really don't have a clue what bandwidth deficit you're talking about. The bandwidth wall is an argument in favor of unification of the CPU and integrated GPU. They share the same bandwidth and the CPU's computing power can grow faster than the bandwidth while the GPU already exhausts it so there's continuous convergence and eventually unification.
You make it sound instead like it's something that would need fixing to make unification work. So if there's some other bandwidth deficit you had in mind that impedes unification, I'd really love to know about it and I would happily answer how I expect it will be overcome.
OpenGL guy
Veteran
This is not true for GCN.
But I don't have to formulate such a corner case to see OpenCL applications run faster on a CPU than on a heterogeneous architecture. In other words, workloads that people assume would run faster by making use of the GPU, aren't running faster at all due to inherent inefficiencies in the GPU architecture and/or due to the heterogeneous overhead.
I think there's a slight difference in terminology here. Here's how the vector part of GCN works AFAIK:
- Single shared decoder that issues 1 vector instruction/cycle.
- 4x16-wide ALU pipelines with 4 cycles latency.
- One active 64-wide thread per ALU pipeline.
This results in the following behaviour:
- Decoder issues one instruction to each ALU pipeline every 4 cycles (round-robin).
- Each instruction takes 4 cycles to process as the threads as 4x as wide as the ALUs.
-> By the time the scheduler comes back to this pipeline, it is ready to process the next dependent instruction.
Now in practice it's a bit more complex than that (the register file is probably the really interesting bit), but basically you don't need any ILP to achieve full throughput with the minimum number of threads. While you do need 256 vector elements (4x64-wide threads) to feed 64 vector elements of processing (4x16-wide ALUs), I think this is basically the exact same trade-off Nick is suggesting for AVX1024.
Handbrake only uses OpenCL for certain parts of the encoding pipeline. This is not a good benchmark, because the slow CPU cores of AMD drag the GPU down (Amdahl's law).
But indeed, video encoding is one of those tasks where a throughput core can't shine because of entropy coding, which is inherently non parallel.
The benefits would be comparable to the unification of vertex and pixel processing: performance, cost, and new applications. For a long time the GPU's unification didn't make sense from a performance point of view due to the inherent differences between vertices and pixels. But soon after they found their common ground they did become unified. Performance wasn't an obstacle any more because unification did not imply sacrificing the qualities of either type of core: it unifies them.
It eliminates duplicate logic, and it eliminates communication overhead between heterogeneous cores (data, code, synchronization, etc.). So it lowers the hardware cost and improves performance, especially for applications that have workloads that fit the 'common ground' or have a mix of workloads that would otherwise be impeded by the heterogeneous overhead. This sparks the development of new applications, which then in turn solidifies the need for a unified architecture.
While the ability to create new high performance applications is already an interesting financial prospect for developers, a unified architecture is also vastly simpler to program than a heterogeneous one and thus reduces the software cost. Compilers can use the UPU's SIMD instructions, with practically no risk of ending up with lower performance like with a heterogeneous architecture. And you can use any programming language and tool set you prefer. Of course the brave still have access to assembly intrinsics and such to squeeze every drop of performance out of it, but the important thing is you're not forced to know anything about the hardware configuration or its characteristics to be able to benefit from it, like you would with all the different heterogeneous setups. So it becomes accessible to the average developer, accelerating his application without effort.
Lastly, the software ecosystem for unified computing is more thriving. You can exchange code, algorithms, libraries, frameworks, etc. much more easily than for a heterogeneous architecture. The amount of OpenCL code pales in comparison to the amount of C, C++, C#, Objective-C, Java, that would benefit from unified acceleration.
Also note that something like OpenCL requires a committee to define and refine the API, and it requires hardware manufacturers to invest into creating drivers for it. So again unified computing would reduce costs at this level as well and broaden the possibilities.
Not really. It only takes 8 cores with AVX-1024 to create a 3+ TFLOPS UPU. TSX will make it feasible to use use that many cores efficiently for a wider range of applications.
Also keep in mind that leaving cores idle when there's nothing to do isn't a bad thing. If an architecture provides high ILP, high DLP and high TLP it can run any workload in the spectrum, but some will inherently not use all these features.
Transactional memory ... easy to write wrong, impossible to debug ... such a great combo. It's never going to be the saviour which will make parallel programming more ubiquitous. Parallel programming isn't easy and transactional memory is not always an efficient solution.
Any way, if we are going to unify around something I'd like it to be something which can efficiently handle divergent workloads ... MIMD down to at least a VLIW level, perhaps scalar ... certainly not a 32 wide vector.
Any way, if we are going to unify around something I'd like it to be something which can efficiently handle divergent workloads ... MIMD down to at least a VLIW level, perhaps scalar ... certainly not a 32 wide vector.
OpenGL guy
Veteran
The 4 vector units in a CU can process separate instructions. This is why I recommend at least four 64 thread work groups per CU as opposed to 256 thread work groups.
My only point was correcting the notion that all GPUs need to swap in other threads to hide ALU latency. As others have said there's effectively zero latency for AMD's GCN to execute dependent instructions. The actual latency is 4 cycles, but that's hidden by the wavefront executing in 4 groups.
You of course need a lot of data parallel work to fill up the GPU in the first place.
That's only because Intel already mentioned AVX-1024 in 2008. Of all the current throughput-oriented architectures, Xeon Phi is 512-bit, NVIDIA's are 1024-bit, and AMD's are 2048-bit, so it's good to know that the CPU's VEX encoding can be extended to 1024-bit. The instruction set certainly needs to be fleshed out (which is what this thread was supposed to mainly be about), but things are going to be straightforward to add as extensions (at least from the ISA point of view). The x86 encoding format is ready for unified high-throughput computing.
That is certainly true. It's even a consequence of any increase in performance. If the average scalar ILP increases, the hit rate of an equal cache would have indeed gone down. That decrease doesn't prevent the performance increase from happening in the first place though. And if you increased your ILP, you probably also have the transistor budget to sufficiently or fully restore the hit rate again.
What's also critical to realize here is that this is every bit as bad for GPUs. They may not care much about hit rate for the latency impact, but they do care a lot about the bandwidth impact!
Finally, note that Xeon Phi has 512-bit SIMD and only 4-way round-robin SMT to hide latencies from cache misses, and seems to be doing alright. Haswell has out-of-order execution, and 2-way anything-goes SMT. Of course you need to take core count and SIMD count and cache sizes into account too, but I think you'll find that it's a very broad design space and the CPU should have no major trouble to further increase its SIMD width. One of the techniques that will go a long way is the use of long-running SIMD instructions, which could double the latency that can be hidden or more.
So this shows that a slightly lower cache hit rate could merely be the 'symptom' of much high performance, for either architecture. I don' t regard it as something that will require substantial changes beyond regular progress and the addition of long-running instructions. During high-throughput workloads CPUs would behave more similar to GPUs (the good and the bad), which is exactly what we want from a unified architecture.
No need to call this a rant as far as I'm concerned. I fully agree it should be replaced with something better. I'll look around for potential options. Thank you.
As I've discussed before, CPUs can achieve high throughput with fewer threads and registers by having lower latencies and by hiding them with out-of-order execution. It is highly debatable whether increasing the thread count would improve performance since Hyper-Threading only increases performance by 30% for the best known cases, and even with just two threads it is known to cause cache contention which nullifies any advantage in the worst cases. It is probably a better idea to hide more latency with long-running instructions since they keep the cache accesses coherent.
That said, long-running SIMD instructions do in fact increase the number of strands (i.e. work-items or scalar threads if you like), and require increasing the register space. But you still won't need anywhere near the same amount of work in flight or registers as the GPU does to achieve competitive performance. And that's a good thing considering Amdahl's law.
CPUs already use lower clocks when all cores are active, than when just one or a few are active. By keeping track of the amount of SIMD instructions they could adjust even more dynamically to high ILP or high DLP workloads or a mix of them.
That is clearly a false assumption. Haswell will be substantially faster at throughput computing than its predecessor and yet its single-threaded performance will also improve. I also see no reason to assume that widening the SIMD units to 512-bit and adding support for AVX-1024 would have any negative effect on single-threaded performance either. The new process node used for that would most likely allow to increase it again.
Unification doesn't have to be a compromise. It can be the best of both worlds.
The logical width of the SIMD operations is the minimum granularity for both the GPU and CPU. NVIDIA's warps are 32 elements while AMD's wavefronts are 64 elements. You can't "partition" it into anything smaller. The CPU's AVX-256 instructions can be used to process 8 32-bit elements. So its granularity is in fact smaller than that of the GPUs.
No. All of the lanes of the GPUs SIMD units perform the same operation. That's why it's called Single Instruction Multiple Data. It's the exact same thing on a CPU, just with different width (for now).
