[keldor314]

Originally Posted by rpg.314

But that's very inefficient to transfer a multilevel data structure, like a BVH, which is where you started out from. For something like BVH, you really need a cache coherence protocol. Abusing page fault handlers just wont do.

What I'm describing is a cache coherency protocol, operating at a memory page level rather than cache line level. Trying to keep track of it at a cache line level would be far too slow, since the data transfer would be dominated by overhead, since cache lines are so small. In addition, the directory would be huge.

The reason to do it at the page level in particular is that you already have dedicated hardware. The big limitation is that you wouldn't be able to do read only sharing, since I don't believe the TLB has any concept of read-only, and thus can't throw an interrupt when the first memory write is encountered. You could however have a state in your handler emulating read only sharing by simply not requesting an invalidation whenever a page belonging to a read sharable memory space is brought in, though this would have to be respected by the programmer.

How efficient transferring a multilevel structure would depend on how well malloc can cluster objects together. If the memory page had a large number of other elements of the structure, it would be much more efficient to transfer the entire page than to do each element separately and update pointers (a BVH is, admittedly, an extreme example, but there are many cases where you have small multilevel structures you want to use on both devices). However, false sharing would be *bad*.

What you'd need then is some concept of memory spaces for malloc, so that it wouldn't inadvertently allocate some CPU scratch variable in the middle of a memory page containing GPU data. A memory space would simply be some set of memory allocations that's guaranteed not to alias pages with memory allocations associated with any other memory space. It'd also be important to ensure that any page from a memory space that can be read by multiple devices is small.

The ultimate result would be a system that, while not an ideal coherency protocol, could still retain many of the benefits with regards to usability, and would not make anything slower in the traditional usage pattern. The important part is that this could be implemented in software only, so the only thing that would have to be updated would be the OS and the drivers.

[Andrew Lauritzen]

Originally Posted by keldor314

Trying to keep track of it at a cache line level would be far too slow, since the data transfer would be dominated by overhead, since cache lines are so small. In addition, the directory would be huge.

Maybe for a discrete GPU, but I don't see a problem on integrated where it's using the same memory hierarchy anyways... both AMD and Intel are going this direction.

How much effort to throw at discrete GPUs/memory spaces depends on how much you believe they are going to matter in the future. With all of the consoles going unified and arguably everything laptop level and down as well, it's only going to be the very high end desktop stuff left as discrete. One could make an argument that those systems could take a more brute force path and still be acceptable. I have a hard time accepting that APIs should be designed around their constraints going forward, even though I love my massive discrete GPUs

[rpg.314]

Originally Posted by keldor314

What I'm describing is a cache coherency protocol, operating at a memory page level rather than cache line level. Trying to keep track of it at a cache line level would be far too slow, since the data transfer would be dominated by overhead, since cache lines are so small. In addition, the directory would be huge.

The reason to do it at the page level in particular is that you already have dedicated hardware. The big limitation is that you wouldn't be able to do read only sharing, since I don't believe the TLB has any concept of read-only, and thus can't throw an interrupt when the first memory write is encountered. You could however have a state in your handler emulating read only sharing by simply not requesting an invalidation whenever a page belonging to a read sharable memory space is brought in, though this would have to be respected by the programmer.

How efficient transferring a multilevel structure would depend on how well malloc can cluster objects together. If the memory page had a large number of other elements of the structure, it would be much more efficient to transfer the entire page than to do each element separately and update pointers (a BVH is, admittedly, an extreme example, but there are many cases where you have small multilevel structures you want to use on both devices). However, false sharing would be *bad*.

What you'd need then is some concept of memory spaces for malloc, so that it wouldn't inadvertently allocate some CPU scratch variable in the middle of a memory page containing GPU data. A memory space would simply be some set of memory allocations that's guaranteed not to alias pages with memory allocations associated with any other memory space. It'd also be important to ensure that any page from a memory space that can be read by multiple devices is small.

The ultimate result would be a system that, while not an ideal coherency protocol, could still retain many of the benefits with regards to usability, and would not make anything slower in the traditional usage pattern. The important part is that this could be implemented in software only, so the only thing that would have to be updated would be the OS and the drivers.

Too high latency. Every page fault kicks to the kernel. This is not a solution to any real problem. Much better to batch up the stuff and do a single copy call.

[keldor314]

Originally Posted by rpg.314

Too high latency. Every page fault kicks to the kernel. This is not a solution to any real problem. Much better to batch up the stuff and do a single copy call.

Well, of course. An explicit copy will always be faster. That said, batching up and copying anything but trivial structures is quite ugly. Also, index based indirection doesn't really cut it, since it forces you to use an extra register to access anything (base address + offset), in addition to extra address math. There really needs to be some way of dealing with pointers without jumping through hoops.

HSA is interesting, but until the CPU socket catches up and closes the order of magnitude wide memory bandwidth gap with the GPU socket, it's useless for high end stuff.

[rpg.314]

Originally Posted by keldor314

Well, of course. An explicit copy will always be faster. That said, batching up and copying anything but trivial structures is quite ugly. Also, index based indirection doesn't really cut it, since it forces you to use an extra register to access anything (base address + offset), in addition to extra address math. There really needs to be some way of dealing with pointers without jumping through hoops.

HSA is interesting, but until the CPU socket catches up and closes the order of magnitude wide memory bandwidth gap with the GPU socket, it's useless for high end stuff.

It's ugly vs slower. Ugly will win because performance matters.