Great response. For some reason it didn't occur to me that the pointers could just be redirected and you'd hold a translation table.
A first thought however The setup will still suffer some loss regardless. Data is likely to be setup serially for pre fetch. With this translation your likely going to be jumping all over memory. There still will be heavy penalties I think.
I think any inefficiencies introduced there will more than be made up for by the inherent efficiency of having a single memory pool where all memory accesses have the ability to leverage GDDR5-class bandwidth.