I'm sitting on the "don't worry about it" side of the fence. What you're asking for is a way to spend tons of times and resources on something that you will simply not be able to do as well as the existing drivers. You might think you need absolute 100% control over it, but chances are you'd make the performance go DOWN and not up by managing the memory manually. Yes, it's a dead end. But it's also a dead end for a path that is really unnecessary to go down. Focus on other parts of your renderer that are more dire. Try to do some performance testing to figure out where your program can be sped up, and focus your time and energy on that instead.