I can't find much information about the game. Is it resolution sensitive or Geometry sensitive? Meaning does the VRAM usage go up quickly with resolution more than geometry or vice versa? Did you try reducing the resolution to see if it clears up enough space?
Most of the VRAM usage is related to the dimension of the world, the distance of the LOD and how much you alter the landscape. The graphics quality settings does not impact so much on performance.
I do not have any detailed informations about how the engine works, but I guess it does not use a simple vertex-geometry approach to store the modifications you do to the world. I notice this issues only today, so this issue could be just a bug introduced with the last patch.
The game also have a simple profiler showing GPU and CPU timings, but nothing more.
EDIT: as for performance, only shadow quality and LOD distance impact a lot on performance, everything else (ambient occlusion, blur, FSAA, characters shadows quality, geometry quality, texture quality, depth of field etc...) does not affect so much the framerate.
EDIT2: to be clear, you need a decent CPU, most of times I am CPU bound (running on an old Nehalem i5 750 machine, +50% overclock on fsb and final frequency). I also observed over 8GB of commited memory on RAM, and over 4GB of working set. So, at least 16GB of RAM is recommended for huge worlds.