Well, in Linux you can use 1920 x 1200 (WUXGA = ps3videomode -v 13) which is a 16:10 format.
As for 720p, 1080i and 1080p are concerned it's a question of how your display handles those resolutions.
*scares that i will get stretched 16:9 pic on 16:10 screen*
so if the monitor letterboxes it, it means you will get 1280 x 720 for 720p??? yeah???
also, what do you get if you let ps3 do the maths for you (aka auto setting )