If it is fixed amout of workload, dropping cores doesn't scale linearly. Largest impact would be going to single core, as it cant render and compute the same time, stalling more and more the start of the calculation of next frames.
ie. when going under 1fps, the processor cant keep up with...