Was the guard band too small for very large quads, or too slow...?
Both were issues, the guardband was only 11-bit (PS1 was 12-bit, which was still way too small but a lot better) and you had a choice between spending cycles for every quad line to get imperfect fillrate saving for pixels outside the clip rectangle or no fillrate saving at all.
But the bigger issue was that you couldn't arbitrarily tessellate quads into smaller polygons to prevent them from going outside the guard band when they did. Quads mapped to axis-aligned sprites, whose widths had to be a multiple of 8 pixels no less. So while you could tessellate the vertices you couldn't match it with the exactly right texture coordinates. I guess your best bet is to scale the quad to get the offending vertexes inside the guardband and hope that the other vertexes are all outside the clipping rectangle. The games with obvious edge distortion probably didn't even do this.