Then of course, there's the whole thing with how there are always people who are there strictly to leech and then you might have a lot of failed pieces because someone you were downloading from may just sign off suddenly... and then a few moments spent using x amount of bandwidth gets wasted. Similar things happen if an active piece gets dropped due to a timeout. This can happen just because the person has way too many pieces being downloaded and something may just be limiting them from being able to manage that many simultaneous transfers.
Indeed, there are ISPs -- some would argue that all ISPs in the US, at least -- which try to minimize the use of P2P serving of data by limiting how many simultaneous transfers are allowed per customer over most ports (though they may not limit it over HTTP/FTP), and if you're limited in that way, there's no way you can ever really reach your connection's max available speed.
The other aspect of it is that the bandwidth you get per transfer is generally pretty low since people are sharing their bandwidth over multiple peers, and the vast majority of people out there have asymmetric connections. The fastest transfer rate I've ever gotten from a single peer sending one block is only about 20 KB/sec. In general, the transfer rates per peer are anywhere from 1.5 KB/sec to about 12 KB/sec if I'm lucky. So even if you get a total sum bandwidth usage of 6 MB/sec, it's only because there are a lot of pieces coming your way at once. That doesn't really change the fact that each piece is coming at you very very slowly.
I've also found that firewall configuration can vastly change your torrent experience (and negatively impact the functioning of your internet connection while 'in the cloud' if set up wrong).
True, but the difference in a case like that is much larger than seeing only 200 KB/sec on a 6 MB/sec connection. 200 bytes/sec is more like it. Historically, I've seen that if the firewall doesn't have the right ports open, the speeds you'll get will never step outside the range of what you could achieve through a dial-up connection, no matter what your actual bandwidth limit is.