When your PC opens a TCP connection to or from a remote server, your PC tells the remote server what your PC's TCP Receive Window (RWIN) is. The basic flow control mechanism of TCP is that the sender will stop sending data if it has not yet received a TCP acknowledgment (ACK) for any of the last RWIN bytes of data sent, and the receiver should transmit an ACK packet back to the sender when there is about half an RWIN's worth of data received but not yet ACKed. The sender uses the arrival of the ACK packet to pace itself on how fast to send data to the receiver. Pulling in a fast download requires not only the download packets to arrive quickly, but also for the ACK packets to get back to the sender in a timely fashion.
If you saturate your upload, the ACK packets of your downloads (also including simultaneous web surfing, which tend to be many requests) will have to queue up waiting for a gap between the congested upload data packets. So your ACKs will be delayed getting back to the remote download server, and it will therefore believe you are on a very slow link, and slow down the transmission of further data to you.