Sebbbi mentions peak hours, this can impact both PSN and your ISP's access network. Assuming it's not PSN (since I do not know much about it), then problems with peak hours for ISPs. Just rules of thumb
If your internet is xDSL, during peak hours the activity on the copper links in a bundle (multiple DSL links going into a DSLAM) can cause noise across the links in the bundle. If the termination of the links in the MDF are crap, then more noise etc.
If your internet is Cable, then shared medium at peak hours.
If your internet is Fiber, this applies to both DSL and Cable also, uplink from the COE equipment at the POP might be saturated. Few providers build networks with no overbooking built in, ie if they have 100 customers on 1G links then they might just use a 10G for the uplink from the POP. Most residential ISP setups use an overbooking of 1:100 and business links around 1:25 or 1:50. There is no rule, but this is in my experience. Ohh yeah, overprovisioning does not need to be no the uplink, could be the backplane/switching fabric capacity of the DSLAM, CMTS and Switches also
Your internet is radio based, then there can be multiple reasons that a link becomes worse.
Add all fun stuff as impuls noise, intermittent noise sources for both Radio and Copper can wreak havoc.
I love my job.....
//edit
Ohh yeah, voice, is realtime stuff and hard to transport well enough. I just heard that mobile operators are looking at dropping doing voice over data and fallback to doing voice the old way. Due to all the issues cropping up, we are now talking carrier grade voice and not lower grade solutions, that I suspect that gaming uses.
Voice and Wifi is also bad, especially if the voice packets are not prioritized correctly, newer wifi uses traffic classification and voice should go into a low latency queue. But if its tagged as best effort, then that wont happen. And it might end up being aggregated,
https://en.wikipedia.org/wiki/Frame_aggregation
Then you might hit jitter and buffer thresholds for the voice application and your brain does not like it when the delay is higher than 20ms. More than 500ms you get really annoyed