It does with a i3 10100 (64K L1, 256K L2, 6MB L3) and ryzen 3600 (64K L1, 512K L2, 16MB L3 with access latency penalties for half), but not really with a ryzen 5600 (64K L1, 512K L2, 32MB L3 with no penalties), which is why I think it's cache related. (edit: also that anecdotal but very similar BFV user video with an old i7)
Edit: Look at the gamegpu aida scores
L1 1.3ns
L2 3.9ns
L3 14.5ns (will be worse if you read across CCX boundary)
RAM 85.4ns
All it would take is for the nvidia to hit higher levels in cache more often or RAM more often and you can get a 10-20% performance difference.
Modern games probably hit caches hard which will cause more misses for other threads. Open world games like Watch Dogs would probably be the worst. You might not see the issues if you're gpu limited because maybe the memory system is able to keep up since the cpu is waiting on the gpu. If you become cpu limiited, suddenly the cpu threads start going as fast as they can and maybe these smaller or higher latency caches cause the nvidia driver a little more pain.