It is not about being realistic.
The thing is, when I use a traditionally controller I'm not expecting that if I swing it, the character behaves naturally on the screen. But if I use the Wiimote, if I swing it, why doesn't it swing like that on the screen? If I swing the control 180 degrees, why does it not swing 180 degrees on the screen?
This is why it's similar to the uncanny valley. It's introducing a control paradigm that is to emulate real-life actions, yet the actions on the screen do not follow from the actions you are performing. In the attempt to make it more realistic, it has made it "almost but not quite", which is more frustrating than a control scheme which does not attempt to be realistic.
I'm frustrated with gesture-based Wii games, like Zelda: TP. I find them frustrating, just like I find mouse gestures in browsing to be frustrated. With a mouse or a control pad, I can have 100% accuracy in what I want to do (press a button, move the stick precisely). With gesture-based movements, I have sub-100% accuracy -- and any inaccuracy is immensely frustrating to me.
Perhaps you don't have a problem with this. That's fine, it's probably personal preference. I just don't like things half-assed, and in my estimation, every Wii control scheme I've encountered thus far has been half-assed. Gesture based controls for games are fun for silly mini games, but I've not enjoyed them as part of deeper games that require more precision.