That's certainly one danger of a streaming focused console.
As opposed to a traditional console where the gaming experience is consistent regardless of where a person hooks up the console and plays their games (on a farm miles away from a city versus in the heart of a large city, for example), a streaming only device is likely going to offer an extremely variable gaming experience depending on the location of the console, distance from the closest data center, console owners internet quality, etc.
While, the console maker (MS in this case) can put all kinds of warnings, etc. on the box stating the optimal conditions for a streaming box, it's still quite possible that someone might buy it without looking at that stuff (Grandpa getting a console for his grandkid's birthday, for example).
There's a fair bit of potential for negative press or negative word of mouth if someone gets one of these streaming consoles and has a bad or non-optimal experience. I'm certainly expecting reddit, twitter, etc. to have mini-explosions when/if MS releases a mainline streaming only console and people start to play games on it. Smaller players can get away with this as it's kind of expected that they're going to get stuff wrong. Expectations of an established player are going to be much higher. Especially if they release it as a mainline product (with associated advertising) versus something that is viewed as just an almost inconsequential side product (like Sony's PlayStation TV, for example).
Regards,
SB