The X GPU is much faster but the CPU is similar and almost all the advances including 4K are GPU related. If you forget about the OG XBO, what could they have done differently on the X at 4K with that Jaguar CPU?
I can answer that... but first the CPU. The One and X CPU may be quite similar but there is one diference. The clock speeds. And in that term the X is about 30% faster!
With this in mind lets imagine you would use all the X CPU power for something like... AI. How would you downgrade this for the One? Making NPCs more stupid?
In other words, there are restrictions on how you can use that extra CPU power because One specs must be kept in mind. Developing on the X instead of the One, does not change this.
Now about the GPU. It's a fact that 1080p 60 fps can be achieved on the X over 1080p 30 fps on the One, using only half the required power it would take to go for 4K 30 fps.
But make this game use the same GPU with only 1080p 30 fps on the X and what would you have on the One? Assuming you were limiting yourself and only using the same power as before, you would get 1080p 15 fps. Make this a full 6 Tflops 1080p 30 fps game, and you get a 1080p 7 fps game on the One.
Now for 4K as you mention, the X can reach 4K, and sometimes even improve on some places. If these improvements are not asthetics only, you may affect gameplay on downgrade.
Imagine a 4K game heavy on alpha effects, using all the X power, where you could not downgrade to 1080p, same fps 30 fps, keeping all the transparencies. This could affect gameplay heavily, making the game different.
Note that even mere asthetics can not influence on the game. For instance I remember playing Unreal 2003 with my clan back in the days, and cranking up all detail to max. I was moving crouched in the middle of the grass, having trouble seeing my opponents, and I kept getting killed. I complained on Team Speak: " How the hell do you see me in the middle of the grass?", and got this simple reply: "What grass?".
Yeap, you guessed, on lower detail there was no grass, and they had a player completely uncovered and exposed moving around. Time to lower detail!
These are the kind of things that cannot happen. And for this you must always have in mind that the specs of a lower machine cannot give you neither advantages, neither disadvantages, in terms of gameplay.