I don't follow the sentiment here at all. What's the point of comparing two things that do completely different things? A hair dryer consumes more energy than either a refrigerator or a GPU. So does a car. It doesn't make any sense to me to use any of these as a benchmark to judge the excessiveness of the other because they solve entirely different problems. A GPU could be consuming 1mW or 1MW, that doesn't make it more or less effective at its job in relation to a refrigerator.
Instead we can compare it against other computing devices. And you know how that story goes -- we've enjoyed a freaking *exponential* increase in efficiency over the past several decades.
Yeah it sucks is that that exponential efficiency growth is slowing down, and it's getting expensive to sustain it. We're just spoiled. I doubt there is any other field in the history of human civilization that has seen a sustained exponential efficiency growth. And that growth hasn't happened by magic -- it has happened due to the efforts of many, many smart human beings. Let's give them their due credit.