I'm not sure you necessarily can - apart from having multiple decoders and some AI to learn your channel hopping habits.
It's been many years since I looked at this (so take with a grain of salt) but, I seem to recall that, due to the extremely variable rate of encoding, the actual H.264 spec might mandate that the decoder maintain a large buffer of the undecoded stream, which will add to latency.
Further, there are 3 methods of encoding frames: I, P, & B. If you change channels, the decoder has to receive an I frame in the stream before it can produce sensible output, but you don't want too many of those in the compressed stream as they are the largest. Further, though I don't know if this is used in TV broadcasts, B frames can reference pixels of both past and future frames, which means frames may be sent out of order, again requiring quite a bit of buffering (and latency) to cope.