First of all, I'm a huge rally fan, being born of a rallying father and been somewhat involved in the sport (marshalling).
I think there are a few misconceptions about what rallying is and was about. Group B for instance, wasn't really what rallying was about. People see it as the end-all-be-all of rallying, but in the grand scheme of things it was merely a blip on the radar. Group B existed between 1983 and 1986 and quickly went out of control. In 1979 they drove slightly modified Ford Escorts (they even ran w/ leaf-springs initially! Although with 270hp Cossies in them at the end) and by 1985 there were tubular spaceframed 600hp cars with plastic/carbon fibre body shells. Does anyone see a disconnect somewhere between those? Not to mention one or two decades before that... (think Mini Cooper, SAAB 92, Cortinas)
Rallying has it's roots not in "Formula One on dirt", rather in "Getting a car across Europe with some trials along the way". Stage rallying, as it's known today, only came about in the '60s.
That's not to say that I didn't appreciate the Audi S1 or the Lancia S4, but from my viewpoint, the sport kind of took a detour there. It wasn't supposed to be, really.