It seems both depend on what kind of problem you are dealing with
One limitation is that some kind of signal or respond cannot be represented using a closed form equation, i.e. step function, discontinuity, or some kind of piecewise linear. Thus, even you are trying to use algebraic to solve you still need to compromise by trade off some accuracy by using an approximation function such as Fouier series (transform), polynomian or Chebyshev. Also, to make PDE and ODE to become such a linear problem in order to be solved by linear algebra method, you still need to make assumptions by neglecting or ignoring some parameters by reducing it to be a constant value or sometime disable it to be zero value (mostly appears in fluid dynamic and heat problem). Thus, only simple problem is very efficient for algebraic solver but I am agree with you the algebraic solver is high accuracy and, yes, speedwise but painful at the same time.
However, on a big scale dynamic model, algebraic method is sometime not appropriate as we seem not be able to produce such a function to accurately describ the model, thus reduced form model with discrete degree of freedom is introduced to aid our pain (yes, approximation comes to play again). It's a long long long story.
On engineering problems, mostly we feel alright to trade off an accuracy within +/- 5% (depend on what kind of problem) if the numerical method can give us an answer of what we want to know. It is better than sitting in pain without noting to discuss
Hope this help, and hope you are lucky on solving your problem. Keep the keyword in mine that it is better get a good approximated solution, but talkable, than a complex algebraic one.
Edit: correcting some typo