What do you have to know to be a Physics programmer?
Depend what you mean by that. If you mean that you want to be able to create physic simulations from scratch, or understand the low level intricacies of physic engines, you'd need different skills than if you want to code on physic API platforms such as Ageia or Havok's.
Obviously, the latter is a lot less demanding than the former. With that said, I'm not certain that a lot of dev-houses out there employ programmers which jobs are to hack a third party physic API on a full time basis. It's true that in the future most dev-houses will have dedicated folks working on the physic side of their projects, but these folks would need to be full fledged programmers, who are able to create simple (or more demanding) physic engines from scratch, and/or update a, third party or not, physic code base when necessary.
In other words, you definitely would need to learn a programming language, and also learn more calculus/vectorial maths on the side.
If you don't feel like jumping straight into learning C and calculus, you can always start, slowly, by learning a managed programming language, such as C# (since it has some game tutorials for it on MSDN) or Python (it's really easy to use), and try some 2D physics tutorials. That way, you'll get some visible results faster, which will help you not giving up on learning more intricate and complex techniques.