It is their own engine and phsyics engine.
Interview link to PCGH
Interview link to PCGH
PCGH: Did you develop your own engine for Aliens versus Predator or did you license a technology? What were the reasons to do so? What are the advantages when utilizing your own technology or using a commercial product?
Kevin Floyer Lea: We use our own engine for all the games we make at Rebellion. It means that we have full control over every feature available in the engine and can tailor it to the needs of every game we make.