Please elaborate on which game types you think fit in this niche category. There are NPC's in the majority of games I play. Not all of the same type, but all need AI routines.
Living world games require non-combat AI as Joker describes. For most games, AI is for opponents, from shooters to racers to sports games to platformers to beat-'em-ups to dungeon crawlers, etc. The vast majority of games only need AI enough to present an interesting challenge, and in most of these games, online provides a 'cheap' solution to a challenging AI. I agree that offline bots etc. can do with being smarter for those that play offline, but I can also see why devs won't be investing heavily into AI when they have an easier solution at their disposal.
Is AI RAM/processor usage proportional to advances in hardware like, for example, rendering is?
The number of concurrent AI in any given scene might be trending upward (I don't know, haven't really looked into it) so I can see higher resource usage in that scenario, but as more system resources become available with every generation surely the percentage allocation to AI is shrinking?
Can you give any concrete examples of resource usage for AI?
I can't give any specific data. I can say that you end up with exponential and messy growth akin to raytracing or physics modelling. For each agent you need to test scenery for possible interactions and other agents, and quite often ray/path calculations. eg. To fire a gun with leading you need some non-trivial maths. To predict where they might change direction to also requires maths. 100 agents all testing rays against each other and scenery gets inefficient. Also the basic premise of decision making boils down to a load of conditionals, meaning lots of branches and random access.
Let's look at something like an Elder Scrolls game. You have an agent that can move objects, and want this agent to set the table for dinner. They need to evaluate all the local objects to decide if it should be on the table or not, test every object being placed to make sure there's room (if you put a jug in the way of where they are going to put the plate, they need to stop their action and respond), respond to changes in the room state to see if anything's making a mess of their task (you putting knives and forks back) and to determine when the task is complete. They need to respond to a knife not being present because another AI agent had taken it out before to use for some job, and needs a start a search that doesn't lead them to the store to buy a new knife or wandering the woods performing a simple locality test for the presence of a knife. And with the interactions becoming more complex, the chance for bugs and errors increases exponentially. There's a good chance IMO that you'll hit artificial madness before you hit artificial intelligence. We've seen that already with NPCs walking on the spot into fences etc.
Considering that most games don't need anything like this level of complexity, just needing NPCs to cover, shoot, not stand still getting shot at, and have some sort of tactics (or other behaviours necessary for the game), it's not surprising that there's no significant investment in AI and it's down to the open-world games to create their own.
And here's an example of poor AI bringing entertainment value!
http://www.express.co.uk/entertainm...r-GTA-5-PC-Mount-Chiliad-GTA-5-Taxi-challenge
I think GTA is a great example. Improving the visuals is a far, far simpler task than creating intelligent virtual people fully aware of their surroundings and responding with simulated emotions and convincing interactions to each other.