Where's the PS3 hypervisor?

[seebs]

I am not convinced that it is even theoretically possible to implement affinity, at least to the levels described in some of the research papers from the IBM side.

Imagine, if you will, that I want to have three adjacent SPEs running a pipeline between XDR and BIO, and I don't want traffic from XDR to them, or from them to BIO, to compete with the CPU's access to EIB. So, using the 0-3 top, 4-7 bottom labeling, I need three of the 4-7 SPEs.

That may not be possible if the reserved SPE and the defective/disabled one are both on that side of the chip.

So there's cases you can describe where a given layout would be advantageous, but would be possible on some PS3 systems and not others. So you have to go without.

I don't know how much of a difference affinity makes in the real world; the IBM research papers I saw suggested it was noticable but not huge.

 

[Arwin]

I am not convinced that it is even theoretically possible to implement affinity, at least to the levels described in some of the research papers from the IBM side.

Imagine, if you will, that I want to have three adjacent SPEs running a pipeline between XDR and BIO, and I don't want traffic from XDR to them, or from them to BIO, to compete with the CPU's access to EIB. So, using the 0-3 top, 4-7 bottom labeling, I need three of the 4-7 SPEs.

That may not be possible if the reserved SPE and the defective/disabled one are both on that side of the chip.

So there's cases you can describe where a given layout would be advantageous, but would be possible on some PS3 systems and not others. So you have to go without.

I don't know how much of a difference affinity makes in the real world; the IBM research papers I saw suggested it was noticable but not huge.

Ok, now that we've talked about this, established some of the basic facts, and I've thought about it some more, I'd like to point out the following:

1. As said earlier, there are currently no games developers even close to maxing out the capacity of the SPEs. The likelyhood that whatever they are doing with the SPEs is at this stage maxing out the EIB seems extremely unlikely. Therefore there are at this point not likely to be issues with games in that respect. This is also very likely why affinity is not discussed in the context of games - to use SPEs at all is enough of a challenge, and the performance people get out of them so far has been way more than they've needed, and even if they need more, it may never happen that bandwidth is enough of a concern for it.

2. Considering that any of the SPEs can be disabled, then choosing which SPE falls under hypervisor exclusive control can't be a random choice either. I'm fairly certain that there will be a fixed plan that has a fixed table which physically disabled SPE should go with which hypervisor controlled SPE. In fact, to make this random just seems more work. So at this stage, it seems to me extremely likely that they will, in fact, make sure that there are two pairs of three on either side of the chip, and affinity is at least theoretically possible.

 

[seebs]

Hmm, you have a good point. If they committed to a consistent pattern, then you could at least guarantee... hmm.

Yes. If the reserved-SPE is designed with the goal of leaving some affinity options open, you can always ensure that there is at least one pair on each "side" of the Cell (one pair between MIC and BIO, one between PPE and BIO) that could be used with affinity. The Cell SDK under Linux can't do affinity, but there's no reason that games in theory couldn't do it, if Sony supported it.

I think it's true that maxing out EIB is fairly unlikely, although it seems that the available bandwidth among the available SPEs, PPE, and so on, is enough that you could probably get into the range where you start seeing performance hiccups. After all, the 300GB/s bandwidth is theoretical; that doesn't mean that, if you're using 200GB/s, you can simply do another 100GB however you want.

So, in practice, it sounds like for most games, the practical loss is just "two SPEs", and it's only on the still-theoretical end that the EIB bandwidth questions could come up. It'll be interesting to see what devs are saying about it in about three years.

 

[ADEX]

I think there's bit of confusion here over exactly how a Hypervisor is defined as and that's leading to confusion.

The hypervisor in Cell is a mixture of hardware and software. The hardware, located in the PPE gives it an extra privilege level. The software (confusingly also called hypervisor) runs on the PPE.

Then you get this:

"On the PS3, one SPE is dedicated to the built-in Sony GameOS, also known as the hypervisor"

This is effectively changing the definition of hypervisor to include not only the hypervisor but also additional stuff which could be running anywhere.
It obviously doesn't mean the entire GameOS but probably the "security OS" section. The reserved SPE is probably doing some crypto stuff as part of this and this definition is including it as part of the hypervisor.

Hence the confusion...


Hmm, should really update my profile, I'm in the same place as Dean A these days.

 

[seebs]

Okay. I think I may see the issue.

Back in 2005, when I was doing a cram course in "what's a Cell and who cares" to do writing at IBM, they told us that the hypervisor usually ran on the PPE, but that the hardware was designed to allow it to hand privilege off to an SPE, and thereafter have the hypervisor on the SPE.

It sounds like you're saying that's just plain false, and in fact, people have mistaken the secure processing vault feature for a hypervisor.

Or, at the very least, that on the PS3, the hypervisor actually runs on the PPE, and the reserved SPE, while under its control, isn't actually running in hypervisor mode.

So, IBM has published statements like "The SPEs can even be used to support hypervisor functionality.", and I think everyone's just assumed that they meant that the PPE could hand-off the hypervisor magic bits to an SPE. It sounds like, from what you're saying, what it actually means is that an SPE, once walled off, can provide support services to a hypervisor.

It sounds as though SPEs have no privelege model internally, but:

Devices can utilize the signal notification feature of Cell to quickly and efficiently inform the SPE code of the completion of commands. More often than not, SPEs utilized in this model will run in privileged mode.​

This kind of thing sure makes it sound like at least one of the two privilege modes can be applied to an SPE, giving it access to privileged chunks of main memory.

So I guess we have two separate questions here. One is, if you had a completely unfettered Cell, and you could write any code you wanted on it, could you write code that resulted in one of the SPEs having access to all the privilege one registers and memory maps, and no running PPE code with access to them?

The other would be what the PS3 is doing, which seems to be that the privilege 1 mode code still exists on the PPE, and has carefully partitioned off one SPE to serve as sort of an unholy legion of the night or something.

I want to be sure I get this right, because I have an article on PS3 Linux going through editing, and if every other article on the topic has been wrong, getting it right will be worth the effort.

 

[ADEX]

So I guess we have two separate questions here. One is, if you had a completely unfettered Cell, and you could write any code you wanted on it, could you write code that resulted in one of the SPEs having access to all the privilege one registers and memory maps, and no running PPE code with access to them?

I'm not 100% sure but as far as I'm aware the SPEs cannot access the hypervisor registers. So you could set a SPE to have privilege 1 level and everything else privilege 2 but only the hypervisor on the PPE could set this up (or tear it down).

The other would be what the PS3 is doing, which seems to be that the privilege 1 mode code still exists on the PPE, and has carefully partitioned off one SPE to serve as sort of an unholy legion of the night or something.

I'd imagine it's decrypting stuff that is on game discs, you don't want any interference in this so it makes sense to lock it off. It can also be doing other stuff like recording TV streams.

I want to be sure I get this right, because I have an article on PS3 Linux going through editing, and if every other article on the topic has been wrong, getting it right will be worth the effort.

For a full clarification I'd ask at the IBM Cell forum, there's people there who know Cell in intricate (intimate?) detail.

 

[seebs]

Well, my original information came from IBM -- but mostly on the compilers side, so they may have just been confused. I'll poke around more.