Where's the PS3 hypervisor?

Discussion in 'CellPerformance@B3D' started by seebs, Nov 30, 2007.

  1. seebs

    Newcomer

    Joined:
    Nov 29, 2007
    Messages:
    44
    Likes Received:
    0
    Location:
    Minnesota
    (Thread was closed without a chance for me to ask this there, so...)

    This is pretty interesting. IBM's technical folks never caught that error, and if it's true, it's a pretty major shift from everything I was told when I was doing a lot of Cell stuff over on that end. Of course, this was IBM, not Sony.

    Are any more details available on this outside of NDA? Every article I could find said that the reserved SPE was being used for the hypervisor, including a number up on developerWorks, which normally vets things on this topic with the Cell experts.

    (It actually has no real impact on my original point, which is that having an SPU dedicated to unknown processing would be expected to have an impact on available bandwidth, but it's still very interesting.)
     
  2. patsu

    Legend

    Joined:
    Jun 25, 2005
    Messages:
    27,614
    Likes Received:
    60
    The counter-argument is if one SPU is already disabled, the EIB should have 1 SPU worth of spare bandwidth to handle added load.
     
  3. seebs

    Newcomer

    Joined:
    Nov 29, 2007
    Messages:
    44
    Likes Received:
    0
    Location:
    Minnesota
    Yeah, but the system has the capacity to saturate EIB, and I think it probably in theory still could even with two SPEs missing. Furthermore, even apart from the EIB question, there's the secondary question of main memory bandwidth; if the Black Ops SPE chews up main memory bandwidth, what can you do?

    I don't think it's likely to be a serious problem, but I have no idea how you'd establish a safe boundary for what you can rely on having access to, in the absence of documentation that I wouldn't exactly expect to get.

    Sadly, I'm not likely to get access to a non-Linux Cell SDK, so I don't know what the affinity options and the like are for the Game OS side of things.

    Mostly, though, I'm curious. Everything I've previously read says that the PS3 has one SPE dedicated to running the hypervisor, and the Cell docs were very clear on the fact that the hypervisor could live on an SPE if it wanted to. Now someone with reasonable credibility says that, in fact, the hypervisor runs on the PPE, and the SPE in question is running unspecified "security stuff" -- which is a very good description in some ways of what the hypervisor appeared to be doing, such as filtering access to hardware to keep user-written code from getting uppity. :)
     
  4. patsu

    Legend

    Joined:
    Jun 25, 2005
    Messages:
    27,614
    Likes Received:
    60
    Only 1 is disabled. All remaining 7 can use the EIB bandwidth. Even the locked down SPU may be doing some needed work.

    That's like asking what if one of the Xenon cores or Intel cores eat up all the bandwidth. It's not going to happen unless normal circumstances.
     
  5. seebs

    Newcomer

    Joined:
    Nov 29, 2007
    Messages:
    44
    Likes Received:
    0
    Location:
    Minnesota
    Right.

    I was just pointing out that the "absent" SPU doesn't guarantee free bandwidth. From another thread, someone said two SPEs could in theory saturate XDR.

    I would hope not. Is there any kind of committed spec available for the maximum bandwidth usage of the Black Ops SPE, so developers can reliably assume that the rest is available?

    I've been trying to find this stuff out asking people on PS3Forums, where I ended up because I was trying to find out why a computer display wasn't working on the PS3 (HDCP, it turns out), and, uhm, it's not a world-class source for detailed technical information on the guts of the PS3. I've mostly been going off bits and pieces of the more-generic Cell docs.
     
  6. patsu

    Legend

    Joined:
    Jun 25, 2005
    Messages:
    27,614
    Likes Received:
    60
    You can saturate main memory bandwidth on other CPUs too. Why concern for Cell ?

    The devs will probably have the necessary info.
     
  7. seebs

    Newcomer

    Joined:
    Nov 29, 2007
    Messages:
    44
    Likes Received:
    0
    Location:
    Minnesota
    This all grew out of a side discussion over on PS3 forums about the ability to get "the most" out of a system.

    My argument was that there were going to be more unknowns, and thus, more need to stay conservative, with Cell than there were on the PS2.

    (BTW, several people there assumed I was somehow implying that the PS3 couldn't keep up with the 360, but I never said, or believed, anything of the sort.)

    That's always the question. I have been wondering, given some of the delays and performance problems, whether some of that information is too secret. :)
     
  8. patsu

    Legend

    Joined:
    Jun 25, 2005
    Messages:
    27,614
    Likes Received:
    60
    I am not a dev and so cannot comment. In general, if there are more factors/variables/unknowns, they can all swing both ways... you can be more conservative, or more aggressive in each of the unknowns depending on your peculiar needs. I don't think there is any hard and fast rules.
     
    #8 patsu, Nov 30, 2007
    Last edited by a moderator: Nov 30, 2007
  9. seebs

    Newcomer

    Joined:
    Nov 29, 2007
    Messages:
    44
    Likes Received:
    0
    Location:
    Minnesota
    Huh? How can unknowns "swing both ways"?

    If I have a requirement of a steady 30fps, that means I have a fixed budget of time to render each frame. If I have an unknown factor, the only way I can be sure of meeting that budget is to allow for the worst case. If I'm "more aggressive", then I'll drop frames...
     
  10. DeanA

    Newcomer

    Joined:
    Oct 26, 2005
    Messages:
    244
    Likes Received:
    36
    Location:
    Cambridge, UK
    This is key. The CPU and hypervisor are implementations that can be specific to PS3. There's no one hypervisor for CELL, so there may be changes as to where it runs (and what it does) between platforms.

    Obviously the issue of backwards compatibility (with previous PS3 revisions, or titles that were built against earlier PS3 firmware) would, in my view, mean that any reserved SPU is not likely to be suddely re-tasked to something that required the use of large amounts of bandwidth during game execution. Old games would choke, and that would make Baby Jesus cry.

    Dean
     
  11. seebs

    Newcomer

    Joined:
    Nov 29, 2007
    Messages:
    44
    Likes Received:
    0
    Location:
    Minnesota
    Well, yes.

    But everything I've seen (and a few things I've written that were checked by people who supposedly knew) says that the 7th SPE on the Cell is where the hypervisor lives on the PS3. Much was made of how important it was to the PS3 that it was possible to have one of the SPEs do the hypervisor, instead of running it on the PPE.

    Do you have any citations or links I could point to? I have an article on PS3 Linux in editing, and given the number of pages I've seen claiming that the 7th SPE is running the hypervisor, if I claim otherwise without published documentation, I'm going to get DOZENS of corrections, and my editor will complain.

    This is a good point, and obviously, I wouldn't expect Sony to release a firmware update in which the OS suddenly starts doing bandwidth-intensive distributed computing using that SPE. However, it does seem as though it would mean that, when designing a game, I'd want to try to leave a chunk of headroom for possible unanticipated actions from there.

    I'm still thinking through the question of how to try to figure out what the impact of this is on bandwidth-tuning hackery. I know that some of the Cell SDK apps show off tricks that depend on being able to arrange SPEs in particular patterns -- and some of this isn't available, at least on the PS3 Linux side of the SDK.
     
  12. DeanA

    Newcomer

    Joined:
    Oct 26, 2005
    Messages:
    244
    Likes Received:
    36
    Location:
    Cambridge, UK
    Links stating that the 7th SPU is used for security stuff? Or regarding positioning of the hypervisor? For the security side, I can't find any public announcement, although pretty much all the PS3 overviews mention it.. like this one at PS3 Forums. For the hypervisor, try the basic architectural overview present for PS3 linux , linked here. I know the document doesn't explicitly state that the hypervisor is running on the PPU, but if you think about the implications of being able to manage/virtualise device resources (via syscalls, interrupts and so on), it becomes fairly clear that pushing device and address I/O management up to an SPU isn't a particularly great way of doing things.

    I don't think you would.. for a couple of reasons. First up (and all IMHO) game designers really don't care about memory bandwidth. Some wouldn't even know what it is! Second, the PS3 devkits have the same SPU setup as retail units (7 active SPUs, 1 of 'em reserved). Performance is the same, so you work to what you have on the devkit. For all you & I know, the 7th SPU could already be configured to 'steal' a small amount of bandwidth in a consistent manner, effectively giving headroom for future changes in this area. Who can say?

    Cheers,
    Dean
     
  13. seebs

    Newcomer

    Joined:
    Nov 29, 2007
    Messages:
    44
    Likes Received:
    0
    Location:
    Minnesota
    Well, what's the distinction? I thought the purpose of keeping everything under a hypervisor all the time WAS security.

    And most of them call it "the hypervisor". That's what made me so very surprised to find out that it might not be.

    Obviously not, but after the stress placed on telling me that it was important that we cover that it was possible to do so, during the fall 2005 Cell article blitz for developerWorks, I just assumed that the theory was that the added security of keeping the hypervisor separated was deemed worth it. After all, the SPU wouldn't be exceptionally fast at it, but the hypervisor doesn't really have to do very much, and the SPUs have the ability to initiate DMA requests too.

    This could just be one of those things where everyone assumed that, since there was an SPU devoted to Something Unknown To Do With Security, and the hypervisor could be offloaded, and had sort of to do with security, that must be what's running on the 7th SPU. I haven't been able to find an actual primary source either way. (And, with your sig in mind, I don't know whether you are wearing your Primary Source hat.)

    I guess what I'd really, really, like to do is track down something official but not NDA'd that explicitly asserts that the hypervisor is running on the PPE, and that the "reserved" SPE isn't running hypervisor code. It seems reasonable enough, but if I go around saying "we should contradict every published source I can find, because some guy on a forum said so; no, it's okay, he's really very knowledgeable", my editor will stab me repeatedly in the face. (It's an occupational hazard.)

    I'd say I don't believe you, but I've been doing embedded lately, and I've found that there are embedded developers who don't know about code size. This might explain occasional complaints of frame dropping problems, though -- if that usage isn't utterly consistent, maybe that's why some games occasionally drop a frame. I could see developers failing to realize that they had to account for that.

    That would actually make a lot of sense, especially if it were documented what the approximate bandwidth usage would be.
     
  14. Arwin

    Arwin Now Officially a Top 10 Poster
    Moderator Legend

    Joined:
    May 17, 2006
    Messages:
    17,674
    Likes Received:
    1,194
    Location:
    Maastricht, The Netherlands
    Have you visited ps2dev.org yet? You could PM one of the hardcore guys in this thread (Glaurung or particularly IronPeter) and see if they have come across anything that indicates what is running where ... I'm not sure that they have, mind, but what they are doing is pretty hardcore (and interesting, I love the post on the second page on OpenGL vs directly accessing RSX)

    http://forums.ps2dev.org/viewtopic.php?t=8364&postdays=0&postorder=asc&start=210
     
  15. seebs

    Newcomer

    Joined:
    Nov 29, 2007
    Messages:
    44
    Likes Received:
    0
    Location:
    Minnesota
    Thanks for the link. Hadn't been there yet. So far, just about everything I've gotten has come from IBM paperwork -- Cell SDK and so on, and a lot of research materials I got from dW back in 2005. Anything on the PS3 implementation has been, of necessity, guesswork.

    But, as an example:

    http://www.ibm.com/developerworks/power/library/pa-soc12/index.html

    "On the PS3, one SPE is dedicated to the built-in Sony GameOS, also known as the hypervisor, and one SPE is disabled to increase PS3 yield and lower cost. So, on the Yellow Dog Linux PS3 platform used in this article, the PPE has six SPEs to which it can offload over the EIB."

    Similarly:

    http://eric_rollins.home.mindspring.com/ray/ray.html

    "On the Playstation 3 seven of eight SPEs are active -- the last failed verification (if it passed it went into an IBM blade system instead :). One of these is running a dedicated Hypervisor, so 6 SPEs are available to a Linux application."

    I am still not quite clear on why my assertion of something that is confirmed by every document I have ever found that expresses an opinion on the topic was grounds for thread closure, but I'm a newbie here and not quite clear on how they run things. :)
    I'd love to see something published by IBM, Sony, or Toshiba that says otherwise; it'd be something of a scoop!
     
  16. Arwin

    Arwin Now Officially a Top 10 Poster
    Moderator Legend

    Joined:
    May 17, 2006
    Messages:
    17,674
    Likes Received:
    1,194
    Location:
    Maastricht, The Netherlands
    I'm not knowledgeable enough to tell anything about this, but this overview of some of the hypervisor function calls may also help determine if that matches up with what would be possible if it resided on an SPE:

    http://wiki.ps2dev.org/ps3:hypervisor

    EDIT: Also, when I read this official Sony announcement,

    http://cell.scei.co.jp/index_e.html

    I read as one of the prime features:
    That seems to suggest that an SPE can be assigned to be under hypervisor control much more than that the hypervisor actually resides on the PPE, but has an SPE available to it exclusively and securely to perform security features like encryption (for which an SPE seems to me very suitable ;) ) and run other stuff and code that you don't want to be visible.

    It also explains the confusion about the SPE running the hypervisor. In a strict sense, it is under control of the hypervisor and protected, so the distinction is small enough to account for the current way it has generally been reported on.

    Not that I doubt DeanA's 'suggestions' for a minute, but I'm as curious and information hungry as the next guy. ;) (ok, maybe a tiny bit more)
     
    #16 Arwin, Nov 30, 2007
    Last edited by a moderator: Nov 30, 2007
  17. DeanA

    Newcomer

    Joined:
    Oct 26, 2005
    Messages:
    244
    Likes Received:
    36
    Location:
    Cambridge, UK
    For what it's worth, this would be my take too.

    Dean
     
  18. one

    one Unruly Member
    Veteran

    Joined:
    Jul 26, 2004
    Messages:
    4,823
    Likes Received:
    153
    Location:
    Minato-ku, Tokyo
    Technically I don't think they contradict each other, the hypervisor itself runs multiple services/daemons/OSs concurrently in each partition with assgined resources in PPE or SPE. From within contexts of these services, including a Linux kernel or GameOS, they can't see what others are doing in the outside world they can't touch so it looks as if it were one whole hypervisor - in other words they can't even know they are running on a hypervisor without accessing hypervisor-provided APIs, let alone what other services are. This is the very definition of virtualization. As these articles are written from the viewpoint of Linux users, it doesn't matter if they call everything they can't touch as a blanket hypervisor. But if you look at it from the context of the hypervisor and Cell, it's not running on a SPE, that SPE belongs to just another virtual partition under the control of the hypervisor. The hypervisor doesn't need an SPE to run.
     
  19. seebs

    Newcomer

    Joined:
    Nov 29, 2007
    Messages:
    44
    Likes Received:
    0
    Location:
    Minnesota
    I think it's clear that the hypervisor could have been anywhere. The impression I got was that "hypervisor calls" consisted of dropping data somewhere for the security SPE to Make Things Happen; that code running on the PPE simply wasn't allowed to do the hypervisor stuff once the security SPE was up.

    Hmm.

    Actually, it occurs to me that the RSX pseudo-crack under Linux is probably evidence of some sort on the topic. The hypervisor is the one doing the work; the question is, is the PPE simply asking that SPE to Do The Magic, or is the PPE (not under user control) doing some magic, with the SPE doing the rest?

    Anyway, thanks for the clarification. I think I'll talk to my editor about clearing this up in that article before it goes live. :)
     
  20. patsu

    Legend

    Joined:
    Jun 25, 2005
    Messages:
    27,614
    Likes Received:
    60
    Yes... that would form the lower bound. I don't see the unknowns as some sort of "law" that caps PS3/Cell's performance throughout its lifecycle. Rather, the "awareness" is only a starting point. As time goes by, more details about the unknown may be revealed (via Sony or experimentation) :) people can then optimize around it for their own tasks. Even the missing affinity can still be implemented in the future since it's likely a system software bullet. However given that developers claim they only use up to 50% of SPU, I don't think we will see this (yet).

    For non-critical task like background download, it is conceivable that the downloader can be aggressive/speculative but extremely low priority to take up slacks from the unknown.

    Finally, there are different economic models to extend PS3 power. e.g., Given the unknown, just throw in more cheap PS3s. Afterall a live petaflop network is already solving F@H issue everyday.
     
Loading...

Share This Page

  • About Us

    Beyond3D has been around for over a decade and prides itself on being the best place on the web for in-depth, technically-driven discussion and analysis of 3D graphics hardware. If you love pixels and transistors, you've come to the right place!

    Beyond3D is proudly published by GPU Tools Ltd.
Loading...