SCE building PS3/Cell-based cluster with Terra Soft; Yellow Dog Linux distro on Cell

Titanio

Legend
I posted this on GAF a little while ago, and am slightly surprised it hasn't made its way here yet, so I guess I'll just post it here too myself ;)

The formal announcement is supposedly due tomorrow, but HPCWire got whiff of it early:

http://www.hpcwire.com/hpc/967146.html

Tomorrow, Terra Soft will officially announce the construction of the world's first Cell-based supercomputing cluster.

In the fall of '05, Terra Soft was contacted by Sony Computer Entertainment, Inc. (SCEI) to develop and manage a supercomputing cluster built upon the IBM Cell Broadband Engine and the Linux OS. This spring, Terra Soft was contracted by Sony and in August completed the construction of a 3000 sq-ft supercomputing facility capable of housing 2400 1U systems. In this remodeled extension to the Loveland, Colorado headquarters, Terra Soft will construct a test cluster and a substantially larger production cluster, dubbed "E.coli" and "Amoeba" respectively.

Terra Soft will use the test cluster "E.coli" to conduct advanced software development, optimization, and testing with emphasis on Y-HPC and Y-Bio applied to the Cell Broadband Engine. The production cluster "Amoeba" will be made available to select University and Department of Energy laboratories to further life sciences research.

The clusters will incorporate, in part, Cell-based PS3 systems. The Cell Broadband Engine provides a "1 + 8" multi-core processing environment, enabling optimized code to function at a superior level of performance over traditional single or dual core CPUs. With all 8 cores on a single chip, the code processes do not lose performance by dropping down to the memory bus as with historic, multiple CPU configurations.

Glen Otero, Director of Life Sciences Research for Terra Soft Solutions explains, "This cluster represents a two-fold opportunity: to optimize a suite of open-source life science applications for the Cell processor; to develop a hands-on community around this world-first cluster whereby researchers and life science studies at all levels may benefit. Once up and running with our first labs engaged, we will expand the community through invitations and referrals, supporting a growing knowledge base and library of Cell optimized code, open and available to life science researchers everywhere."

Lawrence Berkeley National Lab is working with Terra Soft to optimize a suite of life science applications. Los Alamos and Oak Ridge National Labs are also engaged, with select universities coming on-board early in 2007. Terra Soft is working to optimize the entire Y-Bio bioinformatics suite.

Thomas Swidler, Sr. Director of Research & Development at SCEI states, "This cluster is for Sony a means of demonstrating the diversity of the PS3, taking it well beyond the traditional role of a game box. While we are not in the business of competing for the Top500.org nor building cluster components, this creative use of the PS3 beta systems enables Sony to support a level of real world research that may produce very positive, beneficial results."

More reporting:

http://www.coloradoan.com/apps/pbcs.dll/article?AID=/20061009/NEWS01/61009002

http://grids.itmanagersjournal.com/gridsclusters/06/10/09/1345209.shtml?tid=66

Terra Softs site (+ video of the cluster site's construction):

http://www.terrasoftsolutions.com/

For those that don't know, Terra Soft is a (very) small company who previously worked on Power-based Linux distros for Apple products primarily. Now that Apple has switched to Intel, Terra Soft seems to be embracing Cell as its future.

Interesting news for a number of reasons, I guess. It highlights Yellow Dog as perhaps the most likely 'standard' PS3 Linux distro given this deal, since it's there and ready basically. It also is the beginning of dependent business forming around Cell, which is fairly significant. It'll also be the first Cell-based cluster/supercomputer, being ready by the end of the year (the IBM Los Alamos one isn't due to be finished so soon, I don't think). And then I guess there's the whole development of SCE moving into the business of apparently hiring out compute power from its own cluster (as cited above, to Lawrence Berkeley, Oak Ridge and Los Alamos + more).
 
In the fall of '05, Terra Soft was contacted by Sony Computer Entertainment, Inc. (SCEI) to develop and manage a supercomputing cluster built upon the IBM Cell Broadband Engine and the Linux OS. This spring, Terra Soft was contracted by Sony and in August completed the construction of a 3000 sq-ft supercomputing facility capable of housing 2400 1U systems. In this remodeled extension to the Loveland, Colorado headquarters, Terra Soft will construct a test cluster and a substantially larger production cluster, dubbed "E.coli" and "Amoeba" respectively.

I take it the cluster will be used to simulate optimal spinach irrigation systems...
 
It's called e-coli because that was the first life form to have its genetic code sequenced, apparently :p Always a method to madness..
 
WOW!
The amazing thing is that even if two-thirds of the performance of the PS3 cluster went up the chimney, at $599 for a top-end PS3, a 30 teraflops cluster for under $300,000--and one that you can play games on when the super isn't busy--is a remarkable idea.
So it would seem that the PS3 would be an ideal kind of supercomputer node. But what Terra Soft and its partner, Mercury Computer, believe is that a network of PS3s might be an excellent front end to a giant, high-speed cluster built from Cell-based blade servers that have InfiniBand interconnections. The same software would run on both machines, of course, and would be supplied by Terra Soft.

Damn can anybody else see Sony trying to sell companies retail PS3s next year for research like this to Universities? I mean their sales will be a lot higher from a hardware stand point for marketing, but also using CELL in the PS3s for research like this is a cheap way to introduce people to CELL programming.

If the people like it Sony/IBM can sell them to more expensive CELL blade. Would Sony actually think about doing this?

Thoughts.
 
Damn can anybody else see Sony trying to sell companies retail PS3s next year for research like this to Universities?

I am pretty sure based on the Console Model of pricing, Sony would much prefer that consumers also desiring software buy PS3 units. Now a couple for PR, fine, but if 10,000 launch units go to these clusters that is cash out of Sony's pocket and lost potential revenue from games and movies.
 
I am pretty sure based on the Console Model of pricing, Sony would much prefer that consumers also desiring software buy PS3 units. Now a couple for PR, fine, but if 10,000 launch units go to these clusters that is cash out of Sony's pocket and lost potential revenue from games and movies.

The main problem with PS3 was Blu-ray drives not being manufactured in large enough quantities.

If this is a supercomputing cluster, there will be no such adversity (nor there is the need for built-in WiFi, card readers, HDMI/component, RSX, laptop hard drives, or even designer-casing).

Plus, advanced computing stuff tends to be payed very generously by who really needs it, so there's no trouble in getting the money back -and then some-.
 
Due to power, space and application/memory requirements, the units could be like the devkits (instead of retail PS3).

EDIT: And yes, Blu-ray may not be needed.
 
The main problem with PS3 was Blu-ray drives not being manufactured in large enough quantities.

No, the main problem with using PS3 retail units, from Sony's perspective, is cost. Console makers routinely lose money on the units--even years after launch. Why would Sony want researches to buy hoards of PS3s (at a loss to Sony) when they won't be buying software (which makes them money)?

The lack of Blu-ray diodes is inconsequential in that their scarcity is only one factor in the retail price and based on their own projected losses the PS3 will be sold at a loss.

I am not saying cheap PS3-like units could not be produced and sold for the purpose of supercomputing clusters, only that the using retail units (as the question was) is not something Sony will push.

That said, that didn't prevent people from using Xbox consoles for media centers or a number of people experimenting with 16 Xbox Linux clusters... but again, not something MS pushed for the same above reasons.
 
No, the main problem with using PS3 retail units, from Sony's perspective, is cost. Console makers routinely lose money on the units--even years after launch. Why would Sony want researches to buy hoards of PS3s (at a loss to Sony) when they won't be buying software (which makes them money)?
.


One reason was explained by the article. It would be to introduce them to CELL hardware and an opening to purchase the real deal. The real deal being the CELL blade that can cost up to tens of thousands of dollars.

You know practice and test your code at amazing speeds for a year. Spend $300,000 and get 50 teraflops of power. Can't disagree with that.
 
No, the main problem with using PS3 retail units, from Sony's perspective, is cost. Console makers routinely lose money on the units--even years after launch. Why would Sony want researches to buy hoards of PS3s (at a loss to Sony) when they won't be buying software (which makes them money)?

The lack of Blu-ray diodes is inconsequential in that their scarcity is only one factor in the retail price and based on their own projected losses the PS3 will be sold at a loss.

I am not saying cheap PS3-like units could not be produced and sold for the purpose of supercomputing clusters, only that the using retail units (as the question was) is not something Sony will push.

That said, that didn't prevent people from using Xbox consoles for media centers or a number of people experimenting with 16 Xbox Linux clusters... but again, not something MS pushed for the same above reasons.

You're assuming that that version of Yellow Dog Linux (and computing time on the cluster) will be free.
I'm sure that's not the case.

They will be renting (not buying) software + hardware, and that on a limited scale.
After all, this is mainly for demonstration purposes. It's not a cluster vending operation, as they said.
 
Last edited by a moderator:
The other one should've been salmonella not ameoba...

I wonder if the PS3's could be used a'la flashmob...

I wonder how much computing will it be if it's done globally...
 
To clarify, it is reported in another article that they're using PS3 beta kits recycled from developers, as they phase them out in favour of final kits or whatever.

I wonder how much computing will it be if it's done globally...

It could be used as a testing ground for rolling out distributed apps across a wider network of PS3s, which is interesting to consider. Obviously it's already happening with F@H, but Sony could be eyeing its own generic software infrastructure for supporting a wider class of applications. The only difference would be the amount of bandwidth between each node on a global network versus a local one as in this cluster - but there are obviously applications for which that would be relatively unimportant.

It will be interesting to see, though, if institutions favour renting time at one of these clusters versus building their own using $500 boxes. Obviously using Terrasoft's software you get a leg-up, and you get help etc. but some that have already developed distributed apps may be comfortable rolling their own using (hopefully) whatever distro is made available on PS3.
 
Last edited by a moderator:
You know practice and test your code at amazing speeds for a year. Spend $300,000 and get 50 teraflops of power. Can't disagree with that.

Well, if FLOPS is all you want, then sure, it's a great machine. I'm sure there are more factors to take into consideration than just the OMG TERAFLOPS ROXORZ?!!!1 figures.
 
It will be interesting to see, though, if institutions favour renting time at one of these clusters versus building their own using $500 boxes.
I would assume support etc to be great when using the cluster, and totally nonexistant when using home consoles. :p That might be worth much more in the end than any purchase price.
 
Thread pruned for now. The whole E.Coli discussion is absolutely off-topic and will not be tolerated within this thread. If you want to discuss the aspects of that, I suggest opening a thread in RPSC.
 
Damn can anybody else see Sony trying to sell companies retail PS3s next year for research like this to Universities? I mean their sales will be a lot higher from a hardware stand point for marketing, but also using CELL in the PS3s for research like this is a cheap way to introduce people to CELL programming.
They'll only agree to those prices if the companies agree to buy at least 3 Sony-published titles apiece for each machine as well. :p

But hey, then the researchers are at least guaranteed to NEVER get bored!
 
To clarify, it is reported in another article that they're using PS3 beta kits recycled from developers, as they phase them out in favour of final kits or whatever.

That's just awesome... environmentally friendly and incredibly practical.
 
University of Manchester to adopt Cell Broadband Engine (Cell BE) processors for
bioinformatics, molecular modelling and engineering programs.

Researchers will use a prototype of IBM's BladeCentre QS20 system, which incorporates super-fast Cell Broadband Engine (Cell BE) processors, to handle complex bioinformatics, molecular modelling and engineering programs.

By employing the IBM system, the University is giving staff and students the opportunity to use world-leading, high-performance computer facilities to drive their research forward.

"We are early adopters of the IBM Cell BE system because it has the potential to give us significantly improved performance, take up less space, and consume less power," said Terry Hewitt, Director of Research Computing at The University of Manchester.

"High performance computing systems built from systems based on the Cell Broadband Engine have the potential to change the economics associated with supercomputing.

Source: University of Manchester

http://www.physorg.com/news79788433.html
 
I really, really hope that Sony is going to make good on that Linux promise for the retail unit. It's gonna be a huge letdown for me if they screw that up. I'm sure I can think of understandable reasons for them to let it go, but I'd be very unhappy if they did regardless.
 
Back
Top