building up a render farm

[MfA]

http://www.rackmount.com/Rackmt/ATXBladeS100.htm

You might need this one for EATX motherboards :

http://www.rackmount.com/Rackmt/ATXBladeS800.htm

Regardless, at 325$ for casing+PSU you hardly save anything compared to the supermicro I linked earlier taking into account you still need heatsinks ... actually using the heatsinks from their site, which are hugely expensive, you would have to spend more money!

The cheapest solution is buying 2U or 4U EATX rackmount cases, which will fit relatively cheap heatsinks (Shuriken Rev. B for instance) if you buy the 100$ versions, 30$ heatsinks and really cheap racks you could save around 300$ per node compared to the supermicros.

PS. can't judge their RAID solutions ... but since you are doing so much yourself I'd just get some cheap rackmounted drive cage and DIY a raid with a LSI MegaRAID SAS 9211-8i (only does RAID10, but as I said earlier that is the superior RAID to start with).

 

[-tkf-]

Here is what i am doing right now.. this is from a pure Hardware view, and my savior is the new motherboards with build in graphics..

i5 Motherboard from intel with graphics
i5 CPU @ 2.66 GHZ
4GB Ram
250 GB Harddrive

All this fits in a cheap 2U rack Cabinet making it easy and cheap to get into racks.

I tried the "corporate" way but it´s way to costly and i can buy the needed spare parts for this.

Since our jobs are easy to manage and a dead "node" wont kill us we don´t really care that much about reliability on these number crunchers. We focus the reliability on the Machines that handle the Jobs (dual brand name Pizza servers).

If you need fast storage i got some stuff on the drawing board as well, but i our case a 1GB network with a 1GB linkup to the jobs manangers is enough.

You could go for some HP switches that support their 10Gbit stuff and add a storage server to your solution, i got something on paper for that as well.

We expect 213 watt pr node when our i5 machines is running on full speed.

 

[Silent_Buddha]

Weren't you going with Quad cores before? With Clarksdale you'll be moving down to dual core, which means you'd need double the number of machines you previously used? It's the main reason that while Clarksdale is interesting, I've discarded it from consideration for my own uses.

Regards,
SB

 

[-tkf-]

Weren't you going with Quad cores before? With Clarksdale you'll be moving down to dual core, which means you'd need double the number of machines you previously used? It's the main reason that while Clarksdale is interesting, I've discarded it from consideration for my own uses.

Regards,
SB

Where did he mention anything about dual cores?

 

[MfA]

Weren't you going with Quad cores before?

There are quad core i5's at 2.66 GHz (i5-750) so I assume that's the one tkf meant, but the OP wanted dual processor nodes since his software license is per node ... so it ain't an option.

PS. I still don't see the use for 10 Gb ethernet, Infiniband is faster and cheaper ... what's the point?

 

[Silent_Buddha]

Ah, I'd forgotten about the i5 750.

Except for one thing. i5 750 doesn't have onboard video on any MB's.

So again he's still stuck with a dual core Clarksdale. The only CPU with onboard video for the 1156 platform.

Ergo, he's either using a dual core Clarksdale with onboard video or he's using a quad core Lynnsfield in which case he doesn't have onboard video making his configuration a little nonsensical.

Both the H55 and H57 chipsets only add a display path for the on package video of Clarksdale CPUs, they don't add an integrated video option.

Where did he mention anything about dual cores?

See above.

Regards,
SB

 

[rgbaguy]

You might need this one for EATX motherboards :

http://www.rackmount.com/Rackmt/ATXBladeS800.htm

Regardless, at 325$ for casing+PSU you hardly save anything compared to the supermicro I linked earlier taking into account you still need heatsinks ... actually using the heatsinks from their site, which are hugely expensive, you would have to spend more money!

The cheapest solution is buying 2U or 4U EATX rackmount cases, which will fit relatively cheap heatsinks (Shuriken Rev. B for instance) if you buy the 100$ versions, 30$ heatsinks and really cheap racks you could save around 300$ per node compared to the supermicros.

PS. can't judge their RAID solutions ... but since you are doing so much yourself I'd just get some cheap rackmounted drive cage and DIY a raid with a LSI MegaRAID SAS 9211-8i (only does RAID10, but as I said earlier that is the superior RAID to start with).

Thanks for the reply MfA! Apologies for the late reply, been busy. Cool, cool, I'm just so overwhelmed with all the options for hardware etc. It's pretty exhausting work. Am looking for EATX chassis now, heres a starting point from a search on google:

http://www.google.com/products?oe=U...QwAA&price1=&price2=150.00&lnk=prsugg&show=dd

http://store.chassisoutlet.com/

http://sewelldirect.com/Norco-RPC-9...=cse&cvsfa=1306&cvsfe=2&cvsfhu=53572d35373637

http://www.chassis-plans.com/rackmount_chassis.html

I will do more searching and find better prices, then will contact for a quote. You were right about the Rackmount chassis, Rackmount replied with a quote. They said the S5500WB board would only fit into the ATXBladeS800. The quote was a big improvement, 13750 US for 5 x ATXBlade S800's, giving the full 40 blades we require. Each blade comes with a 400 W PSU. They also linked me to heatsinks, US 35 each from here:

http://www.dynatron-corp.com/en/product_detail_1.aspx?cv=1-4-153&id=6&in=0

But will heed your advice and look for cheaper EATX chassis options.

Will also look around for a drive chasis as well.

Here is what i am doing right now.. this is from a pure Hardware view, and my savior is the new motherboards with build in graphics..

i5 Motherboard from intel with graphics
i5 CPU @ 2.66 GHZ
4GB Ram
250 GB Harddrive

All this fits in a cheap 2U rack Cabinet making it easy and cheap to get into racks.

I tried the "corporate" way but it´s way to costly and i can buy the needed spare parts for this.

Since our jobs are easy to manage and a dead "node" wont kill us we don´t really care that much about reliability on these number crunchers. We focus the reliability on the Machines that handle the Jobs (dual brand name Pizza servers).

If you need fast storage i got some stuff on the drawing board as well, but i our case a 1GB network with a 1GB linkup to the jobs manangers is enough.

You could go for some HP switches that support their 10Gbit stuff and add a storage server to your solution, i got something on paper for that as well.

We expect 213 watt pr node when our i5 machines is running on full speed.

Thanks for this, I'm afraid we can only opt for dual socket blades. I believe gigabit ethernet is plenty enough for our needs, since we will be installing SATA drives, having an incredibly fast network isn't really going to help things as the main bottleneck is the drive speed. I don't really think it's worth investing in SAS drives as the cost difference is quite significant.

400 W PSU should be fine for dual socket E5520's, each uses 80 W of power. I can't find power stats for the motherboard, RAM, drive etc. 400 W should be more than enough to handle the upper bound on power usage per blade. What do you guys reckon?

There are quad core i5's at 2.66 GHz (i5-750) so I assume that's the one tkf meant, but the OP wanted dual processor nodes since his software license is per node ... so it ain't an option.

PS. I still don't see the use for 10 Gb ethernet, Infiniband is faster and cheaper ... what's the point?

Yeah, dual socket is really our only option. On the network, gigabit should be fine. Unless anyone would disagree, I'm very open to input, I mean I'm learning as I go

Thanks for the very helpful replies guys. They have helped me a great deal, and am getting there

 

[rgbaguy]

The boards I'm looking at, S5500WB are rack mount 1U, so doesn't it make more sense to get a 1U blade chassis, or am I just lost again?

 

[rgbaguy]

So far, the only 1U chassis I have found that fit the S5500WB (12"x13") are:

http://www.chassis-plans.com/rackmount-industrial-pc-computer-chassis/F1-1u-rackmount-chassis.html

http://www.chassis-plans.com/rackmo...uter-chassis/F2-1u-rackmount-atx-chassis.html

http://www.chassis-plans.com/rackmo...-chassis/F8-1u-rackmount-atx-pc-computer.html

http://www.chassis-plans.com/rackmo...chassis/F11-1u-rackmount-atx-pc-computer.html

 

[rgbaguy]

These look good, just have to find PSU, 1U 400 W should do the trick. Also need to get other parts, like fans etc. I'll give these and the guys above a call for a quote.

http://store.chassisoutlet.com/1u5baramochi.html

http://store.chassisoutlet.com/1uea12x13mbr.html

What you guys reckon?

 

[rgbaguy]

Hmm, PSU going for US 85. Might be kinda hard to keep the blade chassis, fans and PSU under US 300?

1U 400 W PSU: http://store.chassisoutlet.com/tc1u40dufanp.html

 

[Cheezdoodles]

Buy 100 PS3's for 250 a pop.

 

[-tkf-]

There are quad core i5's at 2.66 GHz (i5-750) so I assume that's the one tkf meant, but the OP wanted dual processor nodes since his software license is per node ... so it ain't an option.

PS. I still don't see the use for 10 Gb ethernet, Infiniband is faster and cheaper ... what's the point?

Oops pr node, then you need more cores... grr

My idea with the 10GBit network is pretty simple and just an idea without actually knowing the op needs

My storage solution is not related to our 20 server cluster, it´s related to Storage of video and that kind of stuff but it´s easy to combine.

Again with focus on cheap, i will build a 16 Disk server with a 10Gbit nic and connect that to a 10gbit port on a switch where all clients are connected via gbit. Simple and plain and should provide the clients with enough throughput and they can just rely on the build in nic on the motherboard.

 

[rgbaguy]

Buy 100 PS3's for 250 a pop.

Ah if only it were as simple as that

 

[-tkf-]

Maybe wait it out until the i7 with 6 cores arrive?... if you can

Did you do some math on the cost on license vs hardware costs? How much is one license vs cheaper hardware.

 

[rgbaguy]

Maybe wait it out until the i7 with 6 cores arrive?... if you can

Did you do some math on the cost on license vs hardware costs? How much is one license vs cheaper hardware.

A 6 core i7 isnt going to beat dual socket 4 core xeon for software costs But I'll look into this, perhaps a higher spec i7, definitely going to cost less than an equivalent Xeon, and could give more CPU power. What do you think?

Yeah, I have done numbers on the software, still working on the hardware, looking for a cheaper board as well. Maybe I can even adjust the Xeon's, maybe to a E5502 for example.

I will work with rand figures, it's easier for me.

On the software, one mental ray standalone license comes to ~ R 5400, an equivalent Maya 2010 batch license costs ~ R 6300 (theoretically).

So the cost for 80 sockets mr standalone: ~ R 432 000.

The cost for 80 sockets batch: ~ R 252 000.

On top of that, on the mental ray standalone, that doesn't include our Maya licenses, while the 2nd option does. Huge savings on software.

On the hardware front, I'm pretty sure I haven't optimized costs for the hardware yet. Dual socket is definitely the way to go. On the hardware, sure, the E5520 costs more, but the gains in performance is around 143 % over the E5502 (perhaps more, I'm no hardware expert), which is think justifies spending roughly 92 % more as the hardware will serve for quite a long time, and will serve well for more computationally expensive jobs. Some people might think this crazy, I'm open to input.

I'll however look into the E5502 as the lower bound and the E5520 as the upper bound and run some numbers. I just wish I could actually test this hardware. I looked at passmark a while ago to do some comparisons, I can't look right now don't have flash installed on this machine.Any suggestions on benchmarking?

The L55xx processors cost more, use less power. That's not really going to help in this case, ran some rough numbers and it certainly isn't going to help to invest in the L55xx's as the power costs are actually negligible in relation to cpu costs. Obviously based on rough estimates on farm running time.

 

[rgbaguy]

Specs so you can have a look at the diff between the E5502, E5504, E5506 and E5520s

http://ark.intel.com/Compare.aspx?ids=40200,37096,40711,37092,

 

[rgbaguy]

Here are some comparisons I did, deviating from the E5502.

CPU E5502 E5504 E5506 E5520
clock 1.86 2.0 2.13 2.26
# cores 2 4 4 4
# threads 2 4 4 8
cache (MB) 4 4 4 8
system bus (GT/s) 4.8 4.8 4.8 5.86
ram (Mhz) 800 800 800 800/1066
mem bandwidth (GB/s) 19.2 19.2 19.2 25.6
turbo boost? NO NO NO YES
hyperthreading? NO NO NO YES
cost est 1x ~1.2x ~1.42x ~1.98x
perf est 1x ~4.3x ~4.58x ~9.72x

Please excuse my terrible formatting skills

I could be wrong on my numbers, but the E5520 definitely seems the best option.

I'm not sure how turbo boost or hyper threading will improve our performance, but my guess is that it will at least have a positive effect.

Any thoughts?

 

[rgbaguy]

... On the hardware, sure, the E5520 costs more, but the gains in performance is around 143 % over the E5502 (perhaps more, I'm no hardware expert), ...

Forgive my mistake, I mistook the E5502 for having 4 threads and the E5520 for having 4 as well.

 

[-tkf-]

A 6 core i7 isnt going to beat dual socket 4 core xeon for software costs But I'll look into this, perhaps a higher spec i7, definitely going to cost less than an equivalent Xeon, and could give more CPU power. What do you think?

Yeah, I have done numbers on the software, still working on the hardware, looking for a cheaper board as well. Maybe I can even adjust the Xeon's, maybe to a E5502 for example.

I will work with rand figures, it's easier for me.

On the software, one mental ray standalone license comes to ~ R 5400, an equivalent Maya 2010 batch license costs ~ R 6300 (theoretically).

So the cost for 80 sockets mr standalone: ~ R 432 000.

The cost for 80 sockets batch: ~ R 252 000.

On top of that, on the mental ray standalone, that doesn't include our Maya licenses, while the 2nd option does. Huge savings on software.

On the hardware front, I'm pretty sure I haven't optimized costs for the hardware yet. Dual socket is definitely the way to go. On the hardware, sure, the E5520 costs more, but the gains in performance is around 143 % over the E5502 (perhaps more, I'm no hardware expert), which is think justifies spending roughly 92 % more as the hardware will serve for quite a long time, and will serve well for more computationally expensive jobs. Some people might think this crazy, I'm open to input.

I'll however look into the E5502 as the lower bound and the E5520 as the upper bound and run some numbers. I just wish I could actually test this hardware. I looked at passmark a while ago to do some comparisons, I can't look right now don't have flash installed on this machine.Any suggestions on benchmarking?

The L55xx processors cost more, use less power. That's not really going to help in this case, ran some rough numbers and it certainly isn't going to help to invest in the L55xx's as the power costs are actually negligible in relation to cpu costs. Obviously based on rough estimates on farm running time.

Depending on alot of things in regards to how your renders work, it would be fair to say that the new 6 CORE i7 what be the same as 1.5 Xeon CPU´s at the same Ghz speed. The thing is that usually Xeon CPU´s are clocked a little lower compared to the Desktop version. So for example a 6 Core i7 @ 3ghz would yield you "18 ghz" while a dual xeon 2.66 ghz with 4 cores would yield you: 21.3 ghz, excuse me for using these ghz as a measurement but i think you get the point. The question of course, is when the new i7 is gonna be here and if it will be possible to use them in a small 2U cabinet. But the cost reduction could be significant pr server in terms of hardware and the price would be 10-15% in performance.