building up a render farm

[Blazkowicz]

well raid 6 is :
six drives with an evenly spread 4/6 data, and 2/6 parity. it's 5/6 data, 1/6 parity for raid 5.

raid 10 is :
a raid 0 of three raid 1

dedicated parity is only to be found on raid 3 which no one ever uses.

so raid 5, 6 and 10 leave 5TB, 4TB and 3TB respectively ; raid 10 is the fastest, raid 6 the slowest (slowness is mostly about writes ; presumably not a big problem if write is when you feed it with external data from a gigabit link, or rendered frames)

 

[rgbaguy]

http://en.wikipedia.org/wiki/RAID_10#RAID_10_.28RAID_1.2B0.29

Redundancy and data-loss recovery capability

All but one drive from each RAID 1 set could fail without damaging the data. However, if the failed drive is not replaced, the single working hard drive in the set then becomes a single point of failure for the entire array. If that single hard drive then fails, all data stored in the entire array is lost. As is the case with RAID 0+1, if a failed drive is not replaced in a RAID 10 configuration then a single uncorrectable media error occurring on the mirrored hard drive would result in data loss. Some RAID 10 vendors address this problem by supporting a "hot spare" drive, which automatically replaces and rebuilds a failed drive in the array.

Given these increasing risks with RAID 10, many business and mission critical enterprise environments are beginning to evaluate more fault tolerant RAID setups that add underlying disk parity.[citation needed] Among the most promising are hybrid approaches such as RAID 50 (stripe above single parity) or RAID 60 (stripe above dual parity).
[edit] Performance (speed)

According to manufacturer specifications[3] and official independent benchmarks[4][5], in most cases RAID 10 provides better performance than all other RAID levels except RAID 0.

It is the preferable RAID level for I/O-intensive applications such as database, email, and web servers, as well as for any other use requiring high disk performance.[6]

Thanks for your reply Blazkowicz, it helped alot

I think RAID 10 will be perfect for us

 

[Silent_Buddha]

Aye, Raid 6 and Raid 5 are fairly good compromises if your disk subsystem is a significant portion of your build's cost. Balancing space versus cost versus speed versus data protection.

However, in your render farm, the disk subsystem is going to be a small fraction of your overall costs.

Just make sure to keep multiple spare drives handy in case of one or more drive failures. You'll want to replace any failed drive, ASAP.

Regards,
SB

 

[rgbaguy]

Aye, Raid 6 and Raid 5 are fairly good compromises if your disk subsystem is a significant portion of your build's cost. Balancing space versus cost versus speed versus data protection.

However, in your render farm, the disk subsystem is going to be a small fraction of your overall costs.

Just make sure to keep multiple spare drives handy in case of one or more drive failures. You'll want to replace any failed drive, ASAP.

Regards,
SB

Yeah, totally. Will definitely keep spares on hand. I'm close to getting a final spec together, still waiting for quotes from our supplier.

Will post the specs as soon as I have worked them out fully.

Thanks for help guys

 

[rgbaguy]

Hey guys, I'm having trouble finding a switch. I just can't seem to find switches with 10 gig uplink included (appears they come as a separate module?)

3Com Switch 4200G 48-port (3CR17662-91) with 1-port 10-gigabit Module (3C17666)
http://www.shopmania.co.za/shopping~online-switches~buy-3com-3cr17662-91~p-1103783.html
http://www.3com.com/products/en_US/detail.jsp?tab=prodspec&sku=3CR17662-91&pathtype=purchase

D-Link DGS-3450 (confirm 48-port gigabit L2 switch, 4 combo SPF and 2 10-gig slots)
http://www.shopmania.co.za/shopping~online-switches~buy-d-link-dgs-3450~p-1203526.html
http://www.dlink.ca/products/?pid=dgs-3450

D-Link DGS-3100-48
http://www.wantitall.co.za/PC-Hardw...alone-L2-Gigabit-Stackable-Switch__B000UKJM9U

The above seem what we need right? I'm using those links for a price guideline. Have contacted our supplier, the only 48-port gigabit switch they have is:

D-Link DGS-1248T web smart 48 port 10/100/1000 gigabit + 4 combo SFP fiber connection , un-managed layer2 switch - auto-MDI/MDIX , port mirroring , 8k MAC address , 1.632mb buffer per device , rackmount ready

Which doens't seem to have the 10 gig uplinks?

I'm gonna go back through this post and find all the switch options you guys have posted, see if I can find them here.

 

[rgbaguy]

Hi guys,

Is this card suitable (for workstation and server)?

Intel NetEffect E10G81GP

http://www.intel.com/support/network/adapter/pro100/sb/CS-030160.htm
http://www.intel.com/Assets/PDF/prodbrief/320918.pdf

I'm still confused with the switches, is this one suitable?

http://www.dlink.com/products/?pid=637

Does it include the 10Gb uplink modules?

Hope you can help!

Thanks

 

[MfA]

Does it include the 10Gb uplink modules?

The stacking ports don't seem usable as pure ethernet ports from the sound of it (they are 5 Gb/s full duplex not 10Gb).

DXS-3250 and DGS-3450 can have the 10 Gb uplinks.

PS. your awfully nice to your supplier by trying to hunt down a 10 Gb uplink capable switch for him ... really if you are willing to use him exclusively he should be the one doing it. Personally I'd just say "find something for me or I'll buy the 3CR17662-91 with 10 Gb module from another supplier".

PPS. the 10 Gb modules seem to be always sold separately.

 

[rgbaguy]

The stacking ports don't seem usable as pure ethernet ports from the sound of it (they are 5 Gb/s full duplex not 10Gb).

DXS-3250 and DGS-3450 can have the 10 Gb uplinks.

PS. your awfully nice to your supplier by trying to hunt down a 10 Gb uplink capable switch for him ... really if you are willing to use him exclusively he should be the one doing it. Personally I'd just say "find something for me or I'll buy the 3CR17662-91 with 10 Gb module from another supplier".

PPS. the 10 Gb modules seem to be always sold separately.

Thanks for the reply MfA Ok, will look at those. Oh hardly, it's difficult getting information out of them! I have said to them, look, we need so and so. They aren't doing that, its getting annoying. Will tell them if they aren't prepared to find something that fits our requirements then I'll look elsewhere.

Yeah, that does seem the case. I've asked IF the uplink modules ARE INCLUDED. They never answer that question. Aaaaaaaargh!

Will keep pushing them, otherwise will try someone else. It's hard for me, I have zero experience with networking

 

[Karoshi]

GBIC
SFP
GBIC description includes the rationale:

The appeal of the GBIC standard in networking equipment, as opposed to fixed physical interface configurations, is its flexibility. Where multiple different optical technologies are in use, an administrator can purchase GBICs as needed, not in advance, and they can be the specific type needed for each link. This lowers the cost of the base system and gives the administrator far more flexibility. On the other hand if it is known that a switch will mostly have one port type (especially if that port type is copper) purchasing a switch with that port type built in will probably be cheaper and take up less space per port.

Same applies to SFP.

They dont know if you want to connect to a switch 2m away (get a cheapo laser) or 40km away (we welcome your monies!).
So unless it's a cheaper switch for local distribution it's probably not gonna have plugged in modules. There are possibly cheaper bundle offers, maybe, haggle with them?

 

[Blazkowicz]

That reminds asking for bootroms in shops

reactions are a blank stare, astonishment or outright denial (what? booting with no storage media? that's impossible!). Because you're the 1% of retail shoppers asking for that feature.
yet as a bootrom is a immensely useful $1 piece of hardware.
End result of that story, a friend bought a bunch of so-called "bootable network cards" from a store, that supported a boot rom but did not provide it. Regular, non-bootable network cards.

10gigabit module isn't a 30-year-old commodity tech. But similarly, small companies never buy them and they are probably bought and installed by the big ones, who have their own procurement channels.
You need to get higher in the food chain it seems.

 

[MfA]

Same applies to SFP.

And for 10 Gb/s to XFP/XENPAK.

BTW rgbaguy, the manufacturers will generally on their website tell you what's included as standard ... anyway, the 3com seems by and far the cheapest ... for that you need a 3CXENPAK95 module.

 

[Blazkowicz]

good, by googling that product name you can easily find them, just not in your usual well known online shop. good to know it's called a "transceiver" (like the modules I've never ever seen, that go on the "joystick plug" at the back of older network cards)

 

[rgbaguy]

GBIC
SFP
GBIC description includes the rationale:
Same applies to SFP.

They dont know if you want to connect to a switch 2m away (get a cheapo laser) or 40km away (we welcome your monies!).
So unless it's a cheaper switch for local distribution it's probably not gonna have plugged in modules. There are possibly cheaper bundle offers, maybe, haggle with them?

Thanks for this Karoshi, that definitely cleared up some of the tech jargon

That reminds asking for bootroms in shops

reactions are a blank stare, astonishment or outright denial (what? booting with no storage media? that's impossible!). Because you're the 1% of retail shoppers asking for that feature.
yet as a bootrom is a immensely useful $1 piece of hardware.
End result of that story, a friend bought a bunch of so-called "bootable network cards" from a store, that supported a boot rom but did not provide it. Regular, non-bootable network cards.

10gigabit module isn't a 30-year-old commodity tech. But similarly, small companies never buy them and they are probably bought and installed by the big ones, who have their own procurement channels.
You need to get higher in the food chain it seems.

Ouch, that must have pissed him off You make a good point, well this is pretty much the biggest supplier here, so no other options.

And for 10 Gb/s to XFP/XENPAK.

BTW rgbaguy, the manufacturers will generally on their website tell you what's included as standard ... anyway, the 3com seems by and far the cheapest ... for that you need a 3CXENPAK95 module.

Yeah, been struggling to find out what comes standard. Thanks for that, will ask the supplier if they can get that So 3Com 4200G switch and two 10GBASE-CX4 XENPAK modules should do the trick then

good, by googling that product name you can easily find them, just not in your usual well known online shop. good to know it's called a "transceiver" (like the modules I've never ever seen, that go on the "joystick plug" at the back of older network cards)

True, Google is awesome. Suppliers not

Thanks guys, hopefully they will be able to get...

 

[rgbaguy]

Hey guys, all sorted on the hardware side Have got costs for everything we need including switches and uplink modules (finally!) One question on the network side of things, any standard CX4 cable will work? Will need a 50cm/1m CX4 for switch-to-server link and a 5 m for the workstation-to-switch link. Those cables are quite pricey!

Would like to ask what you guys recommend on the OS side of things. For our blades I'm looking at Open SUSE as it's free and from my experience very stable. Fedora not so much from my experience. What do you reckon? Ideally would like to install a distro that uses as few system resources as possible.

On the workstation side, looking at Windows 7 64-bit, mainly for ease of use (not everyone is comfortable with linux). On the server side of things, not so sure myself?

I reckon Windows Server 2008 would be good. Have asked some people I know and they recommend it. What do you guys think?

What about Windows HPC Server 2008? As I recall from previous posts a good server can optimize network transfers with caching, what do I look at for that?

Have looked to Clonezilla for drive cloning, anyone have experience with this? Looks production stable, what do you reckon? http://www.clonezilla.org/

Would like to say thanks to everyone who has posted on this thread, your help has been fantastic and would be pretty lost without it

Look forward to hearing back!

 

[Laa-Yosh]

Have you found a background storage solution yet? A simple HDD based system on its own usually can't support even a small studio and renderfarm, so you'll probably have to invest in something a bit more advanced. Then the driver and software options of that device are also going to become a factor in choosing your OS.
All I know about ours is that it's something from StorNext... and we have some sort of tape backup system too.

 

[rgbaguy]

Have you found a background storage solution yet? A simple HDD based system on its own usually can't support even a small studio and renderfarm, so you'll probably have to invest in something a bit more advanced. Then the driver and software options of that device are also going to become a factor in choosing your OS.
All I know about ours is that it's something from StorNext... and we have some sort of tape backup system too.

Hey Laa-Yosh, thanks for the reply! Well have looked at a 6 x 1TB drive storage server in RAID 10 for storage. That gives us 3 TB, which should be fine. Once the drives are full we just backup...

This will also be the server that runs our queue master and licensing...

As for tape, I guess I should have a look into that.

 

[-tkf-]

Hey guys, all sorted on the hardware side Have got costs for everything we need including switches and uplink modules (finally!) One question on the network side of things, any standard CX4 cable will work? Will need a 50cm/1m CX4 for switch-to-server link and a 5 m for the workstation-to-switch link. Those cables are quite pricey!

Would like to ask what you guys recommend on the OS side of things. For our blades I'm looking at Open SUSE as it's free and from my experience very stable. Fedora not so much from my experience. What do you reckon? Ideally would like to install a distro that uses as few system resources as possible.

On the workstation side, looking at Windows 7 64-bit, mainly for ease of use (not everyone is comfortable with linux). On the server side of things, not so sure myself?

I reckon Windows Server 2008 would be good. Have asked some people I know and they recommend it. What do you guys think?

What about Windows HPC Server 2008? As I recall from previous posts a good server can optimize network transfers with caching, what do I look at for that?

Have looked to Clonezilla for drive cloning, anyone have experience with this? Looks production stable, what do you reckon? http://www.clonezilla.org/

Would like to say thanks to everyone who has posted on this thread, your help has been fantastic and would be pretty lost without it

Look forward to hearing back!

In regards to Partition Backup and Restore, i have been playing around with:
http://www.paragon-software.com/home/db-express/index.html
Boots from CD or USB and saves/restores from a USB / Firewire Disk

On the server side i am very curious as well, with our plans we end up having a Dual Xeon Server, with 16 1.5 TB Disks serving our workstations over a 10GBit nic. My plan is to cram alot of ram into the machine and run Server 2008. But i have been wondering if there is a High Performace or Slimmed down option that focus on File serving and does a better job than a standard Server 2008

 

[MfA]

Using windows as NAS for Linux clients is just asking for headaches ... why not use a nice linux install for that as well? (Also use NFS rather than SMB.)

 

[Blazkowicz]

I believe SMB / CIFS is the easier way to have a high performance file share. NFS might be not so good. unless maybe you tweak it.

I don't have experience in tuning things but you probably want jumbo frames on the ethernet layer, tuning the file system / choice of file system, raid strip size etc.
(I suppose file sizes are big for 3D rendering, and that you optimize for that)

 

[rgbaguy]

Hey guys, my apologies for the very late reply, and wow, have I been gone a while, whole new look to the forum! Had so much to deal with lately. I'm back now!

Thanks for the replies!

In regards to Partition Backup and Restore, i have been playing around with:
http://www.paragon-software.com/home...ess/index.html
Boots from CD or USB and saves/restores from a USB / Firewire Disk

On the server side i am very curious as well, with our plans we end up having a Dual Xeon Server, with 16 1.5 TB Disks serving our workstations over a 10GBit nic. My plan is to cram alot of ram into the machine and run Server 2008. But i have been wondering if there is a High Performace or Slimmed down option that focus on File serving and does a better job than a standard Server 2008

Will have a look at paragon, thanks! Any luck on your research on the server OS? I'm afraid I'm the last person to ask on that front

Using windows as NAS for Linux clients is just asking for headaches ... why not use a nice linux install for that as well? (Also use NFS rather than SMB.)

Perhaps just use Open SUSE for the file server? Guess that should be fine? Why NFS over SMB?

I believe SMB / CIFS is the easier way to have a high performance file share. NFS might be not so good. unless maybe you tweak it.

I don't have experience in tuning things but you probably want jumbo frames on the ethernet layer, tuning the file system / choice of file system, raid strip size etc.
(I suppose file sizes are big for 3D rendering, and that you optimize for that)

Cool, why SMB over NFS?

Looking forward to more suggestions? Thanks guys