Server 2003: Dual Gigabit

M

MatiasZ

Regular
Hi everybody,

I'm about to receive my new server for the office. It is a Quad Xeon, with 4GB of ram and 6TB of HD in a RAID10 setup (3TB effective storage), which will be used to serve about 4-6 concurrent clients for HDV video editing. Since the network link could become the bottleneck really fast, I was wondering if there's any way to use both Gigabit ports on the server, to get 2Gb of bandwith instead of 1. The server will be running under Windows Server 2003 R2 Enterprise.

Thanks in advance for any ideas or suggestions.
Regards,
Matias
 
You need gigabit nics and a switch that supports channel bonding. Pretty much any decent Gigabit card supported in Server 2k3 should be fine, you just need a switch that can handle it.

If you are using unmanaged switches, you'll have to give each nic a different IP and point half the clients to one IP and half to the other.

EDIT: Also, are you running jumbo frames? I usually don't see more than 250 mbit or so from a Gigabit network running 1500 byte frames.
 
Jumbo frames are great, so long as your clients are using the same frame size. If you think performance is bad with 1500 byte frames, imagine how much worse it gets if the switch has to break the 9000 byte packets apart into seperate 1500 byte chunks for the clients who are still using stock frame size.

Although if they're all on the same physical network, then the server and client will simply auto-negotiate to the proper frame size anyway -- which then pretty much just nullified your jumbo frames to begin with.

Back to the original topic -- one problem with that kind of bandwidth, stock-sized eithernet frames and Server 2003 is the massive CPU loading on the first physical processor if your server is really busy. Go grab the "Server 2003 Scalable Network Pack" and install it. This will give you a small number of the Longhorn Server features such as multithreaded network stack handling, TCP Chimney support and receive-side scaling (a working version of the TCP autotuner from Vista).

Some of those features require hardware support (TCP Chimney for example), so hopefully your nics are new enough that they provide those features. This will drastically help CPU usage on your first logical processor, which then helps your server handle far more load without so much drama.
 
the maddman said:
You need gigabit nics and a switch that supports channel bonding. Pretty much any decent Gigabit card supported in Server 2k3 should be fine, you just need a switch that can handle it.

If you are using unmanaged switches, you'll have to give each nic a different IP and point half the clients to one IP and half to the other.
Click to expand...

The mainboard has dual gigabit, with this controller: Intel® 82563EB and Intel® 82564EB Gigabit Ethernet Controllers

Since I don't have a managed switch, using two different IP's seems like the only option, but a little hassle. Any good not-too-expensive managed switch you could recommend (8 ports should be enough)?

Albuquerque said:
Jumbo frames are great, so long as your clients are using the same frame size.
Click to expand...

I'm still deciding how everything will be connected within the network. I need internet access on some of the clients, and the router is a simple ADSL router. How would the network behave if I setup all clients to use Jumbo Frames, but also plug the router to one of the switches (so it would be on the same physical and logical network)? The other option is to use internet through WiFi, as it's easier to add a couple of wireless NIC's than to have a "dual cabled" office.

Can the MTU be set differently for different subnets under the same interface? I could therefore setup a different subnet for Internet, using Regular (?) Frames, and a subnet for the server, using Jumbo Frames.

Although if they're all on the same physical network, then the server and client will simply auto-negotiate to the proper frame size anyway -- which then pretty much just nullified your jumbo frames to begin with.
Click to expand...

How does this work?? I got lost here ;)

Back to the original topic -- one problem with that kind of bandwidth, stock-sized eithernet frames and Server 2003 is the massive CPU loading on the first physical processor if your server is really busy. Go grab the "Server 2003 Scalable Network Pack" and install it. This will give you a small number of the Longhorn Server features such as multithreaded network stack handling, TCP Chimney support and receive-side scaling (a working version of the TCP autotuner from Vista).

Some of those features require hardware support (TCP Chimney for example), so hopefully your nics are new enough that they provide those features. This will drastically help CPU usage on your first logical processor, which then helps your server handle far more load without so much drama.
Click to expand...

I can't seem to find anything wether these controllers support TCP Chimney or not, but apparently it supports Intel I/OAT technologies which is discussed together with the 2K3 Scalable Network Pack in this document so that should do, right? I hope having a single processor (although a quad one) is not too cumbersome for this setup.

Thank you for all the help so far, it's been both usefull and enlightening ;)
Regards,
Matias
 
Albuquerque said:
Jumbo frames are great, so long as your clients are using the same frame size. If you think performance is bad with 1500 byte frames, imagine how much worse it gets if the switch has to break the 9000 byte packets apart into seperate 1500 byte chunks for the clients who are still using stock frame size.

Although if they're all on the same physical network, then the server and client will simply auto-negotiate to the proper frame size anyway -- which then pretty much just nullified your jumbo frames to begin with.
Click to expand...

Yeah, I made that assumption. The switch has to have enough packet buffer to transmit the larger frames too.
 
MatiasZ said:
The mainboard has dual gigabit, with this controller: Intel® 82563EB and Intel® 82564EB Gigabit Ethernet Controllers

Since I don't have a managed switch, using two different IP's seems like the only option, but a little hassle. Any good not-too-expensive managed switch you could recommend (8 ports should be enough)?
Click to expand...

What's the budget? I'm guessing not a lot since you're looking at 8 ports. I'm using HP Procurve switches right now and I'm happy with them. The Procurve 1800-8G would be the cheapest switch I'd recommend.

http://www.newegg.com/Product/Product.aspx?Item=N82E16833316053

It does support link aggregation, so that would take care of everything. If you are trying to push multi-gigabit traffic, get a managed switch. The little ones just can't handle that much traffic.
 
the maddman said:
What's the budget? I'm guessing not a lot since you're looking at 8 ports. I'm using HP Procurve switches right now and I'm happy with them. The Procurve 1800-8G would be the cheapest switch I'd recommend.

http://www.newegg.com/Product/Product.aspx?Item=N82E16833316053

It does support link aggregation, so that would take care of everything. If you are trying to push multi-gigabit traffic, get a managed switch. The little ones just can't handle that much traffic.
Click to expand...

I can't seem to find those HP easy here in Spain. What about this one: Dell PowerConnect 2708? It says it supports link agreggation and has a web-managed interface (although it is decribed as unmanaged¿?), would that be enough, or at least better than what I have right now?? I can get it for about 100€ which seems a reasonable budget for a switch to me.

EDIT: they seem to have a pretty good offer for a 16 port equivalent switch, the PowerConnect 2716, for only 129€. If this type of solution is reasonable for my setup I would get one right away as it seems pretty decent.
EDIT2: the 2716 comes with Jumbo packet support (so apparently the 2708 doesn't), so it seems a better option yet.
 
MatiasZ said:
I can't seem to find those HP easy here in Spain. What about this one: Dell PowerConnect 2708? It says it supports link agreggation and has a web-managed interface (although it is decribed as unmanaged¿?), would that be enough, or at least better than what I have right now?? I can get it for about 100€ which seems a reasonable budget for a switch to me.

EDIT: they seem to have a pretty good offer for a 16 port equivalent switch, the PowerConnect 2716, for only 129€. If this type of solution is reasonable for my setup I would get one right away as it seems pretty decent.
EDIT2: the 2716 comes with Jumbo packet support (so apparently the 2708 doesn't), so it seems a better option yet.
Click to expand...

I've never worked with Dell's switches, but the spec sheets look alright. I'd agree with you, get the 2716 for the jumbo frame support, and then you can link the server with both ports.

Also, get good Gig-E cards for the clients, that makes a big difference too. 32 bit PCI Realtek cards bad. :)
 
Best thing to do is likely get your desktops with onboard Intel gigabit cards. They make great gig cards with very reasonable throughput and CPU offload capabilities.

With 9000-byte frames, our Lenovo T61 laptops with their onboard Intel Gig-E nics can do bursts to 750Mbit (they can't sustain that for long, as it exceeds the throughput of the laptop harddrive). That's all copper connections to an older Cisco 3850 24-port switch.
 
Albuquerque said:
Best thing to do is likely get your desktops with onboard Intel gigabit cards. They make great gig cards with very reasonable throughput and CPU offload capabilities.

With 9000-byte frames, our Lenovo T61 laptops with their onboard Intel Gig-E nics can do bursts to 750Mbit (they can't sustain that for long, as it exceeds the throughput of the laptop harddrive). That's all copper connections to an older Cisco 3850 24-port switch.
Click to expand...

Clients have (most of them) the onboard NIC that comes with Asus P5KR, which seems to be some "Attansic" chipset. However, I've seen transfer rates of constant 50% BW usage (500Mbit/s, reading through Task Manager->Network, don't know if this is reliable) with the current setup, where the server is an XP PC with a similar NIC, so I'm hoping to at least attain the same with the new setup. I'll get the Dell Switch then and let you know how it goes.

Resuming the other question, do you have any feedback regarding jumbo frames? I'm a little lost there:

MatiasZ said:
Albuquerque said:
Jumbo frames are great, so long as your clients are using the same frame size.
Click to expand...
I'm still deciding how everything will be connected within the network. I need internet access on some of the clients, and the router is a simple ADSL router. How would the network behave if I setup all clients to use Jumbo Frames, but also plug the router to one of the switches (so it would be on the same physical and logical network)? The other option is to use internet through WiFi, as it's easier to add a couple of wireless NIC's than to have a "dual cabled" office.

Can the MTU be set differently for different subnets under the same interface? I could therefore setup a different subnet for Internet, using Regular (?) Frames, and a subnet for the server, using Jumbo Frames.
Click to expand...
Thank you guys ;)

Matias
 
Frame size / MTU is a per-interface setting, but they both relate to maximum size; hence the acronym Maximum Transmission Unit. So, you can have network clients that use smaller frames and they shouldn't cause lots of issues -- so long as the two clients can communicate sufficiently between eachother that they can indeed agree on a proper frame size.

Here's where the problem lies:

You have Client A who is plugged into a "good" switch and can use 9000 byte frames.
You have Client B who is also plugged into a "good" (but seperate) switch and can use 9000 byte frames
A router or switch that connects Switch A and Switch B cannot use 9000 byte frames.

Now you have two clients who can potentially negotiate a full 9000-byte frame, but that router in the middle will have to deconstruct the HUGE incoming packet into six little ones for transmission -- multiply by as many clients as you will have talking across that device, and your network goes straight to packet-fragmentation hell.

So, as long as they're all on the same level of network hardware that is all configured to handle the jumbo frames, you'll be ok. No need to maintain two networks :)
 
Albuquerque said:
Frame size / MTU is a per-interface setting, but they both relate to maximum size; hence the acronym Maximum Transmission Unit. So, you can have network clients that use smaller frames and they shouldn't cause lots of issues -- so long as the two clients can communicate sufficiently between eachother that they can indeed agree on a proper frame size.

Here's where the problem lies:

You have Client A who is plugged into a "good" switch and can use 9000 byte frames.
You have Client B who is also plugged into a "good" (but seperate) switch and can use 9000 byte frames
A router or switch that connects Switch A and Switch B cannot use 9000 byte frames.

Now you have two clients who can potentially negotiate a full 9000-byte frame, but that router in the middle will have to deconstruct the HUGE incoming packet into six little ones for transmission -- multiply by as many clients as you will have talking across that device, and your network goes straight to packet-fragmentation hell.

So, as long as they're all on the same level of network hardware that is all configured to handle the jumbo frames, you'll be ok. No need to maintain two networks :)
Click to expand...

Ohhhh I understand now (I think!). Thank you very much for clearing that up. What I'll do, is make a little diagram of how I have everything connected (once it's done), so you guys can perhaps make some suggestions ;)

I actually received the server today, and it looks great, but I have some doubts on the RAID configuration, which I'm about to ask in a different thread to keep things aside.

I am now deciding which Switch to buy, either the PowerConnect 2716, or this one. Any suggestions between the two?
 
MatiasZ said:
Ohhhh I understand now (I think!). Thank you very much for clearing that up. What I'll do, is make a little diagram of how I have everything connected (once it's done), so you guys can perhaps make some suggestions ;)

I actually received the server today, and it looks great, but I have some doubts on the RAID configuration, which I'm about to ask in a different thread to keep things aside.

I am now deciding which Switch to buy, either the PowerConnect 2716, or this one. Any suggestions between the two?
Click to expand...

I'd trust the Dell over the Trendnet. More people will be using the Dell models.
 
You must log in or register to reply here.
Back
Top