I’ve completed testing of the NIC teaming on our new Dell PowerEdge 1950 servers. I’m more than a little bit surprised by the results, which I’ll get to in a moment. My initial assumptions were that the network adapters would perform in the following order, from best to worst performing:
1) Teamed Intel NICs
2) Teamed Broadcom NICs
3) Single Intel NIC
4) Single Broadcom NIC
I tested each configuration by copying a large file or directory of files from server PO1 to server PO2. Both servers booted from SAN, ran Windows 2003 with the latest Windows patches and updates from our Patchlink server. PO2 was cloned from PO1 after being sysprep’d. The servers were configured identically, each plugged into the same module on the same HP Procurve 5304xl switch. The switch was configured with 802.3ad link aggregation.
The NICs that were tested were:
1 quad port Intel VT 1000 gigabit NIC PCI-X
2 integrated Broadcom Netxtreme II BCM5708C gigabit NICs
The files I used to test were:
- OM_5.4.1_SUU_A00.iso, a 1.85 GB ISO image file
- gw700.iso, a 689MB ISO image file
- A 2.23 GB directory of 509 text files, each averaging 5MB in size
The methodology I used to test with was:
- Install the NIC drivers and configure team’s static IP, subnet mask, default gateway, and DNS on each server. Default team settings were used, including TCP Offload Engine (TOE), Large Send Offload (LSO), and Checksum Offload (CO)
- Disable all unused NICs
- Restart both servers
- Copied the first test file from PO1 to PO2 using the following syntax:
copy filename \\po2\c$\temp\test\
- Timed how many seconds it took to copy the file from PO1 to PO2
- Deleted the copied file from PO2
- Copied the test file again from PO1 to PO2 until 5 passes were completed
- Repeated the process for the next test file(s)
The following configurations were tested:
1) Single Intel NIC to Single Intel NIC using driver dated 6/13/08
2) Single Broadcom NIC to Single Broadcom NIC using driver dated 2/21/08
3) Single Intel NIC using driver dated 6/13/08 to Single Broadcom NIC using driver dated 2/21/08
4) Teamed Intel NIC to Teamed Intel NIC using driver dated 6/13/08
5) Teamed Intel NIC using driver dated 8/23/07 to Teamed Intel NIC using driver dated 6/13/08
6) Teamed Intel NIC to Teamed Intel NIC using driver dated 8/23/07
7) Teamed Broadcom NIC to Teamed Broadcom NIC using driver dated 2/21/08
1) The teamed Intel NICs performed the worst – even worse than using single Intel NICs
2) The single Broadcom NIC outperformed the single Intel NIC
3) The teamed Broadcom NICs were the highest performing
I have no clue why the results are what they are. In the past, I’ve experienced horrendous performance with the Broadcoms, and great performance from the Intels. Does anyone have any idea as to why the teamed Intel NICs would perform so poorly?
The only real difference I could see was that when copying files, Windows Task Manager showed Network Utilization at ~51-56% for the Broadcom tests, and ~16-17% for the Intel tests. Why, I’m not sure.
The data in the spreadsheet shows actual averages, which was the average number of seconds it took to copy a file over five tries, and what is called adjusted average. Adjusted average is something I learned about long ago in a stats class I had that said it’s a best practice to disregard the lowest and highest value in your sample. Either way you look at it, the findings are the same: The Intel performance is horrible while the Broadcoms perform great.
Based upon these tests I’m going to recommend going with the teamed Broadcom NICs in the new server deployment.