10GB ethernet Card?

Switches are communicating at 10GB. Next piece will be the Chelsio card. I have the card, I've got some modules on order (they're expensive AF!).

The issue with the switches not communicating had nothing to do with the jacks (though they state they are rated as CAT6, and I don't think they were CAT6A), I had confused two wall jacks.

I found this out even after having switched the wall jacks out with ones specifically rated for CAT7, which were expensive. Unfortunately, while the CAT7 jacks appeared to be of good quality, they were quite defective/flawed in design. Plugs going into the jack never seemed to make good contact, and the tooless crimping of wires were also not making good contact.

 

Once you are up and running -You will need to list all the components, by model, brand and price for anyone else thinking of going up to 10Gbit.

It would be worth real world (Stopwatch) timing of a file transfer of say 50GB file and Folder containing files across your various machines.

I would be very interested in the bottlenecks. For instance 4X, 8X or 16X PCI Express and even the comparison between your 1Gbit cards and 10Gbit.

With the testing we did, admittedly over only one Sunday afternoon and early evening; There was a very definite conclusion that the biggest improvement came with a managed switch not the cables, number of cables or ports. HOWEVER – our tests were far more restricted than anything you are now doing.

NetgearGS108
NetgearGS116
Cisco SG300
Intel 4 Port I340 used on all 3 workstations at one time.
Intel Dual port Pro1000 and a Single port Pro1000 use on the X6 and X7
On Board Dual and Single 1Gbit from following used on all 3 workstations at one time
Supermicro
X6DAE (Network) Dual / Single Onboard and PCI-X 133 adapter - - 4 Port from PCI Express
X7DAE (Network) Dual / Single Onboard and PCI-X 133 adapter - - 4 Port from PCI Express
X9QRI F+ (Network) Dual / Single Onboard and 4 Port from PCI Express
All on Xeon’s at top spec
Lenovo Thinkpad P53
Dell 1737
All cable was Cat 5E blue, yellow and red. Blue being my favourite.

One thing if I remember correctly was that we didn’t seem to get anymore speed from onboard, however this is what I have eventually ended up using for simple convenience, since I can remove cards and we didn’t perceive an increase in speed with the quad port over dual port.

 

Still doing some work. I'm going to need to replace fans on one of the Switches, because they are rather loud.

Here is a picture, detailing the prices I paid for the switches.



SG350X-24P was $300 (best offer accepted, and it was the first offer).

SG350X-48MP was about $550 (best offer accepted, seller insisted that they couldn't go below this point, but still tested that fact with an offer of $500, but they didn't budge).


The two switches are linked via copper RJ45 10gig ports. They are communicating at 10 gig, according to both of their WebGUIs.

Haven't bought a NAS yet. Looking at DS1819+ or DS1821+.

Haven't yet installed the Chelsio 110-1088-30 card as of yet. I am needing to get a full height bracket, on order.

Order for 4x SFP+ receivers, which claimed support for Cisco and Chelsio in listing:
https://www.ebay.com/itm/224100336491

Very pricey. Bought extras, to maximize value in purchase and serve future needs.

Transceivers serves up to (30m) 93-100ft over Cat6A/Cat7, which is more than I will need most of the time. The longest run on cables is between the two switches is about 66 feet max, so everything else is sure to be less.


In the final setup:
Station 1:
- Netgear R8500.1 (DHCP, Gateway)
- Cisco SG350X-24P (10G to SG350X-48MP and 10G link to main computer)
- Main Computer
- - 10Gig dedicated to local LAN only.
- - 1Gig (mobo) link for internet.
- - 1Gig (mobo) link for specific traffic.
- Other hardware (printers, aux routers).
- 1gig Link to Station 3 from Switch.
- 1gig Link to Station 4 from Switch.

Station 2:
- SG350X-48MP (10G link to SG350X-24P)
- Dahua NVR (two 1gig links set to Load balance)
- POE IP cameras.
- NAS (near future) on a 10G link, or 4x 1G aggregate.
- Other hardware (aux router, VOIP and phone base).

There's two other stations, one for a wireless access point (R8500.2), and another just basically serving 1 device in a room. Station 3 was setup with Cat7, but no device to link at 10G with.

It may be a month or two before I can close the gap in finalizing the overall project.

 

I find myself in an unusual place.

I've got these 10G RJ45 transceivers and whenever I plug them into the Chelsio card, it's not long before they fire up and get HOT. Then the connection rapidly drops in and out.

This sounds like an issue with the card, or is it?

All four RJ45 transceivers that I bought will each become really very hot.

 

I haven’t a clue. They had Chelsio cards at one work place. I know all their kit was definitely running at 10Gbit with no drop outs, as far as I know.

First a very important question. Are they heating up when you send data? Or are they heating up when just connected.

If your system is dropping out when there is no data being thrown across the cables, then it is a different problem to - - “It’s ok until I start loading it”

You will need to correct me if I am making the wrong assumptions

You have 1 Chelsio card and 4 RJ-45 adaptors. You also have 2 managed switches.

You need to isolate to test. one idea would be what you have done and connecting up the switches using the RJ-45 adaptors, but depending on the answer to the question above you may need to load them IE run some date through.

It’s obviously either the transceivers or the Chelsio Cards. It is very unlikely to be you Cisco switches.

Chelsio have been around for many years and are based in the USA and India. Who made your transceivers?

Do your Chelsio cards get hot?

You know what you need to do already you need to temporarily have access to another 10Gbit card, it doesn’t matter what brand – you only need Chelsio on your XP system. You also need access to another brand of transceiver.

Have you tried the Transceivers in another 10Gbit card yet?

What about an email to or phone call to Chelsio?

There is no way in hell they would want to be seen to be incompatible with Cisco. See what they recommend transceiver wise. I have been lucky in the past and got free kit from companies.

Headquarters
Chelsio Communications is headquartered in the heart of California’s Silicon Valley.

Chelsio Communications (Headquarters)
735 N Pastoria Avenue,
Sunnyvale, CA 94085
T: 408.962.3600

Sales
For all sales inquiries please send email to [email protected]

Support
For all support related questions please send email to [email protected]

 

The cable doesn't even need to be plugged into the transceiver before the transceiver gets hot.

Also, the card itself is VERY hot. Had a tough time moving it to a different slot.

The transceivers work perfectly in the cisco switches. In fact, one is plugged in and not getting hot at all, with or without a cable plugged into it.

I think it's a small chance that the Chelsio card is the only piece that itself is just getting really hot at the connected transceiver is getting heat transference from it.

Definitely have some investigating to do.

 

Having had a quick browse on the web thingy

Is the warning led lit on the Chelsio after Boot? It should only be on during Boot.

I think you need to put active cooling on your Chelsio card. (The clue is in the 10Gbit – there will be a crystal oscillating really fast on that card.)

https://forum.netgate.com/topic/100796/resolved-new-chelsio-t520-running-very-hot/2

To be honest most workstations use serious fans and tend to be a little loud. The way I keep mine quiet at home is I got my husband to cut a massive hole in the side of the Supermicro case and run a 16”fan bolted to the side – so it is nice and quiet. I only need the radiator on in my office for an hour in the morning then I turn off, as the workstations act as fan heaters.

I haven’t had heat problems with my network cards but then I run really slow Gigabit – you are running at 10 times the speed. I did have overheating problems with SCSI Raid and had to add active cooling to a card that was really designed to be in a 2U server. Very often servers in racks have 4 fans screaming away at the front and 4 screaming away at the back. Using enterprise kit can very often be like working with a helicopter next to the desk.

Basically 10Gbit cards overheat in the home. Solid copper heat sinks are best. Mr Siamese cat has gone for solid silver inserts into 1KG Supermicro solid copper coolers. As I don’t really have the time or a precision lathe - I went for standard watercooling.

Fitting fans is easy if you can’t be bothered using water-cooling.



Also Chelsio Cards run Hot

https://www.reddit.com/r/homelab/comments/5q3czx/any_gotchas_with_chelsio_10gb_in_windows/

I only have experience with real workstation boards. Non – workstation boards might not be able to provide enougth power to the PCI express.

 

Unfortunately, the Chelsio card on hand does not appear to have any screw holes by which to work.

I'm actually puzzled as to how the tiny heatsink is even attached.

 

Also, I find other 10GBe cards that seem like they can support an XP install, but I'm having a terrible time finding drivers.

Dell Broadcom 57810S Dual Port 10Gb Base-T Server Adapter
https://www.newegg.com/p/1A0-002Y-000R2+&cd=1&hl=en&ct=clnk&gl=us


Intel X540-T2
https://www.ebay.com/itm/INTEL-X540...DAPTER-DELL-K7H46-3DFV8-540-BBHD/181805397920


Dell and Intel seem to make it impossible to find their "XP" drivers.

 

I have found cards from Mellanox and ATTO that are 10Gig speeds and have drivers for XP 32/64 bit. I just put in an order for an ATTO that has RJ45 native connection, though it is only a single RJ45 (darn, only TEN gigs, right?).

If I have heating issues with this ATTO card, It looks like I may be able to fit a fan onto it.

ATTO Drivers
https://www.atto.com/downloads/128

ATTO FastFrame NT11


Mellanox Drivers
https://www.mellanox.com/products/ethernet-drivers/windows/connectx-en-10gbe

Mellanox MCX311A-XCAT CX311A ConnectX-3

 

Here is an additional link that may be useful in the future:

Intel 10 Gigabit X520 Adapters Driver
https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=v76gk

Does not have "XP" folder inside the archive, but is reported on the website to support XP 32 and XP 64.

There is a Vista folder within, which may be what the executable/setup will pull from when detecting 2003/XP.

 

I am also going to be trying an Intel X540-T2.

https://www.ebay.com/itm/Addon-Add-...ort-Nic-Intel-X540-T2-10-Gigabit/324455438673

I will attempt to be using these Intel drivers found on Lenovo's site for "Windows Server 2003"

https://support.lenovo.com/us/en/do...r-2003-r2-thinkserver-rd330-rd430-rd530-rd630

Akorse, I will share all that I learn in my successes and set backs.

 

The heatsink will be applied by thermal glue or an adhesive thermal pad. You can even buy thermal resin. Just type in “heatsink adhesive” into Ebay you will get a tube for under $5.

This is how you will attach your fan too.

I usually go with the largest solid copper heatsink I can squeeze on any card, when not putting a water block on. I have loads of copper sheet that I bought off ebay as well.

I suggest you type in “solid copper heatsink” to Goggle and ebay. Then type in “Supermicro solid copper heatsink” This is what you should have mounted everywhere - solid copper.

As for running dual cards - IE bridging them. May I suggest you just get single up and running first.

As I said in my original post unless you are building a Beowulf Cluster – 1Gbit is fine even for 8K video. Editing video is all CPU and GPU usage.

That’s why I stuck with 1Gbit and went for a bridged set up. I could easily manage with out the bridge –its just that I had the cables. The biggest jump in speed was with out one shred of doubt moving to a managed switch over a non-managed.

Many small companies will only have their server on the 10Gbit line with a 10Gbit card. The rest will be running at 1Gbit. That is why there are so many of those style of boxes available.

First see how hot you get and if you can handle the heat on one 10Gbit line first and do some testing. If you think the card gets hot on a single 10Gbit – wait and see what happens when you bridge with 2 lines now that will be hot.

I understand why Architectural companies like Gensler will run a Beowulf set up, but then they are Using Revit – that makes all the difference when you have real time editing of an entire hospital in 3D including all the wiring, plumbing and materials. They tend to be big files and then the “High Heel Plummy Accents” will always want “a quick render for a client conference” that is virtually always completely unnecessary. Rant over.

 

I will definitely mind what you've said for the copper heatsinks.

I'm only interested in a single 10gig connection from my main computer, for now.

My plan is only to use this on the local network anyway, having the gateway block it, as well as the local software firewall, from reaching the internet.

Main reason for a 10Gig connection will be to maximize my connection to a NAS to basically have it connect as fast as if it were installed locally.

Second reason is to get XP one last hard push in the hardware department.

Linux, Windows 7 and Windows 10 no doubt have a great deal more hardware options due to driver availability.


I did find it especially odd that the Chelsio card was heating up so much without any actual traffic on the card.

 

I totally agree about the Chelsio card. I would be questioning why on earth it was so hot without a load. A lot of other people seem to be having the same issue. I have no personal experience. I can’t really count sitting in a companies offices and using their kit. I do know that whenever I have, they are using workstations and workstation cases – which to be honest are quite often much better at cooling than customised towers.

Because I am also curious and have done some Google searching it seems that it is a common problem. It doesn’t seem to be a Windows XP problem.

Link below worth a look.

https://forums.servethehome.com/index.php?threads/adding-active-cooling-to-a-10g-pci-e-nic.14016/

One other thing I did come across was about setting up a network with 10Gbit and 1Gbit combined and it looks, as though there can be problems if they are not on different IP networks. To be honest looks as though you might have quite a bit of research to do.

https://www.truenas.com/community/threads/problem-with-10g-chelsio-card.71463/

I really think a phone call to Chelsio is worthwhile. Just so you have reference point.

Good luck with it anyway.

 

I received my ATTO FastFrame NT11 10GBASE-T Network card sooner than expected.

After some configuration on the adapter properties, allocating this new card as my main computer's main Local Network connection, everything appears to be going well.

I manually set the IP address on the two NICs that are not used for direct internet connection. I intentionally left their gateways blank, so they would not seek a gateway.

The drivers for this card are located at: https://www.atto.com/downloads/128/1

The website specifically makes mention of Windows XP drivers.

When going through the installation of drivers, I told the computer I would manually install the hardware. I selected for Network Adapter and then the "I have a Disk" button.

When selecting drivers, I went through each of the available folders (after extracting/unzipping). The drivers that displayed were located in: FastFrame\NIC\x64\PROXGB\Winx64\NDIS5x

From here, I had to select driver for Model NS11.

Akorse, Windows tried telling me it couldn't verify the drivers, but it worked. I selected.

Visuals:



Now working.



As of yet, it is not actively being used for 10G speeds. However, compared to the Chelsio card, this thing is not hot to the touch at all of yet.

Next step will be testing speeds when I get my NAS purchased and setup. This might be awhile (not near future, but I can't give a time table).

 

Another key piece in getting this 10G card to work strictly as a LAN is to configure the adapter metric.

10GLAN#1 has a metric of 100
LAN#1 has a metric of 300
LAN#2 has a metric of 200.

The importance is the one with the lower number will cause the adapter to work first.



Since 10GLAN#1 is only for Local connections, I want this one to be used first. 10GLAN#1 is also blocked by its MAC address from accessing the internet on the gateway/edge router. So, even if it has a Gateway, it's blocked at the pass.

However... a 10GLAN#1 with a gateway would ruin my ability to connect to the internet, because it is blocked at the router. The computer would be trying to use that adapter first to reach the internet, but would never succeed. So it is rather important that it does NOT have a gateway designated in this setup.

LAN#2 is setup for only certain ports and assigned for incoming connections from the internet.


LAN#1 is for everything, including internet connections and excluding certain ports, as seen by LAN#2 TCP/IP filtering. But with the LAN#1 metric set for being last in line, and all other adapters not having a gateway, it doesn't matter for internet connections as LAN#1 is the only one able to reach the internet. LAN#1 will just be last in line for local connections.

I could very well merge LAN#1 and LAN#2 (built into motherboard) in a teaming configuration for internet later down the line... but LAN#2 serves its distinct purposes, except for "File and Printer Sharing Services" now.


This is the ATTO NT11 card. Only 1 RJ45 port on it.




Hope this is helping someone out there "googling" for answers!

 

Hi Secpar

I am really interested in your results I haven’t ever touched Metrics before. Totally new to me.Mind you, I didn’t even know about Checksum Offload, which was my initial problem, when I joined this site. Your expense, experimentation and pain will certainly save anyone thinking of going the 10Gbit route a lot of time and money.

From your results, so far. If I ever upgrade to 10Gbit - I will be most likely buying the “ATTO FastFrame NT11” over any other card. I went to the website – registered and downloaded all the drivers for all operating systems I am likely to use.

There seems to be so much variation in performance and speed depending on Lan kit, set up and the computer you use. There are numerous video’s on YouTube on 10Gbit. I was really interested in this one and I suggest you have a quick look.



However, when we tried 4 port bridging (Link Aggregation) – I was under-whelmed by performance increase over 2 port bridging. All our tests were solid state to solid state and a quick test to15K SCSI RAID 6. Which is why I dropped back to 2 port bridging, because it had the huge advantage of freeing up extra PCI Express slots.

All the best

Madeleine

 

I did order another 10G network adapter that will be coming in soon. I'll see if that's a bust or not also. Ordered it the same time I ordered the ATTO.

ADD-PCIE-2RJ45-10G by AddOn and it's basically an Intel X540 T2-A0.

I don't recall the website I got the drivers from offhand, but the file name is "Intel X520 - DRVR_Network_Intel_A01-V76GK_setup_ZPE.exe"


Clearly contains Windows XP in that list. However, the contents of the executable does not contain a folder pertaining to XP. Just from Vista to Windows 8.

I will update how that goes.

 

This appears to be the link where I got the X540 T2 drivers. A Dell website.

https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=v76gk

So, these are Intel drivers from Dell for XP, for X520, X540, X550.

Should work for XP, but I will find out with this other card I bought.