I figured that the switch wasn't setup for 100Mbit/full, so I sent an email to the manager of that switch and he replied that all ports were forced to 100-fd.
After googling, I came accross this table which outlines all of the possible mis-matches in switch/nic configuration that could cause duplex-mismatch and create a ton of collissions.
I forced the server's nic to 100fd and was able to transfer at 80Mb/s again.
Then I started thinking...At our remote location we were experiencing terrible transfer rates of ~200Kb/s on the Internet. I double checked, and found that the firewall's (a freebsd 6.1 machine) internal nic which is serving as an 802.1q vlan trunk had autonegotiated to 100Mb-half-duplex!
I logged into the switch at the co-location and forced all ports to 100-full then did the same with all nics on the server.
After the switch & firewall renegotiated, I was able to get 40Mb/s transfer rates through the firewall to the internal servers.
Unbelievable.
