The replacement SATA adapter came in this week. I chose a Silicon Images 3124 based PCI-X adapter:
I've had good experience with Silicon Images adapters in the past, but I always run them in HBA mode with software RAID. This particular board seems to be their "reference implementation" for the 3124 chipset. As it came, it would lock up on boot when a drive > 500 GB was connected to it. I had to flash the latest firmware (version 6.6.0) available from their website.
Apparently hardware manufacturers still use MS-DOS for BIOS updates. This is a problem because most people don't have a DOS system running anymore, and the machine I was installing on has no floppy/CD drive. Fortunately, you can use FreeDOS to do it. I used a prebuilt 256 MB image from here:
Copy it to your flash drive with `dd` and then mount the drive to copy over the BIOS image and Flash loader. Silicon Images provides two images for the 3124 in a standalone adapter. One is a "base" image, which is SATA HBA mode only, the other is a SoftRAID image. I used the base image as I have no interest in software RAID.
Here's a pic of the card installed:
The card fit fine even though the previous card had the SATA connectors on the rear of the card (opposite the card bracket). The new BIOS fixed the system hang and FreeBSD booted from the old root drive -- I had to mount the root partition manually since the device name changed from da0 to ada0.
As soon as I tried to create a new raidz volume, I immediately started getting hardware disconnect errors from the SATA controller. I'd experienced these errors before, so I grabbed the hot swap tray out of the second Rackable Systems chassis I have. Same errors on the same slots. I was mostly convinced that either both hot swap backplanes were bad or they wouldn't support SATA II datarates, but decided to try the Silicon Images card in the second system:
Good thing I did, because this system is currently up and running with zero errors! I haven't yet determined if the problem is in the motherboard or the PCI-X right angle riser from the other chassis. In any case, I've been copying data back and forth to the box all afternoon in an effort to give the disks a workout. Here's a `df -h`:
$ df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/ada0p2 897G 28G 797G 3% /
devfs 1.0k 1.0k 0B 100% /dev
storage 3.6T 120G 3.5T 3% /storage
Excellent, 3.6 TB of storage! Disk-to-raidz transfers are > 150 MB/sec on the same controller, and I've achieved > 290 MB/sec from ramdisk to the raidz array with compression turned on.
I also gave link aggregation a try using the FreeBSD lagg(4) driver. My switch is a layer 2 managed device, and supports LACP, so I tried it. It works in the sense that I get an aggregated link with fault tolerance. The bandwidth with the two onboard Broadcom PCI-X GbE interfaces is less than 45 MB/sec. So Broadcom cards still suck.