How (not?) to create a large, expandable storage/backup server

Published by on Sat, 10/23/2010 - 01:00

Backup to tape sucks. That's why we started looking at large NAS devices to be able to back up all of our data to disk. It would be much easier and faster than our current tape jukeboxes that consistantly have hardware issues, or have tapes failing, or run out of room. The problem is, we have a lot of data, and we'd like to be able to keep a few versions of said data. We also have been burned enough times by RAID 5 that we wanted something more redundant. And, to top it all off, we're on a budget. So here's a basic rundown of our requirements:

  • ~$1800 or less
  • ~6TB of usable storage space, but expandable
  • Can handle more than a single disk failure
  • Preferably not tied to particular hardware
  • Supports at least CIFS/SMB, NFS, SSH and preferably some RSYNC

It's a fairly short list, true, but much harder than you would think to put together. Pre-made NAS devices just weren't going to fit all of our criteria, so it was time to look at rolling our own. First, I found this case on sale for $300. This was the starting point I needed. It has 20(!) slots for drives, plenty of cooling fans, and we can put whatever hardware we want in it.

So I came up with a plan. I would get this case, a motherboard, power supply, etc. and use ZFS to create a nice large pool of storage that we can carve up as we need. ZFS is an amazing filesystem for large storage pools like this, but it's only supported on a few OSes, none of which are my forté; basically some flavor of Solaris or FreeBSD. I decided to go with OpenSolaris, since it has the best native support for ZFS, and at the time (this was several months ago), arguably the largest community of the Solaris ports.

I also briefly considered NexentaStor Community Edition, but it only allows up to 12TB, and this box has the potential for far more than that. Nexenta also has a Core Platform, but like BeleniX and SchilliX, there didn't seem like any real advantage over plain OpenSolaris. FreeBSD seems to have pretty good support for ZFS, but setting it up as the filesystem for the root drive (thereby allowing you to mirror the root drive, yay, redundancy!) is difficult to say the least, and the learning curve of FreeBSD can be a bit much when coming from LinuxLand.

Why ZFS instead of a nice fast hardware RAID? Well, for one thing, price. For another, we didn't want to be tied to a single type of RAID card. And most imortantly, because ZFS rocks! It has RAID built in, at single, double or triple parity. It can have as much space in one single pool as you can possibly throw at it. It has built in error checking and checksumming. It can compress files on the fly and de-dupe them at the same time. You can snapshot it to your heart's content. It slices, it dices, etc., etc. It really is pretty amazing. I'm surprised Sun and Nexenta are the only companies that seem to be taking advantage of this.

I started shopping around for the rest of the stuff I would need to make this happen. Here's the hardware I purchased out of the gate:

The idea was to hook 4 drives up to the motherboard, 4 up to the RAID card (which we wouldn't actually use the RAID on) and 2 laptop drives we had hanging around up to the remaining 2 SATA ports on the motherboard for the OS. Then, when we needed more space, we could add another PCIe x1 card with 4 more SATA ports, add 4 more drives, rinse, repeat. I went with the HighPoint RAID card because budget constraints were such that a beefier card that could handle all of the SATA/SAS connections in one was out of the question. And I wasn't having any luck finding a cheap board with at least 4 PCIe x4+ slots that was cheap enough, either. The Gigabyte board has 4 PCIe x1 slots, though, and it's a really nice board.

Problem #1, the Promise breakout cables I got didn't work. Little did I know I actually needed to have reverse breakout cables. So I returned the Promise cables and got new reverse (and cheaper) ones, plus an adapter to make it easier to plug in the cooling fans behind the drives, and some SATA power to Molex connector adapters:

So I got it all put together, and everything was up and running. Then the next problem reared it's ugly head. OpenSolaris doesn't support the HighPoint RAID card I bought. Not a great start so far.

I started looking for cards that were compatible with OpenSolaris, and would fit in the 4 PCIe x1 slots I had available. I came across This set of articles talking about doing something very similar to what I wanted to do. Would that I had found it prior to making my order. He had no luck finding a card that would work in OpenSolaris that was cheap and didn't have RAID capability, so he ended up going with a very expensive hardware RAID card, and gave up on the dream of ZFS. I decided to use an OS that supported the card, instead. ZFS is supposedly fairly stable in FreeBSD, so I figured I'd go with that.

I loaded up Freebsd 8, and find out that the card isn't supported out of the box, you have to install HighPoint's binary drivers. Unfortunately, there aren't any drivers for FreeBSD 8. Argh!

I don't really know FreeBSD very well, but I have used FreeNAS effectively for another NAS, and it's based on FreeBSD 7.2, which is supported by HighPoint. So FreeNAS it is. FreeNAS uses version 6 of ZFS, which I wasn't particularly crazy about, but it has a great interface, is easy to set up and maintain, and allows us to use the hardware we already have.

Set up was a breeze, getting the drivers up and running wasn't a problem, but in order to get the drives hooked up to the RAID card to show up I had to make each disk it's own JBOD. ZFS likes to have access to the raw drives, but it didn't seem to hurt performance much, and it worked, which is more than I can say for previous attempts. I was pretty happy with the results. I had a big storage server with a good web interface that fit all the criteria.

One oddity that came up was that the S.M.A.R.T. status of the drives hooked up to the RAID card were unavailable. Not a big deal, but it should have been and indicator of what was to come. I started testing to see what would happen in the event of drive failures. I pulled a drive from the 4 hooked up to the motherboard, and a drive from th 4 hooked up to the RAID card. The one hooked up to the mainboard showed up as missing, but the RAID attached drive did not. Not good. Would the OS keep trying to write data to the missing drive? I have no idea. And I wasn't about to find out.

The search for my holy grail continued. The RAID card wasn't going to work. I needed a PCIe x1 card that could handle 4 SATA connections, and didn't have a RAID. I searched high and low, and finally found the SYBA SY-PEX40008 at mwave. I was a little worried as the reviews on newegg were not very flattering, and it has RAID capability, but it was only $50 and I was desperate. I also found this page, on which a Mr. Stephen Green described having pretty good luck with it as a non-RAID card. Under OpenSolaris, nonetheless!

I ordered the card and came full circle back to OpenSolaris. I installed the latest stable version, 2009.06, which went very smooth. There was plenty of help online to get me going on an unfamiliar OS. Now I had native ZFS for both the root drives as a mirror and the storage drives as RAIDZ2. I can create a dataset in the pool with one command. I can add NFS or CIFS/SMB sharing to that dataset at the same time, no problem.

It's great! Except for the fact that the SMB server keeps hanging. What. The. Hell?

I started searching online, and it seems I'm far from the only one experienceing this problem. At least one person seemed to think that it was triggered when you tried to browse the share using Windows Explorer. Not that anyone would ever browse a SMB share using Windows Explorer. People were saying it seemed to be fixed in later versions of the SMB server, but you can't just install the new version as it's part of the dev version of OpenSolaris that will become 2010.03 (which apparently will be at least 2010.04 at this point).

I installed the dev version. It wouldn't be my first choice, but SMB was unusable as it was, and that made the whole server somewhat less than usable. The install itself went fine. I love how it creates new Boot Enviroments when you do a full upgrade. It makes it much easier to go back if things go awry. Did I say 'if?' Oh, I meant 'when.' It turns out there are a bunch of caveats when doing an upgrade to the dev version. Like not being able to use the command line at all, etc. After a little searching I found a whole page dedicated to fixing the little issues that crop up after the upgrade. Luckily, all the fixes Just Worked™ and I was up and running.

We moved a bunch of files over to the new box and started sharing them through SMB, but some of the files were coming up as missing, even though they were definitely there. I banged my head against permissions and ACLs and such to no avail. Then I noticed that the file that was being looked for was mixed case on the OpenSolaris server and lowercase on the Windows box. Turns out the best way to create a dataset for SMB sharing isn't the default. After following the advice on this page, we were officially up and running.

Or so I thought. The pain continued. Everything works, but mysteriously the machine will just hang when making data transfers through SMB. And, worse, OpenSolaris has been killed by Oracle. So where to go from here? My best 2 options are Nexenta Core and OpenIndiana. At the moment, I'm leaning toward OpenIndiana, for ease of installation and minimal reconfiguration. I basically just upgrade my current OpenSolaris install. I'm going to keep my eye on it for a while before making my decision, though. I think the future of whatever OS I go with will be the determining factor here.

Here are some the most helpful OpenSolaris pages not mentioned above that I found: