This guide is a good way of doing an inexpensive to moderately priced NAS or SAN while delivering almost enterprise grade performance. The “not” in the title refers to my impatience waiting for a case and PSU to be delivered for 2 weeks.
NAS/SAN vs Drobo
Since I have been encoding all of my Blu-ray discs and DVDs for use over my LAN, I have used all of my hard drive space pretty quickly. Computers don’t perform well when they get down to around 10% free space (depending on the operating system). I tried to fix this problem by buying an incredibly overpriced Drobo but this slows down computers even when it has nothing on it (my computer hangs for about 4 minutes when trying to load the 16TB volume over USB and also adds the same amount of time to your boot up if you leave it plugged in). Add this to the insane price of £315 without any hard drives or network capability (to turn it from a DAS into a NAS you need to buy a DroboShare for an additional £140 and this still only gives you USB performance because that’s how it’s connected) and the fact that it has a lot of bugs that you can’t even see before you buy it because a serial number is required to access their community support and you end up with a big shiny black POS.
Compare this £450 lump to what you can build yourself for a NAS. I got a Gigabyte GA-MA785GT-UD3H motherboard (£75), which includes onboard graphics (VGA, DVI and HDMI ports), gigabit ethernet and 6 SATA 3gbps ports as well as an AM3 socket and DDR3 memory slots. I also got an AMD Athlon II X2 240 CPU (£45) and a 2GiB kit of Corsair XMS3 1333MHz 9-9-924 (£37). I would have gone all AM2 for about half the price but I couldn’t find any AM2 CPUs any more.
Any bootable USB stick is fine to put an embedded OS on (I use FreeNAS) and I’m using the Lycom UB-109 internal to external USB adapter to mount it directly onto the motherboard’s internal USB port, though just putting it in one of the rear USB ports would also be OK. This is just tidier and you don’t have to worry about it being pulled out by someone that thinks it was left there accidentally.
So far we are up to just £157 and only need a case and a PSU. If you were to go for a straight comparison, any old case that holds 4 hard drives would do and any 300W or so power supply would also be fine. However, it’s not hard to do better than that. As I have a small rack already, I got a 2U Chenbro RM23212 case (£217). This also meant I needed a 2U PSU. This is a rack mounted case that allows 12 3.5″ SAS/SATA hard drives to be hot swapped in the front. I found that prices for it varied wildly so I got the cheapest one I could find (still not cheap!). I assume this just means it comes with no PSU or backplane but both of those are fine by me, even if it means that hot swapping isn’t so hot. For the PSU I got a SuperMicro PWS-652-2H (£127.55). Also not cheap but I needed a PSU that could fit and potentially power 12 3.5″ hard drives.
That brings us neatly up to £500. £50 more than a Drobo with DroboShare but the same thing could easily be done much cheaper if rack mounts weren’t involved (probably around £200 – cheaper if you have some old parts lying around). You do get a lot more for the money though. As the hard drives are not included in the price of the Drobo, I have not included them here.
Storage
For hard drives, I got 4x 1TB Seagate 7200.11 drives. These were about £55 each and in my experience, transfer data dramatically faster than any other 7200rpm SATA drives. The reason that I only got 4 drives is that I am using ZFS and zpools on FreeNAS. It’s pretty much a form of software RAID so the CPU will be used but this is a NAS box – that’s all the CPU is there for. It means that I can create a raidz with single parity (essentially RAID5 without the write hole – dual parity RAID6 also exists as raidz2) for 4 drives, getting 75% of the space available for data and then add more raidz groups to the pool later. Note that it is still experimental in FreeBSD but the alternative is splashing out on massive RAID cards and either spending much more on RAID10 or dealing with the RAID5 write hole. The only other platform that supports ZFS at all is Solaris.
I could later add another 4x 1TB drives and then, when they are more economical, 4x 2TB drives. All of the drives in a single raidz have to be the same size but the sizes of the raidz groups do not have to match inside the zpool. After I have run out of space for adding new hard drives, I can replace the drives in the raidz and as soon as all 4 have been replaced it will grow to use the new space. This makes the possibilities for expansion almost unlimited. Because it is done in software, adding more drives can be done with either drive multipliers/expanders or dirt cheap RAID cards that have a lot of ports but no real processor or memory.
Getting impatient
I ordered the internal components from scan.co.uk and got them the next day. Two weeks later and I still don’t have a case (ordered from memory-express.co.uk) or PSU (ordered from pconestopshop.com). Needless to say, I will never be buying anything from either of them again. I’ve already called up about the case twice.
Getting desperate, as both of my hard drives are full as well as my MythTV/video encoding server, I continued anyway. I stole a PSU from my brother’s PC, which now looks like it’s been passed through a shredder with the amount of parts missing and cables hanging out. I didn’t use a case at all. I used a corsair dominator memory cooler and a big desk fan for cooling. I also laid out the hard drives separately to make sure heat didn’t build up and to avoid any short circuits.
This is how not to NAS. (use a case!)
Installing FreeNAS
I had major problems trying to install FreeNAS from CD. It seems to only want to install from the primary master IDE even though I was using a SATA drive. I tried using IDE emulation but still had problems unless I plugged it into one specific SATA port after trying all 6. I also had to try all of these with and without the real IDE turned on. There aren’t many things more annoying than loading an installer from a CD and then being told that you have no CD drive. In the end, it finally did find the CD drive but then wouldn’t let me install to the USB drive because it had already mounted it and put a config.xml on it (the default behaviour is to boot off the CD and just use the USB for some persistent storage). It wouldn’t let me unmount the USB drive either.
Another problem was that I was getting a panic: ohci_add_done error. I resolved this simply by unplugging the USB keyboard and plugging it into a PS/2 port. Strangely, removing the keyboard during boot and then plugging it in just when I needed it (to assign a network address) also worked.
The best way I found to install it to the USB drive was to put the CD and the USB drive in a Linux machine, gunzip the FreeNAS embedded image file and then use “dd” to image it on to the USB drive.
cp /media/cdrom/FreeNAS-amd64-embedded.gz . gunzip FreeNAS-amd64-embedded.gz dd if=FreeNAS-amd64-embedded of=/dev/sdb
where /dev/sdb was my USB drive – this is very important or you may wipe a hard drive.
I then just put the USB drive back in to the new machine and unplugged the CD drive’s SATA cable. It booted without further problems.
You may need to load the console and set an IP address for the network interface but after this you should not need the console again. If you do, you can always turn on SSH from the web GUI. I unplugged the keyboard and screen at this point.
Setting up these drives is incredibly easy. It can be done from the FreeNAS web UI but I used the following command:
zpool create datapool raidz ad4 ad6 ad8 ad10
Where ad4 to ad10 were my 4 hard drives set to AHCI mode in the BIOS.
Though they can be added later with a similar syntax, creating it with more than one raidz in the pool is also easy:
zpool create datapool raidz da1 da2 da3 da4 raidz da5 da6 da7 da8
You should “synchronize” the web UI after doing this under the ZFS tab, just so that it shows the correct information.
I then added a read-only samba share to /datapool/ and a read/write FTP with a user name and password, unchrooted (though chroot to /datapool would be ideal). This allows me to upload files to the NAS via password protected FTP while still allowing all of the computers in my house easy access to the files without needing user names and passwords and without being able to accidentally damage anything. I changed the file creation umask to 022 (default 077) so that files are created as 0644 and directories are created as 0755. To be honest, these seemingly inverted (XORed?) umasks confuse me but I just stick to 022 and find that ideal.
I get write speeds of roughly 65 to 90MB/s over FTP on copper gigabit Ethernet. The little Athlon II X2 CPU is very busy during these writes but out of the 2GiB of RAM, only a constant 30% is ever used. It is well known that the ZFS code on FreeBSD needs optimising so I will probably check back in the future and try a patch. This also means that write speeds could be increased by adding a faster CPU. I haven’t actually checked if it is using both cores at the moment (SMP) but I assume it is. ZFS performs better on 64bit apparently and that is what I am using. It is also possible that the client uploading the files was causing some bottleneck but I usually get 100 – 110MB/s between the two used.
In the future, I may add a second Ethernet card and use LAGG for load balancing. However, this only really helps for multiple concurrent clients and it requires a switch that supports it. I have a Dell PowerConnect 2716 switch (it was £160 with a £100 off deal at the time) which does support this.