close

Privacy guaranteed - Your email is not shared with anyone.

Built a new NAS and it's fast

Discussion in 'Tech Talk' started by JimmyN, Aug 29, 2011.

  1. JimmyN

    JimmyN

    Joined:
    Sep 29, 2006
    Messages:
    1,266
    Likes Received:
    7
    Location:
    Virginia
    I built another NAS box, but this time using new hardware rather than 10+ year old stuff from my junk boxes. I knew it would be faster, but it has turned out even better than I expected. It's at least 3 times as fast and I was already getting 200Gbs+ transfers from my junk parts.

    Here's what I got from Newegg:
    Lian Li Lancool Case 6-3.5" internal bays
    AMD Athlon II X4 3.0gz CPU
    Asus motherboard 6 -SATA6 ports, USB 3.0
    Corsair CX 500 Power Supply
    4 gigs RAM

    Total price delivered to my door was $301.95 + $1.92 shipping.
    That is competitive with an off the shelf NAS with 6 drive capacity. The quad core CPU and 4gig RAM are overkill and I could have come in lower if I had backed off a bit but the price was OK, and it was a combo deal so I went with those choices.

    I used existing drives I had on hand, some SATA3, and some SATA6, 1.5TB and 2TB capacities. Installed FreeNAS .7.2.6694 (64bit), takes about 20 minutes. Finished it up yesterday after the power came back on from the hurricane, so I've been doing some testing this afternoon.

    The problem is I can't really load it up, the PC's I'm connecting with just won't read and write fast enough. I have to double up so the NAS is handling two PC's at the same time in order to get some idea of what it's capable of doing. If things aren't too busy tomorrow I'll do some system tweaking and add a 3rd PC for concurrent transfers.

    I was copying a Windows 7 iso file to test with, it's 3.1gigs in size. Just drag and drop it in Windows Explorer to the NAS and back. So these file transfers are using Samba/CIFS shares, which adds some overhead.

    Simultaneous writes to the NAS
    The left side of this first graph shows a ~575Gbs write from a Windows XP system to the NAS. That's as fast as that PC can read it from the drive and send it. The right half of the graph I started the copy again, plus added another file copy to the NAS from a second PC. The second PC only added ~260Gbs, it's slow and that's all it will do (IDE drives and PCI NIC), but the total throughput increased to ~680Gbs.

    You can see the first write start at ~575, then the second write from the slower PC starts and it jumps to ~680Gbs. The first one finishes up and it drops back to ~260 as the slow one finishes. I can copy a 3gig file to the NAS in 52~53 seconds with the faster PC.


    View attachment 218690


    Simultaneous write and read

    The second graph shows a simultaneous write and read. The slower PC is copying a file to the NAS, the faster PC is copying a file from the NAS. That is the fastest write speed I can get with that PC and the fastest read speed with the slow PC. So those speeds are actually PC limitations rather than NAS, but I wanted to see it with good traffic flow in both directions.

    View attachment 218688

    The Hardware
    The highest CPU load has had peaks to 12% (averages 6~8%) and temp is running 18C while it's working. Max RAM used has been 9% of 4gigs. If I let it sit for 20 minutes it idles down. All the drives stop and CPU drops to 900mhz while it waits. My Kill-a-Watt meter shows it's only drawing 38 watts when idle. The 3 - 140mm fans that came in the case are a big part of that, I need to slow them down.

    Anything this fast has got to be illegal, and I haven't even starting tweaking yet. I'm going to experiment with dual NIC's later using link aggregation to increase throughput and provide failover, my problem is going to be rounding up enough PC's to put a good load on it.

    So that's my latest FreeNAS box, and I thought some of you might be interested in hearing about the build and results so far.
     
    Last edited: Aug 29, 2011
  2. Pierre!

    Pierre! NRA Life Member

    Joined:
    Jun 20, 2003
    Messages:
    4,202
    Likes Received:
    388
    Location:
    Lovin Sparks Nv!
    ***> NICE <***​


    You got me drooling to put this together! It would be the *bomb* to attach Win7 Images to it... with all that speed!

    Nice Job JimmyN! Yer gonna get me to learn about Samba shares one way or the other... (LOL)

    Congrats!

    Looking forward to more *benchmark* mania!

    Patrick
     

  3. Linux3

    Linux3

    Joined:
    Dec 31, 2008
    Messages:
    1,399
    Likes Received:
    0
    Interesting. Which mb did you get? I looked at the combo deals and there are too many choices.

    I don't think you over did it with the 4 core AMD.
    Remember, each Ethernet controller needs it own core if you are seeking performance.

    Get yourself a 4 port Ethernet card for the NAS. If your systems you want to back up have 2 Ethernet controllers use 1 for the LAN and one for point to point to the NAS. Then you will appreciate the 4 cores and 4 gigs of RAM.

    Or better yet. Get this managed switch;
    http://www.dell.com/us/business/p/powerconnect-2808/pd

    Then trunk, bond or mirror (whatever you want to call it) the 4 ports on the NAS to the switch and configure them as one 4 gig pipe. Then you can back up and feed multiple systems all at the same time.

    I use Bonding a lot. Very cool technology.
    http://www.linuxhorizon.ro/bonding.html
     
  4. BlownFiveLiter

    BlownFiveLiter

    Joined:
    Mar 24, 2011
    Messages:
    194
    Likes Received:
    38
    Location:
    Chicagoland
    Wow, that's way overkill for anything I could ever need! I was thinking about buying a very cheap Atom board with RAID support and a picoPSU. Nice low power, low cost solution, just for serving up files.
     
  5. IndyGunFreak

    IndyGunFreak

    Joined:
    Jan 26, 2001
    Messages:
    26,595
    Likes Received:
    1,835
    Location:
    Indiana
    http://www.newegg.com/Product/Product.aspx?Item=N82E16813131731 -- Only thing that sucks about that one, is they went w/ Laptop RAM...

    http://www.newegg.com/Product/Product.aspx?Item=N82E16813131732 -- This board, is just awesome. If you don't have major, super power needs, it will work for you perfectly. Runs cool, and has plenty of SATA ports. I think Asus has a Mini-ATX version for a little more, if you're afraid of ITX.
     
  6. JimmyN

    JimmyN

    Joined:
    Sep 29, 2006
    Messages:
    1,266
    Likes Received:
    7
    Location:
    Virginia
    It was an e-blast special or a Shock 48 hour sale, everything I bought except the RAM was one or the other. They may not have that specific combo still available. I'll check my invoice for the number and see if they still have it listed.


    When you shop for a motherboard avoid the Realtek chipset for onboard LAN. If the motherboard you really want has a Realtek 8000 series (8111, 8168) you'll need another NIC adapter and disable the onboard LAN.

    FreeBSD UNIX and Realtek don't play well together. FreeBSD includes a Realtek driver but the Realtek chipset works so poorly with FreeBSD that they don't even include the driver in FreeNAS since it's primary function is networking.

    I have a Realtek 8111/8168 driver that I compiled from source from Realtek, using FreeBSD 7.3, You can put it in the /boot/kernel directory and add an "enable" line to loader.conf so it will load at boot. But it won't give you the best speed. And if you use the 64bit FreeNAS I can't help you at all with that, since I've never found Realtek source code for a 64 bit driver.

    If you install FreeNAS .7.x on a system that has a Realtek 8111 it won't even recognize the chipset. Your only option when installing and assigning the adapter will be "Auto Detect", since the adapter won't be listed to select, and it's never successful since there is no driver.

    You'll get the best results with Intel chipsets, especially the CT series which are well supported in FreeBSD. This is an excellent adapter for FreeNAS, it's PCI-e, and reasonably priced (free shipping too).
    http://www.newegg.com/Product/Product.aspx?Item=N82E16833106036

    Two or more of those will parallel real well with FreeNAS if you want to use link aggregation (LAGG in FreeNAS). But if you don't have a switch or router that can handle LACP you'll need to use load balancing for the dual NICs rather than LACP.
     
  7. JimmyN

    JimmyN

    Joined:
    Sep 29, 2006
    Messages:
    1,266
    Likes Received:
    7
    Location:
    Virginia
    Sorry I was late getting back to everyone this morning it's been busy. I thought some might be thinking about building a FreeNAS box, so I wanted to throw out a few tips.

    The motherboard selection is not critical, having enough SATA ports for expansion is the main criteria. And avoiding the Realtek LAN chipsets, or plan from the beginning to use the Intel adapter I linked to above.

    I would recommend the 64 bit FreeNAS and a "Full" install (the OS+Swap+Data option), rather than "Embedded". The embedded is very small and the OS only takes 128megs, you can install it on a thumbdrive or CF adapter and boot from that each time. But it limits what you can do later, since none of the OS resides on a hard drive each boot is a brand new world and it's hard to make some configuration changes outside the webgui and have them survive a reboot.

    I use an old IDE drive that's too small for anything else, I think it was a WD 40gig that I put in the new box. You'll have an IDE controller to plug it into so it won't take any of your SATA ports. Then you can do a full install and you'll even be able to add some FreeBSD packages since the OS will have space if you're not using "Embedded".

    I installed "iperf" last night using 'pkg_add -r iperf' from the console command line. It will fetch the correct version for your kernel and install it automatically. When the prompt comes back type in "iperf --help" and you'll see it's installed and ready to go.

    I'd set the OS partition at 1gig or more, no swap partition since it won't use it anyway, and the rest of the drive will be formatted as a DATA drive. You can use it or ignore it. I store php scripts I'm playing with in that space, text files with system notes on what I've done, etc.

    If you plan to use the 128bit ZFS filesystem, which I highly recommend for data protection, you'll need to use the 64bit version of FreeNAS and have at least 2gigs of ram. ZFS uses RAM for cache space and will use all it can get.

    Do not use a hardware RAID if you use ZFS, it won't be able to provide data integrity checking and automatic error correction. ZFS is unlike any other file system since it's both a filesystem and RAID combined, it's probably the best filesystem ever developed as it will find and correct disk errors. Every week or so do a "scrub" command and it will look for errors and fix them. If you want to use RAID use ZFS RAIDZ, which is equivalent to a RAID 5, or use RAIDZ-2 which is double parity and similar to a RAID 6. But unlike RAID 5 or 6 ZFS will actually repair any corrupted files on the disk using information from the other drives in the array. With a 4 disk RAIDZ-2 any two of the 4 drives can fail simultaneously and keep on working. Replace the drive(s) and it will rebuild the array from information on the remaining two or three drives. The rebuild process will take quite a while.

    All the configuration is stored in a single file, "config.xml" That is the only file you'll need to restore the OS and your setup should your boot drive fail or you manage to mangle the system up. It's in the conf directory. "/conf/config.xml"

    Put it on a thumbdrive and save it. That file plus your install CD is your complete operating system backup. If you need to re-install the OS you can then copy the xml file back to /conf and all your configuration, drives, mount points, shares, users, etc, will be just like they were.

    I have quite a few years experience with FreeNAS, plus I use FreeBSD, so if anybody has problems, like how to mount the thumbdrive, I can help you out.

    I ran iperf last night to test the network, from PC through the switch to the NAS, and I'm getting 940mbps. That's no load speed, I'll test some more with a load on the network. It's probably about as fast as you can get on a 1gig LAN.
     
  8. BlownFiveLiter

    BlownFiveLiter

    Joined:
    Mar 24, 2011
    Messages:
    194
    Likes Received:
    38
    Location:
    Chicagoland
    I actually have an Atom ITX board already that I could use, but I was thinking of using it for a server on colocation somewhere (Supermicro X7SPA-HF). For home, I was thinking more along these lines. I don't need anything more powerful than that. I'd probably pick up a PCI RAID card and a few 1-1.5TB drives, so I could set up RAID 5. I'd really only use it for a file server, so I don't have to store everything locally on my PC. I have my laptop, plus two other PC's that I use, so having the files available on each system would be nice. I've contemplated building a HTPC, so the home server system would be nice, for storing movies that I could stream over the network.
     
  9. IndyGunFreak

    IndyGunFreak

    Joined:
    Jan 26, 2001
    Messages:
    26,595
    Likes Received:
    1,835
    Location:
    Indiana
    The board itself, has some limitations, mainly, the PCI slot. If it were PCI-e slot, at that price, it's tough to go wrong, but don't expect much out of Foxconn boards PCI is quickly becoming a relic, and it seems counter productive to put something like that in a new NAS.

    Guess it's all about what you're looking for though.
     
  10. Linux3

    Linux3

    Joined:
    Dec 31, 2008
    Messages:
    1,399
    Likes Received:
    0
    JimmyN. I bow before you!
    I did not realize FreeNas comes with ZFS support. I am a huge fan.
    My company buys some pretty expensive Sun /Oracle Sun boxes and Solaris 10.X to run ZFS.
    I have to try this.

    Learn something new every day.
     
  11. IndyGunFreak

    IndyGunFreak

    Joined:
    Jan 26, 2001
    Messages:
    26,595
    Likes Received:
    1,835
    Location:
    Indiana
    ZFS is pretty awesome, I admit.
     
  12. JimmyN

    JimmyN

    Joined:
    Sep 29, 2006
    Messages:
    1,266
    Likes Received:
    7
    Location:
    Virginia
    Linux3, please don't bow... my head will swell up and bust wide open.:supergrin:


    I didn't have much time yesterday, it was a busy day but I got back to working on it again this morning. It killed me not getting to play with my new stuff, but that pesky job gets in the way.

    I've adjusted window sizes, send/receive buffers, etc, and I still can't get above my original setup and ~680mbps on file transfers. I'm thinking I've hit a Samba/CIFS limit. I can probably do better with some FTP or NFS from my FreeBSD box thrown in, but that's gravy anyway. Since it's on my home network 95% of my file transfers will be to/from a Windows system so CIFS speed was my biggest concern and the reason I was using Windows boxes to test file transfer speed.

    I did load the network to see what it's cap is. The graph shows two file transfers to the NAS and while they were copying I also ran iperf to max it out.
    I'm getting 968~975mbps now which is probably the best I can hope for. Only in IT heaven can you get 1000mbps through a 1gig pipe.

    Jitter is only .34 to .43ms at ~970mbps so I'm happy with that, everything seems to be in harmony. I tried using up to 128K window sizes but speed didn't increase and jitter did. I get the best overall performance with 64K windows.

    So today I'll get back to stress testing the hardware. I started running memtest yesterday while I was out and got 8000% coverage with no errors, RAM looks good. Right now it's calculating Mersenne primes which really gives the CPU a workout.

    Normally I would do hardware burn-in and stress testing first, but being 62 years old going on 16 I got impulsive and had to play some first.

    View attachment 218715
     
  13. Linux3

    Linux3

    Joined:
    Dec 31, 2008
    Messages:
    1,399
    Likes Received:
    0
    ~680mbps sounds pretty good to me. That's ~85mB/sec. How many systems are you backing up at one time?
    Don't forget the real speed limit is the read speed of the client drives. You can write to the best RAID in the world but it will run no faster than the speed of the client drive.
    Samba has HUGE overhead but then any kind of GUI based file transfer app will too. I believe you can get NFS for Windows and I think that would be faster.

    Every kind of file transfer has overhead so that 1 gig pipe thing is really just a marketing ploy......


    I can sign up for Social Security at any time.
    Told the boss, Don't piss me off or I might!
     
  14. JimmyN

    JimmyN

    Joined:
    Sep 29, 2006
    Messages:
    1,266
    Likes Received:
    7
    Location:
    Virginia
    It's scheduled and only backs up one system or remote NAS at a time. But my old FreeNAS has been taking care of that chore for over 3 years now, it's set up and has always worked perfectly, so it will still do all the backups. This box is primarily for media streaming on my network and HDTV recording/playback.

    As the collection of Music, DVD, and Bluray has grown I was running out of space since it had a limited number of drive bays and controllers in that box. An old Gateway Pentium 4. At 23~48gigs each Blurays eat a lot of space. And it wasn't fast enough either. I could stream a DVD and a Bluray, but now I'll be able stream Bluray upstairs in the bedroom and on the HTPC downstairs at the same time. That need arises when the kids and grandkids are around on the weekends, and I can also record HDTV now while I'm streaming a movie out.

    Before anyone says anything I don't condone piracy. Every movie on my NAS I have the original disk in the rack. But 8 year olds, even if they are actually 30, and optical disks are an accident waiting to happen, so nobody touches anything in my disk racks.

    I know everybody loves pictures, so here are some photos. I really like Lian Li cases, good cable management and the punched holes have rolled edges so there's nothing sharp. All the drives are mounted with rubber grommets and thumbscrews to cut vibration. Really nice cases.

    You can click on an image to see a larger version

    [​IMG]


    [​IMG]


    [​IMG]


    The back side. I took this before the front views, so it's still missing a drive.

    [​IMG]


    [​IMG]


    You may notice in the pictures that is has two Intel NIC's in the PCI-e slots. A local computer store that I have built about a dozen custom gaming rigs for had asked me about NAS's. They wanted me to OEM them so they could brand them under their own name, like they have done with the gaming rigs. I tend to be a perfectionist so not only do they look good internally but they've never had any problems with systems I've built. So now they're interested in NAS's. And NAS's are much easier since I won't have to do UV sleeving and connectors, lighting, LED temp/fan displays, or background music.

    But they've wanted a prototype to test with two NIC's included for failover and load sharing, and at least a 6 drive capacity. So I set it up with dual NIC's and took the box over to them this morning so they can play with it this weekend. They already called me back and want to keep it, "What's it gonna cost us?" NO, NO, that's mine. "Well can we keep it here as a demo?" NO, NO, NO, that's mine. Give me some specs on what you want and I'll give you a quote, but Tuesday morning I WILL pick that one up. :rofl:
     
    Last edited: Sep 2, 2011
  15. gemeinschaft

    gemeinschaft AKA Fluffy316

    Joined:
    Feb 7, 2004
    Messages:
    2,202
    Likes Received:
    58
    Location:
    Houston, TX
    Very nice.

    I like it.

    I would also like to comment that your work is very nice and tidy, that says a lot.
     
  16. Linux3

    Linux3

    Joined:
    Dec 31, 2008
    Messages:
    1,399
    Likes Received:
    0
    JimmyN, a new career is always fun and exciting.

    Nice workmanship. It's something that is hard to find.
     
  17. pal2511

    pal2511

    Joined:
    Sep 15, 2002
    Messages:
    3,982
    Likes Received:
    0
    Location:
    KS
    You have a link to the exact lian li case that is on newegg or another site?

    I am looking for a case that will let me grow and add multiple drives to in the future. I am going to use it for my htpc and just put it behind my tv stand.
     
  18. JimmyN

    JimmyN

    Joined:
    Sep 29, 2006
    Messages:
    1,266
    Likes Received:
    7
    Location:
    Virginia
    This is the one I got for the FreeNAS, it's $79.95, free shipping. It is steel with 6 - 3.5" internal bays. Or you can put 2 -2.5" drives and 4 -3.5" in the six bays. I got mine on an e-blast special when it was on sale for $59.95
    http://www.newegg.com/Product/Product.aspx?Item=N82E16811112322

    This one appears to be identical except for the smaller window in the side panel, but it's $10.00 more + $9.99 shipping.
    http://www.newegg.com/Product/Product.aspx?Item=N82E16811112319

    If you'd prefer aluminum this one has 5 - 3.5" internal and 1 - 3.5" external, which you can use as an internal. But it's a top mount PSU and it's $89.90 but shipping is free.
    http://www.newegg.com/Product/Product.aspx?Item=N82E16811112154

    They don't have any special sale on cases at the moment.
     
  19. pal2511

    pal2511

    Joined:
    Sep 15, 2002
    Messages:
    3,982
    Likes Received:
    0
    Location:
    KS
    Case looks very good. Im debating between that or a cooler master storm

    Any ideas on what I should get for my main drive? I am using it for HPTC/file backup. I have a 1.5tb along with a 160 gig ide right now. I want to dump the iDe that is my system drive for a 1 tb or 2tb drive. Just trying to find one under $100
     
  20. JimmyN

    JimmyN

    Joined:
    Sep 29, 2006
    Messages:
    1,266
    Likes Received:
    7
    Location:
    Virginia
    A 1TB under $100 won't be hard to find. Newegg has a 1.5TB Seagate Barricuda on sale, it's a SATA 6.0 5800rpm green drive. $64.95 plus another $10 off with promo code EMCKAKD22, and free shipping.

    And they have a HITACHI Deskstar 3TB 7200 RPM SATA 6.0Gb that is normally $179.99 for $149.99 with promo code EMCKAKD23 and free shipping.

    Note that these are OEM drives. No cables, no screws, no CD. Just the bare drive in a brown cardboard box. They just had a bunch of sales over Labor Day Weekend and back to school specials, so today there aren't many hard drive deals as they ended yesterday. But if you wait a week or so some more will show up.