GlockTalk.com
Home Forums Classifieds Blogs Today's Posts Search Social Groups



  
SIGN-UP
Notices

Glock Talk
Welcome To The Glock Talk Forums.
Reply
 
Thread Tools Display Modes
Old 06-30-2013, 17:11   #1
CitizenOfDreams
Senior Member
 
CitizenOfDreams's Avatar
 
Join Date: Feb 2009
Location: Orlando, FL
Posts: 8,117
What would I gain from a "real" RAID?

I know "fake" software RAIDs are laughed at in the IT community, but I have been happy with mine (4 drives in RAID10, Intel ICH10R). It had adequate speeds and no noticeable CPU load. The array properly rebuilt itself when I faked a drive failure.

Now I am planning to build another computer, so I have the following choices in storage configuration:

1. Intel motherboard with software 3Gb/s RAID;

2. A $300 more expensive Asrock motherboard with "half-hardware" 6Gb/s LSI SAS2308 onboard controller;

3. A separate "half-hardware" RAID controller (which one?);

4. A real hardware RAID controller (does not fit my budget).

So, which option should I choose, and what exactly will I gain from a better RAID controller? I am planning to use the machine for video editing and light gaming (think Starcraft, not Halo 6).
CitizenOfDreams is offline   Reply With Quote
Old 06-30-2013, 20:05   #2
harrygunner
Senior Member
 
Join Date: Sep 2010
Posts: 408
Really should think about RAID with today's multi-terabyte drives. We're getting to the point where the "Unrecoverable Bit Error" (UBE) of drives is near the size of drives.

So re-silvering (rebuilding) a RAID group after a drive failure requires nearly the number read/writes as the drives are spec'ed for. If a UBE occurs or two or more drives fail, you've effectively lost your data.

Personally, I'd go either go RAID1 or forget RAID. In all cases, back up your data. Tape drives cost a bit, but you can store multiple copies off-site and recover data much faster than over Internet. Pipe data through compression, then encryption before writing to tape and it's secure.

As far as hardware RAID controllers, the main downside is if anything proprietary is written by the controller and either that version becomes unavailable or the company goes out of business, you'll have trouble if the controller fails.
__________________
People who've had to deal with their karma are more interesting to talk to.
harrygunner is offline   Reply With Quote
Old 07-02-2013, 22:14   #3
CitizenOfDreams
Senior Member
 
CitizenOfDreams's Avatar
 
Join Date: Feb 2009
Location: Orlando, FL
Posts: 8,117
Thanks for the input!

RAID10 is what I want. A combination of capacity and speed, plus some level of fault tolerance. And yes, I know the importance of backups (learned it the hard way, like most of us).

For now, I am going to settle on onboard "fake RAID", for budgetary reasons. Maybe one day I will uprade to a real controller, but I would like to play with one before I commit.
CitizenOfDreams is offline   Reply With Quote
Old 07-03-2013, 00:50   #4
Bushflyr
ʇno uıƃuɐɥ ʇsnɾ
 
Bushflyr's Avatar
 
Join Date: Mar 1999
Location: Western WA
Posts: 4,464
Quote:
Originally Posted by harrygunner View Post
So re-silvering (rebuilding) a RAID group after a drive failure requires nearly the number read/writes as the drives are spec'ed for.
Sorry, but you have no idea WTF you're you're talking about. I've rebuilt my RAID after failure about a dozen times, 5 and 6,
there's no way possible that it's even coming close to wearing out the drives.


Quote:
Tape drives cost a bit, but you can store multiple copies off-site and recover data much faster than over Internet.
Tape!?! TAPE ?!!? Sorry bro, you're stuck in the last century. Tape drives are beyond dead tech.

Quote:
As far as hardware RAID controllers, the main downside is if anything proprietary is written by the controller and either that version becomes unavailable or the company goes out of business, you'll have trouble if the controller fails.
At least you got one thing right.

Which is why I recommend software RAID. You give up a little performance, which most of us never notice, and gain the comfort of never having to rely on hardware that will likely be obsolete and difficult to replace when it fails. I've had hardware that I need go obsolete then fail so many times it isn't even funny. We still use 1gb PCMCIA cards at work that are stupid expensive compared to modern tech but we can't change because they're TSO'd.
__________________
...the secret is to bang the rocks together, guys.

That which does not kill you has made a tactical error. --Tayler
Bushflyr is offline   Reply With Quote
Old 07-03-2013, 10:07   #5
Detectorist
Senior Member
 
Detectorist's Avatar
 
Join Date: Jul 2008
Location: Robertsville, MO
Posts: 7,554
Tape? My Grandparents told me stories about that. lol
__________________
NASM-Certified Personal Trainer

The single biggest problem in communication is the illusion that it has taken place”. George Bernard Shaw

Last edited by Detectorist; 07-03-2013 at 14:12..
Detectorist is offline   Reply With Quote
Old 07-03-2013, 15:30   #6
harrygunner
Senior Member
 
Join Date: Sep 2010
Posts: 408
I read this paper a while back http://www.cs.cmu.edu/~bianca/fast07.pdf I just like to read stuff and I don't like to lose data. That paper had me thinking what to do given I have dozens of multi-terabyte drives with critical data on them.

Here's two articles written by people who appear to be unconcerned about failures while rebuilding a RAID5 array.

They has links to other articles and calculations.

http://www.pizzaandcode.com/posts/780

http://subnetmask255x4.wordpress.com...-impacts-raid/

Some posted comments about still being concerned.

Quote:
KD Mann: In addition, the RAID-5 rebuild process itself is hard on all the drives in the array, not just the “failing” drive, and UREs will increase with drive temperature across the entire array — further increasing the chances of a catastrophic failure.

Finally, with 2, 3 and soon 4TB drives, the amount of time requires to complete a RAID-5 rebuild can now stretch into DAYS! Desktop-class disks weren’t built to withstand that kind of abuse even for a few hours…

Given the ratio of URE probability to disk capacity, RAID-5 is dead, there’s no getting around it. Best to just get used to the idea and get on with life….

Disclaimer: Yes, I do work for a RAID manufacturer, and yes, I have (in the past) designed RAID hardware and software.

And about tape drives, the idea is to backup to devices that have different failure modes and lower failure rates. Backing data to devices that have similar failure rates as the original devices is delusional.

And equally important, faster recovery times. Backing up to the "cloud" sounds cool, but if you need to recover those terabytes, it's going to take a while. If you can perform the calculations to predict probabilities of failures of disk arrays, calculating the time to download your backup is easy.
__________________
People who've had to deal with their karma are more interesting to talk to.
harrygunner is offline   Reply With Quote
Old 07-04-2013, 15:40   #7
Linux3
Senior Member
 
Linux3's Avatar
 
Join Date: Dec 2008
Posts: 1,396
Quote:
Originally Posted by Detectorist View Post
Tape? My Grandparents told me stories about that. lol
Oh my child. Tape is still very much in use.

If you want to back up a 50Tbyte virtual drive it is the only way.
If you want to digitize, FX and store a movie, it's the way to fly.
It is the standard for long term storage.

http://www.fujitsu.com/global/servic...ts/tape/lt250/
__________________
It it's not on fire,
It's a software problem.

Linux3 is offline   Reply With Quote
Old 07-04-2013, 18:36   #8
Detectorist
Senior Member
 
Detectorist's Avatar
 
Join Date: Jul 2008
Location: Robertsville, MO
Posts: 7,554
That's awesome, Linux3.

The last time I dealt with tapes was way back in 1988ish. IBM reel to reel 3450? density. I was the SSO and the shipping company lost a shipment of them...
__________________
NASM-Certified Personal Trainer

The single biggest problem in communication is the illusion that it has taken place”. George Bernard Shaw
Detectorist is offline   Reply With Quote
Old 07-05-2013, 06:25   #9
dwhite53
Senior Member
 
dwhite53's Avatar
 
Join Date: Oct 2008
Location: Central North Carolina
Posts: 1,114
I went through this too building my wife's latest machine. Wanted something to make sure her data was secure.

Windows 7 has software mirroring but it has to be set up from the original install, we were moving her hard drive to the new machine so that was out.

I then did a search for "real time backup" and found a few options. I ended up with this:

http://www.codessentials.com/products/yadisbackup.html

You can set it to mirror whatever files from your main drive to a second, third or however many drives. It's not true mirroring but it will save your data. Any time you modify or create a file, Yadis! copies it to the backup drive. It has to be set to delete saved files when they're deleted from the main drive.

I installed this on my laptop too removing the cd drive and installing a second hard drive. Doesn't affect batter life noticeably.

All the Best,
D. White
__________________
Amendment 10.
The powers not delegated to the United States by the Constitution, nor prohibited by it to the states, are reserved to the states respectively, or to the people.

Last edited by dwhite53; 07-05-2013 at 06:25..
dwhite53 is offline   Reply With Quote
Old 07-05-2013, 18:06   #10
srhoades
Senior Member
 
srhoades's Avatar
 
Join Date: Jul 2000
Posts: 4,536
Quote:
Originally Posted by dwhite53 View Post
I went through this too building my wife's latest machine. Wanted something to make sure her data was secure.

Windows 7 has software mirroring but it has to be set up from the original install, we were moving her hard drive to the new machine so that was out.

I then did a search for "real time backup" and found a few options. I ended up with this:

http://www.codessentials.com/products/yadisbackup.html

You can set it to mirror whatever files from your main drive to a second, third or however many drives. It's not true mirroring but it will save your data. Any time you modify or create a file, Yadis! copies it to the backup drive. It has to be set to delete saved files when they're deleted from the main drive.

I installed this on my laptop too removing the cd drive and installing a second hard drive. Doesn't affect batter life noticeably.

All the Best,
D. White
Does it save versions? What happens if I save over a file and it copies that over across all the drives?
__________________
Let's see. You drive a Ford, you drink crap beer, you quote Hitler, and you haven't had sex in years. Nope, ain't taking your advice. - BALIFF
srhoades is offline   Reply With Quote
Old 07-06-2013, 12:30   #11
dwhite53
Senior Member
 
dwhite53's Avatar
 
Join Date: Oct 2008
Location: Central North Carolina
Posts: 1,114
Quote:
Originally Posted by srhoades View Post
Does it save versions? What happens if I save over a file and it copies that over across all the drives?
Yadis will do versioning though I don't use it and am thus not familiar with it.

Pretty easy to just download, install, and configure. Select a few files from your machine to save to an SD card or flash drive and see how it works.

As fast as computers are now the overhead of the machine making copies of files to another drive is minimal. For most people RAID is just too much.

I've read that if you do hardware RAID and your controller/motherboard die you can have issues accessing your data unless you use another board with the exact same hardware. RAID only has to do what is described in it's ANSI/ISO/IEEE spec's. How the hardware manages it is up to the hardware manufacturer.

Using Windows built in RAID eliminates this problem however it only does the same thing as Yadis! does with a lot more overhead.

All the Best,
D. White
__________________
Amendment 10.
The powers not delegated to the United States by the Constitution, nor prohibited by it to the states, are reserved to the states respectively, or to the people.

Last edited by dwhite53; 07-06-2013 at 12:30..
dwhite53 is offline   Reply With Quote
Old 07-06-2013, 18:25   #12
CitizenOfDreams
Senior Member
 
CitizenOfDreams's Avatar
 
Join Date: Feb 2009
Location: Orlando, FL
Posts: 8,117
Quote:
Originally Posted by dwhite53 View Post
Using Windows built in RAID eliminates this problem however it only does the same thing as Yadis! does with a lot more overhead.
I have used Windows software RAID (stripe) with good success. The CPU usage was somewhat noticeable, but I didn't have much of a CPU back then.

Unfortunately, Windows does not do RAID10 in software.
CitizenOfDreams is offline   Reply With Quote
Old 07-08-2013, 14:49   #13
Drjones
Senior Member
 
Join Date: Sep 2002
Location: CA, just outside the United States
Posts: 17,810
Blog Entries: 1
Send a message via AIM to Drjones


Quote:
Originally Posted by CitizenOfDreams View Post
So, which option should I choose, and what exactly will I gain from a better RAID controller? I am planning to use the machine for video editing and light gaming (think Starcraft, not Halo 6).

Apart from a RAID 0 for a possible performance increase, I'm not sure what you'd gain from RAID.

I mean, there's the obvious redundancy advantage, but as some here have pointed out, there are real dangers to proprietary RAID controllers.

Dunno if you saw my other thread about the NAS I want to build, but for now, I've decided against RAID, mostly because I know that the non-enterprise-level RAID stuff is not that great.

Many/most video/graphics editing programs are designed to have a "scratch" drive anyway, right? This wouldn't apply to a RAID config, I wonder if you might not get *worse* performance, depending exactly what software you use & how you use it.

Dunno about your level of tolerance for messing with stuff like this, but if I were you, I'd forget the RAID, have multiple internal drives, and make positive that you have 2-3 different backups that run as often as you need them to.
__________________
The danger to America is not Barack Obama, but a citizenry capable of entrusting a man like him with the Presidency.

Last edited by Drjones; 07-08-2013 at 14:53..
Drjones is offline   Reply With Quote
Old 07-08-2013, 19:10   #14
CitizenOfDreams
Senior Member
 
CitizenOfDreams's Avatar
 
Join Date: Feb 2009
Location: Orlando, FL
Posts: 8,117
Quote:
Originally Posted by Drjones View Post
Apart from a RAID 0 for a possible performance increase, I'm not sure what you'd gain from RAID.
Mainly, not having to restore data from backup when one of the hard drives goes "click-click-click". Also, storing "semi-important" data that's not worth backing up (e.g. downloaded videos to watch and forget).

Quote:
Dunno if you saw my other thread about the NAS I want to build, but for now, I've decided against RAID, mostly because I know that the non-enterprise-level RAID stuff is not that great.
I think I do remember that thread. That's where you had to reformat your array after a firmware upgrade? My software RAID (Intel Rapid Storage) didn't seem to have those problems. It kept working after a BIOS upgrade, an HDD firmware upgrade, and a driver upgrade.

Quote:
Many/most video/graphics editing programs are designed to have a "scratch" drive anyway, right? This wouldn't apply to a RAID config, I wonder if you might not get *worse* performance, depending exactly what software you use & how you use it.
I don't see a problem there. Programs either use %TEMP% (which would be on a separate SSD along with the OS), or let you set the scratch disk manually.

Quote:
Dunno about your level of tolerance for messing with stuff like this, but if I were you, I'd forget the RAID, have multiple internal drives, and make positive that you have 2-3 different backups that run as often as you need them to.
I don't see how RAID10, even a fake RAID10, can be less reliable than a standalone drive.
CitizenOfDreams is offline   Reply With Quote
Old 07-08-2013, 21:21   #15
Drjones
Senior Member
 
Join Date: Sep 2002
Location: CA, just outside the United States
Posts: 17,810
Blog Entries: 1
Send a message via AIM to Drjones


Quote:
Originally Posted by CitizenOfDreams View Post
Mainly, not having to restore data from backup when one of the hard drives goes "click-click-click". Also, storing "semi-important" data that's not worth backing up (e.g. downloaded videos to watch and forget).
True, I just figure that ANYTHING worth keeping around at all, is worth backing up. Maybe not making multiple backups, and off-site backups, but backups nonetheless.



Quote:
I think I do remember that thread. That's where you had to reformat your array after a firmware upgrade? My software RAID (Intel Rapid Storage) didn't seem to have those problems. It kept working after a BIOS upgrade, an HDD firmware upgrade, and a driver upgrade.

Nope, this thread: http://glocktalk.com/forums/showthread.php?t=1492327



Quote:
I don't see how RAID10, even a fake RAID10, can be less reliable than a standalone drive.
Oh, I never said it would be less reliable. Just that it's a little more work, and again some of the comments I've read in various places on the internet make me a little nervous about consumer grade RAID.
__________________
The danger to America is not Barack Obama, but a citizenry capable of entrusting a man like him with the Presidency.
Drjones is offline   Reply With Quote
Old 07-08-2013, 21:58   #16
nostalgiaguy
Senior Member
 
nostalgiaguy's Avatar
 
Join Date: Apr 2012
Location: Lenoir NC
Posts: 168
In regards to the very first post.......

You lost me at ...." I know "fake" software RAIDs... "

After that I just mumbled to myself abut getting old and not understanding anything anymore.

Damn whippersnappers. Get off my lawn!! LOL
nostalgiaguy is offline   Reply With Quote
Old 07-08-2013, 22:20   #17
CitizenOfDreams
Senior Member
 
CitizenOfDreams's Avatar
 
Join Date: Feb 2009
Location: Orlando, FL
Posts: 8,117
Quote:
Originally Posted by nostalgiaguy View Post
In regards to the very first post.......

You lost me at ...." I know "fake" software RAIDs... "
You know what a hard drive is? No? Magnetic drum storage? Tape? Punch card? How about abacus?
CitizenOfDreams is offline   Reply With Quote
Old 07-09-2013, 04:20   #18
CitizenOfDreams
Senior Member
 
CitizenOfDreams's Avatar
 
Join Date: Feb 2009
Location: Orlando, FL
Posts: 8,117
Quote:
Originally Posted by Drjones View Post
True, I just figure that ANYTHING worth keeping around at all, is worth backing up. Maybe not making multiple backups, and off-site backups, but backups nonetheless.
In ideal world, yes, all data should be backed up. In the real world, however, backup storage and bandwidth are not unlimited. So you have to range your data from "irreplaceable, backup at all costs" to "somewhat hard to recreate, backup whenever the budget allows" to "commonly found, no big deal if it gets lost".

Scary stuff there. There is something fundamentally wrong about a RAID that does not report a dead array member.

Quote:
Oh, I never said it would be less reliable. Just that it's a little more work, and again some of the comments I've read in various places on the internet make me a little nervous about consumer grade RAID.
As long as you not replacing regular backups with RAID, I don't see a problem. As far as work, rebuilding a RAID is far less work than restoring a backup (and all your data stays accessible in the process).
CitizenOfDreams is offline   Reply With Quote
Old 07-09-2013, 07:35   #19
dwhite53
Senior Member
 
dwhite53's Avatar
 
Join Date: Oct 2008
Location: Central North Carolina
Posts: 1,114
My machine is configured with a single, two partition, hard drive. C:\ is for operating system and program installation. Nothing gets saved to the C:\ drive.

D:\ is for all personal data. Everything gets saved to the D:\ drive. I've got D:\ configured with Yadis! to back-up real time to a second hard drive. I periodically (like most, not often enough) do a full imaging of C:\.

Having an OS/program drive and a separate data drive has worked well for me for years.

All the Best,
D. White
__________________
Amendment 10.
The powers not delegated to the United States by the Constitution, nor prohibited by it to the states, are reserved to the states respectively, or to the people.

Last edited by dwhite53; 07-09-2013 at 07:38..
dwhite53 is offline   Reply With Quote
Old 07-09-2013, 09:29   #20
CitizenOfDreams
Senior Member
 
CitizenOfDreams's Avatar
 
Join Date: Feb 2009
Location: Orlando, FL
Posts: 8,117
Quote:
Originally Posted by dwhite53 View Post
Having an OS/program drive and a separate data drive has worked well for me for years.
Yep, that's exactly how I keep it. Partition C: is for the OS and temporary files only, so it can be formatted and reinstalled at any time without worrying about user data.
CitizenOfDreams is offline   Reply With Quote
Old 07-09-2013, 17:34   #21
srhoades
Senior Member
 
srhoades's Avatar
 
Join Date: Jul 2000
Posts: 4,536
Quote:
Originally Posted by dwhite53 View Post
My machine is configured with a single, two partition, hard drive. C:\ is for operating system and program installation. Nothing gets saved to the C:\ drive.

D:\ is for all personal data. Everything gets saved to the D:\ drive. I've got D:\ configured with Yadis! to back-up real time to a second hard drive. I periodically (like most, not often enough) do a full imaging of C:\.

Having an OS/program drive and a separate data drive has worked well for me for years.

All the Best,
D. White
Those are only separate partitions and hence separate logical, not physical drives. If the drive fails you still lose everything on C and D.
__________________
Let's see. You drive a Ford, you drink crap beer, you quote Hitler, and you haven't had sex in years. Nope, ain't taking your advice. - BALIFF
srhoades is offline   Reply With Quote
Old 07-09-2013, 18:36   #22
CitizenOfDreams
Senior Member
 
CitizenOfDreams's Avatar
 
Join Date: Feb 2009
Location: Orlando, FL
Posts: 8,117
Quote:
Originally Posted by srhoades View Post
Those are only separate partitions and hence separate logical, not physical drives. If the drive fails you still lose everything on C and D.
He probably realizes that. But keeping OS and data on separate partitions still has irrisputable(tm) advantages, even if the partitions are on the same physical drive.
CitizenOfDreams is offline   Reply With Quote
Old 07-10-2013, 08:12   #23
dwhite53
Senior Member
 
dwhite53's Avatar
 
Join Date: Oct 2008
Location: Central North Carolina
Posts: 1,114
Quote:
Originally Posted by srhoades View Post
Those are only separate partitions and hence separate logical, not physical drives. If the drive fails you still lose everything on C and D.
Yeah, I realize that but, in times of having to re-install the OS (and we know that NEVER happens with Windows) all my personal data is secure on its own partition plus it's all backed up to the second hard drive.

In all honesty Windows reliability is much better than it used to be. Years ago when I could run Linux for everything I got spoiled on the rock solid reliability of Linux.

All the Best,
D. White
__________________
Amendment 10.
The powers not delegated to the United States by the Constitution, nor prohibited by it to the states, are reserved to the states respectively, or to the people.

Last edited by dwhite53; 07-10-2013 at 08:14..
dwhite53 is offline   Reply With Quote
Old 07-22-2013, 19:33   #24
Dieter
Senior Member
 
Dieter's Avatar
 
Join Date: Jul 2006
Location: SF Bay Area
Posts: 147
Quote:
Originally Posted by CitizenOfDreams View Post
I know "fake" software RAIDs are laughed at in the IT community, but I have been happy with mine (4 drives in RAID10, Intel ICH10R). It had adequate speeds and no noticeable CPU load. The array properly rebuilt itself when I faked a drive failure.
Well, if you are happy with that, I think you've answered your own question!

Rather than spend money on a mobo that might cost more with a fancier, built in RAID controller, you could save that money and consider an SSD Blade PCIe card - more expensive than a normal SSD, but the performance is there for sure. But you don't need a big one - just something to house the OS and your applications.

You can use a separate "spinny" drive for your video editing, and then build a RAID array to use a dump site for your data, sync it up to make backups at various times when you're not using the machine, and such if you're really concerned about software RAID performance. (Though with modern processors, I don't think this is a big deal, especially not with a single array, or even two.)

Hardware solutions I think are the way to go with multiple large arrays but for the home user, or small business, software RAID gets the job done - slower, but cheaper, and in my opinion, more reliable.

Sure, people will argue with that, but as already mentioned - you can run into proprietary issues with hardware, and controllers can fail too, just like anything else.

I have a machine at my office that is supporting 4 x RAID 1 arrays (28TB/2 = 14TB mirrored), 2 internal sets (3TB@) and 2 external sets (4TB@), all SATA III scaled back to SATA I as that is the max the machine can support anyway, and it does just fine. ANY machine you get today will blow the doors off this 9-year old thing.
Dieter is offline   Reply With Quote
Old 07-23-2013, 12:51   #25
Paul53
Geezer Boomer
 
Paul53's Avatar
 
Join Date: Nov 2011
Location: Rosa's Cantina
Posts: 3,818
Anybody ever wonder why 535 Congressmen can't come to an agreement?
__________________
"A little learning is a dang'rous thing; Drink deep, or taste not the Pierian spring: There shallow draughts intoxicate the brain, And drinking largely sobers us again."

Paul53 is offline   Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump




All times are GMT -6. The time now is 04:20.



Homepage
FAQ
Forums
Calendar
Advertise
Gallery
GT Wiki
GT Blogs
Social Groups
Classifieds


Users Currently Online: 788
240 Members
548 Guests

Most users ever online: 2,244
Nov 11, 2013 at 11:42