IPB

Welcome Guest ( Log In | Register )

Help ripping ~30,000 CDs, Was “Help digitizing […]” ;)
trail
post Mar 29 2012, 19:08
Post #1





Group: Members
Posts: 9
Joined: 29-March 12
Member No.: 98189



Hey! I'm a newbie around these forums, but hopefully I'll be around quite a bit. I really like the community here, and hope I can contribute in the future. But enough introduction, here's the interesting stuff:

I work for a college radio station, and we've decided to undertake the rather ambitious project of digitizing the CDs we've acquired over the years. This is a pretty monumental undertaking, so I'm looking to make this as painless and quick as possible. We have a very rough approximation of about 30,000 CDs that we're looking to convert to digital files, and it's my job to work out many of the more technical aspects of the project.

The problem with being a college radio station is that we're on a pretty limited budget. We can't afford any sort of robot or anything like that to help the process along, nor can we afford any sort of service, so we're stuck doing it ourselves. Thankfully, we have a bunch of people willing to put the time and effort in. We also aren't terribly picky about getting every rip totally 100% perfect. But I've done a fair bit of research, and here's the kind of plan I had in mind:

Ideally, we have one pretty decent quad-core desktop that we're planning to outfit with four CD drives. We have software that allows us to rip multiple discs at once to V0 MP3s which are stored on a small RAID 1 array inside the computer. I've done some informal ripping tests, and have narrowed down the two pieces of software that seem to work best to fre:ac and dBpoweramp. I have also tried EAC and simply ripping with MediaMonkey, but freac and dBpoweramp seemed the most efficient and easy to use. Now, if I decide to use one of these pieces of software (if anyone has any suggestions, I'm 100% open to them!) how can I configure them to make them as painless as possible? Would using multiple drives be an option? I found very little information about software that provided ripping from multiple drives simultaneously, so I'm assuming this is not a common feature. If not, would using different computers be our best bet? If anyone has any other suggestions about ripping multiple discs at the same time or other ways to improve efficiency then that would probably make my life much easier.

thanks for your time!
Go to the top of the page
+Quote Post
 
Start new topic
Replies
Porcus
post Apr 2 2012, 14:57
Post #2





Group: Members
Posts: 1842
Joined: 30-November 06
Member No.: 38207



Since this is turning into a discussion of storage (on a budget), here's my uneducated two cents, subject to change upon anyone's better arguments:

- RAID is not backup. RAID is a way to reduce the number of times you need to resort to your backup. RAID does not protect against a thief, a lightening strike, or a 'holy s**t, what did I just do?'. RAID5/6/Z/2Z gives you a limited time to replace a broken drive, that's all. That's a big deal if you care about uptime, but on a budget, you don't. You would rather take the array offline until you are sure it is OK again.

- Striping -- i.e., spreading one file over multiple drives -- (basically all RAIDs except RAID1 ... and some nonstandard solutions) is a bit dangerous: even if you have a fault-tolerance of 1 faulty drive of, say, 4 then you need all the other 3 in order to read a single file. You also need the RAID setup. That is, you cannot take a single drive out of the array and get anything out of it -- and if you will take the 3 working drives out, then you need to mount them in a RAID array that can read it.

- There is a proprietary solution called UnRAID which eliminates the issues of striping: it simply dedicates a drive as parity, monitors the other drives, and whenever you write to a drive, it also updates the parity drive. That means, you can take drive #2 out of the array, mount it on a different computer, and every file on drive #2 is readable. If drive #2 AND the parity drive is ruined -- then retrieve merely drive #2 from your backup and clone it. There is a performance loss (writing takes twice the time), but if media files are basically write-once-read-many, that is no issue.


If you still want to do striping (like, RAID5):

- Enclosure RAID with port multiplier? The www is full of complaints about data loss, so I dare not even try. Yes port multipliers slow things down (everything has to go through the same channel), and that might be one reason for issues -- the OS might give up because it sees the drive as unresponsive.
(I'm using a port multiplier myself, but with 5 individual drives, no striping, and it is still a bit stressful: I thought it would be no issue as I only read the file I'm playing, no writing -- so I thought: but Windows writes to the NTFS journal all the time, or something like that.)

- Stay away from 'hardware RAID'. Mainly because you won't actually get hardware RAID on a budget, even though some weasels market it as such -- it is done in the drivers, and kind of gives you all issues of hardware RAID and all issues of software RAID. And if you actually go for a hardware RAID card, then you need two identicals, in order to have a backup if the card breaks, further violating the 'on a budget' purpose.

- Linux software RAID? Less issues. FreeNAS with ZFS' RAID-Z? Tried it once on a too old box, ZFS does require a bit of resources.


--------------------
One day in the Year of the Fox came a time remembered well
Go to the top of the page
+Quote Post
phofman
post Apr 2 2012, 15:57
Post #3





Group: Members
Posts: 300
Joined: 14-February 12
Member No.: 97162



We have been using linux raid in our company for a few years, from 2 to 8 drives configurations. This is just my experience:

Raid is definitely not a backup, I absolutely agree. In fact I was going to ask about the planned backup procedure. However, backing up 10TB of data on low budget is not a trivial task.

I assume the server should run 24/7 since it is the source of music for the radio station. That is why I suggested redundant raid and server hardware (yet inexpensive one).

Linux raid has no problem being used and running synchronization at the same time. Synchronizing 12TB software raid will take many hours, easily a day or two on a mildly loaded server. I agree that raid5 with only one drive of redundancy is not very safe. Six 3TB drives would allow the much safer raid6 of 12TB.

As for the backup, either the rather expensive tape, or IMO a much more flexible solution is a simple desktop PC with large case and motherboard with 6 sata ports, 5 3TB drives in RAID5, one small drive for system with linux (to make life easier) and run rsync every night/week over gigabit ethernet. I do not assume many changes on the main server data array so the synchronization would take just a few seconds. Preferrably the machine should be located in a different building. This solution would have the advantage of being able to take over the file-serving role of the main server quickly in case of hardware failure. It can be booted and shut-down automatically, by bios/halt command, to minimize electricity costs and hard drives wear.

This post has been edited by phofman: Apr 2 2012, 16:00
Go to the top of the page
+Quote Post
Porcus
post Apr 3 2012, 16:29
Post #4





Group: Members
Posts: 1842
Joined: 30-November 06
Member No.: 38207



QUOTE (phofman @ Apr 2 2012, 16:57) *
the rather expensive tape


The tape drives are expensive, but tapes are only slightly more than hard drives, so ... are there rental services? Or backup services where you would show up physically, have your drives copied, and if you are so unlucky you need restoration, show up with tapes and new drives? (I suspect that if such exist, they are at industry-grade pricing.)


--------------------
One day in the Year of the Fox came a time remembered well
Go to the top of the page
+Quote Post

Posts in this topic
- trail   Help ripping ~30,000 CDs   Mar 29 2012, 19:08
- - frozenspeed   I would use cueripper & foobar2000 but that...   Mar 29 2012, 19:11
|- - trail   QUOTE (frozenspeed @ Mar 29 2012, 13:11) ...   Mar 29 2012, 19:13
- - Dario   What about the dBpoweramp batch ripper? Is there a...   Mar 29 2012, 19:23
|- - garym   QUOTE (Dario @ Mar 29 2012, 13:23) What a...   Mar 29 2012, 21:50
|- - Roseval   You might try running multiple instances of dbPowe...   Mar 29 2012, 22:06
||- - trail   QUOTE (Roseval @ Mar 29 2012, 17:06) You ...   Mar 29 2012, 22:24
|- - trail   QUOTE (garym @ Mar 29 2012, 16:50) QUOTE ...   Mar 29 2012, 22:36
- - DVDdoug   QUOTE Ideally, we have one pretty decent quad-core...   Mar 29 2012, 23:33
|- - trail   QUOTE (DVDdoug @ Mar 29 2012, 18:33) QUOT...   Mar 30 2012, 01:52
- - Destroid   OP: I would concur with others to consider lossles...   Mar 30 2012, 00:08
- - dumdidum   i second the suggestion of ripping to a lossless f...   Mar 30 2012, 08:56
- - Porcus   I ripped about 7000 CDs to FLAC using dBpoweramp, ...   Mar 30 2012, 10:07
- - LosMintos   Sorry, I didn't read all posts carefully, neve...   Mar 30 2012, 14:28
|- - Porcus   QUOTE (LosMintos @ Mar 30 2012, 15:28) Yo...   Mar 30 2012, 16:38
- - pdq   On the cost of storing lossless files - consider t...   Mar 30 2012, 16:40
- - spoon   >My experience with dBpoweramp is that it might...   Mar 30 2012, 16:41
- - trail   First off, I want to say thanks so much to everyon...   Mar 30 2012, 17:44
|- - Porcus   QUOTE (trail @ Mar 30 2012, 18:44) If at ...   Mar 30 2012, 20:23
|- - trail   Bad news, our budget means we probably won't b...   Apr 2 2012, 01:42
|- - phofman   QUOTE (trail @ Apr 2 2012, 01:42) Bad new...   Apr 2 2012, 11:05
|- - phofman   My experience with those SiliconImage SATA control...   Apr 2 2012, 11:22
|- - Destroid   QUOTE (trail @ Apr 2 2012, 00:42) Bad new...   Apr 4 2012, 11:54
|- - trail   QUOTE (Destroid @ Apr 4 2012, 06:54) QUOT...   Apr 5 2012, 20:41
|- - shadowking   QUOTE (Destroid @ Apr 4 2012, 21:54) QUOT...   Apr 6 2012, 15:50
- - .hx   Little side note - CD ripping is not digitizing.   Apr 2 2012, 03:06
- - spoon   Word of advice, if you want a stress free life, st...   Apr 2 2012, 09:49
- - Porcus   Since this is turning into a discussion of storage...   Apr 2 2012, 14:57
|- - rick.hughes   QUOTE (Porcus @ Apr 2 2012, 09:57) - RAID...   Apr 2 2012, 15:40
|- - phofman   We have been using linux raid in our company for a...   Apr 2 2012, 15:57
|- - Porcus   QUOTE (phofman @ Apr 2 2012, 16:57) the r...   Apr 3 2012, 16:29
- - LosMintos   QUOTE (spoon @ Mar 30 2012, 17:41) The la...   Apr 2 2012, 15:48
- - krabapple   Nothing to add except again: go lossless if you c...   Apr 2 2012, 16:30
- - pdq   For backup I would suggest AudioSAFE. If you never...   Apr 2 2012, 16:39
|- - garym   QUOTE (pdq @ Apr 2 2012, 10:39) For backu...   Apr 2 2012, 22:25
|- - Jan S.   QUOTE (garym @ Apr 2 2012, 22:25) QUOTE (...   Apr 3 2012, 10:42
|- - Nessuno   QUOTE (Jan S. @ Apr 3 2012, 11:42) If the...   Apr 3 2012, 17:52
- - rick.hughes   The original CDs could also be considered the back...   Apr 2 2012, 19:12
|- - phofman   QUOTE (rick.hughes @ Apr 2 2012, 19:12) T...   Apr 2 2012, 19:25
- - spoon   Your deletions are kept on audiosafe (the last cha...   Apr 3 2012, 19:33
- - mixminus1   @Destroid: Yes, the OP's budget is limited, a...   Apr 5 2012, 21:06
|- - trail   QUOTE (mixminus1 @ Apr 5 2012, 16:06) @De...   Apr 9 2012, 04:20
- - JJZolx   If the budget is so limited that the station can...   Apr 6 2012, 01:43
- - Porcus   OK, so OP cannot afford drives enough for lossless...   Apr 7 2012, 02:58
- - gorob   I second dbpoweramp as well, it works great for ri...   Mar 11 2014, 03:16
- - JJZolx   I wonder how this project turned out. Or if it eve...   Mar 11 2014, 04:01
- - eahm   30,000 discs...I can't even imagine that many,...   Mar 11 2014, 05:43
|- - Porcus   QUOTE (eahm @ Mar 11 2014, 05:43) It must...   Mar 11 2014, 08:44
- - kennedyb4   Hi. I would like to add my advice to those suggest...   Mar 11 2014, 13:10
- - yourlord   This thread is 2 years old and the original poster...   Mar 11 2014, 18:23
- - eahm   QUOTE (Porcus @ Mar 11 2014, 00:44) Yep -...   Mar 11 2014, 23:46


Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 29th August 2014 - 04:32