IPB

Welcome Guest ( Log In | Register )

2 Pages V   1 2 >  
Reply to this topicStart new topic
Best Windows audio file backup/sync solution in 2012 ?
wcs13
post May 23 2012, 14:56
Post #1





Group: Members
Posts: 60
Joined: 10-October 10
Member No.: 84514



Hi all,

I am looking for a Windows (Seven) backup/sync solution adapted to all my audio files. Let me explain.

My music library consists in over 40,000 music files (800 GB), and it's saved on an external HD at regular intervals.
The problem is that I regularly change the tags of many files at once using foobar.
With a classic backup solution, every time a tag has been changed, the entire audio file must be re-saved ... even though all the "audio" part has remained unchanged and it would have been enough to save the "tag" part of the file!
Thus, I'm looking for a solution capable of backing up the whole file when needed, or only the "tag" part when only tags habe been modified.

On Linux there are solutions like rsync for this kind of advanced features.
On Seven I do not know what can exist. So I hope you can help me find the right solution.
FYI, currently I use SyncBackPro, but it does not seem to work the way I want (or maybe there is a setting that I have not found??).

BTW, I'm not really a geek, so I'm looking for a really EASY TO USE solution, not some command line thing with lots of parameters.
The idea would be to have a windows GUI that would allow me to synchronize ("mirror" mode) a source folder (e.g. C:\AUDIO) with a destination folder (e.g. D:\AUDIO), and that will NOT copy 800 GB of data every time I change ONE single tag on all my audio files. I hope you get the idea.
Last but not least, the solution would have to handle file names and tags involving sometimes accented characters (,, , , etc.).

Thanks in advance. smile.gif
Go to the top of the page
+Quote Post
washu
post May 23 2012, 15:16
Post #2





Group: Members
Posts: 135
Joined: 16-February 03
From: Ottawa
Member No.: 5032



What you are looking for exists, but is useless in your situation. You are backing up to local storage, so tools like rsync would actually be slower if used in the way you are thinking.

Tools like rsync are useful when you have two computers and a slow network between the source and destination. Doing differential copies on local storage is a waste of time and involves more disk access.

If you can use the command line then robocopy is simple and included with Windows 7. Open a command prompt and run the following:

robocopy /mir C:\audio D:\audio

It will make D:\audio a perfect copy of C:\audio (a mirror). It will only copy files that are different. Be aware that it will also delete files from the backup if they are missing from the source.

It will still copy every file if you change a tag in each one, but again that is faster then figuring out the differentials on local storage.

This post has been edited by washu: May 23 2012, 15:18
Go to the top of the page
+Quote Post
eahm
post May 23 2012, 15:18
Post #3





Group: Members
Posts: 1080
Joined: 11-February 12
Member No.: 97076



I use your very same setup as well as SyncBackPro. Sometimes I change "things" and I have to recopy 300GB at once.

When you change the tag the file changes and of course you have to recopy it, how does rsync work? How can it copy only the tag without copying the file if the tag is inside the file?

This post has been edited by eahm: May 23 2012, 15:19


--------------------
/lwAsIimz
Go to the top of the page
+Quote Post
wcs13
post May 23 2012, 15:25
Post #4





Group: Members
Posts: 60
Joined: 10-October 10
Member No.: 84514



washu, I see your point, but my concern is not about the time it takes, but about the writing operation.
I want a backup solution that will write as less data as possible, even if it takes more time.
Like I said, I need a solution that will NOT write 800 GB of data every time I change ONE single tag on all my audio files. That is my choice.

Besides, I'm not really on local storage. All my audio files are on a NAS, and the NAS drives are mounted as network drives under Seven. My example with C: and D: was just an example.
So the real backup/sync would be as follows : X:\AUDIO (network drive) ----> F:\AUDIO (external hard drive).

SyncBackPro does not copy everything even if I change one file. It does copy only the changed files, of course. But that is not enough for me.
I don't need a solution that will only copy files that are different. I need a solution that will copy only tags when only tags are changed, and that will copy the whole file only when there's no other choice.

eahm, I suggest you find some doc about rsync (EDIT : here for instance : http://rsync.samba.org/tech_report/ ). I can't explain you precisely how it works, but I can promise you that it works exactly that way. wink.gif

This post has been edited by wcs13: May 23 2012, 15:48
Go to the top of the page
+Quote Post
washu
post May 23 2012, 15:56
Post #5





Group: Members
Posts: 135
Joined: 16-February 03
From: Ottawa
Member No.: 5032



Why would you care about the writing? It's not going to hurt anything. In fact, it is safer as you eliminate a point of failure (the diffing program). Not that rsync has had any such bugs that I'm aware of, but keeping it simple is always better.

If you are worried about drive life then what you are proposing is worse, as the drives will have to do a full read and then at least a partial write. A full write is less work.

Having your data on a NAS doesn't change anything. It's still going to be way faster and less work to simply copy the files instead of diffing them, unless your network is horribly slow.


If you really must go down this route then you can get versions of rsync for Windows if you look around. I'm going to strongly advise that you do not go this route. Besides it being a worse solution, you do not appear to have the technical level required to use it properly (no insult intended). You said yourself "I'm not really a geek", and rsync on windows has some technical issues due to it being built for UNIX systems.
Go to the top of the page
+Quote Post
Remedial Sound
post May 23 2012, 16:14
Post #6





Group: Members
Posts: 508
Joined: 5-January 06
From: Dublin
Member No.: 26898



I periodically back up my music (and other data) to an external HD and have found this tool to be well suited for the purpose:

http://synkron.sourceforge.net/

One word of caution, in setting up folder pairs (e.g. master/slave), make sure "Do not back up updated files" is checked, otherwise it does a redundant backup of files you've deleted/overwritten/etc. in a local synkron folder (unless of course you really want this behavior).

Go to the top of the page
+Quote Post
eahm
post May 23 2012, 16:35
Post #7





Group: Members
Posts: 1080
Joined: 11-February 12
Member No.: 97076



washu, you still don't get what he needs. The solution he needs is actually better than you think. Your opinion is too strong, some people just don't care about opinions.

Remedial Sound, I suggest you to test SyncBackPro. I use it for my customers as well, I ensure you it's the best Backup/Sync tool right after Symantec Backup Exec. Also check the prices.

This post has been edited by eahm: May 23 2012, 17:24


--------------------
/lwAsIimz
Go to the top of the page
+Quote Post
wcs13
post May 23 2012, 17:01
Post #8





Group: Members
Posts: 60
Joined: 10-October 10
Member No.: 84514



Hi guys,

Remedial Sound, I have checked Synkron's doc, and I don't see it using any delta-transmission algorithm like rsync. Is there something you can tell me about it ?

@all : I am currently testing Liciel Rsync ( http://code.google.com/p/licielrsync/ ) which seems to be pretty much exactly what I've been looking for. I suggest you guys check it out.

I've done a first test with a 6.95 GB folder. First backup of all the files, then I changed ONE tag on all the test files (with foobar), and I did a new backup. Here are the results of the second backup :
sent 12.20M bytes received 14.46M bytes 23.52K bytes/sec
total size is 6.95G speedup is 260.51

... so, considering the small amount of sent/received data, I think it really works ! What do you think ? wink.gif

This post has been edited by wcs13: May 23 2012, 17:02
Go to the top of the page
+Quote Post
smok3
post May 23 2012, 17:26
Post #9


A/V Moderator


Group: Moderator
Posts: 1729
Joined: 30-April 02
From: Slovenia
Member No.: 1922



so the disk-trashing/overall longevity-uptime question's could be:

Is it wiser to:
a. run a delta-aware sync to "dumb" share (that is: rsync not running in daemon mode)
b. run a "dumber" utility like pathsync on a "dumb" share http://www.cockos.com/pathsync/
c. replace the "dumb" share with some nix server and use rsync in daemon mode

?

p.s. if updates to backup procedure are expected, like more machines syncing with some share - then being aware of rsync limits, afaik it is not really bidirectional.

This post has been edited by smok3: May 23 2012, 17:31


--------------------
PANIC: CPU 1: Cache Error (unrecoverable - dcache data) Eframe = 0x90000000208cf3b8
NOTICE - cpu 0 didn't dump TLB, may be hung
Go to the top of the page
+Quote Post
eahm
post May 23 2012, 17:48
Post #10





Group: Members
Posts: 1080
Joined: 11-February 12
Member No.: 97076



Also check DeltaCopy and Syncrify. http://web.synametrics.com/SyncrifyVsDeltaCopy.htm

They seem to be more complete backup solutions, I think I am going to switch two/three companies to Syncrify and see how it runs.

This post has been edited by eahm: May 23 2012, 17:48


--------------------
/lwAsIimz
Go to the top of the page
+Quote Post
wcs13
post May 23 2012, 18:00
Post #11





Group: Members
Posts: 60
Joined: 10-October 10
Member No.: 84514



Smok3, I think that for the intended purposes, there's no need for any bi-directional software. IMHO it's best to have all audio in ONE location, and simply backup it to another location for security purposes (or even for nomad use). By doing so, you can modify or even delete all the copied data, but the "reference audio" remains stored in one location and will be copied again during the next backup. So my method is : "if I want to modify anything in my music library, I need to do it within the "reference audio", because any modification in the "copied audio" will disappear with the next backup".

As for your previous question, honestly I have no idea. I hardly understand the question anyway, sorry. emot-haw.gif

eahm, Synkrify is a paid solution, so I guess I'll stick with Liciel Rsync...

This post has been edited by wcs13: May 23 2012, 18:01
Go to the top of the page
+Quote Post
eahm
post May 23 2012, 18:09
Post #12





Group: Members
Posts: 1080
Joined: 11-February 12
Member No.: 97076



wcs13, it's 100% free for personal use.

...but you will need to install the Server, the Client and Java. I think it's more for businesses.

This post has been edited by eahm: May 23 2012, 18:13


--------------------
/lwAsIimz
Go to the top of the page
+Quote Post
washu
post May 23 2012, 18:22
Post #13





Group: Members
Posts: 135
Joined: 16-February 03
From: Ottawa
Member No.: 5032



QUOTE (wcs13 @ May 23 2012, 12:01) *
I've done a first test with a 6.95 GB folder. First backup of all the files, then I changed ONE tag on all the test files (with foobar), and I did a new backup. Here are the results of the second backup :
sent 12.20M bytes received 14.46M bytes 23.52K bytes/sec
total size is 6.95G speedup is 260.51

... so, considering the small amount of sent/received data, I think it really works ! What do you think ? wink.gif


The tool is not reporting the total I/O, just the parts it worked on.

Assuming I did the math right:

You "read" 14.46M bytes at 23.52K bytes/sec. That's about 630 sec.

6.95 GB transfered in 630 sec is about 11.8 MB/sec, around the realistic speed of a 100 mbit network.

So the tool is reading all your files over the network and then writing just the changes. You've saved no time and put more load on your drives, NAS and CPU. You have introduced an additional point of failure and complication.

Have you confirmed it works with your filenames? UNIX tools wrapped in cygwin don't always handle that properly.
Go to the top of the page
+Quote Post
JJZolx
post May 23 2012, 19:34
Post #14





Group: Members
Posts: 396
Joined: 26-November 04
Member No.: 18345



You're doing something drastically wrong if you're regularly changing tags on all the files in your audio collection. I can understand it to some extent as you figure out what's what, but at some point you should find that your collection becomes a very static entity.
Go to the top of the page
+Quote Post
wcs13
post May 23 2012, 20:48
Post #15





Group: Members
Posts: 60
Joined: 10-October 10
Member No.: 84514



washu, I hear you. However, I don't agree with you. I prefer reading 800 GB and writing a very small % of it, rather than writing 800 GB. I don't see why there should be more load on the drives. Writing is more stressful than just reading.
BTW, yes, I have confirmed it works with my filenames.

JJZolx, you are off topic but I will answer you. I've been improving my music library for the last few years, and yes, it has become a very static entity. So you are right, all my tags are very stable now.
The thing is that very recently, I have felt the need to add a new tag field to ALL my music library. FYI it's the Dynamic Range tag, automatically generated by the foobar component.
This sort of thing happens once in a while. You just need to add a new tag to all your music, even if for all the other tags it has become a very static entity, that's all.
Go to the top of the page
+Quote Post
JJZolx
post May 23 2012, 21:06
Post #16





Group: Members
Posts: 396
Joined: 26-November 04
Member No.: 18345



QUOTE (wcs13 @ May 23 2012, 13:48) *
So you are right, all my tags are very stable now.
The thing is that very recently, I have felt the need to add a new tag field to ALL my music library. FYI it's the Dynamic Range tag, automatically generated by the foobar component.
This sort of thing happens once in a while. You just need to add a new tag to all your music, even if for all the other tags it has become a very static entity, that's all.


And before that, when was the last time you retagged every file in your library? If you're doing it so often that you really need this type of utility, then I'd be surprised. Backing up the whole 800 GB library to disk on the rare occasion that you edit every single file should be trivial for any disk drive. I think your concerns for the health of your hard drives are wildly unfounded.
Go to the top of the page
+Quote Post
washu
post May 23 2012, 21:17
Post #17





Group: Members
Posts: 135
Joined: 16-February 03
From: Ottawa
Member No.: 5032



QUOTE (wcs13 @ May 23 2012, 15:48) *
washu, I hear you. However, I don't agree with you. I prefer reading 800 GB and writing a very small % of it, rather than writing 800 GB. I don't see why there should be more load on the drives. Writing is more stressful than just reading.


Again I'm not trying to be insulting, but your computer knowledge is obviously very limited. Writing is not more stressful than reading. Additional head movements on your drives is way more stressful then either. You are at the very minimum doubling the number of seeks your drive has to perform. More likely you are increasing them 50X or more as just writing large files would be mostly sequential. You are turning an easy sequential write job into a seek intensive read-write job. Also if your tag changes make the file sizes differ by more than a block then doing this your way will heavily fragment the data. This would cause excessive head movements every time you accessed your data, not just when you updated it.

I'm not saying you will kill your drives, but if anything will do it prematurely it is excessive head seeks. It will also kill performance, you don't notice because you are only on 100 mbit. Upgrade to gig-e in the future and it will hurt your potential transfer greatly.


Go to the top of the page
+Quote Post
Ouroboros
post May 23 2012, 21:27
Post #18





Group: Members
Posts: 291
Joined: 30-May 08
From: UK
Member No.: 53927



washu, you are wrong about rsync - that's not how it works. The numbers mean what they say:

- 6.95GB is the total size of the files synchronised
- 12.20MB is the amount sent, this is actual file update data
- 14.46MB is the amount received, all checksum data from the far end
- the rate (23.52K bytes/sec) is simply the amount actually transferred divided by the time taken for the total synchronisation, much of which is time spent creating checksums at each end of the link, and checking the checksums at the sending end.

I use rsync to synchronise around 6TB of data regularly, and I can assure you that it only sends the changes over the network, not the whole 6TB. How can I be so certain - well, I monitor the link utilisation, which has odd spikes of a few seconds at ~250Mbit/s when data is actually transferred, interspersed with long periods of almost no network utilisation but high levels of CPU utilisation at both ends of the link while checksums are being calculated. Also, the whole thing takes around 15-30 minutes on a typical day, which is nowhere near enough to transfer 6TB.
Go to the top of the page
+Quote Post
washu
post May 23 2012, 21:36
Post #19





Group: Members
Posts: 135
Joined: 16-February 03
From: Ottawa
Member No.: 5032



Ouroboros:

What you say is completely correct, IF and ONLY IF rsync is being used on both ends and it is talking over the rysnc protocol. In this case it is not. His NAS is not running rsync, only his Windows PC is. If only one end is using rsync then it has no choice but to read the data in its entirety in order to do the comparison. In this case it is reading it over SMB from the NAS.

Also, you obviously aren't modifying the whole 6 TB each time like wcs13 is. For rsync to even do the compare on 6 TB of changed files in 30 min you would need storage capable of delivering about 3.7 GB/sec. A high end SAN might be able to do that, otherwise it's just not happening.

This post has been edited by washu: May 23 2012, 21:37
Go to the top of the page
+Quote Post
Ouroboros
post May 23 2012, 22:38
Post #20





Group: Members
Posts: 291
Joined: 30-May 08
From: UK
Member No.: 53927



We don't know how his external HD is connected. It sounds as though it might even be a local disk.

Your disk I/O throughput numbers assume that you are running rsync in non-default mode. In default mode it checks the file size and date/time, and only checks the actual file if the size / date /time have changed. Consequently it reads nowhere near the full 6TB, it simply reads the inodes and directory files, around 1% of the total disk size. Consequently the 3.7 GB/s drops to 37 MB/s - easily achievable.

It's only if you use the -I option that you have to read the whole of the 6TB.
Go to the top of the page
+Quote Post
wcs13
post May 23 2012, 22:40
Post #21





Group: Members
Posts: 60
Joined: 10-October 10
Member No.: 84514



I understand both of you, thank you for your comments.

washu is right in saying that currently my NAS is not running rsync. But in fact... my NAS HAS rsync installed, it's simply not running right now, because I don't know how to install rsync (or a rsync-like) in my Windows part and make both rsync talk over the network. If someone knows how to do this (is there a tuto somewhere ?), I think we'd all be happy !

FYI my NAS is simply a home-made Debian server with no GUI, only a command-line interface. It was installed by a friend of mine. I'm not really in touch with him anymore, but I can still try to type some Unix commands and see what happens, if someone helps me of course...

This post has been edited by wcs13: May 23 2012, 22:43
Go to the top of the page
+Quote Post
washu
post May 23 2012, 22:52
Post #22





Group: Members
Posts: 135
Joined: 16-February 03
From: Ottawa
Member No.: 5032



Ouroboros:

We are actually on the same page here, we are agreeing but saying it differently. Yes, in most cases rsync will do exactly what you said and only have to read a tiny % of the files. That is why I said "you obviously aren't modifying the whole 6 TB each time". This isn't specific to rsync, any good file sync program will check sizes and date/time to avoid copying unmodified files.

However, in wcs13's case he is modifying every file each time. Even though a tag update is a tiny change, rsync still has to scan every file in it's entirety to find out what the change was. Unless rsync is being run on both the source and destination, rsync must read the whole file using whatever disk or network transport is being used. The differences in the end are small, which is why rsync reports such small numbers. It doesn't change the fact that the whole data set was transferred anyway.

Unless the rsync protocol is being used then it doesn't gain any time or bandwidth savings over any other sync tool.
Go to the top of the page
+Quote Post
washu
post May 23 2012, 23:04
Post #23





Group: Members
Posts: 135
Joined: 16-February 03
From: Ottawa
Member No.: 5032



wcs13:

If you have a real PC running debian then that does change things a bit. Most of the time when I hear "NAS" I assume a consumer grade limited device.

Assuming the PC isn't really slow, then it can probably do disk transfers far faster than your network works, so you could actually save some significant time. The bottleneck will likely shift to your USB hard drive.

It's been a while since I've setup rsync on debian, (I'm more into BSDs), but I'll see what I can find. A quick Google shows several tutorials so that should at least get you going. I don't mind trying to help out, if you run into some specific problems I'll try to help out.

This doesn't change my caution about drive wear. The I/O load will be the same or greater. If you are concerned about that then you should look at upgrading your network to gig-e for faster transfers and just use a normal sync program.
Go to the top of the page
+Quote Post
Ouroboros
post May 23 2012, 23:30
Post #24





Group: Members
Posts: 291
Joined: 30-May 08
From: UK
Member No.: 53927



@washu.

Got it! Yes, we are both saying the same thing - almost. smile.gif

Even if you only run rsync at one end, you still don't transfer the whole file to do the initial checking. You use whatever file access protocol you are running to check the date / time / size, and only if they don't match do you need to transfer the whole file for checking. However, it defeats the object of using rsync entirely - if you are transferring the whole file back to the source to check for changed blocks you might as well just transfer the whole file to the destination.

For clarity, you don't need to use the rsync protocol, but you do do need to use the rsync algorithm at both ends to get the benefit.

@wcs13

Set up an rsync daemon on the NAS box. I used this tutorial.

Then from the Windows machine you run something like:

rsync -avh /Localdirectory/ wcs13@NAS::NAS/Remotedirectory

This post has been edited by Ouroboros: May 23 2012, 23:40
Go to the top of the page
+Quote Post
wcs13
post May 24 2012, 03:42
Post #25





Group: Members
Posts: 60
Joined: 10-October 10
Member No.: 84514



QUOTE (washu @ May 24 2012, 00:04) *
If you have a real PC running debian then that does change things a bit. Most of the time when I hear "NAS" I assume a consumer grade limited device.

Yes, it's a real hand-mounted PC. It's not really slow, so it shouldn't be the bottleneck. The external hard drive shouldn't be either, since it's not USB but eSATA.

QUOTE (washu @ May 24 2012, 00:04) *
It's been a while since I've setup rsync on debian, (I'm more into BSDs), but I'll see what I can find. A quick Google shows several tutorials so that should at least get you going. I don't mind trying to help out, if you run into some specific problems I'll try to help out.

I think rsync is already set up on Debian, as a daemon. Could you provide me with some way of checking that out ?

QUOTE (washu @ May 24 2012, 00:04) *
This doesn't change my caution about drive wear. The I/O load will be the same or greater. If you are concerned about that then you should look at upgrading your network to gig-e for faster transfers and just use a normal sync program.

Network upgrade to Gig-E is planned but not immediately. BTW, if the network gets upgraded, then I think the Debian PC will become the new bottleneck (it's not really slow, but it's not lightning fast either laugh.gif ).

QUOTE (Ouroboros @ May 24 2012, 00:30) *
Set up an rsync daemon on the NAS box. I used this tutorial.
Then from the Windows machine you run something like:
rsync -avh /Localdirectory/ wcs13@NAS::NAS/Remotedirectory

Ouroboros, I have some questions for you :
- What is this command supposed to do exactly ? Thanks in advance.
- How could I possibly type this command without installint some rsync clone first ?
- If this command creates a "link" between the two intended folders, how can I make that link "permanent" ?
- If this link is created, can I use LicielRsync to backup the folders efficiently, or must I use something else ?
Sorry for all the dumb questions, I'm just trying to figure this out.

Just in case, I remind you that my sync must go in ony ONE direction : from the NAS (where all the "master audio files" are located) to the Windows PC (where only backups are located), not the opposite. Thanks.
Go to the top of the page
+Quote Post

2 Pages V   1 2 >
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 17th September 2014 - 05:22