IPB

Welcome Guest ( Log In | Register )

2 Pages V   1 2 >  
Reply to this topicStart new topic
Transferring Flac Collection over 802.11G
windmiller
post Jun 8 2005, 01:49
Post #1





Group: Members
Posts: 205
Joined: 6-March 04
From: Chapel Hill, NC
Member No.: 12500



Ive got a win2K3 server setup and am going to transfer around 200+GB of Flac & Ogg files to it from my HTPC in the living room. The HTPC is currently connected via 802.11g (125). I get really good transfer speeds but I am worried about corruption. Question is should I worry about corruption transferring this way? I transfer 1-2GB movies all the time. But this is my sacred Flac collection and the copies being trasnferred are backups.


I tried using and external hard drive but for some reason when I plug it in to the Win2K3 server the server freezes up.

* I am going to install a gigabit card on the HTPC in a week or two, should I just wait for that?

This post has been edited by windmiller: Jun 8 2005, 01:50


--------------------
www.losslessaudioblog.com
Go to the top of the page
+Quote Post
guest0101
post Jun 8 2005, 02:09
Post #2





Group: Banned
Posts: 446
Joined: 15-July 03
Member No.: 7789



QUOTE (windmiller @ Jun 7 2005, 05:49 PM)
I get really good transfer speeds but I am worried about corruption. Question is should I worry about corruption transferring this way? I transfer 1-2GB movies all the time. But this is my sacred Flac collection and the copies being trasnferred are backups.
*

Use something like the GUI Windows FlacTester application (http://www.vuplayer.com/other.htm) to verify your FLAC files once they are transferred for MD5 checksum errors. That should give you more peace of mind.

This post has been edited by guest0101: Jun 8 2005, 02:09
Go to the top of the page
+Quote Post
ectotropic
post Jun 8 2005, 12:09
Post #3





Group: Developer
Posts: 137
Joined: 25-September 04
From: Scotland
Member No.: 17297



I wouldn't worry too much about corruption at all – no more so than for a more traditional wired network. In fact I wouldn’t be much more worried about transferring your data via 802.11g than I would be transferring them between 2 hard drives on the same machine (actually if data corruption did occur transferring between 2 hard drives would be a worse case than transferring over any modern network).

Network standards are designed to be error tolerant (that is have the ability to detect and correct errors) wireless networks doubly so - they have to be, given the data transferred could be anything (e.g. medical data) it has to reach the destination entirely intact, bit identical to source. Hence there are several layers of protection in place between your data and the actual transfer. Worst case there is an error that is detected but can’t be corrected, so the receiver will inform the sender and the sender will resend the data.

But if you want that extra piece of mind (understandable tongue.gif ) you could test the resultant files as suggested by guest0101 and/or even go as far as to wrap the flac files up in a container supporting checksums (e.g. rar), which would tell you if the file had received errors in transit. You also may want to check up on the wireless network hardware involved – depending on your hardware you may be able to select a more secure method of transfer rather than a faster method of transfer (again, not that it’s likely to make much difference – especially not if signal strength is good).

This post has been edited by ectotropic: Jun 8 2005, 13:23
Go to the top of the page
+Quote Post
HbG
post Jun 8 2005, 12:47
Post #4





Group: Members
Posts: 289
Joined: 12-May 03
From: The Hague
Member No.: 6555



Yeah, wireless has some good ways of looking after data integrity, it's at least as good as wired. That doesn't mean errors cannot slip through, there is a specific error rate, but i'm not sure what it is, a quick google didn't help much. It's likely that you need to push many terabytes before you see an error coming through undetected though. As Guest0101 said, check your files before you delete the originals.


--------------------
Veni Vidi Vorbis.
Go to the top of the page
+Quote Post
user
post Jun 8 2005, 13:25
Post #5





Group: Members
Posts: 873
Joined: 12-October 01
From: the great wide open
Member No.: 277



I have had a corrupted file once whilst copying/moving from 1 HD to another HD via wireless lan, iirc the connection broke down by some reason.
So, I recommend to copy instead of moving.
If during moving the file gets corrupted, it is lost with high probability.
If you copy the file, you have a backup.
After successful copying, you can erase the original, if u wanna.

This post has been edited by user: Jun 8 2005, 13:27


--------------------
www.High-Quality.ch.vu -- High Quality Audio Archiving Tutorials
Go to the top of the page
+Quote Post
Busemann
post Jun 8 2005, 13:40
Post #6





Group: Members
Posts: 730
Joined: 5-January 04
Member No.: 10970



QUOTE (user @ Jun 8 2005, 04:25 AM)
I have had a corrupted file once whilst copying/moving from 1 HD to another HD via wireless lan, iirc the connection broke down by some reason.
So, I recommend to copy instead of moving.
If during moving the file gets corrupted, it is lost with high probability.
If you copy the file, you have a backup.
After successful copying, you can erase the original, if u wanna.
*


The files are copying if you transfer over a wireless network as well.

There's no risk involved at all.
Go to the top of the page
+Quote Post
gharris999
post Jun 8 2005, 18:24
Post #7





Group: Members
Posts: 99
Joined: 13-January 04
Member No.: 11204



QUOTE (windmiller @ Jun 7 2005, 04:49 PM)
I am worried about corruption. Question is should I worry about corruption transferring this way?
*


Try using robocopy.exe (stands for robust copy -- part of the winserver 2003 rtk, I think...go to the microsoft site and search for "robocopy.exe.") It's what I use for slinging flac files around between machines.

Robocopy is a command line tool that has better verification and error recovery options than xcopy (or the explorer shell, for that matter.) It also works well over slow or intermittent connections.
Go to the top of the page
+Quote Post
windmiller
post Jun 8 2005, 20:26
Post #8





Group: Members
Posts: 205
Joined: 6-March 04
From: Chapel Hill, NC
Member No.: 12500



Thanks for much for all the replies! I have to say I have been pretty impressed with the transfer speeds over 802.11g so that is good news.

I am going to check out RoboCopy for sure as I transfer large files all the time.
Regarding "wrap the flac files up in a container supporting checksums". How would I go about doing this? I know how to pack a .rar with flac files, would I just create a checksum and include it in the folder?

Thanks again!


--------------------
www.losslessaudioblog.com
Go to the top of the page
+Quote Post
johny5
post Jun 8 2005, 20:34
Post #9





Group: Members
Posts: 110
Joined: 27-May 04
Member No.: 14369



QUOTE (windmiller @ Jun 8 2005, 11:26 AM)
Thanks for much for all the replies! I have to say I have been pretty impressed with the transfer speeds over 802.11g so that is good news.

I am going to check out RoboCopy for sure as I transfer large files all the time.
Regarding "wrap the flac files up in a container supporting checksums". How would I go about doing this? I know how to pack a .rar with flac files, would I just create a checksum and include it in the folder?

Thanks again!
*


Rar has a data-integrity test itself. If you also include a recovery record you can repair the file if an error does occure.
Go to the top of the page
+Quote Post
Jebus
post Jun 8 2005, 22:12
Post #10





Group: Developer
Posts: 1294
Joined: 17-March 03
From: Calgary, AB
Member No.: 5541



Its not something to worry about... everything gets checked on transmission automatically. Stuff like accidental data corruption when a file is being read or written to during normal day-to-day operations is much more common because there is no correction then.
Go to the top of the page
+Quote Post
Martin H
post Jun 9 2005, 00:20
Post #11





Group: Members
Posts: 857
Joined: 5-March 05
From: Denmark
Member No.: 20365



QUOTE (ectotropic @ Jun 8 2005, 01:09 PM)
...and/or even go as far as to wrap the flac files up in a container supporting checksums (e.g. rar), which would tell you if the file had received errors in transit.
*

The FLAC format itself, stores a MD5 signature of the original unencoded data in the file header during encoding. Later, when the file is decoded or FLACīs Test feature is used, the MD5 signature is compared with the file, to check for corruption... -Martin.
Go to the top of the page
+Quote Post
windmiller
post Jun 9 2005, 01:09
Post #12





Group: Members
Posts: 205
Joined: 6-March 04
From: Chapel Hill, NC
Member No.: 12500



Gotcha so no "real" need. Much thanks! Transferring as we speak!


--------------------
www.losslessaudioblog.com
Go to the top of the page
+Quote Post
user
post Jun 9 2005, 11:16
Post #13





Group: Members
Posts: 873
Joined: 12-October 01
From: the great wide open
Member No.: 277



QUOTE (Busemann @ Jun 8 2005, 01:40 PM)
The files are copying if you transfer over a wireless network as well.

There's no risk involved at all.



Theory & Practice/experience...
As I wrote above, make sure, you copy files (independent if via wires or wireless), not move them !


--------------------
www.High-Quality.ch.vu -- High Quality Audio Archiving Tutorials
Go to the top of the page
+Quote Post
Defsac
post Jun 9 2005, 11:57
Post #14





Group: Members
Posts: 347
Joined: 17-May 05
Member No.: 22107



If you don't want to wrap the files Microsoft has a command line tool which can output MD5 or SHA1 hashes for files of any type.

QUOTE (Jebus @ Jun 9 2005, 07:12 AM)
Its not something to worry about... everything gets checked on transmission automatically. Stuff like accidental data corruption when a file is being read or written to during normal day-to-day operations is much more common because there is no correction then.
*
Ethernet uses CRC for error checking which isn't as robust as SHA1 or MD5 and errors can slip through. I've had files corrupted during transfer accross wired and wireless connections before.

This post has been edited by Defsac: Jun 9 2005, 12:08
Go to the top of the page
+Quote Post
cabbagerat
post Jun 9 2005, 12:53
Post #15





Group: Members
Posts: 1018
Joined: 27-September 03
From: Cape Town
Member No.: 9042



I took a look at some of my resources and it seems that the 802.11 specs would dictate an uncorrected bit error probability of 2.5e-22. The error rate of the physical channel is much higher (around 1e-6) but packet CRCs and other error correction mechanisms catch the vast majority of these errors.

Now, you have around 1.6e12 bits of data to transfer over the wire. We can assume that the probability of an uncorrected error in any bit is independent of errors in other bits (this is not quite true, but is good enough for now). So - the probability that at least one bit in the set will be incorrect is 1-(1-2.5e-22)^(1.6e12). This is a pretty small number.

However, running something along the lines of (I'm sure there are Windows equivalents of these commands, otherwise you could try Cygwin) "find -name "*" -exec md5sum '{};'" on the directories on both ends might be worth it. A diff tool can tell you if any of the lines differ.


--------------------
Simulate your radar: http://www.brooker.co.za/fers/
Go to the top of the page
+Quote Post
windmiller
post Jun 9 2005, 23:54
Post #16





Group: Members
Posts: 205
Joined: 6-March 04
From: Chapel Hill, NC
Member No.: 12500



QUOTE
However, running something along the lines of (I'm sure there are Windows equivalents of these commands, otherwise you could try Cygwin) "find -name "*" -exec md5sum '{};'" on the directories on both ends might be worth it. A diff tool can tell you if any of the lines differ.


I am going to do some comparisions between the two sets of flac files. The information I have gotten from this thread has eased my mind a bit because over the last few months I have done alot of transferring of files. So far I have not had any problems.


--------------------
www.losslessaudioblog.com
Go to the top of the page
+Quote Post
Martin H
post Jun 10 2005, 02:34
Post #17





Group: Members
Posts: 857
Joined: 5-March 05
From: Denmark
Member No.: 20365



QUOTE (windmiller @ Jun 10 2005, 12:54 AM)
I am going to do some comparisions between the two sets of flac files...
*

There is no need to compare the two sets of files... Just copy the files, and when done, you open FLAC frontend, and add all the directories with the FLACīs that you have copied, and then you just click "Test". Now all the FLACīs will be decoded and an MD5 checsum will be calculated for each decoded FLAC, and that MD5 checksum will be compared against an internal MD5 checksum of the original unencoded data stored inside the FLACīs, so this test is all thatīs needed to check for corruption... -Martin.
Go to the top of the page
+Quote Post
windmiller
post Jun 10 2005, 15:33
Post #18





Group: Members
Posts: 205
Joined: 6-March 04
From: Chapel Hill, NC
Member No.: 12500



QUOTE (Martin H @ Jun 9 2005, 05:34 PM)
QUOTE (windmiller @ Jun 10 2005, 12:54 AM)
I am going to do some comparisions between the two sets of flac files...
*

There is no need to compare the two sets of files... Just copy the files, and when done, you open FLAC frontend, and add all the directories with the FLACīs that you have copied, and then you just click "Test". Now all the FLACīs will be decoded and an MD5 checsum will be calculated for each decoded FLAC, and that MD5 checksum will be compared against an internal MD5 checksum of the original unencoded data stored inside the FLACīs, so this test is all thatīs needed to check for corruption... -Martin.
*




Thanks for the clarification.
It seems like with the Flac Front End I can only do one folder at a time. Am I missing something?

This post has been edited by windmiller: Jun 10 2005, 20:36


--------------------
www.losslessaudioblog.com
Go to the top of the page
+Quote Post
Martin H
post Jun 11 2005, 04:00
Post #19





Group: Members
Posts: 857
Joined: 5-March 05
From: Denmark
Member No.: 20365



QUOTE (windmiller @ Jun 10 2005, 04:33 PM)
Thanks for the clarification.
It seems like with the Flac Front End I can only do one folder at a time. Am I missing something?
*

You can drag and drop multiple folders into FLAC frontend... -Martin.
Go to the top of the page
+Quote Post
atici
post Jun 11 2005, 05:55
Post #20





Group: Members (Donating)
Posts: 1180
Joined: 21-February 02
From: Chicago
Member No.: 1367



I would suggest using a file comparison program like Beyond Compare for this purpose. After the transfer you can compare the MD5 checksums automatically. The two group of files do not have to be simultaneously accessible to the same computer. You can take a snapshot (including MD5 checksums) of a group of folders -> transfer the snapshot file -> compare the snapshot to another group of folders on another computer.

This post has been edited by atici: Jun 11 2005, 05:57


--------------------
The object of mankind lies in its highest individuals.
One must have chaos in oneself to be able to give birth to a dancing star.
Go to the top of the page
+Quote Post
windmiller
post Jun 11 2005, 16:00
Post #21





Group: Members
Posts: 205
Joined: 6-March 04
From: Chapel Hill, NC
Member No.: 12500



Thanks for the link, I will check it out!

What do you guys prefer when backing up your flac files? A .bak file or duplicate of the raw files themselves?

What I want to do is set it up so that whenever I add new flac files to my HTPC it is detected and syncs it with the backups on the server. I was going to use either Windows built-in backup program but I was affraid of using a .bak file instead of the actual files. I was also looking at a program called SURESYNC.

http://www.spursuits.com/suresync/suresync.htm

What format do you guys store your backups in?

This post has been edited by windmiller: Jun 11 2005, 16:02


--------------------
www.losslessaudioblog.com
Go to the top of the page
+Quote Post
atici
post Jun 11 2005, 18:20
Post #22





Group: Members (Donating)
Posts: 1180
Joined: 21-February 02
From: Chicago
Member No.: 1367



The software I recommended earlier, Beyond Compare, does synchronization too. I think it's best to have an offline drive which you sync with the online drive periodically. Beyond Compare would do this very well. If you choose to have two online drives I think the failure probability (of one of them failing) is higher. The only advantage of that solution is clearly that system would be more up-to-date in case of a failure. But also remember that sometimes you might perform operations on your hard drive by mistake (or virus can infect, etc), which you don't want replicated on the backup drive. This would be problematic for always online backup solution.

This post has been edited by atici: Jun 13 2005, 23:55


--------------------
The object of mankind lies in its highest individuals.
One must have chaos in oneself to be able to give birth to a dancing star.
Go to the top of the page
+Quote Post
windmiller
post Jun 11 2005, 22:07
Post #23





Group: Members
Posts: 205
Joined: 6-March 04
From: Chapel Hill, NC
Member No.: 12500



Thanks for the idea Atici, I am going to install and run it tonight.


--------------------
www.losslessaudioblog.com
Go to the top of the page
+Quote Post
windmiller
post Jun 13 2005, 22:09
Post #24





Group: Members
Posts: 205
Joined: 6-March 04
From: Chapel Hill, NC
Member No.: 12500



I just wanted to say that Beyond Compare is a great program! I have writing scripts and adding filters today and ran my first few tests. There are alot of sitautions where this app be a good solution.


--------------------
www.losslessaudioblog.com
Go to the top of the page
+Quote Post
atici
post Jun 13 2005, 23:45
Post #25





Group: Members (Donating)
Posts: 1180
Joined: 21-February 02
From: Chicago
Member No.: 1367



QUOTE (windmiller @ Jun 13 2005, 04:09 PM)
I just wanted to say that Beyond Compare is a great program! I have writing scripts and adding filters today and ran my first few tests. There are alot of sitautions where this app be a good solution.
*

Yes it is indeed. laugh.gif I got confused, why do you need scripts and filters for? huh.gif Let me know your experiences and how you solved the backup problem. Also by default Beyond Compare do not compare CRCs of the files, I'd suggest turning it on for higher confidence (if copy failed or not).

I know of this thread which you might also find interesting.

This post has been edited by atici: Jun 14 2005, 00:22


--------------------
The object of mankind lies in its highest individuals.
One must have chaos in oneself to be able to give birth to a dancing star.
Go to the top of the page
+Quote Post

2 Pages V   1 2 >
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 30th September 2014 - 21:24