IPB

Welcome Guest ( Log In | Register )

 
Reply to this topicStart new topic
ARFlac.pl - Check flac files with AccurateRip, Split from ARCue.pl (Topic ID: 53583)
Cerebus
post Feb 25 2008, 20:21
Post #1





Group: Members
Posts: 158
Joined: 31-October 02
Member No.: 3664



Here's the script, which I called ARFlac.pl. It requires that you have metaflac in your path. Note that much of this code came from ARCue.pl, and so I can't be expected to know everything about the internals of the script - I just adapted it to my needs.

Note:
  • This script does not currently work on rips with other than a 2 second pregap on track 1. All tracks will come back as being incorrect or not in the database.
  • This script does not currently work on rips from cds with a datatrack.
  • If the AR values that come back are all wrong, it is likely a different pressing or a different track 1 pregap. If only ONE doesn't match, that track is probably bad.
CODE
#!/usr/bin/perl

###############################################################################
# #
# ARFlac.pl #
# #
# KitchenStaff (Kitchen.Staff.Supervisor@gmail.com) #
# #
# Many thanks to Mr. Spoon for his kind permission to use the AccurateRip #
# technology and database. Access to AccurateRip is regulated, see #
# http://www.accuraterip.com/3rdparty-access.htm for details. #
# #
# Original ARCue script by Christopher Key <cjk32@cam.ac.uk> #
# #
# This script allows the usage of AccurateRip's extensive track checksum #
# database to verify the accuracy of directories of flac files. To use, #
# simply run: #
# #
# ARFlac.pl Flacdir1 Flacdir2 etc... #
# Note that this is not a perfect system. If the disc originally had a data #
# track, this routine may not find a match. This is also dependent on the #
# Accurate Rip database having seen the CD a sufficient number of times. #
# This tool should be used to verify that a set of files is good, but not to #
# necessarily indicate that the entire set is bad (different pressings, etc.) #
# #
###############################################################################

use strict;

use LWP;
use Carp;
use POSIX;

my $lwpUserAgent = LWP::UserAgent->new;

foreach my $flacDir (@ARGV) {

# Get the list of flac files from the directory
print $flacDir . ":";

opendir CURDIR, $flacDir;
my @flaclist = readdir CURDIR;
closedir CURDIR;

my @filelist = grep {/flac$/i} @flaclist;
@filelist = sort @filelist;

# We need track offsets in 'frames'
# can get these from metaflac
my (@trackOffsets,@trackLengths,@trackSamples);
my $tn=0;
my $tmpl=0;
foreach my $fl (@filelist) {

$tmpl = `metaflac --show-total-samples "$flacDir/$fl"`;
$trackSamples[$tn] = floor($tmpl/1);
$trackLengths[$tn] = ceil($tmpl/588);
printf $trackSamples[$tn] . ":";
$tn++;
}

my $trackCount = $tn;
my $curoff = 0;
$trackOffsets[0] = 0;
for ($tmpl = 1; $tmpl <= $trackCount; $tmpl++) {
$curoff += $trackLengths[$tmpl-1];
$trackOffsets[$tmpl] = $curoff;
}

# Calculate the three disc ids used by AR
my ($discId1, $discId2, $cddbDiscId) = (0, 0, 0);

{
use integer;

for (my $trackNo = 0; $trackNo <= $trackCount; $trackNo++) {
my $trackOffset = $trackOffsets[$trackNo];

$discId1 += $trackOffset;
$discId2 += ($trackOffset ? $trackOffset : 1) * ($trackNo + 1);
if ($trackNo < $trackCount) {
$cddbDiscId = $cddbDiscId + sumDigits(int($trackOffset/75)+2);
}
}

$cddbDiscId = (($cddbDiscId % 255) << 24) + ((int($trackOffsets[$trackCount]/75) - int($trackOffsets[0]/75)) << 8) + $trackCount;

$discId1 &= 0xFFFFFFFF;
$discId2 &= 0xFFFFFFFF;
$cddbDiscId &= 0xFFFFFFFF;
}

print "\nChecking AccurateRip database\n\n";

# See if we can find the disc in the database
my $arUrl = sprintf("http://www.accuraterip.com/accuraterip/%.1x/%.1x/%.1x/dBAR-%.3d-%.8x-%.8x-%.8x.bin",
$discId1 & 0xF, $discId1>>4 & 0xF, $discId1>>8 & 0xF, $trackCount, $discId1, $discId2, $cddbDiscId);

my $arDiscNotInDb = 0;
my $arNetworkFailed = 0;

my $response = $lwpUserAgent->get($arUrl);

if (!$response->is_success) {
if ($response->status_line =~ m/^404/) {
$arDiscNotInDb = 1;
}else{
$arNetworkFailed = $response->status_line;
}
}

# Extract CRCs from response data
my $arCrcCount = 0;
my @arTrackConfidences = ();
my @arTrackCRCs = ();

if (!($arDiscNotInDb || $arNetworkFailed)) {
my $arCrcData = $response->content;
my $ptr = 0;

while ($ptr < length($arCrcData)) {
my ($chunkTrackCount, $chunkDiscId1, $chunkDiscId2, $chunkCddbDiscId);

# Force perl to interpret these values as signed integers
{
use integer;

$chunkTrackCount = unpack("c",substr($arCrcData,$ptr,1));
$chunkDiscId1 = unpack("V",substr($arCrcData,$ptr+1,4)) + 0;
$chunkDiscId2 = unpack("V",substr($arCrcData,$ptr+5,4)) + 0;
$chunkCddbDiscId = unpack("V",substr($arCrcData,$ptr+9,4)) + 0;
}

$ptr +=13;

if ( $chunkTrackCount != $trackCount
|| $chunkDiscId1 != $discId1
|| $chunkDiscId2 != $discId2
|| $chunkCddbDiscId != $cddbDiscId ) {

croak("Track count or Disc IDs don't match.");
}

# How if it flagged that a track is not in the database?
for (my $track = 0; $track < $trackCount; $track++) {
my ($trackConfidence, $trackCrc);

# Force perl to interpret these values as signed integers
{
use integer;

$trackConfidence = unpack("c",substr($arCrcData,$ptr,1));
$trackCrc = unpack("V",substr($arCrcData,$ptr+1,4)) + 0;
$ptr += 9;
}

if ($arCrcCount == 0){
$arTrackConfidences[$track] = [];
$arTrackCRCs[$track] = [];
}

$arTrackConfidences[$track]->[$arCrcCount] = $trackConfidence;
$arTrackCRCs[$track]->[$arCrcCount] = $trackCrc;
}
$arCrcCount++;
}
}


printf "Track\tRipping Status\t\t[Disc ID: %08x-%08x]\n", $discId1, $cddbDiscId;

# Calculate a CRC for each track
my $errLevel = 0;

# Calculate a CRC for each track
my @trackCRCs = ();
my $FH;
my ($accuratelyRipped, $notAccuratelyRipped, $notInDatabase) = (0, 0, 0);
for (my $trackNo = 0; $trackNo < $trackCount; $trackNo++) {

# Open a pipe to flac decode
open($FH, "flac -d -c -f --force-raw-format --totally-silent --endian=little --sign=signed \"$flacDir/$filelist[$trackNo]\" |");
binmode $FH;

my ($frame, $CRC);
$CRC = 0;
$CRC = processFile($FH, $trackLengths[$trackNo], $trackNo==0, $trackNo==$trackCount-1);

close($FH);

{
use integer;
$trackCRCs[$trackNo] = $CRC & 0xFFFFFFFF;
}


if ($arDiscNotInDb) {
printf " %d\tTrack not present in database. [%08x]\n",
$trackNo + 1, $trackCRCs[$trackNo];

$notInDatabase++;
} elsif ($arNetworkFailed) {
printf " %d\t [%08x]\n",
$trackNo + 1, $trackCRCs[$trackNo];

} else {

my $foundCrc = 0;
my $foundCrcMatch = 0;

for (my $arCrcNo=0; $arCrcNo < $arCrcCount; $arCrcNo++) {
if ($arTrackConfidences[$trackNo]->[$arCrcNo] != 0){
$foundCrc = 1;

if ($arTrackCRCs[$trackNo]->[$arCrcNo] == $trackCRCs[$trackNo]) {
printf " %d\tAccurately Ripped (confidence %d) [%08x]\n",
$trackNo + 1, $arTrackConfidences[$trackNo]->[$arCrcNo], $arTrackCRCs[$trackNo]->[$arCrcNo];

$accuratelyRipped++;

$foundCrcMatch = 1;
last;
}
}
}
if (!$foundCrc) {
printf " %d\tTrack not present in database. [%08x]\n",
$trackNo + 1, $trackCRCs[$trackNo];

$notInDatabase++;
}elsif (!$foundCrcMatch) {
printf " %d\t** Rip not accurate ** (confidence %d) [%08x] [%08x]\n",
$trackNo + 1, $arTrackConfidences[$trackNo]->[0], $arTrackCRCs[$trackNo]->[0], $trackCRCs[$trackNo];

$notAccuratelyRipped++;
}
}
}

if ($arDiscNotInDb) {
print "Disc not present in AccurateRip database.\n";
$errLevel = 2;
} elsif ($arNetworkFailed) {
print "Failed to get $arUrl : " . $arNetworkFailed . "\n";
$errLevel = 3;
} elsif ($accuratelyRipped == $trackCount) {
print "All Tracks Accurately Ripped.\n";
} else {
if ($notAccuratelyRipped >= 3) {
print "Your CD disc is possibly a different pressing to the one(s) stored in AccurateRip.\n"
}

printf "Track(s) Accurately Ripped: %d\n", $accuratelyRipped;
printf "**** Track(s) Not Ripped Accurately: %d ****\n", $notAccuratelyRipped;
printf "Track(s) Not in Database: %d\n", $notInDatabase;

$errLevel = 1;
}

print "\n\n\n";
}

sub processFile {
use integer;
my ($FH, $tracklength, $firstTrack, $lastTrack) = @_;
my ($frame,$CRC,$frameOffset,$frameNo,$sample,$endFrame,$frmloop);

$CRC=0;

if ($firstTrack) {
# Skip first 4 frames
if ($tracklength<=4) {
return 0;
}

if (read($FH, $frame, 4*2352) != 4*2352) { croak ("read failed.") };

if (read($FH, $frame, 2352) != 2352) { croak ("read failed.") };
$sample = unpack("V",substr($frame,2348,4));
$CRC += $sample;
$frameNo = 5;
} else {
$frameNo = 0;
}

if ($lastTrack) {
$endFrame = $tracklength-5;
} else {
$endFrame = $tracklength;
}


for ($frmloop=$frameNo; $frmloop<$endFrame;$frmloop++) {

if (read($FH, $frame, 2352) != 2352) { croak ("read failed.") };
$frameOffset = $frmloop * 588;

foreach (unpack("V588", $frame)) {
$CRC += $_ * (++$frameOffset);
}
}

return $CRC;
}



sub sumDigits {
my $n = shift;
my $r = 0;
while ($n > 0) {
$r = $r + ($n % 10);
$n = int($n / 10);
}
return $r;
}

Moderation: Codebox.

This post has been edited by greynol: Apr 9 2008, 06:29
Go to the top of the page
+Quote Post
rohangc
post Apr 9 2008, 05:31
Post #2





Group: Members
Posts: 570
Joined: 1-December 02
From: India
Member No.: 3948



Thanks for ARFlac.pl Cerebus. This is exactly what I was looking for.

This post has been edited by greynol: Apr 9 2008, 06:12
Go to the top of the page
+Quote Post
Jean Tourrilhes
post Apr 10 2008, 21:59
Post #3





Group: Members
Posts: 37
Joined: 13-March 08
Member No.: 52008



Hi,

First, thank you for this excellent tool. Despite all the limitation (not your fault), I've started using it.

I've started trying a few disks that I have ripped and for which I have the AR results. For most disks, it works ok. However, for a few disks, it calculate a wrong CRC for the first track.

Does anybody has any clue about this, and why it's happening. Maybe we could brute force a solution for that.

Second, in many cases I already have a EAC log, containing the TOC. Could that be used optionally ? The idea is that it may help cases where some tracks are missing, and maybe also fix issues with track 1.

Thanks for the attention, and keep up the good work...

Jean
Go to the top of the page
+Quote Post
AliL
post Apr 11 2008, 14:24
Post #4





Group: Members
Posts: 95
Joined: 18-August 07
Member No.: 46314



Wow, I'd been wondering if there was a program out there that could do exactly this, brilliant! It is a bit slow though tongue.gif but I'm just grateful that someone actually put their time and effort into making this, thank you Cerebus (and Chris for the ortiginal arcue.pl script).

One thing though, as it's quite slow and I have a lot of albums to get through I was wondering if there was a way of it outputting the results to a log file or something similar as I could just point it to my files and let it get on with it overnight and as the command prompt only scrolls back a limited way eventually I'd lose some of the output.

Secondly, I have an album that says the tracks are accurately ripped but I get a confidence rating of -56. A bit odd I must say. Any reason this might be happening?

Thanks, AliL

P.S. I've thought of another question, however this is more related to the command prompt behaviour. Is there a way to batch add directories to the commandline rather than typing them in one at a time (or in my case dragging and dropping the path from the address bar into the cmd window)? Or is the script able to recursively check all folders within a directory (which would automatically fix the above question/problem)? Thanks.

This post has been edited by AliL: Apr 11 2008, 14:42
Go to the top of the page
+Quote Post
Corwin
post Jun 29 2008, 02:29
Post #5





Group: Members
Posts: 37
Joined: 20-May 07
Member No.: 43617



There's something not quite right here... accuraterip-crcgen does exactly the same thing.

$ ARFlac.pl /amber/music/artists/Metallica/Metallica\ -\ 1984\ -\ Ride\ the\ Lightning/
/amber/music/artists/Metallica/Metallica - 1984 - Ride the Lightning/:12561444:17529456:13734504:18214476:10776864:11652396:17490060:23554104:
Checking AccurateRip database

Track Ripping Status [Disc ID: 000dd6b4-730b1e08]
1 Accurately Ripped (confidence 2) [2019a2aa]
2 Accurately Ripped (confidence 2) [201f0a88]
3 Accurately Ripped (confidence 2) [868abd27]
4 Accurately Ripped (confidence 2) [9bdc2556]
5 Accurately Ripped (confidence 2) [59d21bf6]
6 Accurately Ripped (confidence 2) [aa48f821]
7 Accurately Ripped (confidence 2) [61113500]
8 Accurately Ripped (confidence 2) [f14421bc]
All Tracks Accurately Ripped.

I merged the files together into a single file wav using CUEtools (on W2K in a virtual machine), then tried this:

$ ARCue.pl Metallica\ -\ Ride\ the\ Lightning.cue
Metallica - Ride the Lightning.cue:

Checking AccurateRip database

Track Ripping Status [Disc ID: 000dd7d4-760b1e08]

1 Accurately Ripped (confidence 34) [2019a2aa]
2 Accurately Ripped (confidence 33) [201f0a88]
3 Accurately Ripped (confidence 38) [868abd27]
4 Accurately Ripped (confidence 36) [9bdc2556]
5 Accurately Ripped (confidence 36) [59d21bf6]
6 Accurately Ripped (confidence 36) [aa48f821]
7 Accurately Ripped (confidence 34) [61113500]
8 Accurately Ripped (confidence 30) [f14421bc]

_______________________

All Tracks Accurately Ripped.



All the checksums are the same but the confidence count is about 15 times lower....

BTW this is from the original log file when it was ripped over a year ago:

Track Ripping Status [Disc ID: 000dd7d4-760b1e08]

1 Accurately Ripped (confidence 14) [2019a2aa]
2 Accurately Ripped (confidence 14) [201f0a88]
3 Accurately Ripped (confidence 17) [868abd27]
4 Accurately Ripped (confidence 15) [9bdc2556]
5 Accurately Ripped (confidence 15) [59d21bf6]
6 Accurately Ripped (confidence 15) [aa48f821]
7 Accurately Ripped (confidence 14) [61113500]
8 Accurately Ripped (confidence 11) [f14421bc]

This post has been edited by Corwin: Jun 29 2008, 02:39
Go to the top of the page
+Quote Post
spoon
post Jun 29 2008, 07:07
Post #6


dBpowerAMP developer


Group: Developer (Donating)
Posts: 2749
Joined: 24-March 02
Member No.: 1615



The first has a different disc id.


--------------------
Spoon http://www.dbpoweramp.com
Go to the top of the page
+Quote Post
zer
post Jan 8 2009, 18:58
Post #7





Group: Members
Posts: 4
Joined: 8-January 09
Member No.: 65346



Ok, my programming skills are really minimal, I am not even sure how to properly add this option to the perl script,
but it is really easy to make it work on discs with some pregap.
simply initialize $curoff, and $trackOffsets[0] to the right offset sector instead of 0.

I am wanting to check a bunch check a bunch of old rips on linux.
I know CUEtools.NET does some real offset detection,
but without flac support on linux, it is cumbersome to use,
and I have not yet understood its algorithm.

but it appears to me that an ignorant approach will be fine
for 99+% of my rips with correct drive offset, but no cuesheet. (or without need to read cuesheet)
specifically, those with none, or a small silent pregap.

just noting that at least in my collection, some pregap lengths appear much more common than others.
for a subset of my rips (just those with eac logs w/ toc table section):
pregap len: count:
0 173
32 23
30 5
33 3
25,37,72,75 1 each

So, although an explanation of how real offset detection works would be nice,
I would appreciate if someone could help me make this scrips useful for bulk scanning
old rips that do not have AR results. An option to set the starting sector would work,
but maybe for this type of "cueless scanner" it would make sense to try at least a few
common pregap lengths if there is no match for the default assumption of 0?
maybe some more data on these pregap lengths would be useful,
but it appears to me that I could reduce the 'special case' from over 15% to under 1% by just doing a few more lookups.

what would be really great is adding the tags like dBpoweramp and CUEtools.NET do, but thats really over my head at this point.
Go to the top of the page
+Quote Post
patmcg
post Aug 25 2009, 12:32
Post #8





Group: Members
Posts: 74
Joined: 18-March 08
Member No.: 52123



Is this script still current and working. Also, is there a tool that will detect if only the offset is wrong? Thanks.


--------------------
Is your perfect hearing worth <$200? -- USE EAR PLUGS
Go to the top of the page
+Quote Post
edwardar
post Aug 25 2009, 19:03
Post #9





Group: Members
Posts: 98
Joined: 8-July 04
Member No.: 15139



QUOTE (patmcg @ Aug 25 2009, 12:32) *
Is this script still current and working. Also, is there a tool that will detect if only the offset is wrong? Thanks.


Cuetools checks rips against the accuraterip database, and allows you to correct offsets too!

http://www.hydrogenaudio.org/forums/index....c=66233&hl=
Go to the top of the page
+Quote Post

Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 1st October 2014 - 14:43