IPB

Welcome Guest ( Log In | Register )

24 Pages V  « < 5 6 7 8 9 > »   
Reply to this topicStart new topic
MP3 repacker
bukem
post Jun 24 2006, 21:40
Post #151





Group: Members
Posts: 120
Joined: 14-September 03
From: Poland
Member No.: 8837



Unfortunately the file is 12MB, but I've managed to upload it to the RapidShare. I hope that it will help you debug the problem. BTW, I should mention it before - thank you for mp3packer, it's great tool!

Ps. and here comes the link -> suspicious mp3 file
Go to the top of the page
+Quote Post
Omion
post Jun 25 2006, 07:48
Post #152





Group: Developer
Posts: 432
Joined: 22-February 04
From: San Diego, CA
Member No.: 12180



Thanks for the file. It looks like the problem is a buffer overrun, right where I added the code to protect against a buffer underrun in 1.02. The odd thing is that the file looks perfectly OK, but it's requesting 9 bytes from the frame following the problem frame. Then the next frame wants 469 bytes from the bit reservoir... the file just seems to be screwed up right there, and my program is panicking.

I just checked, and Foobar will skip to the next song at 6:05, which is right where the problem frame occurs. [edit: I just checked again, and Foobar is no longer skipping to the next song there... odd]
No other mp3 checker programs would have caught it, because I don't think they check the bit reservoir for invalid stuff.

So I'll do a check for buffer overruns, and put up a warning and pad with garbage if that happens (just like I do with underruns)


@psyllium:
Your /dev/shm comment pushed me over the edge...
I got my diskless server running Ubuntu Linux, and it's quite awesome. I make it automatically mount a ramdisk on startup, fill it up with my web site, and synchronize new files every few hours. All without 3rd party tools. MAN, Linux is powerful. cool.gif
The only thing I miss from Windows is filesystem-based compression like NTFS has. That would have saved me a lot of headaches. Well, it's only a matter of time before some Linux FS supports it.

This post has been edited by Omion: Jun 25 2006, 10:26


--------------------
"We demand rigidly defined areas of doubt and uncertainty!" - Vroomfondel, H2G2
Go to the top of the page
+Quote Post
bukem
post Jun 25 2006, 11:32
Post #153





Group: Members
Posts: 120
Joined: 14-September 03
From: Poland
Member No.: 8837



Thanks Omion for your help. I'm looking forward for new version of mp3packer.
Go to the top of the page
+Quote Post
kerminen
post Jun 25 2006, 13:30
Post #154





Group: Members
Posts: 88
Joined: 6-February 03
Member No.: 4884



@Omion:

http://www.cenatek.com/product_ramdisk.cfm

ramdisks up to 3GB, there is a comprehensive Users manual to download from that web page...


--------------------
HA Folding
http://fah-web.stanford.edu/teamstats/team32639.html
Go to the top of the page
+Quote Post
Firon
post Jun 25 2006, 19:36
Post #155





Group: Members
Posts: 830
Joined: 3-November 05
Member No.: 25526



Costs like $50 for a license, unfortunately.
Go to the top of the page
+Quote Post
Omion
post Jun 26 2006, 07:56
Post #156





Group: Developer
Posts: 432
Joined: 22-February 04
From: San Diego, CA
Member No.: 12180



Allrighty, 1.04's out.

Fixed bukem's issue, so it will now put up a warning that something is wrong instead of crashing. Note that there will still be a minor glitch where the error occurred.

Also tweaked the silent-frame detection algorithm, and made it optional. If you have a song which has a lot of silence in it, but mp3packer isn't doing anything to it, use the -z option. The reason I made it an option is that I don't know if my criteria for silent frames will actually include some non-silent frames.

There was another obscure LAME/XING tag related bug which is also squashed.


@kerminen:
Firon pointed out why I didn't consider that. I actually had a pretty good ramdisk for Windows, but it was also a trial to a $50 program. In the end, Linux is free. cool.gif

I'm still having some issues with Linux (well, I think it's my CF card) and wireless support is no good, but I think I'll stick with Linux, at least on that computer. It just feels better.


--------------------
"We demand rigidly defined areas of doubt and uncertainty!" - Vroomfondel, H2G2
Go to the top of the page
+Quote Post
bukem
post Jul 14 2006, 17:09
Post #157





Group: Members
Posts: 120
Joined: 14-September 03
From: Poland
Member No.: 8837



@Omion:

Sorry for bothering you again, but I've found another problematic mp3 files:

1. Fatal error: exception Mp3types.Too_many_bytes case:

Again, the suspicious file seems to have problems with the bit reservoir, but nevertheless to implemented by you buffer checks it's crashing mp3packer v1.04-126. Here you have output:

CODE
D:\tmp>mp3packer -i "02. the method - i've got a cat.mp3"
*** '02. the method - i've got a cat.mp3'
INFO:
MPEG1 layer 3
7489 frames
44100 Hz
38.281250 frames per second
195.631020 seconds
6260727 bytes in file (256.021851 kbps)
6260192 bytes in MP3 frames (255.999973 kbps) = current bitrate
47901297 bits of payload data (244.855325 kbps)
5990964 bytes of payload data (244.990349 kbps)
26415 bits wasted from partially-full bytes (0.135025 kbps)
6260568 bytes of MP3 data (256.015349 kbps) = minimum bitrate possible
-376 bytes of padding (-0.015376 kbps)
535 bytes outside MP3 frames (0.021878 kbps)
Bitrate distribution:
  256: 612,6877
Largest frame uses 8704 bits = 1088 bytes = 333.200000 kbps
Smallest bitrate for CBR is 320

D:\tmp>mp3packer "02. the method - i've got a cat.mp3"
*** '02. the method - i've got a cat.mp3' -> '02. the method - i've got a cat-vbr.mp3'
0% done on frame 0
Buffer underflow; added 451 zeros to the beginning of frame 0
Fatal error: exception Mp3types.Too_many_bytes


2. Fatal error: exception End_of_file case:

That file this time crash the mp3packer completely.

CODE
D:\tmp>mp3packer -i "01. stylin' - drifting sun.mp3"
*** '01. stylin' - drifting sun.mp3'
Fatal error: exception End_of_file

D:\tmp>mp3packer "01. stylin' - drifting sun.mp3"
*** '01. stylin' - drifting sun.mp3' -> '01. stylin' - drifting sun-vbr.mp3'
Fatal error: exception End_of_file


FYI -> mp3trim doesn't report any problems with these files.
Go to the top of the page
+Quote Post
kevinsham
post Jul 15 2006, 13:07
Post #158





Group: Members
Posts: 47
Joined: 23-April 02
Member No.: 1856



QUOTE (jaybeee @ Feb 13 2006, 21:28) *
QUOTE (kevinsham @ Feb 13 2006, 01:21 PM)
A feature request:
Rename the original file and use the input file name as output.
*
Ah yes.
As in: shove '-orig' on the end of the original input file, and thus the resulting output file can have the original's original name (follow me!?).


Is this feature planned to be included in the future?
Thank you very much.
Go to the top of the page
+Quote Post
Omion
post Jul 15 2006, 16:00
Post #159





Group: Developer
Posts: 432
Joined: 22-February 04
From: San Diego, CA
Member No.: 12180



@kevinsham (and others):
I should be able to add original-file renaming to the next version.

QUOTE (bukem @ Jul 14 2006, 09:09) *
CODE
D:\tmp>mp3packer -i "02. the method - i've got a cat.mp3"
*** '02. the method - i've got a cat.mp3'
-376 bytes of padding (-0.015376 kbps)

D:\tmp>mp3packer "02. the method - i've got a cat.mp3"
*** '02. the method - i've got a cat.mp3' -> '02. the method - i've got a cat-vbr.mp3'
0% done on frame 0
Buffer underflow; added 451 zeros to the beginning of frame 0
Fatal error: exception Mp3types.Too_many_bytes


CODE
D:\tmp>mp3packer -i "01. stylin' - drifting sun.mp3"
*** '01. stylin' - drifting sun.mp3'
Fatal error: exception End_of_file

D:\tmp>mp3packer "01. stylin' - drifting sun.mp3"
*** '01. stylin' - drifting sun.mp3' -> '01. stylin' - drifting sun-vbr.mp3'
Fatal error: exception End_of_file

I think I know the problems with both files. The first one appears to have been cut improperly (hence the buffer underflow warning and negative padding). The problem is that whereas the original file could put those bytes in the previous (non-existant) frame, mp3packer has no such luxury and panics because there are too many bytes.

The second problem is due to the XING frame not having CRC data, but the rest of the frames do. My program reads the XING tag, assumes that all valid frames have no CRC data, and throws out the rest of the frames.

I should be able to fix both problems tomorrow.

This post has been edited by Omion: Jul 15 2006, 16:01


--------------------
"We demand rigidly defined areas of doubt and uncertainty!" - Vroomfondel, H2G2
Go to the top of the page
+Quote Post
Omion
post Jul 17 2006, 07:47
Post #160





Group: Developer
Posts: 432
Joined: 22-February 04
From: San Diego, CA
Member No.: 12180



All right. Got 1.05 out.

I added the -u switch to do original-file renaming as many have suggested. For example:
CODE
mp3packer -u file.mp3 file-orig.mp3
will rename file.mp3 to file-orig.mp3, then save the packed version as file.mp3.

Note that if you use the -u option and leave off the output file, the original file will have "-vbr" appended to it, instead of the new file. This is the default from before, but it doesn't make much sense with -u. So always either specify the output file or use something like "-a -orig" to override the appending.

I also sort of fixed the problems bukem was having. The second file now works flawlessly, but the first file is still a bit odd. There are at least 3 things wrong with it:
  1. The XING tag has no CRC data, but the rest of the frames do. This was worked around in 1.05.
  2. The XING tag has some LAME info that Foobar reads, but my program doesn't because the internal LAME CRC is not there (this is a different CRC from the one mentioned in #1) The result of this is that Foobar will trim 576 samples off the original file, but not from the repacked one, resulting in "bit compare tracks" resulting in quite a large failure.
  3. Even if I manually fixed that in the file, the resulting files are a tiny bit different. I have NO idea where this is coming from. The maximum difference is -135dB, so it will be impossible to hear, but that means that something along the way is lossy for ONLY this one file. I really don't know why. blink.gif
Everything else should be OK though.

This post has been edited by Omion: Jul 20 2006, 06:44


--------------------
"We demand rigidly defined areas of doubt and uncertainty!" - Vroomfondel, H2G2
Go to the top of the page
+Quote Post
bukem
post Jul 17 2006, 13:03
Post #161





Group: Members
Posts: 120
Joined: 14-September 03
From: Poland
Member No.: 8837



@Omion:

Thanks again. I'm going to check new version of mp3packer ASAP and let you know the results.

Edit: [@Omion] I'm in the middle of the conversion process of ~30 GB mp3 test set. That's step one. Next step is to bit compare source tracks with converted tracks to find any inconsistency - if any I'll let you know.

This post has been edited by bukem: Jul 20 2006, 18:50
Go to the top of the page
+Quote Post
Omion
post Jul 20 2006, 21:14
Post #162





Group: Developer
Posts: 432
Joined: 22-February 04
From: San Diego, CA
Member No.: 12180



QUOTE (bukem @ Jul 17 2006, 05:03) *
[@Omion] I'm in the middle of the conversion process of ~30 GB mp3 test set. That's step one. Next step is to bit compare source tracks with converted tracks to find any inconsistency - if any I'll let you know.

Very cool. That would really help improve/confirm the stability of mp3packer. Unfortunately, I don't know of any way to automate the "Bit-compare tracks" command from Foobar, so it may take a while.

Also note that if a file reports an error in mp3packer, the conversion will probably not be lossless. However, if there was an error, then the file was "wrong" to begin with, so being bit-exact won't matter. (If you have too much time on your hands, though, you can check to make sure that the lossy parts are only concentrated around the error)

BTW, I found out that the non-exactness of the "I've got a cat" file was due to Foobar. Testing it with 0.9 reports no differences. I had previously used 0.8.3, which apparently had an extremely minor roundoff issue. Which is a relief, because I was pulling my hair out looking at the hex output of mp3packer, trying to see what went wrong. It's a good thing it's not my fault laugh.gif That means that, if you can, use 0.9 to do the comparison.


--------------------
"We demand rigidly defined areas of doubt and uncertainty!" - Vroomfondel, H2G2
Go to the top of the page
+Quote Post
bukem
post Jul 21 2006, 00:04
Post #163





Group: Members
Posts: 120
Joined: 14-September 03
From: Poland
Member No.: 8837



@Omion

QUOTE
BTW, I found out that the non-exactness of the "I've got a cat" file was due to Foobar. Testing it with 0.9 reports no differences. I had previously used 0.8.3, which apparently had an extremely minor roundoff issue.


Really good news smile.gif. I was a bit afraid that mp3packer may lose some bits.

QUOTE
Unfortunately, I don't know of any way to automate the "Bit-compare tracks" command from Foobar, so it may take a while.


That's not a problem. With f2k you can bit-compare several tracks at once. Just add folder with source mp3's to the playlist and then add folder with converted mp3's to the same playlist, select all tracks, rightclick and choose bit-compare and voila!

BTW. It seems that I've found a small glitch in WinMp3Packer GUI - I have to write simple batch file to convert the test set again. Edit: My fault smile.gif

This post has been edited by bukem: Jul 21 2006, 00:12
Go to the top of the page
+Quote Post
Omion
post Jul 21 2006, 00:10
Post #164





Group: Developer
Posts: 432
Joined: 22-February 04
From: San Diego, CA
Member No.: 12180



QUOTE (bukem @ Jul 20 2006, 16:04) *
That's not a problem. With f2k you can bit-compare several tracks at once. Just add folder with source mp3's to the playlist and then add folder with converted mp3's to the same playlist, select all tracks, rightclick and choose bit-compare and voila!

Whoa. w00t.gif That is AWESOME. I just keep liking Foobar more and more.


--------------------
"We demand rigidly defined areas of doubt and uncertainty!" - Vroomfondel, H2G2
Go to the top of the page
+Quote Post
bukem
post Jul 21 2006, 00:16
Post #165





Group: Members
Posts: 120
Joined: 14-September 03
From: Poland
Member No.: 8837



@Omion
Yeah, F2K is the tool you can fall in love with wink.gif. BTW, I just had to start the conversion process again because of my fault (too drunk?) so I'll update you about results tomorrow.

This post has been edited by bukem: Jul 21 2006, 00:17
Go to the top of the page
+Quote Post
Firon
post Jul 21 2006, 00:59
Post #166





Group: Members
Posts: 830
Joined: 3-November 05
Member No.: 25526



Yeah, it compares the first half of the tracks with the counterpart in the second half of the list. Works the same for the masstagger's copy tags function.
Go to the top of the page
+Quote Post
bukem
post Jul 21 2006, 01:49
Post #167





Group: Members
Posts: 120
Joined: 14-September 03
From: Poland
Member No.: 8837



@Omion

I've found first broken mp3 file in my test set. It's negative padding case again. In fact all tracks from that particular album have the same issue.

Here's output from mp3packer -i:
CODE
*** '01. fatboy slim - right here, right now.mp3'
INFO:
MPEG1 layer 3
14851 frames
44100 Hz
38.281250 frames per second
387.944490 seconds
9091943 bytes in file (187.489566 kbps)
9091413 bytes in MP3 frames (187.478637 kbps) = current bitrate
106667278 bits of payload data (274.955002 kbps)
13339229 bytes of payload data (275.075004 kbps)
46554 bits wasted from partially-full bytes (0.120002 kbps)
13873865 bytes of MP3 data (286.100004 kbps) = minimum bitrate possible
-4782452 bytes of padding (-98.621367 kbps)
530 bytes outside MP3 frames (0.010929 kbps)
Bitrate distribution:
   32: 9,0
   48: 1,0
  128: 898,0
  160: 5798,0
  192: 5435,0
  224: 1253,0
  256: 718,0
  320: 739,0
Largest frame uses 15488 bits = 1936 bytes = 592.900000 kbps
Fatal error: exception Mp3types.Too_many_bytes


Edit:
I've found solution in that case -> Fix MP3 Header from F2K did the trick! After that mp3packer has no problems with these tracks.

CODE
*** '01. fatboy slim - right here, right now.mp3.mp3'
INFO:
MPEG1 layer 3
14850 frames
44100 Hz
38.281250 frames per second
387.918367 seconds
9091995 bytes in file (187.503264 kbps)
9091257 bytes in MP3 frames (187.488044 kbps) = current bitrate
68159667 bits of payload data (175.706212 kbps)
8526466 bytes of payload data (175.840418 kbps)
52061 bits wasted from partially-full bytes (0.134206 kbps)
9061066 bytes of MP3 data (186.865418 kbps) = minimum bitrate possible
30191 bytes of padding (0.622626 kbps)
738 bytes outside MP3 frames (0.015220 kbps)
Bitrate distribution:
   32: 9,0
  128: 898,0
  160: 5798,0
  192: 5435,0
  224: 1253,0
  256: 718,0
  320: 739,0
Largest frame uses 8918 bits = 1115 bytes = 341.392187 kbps
Smallest bitrate for CBR is 320


The strange thing, though, that mp3trim doesn't found any problems with these tracks?

This post has been edited by bukem: Jul 21 2006, 02:03
Go to the top of the page
+Quote Post
Omion
post Jul 21 2006, 02:23
Post #168





Group: Developer
Posts: 432
Joined: 22-February 04
From: San Diego, CA
Member No.: 12180



Wow. There's 4MB of MP3 data which just isn't in the file! I'll find out what's going on. Just keep sending me the broken ones biggrin.gif

This post has been edited by Omion: Jul 21 2006, 02:27


--------------------
"We demand rigidly defined areas of doubt and uncertainty!" - Vroomfondel, H2G2
Go to the top of the page
+Quote Post
bukem
post Jul 21 2006, 02:33
Post #169





Group: Members
Posts: 120
Joined: 14-September 03
From: Poland
Member No.: 8837



In fact I just found and fixed another album which had the same symptoms like that one before. Fix MP3 Header helped again wink.gif

This post has been edited by bukem: Jul 21 2006, 02:33
Go to the top of the page
+Quote Post
Omion
post Jul 21 2006, 02:57
Post #170





Group: Developer
Posts: 432
Joined: 22-February 04
From: San Diego, CA
Member No.: 12180



Bleh. I found the problem. The XING frame is not parsed as an XING frame (it may be invalid... I'll have to check) and so something happens which causes mp3packer to think that the CRC in each frame is actually the bit-reservoir offset. Hilarity ensues.

I'll get this fixed soon.

EDIT: I fixed the errors all over the place, but now it throws out all the frames. Keep hacking...

This post has been edited by Omion: Jul 21 2006, 03:07


--------------------
"We demand rigidly defined areas of doubt and uncertainty!" - Vroomfondel, H2G2
Go to the top of the page
+Quote Post
Omion
post Jul 21 2006, 04:51
Post #171





Group: Developer
Posts: 432
Joined: 22-February 04
From: San Diego, CA
Member No.: 12180



I think I fixed the problem. Get 1.06.

The problem, if you really want to know, is that the XING tag's frame does not have a CRC on it, and it also does not specify an encoder. mp3packer sees that it doesn't have an encoder, assumes that it's not an XING frame, then assumes that all other valid frames don't have CRCs either. HOWEVER, I never actually check if the other frames have CRCs, so the CRC is used for the bit reservoir offset. This creates errors all over the place, until there is eventually too much data to even fit into a 320kbps frame. The program then croaks.

Aren't you glad you asked? (oh wait. You didn't ask...)


--------------------
"We demand rigidly defined areas of doubt and uncertainty!" - Vroomfondel, H2G2
Go to the top of the page
+Quote Post
bukem
post Jul 21 2006, 14:31
Post #172





Group: Members
Posts: 120
Joined: 14-September 03
From: Poland
Member No.: 8837



Omion, how about another nasty mp3 track ? tongue.gif . It fails in mp3packer with too_many_bytes error, but when treat with f2k's fix mp3 header magic happens and mp3packer has no problems with processing that file anymore.

Edit: Do anybody know what exactly f2k's fix mp3 header fuction do? Forget it - I've found answer rolleyes.gif

This post has been edited by bukem: Jul 21 2006, 14:58
Go to the top of the page
+Quote Post
Omion
post Jul 21 2006, 15:55
Post #173





Group: Developer
Posts: 432
Joined: 22-February 04
From: San Diego, CA
Member No.: 12180



Bleh. It's that LAME tag being inconsistant again. The LAME tag header says that it's not original, but the rest of the frames are original. I fixed it, but I don't have time right now to release it, but here's an EXE that should be fixed.


--------------------
"We demand rigidly defined areas of doubt and uncertainty!" - Vroomfondel, H2G2
Go to the top of the page
+Quote Post
bukem
post Jul 22 2006, 13:20
Post #174





Group: Members
Posts: 120
Joined: 14-September 03
From: Poland
Member No.: 8837



Great! I've just finished mp3packing 37 531 877 KB of mp3 music (4862 tracks) and I saved 166 657 KB which is about 0.44 %. Now I've started bit-comparing source and mp3packed tracks to find any inconsistency (hope not smile.gif).

Edit 1:
I've found first inconsistent track crying.gif ).

CODE
Comparing:
"R:\music\..good looking records\.other\studio x\01. scalar - solar.mp3"
"D:\documents\music\.streams\music\..good looking records\.other\studio x\01. scalar - solar.mp3"
Comparing failed (length mismatch).


Edit 2:
Something is wrong here. Till now in 722 mp3 tracks, f2k bit-compare found 65 files with differences, 125 files failed to compare because of Comparing failed (length mismatch) blink.gif

And here an example of mp3 with differences:
CODE
Comparing:
"R:\music\..good looking records\.other\.singles\looking good - (lgr019) 01. moonchild - seatown.mp3"
"D:\documents\music\.streams\music\..good looking records\.other\.singles\looking good - (lgr019) 01. moonchild - seatown.mp3"
differences found: 3255 sample(s), starting at 25.5749887 second(s), peak: 0.1141328 at 25.5886621 second(s), 2ch


Edit 3:
Huston, we have a problem -> another 614 files processed and here are results:
83 files - Comparing failed (length mismatch)
109 files - Differences found

This post has been edited by bukem: Jul 22 2006, 20:43
Go to the top of the page
+Quote Post
Omion
post Jul 23 2006, 04:09
Post #175





Group: Developer
Posts: 432
Joined: 22-February 04
From: San Diego, CA
Member No.: 12180



QUOTE (bukem @ Jul 22 2006, 05:20) *
Huston, we have a problem -> another 614 files processed and here are results:
83 files - Comparing failed (length mismatch)
109 files - Differences found

Run a few of the files that had errors through mp3packer again and look for any errors that may have occurred. There may have been over/underruns in the files which mp3packer threw out. Your example in "Edit 2" looks like there may have been a bad frame or two.

Also, I don't think I've ever run across a "Comparing failed (length mismatch)" error... What version of Foobar are you using? Could you upload an mp3 that does it? [ edit: Oops. You did upload one. Testing... ]


Actually, I just realized that mp3packer does not give a warning for sync errors, which may make it difficult to see if the error was in the original or in the conversion.

One more thing: you didn't use the -z switch, did you?

This post has been edited by Omion: Jul 23 2006, 04:10


--------------------
"We demand rigidly defined areas of doubt and uncertainty!" - Vroomfondel, H2G2
Go to the top of the page
+Quote Post

24 Pages V  « < 5 6 7 8 9 > » 
Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 25th November 2014 - 00:02