IPB

Welcome Guest ( Log In | Register )

 
Reply to this topicStart new topic
My Album Art Downloader Scripts
Abee
post Aug 31 2007, 06:18
Post #1





Group: Members
Posts: 3
Joined: 31-August 07
Member No.: 46650



Thanks for Marc Landis' excellent work, I can now much more efficiently acquire and manage album art images.

What is the Album Art Downloader :
http://www.hydrogenaudio.org/forums/index....showtopic=51896

I wrote three scripts for the Album Art Downloader using mutithread programming techniques, expecting to search and download faster... rolleyes.gif
just extract and release these .boo files to your AlbumArt\Scripts folder

Notice that they may NOT work simultaneously, run one script at a time:
Script searching Amazon.com:(unlike the original amazon.boo, this script searches the html site(not the xml one) and returns much more results)
Attached File  AmazonUSMultiThread.rar ( 1.8K ) Number of downloads: 907


Script searching Amazon.co.jp(efficient in searching japanese album arts)
Attached File  AmazonJPMultiThread.rar ( 1.83K ) Number of downloads: 641


Script searching Last.FM(efficient in searching artists' images)
Attached File  LastFMmultiThread.rar ( 1.1K ) Number of downloads: 796


Attention the mutithread techniques used in these scripts is experimental, and bugs might be encountered.
Expecting feedbacks smile.gif

This post has been edited by Abee: Aug 31 2007, 13:07
Go to the top of the page
+Quote Post
AlexVallat
post Aug 31 2007, 08:34
Post #2





Group: Developer
Posts: 1072
Joined: 31-July 06
Member No.: 33465



Glad to see someone else writing scripts for this now!

Just one point, your amazon US one searches partially amazon.com, and partially amazon.co.jp - I assume not intentional?

I'm interested in why you wrote these threaded - are you using the newer XUI version? There may be problems with cancelling, as to cancel a search (including if it reaches the specified maximum number of results) the current thread is aborted from within the AddThumb method. I did not consider that the thread adding thumbnails might not be the search thread itself.

I'm also trying to figure out what work it is you are trying to parellelise. Is it that you want to be downloading multiple thumbnails at once? XUI is already fairly well multi-threaded, but downloading thumbnails is done one at a time, true. If downloading multiple thumbnails at once offers a benefit, it might be worth having this in XUI itself, rather than requiring scripts to multi-thread, which is a bit more complex than should be required!

If you are interested in writing complex scripts (and using XUI), you might want to take a look at the AlbumArtDownloader.Scripts.dll, which has a public interface on it: AlbumArtDownloader.Scripts.IScript. Any class implementing this interface in a dll placed in the scripts folder will be loaded and used as a script, which frees you up to use any .net language to write scripts. You could even use a Boo script to create a class from it, and it should be automatically compiled up and used, but for boo scripts its probably easier just to stick to the existing magic-named static methods system.

Anyway, if you have any questions, I'll be around!

Alex
Go to the top of the page
+Quote Post
Abee
post Aug 31 2007, 10:59
Post #3





Group: Members
Posts: 3
Joined: 31-August 07
Member No.: 46650



QUOTE (AlexVallat @ Aug 31 2007, 15:34) *
...
Just one point, your amazon US one searches partially amazon.com, and partially amazon.co.jp - I assume not intentional?
...

Not intentional indeed...
It's a shame that i mistakenly submitted a half-finished one... tongue.gif and i have corrected it already.
Thanks for your check of my scripts... Actually i wrote my scripts based upon yours.. biggrin.gif

i am quite a green hand of Boo language...(i started using it just two weeks ago)
The reason why i wrote my scripts using multi-thread methods is that it takes too much time
to connect to Amazon.com, Amazon.co.jp or Last.FM in my position(somewhere in China)...
Occasionally i must spend 20 minutes or more waiting for the results being completely transported.

i've encountered lots of difficulties in managing multi-thread downloading..
first: how to record the number of results and keep the progressbar going...
second: i found Album Art Downloader will crash if i started too many threads, and now i use a
"thread pool" to restrict number of active threads.
these two problems afforementioned are what i solved last week, (using clumsy ways maybe...)

and, as you said, i found many problems which cannot be solved by myself.
the biggest one is what you mentioned:"There may be problems with cancelling" ,yes
when you push the "cancel" button, threads already started won't stop...
i found no way to handle this...
another problem is that the scripts may cause crash when results are too many.

i noticed that you said:
"it might be worth having this in XUI itself, rather than requiring scripts to multi-thread, which is a bit more complex than should be required!"
that is exactly what i want... because it does offer a much faster downloading speed.
you can compare my multi-thread scripts with scripts below:
Attached File  LastFM.rar ( 814bytes ) Number of downloads: 387
Attached File  AmazonUS.rar ( 1.28K ) Number of downloads: 365
Attached File  AmazonJP.rar ( 1.29K ) Number of downloads: 372


i will read more about web crawler and try to write more efficient scripts, simply because it's a fun for me.

BTW, i found trouble when installing Microsoft .NET Framework 3.0.
The Communication Fundation seems not compatible with my computer, but i installed
Workflow Fundation and Presentation Fundation correctly.
Therefore i could only use AlbumArt Downloader ver. 0.4.1.0, any higher version reports a
fatal error:
crying.gif
"...System.IO.FileNotFoundException: Could not load file or assembly 'System.ServiceModel, Version=3.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' or one of its dependencies. 系统找不到指定的文件(System cannot find designated file)。
File name: 'System.ServiceModel, Version=3.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'..."

maybe i should try a Windows Vista system...

This post has been edited by Abee: Aug 31 2007, 12:56
Go to the top of the page
+Quote Post
AlexVallat
post Aug 31 2007, 18:39
Post #4





Group: Developer
Posts: 1072
Joined: 31-July 06
Member No.: 33465



QUOTE (Abee @ Aug 31 2007, 10:59) *
BTW, i found trouble when installing Microsoft .NET Framework 3.0.
The Communication Fundation seems not compatible with my computer, but i installed
Workflow Fundation and Presentation Fundation correctly.
Therefore i could only use AlbumArt Downloader ver. 0.4.1.0, any higher version reports a
fatal error:
crying.gif
"...System.IO.FileNotFoundException: Could not load file or assembly 'System.ServiceModel, Version=3.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' or one of its dependencies. 系统找不到指定的文件(System cannot find designated file)。
File name: 'System.ServiceModel, Version=3.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'..."

maybe i should try a Windows Vista system...

Now that's weird. I've never heard of installing just bits of the .net framework. As far as I know, you just download the installer from here and it installs all of it. It sounds like yours is a bit broken at the moment, you might want to try removing and reinstalling it. I haven't tried it on Windows Vista, but it should work fine too.

If you can't use the latest XUI version, then there probably isn't much I can do about improving performance, though, as you won't be able to try anything out. In general terms though, I think the main factor is going to be number of simultaneous image downloads. In AAD as it stands, this is one. Each thumbnail must finish downloading before it starts on the next. The other extreme of unlimited simultaneous downloads (just spinning off another download thread for each image as the URL is discovered) is also not desirable, though, as it tends to lead to instability and poor performance. I'm not sure whether an optimal number of simultaneous downloads can be found, or whether one at a time is still the best way to go.

You might like to try downloading the search results html pages on a single separate thread, then calling AddThumb on the main search thread in response to html pages being downloaded. That way the image downloading occurs on the main search thread, and will respond to cancelling properly... just an idea, anyway.

Alex
Go to the top of the page
+Quote Post
Abee
post Sep 1 2007, 04:14
Post #5





Group: Members
Posts: 3
Joined: 31-August 07
Member No.: 46650



QUOTE (AlexVallat @ Sep 1 2007, 01:39) *
...
You might like to try downloading the search results html pages on a single separate thread, then calling AddThumb on the main search thread in response to html pages being downloaded. That way the image downloading occurs on the main search thread, and will respond to cancelling properly... just an idea, anyway.
...


The way you mentioned couldn't eliminate speed limitations when searching websites whose structure illustrated below:
Attached Image


Assume that it takes 10 seconds to finish each "go deeper" thread, such as A1 and A2, and it takes 5
seconds to get a new result page (thread D and E)
In the method you suggested, because thread E and A are parallel, total time should be 10*N seconds(N is the number of results).
But in circumstance that we start A1, B1 and C1 together, total time should be around 10*N/M (M is the number of simultaneously working threads) seconds if bandwidth is enough.
I think M should be about 5 to 10 (just a experiential value)

However, I agree that stability should be considered first...

This post has been edited by Abee: Sep 1 2007, 09:03
Go to the top of the page
+Quote Post
AlexVallat
post Sep 1 2007, 08:26
Post #6





Group: Developer
Posts: 1072
Joined: 31-July 06
Member No.: 33465



OK, I see where you're going with this. Yes, those intermediary pages will be a problem. They would also continue to be a problem even if AAD handled simultaneous downloading of thumbnails internally, as even though the AddThumb method would return immediately, you'd still have the download of the detail page running synchronously.

It would be interesting to see what sort of performance just having the detail pages downloaded simultaneously and keeping the image downloading on the main thread handled by AAD. This would indicate where the slow step was.

I'm thinking, the main search thread spins off one worker thread to download results pages (D&E), and a number (say 5) of detail download threads. It should then go into a loop watching an image download queue (probably waiting on a AutoResetEvent signaller to notify it the queue has changed).

As each results page is downloaded and parsed, every link to a detail page is added to a queue. The detail download threads should watch that queue, and when they find something in it, remove it from the queue and start downloading it (A1-C1). Care will have to be taken here to avoid them trampling on each other trying to process the same item. Once a detail download thread has downloaded a page, it parses it for the image link, and adds that link to the image download queue, and pokes the signaller to let the main search thread know that there are images in the queue.

The main search thread should then remove image urls from the image download queue and AddThumb them as results, looping round and returning to wait once the queue is empty.

Two cases need special consideration, ending and cancelling. For cancelling, you should trap the ThreadAbortedException around the queue watching loop in the main search thread, and if caught, abort all other threads and clear all queues too. For ending, I guess the only thing that knows when there are no more results is the results page download thread, so that would have to signal the main thread to stop waiting for the image download queue and return.

Personally, I would find something of this complexity a nightmare to write in Boo, without IDE support, but if you want to have a go, good luck!

Multithreaded programming is always interesting :-)
Go to the top of the page
+Quote Post
tr3
post Feb 6 2008, 05:09
Post #7





Group: Members
Posts: 1
Joined: 6-February 08
Member No.: 51068



is the amazon.co.jp add-on working properly?
i've been trying to download a lot of album art for my albums, but when i try to use this i get none.
if i look on amazon.co.jp manually all the album art is there. am i doing something wrong?
Go to the top of the page
+Quote Post
AstroArk
post Nov 15 2009, 12:46
Post #8





Group: Members
Posts: 1
Joined: 15-November 09
Member No.: 74964



Hi there.This may seem dumb.. But I'm new to foobar2000 and i was wondering, where the hell is the Albumart/Scripts folder? iv looked all over and can not seem to find it.
Go to the top of the page
+Quote Post
Benji99
post Nov 19 2009, 05:12
Post #9





Group: Members
Posts: 80
Joined: 4-October 04
Member No.: 17477



QUOTE (AstroArk @ Nov 15 2009, 06:46) *
Hi there.This may seem dumb.. But I'm new to foobar2000 and i was wondering, where the hell is the Albumart/Scripts folder? iv looked all over and can not seem to find it.


Please read the first post, there's a link to the thread explaining the program.
It's a standalone program that can interface with Foobar through the foo_run component.
Go to the top of the page
+Quote Post

Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 26th October 2014 - 04:57