gmusicbrowser Forum

Main board => Questions => Topic started by: Sven on April 10, 2010, 20:21:33

Title: Slow Library scanning
Post by: Sven on April 10, 2010, 20:21:33
When running gmb from the terminal it's usually spamming this while scanning the library:

Use of uninitialized value in subroutine entry at /usr/bin/gmusicbrowser line 488.

And this:

Invalid frameID 'Albu' (lowercase and/or space)
End of tag reached prematurely while reading frame Albu -> skipping rest of tag

Where Albu can also be replaced by Reco or Arti or something else of that form.

Both seem to make scanning a lot slower. I'm not sure if this is of any help, but is there a way to check what songs have invalid tags? It could be the songs that I ripped in FLAC format for example.
Title: Re: Slow Library scanning
Post by: Quentin Sculo on April 11, 2010, 02:13:32
I recently fixed a bug that could explain these invalid tags, if you have written tags on ape or mpc files with a v1.1.x version, instead of writing the changes in the ape tag, it created a bogus id3 tag :(
I fixed the bug in this commit : http://git.gmusicbrowser.org/commit/62d33403ad
I should release a v1.1.5 soon

I don't think these bogus tags or warnings should slow things too much. But I need to make some improvements to the way the db and the widgets are updated to make it more efficient.

about the "uninitialized value" warning, it's hard to tell, maybe you can send me the complete output by email, maybe with the -debug option.
Title: Re: Slow Library scanning
Post by: Sven on April 11, 2010, 05:03:39
I'm always using the latest git version, so that shouldn't be a problem. And I deleted all my ape files a while ago, because gmb thought they were like 237436943 hours long.

Anyways, I'll just send you the logs by email.

Fact is btw, that sometimes, it takes like 100 songs at a time, and sometimes one at a time. Looking at the progress bar and the text under it of course.
Title: Re: Slow Library scanning
Post by: Quentin Sculo on April 11, 2010, 16:56:29
I've just realized that the bug I mentioned above also affect mp3 files with ape tags. Though I don't think it could cause loss of data, it should be corrected the next time you edit the tag with a recent git version (a version including the commit above).

In the log you sent me, the message "No VBR header found, must count all the frames to determine length." shows what is the problem.
mp3 files that do not have a VBR header do not have length information, so unless the file is CBR (constant bit rate), which can't be sure, gmb needs to scan the whole file in order to know the length of the file.
This full scan is slow, I plan to add an option to only do the full scan of the current song, but in the meantime you'll have to wait until it's done, it's only done the first time the song is scanned, so that's not so bad.

And I deleted all my ape files a while ago, because gmb thought they were like 237436943 hours long.
Did you report it to me ? (it's possible I forgot) Anyway, don't hesitate to report this kind of problems and to send me an example file.
Title: Re: Slow Library scanning
Post by: Sven on April 11, 2010, 17:20:03
Nah, that was a long time ago when I just went linux all the way. I just ripped them again in ogg or something, because that seemed easy enough. So I didn't really report it.

And I figured that that was the problem when running it in debug mode. But it's happening every time my library is being scanned, which is every time I start gmb. Not only the first time. Maybe it's because of the wrong APE tags that it can't save the VBR headers?
Title: Re: Slow Library scanning
Post by: Quentin Sculo on April 11, 2010, 18:30:14
I found the problem, the file size and modification time were never updated  :o
And as the full scan is done every time the file size is different, the full scan was done everytime once their size had changed.
The file size doesn't usually change when the tag is modified, it depends on how much blank space is available, but modifying 2 tags (the id3v2 and ape) makes it much more likely to change the file size.

the fix is in the git repo :
http://git.gmusicbrowser.org/commit/890a1982f3

thanks for reporting this :)
Title: Re: Slow Library scanning
Post by: Sven on April 11, 2010, 18:38:18
Holy s**t, that was fast (both scanning and you fixing it ;)). Instead of it taking like a minute to scan, it now took less than a second. Once again: you're awesome :D

Edit: This was btw what I meant last time when I said it was really slow. I only just figured that it actually could be a bug :P
Title: Re: Slow Library scanning
Post by: Thomas1970 on May 25, 2010, 02:01:25
In the log you sent me, the message "No VBR header found, must count all the frames to determine length." shows what is the problem.
mp3 files that do not have a VBR header do not have length information, so unless the file is CBR (constant bit rate), which can't be sure, gmb needs to scan the whole file in order to know the length of the file.
This full scan is slow, I plan to add an option to only do the full scan of the current song, but in the meantime you'll have to wait until it's done, it's only done the first time the song is scanned, so that's not so bad.
I think I have a problem with this schan method.

I'm running GMB on Ubuntu on an old PC with little memory. My library currently contains more than 18000 songs and located on a NAS in my LAN, which is connected via NFS.
GMB really runs smooth in this environment!

This is my problem description:

After each program start, GMB looks for new and changed files. That's good, and the performance is acceptable.
But after the re-scan, the text in the progress window changes to:
"Prüfe Länge/Bitrate von 12767 Dateien ohne VBR-Header...", which means something like:
"Checking length/bitrate of 12767 files without VBR header..."
The progress indicator shows no progress, although on next launch the displayed number is decreased.
There is no "stop" button in this window (there was one during re-scanning).
You can close the window, but the operation seems to continue in background.

This check operation consumes 100% of my CPU, mainly because of iowait.
And because my PC is old, it becomes unusable in this state...

Three questions:
- What's the use of this frame counting check?
- Can you change this operation to not causing cpu iowait?
- Could you please update the window during the check operation such, that the progress indicator really indicates the progress and the number of files to check counts down and maybe add a "stop" button?

Thanks a lot!
Title: Re: Slow Library scanning
Post by: Quentin Sculo on May 25, 2010, 03:14:48
Three questions:
- What's the use of this frame counting check?
It's to determine the real length of some mp3 files, without this the length (and bitrate) is an estimation which is currently only based on the first audio frame if I remember correctly. The estimation should be all right for constant bitrate mp3 files, but without a VBR header, there's no way to tell if the file is constant bitrate or variable bitrate. Other programs might do an estimate on thefirst few frames, but it is sometimes very wrong.

- Can you change this operation to not causing cpu iowait?
no, the operation needs to read the whole file, which of course is not a good thing when the file are not local.

- Could you please update the window during the check operation such, that the progress indicator really indicates the progress and the number of files to check counts down and maybe add a "stop" button?
I think it updates, it's just that the progress is very slow. The displayed number is meant to be the total number of songs to scan since the start of the operation, and thus doesn't change.

I've changed the wording of this progress bar in v1.1.x, it should be a little clearer.
There is currently no stop button, because it wasn't meant to be stopped, because there was no way to restart it.
If you really want to disable this, you can replace this line in /usr/bin/gmusicbrowser (v1.0.x only) :
Code: [Select]
elsif ($m eq 'l') {push @LengthEstimated,$ID}by this line :
Code: [Select]
elsif ($m eq 'l') { $song[SONG_MISSINGSINCE]=undef } this will clear the list of songs that need a full scan on startup.

I plan to add a way to disable this in v1.1.x
Title: Re: Slow Library scanning
Post by: Thomas1970 on May 25, 2010, 23:33:21
Three questions:
- What's the use of this frame counting check?
It's to determine the real length of some mp3 files, without this the length (and bitrate) is an estimation which is currently only based on the first audio frame if I remember correctly. The estimation should be all right for constant bitrate mp3 files, but without a VBR header, there's no way to tell if the file is constant bitrate or variable bitrate. Other programs might do an estimate on thefirst few frames, but it is sometimes very wrong.

OK, I think I understand.
But my feeling is that a VBR header missing from a VBR file is an error.
And I think you should not try to catch this error with a functionality that consumes a lot of load.
Maybe you could add a "VBR length scan" command to the context menu of files and folders to allow the user to trigger the function?
Also you could trigger the function for each played (and not previously scanned) file?

There is currently no stop button, because it wasn't meant to be stopped, because there was no way to restart it.
If you really want to disable this, you can replace this line in /usr/bin/gmusicbrowser (v1.0.x only) :
Code: [Select]
elsif ($m eq 'l') {push @LengthEstimated,$ID}by this line :
Code: [Select]
elsif ($m eq 'l') { $song[SONG_MISSINGSINCE]=undef } this will clear the list of songs that need a full scan on startup.

I plan to add a way to disable this in v1.1.x

Does this code change affect only the VBR length scan?
Or does it suppress the folder scan on startup, too?

Thanks, Thomas
Title: Re: Slow Library scanning
Post by: Quentin Sculo on May 26, 2010, 02:30:37
But my feeling is that a VBR header missing from a VBR file is an error.
They are valid mp3 files, VBR headers are an addition (at least 2 different VBR header formats exist).

And I think you should not try to catch this error with a functionality that consumes a lot of load.
Maybe you could add a "VBR length scan" command to the context menu of files and folders to allow the user to trigger the function?
Also you could trigger the function for each played (and not previously scanned) file?
Yes, I'll add an option to automatically do that scan of not, and maybe do it on the first play.
To start the scan manually, it is already possible in v1.1.x by holding shift when selecting "re-read tags".

Does this code change affect only the VBR length scan?
Or does it suppress the folder scan on startup, too?
it only affect the VBR length scan
Title: Re: Slow Library scanning
Post by: Thomas1970 on May 26, 2010, 11:37:58
Does this code change affect only the VBR length scan?
Or does it suppress the folder scan on startup, too?
it only affect the VBR length scan
Cool! I've changed that and now the app works fine! Again: Thanks!
And again: I'm impressed by the performance of the app on my old system!
Title: Re: Slow Library scanning
Post by: Thomas1970 on May 26, 2010, 11:52:05
And I think you should not try to catch this error with a functionality that consumes a lot of load.
Maybe you could add a "VBR length scan" command to the context menu of files and folders to allow the user to trigger the function?
Also you could trigger the function for each played (and not previously scanned) file?
Yes, I'll add an option to automatically do that scan of not, and maybe do it on the first play.
To start the scan manually, it is already possible in v1.1.x by holding shift when selecting "re-read tags".
Great! I'm waiting for this change to come through via Ubuntu software updates.
Title: Re: Slow Library scanning
Post by: ochosi on May 28, 2010, 17:02:38
@Thomas1970: you can also try our ppa: https://launchpad.net/~shimmerproject/+archive/ppa
Title: Re: Slow Library scanning
Post by: bananas on August 19, 2011, 02:53:08
Hi, i have this problem, adding music to library take AGES and it is because of the VBR songs, i have never have problems with this songs using other players, i ve been trying all media players i have like 30 000 songs, how can i solve this problem? is there anyway to set to not calculate the lenght?

Im using gmb 1.1.7.

Thanks in advance.
Title: Re: Slow Library scanning
Post by: VastOne on August 19, 2011, 04:56:09
I m using the git version and it now has an option to scan VBR in 3 ways:

Never
When Added
When Current Song
Title: Re: Slow Library scanning
Post by: Quentin Sculo on August 19, 2011, 05:00:34
I've added an option to disable this. So you can either use the git version, wait for v1.1.8, or use ochosi's ppa (that should contain this change).
Title: Re:Slow Library scanning
Post by: bananas on August 19, 2011, 22:00:28
Im new in linux, what is a GIT version of programs? I have never been able to install anything but .deb or from repositories. I'll search for ochosi ppa.
Title: Odp: Slow Library scanning
Post by: VastOne on August 19, 2011, 22:25:02
open terminal and run

Code: [Select]
git clone http://github.com/squentin/gmusicbrowser.git

that will create a “gmusicbrowser” folder with the latest version of gmusicbrowser, it also contains all gmusicbrowser's history since v1.0 (in gmusicbrowser/.git/)

You can run the currently checked-out version simply by running :

Code: [Select]
./gmusicbrowser.pl
You can of course run it from any folder by using the full path, for example :

Code: [Select]
/home/myname/gmusicbrowser/gmusicbrowser.pl
You can use the -C gmusicbrowser option to specify an alternate configuration file :

Code: [Select]
./gmusicbrowser.pl -C mynewconf
You can update the source weekly by running the following in terminal within the directory:

/home/your username/gmusicbrowser

Code: [Select]
git pull

Source of the above

http://wiki.gmusicbrowser.org/doku.php?id=guide:git
Title: Re: Slow Library scanning
Post by: bananas on August 19, 2011, 22:40:17
Thanks! I downloaded and installed GIT, dunno why :P ... and also i cannot even find it in my menu programs.


I got this after running the first line VastOne suggest:



Code: [Select]
/usr/bin/git-clone: 312: curl: not found
Cannot get remote repository information.
Perhaps git-update-server-info needs to be run there?

And also cant finde ochosi ppa :/

EDIT: I found the ppa, now the problem is solved :)

but

Now this version has no more the layout SHIMMER BROWSER from 1.1.7 =( is there anyway to get it back?
Title: Re: Slow Library scanning
Post by: Quentin Sculo on August 20, 2011, 02:33:05
Strange, I'm not responsible for the ppa, but I thought the ppa should have the shimmer layouts.
You can download the shimmer layouts file there :
https://raw.github.com/shimmerproject/gmusicbrowser/master/layouts/shimmer.layout
just put it in ~/.config/gmusicbrowser/layouts/ and restart gmb.
Title: Re: Slow Library scanning
Post by: bananas on August 25, 2011, 17:56:47
thanks again!  :)
Title: Re: Slow Library scanning
Post by: ochosi on August 25, 2011, 22:52:19
Quote
Now this version has no more the layout SHIMMER BROWSER from 1.1.7 =( is there anyway to get it back?

i'm wondering how that can happen. the layouts should definitely be there, i just checked the package again right now.
could you check whether the file "shimmer.layout" exists in /usr/share/gmusicbrowser/layouts ?
thanks in advance!