Slow Library scanning

[Sven]

When running gmb from the terminal it's usually spamming this while scanning the library:

Use of uninitialized value in subroutine entry at /usr/bin/gmusicbrowser line 488.

And this:

Invalid frameID 'Albu' (lowercase and/or space)
End of tag reached prematurely while reading frame Albu -> skipping rest of tag

Where Albu can also be replaced by Reco or Arti or something else of that form.

Both seem to make scanning a lot slower. I'm not sure if this is of any help, but is there a way to check what songs have invalid tags? It could be the songs that I ripped in FLAC format for example.

[Quentin Sculo]

I recently fixed a bug that could explain these invalid tags, if you have written tags on ape or mpc files with a v1.1.x version, instead of writing the changes in the ape tag, it created a bogus id3 tag
I fixed the bug in this commit : http://git.gmusicbrowser.org/commit/62d33403ad
I should release a v1.1.5 soon

I don't think these bogus tags or warnings should slow things too much. But I need to make some improvements to the way the db and the widgets are updated to make it more efficient.

about the "uninitialized value" warning, it's hard to tell, maybe you can send me the complete output by email, maybe with the -debug option.

[Sven]

I'm always using the latest git version, so that shouldn't be a problem. And I deleted all my ape files a while ago, because gmb thought they were like 237436943 hours long.

Anyways, I'll just send you the logs by email.

Fact is btw, that sometimes, it takes like 100 songs at a time, and sometimes one at a time. Looking at the progress bar and the text under it of course.

[Quentin Sculo]

I've just realized that the bug I mentioned above also affect mp3 files with ape tags. Though I don't think it could cause loss of data, it should be corrected the next time you edit the tag with a recent git version (a version including the commit above).

In the log you sent me, the message "No VBR header found, must count all the frames to determine length." shows what is the problem.
mp3 files that do not have a VBR header do not have length information, so unless the file is CBR (constant bit rate), which can't be sure, gmb needs to scan the whole file in order to know the length of the file.
This full scan is slow, I plan to add an option to only do the full scan of the current song, but in the meantime you'll have to wait until it's done, it's only done the first time the song is scanned, so that's not so bad.

And I deleted all my ape files a while ago, because gmb thought they were like 237436943 hours long.

Did you report it to me ? (it's possible I forgot) Anyway, don't hesitate to report this kind of problems and to send me an example file.

[Sven]

Nah, that was a long time ago when I just went linux all the way. I just ripped them again in ogg or something, because that seemed easy enough. So I didn't really report it.

And I figured that that was the problem when running it in debug mode. But it's happening every time my library is being scanned, which is every time I start gmb. Not only the first time. Maybe it's because of the wrong APE tags that it can't save the VBR headers?

[Quentin Sculo]

I found the problem, the file size and modification time were never updated 
And as the full scan is done every time the file size is different, the full scan was done everytime once their size had changed.
The file size doesn't usually change when the tag is modified, it depends on how much blank space is available, but modifying 2 tags (the id3v2 and ape) makes it much more likely to change the file size.

the fix is in the git repo :
http://git.gmusicbrowser.org/commit/890a1982f3

thanks for reporting this

[Sven]

Holy s**t, that was fast (both scanning and you fixing it ). Instead of it taking like a minute to scan, it now took less than a second. Once again: you're awesome

Edit: This was btw what I meant last time when I said it was really slow. I only just figured that it actually could be a bug

[Thomas1970]

In the log you sent me, the message "No VBR header found, must count all the frames to determine length." shows what is the problem.
mp3 files that do not have a VBR header do not have length information, so unless the file is CBR (constant bit rate), which can't be sure, gmb needs to scan the whole file in order to know the length of the file.
This full scan is slow, I plan to add an option to only do the full scan of the current song, but in the meantime you'll have to wait until it's done, it's only done the first time the song is scanned, so that's not so bad.

I think I have a problem with this schan method.

I'm running GMB on Ubuntu on an old PC with little memory. My library currently contains more than 18000 songs and located on a NAS in my LAN, which is connected via NFS.
GMB really runs smooth in this environment!

This is my problem description:

After each program start, GMB looks for new and changed files. That's good, and the performance is acceptable.
But after the re-scan, the text in the progress window changes to:
"Prüfe Länge/Bitrate von 12767 Dateien ohne VBR-Header...", which means something like:
"Checking length/bitrate of 12767 files without VBR header..."
The progress indicator shows no progress, although on next launch the displayed number is decreased.
There is no "stop" button in this window (there was one during re-scanning).
You can close the window, but the operation seems to continue in background.

This check operation consumes 100% of my CPU, mainly because of iowait.
And because my PC is old, it becomes unusable in this state...

Three questions:
- What's the use of this frame counting check?
- Can you change this operation to not causing cpu iowait?
- Could you please update the window during the check operation such, that the progress indicator really indicates the progress and the number of files to check counts down and maybe add a "stop" button?

Thanks a lot!

[Quentin Sculo]

Three questions:
- What's the use of this frame counting check?

It's to determine the real length of some mp3 files, without this the length (and bitrate) is an estimation which is currently only based on the first audio frame if I remember correctly. The estimation should be all right for constant bitrate mp3 files, but without a VBR header, there's no way to tell if the file is constant bitrate or variable bitrate. Other programs might do an estimate on thefirst few frames, but it is sometimes very wrong.

- Can you change this operation to not causing cpu iowait?

no, the operation needs to read the whole file, which of course is not a good thing when the file are not local.

- Could you please update the window during the check operation such, that the progress indicator really indicates the progress and the number of files to check counts down and maybe add a "stop" button?

I think it updates, it's just that the progress is very slow. The displayed number is meant to be the total number of songs to scan since the start of the operation, and thus doesn't change.

I've changed the wording of this progress bar in v1.1.x, it should be a little clearer.
There is currently no stop button, because it wasn't meant to be stopped, because there was no way to restart it.
If you really want to disable this, you can replace this line in /usr/bin/gmusicbrowser (v1.0.x only) :

Code: [Select]

elsif ($m eq 'l') {push @LengthEstimated,$ID}by this line :

Code: [Select]

elsif ($m eq 'l') { $song[SONG_MISSINGSINCE]=undef } this will clear the list of songs that need a full scan on startup.

I plan to add a way to disable this in v1.1.x

[Thomas1970]

Three questions:
- What's the use of this frame counting check?

It's to determine the real length of some mp3 files, without this the length (and bitrate) is an estimation which is currently only based on the first audio frame if I remember correctly. The estimation should be all right for constant bitrate mp3 files, but without a VBR header, there's no way to tell if the file is constant bitrate or variable bitrate. Other programs might do an estimate on thefirst few frames, but it is sometimes very wrong.

OK, I think I understand.
But my feeling is that a VBR header missing from a VBR file is an error.
And I think you should not try to catch this error with a functionality that consumes a lot of load.
Maybe you could add a "VBR length scan" command to the context menu of files and folders to allow the user to trigger the function?
Also you could trigger the function for each played (and not previously scanned) file?

There is currently no stop button, because it wasn't meant to be stopped, because there was no way to restart it.
If you really want to disable this, you can replace this line in /usr/bin/gmusicbrowser (v1.0.x only) :

Code: [Select]

elsif ($m eq 'l') {push @LengthEstimated,$ID}by this line :

Code: [Select]

elsif ($m eq 'l') { $song[SONG_MISSINGSINCE]=undef } this will clear the list of songs that need a full scan on startup.

I plan to add a way to disable this in v1.1.x

Does this code change affect only the VBR length scan?
Or does it suppress the folder scan on startup, too?

Thanks, Thomas

[Quentin Sculo]

But my feeling is that a VBR header missing from a VBR file is an error.

They are valid mp3 files, VBR headers are an addition (at least 2 different VBR header formats exist).

And I think you should not try to catch this error with a functionality that consumes a lot of load.
Maybe you could add a "VBR length scan" command to the context menu of files and folders to allow the user to trigger the function?
Also you could trigger the function for each played (and not previously scanned) file?

Yes, I'll add an option to automatically do that scan of not, and maybe do it on the first play.
To start the scan manually, it is already possible in v1.1.x by holding shift when selecting "re-read tags".

Does this code change affect only the VBR length scan?
Or does it suppress the folder scan on startup, too?

it only affect the VBR length scan

[Thomas1970]

Does this code change affect only the VBR length scan?
Or does it suppress the folder scan on startup, too?

it only affect the VBR length scan

Cool! I've changed that and now the app works fine! Again: Thanks!
And again: I'm impressed by the performance of the app on my old system!

[Thomas1970]

And I think you should not try to catch this error with a functionality that consumes a lot of load.
Maybe you could add a "VBR length scan" command to the context menu of files and folders to allow the user to trigger the function?
Also you could trigger the function for each played (and not previously scanned) file?

Yes, I'll add an option to automatically do that scan of not, and maybe do it on the first play.
To start the scan manually, it is already possible in v1.1.x by holding shift when selecting "re-read tags".

Great! I'm waiting for this change to come through via Ubuntu software updates.

[ochosi]

@Thomas1970: you can also try our ppa: https://launchpad.net/~shimmerproject/+archive/ppa

[bananas]

Hi, i have this problem, adding music to library take AGES and it is because of the VBR songs, i have never have problems with this songs using other players, i ve been trying all media players i have like 30 000 songs, how can i solve this problem? is there anyway to set to not calculate the lenght?

Im using gmb 1.1.7.

Thanks in advance.