Making gmusicbrowser network-aware

[bardo]

I have moved most of my music to a networked server and I'm currently looking for a solution to use gmb from my laptop. I already have an NFS share, but it's not always mounted, so its presence is unreliable. I'm not asking for mpd or mt-daapd support, since I know it would mean an incredible amount of work to fit it in the current structure.

However, it would be nice to able to be able to mark the directories as "networked", so that gmb would show the files inside them only if they are present. Ideally, when the user adds a directory s/he checks a "networked" mark, which enables a drop-down menu containing the system's mountpoints. The user then selects the appropriate one, which is then checked for mounting. Autodetection should also be possible, even though it's not completely trivial.

This however arises the problem of separating the local db from the "mobile" ones... filtering could be done on just a single db, but it probably would come at a performance cost, I don't know...

[Quentin Sculo]

In v1.1.5, there is a "master filter" feature that should help. You just have to create a filter based on the folder name, then you can easily restrict the library to the local songs with just one click.

Separating the db into multiple parts is an interesting idea, but I don't think I will implement this in the near future.

[berarma]

The same happens for removable media, external hdd, etc., and in such cases I've hit another problem. Scanning for new/deleted songs when it's not mounted will erase all the songs there and then they have to be re-added the next time it's mounted. Detecting that the volume isn't mounted and ignoring those songs in the library when scanning would be great.

[Quentin Sculo]

There is no real way to determine if a absence of a file is because of an not-mounted media or not (a media can be mounted anywhere), and anyway, there has to be an easy way to remove songs from a media without having to mount this media (which may no longer exist).
Note that it only happens when checking for updated/deleted songs, not when scanning for new songs. Also songs not in the master filter are not checked, and thus not removed.

But it's true it's not very practical, I'm not sure what is the best way to deal with that.

[bardo]

It's true that a program cannot know if a file is deleted or its partition is simply unmounted, if it happens when it's not running. However, if we take fstab's contents as 'trusted', the workflow seems to be pretty straightforward:

1. gmb is started
2. check the directories listed in the library and understand if they are on a mounted media*
3. compare the directory list with the mounted filesystems list
4. apply the necessary filters

There are some points to take in consideration if things are done this way. First of all, there's no removable media support unless statically defined. This could be addressed with a "manual" addition of a removable media to the library, based on the ID of the drive.

Then, it has to be kept in mind that the approach as described here should work fine for most layouts, but there are some edge cases that are not covered. For example, library dir: /foo. Mountpoint: /foo/bar. However, this one should be easy too (the check on point 2. becomes: match libdir on mountdir OR match mountdir on libdir). There may be other cases I didn't think about.

Finally, while the program is running, just keep an inotify() on the involved medias to apply filters on-the-fly.


* This implies we suppose the user doesn't use mountpoints to "hide" other content. I think it's a sane assumption.

[berarma]

I've given it a spin in my head after reading bardo's post.

When adding directories to the library we should annotate the first mount point going up the directory tree. Mount points can be taken from /proc/mounts.

When searching for updated/deleted songs we should check if the mount point for the directory is mounted. When it's not we should ignore that directory.

Removing the songs when that mount point doesn't exists anymore would be possible just deleting that directory from the library.

Problem is, what happens when we add a directory containing mount points? We should break it down into several directories (or alternatively, ignore anything inside a mount point not being the directory added by the user). We could do it internally and show the internal mount points just when they're gone, with a warning to let the user remove them from the library. This is maybe the hardest part.

[bardo]

When a directory contains a mountpoint, as you say, it should be (internally) broken down to two different library entries. Making the search for songs non-fs-crossing (can be done easily with unix find, I don't know if perl supports it) should help keeping the code simple.

The only thing that is really needed to avoid a mess in the code is the database splitting. Damn, if I was comfortable enough with perl I'd give it a try.

[Quentin Sculo]

I've implemented a quick solution, it looks at defined mount points in /etc/fstab on check which ones are mounted in /etc/mtab. And then exclude songs on the not-mounted mount points from the library (if the option is checked)
there is 3 drawbacks currently :
1) it's only done at startup (or when you toggle the option), once I'll add support for inotify it can easily be done live
2) it ignores symlinks: if the mount path does not appear in the song's path because it uses a symlink, then it won't work. It is harder to fix, as it requires finding the real path of all songs, which imply checking each folder and sub-folder leading to a song for symlinks. Though I could add this "real path" as a song property, it might be hard to know when to update it. inotify would help, though you'd still need to verify them at startup. (I've done a quick test, for me it takes ~20s to determine the real path of the ~2200 folders of my library).
3) it won't work for mount points that are not in fstab

I haven't commited it yet, as I'm wondering if these drawbacks will make the option more confusing (because it doesn't work) than useful.

Any comments ?

[bardo]

Quentin,
first, let me tell you that you are amazing. In ten years of linux and open source, I've never seen a developer so committed to his work and his users. Thanks for your great work.

Now to the technical details. I don't think the drawbacks you point out are really relevant, and in fact working around them in an efficient way doesn't seem to be easy. I'm only curious about #3 (which could be particularly interesting for collections on removable media - nowadays I don't know almost anybody with a static mountpoint for a removable drive): why making it fstab only? Is there any particular reason for not relying on mtab only? Which, by the way, could simplify a lot of the work when inotify support is live.

About #2, I don't think 20s is a lot. You already have to scan the whole tree for songs at least once, which takes a lot more than this, and it could be done at the same time. But then again, I don't know if the network overhead is relevant here.

[Quentin Sculo]

why making it fstab only? Is there any particular reason for not relying on mtab only?

Well mtab only list the currently mounted filesystems, so you know nothing of the unmounted ones. So it woulod be much more complicated, and involve guesses. Such as memorizing what mount point have been used, and assuming they are still mount points. Or treating specific folders such as /mnt and /media differently.

the 20s in 2) is not much if it's done in the background, but is a lot if the user must wait for it at startup. Yes it could only be updated when scanning folders/checking for changes. It's just that after a change of symlinks, gmb will only be aware of it after a scan/check.

The current patch if anyone want to see : https://gist.github.com/795943
(it's very simple currently)