Some albums do not appear after Scan [SOLVED]

[dkl]

Hello,

I have a quite frequent issue with Subsonic/Airsonic: from time to time, some albums are missing in the DB after scan.

Both Subsonic and Airsonic are installed on a RPI4 running raspbian. The files are flac/mp3 files, on a ext4 file system, on a USB HDD. File system is clean.

It looks to me that this is not a DB issue, as my Subsonic instance (6.1.6) is a attached to MariaDB, while Airsonic uses its own DB. But both encounter the same kind of issues.

I frequently add albums (every week more or less). A scan is planned weekly on both Subsonic and Airsonic. Everything works fine, except that, from time to time, one or several albums are missing.

More things: the way I build my music library is the following: each "new" album is stored in one single directory, and I always add two symlinks to this album, to have "faster access" to new albums.

Example:

The "physical" album:
03-Baroque/Archangelo Corelli/F_Bruggen-G_Leonhardt-A_Bylsma-Corelli_-_Sonata_Op.5_No.7-12_La_Follia

The 2 symlinks:
000_NEWS-2022/2022-02-14-F_Bruggen-G_Leonhardt-A_Bylsma-Corelli_-_Sonata_Op.5_No.7-12_La_Follia -> ./F_Bruggen-G_Leonhardt-A_Bylsma-Corelli_-_Sonata_Op.5_No.7-12_La_Follia
000_NEWS-2022/F_Bruggen-G_Leonhardt-A_Bylsma-Corelli_-_Sonata_Op.5_No.7-12_La_Follia -> ../03-Baroque/Archangelo Corelli/F_Bruggen-G_Leonhardt-A_Bylsma-Corelli_-_Sonata_Op.5_No.7-12_La_Follia

For this album, after scanning the DB, it appears that the 2 symlinks are present, but not the "physical" one.

I have exactly the same kind of issue on several other albums. On the file system, everything seems to be fine, but not in Subsonic/Airsonic.

I found two ways to solve this:

- delete and rebuild the complete DB (which is the "nuclear" solution)

- copy the "physical" album (cp -r <old-album> <new-album>, and just update the DB. This generally works...

Using the second solution copies all the files of the old album to the new directory (I generally just change a bit the name of the directory), and gives them "today's date". Generally, this works. More over, after having done this, I usually see the two albums in Subsonic...

For me, the issue comes from the way Subsonic/Airsonic detects new files on the file system during scan. For an obscure reason, this operation sometimes "misses" some files/directories.

Does anybody know how the check is made in Subsonic to "detect" new albums?

Is there something to be checked at the "album" level (the flac files? the album directory?), or even at the parent directory level?

Any hint to understand this issue?

Denis

[dkl]

Update:

I have duplicated the album (remove the "dot" in "Op.5":

Code: Select all

03-Baroque/Archangelo Corelli/F_Bruggen-G_Leonhardt-A_Bylsma-Corelli_-_Sonata_Op5_No.7-12_La_Follia


So I have now 2 duplicated albums:

Code: Select all

drwxrwxr-x 2 denis denis 4096 févr. 14 15:58  F_Bruggen-G_Leonhardt-A_Bylsma-Corelli_-_Sonata_Op.5_No.7-12_La_Follia
drwxrwxr-x 2 denis denis 4096 févr. 22 11:30  F_Bruggen-G_Leonhardt-A_Bylsma-Corelli_-_Sonata_Op5_No.7-12_La_Follia


Then updated DB, both album appear now....

Then, I have deleted the "new" one, and updated again: now I can still see the old one (which was not the case before creating the "duplicate"), the "new" one is deleted also in the DB.

So, it seems to me that the problem does not really come from the album itself, but maybe from the parent directory? Or is it a "date" issue?

Next time I have such issue, I'll try to add a "dummy" album (i.e not a copy of the missing one), and see how it goes. And also I'll check the date of the parent directory, which I did not do today.

Any help appreciated.

Denis

[dkl]

Hi everybody,

Issue solved. The problem comes from the file system: the "mtime" of the parent directory of the album is sometimes not updated (don't know why for the moment), so I guess that the scan does not go into this directory, and the album is just not scanned. On the contrary, the folder that holds the symlinks is up to date, to I can see the symlinks, but not the "physical" album.

Manually performing a "touch" on the parent directory solves the issue.

I have to understand now why this date is not updated when I enter the album, but this has nothing to do with subsonic. Sometimes it is, but sometimes not.... This is only file manipulation in Linux, should be fairly simple to check this.

Denis

[acroyear]

There are some funny things about folders and mtimes for linux (perhaps worse when the source drive is being served via the network as an SMB share from a windows box?).

I find that if I add a file to a folder, all the previous files 'disappear' from Subsonic's DB. The files are still there, but totally inaccessible. The only way that seems to fix it is to rename the folder to something else, scan, clean DB (losing any playlist info), then rename it back, scan, clean DB, then try to reconstruct what playlists got affected. :/

(nope. gone is gone - many of the playlists say they have 100 songs or something, but then have 0 or 28 or whatever anyways)

[dkl]

Yes, that's a bit weird....

The way I do here is that I add new albums, mostly from CD's, or streaming, on a linux computer (ubuntu 14.04). Files are local, copied manually to another local directory (local repository), and then "pushed" on the server (rpi4 / NFS) using rsync.

The problem appears at the beginning of this "chain", i.e when I copy the directory that holds the album into another directroy, both local to my PC.

I used to do this using Gnome, so using the mouse + copy paste... But it finally appears that a LOT of directories copied this way don't update the mtime of the parent directory! Some do, but not all of them. I don't know why...
Doing the same with "cp" or "mv" is fine, so I do it this way now, and everything seems to be fine now.

The way I solved the issue, if this can help anybody:
1) search in the DB the recent albums. For example, all album that I "pushed" since the beginning of this year. This is done through a SQL request (I have installed MariaDB with subsonic). I put the result into a file...

Code: Select all

SELECT path FROM media_file WHERE path NOT LIKE '%flac' AND created LIKE '2022%' into outfile '/tmp/albums_2022.lst';


2) Do the same thing on the file system, using "find"

Code: Select all

find -L /MNT/Music/Music/flac -type d -newermt "2022-01-01" > /tmp/find_albums.lst


3) sort both files, and then perform a diff between the sorted files.

This gives me the list of directories that are on the file system but not in the DB.

For each of this missing directories, perform a touch on it, and also on its parent.

4) Rescan the DB, and recheck

You're done.

You can also do this on the entire DB if nedded, of course.

Antother way to solve the issue (which I used before I discovered the root cause):
1) Erase the DB... For this, I use a dummy directory, holding only one album, and I select it in the WebUI of subsonic, deselecting the "true" root directory. Then Clean up DB + Scan
2) Rebuild the full DB: select again the "true" root dir and scan it again

Apparently, this rebuilds everything, and the time/date of the folders has no influence on this process. But this can be quite long...
It's only when I add albums, after a "full scan", that this date problem appears.

Well, I'm happy that this issue is solved now. For a long while, I thought it was a DB issue, but it's not.

Denis