BETA 4.7 - 2 MEDIA SCAN SSSSSLLLLLOOOOOOWW

[pir8radio]

Hey, 4.6 used to scan my music in about a min or two.. 4.7 beta 1 and beta 2 take forever..... As in DAYS.... i get about 1 mp3 scanned a second...

I do not allow write permission to any user on the music drive... so i dont know if 4.7 is attempting to write something to that drive now? I will not allow anything to write to my music drive, too much crap gets messed up.

So anyone know why 4.7 is scanning the music folder so slow now? I have 112,000 MP3's running on win7 64 with 16gigs or ram, 1000 MB set in subsonic control panel (tried smaller ammount with same issue) MP3's are on an internal raid array. Dual Xenon quad core processors.... any help would be appriciated...

[BKKKPewsey]

Not sure why is that slow - on my server - with much lower specs it took 12 hours to index 110k files
Yep not exactly blindly quick but if I remember correctly the older versions (4.5 I think) took approx 5 hrs to create a new index. (clean install)
Fortunately re-indexing only takes about 40 mins so hopefully 4.7 will be similar.
(I have reverted back to 4.6)

[pir8radio]

Im talking about scanning for changes, not from scratch.. I used to have it scan automatically at midnight for changes but found with 4.7 that it was still running in the afternoon the following day. When i click the scan link in the settings it just starts scanning at about 1 song a second... weird......

[GJ51]

That is wierd. Granted I have a pretty powerful server, but i can scan my 50k+ library in about 10 mins on 4.7 b2.

EDIT: Just timed a refresh scan in about 6 mins.

[pir8radio]

yea i don't know.... my machine isn't stressed either... plenty of ram, plenty of CPU.... just slow... Well if anyone reads this and has any other suggestions let me know.. I don't really want to switch back to 4.6 my users like the personal play lists...

[GJ51]

I would think a clean install might help.

Also, I didn't touch the ram allocation as that is no longer an issue. Many of the functions that were dying in ram on previous version have been fixed by moving more functions into the database. The default 150k should be fine.

[lars mars]

same issue here, it is extremely slow scanning the media. nothing on my server config changed, but what used to take a couple hours with 4.6 is now a still ongoing process in the middle of day 4!!!! and still only about 3/4 of 600K files scanned. that's ridiculous!!!!
considering that i add stuff every other day - i might have to change the schedule to one week adding, one week scanning!!!!

there is no strain on memory, cpu or whatsoever. so i guess it is some kind of screw-up in the code.

[GJ51]

Have you looked at the data transfer speeds of the storage device?

Can you describe your sotrage setup? Local? NAS? OS involved, etc.?

[hakko]

From what I know:

4.6 "scanned" your library in the sense that it quickly traversed your directory tree (no files were actually read until listed in the main window).
4.7 "scans" your library in the sense that it reads meta-data for every single new file (yes, 600.000) and inserts that meta-data into a single file (the hsqldb database).

you can compare it with a search on your filesystem based on filename, vs a search for a string to be found inside a file. the latter is obviously magnitudes slower.

I'm not sure about hsqldb performance in general but I wouldn't be surprised if it turns out to be a bit of a bottleneck. Not sure if Sindre's database structure is very optimized either. If I was a programmer (hey wait! I am a programmer) I would look into optimizing the database structure and try to use batch inserts as much as possible, then force people to use Postgresql instead of hsqldb, then have a look at the performance of JAudioTagger used to read meta-data. But either way... 600.000 songs are gonna take a while. There's always a trade-off and personally, I'm in favor of getting an initial hit, rather than a sluggish system.

[BKKKPewsey]

4.6 "scanned" your library in the sense that it quickly traversed your directory tree (no files were actually read until listed in the main window).

Incorrect - the year and genre tags had to be read from the file tag data - and I believe the title, artist and album name data was also read from tags and used at "song" level.
If otherwise, a random playlist where only XXXX genre and YYYY year was specified wouldn't work
(or would take ages to create)

[lars mars]

Have you looked at the data transfer speeds of the storage device?

Can you describe your sotrage setup? Local? NAS? OS involved, etc.?

to answer that one first, nothing changed there either, 8 esata hdd local on ubuntu 12 headless, AMD Athlon II X4 640, 8gb ram.
that shouldn't be the prob, at all.

[lars mars]

From what I know:

4.6 "scanned" your library in the sense that it quickly traversed your directory tree (no files were actually read until listed in the main window).
4.7 "scans" your library in the sense that it reads meta-data for every single new file (yes, 600.000) and inserts that meta-data into a single file (the hsqldb database).

you can compare it with a search on your filesystem based on filename, vs a search for a string to be found inside a file. the latter is obviously magnitudes slower.

I'm not sure about hsqldb performance in general but I wouldn't be surprised if it turns out to be a bit of a bottleneck. Not sure if Sindre's database structure is very optimized either. If I was a programmer (hey wait! I am a programmer) I would look into optimizing the database structure and try to use batch inserts as much as possible, then force people to use Postgresql instead of hsqldb, then have a look at the performance of JAudioTagger used to read meta-data. But either way... 600.000 songs are gonna take a while. There's always a trade-off and personally, I'm in favor of getting an initial hit, rather than a sluggish system.

now, i am not a programmer , but this explanation makes sense to me. i am all for a fast db.
hell, i can live with it, if it is just the initial scan. but if the scanning speed stays the same at further re-scans i will throw a fit.
maybe some reverse engineering from kplaylist is in order for the db.

[hakko]

Incorrect - the year and genre tags had to be read from the file tag data - and I believe the title, artist and album name data was also read from tags and used at "song" level.
If otherwise, a random playlist where only XXXX genre and YYYY year was specified wouldn't work
(or would take ages to create)

Oh yes, you're right, I'm mixing it up with how caching was implemented. A 600.000 song library must have taken a while to index the first time on 4.6 as well (although probably faster than in 4.7, as it was only written to the subsonic14.index + to memory).

When a random playlist is created in 4.6, meta-data is re-read per file again though when the files are added to playlist. If I'm not wrong again.

[BKKKPewsey]

As a benchmark, as previously posted, mine took just under 12hrs for 110k files and that was with atom 230 processor.
So 2 days max I would expect with your setup (and that being very generous)
Nobody has yet posted how long it takes to re-scan eg normal re-index with 4.7
(perhaps they are still waiting )
A long initial scan to build the new database is somewhat expected but if that time is repeated on every scan
For 4.7 users it may be a good time to add next years Christmas albums into SS now!!!

Edit: bit of a cp problem here this is referring to lars mars post

[lars mars]

As a benchmark, as previously posted, mine took just under 12hrs for 110k files and that was with atom 230 processor.
So 2 days max I would expect with your setup (and that being very generous)
Nobody has yet posted how long it takes to re-scan eg normal re-index with 4.7
(perhaps they are still waiting )

yep well, i thought so to. actually i was more thinking in the 1/2 day range. don't really remember how long it took with 4.6 initially but re-scans took about 1hr.
i guess i will wait and see LOL.

any ideas about memory settings? would there be improvement by upping subsonic mem to maybe 512 or 1024/ or upping java mem? not that i had any out of/ low on errors so far, but....