Indexing, searching, and browsing

[googli]

Nice application!

I have been the main developer of a system with similar targets for a couple of years but It isnt extendable enought to motivate major improvements anymore (http://coolmp3.sf.net). I'm thinking about maybe extending your application instead.

I would like your input on the following things:

* Could the infrastructure be improved so that the whole application works against a sql-server of some sort? Internal/external should be user selectable.
When I try to use this app with my MP3 collection (~60k mp3s) it is very sluggish (mostly due to (re)indexing i guess).
*Is browsing by folder doenst really seem like the optimal solution. I might want to be able to sort albums by year,artist,whatever at different times. Folder browsing can of course be supported as well.
*When browsing directory strucutures I get no feeling for what is an abum and what is just a directory. I used ini-files in all directories containg an album, not very good either. I am thinking in the lines of forcing an folder where there is an mp3 to be considered an "album" or something like that.

Btw. What tools do you use? I am not familiar wil maven and I had to fiddle around in eclipse to set things up, it didnt feel like the optimal way to work, I guess that I should just learn how to use maven...

[googli]

It is almost impossible to work with subsonic and a large quantity of files.. I get out of heap speace exceptions all the time even if I expand the jvm max mem and perm gen alot.. like 256/512 and run no other apps in one tomcat server. This is mostly related to the search function.
When I click on one folder where there is about 200gb mp3's it can take 10-30 seconds to just populate the first browsescreen if the system has been left for a while (and the directory tree is out of the cahe?)

[sindre_mehus]

Hi,

How large is your music collection in total? I have around 120GB / 34k songs, and don't see any significant performance problems (on a fairly recent PC and with default Tomcat memory settings).

If you have an enormous amount of files I can understand that searching causes problems, since the search index is kept in memory (which is something I hope to address later).

As for the 200GB folder you mention; that's a bit "above average" with respect to what Subsonic is designed for. Normally, a folder is supposed to contain all music for a single artist or album. Maybe you could try to configure the 200GB folder as a top-level music folder? I suspect what takes so long is a recursive search for cover arts. That's normally fast, but not if it has to look through a 200GB sub-structure of files and directories.

I am actually working on some performance tuning related to disk access and cover art, but it's too early to tell if it will help in your case.

Regards,
Sindre

[googli]

I'm running it on a fast enough computer.. Tomcat does not use up all cpu resources so it is probably just iterating thru all folders that takes IO time (if subsonic does this every time). I have all my mp3s ordered by the date when i sorted them into the archive like 2006\05\2006-05-03\ and then all albums in that directoy. This is spread over three partitions.

Total songs :: 60274
Total dirs (all) :: 5864
Total albums and single dirs :: 5790
Total album artist :: 2347
Total song artists :: 6159
Total album images :: 3712
...
Total file size :: 421.69 Gb

To be able to build a flexible UI I personally don't see any alternatives to using a proper database for many purposes. Do you have any objections to this? I know that it might make configuration a bit harder.

The reason for me being so boring with you is that I might be interested to help out in development. In that case I want to know if our goals are compatible first.

[sindre_mehus]

Ok, no wonder it takes forever to load the first page. Subsonic requires an artist/album/song disk layout. For other disk layouts (such as yours) I agree there is no alternative but to store the structure in a database.

The disadvantages of using a database are:
- No "real-time" view of the music collection
- More complex to implement

It's on my todo-list to analyze further what a rewrite would involve, but as a matter of fact, very few users have requested this functionality. Hence, it's not highly prioritized.

In conclusion, for the time being I'm willing to live with the requirement that users must have a particular disk layout, and instead focus on implementing new functionality (such as streaming to mobile phones). However, feel free to look around in the source code and get a feel for yourself for what needs to be done to support a database-driven solution.

(By the way, there is already an HSQL database in Subsonic.)

Cheers,
Sindre

[googli]

I wrote an application that moved around my my mp3s instead!

Now there is a new problem, I have troubles finding music some times.

What I currently feel is missing:
* search of any field, (bitrate:<200, performer:X)
* Combination searches: (artist:X and album:b)
* Order results by any field
* Album grouping, (not listing all tracks of an album if artist matches but show full albums first and then single tracks from other albums later)
* Search results paging
* Index some more tags

So I guess I'm back to this "need" for a database backend, at least for searching (even though browsing mostly can be defined as a specific search view)

Are there any plans for the next 2-5 months to improve search?

I am somewhat interested in writing something, I havent had time to do anything other than reading your source code for now.
I am thinking in the lines of investigating db4o for the index storage.

Off to do some stand alone experimentation...

[FreeMan]

Sindre-

This is a much more detailed equivalent to the request I made here http://forum.subsonic.org/forum/viewtopic.php?t=1049.

I really like Subsonic, and I think it's a great program. (I can listen through the corporate firewall! Yea!!!) I believe that it will be even better when it can accommodate user's library collections in whatever format they've been stored and can do searching/ordering based on ID3 tags.

I, for one, would rather see this as a top priority since I'm not interested in streaming music to my phone. Of course, I'm probably in the minority in that one...

[sindre_mehus]

I think most of the requirements listed above can be implemented with the current architecture. Subsonic already parses ID3 tags and saves them in an external file (c:/subsonic/subsonic9.index). The content of the file is kept in memory for better performance.

Sindre

[FreeMan]

...Subsonic already parses ID3 tags and saves them in an external file (c:/subsonic/subsonic9.index)...

Why is it, then, that albums are identified by directory structure?

I just lost my entire library (stupid mistake on my part - don't worry, it's all backed up - on the original CDs!), so I'm rebuilding in the directory structure that Subsonic likes. Before, though, I had lots of 'albums', each a single file, in one directory, and Subsonic didn't see them or list them as separate albums.

Thanks,
FreeMan

[sindre_mehus]

Why is it, then, that albums are identified by directory structure?

I guess the main reasons are:
o It's easier to implement.
o I think people, on average, have better file structure than tag quality.
o Changes to the disk are immediately reflected in the web interface, no rescan necessary.

Sindre

[FreeMan]

o It's easier to implement.

Fair enough. That's always a motivator on *FREE* software!

o I think people, on average, have better file structure than tag quality.

Guess I'm one of those detaily people who makes sure the tags are pretty accurate.

o Changes to the disk are immediately reflected in the web interface, no rescan necessary.

This makes sense, but why is there provision for scheduled and immediate re-scans?

[mw]

3,950 artists
4,461 albums
61,943 songs
381.85 GB (~ 5,931 hours)

All pages are parsed/shown in under one second on a 2GHz dual core CPU. I can provide full system specs if it's important. However, I am amazed at how well Subsonic has scaled with my growing collection. My tags and directory structure are very clean and uniform, however.

Searches are snappy, and streaming is flawless. Well done, Sindre!

[googli]

I still have quite long load-times on my various artists folder which have about 400 albums.


Back to the search stuff...

To avoid too much re-invention i chose to go with Apache solr as my newish search back end after spending some time experimenting. '

I think i'm gonna look into if this also can be adapted for search over multiple subsonic instances, solr could be very fitting for this.

I know there is another thread on multiple instances searching, so maybe I won't do that if someone else does it and it it possible to make it compatible with my feature.

I don't know when all of this will be finished, probably months away, I am extremely busy at work right now and I don't know spring at all.

[DandyDon]

3,080 artists
4,614 albums
64,901 songs
328.05 GB (~ 5,096 hours)

Running Vista x64 Ultimate on Athlon x64 dual core @ 2 GHz everything is blazing fast, and streaming is perfect, but I am very picky about my structure and tags. Thanks Sindre !

[pampelmusenmann]

Why is it, then, that albums are identified by directory structure?

o I think people, on average, have better file structure than tag quality.

Damn youre right!
You're right, this argument counts a lot, as at least 10% of the files have no id3-tag at all...anyway - would be great to be able to browse through 'Genres' or st. like that