MusicCabinet - artist radio, genre radio & related artists

[hakko]

Ultraviolet: I'm sorry to hear that. The only thing that worries me is whether the slowing down is related to my code (for example, holding too many artists names meant to be updated in memory), or if last.fm has some sort of "unfair usage" detection that identifies a lot of web service calls from your machine and delays the answers.

Obviously, there is some sort of upper limit given that last.fm allows 5 calls per second. If your library is humongous and you want it kept up-to-date, there is some point where your machine can ask last.fm for updates 24/7 and still not get as much as needed within that time frame.

But you've effectively made me aware that I haven't really written my code to handle this gracefully. Currently, it's just "get all artists names lacking last.fm metadata, put them into memory, iterate over all of them, ask last.fm for metadata, batch insert into database import table for every 1000 answers, when all is done, run the script that updates the real database tables from import table". This is fairly efficient for a library of my size but obviously not of yours.

I'll change this to keep less artist names in memory during the scanning, and to update the real database tables during the process. And also to calculate some upper limit for what's realistic to handle given the 5 calls per second limit, and then handle larger libraries differently than trying and failing.

I cannot give you a time frame of when this will happen. I'll finish a new feature that I'm halfway through and then I'll prioritize this, but I'll have significantly less spare time for the next weeks to come. So you can either wait for that, or try adding your music in batches (start off by having just one media folder containing 50.000 tracks, update search index, add another media folder with 50.000 more tracks, update search index, repeat until done).

[TheWarden]

Hey,

I haven't had a chance to try the debug version on Linux yet (tomorrow) but I did have a play around with MusicCabinet a bit on a windows install and I have some (personal) feedback for you. Firstly I think the Genre cloud and using the Last.fm tags for that is a great idea and seems to work really well. In the absence of mp3 tag browsing in Subsonic this seems like the best and simplest way to achieve this functionality. Here is a list a requests most of which are purely personal preference:

[*]In order to avoid having to create icons for every theme maybe it would be better to make the Genres and Radio buttons more like "shortcuts" and just have them as text links in the left hand frame? If you are planning on creating icons for all the themes then I'm fine with that too.
[*] Because Subsonic lacks genres it would be nice to be able to browse ALL the artists in your collection that match a certain genre instead of just showing the "top" 20 artists. It would have to be a list view like the left hand frame. So you could have a "Browse Genre" link along with Play radio/browse last.fm. Also can you make the number of top artists shown configurable (ie more than 20)?
[*] I think I prefer how albums were listed in the original subsonic with the album art to the right. The problem is that now you have to scroll to browse many albums - especially now with the artist bio (which I love btw). Also as a purely asthetic thing I don't like having the album art on the left and then having the album list indented such that the text doesn't line up with the playlist at the bottom. I just don't think it's the most efficient use of space and makes browsing albums a little slower than it used to be.
[*] I'm not sure exactly how last.fm's top tracks is worked out but it seems to include quite a lot of intros and filler tracks. Is it most played (that might explain the intros). Also for artist radio are you using the top tracks from the other similar artists or just randomly selecting the tracks? Again these seem to include a lot of tracks you would never hear on a real radio (tracks from mix comilations, intros, skits, crap songs!).

Like I said I think the Genre cloud and last.fm integration is a fantastic idea and well implemented but do we have to change the default album view so much? Could it at least be configurable? And my most important feature request is to include a genre browser for all artists in your collection. I hope Sindre takes note of this mod - keep up the good work!

[Alsaya]

Hello hakko,

is MusicCabinet compatible with the 4.7 beta?

Thanks

[hakko]

Thanks for the feedback!

I should probably make clear that I'm not really a front-end developer or a designer. I've added a few things to fit my personal needs (the new icons, the new album view) but I don't enjoy delving into that. So if you want new icons, or a better layout in general, I'm open for it but I'd need input. Either plain HTML (mine is horrible), new icon packs from someone who knows how to create that, or new code (fork+push at Github) that achieves it. Otherwise it tends to end up pretty far down on my list. But if you mod my code and I like it, I'll gladly put it into my code base.

The top 20 should definitely be configurable and most of all, it needs paging. I'll add that sooner or later (it works on the back-end, just needs GUI).

Also, it's a good idea to extend the left frame. You know how Subsonic adds a drop down menu for multiple media folders? I imagine adding a second drop down menu just like that, with all genres. That would limit the left frame to only show artists matching that. Maybe it's not exactly what you meant but thanks for that idea!

The tracks for the radio are chosen by random among the top 50 most played, fetched from last.fm. The higher the rank a track has, the more likely it is to be picked for the radio, plus that songs you've played a lot yourself and scrobbled to last.fm are favored. The same goes for related artists radio. That means intros, remixes and stuff might be chosen if they've been played a lot by other people. Or if you only have those tracks from an artist and nothing better, MusicCabinet might pick it just to get something. It would be good to allow the track picking algorithm to be more configurable as well, I'll do that some day too. So much code to write, so less time!

[hakko]

Hi Alsaya! I'm not sure what to answer. MusicCabinet is a full replacement (built on Subsonic 4.6) so the stuff Sindre has added to 4.7 will be gone if you start using MusicCabinet. I'm looking into grabbing what Sindre has done for 4.7 and incorporate it into MusicCabinet, but I'm not there yet. So right now you have to choose between the two of them.

[Alsaya]

Hello Hakko,

then I've to wait, because 4.6 has a problem with playing on mobile devices, it skips the songs after 1:30 and 2 mins and therefore I switched to the beta - with some new problems for me
Thanks for your answer

[hakko]

Minor update:

Every now and then, I check the stats at last.fm, summarizing listening habits over the pasts months. So, why not integrate that into Subsonic for easy access?



For this to work, download http://dilerium.se/musiccabinet/subsonic.war (0.5.41) just as usual.
Before using this new feature, go to Settings -> MusicCabinet and press "Update database" and then update the search index.

This works for every user who has entered their last.fm username in the personal settings.

[tzortst]

Just curious, why is it bad taking genres from last.fm, not from ID3 tags?

I believe that ID3 tags in mp3 files are most often wrong or lacking, and there's only room for one tag (even though all music players have their own ways to circumvent this, meaning it's impossible to actually use that value). Also, you'd have to tag your whole library yourself, more or less.

Using last.fm, you get automatic tags with a weighting factor, like Madonna is currently 100% pop and 48% dance. I think that adds for much more sophisticated mechanisms to generate playlists and find similar artists and so on.

Anyway, sorry to let you down! Hopefully, you'll find somebody else to do an ID3 tag version for you.

Reason being these are wrong half the time, it isn't 100% reliable, I guess some of us are very picky about our music collection. I tagged all my my music with genres, length process but well worth it. Guess other reason is all other music applications just use the ID3 Tags.

Again thanks for the mod, keep it up, need more people like you unfortunately this isn't for me

[TheWarden]

Thanks! I have a new subsonic.war with added logging (no new features) up here: http://dilerium.se/musiccabinet/subsonic-thewarden.war

Here's the log output:

Code: Select all

2012-05-21 10:25:05] DEBUG JdbcDatabaseAdministrationDao - Assume database is down.
[2012-05-21 10:25:05] DEBUG JdbcDatabaseAdministrationDao - Connect socket to hostlocalhost, port 5432.
[2012-05-21 10:25:05] DEBUG JdbcDatabaseAdministrationDao - Open socket output stream.
[2012-05-21 10:25:05] DEBUG JdbcDatabaseAdministrationDao - Print 0 0 0 42 0 3 0 0 u s e r 0 p o s t g r es 0 etc.
[2012-05-21 10:25:05] DEBUG JdbcDatabaseAdministrationDao - Open socket input stream.
[2012-05-21 10:25:05] DEBUG JdbcDatabaseAdministrationDao - Read one character.
[2012-05-21 10:25:05] DEBUG JdbcDatabaseAdministrationDao - Character returned was E.
[2012-05-21 10:25:05] DEBUG JdbcDatabaseAdministrationDao - Close socket, running = false

With regards to the change in album view I just thought that it would be nice to have the option of keeping the default Subsonic layout while keeping the new Genres. If I don't use the genre features at all and just browse using the left frame then I still get your new album layout. Or maybe you can separate out your changes into individual modules of code so that we can add the bits we like to our own servers? But I realise what a pain that would be for you. Anyway, I look forward to seeing how this mod progresses - a great start.

[hakko]

Ultraviolet: I made the decision not to re-write the import logic completely to support your 500.000 tracks library. Instead, I've set an upper limit for the search index update, so that only a subset of your artists will be picked the first time you update your index. The second time, another subset will be chosen, and over the course of a week or so, you should have it all (if you update once per night). Hopefully, this will work better and put less strain on memory usage and the last.fm web services!

TheWarden: That's super interesting.. reading the PostgreSQL source code, the "E" sent back indicates an error. I can't tell the reason for it, so to move forward, I've changed the detection mechanism slightly. This "E" now gets logged as a warning, but MusicCabinet will consider your PostgreSQL service as running and allow you to update the password for user "postgres". Given the occurrence of the "E", it's not unlikely that the password update will fail with a more detailed error message, though. But if it does, that should hopefully give some more insight into what's going wrong.

I've added both these changes and uploaded them to http://dilerium.se/musiccabinet/subsonic.war (0.5.42). Feedback is very welcome as usual! On errors, please check the file musiccabinet.log in your temp directory and the subsonic.log file.

Also, sorry, but re-writing the album view isn't top priority at the moment. The top list is
- "there shouldn't be bugs stopping people from using it" and then
- "I should implement stuff people wish for, that I'd like to see too" and then
- "there should be even more features I'd like to see, personally" and then
- "I should help people out by implementing feature requests, even though I won't use them myself".
The modularization would be fun (because it would be interesting) but I still think that adding new features is more prioritized, at the moment. This is a hobby project for me after all and I'm not asking for donations like Sindre. But it might change of course.

If you want to change something badly, just grab the code from https://github.com/hakko and mod it! It's very rewarding to create stuff and then have a music player that behaves just the way you like it.

[TheWarden]

Yes this change gets me a bit further. Now I get:

Code: Select all

com.github.hakko.musiccabinet.exception.ApplicationException: Could not create database!
<snip>
Caused by: org.springframework.dao.DataIntegrityViolationException: StatementCallback; SQL [create database "musiccabinet" with owner=postgres template=template0 encoding='UTF8' connection limit=-1;]; ERROR: encoding UTF8 does not match locale en_GB
  Detail: The chosen LC_CTYPE setting requires encoding LATIN1.; nested exception is org.postgresql.util.PSQLException: ERROR: encoding UTF8 does not match locale en_GB
  Detail: The chosen LC_CTYPE setting requires encoding LATIN1.
<snip>
Caused by: org.postgresql.util.PSQLException: ERROR: encoding UTF8 does not match locale en_GB
  Detail: The chosen LC_CTYPE setting requires encoding LATIN1.

Which is what a default out of the box Fedora Linux install is going to give.

[hakko]

Thanks for digging into it!

I haven't seen your error before but I think this is applicable http://www.battestini.net/blog/2011/11/ ... g-locales/

I guess en_GB.UTF-8 isn't part of the output from locale -a?

So.. how do you then add supported locales and character encodings in Fedora? You probably know more about that than me, as I can't lay my hands on such a machine right now. But it might be as simple as running the command "system-config-language". Also, it would be interesting to see the content of /etc/sysconfig/i18n. You'd want "en_GB.UTF-8" as one of your "SUPPORTED" entries.

Do you have any idea why UTF-8 isn't a supported character encoding for your box (if that's the case)? Are you running a super stripped-down install? I hadn't taken into account that it wouldn't be available everywhere. I thought the days of us-ascii and such were long gone...

[Ultraviolet]

So I am absolutely blown away by how awesome this mod is. I've already discovered all sorts of awesome music I've never heard (or heard of) before. I completely understand not wanting to tailor this to my edge-case library size. I'm running the latest war you've posted--we'll see if that does the trick. Frankly, I don't care if it takes a month or more to get everything examined. My library is large, but fairly static at this point so there shouldn't be an ongoing issue.

One question: what is "tag descriptions [not started]"? That has never (that I've noticed at least) started for me except for one time, right after dropping the latest war in, it went to "-1". However, scans have completed before (to where search is functional again), so it's possible something happened here when I didn't notice it.

Finally, a big thank you for making the update asynchronous. It makes the difference between me enjoying this for however many days/weeks/months it takes to analyze my library and not being able to use it at all.

[hakko]

Thanks for your kind words! It's good to hear the positive feedback too, as people tend to just report errors. : )

The "tag descriptions" is a service that fetches.. tag descriptions. After having imported tags for your artists (that's why it's not started until the other imports are done), MusicCabinet makes a profile of your music. Is it all disco? Then, only a textual description of "disco" needs to be fetched from last.fm. ("Disco is a genre of dance-oriented music that originated in African American, gay and Hispanic American communities in 1970s. [...]") Is it also a bit hard rock, bebop jazz, and piano? Then, descriptions for those tags are fetched as well. It's done at the very end of the import and pretty quick. -1 means no tag descriptions needed update. (Should be display as 0 or "no" instead, really.)

If you don't mind, I'd be interested to hear how it works one week from now too! I've tried to make the playlist generation as efficient as possible, but I still need to test it with more data to look for bottlenecks. And since your library is ten times as big as mine, it would provide a very good data set for looking for optimizations.

[Ultraviolet]

Thanks for the explanation. I definitely have artist descriptions populating postgres, so it must be completing quickly when I don't notice. Everything completes fairly quickly except for "artist top tracks". Everything but this and artist descriptions generally will complete in an hour or two. The "artist top tracks" then typically takes at least 12+ hours. I've roughly timed its progress at ~1.7/minute. No idea if this is typical, a bug, or just my IP being throttled by last.fm.

There are also some strange genres being returned. For instance, "L3a1e5da4n15l1" (no description for that one), but there are associated songs; a "Download" genre with a description of, "Personal sticky note of sorts for those with a poor understanding of the Last.fm tagging system." I don't know if this is the result of my id3 tags, the mod or last.fm data--I just thought I would point it out.