Amazon hosted Subsonic install

[Shahid]

Hi,

I have Subsonic up and working great on my local, physical linux box ... but I'm thinking about starting up an Amazon EC2 instance with an S3 storage area for the music files themselves. That way I should be able to access my music with no worries about whether it's on or not.

Has anyone tried this before? Anything I should be on the lookout for?

Thanks for your help!

[brett]

Hi,

I have Subsonic up and working great on my local, physical linux box ... but I'm thinking about starting up an Amazon EC2 instance with an S3 storage area for the music files themselves. That way I should be able to access my music with no worries about whether it's on or not.

Has anyone tried this before? Anything I should be on the lookout for?

Thanks for your help!

getting hit with some kind of copyright violation?

[pastelero]

Hi!,

I was in the same situation. Just bought a nice MacBook Air (64GB storage), and I don't want to carry around with an external hard drive to hear my music (I constantly drop the hard drive it to the floor, and always fear for the worst ).

So I did what you where thinking, and everything (well, almost everything) ok. I now use Subsonic to hear my music collection through Amazon EC2/S3, so I don't have to worry about disk space.

I used the following to get it working:

* Transmit to put my music in Amazon S3 easily
* Alestic.com AMI for an Ubuntu 10.10 32 bit server in Amazon EC2. When you enter this EC2 instance via ssh, you have an option to quickly install Tomcat (don't forget to 'sudo apt-get update' previously so that the quick install of Tomcat works).
* S3FS (S3 File System: http://code.google.com/p/s3fs/wiki/FuseOverAmazon) for mounting the S3 bucket as a local directory in EC2. You have to download it and compile it (this went smoothly in my case; just had to install some dependencies previously).

There's some things to watch out for, that took me some time to figure out:

* It's important that the s3fs has read/write access:


Code: Select all

sudo /usr/bin/s3fs -o allow_other bucket_name /mnt/bucket_name -ouse_cache=/tmp


* S3FS doesn't read directories created through the Amazon S3 control panel, or through Transmit. When you copy folders to your S3 bucket (for example in bucket_name->music->Michael Jackson), you have to make a ' mkdir "Michael Jackson" ' in your EC2 in order to have s3fs realize that the directory exists. Only then will the s3fs will show the files you previously copied within the 'michael jackson' directory. I had to play a little bit to automate this process since making the 'mkdir' of directories and subdirectories manually would be endless if your music collection is big.
* Watch out for directories and subdirectories with accents (áéíóúñö...). Couldn't get them working with subsonic/s3fs, even after reading a forum that said how to solve it.
* For some reason, the main music folder in 'Settings->Music Folder' doesn't update your artist left column list when you copy new folders in your S3 (and do the manual local 'mkdir' so that files show). Not even saving changes in 'Settings->Music Folder', or updating the 'Settings->Search' search index. The two ways Subsonic finds out this new copied albums is by explicitly entering the specific new music folder in 'Settings->Music Folder' (for example /mnt/bucket_name/music/Michael Jackson), or restarting the subsonic server (sudo service subsonic restart).
* The first time you access an artist/album, it takes ages to show the tracks information (for me, some 12-20 seconds). The following times you enter the same album it just takes 2 seconds, but the first time is painful. I think it's got to do with the way s3fs works (in order to read the id3 tag of each file, I think it has to transfer the whole file from S3 to EC2).

Hope this works for you. For me it's been great. No more carrying around with an external hard disk, and having to worry for limited local disk space for music .

[Shahid]

Pastelero, you are a star - this is exactly the information I needed! Now, to give it crack on my own ...

Hope this thread will become useful for others too, and this use case becomes more popular. For people with large music collections this seems to be the best solution for hosting it.

[gcubed]

wow, i didnt even think about this solution. Thanks for the info, gotta dust off some of my rather limited linux knowledge

[steez]

How much does this cost with S3? Can you estimate how much bandwith goes ththrough amazon?
Let's say a 200GB library with 5gb down A month.
It's a rather nice solution and keeps a online backup just in case.

[HexCore]

Great thread, I'll also try to set up my subsonic on Amazon EC2/S3.

Anyone got some info on the cost ?

[HexCore]

Hi,

I also installed Subsonic on Amazon EC2 and using S3 to store my files.
It works great !
I plan to test it a while before transfering all my mp3s there.
I still have a problem with the directory not being seen, like pastelero, but I don't know how to automate the process of mkdir :/
If somebody can help

I'd also like to use two different buckets, I'll try that later ...

Anyway, it works pretty smoothly, I'll test it more and let you guys know...

Btw, I made a quick calculation for the price and for 50 Go of data, 5Go of transfert, it's less than 10$ per month

[3R3]

* S3FS doesn't read directories created through the Amazon S3 control panel, or through Transmit. When you copy folders to your S3 bucket (for example in bucket_name->music->Michael Jackson), you have to make a ' mkdir "Michael Jackson" ' in your EC2 in order to have s3fs realize that the directory exists. Only then will the s3fs will show the files you previously copied within the 'michael jackson' directory. I had to play a little bit to automate this process since making the 'mkdir' of directories and subdirectories manually would be endless if your music collection is big.
* Watch out for directories and subdirectories with accents (áéíóúñö...). Couldn't get them working with subsonic/s3fs, even after reading a forum that said how to solve it.
* For some reason, the main music folder in 'Settings->Music Folder' doesn't update your artist left column list when you copy new folders in your S3 (and do the manual local 'mkdir' so that files show). Not even saving changes in 'Settings->Music Folder', or updating the 'Settings->Search' search index. The two ways Subsonic finds out this new copied albums is by explicitly entering the specific new music folder in 'Settings->Music Folder' (for example /mnt/bucket_name/music/Michael Jackson), or restarting the subsonic server (sudo service subsonic restart).
* The first time you access an artist/album, it takes ages to show the tracks information (for me, some 12-20 seconds). The following times you enter the same album it just takes 2 seconds, but the first time is painful. I think it's got to do with the way s3fs works (in order to read the id3 tag of each file, I think it has to transfer the whole file from S3 to EC2).

So taking all these points into consideration, I would advise anyone against just switching over until he knows how to solve the problems. Automating transfer and mkdir calls, making umlauts and accents work and the rest makes using amazon hosting compared to setting up a small home server seem quite painful. The only difference seems to be that you give amazon the cash to pay for utility bill and maintenance, but with this amount of hacking you could just as well do it yourself and have a better service quality (ok, apart from upload speeds on common dsl lines. or for rural areas). So whats the point?

After thinking twice, I admit that its a nice passtime for hackers in rural areas with bad cable at home
No bad feelings here, but what I read about all this S3FS limitations/speed and the umlauts makes me setting up a debian box and fiddling with sambas umlaut handling and codepages look not so complicated after all.

But then again, maybe I misunderstood the motivation. If this is about fast access to your files worldwide and a little streaming and music on the side, it's all pretty legitimate to go through the troubles.

[HexCore]

I don't have a spare computer at home, I don't want to use my main computer for that and I don't want to have to deal with the hardware.
This solution may have some downsides, they don't seem impossible to resolve.
I thought of using a normal dedicated server, but the cost would be much higher due to the large size of my music library.

Plus, it's fun to hack ...

[jaquense]

How many users do you plan on having? I run my subsonic server off of my linux box, which functions as my main desktop computer. I occasionally run into performance issues when a bunch of people are on and i'm working on the computer, but nothing to the point of wanting to pay for hosting. Hell I also host 3 web sites, a vpn, and a plethora of other web services on my 5 year old desktop.

I wouldn't be to afraid of letting SS run on that main comp.

[HexCore]

Well, thanks for trying to convince me to quit Amazon, but I still like this solution (so far).
The cost of leaving the pc ON 24/7 at home for this is more expensive than the amazon solution (7$/month for Amazon S3 vs ~12€/month for electricity http://www.codinghorror.com/blog/2005/1 ... pc-on.html ).
Again, I don't WANT to host subsonic at home.
So, the cheapest solution is still amazon S3 + EC2 ....

[HexCore]

just a quick feedback :
Subsonic can't read the directory if you don't make a mkdir to create them. You have to upload the mp3s with Linux, to do so, I mounted the s3 bucket on my personal computer at home and upload the mp3s using sabnzbd. The bucket is therefore mounted on the s2 instance and on my linux box.
I had once a problem with synchronization, but other than that, it works fine.
I don't have any directory or file with accent (=umlaut), so I have dealt with this problem yet.

The micro instance seems a bit slow though, the interface is not so fast. I'll try with a faster instance (small instance).

Anyway, I'm still happy with this solution and try to improve it.

If questions, don't hesitate to ask.

[Kirk]

Depending on the type of music you're hosting (if you're hosting copyrighted top title bands) you may need to watch out for the law...

In particular, there is some legislation in its early stages which could pose serious threats for many Subsonic users in the United States in the time to come...

-Kirk