memory usage on large collection

[dorisane]

Does anyone have any tips for creating the search index for a large collection ? Currently it gets a fair way into the files and then seems to slow down more and more until the updates at every 1000 files added stop coming (also free ram goes to about 100mb about then). I've upped the java heap size to 1200 in the /usr/bin/subsonic file, but i dont think it's going to be enough.

Any tips on making subsonic work with indexing larger collections or lowering memory usage ?

[dorisane]

to chase this up, after a partial indexing now subsonic is just crashing on every login with:

GC overhead limit exceeded (java out of memory error).

it got to about 120,000 files on the best attempt before going down the pan.

Is there anything like subsonic that works well on large collections ?

[bushman4]

Um, subsonic seems to work fine for me for large collections. See my sig.

I have my java memory limit set at 400MB, and it uses every bit of it.

YMMV,

Glenn

[dorisane]

Wondering if its the 4gb ram on your machine that makes the difference. i'm running with 2 on that machine. With the java limit it shouldnt matter though i'd have thought. ;-/

I'll try setting java limit to 400.

[dorisane]

tried several java limits, and tried putting folders as 26 seperate media folders, still not working. Up until about 50000 tracks it works fine, beyond that it seems to say its indexed but the files arnt actually searchable, and beyond about 100000 it just totally packs in for me.

Did you do anything to tweak your setup, particularly in terms of the database ?

[GJ51]

I've got 65k songs + tons of video running on a 2GB WHS 2011 VM with the java memory set to 256mb.

No problems.

You might consider uninstalling all Java files and doing a clean install.

http://www.java.com/en/download/manual.jsp

EDIT: If you have all you trarcks in a small number of folders, that's probably what's causing it.

SS likes the collection in folders:

X:\Music\Artist\Album\Tracks

[lars mars]

hi, i have kinda the same prob. on a linux server amd x2 5000+ with 6gb ram. have a music collection of about 500k+ change files. memory for subsonic set to 1024 (under etc/default/subsonic).
sometimes it starts up just fine (well, kind of - it takes forever initially to load) and all is good finally. once at that point it is running with minor delays fine. other times i end up with halfloaded gui and the above mentioned 'GC overhead limit exceeded' and thats it, period. kinda annoying!
are there any tweaks to get subsonic stable and reliable running? i know its a rather large collection, but considering that it is supposed to run stable up to 100.000 files with about 120mb memory - i dont see the prob for it to keep crashing with 1024 for 520.000 files.
while i like subsonic in general, i think i stick with kplaylist for now on my other server, which runs rocksolid.
i am open for any suggestions to get subsonic running reliable and stable, without the maybe or maybe not today factor.

thx in advance, lsdeep

[lars mars]

I've got 65k songs + tons of video running on a 2GB WHS 2011 VM with the java memory set to 256mb.

No problems.

You might consider uninstalling all Java files and doing a clean install.

http://www.java.com/en/download/manual.jsp

EDIT: If you have all you trarcks in a small number of folders, that's probably what's causing it.

SS likes the collection in folders:

X:\Music\Artist\Album\Tracks

just to add that, alll my music is in a folder structure like 'letter of alphabet/ band/album/' - dont know if you can get more anal....

[BKKKPewsey]

Have a try at reducing the memory allocation to say 512mb.
I have noticed a few posts where the symptoms appear to be a low Java memory allocation but they have said that their memory allocation was 1 Gb or higher.
I just wonder if there maybe a maximum (undocumented?) limit which if you exceed results in it going back to a default
Might be worth a shot

[lars mars]

Have a try at reducing the memory allocation to say 512mb.
I have noticed a few posts where the symptoms appear to be a low Java memory allocation but they have said that their memory allocation was 1 Gb or higher.
I just wonder if there maybe a maximum (undocumented?) limit which if you exceed results in it going back to a default
Might be worth a shot

sounds kinda weird. but end of the day - what isnt with computers & os this days! i will give it a shot... can't hurt, can it!? i will post the results later on.

thx lsdeep

[lars mars]

well, it doesnt really make sense, but lowering memory to 512, as suggested improved the situation alot!? at the moment i play around with minor increases, since it runs stable but very slow - getting slowly towards 700 now. anybody else experienced this??? it just seems weird to me - less memory improves stability!!!???
happy to hear about your exp.
thx BKKKPewsey for the tip! while it seems to work sofar, i still dont get my head around it why!

[BKKKPewsey]

Its a computer if you you try to reason why it works (or not) you will be end up on the funny farm

[fventura03]

How do you change the memory allocation? I have my subsonic tray service memory limit at 512mb. (right now it says 53mb).

Checking how much subsonic-service.exe it says 154mb. Everything runs good but navigating through pages seems to be slow.

268 artists
836 albums
11,944 songs
7207.97 GB (~ 111,971 hours)

^all video.

[lars mars]

Its a computer if you you try to reason why it works (or not) you will be end up on the funny farm

thx 4 reenforcing this concept!!! ROFL!!!! i can't agree more... just returned from there

[lars mars]

How do you change the memory allocation? I have my subsonic tray service memory limit at 512mb. (right now it says 53mb).

Checking how much subsonic-service.exe it says 154mb. Everything runs good but navigating through pages seems to be slow.

268 artists
836 albums
11,944 songs
7207.97 GB (~ 111,971 hours)

^all video.

well, under linux it's in etc/default/subsonic. considering your mentioned .exe ending, sounds like win. no friggin clue - don't use that for several years anymore.
but i'am sure sb here can help you there.

gretz, lsdeep