Search index process still running but stopped indexing

[mhendu]

I've been using Subsonic for a few weeks now and have had a recurring problem when the program tries to build the search index. The log file shows the index processes up to 59000 entries (roughly 1/3 of my collection) but still continues to take up as much as 40% of my CPU time even though it appears not to be adding any more entries. If I quit out of the process and restart Subsonic I can confirm that it has indexed up to about the folders starting with the letter 'G.' Anything after that is not indexed, no matter how long I leave the process running. I've seen other threads about the indexing so I increased the memory allocated to Subsonic to 256 MB but that has had no effect. I'd really love to get the search index completed - any thoughts?

Thanks!

[3R3]

any error messages in the log?

[mhendu]

No error messages in the logs. I've updated to the latest beta but this hasn't solved my problem. The search index now stops updating after indexing 61000 files - any idea why that might be the case??

[sindre_mehus]

Since you have such a big music collection I think it's a memory issue. Please make sure that you have increased the Java heap size. To see how much memory Subsonic is actually using, click "About" in the web interface.

[mhendu]

Perfect, thanks! It indexed 179000 entries, which is about 10% more entries than songs that are actually on disk, but my whole collection was indexed. Thanks for the tip!

[sindre_mehus]

Glad that it worked

"Entries" in this context includes directories, so that explains the extra 10%, I guess.

Cheers

[wdpaynter]

Just wondering how to increase Java heap size. My collection is around 150,000 songs, but the interface says that their are only around 70,000. I increased the memory limit in subsonic, but it sounds like I may be experiencing the same thing as mhendu. I just don't want to screw anything up.

Is it possible that the count on subsonic is wrong?

Thanks
BP

[sindre_mehus]

"Java heap size" is identical to "memory limit in subsonic".

To see if you have enough memory, you can start a manual search index update (Settings > Search), and follow the progress in the log (c:/subsonic/subsonic.log)

Hope this helps

[wdpaynter]

Sindre - thanks for the reply

I did as you suggested the log shows that it sacnned 176,000 entries, but still shows the same amount of songs (70,743) on the "stats" area in the left navigation pane.

Here is the end of my log if this helps. I am getting errors in regards to the header of certain files. (There are more of these errors further back in the log as well.) PLEASE HELP! Thanks

[2010-04-27 11:38:00,581] INFO SearchService - Created search index with 114000 entries.
[2010-04-27 11:38:15,880] WARN EntaggedParser - Error when parsing tags in \\William-office\l\Rock, Metal, & Punk\Motorhead\What's Words Worth\04 - Motorhead - White Line Fever.mp3
entagged.audioformats.exceptions.CannotReadException: "\\William-office\l\Rock, Metal, & Punk\Motorhead\What's Words Worth\04 - Motorhead - White Line Fever.mp3" :entagged.audioformats.exceptions.CannotReadException: Error while reading header(maybe file is corrupted, or missing first mpeg frame before xing header)
at entagged.audioformats.generic.AudioFileReader.read(Unknown Source)
at entagged.audioformats.AudioFileIO.readFile(Unknown Source)
at entagged.audioformats.AudioFileIO.read(Unknown Source)
at net.sourceforge.subsonic.domain.EntaggedParser.getRawMetaData(EntaggedParser.java:61)
at net.sourceforge.subsonic.domain.MetaDataParser.getMetaData(MetaDataParser.java:42)
at net.sourceforge.subsonic.domain.MusicFile.getMetaData(MusicFile.java:230)
at net.sourceforge.subsonic.service.SearchService$Line.forFile(SearchService.java:764)
at net.sourceforge.subsonic.service.SearchService$Scanner.visit(SearchService.java:830)
at net.sourceforge.subsonic.domain.MusicFile.accept(MusicFile.java:336)
at net.sourceforge.subsonic.domain.MusicFile.accept(MusicFile.java:343)
at net.sourceforge.subsonic.domain.MusicFile.accept(MusicFile.java:343)
at net.sourceforge.subsonic.domain.MusicFile.accept(MusicFile.java:343)
at net.sourceforge.subsonic.service.SearchService.doCreateIndex(SearchService.java:138)
at net.sourceforge.subsonic.service.SearchService.access$000(SearchService.java:60)
at net.sourceforge.subsonic.service.SearchService$1.run(SearchService.java:112)
Caused by: entagged.audioformats.exceptions.CannotReadException: Error while reading header(maybe file is corrupted, or missing first mpeg frame before xing header)
at entagged.audioformats.mp3.util.Mp3InfoReader.read(Unknown Source)
at entagged.audioformats.mp3.Mp3FileReader.getEncodingInfo(Unknown Source)
... 15 more

[2010-04-27 11:39:01,873] INFO SearchService - Created search index with 115000 entries.
[2010-04-27 11:40:01,771] INFO SearchService - Created search index with 116000 entries.
[2010-04-27 11:41:02,141] INFO SearchService - Created search index with 117000 entries.
[2010-04-27 11:42:02,260] INFO SearchService - Created search index with 118000 entries.
[2010-04-27 11:43:02,394] INFO SearchService - Created search index with 119000 entries.
[2010-04-27 11:44:01,427] INFO SearchService - Created search index with 120000 entries.
[2010-04-27 11:44:59,276] INFO SearchService - Created search index with 121000 entries.
[2010-04-27 11:45:59,123] INFO SearchService - Created search index with 122000 entries.
[2010-04-27 11:46:59,680] INFO SearchService - Created search index with 123000 entries.
[2010-04-27 11:47:58,806] INFO SearchService - Created search index with 124000 entries.
[2010-04-27 11:49:01,269] INFO SearchService - Created search index with 125000 entries.
[2010-04-27 11:50:04,021] INFO SearchService - Created search index with 126000 entries.
[2010-04-27 11:50:07,379] WARN EntaggedParser - Error when parsing tags in \\William-office\l\Rock, Metal, & Punk\Radiohead\Pablo Honey\12 - Radiohead - Blow Out.mp3
entagged.audioformats.exceptions.CannotReadException: "\\William-office\l\Rock, Metal, & Punk\Radiohead\Pablo Honey\12 - Radiohead - Blow Out.mp3" :entagged.audioformats.exceptions.CannotReadException: Error while reading header(maybe file is corrupted, or missing first mpeg frame before xing header)
at entagged.audioformats.generic.AudioFileReader.read(Unknown Source)
at entagged.audioformats.AudioFileIO.readFile(Unknown Source)
at entagged.audioformats.AudioFileIO.read(Unknown Source)
at net.sourceforge.subsonic.domain.EntaggedParser.getRawMetaData(EntaggedParser.java:61)
at net.sourceforge.subsonic.domain.MetaDataParser.getMetaData(MetaDataParser.java:42)
at net.sourceforge.subsonic.domain.MusicFile.getMetaData(MusicFile.java:230)
at net.sourceforge.subsonic.service.SearchService$Line.forFile(SearchService.java:764)
at net.sourceforge.subsonic.service.SearchService$Scanner.visit(SearchService.java:830)
at net.sourceforge.subsonic.domain.MusicFile.accept(MusicFile.java:336)
at net.sourceforge.subsonic.domain.MusicFile.accept(MusicFile.java:343)
at net.sourceforge.subsonic.domain.MusicFile.accept(MusicFile.java:343)
at net.sourceforge.subsonic.domain.MusicFile.accept(MusicFile.java:343)
at net.sourceforge.subsonic.service.SearchService.doCreateIndex(SearchService.java:138)
at net.sourceforge.subsonic.service.SearchService.access$000(SearchService.java:60)
at net.sourceforge.subsonic.service.SearchService$1.run(SearchService.java:112)
Caused by: entagged.audioformats.exceptions.CannotReadException: Error while reading header(maybe file is corrupted, or missing first mpeg frame before xing header)
at entagged.audioformats.mp3.util.Mp3InfoReader.read(Unknown Source)
at entagged.audioformats.mp3.Mp3FileReader.getEncodingInfo(Unknown Source)
... 15 more

[2010-04-27 11:51:15,041] INFO SearchService - Created search index with 127000 entries.
[2010-04-27 11:52:19,854] INFO SearchService - Created search index with 128000 entries.
[2010-04-27 11:53:22,259] INFO SearchService - Created search index with 129000 entries.
[2010-04-27 11:54:24,168] INFO SearchService - Created search index with 130000 entries.
[2010-04-27 11:55:25,818] INFO SearchService - Created search index with 131000 entries.
[2010-04-27 11:56:34,634] INFO SearchService - Created search index with 132000 entries.
[2010-04-27 11:57:41,261] INFO SearchService - Created search index with 133000 entries.
[2010-04-27 11:58:49,551] INFO SearchService - Created search index with 134000 entries.
[2010-04-27 11:59:55,746] INFO SearchService - Created search index with 135000 entries.
[2010-04-27 12:01:01,126] INFO SearchService - Created search index with 136000 entries.
[2010-04-27 12:02:07,583] INFO SearchService - Created search index with 137000 entries.
[2010-04-27 12:03:16,071] INFO SearchService - Created search index with 138000 entries.
[2010-04-27 12:04:20,035] INFO SearchService - Created search index with 139000 entries.
[2010-04-27 12:05:23,049] INFO SearchService - Created search index with 140000 entries.
[2010-04-27 12:06:35,382] INFO SearchService - Created search index with 141000 entries.
[2010-04-27 12:07:36,964] INFO SearchService - Created search index with 142000 entries.
[2010-04-27 12:08:02,632] DEBUG NetworkService - Starting PortForwardingTask
[2010-04-27 12:08:05,710] INFO NetworkService - Successfully forwarding public port 80 to local port 80.
[2010-04-27 12:08:05,710] DEBUG NetworkService - Completed PortForwardingTask
[2010-04-27 12:08:40,464] INFO SearchService - Created search index with 143000 entries.
[2010-04-27 12:09:43,850] INFO SearchService - Created search index with 144000 entries.
[2010-04-27 12:10:47,685] INFO SearchService - Created search index with 145000 entries.
[2010-04-27 12:11:50,514] INFO SearchService - Created search index with 146000 entries.
[2010-04-27 12:12:54,859] INFO SearchService - Created search index with 147000 entries.
[2010-04-27 12:13:56,335] INFO SearchService - Created search index with 148000 entries.
[2010-04-27 12:14:59,176] INFO SearchService - Created search index with 149000 entries.
[2010-04-27 12:16:00,348] INFO SearchService - Created search index with 150000 entries.
[2010-04-27 12:17:00,715] INFO SearchService - Created search index with 151000 entries.
[2010-04-27 12:18:02,962] INFO SearchService - Created search index with 152000 entries.
[2010-04-27 12:19:03,801] INFO SearchService - Created search index with 153000 entries.
[2010-04-27 12:20:03,708] INFO SearchService - Created search index with 154000 entries.
[2010-04-27 12:20:16,098] INFO SearchService - Created search index with 155000 entries.
[2010-04-27 12:20:28,898] INFO SearchService - Created search index with 156000 entries.
[2010-04-27 12:20:41,403] INFO SearchService - Created search index with 157000 entries.
[2010-04-27 12:20:55,394] INFO SearchService - Created search index with 158000 entries.
[2010-04-27 12:21:08,429] INFO SearchService - Created search index with 159000 entries.
[2010-04-27 12:21:20,928] INFO SearchService - Created search index with 160000 entries.
[2010-04-27 12:21:34,125] INFO SearchService - Created search index with 161000 entries.
[2010-04-27 12:21:46,865] INFO SearchService - Created search index with 162000 entries.
[2010-04-27 12:22:00,467] INFO SearchService - Created search index with 163000 entries.
[2010-04-27 12:22:12,836] INFO SearchService - Created search index with 164000 entries.
[2010-04-27 12:22:25,840] INFO SearchService - Created search index with 165000 entries.
[2010-04-27 12:22:38,948] INFO SearchService - Created search index with 166000 entries.
[2010-04-27 12:23:05,656] INFO SearchService - Created search index with 167000 entries.
[2010-04-27 12:23:34,368] INFO SearchService - Created search index with 168000 entries.
[2010-04-27 12:23:57,729] INFO SearchService - Created search index with 169000 entries.
[2010-04-27 12:24:28,499] INFO SearchService - Created search index with 170000 entries.
[2010-04-27 12:24:41,176] INFO SearchService - Created search index with 171000 entries.
[2010-04-27 12:24:53,594] INFO SearchService - Created search index with 172000 entries.
[2010-04-27 12:25:06,347] INFO SearchService - Created search index with 173000 entries.
[2010-04-27 12:25:19,919] INFO SearchService - Created search index with 174000 entries.
[2010-04-27 12:25:32,992] INFO SearchService - Created search index with 175000 entries.
[2010-04-27 12:25:47,039] INFO SearchService - Created search index with 176000 entries.
[2010-04-27 13:07:54,478] DEBUG NetworkService - Starting URLRedirectionTask
[2010-04-27 13:07:54,808] DEBUG NetworkService - Completed URLRedirectionTask
[2010-04-27 13:08:05,664] DEBUG NetworkService - Starting PortForwardingTask
[2010-04-27 13:08:08,741] INFO NetworkService - Successfully forwarding public port 80 to local port 80.
[2010-04-27 13:08:08,741] DEBUG NetworkService - Completed PortForwardingTask
[2010-04-27 14:08:08,740] DEBUG NetworkService - Starting PortForwardingTask
[2010-04-27 14:08:11,788] INFO NetworkService - Successfully forwarding public port 80 to local port 80.
[2010-04-27 14:08:11,788] DEBUG NetworkService - Completed PortForwardingTask

[sindre_mehus]

I have two suggestions:

1. Look for anything suspicious in the log files c:/program files/subsonic/log. Look in particular for "OutOfMemoryError".

2. Increase the memory limit, restart Subsonic, and run the search index again. See if it gets further.

[wdpaynter]

Well immediately after I posted this, the tallies updated, but still do not match up. I realize they may not be exact, but should be closer than this:

According to Subsonic I have the following:
4,758 artists
11,903 albums
160,724 songs
924.46 GB (~ 14,361 hours)

According to MediaMonkey (what I use locally to play music):
4,824 Artists
12,950 Albums
162,209 Songs
936 GB

Should I up the memory limit and search again? It seems to me that if I have 4,824 + 12,950 + 162,209 = I should have 179,983 or 180,000 entries searched/created in my log

Also, why am I getting the header errors?

Please let me know if you follow, or what you think I should do. I appreciate anyone's help!

Thanks
BP

[wdpaynter]

Damn sindre, quick reply!

I'll do so and report back.
Thanks

[wdpaynter]

sindre

I searched in the Notepad file (log) for "OutOfMemoryError", but it was not present. Below are the only (3) errors I had, do you think that rescanning will do anything considering there were no memory errors?? - Sorry to be such a pain in the butt - I did donate already and will certainly contribute again if i can get this working the way I want

Thanks!!

[2010-04-27 10:20:45,168] WARN EntaggedParser - Error when parsing tags in \\William-Office\l\All Other Genres\Rap, Hip-Hop, & R&B\Compilation\Eastern Conference All Stars Vol. 4\01 - High & Mighty - Ice Breaker (Intro).mp3
entagged.audioformats.exceptions.CannotReadException: "\\William-Office\l\All Other Genres\Rap, Hip-Hop, & R&B\Compilation\Eastern Conference All Stars Vol. 4\01 - High & Mighty - Ice Breaker (Intro).mp3" :entagged.audioformats.exceptions.CannotReadException: Error while reading header(maybe file is corrupted, or missing first mpeg frame before xing header)

[2010-04-27 11:38:15,880] WARN EntaggedParser - Error when parsing tags in \\William-office\l\Rock, Metal, & Punk\Motorhead\What's Words Worth\04 - Motorhead - White Line Fever.mp3
entagged.audioformats.exceptions.CannotReadException: "\\William-office\l\Rock, Metal, & Punk\Motorhead\What's Words Worth\04 - Motorhead - White Line Fever.mp3" :entagged.audioformats.exceptions.CannotReadException: Error while reading header(maybe file is corrupted, or missing first mpeg frame before xing header)

[2010-04-27 11:50:07,379] WARN EntaggedParser - Error when parsing tags in \\William-office\l\Rock, Metal, & Punk\Radiohead\Pablo Honey\12 - Radiohead - Blow Out.mp3
entagged.audioformats.exceptions.CannotReadException: "\\William-office\l\Rock, Metal, & Punk\Radiohead\Pablo Honey\12 - Radiohead - Blow Out.mp3" :entagged.audioformats.exceptions.CannotReadException: Error while reading header(maybe file is corrupted, or missing first mpeg frame before xing header)

[sindre_mehus]

If there are only three files with bad tags you could just fix them in any tag editor.

Forgot to say: The actual search index is stored in c:/subsonic/subsonic10.index. Look if something is missing towards the end of the file.

[wdpaynter]

1.) I use Mediamonkey to edit tags and they appear to be fine when I look at them inside of the program - is there anything in particular that may be causing this?

2.)I opened the "subsonic10.index in Notepad - how can I tell if something is "missing towards the end" - there are 176,659 lines in the file

Thanks
BP