Out of memory failures since 6.0 upgrade

[ajkessel]

I was using Subsonic 6.0b2 until recently with no problems, on a Debian GNU/Linux system. Since upgrading to the recent full 6.0 release, the Subsonic server has been crashing daily. I also get errors like below in my subsonic_sh.log, as well as the image below on my iSub client. I've boosted the /etc/default/subsonic memory setting up several times; now it's at 1500M (was something like 512M before) and still crashing/failing/becoming nonresponsive regularly and needing to be reset. Any suggestions? Is this a 6.0 bug that wasn't present in the last beta?

Code: Select all

Exception in thread "pool-4-thread-1" org.springframework.dao.TransientDataAccessResourceException: PreparedStatementCallback; SQL [select distinct artist from media_file where type=? and artist is not null]; out of memory; nested exception is java.sql.SQLException: out of memory
        at org.springframework.jdbc.support.SQLStateSQLExceptionTranslator.doTranslate(SQLStateSQLExceptionTranslator.java:106)
        at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:72)
        at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:80)
        at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:80)
        at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:607)
        at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:641)
        at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:670)
        at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:678)
        at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:710)
        at org.springframework.jdbc.core.JdbcTemplate.queryForList(JdbcTemplate.java:768)
        at net.sourceforge.subsonic.dao.AbstractDao.queryForStrings(AbstractDao.java:111)
        at net.sourceforge.subsonic.dao.MediaFileDao.getArtistNames(MediaFileDao.java:571)
        at net.sourceforge.subsonic.service.LastFmService$ArtistBioLoader.run(LastFmService.java:520)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)
Caused by: java.sql.SQLException: out of memory
        at org.hsqldb.jdbc.Util.throwError(Unknown Source)
        at org.hsqldb.jdbc.jdbcPreparedStatement.executeQuery(Unknown Source)
        at org.springframework.jdbc.core.JdbcTemplate$1.doInPreparedStatement(JdbcTemplate.java:648)
        at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:591)
        ... 11 more


Finally, here's the error (I believe the same one) via the web interface:

Code: Select all

Subsonic encountered an internal error. You can report this error in the Subsonic Forum. Please include the information below.

Exception   org.springframework.dao.TransientDataAccessResourceException
Message   PreparedStatementCallback; SQL [select id, path, folder, type, format, title, album, artist, album_artist, disc_number, track_number, year, genre, bit_rate, variable_bit_rate, duration_seconds, file_size, width, height, cover_art_path, parent_path, play_count, last_played, comment, created, changed, last_scanned, children_last_updated, present, version from media_file where type = ? and folder in (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) and present order by created desc limit ? offset ?]; out of memory; nested exception is java.sql.SQLException: out of memory
Java version   Oracle Corporation 1.7.0_05
Operating system   Linux 3.16.0-4-686-pae
Server   jetty-6.1.x
Memory   Used 39 of 956 MB
Stack trace   org.springframework.dao.TransientDataAccessResourceException: PreparedStatementCallback; SQL [select id, path, folder, type, format, title, album, artist, album_artist, disc_number, track_number, year, genre, bit_rate, variable_bit_rate, duration_seconds, file_size, width, height, cover_art_path, parent_path, play_count, last_played, comment, created, changed, last_scanned, children_last_updated, present, version from media_file where type = ? and folder in (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) and present order by created desc limit ? offset ?]; out of memory; nested exception is java.sql.SQLException: out of memory
   at org.springframework.jdbc.support.SQLStateSQLExceptionTranslator.doTranslate(SQLStateSQLExceptionTranslator.java:106)
   at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:72)
   at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:80)
   at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:80)
   at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:607)
   at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:641)
   at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:666)
   at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:698)
   at org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate.query(NamedParameterJdbcTemplate.java:133)
   at org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate.query(NamedParameterJdbcTemplate.java:137)
   at net.sourceforge.subsonic.dao.AbstractDao.namedQuery(AbstractDao.java:104)
   at net.sourceforge.subsonic.dao.MediaFileDao.getNewestAlbums(MediaFileDao.java:280)
   at net.sourceforge.subsonic.service.MediaFileService.getNewestAlbums(MediaFileService.java:284)
   at net.sourceforge.subsonic.controller.HomeController.getNewest(HomeController.java:185)
   at net.sourceforge.subsonic.controller.HomeController.handleRequestInternal(HomeController.java:103)
   at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153)
   at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
   at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:875)
   at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:807)
   at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:571)
   at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:501)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
   at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:487)
   at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1093)
   at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:265)
   at org.acegisecurity.intercept.web.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:107)
   at org.acegisecurity.intercept.web.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:72)
   at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
   at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:166)
   at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
   at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
   at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
   at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
   at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
   at org.acegisecurity.wrapper.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:81)
   at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
   at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:173)
   at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
   at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
   at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
   at org.acegisecurity.ui.logout.LogoutFilter.doFilter(LogoutFilter.java:110)
   at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
   at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
   at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
   at org.acegisecurity.util.FilterChainProxy.doFilter(FilterChainProxy.java:149)
   at org.acegisecurity.util.FilterToBeanProxy.doFilter(FilterToBeanProxy.java:98)
   at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1084)
   at net.sourceforge.subsonic.filter.ResponseHeaderFilter.doFilter(ResponseHeaderFilter.java:47)
   at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1084)
   at net.sourceforge.subsonic.filter.RequestEncodingFilter.doFilter(RequestEncodingFilter.java:43)
   at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1084)
   at net.sourceforge.subsonic.filter.ParameterDecodingFilter.doFilter(ParameterDecodingFilter.java:54)
   at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1084)
   at net.sourceforge.subsonic.filter.BootstrapVerificationFilter.doFilter(BootstrapVerificationFilter.java:59)
   at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1084)
   at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:360)
   at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
   at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
   at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712)
   at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
   at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)
   at org.mortbay.jetty.Server.handle(Server.java:313)
   at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:506)
   at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:830)
   at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:514)
   at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:211)
   at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:381)
   at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:396)
   at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)
Caused by: java.sql.SQLException: out of memory
   at org.hsqldb.jdbc.Util.throwError(Unknown Source)
   at org.hsqldb.jdbc.jdbcPreparedStatement.executeQuery(Unknown Source)
   at org.springframework.jdbc.core.JdbcTemplate$1.doInPreparedStatement(JdbcTemplate.java:648)
   at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:591)
   ... 65 more


[ajkessel]

Update - I switched back to 6.0b2 and have the same problem, so maybe the upgrade is just coincidental. Now I can't get Subsonic to start at all, although CPU usage for the process is skyrocketing. subsonic_sh.log shows continual "full GC" updates, followed by

Code: Select all

Exception in thread "Timer-0" java.lang.OutOfMemoryError: Java heap space
        at java.util.HashMap.newValueIterator(HashMap.java:971)
        at java.util.HashMap$Values.iterator(HashMap.java:1038)
        at org.mortbay.jetty.servlet.HashSessionManager.scavenge(HashSessionManager.java:180)
        at org.mortbay.jetty.servlet.HashSessionManager.access$000(HashSessionManager.java:36)
        at org.mortbay.jetty.servlet.HashSessionManager$1.run(HashSessionManager.java:144)
        at java.util.TimerThread.mainLoop(Timer.java:555)
        at java.util.TimerThread.run(Timer.java:505)
3076665 [main] INFO org.springframework.beans.factory.xml.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [org/springframework/jdbc/support/sql-error-codes.xml]
3076837 [main] INFO org.springframework.jdbc.support.SQLErrorCodesFactory - SQLErrorCodes loaded: [DB2, Derby, H2, HSQL, Informix, MS-SQL, MySQL, Oracle, PostgreSQL, Sybase]


Any ideas for how to get my server working again?

[sfgreenwood]

I have a similar problem without the memory failure: the server is responding very slowly and continually running a full GC check. I removed 6 beta 2 and installed 6.0 from clean. I have two folders, one with MP3s and once with a large number of flacs. I scanned the MP3 folder first and all seemed fine, then reintroduced the flacs.

At --max-memory=150 it was looping every minute or so, but at --max-memory=500 it seems to be processing with a maximum of 280M but stuck like this:

Code: Select all

[GC 209479K->106543K(280576K), 0.0360570 secs]
[GC 217135K->101463K(281600K), 0.0279580 secs]
[GC 212055K->96511K(281600K), 0.0730180 secs]
[GC 208127K->105295K(281600K), 0.0372470 secs]
[GC 216911K->100967K(281088K), 0.0361130 secs]
[GC 213095K->96839K(281600K), 0.0306130 secs]
[GC 208967K->105839K(281088K), 0.0362560 secs]

The java process is taking 75% of CPU but load isn't high (HP Microserver with 8GB RAM and AMD Turion(tm) II Neo N40L Dual-Core Processor).

[sfgreenwood]

For information I have reverted to a clean 5.3 install and it seems to be handling both folders well with no issues.

[ajkessel]

I was able to fix my own problem by wiping the database and starting over. Never figured it out otherwise.